Brevet/dev/tools/nextcloud-S3-migration

nextcloud local S3 migration

Script for migrating Nextcloud primary storage from local to S3 storage :floppy_disk: to :cloud:

Fork from mrAceT

Revision main - 4ab29ef

local to S3

It will transfer files from local based primary storage to a S3 primary storage.

Before you start, it is probably wise to set $DO_FILES_CLEAN (occ files:cleanup) and $DO_FILES_SCAN (occ files:scan --all) to '1' once, let the 'Nextcloud' do some checking.. then you'll start out as clean as possible

• make sure that the Nextcloud cron job is disabled and make sure that you local data is sufficiently large
• go to the nextcloud root folder and install composer

wget https://getcomposer.org/installer
mv installer composer-setup.php
mkdir bin
php composer-setup.php --install-dir /var/www/html/nextcloud/bin
chown -R apache:apache bin
rm -f composer-setup.php
chmod 755 bin/*

• install 'aws/aws-sdk-php'

runuser -u apache -- ./bin/composer.phar require aws/aws-sdk-php

• place 'storage.config.php', 'localtos3.php' and 'tests3.php' in the nextcloud root folder (and set your S3 credentials!)

• set & check all the config variables in the beginning of the scripts!

• check your S3 config

sudo -u apache php81 -d memory_limit=1024M tests3.php

• start with the highest $TEST => 2 (complete dry run, just checks en dummy uploads etc. NO database changes what so ever!)

sudo -u apache php81 -d memory_limit=1024M localtos3.php

• start the migration :

DO NOT skip ahead and go live ($TEST=0) as the first step.. then your downtime will be very long!

1. set $TEST to a 'small test user", upload the data to S3 for only that user (NO database changes what so ever!)
2. set $TEST to 1 and run the script yet again, upload (and check) all the data to S3 (NO database changes what so ever!)
3. set $TEST to 0 and run the script again (this is LIVE, nextcloud will be set into maintenance:mode --on while working ! database changes!)

With performing 'the move' at second step you will decrease the downtime (with maintenance mode:on) immensely! This because the script will first check if it already has uploaded the latest file, then it can skip to the next and does not need to (slowly) upload it to your S3 bucket! With a litte luck the final run (with $TEST=0) can be done within a minute!

NOTE second step will take a very long time when you have a lot of data to upload!

If everything worked you might want to delete the data in data folder. Also you probably want to delete this script (and the 'storage.config.php') after running it. If all went as it should the config data in 'storage.config.php' is included in the 'config/config.php'. Then the 'storage.config.php' can also be removed from your config folder (no sense in having a double config)

S3 sanity check!

When you

1. have S3 as your primary storage
2. set $TEST to 0
3. optionally set $SET_MAINTENANCE to 0
4. (have set/checked all the other variables..)

Then the script 'localtos3.php' will:

• look for entries in S3 and not in the database and vice versa and remove them. This can happen sometimes upon removing an account, preview files might not get removed.. stuff like that..

• check for canceled uploads. Inspired upon otherguy/nextcloud-cleanup. I have not had this problem, so can not test.. => check only!

• preview cleanup. Removes previews of files that no longer exist. There is some initial work for clearing previews.. that is a work in progress, use at your own risc!

The script will do the "sanity check" when migrating also (we want a good and clean migrition, won't we? ;)

Output example

sudo -u apache php81 -d memory_limit=2048M localtos3.php

#########################################################################################
 Migration tool for Nextcloud local to S3 version 0.35

 Reading config...

#########################################################################################
Setting up local migration to S3 (sync)...

first load the nextcloud config...
S3 NOT configured in config.php, using $CONFIG_OBJECTSTORE
connect to sql-database...
FOUND 'local::/data', good. 
The local store  id is:1
NOTE: No 'object::store:' > S3 storage  = defined


 Upon migration local will be renamed to object::store

######################################################################################### 0

NOTE: THIS IS THE REAL THING!!

Base init complete, continue?

######################################################################################### 
database backup...
(to restore: mysql -u nextcloudb -p nextcloudb < backup.sql)

backup config.php...not needed
connect to S3...

#########################################################################################
Setting everything up finished ##########################################################

#########################################################################################
appdata preview size... ($PREVIEW_MAX_AGE = 0 days, stats only)
appdata preview size before :  354.54 Mb	(974 files)
appdata preview > 1 year old:    0.00 Mb	(0 files)
appdata preview size cleared:    0.00 Mb	(0 files)
appdata preview size cleared:    0.00 Mb	(0 files)
appdata preview size now    :  354.54 Mb	(974 files/ -0%)

#########################################################################################
read files in S3............................................................
Objects to process in S3: 251548  DONE                                                    
objects removed from  S3: 0	(0 bytes)
objects updated to    S3: 0	(0 bytes)
objects skipped on    S3: 0	(0 bytes)
objects in sync on    S3: 251548	(475.64 Gb)

#########################################################################################
check files in oc_filecache... 
Number of objects in oc_filecache: 251548  DONE                                     
Files in oc_filecache added to S3: 0	(0 bytes)
Copying files finished

#########################################################################################
check for canceled uploads in oc_filecache...
=> EXPERIMENTAL, I have not had this problem, so can not test.. => check only!

#########################################################################################
oc_storages altered (1929/1)

#########################################################################################
NOTE: you can remove the user folder of appdata_ocadc83b19e7	by: rm -rf /data/appdata_ocadc83b19e7
NOTE: you can remove the user folder of pedro	by: rm -rf /data/pedro
NOTE: you can remove the user folder of 14a414cc-89a0-1037-9269-0fd2eda7b977	by: rm -rf /data/14a414cc-89a0-1037-9269-0fd2eda7b977
NOTE: you can remove the user folder of admin_dfr	by: rm -rf /data/admin_dfr
NOTE: you can remove the user folder of admin_tkg	by: rm -rf /data/admin_tkg
NOTE: you can remove the user folder of vdbroecke	by: rm -rf /data/vdbroecke

Copy storage.config.php to the config folder...1
#########################################################################################

ALL DONE!

Log into your Nextcloud instance and check!

If all looks well: do not forget to remove '/config/storage.config.php' (it should be
                   included in your config: having double config data is a risk..)
If it's not OK   : set your instance in 'maintenance:mode --on' & restore your SQL backup
                   you'll be back to 'local' (let me know, via GitHub, I'll try to help)
#########################################################################################