Learn about Centmin Mod LEMP Stack today
Register Now

Restore from Backup Scenario - transfer to new server

Discussion in 'Centmin Mod Insights' started by dorobo, Dec 19, 2015.

  1. dorobo

    dorobo Active Member

    420
    104
    43
    Jun 6, 2014
    Ratings:
    +161
    Local Time:
    6:18 AM
    latest
    latest
    Supposed your server crashed and you have a number of domains that you need to be restored from backup and you need to restore it as fast as possible.

    Would this work?

    You install centmin on the new server and then

    rsync your backups of

    /usr/local/nginx/conf/conf.d
    /home/nginx/domains

    from backup server to new server

    $ chown nginx.nginx -R /home/nginx/domains

    restore databases

    change IP of the domains

    anything else?
     
  2. eva2000

    eva2000 Administrator Staff Member

    30,156
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    8:18 AM
    Nginx 1.13.x
    MariaDB 5.5
  3. RoldanLT

    RoldanLT Well-Known Member

    3,901
    949
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,298
    Local Time:
    6:18 AM
    1.11
    10.2
    Yes, this is the only important directory to sync.

    And also, if you have huge amount of data, then provision the closest VPS location so the transfer is also quick.

    I always have this command on crontab :D.
     
    • Informative Informative x 2
  4. eva2000

    eva2000 Administrator Staff Member

    30,156
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    8:18 AM
    Nginx 1.13.x
    MariaDB 5.5
    FYI, I do have future plans for Centmin Mod transfer data script similar to WHM to WHM transfer feature ;)

    For now some tips that you may want to test (no guarantees it works 100% and no support essentially), so @RoldanLT @Matt @Matt Williams @pamamolf @dorobo and other folks may want to test this on test VPS/data first

    Requirements
    • Needs to use ssh key (private/public key) setup and connection from source and destination servers
    All commands in SSH as root user

    Setup variables for remote destination server ip address, port and your ssh private key name and backup directory on local source server
    Code (Text):
    remoteip=111.222.333.444
    port=22
    privatekey=private.key
    backupdir=/home/backupsql
    DUMPOPT='-Q -K --max_allowed_packet=256M --net_buffer_length=65536 --routines --events --triggers --hex-blob'
    CLIENTOPT='--max_allowed_packet=256M --net_buffer_length=65536'
    mkdir -p $backupdir
    


    rsync relevant data which assumes on remote destination server, centmin mod is installed but no vhosts or mysql databases have been created. It also assumes /root/.my.cnf is setup for mysql root user password which is automatically done on initial Centmin Mod installs

    file data with rsync options -avzin where n is for dry run so no data is actually transferred. If all the dry runs look good, remove n to become -avzi for the below commands

    dry run with 'n'
    Code (Text):
    cd /home
    rsync -avzin --progress --stats -e "ssh -p $port -i $HOME/.ssh/$privatekey" --exclude 'domains/*/log' --exclude 'domains/*/backup' /home/nginx/ root@$remoteip:/home/nginx
    rsync -avzin --progress --stats -e "ssh -p $port -i $HOME/.ssh/$privatekey" /usr/local/nginx/conf/ root@$remoteip:/usr/local/nginx/conf
    rsync -avzin --progress --stats -e "ssh -p $port -i $HOME/.ssh/$privatekey" /etc/centminmod/custom_config.inc root@$remoteip:/etc/centminmod/custom_config.inc
    rsync -avzin --progress --stats -e "ssh -p $port -i $HOME/.ssh/$privatekey" /usr/local/etc/php-fpm.conf root@$remoteip:/usr/local/etc/php-fpm.conf
    rsync -avzin --progress --stats -e "ssh -p $port -i $HOME/.ssh/$privatekey" /etc/my.cnf root@$remoteip:/etc/my.cnf
    rsync -avzin --progress --stats -e "ssh -p $port -i $HOME/.ssh/$privatekey" /etc/csf/csf.conf root@$remoteip:/etc/csf/csf.conf
    rsync -avzin --progress --stats -e "ssh -p $port -i $HOME/.ssh/$privatekey" /etc/init.d/memcached root@$remoteip:/etc/init.d/memcached

    live run without 'n'
    Code (Text):
    cd /home
    rsync -avzi --progress --stats -e "ssh -p $port -i $HOME/.ssh/$privatekey" --exclude 'domains/*/log' --exclude 'domains/*/backup' /home/nginx/ root@$remoteip:/home/nginx
    rsync -avzi --progress --stats -e "ssh -p $port -i $HOME/.ssh/$privatekey" /usr/local/nginx/conf/ root@$remoteip:/usr/local/nginx/conf
    rsync -avzi --progress --stats -e "ssh -p $port -i $HOME/.ssh/$privatekey" /etc/centminmod/custom_config.inc root@$remoteip:/etc/centminmod/custom_config.inc
    rsync -avzi --progress --stats -e "ssh -p $port -i $HOME/.ssh/$privatekey" /usr/local/etc/php-fpm.conf root@$remoteip:/usr/local/etc/php-fpm.conf
    rsync -avzi --progress --stats -e "ssh -p $port -i $HOME/.ssh/$privatekey" /etc/my.cnf root@$remoteip:/etc/my.cnf
    rsync -avzi --progress --stats -e "ssh -p $port -i $HOME/.ssh/$privatekey" /etc/csf/csf.conf root@$remoteip:/etc/csf/csf.conf
    rsync -avzi --progress --stats -e "ssh -p $port -i $HOME/.ssh/$privatekey" /etc/init.d/memcached root@$remoteip:/etc/init.d/memcached


    To transfer individual database name setup DBNAME variable with mysql database name, remote login to create database name and then mysqldump + pipe to remote db server the data
    Code (Text):
    DBNAME='dbname'
    ssh root@$remoteip -p $port -i ~/.ssh/$privatekey mysqladmin create $DBNAME
    mysqldump $DUMPOPT $DBNAME | ssh root@$remoteip -p $port -i ~/.ssh/$privatekey mysql $CLIENTOPT $DBNAME
    

    list of database names, same as above just space separated list of mysql database names
    Code (Text):
    DBNAME='dbname dbname2 dbname3'
    for db in $DBNAME; do ssh root@$remoteip -p $port -i ~/.ssh/$privatekey mysqladmin create $db; mysqldump $DUMPOPT $db | ssh root@$remoteip -p $port -i ~/.ssh/$privatekey mysql $CLIENTOPT $db; done
    

    Note, this won't transfer custom created mysql usernames and passwords created on old server. You can easily backup mysql grants for this

    this command outputs the list of db user grant statements which need replaying on destination mysql server (import into mysql system database)
    Code (Text):
    mysql --batch --skip-column-names -e "SELECT user, host FROM user" mysql | while read user host; do if [[ "$user" != 'root' ]]; then mysql --batch --skip-column-names -e "SHOW GRANTS FOR '$user'@'$host'" | sed -e 's|$|;|'; fi; done


    Code (Text):
    mysql --batch --skip-column-names -e "SELECT user, host FROM user" mysql | while read user host; do if [[ "$user" != 'root' ]]; then mysql --batch --skip-column-names -e "SHOW GRANTS FOR '$user'@'$host'" | sed -e 's|$|;|'; fi; done
    
    GRANT USAGE ON *.* TO 'dbuser1'@'localhost' IDENTIFIED BY PASSWORD '*196BDEDE2AE4F84CA44C47D54D78478C7E2BD7B7';
    GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE ON `dbname2`.* TO 'dbuser1'@'localhost';
    

    If anyone has something to add or if I missed anything, let me know :)
     
    Last edited: Nov 14, 2016
    • Like Like x 3
    • Informative Informative x 2
  5. pamamolf

    pamamolf Well-Known Member

    2,721
    242
    63
    May 31, 2014
    Ratings:
    +433
    Local Time:
    1:18 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    Santa is coming with something cool :)
     
    • Like Like x 1
  6. dorobo

    dorobo Active Member

    420
    104
    43
    Jun 6, 2014
    Ratings:
    +161
    Local Time:
    6:18 AM
    latest
    latest
    What is the $HOME variable and why was it not defined in the first code?
     
  7. eva2000

    eva2000 Administrator Staff Member

    30,156
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    8:18 AM
    Nginx 1.13.x
    MariaDB 5.5
    $HOME is system environment variable for the user you're logged in with

    for root user = /root

    for any linux system user you create or use it will be /home/username

    you can find the system variable via 2 commands
    Code:
    echo $HOME
    or
    Code:
    env
     
    • Informative Informative x 2
  8. dorobo

    dorobo Active Member

    420
    104
    43
    Jun 6, 2014
    Ratings:
    +161
    Local Time:
    6:18 AM
    latest
    latest
    Thanks George.

    I will test this and report back.
     
  9. dorobo

    dorobo Active Member

    420
    104
    43
    Jun 6, 2014
    Ratings:
    +161
    Local Time:
    6:18 AM
    latest
    latest
    are those commands still on the new server?

    and does this command

    ssh root@$remoteip -p $port -i ~/.ssh/$privatekey mysqladmin create $DBNAME

    mean that it will create the database name on the old server?
     
  10. eva2000

    eva2000 Administrator Staff Member

    30,156
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    8:18 AM
    Nginx 1.13.x
    MariaDB 5.5
    source server :) and yup remotely connect to create new database name
     
  11. dorobo

    dorobo Active Member

    420
    104
    43
    Jun 6, 2014
    Ratings:
    +161
    Local Time:
    6:18 AM
    latest
    latest
    I tested this and it copied the files into

    /home/nginx/nginx/domains

    instead of

    /home/nginx/domains
     
  12. dorobo

    dorobo Active Member

    420
    104
    43
    Jun 6, 2014
    Ratings:
    +161
    Local Time:
    6:18 AM
    latest
    latest
    and also

    /usr/local/nginx/conf/conf/conf.d

    /usr/local/nginx/conf/conf/

    I put the commands in a bash script

    /root/test2.sh

    another thing is since the log files were excluded (supposed the files were copied on the right folders) wouldn't that result in an nginx error when you restart since the log files are not there?
     
  13. eva2000

    eva2000 Administrator Staff Member

    30,156
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    8:18 AM
    Nginx 1.13.x
    MariaDB 5.5
    missing forward slashes on source parent directories doh
    Code:
    cd /home
    rsync -avzin --progress --stats -e "ssh -p $port -i $HOME/.ssh/$privatekey" --exclude 'nginx/domains/*/log' --exclude 'nginx/domains/*/backup' /home/nginx/ root@$remoteip:/home/nginx
    rsync -avzin --progress --stats -e "ssh -p $port -i $HOME/.ssh/$privatekey" /usr/local/nginx/conf/ root@$remoteip:/usr/local/nginx/conf
     
  14. raciasolvo

    raciasolvo Member

    98
    10
    8
    Oct 7, 2016
    Ratings:
    +27
    Local Time:
    1:18 AM
    Nginx 1.11.6
    MariaDB 10.0.27
    May be add --exclude 'nginx/domains/*/public/wp-content/cache'?
     
  15. eva2000

    eva2000 Administrator Staff Member

    30,156
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    8:18 AM
    Nginx 1.13.x
    MariaDB 5.5
    it should get cleared anyway later on so doesn't matter really
     
  16. dorobo

    dorobo Active Member

    420
    104
    43
    Jun 6, 2014
    Ratings:
    +161
    Local Time:
    6:18 AM
    latest
    latest
    the log files are still copied with this
     
    • Informative Informative x 1
  17. eva2000

    eva2000 Administrator Staff Member

    30,156
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    8:18 AM
    Nginx 1.13.x
    MariaDB 5.5
    try instead of
    Code (Text):
    --exclude 'nginx/domains/*/log' --exclude 'nginx/domains/*/backup'
    

    use
    Code (Text):
    --exclude 'domains/*/log' --exclude 'domains/*/backup'
    

    it's relative to /home/nginx/
     
    • Like Like x 1
  18. dorobo

    dorobo Active Member

    420
    104
    43
    Jun 6, 2014
    Ratings:
    +161
    Local Time:
    6:18 AM
    latest
    latest
    it worked!

    Thanks George.
     
    • Like Like x 1
  19. eva2000

    eva2000 Administrator Staff Member

    30,156
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    8:18 AM
    Nginx 1.13.x
    MariaDB 5.5