Welcome to Centmin Mod Community
Become a Member

Auto Database Backup Script 1.0

Export and Upload your Database to Offsite location with Cron

  1. eva2000

    eva2000 Administrator Staff Member

    53,248
    12,117
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,655
    Local Time:
    12:38 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    yeah you can change the port in upload.pl if you need to

    cswaits = context switch waits http://en.wikipedia.org/wiki/Context_switch and http://www.linfo.org/context_switch.html


     
  2. pamamolf

    pamamolf Premium Member Premium Member

    4,068
    427
    83
    May 31, 2014
    Ratings:
    +832
    Local Time:
    5:38 PM
    Nginx-1.25.x
    MariaDB 10.3.x
    Is there an option for pigz to let it now how many cores can use?

    For example if i have 8 unused cores to give to pigz 6 of them so it will not use all cores and kill server performance?

    As i can see on the net server loads up to 700% in some cases.

    Code:
    pigz -4 -p 6 backup_070712.sql
    Is this syntax ok?

    Thanks
     
  3. eva2000

    eva2000 Administrator Staff Member

    53,248
    12,117
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,655
    Local Time:
    12:38 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  4. pamamolf

    pamamolf Premium Member Premium Member

    4,068
    427
    83
    May 31, 2014
    Ratings:
    +832
    Local Time:
    5:38 PM
    Nginx-1.25.x
    MariaDB 10.3.x
    Yes i found it from the help option but i was not sure if i must use it as:

    Code:
    pigz -4 -p 6 backup_070712.sql
    or

    Code:
    pigz -4p 6 backup_070712.sql

    Also:

    (default is the number of online processors)

    does that means total cores and threads?
     
  5. eva2000

    eva2000 Administrator Staff Member

    53,248
    12,117
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,655
    Local Time:
    12:38 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    for pigz -p is number of physical cores only (not including virtual / hyperthreaded threads). So for a quad core Xeon E3-1230 would auto use 4 physical cpu cores.
    Code:
    pigz -4R -p6 backup.sql
     
  6. pamamolf

    pamamolf Premium Member Premium Member

    4,068
    427
    83
    May 31, 2014
    Ratings:
    +832
    Local Time:
    5:38 PM
    Nginx-1.25.x
    MariaDB 10.3.x
    Ok thanks i thought that was counting there also the threads :)

    How can i add a path here:

    Code:
    pigz -4 backup_070712.sql
     
  7. eva2000

    eva2000 Administrator Staff Member

    53,248
    12,117
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,655
    Local Time:
    12:38 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    not sure I understand, if you meant path to sql

    Code:
    pigz -4R -p4 /path/to/backup.sql
     
  8. pamamolf

    pamamolf Premium Member Premium Member

    4,068
    427
    83
    May 31, 2014
    Ratings:
    +832
    Local Time:
    5:38 PM
    Nginx-1.25.x
    MariaDB 10.3.x
    Yes this is what i want :)

    Thanks !
     
    Last edited: Jun 24, 2014
  9. eva2000

    eva2000 Administrator Staff Member

    53,248
    12,117
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,655
    Local Time:
    12:38 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    it will overwrite the sql file that you specified but if you have sql file with timestamp name, it won't
     
  10. rdan

    rdan Well-Known Member

    5,439
    1,398
    113
    May 25, 2014
    Ratings:
    +2,187
    Local Time:
    10:38 PM
    Mainline
    10.2
    Is it possible to use sftp instead of ftp?
     
  11. rdan

    rdan Well-Known Member

    5,439
    1,398
    113
    May 25, 2014
    Ratings:
    +2,187
    Local Time:
    10:38 PM
    Mainline
    10.2
    Noob question, How can I verify if I have pigz or pbzip2 installed on the server? :)
     
  12. eva2000

    eva2000 Administrator Staff Member

    53,248
    12,117
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,655
    Local Time:
    12:38 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    type

    Code:
    which pigz
    which pbzip2
    pigz -h
    pbzip2 -h
    Centmin Mod doesn't install them by default, you need to run centmin.sh menu option 17 ;) See https://community.centminmod.com/th...ssion-tools-pigz-pbzip2-lbzip2-plzip-p7zip.92

    Code:
    --------------------------------------------------------
    Centmin Mod 1.2.3-eva2000.07 - http://centminmod.com
    --------------------------------------------------------
                       Centmin Mod Menu                
    --------------------------------------------------------
    1).  Centmin Install
    2).  Add Nginx vhost domain
    3).  NSD setup domain name DNS
    4).  Nginx Upgrade / Downgrade
    5).  PHP Upgrade / Downgrade
    6).  XCache Re-install
    7).  APC Cache Re-install
    8).  XCache Install
    9).  APC Cache Install
    10). Memcached Server Re-install
    11). MariaDB 5.2, 5.5, 10 Upgrade Sub-Menu
    12). Zend OpCache Install/Re-install
    13). Install ioping.sh vbtechsupport.com/1239/
    14). SELinux disable
    15). Install/Re-install ImageMagick PHP Extension
    16). Change SSHD Port Number
    17). Multi-thread compression: pigz,pbzip2,lbzip2,p7zip etc
    18). Suhosin PHP Extension install
    19). Install FFMPEG and FFMPEG PHP Extension
    20). NSD Re-install
    21). Update - Nginx + PHP-FPM + Siege
    22). Exit
    --------------------------------------------------------
    Enter option [ 1 - 22 ]
    --------------------------------------------------------
    Code:
    pigz -h
    Usage: pigz [options] [files ...]
      will compress files in place, adding the suffix '.gz'.  If no files are
      specified, stdin will be compressed to stdout.  pigz does what gzip does,
      but spreads the work over multiple processors and cores when compressing.
    
    Options:
      -0 to -9, -11        Compression level (11 is much slower, a few % better)
      --fast, --best       Compression levels 1 and 9 respectively
      -b, --blocksize mmm  Set compression block size to mmmK (default 128K)
      -c, --stdout         Write all processed output to stdout (won't delete)
      -d, --decompress     Decompress the compressed input
      -f, --force          Force overwrite, compress .gz, links, and to terminal
      -F  --first          Do iterations first, before block split for -11
      -h, --help           Display a help screen and quit
      -i, --independent    Compress blocks independently for damage recovery
      -I, --iterations n   Number of iterations for -11 optimization
      -k, --keep           Do not delete original file after processing
      -K, --zip            Compress to PKWare zip (.zip) single entry format
      -l, --list           List the contents of the compressed input
      -L, --license        Display the pigz license and quit
      -M, --maxsplits n    Maximum number of split blocks for -11
      -n, --no-name        Do not store or restore file name in/from header
      -N, --name           Store/restore file name and mod time in/from header
      -O  --oneblock       Do not split into smaller blocks for -11
      -p, --processes n    Allow up to n compression threads (default is the
                           number of online processors, or 8 if unknown)
      -q, --quiet          Print no messages, even on error
      -r, --recursive      Process the contents of all subdirectories
      -R, --rsyncable      Input-determined block locations for rsync
      -S, --suffix .sss    Use suffix .sss instead of .gz (for compression)
      -t, --test           Test the integrity of the compressed input
      -T, --no-time        Do not store or restore mod time in/from header
      -v, --verbose        Provide more verbose output
      -V  --version        Show the version of pigz
      -z, --zlib           Compress to zlib (.zz) instead of gzip format
      --                   All arguments after "--" are treated as files
    
    Code:
    pbzip2 -h
    Parallel BZIP2 v1.1.8 - by: Jeff Gilchrist [http://compression.ca]
    [Jun. 10, 2012]               (uses libbzip2 by Julian Seward)
    Major contributions: Yavor Nikolov <nikolov.javor+pbzip2@gmail.com>
    
    Usage: pbzip2 [-1 .. -9] [-b#cdfhklm#p#qrS#tVz] <filename> <filename2> <filenameN>
    -1 .. -9        set BWT block size to 100k .. 900k (default 900k)
    -b#             Block size in 100k steps (default 9 = 900k)
    -c,--stdout     Output to standard out (stdout)
    -d,--decompress Decompress file
    -f,--force      Overwrite existing output file
    -h,--help       Print this help message
    -k,--keep       Keep input file, don't delete
    -l,--loadavg    Load average determines max number processors to use
    -m#             Maximum memory usage in 1MB steps (default 100 = 100MB)
    -p#             Number of processors to use (default: autodetect [4])
    -q,--quiet      Quiet mode (default)
    -r,--read       Read entire input file into RAM and split between processors
    -S#             Child thread stack size in 1KB steps (default stack size if unspecified)
    -t,--test       Test compressed file integrity
    -v,--verbose    Verbose mode
    -V,--version    Display version info for pbzip2 then exit
    -z,--compress   Compress file (default)
    --ignore-trailing-garbage=# Ignore trailing garbage flag (1 - ignored; 0 - forbidden)
    
    If no file names are given, pbzip2 compresses or decompresses from standard input to standard output.
    
    Example: pbzip2 -b15vk myfile.tar
    Example: pbzip2 -p4 -r -5 myfile.tar second*.txt
    Example: tar cf myfile.tar.bz2 --use-compress-prog=pbzip2 dir_to_compress/
    Example: pbzip2 -d -m500 myfile.tar.bz2
    Example: pbzip2 -dc myfile.tar.bz2 | tar x
    Example: pbzip2 -c < myfile.txt > myfile.txt.bz2 
     
  13. rdan

    rdan Well-Known Member

    5,439
    1,398
    113
    May 25, 2014
    Ratings:
    +2,187
    Local Time:
    10:38 PM
    Mainline
    10.2
    I have them installed. Thanks!

    So from your experience what is the most in compression?
    I mean with the very small size output?
    I need the most small size archive to upload it to my dropbox :)
     
  14. eva2000

    eva2000 Administrator Staff Member

    53,248
    12,117
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,655
    Local Time:
    12:38 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  15. Omer

    Omer New Member

    10
    8
    3
    Sep 11, 2014
    Ratings:
    +8
    Local Time:
    5:38 PM
    Nginx 1.7.5
    MariaDB 5.5
  16. rdan

    rdan Well-Known Member

    5,439
    1,398
    113
    May 25, 2014
    Ratings:
    +2,187
    Local Time:
    10:38 PM
    Mainline
    10.2
  17. BobbyWibowo

    BobbyWibowo Active Member

    197
    42
    28
    Jul 30, 2015
    Indonesia
    Ratings:
    +71
    Local Time:
    9:38 PM
    1.17.x
    10.3.x
    I started using this code as the base for my database backup script. Added some array thingy to backup multiple databases, switched to pigz, and email notification as well. But it seems people do backup their site's root directory as well huh? I didn't setup the script to host database backup anywhere since I'm using Linode daily backup as well. I was just slightly concerned about the fact that they suggested to dump database before their backup system kicks in, for safety measure due to failure in saving the database state, or something.
    So, yeah, that's that. And my question is, should I also backup my site's root directory? I mean, since Linode's backup should backup everything anyway. And also, what's the actual risk of not doing it? Thanks in advance.
     
  18. eva2000

    eva2000 Administrator Staff Member

    53,248
    12,117
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,655
    Local Time:
    12:38 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    probably best to start own backup script thread in system admin forums System Administration | Centmin Mod Community there's other backup script discussions in that forum too :)
     
  19. bojan horvatic

    bojan horvatic New Member

    6
    1
    3
    Feb 6, 2016
    Ratings:
    +1
    Local Time:
    4:38 PM
    Hi :)

    OK, so this script is exactly what I need, but since I'm a noob, I ran into a problem and need someone to help me out.

    CentOS 7
    OVH dedicated server
    Centminmod beta

    After figuring out permission thing with chmod 700 I bumped into this problem:
    Code:
    Can't locate Date/Manip.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/l                                     ocal/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/                                     lib64/perl5 /usr/share/perl5 .) at /home/nginx/domains/eliterehabplacement.com/b                                     ackup/upload.pl line 6.
    BEGIN failed--compilation aborted at /home/nginx/domains/eliterehabplacement.com                                     /backup/upload.pl line 6.
    Thanks!
     
  20. eva2000

    eva2000 Administrator Staff Member

    53,248
    12,117
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,655
    Local Time:
    12:38 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    @Matt's script is in perl and seems your system is missing a perl yum package

    using yum whatprovides command see which yum package provides */Manip.pm
    Code (Text):
    yum whatprovides */Manip.pm -q
    perl-Date-Manip-6.41-2.el7.noarch : Date manipulation routines
    Repo        : base
    Matched from:
    Filename    : /usr/share/perl5/vendor_perl/Date/Manip.pm

    looks to be perl-Date-Manip so install via yum
    Code (Text):
    yum install perl-Date-Manip