Join the community today
Become a Member

Sysadmin VPS crashed with hardware failures, migrated over to new VPS node and mysql consuming xxx%v cpu

Discussion in 'System Administration' started by Dannymh, Mar 18, 2019.

  1. Dannymh

    Dannymh New Member

    12
    3
    3
    Oct 23, 2014
    Ratings:
    +3
    Local Time:
    7:52 PM
    1.7.x
    5.5
    Hi,

    My SSD and RAID device failed on my VPS node and had to be mrigrated to a new instance on another node. Ever since the performance of my website has been woeful.
    SQL is taking up 300% of the CPU or more. Below is is an
    Code:
    htop
    from straight after a reboot

    Screen Shot 2019-03-18 at 7.18.07 pm.png

    THere is nothing stuck or delayed in when I run SHOW PROCESSLIST; in mysql.

    If I try to run dynimizer on the server to try and dynimize the mysql and cpu usage, it just locks out the server until it needs to be rebooted.

    On top of this its running super slow on the fromnt end.

    On my site I simply run xenforo forums with about 1m posts and about 70k threads roughly

    Dan
     
  2. eva2000

    eva2000 Administrator Staff Member

    44,785
    10,212
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,828
    Local Time:
    3:52 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    ??? Is it Dynimizer: CPU performance virtualization ? if it is how long have you been running this product ? try uninstalling it and reverting to centmin mod setup server config/mysql settings which are dynamically configured based on your server environment by default.

    The problem could be dynimize itself i.e. Dynimizer User Reference i.e. it requires at least 4GB of virtual memory while your htop stats show you have less than 1GB with very small 19MB swap disk.
    If running Centmin Mod 123.09beta01 or higher, you will have access to a cminfo top command which can also provide a summary of statistics - some of which are explained in info and linked articles below.

    FYI, CPU load optimal is 1.0 per cpu thread so 8 cpu thread normal = 8x1 = 8.0. 16 cpu server normal is 16x1 = 16.0

    so cpu load of 8 for
    • 2 cpu server = 8/2 = 4.0 load average
    • 4 cpu server = 8/4 = 2.0 load average
    • 8 cpu server = 8/8 = 1.0 load average
    When you type top command, hit number 1 to break down per cpu stats

    Centmin Mod is provided as is, so troubleshooting load issues is left to end user to do. However, there's many linux tools and scripts that can help you figure out what was causing the load issues and when.

    If you're providing info on this forum, more info might be helpful
    1. What version of Centmin Mod ? .08 stable or .09 beta ? If .09 beta when was it installed and when was last time you updated ?
    2. What's your VPS/Server hardware specifications ? cpu type ? memory available ? disk space ?
    3. Who's your web host ? Different hosts have different limits for server resource usage and some are more restrictive that others so it could just be their resource usage policy you tripped which may or may not be restrictive. If restrictive, then real solution would be finding a better web host. If not restrictive, then it's finding out what caused high cpu load.
    4. If running Centmin Mod 123.09beta01 or higher, you will have access to a cminfo top command which can also provide a summary of statistics - some of which are explained in info and linked articles below.
    Tools and commands you will want to read up on and learn for basic system admin tasks and troubleshooting.
    Notes:
    For example what's output for these commands
    Code (Text):
    top -bn1
    

    Code (Text):
    ps aufxw
    

    Code (Text):
    sar -q
    

    Code (Text):
    sar -r
    

    Code (Text):
    sar -q -f /var/log/sa/sa$(date +%d -d yesterday)
    

    Code (Text):
    sar -r -f /var/log/sa/sa$(date +%d -d yesterday)
    

    If you want to look at per process usage stats use pidstat run pidstat every 1 sec for 5 runs and sed is just to replace your main hostname with word hostname so mask your domain for posting on public places
    Code (Text):
    pidstat -durh 1 5 | sed -e "s|$(hostname)|hostname|g"
    

    for posting code you might want to use CODE tags for code How to use forum BBCODE code tags :)
     
  3. Dannymh

    Dannymh New Member

    12
    3
    3
    Oct 23, 2014
    Ratings:
    +3
    Local Time:
    7:52 PM
    1.7.x
    5.5

    I upped the swap to 5Gb and the memory is actuall 6GB not 1GB (htop is showing only 800mb being used not available)

    The server is out of control with CPU usage mainly around mysql with centminmod default setup. I am trying to get dynimizer up and running as generally that helps with server CPU usage. Looks like the system is basically not switching mysql to memory at all and instead using CPU for everything.


    Centminmod version: 123.09beta01
    VPS details: 6 Cores @ 3.5Ghz Intel CPUs
    Memory: 6GB
    SSD: 70gb
    Host: networkpresence.com.au

    Cminfo log attached
     

    Attached Files:

  4. eva2000

    eva2000 Administrator Staff Member

    44,785
    10,212
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,828
    Local Time:
    3:52 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    From that cminfo top output majority of pidstats cpu load is coming from php-fpm and not mysql process at that time - you may need to run cminfo at other times to capture more details. Your php issues most likely are due to PHP-FPM needing tuning. Read PHP-FPM - How to troubleshoot & optimize PHP-FPM server?

    Code (Text):
    ------------------------------------------------------------------
    top -bcn1 -w200
    top - 23:06:18 up 30 min,  1 user,  load average: 4.74, 3.88, 3.89
    Tasks: 131 total,   9 running, 122 sleeping,   0 stopped,   0 zombie
    %Cpu(s): 17.4 us,  7.6 sy,  0.0 ni,  3.5 id,  4.1 wa,  0.0 hi, 44.2 si, 23.3 st
    KiB Mem :  5798760 total,  3064896 free,  1288308 used,  1445556 buff/cache
    KiB Swap:  4213752 total,  4213752 free,        0 used.  4234472 avail Mem
    
      PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    14360 nginx     20   0  718504  31636  14512 R  87.0  0.5   0:36.60 php-fpm: pool www
    14361 nginx     20   0  720312  33576  14504 R  73.9  0.6   0:26.81 php-fpm: pool www
       24 root      20   0       0      0      0 S  60.9  0.0  17:09.56 [ksoftirqd/3]
    14028 nginx     20   0  718776  35312  17796 R  60.9  0.6   1:36.80 php-fpm: pool www
     3951 mysql     20   0 4759580 926104  12276 S  43.5 16.0   9:21.17 /usr/sbin/mysqld
    14808 root      20   0  161960   2204   1616 R  34.8  0.0   0:00.20 top -bcn1 -w200
    14015 nginx     20   0  718576  33840  16576 R  26.1  0.6   1:26.93 php-fpm: pool www
    14014 nginx     20   0  716640  31412  16172 S  21.7  0.5   1:34.47 php-fpm: pool www
       69 root      20   0       0      0      0 S  17.4  0.0   1:36.36 [kworker/3:1]
     4158 nginx     10 -10  169004  57404   3316 S   8.7  1.0   1:14.11 nginx: worker process
    

    Code (Text):
    #      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
     1552950387     0         3    0.00    0.96    0.00    0.96     0      0.00      0.00       0      0   0.00      0.00      0.00      0.00  ksoftirqd/0
     1552950387     0         7    0.00    0.96    0.00    0.96     0      0.00      0.00       0      0   0.00      0.00      0.00      0.00  migration/0
     1552950387     0         9    0.00    3.85    0.00    3.85     0      0.00      0.00       0      0   0.00      0.00      0.00      0.00  rcu_sched
     1552950387     0        13    0.00    0.96    0.00    0.96     1      0.00      0.00       0      0   0.00      0.00      0.00      0.00  migration/1
     1552950387     0        14    0.00    5.77    0.00    5.77     1      0.00      0.00       0      0   0.00      0.00      0.00      0.00  ksoftirqd/1
     1552950387     0        19    0.00    1.92    0.00    1.92     2      0.00      0.00       0      0   0.00      0.00      0.00      0.00  ksoftirqd/2
     1552950387     0        24    0.00   54.81    0.00   54.81     3      0.00      0.00       0      0   0.00      0.00      0.00      0.00  ksoftirqd/3
     1552950387     0        29    0.00    0.96    0.00    0.96     4      0.00      0.00       0      0   0.00      0.00      0.00      0.00  ksoftirqd/4
     1552950387     0        33    0.00    0.96    0.00    0.96     5      0.00      0.00       0      0   0.00      0.00      0.00      0.00  migration/5
     1552950387     0        34    0.00    1.92    0.00    1.92     5      0.00      0.00       0      0   0.00      0.00      0.00      0.00  ksoftirqd/5
     1552950387     0        52    0.00    0.96    0.00    0.96     2      0.00      0.00       0      0   0.00      0.00      0.00      0.00  kworker/2:1
     1552950387     0        69    0.00   23.08    0.00   23.08     3      0.00      0.00       0      0   0.00      0.00      0.00      0.00  kworker/3:1
     1552950387     0       345    0.00    0.96    0.00    0.96     1      0.00      0.00       0      0   0.00      0.00      0.00      0.00  kworker/1:2
     1552950387     0      1543    0.00    0.96    0.00    0.96     4      0.00      0.00       0      0   0.00      0.00     15.38      0.00  loop0
     1552950387     0      2409    0.96    0.96    0.00    1.92     4      0.00      0.00   90548   3208   0.06      0.00      0.00      0.00  rngd
     1552950387     0      2524    0.00    0.96    0.00    0.96     2      0.00      0.00   12184   5440   0.09      0.00      0.00      0.00  haveged
     1552950387     0      3375    0.00    0.96    0.00    0.96     4      0.00      0.00       0      0   0.00      0.00      0.00      0.00  kworker/4:2
     1552950387     0      3872    0.00    0.96    0.00    0.96     4      0.00      0.00  710256  10580   0.18      0.00      0.00      0.00  php-fpm
     1552950387   996      3951    7.69   35.58    0.00   43.27     2    266.35      0.00 4759580 925916  15.97     76.92    234.62    230.77  mysqld
     1552950387  1000      4153    0.00    2.88    0.00    2.88     5      2.88      0.00  169004  55764   0.96      0.00      0.00      0.00  nginx
     1552950387  1000      4155    1.92   13.46    0.00   15.38     0      0.00      0.00  169004  56124   0.97      0.00      0.00      0.00  nginx
     1552950387  1000      4158    0.00    0.96    0.00    0.96     0      0.00      0.00  169004  57404   0.99      0.00      0.00      0.00  nginx
     1552950387     0      4163    0.96    3.85    0.00    4.81     2      0.00      0.00  154704   5556   0.10      0.00      0.00      0.00  sshd
     1552950387  1000     14014    6.73   18.27    0.00   25.00     4      0.00      0.00  716640  31428   0.54      0.00      7.69      0.00  php-fpm
     1552950387  1000     14023   10.58   28.85    0.00   39.42     4      0.96      0.00  719088  36048   0.62      0.00      7.69      0.00  php-fpm
     1552950387  1000     14028    5.77   15.38    0.00   21.15     2      0.00      0.00  718776  35312   0.61      0.00      0.00      0.00  php-fpm
     1552950387  1000     14361    3.85    9.62    0.00   13.46     5      0.00      0.00  718264  31528   0.54      0.00      0.00      0.00  php-fpm
     1552950387  1000     14371   17.31   40.38    0.00   57.69     5      3.85      0.00  718504  31220   0.54      0.00      0.00      0.00  php-fpm
     1552950387     0     14596    0.00    0.00    0.00    0.00     0      0.00      0.00  107952    668   0.01      0.00      3.85      0.00  tee
     1552950387     0     14812    1.92   19.23    0.00   21.15     0    382.69      0.00  108324   1176   0.02      0.00      0.00      0.00  pidstat
    


    Centmin Mod is provide as is, so short of scripted related bugs or issues, any further optimisation to the web stack components - nginx, php-fpm, mariadb mysql, csf firewall etc or web app specific configurations are left to the Centmin Mod user to deal with. So I do not provide any free support for such.

    Are you getting any gateway 502 /504 timeouts ? Bad gateway 502 /504 timeouts are usually related to Nginx timing out waiting on PHP-FPM to respond as PHP-FPM is overloaded or overwhelmed with requests, so may need to tune PHP-FPM values. It also maybe due to PHP-FPM in turn being queued and backed up waiting on MariaDB MySQL server to respond - so also need to look at MySQL.

    You'll need to tune your PHP-FPM settings with php-fpm main pool config file at /usr/local/etc/php-fpm.conf (overview of config files) and this is left up to end user to do but here's a thread for starters to enable PHP-FPM status page output outlined at
    Enabling PHP-FPM status also allows setting up 3rd party PHP-FPM status metric monitoring from services like:

    Checking PHP-FPM etc logs



    You'll also need to check into your PHP-FPM, Nginx and MariaDB logs which you can find as outlined at How to troubleshoot Centmin Mod initial install issues

    Server logs include Nginx, PHP-FPM, MariaDB MySQL error logs as well as others. You can find your Centmin Mod install/menu logs at FAQ 7 and server logs at FAQ 19 at Centmin Mod FAQ (most up to date info in FAQ so always read that first). Spoiler tag below has info too but may not be up to date.

    Some of Centmin Mod's installed software will have their own access and error logs which maybe useful for diagnosing errors or give info, notes, or warning notices.

    Note: There's no support provided by me for diagnosing such errors which may occur for various reasons including misconfiguration of installed php/mysql scripts or applications.

    In SSH2 telnet you can use tail command to view the last X number of lines in the file.

    For example for viewing last 10 lines in the file for:

    For Nginx access and error logs:
    Code:
      tail -10 /usr/local/nginx/logs/access.log
      tail -10 /usr/local/nginx/logs/error.log
    
    For specific domainname.com access and error log:
    Code:
      tail -10 /home/nginx/domains/domainname.com/log/access.log
      tail -10 /home/nginx/domains/domainname.com/log/error.log
    
    For other system error logs located at /var/log:

    list /var/log files in ascending time order so the most recently modified files are at the bottom
    Code:
      ls -lhrt /var/log
    
    Code:
    total 2.7M
    -rw------- 1 root  root    0 Aug 29 15:33 tallylog
    -rw------- 1 root  root    0 Aug 29 15:33 spooler
    drwx------ 3 root  root 4.0K Aug 29 15:35 samba
    drwxr-xr-x 2 root  root 4.0K Aug 29 15:35 mail
    -rw-r--r-- 1 root  500     0 Oct  8 18:13 dmesg.old
    -rw------- 1 root  500     0 Oct  8 18:13 boot.log
    -rw-r--r-- 1 root  500     0 Oct  8 18:14 dmesg
    drwx------ 2 root  root 4.0K Oct  8 18:14 httpd
    drwxr-xr-x 2 root  root 4.0K Oct  8 19:08 php-fpm
    -rw-rw---- 1 mysql root 2.3K Oct  9 12:38 mysqld.log
    -rw------- 1 root  root 9.2K Oct 26 10:48 yum.log
    -rw------- 1 root  utmp  94K Nov  7 22:59 btmp
    drwxr-xr-x 2 root  root 4.0K Nov  8 00:00 sa
    -rw------- 1 root  root 269K Nov  8 21:39 messages
    -rw------- 1 root  root 110K Nov  8 23:08 secure
    -rw-rw-r-- 1 root  utmp  43K Nov  8 23:08 wtmp
    -rw-r--r-- 1 root  root 144K Nov  8 23:08 lastlog
    -rw------- 1 root  root  69K Nov  8 23:08 lfd.log
    -rw------- 1 root  root 332K Nov  8 23:08 maillog
    -rw------- 1 root  500  1.6M Nov  8 23:10 cron
    
    For PHP-FPM error log:
    Code:
      tail -10 /var/log/php-fpm/www-error.log
    
    and/or
    Code:
      /var/log/php-fpm/www-php.error.log
    
    For MySQL / MariaDB error log:
    Code:
      tail -10 /var/log/mysqld.log
    
    For CSF firewall LFD log:
    Code:
      tail -10 /var/log/lfd.log
    
    For Mail log:
    Code:
      tail -10 /var/log/maillog
    
    For Cron job logs:
    Code:
      tail -10 /var/log/cron
    

    How to edit php.ini and php-fpm configuration files ?



    Centmin Mod install created command short cuts outlined here to allow you to quickly edit your /usr/local/lib/php.ini file and your /usr/local/etc/php-fpm.conf file. Full list of command shortcuts below:
    • Edit php.ini = phpedit ( /usr/local/lib/php.ini )
    • Edit my.cnf = mycnf ( /etc/my.cnf )
    • Edit php-fpm.conf = fpmconf ( /usr/local/etc/php-fpm.conf )
    • Edit nginx.conf = nginxconf ( /usr/local/nginx/conf/nginx.conf )
    • Edit (nginx) virtual.conf = vhostconf - only edits /usr/local/nginx/conf/conf.d/virtual.conf not the additional vhost domain.com.conf files added later
    • Edit (nginx) php.conf = phpinc ( /usr/local/nginx/conf/php.conf )
    • Edit (nginx) drop.conf = dropinc ( /usr/local/nginx/conf/drop.conf )
    • Edit (nginx) staticfiles.conf = statfilesinc ( /usr/local/nginx/conf/staticfiles.conf )
    • nginx stop/start/restart = ngxstop/ngxstart/ngxrestart
    • php-fpm stop/start/restart = fpmstop/fpmstart/fpmrestart
    • mysql stop/start/restart = mysqlstop/mysqlstart/mysqlrestart
    • nginx + php-fpm stop/start/restart = npstop/npstart/nprestart
    • memcached stop/start/restart =memcachedstop/memcachedstart/memcachedrestart
    • csf stop/start/restart = csfstop/csfstart/csfrestart

    Troubleshooting Tools



    However, there's many linux tools and scripts that can help you figure out what was causing the load issues and when.

    Tools and commands you will want to read up on and learn for basic system admin tasks and troubleshooting.
    Notes:
    However, Centmin Mod users are free to help each other out and ask questions or give answers on this community forum. My hopes are that this community forum evolves so that more veteran long time Centmin Mod users help new Centmin Mod users out :)
     
  5. eva2000

    eva2000 Administrator Staff Member

    44,785
    10,212
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,828
    Local Time:
    3:52 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    FYi, your top stats from cminfo top output show ~23.3% cpu steal too (23.3 st) meaning load coming from neighbouring VPS tenants sharing your server and competing for cpu resources !
    Code (Text):
    top -bcn1 -w200
    top - 23:06:18 up 30 min,  1 user,  load average: 4.74, 3.88, 3.89
    Tasks: 131 total,   9 running, 122 sleeping,   0 stopped,   0 zombie
    %Cpu(s): 17.4 us,  7.6 sy,  0.0 ni,  3.5 id,  4.1 wa,  0.0 hi, 44.2 si, 23.3 st
    KiB Mem :  5798760 total,  3064896 free,  1288308 used,  1445556 buff/cache
    KiB Swap:  4213752 total,  4213752 free,        0 used.  4234472 avail Mem
    

    cpu steal time = time or percentage of cpu cycles that your VPS could of used but were not able to use due to either VPS provider's artificial VPS cpu utilisation limits and/or other VPS neighbours and/or overloaded VPS host node.

    Top stats output with 0.0 st steal time reported = zero cpu steal time
    Code (Text):
    top -bn1
    top - 07:35:18 up 18 days, 21:38,  1 user,  load average: 0.00, 0.01, 0.05
    Tasks: 168 total,   1 running, 167 sleeping,   0 stopped,   0 zombie
    %Cpu0  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu1  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu2  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu3  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu4  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu5  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu6  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu7  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    KiB Mem : 16236164 total,  1866764 free,  1687184 used, 12682216 buff/cache
    KiB Swap:  8982524 total,  8980476 free,     2048 used. 13676848 avail Mem
    

    So if you have a very high consistent cpu steal time/percentage, probably time to move your VPS server to a less loaded VPS host node or different host/server.

    You can use sar command to look at your current and past historic cpu utilisation metrics which include cpu steal time

    To view today's cpu utilisation via sar command with cpu utilisation flag -u (sed is just to mask your real server hostname for word hostname for privacy)
    Code (Text):
    sar -u | sed -e "s|$(hostname)|hostname|"
    

    example output for cpu utilisation metrics including cpu steal percentage = %steal. Centmin Mod logs sar metrics every 5 minutes which differs from default sar on CentOS and other Linux distributions which logs sar metrics every 10 minutes by default
    Code (Text):
    sar -u | sed -e "s|$(hostname)|hostname|"
    Linux 4.9.15-x86_64-linode81 (hostname)         06/19/2017      _x86_64_        (1 CPU)
    
    12:00:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
    12:05:01 AM     all      3.28      0.00      2.22      0.07      0.95     93.48
    12:10:01 AM     all      2.60      0.00      1.79      0.03      0.36     95.23
    12:15:01 AM     all      5.80      0.00      2.14      0.04      0.29     91.74
    12:20:01 AM     all      2.80      0.00      2.04      0.04      0.44     94.68
    12:25:01 AM     all      2.72      0.00      1.85      0.04      0.58     94.82
    12:30:01 AM     all      2.55      0.00      1.71      0.02      0.45     95.27
    12:35:01 AM     all      2.57      0.00      1.91      0.03      0.83     94.67
    12:40:02 AM     all      2.65      0.00      1.89      0.02      0.69     94.76
    12:45:01 AM     all      2.59      0.00      1.74      0.02      0.52     95.13
    12:50:02 AM     all      2.59      0.00      1.88      0.08      0.54     94.92
    12:55:01 AM     all      2.58      0.00      1.88      0.01      0.47     95.05
    01:00:01 AM     all      2.35      0.00      1.76      0.03      0.50     95.37
    01:05:01 AM     all      3.09      0.00      2.38      0.04      1.16     93.32
    01:10:02 AM     all      2.74      0.00      2.01      0.03      1.00     94.22
    01:15:01 AM     all      2.61      0.00      1.84      0.02      0.96     94.57
    01:20:01 AM     all      5.85      0.00      2.24      0.07      0.81     91.02
    01:25:01 AM     all      3.01      0.00      1.94      0.02      1.20     93.82
    01:30:02 AM     all      2.51      0.00      1.69      0.02      0.91     94.87
    01:35:02 AM     all      2.76      0.00      1.98      0.02      1.36     93.89
    01:40:02 AM     all      2.67      0.00      1.93      0.04      1.34     94.02
    01:45:02 AM     all      2.78      0.00      1.86      0.02      1.33     94.01
    01:50:01 AM     all      2.82      0.00      1.99      0.02      1.19     93.98
    01:55:01 AM     all      2.74      0.00      2.03      0.03      1.37     93.83
    02:00:01 AM     all      2.45      0.00      1.75      0.02      0.83     94.94
    02:05:01 AM     all      3.30      0.00      2.34      0.02      2.17     92.17
    02:10:02 AM     all      6.19      0.00      2.50      0.11      1.63     89.57
    02:15:02 AM     all      2.73      0.00      1.87      0.03      1.57     93.81
    02:20:01 AM     all      2.86      0.00      2.25      0.06      2.11     92.73
    02:25:01 AM     all      3.07      0.00      2.28      0.03      2.47     92.14
    02:30:02 AM     all      2.72      0.00      2.06      0.03      2.73     92.46
    02:35:01 AM     all      3.06      0.00      2.37      0.04      4.59     89.94
    02:40:01 AM     all      2.97      0.00      2.21      0.03      3.53     91.27
    02:45:01 AM     all      3.00      0.00      2.21      0.04      3.82     90.93
    02:50:02 AM     all      2.93      0.00      2.27      0.04      3.28     91.49
    02:55:01 AM     all      3.03      0.00      2.39      0.06      3.23     91.29
    03:00:02 AM     all      2.88      0.00      2.12      0.03      3.08     91.89
    03:05:01 AM     all      3.35      0.00      2.82      0.10      3.93     89.80
    03:10:01 AM     all      6.70      0.00      2.77      0.25      2.67     87.60
    03:15:01 AM     all      2.83      0.07      2.04      0.03      2.88     92.14
    03:20:02 AM     all      3.07      0.00      2.17      0.03      1.26     93.47
    03:25:01 AM     all      3.61      0.00      3.01      0.01      1.79     91.58
    03:30:02 AM     all      2.70      0.00      2.42      0.02      1.69     93.16
    03:35:02 AM     all      2.89      0.00      2.09      0.03      2.23     92.75
    03:40:02 AM     all      2.98      0.00      2.16      0.04      2.31     92.52
    03:45:01 AM     all      2.91      0.00      2.02      0.03      2.77     92.27
    03:50:01 AM     all      2.97      4.96      2.94      0.35      2.61     86.18
    03:55:01 AM     all      5.17      0.22      2.95      1.15      2.11     88.40
    04:00:02 AM     all      3.54      0.00      2.46      0.04      2.71     91.26
    04:05:01 AM     all      3.81      0.00      3.47      0.10      4.38     88.25
    04:10:01 AM     all      4.56      0.00      2.80      0.05      2.39     90.20
    04:15:02 AM     all      4.40      0.00      2.03      0.11      2.10     91.37
    04:20:01 AM     all      2.89      0.00      2.08      0.03      2.44     92.56
    04:25:01 AM     all      3.08      0.00      2.06      0.02      2.52     92.31
    04:30:01 AM     all      2.67      0.00      1.88      0.03      2.85     92.57
    04:35:02 AM     all      2.91      0.00      2.13      0.04      3.08     91.83
    04:40:01 AM     all      2.94      0.00      2.13      0.02      2.89     92.02
    04:45:02 AM     all      2.97      0.00      1.96      0.03      3.15     91.89
    04:50:01 AM     all      2.96      0.00      2.01      0.02      2.43     92.58
    04:55:02 AM     all      2.90      0.00      2.03      0.04      2.55     92.49
    05:00:01 AM     all      2.70      0.00      1.96      0.02      2.72     92.60
    05:05:01 AM     all      3.55      0.00      2.33      0.06      5.01     89.05
    05:10:01 AM     all      3.06      0.00      2.18      0.05      5.33     89.39
    05:15:01 AM     all      2.86      0.00      1.98      0.03      4.94     90.19
    05:20:01 AM     all      6.89      0.00      2.53      0.06      4.63     85.89
    05:25:01 AM     all      3.09      0.00      2.07      0.03      4.05     90.76
    05:30:02 AM     all      2.76      0.00      1.91      0.03      3.70     91.61
    05:35:01 AM     all      2.95      0.00      2.15      0.03      3.68     91.19
    05:40:01 AM     all      2.82      0.00      2.01      0.02      2.79     92.35
    05:45:01 AM     all      3.06      0.00      2.03      0.04      3.67     91.20
    05:50:01 AM     all      2.83      0.00      2.07      0.04      3.30     91.76
    05:55:01 AM     all      3.15      0.00      2.22      0.04      2.79     91.79
    06:00:01 AM     all      2.74      0.00      1.94      0.05      2.73     92.54
    06:05:01 AM     all      3.32      0.00      2.43      0.03      4.11     90.10
    06:10:01 AM     all      6.74      0.00      2.52      0.05      3.46     87.24
    06:15:01 AM     all      2.69      0.00      1.83      0.03      3.57     91.87
    06:20:01 AM     all      3.02      0.00      2.15      0.04      3.18     91.61
    06:25:03 AM     all      3.13      0.00      2.14      0.05      5.09     89.60
    06:30:01 AM     all      3.07      0.00      2.10      0.04      7.47     87.32
    06:35:02 AM     all      3.12      0.00      2.37      0.04      7.51     86.96
    06:40:01 AM     all      3.09      0.00      2.18      0.04      6.85     87.84
    06:45:02 AM     all      3.13      0.00      2.05      0.05      6.70     88.07
    06:50:01 AM     all      2.91      0.00      2.08      0.10      3.36     91.56
    06:55:01 AM     all      3.15      0.00      2.13      0.02      3.67     91.02
    07:00:01 AM     all      2.69      0.00      1.96      0.02      2.70     92.63
    07:05:01 AM     all      3.45      0.00      2.55      0.03      4.53     89.44
    07:10:01 AM     all      3.10      0.00      2.15      0.04      4.46     90.25
    07:15:02 AM     all      4.00      0.00      2.42      0.12      4.59     88.87
    07:20:02 AM     all      6.28      0.00      2.59      0.59      3.74     86.80
    07:25:01 AM     all      3.19      0.00      2.18      0.04      3.68     90.91
    07:30:02 AM     all      2.92      0.00      2.05      0.04      4.02     90.97
    07:35:02 AM     all      3.05      0.00      2.14      0.05      4.38     90.38
    07:40:02 AM     all      3.10      0.00      2.05      0.03      4.01     90.82
    07:45:01 AM     all      2.89      0.00      2.08      0.04      5.54     89.46
    07:50:01 AM     all      2.84      0.00      2.14      0.06      4.06     90.90
    07:55:01 AM     all      3.11      0.00      2.23      0.06      3.99     90.61
    08:00:01 AM     all      2.89      0.00      2.00      0.03      3.42     91.66
    08:05:02 AM     all      3.34      0.00      2.76      0.12      6.75     87.04
    08:10:02 AM     all      4.80      0.00      2.54      0.06      4.98     87.62
    08:15:02 AM     all      4.81      0.00      2.20      0.13      4.08     88.77
    08:20:02 AM     all      3.39      0.00      2.40      0.13      5.17     88.92
    08:25:02 AM     all      3.22      0.00      2.32      0.05      5.89     88.52
    08:30:01 AM     all      2.96      0.00      1.95      0.04      4.73     90.32
    08:35:01 AM     all      3.17      0.00      2.19      0.06      6.10     88.49
    08:40:01 AM     all      2.97      0.00      2.23      0.06      4.57     90.17
    08:45:01 AM     all      3.10      0.00      2.22      0.04      6.82     87.82
    08:50:01 AM     all      2.90      0.00      2.23      0.06      4.91     89.90
    08:55:02 AM     all      2.95      0.00      2.11      0.03      5.20     89.70
    09:00:02 AM     all      2.86      0.00      2.00      0.03      4.59     90.52
    09:05:02 AM     all      5.36      0.00      2.65      0.05      6.29     85.65
    09:10:01 AM     all      5.52      0.00      2.27      0.04      6.72     85.44
    09:15:02 AM     all      2.91      0.00      1.95      0.05      5.63     89.45
    09:20:01 AM     all      3.03      0.00      2.13      0.04      5.82     88.99
    09:25:02 AM     all      3.43      0.00      2.28      0.05      4.72     89.52
    09:30:01 AM     all      3.03      0.00      1.96      0.03      5.18     89.79
    09:35:02 AM     all      3.10      0.00      2.23      0.02      6.18     88.46
    09:40:01 AM     all      3.04      0.00      2.20      0.05      5.20     89.52
    09:45:02 AM     all      3.11      0.00      2.14      0.08      4.59     90.08
    09:50:01 AM     all      2.94      0.00      2.13      0.04      5.62     89.27
    09:55:01 AM     all      2.93      0.00      2.19      0.04      4.94     89.91
    10:00:02 AM     all      2.81      0.00      2.06      0.41      3.31     91.42
    10:05:02 AM     all      3.43      0.00      2.63      0.06      5.88     88.00
    10:10:01 AM     all      3.05      0.00      2.14      0.04      3.25     91.52
    10:15:02 AM     all      4.53      0.00      2.34      0.05      3.31     89.78
    10:20:01 AM     all      5.22      0.00      2.51      0.46      4.10     87.71
    10:25:01 AM     all      3.33      0.00      2.15      0.04      4.82     89.66
    10:30:02 AM     all      2.88      0.00      2.05      0.05      4.27     90.76
    10:35:01 AM     all      3.04      0.00      2.18      0.03      4.74     90.01
    10:40:01 AM     all      3.17      0.00      2.26      0.04      3.60     90.93
    10:45:01 AM     all      2.97      0.00      2.02      0.02      3.26     91.73
    10:50:01 AM     all      2.90      0.00      2.03      0.02      2.32     92.73
    10:55:01 AM     all      3.20      0.00      2.16      0.05      2.82     91.77
    11:00:01 AM     all      2.82      0.00      2.01      0.04      2.62     92.51
    11:05:01 AM     all      3.29      0.00      2.48      0.07      4.30     89.86
    11:10:01 AM     all      7.31      0.00      2.79      0.26      3.66     85.97
    11:15:01 AM     all      2.76      0.00      2.02      0.04      3.42     91.76
    11:20:02 AM     all      3.05      0.00      2.21      0.06      3.23     91.45
    11:25:02 AM     all      3.13      0.00      2.30      0.03      3.20     91.34
    11:30:01 AM     all      2.82      0.00      1.93      0.03      2.54     92.67
    11:35:01 AM     all      3.04      0.00      2.32      0.04      2.96     91.64
    11:40:01 AM     all      2.99      0.00      2.15      0.04      3.41     91.40
    11:45:02 AM     all      2.95      0.00      2.03      0.03      3.59     91.40
    11:50:02 AM     all      2.93      0.00      2.21      0.03      3.36     91.48
    11:55:01 AM     all      2.97      0.00      2.22      0.02      2.94     91.85
    12:00:02 PM     all      3.00      0.00      2.22      0.03      2.95     91.80
    12:05:01 PM     all      3.29      0.00      2.66      0.08      3.69     90.29
    12:10:01 PM     all      2.91      0.00      2.07      0.05      2.80     92.18
    12:15:01 PM     all      3.37      0.00      1.88      0.03      2.47     92.25
    12:20:02 PM     all      6.43      0.00      2.53      0.39      3.90     86.75
    12:25:02 PM     all      3.08      0.00      2.12      0.07      4.89     89.84
    12:30:01 PM     all      2.95      0.00      1.93      0.03      3.29     91.80
    12:35:02 PM     all      2.98      0.00      2.07      0.04      3.74     91.17
    12:40:01 PM     all      3.08      0.00      2.15      0.04      2.95     91.78
    12:45:01 PM     all      2.96      0.00      1.93      0.04      3.78     91.29
    12:50:01 PM     all      2.94      0.00      2.09      0.03      2.41     92.54
    Average:        all      3.30      0.03      2.18      0.06      3.38     91.03
    


    To view yesterday's sar cpu utilisation use command where yesterday's sar log file is at /var/log/sa/sa$(date +%d -d yesterday).
    Code (Text):
    sar -u -f /var/log/sa/sa$(date +%d -d yesterday) | sed -e "s|$(hostname)|hostname|"
    

    Each day's sar log is at /var/log/sa/saXX where XX is date of the month
    Code (Text):
    ls -lAhrt /var/log/sa/sa* | grep -v sar
    -rw-r--r-- 1 root root  1.2M Aug 11 23:55 /var/log/sa/sa11
    -rw-r--r-- 1 root root  1.2M Aug 12 23:55 /var/log/sa/sa12
    -rw-r--r-- 1 root root  1.2M Aug 13 23:55 /var/log/sa/sa13
    -rw-r--r-- 1 root root  1.4M Aug 14 23:55 /var/log/sa/sa14
    -rw-r--r-- 1 root root  2.7M Aug 15 23:55 /var/log/sa/sa15
    -rw-r--r-- 1 root root  1.3M Aug 16 23:55 /var/log/sa/sa16
    -rw-r--r-- 1 root root  1.3M Aug 17 23:55 /var/log/sa/sa17
    -rw-r--r-- 1 root root  1.6M Aug 18 23:59 /var/log/sa/sa18
    -rw-r--r-- 1 root root  9.8M Aug 19 23:55 /var/log/sa/sa19
    -rw-r--r-- 1 root root  1.2M Aug 20 23:55 /var/log/sa/sa20
    -rw-r--r-- 1 root root  1.2M Aug 21 23:55 /var/log/sa/sa21
    -rw-r--r-- 1 root root  1.2M Aug 22 23:55 /var/log/sa/sa22
    -rw-r--r-- 1 root root  1.2M Aug 23 23:55 /var/log/sa/sa23
    -rw-r--r-- 1 root root  1.2M Aug 24 23:55 /var/log/sa/sa24
    -rw-r--r-- 1 root root  1.2M Aug 25 23:55 /var/log/sa/sa25
    -rw-r--r-- 1 root root  1.2M Aug 26 23:55 /var/log/sa/sa26
    -rw-r--r-- 1 root root  1.2M Aug 27 23:55 /var/log/sa/sa27
    -rw-r--r-- 1 root root  1.4M Aug 28 23:55 /var/log/sa/sa28
    -rw-r--r-- 1 root root  1.3M Aug 29 23:58 /var/log/sa/sa29
    -rw-r--r-- 1 root root  1.2M Aug 30 23:55 /var/log/sa/sa30
    -rw-r--r-- 1 root root  1.2M Aug 31 23:55 /var/log/sa/sa31
    -rw-r--r-- 1 root root  1.2M Sep  1 23:55 /var/log/sa/sa01
    -rw-r--r-- 1 root root  1.2M Sep  2 23:55 /var/log/sa/sa02
    -rw-r--r-- 1 root root  1.3M Sep  3 23:55 /var/log/sa/sa03
    -rw-r--r-- 1 root root  1.2M Sep  4 23:55 /var/log/sa/sa04
    -rw-r--r-- 1 root root  1.2M Sep  5 23:55 /var/log/sa/sa05
    -rw-r--r-- 1 root root  1.3M Sep  6 23:55 /var/log/sa/sa06
    -rw-r--r-- 1 root root  1.2M Sep  7 23:55 /var/log/sa/sa07
    -rw-r--r-- 1 root root  1.2M Sep  8 23:55 /var/log/sa/sa08
    -rw-r--r-- 1 root root  1.2M Sep  9 23:55 /var/log/sa/sa09
    -rw-r--r-- 1 root root  365K Sep 10 07:20 /var/log/sa/sa10
    

    So to view 2nd of this month would be
    Code (Text):
    sar -u -f /var/log/sa/sa02 | sed -e "s|$(hostname)|hostname|"
    

    Or view 2nd of this month after 10:30 pm with -s flag
    Code (Text):
    sar -u -f /var/log/sa/sa02 -s 22:30:00 | sed -e "s|$(hostname)|hostname|"
    
     
  6. Dannymh

    Dannymh New Member

    12
    3
    3
    Oct 23, 2014
    Ratings:
    +3
    Local Time:
    7:52 PM
    1.7.x
    5.5
    Thanks for the heads up on that one. On my setup that isn't meant to happen, they are meant to be more isolated than that.
     
  7. eva2000

    eva2000 Administrator Staff Member

    44,785
    10,212
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,828
    Local Time:
    3:52 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    yeah i'll add some sar -u cpu utilisatin stats to cminfo top command too in next 123.09beta01 update later today :)