Welcome to Centmin Mod Community
Become a Member

Sysadmin Swap increases slowly, but continuously until 100%

Discussion in 'System Administration' started by Simon Brown, Jun 24, 2019.

  1. Simon Brown

    Simon Brown Member

    41
    5
    8
    Feb 9, 2017
    Ratings:
    +17
    Local Time:
    11:00 PM
    1.11.9
    Please fill in any relevant information that applies to you:
    • CentOS Version: CentOS 7.6.1810 - 64bit
    • Centmin Mod Version Installed: 123.09beta01
    • Nginx Version Installed: 1.17.0
    • PHP Version Installed: 7.0.33
    • MariaDB MySQL Version Installed: 10.1.40
    • When was last time updated Centmin Mod code base ? : run cmupdate command
    • Persistent Config:
      Code (Text):
      NGXDYNAMIC_NGXPAGESPEED='n'
      NGINX_PAGESPEED='n'
      NGXDYNAMIC_BROTLI='y'
      NGINX_LIBBROTLI='y'
      LETSENCRYPT_DETECT='y'
      NGINX_VHOSTSTATS=n
      MARCH_TARGETNATIVE='n'
      NGINX_VIDEO='y'
      PHP_PGO='y'
      
    This is a strange issue that a little beyond my tech knowledge. Has anyone else seen this issue...

    The setup:
    I have 1x new Vultr bare metal dedicated server, 2x new Vultr servers and 1x new UpCloud server all using the standard Centos 7 install. I've installed centmin 123.09beta01.

    The servers all have a minimum of 4 processors & minimum of 8Gb Memory.

    The only custom setup is the servers use Galera for database clustering. While I test the setup all the servers are behind a firewall that's blocking access on all ports to the outside world. The IP's of all the servers are whitelisted in the external firewalls and the CSF firewalls, so all ports are available only to the network of servers. Galera is working well, ssh, ping, etc. is working perfectly between all the servers.
    centmin & all yum updates are the latest versions.

    The issue:
    For some reason, the servers all start to use swap even though there's plenty of memory.
    The results below are for 24 hours after a server reboot.

    I've run a script to find only the prosesses using swap. As you can see Nginx uses most of it:

    Overall swap used: 120380 kB
    ========================================
    kB pid name
    ========================================
    33808 4352 nginx
    31464 4357 nginx
    31356 4353 nginx
    8900 3344 tuned
    5024 1884 polkitd
    3728 1906 haveged
    1572 4759 amplify-agent
    912 3340 sshd
    824 1887 rngd
    576 1543 systemd-udevd
    532 3348 rsyslogd
    328 1958 chronyd
    308 1521 systemd-journal
    284 1926 dbus-daemon
    264 2034 crond
    220 1 systemd
    148 1892 irqbalance
    132 2012 systemd-logind

    I've found that restarting Nginx via the 'nprestart' helps to lower the swap usage but obviously, it's not a fix.

    Free mem command: free -m
    total used free shared buff/cache available
    Mem: 7821 1484 795 19 5541 5981
    Swap: 1023 57 966



    Swap usage over 24 hours. You can see where the swap got to after a few days to the left, then I rebooted the server.
    Screenshot 2019-06-24 at 11.23.28 am.jpg

    Same server over 4 days:
    Screenshot 2019-06-24 at 11.54.24 am.jpg

    Another server over 4 days:
    Screenshot 2019-06-24 at 11.53.17 am.jpg

    The high CPU/Disk usage to the right of these graphs is from recompiling PHP to use version 7.3.5. It's the only software that isn't the latest version. I don't think it's the issue though.
    Screenshot 2019-06-24 at 11.52.25 am.jpg Screenshot 2019-06-24 at 11.51.35 am.jpg


    Memory usage:
    Screenshot 2019-06-24 at 11.50.45 am.jpg


    Has anyone had the same swap issue?

    I've been trying to solve this issue for a few weeks. I'm running out of ideas!!!
     
  2. eva2000

    eva2000 Administrator Staff Member

    41,646
    9,380
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,408
    Local Time:
    8:00 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    For each server post the output for commands below? Wrap in quote bbcode tags
    Code (Text):
    nginx - V

    And next one wrap in code bbcode tags
    Code (Text):
    cminfo top

    When was last time updated Centmin Mod code base ? Using cm update command before nginx compiles/updates?

    Try disabling these set to no and recompile nginx via centmin.sh menu option 4 and see
     
    Last edited: Jun 25, 2019
  3. Simon Brown

    Simon Brown Member

    41
    5
    8
    Feb 9, 2017
    Ratings:
    +17
    Local Time:
    11:00 PM
    1.11.9
    I regularly update centmin and the yum updates when I log out of centmin. The server(s) is up-to-date.

    Code:
     Centmin Mod local code is up to date at /usr/local/src/centminmod
     no available updates at this time...
    The NGINX_VIDEO='y' is only enabled on one server. I enabled it a few days ago. The server was swapping for a few weeks before I enabled the video.

    These options are enabled on all the servers NGXDYNAMIC_BROTLI='y' NGINX_LIBBROTLI='y' so I'll disable it on one server and see if that make any difference.

    nginx -V
    Code:
    nginx version: nginx/1.17.0 (190619-150338-centos7-kvm)
    built by gcc 8.2.1 20180905 (Red Hat 8.2.1-3) (GCC)
    built with OpenSSL 1.1.1c  28 May 2019
    TLS SNI support enabled
    configure arguments: --with-ld-opt='-Wl,-E -L/usr/local/zlib-cf/lib -L/usr/local/lib -ljemalloc -Wl,-z,relro -Wl,-rpath,/usr/local/zlib-cf/lib:/usr/local/lib' --with-cc-opt='-I/usr/local/zlib-cf/include -I/usr/local/include -m64 -march=x86-64 -mavx -mavx2 -mpclmul -msse4 -msse4.1 -msse4.2 -DTCP_FASTOPEN=23 -g -O3 -fstack-protector-strong -flto -fuse-ld=gold --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wimplicit-fallthrough=0 -fcode-hoisting -Wp,-D_FORTIFY_SOURCE=2 -Wno-deprecated-declarations' --sbin-path=/usr/local/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --build=190619-150338-centos7-kvm --with-compat --with-http_stub_status_module --with-http_secure_link_module --with-http_flv_module --with-http_mp4_module --add-module=../nginx-rtmp-module --with-libatomic --with-http_gzip_static_module --add-dynamic-module=../ngx_brotli --with-http_sub_module --with-http_addition_module --with-http_image_filter_module=dynamic --with-http_geoip_module --with-stream_geoip_module --with-stream_realip_module --with-stream_ssl_preread_module --with-threads --with-stream --with-stream_ssl_module --with-http_slice_module --with-http_realip_module --add-dynamic-module=../ngx-fancyindex-0.4.2 --add-module=../ngx_cache_purge-2.5 --add-dynamic-module=../ngx_devel_kit-0.3.0 --add-dynamic-module=../set-misc-nginx-module-0.32 --add-dynamic-module=../echo-nginx-module-0.61 --add-module=../redis2-nginx-module-0.15 --add-module=../ngx_http_redis-0.3.7 --add-module=../memc-nginx-module-0.18 --add-module=../srcache-nginx-module-0.31 --add-dynamic-module=../headers-more-nginx-module-0.33 --with-pcre-jit --with-zlib=../zlib-cloudflare-1.3.0 --with-http_ssl_module --with-http_v2_module --with-openssl=../openssl-1.1.1c --with-openssl-opt='enable-ec_nistp_64_gcc_128 enable-tls1_3'

    cminfo top:

    Code:
    PHP Warning:  PHP Startup: Unable to load dynamic library 'geoip.so' (tried: /usr/local/lib/php/extensions/no-debug-non-zts-20180731/geoip.so (/usr/local/lib/php/extensions/no-debug-non-zts-20180731/geoip.so: cannot open shared object file: No such file or directory), /usr/local/lib/php/extensions/no-debug-non-zts-20180731/geoip.so.so (/usr/local/lib/php/extensions/no-debug-non-zts-20180731/geoip.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
    PHP Warning:  PHP Startup: Unable to load dynamic library 'mailparse.so' (tried: /usr/local/lib/php/extensions/no-debug-non-zts-20180731/mailparse.so (/usr/local/lib/php/extensions/no-debug-non-zts-20180731/mailparse.so: cannot open shared object file: No such file or directory), /usr/local/lib/php/extensions/no-debug-non-zts-20180731/mailparse.so.so (/usr/local/lib/php/extensions/no-debug-non-zts-20180731/mailparse.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
    PHP Warning:  PHP Startup: Unable to load dynamic library 'redis.so' (tried: /usr/local/lib/php/extensions/no-debug-non-zts-20180731/redis.so (/usr/local/lib/php/extensions/no-debug-non-zts-20180731/redis.so: cannot open shared object file: No such file or directory), /usr/local/lib/php/extensions/no-debug-non-zts-20180731/redis.so.so (/usr/local/lib/php/extensions/no-debug-non-zts-20180731/redis.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
    ------------------------------------------------------------------
     Centmin Mod Top Info:
    ------------------------------------------------------------------
     Server Location Info
    
      city: Spitalfields
      region: Essex
      country: GB
      ASN: AS20473 Choopa, LLC
    
     Processors physical = 1, cores = 4, virtual = 4, hyperthreading = no
    
          4  2593.904
          4  Virtual CPU 82d9ed4018dd
          4  16384 KB
    
     System Up Since:     2019-06-23 11:55:31
     System Uptime:     up 1 day, 4 hours, 24 minutes
     MySQL Server Started     2019-06-23 11:55:47
     MySQL Uptime:         1 day 4 hours 24 min 11 sec
     MySQL Uptime (secs):     102251
     Server Type:         kvm
     CentOS Version:     7.6
     Centmin Mod:         123.09beta01.b193
     Nginx PageSpeed:     OFF
     Nginx Version:     1.17.0 (190619-150338-centos7-kvm)
     PHP-FPM Version:     Warning:
     MariaDB Version:     10.1.40
     CSF Firewall:         v12.12
     Memcached Server:     1.5.16
     NSD Version:          -
     Siege Version:     4.0.4
     Maldet Version:     not installed
     ClamAV Version:     not installed
     ElasticSearch:     not installed
    
    ------------------------------------------------------------------
    free -mtl
                  total        used        free      shared  buff/cache   available
    Mem:           7821        1456         248          19        6115        6008
    Low:           7821        7572         248
    High:             0           0           0
    Swap:          1023          56         967
    Total:         8845        1512        1216
    
    ------------------------------------------------------------------
    df -hT
    Filesystem     Type      Size  Used Avail Use% Mounted on
    /dev/vda1      ext4      158G   59G   91G  40% /
    devtmpfs       devtmpfs  3.9G     0  3.9G   0% /dev
    tmpfs          tmpfs     3.9G     0  3.9G   0% /dev/shm
    tmpfs          tmpfs     3.9G   17M  3.9G   1% /run
    tmpfs          tmpfs     3.9G     0  3.9G   0% /sys/fs/cgroup
    /dev/loop0     ext4      5.8G   25M  5.5G   1% /tmp
    tmpfs          tmpfs     783M     0  783M   0% /run/user/0
    
    ------------------------------------------------------------------
    php-fpm stats
    
    Ready to handle connections
    pool:                 www
    process manager:      ondemand
    start time:           24/Jun/2019:16:20:00 +0100
    start since:          0
    accepted conn:        2
    listen queue:         0
    max listen queue:     0
    listen queue len:     8151
    idle processes:       0
    active processes:     1
    total processes:      1
    max active processes: 1
    max children reached: 0
    slow requests:        0
    
    ------------------------------------------------------------------
    Filter sar -q for times cpu load avg (1min) hit/exceeded cpu threads max
    
    12:05:01 AM         0       158      0.19      0.28      0.30         0
    12:10:01 AM         0       158      0.11      0.20      0.27         0
    12:15:01 AM         0       159      0.25      0.25      0.28         0
    12:20:01 AM         0       158      0.14      0.18      0.24         0
    12:25:01 AM         0       159      0.25      0.27      0.27         0
    12:30:01 AM         0       172      0.24      0.24      0.26         0
    12:35:01 AM         0       159      0.45      0.36      0.30         0
    12:40:01 AM         0       158      0.51      0.51      0.40         0
    12:45:01 AM         1       162      0.47      0.47      0.41         0
    12:50:01 AM         0       158      0.19      0.32      0.37         0
    12:55:01 AM         0       159      0.50      0.43      0.40         0
    01:00:01 AM         1       191      0.63      0.38      0.38         0
    01:05:01 AM         0       166      0.35      0.32      0.35         0
    01:10:01 AM         1       159      0.33      0.30      0.32         0
    01:15:01 AM         1       158      0.16      0.29      0.31         0
    01:20:01 AM         0       158      0.33      0.34      0.33         0
    01:25:02 AM         0       164      0.36      0.31      0.32         0
    01:30:01 AM         0       172      0.24      0.29      0.31         0
    01:35:01 AM         1       162      0.39      0.29      0.31         0
    01:40:01 AM         0       158      0.26      0.26      0.30         0
    01:45:01 AM         1       163      0.33      0.38      0.35         0
    01:50:01 AM         2       164      0.11      0.21      0.28         0
    01:55:01 AM         0       158      0.37      0.33      0.32         0
    02:00:01 AM         6       239      1.19      0.57      0.41         0
    02:05:01 AM         0       160      0.18      0.35      0.37         0
    02:10:01 AM         0       166      0.34      0.31      0.34         0
    02:15:01 AM         0       159      0.19      0.25      0.31         0
    02:20:01 AM         0       159      0.33      0.25      0.29         0
    02:25:01 AM         0       159      0.26      0.31      0.31         0
    02:30:01 AM         6       187      0.71      0.40      0.34         0
    02:35:01 AM         1       163      0.33      0.30      0.31         0
    02:40:01 AM         0       165      0.16      0.24      0.29         0
    02:45:01 AM         0       160      0.36      0.32      0.32         0
    02:50:01 AM         0       159      0.50      0.33      0.32         0
    02:55:01 AM         2       163      0.44      0.36      0.33         0
    03:00:01 AM         0       180      0.27      0.34      0.33         0
    03:05:01 AM         1       165      0.20      0.34      0.33         0
    03:10:01 AM         0       160      0.37      0.37      0.34         0
    03:15:01 AM         0       159      0.33      0.40      0.37         0
    03:20:01 AM         0       159      0.36      0.34      0.34         0
    03:25:01 AM         0       159      0.34      0.36      0.35         0
    03:30:02 AM         0       173      0.37      0.37      0.36         0
    03:35:01 AM         0       159      0.20      0.29      0.32         0
    03:40:01 AM         5       167      0.32      0.30      0.32         0
    03:45:01 AM         1       160      0.30      0.27      0.30         0
    03:50:01 AM         0       159      0.27      0.23      0.28         0
    03:55:01 AM         0       160      0.10      0.20      0.26         0
    04:00:01 AM         2       184      0.38      0.30      0.29         0
    04:05:01 AM         0       160      0.42      0.35      0.32         0
    04:10:01 AM         0       159      0.32      0.32      0.32         0
    04:15:01 AM         0       159      0.31      0.35      0.33         0
    04:20:01 AM         1       164      0.39      0.37      0.34         0
    04:25:01 AM         0       160      0.24      0.36      0.34         0
    04:30:01 AM         1       173      0.47      0.38      0.34         0
    04:35:01 AM         0       160      0.46      0.37      0.34         0
    04:40:01 AM         0       160      0.22      0.29      0.31         0
    04:45:01 AM         1       163      0.41      0.37      0.34         0
    04:50:01 AM         1       160      0.39      0.36      0.33         0
    04:55:01 AM         2       168      0.20      0.31      0.32         0
    05:00:01 AM         1       179      0.19      0.23      0.29         0
    05:05:01 AM         1       160      0.23      0.27      0.30         0
    05:10:01 AM         0       161      0.51      0.36      0.33         0
    05:15:01 AM         0       160      0.34      0.37      0.33         0
    05:20:01 AM         1       171      0.25      0.31      0.31         0
    05:25:01 AM         0       160      0.39      0.38      0.35         0
    05:30:01 AM         0       174      0.67      0.41      0.36         0
    05:35:01 AM         1       165      0.24      0.36      0.36         0
    05:40:01 AM         1       166      0.29      0.32      0.34         0
    05:45:01 AM         0       160      0.51      0.37      0.34         0
    05:50:01 AM         0       160      0.54      0.38      0.34         0
    05:55:01 AM         0       160      0.30      0.32      0.32         0
    06:00:01 AM         0       181      0.12      0.25      0.29         0
    06:05:01 AM         0       160      0.29      0.35      0.33         0
    06:10:01 AM         0       160      0.28      0.30      0.31         0
    06:15:01 AM         0       160      0.59      0.39      0.34         0
    06:20:01 AM         0       160      0.34      0.36      0.34         0
    06:25:01 AM         0       167      0.76      0.42      0.35         0
    06:30:01 AM         2       174      0.40      0.35      0.33         0
    06:35:01 AM         1       161      0.36      0.31      0.32         0
    06:40:01 AM         0       160      0.19      0.27      0.31         0
    06:45:01 AM         0       160      0.32      0.37      0.35         0
    06:50:01 AM         1       164      0.32      0.38      0.36         0
    06:55:01 AM         0       161      0.25      0.32      0.34         0
    07:00:01 AM         2       192      0.66      0.40      0.35         0
    07:05:01 AM         0       162      0.26      0.45      0.40         0
    07:10:01 AM         0       160      0.36      0.39      0.39         0
    07:15:01 AM         1       164      0.18      0.27      0.34         0
    07:20:01 AM         2       172      0.12      0.24      0.31         0
    07:25:01 AM         0       173      0.47      0.30      0.32         0
    07:30:01 AM         4       178      0.40      0.40      0.36         0
    07:35:01 AM         0       166      0.34      0.38      0.36         0
    07:40:01 AM         1       177      0.37      0.34      0.34         0
    07:45:01 AM         1       164      0.44      0.39      0.36         0
    07:50:01 AM         0       166      0.44      0.38      0.36         0
    07:55:01 AM         0       168      0.45      0.43      0.39         0
    08:00:01 AM         2       189      0.51      0.43      0.41         0
    08:05:01 AM         0       160      0.15      0.31      0.38         0
    08:10:01 AM         1       164      0.35      0.37      0.40         0
    08:15:01 AM         0       160      0.46      0.42      0.41         0
    08:20:01 AM         1       161      0.17      0.30      0.37         0
    08:25:01 AM         0       160      0.31      0.33      0.37         0
    08:30:01 AM         4       175      0.22      0.33      0.36         0
    08:35:01 AM         1       160      0.30      0.34      0.36         0
    08:40:01 AM         2       164      0.44      0.37      0.36         0
    08:45:01 AM         0       161      0.38      0.35      0.34         0
    08:50:01 AM         0       160      0.23      0.37      0.35         0
    08:55:01 AM         0       160      0.31      0.34      0.35         0
    09:00:02 AM         4       182      0.39      0.32      0.33         0
    09:05:01 AM         0       160      0.40      0.37      0.34         0
    09:10:01 AM         0       160      0.41      0.32      0.33         0
    09:15:01 AM         0       166      0.39      0.36      0.33         0
    09:20:01 AM         0       160      0.31      0.34      0.33         0
    09:25:01 AM         0       160      0.24      0.30      0.31         0
    09:30:01 AM         1       173      0.39      0.35      0.33         0
    09:35:01 AM         1       164      0.34      0.30      0.32         0
    09:40:01 AM         0       166      0.20      0.25      0.30         0
    09:45:01 AM         0       160      0.48      0.40      0.35         0
    09:50:01 AM         1       164      0.36      0.43      0.39         0
    09:55:01 AM         0       160      0.64      0.43      0.40         0
    10:00:01 AM         1       185      0.40      0.41      0.40         0
    10:05:01 AM         1       171      0.48      0.45      0.42         0
    10:10:01 AM         0       160      0.29      0.40      0.41         0
    10:15:01 AM         2       165      0.36      0.32      0.38         0
    10:20:01 AM         1       163      0.26      0.42      0.41         0
    10:25:01 AM         0       159      0.09      0.25      0.35         0
    10:30:01 AM         2       174      0.24      0.31      0.35         0
    10:35:01 AM         0       159      0.31      0.31      0.34         0
    10:40:01 AM         0       159      0.39      0.39      0.39         0
    10:41:35 AM       LINUX RESTART
    10:50:01 AM         0       161      0.38      0.37      0.39         0
    10:55:01 AM         0       161      0.19      0.26      0.34         0
    11:00:01 AM         0       186      0.45      0.35      0.34         0
    11:05:01 AM         0       165      0.11      0.30      0.33         0
    11:10:02 AM         0       165      0.64      0.38      0.35         0
    11:15:01 AM         1       166      0.27      0.26      0.30         0
    11:15:27 AM       LINUX RESTART
    11:17:30 AM       LINUX RESTART
    11:17:43 AM         1       170      0.37      0.30      0.32         0
    11:17:55 AM         1       172      0.39      0.31      0.32         0
    11:18:11 AM         1       173      0.53      0.34      0.33         0
    11:18:54 AM         1       173      0.74      0.42      0.36         0
    11:20:01 AM         2       182      1.56      0.64      0.43         0
    11:21:48 AM         3       175      1.95      1.25      0.69         0
    11:25:01 AM         4       177      4.73      3.17      1.56         0
    11:30:01 AM         6       183      1.36      1.92      1.46         0
    11:34:12 AM         0       168      5.49      3.21      2.02         0
    11:34:28 AM         1       171      4.60      3.15      2.03         0
    11:35:01 AM         2       175      3.25      2.96      2.00         0
    11:40:01 AM         0       169      0.44      1.46      1.63         0
    11:45:01 AM         0       166      0.22      0.66      1.24         0
    11:50:01 AM         0       167      0.42      0.47      1.00         0
    11:55:01 AM         0       167      0.28      0.38      0.81         0
    12:00:01 PM         3       200      0.54      0.35      0.67         0
    12:05:01 PM         0       173      0.58      0.38      0.59         0
    12:10:01 PM         0       171      0.24      0.32      0.51         0
    12:15:01 PM         0       180      0.43      0.38      0.49         0
    12:20:01 PM         0       165      0.23      0.32      0.44         0
    12:25:01 PM         1       169      0.26      0.27      0.38         0
    12:30:01 PM         5       186      0.17      0.29      0.37         0
    12:35:01 PM         0       165      0.36      0.39      0.39         0
    12:40:01 PM         1       165      0.18      0.31      0.37         0
    12:45:01 PM         0       159      0.18      0.27      0.34         0
    12:50:01 PM         0       159      0.36      0.28      0.32         0
    12:55:01 PM         0       159      0.26      0.24      0.30         0
    01:00:01 PM         1       192      0.27      0.27      0.30         0
    01:05:01 PM         1       166      0.22      0.23      0.27         0
    01:10:01 PM         1       163      0.16      0.24      0.28         0
    01:15:01 PM         0       159      0.30      0.31      0.31         0
    01:20:01 PM         0       159      0.53      0.33      0.32         0
    01:25:02 PM         1       165      0.43      0.36      0.33         0
    01:30:01 PM         0       163      0.31      0.38      0.35         0
    01:35:01 PM         2       164      0.42      0.42      0.38         0
    01:40:01 PM         0       163      0.27      0.35      0.37         0
    01:45:01 PM         0       163      0.26      0.29      0.33         0
    01:50:01 PM         0       169      0.18      0.24      0.30         0
    01:55:01 PM         0       164      0.38      0.32      0.32         0
    02:00:01 PM         5       184      0.29      0.38      0.37         0
    02:05:01 PM         0       163      0.36      0.30      0.33         0
    02:10:01 PM         1       163      0.63      0.39      0.34         0
    02:12:10 PM       LINUX RESTART
    02:20:01 PM         1       163      0.21      0.29      0.31         0
    02:25:01 PM         0       159      0.36      0.30      0.31         0
    02:30:01 PM         3       185      0.38      0.36      0.33         0
    02:35:01 PM         0       159      0.28      0.30      0.31         0
    02:40:01 PM         1       165      0.18      0.35      0.34         0
    02:45:01 PM         0       159      0.30      0.29      0.32         0
    02:50:01 PM         0       159      0.18      0.25      0.30         0
    02:55:01 PM         1       163      0.44      0.37      0.34         0
    03:00:01 PM         0       186      0.58      0.40      0.34         0
    03:05:01 PM         0       159      0.29      0.32      0.32         0
    03:10:01 PM         1       163      0.49      0.38      0.33         0
    03:15:01 PM         2       163      0.32      0.40      0.36         0
    03:20:01 PM         0       165      0.53      0.34      0.33         0
    03:25:01 PM         0       165      0.14      0.21      0.28         0
    03:30:01 PM         0       173      0.15      0.21      0.26         0
    03:35:01 PM         1       164      0.33      0.25      0.26         0
    03:40:02 PM         0       159      0.17      0.22      0.24         0
    03:45:01 PM         0       165      0.38      0.35      0.30         0
    03:50:01 PM         0       159      0.22      0.28      0.29         0
    03:55:01 PM         0       160      0.44      0.34      0.31         0
    04:00:01 PM         1       190      0.15      0.30      0.31         0
    04:05:01 PM         0       159      0.16      0.30      0.31         0
    04:10:01 PM         4       167      0.22      0.28      0.31         0
    04:15:01 PM         1       165      0.38      0.30      0.32         0
    
    ------------------------------------------------------------------
    Filter sar -q for times cpu load avg (5min) hit/exceeded cpu threads max
    
    11:25:01 AM         4       177      4.73      3.17      1.56         0
    11:34:12 AM         0       168      5.49      3.21      2.02         0
    11:34:28 AM         1       171      4.60      3.15      2.03         0
    
    ------------------------------------------------------------------
    sar -q | sed -e "s|$(hostname)|hostname|g"
    Linux 3.10.0-957.21.3.el7.x86_64 (hostname)     06/24/2019     _x86_64_    (4 CPU)
    
    12:00:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
    12:05:01 AM         0       158      0.19      0.28      0.30         0
    12:10:01 AM         0       158      0.11      0.20      0.27         0
    12:15:01 AM         0       159      0.25      0.25      0.28         0
    12:20:01 AM         0       158      0.14      0.18      0.24         0
    12:25:01 AM         0       159      0.25      0.27      0.27         0
    12:30:01 AM         0       172      0.24      0.24      0.26         0
    12:35:01 AM         0       159      0.45      0.36      0.30         0
    12:40:01 AM         0       158      0.51      0.51      0.40         0
    12:45:01 AM         1       162      0.47      0.47      0.41         0
    12:50:01 AM         0       158      0.19      0.32      0.37         0
    12:55:01 AM         0       159      0.50      0.43      0.40         0
    01:00:01 AM         1       191      0.63      0.38      0.38         0
    01:05:01 AM         0       166      0.35      0.32      0.35         0
    01:10:01 AM         1       159      0.33      0.30      0.32         0
    01:15:01 AM         1       158      0.16      0.29      0.31         0
    01:20:01 AM         0       158      0.33      0.34      0.33         0
    01:25:02 AM         0       164      0.36      0.31      0.32         0
    01:30:01 AM         0       172      0.24      0.29      0.31         0
    01:35:01 AM         1       162      0.39      0.29      0.31         0
    01:40:01 AM         0       158      0.26      0.26      0.30         0
    01:45:01 AM         1       163      0.33      0.38      0.35         0
    01:50:01 AM         2       164      0.11      0.21      0.28         0
    01:55:01 AM         0       158      0.37      0.33      0.32         0
    02:00:01 AM         6       239      1.19      0.57      0.41         0
    02:05:01 AM         0       160      0.18      0.35      0.37         0
    02:10:01 AM         0       166      0.34      0.31      0.34         0
    02:15:01 AM         0       159      0.19      0.25      0.31         0
    02:20:01 AM         0       159      0.33      0.25      0.29         0
    02:25:01 AM         0       159      0.26      0.31      0.31         0
    02:30:01 AM         6       187      0.71      0.40      0.34         0
    02:35:01 AM         1       163      0.33      0.30      0.31         0
    02:40:01 AM         0       165      0.16      0.24      0.29         0
    02:45:01 AM         0       160      0.36      0.32      0.32         0
    02:50:01 AM         0       159      0.50      0.33      0.32         0
    02:55:01 AM         2       163      0.44      0.36      0.33         0
    03:00:01 AM         0       180      0.27      0.34      0.33         0
    03:05:01 AM         1       165      0.20      0.34      0.33         0
    03:10:01 AM         0       160      0.37      0.37      0.34         0
    03:15:01 AM         0       159      0.33      0.40      0.37         0
    03:20:01 AM         0       159      0.36      0.34      0.34         0
    03:25:01 AM         0       159      0.34      0.36      0.35         0
    03:30:02 AM         0       173      0.37      0.37      0.36         0
    03:35:01 AM         0       159      0.20      0.29      0.32         0
    03:40:01 AM         5       167      0.32      0.30      0.32         0
    03:45:01 AM         1       160      0.30      0.27      0.30         0
    03:50:01 AM         0       159      0.27      0.23      0.28         0
    03:55:01 AM         0       160      0.10      0.20      0.26         0
    04:00:01 AM         2       184      0.38      0.30      0.29         0
    04:05:01 AM         0       160      0.42      0.35      0.32         0
    04:10:01 AM         0       159      0.32      0.32      0.32         0
    04:15:01 AM         0       159      0.31      0.35      0.33         0
    04:20:01 AM         1       164      0.39      0.37      0.34         0
    04:25:01 AM         0       160      0.24      0.36      0.34         0
    04:30:01 AM         1       173      0.47      0.38      0.34         0
    04:35:01 AM         0       160      0.46      0.37      0.34         0
    04:40:01 AM         0       160      0.22      0.29      0.31         0
    04:45:01 AM         1       163      0.41      0.37      0.34         0
    04:50:01 AM         1       160      0.39      0.36      0.33         0
    04:55:01 AM         2       168      0.20      0.31      0.32         0
    05:00:01 AM         1       179      0.19      0.23      0.29         0
    05:05:01 AM         1       160      0.23      0.27      0.30         0
    05:10:01 AM         0       161      0.51      0.36      0.33         0
    05:15:01 AM         0       160      0.34      0.37      0.33         0
    05:20:01 AM         1       171      0.25      0.31      0.31         0
    05:25:01 AM         0       160      0.39      0.38      0.35         0
    05:30:01 AM         0       174      0.67      0.41      0.36         0
    05:35:01 AM         1       165      0.24      0.36      0.36         0
    05:40:01 AM         1       166      0.29      0.32      0.34         0
    05:45:01 AM         0       160      0.51      0.37      0.34         0
    05:50:01 AM         0       160      0.54      0.38      0.34         0
    05:55:01 AM         0       160      0.30      0.32      0.32         0
    06:00:01 AM         0       181      0.12      0.25      0.29         0
    06:05:01 AM         0       160      0.29      0.35      0.33         0
    06:10:01 AM         0       160      0.28      0.30      0.31         0
    06:15:01 AM         0       160      0.59      0.39      0.34         0
    06:20:01 AM         0       160      0.34      0.36      0.34         0
    06:25:01 AM         0       167      0.76      0.42      0.35         0
    06:30:01 AM         2       174      0.40      0.35      0.33         0
    06:35:01 AM         1       161      0.36      0.31      0.32         0
    06:40:01 AM         0       160      0.19      0.27      0.31         0
    06:45:01 AM         0       160      0.32      0.37      0.35         0
    06:50:01 AM         1       164      0.32      0.38      0.36         0
    06:55:01 AM         0       161      0.25      0.32      0.34         0
    07:00:01 AM         2       192      0.66      0.40      0.35         0
    07:05:01 AM         0       162      0.26      0.45      0.40         0
    07:10:01 AM         0       160      0.36      0.39      0.39         0
    07:15:01 AM         1       164      0.18      0.27      0.34         0
    07:20:01 AM         2       172      0.12      0.24      0.31         0
    07:25:01 AM         0       173      0.47      0.30      0.32         0
    07:30:01 AM         4       178      0.40      0.40      0.36         0
    07:35:01 AM         0       166      0.34      0.38      0.36         0
    07:40:01 AM         1       177      0.37      0.34      0.34         0
    07:45:01 AM         1       164      0.44      0.39      0.36         0
    07:50:01 AM         0       166      0.44      0.38      0.36         0
    07:55:01 AM         0       168      0.45      0.43      0.39         0
    08:00:01 AM         2       189      0.51      0.43      0.41         0
    08:05:01 AM         0       160      0.15      0.31      0.38         0
    08:10:01 AM         1       164      0.35      0.37      0.40         0
    08:15:01 AM         0       160      0.46      0.42      0.41         0
    08:20:01 AM         1       161      0.17      0.30      0.37         0
    08:25:01 AM         0       160      0.31      0.33      0.37         0
    08:30:01 AM         4       175      0.22      0.33      0.36         0
    08:35:01 AM         1       160      0.30      0.34      0.36         0
    08:40:01 AM         2       164      0.44      0.37      0.36         0
    08:45:01 AM         0       161      0.38      0.35      0.34         0
    08:50:01 AM         0       160      0.23      0.37      0.35         0
    08:55:01 AM         0       160      0.31      0.34      0.35         0
    09:00:02 AM         4       182      0.39      0.32      0.33         0
    09:05:01 AM         0       160      0.40      0.37      0.34         0
    09:10:01 AM         0       160      0.41      0.32      0.33         0
    09:15:01 AM         0       166      0.39      0.36      0.33         0
    09:20:01 AM         0       160      0.31      0.34      0.33         0
    09:25:01 AM         0       160      0.24      0.30      0.31         0
    09:30:01 AM         1       173      0.39      0.35      0.33         0
    09:35:01 AM         1       164      0.34      0.30      0.32         0
    09:40:01 AM         0       166      0.20      0.25      0.30         0
    09:45:01 AM         0       160      0.48      0.40      0.35         0
    09:50:01 AM         1       164      0.36      0.43      0.39         0
    09:55:01 AM         0       160      0.64      0.43      0.40         0
    10:00:01 AM         1       185      0.40      0.41      0.40         0
    10:05:01 AM         1       171      0.48      0.45      0.42         0
    10:10:01 AM         0       160      0.29      0.40      0.41         0
    10:15:01 AM         2       165      0.36      0.32      0.38         0
    10:20:01 AM         1       163      0.26      0.42      0.41         0
    10:25:01 AM         0       159      0.09      0.25      0.35         0
    10:30:01 AM         2       174      0.24      0.31      0.35         0
    10:35:01 AM         0       159      0.31      0.31      0.34         0
    10:40:01 AM         0       159      0.39      0.39      0.39         0
    Average:            1       165      0.34      0.34      0.34         0
    
    10:41:35 AM       LINUX RESTART
    
    10:45:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
    10:50:01 AM         0       161      0.38      0.37      0.39         0
    10:55:01 AM         0       161      0.19      0.26      0.34         0
    11:00:01 AM         0       186      0.45      0.35      0.34         0
    11:05:01 AM         0       165      0.11      0.30      0.33         0
    11:10:02 AM         0       165      0.64      0.38      0.35         0
    11:15:01 AM         1       166      0.27      0.26      0.30         0
    Average:            0       167      0.34      0.32      0.34         0
    
    11:15:27 AM       LINUX RESTART
    
    11:17:30 AM       LINUX RESTART
    
    11:17:38 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
    11:17:43 AM         1       170      0.37      0.30      0.32         0
    11:17:55 AM         1       172      0.39      0.31      0.32         0
    11:18:11 AM         1       173      0.53      0.34      0.33         0
    11:18:54 AM         1       173      0.74      0.42      0.36         0
    11:20:01 AM         2       182      1.56      0.64      0.43         0
    11:21:48 AM         3       175      1.95      1.25      0.69         0
    11:25:01 AM         4       177      4.73      3.17      1.56         0
    11:30:01 AM         6       183      1.36      1.92      1.46         0
    11:34:12 AM         0       168      5.49      3.21      2.02         0
    11:34:28 AM         1       171      4.60      3.15      2.03         0
    11:35:01 AM         2       175      3.25      2.96      2.00         0
    11:40:01 AM         0       169      0.44      1.46      1.63         0
    11:45:01 AM         0       166      0.22      0.66      1.24         0
    11:50:01 AM         0       167      0.42      0.47      1.00         0
    11:55:01 AM         0       167      0.28      0.38      0.81         0
    12:00:01 PM         3       200      0.54      0.35      0.67         0
    12:05:01 PM         0       173      0.58      0.38      0.59         0
    12:10:01 PM         0       171      0.24      0.32      0.51         0
    12:15:01 PM         0       180      0.43      0.38      0.49         0
    12:20:01 PM         0       165      0.23      0.32      0.44         0
    12:25:01 PM         1       169      0.26      0.27      0.38         0
    12:30:01 PM         5       186      0.17      0.29      0.37         0
    12:35:01 PM         0       165      0.36      0.39      0.39         0
    12:40:01 PM         1       165      0.18      0.31      0.37         0
    12:45:01 PM         0       159      0.18      0.27      0.34         0
    12:50:01 PM         0       159      0.36      0.28      0.32         0
    12:55:01 PM         0       159      0.26      0.24      0.30         0
    01:00:01 PM         1       192      0.27      0.27      0.30         0
    01:05:01 PM         1       166      0.22      0.23      0.27         0
    01:10:01 PM         1       163      0.16      0.24      0.28         0
    01:15:01 PM         0       159      0.30      0.31      0.31         0
    01:20:01 PM         0       159      0.53      0.33      0.32         0
    01:25:02 PM         1       165      0.43      0.36      0.33         0
    01:30:01 PM         0       163      0.31      0.38      0.35         0
    01:35:01 PM         2       164      0.42      0.42      0.38         0
    01:40:01 PM         0       163      0.27      0.35      0.37         0
    01:45:01 PM         0       163      0.26      0.29      0.33         0
    01:50:01 PM         0       169      0.18      0.24      0.30         0
    01:55:01 PM         0       164      0.38      0.32      0.32         0
    02:00:01 PM         5       184      0.29      0.38      0.37         0
    02:05:01 PM         0       163      0.36      0.30      0.33         0
    02:10:01 PM         1       163      0.63      0.39      0.34         0
    Average:            1       170      0.84      0.70      0.63         0
    
    02:12:10 PM       LINUX RESTART
    
    02:15:01 PM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
    02:20:01 PM         1       163      0.21      0.29      0.31         0
    02:25:01 PM         0       159      0.36      0.30      0.31         0
    02:30:01 PM         3       185      0.38      0.36      0.33         0
    02:35:01 PM         0       159      0.28      0.30      0.31         0
    02:40:01 PM         1       165      0.18      0.35      0.34         0
    02:45:01 PM         0       159      0.30      0.29      0.32         0
    02:50:01 PM         0       159      0.18      0.25      0.30         0
    02:55:01 PM         1       163      0.44      0.37      0.34         0
    03:00:01 PM         0       186      0.58      0.40      0.34         0
    03:05:01 PM         0       159      0.29      0.32      0.32         0
    03:10:01 PM         1       163      0.49      0.38      0.33         0
    03:15:01 PM         2       163      0.32      0.40      0.36         0
    03:20:01 PM         0       165      0.53      0.34      0.33         0
    03:25:01 PM         0       165      0.14      0.21      0.28         0
    03:30:01 PM         0       173      0.15      0.21      0.26         0
    03:35:01 PM         1       164      0.33      0.25      0.26         0
    03:40:02 PM         0       159      0.17      0.22      0.24         0
    03:45:01 PM         0       165      0.38      0.35      0.30         0
    03:50:01 PM         0       159      0.22      0.28      0.29         0
    03:55:01 PM         0       160      0.44      0.34      0.31         0
    04:00:01 PM         1       190      0.15      0.30      0.31         0
    04:05:01 PM         0       159      0.16      0.30      0.31         0
    04:10:01 PM         4       167      0.22      0.28      0.31         0
    04:15:01 PM         1       165      0.38      0.30      0.32         0
    Average:            1       166      0.30      0.31      0.31         0
    
    ------------------------------------------------------------------
    sar -r | sed -e "s|$(hostname)|hostname|g"
    Linux 3.10.0-957.21.3.el7.x86_64 (hostname)     06/24/2019     _x86_64_    (4 CPU)
    
    12:00:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
    12:05:01 AM    244980   7763876     96.94     92160     76424   6761856     74.66    719504    891624       172
    12:10:01 AM    174544   7834312     97.82     88244     76716   6761852     74.66    717440    889952       140
    12:15:01 AM    168776   7840080     97.89     85024     75516   6761852     74.66    715680    887080       180
    12:20:01 AM    234920   7773936     97.07     68716     72984   6761852     74.66    707392    876536       148
    12:25:01 AM    179680   7829176     97.76     63620     73376   6762880     74.67    707920    872904       128
    12:30:01 AM    172072   7836784     97.85     51180     72648   6794972     75.02    713376    864264       160
    12:35:01 AM    156536   7852320     98.05     46312     69916   6762876     74.67    703092    856956        48
    12:40:01 AM    184228   7824628     97.70     34844     69240   6762876     74.67    698864    848940       148
    12:45:01 AM    156604   7852252     98.04     25264     76872   6798660     75.06    717464    855232       192
    12:50:01 AM    186164   7822692     97.68     17676     75052   6771072     74.76    687852    848284       168
    12:55:01 AM    210024   7798832     97.38     13168     68772   6771020     74.76    686088    839460        24
    01:00:01 AM    158552   7850304     98.02     13020     78160   6833488     75.45    708348    843972       204
    01:05:01 AM    190204   7818652     97.63     12404     78000   6787112     74.93    700828    837660        80
    01:10:01 AM    211968   7796888     97.35     12564     66668   6771072     74.76    688952    833204       148
    01:15:01 AM    189524   7819332     97.63     11592     63916   6771068     74.76    686716    831840       164
    01:20:01 AM    220904   7787952     97.24     11248     65836   6771068     74.76    691172    828700       152
    01:25:02 AM    225548   7783308     97.18     11012     60756   6784948     74.91    691308    826768       124
    01:30:01 AM    166260   7842596     97.92     11660     69644   6803104     75.11    698816    834880       188
    01:35:01 AM    166948   7841908     97.92     11408     67508   6804848     75.13    722672    831332        80
    01:40:01 AM    212804   7796052     97.34     11756     60564   6771036     74.76    688028    826672       164
    01:45:01 AM    183396   7825460     97.71     11496     66604   6804848     75.13    725496    827304       172
    01:50:01 AM    207960   7800896     97.40     11520     67980   6783908     74.90    694936    828544       184
    01:55:01 AM    286788   7722068     96.42     11164     65220   6771036     74.76    695180    822068       184
    02:00:01 AM    118812   7890044     98.52     11600     75368   6874288     75.90    730720    831312       236
    02:05:01 AM    205260   7803596     97.44     12876     79904   6771368     74.76    694112    834572       172
    02:10:01 AM    202880   7805976     97.47     13132     75940   6785248     74.91    698604    831788       184
    02:15:01 AM    203600   7805256     97.46     12740     72728   6771372     74.76    692860    829644       200
    02:20:01 AM    202916   7805940     97.47     11932     70088   6771340     74.76    690792    828088       176
    02:25:01 AM    182440   7826416     97.72     11692     68644   6771372     74.76    691916    825088       204
    02:30:01 AM    115244   7893612     98.56     11884     70180   6871768     75.87    765484    827260       160
    02:35:01 AM    160816   7848040     97.99     12040     68096   6805148     75.13    724492    824084       152
    02:40:01 AM    201464   7807392     97.48     11328     63380   6785212     74.91    694804    818400       148
    02:45:01 AM    197548   7811308     97.53     11552     63292   6771364     74.76    690000    819192       124
    02:50:01 AM    220572   7788284     97.25     11504     62528   6771340     74.76    689052    818620       180
    02:55:01 AM    221728   7787128     97.23     11700     62592   6775560     74.81    695176    815644       192
    03:00:01 AM    160600   7848256     97.99     11408     69932   6819508     75.29    702796    823336       180
    03:05:01 AM    161896   7846960     97.98     12376     74308   6805656     75.14    727012    824176       160
    03:10:01 AM    192268   7816588     97.60     12856     66456   6771872     74.77    693136    817512       188
    03:15:01 AM    155744   7853112     98.06     71876     91708   6771372     74.76    730208    867844       148
    03:20:01 AM    205064   7803792     97.44     64252     82856   6771372     74.76    728272    853456       172
    03:25:01 AM    201988   7806868     97.48     60788     75348   6771368     74.76    720208    850692       184
    03:30:02 AM    184568   7824288     97.70     54844     69968   6803456     75.11    715532    853824       208
    03:35:01 AM    237096   7771760     97.04     41748     68328   6771368     74.76    697588    847056       840
    03:40:01 AM    137588   7871268     98.28     39764     64196   6838936     75.51    760396    844056       140
    03:45:01 AM    179840   7829016     97.75     33096     65400   6771796     74.77    695436    837592        48
    03:50:01 AM    185920   7822936     97.68     24256     63468   6771368     74.76    694888    827116        48
    03:55:01 AM    174796   7834060     97.82     18544     64552   6771368     74.76    694684    822808       180
    04:00:01 AM    156888   7851968     98.04     13588     70808   6821980     75.32    707208    823656       136
    04:05:01 AM    228080   7780776     97.15     12024     71516   6771340     74.76    697652    820068       204
    04:10:01 AM    256192   7752664     96.80     12092     63184   6771372     74.76    691548    817808       180
    04:15:01 AM    190944   7817912     97.62     12220     61720   6771368     74.76    693388    814364       180
    04:20:01 AM    151672   7857184     98.11     11908     63884   6815904     75.25    734132    810888       240
    04:25:01 AM    157580   7851276     98.03     11720     62212   6782120     74.88    700844    809388       108
    04:30:01 AM    194016   7814840     97.58     11180     67360   6799760     75.07    704516    813588       148
    04:35:01 AM    217160   7791696     97.29     11352     65996   6782128     74.88    704056    808392       220
    04:40:01 AM    225428   7783428     97.19     11548     61336   6782088     74.88    700024    806792       112
    04:45:01 AM    156936   7851920     98.04     11604     60448   6805148     75.13    731904    804048       172
    04:50:01 AM    221816   7787040     97.23     11296     60996   6782132     74.88    702256    802980        44
    04:55:01 AM    240492   7768364     97.00     11556     60144   6785488     74.92    702508    803196       156
    05:00:01 AM    149876   7858980     98.13     11540     67572   6848988     75.62    747136    807724       176
    05:05:01 AM    258124   7750732     96.78     12072     63868   6782128     74.88    702996    804968       184
    05:10:01 AM    181868   7826988     97.73     12676     64848   6782132     74.88    703432    805768       196
    05:15:01 AM    240384   7768472     97.00     11280     73688   6782100     74.88    708068    807576       172
    05:20:01 AM    151148   7857708     98.11     12052     73840   6796304     75.04    712012    807772       192
    05:25:01 AM    204480   7804376     97.45     12408     66476   6782124     74.88    702340    806928       120
    05:30:01 AM    192156   7816700     97.60     11868     63212   6814220     75.23    712680    802484       140
    05:35:01 AM    245500   7763356     96.93     11244     64248   6784956     74.91    707956    798328       176
    05:40:01 AM    199824   7809032     97.50     11476     62504   6796004     75.03    707748    799544       156
    05:45:01 AM    201664   7807192     97.48     11440     60712   6782132     74.88    703896    797656       108
    05:50:01 AM    201368   7807488     97.49     11008     63748   6782100     74.88    706912    797128       164
    05:55:01 AM    204408   7804448     97.45     11316     59532   6782128     74.88    704660    795076         0
    06:00:01 AM    128216   7880640     98.40     11376     67960   6830264     75.41    722736    800372       120
    06:05:01 AM    194924   7813932     97.57     11916     67128   6782100     74.88    704108    803700       116
    06:10:01 AM    201668   7807188     97.48     12044     66520   6782128     74.88    706244    800596       172
    06:15:01 AM    260968   7747888     96.74     11252     61608   6782096     74.88    705520    795248       152
    06:20:01 AM    225744   7783112     97.18     11724     63812   6782128     74.88    706376    797100       144
    06:25:01 AM    185576   7823280     97.68     11612     63088   6796004     75.03    711356    794652       160
    06:30:01 AM    165944   7842912     97.93     11888     69492   6812152     75.21    715204    801384       252
    06:35:01 AM    206656   7802200     97.42     11596     67444   6789988     74.97    707176    798752       172
    06:40:01 AM    190868   7817988     97.62     11936     63004   6782128     74.88    705924    795412       172
    06:45:01 AM    189760   7819096     97.63     11784     63308   6782096     74.88    705924    795080       240
    06:50:01 AM    154380   7854476     98.07     11672     64724   6815908     75.25    743040    791052       168
    06:55:01 AM    187448   7821408     97.66     11484     64528   6782128     74.88    706844    792640       108
    07:00:01 AM    141448   7867408     98.23     11348     68256   6852112     75.65    743608    802664       136
    07:05:01 AM    195536   7813320     97.56     12104     71056   6782128     74.88    700564    805524       140
    07:10:01 AM    187212   7821644     97.66     12216     67760   6782128     74.88    696816    805292       204
    07:15:01 AM    178684   7830172     97.77     11516     63472   6815916     75.25    727548    802248       156
    07:20:01 AM    213152   7795704     97.34     11544     63204   6796612     75.04    699288    799128       144
    07:25:01 AM    213696   7795160     97.33     11380     64184   6795736     75.03    700216    798200        52
    07:30:01 AM    133264   7875592     98.34     11708     67800   6847248     75.60    735904    804616        72
    07:35:01 AM    205192   7803664     97.44     11376     64736   6796008     75.03    697740    801212       164
    07:40:01 AM    176496   7832360     97.80     11848     63132   6810084     75.19    701012    798976        76
    07:45:01 AM    159340   7849516     98.01     11492     62320   6815912     75.25    730656    793600        44
    07:50:01 AM    227888   7780968     97.15     11288     61348   6795976     75.03    699348    793152       160
    07:55:01 AM    219852   7789004     97.25     11444     62048   6796004     75.03    699868    792812       140
    08:00:01 AM    133200   7875656     98.34     11852     67200   6868436     75.83    746668    796956       208
    08:05:01 AM    220248   7788608     97.25     11620     67720   6782128     74.88    699676    794112       148
    08:10:01 AM    172788   7836068     97.84     11568     62808   6815912     75.25    731912    789348       192
    08:15:01 AM    194176   7814680     97.58     11340     73064   6782076     74.88    696672    801540       140
    08:20:01 AM    195076   7813780     97.56     11592     71532   6782264     74.88    700236    796436       160
    08:25:01 AM    175424   7833432     97.81     14908     68416   6782132     74.88    697688    798744       196
    08:30:01 AM    172192   7836664     97.85     12180     65100   6855088     75.68    759196    800024        60
    08:35:01 AM    181488   7827368     97.73     12484     70100   6782100     74.88    700408    795232       188
    08:40:01 AM    152576   7856280     98.09     12896     69752   6808000     75.16    726344    793928       196
    08:45:01 AM    198660   7810196     97.52     12380     64728   6782132     74.88    695796    792792       240
    08:50:01 AM    189384   7819472     97.64     12236     62224   6782128     74.88    695796    789564       196
    08:55:01 AM    166436   7842420     97.92     11956     64576   6782128     74.88    698488    788328       128
    09:00:02 AM    167648   7841208     97.91     11840     68104   6827480     75.38    711028    790232       208
    09:05:01 AM    224232   7784624     97.20     11828     66872   6782128     74.88    696140    790532       228
    09:10:01 AM    212796   7796060     97.34     11880     65212   6782128     74.88    698004    786560       256
    09:15:01 AM    182140   7826716     97.73     11684     63780   6796004     75.03    701172    784820       168
    09:20:01 AM    223920   7784936     97.20     11460     63916   6782100     74.88    703144    780628       180
    09:25:01 AM    190428   7818428     97.62     11504     62224   6782136     74.88    701364    779688       156
    09:30:01 AM    199988   7808868     97.50     12184     67992   6802172     75.10    709580    784772       176
    09:35:01 AM    187632   7821224     97.66     11680     68644   6815908     75.25    735048    784100       196
    09:40:01 AM    202424   7806432     97.47     12232     64456   6796008     75.03    703900    781800       216
    09:45:01 AM    180592   7828264     97.75     13512     63472   6782136     74.88    701012    781204       144
    09:50:01 AM    155208   7853648     98.06     13596     64196   6811928     75.21    731304    779844       176
    09:55:01 AM    186200   7822656     97.68     12800     64160   6782128     74.88    701748    778952       216
    10:00:01 AM    125956   7882900     98.43     12312     67548   6864048     75.78    751340    779352       188
    10:05:01 AM    174164   7834692     97.83     14104     70132   6796180     75.03    706684    782452       132
    10:10:01 AM    204164   7804692     97.45     14036     67096   6782128     74.88    703352    779204       184
    10:15:01 AM    142316   7866540     98.22     14112     65252   6815912     75.25    733432    778416       176
    10:20:01 AM    157448   7851408     98.03     13048     63960   6805148     75.13    731976    775392       180
    10:25:01 AM    202536   7806320     97.47     12800     63888   6771372     74.76    698596    774148       200
    10:30:01 AM    157364   7851492     98.04     11960     69156   6801268     75.09    705628    778780        48
    10:35:01 AM    192116   7816740     97.60     11752     67436   6771380     74.76    699820    774436       184
    10:40:01 AM    239180   7769676     97.01     11624     62296   6771332     74.76    694696    773520       148
    Average:       190094   7818762     97.63     17874     67269   6792272     74.99    708719    812052       164
    
    10:41:35 AM       LINUX RESTART
    
    10:45:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
    10:50:01 AM    272092   7736764     96.60     89084    420264   6783168     74.89    863928   1044336       160
    10:55:01 AM    245464   7763392     96.94     86604    416784   6783244     74.89    854824   1047768       200
    11:00:01 AM    182852   7826004     97.72     78092    423600   6841760     75.54    867848   1060936       256
    11:05:01 AM    191088   7817768     97.61     65360    422856   6793624     75.01    842712   1057568       188
    11:10:02 AM    216840   7792016     97.29     57520    410512   6793620     75.01    835300   1044780       208
    11:15:01 AM    197040   7811816     97.54     45068    412624   6791780     74.99    822088   1044596        84
    Average:       217563   7791293     97.28     70288    417773   6797866     75.05    847783   1049997       183
    
    11:15:27 AM       LINUX RESTART
    
    11:17:30 AM       LINUX RESTART
    
    11:17:38 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
    11:17:43 AM    295456   7713400     96.31     47384    443800   6849144     75.62    895200   1049872      1964
    11:17:55 AM    281588   7727268     96.48     47976    452496   6862980     75.77    903052   1051528      2760
    11:18:11 AM    297400   7711456     96.29     48032    456308   6845672     75.58    880216   1051936      1752
    11:18:54 AM    279588   7729268     96.51     55812    510788   6873736     75.89   1025948    996080     24660
    11:20:01 AM    310320   7698536     96.13     70124    824620   6834364     75.46   1084972   1235776     68212
    11:21:48 AM    621596   7387260     92.24     71208    956216   6872276     75.87   1190964   1294180      2004
    11:25:01 AM   1919240   6089616     76.04     39904    678984   6874680     75.90    806000   1356152     10088
    11:30:01 AM   1547640   6461216     80.68     42548    870932   6941080     76.63   1018336   1401700        12
    11:34:12 AM   1627260   6381596     79.68     46260    781364   6849744     75.63    895892   1358744     70944
    11:34:28 AM   1491936   6516920     81.37     47848    875516   6863804     75.78    943420   1435960    103928
    11:35:01 AM   1482316   6526540     81.49     49660    894148   6859264     75.73    949644   1422912     31928
    11:40:01 AM   1353016   6655840     83.11     51700    914524   6838644     75.50   1001780   1390664       132
    11:45:01 AM   1270936   6737920     84.13     51964    914840   6820512     75.30   1001576   1390852       140
    11:50:01 AM   1161536   6847320     85.50     52408    916280   6838644     75.50   1002644   1392112       180
    11:55:01 AM   1056844   6952012     86.80     53064    915876   6838648     75.50   1003076   1391884       180
    12:00:01 PM    878844   7130012     89.03     53592    917256   6930332     76.52   1057148   1393180        60
    12:05:01 PM    821764   7187092     89.74     53968    917936   6844884     75.57   1008548   1393608       184
    12:10:01 PM    716072   7292784     91.06     54332    918332   6844888     75.57   1008876   1393968       192
    12:15:01 PM    462480   7546376     94.23     55904    968528   6941832     76.64   1143972   1396660      1744
    12:20:01 PM    489928   7518928     93.88     57964    969120   6771708     74.76   1037568   1375476       148
    12:25:01 PM    343012   7665844     95.72     58196    969500   6805456     75.14   1071212   1375572       204
    12:30:01 PM    196668   7812188     97.54     58428    969932   6857532     75.71   1109924   1375724       184
    12:35:01 PM    224880   7783976     97.19     58496    958436   6770256     74.75   1027032   1375128        40
    12:40:01 PM    220984   7787872     97.24     58784    940748   6762824     74.67    986116   1382248       128
    12:45:01 PM    175528   7833328     97.81     59100    928940   6750116     74.53    966644   1387936       160
    12:50:01 PM    234204   7774652     97.08     59460    897864   6750116     74.53    924720   1399136       164
    12:55:01 PM    193308   7815548     97.59     59756    885464   6750112     74.53    909128   1402552       160
    01:00:01 PM    167924   7840932     97.90     60052    866696   6803028     75.11    903812   1406412       144
    01:05:01 PM    215824   7793032     97.31     56416    844952   6762928     74.67    860644   1409580       156
    01:10:01 PM    171992   7836864     97.85     56356    832648   6765420     74.69    859740   1410256       136
    01:15:01 PM    183168   7825688     97.71     56216    814028   6750116     74.53    829088   1407504        36
    01:20:01 PM    285116   7723740     96.44     49956    783508   6750080     74.53    808560   1391416       180
    01:25:02 PM    246336   7762520     96.92     50148    773256   6752608     74.55    804732   1386512        64
    01:30:01 PM    207256   7801600     97.41     50008    762096   6756208     74.59    799760   1380812        96
    01:35:01 PM    163020   7845836     97.96     47080    753560   6761680     74.65    794272   1374912       208
    01:40:01 PM    214944   7793912     97.32     46820    725468   6756960     74.60    785160   1358884       160
    01:45:01 PM    204652   7804204     97.44     46556    709288   6756956     74.60    778040   1349632       104
    01:50:01 PM    184496   7824360     97.70     46880    692944   6770828     74.75    772048   1343384       148
    01:55:01 PM    240316   7768540     97.00     47140    665840   6756956     74.60    753620   1331220        96
    02:00:01 PM    194052   7814804     97.58     47004    655244   6792396     74.99    758404   1325888       176
    02:05:01 PM    194324   7814532     97.57     44828    642464   6756948     74.60    740640   1318416       160
    02:10:01 PM    151344   7857512     98.11     45024    639804   6783896     74.90    767032   1317172       168
    Average:       547122   7461734     93.17     52723    805013   6812387     75.21    925456   1342465      7721
    
    02:12:10 PM       LINUX RESTART
    
    02:15:01 PM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
    02:20:01 PM    280684   7728172     96.50     54888    672276   6752384     74.55    733328   1361452        24
    02:25:01 PM    171416   7837440     97.86     55204    672680   6750248     74.53    733020   1361676       108
    02:30:01 PM    168476   7840380     97.90     53496    654508   6784280     74.90    725404   1359228       204
    02:35:01 PM    199204   7809652     97.51     52956    635072   6750248     74.53    702156   1352716       176
    02:40:01 PM    210684   7798172     97.37     52480    615920   6764128     74.68    695292   1343488       104
    02:45:01 PM    227452   7781404     97.16     51968    599432   6750244     74.53    682268   1335720       140
    02:50:01 PM    176992   7831864     97.79     51952    590720   6750244     74.53    678904   1330740       108
    02:55:01 PM    154992   7853864     98.06     51296    574824   6782052     74.88    702024   1321852        28
    03:00:01 PM    163760   7845096     97.96     51132    564320   6812252     75.21    687424   1313352       136
    03:05:01 PM    190692   7818164     97.62     50500    549088   6750248     74.53    657836   1308500       156
    03:10:01 PM    161532   7847324     97.98     50044    532452   6784028     74.90    681084   1300496        36
    03:15:01 PM    221568   7787288     97.23     47988    513884   6751528     74.54    639324   1290168       164
    03:20:01 PM    239140   7769716     97.01     41552    500276   6764120     74.68    632008   1280128       172
    03:25:01 PM    182648   7826208     97.72     37184    497328   6764120     74.68    628144   1276448       120
    03:30:01 PM    380260   7628596     95.25     32744    488996   6782340     74.88    628424   1269980       164
    03:35:01 PM    250440   7758416     96.87     33060    489332   6784028     74.90    652052   1270256       192
    03:40:02 PM    179472   7829384     97.76     33400    489660   6750248     74.53    619364   1270580       160
    03:45:01 PM    189772   7819084     97.63     28400    477372   6764120     74.68    613700   1262160       116
    03:50:01 PM    203864   7804992     97.45     28664    461564   6750244     74.53    602716   1254336       156
    03:55:01 PM    198396   7810460     97.52     28956    446656   6750248     74.53    595264   1247000       152
    04:00:01 PM    172192   7836664     97.85     29224    432452   6815644     75.25    607980   1240108       120
    04:05:01 PM    188712   7820144     97.64     29636    418960   6750244     74.53    582040   1233444       128
    04:10:01 PM    122876   7885980     98.47     29908    409464   6799476     75.07    624520   1228864       160
    04:15:01 PM    157048   7851808     98.04     30056    395484   6787448     74.94    605852   1222028       220
    Average:       199678   7809178     97.51     41945    528447   6768507     74.73    654589   1293113       135
    
    ------------------------------------------------------------------
    sar -u | sed -e "s|$(hostname)|hostname|g"
    Linux 3.10.0-957.21.3.el7.x86_64 (hostname)     06/24/2019     _x86_64_    (4 CPU)
    
    12:00:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
    12:05:01 AM     all      3.39      0.00      7.13      0.03      0.02     89.43
    12:10:01 AM     all      2.76      0.00      5.86      0.01      0.02     91.35
    12:15:01 AM     all      3.11      0.00      6.67      0.02      0.02     90.18
    12:20:01 AM     all      2.86      0.00      6.25      0.02      0.02     90.85
    12:25:01 AM     all      3.17      0.00      6.76      0.02      0.02     90.04
    12:30:01 AM     all      2.82      0.00      6.14      0.01      0.02     91.02
    12:35:01 AM     all      3.36      0.00      7.30      0.02      0.02     89.30
    12:40:01 AM     all      2.94      0.00      6.50      0.02      0.02     90.53
    12:45:01 AM     all      3.04      0.00      6.69      0.02      0.02     90.24
    12:50:01 AM     all      2.93      0.00      6.46      0.02      0.02     90.58
    12:55:01 AM     all      3.01      0.00      7.07      0.02      0.02     89.89
    01:00:01 AM     all      2.94      0.00      6.46      0.02      0.01     90.56
    01:05:01 AM     all      3.30      0.00      7.23      0.02      0.02     89.43
    01:10:01 AM     all      2.93      0.00      6.42      0.02      0.02     90.62
    01:15:01 AM     all      3.03      0.00      6.60      0.01      0.02     90.34
    01:20:01 AM     all      2.76      0.00      6.21      0.02      0.02     90.99
    01:25:02 AM     all      3.04      0.00      6.73      0.02      0.02     90.19
    01:30:01 AM     all      2.92      0.00      6.51      0.02      0.02     90.54
    01:35:01 AM     all      3.36      0.00      7.39      0.02      0.02     89.22
    01:40:01 AM     all      2.75      0.00      6.17      0.02      0.02     91.04
    01:45:01 AM     all      2.94      0.00      6.60      0.02      0.02     90.42
    01:50:01 AM     all      2.78      0.00      6.15      0.02      0.02     91.03
    01:55:01 AM     all      3.03      0.00      6.72      0.02      0.02     90.21
    02:00:01 AM     all      2.83      0.00      6.21      0.01      0.02     90.92
    02:05:01 AM     all      4.57      0.00      9.69      0.02      0.04     85.68
    02:10:01 AM     all      2.77      0.00      6.15      0.02      0.02     91.04
    02:15:01 AM     all      3.18      0.00      6.98      0.02      0.02     89.80
    02:20:01 AM     all      2.75      0.00      6.24      0.02      0.02     90.98
    02:25:01 AM     all      2.98      0.00      6.77      0.02      0.02     90.21
    02:30:01 AM     all      3.06      0.00      6.91      0.03      0.02     89.98
    02:35:01 AM     all      3.19      0.00      7.15      0.02      0.02     89.62
    02:40:01 AM     all      2.96      0.00      6.74      0.03      0.03     90.24
    02:45:01 AM     all      2.91      0.00      6.67      0.02      0.02     90.38
    02:50:01 AM     all      2.95      0.00      6.82      0.03      0.02     90.18
    02:55:01 AM     all      2.91      0.00      6.70      0.02      0.02     90.35
    03:00:01 AM     all      2.87      0.00      6.60      0.02      0.02     90.50
    03:05:01 AM     all      3.48      0.00      7.68      0.06      0.03     88.75
    03:10:01 AM     all      2.95      0.00      6.74      0.01      0.02     90.27
    03:15:01 AM     all      2.97      0.05      6.93      0.21      0.02     89.81
    03:20:01 AM     all      2.93      0.00      6.80      0.02      0.02     90.23
    03:25:01 AM     all      2.92      0.00      6.67      0.02      0.02     90.37
    03:30:02 AM     all      2.94      0.00      6.71      0.02      0.03     90.31
    03:35:01 AM     all      3.24      0.00      7.54      0.02      0.02     89.18
    03:40:01 AM     all      2.97      0.00      6.83      0.02      0.02     90.16
    03:45:01 AM     all      2.95      0.00      6.90      0.02      0.02     90.11
    03:50:01 AM     all      2.91      0.00      6.79      0.02      0.02     90.27
    03:55:01 AM     all      2.94      0.00      6.87      0.02      0.02     90.15
    04:00:01 AM     all      3.05      0.00      7.03      0.02      0.02     89.88
    04:05:01 AM     all      3.17      0.00      7.40      0.02      0.03     89.38
    04:10:01 AM     all      2.78      0.00      6.54      0.02      0.02     90.63
    04:15:01 AM     all      2.94      0.00      6.92      0.02      0.02     90.10
    04:20:01 AM     all      2.83      0.00      6.73      0.02      0.02     90.39
    04:25:01 AM     all      2.89      0.00      6.85      0.02      0.02     90.22
    04:30:01 AM     all      2.92      0.00      6.75      0.02      0.02     90.29
    04:35:01 AM     all      3.20      0.00      7.52      0.02      0.02     89.24
    04:40:01 AM     all      2.94      0.00      6.89      0.03      0.03     90.12
    04:45:01 AM     all      3.03      0.00      7.15      0.02      0.02     89.78
    04:50:01 AM     all      2.82      0.00      6.58      0.02      0.02     90.56
    04:55:01 AM     all      2.85      0.00      6.75      0.02      0.02     90.35
    05:00:01 AM     all      3.01      0.00      7.07      0.02      0.02     89.88
    05:05:01 AM     all      3.17      0.00      7.51      0.03      0.03     89.26
    05:10:01 AM     all      2.82      0.00      6.81      0.02      0.02     90.33
    05:15:01 AM     all      2.97      0.00      7.02      0.02      0.02     89.96
    05:20:01 AM     all      2.77      0.00      6.76      0.03      0.02     90.42
    05:25:01 AM     all      3.02      0.00      7.56      0.03      0.03     89.37
    05:30:01 AM     all      2.85      0.00      6.86      0.05      0.02     90.23
    05:35:01 AM     all      3.19      0.00      7.64      0.03      0.03     89.11
    05:40:01 AM     all      2.96      0.00      7.09      0.02      0.02     89.90
    05:45:01 AM     all      2.93      0.00      7.16      0.02      0.02     89.87
    05:50:01 AM     all      2.76      0.00      6.77      0.02      0.02     90.43
    05:55:01 AM     all      2.83      0.00      6.86      0.02      0.02     90.27
    06:00:01 AM     all      2.84      0.00      6.90      0.02      0.02     90.21
    06:05:01 AM     all      3.23      0.00      7.89      0.03      0.05     88.81
    06:10:01 AM     all      2.79      0.00      6.87      0.02      0.02     90.29
    06:15:01 AM     all      3.04      0.00      7.46      0.02      0.03     89.45
    06:20:01 AM     all      2.70      0.00      6.54      0.02      0.02     90.72
    06:25:01 AM     all      3.02      0.00      7.49      0.02      0.02     89.44
    06:30:01 AM     all      2.80      0.00      6.92      0.02      0.02     90.23
    06:35:01 AM     all      3.16      0.00      7.75      0.03      0.03     89.03
    06:40:01 AM     all      2.86      0.00      7.08      0.02      0.02     90.01
    06:45:01 AM     all      2.98      0.00      7.35      0.02      0.02     89.63
    06:50:01 AM     all      2.78      0.00      6.93      0.02      0.02     90.25
    06:55:01 AM     all      2.98      0.00      7.40      0.02      0.02     89.57
    07:00:01 AM     all      2.82      0.00      7.03      0.02      0.02     90.10
    07:05:01 AM     all      3.46      0.00      8.53      0.02      0.04     87.96
    07:10:01 AM     all      2.83      0.00      6.98      0.02      0.02     90.16
    07:15:01 AM     all      3.00      0.00      7.52      0.02      0.02     89.43
    07:20:01 AM     all      2.77      0.00      6.94      0.02      0.02     90.24
    07:25:01 AM     all      3.11      0.00      7.71      0.02      0.04     89.12
    07:30:01 AM     all      2.67      0.00      6.60      0.02      0.03     90.68
    07:35:01 AM     all      3.23      0.00      8.04      0.02      0.03     88.68
    07:40:01 AM     all      2.84      0.00      7.15      0.02      0.02     89.97
    07:45:01 AM     all      3.08      0.00      7.74      0.02      0.02     89.13
    07:50:01 AM     all      2.78      0.00      7.01      0.02      0.02     90.17
    07:55:01 AM     all      3.04      0.00      7.63      0.02      0.02     89.28
    08:00:01 AM     all      2.87      0.00      7.14      0.02      0.03     89.94
    08:05:01 AM     all      3.25      0.00      8.20      0.03      0.03     88.49
    08:10:01 AM     all      2.78      0.00      6.93      0.02      0.02     90.24
    08:15:01 AM     all      3.21      0.00      7.83      0.03      0.02     88.90
    08:20:01 AM     all      2.74      0.00      6.98      0.03      0.02     90.23
    08:25:01 AM     all      3.09      0.00      7.61      0.03      0.02     89.25
    08:30:01 AM     all      2.83      0.00      7.09      0.02      0.03     90.03
    08:35:01 AM     all      3.21      0.00      8.01      0.02      0.03     88.73
    08:40:01 AM     all      2.91      0.00      7.38      0.03      0.02     89.66
    08:45:01 AM     all      2.97      0.00      7.50      0.02      0.02     89.49
    08:50:01 AM     all      2.79      0.00      7.11      0.02      0.02     90.06
    08:55:01 AM     all      3.06      0.00      7.74      0.02      0.02     89.16
    09:00:02 AM     all      2.76      0.00      6.91      0.02      0.03     90.28
    09:05:01 AM     all      3.44      0.00      8.65      0.02      0.03     87.86
    09:10:01 AM     all      2.78      0.00      7.04      0.04      0.02     90.12
    09:15:01 AM     all      2.89      0.00      7.32      0.02      0.02     89.74
    09:20:01 AM     all      2.79      0.00      7.11      0.02      0.02     90.06
    09:25:01 AM     all      3.03      0.00      7.65      0.03      0.02     89.26
    09:30:01 AM     all      2.88      0.00      7.25      0.04      0.02     89.81
    09:35:01 AM     all      3.16      0.00      8.03      0.03      0.02     88.76
    09:40:01 AM     all      3.05      0.00      7.62      0.02      0.02     89.29
    09:45:01 AM     all      2.93      0.00      7.45      0.02      0.02     89.58
    09:50:01 AM     all      2.93      0.00      7.45      0.03      0.02     89.58
    09:55:01 AM     all      3.06      0.00      7.96      0.02      0.02     88.93
    10:00:01 AM     all      2.88      0.00      7.24      0.02      0.02     89.84
    10:05:01 AM     all      3.23      0.00      8.16      0.02      0.03     88.56
    10:10:01 AM     all      2.81      0.00      7.07      0.02      0.02     90.09
    10:15:01 AM     all      2.94      0.00      7.35      0.02      0.02     89.66
    10:20:01 AM     all      2.79      0.00      7.05      0.02      0.02     90.12
    10:25:01 AM     all      3.00      0.00      7.56      0.03      0.02     89.38
    10:30:01 AM     all      2.86      0.00      7.27      0.02      0.02     89.82
    10:35:01 AM     all      3.17      0.00      7.95      0.02      0.02     88.83
    10:40:01 AM     all      3.14      0.00      7.71      0.02      0.02     89.10
    Average:        all      2.98      0.00      7.10      0.02      0.02     89.87
    
    10:41:35 AM       LINUX RESTART
    
    10:45:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
    10:50:01 AM     all      2.86      0.00      6.57      0.03      0.02     90.52
    10:55:01 AM     all      2.98      0.00      6.94      0.02      0.02     90.03
    11:00:01 AM     all      3.02      0.00      6.92      0.02      0.02     90.01
    11:05:01 AM     all      3.40      0.00      7.76      0.02      0.03     88.80
    11:10:02 AM     all      2.96      0.00      6.88      0.02      0.02     90.12
    11:15:01 AM     all      3.10      0.00      6.96      0.02      0.02     89.90
    Average:        all      3.05      0.00      7.00      0.02      0.02     89.90
    
    11:15:27 AM       LINUX RESTART
    
    11:17:30 AM       LINUX RESTART
    
    11:17:38 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
    11:17:43 AM     all     28.08      0.00     13.61      0.00      0.06     58.25
    11:17:55 AM     all     19.94      0.00      5.39      0.10      0.04     74.52
    11:18:11 AM     all     27.47      0.00     14.50      0.07      0.07     57.89
    11:18:54 AM     all     18.93      0.00      7.74      0.51      0.04     72.78
    11:20:01 AM     all     11.74      0.00     12.18      0.34      0.04     75.70
    11:21:48 AM     all     26.00      0.00     22.81      0.11      0.08     51.00
    11:25:01 AM     all     69.72      0.00     25.41      0.09      0.12      4.65
    11:30:01 AM     all     20.21      0.00      7.50      0.09      0.04     72.16
    11:34:12 AM     all     47.09      0.00     23.19      0.03      0.10     29.59
    11:34:28 AM     all     13.82      0.00     25.00      0.16      0.08     60.94
    11:35:01 AM     all     19.20      0.00     10.40      0.21      0.07     70.12
    11:40:01 AM     all      7.35      0.00      7.46      0.03      0.04     85.11
    11:45:01 AM     all      2.07      0.00      3.93      0.01      0.01     93.97
    11:50:01 AM     all      3.00      0.00      5.76      0.02      0.02     91.21
    11:55:01 AM     all      3.21      0.00      6.13      0.01      0.02     90.63
    12:00:01 PM     all      3.24      0.00      6.33      0.02      0.02     90.40
    12:05:01 PM     all      3.41      0.00      6.86      0.02      0.02     89.69
    12:10:01 PM     all      2.96      0.00      6.03      0.02      0.02     90.97
    12:15:01 PM     all      3.25      0.00      6.53      0.02      0.02     90.18
    12:20:01 PM     all      2.78      0.00      5.73      0.02      0.01     91.46
    12:25:01 PM     all      3.04      0.00      6.35      0.02      0.02     90.58
    12:30:01 PM     all      2.91      0.00      6.11      0.01      0.01     90.96
    12:35:01 PM     all      2.96      0.00      6.37      0.02      0.02     90.64
    12:40:01 PM     all      2.86      0.00      6.15      0.02      0.02     90.96
    12:45:01 PM     all      3.10      0.00      6.61      0.02      0.02     90.26
    12:50:01 PM     all      2.81      0.00      5.97      0.01      0.02     91.19
    12:55:01 PM     all      2.87      0.00      6.21      0.02      0.02     90.89
    01:00:01 PM     all      2.86      0.00      6.14      0.01      0.01     90.97
    01:05:01 PM     all      3.25      0.00      6.90      0.01      0.02     89.82
    01:10:01 PM     all      2.61      0.00      5.67      0.01      0.02     91.69
    01:15:01 PM     all      3.02      0.00      6.47      0.02      0.01     90.48
    01:20:01 PM     all      2.82      0.00      6.11      0.02      0.02     91.04
    01:25:02 PM     all      2.83      0.00      6.21      0.02      0.01     90.92
    01:30:01 PM     all      2.78      0.00      6.12      0.02      0.01     91.08
    01:35:01 PM     all      3.20      0.00      6.78      0.01      0.02     89.97
    01:40:01 PM     all      2.86      0.00      6.22      0.02      0.02     90.88
    01:45:01 PM     all      3.13      0.00      6.75      0.02      0.02     90.08
    01:50:01 PM     all      2.68      0.00      6.08      0.02      0.02     91.20
    01:55:01 PM     all      3.00      0.00      6.50      0.01      0.02     90.47
    02:00:01 PM     all      2.85      0.00      6.11      0.01      0.02     91.01
    02:05:01 PM     all      3.29      0.00      7.17      0.02      0.02     89.51
    02:10:01 PM     all      2.74      0.00      5.99      0.02      0.02     91.24
    Average:        all      6.26      0.00      7.28      0.03      0.02     86.41
    
    02:12:10 PM       LINUX RESTART
    
    02:15:01 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
    02:20:01 PM     all      2.70      0.00      5.73      0.02      0.02     91.53
    02:25:01 PM     all      2.96      0.00      6.32      0.02      0.02     90.68
    02:30:01 PM     all      2.79      0.00      6.01      0.02      0.02     91.16
    02:35:01 PM     all      3.20      0.00      6.75      0.02      0.04     90.00
    02:40:01 PM     all      2.77      0.00      5.91      0.01      0.02     91.29
    02:45:01 PM     all      3.05      0.00      6.45      0.02      0.03     90.45
    02:50:01 PM     all      2.75      0.00      5.87      0.02      0.02     91.34
    02:55:01 PM     all      2.98      0.00      6.28      0.02      0.02     90.70
    03:00:01 PM     all      2.90      0.00      6.04      0.02      0.02     91.02
    03:05:01 PM     all      3.24      0.00      6.78      0.02      0.04     89.92
    03:10:01 PM     all      2.72      0.00      5.73      0.01      0.02     91.51
    03:15:01 PM     all      2.91      0.00      6.17      0.02      0.02     90.88
    03:20:01 PM     all      2.88      0.00      6.11      0.01      0.03     90.97
    03:25:01 PM     all      2.91      0.00      6.25      0.01      0.03     90.80
    03:30:01 PM     all      2.80      0.00      5.99      0.02      0.02     91.17
    03:35:01 PM     all      3.20      0.00      6.51      0.01      0.03     90.25
    03:40:02 PM     all      2.70      0.00      5.80      0.02      0.03     91.45
    03:45:01 PM     all      3.02      0.00      6.37      0.02      0.03     90.56
    03:50:01 PM     all      2.62      0.00      5.64      0.02      0.03     91.70
    03:55:01 PM     all      3.00      0.00      6.43      0.02      0.02     90.53
    04:00:01 PM     all      2.84      0.00      6.10      0.02      0.02     91.01
    04:05:01 PM     all      3.25      0.00      6.93      0.02      0.04     89.76
    04:10:01 PM     all      2.73      0.00      5.93      0.01      0.02     91.31
    04:15:01 PM     all      3.10      0.00      6.79      0.02      0.02     90.07
    Average:        all      2.92      0.00      6.20      0.02      0.03     90.84
    
    ------------------------------------------------------------------
    top -bHn1 -w200
    top - 16:20:00 up 1 day,  4:24,  1 user,  load average: 0.50, 0.35, 0.33
    Threads: 164 total,   1 running, 163 sleeping,   0 stopped,   0 zombie
    %Cpu(s):  1.4 us,  2.8 sy,  0.0 ni, 95.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    KiB Mem :  8008856 total,   236308 free,  1472920 used,  6299628 buff/cache
    KiB Swap:  1048572 total,  1025020 free,    23552 used.  6171576 avail Mem
    
      PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
    24824 root      20   0  162064   2180   1540 R  5.9  0.0   0:00.02 top
        1 root      20   0  191140   3408   2212 S  0.0  0.0   0:16.99 systemd
        2 root      20   0       0      0      0 S  0.0  0.0   0:00.01 kthreadd
        3 root      20   0       0      0      0 S  0.0  0.0   1:07.50 ksoftirqd/0
        5 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kworker/0:0H
        7 root      rt   0       0      0      0 S  0.0  0.0   0:08.84 migration/0
        8 root      20   0       0      0      0 S  0.0  0.0   0:00.00 rcu_bh
        9 root      20   0       0      0      0 S  0.0  0.0   1:22.39 rcu_sched
       10 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 lru-add-drain
       11 root      rt   0       0      0      0 S  0.0  0.0   0:00.42 watchdog/0
       12 root      rt   0       0      0      0 S  0.0  0.0   0:00.35 watchdog/1
       13 root      rt   0       0      0      0 S  0.0  0.0   0:08.89 migration/1
       14 root      20   0       0      0      0 S  0.0  0.0   0:53.48 ksoftirqd/1
       16 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kworker/1:0H
       17 root      rt   0       0      0      0 S  0.0  0.0   0:00.33 watchdog/2
       18 root      rt   0       0      0      0 S  0.0  0.0   0:09.09 migration/2
       19 root      20   0       0      0      0 S  0.0  0.0   1:02.02 ksoftirqd/2
       21 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kworker/2:0H
       22 root      rt   0       0      0      0 S  0.0  0.0   0:00.39 watchdog/3
       23 root      rt   0       0      0      0 S  0.0  0.0   0:09.85 migration/3
       24 root      20   0       0      0      0 S  0.0  0.0   1:22.78 ksoftirqd/3
       26 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kworker/3:0H
       28 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kdevtmpfs
       29 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 netns
       30 root      20   0       0      0      0 S  0.0  0.0   0:00.05 khungtaskd
       31 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 writeback
       32 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kintegrityd
       33 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 bioset
       34 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 bioset
       35 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 bioset
       36 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kblockd
       37 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 md
       38 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 edac-poller
       39 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 watchdogd
       45 root      20   0       0      0      0 S  0.0  0.0   1:26.15 kswapd0
       46 root      25   5       0      0      0 S  0.0  0.0   0:00.00 ksmd
       47 root      39  19       0      0      0 S  0.0  0.0   0:00.44 khugepaged
       48 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 crypto
       56 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kthrotld
       58 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kmpath_rdacd
       59 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kaluad
       61 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kpsmoused
       62 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 ipv6_addrconf
       75 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 deferwq
      109 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kauditd
      638 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 ata_sff
      663 root      20   0       0      0      0 S  0.0  0.0   0:00.00 scsi_eh_0
      665 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 scsi_tmf_0
      667 root      20   0       0      0      0 S  0.0  0.0   0:00.00 scsi_eh_1
      668 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 scsi_tmf_1
      770 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 ttm_swap
     1304 root       0 -20       0      0      0 S  0.0  0.0   0:00.38 kworker/0:1H
     1414 root      20   0       0      0      0 S  0.0  0.0   0:06.52 kworker/0:2
     1418 root       0 -20       0      0      0 S  0.0  0.0   0:00.43 kworker/1:1H
     1448 root      20   0       0      0      0 S  0.0  0.0   0:03.59 jbd2/vda1-8
     1449 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 ext4-rsv-conver
     1457 root       0 -20       0      0      0 S  0.0  0.0   0:00.96 kworker/3:1H
     1504 root       0 -20       0      0      0 S  0.0  0.0   0:00.29 kworker/2:1H
     1521 root      20   0   55912  20124  19856 S  0.0  0.3   0:06.80 systemd-journal
     1543 root      20   0   44680   1052    936 S  0.0  0.0   0:00.19 systemd-udevd
     1557 root       0 -20       0      0      0 S  0.0  0.0   0:00.30 loop0
     1568 root      20   0       0      0      0 S  0.0  0.0   0:00.07 jbd2/loop0-8
     1571 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 ext4-rsv-conver
     1596 root      20   0       0      0      0 S  0.0  0.0   0:06.79 kworker/2:2
     1884 polkitd   20   0  612408   7088   2480 S  0.0  0.1   0:01.51 polkitd
     1993 polkitd   20   0  612408   7088   2480 S  0.0  0.1   0:00.00 gmain
     2008 polkitd   20   0  612408   7088   2480 S  0.0  0.1   0:01.92 gdbus
     2027 polkitd   20   0  612408   7088   2480 S  0.0  0.1   0:00.00 polkitd
     2071 polkitd   20   0  612408   7088   2480 S  0.0  0.1   0:00.00 JS GC Helper
     2077 polkitd   20   0  612408   7088   2480 S  0.0  0.1   0:00.00 JS Sour~ Thread
     2097 polkitd   20   0  612408   7088   2480 S  0.0  0.1   0:00.00 polkitd
     1887 root      20   0   90552   2224   2128 S  0.0  0.0   0:14.89 rngd
     1892 root      20   0   21716    968    832 S  0.0  0.0   0:12.89 irqbalance
     1906 root      20   0   12184   2668    584 S  0.0  0.0   0:15.31 haveged
     1926 dbus      20   0   58172   1892   1580 S  0.0  0.0   0:08.66 dbus-daemon
     1958 chrony    20   0  117784   1356   1212 S  0.0  0.0   0:00.32 chronyd
     2012 root      20   0   26560   1616   1312 S  0.0  0.0   0:05.19 systemd-logind
     2034 root      20   0  126364   1352    936 S  0.0  0.0   0:00.50 crond
     2199 root      20   0       0      0      0 S  0.0  0.0   0:00.33 kworker/u8:1
     2216 memcach+  20   0  449820   1720    672 S  0.0  0.0   0:00.81 memcached
     2217 memcach+  20   0  449820   1720    672 S  0.0  0.0   0:00.64 memcached
     2218 memcach+  20   0  449820   1720    672 S  0.0  0.0   0:00.00 memcached
     2219 memcach+  20   0  449820   1720    672 S  0.0  0.0   0:00.00 memcached
     2220 memcach+  20   0  449820   1720    672 S  0.0  0.0   0:00.00 memcached
     2221 memcach+  20   0  449820   1720    672 S  0.0  0.0   0:00.00 memcached
     2222 memcach+  20   0  449820   1720    672 S  0.0  0.0   0:00.00 memcached
     2223 memcach+  20   0  449820   1720    672 S  0.0  0.0   0:00.00 memcached
     2224 memcach+  20   0  449820   1720    672 S  0.0  0.0   0:01.28 memcached
     2225 memcach+  20   0  449820   1720    672 S  0.0  0.0   0:00.00 memcached
     2286 root      20   0       0      0      0 S  0.0  0.0   0:00.77 hwrng
     2854 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 nfit
     3340 root      20   0  112916   3408   3292 S  0.0  0.0   0:01.17 sshd
     3344 root      20   0  573976   7752   3608 S  0.0  0.1   0:00.13 tuned
     4207 root      20   0  573976   7752   3608 S  0.0  0.1   0:00.00 gmain
     4208 root      20   0  573976   7752   3608 S  0.0  0.1   0:11.13 tuned
     4215 root      20   0  573976   7752   3608 S  0.0  0.1   0:00.00 tuned
     4309 root      20   0  573976   7752   3608 S  0.0  0.1   0:00.00 tuned
     3348 root      20   0  232788  14680  11860 S  0.0  0.2   0:00.01 rsyslogd
     3388 root      20   0  232788  14680  11860 S  0.0  0.2   0:07.00 in:imjournal
     3402 root      20   0  232788  14680  11860 S  0.0  0.2   0:00.70 rs:main Q:Reg
     3390 root      20   0  110104    808    676 S  0.0  0.0   0:00.00 agetty
     3519 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kworker/0:1
     4388 root      20   0  154708   5536   4232 S  0.0  0.1   0:00.03 sshd
     4390 root      20   0  116372   3092   1708 S  0.0  0.0   0:00.03 bash
     4735 root      20   0       0      0      0 S  0.0  0.0   0:30.39 kworker/1:1
     5346 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 dio/vda1
     6365 root      20   0       0      0      0 S  0.0  0.0   0:07.34 kworker/1:0
     6693 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:01.18 mysqld
     6874 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.00 mysqld
     6890 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.00 mysqld
     6903 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   1:44.23 mysqld
     8090 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.07 mysqld
     8091 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.00 mysqld
     8094 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.01 mysqld
     8197 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.15 mysqld
     8674 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:05.37 mysqld
     8675 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:05.85 mysqld
     8676 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:05.60 mysqld
     8677 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:05.51 mysqld
     8678 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:05.48 mysqld
     8679 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:05.45 mysqld
     8680 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:06.16 mysqld
     8681 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:05.64 mysqld
     8682 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:05.54 mysqld
     8683 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:05.54 mysqld
     8759 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.75 mysqld
     8959 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:03.67 mysqld
     8960 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:05.42 mysqld
     8961 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:11.26 mysqld
     8962 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.37 mysqld
     8963 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.80 mysqld
     8964 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.00 mysqld
     9014 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:05.01 mysqld
     9015 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:04.32 mysqld
     9034 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.00 mysqld
     9035 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:03.09 mysqld
     9038 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.00 mysqld
     9074 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.00 mysqld
     9075 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.00 mysqld
     9076 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.00 mysqld
     9077 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.00 mysqld
     9706 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   1:38.32 mysqld
    10357 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   5:40.07 mysqld
    25937 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:06.27 mysqld
    29971 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:06.81 mysqld
    25473 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:06.24 mysqld
    25474 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:06.15 mysqld
     8643 root      20   0  171904  23956   2184 S  0.0  0.3   0:01.31 lfd - sleeping
    11883 root      20   0       0      0      0 S  0.0  0.0   0:00.51 kworker/3:1
    15130 root      20   0  154988   5840   4264 S  0.0  0.1   0:00.09 sshd
    17530 root      20   0       0      0      0 S  0.0  0.0   0:05.66 kworker/1:2
    20634 root      20   0  154988   5688   4260 S  0.0  0.1   0:00.02 sshd
    21779 root      20   0       0      0      0 S  0.0  0.0   0:00.84 kworker/u8:0
    22499 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kworker/3:0
    23805 root      20   0  113440   1744   1304 S  0.0  0.0   0:00.00 cminfo
    23916 root      20   0  154992   5836   4264 S  0.0  0.1   0:00.06 sshd
    24008 root      20   0  113524   1344    752 S  0.0  0.0   0:00.01 cminfo
    24009 root      20   0  107952    668    564 S  0.0  0.0   0:00.00 tee
    24572 root      20   0  317240 117996    448 S  0.0  1.5   0:00.03 nginx
    24578 nginx     10 -10  354104 153892   1268 S  0.0  1.9   0:00.06 nginx
    24581 nginx     10 -10  354104 154620   1304 S  0.0  1.9   0:00.07 nginx
    24762 root      10 -10  631772  15104   1252 S  0.0  0.2   0:00.00 php-fpm
    24790 nginx     10 -10  631772  15736   1716 S  0.0  0.2   0:00.00 php-fpm
    26603 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kworker/2:0
    
    ------------------------------------------------------------------
    iotop -bton1 -P
    16:20:01 Total DISK READ :       0.00 B/s | Total DISK WRITE :       0.00 B/s
    16:20:01 Actual DISK READ:       0.00 B/s | Actual DISK WRITE:       0.00 B/s
        TIME  PID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
    
    ------------------------------------------------------------------
    pidstat -durh 1 10 | sed -e "s|$(hostname)|hostname|g"
    Linux 3.10.0-957.21.3.el7.x86_64 (hostname)     06/24/2019     _x86_64_    (4 CPU)
    
    #      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
     1561389602     0         1    0.00    0.00    0.00    0.00     3     25.49      0.00  191140   3408   0.04      0.00      0.00      0.00  systemd
     1561389602     0      1521    0.00    0.00    0.00    0.00     2     11.76      0.00   55912  20124   0.25      0.00      0.00      0.00  systemd-journal
     1561389602     0      1892    0.00    0.00    0.00    0.00     0     11.76      0.00   21716    968   0.01      0.00      0.00      0.00  irqbalance
     1561389602    81      1926    0.00    0.00    0.00    0.00     1      1.96      0.00   58172   1892   0.02      0.00      0.00      0.00  dbus-daemon
     1561389602     0      2012    0.00    0.00    0.00    0.00     3     11.76      0.00   26560   1624   0.02      0.00      0.00      0.00  systemd-logind
     1561389602     0      2034    0.00    0.00    0.00    0.00     1      0.98      0.00  126364   1352   0.02      0.00      7.84      0.00  crond
     1561389602     0      2199    0.00    0.98    0.00    0.98     3      0.00      0.00       0      0   0.00      0.00      0.00      0.00  kworker/u8:1
     1561389602     0      3348    0.00    0.00    0.00    0.00     2      1.96      0.00  232788  14688   0.18      0.00      3.92      0.00  rsyslogd
     1561389602   996      6693    0.98    0.98    0.00    1.96     2     46.08      0.00 6734520 1092108  13.64      0.00      0.00      0.00  mysqld
     1561389602  1000     24581    0.98    0.00    0.00    0.98     0      0.00      0.00  354104 154620   1.93      0.00      0.00      0.00  nginx
     1561389602     0     25050    0.00    0.98    0.00    0.98     3    318.63      0.00  108192   1060   0.01      0.00      0.00      0.00  pidstat
    
    #      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
     1561389603     0     17530    0.00    2.00    0.00    2.00     1      0.00      0.00       0      0   0.00      0.00      0.00      0.00  kworker/1:2
     1561389603     0     25050    0.00    1.00    0.00    1.00     3    336.00      0.00  108192   1136   0.01      0.00      0.00      0.00  pidstat
     1561389603     0     25051    0.00    0.00    0.00    0.00     0      1.00      0.00  116992    936   0.01      0.00      0.00      0.00  sed
    
    #      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
     1561389604   996      6693    0.00    0.00    0.00    0.00     2    542.00      0.00 6734520 1092108  13.64      0.00      0.00      0.00  mysqld
     1561389604     0     25050    0.00    1.00    0.00    1.00     3    317.00      0.00  108192   1136   0.01      0.00      0.00      0.00  pidstat
    
    #      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
     1561389605     0     25050    1.00    1.00    0.00    2.00     3    317.00      0.00  108192   1136   0.01      0.00      0.00      0.00  pidstat
    
    #      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
     1561389606     0         1    0.00    1.00    0.00    1.00     3      0.00      0.00  191140   3408   0.04      0.00      0.00      0.00  systemd
     1561389606     0      1448    0.00    0.00    0.00    0.00     3      0.00      0.00       0      0   0.00      0.00     48.00      0.00  jbd2/vda1-8
     1561389606     0      1557    0.00    0.00    0.00    0.00     1      0.00      0.00       0      0   0.00      0.00     64.00      0.00  loop0
     1561389606   996      6693    1.00    1.00    0.00    2.00     2    542.00      0.00 6734520 1092108  13.64      0.00      0.00      0.00  mysqld
     1561389606     0      8643    0.00    0.00    0.00    0.00     0      0.00      0.00       0      0   0.00      0.00      4.00      0.00  lfd
     1561389606     0     25050    0.00    1.00    0.00    1.00     3    317.00      0.00  108192   1136   0.01      0.00      0.00      0.00  pidstat
    
    #      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
     1561389607   996      6693    0.00    0.00    0.00    0.00     2    542.00      0.00 6734520 1092108  13.64      0.00      0.00      0.00  mysqld
     1561389607     0     24009    0.00    0.00    0.00    0.00     1      0.00      0.00  107952    668   0.01      0.00      4.00      0.00  tee
     1561389607     0     25050    1.00    0.00    0.00    1.00     3    317.00      0.00  108192   1136   0.01      0.00      0.00      0.00  pidstat
    
    #      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
     1561389608   996      6693    1.00    0.00    0.00    1.00     2      0.00      0.00 6734520 1092108  13.64      0.00      0.00      0.00  mysqld
     1561389608     0     25050    0.00    1.00    0.00    1.00     3    317.00      0.00  108192   1136   0.01      0.00      0.00      0.00  pidstat
    
    #      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
     1561389609   996      6693    0.00    0.00    0.00    0.00     2    542.00      0.00 6734520 1092108  13.64      0.00      0.00      0.00  mysqld
     1561389609     0     15130    0.00    0.00    0.00    0.00     1     38.00      0.00  154988   5840   0.07      0.00      0.00      0.00  sshd
     1561389609     0     25050    0.00    2.00    0.00    2.00     3    317.00      0.00  108192   1136   0.01      0.00      0.00      0.00  pidstat
     1561389609     0     25066    0.00    0.00    0.00    0.00     3    881.00      0.00    9556   1460   0.02      0.00      8.00      0.00  bash
     1561389609     0     25077    0.00    0.00    0.00    0.00     1    949.00      0.00  113176   1560   0.02      0.00      4.00      0.00  dmotd
     1561389609     0     25109    0.00    0.00    0.00    0.00     2    490.00      0.00  113180   1032   0.01      0.00      0.00      0.00  dmotd
     1561389609     0     25110    0.00    0.00    0.00    0.00     0    316.00      0.00  107952    668   0.01      0.00      4.00      0.00  tee
     1561389609     0     25148    0.00    0.00    0.00    0.00     0    529.00      0.00  116668   1396   0.02      0.00      0.00      0.00  git
     1561389609     0     25149    1.00    0.00    0.00    1.00     2   1262.00      0.00  163908   4296   0.05      0.00      0.00      0.00  git-remote-http
    
    #      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
     1561389610     0        14    0.00    1.00    0.00    1.00     1      0.00      0.00       0      0   0.00      0.00      0.00      0.00  ksoftirqd/1
     1561389610   996      6693    0.00    1.00    0.00    1.00     2    542.00      0.00 6734520 1092108  13.64      0.00      0.00      0.00  mysqld
     1561389610     0     15130    0.00    0.00    0.00    0.00     2      2.00      0.00  154988   5840   0.07      0.00      0.00      0.00  sshd
     1561389610     0     25050    0.00    1.00    0.00    1.00     0    335.00      0.00  108192   1136   0.01      0.00      0.00      0.00  pidstat
     1561389610     0     25077    0.00    0.00    0.00    0.00     1    109.00      0.00  113180   1564   0.02      0.00      8.00      0.00  dmotd
     1561389610     0     25166    1.00    6.00    0.00    7.00     3   8509.00      0.00  153004  33312   0.42      0.00      0.00      0.00  find
     1561389610     0     25167    0.00    0.00    0.00    0.00     3    108.00      0.00  113180    452   0.01      0.00      0.00      0.00  dmotd
    
    #      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
     1561389611     0         1    0.00    0.00    0.00    0.00     2     22.00      0.00  191140   3408   0.04      0.00      0.00      0.00  systemd
     1561389611     0      1448    0.00    0.00    0.00    0.00     2      0.00      0.00       0      0   0.00      0.00     28.00      0.00  jbd2/vda1-8
     1561389611     0      1521    0.00    1.00    0.00    1.00     1     38.00      0.00   55912  20136   0.25      0.00      0.00      0.00  systemd-journal
     1561389611    81      1926    0.00    1.00    0.00    1.00     2      2.00      0.00   58172   1892   0.02      0.00      0.00      0.00  dbus-daemon
     1561389611     0      2012    0.00    0.00    0.00    0.00     0     12.00      0.00   26560   1624   0.02      0.00      0.00      0.00  systemd-logind
     1561389611     0      3340    0.00    1.00    0.00    1.00     3     15.00      0.00  112916   3408   0.04      0.00    500.00    112.00  sshd
     1561389611     0      3348    0.00    0.00    0.00    0.00     2      2.00      0.00  232788  14688   0.18      0.00      0.00      0.00  rsyslogd
     1561389611   996      6693    1.00    0.00    0.00    1.00     2      0.00      0.00 6734520 1092108  13.64      0.00      0.00      0.00  mysqld
     1561389611     0     24762    0.00    0.00    0.00    0.00     1      9.00      0.00  631772  15104   0.19      0.00      0.00      0.00  php-fpm
     1561389611     0     25050    0.00    1.00    0.00    1.00     0    326.00      0.00  108192   1136   0.01      0.00      0.00      0.00  pidstat
     1561389611     0     25171    1.00    1.00    0.00    2.00     1   1768.00      0.00  154992   5668   0.07      0.00      0.00      0.00  sshd
     1561389611     0     25174    0.00    0.00    0.00    0.00     3    882.00      0.00    9556   1460   0.02      0.00      8.00      0.00  bash
     1561389611     0     25185    0.00    0.00    0.00    0.00     3    950.00      0.00  113176   1560   0.02      0.00      4.00      0.00  dmotd
    ------------------------------------------------------------------
    Stats saved at: /root/centminlogs/cminfo-top-240619-161950.log
    ------------------------------------------------------------------
    

    Another thing I've noticed thanks to your suggestion to use 'cminfo top'. When I checked the 'centminlogs' directory there were many thousands of 'cmm-login-git-checks_*.log' files. These are created everytime there's an Ssh connection. Galera is making many SSH connections to all the servers all the time. This means there is a git check every time an SSH connection is made. Maybe this is the cause of the SWAP being used.

    I've found these for the custom_config:

    Code:
    ENABLEMOTD='n'
    ENABLEMOTD_CSFMSG='n'
    ENABLEMOTD_LINKSMSG='n' 
    But, ENABLEMOTD='n' doesn't seem to work.

    I have a feeling the many system statuses and git checks are part, if not the issue here.

    Is there a way to perminantly turn off the status and git checks when there's an SSH login?
     
  4. Simon Brown

    Simon Brown Member

    41
    5
    8
    Feb 9, 2017
    Ratings:
    +17
    Local Time:
    11:00 PM
    1.11.9
    Quick update, I didn't launch centmin.sh after I updated the custom_config. I've now done it on all the servers. I'll wait to see if it make any difference :)
     
    Last edited: Jun 25, 2019
  5. eva2000

    eva2000 Administrator Staff Member

    41,646
    9,380
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,408
    Local Time:
    8:00 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    From cminfo top output not seeing nginx as a swap user there so probably it's a victim not a culprit of low memory free availability. Most likely it's MySQL itself given it's virtual memory size. I'd be tuning MySQL buffers to lower memory usage. But the current cminfo top shows no swapping to disk right now so probably not a good indicator.

    also suggests it's normal occurrence of swap as you're reaching low memory free availability free with high/low reported memory usage shows low stage you have 248MB free
    Code (Text):
    ------------------------------------------------------------------
    free -mtl
                 total        used        free      shared  buff/cache   available
    Mem:           7821        1456         248          19        6115        6008
    Low:           7821        7572         248
    High:             0           0           0
    Swap:          1023          56         967
    Total:         8845        1512        1216
    

    sar -r output shows memory free can get down to 115,244 KB or 112.5MB with 6,871,768 KB or ~6.55GB in committed memory at 2:30:01 AM on June 24 so it's would be expected that swap would be avenue for other apps/processes fighting for memory availability when it gets that low
    Code (Text):
    02:15:01 PM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
    
    02:20:01 AM    202916   7805940     97.47     11932     70088   6771340     74.76    690792    828088       176
    02:25:01 AM    182440   7826416     97.72     11692     68644   6771372     74.76    691916    825088       204
    02:30:01 AM    115244   7893612     98.56     11884     70180   6871768     75.87    765484    827260       160
    02:35:01 AM    160816   7848040     97.99     12040     68096   6805148     75.13    724492    824084       152
    02:40:01 AM    201464   7807392     97.48     11328     63380   6785212     74.91    694804    818400       148
    02:45:01 AM    197548   7811308     97.53     11552     63292   6771364     74.76    690000    819192       124
    02:50:01 AM    220572   7788284     97.25     11504     62528   6771340     74.76    689052    818620       180
    02:55:01 AM    221728   7787128     97.23     11700     62592   6775560     74.81    695176    815644       192
    03:00:01 AM    160600   7848256     97.99     11408     69932   6819508     75.29    702796    823336       180
    03:05:01 AM    161896   7846960     97.98     12376     74308   6805656     75.14    727012    824176       160
    03:10:01 AM    192268   7816588     97.60     12856     66456   6771872     74.77    693136    817512       188
    03:15:01 AM    155744   7853112     98.06     71876     91708   6771372     74.76    730208    867844       148
    03:20:01 AM    205064   7803792     97.44     64252     82856   6771372     74.76    728272    853456       172
    03:25:01 AM    201988   7806868     97.48     60788     75348   6771368     74.76    720208    850692       184
    03:30:02 AM    184568   7824288     97.70     54844     69968   6803456     75.11    715532    853824       208
    03:35:01 AM    237096   7771760     97.04     41748     68328   6771368     74.76    697588    847056       840
    03:40:01 AM    137588   7871268     98.28     39764     64196   6838936     75.51    760396    844056       140
    03:45:01 AM    179840   7829016     97.75     33096     65400   6771796     74.77    695436    837592        48
    03:50:01 AM    185920   7822936     97.68     24256     63468   6771368     74.76    694888    827116        48
    03:55:01 AM    174796   7834060     97.82     18544     64552   6771368     74.76    694684    822808       180
    04:00:01 AM    156888   7851968     98.04     13588     70808   6821980     75.32    707208    823656       136
    04:05:01 AM    228080   7780776     97.15     12024     71516   6771340     74.76    697652    820068       204
    


    currently no way to disable that though those logs get purged every 5 days. But pretty sure it isn't the cause of swapping.

    you can use cminfo top command via cronjob to record your usage if you want as the cminfo top output is saved to a log in /root/centminlogs/ directory so you can see your specific process and system usage.

    For CentOS 7, you can try changing in /etc/sysctl.d/101-sysctl.conf the vm.swappiness from 100-10 = swap to disk when 90% full memory
    Code (Text):
    vm.swappiness=10

    to swap to disk when 100-1 = 99% full memory
    Code (Text):
    vm.swappiness=1

    then for changes to take affect run
    Code (Text):
    sysctl --system
     
  6. Simon Brown

    Simon Brown Member

    41
    5
    8
    Feb 9, 2017
    Ratings:
    +17
    Local Time:
    11:00 PM
    1.11.9
    Just wanted to update this ticket... It seems that using the ENABLEMOTD config has worked. The three servers I used configured are no longer using SWAP and there aren't thousands of centminlog files anymore. IF people are syncing servers via SSH with a service like Lsyncd and/or cluster/server monitoring services that use SSH, then you should set the centmin config with this:
    Code:
    ENABLEMOTD='n'
    ENABLEMOTD_CSFMSG='n'
    ENABLEMOTD_LINKSMSG='n' 
    Many thanks as ever for your detailed replies.
     
  7. buik

    buik Well-Known Member

    1,193
    318
    83
    Apr 29, 2016
    Ratings:
    +937
    Local Time:
    12:00 AM
    I have seen this more often. Might be low memory free availability or oversold node.
    Have you tested this setup also on bare metal dedicated?

    So you're sure you don't suffer from neighbors.
     
  8. Simon Brown

    Simon Brown Member

    41
    5
    8
    Feb 9, 2017
    Ratings:
    +17
    Local Time:
    11:00 PM
    1.11.9
    The issue was happening on 4 VPS's using two vendors and a dedicated server with 32Gb memory.
    After over 24 hours the swap still isn't used. Fingers crossed.
     
  9. buik

    buik Well-Known Member

    1,193
    318
    83
    Apr 29, 2016
    Ratings:
    +937
    Local Time:
    12:00 AM
    Have you tested this setup also on bare metal dedicated only?
    VPS servers are always overpopulated on lowend (Vultr, Digitalocean etc etc). It's just the question of how far.
     
  10. Simon Brown

    Simon Brown Member

    41
    5
    8
    Feb 9, 2017
    Ratings:
    +17
    Local Time:
    11:00 PM
    1.11.9
    Yes, I have a bare metal dedicated with 32Gb memory. It was swapping too. So far, swap is 0 on all VPS's and the dedicated. I've been testing the cluster for weeks now, trying to work out why all nodes were swapping all the time.
    This is the longest and most stable it's been. I even enabled lsyncd on the nodes. So far, so good.
     
  11. buik

    buik Well-Known Member

    1,193
    318
    83
    Apr 29, 2016
    Ratings:
    +937
    Local Time:
    12:00 AM
    @Simon Brown I apologize.
    I didn't read it correctly.
    And should have known better.

    As you are using various VPS providers and 1 dedi
    (Vultr has been offering dedis for 1.5 years. I didn't even know. Haha.),
    this problem has more of a bug, mis-config, than of overselling.

    Completely in view of the fact that the problem seems to be corrected by adjusting:
    ENABLEMOTD.

    And you wrote this before I responded.
    Again excuses.
     
  12. eva2000

    eva2000 Administrator Staff Member

    41,646
    9,380
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,408
    Local Time:
    8:00 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    FYI, ENABLEMOTD='y' is force enabled now in 123.09beta01 so setting ENABLEMOTD='n' won't have any effect so wouldn't be that.

    might want to make use of cminfo top stats to monitor your processes, I updated 123.09beta01's cminfo top stats to also check top 10 VmSwap and VmSize/VSZ using processes Beta Branch - update tools/cminfo.sh in 123.09beta01

    cminfo command is updated every 4hrs on centmin mod servers via a cronjob. You can manually update it to use updated cminfo code via manually running cronjob updating script
    Code (Text):
    /usr/bin/cminfo_updater