Sysadmin Swap increases slowly, but continuously until 100%

Simon Brown · Jun 24, 2019

Please fill in any relevant information that applies to you:
CentOS Version: CentOS 7.6.1810 - 64bit

Centmin Mod Version Installed: 123.09beta01

Nginx Version Installed: 1.17.0

PHP Version Installed: 7.0.33

MariaDB MySQL Version Installed: 10.1.40

When was last time updated Centmin Mod code base ? : run cmupdate command
Persistent Config:
Code (Text):
NGXDYNAMIC_NGXPAGESPEED='n'
NGINX_PAGESPEED='n'
NGXDYNAMIC_BROTLI='y'
NGINX_LIBBROTLI='y'
LETSENCRYPT_DETECT='y'
NGINX_VHOSTSTATS=n
MARCH_TARGETNATIVE='n'
NGINX_VIDEO='y'
PHP_PGO='y'
This is a strange issue that a little beyond my tech knowledge. Has anyone else seen this issue...

The setup:
I have 1x new Vultr bare metal dedicated server, 2x new Vultr servers and 1x new UpCloud server all using the standard Centos 7 install. I've installed centmin 123.09beta01.

The servers all have a minimum of 4 processors & minimum of 8Gb Memory.

The only custom setup is the servers use Galera for database clustering. While I test the setup all the servers are behind a firewall that's blocking access on all ports to the outside world. The IP's of all the servers are whitelisted in the external firewalls and the CSF firewalls, so all ports are available only to the network of servers. Galera is working well, ssh, ping, etc. is working perfectly between all the servers.
centmin & all yum updates are the latest versions.

The issue:
For some reason, the servers all start to use swap even though there's plenty of memory.
The results below are for 24 hours after a server reboot.

I've run a script to find only the prosesses using swap. As you can see Nginx uses most of it:

Overall swap used: 120380 kB
========================================
kB pid name
========================================
33808 4352 nginx
31464 4357 nginx
31356 4353 nginx
8900 3344 tuned
5024 1884 polkitd
3728 1906 haveged
1572 4759 amplify-agent
912 3340 sshd
824 1887 rngd
576 1543 systemd-udevd
532 3348 rsyslogd
328 1958 chronyd
308 1521 systemd-journal
284 1926 dbus-daemon
264 2034 crond
220 1 systemd
148 1892 irqbalance
132 2012 systemd-logind

I've found that restarting Nginx via the 'nprestart' helps to lower the swap usage but obviously, it's not a fix.

Free mem command: free -m
total used free shared buff/cache available
Mem: 7821 1484 795 19 5541 5981
Swap: 1023 57 966

Swap usage over 24 hours. You can see where the swap got to after a few days to the left, then I rebooted the server.

Same server over 4 days:

Another server over 4 days:

The high CPU/Disk usage to the right of these graphs is from recompiling PHP to use version 7.3.5. It's the only software that isn't the latest version. I don't think it's the issue though.

Memory usage:

Has anyone had the same swap issue?

I've been trying to solve this issue for a few weeks. I'm running out of ideas!!!

eva2000 · Jun 25, 2019

For each server post the output for commands below? Wrap in quote bbcode tags
Code (Text):
nginx - V
And next one wrap in code bbcode tags
Code (Text):
cminfo top
When was last time updated Centmin Mod code base ? Using cm update command before nginx compiles/updates?

Simon Brown said: ↑

NGXDYNAMIC_BROTLI='y' NGINX_LIBBROTLI='y'
NGINX_VIDEO='y'
Click to expand...

Try disabling these set to no and recompile nginx via centmin.sh menu option 4 and see

Simon Brown · Jun 25, 2019

I regularly update centmin and the yum updates when I log out of centmin. The server(s) is up-to-date.

Code:

 Centmin Mod local code is up to date at /usr/local/src/centminmod
 no available updates at this time...

The NGINX_VIDEO='y' is only enabled on one server. I enabled it a few days ago. The server was swapping for a few weeks before I enabled the video.

These options are enabled on all the servers NGXDYNAMIC_BROTLI='y' NGINX_LIBBROTLI='y' so I'll disable it on one server and see if that make any difference.

nginx -V

Code:

nginx version: nginx/1.17.0 (190619-150338-centos7-kvm)
built by gcc 8.2.1 20180905 (Red Hat 8.2.1-3) (GCC)
built with OpenSSL 1.1.1c  28 May 2019
TLS SNI support enabled
configure arguments: --with-ld-opt='-Wl,-E -L/usr/local/zlib-cf/lib -L/usr/local/lib -ljemalloc -Wl,-z,relro -Wl,-rpath,/usr/local/zlib-cf/lib:/usr/local/lib' --with-cc-opt='-I/usr/local/zlib-cf/include -I/usr/local/include -m64 -march=x86-64 -mavx -mavx2 -mpclmul -msse4 -msse4.1 -msse4.2 -DTCP_FASTOPEN=23 -g -O3 -fstack-protector-strong -flto -fuse-ld=gold --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wimplicit-fallthrough=0 -fcode-hoisting -Wp,-D_FORTIFY_SOURCE=2 -Wno-deprecated-declarations' --sbin-path=/usr/local/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --build=190619-150338-centos7-kvm --with-compat --with-http_stub_status_module --with-http_secure_link_module --with-http_flv_module --with-http_mp4_module --add-module=../nginx-rtmp-module --with-libatomic --with-http_gzip_static_module --add-dynamic-module=../ngx_brotli --with-http_sub_module --with-http_addition_module --with-http_image_filter_module=dynamic --with-http_geoip_module --with-stream_geoip_module --with-stream_realip_module --with-stream_ssl_preread_module --with-threads --with-stream --with-stream_ssl_module --with-http_slice_module --with-http_realip_module --add-dynamic-module=../ngx-fancyindex-0.4.2 --add-module=../ngx_cache_purge-2.5 --add-dynamic-module=../ngx_devel_kit-0.3.0 --add-dynamic-module=../set-misc-nginx-module-0.32 --add-dynamic-module=../echo-nginx-module-0.61 --add-module=../redis2-nginx-module-0.15 --add-module=../ngx_http_redis-0.3.7 --add-module=../memc-nginx-module-0.18 --add-module=../srcache-nginx-module-0.31 --add-dynamic-module=../headers-more-nginx-module-0.33 --with-pcre-jit --with-zlib=../zlib-cloudflare-1.3.0 --with-http_ssl_module --with-http_v2_module --with-openssl=../openssl-1.1.1c --with-openssl-opt='enable-ec_nistp_64_gcc_128 enable-tls1_3'

cminfo top:

Code:

PHP Warning:  PHP Startup: Unable to load dynamic library 'geoip.so' (tried: /usr/local/lib/php/extensions/no-debug-non-zts-20180731/geoip.so (/usr/local/lib/php/extensions/no-debug-non-zts-20180731/geoip.so: cannot open shared object file: No such file or directory), /usr/local/lib/php/extensions/no-debug-non-zts-20180731/geoip.so.so (/usr/local/lib/php/extensions/no-debug-non-zts-20180731/geoip.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library 'mailparse.so' (tried: /usr/local/lib/php/extensions/no-debug-non-zts-20180731/mailparse.so (/usr/local/lib/php/extensions/no-debug-non-zts-20180731/mailparse.so: cannot open shared object file: No such file or directory), /usr/local/lib/php/extensions/no-debug-non-zts-20180731/mailparse.so.so (/usr/local/lib/php/extensions/no-debug-non-zts-20180731/mailparse.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library 'redis.so' (tried: /usr/local/lib/php/extensions/no-debug-non-zts-20180731/redis.so (/usr/local/lib/php/extensions/no-debug-non-zts-20180731/redis.so: cannot open shared object file: No such file or directory), /usr/local/lib/php/extensions/no-debug-non-zts-20180731/redis.so.so (/usr/local/lib/php/extensions/no-debug-non-zts-20180731/redis.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
------------------------------------------------------------------
 Centmin Mod Top Info:
------------------------------------------------------------------
 Server Location Info

  city: Spitalfields
  region: Essex
  country: GB
  ASN: AS20473 Choopa, LLC

 Processors physical = 1, cores = 4, virtual = 4, hyperthreading = no

      4  2593.904
      4  Virtual CPU 82d9ed4018dd
      4  16384 KB

 System Up Since:     2019-06-23 11:55:31
 System Uptime:     up 1 day, 4 hours, 24 minutes
 MySQL Server Started     2019-06-23 11:55:47
 MySQL Uptime:         1 day 4 hours 24 min 11 sec
 MySQL Uptime (secs):     102251
 Server Type:         kvm
 CentOS Version:     7.6
 Centmin Mod:         123.09beta01.b193
 Nginx PageSpeed:     OFF
 Nginx Version:     1.17.0 (190619-150338-centos7-kvm)
 PHP-FPM Version:     Warning:
 MariaDB Version:     10.1.40
 CSF Firewall:         v12.12
 Memcached Server:     1.5.16
 NSD Version:          -
 Siege Version:     4.0.4
 Maldet Version:     not installed
 ClamAV Version:     not installed
 ElasticSearch:     not installed

------------------------------------------------------------------
free -mtl
              total        used        free      shared  buff/cache   available
Mem:           7821        1456         248          19        6115        6008
Low:           7821        7572         248
High:             0           0           0
Swap:          1023          56         967
Total:         8845        1512        1216

------------------------------------------------------------------
df -hT
Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/vda1      ext4      158G   59G   91G  40% /
devtmpfs       devtmpfs  3.9G     0  3.9G   0% /dev
tmpfs          tmpfs     3.9G     0  3.9G   0% /dev/shm
tmpfs          tmpfs     3.9G   17M  3.9G   1% /run
tmpfs          tmpfs     3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/loop0     ext4      5.8G   25M  5.5G   1% /tmp
tmpfs          tmpfs     783M     0  783M   0% /run/user/0

------------------------------------------------------------------
php-fpm stats

Ready to handle connections
pool:                 www
process manager:      ondemand
start time:           24/Jun/2019:16:20:00 +0100
start since:          0
accepted conn:        2
listen queue:         0
max listen queue:     0
listen queue len:     8151
idle processes:       0
active processes:     1
total processes:      1
max active processes: 1
max children reached: 0
slow requests:        0

------------------------------------------------------------------
Filter sar -q for times cpu load avg (1min) hit/exceeded cpu threads max

12:05:01 AM         0       158      0.19      0.28      0.30         0
12:10:01 AM         0       158      0.11      0.20      0.27         0
12:15:01 AM         0       159      0.25      0.25      0.28         0
12:20:01 AM         0       158      0.14      0.18      0.24         0
12:25:01 AM         0       159      0.25      0.27      0.27         0
12:30:01 AM         0       172      0.24      0.24      0.26         0
12:35:01 AM         0       159      0.45      0.36      0.30         0
12:40:01 AM         0       158      0.51      0.51      0.40         0
12:45:01 AM         1       162      0.47      0.47      0.41         0
12:50:01 AM         0       158      0.19      0.32      0.37         0
12:55:01 AM         0       159      0.50      0.43      0.40         0
01:00:01 AM         1       191      0.63      0.38      0.38         0
01:05:01 AM         0       166      0.35      0.32      0.35         0
01:10:01 AM         1       159      0.33      0.30      0.32         0
01:15:01 AM         1       158      0.16      0.29      0.31         0
01:20:01 AM         0       158      0.33      0.34      0.33         0
01:25:02 AM         0       164      0.36      0.31      0.32         0
01:30:01 AM         0       172      0.24      0.29      0.31         0
01:35:01 AM         1       162      0.39      0.29      0.31         0
01:40:01 AM         0       158      0.26      0.26      0.30         0
01:45:01 AM         1       163      0.33      0.38      0.35         0
01:50:01 AM         2       164      0.11      0.21      0.28         0
01:55:01 AM         0       158      0.37      0.33      0.32         0
02:00:01 AM         6       239      1.19      0.57      0.41         0
02:05:01 AM         0       160      0.18      0.35      0.37         0
02:10:01 AM         0       166      0.34      0.31      0.34         0
02:15:01 AM         0       159      0.19      0.25      0.31         0
02:20:01 AM         0       159      0.33      0.25      0.29         0
02:25:01 AM         0       159      0.26      0.31      0.31         0
02:30:01 AM         6       187      0.71      0.40      0.34         0
02:35:01 AM         1       163      0.33      0.30      0.31         0
02:40:01 AM         0       165      0.16      0.24      0.29         0
02:45:01 AM         0       160      0.36      0.32      0.32         0
02:50:01 AM         0       159      0.50      0.33      0.32         0
02:55:01 AM         2       163      0.44      0.36      0.33         0
03:00:01 AM         0       180      0.27      0.34      0.33         0
03:05:01 AM         1       165      0.20      0.34      0.33         0
03:10:01 AM         0       160      0.37      0.37      0.34         0
03:15:01 AM         0       159      0.33      0.40      0.37         0
03:20:01 AM         0       159      0.36      0.34      0.34         0
03:25:01 AM         0       159      0.34      0.36      0.35         0
03:30:02 AM         0       173      0.37      0.37      0.36         0
03:35:01 AM         0       159      0.20      0.29      0.32         0
03:40:01 AM         5       167      0.32      0.30      0.32         0
03:45:01 AM         1       160      0.30      0.27      0.30         0
03:50:01 AM         0       159      0.27      0.23      0.28         0
03:55:01 AM         0       160      0.10      0.20      0.26         0
04:00:01 AM         2       184      0.38      0.30      0.29         0
04:05:01 AM         0       160      0.42      0.35      0.32         0
04:10:01 AM         0       159      0.32      0.32      0.32         0
04:15:01 AM         0       159      0.31      0.35      0.33         0
04:20:01 AM         1       164      0.39      0.37      0.34         0
04:25:01 AM         0       160      0.24      0.36      0.34         0
04:30:01 AM         1       173      0.47      0.38      0.34         0
04:35:01 AM         0       160      0.46      0.37      0.34         0
04:40:01 AM         0       160      0.22      0.29      0.31         0
04:45:01 AM         1       163      0.41      0.37      0.34         0
04:50:01 AM         1       160      0.39      0.36      0.33         0
04:55:01 AM         2       168      0.20      0.31      0.32         0
05:00:01 AM         1       179      0.19      0.23      0.29         0
05:05:01 AM         1       160      0.23      0.27      0.30         0
05:10:01 AM         0       161      0.51      0.36      0.33         0
05:15:01 AM         0       160      0.34      0.37      0.33         0
05:20:01 AM         1       171      0.25      0.31      0.31         0
05:25:01 AM         0       160      0.39      0.38      0.35         0
05:30:01 AM         0       174      0.67      0.41      0.36         0
05:35:01 AM         1       165      0.24      0.36      0.36         0
05:40:01 AM         1       166      0.29      0.32      0.34         0
05:45:01 AM         0       160      0.51      0.37      0.34         0
05:50:01 AM         0       160      0.54      0.38      0.34         0
05:55:01 AM         0       160      0.30      0.32      0.32         0
06:00:01 AM         0       181      0.12      0.25      0.29         0
06:05:01 AM         0       160      0.29      0.35      0.33         0
06:10:01 AM         0       160      0.28      0.30      0.31         0
06:15:01 AM         0       160      0.59      0.39      0.34         0
06:20:01 AM         0       160      0.34      0.36      0.34         0
06:25:01 AM         0       167      0.76      0.42      0.35         0
06:30:01 AM         2       174      0.40      0.35      0.33         0
06:35:01 AM         1       161      0.36      0.31      0.32         0
06:40:01 AM         0       160      0.19      0.27      0.31         0
06:45:01 AM         0       160      0.32      0.37      0.35         0
06:50:01 AM         1       164      0.32      0.38      0.36         0
06:55:01 AM         0       161      0.25      0.32      0.34         0
07:00:01 AM         2       192      0.66      0.40      0.35         0
07:05:01 AM         0       162      0.26      0.45      0.40         0
07:10:01 AM         0       160      0.36      0.39      0.39         0
07:15:01 AM         1       164      0.18      0.27      0.34         0
07:20:01 AM         2       172      0.12      0.24      0.31         0
07:25:01 AM         0       173      0.47      0.30      0.32         0
07:30:01 AM         4       178      0.40      0.40      0.36         0
07:35:01 AM         0       166      0.34      0.38      0.36         0
07:40:01 AM         1       177      0.37      0.34      0.34         0
07:45:01 AM         1       164      0.44      0.39      0.36         0
07:50:01 AM         0       166      0.44      0.38      0.36         0
07:55:01 AM         0       168      0.45      0.43      0.39         0
08:00:01 AM         2       189      0.51      0.43      0.41         0
08:05:01 AM         0       160      0.15      0.31      0.38         0
08:10:01 AM         1       164      0.35      0.37      0.40         0
08:15:01 AM         0       160      0.46      0.42      0.41         0
08:20:01 AM         1       161      0.17      0.30      0.37         0
08:25:01 AM         0       160      0.31      0.33      0.37         0
08:30:01 AM         4       175      0.22      0.33      0.36         0
08:35:01 AM         1       160      0.30      0.34      0.36         0
08:40:01 AM         2       164      0.44      0.37      0.36         0
08:45:01 AM         0       161      0.38      0.35      0.34         0
08:50:01 AM         0       160      0.23      0.37      0.35         0
08:55:01 AM         0       160      0.31      0.34      0.35         0
09:00:02 AM         4       182      0.39      0.32      0.33         0
09:05:01 AM         0       160      0.40      0.37      0.34         0
09:10:01 AM         0       160      0.41      0.32      0.33         0
09:15:01 AM         0       166      0.39      0.36      0.33         0
09:20:01 AM         0       160      0.31      0.34      0.33         0
09:25:01 AM         0       160      0.24      0.30      0.31         0
09:30:01 AM         1       173      0.39      0.35      0.33         0
09:35:01 AM         1       164      0.34      0.30      0.32         0
09:40:01 AM         0       166      0.20      0.25      0.30         0
09:45:01 AM         0       160      0.48      0.40      0.35         0
09:50:01 AM         1       164      0.36      0.43      0.39         0
09:55:01 AM         0       160      0.64      0.43      0.40         0
10:00:01 AM         1       185      0.40      0.41      0.40         0
10:05:01 AM         1       171      0.48      0.45      0.42         0
10:10:01 AM         0       160      0.29      0.40      0.41         0
10:15:01 AM         2       165      0.36      0.32      0.38         0
10:20:01 AM         1       163      0.26      0.42      0.41         0
10:25:01 AM         0       159      0.09      0.25      0.35         0
10:30:01 AM         2       174      0.24      0.31      0.35         0
10:35:01 AM         0       159      0.31      0.31      0.34         0
10:40:01 AM         0       159      0.39      0.39      0.39         0
10:41:35 AM       LINUX RESTART
10:50:01 AM         0       161      0.38      0.37      0.39         0
10:55:01 AM         0       161      0.19      0.26      0.34         0
11:00:01 AM         0       186      0.45      0.35      0.34         0
11:05:01 AM         0       165      0.11      0.30      0.33         0
11:10:02 AM         0       165      0.64      0.38      0.35         0
11:15:01 AM         1       166      0.27      0.26      0.30         0
11:15:27 AM       LINUX RESTART
11:17:30 AM       LINUX RESTART
11:17:43 AM         1       170      0.37      0.30      0.32         0
11:17:55 AM         1       172      0.39      0.31      0.32         0
11:18:11 AM         1       173      0.53      0.34      0.33         0
11:18:54 AM         1       173      0.74      0.42      0.36         0
11:20:01 AM         2       182      1.56      0.64      0.43         0
11:21:48 AM         3       175      1.95      1.25      0.69         0
11:25:01 AM         4       177      4.73      3.17      1.56         0
11:30:01 AM         6       183      1.36      1.92      1.46         0
11:34:12 AM         0       168      5.49      3.21      2.02         0
11:34:28 AM         1       171      4.60      3.15      2.03         0
11:35:01 AM         2       175      3.25      2.96      2.00         0
11:40:01 AM         0       169      0.44      1.46      1.63         0
11:45:01 AM         0       166      0.22      0.66      1.24         0
11:50:01 AM         0       167      0.42      0.47      1.00         0
11:55:01 AM         0       167      0.28      0.38      0.81         0
12:00:01 PM         3       200      0.54      0.35      0.67         0
12:05:01 PM         0       173      0.58      0.38      0.59         0
12:10:01 PM         0       171      0.24      0.32      0.51         0
12:15:01 PM         0       180      0.43      0.38      0.49         0
12:20:01 PM         0       165      0.23      0.32      0.44         0
12:25:01 PM         1       169      0.26      0.27      0.38         0
12:30:01 PM         5       186      0.17      0.29      0.37         0
12:35:01 PM         0       165      0.36      0.39      0.39         0
12:40:01 PM         1       165      0.18      0.31      0.37         0
12:45:01 PM         0       159      0.18      0.27      0.34         0
12:50:01 PM         0       159      0.36      0.28      0.32         0
12:55:01 PM         0       159      0.26      0.24      0.30         0
01:00:01 PM         1       192      0.27      0.27      0.30         0
01:05:01 PM         1       166      0.22      0.23      0.27         0
01:10:01 PM         1       163      0.16      0.24      0.28         0
01:15:01 PM         0       159      0.30      0.31      0.31         0
01:20:01 PM         0       159      0.53      0.33      0.32         0
01:25:02 PM         1       165      0.43      0.36      0.33         0
01:30:01 PM         0       163      0.31      0.38      0.35         0
01:35:01 PM         2       164      0.42      0.42      0.38         0
01:40:01 PM         0       163      0.27      0.35      0.37         0
01:45:01 PM         0       163      0.26      0.29      0.33         0
01:50:01 PM         0       169      0.18      0.24      0.30         0
01:55:01 PM         0       164      0.38      0.32      0.32         0
02:00:01 PM         5       184      0.29      0.38      0.37         0
02:05:01 PM         0       163      0.36      0.30      0.33         0
02:10:01 PM         1       163      0.63      0.39      0.34         0
02:12:10 PM       LINUX RESTART
02:20:01 PM         1       163      0.21      0.29      0.31         0
02:25:01 PM         0       159      0.36      0.30      0.31         0
02:30:01 PM         3       185      0.38      0.36      0.33         0
02:35:01 PM         0       159      0.28      0.30      0.31         0
02:40:01 PM         1       165      0.18      0.35      0.34         0
02:45:01 PM         0       159      0.30      0.29      0.32         0
02:50:01 PM         0       159      0.18      0.25      0.30         0
02:55:01 PM         1       163      0.44      0.37      0.34         0
03:00:01 PM         0       186      0.58      0.40      0.34         0
03:05:01 PM         0       159      0.29      0.32      0.32         0
03:10:01 PM         1       163      0.49      0.38      0.33         0
03:15:01 PM         2       163      0.32      0.40      0.36         0
03:20:01 PM         0       165      0.53      0.34      0.33         0
03:25:01 PM         0       165      0.14      0.21      0.28         0
03:30:01 PM         0       173      0.15      0.21      0.26         0
03:35:01 PM         1       164      0.33      0.25      0.26         0
03:40:02 PM         0       159      0.17      0.22      0.24         0
03:45:01 PM         0       165      0.38      0.35      0.30         0
03:50:01 PM         0       159      0.22      0.28      0.29         0
03:55:01 PM         0       160      0.44      0.34      0.31         0
04:00:01 PM         1       190      0.15      0.30      0.31         0
04:05:01 PM         0       159      0.16      0.30      0.31         0
04:10:01 PM         4       167      0.22      0.28      0.31         0
04:15:01 PM         1       165      0.38      0.30      0.32         0

------------------------------------------------------------------
Filter sar -q for times cpu load avg (5min) hit/exceeded cpu threads max

11:25:01 AM         4       177      4.73      3.17      1.56         0
11:34:12 AM         0       168      5.49      3.21      2.02         0
11:34:28 AM         1       171      4.60      3.15      2.03         0

------------------------------------------------------------------
sar -q | sed -e "s|$(hostname)|hostname|g"
Linux 3.10.0-957.21.3.el7.x86_64 (hostname)     06/24/2019     _x86_64_    (4 CPU)

12:00:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
12:05:01 AM         0       158      0.19      0.28      0.30         0
12:10:01 AM         0       158      0.11      0.20      0.27         0
12:15:01 AM         0       159      0.25      0.25      0.28         0
12:20:01 AM         0       158      0.14      0.18      0.24         0
12:25:01 AM         0       159      0.25      0.27      0.27         0
12:30:01 AM         0       172      0.24      0.24      0.26         0
12:35:01 AM         0       159      0.45      0.36      0.30         0
12:40:01 AM         0       158      0.51      0.51      0.40         0
12:45:01 AM         1       162      0.47      0.47      0.41         0
12:50:01 AM         0       158      0.19      0.32      0.37         0
12:55:01 AM         0       159      0.50      0.43      0.40         0
01:00:01 AM         1       191      0.63      0.38      0.38         0
01:05:01 AM         0       166      0.35      0.32      0.35         0
01:10:01 AM         1       159      0.33      0.30      0.32         0
01:15:01 AM         1       158      0.16      0.29      0.31         0
01:20:01 AM         0       158      0.33      0.34      0.33         0
01:25:02 AM         0       164      0.36      0.31      0.32         0
01:30:01 AM         0       172      0.24      0.29      0.31         0
01:35:01 AM         1       162      0.39      0.29      0.31         0
01:40:01 AM         0       158      0.26      0.26      0.30         0
01:45:01 AM         1       163      0.33      0.38      0.35         0
01:50:01 AM         2       164      0.11      0.21      0.28         0
01:55:01 AM         0       158      0.37      0.33      0.32         0
02:00:01 AM         6       239      1.19      0.57      0.41         0
02:05:01 AM         0       160      0.18      0.35      0.37         0
02:10:01 AM         0       166      0.34      0.31      0.34         0
02:15:01 AM         0       159      0.19      0.25      0.31         0
02:20:01 AM         0       159      0.33      0.25      0.29         0
02:25:01 AM         0       159      0.26      0.31      0.31         0
02:30:01 AM         6       187      0.71      0.40      0.34         0
02:35:01 AM         1       163      0.33      0.30      0.31         0
02:40:01 AM         0       165      0.16      0.24      0.29         0
02:45:01 AM         0       160      0.36      0.32      0.32         0
02:50:01 AM         0       159      0.50      0.33      0.32         0
02:55:01 AM         2       163      0.44      0.36      0.33         0
03:00:01 AM         0       180      0.27      0.34      0.33         0
03:05:01 AM         1       165      0.20      0.34      0.33         0
03:10:01 AM         0       160      0.37      0.37      0.34         0
03:15:01 AM         0       159      0.33      0.40      0.37         0
03:20:01 AM         0       159      0.36      0.34      0.34         0
03:25:01 AM         0       159      0.34      0.36      0.35         0
03:30:02 AM         0       173      0.37      0.37      0.36         0
03:35:01 AM         0       159      0.20      0.29      0.32         0
03:40:01 AM         5       167      0.32      0.30      0.32         0
03:45:01 AM         1       160      0.30      0.27      0.30         0
03:50:01 AM         0       159      0.27      0.23      0.28         0
03:55:01 AM         0       160      0.10      0.20      0.26         0
04:00:01 AM         2       184      0.38      0.30      0.29         0
04:05:01 AM         0       160      0.42      0.35      0.32         0
04:10:01 AM         0       159      0.32      0.32      0.32         0
04:15:01 AM         0       159      0.31      0.35      0.33         0
04:20:01 AM         1       164      0.39      0.37      0.34         0
04:25:01 AM         0       160      0.24      0.36      0.34         0
04:30:01 AM         1       173      0.47      0.38      0.34         0
04:35:01 AM         0       160      0.46      0.37      0.34         0
04:40:01 AM         0       160      0.22      0.29      0.31         0
04:45:01 AM         1       163      0.41      0.37      0.34         0
04:50:01 AM         1       160      0.39      0.36      0.33         0
04:55:01 AM         2       168      0.20      0.31      0.32         0
05:00:01 AM         1       179      0.19      0.23      0.29         0
05:05:01 AM         1       160      0.23      0.27      0.30         0
05:10:01 AM         0       161      0.51      0.36      0.33         0
05:15:01 AM         0       160      0.34      0.37      0.33         0
05:20:01 AM         1       171      0.25      0.31      0.31         0
05:25:01 AM         0       160      0.39      0.38      0.35         0
05:30:01 AM         0       174      0.67      0.41      0.36         0
05:35:01 AM         1       165      0.24      0.36      0.36         0
05:40:01 AM         1       166      0.29      0.32      0.34         0
05:45:01 AM         0       160      0.51      0.37      0.34         0
05:50:01 AM         0       160      0.54      0.38      0.34         0
05:55:01 AM         0       160      0.30      0.32      0.32         0
06:00:01 AM         0       181      0.12      0.25      0.29         0
06:05:01 AM         0       160      0.29      0.35      0.33         0
06:10:01 AM         0       160      0.28      0.30      0.31         0
06:15:01 AM         0       160      0.59      0.39      0.34         0
06:20:01 AM         0       160      0.34      0.36      0.34         0
06:25:01 AM         0       167      0.76      0.42      0.35         0
06:30:01 AM         2       174      0.40      0.35      0.33         0
06:35:01 AM         1       161      0.36      0.31      0.32         0
06:40:01 AM         0       160      0.19      0.27      0.31         0
06:45:01 AM         0       160      0.32      0.37      0.35         0
06:50:01 AM         1       164      0.32      0.38      0.36         0
06:55:01 AM         0       161      0.25      0.32      0.34         0
07:00:01 AM         2       192      0.66      0.40      0.35         0
07:05:01 AM         0       162      0.26      0.45      0.40         0
07:10:01 AM         0       160      0.36      0.39      0.39         0
07:15:01 AM         1       164      0.18      0.27      0.34         0
07:20:01 AM         2       172      0.12      0.24      0.31         0
07:25:01 AM         0       173      0.47      0.30      0.32         0
07:30:01 AM         4       178      0.40      0.40      0.36         0
07:35:01 AM         0       166      0.34      0.38      0.36         0
07:40:01 AM         1       177      0.37      0.34      0.34         0
07:45:01 AM         1       164      0.44      0.39      0.36         0
07:50:01 AM         0       166      0.44      0.38      0.36         0
07:55:01 AM         0       168      0.45      0.43      0.39         0
08:00:01 AM         2       189      0.51      0.43      0.41         0
08:05:01 AM         0       160      0.15      0.31      0.38         0
08:10:01 AM         1       164      0.35      0.37      0.40         0
08:15:01 AM         0       160      0.46      0.42      0.41         0
08:20:01 AM         1       161      0.17      0.30      0.37         0
08:25:01 AM         0       160      0.31      0.33      0.37         0
08:30:01 AM         4       175      0.22      0.33      0.36         0
08:35:01 AM         1       160      0.30      0.34      0.36         0
08:40:01 AM         2       164      0.44      0.37      0.36         0
08:45:01 AM         0       161      0.38      0.35      0.34         0
08:50:01 AM         0       160      0.23      0.37      0.35         0
08:55:01 AM         0       160      0.31      0.34      0.35         0
09:00:02 AM         4       182      0.39      0.32      0.33         0
09:05:01 AM         0       160      0.40      0.37      0.34         0
09:10:01 AM         0       160      0.41      0.32      0.33         0
09:15:01 AM         0       166      0.39      0.36      0.33         0
09:20:01 AM         0       160      0.31      0.34      0.33         0
09:25:01 AM         0       160      0.24      0.30      0.31         0
09:30:01 AM         1       173      0.39      0.35      0.33         0
09:35:01 AM         1       164      0.34      0.30      0.32         0
09:40:01 AM         0       166      0.20      0.25      0.30         0
09:45:01 AM         0       160      0.48      0.40      0.35         0
09:50:01 AM         1       164      0.36      0.43      0.39         0
09:55:01 AM         0       160      0.64      0.43      0.40         0
10:00:01 AM         1       185      0.40      0.41      0.40         0
10:05:01 AM         1       171      0.48      0.45      0.42         0
10:10:01 AM         0       160      0.29      0.40      0.41         0
10:15:01 AM         2       165      0.36      0.32      0.38         0
10:20:01 AM         1       163      0.26      0.42      0.41         0
10:25:01 AM         0       159      0.09      0.25      0.35         0
10:30:01 AM         2       174      0.24      0.31      0.35         0
10:35:01 AM         0       159      0.31      0.31      0.34         0
10:40:01 AM         0       159      0.39      0.39      0.39         0
Average:            1       165      0.34      0.34      0.34         0

10:41:35 AM       LINUX RESTART

10:45:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
10:50:01 AM         0       161      0.38      0.37      0.39         0
10:55:01 AM         0       161      0.19      0.26      0.34         0
11:00:01 AM         0       186      0.45      0.35      0.34         0
11:05:01 AM         0       165      0.11      0.30      0.33         0
11:10:02 AM         0       165      0.64      0.38      0.35         0
11:15:01 AM         1       166      0.27      0.26      0.30         0
Average:            0       167      0.34      0.32      0.34         0

11:15:27 AM       LINUX RESTART

11:17:30 AM       LINUX RESTART

11:17:38 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
11:17:43 AM         1       170      0.37      0.30      0.32         0
11:17:55 AM         1       172      0.39      0.31      0.32         0
11:18:11 AM         1       173      0.53      0.34      0.33         0
11:18:54 AM         1       173      0.74      0.42      0.36         0
11:20:01 AM         2       182      1.56      0.64      0.43         0
11:21:48 AM         3       175      1.95      1.25      0.69         0
11:25:01 AM         4       177      4.73      3.17      1.56         0
11:30:01 AM         6       183      1.36      1.92      1.46         0
11:34:12 AM         0       168      5.49      3.21      2.02         0
11:34:28 AM         1       171      4.60      3.15      2.03         0
11:35:01 AM         2       175      3.25      2.96      2.00         0
11:40:01 AM         0       169      0.44      1.46      1.63         0
11:45:01 AM         0       166      0.22      0.66      1.24         0
11:50:01 AM         0       167      0.42      0.47      1.00         0
11:55:01 AM         0       167      0.28      0.38      0.81         0
12:00:01 PM         3       200      0.54      0.35      0.67         0
12:05:01 PM         0       173      0.58      0.38      0.59         0
12:10:01 PM         0       171      0.24      0.32      0.51         0
12:15:01 PM         0       180      0.43      0.38      0.49         0
12:20:01 PM         0       165      0.23      0.32      0.44         0
12:25:01 PM         1       169      0.26      0.27      0.38         0
12:30:01 PM         5       186      0.17      0.29      0.37         0
12:35:01 PM         0       165      0.36      0.39      0.39         0
12:40:01 PM         1       165      0.18      0.31      0.37         0
12:45:01 PM         0       159      0.18      0.27      0.34         0
12:50:01 PM         0       159      0.36      0.28      0.32         0
12:55:01 PM         0       159      0.26      0.24      0.30         0
01:00:01 PM         1       192      0.27      0.27      0.30         0
01:05:01 PM         1       166      0.22      0.23      0.27         0
01:10:01 PM         1       163      0.16      0.24      0.28         0
01:15:01 PM         0       159      0.30      0.31      0.31         0
01:20:01 PM         0       159      0.53      0.33      0.32         0
01:25:02 PM         1       165      0.43      0.36      0.33         0
01:30:01 PM         0       163      0.31      0.38      0.35         0
01:35:01 PM         2       164      0.42      0.42      0.38         0
01:40:01 PM         0       163      0.27      0.35      0.37         0
01:45:01 PM         0       163      0.26      0.29      0.33         0
01:50:01 PM         0       169      0.18      0.24      0.30         0
01:55:01 PM         0       164      0.38      0.32      0.32         0
02:00:01 PM         5       184      0.29      0.38      0.37         0
02:05:01 PM         0       163      0.36      0.30      0.33         0
02:10:01 PM         1       163      0.63      0.39      0.34         0
Average:            1       170      0.84      0.70      0.63         0

02:12:10 PM       LINUX RESTART

02:15:01 PM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
02:20:01 PM         1       163      0.21      0.29      0.31         0
02:25:01 PM         0       159      0.36      0.30      0.31         0
02:30:01 PM         3       185      0.38      0.36      0.33         0
02:35:01 PM         0       159      0.28      0.30      0.31         0
02:40:01 PM         1       165      0.18      0.35      0.34         0
02:45:01 PM         0       159      0.30      0.29      0.32         0
02:50:01 PM         0       159      0.18      0.25      0.30         0
02:55:01 PM         1       163      0.44      0.37      0.34         0
03:00:01 PM         0       186      0.58      0.40      0.34         0
03:05:01 PM         0       159      0.29      0.32      0.32         0
03:10:01 PM         1       163      0.49      0.38      0.33         0
03:15:01 PM         2       163      0.32      0.40      0.36         0
03:20:01 PM         0       165      0.53      0.34      0.33         0
03:25:01 PM         0       165      0.14      0.21      0.28         0
03:30:01 PM         0       173      0.15      0.21      0.26         0
03:35:01 PM         1       164      0.33      0.25      0.26         0
03:40:02 PM         0       159      0.17      0.22      0.24         0
03:45:01 PM         0       165      0.38      0.35      0.30         0
03:50:01 PM         0       159      0.22      0.28      0.29         0
03:55:01 PM         0       160      0.44      0.34      0.31         0
04:00:01 PM         1       190      0.15      0.30      0.31         0
04:05:01 PM         0       159      0.16      0.30      0.31         0
04:10:01 PM         4       167      0.22      0.28      0.31         0
04:15:01 PM         1       165      0.38      0.30      0.32         0
Average:            1       166      0.30      0.31      0.31         0

------------------------------------------------------------------
sar -r | sed -e "s|$(hostname)|hostname|g"
Linux 3.10.0-957.21.3.el7.x86_64 (hostname)     06/24/2019     _x86_64_    (4 CPU)

12:00:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
12:05:01 AM    244980   7763876     96.94     92160     76424   6761856     74.66    719504    891624       172
12:10:01 AM    174544   7834312     97.82     88244     76716   6761852     74.66    717440    889952       140
12:15:01 AM    168776   7840080     97.89     85024     75516   6761852     74.66    715680    887080       180
12:20:01 AM    234920   7773936     97.07     68716     72984   6761852     74.66    707392    876536       148
12:25:01 AM    179680   7829176     97.76     63620     73376   6762880     74.67    707920    872904       128
12:30:01 AM    172072   7836784     97.85     51180     72648   6794972     75.02    713376    864264       160
12:35:01 AM    156536   7852320     98.05     46312     69916   6762876     74.67    703092    856956        48
12:40:01 AM    184228   7824628     97.70     34844     69240   6762876     74.67    698864    848940       148
12:45:01 AM    156604   7852252     98.04     25264     76872   6798660     75.06    717464    855232       192
12:50:01 AM    186164   7822692     97.68     17676     75052   6771072     74.76    687852    848284       168
12:55:01 AM    210024   7798832     97.38     13168     68772   6771020     74.76    686088    839460        24
01:00:01 AM    158552   7850304     98.02     13020     78160   6833488     75.45    708348    843972       204
01:05:01 AM    190204   7818652     97.63     12404     78000   6787112     74.93    700828    837660        80
01:10:01 AM    211968   7796888     97.35     12564     66668   6771072     74.76    688952    833204       148
01:15:01 AM    189524   7819332     97.63     11592     63916   6771068     74.76    686716    831840       164
01:20:01 AM    220904   7787952     97.24     11248     65836   6771068     74.76    691172    828700       152
01:25:02 AM    225548   7783308     97.18     11012     60756   6784948     74.91    691308    826768       124
01:30:01 AM    166260   7842596     97.92     11660     69644   6803104     75.11    698816    834880       188
01:35:01 AM    166948   7841908     97.92     11408     67508   6804848     75.13    722672    831332        80
01:40:01 AM    212804   7796052     97.34     11756     60564   6771036     74.76    688028    826672       164
01:45:01 AM    183396   7825460     97.71     11496     66604   6804848     75.13    725496    827304       172
01:50:01 AM    207960   7800896     97.40     11520     67980   6783908     74.90    694936    828544       184
01:55:01 AM    286788   7722068     96.42     11164     65220   6771036     74.76    695180    822068       184
02:00:01 AM    118812   7890044     98.52     11600     75368   6874288     75.90    730720    831312       236
02:05:01 AM    205260   7803596     97.44     12876     79904   6771368     74.76    694112    834572       172
02:10:01 AM    202880   7805976     97.47     13132     75940   6785248     74.91    698604    831788       184
02:15:01 AM    203600   7805256     97.46     12740     72728   6771372     74.76    692860    829644       200
02:20:01 AM    202916   7805940     97.47     11932     70088   6771340     74.76    690792    828088       176
02:25:01 AM    182440   7826416     97.72     11692     68644   6771372     74.76    691916    825088       204
02:30:01 AM    115244   7893612     98.56     11884     70180   6871768     75.87    765484    827260       160
02:35:01 AM    160816   7848040     97.99     12040     68096   6805148     75.13    724492    824084       152
02:40:01 AM    201464   7807392     97.48     11328     63380   6785212     74.91    694804    818400       148
02:45:01 AM    197548   7811308     97.53     11552     63292   6771364     74.76    690000    819192       124
02:50:01 AM    220572   7788284     97.25     11504     62528   6771340     74.76    689052    818620       180
02:55:01 AM    221728   7787128     97.23     11700     62592   6775560     74.81    695176    815644       192
03:00:01 AM    160600   7848256     97.99     11408     69932   6819508     75.29    702796    823336       180
03:05:01 AM    161896   7846960     97.98     12376     74308   6805656     75.14    727012    824176       160
03:10:01 AM    192268   7816588     97.60     12856     66456   6771872     74.77    693136    817512       188
03:15:01 AM    155744   7853112     98.06     71876     91708   6771372     74.76    730208    867844       148
03:20:01 AM    205064   7803792     97.44     64252     82856   6771372     74.76    728272    853456       172
03:25:01 AM    201988   7806868     97.48     60788     75348   6771368     74.76    720208    850692       184
03:30:02 AM    184568   7824288     97.70     54844     69968   6803456     75.11    715532    853824       208
03:35:01 AM    237096   7771760     97.04     41748     68328   6771368     74.76    697588    847056       840
03:40:01 AM    137588   7871268     98.28     39764     64196   6838936     75.51    760396    844056       140
03:45:01 AM    179840   7829016     97.75     33096     65400   6771796     74.77    695436    837592        48
03:50:01 AM    185920   7822936     97.68     24256     63468   6771368     74.76    694888    827116        48
03:55:01 AM    174796   7834060     97.82     18544     64552   6771368     74.76    694684    822808       180
04:00:01 AM    156888   7851968     98.04     13588     70808   6821980     75.32    707208    823656       136
04:05:01 AM    228080   7780776     97.15     12024     71516   6771340     74.76    697652    820068       204
04:10:01 AM    256192   7752664     96.80     12092     63184   6771372     74.76    691548    817808       180
04:15:01 AM    190944   7817912     97.62     12220     61720   6771368     74.76    693388    814364       180
04:20:01 AM    151672   7857184     98.11     11908     63884   6815904     75.25    734132    810888       240
04:25:01 AM    157580   7851276     98.03     11720     62212   6782120     74.88    700844    809388       108
04:30:01 AM    194016   7814840     97.58     11180     67360   6799760     75.07    704516    813588       148
04:35:01 AM    217160   7791696     97.29     11352     65996   6782128     74.88    704056    808392       220
04:40:01 AM    225428   7783428     97.19     11548     61336   6782088     74.88    700024    806792       112
04:45:01 AM    156936   7851920     98.04     11604     60448   6805148     75.13    731904    804048       172
04:50:01 AM    221816   7787040     97.23     11296     60996   6782132     74.88    702256    802980        44
04:55:01 AM    240492   7768364     97.00     11556     60144   6785488     74.92    702508    803196       156
05:00:01 AM    149876   7858980     98.13     11540     67572   6848988     75.62    747136    807724       176
05:05:01 AM    258124   7750732     96.78     12072     63868   6782128     74.88    702996    804968       184
05:10:01 AM    181868   7826988     97.73     12676     64848   6782132     74.88    703432    805768       196
05:15:01 AM    240384   7768472     97.00     11280     73688   6782100     74.88    708068    807576       172
05:20:01 AM    151148   7857708     98.11     12052     73840   6796304     75.04    712012    807772       192
05:25:01 AM    204480   7804376     97.45     12408     66476   6782124     74.88    702340    806928       120
05:30:01 AM    192156   7816700     97.60     11868     63212   6814220     75.23    712680    802484       140
05:35:01 AM    245500   7763356     96.93     11244     64248   6784956     74.91    707956    798328       176
05:40:01 AM    199824   7809032     97.50     11476     62504   6796004     75.03    707748    799544       156
05:45:01 AM    201664   7807192     97.48     11440     60712   6782132     74.88    703896    797656       108
05:50:01 AM    201368   7807488     97.49     11008     63748   6782100     74.88    706912    797128       164
05:55:01 AM    204408   7804448     97.45     11316     59532   6782128     74.88    704660    795076         0
06:00:01 AM    128216   7880640     98.40     11376     67960   6830264     75.41    722736    800372       120
06:05:01 AM    194924   7813932     97.57     11916     67128   6782100     74.88    704108    803700       116
06:10:01 AM    201668   7807188     97.48     12044     66520   6782128     74.88    706244    800596       172
06:15:01 AM    260968   7747888     96.74     11252     61608   6782096     74.88    705520    795248       152
06:20:01 AM    225744   7783112     97.18     11724     63812   6782128     74.88    706376    797100       144
06:25:01 AM    185576   7823280     97.68     11612     63088   6796004     75.03    711356    794652       160
06:30:01 AM    165944   7842912     97.93     11888     69492   6812152     75.21    715204    801384       252
06:35:01 AM    206656   7802200     97.42     11596     67444   6789988     74.97    707176    798752       172
06:40:01 AM    190868   7817988     97.62     11936     63004   6782128     74.88    705924    795412       172
06:45:01 AM    189760   7819096     97.63     11784     63308   6782096     74.88    705924    795080       240
06:50:01 AM    154380   7854476     98.07     11672     64724   6815908     75.25    743040    791052       168
06:55:01 AM    187448   7821408     97.66     11484     64528   6782128     74.88    706844    792640       108
07:00:01 AM    141448   7867408     98.23     11348     68256   6852112     75.65    743608    802664       136
07:05:01 AM    195536   7813320     97.56     12104     71056   6782128     74.88    700564    805524       140
07:10:01 AM    187212   7821644     97.66     12216     67760   6782128     74.88    696816    805292       204
07:15:01 AM    178684   7830172     97.77     11516     63472   6815916     75.25    727548    802248       156
07:20:01 AM    213152   7795704     97.34     11544     63204   6796612     75.04    699288    799128       144
07:25:01 AM    213696   7795160     97.33     11380     64184   6795736     75.03    700216    798200        52
07:30:01 AM    133264   7875592     98.34     11708     67800   6847248     75.60    735904    804616        72
07:35:01 AM    205192   7803664     97.44     11376     64736   6796008     75.03    697740    801212       164
07:40:01 AM    176496   7832360     97.80     11848     63132   6810084     75.19    701012    798976        76
07:45:01 AM    159340   7849516     98.01     11492     62320   6815912     75.25    730656    793600        44
07:50:01 AM    227888   7780968     97.15     11288     61348   6795976     75.03    699348    793152       160
07:55:01 AM    219852   7789004     97.25     11444     62048   6796004     75.03    699868    792812       140
08:00:01 AM    133200   7875656     98.34     11852     67200   6868436     75.83    746668    796956       208
08:05:01 AM    220248   7788608     97.25     11620     67720   6782128     74.88    699676    794112       148
08:10:01 AM    172788   7836068     97.84     11568     62808   6815912     75.25    731912    789348       192
08:15:01 AM    194176   7814680     97.58     11340     73064   6782076     74.88    696672    801540       140
08:20:01 AM    195076   7813780     97.56     11592     71532   6782264     74.88    700236    796436       160
08:25:01 AM    175424   7833432     97.81     14908     68416   6782132     74.88    697688    798744       196
08:30:01 AM    172192   7836664     97.85     12180     65100   6855088     75.68    759196    800024        60
08:35:01 AM    181488   7827368     97.73     12484     70100   6782100     74.88    700408    795232       188
08:40:01 AM    152576   7856280     98.09     12896     69752   6808000     75.16    726344    793928       196
08:45:01 AM    198660   7810196     97.52     12380     64728   6782132     74.88    695796    792792       240
08:50:01 AM    189384   7819472     97.64     12236     62224   6782128     74.88    695796    789564       196
08:55:01 AM    166436   7842420     97.92     11956     64576   6782128     74.88    698488    788328       128
09:00:02 AM    167648   7841208     97.91     11840     68104   6827480     75.38    711028    790232       208
09:05:01 AM    224232   7784624     97.20     11828     66872   6782128     74.88    696140    790532       228
09:10:01 AM    212796   7796060     97.34     11880     65212   6782128     74.88    698004    786560       256
09:15:01 AM    182140   7826716     97.73     11684     63780   6796004     75.03    701172    784820       168
09:20:01 AM    223920   7784936     97.20     11460     63916   6782100     74.88    703144    780628       180
09:25:01 AM    190428   7818428     97.62     11504     62224   6782136     74.88    701364    779688       156
09:30:01 AM    199988   7808868     97.50     12184     67992   6802172     75.10    709580    784772       176
09:35:01 AM    187632   7821224     97.66     11680     68644   6815908     75.25    735048    784100       196
09:40:01 AM    202424   7806432     97.47     12232     64456   6796008     75.03    703900    781800       216
09:45:01 AM    180592   7828264     97.75     13512     63472   6782136     74.88    701012    781204       144
09:50:01 AM    155208   7853648     98.06     13596     64196   6811928     75.21    731304    779844       176
09:55:01 AM    186200   7822656     97.68     12800     64160   6782128     74.88    701748    778952       216
10:00:01 AM    125956   7882900     98.43     12312     67548   6864048     75.78    751340    779352       188
10:05:01 AM    174164   7834692     97.83     14104     70132   6796180     75.03    706684    782452       132
10:10:01 AM    204164   7804692     97.45     14036     67096   6782128     74.88    703352    779204       184
10:15:01 AM    142316   7866540     98.22     14112     65252   6815912     75.25    733432    778416       176
10:20:01 AM    157448   7851408     98.03     13048     63960   6805148     75.13    731976    775392       180
10:25:01 AM    202536   7806320     97.47     12800     63888   6771372     74.76    698596    774148       200
10:30:01 AM    157364   7851492     98.04     11960     69156   6801268     75.09    705628    778780        48
10:35:01 AM    192116   7816740     97.60     11752     67436   6771380     74.76    699820    774436       184
10:40:01 AM    239180   7769676     97.01     11624     62296   6771332     74.76    694696    773520       148
Average:       190094   7818762     97.63     17874     67269   6792272     74.99    708719    812052       164

10:41:35 AM       LINUX RESTART

10:45:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
10:50:01 AM    272092   7736764     96.60     89084    420264   6783168     74.89    863928   1044336       160
10:55:01 AM    245464   7763392     96.94     86604    416784   6783244     74.89    854824   1047768       200
11:00:01 AM    182852   7826004     97.72     78092    423600   6841760     75.54    867848   1060936       256
11:05:01 AM    191088   7817768     97.61     65360    422856   6793624     75.01    842712   1057568       188
11:10:02 AM    216840   7792016     97.29     57520    410512   6793620     75.01    835300   1044780       208
11:15:01 AM    197040   7811816     97.54     45068    412624   6791780     74.99    822088   1044596        84
Average:       217563   7791293     97.28     70288    417773   6797866     75.05    847783   1049997       183

11:15:27 AM       LINUX RESTART

11:17:30 AM       LINUX RESTART

11:17:38 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
11:17:43 AM    295456   7713400     96.31     47384    443800   6849144     75.62    895200   1049872      1964
11:17:55 AM    281588   7727268     96.48     47976    452496   6862980     75.77    903052   1051528      2760
11:18:11 AM    297400   7711456     96.29     48032    456308   6845672     75.58    880216   1051936      1752
11:18:54 AM    279588   7729268     96.51     55812    510788   6873736     75.89   1025948    996080     24660
11:20:01 AM    310320   7698536     96.13     70124    824620   6834364     75.46   1084972   1235776     68212
11:21:48 AM    621596   7387260     92.24     71208    956216   6872276     75.87   1190964   1294180      2004
11:25:01 AM   1919240   6089616     76.04     39904    678984   6874680     75.90    806000   1356152     10088
11:30:01 AM   1547640   6461216     80.68     42548    870932   6941080     76.63   1018336   1401700        12
11:34:12 AM   1627260   6381596     79.68     46260    781364   6849744     75.63    895892   1358744     70944
11:34:28 AM   1491936   6516920     81.37     47848    875516   6863804     75.78    943420   1435960    103928
11:35:01 AM   1482316   6526540     81.49     49660    894148   6859264     75.73    949644   1422912     31928
11:40:01 AM   1353016   6655840     83.11     51700    914524   6838644     75.50   1001780   1390664       132
11:45:01 AM   1270936   6737920     84.13     51964    914840   6820512     75.30   1001576   1390852       140
11:50:01 AM   1161536   6847320     85.50     52408    916280   6838644     75.50   1002644   1392112       180
11:55:01 AM   1056844   6952012     86.80     53064    915876   6838648     75.50   1003076   1391884       180
12:00:01 PM    878844   7130012     89.03     53592    917256   6930332     76.52   1057148   1393180        60
12:05:01 PM    821764   7187092     89.74     53968    917936   6844884     75.57   1008548   1393608       184
12:10:01 PM    716072   7292784     91.06     54332    918332   6844888     75.57   1008876   1393968       192
12:15:01 PM    462480   7546376     94.23     55904    968528   6941832     76.64   1143972   1396660      1744
12:20:01 PM    489928   7518928     93.88     57964    969120   6771708     74.76   1037568   1375476       148
12:25:01 PM    343012   7665844     95.72     58196    969500   6805456     75.14   1071212   1375572       204
12:30:01 PM    196668   7812188     97.54     58428    969932   6857532     75.71   1109924   1375724       184
12:35:01 PM    224880   7783976     97.19     58496    958436   6770256     74.75   1027032   1375128        40
12:40:01 PM    220984   7787872     97.24     58784    940748   6762824     74.67    986116   1382248       128
12:45:01 PM    175528   7833328     97.81     59100    928940   6750116     74.53    966644   1387936       160
12:50:01 PM    234204   7774652     97.08     59460    897864   6750116     74.53    924720   1399136       164
12:55:01 PM    193308   7815548     97.59     59756    885464   6750112     74.53    909128   1402552       160
01:00:01 PM    167924   7840932     97.90     60052    866696   6803028     75.11    903812   1406412       144
01:05:01 PM    215824   7793032     97.31     56416    844952   6762928     74.67    860644   1409580       156
01:10:01 PM    171992   7836864     97.85     56356    832648   6765420     74.69    859740   1410256       136
01:15:01 PM    183168   7825688     97.71     56216    814028   6750116     74.53    829088   1407504        36
01:20:01 PM    285116   7723740     96.44     49956    783508   6750080     74.53    808560   1391416       180
01:25:02 PM    246336   7762520     96.92     50148    773256   6752608     74.55    804732   1386512        64
01:30:01 PM    207256   7801600     97.41     50008    762096   6756208     74.59    799760   1380812        96
01:35:01 PM    163020   7845836     97.96     47080    753560   6761680     74.65    794272   1374912       208
01:40:01 PM    214944   7793912     97.32     46820    725468   6756960     74.60    785160   1358884       160
01:45:01 PM    204652   7804204     97.44     46556    709288   6756956     74.60    778040   1349632       104
01:50:01 PM    184496   7824360     97.70     46880    692944   6770828     74.75    772048   1343384       148
01:55:01 PM    240316   7768540     97.00     47140    665840   6756956     74.60    753620   1331220        96
02:00:01 PM    194052   7814804     97.58     47004    655244   6792396     74.99    758404   1325888       176
02:05:01 PM    194324   7814532     97.57     44828    642464   6756948     74.60    740640   1318416       160
02:10:01 PM    151344   7857512     98.11     45024    639804   6783896     74.90    767032   1317172       168
Average:       547122   7461734     93.17     52723    805013   6812387     75.21    925456   1342465      7721

02:12:10 PM       LINUX RESTART

02:15:01 PM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
02:20:01 PM    280684   7728172     96.50     54888    672276   6752384     74.55    733328   1361452        24
02:25:01 PM    171416   7837440     97.86     55204    672680   6750248     74.53    733020   1361676       108
02:30:01 PM    168476   7840380     97.90     53496    654508   6784280     74.90    725404   1359228       204
02:35:01 PM    199204   7809652     97.51     52956    635072   6750248     74.53    702156   1352716       176
02:40:01 PM    210684   7798172     97.37     52480    615920   6764128     74.68    695292   1343488       104
02:45:01 PM    227452   7781404     97.16     51968    599432   6750244     74.53    682268   1335720       140
02:50:01 PM    176992   7831864     97.79     51952    590720   6750244     74.53    678904   1330740       108
02:55:01 PM    154992   7853864     98.06     51296    574824   6782052     74.88    702024   1321852        28
03:00:01 PM    163760   7845096     97.96     51132    564320   6812252     75.21    687424   1313352       136
03:05:01 PM    190692   7818164     97.62     50500    549088   6750248     74.53    657836   1308500       156
03:10:01 PM    161532   7847324     97.98     50044    532452   6784028     74.90    681084   1300496        36
03:15:01 PM    221568   7787288     97.23     47988    513884   6751528     74.54    639324   1290168       164
03:20:01 PM    239140   7769716     97.01     41552    500276   6764120     74.68    632008   1280128       172
03:25:01 PM    182648   7826208     97.72     37184    497328   6764120     74.68    628144   1276448       120
03:30:01 PM    380260   7628596     95.25     32744    488996   6782340     74.88    628424   1269980       164
03:35:01 PM    250440   7758416     96.87     33060    489332   6784028     74.90    652052   1270256       192
03:40:02 PM    179472   7829384     97.76     33400    489660   6750248     74.53    619364   1270580       160
03:45:01 PM    189772   7819084     97.63     28400    477372   6764120     74.68    613700   1262160       116
03:50:01 PM    203864   7804992     97.45     28664    461564   6750244     74.53    602716   1254336       156
03:55:01 PM    198396   7810460     97.52     28956    446656   6750248     74.53    595264   1247000       152
04:00:01 PM    172192   7836664     97.85     29224    432452   6815644     75.25    607980   1240108       120
04:05:01 PM    188712   7820144     97.64     29636    418960   6750244     74.53    582040   1233444       128
04:10:01 PM    122876   7885980     98.47     29908    409464   6799476     75.07    624520   1228864       160
04:15:01 PM    157048   7851808     98.04     30056    395484   6787448     74.94    605852   1222028       220
Average:       199678   7809178     97.51     41945    528447   6768507     74.73    654589   1293113       135

------------------------------------------------------------------
sar -u | sed -e "s|$(hostname)|hostname|g"
Linux 3.10.0-957.21.3.el7.x86_64 (hostname)     06/24/2019     _x86_64_    (4 CPU)

12:00:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
12:05:01 AM     all      3.39      0.00      7.13      0.03      0.02     89.43
12:10:01 AM     all      2.76      0.00      5.86      0.01      0.02     91.35
12:15:01 AM     all      3.11      0.00      6.67      0.02      0.02     90.18
12:20:01 AM     all      2.86      0.00      6.25      0.02      0.02     90.85
12:25:01 AM     all      3.17      0.00      6.76      0.02      0.02     90.04
12:30:01 AM     all      2.82      0.00      6.14      0.01      0.02     91.02
12:35:01 AM     all      3.36      0.00      7.30      0.02      0.02     89.30
12:40:01 AM     all      2.94      0.00      6.50      0.02      0.02     90.53
12:45:01 AM     all      3.04      0.00      6.69      0.02      0.02     90.24
12:50:01 AM     all      2.93      0.00      6.46      0.02      0.02     90.58
12:55:01 AM     all      3.01      0.00      7.07      0.02      0.02     89.89
01:00:01 AM     all      2.94      0.00      6.46      0.02      0.01     90.56
01:05:01 AM     all      3.30      0.00      7.23      0.02      0.02     89.43
01:10:01 AM     all      2.93      0.00      6.42      0.02      0.02     90.62
01:15:01 AM     all      3.03      0.00      6.60      0.01      0.02     90.34
01:20:01 AM     all      2.76      0.00      6.21      0.02      0.02     90.99
01:25:02 AM     all      3.04      0.00      6.73      0.02      0.02     90.19
01:30:01 AM     all      2.92      0.00      6.51      0.02      0.02     90.54
01:35:01 AM     all      3.36      0.00      7.39      0.02      0.02     89.22
01:40:01 AM     all      2.75      0.00      6.17      0.02      0.02     91.04
01:45:01 AM     all      2.94      0.00      6.60      0.02      0.02     90.42
01:50:01 AM     all      2.78      0.00      6.15      0.02      0.02     91.03
01:55:01 AM     all      3.03      0.00      6.72      0.02      0.02     90.21
02:00:01 AM     all      2.83      0.00      6.21      0.01      0.02     90.92
02:05:01 AM     all      4.57      0.00      9.69      0.02      0.04     85.68
02:10:01 AM     all      2.77      0.00      6.15      0.02      0.02     91.04
02:15:01 AM     all      3.18      0.00      6.98      0.02      0.02     89.80
02:20:01 AM     all      2.75      0.00      6.24      0.02      0.02     90.98
02:25:01 AM     all      2.98      0.00      6.77      0.02      0.02     90.21
02:30:01 AM     all      3.06      0.00      6.91      0.03      0.02     89.98
02:35:01 AM     all      3.19      0.00      7.15      0.02      0.02     89.62
02:40:01 AM     all      2.96      0.00      6.74      0.03      0.03     90.24
02:45:01 AM     all      2.91      0.00      6.67      0.02      0.02     90.38
02:50:01 AM     all      2.95      0.00      6.82      0.03      0.02     90.18
02:55:01 AM     all      2.91      0.00      6.70      0.02      0.02     90.35
03:00:01 AM     all      2.87      0.00      6.60      0.02      0.02     90.50
03:05:01 AM     all      3.48      0.00      7.68      0.06      0.03     88.75
03:10:01 AM     all      2.95      0.00      6.74      0.01      0.02     90.27
03:15:01 AM     all      2.97      0.05      6.93      0.21      0.02     89.81
03:20:01 AM     all      2.93      0.00      6.80      0.02      0.02     90.23
03:25:01 AM     all      2.92      0.00      6.67      0.02      0.02     90.37
03:30:02 AM     all      2.94      0.00      6.71      0.02      0.03     90.31
03:35:01 AM     all      3.24      0.00      7.54      0.02      0.02     89.18
03:40:01 AM     all      2.97      0.00      6.83      0.02      0.02     90.16
03:45:01 AM     all      2.95      0.00      6.90      0.02      0.02     90.11
03:50:01 AM     all      2.91      0.00      6.79      0.02      0.02     90.27
03:55:01 AM     all      2.94      0.00      6.87      0.02      0.02     90.15
04:00:01 AM     all      3.05      0.00      7.03      0.02      0.02     89.88
04:05:01 AM     all      3.17      0.00      7.40      0.02      0.03     89.38
04:10:01 AM     all      2.78      0.00      6.54      0.02      0.02     90.63
04:15:01 AM     all      2.94      0.00      6.92      0.02      0.02     90.10
04:20:01 AM     all      2.83      0.00      6.73      0.02      0.02     90.39
04:25:01 AM     all      2.89      0.00      6.85      0.02      0.02     90.22
04:30:01 AM     all      2.92      0.00      6.75      0.02      0.02     90.29
04:35:01 AM     all      3.20      0.00      7.52      0.02      0.02     89.24
04:40:01 AM     all      2.94      0.00      6.89      0.03      0.03     90.12
04:45:01 AM     all      3.03      0.00      7.15      0.02      0.02     89.78
04:50:01 AM     all      2.82      0.00      6.58      0.02      0.02     90.56
04:55:01 AM     all      2.85      0.00      6.75      0.02      0.02     90.35
05:00:01 AM     all      3.01      0.00      7.07      0.02      0.02     89.88
05:05:01 AM     all      3.17      0.00      7.51      0.03      0.03     89.26
05:10:01 AM     all      2.82      0.00      6.81      0.02      0.02     90.33
05:15:01 AM     all      2.97      0.00      7.02      0.02      0.02     89.96
05:20:01 AM     all      2.77      0.00      6.76      0.03      0.02     90.42
05:25:01 AM     all      3.02      0.00      7.56      0.03      0.03     89.37
05:30:01 AM     all      2.85      0.00      6.86      0.05      0.02     90.23
05:35:01 AM     all      3.19      0.00      7.64      0.03      0.03     89.11
05:40:01 AM     all      2.96      0.00      7.09      0.02      0.02     89.90
05:45:01 AM     all      2.93      0.00      7.16      0.02      0.02     89.87
05:50:01 AM     all      2.76      0.00      6.77      0.02      0.02     90.43
05:55:01 AM     all      2.83      0.00      6.86      0.02      0.02     90.27
06:00:01 AM     all      2.84      0.00      6.90      0.02      0.02     90.21
06:05:01 AM     all      3.23      0.00      7.89      0.03      0.05     88.81
06:10:01 AM     all      2.79      0.00      6.87      0.02      0.02     90.29
06:15:01 AM     all      3.04      0.00      7.46      0.02      0.03     89.45
06:20:01 AM     all      2.70      0.00      6.54      0.02      0.02     90.72
06:25:01 AM     all      3.02      0.00      7.49      0.02      0.02     89.44
06:30:01 AM     all      2.80      0.00      6.92      0.02      0.02     90.23
06:35:01 AM     all      3.16      0.00      7.75      0.03      0.03     89.03
06:40:01 AM     all      2.86      0.00      7.08      0.02      0.02     90.01
06:45:01 AM     all      2.98      0.00      7.35      0.02      0.02     89.63
06:50:01 AM     all      2.78      0.00      6.93      0.02      0.02     90.25
06:55:01 AM     all      2.98      0.00      7.40      0.02      0.02     89.57
07:00:01 AM     all      2.82      0.00      7.03      0.02      0.02     90.10
07:05:01 AM     all      3.46      0.00      8.53      0.02      0.04     87.96
07:10:01 AM     all      2.83      0.00      6.98      0.02      0.02     90.16
07:15:01 AM     all      3.00      0.00      7.52      0.02      0.02     89.43
07:20:01 AM     all      2.77      0.00      6.94      0.02      0.02     90.24
07:25:01 AM     all      3.11      0.00      7.71      0.02      0.04     89.12
07:30:01 AM     all      2.67      0.00      6.60      0.02      0.03     90.68
07:35:01 AM     all      3.23      0.00      8.04      0.02      0.03     88.68
07:40:01 AM     all      2.84      0.00      7.15      0.02      0.02     89.97
07:45:01 AM     all      3.08      0.00      7.74      0.02      0.02     89.13
07:50:01 AM     all      2.78      0.00      7.01      0.02      0.02     90.17
07:55:01 AM     all      3.04      0.00      7.63      0.02      0.02     89.28
08:00:01 AM     all      2.87      0.00      7.14      0.02      0.03     89.94
08:05:01 AM     all      3.25      0.00      8.20      0.03      0.03     88.49
08:10:01 AM     all      2.78      0.00      6.93      0.02      0.02     90.24
08:15:01 AM     all      3.21      0.00      7.83      0.03      0.02     88.90
08:20:01 AM     all      2.74      0.00      6.98      0.03      0.02     90.23
08:25:01 AM     all      3.09      0.00      7.61      0.03      0.02     89.25
08:30:01 AM     all      2.83      0.00      7.09      0.02      0.03     90.03
08:35:01 AM     all      3.21      0.00      8.01      0.02      0.03     88.73
08:40:01 AM     all      2.91      0.00      7.38      0.03      0.02     89.66
08:45:01 AM     all      2.97      0.00      7.50      0.02      0.02     89.49
08:50:01 AM     all      2.79      0.00      7.11      0.02      0.02     90.06
08:55:01 AM     all      3.06      0.00      7.74      0.02      0.02     89.16
09:00:02 AM     all      2.76      0.00      6.91      0.02      0.03     90.28
09:05:01 AM     all      3.44      0.00      8.65      0.02      0.03     87.86
09:10:01 AM     all      2.78      0.00      7.04      0.04      0.02     90.12
09:15:01 AM     all      2.89      0.00      7.32      0.02      0.02     89.74
09:20:01 AM     all      2.79      0.00      7.11      0.02      0.02     90.06
09:25:01 AM     all      3.03      0.00      7.65      0.03      0.02     89.26
09:30:01 AM     all      2.88      0.00      7.25      0.04      0.02     89.81
09:35:01 AM     all      3.16      0.00      8.03      0.03      0.02     88.76
09:40:01 AM     all      3.05      0.00      7.62      0.02      0.02     89.29
09:45:01 AM     all      2.93      0.00      7.45      0.02      0.02     89.58
09:50:01 AM     all      2.93      0.00      7.45      0.03      0.02     89.58
09:55:01 AM     all      3.06      0.00      7.96      0.02      0.02     88.93
10:00:01 AM     all      2.88      0.00      7.24      0.02      0.02     89.84
10:05:01 AM     all      3.23      0.00      8.16      0.02      0.03     88.56
10:10:01 AM     all      2.81      0.00      7.07      0.02      0.02     90.09
10:15:01 AM     all      2.94      0.00      7.35      0.02      0.02     89.66
10:20:01 AM     all      2.79      0.00      7.05      0.02      0.02     90.12
10:25:01 AM     all      3.00      0.00      7.56      0.03      0.02     89.38
10:30:01 AM     all      2.86      0.00      7.27      0.02      0.02     89.82
10:35:01 AM     all      3.17      0.00      7.95      0.02      0.02     88.83
10:40:01 AM     all      3.14      0.00      7.71      0.02      0.02     89.10
Average:        all      2.98      0.00      7.10      0.02      0.02     89.87

10:41:35 AM       LINUX RESTART

10:45:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
10:50:01 AM     all      2.86      0.00      6.57      0.03      0.02     90.52
10:55:01 AM     all      2.98      0.00      6.94      0.02      0.02     90.03
11:00:01 AM     all      3.02      0.00      6.92      0.02      0.02     90.01
11:05:01 AM     all      3.40      0.00      7.76      0.02      0.03     88.80
11:10:02 AM     all      2.96      0.00      6.88      0.02      0.02     90.12
11:15:01 AM     all      3.10      0.00      6.96      0.02      0.02     89.90
Average:        all      3.05      0.00      7.00      0.02      0.02     89.90

11:15:27 AM       LINUX RESTART

11:17:30 AM       LINUX RESTART

11:17:38 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
11:17:43 AM     all     28.08      0.00     13.61      0.00      0.06     58.25
11:17:55 AM     all     19.94      0.00      5.39      0.10      0.04     74.52
11:18:11 AM     all     27.47      0.00     14.50      0.07      0.07     57.89
11:18:54 AM     all     18.93      0.00      7.74      0.51      0.04     72.78
11:20:01 AM     all     11.74      0.00     12.18      0.34      0.04     75.70
11:21:48 AM     all     26.00      0.00     22.81      0.11      0.08     51.00
11:25:01 AM     all     69.72      0.00     25.41      0.09      0.12      4.65
11:30:01 AM     all     20.21      0.00      7.50      0.09      0.04     72.16
11:34:12 AM     all     47.09      0.00     23.19      0.03      0.10     29.59
11:34:28 AM     all     13.82      0.00     25.00      0.16      0.08     60.94
11:35:01 AM     all     19.20      0.00     10.40      0.21      0.07     70.12
11:40:01 AM     all      7.35      0.00      7.46      0.03      0.04     85.11
11:45:01 AM     all      2.07      0.00      3.93      0.01      0.01     93.97
11:50:01 AM     all      3.00      0.00      5.76      0.02      0.02     91.21
11:55:01 AM     all      3.21      0.00      6.13      0.01      0.02     90.63
12:00:01 PM     all      3.24      0.00      6.33      0.02      0.02     90.40
12:05:01 PM     all      3.41      0.00      6.86      0.02      0.02     89.69
12:10:01 PM     all      2.96      0.00      6.03      0.02      0.02     90.97
12:15:01 PM     all      3.25      0.00      6.53      0.02      0.02     90.18
12:20:01 PM     all      2.78      0.00      5.73      0.02      0.01     91.46
12:25:01 PM     all      3.04      0.00      6.35      0.02      0.02     90.58
12:30:01 PM     all      2.91      0.00      6.11      0.01      0.01     90.96
12:35:01 PM     all      2.96      0.00      6.37      0.02      0.02     90.64
12:40:01 PM     all      2.86      0.00      6.15      0.02      0.02     90.96
12:45:01 PM     all      3.10      0.00      6.61      0.02      0.02     90.26
12:50:01 PM     all      2.81      0.00      5.97      0.01      0.02     91.19
12:55:01 PM     all      2.87      0.00      6.21      0.02      0.02     90.89
01:00:01 PM     all      2.86      0.00      6.14      0.01      0.01     90.97
01:05:01 PM     all      3.25      0.00      6.90      0.01      0.02     89.82
01:10:01 PM     all      2.61      0.00      5.67      0.01      0.02     91.69
01:15:01 PM     all      3.02      0.00      6.47      0.02      0.01     90.48
01:20:01 PM     all      2.82      0.00      6.11      0.02      0.02     91.04
01:25:02 PM     all      2.83      0.00      6.21      0.02      0.01     90.92
01:30:01 PM     all      2.78      0.00      6.12      0.02      0.01     91.08
01:35:01 PM     all      3.20      0.00      6.78      0.01      0.02     89.97
01:40:01 PM     all      2.86      0.00      6.22      0.02      0.02     90.88
01:45:01 PM     all      3.13      0.00      6.75      0.02      0.02     90.08
01:50:01 PM     all      2.68      0.00      6.08      0.02      0.02     91.20
01:55:01 PM     all      3.00      0.00      6.50      0.01      0.02     90.47
02:00:01 PM     all      2.85      0.00      6.11      0.01      0.02     91.01
02:05:01 PM     all      3.29      0.00      7.17      0.02      0.02     89.51
02:10:01 PM     all      2.74      0.00      5.99      0.02      0.02     91.24
Average:        all      6.26      0.00      7.28      0.03      0.02     86.41

02:12:10 PM       LINUX RESTART

02:15:01 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
02:20:01 PM     all      2.70      0.00      5.73      0.02      0.02     91.53
02:25:01 PM     all      2.96      0.00      6.32      0.02      0.02     90.68
02:30:01 PM     all      2.79      0.00      6.01      0.02      0.02     91.16
02:35:01 PM     all      3.20      0.00      6.75      0.02      0.04     90.00
02:40:01 PM     all      2.77      0.00      5.91      0.01      0.02     91.29
02:45:01 PM     all      3.05      0.00      6.45      0.02      0.03     90.45
02:50:01 PM     all      2.75      0.00      5.87      0.02      0.02     91.34
02:55:01 PM     all      2.98      0.00      6.28      0.02      0.02     90.70
03:00:01 PM     all      2.90      0.00      6.04      0.02      0.02     91.02
03:05:01 PM     all      3.24      0.00      6.78      0.02      0.04     89.92
03:10:01 PM     all      2.72      0.00      5.73      0.01      0.02     91.51
03:15:01 PM     all      2.91      0.00      6.17      0.02      0.02     90.88
03:20:01 PM     all      2.88      0.00      6.11      0.01      0.03     90.97
03:25:01 PM     all      2.91      0.00      6.25      0.01      0.03     90.80
03:30:01 PM     all      2.80      0.00      5.99      0.02      0.02     91.17
03:35:01 PM     all      3.20      0.00      6.51      0.01      0.03     90.25
03:40:02 PM     all      2.70      0.00      5.80      0.02      0.03     91.45
03:45:01 PM     all      3.02      0.00      6.37      0.02      0.03     90.56
03:50:01 PM     all      2.62      0.00      5.64      0.02      0.03     91.70
03:55:01 PM     all      3.00      0.00      6.43      0.02      0.02     90.53
04:00:01 PM     all      2.84      0.00      6.10      0.02      0.02     91.01
04:05:01 PM     all      3.25      0.00      6.93      0.02      0.04     89.76
04:10:01 PM     all      2.73      0.00      5.93      0.01      0.02     91.31
04:15:01 PM     all      3.10      0.00      6.79      0.02      0.02     90.07
Average:        all      2.92      0.00      6.20      0.02      0.03     90.84

------------------------------------------------------------------
top -bHn1 -w200
top - 16:20:00 up 1 day,  4:24,  1 user,  load average: 0.50, 0.35, 0.33
Threads: 164 total,   1 running, 163 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.4 us,  2.8 sy,  0.0 ni, 95.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  8008856 total,   236308 free,  1472920 used,  6299628 buff/cache
KiB Swap:  1048572 total,  1025020 free,    23552 used.  6171576 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
24824 root      20   0  162064   2180   1540 R  5.9  0.0   0:00.02 top
    1 root      20   0  191140   3408   2212 S  0.0  0.0   0:16.99 systemd
    2 root      20   0       0      0      0 S  0.0  0.0   0:00.01 kthreadd
    3 root      20   0       0      0      0 S  0.0  0.0   1:07.50 ksoftirqd/0
    5 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kworker/0:0H
    7 root      rt   0       0      0      0 S  0.0  0.0   0:08.84 migration/0
    8 root      20   0       0      0      0 S  0.0  0.0   0:00.00 rcu_bh
    9 root      20   0       0      0      0 S  0.0  0.0   1:22.39 rcu_sched
   10 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 lru-add-drain
   11 root      rt   0       0      0      0 S  0.0  0.0   0:00.42 watchdog/0
   12 root      rt   0       0      0      0 S  0.0  0.0   0:00.35 watchdog/1
   13 root      rt   0       0      0      0 S  0.0  0.0   0:08.89 migration/1
   14 root      20   0       0      0      0 S  0.0  0.0   0:53.48 ksoftirqd/1
   16 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kworker/1:0H
   17 root      rt   0       0      0      0 S  0.0  0.0   0:00.33 watchdog/2
   18 root      rt   0       0      0      0 S  0.0  0.0   0:09.09 migration/2
   19 root      20   0       0      0      0 S  0.0  0.0   1:02.02 ksoftirqd/2
   21 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kworker/2:0H
   22 root      rt   0       0      0      0 S  0.0  0.0   0:00.39 watchdog/3
   23 root      rt   0       0      0      0 S  0.0  0.0   0:09.85 migration/3
   24 root      20   0       0      0      0 S  0.0  0.0   1:22.78 ksoftirqd/3
   26 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kworker/3:0H
   28 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kdevtmpfs
   29 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 netns
   30 root      20   0       0      0      0 S  0.0  0.0   0:00.05 khungtaskd
   31 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 writeback
   32 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kintegrityd
   33 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 bioset
   34 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 bioset
   35 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 bioset
   36 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kblockd
   37 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 md
   38 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 edac-poller
   39 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 watchdogd
   45 root      20   0       0      0      0 S  0.0  0.0   1:26.15 kswapd0
   46 root      25   5       0      0      0 S  0.0  0.0   0:00.00 ksmd
   47 root      39  19       0      0      0 S  0.0  0.0   0:00.44 khugepaged
   48 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 crypto
   56 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kthrotld
   58 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kmpath_rdacd
   59 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kaluad
   61 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kpsmoused
   62 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 ipv6_addrconf
   75 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 deferwq
  109 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kauditd
  638 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 ata_sff
  663 root      20   0       0      0      0 S  0.0  0.0   0:00.00 scsi_eh_0
  665 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 scsi_tmf_0
  667 root      20   0       0      0      0 S  0.0  0.0   0:00.00 scsi_eh_1
  668 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 scsi_tmf_1
  770 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 ttm_swap
 1304 root       0 -20       0      0      0 S  0.0  0.0   0:00.38 kworker/0:1H
 1414 root      20   0       0      0      0 S  0.0  0.0   0:06.52 kworker/0:2
 1418 root       0 -20       0      0      0 S  0.0  0.0   0:00.43 kworker/1:1H
 1448 root      20   0       0      0      0 S  0.0  0.0   0:03.59 jbd2/vda1-8
 1449 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 ext4-rsv-conver
 1457 root       0 -20       0      0      0 S  0.0  0.0   0:00.96 kworker/3:1H
 1504 root       0 -20       0      0      0 S  0.0  0.0   0:00.29 kworker/2:1H
 1521 root      20   0   55912  20124  19856 S  0.0  0.3   0:06.80 systemd-journal
 1543 root      20   0   44680   1052    936 S  0.0  0.0   0:00.19 systemd-udevd
 1557 root       0 -20       0      0      0 S  0.0  0.0   0:00.30 loop0
 1568 root      20   0       0      0      0 S  0.0  0.0   0:00.07 jbd2/loop0-8
 1571 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 ext4-rsv-conver
 1596 root      20   0       0      0      0 S  0.0  0.0   0:06.79 kworker/2:2
 1884 polkitd   20   0  612408   7088   2480 S  0.0  0.1   0:01.51 polkitd
 1993 polkitd   20   0  612408   7088   2480 S  0.0  0.1   0:00.00 gmain
 2008 polkitd   20   0  612408   7088   2480 S  0.0  0.1   0:01.92 gdbus
 2027 polkitd   20   0  612408   7088   2480 S  0.0  0.1   0:00.00 polkitd
 2071 polkitd   20   0  612408   7088   2480 S  0.0  0.1   0:00.00 JS GC Helper
 2077 polkitd   20   0  612408   7088   2480 S  0.0  0.1   0:00.00 JS Sour~ Thread
 2097 polkitd   20   0  612408   7088   2480 S  0.0  0.1   0:00.00 polkitd
 1887 root      20   0   90552   2224   2128 S  0.0  0.0   0:14.89 rngd
 1892 root      20   0   21716    968    832 S  0.0  0.0   0:12.89 irqbalance
 1906 root      20   0   12184   2668    584 S  0.0  0.0   0:15.31 haveged
 1926 dbus      20   0   58172   1892   1580 S  0.0  0.0   0:08.66 dbus-daemon
 1958 chrony    20   0  117784   1356   1212 S  0.0  0.0   0:00.32 chronyd
 2012 root      20   0   26560   1616   1312 S  0.0  0.0   0:05.19 systemd-logind
 2034 root      20   0  126364   1352    936 S  0.0  0.0   0:00.50 crond
 2199 root      20   0       0      0      0 S  0.0  0.0   0:00.33 kworker/u8:1
 2216 memcach+  20   0  449820   1720    672 S  0.0  0.0   0:00.81 memcached
 2217 memcach+  20   0  449820   1720    672 S  0.0  0.0   0:00.64 memcached
 2218 memcach+  20   0  449820   1720    672 S  0.0  0.0   0:00.00 memcached
 2219 memcach+  20   0  449820   1720    672 S  0.0  0.0   0:00.00 memcached
 2220 memcach+  20   0  449820   1720    672 S  0.0  0.0   0:00.00 memcached
 2221 memcach+  20   0  449820   1720    672 S  0.0  0.0   0:00.00 memcached
 2222 memcach+  20   0  449820   1720    672 S  0.0  0.0   0:00.00 memcached
 2223 memcach+  20   0  449820   1720    672 S  0.0  0.0   0:00.00 memcached
 2224 memcach+  20   0  449820   1720    672 S  0.0  0.0   0:01.28 memcached
 2225 memcach+  20   0  449820   1720    672 S  0.0  0.0   0:00.00 memcached
 2286 root      20   0       0      0      0 S  0.0  0.0   0:00.77 hwrng
 2854 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 nfit
 3340 root      20   0  112916   3408   3292 S  0.0  0.0   0:01.17 sshd
 3344 root      20   0  573976   7752   3608 S  0.0  0.1   0:00.13 tuned
 4207 root      20   0  573976   7752   3608 S  0.0  0.1   0:00.00 gmain
 4208 root      20   0  573976   7752   3608 S  0.0  0.1   0:11.13 tuned
 4215 root      20   0  573976   7752   3608 S  0.0  0.1   0:00.00 tuned
 4309 root      20   0  573976   7752   3608 S  0.0  0.1   0:00.00 tuned
 3348 root      20   0  232788  14680  11860 S  0.0  0.2   0:00.01 rsyslogd
 3388 root      20   0  232788  14680  11860 S  0.0  0.2   0:07.00 in:imjournal
 3402 root      20   0  232788  14680  11860 S  0.0  0.2   0:00.70 rs:main Q:Reg
 3390 root      20   0  110104    808    676 S  0.0  0.0   0:00.00 agetty
 3519 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kworker/0:1
 4388 root      20   0  154708   5536   4232 S  0.0  0.1   0:00.03 sshd
 4390 root      20   0  116372   3092   1708 S  0.0  0.0   0:00.03 bash
 4735 root      20   0       0      0      0 S  0.0  0.0   0:30.39 kworker/1:1
 5346 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 dio/vda1
 6365 root      20   0       0      0      0 S  0.0  0.0   0:07.34 kworker/1:0
 6693 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:01.18 mysqld
 6874 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.00 mysqld
 6890 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.00 mysqld
 6903 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   1:44.23 mysqld
 8090 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.07 mysqld
 8091 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.00 mysqld
 8094 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.01 mysqld
 8197 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.15 mysqld
 8674 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:05.37 mysqld
 8675 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:05.85 mysqld
 8676 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:05.60 mysqld
 8677 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:05.51 mysqld
 8678 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:05.48 mysqld
 8679 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:05.45 mysqld
 8680 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:06.16 mysqld
 8681 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:05.64 mysqld
 8682 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:05.54 mysqld
 8683 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:05.54 mysqld
 8759 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.75 mysqld
 8959 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:03.67 mysqld
 8960 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:05.42 mysqld
 8961 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:11.26 mysqld
 8962 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.37 mysqld
 8963 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.80 mysqld
 8964 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.00 mysqld
 9014 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:05.01 mysqld
 9015 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:04.32 mysqld
 9034 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.00 mysqld
 9035 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:03.09 mysqld
 9038 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.00 mysqld
 9074 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.00 mysqld
 9075 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.00 mysqld
 9076 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.00 mysqld
 9077 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:00.00 mysqld
 9706 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   1:38.32 mysqld
10357 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   5:40.07 mysqld
25937 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:06.27 mysqld
29971 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:06.81 mysqld
25473 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:06.24 mysqld
25474 mysql     20   0 6734520   1.0g   7344 S  0.0 13.6   0:06.15 mysqld
 8643 root      20   0  171904  23956   2184 S  0.0  0.3   0:01.31 lfd - sleeping
11883 root      20   0       0      0      0 S  0.0  0.0   0:00.51 kworker/3:1
15130 root      20   0  154988   5840   4264 S  0.0  0.1   0:00.09 sshd
17530 root      20   0       0      0      0 S  0.0  0.0   0:05.66 kworker/1:2
20634 root      20   0  154988   5688   4260 S  0.0  0.1   0:00.02 sshd
21779 root      20   0       0      0      0 S  0.0  0.0   0:00.84 kworker/u8:0
22499 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kworker/3:0
23805 root      20   0  113440   1744   1304 S  0.0  0.0   0:00.00 cminfo
23916 root      20   0  154992   5836   4264 S  0.0  0.1   0:00.06 sshd
24008 root      20   0  113524   1344    752 S  0.0  0.0   0:00.01 cminfo
24009 root      20   0  107952    668    564 S  0.0  0.0   0:00.00 tee
24572 root      20   0  317240 117996    448 S  0.0  1.5   0:00.03 nginx
24578 nginx     10 -10  354104 153892   1268 S  0.0  1.9   0:00.06 nginx
24581 nginx     10 -10  354104 154620   1304 S  0.0  1.9   0:00.07 nginx
24762 root      10 -10  631772  15104   1252 S  0.0  0.2   0:00.00 php-fpm
24790 nginx     10 -10  631772  15736   1716 S  0.0  0.2   0:00.00 php-fpm
26603 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kworker/2:0

------------------------------------------------------------------
iotop -bton1 -P
16:20:01 Total DISK READ :       0.00 B/s | Total DISK WRITE :       0.00 B/s
16:20:01 Actual DISK READ:       0.00 B/s | Actual DISK WRITE:       0.00 B/s
    TIME  PID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND

------------------------------------------------------------------
pidstat -durh 1 10 | sed -e "s|$(hostname)|hostname|g"
Linux 3.10.0-957.21.3.el7.x86_64 (hostname)     06/24/2019     _x86_64_    (4 CPU)

#      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
 1561389602     0         1    0.00    0.00    0.00    0.00     3     25.49      0.00  191140   3408   0.04      0.00      0.00      0.00  systemd
 1561389602     0      1521    0.00    0.00    0.00    0.00     2     11.76      0.00   55912  20124   0.25      0.00      0.00      0.00  systemd-journal
 1561389602     0      1892    0.00    0.00    0.00    0.00     0     11.76      0.00   21716    968   0.01      0.00      0.00      0.00  irqbalance
 1561389602    81      1926    0.00    0.00    0.00    0.00     1      1.96      0.00   58172   1892   0.02      0.00      0.00      0.00  dbus-daemon
 1561389602     0      2012    0.00    0.00    0.00    0.00     3     11.76      0.00   26560   1624   0.02      0.00      0.00      0.00  systemd-logind
 1561389602     0      2034    0.00    0.00    0.00    0.00     1      0.98      0.00  126364   1352   0.02      0.00      7.84      0.00  crond
 1561389602     0      2199    0.00    0.98    0.00    0.98     3      0.00      0.00       0      0   0.00      0.00      0.00      0.00  kworker/u8:1
 1561389602     0      3348    0.00    0.00    0.00    0.00     2      1.96      0.00  232788  14688   0.18      0.00      3.92      0.00  rsyslogd
 1561389602   996      6693    0.98    0.98    0.00    1.96     2     46.08      0.00 6734520 1092108  13.64      0.00      0.00      0.00  mysqld
 1561389602  1000     24581    0.98    0.00    0.00    0.98     0      0.00      0.00  354104 154620   1.93      0.00      0.00      0.00  nginx
 1561389602     0     25050    0.00    0.98    0.00    0.98     3    318.63      0.00  108192   1060   0.01      0.00      0.00      0.00  pidstat

#      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
 1561389603     0     17530    0.00    2.00    0.00    2.00     1      0.00      0.00       0      0   0.00      0.00      0.00      0.00  kworker/1:2
 1561389603     0     25050    0.00    1.00    0.00    1.00     3    336.00      0.00  108192   1136   0.01      0.00      0.00      0.00  pidstat
 1561389603     0     25051    0.00    0.00    0.00    0.00     0      1.00      0.00  116992    936   0.01      0.00      0.00      0.00  sed

#      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
 1561389604   996      6693    0.00    0.00    0.00    0.00     2    542.00      0.00 6734520 1092108  13.64      0.00      0.00      0.00  mysqld
 1561389604     0     25050    0.00    1.00    0.00    1.00     3    317.00      0.00  108192   1136   0.01      0.00      0.00      0.00  pidstat

#      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
 1561389605     0     25050    1.00    1.00    0.00    2.00     3    317.00      0.00  108192   1136   0.01      0.00      0.00      0.00  pidstat

#      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
 1561389606     0         1    0.00    1.00    0.00    1.00     3      0.00      0.00  191140   3408   0.04      0.00      0.00      0.00  systemd
 1561389606     0      1448    0.00    0.00    0.00    0.00     3      0.00      0.00       0      0   0.00      0.00     48.00      0.00  jbd2/vda1-8
 1561389606     0      1557    0.00    0.00    0.00    0.00     1      0.00      0.00       0      0   0.00      0.00     64.00      0.00  loop0
 1561389606   996      6693    1.00    1.00    0.00    2.00     2    542.00      0.00 6734520 1092108  13.64      0.00      0.00      0.00  mysqld
 1561389606     0      8643    0.00    0.00    0.00    0.00     0      0.00      0.00       0      0   0.00      0.00      4.00      0.00  lfd
 1561389606     0     25050    0.00    1.00    0.00    1.00     3    317.00      0.00  108192   1136   0.01      0.00      0.00      0.00  pidstat

#      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
 1561389607   996      6693    0.00    0.00    0.00    0.00     2    542.00      0.00 6734520 1092108  13.64      0.00      0.00      0.00  mysqld
 1561389607     0     24009    0.00    0.00    0.00    0.00     1      0.00      0.00  107952    668   0.01      0.00      4.00      0.00  tee
 1561389607     0     25050    1.00    0.00    0.00    1.00     3    317.00      0.00  108192   1136   0.01      0.00      0.00      0.00  pidstat

#      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
 1561389608   996      6693    1.00    0.00    0.00    1.00     2      0.00      0.00 6734520 1092108  13.64      0.00      0.00      0.00  mysqld
 1561389608     0     25050    0.00    1.00    0.00    1.00     3    317.00      0.00  108192   1136   0.01      0.00      0.00      0.00  pidstat

#      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
 1561389609   996      6693    0.00    0.00    0.00    0.00     2    542.00      0.00 6734520 1092108  13.64      0.00      0.00      0.00  mysqld
 1561389609     0     15130    0.00    0.00    0.00    0.00     1     38.00      0.00  154988   5840   0.07      0.00      0.00      0.00  sshd
 1561389609     0     25050    0.00    2.00    0.00    2.00     3    317.00      0.00  108192   1136   0.01      0.00      0.00      0.00  pidstat
 1561389609     0     25066    0.00    0.00    0.00    0.00     3    881.00      0.00    9556   1460   0.02      0.00      8.00      0.00  bash
 1561389609     0     25077    0.00    0.00    0.00    0.00     1    949.00      0.00  113176   1560   0.02      0.00      4.00      0.00  dmotd
 1561389609     0     25109    0.00    0.00    0.00    0.00     2    490.00      0.00  113180   1032   0.01      0.00      0.00      0.00  dmotd
 1561389609     0     25110    0.00    0.00    0.00    0.00     0    316.00      0.00  107952    668   0.01      0.00      4.00      0.00  tee
 1561389609     0     25148    0.00    0.00    0.00    0.00     0    529.00      0.00  116668   1396   0.02      0.00      0.00      0.00  git
 1561389609     0     25149    1.00    0.00    0.00    1.00     2   1262.00      0.00  163908   4296   0.05      0.00      0.00      0.00  git-remote-http

#      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
 1561389610     0        14    0.00    1.00    0.00    1.00     1      0.00      0.00       0      0   0.00      0.00      0.00      0.00  ksoftirqd/1
 1561389610   996      6693    0.00    1.00    0.00    1.00     2    542.00      0.00 6734520 1092108  13.64      0.00      0.00      0.00  mysqld
 1561389610     0     15130    0.00    0.00    0.00    0.00     2      2.00      0.00  154988   5840   0.07      0.00      0.00      0.00  sshd
 1561389610     0     25050    0.00    1.00    0.00    1.00     0    335.00      0.00  108192   1136   0.01      0.00      0.00      0.00  pidstat
 1561389610     0     25077    0.00    0.00    0.00    0.00     1    109.00      0.00  113180   1564   0.02      0.00      8.00      0.00  dmotd
 1561389610     0     25166    1.00    6.00    0.00    7.00     3   8509.00      0.00  153004  33312   0.42      0.00      0.00      0.00  find
 1561389610     0     25167    0.00    0.00    0.00    0.00     3    108.00      0.00  113180    452   0.01      0.00      0.00      0.00  dmotd

#      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
 1561389611     0         1    0.00    0.00    0.00    0.00     2     22.00      0.00  191140   3408   0.04      0.00      0.00      0.00  systemd
 1561389611     0      1448    0.00    0.00    0.00    0.00     2      0.00      0.00       0      0   0.00      0.00     28.00      0.00  jbd2/vda1-8
 1561389611     0      1521    0.00    1.00    0.00    1.00     1     38.00      0.00   55912  20136   0.25      0.00      0.00      0.00  systemd-journal
 1561389611    81      1926    0.00    1.00    0.00    1.00     2      2.00      0.00   58172   1892   0.02      0.00      0.00      0.00  dbus-daemon
 1561389611     0      2012    0.00    0.00    0.00    0.00     0     12.00      0.00   26560   1624   0.02      0.00      0.00      0.00  systemd-logind
 1561389611     0      3340    0.00    1.00    0.00    1.00     3     15.00      0.00  112916   3408   0.04      0.00    500.00    112.00  sshd
 1561389611     0      3348    0.00    0.00    0.00    0.00     2      2.00      0.00  232788  14688   0.18      0.00      0.00      0.00  rsyslogd
 1561389611   996      6693    1.00    0.00    0.00    1.00     2      0.00      0.00 6734520 1092108  13.64      0.00      0.00      0.00  mysqld
 1561389611     0     24762    0.00    0.00    0.00    0.00     1      9.00      0.00  631772  15104   0.19      0.00      0.00      0.00  php-fpm
 1561389611     0     25050    0.00    1.00    0.00    1.00     0    326.00      0.00  108192   1136   0.01      0.00      0.00      0.00  pidstat
 1561389611     0     25171    1.00    1.00    0.00    2.00     1   1768.00      0.00  154992   5668   0.07      0.00      0.00      0.00  sshd
 1561389611     0     25174    0.00    0.00    0.00    0.00     3    882.00      0.00    9556   1460   0.02      0.00      8.00      0.00  bash
 1561389611     0     25185    0.00    0.00    0.00    0.00     3    950.00      0.00  113176   1560   0.02      0.00      4.00      0.00  dmotd
------------------------------------------------------------------
Stats saved at: /root/centminlogs/cminfo-top-240619-161950.log
------------------------------------------------------------------

Another thing I've noticed thanks to your suggestion to use 'cminfo top'. When I checked the 'centminlogs' directory there were many thousands of 'cmm-login-git-checks_*.log' files. These are created everytime there's an Ssh connection. Galera is making many SSH connections to all the servers all the time. This means there is a git check every time an SSH connection is made. Maybe this is the cause of the SWAP being used.

I've found these for the custom_config:

Code:

ENABLEMOTD='n'
ENABLEMOTD_CSFMSG='n'
ENABLEMOTD_LINKSMSG='n'

But, ENABLEMOTD='n' doesn't seem to work.

I have a feeling the many system statuses and git checks are part, if not the issue here.

Is there a way to perminantly turn off the status and git checks when there's an SSH login?

Simon Brown · Jun 25, 2019

Quick update, I didn't launch centmin.sh after I updated the custom_config. I've now done it on all the servers. I'll wait to see if it make any difference

eva2000 · Jun 25, 2019

From cminfo top output not seeing nginx as a swap user there so probably it's a victim not a culprit of low memory free availability. Most likely it's MySQL itself given it's virtual memory size. I'd be tuning MySQL buffers to lower memory usage. But the current cminfo top shows no swapping to disk right now so probably not a good indicator.

also suggests it's normal occurrence of swap as you're reaching low memory free availability free with high/low reported memory usage shows low stage you have 248MB free
Code (Text):
------------------------------------------------------------------
free -mtl
             total        used        free      shared  buff/cache   available
Mem:           7821        1456         248          19        6115        6008
Low:           7821        7572         248
High:             0           0           0
Swap:          1023          56         967
Total:         8845        1512        1216
sar -r output shows memory free can get down to 115,244 KB or 112.5MB with 6,871,768 KB or ~6.55GB in committed memory at 2:30:01 AM on June 24 so it's would be expected that swap would be avenue for other apps/processes fighting for memory availability when it gets that low
Code (Text):
02:15:01 PM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty

02:20:01 AM    202916   7805940     97.47     11932     70088   6771340     74.76    690792    828088       176
02:25:01 AM    182440   7826416     97.72     11692     68644   6771372     74.76    691916    825088       204
02:30:01 AM    115244   7893612     98.56     11884     70180   6871768     75.87    765484    827260       160
02:35:01 AM    160816   7848040     97.99     12040     68096   6805148     75.13    724492    824084       152
02:40:01 AM    201464   7807392     97.48     11328     63380   6785212     74.91    694804    818400       148
02:45:01 AM    197548   7811308     97.53     11552     63292   6771364     74.76    690000    819192       124
02:50:01 AM    220572   7788284     97.25     11504     62528   6771340     74.76    689052    818620       180
02:55:01 AM    221728   7787128     97.23     11700     62592   6775560     74.81    695176    815644       192
03:00:01 AM    160600   7848256     97.99     11408     69932   6819508     75.29    702796    823336       180
03:05:01 AM    161896   7846960     97.98     12376     74308   6805656     75.14    727012    824176       160
03:10:01 AM    192268   7816588     97.60     12856     66456   6771872     74.77    693136    817512       188
03:15:01 AM    155744   7853112     98.06     71876     91708   6771372     74.76    730208    867844       148
03:20:01 AM    205064   7803792     97.44     64252     82856   6771372     74.76    728272    853456       172
03:25:01 AM    201988   7806868     97.48     60788     75348   6771368     74.76    720208    850692       184
03:30:02 AM    184568   7824288     97.70     54844     69968   6803456     75.11    715532    853824       208
03:35:01 AM    237096   7771760     97.04     41748     68328   6771368     74.76    697588    847056       840
03:40:01 AM    137588   7871268     98.28     39764     64196   6838936     75.51    760396    844056       140
03:45:01 AM    179840   7829016     97.75     33096     65400   6771796     74.77    695436    837592        48
03:50:01 AM    185920   7822936     97.68     24256     63468   6771368     74.76    694888    827116        48
03:55:01 AM    174796   7834060     97.82     18544     64552   6771368     74.76    694684    822808       180
04:00:01 AM    156888   7851968     98.04     13588     70808   6821980     75.32    707208    823656       136
04:05:01 AM    228080   7780776     97.15     12024     71516   6771340     74.76    697652    820068       204
Simon Brown said: ↑

Is there a way to perminantly turn off the status and git checks when there's an SSH login?
Click to expand...

currently no way to disable that though those logs get purged every 5 days. But pretty sure it isn't the cause of swapping.

you can use cminfo top command via cronjob to record your usage if you want as the cminfo top output is saved to a log in /root/centminlogs/ directory so you can see your specific process and system usage.

For CentOS 7, you can try changing in /etc/sysctl.d/101-sysctl.conf the vm.swappiness from 100-10 = swap to disk when 90% full memory
Code (Text):
vm.swappiness=10
to swap to disk when 100-1 = 99% full memory
Code (Text):
vm.swappiness=1
then for changes to take affect run
Code (Text):
sysctl --system

Simon Brown · Jun 26, 2019

Just wanted to update this ticket... It seems that using the ENABLEMOTD config has worked. The three servers I used configured are no longer using SWAP and there aren't thousands of centminlog files anymore. IF people are syncing servers via SSH with a service like Lsyncd and/or cluster/server monitoring services that use SSH, then you should set the centmin config with this:
Code:
ENABLEMOTD='n'
ENABLEMOTD_CSFMSG='n'
ENABLEMOTD_LINKSMSG='n' 
Many thanks as ever for your detailed replies.

buik · Jun 27, 2019

eva2000 said: ↑

so probably it's a victim of low memory free availability.
Click to expand...

I have seen this more often. Might be low memory free availability or oversold node.
Have you tested this setup also on bare metal dedicated?

So you're sure you don't suffer from neighbors.

Simon Brown · Jun 27, 2019

buik said: ↑

I have seen this more often. Might be low memory free availability or oversold node.
Click to expand...

The issue was happening on 4 VPS's using two vendors and a dedicated server with 32Gb memory.
After over 24 hours the swap still isn't used. Fingers crossed.

buik · Jun 27, 2019

Simon Brown said: ↑

The issue was happening on 4 VPS's using two vendors and a dedicated server with 32Gb memory.
After over 24 hours the swap still isn't used. Fingers crossed.
Click to expand...

Have you tested this setup also on bare metal dedicated only?
VPS servers are always overpopulated on lowend (Vultr, Digitalocean etc etc). It's just the question of how far.

Simon Brown · Jun 27, 2019

Yes, I have a bare metal dedicated with 32Gb memory. It was swapping too. So far, swap is 0 on all VPS's and the dedicated. I've been testing the cluster for weeks now, trying to work out why all nodes were swapping all the time.
This is the longest and most stable it's been. I even enabled lsyncd on the nodes. So far, so good.

buik · Jun 27, 2019

@Simon Brown I apologize.
I didn't read it correctly.
And should have known better.

As you are using various VPS providers and 1 dedi
(Vultr has been offering dedis for 1.5 years. I didn't even know. Haha.),
this problem has more of a bug, mis-config, than of overselling.

Completely in view of the fact that the problem seems to be corrected by adjusting:
ENABLEMOTD.

And you wrote this before I responded.
Again excuses.

eva2000 · Jun 27, 2019

buik said: ↑

Completely in view of the fact that the problem seems to be corrected by adjusting:
ENABLEMOTD.
Click to expand...

FYI, ENABLEMOTD='y' is force enabled now in 123.09beta01 so setting ENABLEMOTD='n' won't have any effect so wouldn't be that.

Simon Brown said: ↑

This is the longest and most stable it's been. I even enabled lsyncd on the nodes. So far, so good.
Click to expand...

might want to make use of cminfo top stats to monitor your processes, I updated 123.09beta01's cminfo top stats to also check top 10 VmSwap and VmSize/VSZ using processes Beta Branch - update tools/cminfo.sh in 123.09beta01

cminfo command is updated every 4hrs on centmin mod servers via a cronjob. You can manually update it to use updated cminfo code via manually running cronjob updating script
Code (Text):
/usr/bin/cminfo_updater

Log in / Register

Forums

Want to subscribe to topics you're interested in?

Sysadmin Swap increases slowly, but continuously until 100%

Simon Brown Premium Member Premium Member

eva2000 Administrator Staff Member

Simon Brown Premium Member Premium Member

Simon Brown Premium Member Premium Member

eva2000 Administrator Staff Member

Simon Brown Premium Member Premium Member

buik “The best traveler is one without a camera.”

Simon Brown Premium Member Premium Member

buik “The best traveler is one without a camera.”

Simon Brown Premium Member Premium Member

buik “The best traveler is one without a camera.”

eva2000 Administrator Staff Member

.

Log in / Register

Useful Searches

Want to subscribe to topics you're interested in?

Sysadmin Swap increases slowly, but continuously until 100%

Simon Brown Premium Member Premium Member

eva2000 Administrator Staff Member

Simon Brown Premium Member Premium Member

Simon Brown Premium Member Premium Member

eva2000 Administrator Staff Member

Simon Brown Premium Member Premium Member

buik “The best traveler is one without a camera.”

Simon Brown Premium Member Premium Member

buik “The best traveler is one without a camera.”

Simon Brown Premium Member Premium Member

buik “The best traveler is one without a camera.”

eva2000 Administrator Staff Member

.