Join the community today
Become a Member

CPU 100% what to do?

Discussion in 'System Administration' started by adamus007p, Sep 14, 2020.

  1. adamus007p

    adamus007p Member

    252
    15
    18
    Feb 8, 2019
    Ratings:
    +26
    Local Time:
    2:40 PM
    Hello I have a CPU 100% I see it on htop or top2.


    Code (Text):
    QEMU Virtual CPU version 2.5+ - 2.40/2.40GHz                 CPU -   100.0%  nice:     0.0%  ctx_sw:   628   MEM -   33.1%  active:    6.69G   SWAP -    1.3%   LOAD    8-core
    CPU  [|||||||||||||||||||||||||||||||||||||||||||||100.0%]   user:    19.1%  irq:      0.0%  inter:   4100   total:  15.5G  inactive:  2.43G   total:   7.88G   1 min:   22.43
    MEM  [|||||||||||||||                               33.1%]   system:  30.9%  iowait:   0.0%  sw_int:  4634   used:   5.13G  buffers:    421M   used:     102M   5 min:   20.86
    SWAP [                                               1.3%]   idle:     0.0%  steal:   50.0%                  free:   10.4G  cached:    10.5G   free:    7.78G   15 min:  18.05
    
    NETWORK                  Rx/s   Tx/s   TASKS 188 (289 thr), 21 run, 167 slp, 0 oth sorted automatically by CPU consumption
    docker0                    0b     0b
    eth0                      4Kb   15Kb   Nginx            1    Active connections: 4 server accepts handled requests 263 263 422 Reading: 0 Writing: 2 Waiting: 2
    lo                        3Kb    3Kb
                                           CPU%   MEM%  VIRT  RES     PID USER          TIME+ THR  NI S  R/s W/s  Command
    TCP CONNECTIONS                        31.9   0.1   503M  15.0M 11071 nginx          8:20 1   -10 R    0 0    php-fpm: pool www
    Listen                            13   27.1   0.1   503M  15.2M  8277 nginx       1h25:14 1   -10 R    0 0    php-fpm: pool www
    Initiated                          0   24.3   0.1   505M  16.6M 21793 nginx          2:59 1   -10 R    0 0    php-fpm: pool www
    Established                       16   24.0   0.1   500M  12.5M 10488 nginx         51:53 1   -10 R    0 0    php-fpm: pool www
    Terminated                         0   23.7   0.1   502M  14.4M 10546 nginx         38:42 1   -10 R    0 0    php-fpm: pool www
    Tracked                    46/524288   22.6   0.1   505M  17.2M  8273 nginx       1h14:07 1   -10 R    0 0    php-fpm: pool www
                                           22.3   0.1   503M  14.9M 13102 nginx         19:46 1   -10 R    0 0    php-fpm: pool www
    DefaultGateway                1104ms   18.5   0.1   502M  15.1M  8275 nginx        1h2:29 1   -10 R    0 0    php-fpm: pool www
                                           18.2   0.1   505M  17.3M  8272 nginx        1h7:50 1   -10 R    0 0    php-fpm: pool www
    DISK I/O                 R/s    W/s    17.8   0.1   502M  14.6M  8279 nginx        1h0:19 1   -10 R    0 0    php-fpm: pool www
    sda                         0    14K   17.1   0.1   505M  16.0M  8278 nginx         15:58 1   -10 R    0 0    php-fpm: pool www
    sda1                        0      0   16.2   0.1   502M  14.2M 22305 nginx          2:40 1   -10 R    0 0    php-fpm: pool www
    sda2                        0    14K   15.8   0.1   505M  16.5M 11870 nginx          2:22 1   -10 R    0 0    php-fpm: pool www
    sda3                        0      0   15.8   0.1   502M  14.8M  8276 nginx        1h0:19 1   -10 R    0 0    php-fpm: pool www
    sr0                         0      0   15.8   0.1   502M  14.1M 21765 nginx          5:16 1   -10 R    0 0    php-fpm: pool www
                                           15.8   0.1   500M  13.0M 23191 nginx          0:56 1   -10 R    0 0    php-fpm: pool www
    FILE SYS                 Used  Total   15.8   0.1   500M  12.5M  9481 nginx        1h9:37 1   -10 R    0 0    php-fpm: pool www
    / (sda2)                 327G  1.43T   15.5   0.1   500M  12.4M 23199 nginx          2:21 1   -10 R    0 0    php-fpm: pool www
    _lib/docker/containers   327G  1.43T   15.4   0.1   505M  16.9M  8274 nginx         42:05 1   -10 R    0 0    php-fpm: pool www
    _r/lib/docker/overlay2   327G  1.43T   14.4   0.1   505M  16.8M 16821 nginx          2:21 1   -10 R    0 0    php-fpm: pool www
                                           10.7   0.3   566M  39.8M 23947 root           0:09 1     0 R    0 0    /usr/bin/python2 /usr/bin/glances
    IRQ                           Rate/s   0.3    0.4   106M  59.3M 10915 root          24:21 1     0 S    0 0    /usr/lib/systemd/systemd-journald
    LOC                            13087   0.0    9.1   10.3G 1.41G  7341 mysql          3:12 44    0 S    0 0    /usr/sbin/mysqld
    IWI                               41   0.0    0.6   1.23G 90.3M  8258 nginx          0:36 1   -10 S    0 0    nginx: worker process
    29_virtio0-input.1                23   0.0    0.5   213M  81.3M 20975 root           0:02 1     0 S    0 0    SCREEN
    RES                                5   0.0    0.4   1.15G 71.4M  8255 nginx          0:14 1   -10 S    0 0    nginx: worker process
    15_ata_piix                        4   0.0    0.4   1.15G 71.0M  8257 nginx          0:16 1   -10 S    0 0    nginx: worker process
                                           0.0    0.4   1.14G 70.4M  8254 nginx          0:09 1   -10 S    0 0    nginx: worker process
                                           0.0    0.3   169M  45.2M 29578 root           0:01 1     0 S    0 0    SCREEN
                                           0.0    0.3   213M  44.7M 27472 root           0:00 1     0 S    0 0    SCREEN
                                           0.0    0.2   694M  34.5M  2957 root           7:01 3     0 S    0 1K   /usr/sbin/rsyslogd -n
                                           0.0    0.2   1.09G 27.4M  8253 root           0:00 1     0 S    0 0    nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/c
                                           0.0    0.2   1.36G 26.6M 12701 root           2:39 24    0 S    0 0    /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/d
                                           0.0    0.1   170M  23.7M 29085 root           0:12 1     0 S    0 0    lfd - sleeping
                                           0.0    0.1   503M  16.1M 23202 nginx          0:00 1   -10 S    0 0    php-fpm: pool www
                                           0.0    0.1   503M  15.8M 16827 nginx          0:00 1   -10 S    0 0    php-fpm: pool www
                                           0.0    0.1   503M  15.5M 10466 nginx          0:01 1   -10 S    0 0    php-fpm: pool www
                                           0.0    0.1   124M  15.4M 16130 root           0:00 1     0 S    0 0    /bin/bash /usr/bin/centmin
                                           0.0    0.1   501M  13.8M 22200 nginx          0:00 1   -10 S    0 0    php-fpm: pool www
                                           0.0    0.1   124M  13.0M  3045 root           0:00 1     0 S    0 0    /bin/bash /usr/bin/centmin
                                           0.0    0.1   500M  12.8M 23279 nginx          0:00 1   -10 S    0 0    php-fpm: pool www
                                           0.0    0.1   124M  12.7M 17304 root           0:00 1     0 S    0 0    /bin/bash /usr/bin/centmin
    
                                           System overloaded in the last 5 minutes
    2020-09-13 19:34:33 UTC                2020-09-13 19:33:17 (ongoing) - LOAD (2.2)
    


    Is it an atack? DDOS?


    How to troubleshoot it?






    Code (Text):
    CentOS Linux release 7.8.2003 (Core)
    123.09beta01.b588
    git branch
    * 123.09beta01
    nginx -V
    nginx version: nginx/1.19.0 (120620-175146-centos7-kvm-6cb7a97-br-25f86f0)
    built by gcc 8.3.1 20190311 (Red Hat 8.3.1-3) (GCC)
    built with OpenSSL 1.1.1g  21 Apr 2020
    TLS SNI support enabled
    configure arguments: --with-ld-opt='-Wl,-E -L/usr/local/zlib-cf/lib -L/usr/local/lib -ljemalloc -Wl,-z,relro -Wl,-rpath,/usr/local/zlib-cf/lib:/usr/local/lib -flto=8 -fuse-ld=gold' --with-cc-opt='-I/usr/local/zlib-cf/include -I/usr/local/include -m64 -march=x86-64 -mavx256-split-unaligned-load -mavx256-split-unaligned-store -mpclmul -msse4 -msse4.1 -msse4.2 -DTCP_FASTOPEN=23 -g -O3 -fstack-protector-strong -flto=8 -fuse-ld=gold --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wimplicit-fallthrough=0 -fcode-hoisting -Wno-cast-function-type -Wno-format-extra-args -Wp,-D_FORTIFY_SOURCE=2 -Wno-deprecated-declarations' --sbin-path=/usr/local/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --build=120620-175146-centos7-kvm-6cb7a97-br-25f86f0 --with-compat --with-http_stub_status_module --with-http_secure_link_module --with-libatomic --with-http_gzip_static_module --add-dynamic-module=../ngx_brotli --with-http_sub_module --with-http_addition_module --with-http_image_filter_module=dynamic --with-http_geoip_module --with-stream_geoip_module --with-stream_realip_module --with-stream_ssl_preread_module --with-threads --with-stream --with-stream_ssl_module --with-http_realip_module --add-dynamic-module=../ngx-fancyindex-0.4.2 --add-module=../ngx_cache_purge-2.5 --add-dynamic-module=../ngx_devel_kit-0.3.0 --add-dynamic-module=../set-misc-nginx-module-0.32 --add-dynamic-module=../echo-nginx-module-0.61 --add-module=../redis2-nginx-module-0.15 --add-module=../ngx_http_redis-0.3.7 --add-module=../memc-nginx-module-0.18 --add-module=../srcache-nginx-module-0.32rc1 --add-dynamic-module=../headers-more-nginx-module-0.33 --with-pcre-jit --with-zlib=../zlib-cloudflare-1.3.0 --with-http_ssl_module --with-http_v2_module --with-openssl=../openssl-1.1.1g --with-openssl-opt='enable-ec_nistp_64_gcc_128 enable-tls1_3 -fuse-ld=gold'
    
     
    Last edited: Sep 14, 2020
  2. eva2000

    eva2000 Administrator Staff Member

    45,201
    10,280
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,934
    Local Time:
    10:40 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    If running Centmin Mod 123.09beta01 or higher, you will have access to a cminfo top command which can also provide a summary of statistics - some of which are explained in info and linked articles below.

    FYI, CPU load optimal is 1.0 per cpu thread so 8 cpu thread normal = 8x1 = 8.0. 16 cpu server normal is 16x1 = 16.0

    so cpu load of 8 for
    • 2 cpu server = 8/2 = 4.0 load average
    • 4 cpu server = 8/4 = 2.0 load average
    • 8 cpu server = 8/8 = 1.0 load average
    When you type top command, hit number 1 to break down per cpu stats

    Centmin Mod is provided as is, so troubleshooting load issues is left to end user to do. However, there's many linux tools and scripts that can help you figure out what was causing the load issues and when.

    If you're providing info on this forum, more info might be helpful
    1. What version of Centmin Mod ? .08 stable or .09 beta ? If .09 beta when was it installed and when was last time you updated ?
    2. What's your VPS/Server hardware specifications ? cpu type ? memory available ? disk space ?
    3. Who's your web host ? Different hosts have different limits for server resource usage and some are more restrictive that others so it could just be their resource usage policy you tripped which may or may not be restrictive. If restrictive, then real solution would be finding a better web host. If not restrictive, then it's finding out what caused high cpu load.
    4. If running Centmin Mod 123.09beta01 or higher, you will have access to a cminfo top command which can also provide a summary of statistics - some of which are explained in info and linked articles below.
    Tools and commands you will want to read up on and learn for basic system admin tasks and troubleshooting.
    Notes:
    For example some commands which are also available automatically when you run cminfo top command
    Code (Text):
    top -bn1
    

    Code (Text):
    ps aufxw
    

    Code (Text):
    sar -q
    

    Code (Text):
    sar -r
    

    Code (Text):
    sar -q -f /var/log/sa/sa$(date +%d -d yesterday)
    

    Code (Text):
    sar -r -f /var/log/sa/sa$(date +%d -d yesterday)
    

    If you want to look at per process usage stats use pidstat run pidstat every 1 sec for 5 runs and sed is just to replace your main hostname with word hostname so mask your domain for posting on public places
    Code (Text):
    pidstat -durh 1 5 | sed -e "s|$(hostname)|hostname|g"
    

    for posting code you might want to use CODE tags for code How to use forum BBCODE code tags :)
     
  3. adamus007p

    adamus007p Member

    252
    15
    18
    Feb 8, 2019
    Ratings:
    +26
    Local Time:
    2:40 PM
    Ok problem was with the php script.