Get the most out of your Centmin Mod LEMP stack
Become a Member

/etc/systemd/system/php-fpm.service.d/limit.conf causing issues primarily with WordPress Gutenberg.

Discussion in 'Bug Reports' started by jcat, Apr 30, 2019.

  1. jcat

    jcat Member

    114
    17
    18
    Jun 21, 2015
    Ratings:
    +44
    Local Time:
    10:31 AM
    Please fill in any relevant information that applies to you:
    • CentOS Version: CentOS Linux release 7.6.1810
    • Centmin Mod Version Installed: 123.09beta01
    • Nginx Version Installed: 1.15.12
    • PHP Version Installed: 7.2.16
    • MariaDB MySQL Version Installed: 10.3.14
    • When was last time updated Centmin Mod code base ? : every 8 hours
    • Persistent Config:
    Code:
    NGXDYNAMIC_BROTLI=y
    NGINX_LIBBROTLI=y
    ZSTD_LOGROTATE_NGINX=y
    ZSTD_LOGROTATE_PHPFPM=y
    MARIADB_INSTALLTENTHREE=y
    PHP_BROTLI=y
    PHP_LZFOUR=y
    PHP_LZF=y
    PHP_PGO=y
    PHP_ZSTD=y
    LETSENCRYPT_DETECT='y'
    DUALCERTS='y'
    

    Been seeing this somewhat more and more where limits set in /etc/systemd/system/php-fpm.service.d/limit.conf cause 502's, example:

    Snaggy - easy screenshots

    I meant to report this earlier so I could give more examples.

    I actually randomly came across limit.conf because I was troubleshooting an issue with a 502, spent hours and hours on it, finally tried to debug using gdb, ran something like

    Code:
    # gdb /usr/sbin/php-fpm -F --fpm-config /usr/local/etc/php-fpm.conf
    went to test the issue to get some output and then bam.. the issue went away, was totally confused how running php-fpm like that magically fixed it which gave me the idea to check the systemd conf, commented out the lines, restarted php-fpm, and issue went away. I was thinking this was a one off issue but have been seeing it more and more now, especially with Gutenberg errors relating to the wordpress rest api.
     
    • Informative Informative x 1
  2. eva2000

    eva2000 Administrator Staff Member

    41,723
    9,395
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,431
    Local Time:
    12:31 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    OpenVZ 7 container based CentOS 7 ?

    contents of /etc/systemd/system/php-fpm.service.d/limit.conf ?

    have you tried narrowing down which setting i /etc/systemd/system/php-fpm.service.d/limit.conf causes the issue as it seems to be unique to OpenVZ 7 container based VPSes

    Though it could be 502 from generally php-fpm load issues and conincidentally it resolved at the time you tested the issue


    PHP-FPM might need tuning. Excuse the generic response (helps everyone reading this thread too :) :) Read PHP-FPM - How to troubleshoot & optimize PHP-FPM server? If the case, then Centmin Mod is provide as is, so short of scripted related bugs or issues, any further optimisation to the web stack components - nginx, php-fpm, mariadb mysql, csf firewall etc or web app specific configurations are left to the Centmin Mod user to deal with. So I do not provide any free support for such.

    Bad gateway 502 /504 timeouts are usually related to Nginx timing out waiting on PHP-FPM to respond as PHP-FPM is overloaded or overwhelmed with requests, so may need to tune PHP-FPM values. It also maybe due to PHP-FPM in turn being queued and backed up waiting on MariaDB MySQL server to respond - so also need to look at MySQL.

    You'll need to tune your PHP-FPM settings with php-fpm main pool config file at /usr/local/etc/php-fpm.conf (overview of config files) and this is left up to end user to do but here's a thread for starters to enable PHP-FPM status page output outlined at
    Enabling PHP-FPM status also allows setting up 3rd party PHP-FPM status metric monitoring from services like:

    Checking PHP-FPM etc logs



    You'll also need to check into your PHP-FPM, Nginx and MariaDB logs which you can find as outlined at How to troubleshoot Centmin Mod initial install issues

    Server logs include Nginx, PHP-FPM, MariaDB MySQL error logs as well as others. You can find your Centmin Mod install/menu logs at FAQ 7 and server logs at FAQ 19 at Centmin Mod FAQ (most up to date info in FAQ so always read that first). Spoiler tag below has info too but may not be up to date.

    Some of Centmin Mod's installed software will have their own access and error logs which maybe useful for diagnosing errors or give info, notes, or warning notices.

    Note: There's no support provided by me for diagnosing such errors which may occur for various reasons including misconfiguration of installed php/mysql scripts or applications.

    In SSH2 telnet you can use tail command to view the last X number of lines in the file.

    For example for viewing last 10 lines in the file for:

    For Nginx access and error logs:
    Code:
      tail -10 /usr/local/nginx/logs/access.log
      tail -10 /usr/local/nginx/logs/error.log
    
    For specific domainname.com access and error log:
    Code:
      tail -10 /home/nginx/domains/domainname.com/log/access.log
      tail -10 /home/nginx/domains/domainname.com/log/error.log
    
    For other system error logs located at /var/log:

    list /var/log files in ascending time order so the most recently modified files are at the bottom
    Code:
      ls -lhrt /var/log
    
    Code:
    total 2.7M
    -rw------- 1 root  root    0 Aug 29 15:33 tallylog
    -rw------- 1 root  root    0 Aug 29 15:33 spooler
    drwx------ 3 root  root 4.0K Aug 29 15:35 samba
    drwxr-xr-x 2 root  root 4.0K Aug 29 15:35 mail
    -rw-r--r-- 1 root  500     0 Oct  8 18:13 dmesg.old
    -rw------- 1 root  500     0 Oct  8 18:13 boot.log
    -rw-r--r-- 1 root  500     0 Oct  8 18:14 dmesg
    drwx------ 2 root  root 4.0K Oct  8 18:14 httpd
    drwxr-xr-x 2 root  root 4.0K Oct  8 19:08 php-fpm
    -rw-rw---- 1 mysql root 2.3K Oct  9 12:38 mysqld.log
    -rw------- 1 root  root 9.2K Oct 26 10:48 yum.log
    -rw------- 1 root  utmp  94K Nov  7 22:59 btmp
    drwxr-xr-x 2 root  root 4.0K Nov  8 00:00 sa
    -rw------- 1 root  root 269K Nov  8 21:39 messages
    -rw------- 1 root  root 110K Nov  8 23:08 secure
    -rw-rw-r-- 1 root  utmp  43K Nov  8 23:08 wtmp
    -rw-r--r-- 1 root  root 144K Nov  8 23:08 lastlog
    -rw------- 1 root  root  69K Nov  8 23:08 lfd.log
    -rw------- 1 root  root 332K Nov  8 23:08 maillog
    -rw------- 1 root  500  1.6M Nov  8 23:10 cron
    
    For PHP-FPM error log:
    Code:
      tail -10 /var/log/php-fpm/www-error.log
    
    and/or
    Code:
      /var/log/php-fpm/www-php.error.log
    
    For MySQL / MariaDB error log:
    Code:
      tail -10 /var/log/mysqld.log
    
    For CSF firewall LFD log:
    Code:
      tail -10 /var/log/lfd.log
    
    For Mail log:
    Code:
      tail -10 /var/log/maillog
    
    For Cron job logs:
    Code:
      tail -10 /var/log/cron
    

    How to edit php.ini and php-fpm configuration files ?



    Centmin Mod install created command short cuts outlined here to allow you to quickly edit your /usr/local/lib/php.ini file and your /usr/local/etc/php-fpm.conf file. Full list of command shortcuts below:
    • Edit php.ini = phpedit ( /usr/local/lib/php.ini )
    • Edit my.cnf = mycnf ( /etc/my.cnf )
    • Edit php-fpm.conf = fpmconf ( /usr/local/etc/php-fpm.conf )
    • Edit nginx.conf = nginxconf ( /usr/local/nginx/conf/nginx.conf )
    • Edit (nginx) virtual.conf = vhostconf - only edits /usr/local/nginx/conf/conf.d/virtual.conf not the additional vhost domain.com.conf files added later
    • Edit (nginx) php.conf = phpinc ( /usr/local/nginx/conf/php.conf )
    • Edit (nginx) drop.conf = dropinc ( /usr/local/nginx/conf/drop.conf )
    • Edit (nginx) staticfiles.conf = statfilesinc ( /usr/local/nginx/conf/staticfiles.conf )
    • nginx stop/start/restart = ngxstop/ngxstart/ngxrestart
    • php-fpm stop/start/restart = fpmstop/fpmstart/fpmrestart
    • mysql stop/start/restart = mysqlstop/mysqlstart/mysqlrestart
    • nginx + php-fpm stop/start/restart = npstop/npstart/nprestart
    • memcached stop/start/restart =memcachedstop/memcachedstart/memcachedrestart
    • csf stop/start/restart = csfstop/csfstart/csfrestart

    Troubleshooting Tools



    However, there's many linux tools and scripts that can help you figure out what was causing the load issues and when.

    Tools and commands you will want to read up on and learn for basic system admin tasks and troubleshooting.
    Notes:
    However, Centmin Mod users are free to help each other out and ask questions or give answers on this community forum. My hopes are that this community forum evolves so that more veteran long time Centmin Mod users help new Centmin Mod users out :)
     
    • Like Like x 1
  3. jcat

    jcat Member

    114
    17
    18
    Jun 21, 2015
    Ratings:
    +44
    Local Time:
    10:31 AM
    Yes

    Prior to commenting out the top Limit lines:

    Code:
    [Service]
    LimitNOFILE=262144
    LimitNPROC=16384
    LimitSTACK=262144
    LimitNICE=-15
    #Nice=-10
    StartLimitBurst=50
    #CPUShares=1500
    #CPUSchedulingPolicy=fifo
    #CPUSchedulingPriority=99
    
    Just did now yeah, its this line:

    Code:
    LimitSTACK=262144
    with it, 502, without no issues, just tested back and forth 5 times and each time reproduces.

    No I can 100% ensure this is/was not the case.
     
  4. eva2000

    eva2000 Administrator Staff Member

    41,723
    9,395
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,431
    Local Time:
    12:31 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Ok from what I understand if OpenVZ 7 is like OpenVZ 6, the memory management is different from other virtualizations like KVM/Xen. The stack size tweak in php-fpm systemd service file is handed down from my stack size tweak for init.d php-fpm service file setup. Lowering the stack size helps manage excessive memory used reported by OpenVZ systems as they consider VSZ/virtual memory as used memory unlike other non-OpenVZ systems. So that's why OpenVZ systems can report higher memory used for multithreaded applications as CentOS default stack size is very large. Reducing it = reduces the memory used. This worked with init.d with OpenVZ 6 but not with systemd service file it seems for OpenVZ 7. You can probably verify with php-fpm reported memory usage for ps aufx output with and without LimitSTACK=262144 setting and see. You can also try raising it to LimitSTACK=524288 and see if that helps
     
  5. jcat

    jcat Member

    114
    17
    18
    Jun 21, 2015
    Ratings:
    +44
    Local Time:
    10:31 AM
    How do we disable the stack size script? Its causing to many random issues for us on openvz7, can we disable it in the installer + if it runs during updates / checks
     
  6. eva2000

    eva2000 Administrator Staff Member

    41,723
    9,395
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,431
    Local Time:
    12:31 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    @jcat did you verify if raising it to LimitSTACK=524288 helped or not ?

    could outline the specific problems you getting ?

    Is the problem for initial installed Centmin Mod or on centmin.sh menu option 5 and/or after running cmupdate + centmin.sh ?
     
  7. jcat

    jcat Member

    114
    17
    18
    Jun 21, 2015
    Ratings:
    +44
    Local Time:
    10:31 AM
    It might help initially but it could be random where that limit is then met at some point.

    This particular time, it was from a centminmod install installed back in 05/2017 but site randomly went down, 502 error.

    Checked nginx error log and all we I see is:

    Code:
    2019/10/10 05:41:00 [error] 23367#23367: *192 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 1.1.1.1, server: domain.com.au, request: "GET /somepage/ HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.domain.com.au"
    
    php-fpm was online

    Code:
    # service php-fpm status
    Redirecting to /bin/systemctl status php-fpm.service
    * php-fpm.service - PHP FastCGI Process Manager
       Loaded: loaded (/usr/lib/systemd/system/php-fpm.service; enabled; vendor preset: disabled)
      Drop-In: /etc/systemd/system/php-fpm.service.d
               `-limit.conf
       Active: active (running) since Thu 2019-10-10 05:38:33 CDT; 3s ago
    
    there has been no recent changes to this server.

    To fix, I commented out the lines in /etc/systemd/system/php-fpm.service.d/limit.conf
    and restart php-fpm.

    Site went down at 5AM EST

    Cron ran:

    Code:
    Oct 10 06:00:01 ngx CROND[2612]: (root) CMD (/usr/bin/cmupdate 2>/dev/null ;
    Server time is CST so it occured 2 hours before the cron ran.

    Although when looking at errors log I do see the upstream errors starting around 04:06:45 CST

    The only cron around that time was:

    Code:
    Oct 10 04:00:01 ngx CROND[14735]: (root) CMD (/usr/bin/cminfo_updater 2>/dev/null)