Want to subscribe to topics you're interested in?
Become a Member

Hight load and users down.

Discussion in 'Nginx, PHP-FPM & MariaDB MySQL' started by KinderOvO, Feb 20, 2016.

  1. KinderOvO

    KinderOvO Member

    92
    1
    8
    Jan 13, 2015
    Ratings:
    +1
    Local Time:
    8:17 AM
    1.4
    10
    [​IMG]


    My active connections always stays above 9000, but not now is coming to 3600, when it reaches this value, all threads in vain to 99% connections fall for 2000.
    Google analytics show 25/35 requests per second

    One would give me a help?

    Server Spec: 2x2620v3 / 64memory / 1gbit / 100tb / 960gb ssd / 100tb.com
    2x2620v3
     
  2. rdan

    rdan Well-Known Member

    5,447
    1,408
    113
    May 25, 2014
    Ratings:
    +2,201
    Local Time:
    6:17 AM
    Mainline
    10.2
    Try this PHP-FPM Config:

    Meaning only 35 visitor real time?

    I think your site has been hit by Layer 7 DDOS attack.
    Maybe try Cloudflare or Sucuri protection?

    Or enable CSF DOS protection.
     
  3. KinderOvO

    KinderOvO Member

    92
    1
    8
    Jan 13, 2015
    Ratings:
    +1
    Local Time:
    8:17 AM
    1.4
    10
    No! 35 request per second.
    Online users average 4k simultaneously.
    test your configuration, worse :(
     
  4. rdan

    rdan Well-Known Member

    5,447
    1,408
    113
    May 25, 2014
    Ratings:
    +2,201
    Local Time:
    6:17 AM
    Mainline
    10.2
    What I understand on this is real time.

    How do you measure this/count?
    Via XenForo online user list or what?

    I always use dynamic and use the same formula in all large site I manage (4-8K per second/real time) and it's just 20-40% CPU Load.
     
  5. rdan

    rdan Well-Known Member

    5,447
    1,408
    113
    May 25, 2014
    Ratings:
    +2,201
    Local Time:
    6:17 AM
    Mainline
    10.2
    Basically I use this formula.

    For low CPU clock 3 ghz below:
    For high CPU clock 3 ghz up.
     
  6. KinderOvO

    KinderOvO Member

    92
    1
    8
    Jan 13, 2015
    Ratings:
    +1
    Local Time:
    8:17 AM
    1.4
    10
    I was running before cpanel with nginxcp plugin and is working with 9000 connections, now, cpu is 99% drops all connections.

    is there any limitation? should I set up? something that is limiting users to access?
     
  7. rdan

    rdan Well-Known Member

    5,447
    1,408
    113
    May 25, 2014
    Ratings:
    +2,201
    Local Time:
    6:17 AM
    Mainline
    10.2
    And now you are already running Centminmod based server?
     
  8. rdan

    rdan Well-Known Member

    5,447
    1,408
    113
    May 25, 2014
    Ratings:
    +2,201
    Local Time:
    6:17 AM
    Mainline
    10.2
    How is that SSD setup?
    Only single disk?
     
  9. eva2000

    eva2000 Administrator Staff Member

    54,873
    12,240
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,811
    Local Time:
    8:17 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Centmin Mod is provide as is, so short of scripted related bugs or issues, any further optimisation to the web stack components - nginx, php-fpm, mariadb mysql, csf firewall etc or web app specific configurations are left to the Centmin Mod user to deal with. So I do not provide any free support for such. However, there's many linux tools and scripts that can help you figure out what was causing the load issues and when.

    FYI, cpu load isn't high you have dual Xeon E5-2620v3 which is 2x2x6 = 24 cpu thread server so normal load is 1.0x 24 = 24.0 so under that you're fine Intel® Xeon® Processor E5-2620 v3 (15M Cache, 2.40 GHz) Specifications CPU load of 8 is only 8/24 = 0.333 cpu load average so low cpu usage :)

    If you're providing info on this forum, more info might be helpful
    1. What version of Centmin Mod ? .08 stable or .09 beta ? If .09 beta when was it installed and when was last time you updated ?
    Tools and commands you will want to read up on and learn for basic system admin tasks and troubleshooting.
    Notes:
    Are you getting any bad gateway 502/504 timeouts ? Bad gateway 502 /504 timeouts are usually related to Nginx timing out waiting on PHP-FPM to respond as PHP-FPM is overloaded or overwhelmed with requests, so may need to tune PHP-FPM values. It also maybe due to PHP-FPM in turn being queued and backed up waiting on MariaDB MySQL server to respond - so also need to look at MySQL.

    You'll need to tune your PHP-FPM settings and this is left up to end user to do but here's a thread for starters to enable php status page output outlined at PHP-FPM - CentminMod.com LEMP Nginx web stack for CentOS and PHP-FPM - pm.max_children | Centmin Mod Community which outlines the official PHP-FPM config documentation as well.

    Checking PHP-FPM etc logs



    You'll also need to check into your PHP-FPM, Nginx and MariaDB logs which you can find as outlined at How to troubleshoot Centmin Mod initial install issues

    Server logs include Nginx, PHP-FPM, MariaDB MySQL error logs as well as others. You can find your Centmin Mod install/menu logs at FAQ 7 and server logs at FAQ 19 at Centmin Mod FAQ (most up to date info in FAQ so always read that first). Spoiler tag below has info too but may not be up to date.

    Some of Centmin Mod's installed software will have their own access and error logs which maybe useful for diagnosing errors or give info, notes, or warning notices.

    Note: There's no support provided by me for diagnosing such errors which may occur for various reasons including misconfiguration of installed php/mysql scripts or applications.

    In SSH2 telnet you can use tail command to view the last X number of lines in the file.

    For example for viewing last 10 lines in the file for:

    For Nginx access and error logs:
    Code:
    tail -10 /usr/local/nginx/logs/access.log
    tail -10 /usr/local/nginx/logs/error.log
    
    For specific domainname.com access and error log:
    Code:
    tail -10 /home/nginx/domains/domainname.com/log/access.log
    tail -10 /home/nginx/domains/domainname.com/log/error.log
    
    For other system error logs located at /var/log:

    list /var/log files in ascending time order so the most recently modified files are at the bottom
    Code:
    ls -lhrt /var/log
    
    Code:
    total 2.7M
    -rw------- 1 root root 0 Aug 29 15:33 tallylog
    -rw------- 1 root root 0 Aug 29 15:33 spooler
    drwx------ 3 root root 4.0K Aug 29 15:35 samba
    drwxr-xr-x 2 root root 4.0K Aug 29 15:35 mail
    -rw-r--r-- 1 root 500 0 Oct 8 18:13 dmesg.old
    -rw------- 1 root 500 0 Oct 8 18:13 boot.log
    -rw-r--r-- 1 root 500 0 Oct 8 18:14 dmesg
    drwx------ 2 root root 4.0K Oct 8 18:14 httpd
    drwxr-xr-x 2 root root 4.0K Oct 8 19:08 php-fpm
    -rw-rw---- 1 mysql root 2.3K Oct 9 12:38 mysqld.log
    -rw------- 1 root root 9.2K Oct 26 10:48 yum.log
    -rw------- 1 root utmp 94K Nov 7 22:59 btmp
    drwxr-xr-x 2 root root 4.0K Nov 8 00:00 sa
    -rw------- 1 root root 269K Nov 8 21:39 messages
    -rw------- 1 root root 110K Nov 8 23:08 secure
    -rw-rw-r-- 1 root utmp 43K Nov 8 23:08 wtmp
    -rw-r--r-- 1 root root 144K Nov 8 23:08 lastlog
    -rw------- 1 root root 69K Nov 8 23:08 lfd.log
    -rw------- 1 root root 332K Nov 8 23:08 maillog
    -rw------- 1 root 500 1.6M Nov 8 23:10 cron
    
    For PHP-FPM error log:
    Code:
    tail -10 /var/log/php-fpm/www-error.log
    
    and/or
    Code:
    /var/log/php-fpm/www-php.error.log
    
    For MySQL / MariaDB error log:
    Code:
    tail -10 /var/log/mysqld.log
    
    For CSF firewall LFD log:
    Code:
    tail -10 /var/log/lfd.log
    
    For Mail log:
    Code:
    tail -10 /var/log/maillog
    
    For Cron job logs:
    Code:
    tail -10 /var/log/cron
    

    How to edit php.ini and php-fpm configuration files ?



    Centmin Mod install created command short cuts outlined here to allow you to quickly edit your /usr/local/lib/php.ini file and your /usr/local/etc/php-fpm.conf file. Full list of command shortcuts below:
    • Edit php.ini = phpedit ( /usr/local/lib/php.ini )
    • Edit my.cnf = mycnf ( /etc/my.cnf )
    • Edit php-fpm.conf = fpmconf ( /usr/local/etc/php-fpm.conf )
    • Edit nginx.conf = nginxconf ( /usr/local/nginx/conf/nginx.conf )
    • Edit (nginx) virtual.conf = vhostconf - only edits /usr/local/nginx/conf/conf.d/virtual.conf not the additional vhost domain.com.conf files added later
    • Edit (nginx) php.conf = phpinc ( /usr/local/nginx/conf/php.conf )
    • Edit (nginx) drop.conf = dropinc ( /usr/local/nginx/conf/drop.conf )
    • Edit (nginx) staticfiles.conf = statfilesinc ( /usr/local/nginx/conf/staticfiles.conf )
    • nginx stop/start/restart = ngxstop/ngxstart/ngxrestart
    • php-fpm stop/start/restart = fpmstop/fpmstart/fpmrestart
    • mysql stop/start/restart = mysqlstop/mysqlstart/mysqlrestart
    • nginx + php-fpm stop/start/restart = npstop/npstart/nprestart
    • memcached stop/start/restart =memcachedstop/memcachedstart/memcachedrestart
    • csf stop/start/restart = csfstop/csfstart/csfrestart
     
  10. KinderOvO

    KinderOvO Member

    92
    1
    8
    Jan 13, 2015
    Ratings:
    +1
    Local Time:
    8:17 AM
    1.4
    10
    - I use the stable version of centminmod "1.2.3-eva2000.08".
    - I only use 1 ssd drive.
    I believe that high consumption, this being caused by not using any cache system.
    I do not use cache because my wordpress use random posts every visit or f5 rotate the posts.
    I believe it causes a big mysql consumption and CPU.
    I thought to use only memocache, but did not understand its installation and use with wordpress.
    Note: must the posts are random, so do not use cache.
     
  11. rdan

    rdan Well-Known Member

    5,447
    1,408
    113
    May 25, 2014
    Ratings:
    +2,201
    Local Time:
    6:17 AM
    Mainline
    10.2
    You must at-least enable Redis/Memcached Data Caching.
    And use PHP 7.0
    That will give a major boost.
     
  12. KinderOvO

    KinderOvO Member

    92
    1
    8
    Jan 13, 2015
    Ratings:
    +1
    Local Time:
    8:17 AM
    1.4
    10
    I understand, but I can not make cache, because I need to be always Random posts.
    whenever visitor accesses or updates f5 appears new posts :(
     
  13. KinderOvO

    KinderOvO Member

    92
    1
    8
    Jan 13, 2015
    Ratings:
    +1
    Local Time:
    8:17 AM
    1.4
    10
    I think about splitting into two servers.
    Server 1 Webserver
    Server 2 Mysql

    I use (ovh or online.net)

    I think on 2 servers
    2xE5-2620v3 for webserver + 2xE5-2620v3 for mysql
    or
    2xE5-2640v3 for webserver + 2xE5-2640v3 for mysql
     
  14. rdan

    rdan Well-Known Member

    5,447
    1,408
    113
    May 25, 2014
    Ratings:
    +2,201
    Local Time:
    6:17 AM
    Mainline
    10.2
    Just choose OVH single Dedicated beast Server.
    This specs:
    [​IMG]

    It is really powerful I swear (y).

    Data Caching can't cause you any problem with random post, it will just save Database Work Load.

    And it will also add latency for your MYSQL.
    I find it CPU consuming even on the same private network.
     
  15. KinderOvO

    KinderOvO Member

    92
    1
    8
    Jan 13, 2015
    Ratings:
    +1
    Local Time:
    8:17 AM
    1.4
    10
    - I thought of two servers because the OVH private network has 10Gbps.

    - So this E5-1650v3 server would be more powerful than 2xE5-2640v3?

    - My site has only 5GB, which 4x800SSD benefit Hardware Raid 10? currently use 2x960GB SSD RAID 0 software

    - I tested Redis, all content was being cached, posts stopped rotating.
    I will test memcached, but it only needs to install the server or have to make any changes? You need to use plugin?
    Looking for a way to relieve mysql without stopping to rotate my posts.

    Thank yourself for attention, very rare to find people willing to help today.
     
  16. rdan

    rdan Well-Known Member

    5,447
    1,408
    113
    May 25, 2014
    Ratings:
    +2,201
    Local Time:
    6:17 AM
    Mainline
    10.2
    Personally I don't think you need a separate DB server since your Mysql is not the resource hog here.
    It's PHP-FPM needs attention so you need faster CPU on single core which is CPU with higher Clock Speed.

    In regards to single CPU core works, Single E5-1650v3 is really powerful.
    I would choose that CPU over 2xE5-2640v3.

    4 x 800 GB SSD on Raid 10, will make your site fly!.
    It's not about the disk size but the I/O speed.
    I have this setup and it beats up Linode and Ramnode SSD I/O Speed.

    I'm not referring to Redis Full Page cache on Nginx level.
    But only Data store: Redis Object Cache — WordPress Plugins
     
  17. KinderOvO

    KinderOvO Member

    92
    1
    8
    Jan 13, 2015
    Ratings:
    +1
    Local Time:
    8:17 AM
    1.4
    10
    Hired the server recommended not improved at all :(
     
  18. eva2000

    eva2000 Administrator Staff Member

    54,873
    12,240
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,811
    Local Time:
    8:17 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    did you read https://community.centminmod.com/posts/26971/ your cpu load isn't high at all it's equivalent to 0.333 cpu load oe 33.3% utilisation but yes you need to tune your settings as outlined in linked post
     
  19. dorobo

    dorobo Active Member

    420
    104
    43
    Jun 6, 2014
    Ratings:
    +162
    Local Time:
    6:17 AM
    latest
    latest
    You know eva2000 does paid server tuning. Maybe you can afford it.
     
  20. KinderOvO

    KinderOvO Member

    92
    1
    8
    Jan 13, 2015
    Ratings:
    +1
    Local Time:
    8:17 AM
    1.4
    10
    Cpanel Server + apache + nginxcp
    netstat -n | grep :80 |wc -l
    4500

    Server Centminmod
    netstat -n | grep :80 |wc -l
    2100

    Lol