Get the most out of your Centmin Mod LEMP stack
Become a Member

Magento My Magento website choked today with high traffic, what can I do to improve?

Discussion in 'Ecommerce / Shopping cart usage' started by Afaq, Mar 11, 2018.

  1. Afaq

    Afaq Member

    81
    3
    8
    Aug 5, 2015
    Ratings:
    +5
    Local Time:
    12:29 AM
    So I am running centminmod (stable version) on Linode 8GB server. I have all the default settings as far as Nginx and PPH-FPM is concerned. I am running Magento 1.9 with about 400 products.

    Today, about 1300 concurrent users came to my website but it couldnt handle the load. The CPU usage skyrocketed to 2000%. And PHP-FPM was using it all. Site was crashing repeatedly. I had to restart PHP-FPM after a while to keep it up. I thought its a CPU related issue, so I resized the linode to 64GB one but similar issue. Website was crawling but was able to handle about 1800-2000 users now.

    I am unsure what the issue could be? I am thinking that may be the Linode isnt good for high traffic. I am thinking of getting a SoYouStart server with Intel Xeon E3-1245v2 (4c/8t) and 32GB RAM. However, I am not sure, it this was an optimization issue or resources issue?

    I am running a pretty standard Magento install with its built-in case. Do you guys think memcache would have helped this?

     
  2. Afaq

    Afaq Member

    81
    3
    8
    Aug 5, 2015
    Ratings:
    +5
    Local Time:
    12:29 AM
    And please tell if it would be beneficial to move to AWS (despite its complex setup)? We dont always get this type of traffic but there are a few days every 2-3 months when we get this.
     
  3. rdan

    rdan Well-Known Member

    5,426
    1,389
    113
    May 25, 2014
    Ratings:
    +2,172
    Local Time:
    3:29 AM
    Mainline
    10.2
  4. rdan

    rdan Well-Known Member

    5,426
    1,389
    113
    May 25, 2014
    Ratings:
    +2,172
    Local Time:
    3:29 AM
    Mainline
    10.2
    CPU also has something to do as Linode CPU is not much powerful than the one I linked above.
     
  5. rdan

    rdan Well-Known Member

    5,426
    1,389
    113
    May 25, 2014
    Ratings:
    +2,172
    Local Time:
    3:29 AM
    Mainline
    10.2
    Update to PHP 7.2?
    Enable Zend OPcache.
    Enable data caching via Memcached.
    Enable Full Page Caching via Fastcgi_cache.
     
  6. Afaq

    Afaq Member

    81
    3
    8
    Aug 5, 2015
    Ratings:
    +5
    Local Time:
    12:29 AM
    Thanks for your suggestion. However, it is a little bit out of my budget. I am looking for something around 50USD. Secondly, I think I need something with 1Gbps outbound bandwidth since Linode also sent me a warning that m usage is over the limit (which is 10Mbps).
     
  7. rdan

    rdan Well-Known Member

    5,426
    1,389
    113
    May 25, 2014
    Ratings:
    +2,172
    Local Time:
    3:29 AM
    Mainline
    10.2
  8. Afaq

    Afaq Member

    81
    3
    8
    Aug 5, 2015
    Ratings:
    +5
    Local Time:
    12:29 AM
  9. Revenge

    Revenge Active Member

    469
    93
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +354
    Local Time:
    7:29 PM
    1.9.x
    10.1.x
    Are you sure the 1300 concurrent are real traffic and not a ddos attack?
     
  10. Jon Snow

    Jon Snow Active Member

    727
    150
    43
    Jun 30, 2017
    Ratings:
    +215
    Local Time:
    3:29 PM
    Nginx 1.13.9
    MariaDB 10.1.31
    Isn't the limit 1 Gbps?
     
  11. eva2000

    eva2000 Administrator Staff Member

    50,852
    11,782
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,234
    Local Time:
    5:29 AM
    Nginx 1.25.x
    MariaDB 10.x
    No that is a preset alert you can modify in Linode manager and not the limit of your VPS. It's there to remind you when you reach certain thresholds and not a warning really. You can modify the limit or disable the alert. Usually, I modify the limit based on my usualy monitored 24x7 peak usage. So only when there's spikes, that I get alerts. Very useful to understand when it's time to upgrade etc.

    Probably 65% of the issue is tuning PHP-FPM correctly and implementation Magento caching and needing more cpu threads to service PHP-FPM, so yes 8GB Linode only has 4 cpu core/threads. Haven't used Magento myself though so you'd have to see what other Magento + Centmin Mod users have to say too the prefixed tagged threads here might have some clues. Particularly this thread Magento - Going beyond Magento Speed Limits without HHVM or Varnish Cache

    Just make sure you're using latest Magento 2.2.x branch version as from benchmarks I see it's much better. There's also Magento Optimization Guide | Magento 2 Developer Documentation which suggest using Redis caching instead of Memcached. Centmin Mod users Upgrade - Nginx - Redis - Insight Guide - How to install Redis server on Centmin Mod LEMP stack

    Regarding PHP-FPM, Centmin Mod is provide as is, so short of scripted related bugs or issues, any further optimisation to the web stack components - nginx, php-fpm, mariadb mysql, csf firewall etc or web app specific configurations are left to the Centmin Mod user to deal with. So I do not provide any free support for such.

    PHP-FPM issues will need troubleshooting i.e. bad gateway 502 /504 timeouts are usually related to Nginx timing out waiting on PHP-FPM to respond as PHP-FPM is overloaded or overwhelmed with requests, so may need to tune PHP-FPM values. It also maybe due to PHP-FPM in turn being queued and backed up waiting on MariaDB MySQL server to respond - so also need to look at MySQL.

    You'll need to tune your PHP-FPM settings with php-fpm main pool config file at /usr/local/etc/php-fpm.conf (overview of config files) and this is left up to end user to do but here's a thread for starters to enable PHP-FPM status page output outlined at
    Enabling PHP-FPM status also allows setting up 3rd party PHP-FPM status metric monitoring from services like:

    Checking PHP-FPM etc logs



    You'll also need to check into your PHP-FPM, Nginx and MariaDB logs which you can find as outlined at How to troubleshoot Centmin Mod initial install issues

    Server logs include Nginx, PHP-FPM, MariaDB MySQL error logs as well as others. You can find your Centmin Mod install/menu logs at FAQ 7 and server logs at FAQ 19 at Centmin Mod FAQ (most up to date info in FAQ so always read that first). Spoiler tag below has info too but may not be up to date.

    Some of Centmin Mod's installed software will have their own access and error logs which maybe useful for diagnosing errors or give info, notes, or warning notices.

    Note: There's no support provided by me for diagnosing such errors which may occur for various reasons including misconfiguration of installed php/mysql scripts or applications.

    In SSH2 telnet you can use tail command to view the last X number of lines in the file.

    For example for viewing last 10 lines in the file for:

    For Nginx access and error logs:
    Code:
      tail -10 /usr/local/nginx/logs/access.log
      tail -10 /usr/local/nginx/logs/error.log
    
    For specific domainname.com access and error log:
    Code:
      tail -10 /home/nginx/domains/domainname.com/log/access.log
      tail -10 /home/nginx/domains/domainname.com/log/error.log
    
    For other system error logs located at /var/log:

    list /var/log files in ascending time order so the most recently modified files are at the bottom
    Code:
      ls -lhrt /var/log
    
    Code:
    total 2.7M
    -rw------- 1 root  root    0 Aug 29 15:33 tallylog
    -rw------- 1 root  root    0 Aug 29 15:33 spooler
    drwx------ 3 root  root 4.0K Aug 29 15:35 samba
    drwxr-xr-x 2 root  root 4.0K Aug 29 15:35 mail
    -rw-r--r-- 1 root  500     0 Oct  8 18:13 dmesg.old
    -rw------- 1 root  500     0 Oct  8 18:13 boot.log
    -rw-r--r-- 1 root  500     0 Oct  8 18:14 dmesg
    drwx------ 2 root  root 4.0K Oct  8 18:14 httpd
    drwxr-xr-x 2 root  root 4.0K Oct  8 19:08 php-fpm
    -rw-rw---- 1 mysql root 2.3K Oct  9 12:38 mysqld.log
    -rw------- 1 root  root 9.2K Oct 26 10:48 yum.log
    -rw------- 1 root  utmp  94K Nov  7 22:59 btmp
    drwxr-xr-x 2 root  root 4.0K Nov  8 00:00 sa
    -rw------- 1 root  root 269K Nov  8 21:39 messages
    -rw------- 1 root  root 110K Nov  8 23:08 secure
    -rw-rw-r-- 1 root  utmp  43K Nov  8 23:08 wtmp
    -rw-r--r-- 1 root  root 144K Nov  8 23:08 lastlog
    -rw------- 1 root  root  69K Nov  8 23:08 lfd.log
    -rw------- 1 root  root 332K Nov  8 23:08 maillog
    -rw------- 1 root  500  1.6M Nov  8 23:10 cron
    
    For PHP-FPM error log:
    Code:
      tail -10 /var/log/php-fpm/www-error.log
    
    and/or
    Code:
      /var/log/php-fpm/www-php.error.log
    
    For MySQL / MariaDB error log:
    Code:
      tail -10 /var/log/mysqld.log
    
    For CSF firewall LFD log:
    Code:
      tail -10 /var/log/lfd.log
    
    For Mail log:
    Code:
      tail -10 /var/log/maillog
    
    For Cron job logs:
    Code:
      tail -10 /var/log/cron
    

    How to edit php.ini and php-fpm configuration files ?



    Centmin Mod install created command short cuts outlined here to allow you to quickly edit your /usr/local/lib/php.ini file and your /usr/local/etc/php-fpm.conf file. Full list of command shortcuts below:
    • Edit php.ini = phpedit ( /usr/local/lib/php.ini )
    • Edit my.cnf = mycnf ( /etc/my.cnf )
    • Edit php-fpm.conf = fpmconf ( /usr/local/etc/php-fpm.conf )
    • Edit nginx.conf = nginxconf ( /usr/local/nginx/conf/nginx.conf )
    • Edit (nginx) virtual.conf = vhostconf - only edits /usr/local/nginx/conf/conf.d/virtual.conf not the additional vhost domain.com.conf files added later
    • Edit (nginx) php.conf = phpinc ( /usr/local/nginx/conf/php.conf )
    • Edit (nginx) drop.conf = dropinc ( /usr/local/nginx/conf/drop.conf )
    • Edit (nginx) staticfiles.conf = statfilesinc ( /usr/local/nginx/conf/staticfiles.conf )
    • nginx stop/start/restart = ngxstop/ngxstart/ngxrestart
    • php-fpm stop/start/restart = fpmstop/fpmstart/fpmrestart
    • mysql stop/start/restart = mysqlstop/mysqlstart/mysqlrestart
    • nginx + php-fpm stop/start/restart = npstop/npstart/nprestart
    • memcached stop/start/restart =memcachedstop/memcachedstart/memcachedrestart
    • csf stop/start/restart = csfstop/csfstart/csfrestart

    Troubleshooting Tools



    However, there's many linux tools and scripts that can help you figure out what was causing the load issues and when.

    Tools and commands you will want to read up on and learn for basic system admin tasks and troubleshooting.
    Notes:
    However, Centmin Mod users are free to help each other out and ask questions or give answers on this community forum. My hopes are that this community forum evolves so that more veteran long time Centmin Mod users help new Centmin Mod users out :)
     
  12. eva2000

    eva2000 Administrator Staff Member

    50,852
    11,782
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,234
    Local Time:
    5:29 AM
    Nginx 1.25.x
    MariaDB 10.x
    hmm could of sworn Magento 2 was faster but benchmarks i googled don't show that
    again, I haven't used Magento myself heh

    but wonder if optimisation recommendations at Magento Optimization Guide | Magento 2 Developer Documentation make a difference in Magento 2.2?

    blog at Magento & Performance Optimization Blog by Goivvy.com also has alot of Magento optimisation articles

    i.e.
    Though this article says Magento 2 > 1 Magento 1 Vs Magento 2: Performance, Speed and Features Comparison

    and
    Probably best to focus on optimising Nginx, PHP-FPM and MySQL for Magento 1.9 first as such optimisations can translate over to future Magento version upgrades. Seems Magento 1.x support is until 2018 too!
     
    Last edited: Mar 11, 2018
  13. Afaq

    Afaq Member

    81
    3
    8
    Aug 5, 2015
    Ratings:
    +5
    Local Time:
    12:29 AM
    Thanks for the details response.

    We expect a huge wave of traffic tonight as well. So I am moving to a bigger dedicated server (4c/8t, 16GB). While I move, I want all traffic to go to a static page. How do I do that in my current NGINX config?

    Code:
    # Centmin Mod Getting Started Guide
    # must read http://centminmod.com/getstarted.html
    
    # redirect from non-www to www
    # uncomment, save file and restart Nginx to enable
    # if unsure use return 302 before using return 301
    #server {
    #            listen   80;
    #            server_name domain.com;
    #            return 301 $scheme://www.domain.com$request_uri;
    #       }
    
    server {
      listen 80;
      server_name .domain.com;
    
    
    
    # ngx_pagespeed & ngx_pagespeed handler
    #include /usr/local/nginx/conf/pagespeed.conf;
    #include /usr/local/nginx/conf/pagespeedhandler.conf;
    #include /usr/local/nginx/conf/pagespeedstatslog.conf;
    
      # limit_conn limit_per_ip 16;
      # ssi  on;
    
      access_log /home/nginx/domains/domain.com/log/access.log combined buffer=256k flush=60m;
      error_log /home/nginx/domains/domain.com/log/error.log;
    
      root /home/nginx/domains/domain.com/public;
    
      # prevent access to ./directories and files
      location ~ (?:^|/)\. {
       deny all;
      } 
    
      location / {
    
    # block common exploits, sql injections etc
    #include /usr/local/nginx/conf/block.conf;
    
      # Enables directory listings when index file not found
      #autoindex  on;
    
      # Shows file listing times as local time
      #autoindex_localtime on;
    
      # Enable for vBulletin usage WITHOUT vbSEO installed
      # More example Nginx vhost configurations at
      # http://centminmod.com/nginx_configure.html
      try_files        $uri $uri/ @handler;
      index index.html index.php;
      expires 30d;
    
      }
    
     location @handler {
     rewrite / /index.php;
     }
    
     location ~ .php/ {
       rewrite ^(.*.php)/ $1 last;
     }
    
     location ~ .php$ {
     #if(!-e $request_filename) {
     #  rewrite / /index.php last;
     #}
    
     expires  off;
     fastcgi_pass 127.0.0.1:9000;
     #fastcgi_param HTTPS $fastcgi_https;
     fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
     fastcgi_param MAGE_RUN_CODE default;
     fastcgi_param MAGE_RUN_TYPE store;
     include fastcgi_params;
    }
    
     # include /usr/local/nginx/conf/staticfiles.conf;
     # include /usr/local/nginx/conf/php.conf;
     # include /usr/local/nginx/conf/drop.conf;
      #include /usr/local/nginx/conf/errorpage.conf;
     # include /usr/local/nginx/conf/vts_server.conf;
    }
    
     
  14. eva2000

    eva2000 Administrator Staff Member

    50,852
    11,782
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,234
    Local Time:
    5:29 AM
    Nginx 1.25.x
    MariaDB 10.x
    You're not using HTTPS ? that vhost is non-HTTPS version

    Looks like vhost isn't using any of 123.09beta01 default maintenance mode features because you could enable maintenance mode. You can probably add that back manually if on 123.09beta01 as per 1st post (see Verifying SiteStatus Maintenance Mode Setup section) at Beta Branch - sitestatus maintenance mode

    pretty sure won't help if you don't tune and figure out the load issues i.e. PHP-FPM tuning outlined above
     
  15. Afaq

    Afaq Member

    81
    3
    8
    Aug 5, 2015
    Ratings:
    +5
    Local Time:
    12:29 AM
    I am not using HTTPS and I am running 1.2.3.08. Not the latest beta.
     
  16. eva2000

    eva2000 Administrator Staff Member

    50,852
    11,782
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,234
    Local Time:
    5:29 AM
    Nginx 1.25.x
    MariaDB 10.x
  17. klimbo

    klimbo New Member

    8
    2
    1
    Mar 6, 2019
    Ratings:
    +2
    Local Time:
    5:29 AM
    CentOS 6.5 64bit
    MariaDB 5.5
    i agreed with you bro
    i think that vhost does not use any of the default maintenance mode features of 123.09beta01