Sysadmin Finding the cause of a server crash?

Centmin Mod is provide as is, so short of scripted related bugs or issues, any further optimisation to the web stack components - nginx, php-fpm, mariadb mysql, csf firewall etc or web app specific configurations are left to the Centmin Mod user to deal with. So I do not provide any free support for such. But here's some hints

Nginx 502 or 504 Bad Gateway Errors

Bad gateway 502 /504 timeouts are usually related to Nginx timing out waiting on PHP-FPM to respond as PHP-FPM is overloaded or overwhelmed with requests, so may need to tune PHP-FPM values. It also maybe due to PHP-FPM in turn being queued and backed up waiting on MariaDB MySQL server to respond - so also need to look at MySQL.

You'll need to tune your PHP-FPM settings with php-fpm main pool config file at /usr/local/etc/php-fpm.conf (overview of config files) and this is left up to end user to do but here's a thread for starters to enable php status page output outlined at

• http://centminmod.com/phpfpm.html#browserstatus and
• https://community.centminmod.com/threads/pm-max_children.479/ and
• https://community.centminmod.com/threads/warning-pool-www-server-reached-max_children-setting-50-consider-raising-it.791/ which outlines the official PHP-FPM config documentation as well.

Checking PHP-FPM etc logs

You'll also need to check into your PHP-FPM, Nginx and MariaDB logs which you can find as outlined at How to troubleshoot Centmin Mod initial install issues

Server logs include Nginx, PHP-FPM, MariaDB MySQL error logs as well as others. You can find your Centmin Mod install/menu logs at FAQ 7 and server logs at FAQ 19 at Centmin Mod FAQ (most up to date info in FAQ so always read that first). Spoiler tag below has info too but may not be up to date.

Some of Centmin Mod's installed software will have their own access and error logs which maybe useful for diagnosing errors or give info, notes, or warning notices.

Note: There's no support provided by me for diagnosing such errors which may occur for various reasons including misconfiguration of installed php/mysql scripts or applications.

In SSH2 telnet you can use tail command to view the last X number of lines in the file.

For example for viewing last 10 lines in the file for:

For Nginx access and error logs:

Code:

  tail -10 /usr/local/nginx/logs/access.log
  tail -10 /usr/local/nginx/logs/error.log

For specific domainname.com access and error log:

Code:

  tail -10 /home/nginx/domains/domainname.com/log/access.log
  tail -10 /home/nginx/domains/domainname.com/log/error.log

For other system error logs located at /var/log:

list /var/log files in ascending time order so the most recently modified files are at the bottom

Code:

  ls -lhrt /var/log

Code:

total 2.7M
-rw------- 1 root  root    0 Aug 29 15:33 tallylog
-rw------- 1 root  root    0 Aug 29 15:33 spooler
drwx------ 3 root  root 4.0K Aug 29 15:35 samba
drwxr-xr-x 2 root  root 4.0K Aug 29 15:35 mail
-rw-r--r-- 1 root  500     0 Oct  8 18:13 dmesg.old
-rw------- 1 root  500     0 Oct  8 18:13 boot.log
-rw-r--r-- 1 root  500     0 Oct  8 18:14 dmesg
drwx------ 2 root  root 4.0K Oct  8 18:14 httpd
drwxr-xr-x 2 root  root 4.0K Oct  8 19:08 php-fpm
-rw-rw---- 1 mysql root 2.3K Oct  9 12:38 mysqld.log
-rw------- 1 root  root 9.2K Oct 26 10:48 yum.log
-rw------- 1 root  utmp  94K Nov  7 22:59 btmp
drwxr-xr-x 2 root  root 4.0K Nov  8 00:00 sa
-rw------- 1 root  root 269K Nov  8 21:39 messages
-rw------- 1 root  root 110K Nov  8 23:08 secure
-rw-rw-r-- 1 root  utmp  43K Nov  8 23:08 wtmp
-rw-r--r-- 1 root  root 144K Nov  8 23:08 lastlog
-rw------- 1 root  root  69K Nov  8 23:08 lfd.log
-rw------- 1 root  root 332K Nov  8 23:08 maillog
-rw------- 1 root  500  1.6M Nov  8 23:10 cron

For PHP-FPM error log:

Code:

  tail -10 /var/log/php-fpm/www-error.log

and/or

Code:

  /var/log/php-fpm/www-php.error.log

For MySQL / MariaDB error log:

Code:

  tail -10 /var/log/mysqld.log

For CSF firewall LFD log:

Code:

  tail -10 /var/log/lfd.log

For Mail log:

Code:

  tail -10 /var/log/maillog

For Cron job logs:

Code:

  tail -10 /var/log/cron

How to edit php.ini and php-fpm configuration files ?

Centmin Mod install created command short cuts outlined here to allow you to quickly edit your /usr/local/lib/php.ini file and your /usr/local/etc/php-fpm.conf file. Full list of command shortcuts below:

• Edit php.ini = phpedit ( /usr/local/lib/php.ini )
• Edit my.cnf = mycnf ( /etc/my.cnf )
• Edit php-fpm.conf = fpmconf ( /usr/local/etc/php-fpm.conf )
• Edit nginx.conf = nginxconf ( /usr/local/nginx/conf/nginx.conf )
• Edit (nginx) virtual.conf = vhostconf - only edits /usr/local/nginx/conf/conf.d/virtual.conf not the additional vhost domain.com.conf files added later
• Edit (nginx) php.conf = phpinc ( /usr/local/nginx/conf/php.conf )
• Edit (nginx) drop.conf = dropinc ( /usr/local/nginx/conf/drop.conf )
• Edit (nginx) staticfiles.conf = statfilesinc ( /usr/local/nginx/conf/staticfiles.conf )
• nginx stop/start/restart = ngxstop/ngxstart/ngxrestart
• php-fpm stop/start/restart = fpmstop/fpmstart/fpmrestart
• mysql stop/start/restart = mysqlstop/mysqlstart/mysqlrestart
• nginx + php-fpm stop/start/restart = npstop/npstart/nprestart
• memcached stop/start/restart =memcachedstop/memcachedstart/memcachedrestart
• csf stop/start/restart = csfstop/csfstart/csfrestart

Troubleshooting Tools

However, there's many linux tools and scripts that can help you figure out what was causing the load issues and when.

Tools and commands you will want to read up on and learn for basic system admin tasks and troubleshooting.

• 20 Command Line Tools to Monitor Linux Performance
• 13 Linux Performance Monitoring Tools - Part 2
• 80 Linux Monitoring Tools for SysAdmins - Server Density Blog
• Amazing ! 25 Linux Performance Monitoring Tools
• 20 Linux System Monitoring Tools Every SysAdmin Should Know
• How to monitor disk I/O in Linux from command line - Xmodulo
  
• 10 Useful Sar (Sysstat) Examples for UNIX / Linux Performance Monitoring
  
• Generate CPU, Memory & I/O report using SAR command
  
• My mysqlmymonlite.sh script can help with that mysqlmymonlite.sh README text file has instructions for setting up a cronjob to email you reports i.e. every 10/15min or every hour.

Notes:

• Centmin Mod already installed Glances tool. It can be invoked from SSH command glances or top2. Official Glances site at nicolargo.github.io/glances/ and documentation here.

However, Centmin Mod users are free to help each other out and ask questions or give answers on this community forum. My hopes are that this community forum evolves so that more veteran long time Centmin Mod users help new Centmin Mod users out

 

Oh i just add the standard response to the top of my reply for benefit of all visitors and lurkers reading the post as well

 

Use more than one monitoring/uptime service to double check for false alerts https://community.centminmod.com/threads/monitoring-and-alert-tools.151/

I use nixstats, nodequery, nodeping (paid plan), newrelic, uptimerobot, and amazon route53's health checks among many

 

which version of MariaDB ? 10.0 or 10.1 branch ? that was fixed ages ago so check the dates of error messages in your logs [MDEV-5151] mysql_upgrade does not fix "last_update" in "mysql.innodb_table_stats" - JIRA

what output do you get for this command which is usually run after major mysql version updates only i.e. from MariaDB 5.5 to 10.0 to 10.1 and 10.2 etc

Code (Text):

mysql_upgrade --force --verbose

 

Output looks good but i removed it as it reveals your database names as well

did you update from MariaDB 10.0 to 10.1 or was 10.1 the default ?

 

what's output for this command

Code (Text):

mysql -e "show create table mysql.innodb_index_stats\G;"

example for my MariaDB 10.1.20 server

Code (Text):

mysql -e "show create table mysql.innodb_index_stats\G;"
*************************** 1. row ***************************
       Table: innodb_index_stats
Create Table: CREATE TABLE `innodb_index_stats` (
  `database_name` varchar(64) COLLATE utf8_bin NOT NULL,
  `table_name` varchar(64) COLLATE utf8_bin NOT NULL,
  `index_name` varchar(64) COLLATE utf8_bin NOT NULL,
  `last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  `stat_name` varchar(64) COLLATE utf8_bin NOT NULL,
  `stat_value` bigint(20) unsigned NOT NULL,
  `sample_size` bigint(20) unsigned DEFAULT NULL,
  `stat_description` varchar(1024) COLLATE utf8_bin NOT NULL,
  PRIMARY KEY (`database_name`,`table_name`,`index_name`,`stat_name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin STATS_PERSISTENT=0

 

Code (Text):

`last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMES

did you cut off STAMP, from end of that line in copy and paste ? could be mysql_upgrade run i told you do it fixed it as that's what it's there for to apply system database table structure changes for major mysql/mariadb version updates

you can confirm by restart mysql server and then checking the logs again

 

can also check when mysql.innodb_index_stats was created and updated too

Code (Text):

mysqlshow -i mysql | egrep 'innodb_index_stats|Update_time'

example output

Code (Text):

mysqlshow -i mysql | egrep 'innodb_index_stats|Update_time'
| Name                      | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length    | Index_length | Data_free | Auto_increment | Create_time         | Update_time         | Check_time          | Collation         | Checksum | Create_options     | Comment                                           |
| innodb_index_stats        | InnoDB | 10      | Compact    | 13   | 1260           | 16384       | 0                  | 0            | 0         |                | 2016-11-25 08:20:23 |                     |                     | utf8_bin          |          | stats_persistent=0 |   

Was fresh install so only Create_time = 2016-11-25 08:20:23 for me

 

hmm some serious issues going on there !

First check if all server disk partitions have enough free disk space

Code (Text):

df -hT

getting errors on front end of wordpress installs at all ?

backup ALL databases on server not just one including mysql system databases using mysqldump and then stop mysql cleanly, backup physical data files /var/lib/mysql and move it and use mysql_install_db MySQL :: MySQL 5.6 Reference Manual :: 4.4.3 mysql_install_db — Initialize MySQL Data Directory to repopulate /var/lib/mysql and restore mysqldump sql files from sql backup

create backup directory

Code (Text):

mkdir -p /home/mysqlbackups

To backup where DATABASENAME is name of your database

Code (Text):

DBNAME='DATABASENAME'
mysqldump --opt --routines $DBNAME> /home/mysqlbackups/$DBNAME_backup.sql

this command will list all database names on your server

Code (Text):

mysql -e "show databases;"

so change DBNAME variable for each run

To restore need to create the database name then restore from sql file backup

Code (Text):

DBNAME='DATABASENAME'
mysqladmin create $DBNAME
mysql < /home/mysqlbackups/$DBNAME_backup.sql

Start with just backing up via mysqldump, do you get any errors when backing up ?

 

Those can be ignored as they aren't actual databases per se

So can continue

 

@RB1 full guide of the process at https://community.centminmod.com/th...to-reset-your-mysql-database-directory.10001/