Welcome to Centmin Mod Community
Register Now

Sysadmin Finding the cause of a server crash?

Discussion in 'System Administration' started by RB1, Jan 10, 2017.

  1. RB1

    RB1 Active Member

    281
    72
    28
    Nov 11, 2016
    California
    Ratings:
    +119
    Local Time:
    8:41 AM
    Nginx 1.13.x
    MariaDB 10.1.x
    I'm still a beginner and learning the ropes of server management.

    I have the New Relic server monitor installed on my server, as well as their monitor for HTTP errors. When I was sleeping, my server suffered from about 55 minutes of downtime.

    What should I be looking for in /var/logs (or is that not the right place?).


    EDIT: What do you make of this?

    Jan 8th 2017, 0:39:14 Fremont, CA, USA NetworkError: Connect to example.com:80 [/23.67.2.221] failed: connect timed out

    I'm currently monitoring from three locations (Fremont, Washington D.C., and Dallas). Perhaps it wasn't even a problem with my server and just New Relic's monitoring server going down?
     
    Last edited: Jan 10, 2017
  2. eva2000

    eva2000 Administrator Staff Member

    31,026
    6,928
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,434
    Local Time:
    2:41 AM
    Nginx 1.13.x
    MariaDB 5.5
    Centmin Mod is provide as is, so short of scripted related bugs or issues, any further optimisation to the web stack components - nginx, php-fpm, mariadb mysql, csf firewall etc or web app specific configurations are left to the Centmin Mod user to deal with. So I do not provide any free support for such. But here's some hints :)

    Nginx 502 or 504 Bad Gateway Errors



    Bad gateway 502 /504 timeouts are usually related to Nginx timing out waiting on PHP-FPM to respond as PHP-FPM is overloaded or overwhelmed with requests, so may need to tune PHP-FPM values. It also maybe due to PHP-FPM in turn being queued and backed up waiting on MariaDB MySQL server to respond - so also need to look at MySQL.

    You'll need to tune your PHP-FPM settings with php-fpm main pool config file at /usr/local/etc/php-fpm.conf (overview of config files) and this is left up to end user to do but here's a thread for starters to enable php status page output outlined at

    Checking PHP-FPM etc logs



    You'll also need to check into your PHP-FPM, Nginx and MariaDB logs which you can find as outlined at How to troubleshoot Centmin Mod initial install issues

    Server logs include Nginx, PHP-FPM, MariaDB MySQL error logs as well as others. You can find your Centmin Mod install/menu logs at FAQ 7 and server logs at FAQ 19 at Centmin Mod FAQ (most up to date info in FAQ so always read that first). Spoiler tag below has info too but may not be up to date.

    Some of Centmin Mod's installed software will have their own access and error logs which maybe useful for diagnosing errors or give info, notes, or warning notices.

    Note: There's no support provided by me for diagnosing such errors which may occur for various reasons including misconfiguration of installed php/mysql scripts or applications.

    In SSH2 telnet you can use tail command to view the last X number of lines in the file.

    For example for viewing last 10 lines in the file for:

    For Nginx access and error logs:
    Code:
      tail -10 /usr/local/nginx/logs/access.log
      tail -10 /usr/local/nginx/logs/error.log
    
    For specific domainname.com access and error log:
    Code:
      tail -10 /home/nginx/domains/domainname.com/log/access.log
      tail -10 /home/nginx/domains/domainname.com/log/error.log
    
    For other system error logs located at /var/log:

    list /var/log files in ascending time order so the most recently modified files are at the bottom
    Code:
      ls -lhrt /var/log
    
    Code:
    total 2.7M
    -rw------- 1 root  root    0 Aug 29 15:33 tallylog
    -rw------- 1 root  root    0 Aug 29 15:33 spooler
    drwx------ 3 root  root 4.0K Aug 29 15:35 samba
    drwxr-xr-x 2 root  root 4.0K Aug 29 15:35 mail
    -rw-r--r-- 1 root  500     0 Oct  8 18:13 dmesg.old
    -rw------- 1 root  500     0 Oct  8 18:13 boot.log
    -rw-r--r-- 1 root  500     0 Oct  8 18:14 dmesg
    drwx------ 2 root  root 4.0K Oct  8 18:14 httpd
    drwxr-xr-x 2 root  root 4.0K Oct  8 19:08 php-fpm
    -rw-rw---- 1 mysql root 2.3K Oct  9 12:38 mysqld.log
    -rw------- 1 root  root 9.2K Oct 26 10:48 yum.log
    -rw------- 1 root  utmp  94K Nov  7 22:59 btmp
    drwxr-xr-x 2 root  root 4.0K Nov  8 00:00 sa
    -rw------- 1 root  root 269K Nov  8 21:39 messages
    -rw------- 1 root  root 110K Nov  8 23:08 secure
    -rw-rw-r-- 1 root  utmp  43K Nov  8 23:08 wtmp
    -rw-r--r-- 1 root  root 144K Nov  8 23:08 lastlog
    -rw------- 1 root  root  69K Nov  8 23:08 lfd.log
    -rw------- 1 root  root 332K Nov  8 23:08 maillog
    -rw------- 1 root  500  1.6M Nov  8 23:10 cron
    
    For PHP-FPM error log:
    Code:
      tail -10 /var/log/php-fpm/www-error.log
    
    and/or
    Code:
      /var/log/php-fpm/www-php.error.log
    
    For MySQL / MariaDB error log:
    Code:
      tail -10 /var/log/mysqld.log
    
    For CSF firewall LFD log:
    Code:
      tail -10 /var/log/lfd.log
    
    For Mail log:
    Code:
      tail -10 /var/log/maillog
    
    For Cron job logs:
    Code:
      tail -10 /var/log/cron
    

    How to edit php.ini and php-fpm configuration files ?



    Centmin Mod install created command short cuts outlined here to allow you to quickly edit your /usr/local/lib/php.ini file and your /usr/local/etc/php-fpm.conf file. Full list of command shortcuts below:
    • Edit php.ini = phpedit ( /usr/local/lib/php.ini )
    • Edit my.cnf = mycnf ( /etc/my.cnf )
    • Edit php-fpm.conf = fpmconf ( /usr/local/etc/php-fpm.conf )
    • Edit nginx.conf = nginxconf ( /usr/local/nginx/conf/nginx.conf )
    • Edit (nginx) virtual.conf = vhostconf - only edits /usr/local/nginx/conf/conf.d/virtual.conf not the additional vhost domain.com.conf files added later
    • Edit (nginx) php.conf = phpinc ( /usr/local/nginx/conf/php.conf )
    • Edit (nginx) drop.conf = dropinc ( /usr/local/nginx/conf/drop.conf )
    • Edit (nginx) staticfiles.conf = statfilesinc ( /usr/local/nginx/conf/staticfiles.conf )
    • nginx stop/start/restart = ngxstop/ngxstart/ngxrestart
    • php-fpm stop/start/restart = fpmstop/fpmstart/fpmrestart
    • mysql stop/start/restart = mysqlstop/mysqlstart/mysqlrestart
    • nginx + php-fpm stop/start/restart = npstop/npstart/nprestart
    • memcached stop/start/restart =memcachedstop/memcachedstart/memcachedrestart
    • csf stop/start/restart = csfstop/csfstart/csfrestart

    Troubleshooting Tools



    However, there's many linux tools and scripts that can help you figure out what was causing the load issues and when.

    Tools and commands you will want to read up on and learn for basic system admin tasks and troubleshooting.
    Notes:
    However, Centmin Mod users are free to help each other out and ask questions or give answers on this community forum. My hopes are that this community forum evolves so that more veteran long time Centmin Mod users help new Centmin Mod users out :)
     
    • Like Like x 1
  3. RB1

    RB1 Active Member

    281
    72
    28
    Nov 11, 2016
    California
    Ratings:
    +119
    Local Time:
    8:41 AM
    Nginx 1.13.x
    MariaDB 10.1.x
    I will just reply here 1st before reading.

    Sorry...I know I have been posting on this forum quite a lot. I completely understand that you don't/shouldn't have to respond to every question because it's quite time consuming.
    I'm very appreciative of all the help I've received from the forum and perhaps we can get a conversation among other members going.

    Next time you can just ignore my non-important posts regarding general questions :)
     
  4. eva2000

    eva2000 Administrator Staff Member

    31,026
    6,928
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,434
    Local Time:
    2:41 AM
    Nginx 1.13.x
    MariaDB 5.5
    Oh i just add the standard response to the top of my reply for benefit of all visitors and lurkers reading the post as well :D
     
    • Like Like x 3
  5. eva2000

    eva2000 Administrator Staff Member

    31,026
    6,928
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,434
    Local Time:
    2:41 AM
    Nginx 1.13.x
    MariaDB 5.5
    Use more than one monitoring/uptime service to double check for false alerts https://community.centminmod.com/threads/monitoring-and-alert-tools.151/ :)

    I use nixstats, nodequery, nodeping (paid plan), newrelic, uptimerobot, and amazon route53's health checks among many :D
     
  6. RB1

    RB1 Active Member

    281
    72
    28
    Nov 11, 2016
    California
    Ratings:
    +119
    Local Time:
    8:41 AM
    Nginx 1.13.x
    MariaDB 10.1.x
    Thanks :)
    I'm looking at maybe Route 53 Health check and Rackspace monitor (I am also looking for a CDN, undecided on Cloudfront or Rackspace CDN, I need to figure out how to run some tests for a multitude of countries).

    So far the only errors I see are (I still haven't really tweaked my server; don't know what I'm doing :))
    Code (Text):
    WARNING: [pool www] server reached max_children setting (4), consider raising it
    

    Also some weird mysql errors on a database I'm not using but I don't believe it's related to the crash.
    Code (Text):
    InnoDB: Error: Column last_update in table "mysql"."innodb_table_stats" is INT UNSIGNED NOT NULL but should be BINARY(4) NOT NULL (type mismatch).
    
     
  7. eva2000

    eva2000 Administrator Staff Member

    31,026
    6,928
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,434
    Local Time:
    2:41 AM
    Nginx 1.13.x
    MariaDB 5.5
    which version of MariaDB ? 10.0 or 10.1 branch ? that was fixed ages ago so check the dates of error messages in your logs [MDEV-5151] mysql_upgrade does not fix "last_update" in "mysql.innodb_table_stats" - JIRA

    what output do you get for this command which is usually run after major mysql version updates only i.e. from MariaDB 5.5 to 10.0 to 10.1 and 10.2 etc
    Code (Text):
    mysql_upgrade --force --verbose
     
  8. RB1

    RB1 Active Member

    281
    72
    28
    Nov 11, 2016
    California
    Ratings:
    +119
    Local Time:
    8:41 AM
    Nginx 1.13.x
    MariaDB 10.1.x
    On MariaDB 10.1.20
    Those errors are from today, and I have many of them every day :LOL:
     
  9. eva2000

    eva2000 Administrator Staff Member

    31,026
    6,928
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,434
    Local Time:
    2:41 AM
    Nginx 1.13.x
    MariaDB 5.5
    Output looks good but i removed it as it reveals your database names as well

    did you update from MariaDB 10.0 to 10.1 or was 10.1 the default ?
     
  10. RB1

    RB1 Active Member

    281
    72
    28
    Nov 11, 2016
    California
    Ratings:
    +119
    Local Time:
    8:41 AM
    Nginx 1.13.x
    MariaDB 10.1.x
    Yes I updated from 10.0
     
  11. eva2000

    eva2000 Administrator Staff Member

    31,026
    6,928
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,434
    Local Time:
    2:41 AM
    Nginx 1.13.x
    MariaDB 5.5
    what's output for this command
    Code (Text):
    mysql -e "show create table mysql.innodb_index_stats\G;"
    

    example for my MariaDB 10.1.20 server
    Code (Text):
    mysql -e "show create table mysql.innodb_index_stats\G;"
    *************************** 1. row ***************************
           Table: innodb_index_stats
    Create Table: CREATE TABLE `innodb_index_stats` (
      `database_name` varchar(64) COLLATE utf8_bin NOT NULL,
      `table_name` varchar(64) COLLATE utf8_bin NOT NULL,
      `index_name` varchar(64) COLLATE utf8_bin NOT NULL,
      `last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
      `stat_name` varchar(64) COLLATE utf8_bin NOT NULL,
      `stat_value` bigint(20) unsigned NOT NULL,
      `sample_size` bigint(20) unsigned DEFAULT NULL,
      `stat_description` varchar(1024) COLLATE utf8_bin NOT NULL,
      PRIMARY KEY (`database_name`,`table_name`,`index_name`,`stat_name`)
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin STATS_PERSISTENT=0
     
  12. RB1

    RB1 Active Member

    281
    72
    28
    Nov 11, 2016
    California
    Ratings:
    +119
    Local Time:
    8:41 AM
    Nginx 1.13.x
    MariaDB 10.1.x
    Code (Text):
    *************************** 1. row ***************************
           Table: innodb_index_stats
    Create Table: CREATE TABLE `innodb_index_stats` (
      `database_name` varchar(64) COLLATE utf8_bin NOT NULL,
      `table_name` varchar(64) COLLATE utf8_bin NOT NULL,
      `index_name` varchar(64) COLLATE utf8_bin NOT NULL,
      `last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMES
      `stat_name` varchar(64) COLLATE utf8_bin NOT NULL,
      `stat_value` bigint(20) unsigned NOT NULL,
      `sample_size` bigint(20) unsigned DEFAULT NULL,
      `stat_description` varchar(1024) COLLATE utf8_bin NOT NULL,
      PRIMARY KEY (`database_name`,`table_name`,`index_name`,`stat_name`)
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin STATS_PERSISTENT=0
    


    Looks the same :p
     
  13. eva2000

    eva2000 Administrator Staff Member

    31,026
    6,928
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,434
    Local Time:
    2:41 AM
    Nginx 1.13.x
    MariaDB 5.5
    Code (Text):
    `last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMES

    did you cut off STAMP, from end of that line in copy and paste ? could be mysql_upgrade run i told you do it fixed it as that's what it's there for to apply system database table structure changes for major mysql/mariadb version updates

    you can confirm by restart mysql server and then checking the logs again
     
  14. eva2000

    eva2000 Administrator Staff Member

    31,026
    6,928
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,434
    Local Time:
    2:41 AM
    Nginx 1.13.x
    MariaDB 5.5
    can also check when mysql.innodb_index_stats was created and updated too
    Code (Text):
    mysqlshow -i mysql | egrep 'innodb_index_stats|Update_time'
    

    example output
    Code (Text):
    mysqlshow -i mysql | egrep 'innodb_index_stats|Update_time'
    | Name                      | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length    | Index_length | Data_free | Auto_increment | Create_time         | Update_time         | Check_time          | Collation         | Checksum | Create_options     | Comment                                           |
    | innodb_index_stats        | InnoDB | 10      | Compact    | 13   | 1260           | 16384       | 0                  | 0            | 0         |                | 2016-11-25 08:20:23 |                     |                     | utf8_bin          |          | stats_persistent=0 |   

    Was fresh install so only Create_time = 2016-11-25 08:20:23 for me
     
  15. RB1

    RB1 Active Member

    281
    72
    28
    Nov 11, 2016
    California
    Ratings:
    +119
    Local Time:
    8:41 AM
    Nginx 1.13.x
    MariaDB 10.1.x
    No I didn't cut anything off

    Hmm, some new errors I haven't seen (I changed DB name for the post):

    Create time is: 2017-01-09 22:14:51
    Blank update time
     
  16. eva2000

    eva2000 Administrator Staff Member

    31,026
    6,928
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,434
    Local Time:
    2:41 AM
    Nginx 1.13.x
    MariaDB 5.5
    hmm some serious issues going on there !

    First check if all server disk partitions have enough free disk space
    Code (Text):
    df -hT
    


    getting errors on front end of wordpress installs at all ?

    backup ALL databases on server not just one including mysql system databases using mysqldump and then stop mysql cleanly, backup physical data files /var/lib/mysql and move it and use mysql_install_db MySQL :: MySQL 5.6 Reference Manual :: 4.4.3 mysql_install_db — Initialize MySQL Data Directory to repopulate /var/lib/mysql and restore mysqldump sql files from sql backup

    create backup directory
    Code (Text):
    mkdir -p /home/mysqlbackups
    

    To backup where DATABASENAME is name of your database
    Code (Text):
    DBNAME='DATABASENAME'
    mysqldump --opt --routines $DBNAME> /home/mysqlbackups/$DBNAME_backup.sql


    this command will list all database names on your server
    Code (Text):
    mysql -e "show databases;"


    so change DBNAME variable for each run

    To restore need to create the database name then restore from sql file backup
    Code (Text):
    DBNAME='DATABASENAME'
    mysqladmin create $DBNAME
    mysql < /home/mysqlbackups/$DBNAME_backup.sql
    


    Start with just backing up via mysqldump, do you get any errors when backing up ?
     
    Last edited: Jan 10, 2017
  17. RB1

    RB1 Active Member

    281
    72
    28
    Nov 11, 2016
    California
    Ratings:
    +119
    Local Time:
    8:41 AM
    Nginx 1.13.x
    MariaDB 10.1.x
    No, my Wordpress has no visible errors (only way I found the mysql error was checking the logs).

    Output of "df -hT" looks OK:
    Code (Text):
    Filesystem     Type      Size  Used Avail Use% Mounted on
    /dev/root      ext4       24G  8.0G   15G  35% /
    devtmpfs       devtmpfs  998M     0  998M   0% /dev
    tmpfs          tmpfs     999M     0  999M   0% /dev/shm
    tmpfs          tmpfs     999M   58M  942M   6% /run
    tmpfs          tmpfs     999M     0  999M   0% /sys/fs/cgroup
    tmpfs          tmpfs     200M     0  200M   0% /run/user/993
    tmpfs          tmpfs     200M     0  200M   0% /run/user/0
    tmpfs          tmpfs     200M     0  200M   0% /run/user/991

    The three databases for my websites and mysql database backed up correctly, however:
    Code (Text):
    # mysqldump --opt --routines information_schema> /home/mysqlbackups/information_schema_backup.sql
    mysqldump: Got error: 1044: "Access denied for user 'root'@'localhost' to database 'information_schema'" when using LOCK TABLES
    
    # mysqldump --opt --routines performance_schema> /home/mysqlbackups/performance_schema.sql
    mysqldump: Got error: 1142: "SELECT, LOCK TABLES command denied to user 'root'@'localhost' for table 'accounts'" when using LOCK TABLES

    Shall I continue or do these two need to be addressed? Looks like just a permissions issue, but I don't have access as the root user?
    Maybe I should just rebuild the server if more things keep happening haha :p
     
  18. eva2000

    eva2000 Administrator Staff Member

    31,026
    6,928
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,434
    Local Time:
    2:41 AM
    Nginx 1.13.x
    MariaDB 5.5

    Those can be ignored as they aren't actual databases per se

    So can continue
     
  19. eva2000

    eva2000 Administrator Staff Member

    31,026
    6,928
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,434
    Local Time:
    2:41 AM
    Nginx 1.13.x
    MariaDB 5.5
  20. RB1

    RB1 Active Member

    281
    72
    28
    Nov 11, 2016
    California
    Ratings:
    +119
    Local Time:
    8:41 AM
    Nginx 1.13.x
    MariaDB 10.1.x
    Hey, thanks for adding that second part...I was stuck at the /root/.my.cnf step.
    Now my problem is that after moving it back and running this command I get:
    Code (Text):
    # mysqladmin ver
    
    mysqladmin: connect to server at 'localhost' failed
    error: 'Access denied for user 'root'@'localhost' (using password: YES)'


    Also using mysqladmin_shell.sh I get the error:
    ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

    Good news is that mysql starts up normally and I got no errors importing

    I'm in over my head :confused:
     
    Last edited: Jan 10, 2017
    • Like Like x 1