Discover Centmin Mod today
Register Now

After rebooting server Nginx can not start

Discussion in 'System Administration' started by Sunka, Feb 7, 2019.

  1. Sunka

    Sunka Well-Known Member

    1,150
    325
    83
    Oct 31, 2015
    Pula, Croatia
    Ratings:
    +525
    Local Time:
    7:34 AM
    Nginx 1.17.9
    MariaDB 10.3.22
    Nginx can not start after rebooting because *75460 open socket #29 left in connection 39
    @eva2000 could you please take a look and help me.



    This is some info from my cli.

    Code:
    [root@upcloud ~]# service nginx status
    â— nginx.service - SYSV: Nginx is an HTTP(S) server, HTTP(S) reverse proxy and IMAP/POP3 proxy server
       Loaded: loaded (/etc/rc.d/init.d/nginx; bad; vendor preset: disabled)
       Active: failed (Result: exit-code) since Thu 2019-02-07 06:07:21 CET; 58min left
         Docs: man:systemd-sysv-generator(8)
      Process: 3524 ExecStart=/etc/rc.d/init.d/nginx start (code=exited, status=1/FAILURE)
    
    Feb 07 06:07:20 upcloud.pijanitvor.com systemd[1]: Starting SYSV: Nginx is an HTTP(S) server, HTTP(S) rev......
    Feb 07 06:07:21 upcloud.pijanitvor.com nginx[3524]: Starting nginx: /bin/bash: line 1:  3663 Illegal inst...onf
    Feb 07 06:07:21 upcloud.pijanitvor.com nginx[3524]: [FAILED]
    Feb 07 06:07:21 upcloud.pijanitvor.com systemd[1]: nginx.service: control process exited, code=exited status=1
    Feb 07 06:07:21 upcloud.pijanitvor.com systemd[1]: Failed to start SYSV: Nginx is an HTTP(S) server, HTTP...er.
    Feb 07 06:07:21 upcloud.pijanitvor.com systemd[1]: Unit nginx.service entered failed state.
    Feb 07 06:07:21 upcloud.pijanitvor.com systemd[1]: nginx.service failed.
    Hint: Some lines were ellipsized, use -l to show in full.
    You have new mail in /var/spool/mail/root
    [root@upcloud ~]# nginx -t
    nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
    Illegal instruction
    [root@upcloud ~]# service nginx restart
    Restarting nginx (via systemctl):  Job for nginx.service failed because the control process exited with error code. See "systemctl status nginx.service" and "journalctl -xe" for details.
                                                               [FAILED]
    [root@upcloud ~]# systemctl status nginx.service
    â— nginx.service - SYSV: Nginx is an HTTP(S) server, HTTP(S) reverse proxy and IMAP/POP3 proxy server
       Loaded: loaded (/etc/rc.d/init.d/nginx; bad; vendor preset: disabled)
       Active: failed (Result: exit-code) since Thu 2019-02-07 05:10:12 CET; 12s ago
         Docs: man:systemd-sysv-generator(8)
      Process: 4640 ExecStart=/etc/rc.d/init.d/nginx start (code=exited, status=1/FAILURE)
    
    Feb 07 05:10:12 upcloud.pijanitvor.com systemd[1]: Starting SYSV: Nginx is an HTTP(S) server, HTTP(S) rev......
    Feb 07 05:10:12 upcloud.pijanitvor.com nginx[4640]: Starting nginx: /bin/bash: line 1:  4652 Illegal inst...onf
    Feb 07 05:10:12 upcloud.pijanitvor.com nginx[4640]: [FAILED]
    Feb 07 05:10:12 upcloud.pijanitvor.com systemd[1]: nginx.service: control process exited, code=exited status=1
    Feb 07 05:10:12 upcloud.pijanitvor.com systemd[1]: Failed to start SYSV: Nginx is an HTTP(S) server, HTTP...er.
    Feb 07 05:10:12 upcloud.pijanitvor.com systemd[1]: Unit nginx.service entered failed state.
    Feb 07 05:10:12 upcloud.pijanitvor.com systemd[1]: nginx.service failed.
    Hint: Some lines were ellipsized, use -l to show in full.
    [root@upcloud ~]# systemctl status nginx.service -l
    â— nginx.service - SYSV: Nginx is an HTTP(S) server, HTTP(S) reverse proxy and IMAP/POP3 proxy server
       Loaded: loaded (/etc/rc.d/init.d/nginx; bad; vendor preset: disabled)
       Active: failed (Result: exit-code) since Thu 2019-02-07 05:10:12 CET; 27s ago
         Docs: man:systemd-sysv-generator(8)
      Process: 4640 ExecStart=/etc/rc.d/init.d/nginx start (code=exited, status=1/FAILURE)
    
    Feb 07 05:10:12 upcloud.pijanitvor.com systemd[1]: Starting SYSV: Nginx is an HTTP(S) server, HTTP(S) reverse proxy and IMAP/POP3 proxy server...
    Feb 07 05:10:12 upcloud.pijanitvor.com nginx[4640]: Starting nginx: /bin/bash: line 1:  4652 Illegal instruction     /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
    Feb 07 05:10:12 upcloud.pijanitvor.com nginx[4640]: [FAILED]
    Feb 07 05:10:12 upcloud.pijanitvor.com systemd[1]: nginx.service: control process exited, code=exited status=1
    Feb 07 05:10:12 upcloud.pijanitvor.com systemd[1]: Failed to start SYSV: Nginx is an HTTP(S) server, HTTP(S) reverse proxy and IMAP/POP3 proxy server.
    Feb 07 05:10:12 upcloud.pijanitvor.com systemd[1]: Unit nginx.service entered failed state.
    Feb 07 05:10:12 upcloud.pijanitvor.com systemd[1]: nginx.service failed.
    [root@upcloud ~]# service nginx reload
    Reloading nginx configuration (via systemctl):  Job for nginx.service invalid.
                                                               [FAILED]
    [root@upcloud ~]# journalctl -xe
    Feb 07 05:10:08 upcloud.pijanitvor.com kernel: audit: audit_lost=1301214 audit_rate_limit=100 audit_backlog_lim
    Feb 07 05:10:08 upcloud.pijanitvor.com kernel: audit: rate limit exceeded
    Feb 07 05:10:09 upcloud.pijanitvor.com kernel: audit: audit_lost=1485400 audit_rate_limit=100 audit_backlog_lim
    Feb 07 05:10:09 upcloud.pijanitvor.com kernel: audit: rate limit exceeded
    Feb 07 05:10:12 upcloud.pijanitvor.com polkitd[2820]: Registered Authentication Agent for unix-process:4634:174
    Feb 07 05:10:12 upcloud.pijanitvor.com systemd[1]: Starting SYSV: Nginx is an HTTP(S) server, HTTP(S) reverse p
    -- Subject: Unit nginx.service has begun start-up
    -- Defined-By: systemd
    -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
    --
    -- Unit nginx.service has begun starting up.
    Feb 07 05:10:12 upcloud.pijanitvor.com kernel: traps: nginx[4652] trap invalid opcode ip:53fc70 sp:7ffc2af5c500
    Feb 07 05:10:12 upcloud.pijanitvor.com nginx[4640]: Starting nginx: /bin/bash: line 1:  4652 Illegal instructio
    Feb 07 05:10:12 upcloud.pijanitvor.com nginx[4640]: [FAILED]
    Feb 07 05:10:12 upcloud.pijanitvor.com systemd[1]: nginx.service: control process exited, code=exited status=1
    Feb 07 05:10:12 upcloud.pijanitvor.com systemd[1]: Failed to start SYSV: Nginx is an HTTP(S) server, HTTP(S) re
    -- Subject: Unit nginx.service has failed
    -- Defined-By: systemd
    -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
    --
    -- Unit nginx.service has failed.
    --
    -- The result is failed.
    Feb 07 05:10:12 upcloud.pijanitvor.com systemd[1]: Unit nginx.service entered failed state.
    Feb 07 05:10:12 upcloud.pijanitvor.com systemd[1]: nginx.service failed.
    Feb 07 05:10:12 upcloud.pijanitvor.com polkitd[2820]: Unregistered Authentication Agent for unix-process:4634:1
    Feb 07 05:10:55 upcloud.pijanitvor.com polkitd[2820]: Registered Authentication Agent for unix-process:4671:217
    Feb 07 05:10:55 upcloud.pijanitvor.com systemd[1]: Unit nginx.service cannot be reloaded because it is inactive
    Feb 07 05:10:55 upcloud.pijanitvor.com polkitd[2820]:
    
    
    
     tail -10 /usr/local/nginx/logs/error.log
    2019/02/07 04:54:50 [alert] 11042#11042: *75460 open socket #29 left in connection 39
    2019/02/07 04:54:50 [alert] 11042#11042: aborting


     
  2. eva2000

    eva2000 Administrator Staff Member

    58,893
    12,490
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +19,122
    Local Time:
    3:34 PM
    Nginx 1.31.x
    MariaDB 10.x/11.4+/12.3+
    did you move vps nodes to as server with different cpu ? centmin mod 123.09beta01 nginx is compiled to be optimised against the specific cpu model detected unless you're using linode, digitalocean, vultr which automatically disables compiling against specific cpu models via persistent config file variable MARCH_TARGETNATIVE='n' for this very reason/issue you have as once you change to a vps host node which has differing cpu models, you can get illegal instruction errors for nginx and/or php-fpm. Solution right now is for you to recompile nginx and php-fpm via centmin.sh menu option 4 and 5 to get centmin mod to detect the newly changed cpu model.

    So if you are on a vps provider which allows entire VPS to switch to other VPS host node servers like linode, digitalocean or vultr, you can set MARCH_TARGETNATIVE='n' in persistent config file /etc/centminmod/custom_config.inc prior to running centmin.sh menu option 4 and 5

    btw, what's output for
    Code (Text):
    curl -s https://ipinfo.io/org
    

    This output is used to detect if server belongs on linode, digitalocean or vultr network. IIRC upcloud didn't have a good identifier for that output hence i didn't add it to auto disable routine for setting MARCH_TARGETNATIVE='n' in persistent config file /etc/centminmod/custom_config.inc
     
  3. Sunka

    Sunka Well-Known Member

    1,150
    325
    83
    Oct 31, 2015
    Pula, Croatia
    Ratings:
    +525
    Local Time:
    7:34 AM
    Nginx 1.17.9
    MariaDB 10.3.22
  4. Sunka

    Sunka Well-Known Member

    1,150
    325
    83
    Oct 31, 2015
    Pula, Croatia
    Ratings:
    +525
    Local Time:
    7:34 AM
    Nginx 1.17.9
    MariaDB 10.3.22
    So, before recompiling php and ngnix, i set that into custom config?

    I changed server provider 2 years ago. This started just few months ago. Sometimes nginx does not start after server reboot, but usually i reboot again and than it starts.
     
  5. Sunka

    Sunka Well-Known Member

    1,150
    325
    83
    Oct 31, 2015
    Pula, Croatia
    Ratings:
    +525
    Local Time:
    7:34 AM
    Nginx 1.17.9
    MariaDB 10.3.22
    Yeeeeeeee
    Nginx is alive. Thank you Eva!!!!!

    I set MARCH_TARGETNATIVE='n' in persistent config file /etc/centminmod/custom_config.inc and recompile nginx and php.

    So @eva2000
    What to do now?
    Is it safe to reboot server when needed because I have MARCH_TARGETNATIVE='n' set in persistent config file?
     
  6. eva2000

    eva2000 Administrator Staff Member

    58,893
    12,490
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +19,122
    Local Time:
    3:34 PM
    Nginx 1.31.x
    MariaDB 10.x/11.4+/12.3+
    try a reboot and see it

    it should now be fixed if the problem is changed cpu model when you migrated VPS in place i.e. entire OS was copied from one VPS host node to another with same web host which is supported by linode, digitalocean, vultr and upcloud.

    ok will add upcloud to the list of web hosts to auto set MARCH_TARGETNATIVE='n' for