Join the community today
Register Now

Sysadmin Slow response at login and starting Centmin

Discussion in 'Bug Reports' started by duderuud, Mar 15, 2022.

  1. duderuud

    duderuud Premium Member Premium Member

    249
    82
    28
    Dec 5, 2020
    The Netherlands
    Ratings:
    +178
    Local Time:
    10:04 AM
    1.25 x
    10.6
    Please fill in any relevant information that applies to you:
    • CentOS Version: CentOS 7 7009 X64
    • Centmin Mod Version Installed: 123.09beta01
    • Nginx Version Installed: 1.21.6
    • PHP Version Installed: 8.0.16
    • MariaDB MySQL Version Installed: 10.4.24
    • When was last time updated Centmin Mod code base ? : very recent
    • Persistent Config: Yes
    Moved to a new server and logging in takes a long time (like 15.30 seconds). Looks to be related to the update check:

    Code:
    ==========================================================
     Centmin Mod local code is up to date at /usr/local/src/centminmod
     no available updates at this time...
    ==========================================================
    
    Starting up Centmin takes more than a minute and closing Centmin takes about 20 seconds.
    In the beginning the server responded normally (like my previous VPS) but this just started happening all of a sudden.

    Load is low and the server itself should't be a problem (10 Epic 7702 cores, 64GB mem).

    Any idea how I can troubleshoot this?

    Edit: My old VPS is still up, that has the same problem with logging in. Starting Centmin is a lot quicker though, around 10-15 seconds.

     
    Last edited: Mar 15, 2022
  2. buik

    buik “The best traveler is one without a camera.”

    2,026
    524
    113
    Apr 29, 2016
    Flanders
    Ratings:
    +1,674
    Local Time:
    10:04 AM
    Perhaps a hardware failure, DDOS to the node/neighbor or noisy neighbors.
    As VPS is a shared platform where neighbors or crappy code from those neighbors can exert influence.

    Perhaps you can run a benchmark on both systems at various fixed times?

    In addition, with 10 Epic 7702 cores, 64GB, I would immediately switch to a dedicated server. You never want to be dependent on neighbors with these specs anyway. In addition, the biggest bottleneck with these types of large VPS servers is next to overselling, neighbors, the limitation on I/O.
     
  3. duderuud

    duderuud Premium Member Premium Member

    249
    82
    28
    Dec 5, 2020
    The Netherlands
    Ratings:
    +178
    Local Time:
    10:04 AM
    1.25 x
    10.6
    It's a dedicated vps (guaranteed resources).
     
  4. buik

    buik “The best traveler is one without a camera.”

    2,026
    524
    113
    Apr 29, 2016
    Flanders
    Ratings:
    +1,674
    Local Time:
    10:04 AM
    Otherwise, it seems very likely that these guaranteed resourses cannot be assigned to you. See the examples above.

    You should bench both systems several times at a fixed time. At various times of the day: morning, afternoon, evening, night. If you still experience problems then send a ticket to the provider.
     
  5. eva2000

    eva2000 Administrator Staff Member

    54,529
    12,212
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,781
    Local Time:
    7:04 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Could you provide some of the answers and output to the questions and commands asked at https://community.centminmod.com/threads/troubleshoot-slow-ssh-logins.12124/

    In particular this command which checks SSH login time log's recorded time
    Code (Text):
    find /root/centminlogs/ -type f -name "cmm-login-git-checks_*.log" -exec tail -1 {} \;  | tail -10

    example output
    Code (Text):
    find /root/centminlogs/ -type f -name "cmm-login-git-checks_*.log" -exec tail -1 {} \;  | tail -10
    Total Git & Nginx Check Time: 1.959145877 seconds
    Total Git & Nginx Check Time: 16.832768986 seconds
    Total Git & Nginx Check Time: 1.715314111 seconds
    Total Git & Nginx Check Time: 16.824245948 seconds
    Total Git & Nginx Check Time: 21.068782821 seconds
    Total Git & Nginx Check Time: 2.387668370 seconds
    Total Git & Nginx Check Time: 15.799400361 seconds
    

    Is it only first time login via SSH or everytime login via SSH ?
     
  6. happyhacking

    happyhacking Member

    111
    18
    18
    Apr 23, 2021
    Ratings:
    +61
    Local Time:
    3:04 AM
    1.22.0
    MariadDB 10.4.25
    This same behavior occurred to me today, the login was too slow:
    Code:
    find /root/centminlogs/ -type f -name "cmm-login-git-checks_*.log" -exec tail -1 {} \;  | tail -10
    
    Total Git & Nginx Check Time: 2.797832479 seconds
    Total Git & Nginx Check Time: 161.796007435 seconds
    Total Git & Nginx Check Time: 3.139394959 seconds
    Total Git & Nginx Check Time: 161.843277053 seconds
    Total Git & Nginx Check Time: 161.811475087 seconds
    
    But then i discovered that my server lost IPv4 connectivity while IPv6 was working fine, and the delay appears just before the section where you are notified of updates for nginx or centminmod script. perhaps this check should be limited to some more realistic time period like 10 seconds, since if the update server doesnt answers in that time there is almost sure a conectivity trouble is happening.
     
  7. eva2000

    eva2000 Administrator Staff Member

    54,529
    12,212
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,781
    Local Time:
    7:04 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Interesting heard of IPv6 connectivity issues but rarely for IPv4! I'll look at adding a timeout value to the checks :)
     
  8. happyhacking

    happyhacking Member

    111
    18
    18
    Apr 23, 2021
    Ratings:
    +61
    Local Time:
    3:04 AM
    1.22.0
    MariadDB 10.4.25
    Thank you for adding the timeout of 10 seconds ;)

    BTW: I found a new "bug", if you have IPv4 and IPv6 connectivity but somehow you lost IPv6 connectivity and wants to update centminmod through "cmupdate" it will stall and fail to do it, here is a run of it:

    Code:
    bash -vx /usr/bin/cmupdate
    
    ######################################################
    fupdate
    + fupdate
    + branch_opt=
    + [[ '' = \b\e\t\a ]]
    + [[ '' = \s\t\a\b\l\e ]]
    ++ grep 'HTTP\/'
    ++ curl -sI https://raw.githubusercontent.com/centminmod/centminmod/124.00stable/gitclean.txt
    ++ awk '/200/ {print $2}'
    
    *HERE Hangs UP*
    
    Dont know if the curl command can be instructed to follow IPv6 first then IPv4 or viceversa
    And im wondering if loosing IPv6 temporary while IPv4 is working could have more implications on centmin routines
     
  9. eva2000

    eva2000 Administrator Staff Member

    54,529
    12,212
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,781
    Local Time:
    7:04 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    default version of curl in CentOS 7 system doesn't support such option but newer curl 7.83 does for --happy-eyeballs-timeout-ms
    Code (Text):
    curl --help all 2>&1 | grep -i timeout
         --connect-timeout <fractional seconds> Maximum time allowed for connection
         --expect100-timeout <seconds> How long to wait for 100-continue
         --happy-eyeballs-timeout-ms <milliseconds> Time for IPv6 before trying IPv4
    

    looks like it was added in curl 7.59 curl ootw happy eyeballs timeout | daniel.haxx.se
    This is why Centmin Mod defaults to FORCE_IPVFOUR='y' by default to force curl and wget to use IPv4 first due to flakiness of IPv6 in the past. Unless you set in persistent config FORCE_IPVFOUR='n', then IPv6 and IPv4 can be used where available.
     
    Last edited: Jun 10, 2022
  10. eva2000

    eva2000 Administrator Staff Member

    54,529
    12,212
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,781
    Local Time:
    7:04 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    But I'll update cmupdate command for curl check/timeout too
     
  11. happyhacking

    happyhacking Member

    111
    18
    18
    Apr 23, 2021
    Ratings:
    +61
    Local Time:
    3:04 AM
    1.22.0
    MariadDB 10.4.25
    Thank you so much.
     
  12. happyhacking

    happyhacking Member

    111
    18
    18
    Apr 23, 2021
    Ratings:
    +61
    Local Time:
    3:04 AM
    1.22.0
    MariadDB 10.4.25
    The commit for curl timeout was merged in 130.00beta1 but its missing in 124.00stable does this mean that i should use the newest centminmod beta (even im still on centos 7.9) to get the most recent features or this change will be later merged to 124.00stable ? Thanks in advance eva2000
     
  13. eva2000

    eva2000 Administrator Staff Member

    54,529
    12,212
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,781
    Local Time:
    7:04 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    124.00stable is reserved for any bug fixes/security updates now while 130.00beta01 is for new features - especially extensive ones. cmupdate change probably will end up in 124.00stable eventually