Get the most out of your Centmin Mod LEMP stack
Become a Member

Linode Vultr Move from vultr to linode caused a huge performance decrease

Discussion in 'Virtual Private Server (VPS) hosting' started by redbird, Mar 24, 2018.

  1. redbird

    redbird Member

    84
    13
    8
    Aug 28, 2015
    Web
    Ratings:
    +22
    Local Time:
    4:24 PM
    For years we had a small saas service (around 5k users) running on vultr 4GB ram VPS.

    We monitor all of our VPSs with NodeQuery.com and nixstats.com

    Here is the screenshot: Screenshot Load was always lower than 40% in most cases around 30%.

    It's a CPU intensive service that automates content sharing to blogs and social networks.


    When we moved our saas to 4 core 8GB VPS at linode (same OS - centos 7, latest centminmod 009) it is incredibly sluggish. Load is always 100% 24/7. Here is the screenshot: Screenshot

    The only difference that old VPS had php 5.6 and this one 7.1 (which was supposed to make things faster)

    I checked the CPUs linode's are 2400mhz vs Vultr's 3600mhz but I don't believe that this is the reason.

    mysql, nginx and php-fpm configs are the same. No sudden spikes of subscribers or anything like that.

    So why in the world, vultr's 30% load on 4core/4gb vps converts into 100% load on 4core/8gb on linode?

    Any ideas at where even start digging?

    So frustrated.
     
  2. ArisC

    ArisC Active Member

    125
    30
    28
    Jun 1, 2017
    Ratings:
    +61
    Local Time:
    9:24 AM
    Nginx Latest
    MariaDB Latest
    Why did you moved? :/

    Was going anything weird with Vultr?
     
  3. eva2000

    eva2000 Administrator Staff Member

    53,277
    12,117
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,655
    Local Time:
    4:24 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    That is the exact reason why you're experiencing slowness - as you had Vultr older fast 3.60ghz cpus which were faster with faster disk and memory. Vultr then later switched to slower Vultr 2.40Ghz and that meant Linode cpus were faster but not faster than Vultr old 3.60Ghz - see benchmarks i did at DigitalOcean - Linode - Vultr - Benchmarks - DigitalOcean vs Vultr vs Linode Benchmark Compare. Vultr web site still shows old benchmarks with 3.40-3.60Ghz cpus while they now have switched to much slower 2.40Ghz cpus. So if you had Vultr 3.4-3.6Ghz cpus you should keep them but if you have Vultr 2.4Ghz, then move to Linode.

    upload_2018-3-24_8-34-21.png

    upload_2018-3-24_8-35-6.png

    upload_2018-3-24_8-35-39.png

    upload_2018-3-24_8-36-25.png
     
  4. eva2000

    eva2000 Administrator Staff Member

    53,277
    12,117
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,655
    Local Time:
    4:24 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    but this part could be due to something else as well besides difference cpus.. you should figure out what's causing 100% cpu load

    CPU load optimal is 1.0 per cpu thread so 8 cpu thread normal = 8x1 = 8.0. 16 cpu server normal is 16x1 = 16.0

    so cpu load of 8 for
    • 2 cpu server = 8/2 = 4.0 load average
    • 4 cpu server = 8/4 = 2.0 load average
    • 8 cpu server = 8/8 = 1.0 load average
    When you type top command, hit number 1 to break down per cpu stats

    Centmin Mod is provided as is, so troubleshooting load issues is left to end user to do. However, there's many linux tools and scripts that can help you figure out what was causing the load issues and when.

    Tools and commands you will want to read up on and learn for basic system admin tasks and troubleshooting.
    Notes:
    For example what's output for these commands
    Code (Text):
    top -bn1
    

    Code (Text):
    ps aufxw
    

    Code (Text):
    sar -q
    

    Code (Text):
    sar -r
    

    Code (Text):
    sar -q -f /var/log/sa/sa$(date +%d -d yesterday)
    

    Code (Text):
    sar -r -f /var/log/sa/sa$(date +%d -d yesterday)
    

    for posting code you might want to use CODE tags for code How to use forum BBCODE code tags :)
     
  5. redbird

    redbird Member

    84
    13
    8
    Aug 28, 2015
    Web
    Ratings:
    +22
    Local Time:
    4:24 PM
    Yeah last couple of weeks we had some weird freezes on vultr. Like all the processes are normal but site not responsive. Then everything goes back to normal. Could be a busy host or network issue. They moved us to a diff. node but it did not help.

    George, I got a reply from linode support that we are on a very busy host. They scheduled to migrate our VPS to much more calm place ))

    And thanks for the info man on troubleshooting resources. I'll dig in.

    I love your posts, always learning so much new.
     
  6. redbird

    redbird Member

    84
    13
    8
    Aug 28, 2015
    Web
    Ratings:
    +22
    Local Time:
    4:24 PM
    A little update: linode migrated us to a not so busy host and so far load is around 1 (25%)

    Also I noticed this:
    Code:
    Architecture:          x86_64
    CPU op-mode(s):        32-bit, 64-bit
    Byte Order:            Little Endian
    CPU(s):                4
    On-line CPU(s) list:   0-3
    Thread(s) per core:    1
    Core(s) per socket:    1
    Socket(s):             4
    NUMA node(s):          1
    Vendor ID:             GenuineIntel
    CPU family:            6
    Model:                 62
    Model name:            Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
    Stepping:              4
    CPU MHz:               2799.994
    BogoMIPS:              5602.32
    Hypervisor vendor:     KVM
    Virtualization type:   full
    L1d cache:             32K
    L1i cache:             32K
    L2 cache:              4096K
    L3 cache:              16384K
    NUMA node0 CPU(s):     0-3
    Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm pti fsgsbase tsc_adjust smep erms xsaveopt arat
    So, it looks like they have a slightly better CPU's than 2400. You just have to ask I guess (or complain :)
     
  7. eva2000

    eva2000 Administrator Staff Member

    53,277
    12,117
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,655
    Local Time:
    4:24 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  8. redbird

    redbird Member

    84
    13
    8
    Aug 28, 2015
    Web
    Ratings:
    +22
    Local Time:
    4:24 PM
    Yeah, that's what they confirmed. They even offered me to pick any one of those we want.

    George, what do you think would be the best pick to keep cpu load to minimum?

    Is frequency the only parameter I should pay attention to or not?

    It looks like they all have a different number of threads per core, cache and bus speed.
     
  9. eva2000

    eva2000 Administrator Staff Member

    53,277
    12,117
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,655
    Local Time:
    4:24 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Depends as each generation v2, v3 etc has improved cryptographic ability for HTTPS sites. And meltdown/spectre and Kernel KPTI patch fixes mean reduced performance for all Intel servers. The older the cpu, the more performance reduction impact these KPTI/Spectre fixes have overall and remember that base frequency doesn't mean much it's turbo boost profiles that matter depending on how many cpu core/threads are in use.

    E5-2697v4 probably would be the best bet these days in light of meltdown/spectre vulnerability fixes etc.

    Xeon E5-2697 v4 - Intel - WikiChip Turbo boost still at 2.80Ghz will all 18 cpu cores in use.
    Code (Text):
    Turbo Frequency    Yes
    3,600 MHz (1 core),
    3,600 MHz (2 cores),
    3,400 MHz (3 cores),
    3,300 MHz (4 cores),
    3,200 MHz (5 cores),
    3,100 MHz (6 cores),
    3,000 MHz (7 cores),
    2,900 MHz (8 cores),
    2,800 MHz (9 cores),
    2,800 MHz (10 cores),
    2,800 MHz (11 cores),
    2,800 MHz (12 cores),
    2,800 MHz (13 cores),
    2,800 MHz (14 cores),
    2,800 MHz (15 cores),
    2,800 MHz (16 cores),
    2,800 MHz (17 cores),
    2,800 MHz (18 cores)
    
     
  10. buik

    buik “The best traveler is one without a camera.”

    1,990
    518
    113
    Apr 29, 2016
    Flanders
    Ratings:
    +1,647
    Local Time:
    8:24 AM
    4 CPU's haha.
    Got some new London VPS.
    With: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz - Launch DateQ3'17
     
  11. eva2000

    eva2000 Administrator Staff Member

    53,277
    12,117
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,655
    Local Time:
    4:24 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  12. buik

    buik “The best traveler is one without a camera.”

    1,990
    518
    113
    Apr 29, 2016
    Flanders
    Ratings:
    +1,647
    Local Time:
    8:24 AM
  13. eva2000

    eva2000 Administrator Staff Member

    53,277
    12,117
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,655
    Local Time:
    4:24 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Sweet here's to Linode rolling them out to all datacenters eventually. Looks like a good upgrade, double number of cores compared to E3-2680v2 10 vs 20 cores and only bump TDP from 115 watts to 150 watts and still managed larger L3 cache 27.5MB vs 25MB and faster DDR4-2666Mhz memory !
     
  14. buik

    buik “The best traveler is one without a camera.”

    1,990
    518
    113
    Apr 29, 2016
    Flanders
    Ratings:
    +1,647
    Local Time:
    8:24 AM
    Think you better could get a VPS servers on a quiet host server with an old CPU than a busy host one with a gold CPU. It is still a shared platform withneighbors.

    As can be viewed by the Redbird case.
     
  15. eva2000

    eva2000 Administrator Staff Member

    53,277
    12,117
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,655
    Local Time:
    4:24 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Yeah then it's easier to ask to be moved to another VPS node. I suppose the newer Xeon Gold based VPS host nodes would potentially have more neighbours sharing the server given a dual Xeon Gold 6148 would have 2x 20C/40T = 40 cores and 80 threads !

    Would love to see Linode offer up a US$60/month 6 cpu core plan eventually :)
     
  16. buik

    buik “The best traveler is one without a camera.”

    1,990
    518
    113
    Apr 29, 2016
    Flanders
    Ratings:
    +1,647
    Local Time:
    8:24 AM
    Host is quite now and VPS server is quick.
    But of course they will fill them and then everything will be slower.
    Not going to happen.
    Cores are expensive and thats directly my biggest problem with VPS servers, besides neighbours.
    Relatively much memory, little cores.
    it is disproportionate but all providers do it.
     
  17. eva2000

    eva2000 Administrator Staff Member

    53,277
    12,117
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,655
    Local Time:
    4:24 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    True though reason this forums are on Linode 8GB plan with 4 cores is mainly the solid redundant disk storage layer. and ability to clone VPS and easily migrate between data centers. Snapshot/data migrations can happen at up to 200MB/s speeds. Building out and replicating the same on a dedicated server involving expensive raid disks and faster network connectivity i.e. 10Gbps NICs all add up cost wise. Folks overlook that fact when comparing dedicated server versus VPS like Linode's offerings.
     
  18. eva2000

    eva2000 Administrator Staff Member

    53,277
    12,117
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,655
    Local Time:
    4:24 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    output for
    Code (Text):
    cat /proc/cpuinfo
    
     
  19. buik

    buik “The best traveler is one without a camera.”

    1,990
    518
    113
    Apr 29, 2016
    Flanders
    Ratings:
    +1,647
    Local Time:
    8:24 AM
    Code:
    processor       : 0
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 85
    model name      : Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
    stepping        : 4
    microcode       : 0x1
    cpu MHz         : 2399.996
    cache size      : 16384 KB
    physical id     : 0
    siblings        : 1
    core id         : 0
    cpu cores       : 4
    apicid          : 0
    initial apicid  : 0
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov                                                                              pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc                                                                              arch_perfmon rep_good nopl xtopology cpuid pni pclmulqdq vmx ssse3 fma cx16 pci                                                                             d sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand                                                                              hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti tpr_shadow                                                                              vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invp                                                                             cid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw a                                                                             vx512vl xsaveopt xsavec xgetbv1 xsaves arat pku ospke
    bugs            : cpu_meltdown spectre_v1 spectre_v2
    bogomips        : 4801.99
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 40 bits physical, 48 bits virtual
     
  20. eva2000

    eva2000 Administrator Staff Member

    53,277
    12,117
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,655
    Local Time:
    4:24 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    nice something funky with flags output.. what about for
    Code (Text):
    lscpu