Discover Centmin Mod today
Register Now

Featured Benchmarks Hardware Packet.net bare metal cloud AMD EPYC 7401P review & benchmarks

Discussion in 'Dedicated server hosting' started by eva2000, Feb 26, 2018.

Tags:
Thread Status:
Not open for further replies.
  1. eva2000

    eva2000 Administrator Staff Member

    35,481
    7,828
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,067
    Local Time:
    6:41 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    This thread is dedicated to review and benchmarks for Packet.net - a bare metal cloud provider that offers on demand hourly, reserved, spot and privately deployed bare metal servers. Packet.net offers bare metal servers and recently announced roll out AMD EPYC 7401P 24 core/48 thread servers as well with AMD EPYC 7401P details as well as Intel Xeon Gold 5120 Scalable processors too. If you like this benchmark review and find it useful, please consider a small donation to support Centmin Mod. This time of testing does cost $$$ :)

    Packet.net Bare Metal Server Benchmark Reviews



    Packet.net Plans & Pricing



    • Packet.net Pricing - with bandwidth out prices at US$0.05/GB pay as you go so prices are too high for continuous 24x7x30 day usage for me but they're catering to developers too so if you need high performance bare metal servers for a few hours, then just the perfect fit. Centmin Mod optimisation and testing will be fun now that they have AMD EPYC 7401P cpus on offer :) They have spot pricing, for example their TYPE 1E plan is for Intel Xeon E3-1578Lv5 Skylake based cpu with 32GB DDR4-2400 ECC and 240GB SSD with 10Gbps Intel X710 dual NIC in TLB config runs at US$0.40/hr on demand pricing but I managed to get Los Angeles based TYPE 1E on spot pricing for US$0.08-0.15/hr for testing.
    • Packet.net FAQ
    The plans

    packetnet-plans.png

    And Intel Xeon Gold 5120 Scalable cpus

    packetnet-plans2.png

    The bandwidth pricing is still better than other large cloud providers like Amazon AWS, Google Cloud, Softlayer, Rackspace and Microsoft Azure etc.

    packetnet-bandwidth-price.png
     
    Last edited: Feb 28, 2018
  2. eva2000

    eva2000 Administrator Staff Member

    35,481
    7,828
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,067
    Local Time:
    6:41 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x

    Packet.net TYPE 1E Intel Xeon E3-1578Lv5 Skylake



    I spun up a Packet.net Los Angeles based TYPE 1E Intel Xeon E3-1578Lv5 Skylake bare metal server using spot bidding pricing at US$0.15/hr on current market rate of US$0.08/hr (on demand pricing is US$0.40/hr) for Centmin Mod 123.09beta01 install testing and centminmodbench.sh testing.

    Intel® Xeon® Processor E3-1578L v5 (8M Cache, 2.00 GHz) Product Specifications

    type1e-hardware-ark.png

    Highlights would definitely be 10Gbps network speed i.e. Vultr Los Angeles download speed test managed to pull 442MB/s ! The weakness if you must compare to other providers is disk I/O speed with read/write under 200-230MB/s with 45k/65K random IOPs.

    System
    • 4 Physical Cores @ 2.0 GHz base / 3.40 GHz burst (1 × E3-1578L)
    • Integrated Intel® Iris™ Pro Graphics P580
    • 32GB DDR4-2400 ECC RAM
    • 240 GB of SSD (1 × 240 GB)
    • 10 Gbps Network (2 × Intel X710 NIC's in TLB)
    • CentOS 7.4 64bit
    • Los Angeles
    type1e-hardware-01.png

    Packet.net TYPE 1E Details



    Intel Xeon E3-1578Lv5 Skylake cpu frequencies 2.0Ghz base with 3.4Ghz Turbo for 1 core and 3.2Ghz for all 4 cores.

    Code (Text):
    True Frequency (without accounting Turbo) 1992 MHz
      CPU Multiplier 20x || Bus clock frequency (BCLK) 99.60 MHz
    
    Socket [0] - [physical cores=4, logical cores=8, max online cores ever=4]
      TURBO ENABLED on 4 Cores, Hyper Threading ON
      Max Frequency without considering Turbo 2091.60 MHz (99.60 x [21])
      Max TURBO Multiplier (if Enabled) with 1/2/3/4 Cores is  34x/33x/32x/32x
      Real Current Frequency 3373.64 MHz [99.60 x 33.87] (Max of below)
            Core [core-id]  :Actual Freq (Mult.)      C0%   Halt(C1)%  C3 %   C6 %  Temp      VCore
            Core 1 [0]:       3373.64 (33.87x)         1    96.9       1    1.84    21      1.0911
            Core 2 [1]:       3356.55 (33.70x)         1    97.9       1       1    21      1.0885
            Core 3 [2]:       3356.99 (33.70x)         1    98.8       0    1.05    20      1.1085
            Core 4 [3]:       3354.06 (33.68x)         1    97.9       1       1    20      1.1085
    
    C0 = Processor running without halting
    C1 = Processor running with halts (States >C0 are power saver modes with cores idling)
    C3 = Cores running with PLL turned off and core cache turned off
    C6, C7 = Everything in C3 + core state saved to last level cache, C7 is deeper than C6
      Above values in table are in percentage over the last 1 sec
    

    Code (Text):
    lscpu
    Architecture:          x86_64
    CPU op-mode(s):        32-bit, 64-bit
    Byte Order:            Little Endian
    CPU(s):                8
    On-line CPU(s) list:   0-7
    Thread(s) per core:    2
    Core(s) per socket:    4
    Socket(s):             1
    NUMA node(s):          1
    Vendor ID:             GenuineIntel
    CPU family:            6
    Model:                 94
    Model name:            Intel(R) Xeon(R) CPU E3-1578L v5 @ 2.00GHz
    Stepping:              3
    CPU MHz:               3396.015
    CPU max MHz:           3400.0000
    CPU min MHz:           800.0000
    BogoMIPS:              3984.00
    Virtualization:        VT-x
    L1d cache:             32K
    L1i cache:             32K
    L2 cache:              256K
    L3 cache:              8192K
    NUMA node0 CPU(s):     0-7
    Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
    

    Memory
    Code (Text):
    free -m
                  total        used        free      shared  buff/cache   available
    Mem:          32084        1481       29794          33         808       30166
    Swap:          1949           0        1949
    

    Disk
    Code (Text):
    df -hT
    Filesystem     Type      Size  Used Avail Use% Mounted on
    /dev/sda3      ext4      219G  5.6G  202G   3% /
    devtmpfs       devtmpfs   16G  4.0K   16G   1% /dev
    tmpfs          tmpfs      16G     0   16G   0% /dev/shm
    tmpfs          tmpfs      16G   17M   16G   1% /run
    tmpfs          tmpfs      16G     0   16G   0% /sys/fs/cgroup
    tmpfs          tmpfs     3.2G     0  3.2G   0% /run/user/0
    tmpfs          tmpfs      16G  336K   16G   1% /tmp
    

    Centmin Mod 123.09beta01 Install



    Used default curl installer command below
    Code (Text):
    yum -y update; curl -O https://centminmod.com/betainstaller.sh && chmod 0700 betainstaller.sh && bash betainstaller.sh
    

    Initial Install Time was very decent at ~721 seconds for Intel Xeon E3-1578Lv5 Skylake with 2.0Ghz based and 3.40Ghz Turbo. You can compare that to other times posted at Install - Post your Centmin Mod 123.09beta01 install time stats. Though at as Feb 24th, 2018, the 123.09beta01 installed changed some defaults to more performance at the expense of initial install time.
    Code (Text):
    ---------------------------------------------------------------------------
    Total Curl Installer YUM or DNF Time: 91.2912 seconds
    Total YUM Time: 64.039117358 seconds
    Total YUM or DNF + Source Download Time: 87.3015
    Total Nginx First Time Install Time: 135.2867
    Total PHP First Time Install Time: 141.0516
    Download Zip From Github Time: 1.8695
    Total Time Other eg. source compiles: 264.2750
    Total Centmin Mod Install Time: 627.9147
    ---------------------------------------------------------------------------
    Total Install Time (curl yum + cm install + zip download): 721.0754 seconds
    ---------------------------------------------------------------------------
    

    Post install check list
    Code (Text):
    *************************************************
    * Post-Install Check List....
    *************************************************
    
    --------------------------------------------------------
    Check ccache Version:
    --------------------------------------------------------
    ccache version 3.4.1
    
    Copyright (C) 2002-2007 Andrew Tridgell
    Copyright (C) 2009-2018 Joel Rosdahl
    
    This program is free software; you can redistribute it and/or modify it under
    the terms of the GNU General Public License as published by the Free Software
    Foundation; either version 3 of the License, or (at your option) any later
    version.
    
    --------------------------------------------------------
    Check Nginx Version:
    --------------------------------------------------------
    nginx version: nginx/1.13.9
    built by gcc 7.2.1 20170829 (Red Hat 7.2.1-1) (GCC)
    built with OpenSSL 1.1.0g  2 Nov 2017
    TLS SNI support enabled
    configure arguments: --with-ld-opt='-ljemalloc -Wl,-z,relro -Wl,-rpath,/usr/local/lib' --with-cc-opt='-m64 -march=native -DTCP_FASTOPEN=23 -g -O3 -fstack-protector-strong -flto -fuse-ld=gold --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wimplicit-fallthrough=0 -fcode-hoisting -Wp,-D_FORTIFY_SOURCE=2 -gsplit-dwarf' --sbin-path=/usr/local/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --with-compat --with-http_stub_status_module --with-http_secure_link_module --with-libatomic --with-http_gzip_static_module --with-http_sub_module --with-http_addition_module --with-http_image_filter_module=dynamic --with-http_geoip_module --with-stream_geoip_module --with-stream_realip_module --with-stream_ssl_preread_module --with-threads --with-stream=dynamic --with-stream_ssl_module --with-http_realip_module --add-dynamic-module=../ngx-fancyindex-0.4.2 --add-module=../ngx_cache_purge-2.4.2 --add-module=../ngx_devel_kit-0.3.0 --add-dynamic-module=../set-misc-nginx-module-0.31 --add-dynamic-module=../echo-nginx-module-0.61 --add-module=../redis2-nginx-module-0.14 --add-module=../ngx_http_redis-0.3.7 --add-module=../memc-nginx-module-0.18 --add-module=../srcache-nginx-module-0.31 --add-dynamic-module=../headers-more-nginx-module-0.33 --with-pcre=../pcre-8.41 --with-pcre-jit --with-zlib=../zlib-cloudflare-1.3.0 --with-http_ssl_module --with-http_v2_module --with-openssl=../openssl-1.1.0g --with-openssl-opt='enable-ec_nistp_64_gcc_128'
    
    --------------------------------------------------------
    Check PHP-FPM Version:
    --------------------------------------------------------
    PHP 5.6.33 (cli) (built: Feb 25 2018 19:39:56)
    Copyright (c) 1997-2016 The PHP Group
    Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
        with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2016, by Zend Technologies
    
    --------------------------------------------------------
    Check MariaDB installed RPM Versions:
    --------------------------------------------------------
    MariaDB-devel-10.1.31-1.el7.centos.x86_64
    MariaDB-client-10.1.31-1.el7.centos.x86_64
    MariaDB-server-10.1.31-1.el7.centos.x86_64
    MariaDB-shared-10.1.31-1.el7.centos.x86_64
    MariaDB-common-10.1.31-1.el7.centos.x86_64
    MariaDB-compat-10.1.31-1.el7.centos.x86_64
    
    --------------------------------------------------------
    Check Memcached Server Version:
    --------------------------------------------------------
    memcached 1.5.5
    
    --------------------------------------------------------
    Check CSF Firewall Version:
    --------------------------------------------------------
    csf: v11.07 (generic)
    
    --------------------------------------------------------
    Check Siege Benchmark Version:
    --------------------------------------------------------
    SIEGE 4.0.4
    
    Copyright (C) 2017 by Jeffrey Fulmer, et al.
    This is free software; see the source for copying conditions.
    There is NO warranty; not even for MERCHANTABILITY or FITNESS
    FOR A PARTICULAR PURPOSE.
    
    
    --------------------------------------------------------
    Check mysqlreport version:
    --------------------------------------------------------
    mysqlreport v3.5-maria11 Jul 4 2013
    
    --------------------------------------------------------
    Check pure-ftpd Version:
    --------------------------------------------------------
    pure-ftpd v1.0.42
    
    --------------------------------------------------------
    Check YUM Repo List
    --------------------------------------------------------
    Loaded plugins: fastestmirror, priorities, versionlock
    Loading mirror speeds from cached hostfile
     * base: mirror.fileplanet.com
     * epel: mirrors.develooper.com
     * extras: mirror.fileplanet.com
     * rpmforge: mirror.hmc.edu
     * updates: mirror.fileplanet.com
    228 packages excluded due to repository priority protections
    repo id                        repo name                             status
    base/7/x86_64                  CentOS-7 - Base                       7,420+2,171
    centos-sclo-rh/x86_64          CentOS-7 - SCLo rh                       6,546+94
    centos-sclo-sclo/x86_64        CentOS-7 - SCLo sclo                          500
    epel/x86_64                    Extra Packages for Enterprise Linux 7  11,376+923
    extras/7/x86_64                CentOS-7 - Extras                             388
    mariadb                        MariaDB                                      15+2
    rpmforge                       RHEL 7 - RPMforge.net - dag               139+106
    updates/7/x86_64               CentOS-7 - Updates                      1,499+430
    !varnishcache_varnish41/x86_64 varnishcache_varnish41                         46
    !varnishcache_varnish41-source varnishcache_varnish41-source                   0
    repolist: 27,929
    
    --------------------------------------------------------
    mytimes check:
    /usr/bin/mytimes
    --------------------------------------------------------
    Sun Feb 25 19:42:44 UTC 2018    [UTC]
    Mon Feb 26 05:42:44 AEST 2018   [Australia/Brisbane]
    Sun Feb 25 11:42:44 PST 2018    [America/Los_Angeles]
    Sun Feb 25 13:42:44 CST 2018    [America/Chicago]
    Sun Feb 25 14:42:44 EST 2018    [America/New_York]
    Sun Feb 25 14:42:44 EST 2018    [America/Montreal]
    Sun Feb 25 19:42:44 GMT 2018    [Europe/London]
    Sun Feb 25 20:42:44 CET 2018    [Europe/Berlin]
    Mon Feb 26 02:42:44 +07 2018    [Asia/Bangkok]
    Mon Feb 26 02:42:44 +07 2018    [Asia/Ho_Chi_Minh]
    Mon Feb 26 02:42:44 WIB 2018    [Asia/Jakarta]
    Mon Feb 26 03:42:44 +08 2018    [Asia/Kuala_Lumpur]
    Mon Feb 26 03:42:44 +08 2018    [Asia/Singapore]
    --------------------------------------------------------
    

    Intel Xeon E3-1578Lv5 Skylake cpus
    Code (Text):
    cat /proc/cpuinfo
    processor       : 0
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 94
    model name      : Intel(R) Xeon(R) CPU E3-1578L v5 @ 2.00GHz
    stepping        : 3
    microcode       : 0xba
    cpu MHz         : 3397.187
    cache size      : 8192 KB
    physical id     : 0
    siblings        : 8
    core id         : 0
    cpu cores       : 4
    apicid          : 0
    initial apicid  : 0
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 22
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
    bogomips        : 3984.00
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 39 bits physical, 48 bits virtual
    power management:
    
    processor       : 1
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 94
    model name      : Intel(R) Xeon(R) CPU E3-1578L v5 @ 2.00GHz
    stepping        : 3
    microcode       : 0xba
    cpu MHz         : 3400.468
    cache size      : 8192 KB
    physical id     : 0
    siblings        : 8
    core id         : 1
    cpu cores       : 4
    apicid          : 2
    initial apicid  : 2
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 22
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
    bogomips        : 3984.00
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 39 bits physical, 48 bits virtual
    power management:
    
    processor       : 2
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 94
    model name      : Intel(R) Xeon(R) CPU E3-1578L v5 @ 2.00GHz
    stepping        : 3
    microcode       : 0xba
    cpu MHz         : 3052.343
    cache size      : 8192 KB
    physical id     : 0
    siblings        : 8
    core id         : 2
    cpu cores       : 4
    apicid          : 4
    initial apicid  : 4
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 22
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
    bogomips        : 3984.00
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 39 bits physical, 48 bits virtual
    power management:
    
    processor       : 3
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 94
    model name      : Intel(R) Xeon(R) CPU E3-1578L v5 @ 2.00GHz
    stepping        : 3
    microcode       : 0xba
    cpu MHz         : 3206.718
    cache size      : 8192 KB
    physical id     : 0
    siblings        : 8
    core id         : 3
    cpu cores       : 4
    apicid          : 6
    initial apicid  : 6
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 22
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
    bogomips        : 3984.00
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 39 bits physical, 48 bits virtual
    power management:
    
    processor       : 4
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 94
    model name      : Intel(R) Xeon(R) CPU E3-1578L v5 @ 2.00GHz
    stepping        : 3
    microcode       : 0xba
    cpu MHz         : 3223.671
    cache size      : 8192 KB
    physical id     : 0
    siblings        : 8
    core id         : 0
    cpu cores       : 4
    apicid          : 1
    initial apicid  : 1
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 22
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
    bogomips        : 3984.00
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 39 bits physical, 48 bits virtual
    power management:
    
    processor       : 5
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 94
    model name      : Intel(R) Xeon(R) CPU E3-1578L v5 @ 2.00GHz
    stepping        : 3
    microcode       : 0xba
    cpu MHz         : 3266.484
    cache size      : 8192 KB
    physical id     : 0
    siblings        : 8
    core id         : 1
    cpu cores       : 4
    apicid          : 3
    initial apicid  : 3
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 22
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
    bogomips        : 3984.00
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 39 bits physical, 48 bits virtual
    power management:
    
    processor       : 6
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 94
    model name      : Intel(R) Xeon(R) CPU E3-1578L v5 @ 2.00GHz
    stepping        : 3
    microcode       : 0xba
    cpu MHz         : 3400.234
    cache size      : 8192 KB
    physical id     : 0
    siblings        : 8
    core id         : 2
    cpu cores       : 4
    apicid          : 5
    initial apicid  : 5
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 22
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
    bogomips        : 3984.00
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 39 bits physical, 48 bits virtual
    power management:
    
    processor       : 7
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 94
    model name      : Intel(R) Xeon(R) CPU E3-1578L v5 @ 2.00GHz
    stepping        : 3
    microcode       : 0xba
    cpu MHz         : 3173.593
    cache size      : 8192 KB
    physical id     : 0
    siblings        : 8
    core id         : 3
    cpu cores       : 4
    apicid          : 7
    initial apicid  : 7
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 22
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
    bogomips        : 3984.00
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 39 bits physical, 48 bits virtual
    power management:
    


    centminmodbench.sh results



    All new VPS/Dedicated servers I get, I will run my centminmodbench.sh benchmark script at bench.centminmod.com to fully evaluate the performance of the server. The results are below.

    Highlights would definitely be 10Gbps network speed i.e. Vultr Los Angeles download speed test managed to pull 442MB/s ! The weakness if you must compare to other providers is disk I/O speed with read/write under 200-230MB/s with 45k/65K random IOPs.

    Code (Text):
    -------------------------------------------
    centminmodbench.sh 0.9.5
    http://bench.centminmod.com
    written by: George Liu (eva2000)
    http://centminmod.com
    -------------------------------------------
    
    -------------------------------------------
    System Information
    -------------------------------------------
    
    3.10.0-693.17.1.el7.x86_64
    
    CentOS Linux release 7.4.1708 (Core)
    
    Centmin Mod
    ----------------------------------------------
    Architecture:          x86_64
    CPU op-mode(s):        32-bit, 64-bit
    Byte Order:            Little Endian
    CPU(s):                8
    On-line CPU(s) list:   0-7
    Thread(s) per core:    2
    Core(s) per socket:    4
    Socket(s):             1
    NUMA node(s):          1
    Vendor ID:             GenuineIntel
    CPU family:            6
    Model:                 94
    Model name:            Intel(R) Xeon(R) CPU E3-1578L v5 @ 2.00GHz
    Stepping:              3
    CPU MHz:               3212.265
    CPU max MHz:           3400.0000
    CPU min MHz:           800.0000
    BogoMIPS:              3984.00
    Virtualization:        VT-x
    L1d cache:             32K
    L1i cache:             32K
    L2 cache:              256K
    L3 cache:              8192K
    NUMA node0 CPU(s):     0-7
    Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
    
    CPU Flags
     fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
    
    ----------------------------------------------
    CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE MAXMHZ    MINMHZ
    0   0    0      0    0:0:0:0       yes    3400.0000 800.0000
    1   0    0      1    1:1:1:0       yes    3400.0000 800.0000
    2   0    0      2    2:2:2:0       yes    3400.0000 800.0000
    3   0    0      3    3:3:3:0       yes    3400.0000 800.0000
    4   0    0      0    0:0:0:0       yes    3400.0000 800.0000
    5   0    0      1    1:1:1:0       yes    3400.0000 800.0000
    6   0    0      2    2:2:2:0       yes    3400.0000 800.0000
    7   0    0      3    3:3:3:0       yes    3400.0000 800.0000
    
    ----------------------------------------------
                  total        used        free      shared  buff/cache   available
    Mem:          32084        1421       30201          33         460       30267
    Low:          32084        1882       30201
    High:             0           0           0
    Swap:          1949           0        1949
    
    ----------------------------------------------
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/sda3       219G  5.5G  202G   3% /
    devtmpfs         16G  4.0K   16G   1% /dev
    tmpfs            16G     0   16G   0% /dev/shm
    tmpfs            16G   17M   16G   1% /run
    tmpfs            16G     0   16G   0% /sys/fs/cgroup
    tmpfs           3.2G     0  3.2G   0% /run/user/0
    tmpfs            16G  332K   16G   1% /tmp
    
    

    Code (Text):
    -------------------------------------------
    disk ioping tests
    -------------------------------------------
    
    Running IOPing I/O benchmark...
    
    IOPing I/O: ./ioping -c 10 .
    4.0 KiB from . (ext4 /dev/sda3): request=1 time=196 us
    4.0 KiB from . (ext4 /dev/sda3): request=2 time=216 us
    4.0 KiB from . (ext4 /dev/sda3): request=3 time=226 us
    4.0 KiB from . (ext4 /dev/sda3): request=4 time=207 us
    4.0 KiB from . (ext4 /dev/sda3): request=5 time=210 us
    4.0 KiB from . (ext4 /dev/sda3): request=6 time=226 us
    4.0 KiB from . (ext4 /dev/sda3): request=7 time=233 us
    4.0 KiB from . (ext4 /dev/sda3): request=8 time=199 us
    4.0 KiB from . (ext4 /dev/sda3): request=9 time=246 us
    4.0 KiB from . (ext4 /dev/sda3): request=10 time=212 us
    
    --- . (ext4 /dev/sda3) ioping statistics ---
    10 requests completed in 9.0 s, 4.6 k iops, 18.0 MiB/s
    min/avg/max/mdev = 196 us / 217 us / 246 us / 14 us
    
    IOPing seek rate: ./ioping -RD .
    
    --- . (ext4 /dev/sda3) ioping statistics ---
    17.6 k requests completed in 3.0 s, 5.9 k iops, 23.0 MiB/s
    min/avg/max/mdev = 56 us / 169 us / 2.1 ms / 38 us
    
    IOPing sequential: ./ioping -RL .
    
    --- . (ext4 /dev/sda3) ioping statistics ---
    5.0 k requests completed in 3.0 s, 1.7 k iops, 425.5 MiB/s
    min/avg/max/mdev = 538 us / 587 us / 6.5 ms / 105 us
    
    IOPing cached: ./ioping -RC .
    
    --- . (ext4 /dev/sda3) ioping statistics ---
    4.0 M requests completed in 3.0 s, 1.5 M iops, 5.5 GiB/s
    min/avg/max/mdev = 0 us / 0 us / 25 us / 0 us
    
    

    Code (Text):
    -------------------------------------------
    disk DD tests
    -------------------------------------------
    
    dd if=/dev/zero of=sb-io-test bs=1M count=1k conv=fdatasync
    1024+0 records in
    1024+0 records out
    1073741824 bytes (1.1 GB) copied, 4.48637 s, 239 MB/s
    
    dd if=/dev/zero of=sb-io-test bs=64k count=16k conv=fdatasync
    16384+0 records in
    16384+0 records out
    1073741824 bytes (1.1 GB) copied, 4.5041 s, 238 MB/s
    
    dd if=/dev/zero of=sb-io-test bs=1M count=1k oflag=dsync
    1024+0 records in
    1024+0 records out
    1073741824 bytes (1.1 GB) copied, 4.15684 s, 258 MB/s
    
    dd if=/dev/zero of=sb-io-test bs=64k count=16k oflag=dsync
    16384+0 records in
    16384+0 records out
    1073741824 bytes (1.1 GB) copied, 6.3402 s, 169 MB/s
    

    Code (Text):
    -------------------------------------------
    disk FIO tests
    -------------------------------------------
    
    Running FIO benchmark...
    
    FIO_VERSION = fio-2.0.9
    
    FIO random reads:
    randomreads: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
    fio-2.0.9
    Starting 1 process
    randomreads: Laying out IO file(s) (1 file(s) / 1024MB)
    
    randomreads: (groupid=0, jobs=1): err= 0: pid=7628: Sun Feb 25 19:47:53 2018
      read : io=1024.3MB, bw=183522KB/s, iops=45880 , runt=  5715msec
      cpu          : usr=5.48%, sys=20.46%, ctx=245176, majf=0, minf=83
      IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
         submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
         complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
         issued    : total=r=262207/w=0/d=0, short=r=0/w=0/d=0
    
    Run status group 0 (all jobs):
       READ: io=1024.3MB, aggrb=183521KB/s, minb=183521KB/s, maxb=183521KB/s, mint=5715msec, maxt=5715msec
    
    Disk stats (read/write):
      sda: ios=259116/2, merge=66/1, ticks=359697/24, in_queue=360025, util=98.31%
    
    FIO random writes:
    randomwrites: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
    fio-2.0.9
    Starting 1 process
    
    randomwrites: (groupid=0, jobs=1): err= 0: pid=7632: Sun Feb 25 19:47:58 2018
      write: io=1024.3MB, bw=260126KB/s, iops=65031 , runt=  4032msec
      cpu          : usr=6.52%, sys=29.87%, ctx=254620, majf=0, minf=18
      IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
         submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
         complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
         issued    : total=r=0/w=262207/d=0, short=r=0/w=0/d=0
    
    Run status group 0 (all jobs):
      WRITE: io=1024.3MB, aggrb=260125KB/s, minb=260125KB/s, maxb=260125KB/s, mint=4032msec, maxt=4032msec
    
    Disk stats (read/write):
      sda: ios=0/253692, merge=0/66, ticks=0/248203, in_queue=248405, util=97.55%
    
    

    Code (Text):
    -------------------------------------------
    ramdisk DD tests - memory bandwidth
    -------------------------------------------
    
    dd if=/dev/zero of=sb-io-test bs=128k count=1k conv=fdatasync
    1024+0 records in
    1024+0 records out
    134217728 bytes (134 MB) copied, 0.0338726 s, 4.0 GB/s
    
    dd if=/dev/zero of=sb-io-test bs=8k count=16k conv=fdatasync
    16384+0 records in
    16384+0 records out
    134217728 bytes (134 MB) copied, 0.0412279 s, 3.3 GB/s
    
    dd if=/dev/zero of=sb-io-test bs=128k count=1k oflag=dsync
    1024+0 records in
    1024+0 records out
    134217728 bytes (134 MB) copied, 0.0333627 s, 4.0 GB/s
    
    dd if=/dev/zero of=sb-io-test bs=8k count=16k oflag=dsync
    16384+0 records in
    16384+0 records out
    134217728 bytes (134 MB) copied, 0.0406788 s, 3.3 GB/s
    

    Code (Text):
    -------------------------------------------
    ramdisk ioping tests - memory bandwidth
    -------------------------------------------
    
    Running IOPing I/O ramdisk benchmark...
    
    IOPing I/O: ./ioping -c 10 .
    4.0 KiB from . (tmpfs tmpfs): request=1 time=17 us
    4.0 KiB from . (tmpfs tmpfs): request=2 time=20 us
    4.0 KiB from . (tmpfs tmpfs): request=3 time=3 us
    4.0 KiB from . (tmpfs tmpfs): request=4 time=4 us
    4.0 KiB from . (tmpfs tmpfs): request=5 time=4 us
    4.0 KiB from . (tmpfs tmpfs): request=6 time=4 us
    4.0 KiB from . (tmpfs tmpfs): request=7 time=4 us
    4.0 KiB from . (tmpfs tmpfs): request=8 time=4 us
    4.0 KiB from . (tmpfs tmpfs): request=9 time=3 us
    4.0 KiB from . (tmpfs tmpfs): request=10 time=4 us
    
    --- . (tmpfs tmpfs) ioping statistics ---
    10 requests completed in 9.0 s, 149.3 k iops, 583.0 MiB/s
    min/avg/max/mdev = 3 us / 6 us / 20 us / 5 us
    
    IOPing seek rate: ./ioping -R .
    
    --- . (tmpfs tmpfs) ioping statistics ---
    1.8 M requests completed in 3.0 s, 1.4 M iops, 5.5 GiB/s
    min/avg/max/mdev = 0 us / 0 us / 22 us / 0 us
    
    IOPing sequential: ./ioping -RL .
    
    --- . (tmpfs tmpfs) ioping statistics ---
    73.8 k requests completed in 3.0 s, 40.6 k iops, 9.9 GiB/s
    min/avg/max/mdev = 20 us / 24 us / 83 us / 2 us
    
    IOPing cached: ./ioping -RC .
    
    --- . (tmpfs tmpfs) ioping statistics ---
    3.8 M requests completed in 3.0 s, 1.4 M iops, 5.2 GiB/s
    min/avg/max/mdev = 0 us / 0 us / 60 us / 0 us
    
    

    Code (Text):
    -------------------------------------------
    Running bandwidth benchmark...
    -------------------------------------------
    
    ----------------------------------------------
    Download from Cachefly (http://cachefly.cachefly.net/100mb.test)
    Download Cachefly: 99.8MB/s
    
    -------------------------------------------
    USA bandwidth tests...
    -------------------------------------------
    
    ----------------------------------------------
    Download from Linode, Atlanta, GA, USA (http://speedtest.atlanta.linode.com/100MB-atlanta.bin)
    Download Linode, Atlanta, GA, USA: 43.5MB/s
    ----------------------------------------------
    Download from Linode, Dallas, TX, USA (http://speedtest.dallas.linode.com/100MB-dallas.bin)
    Download Linode, Dallas, TX, USA: 69.1MB/s
    ----------------------------------------------
    Download from Leaseweb, Manassas, VA, USA (http://mirror.us.leaseweb.net/speedtest/100mb.bin)
    Download Leaseweb, Manassas, VA, USA: 49.2MB/s
    ----------------------------------------------
    Download from Softlayer, Seattle, WA, USA (http://speedtest.sea01.softlayer.com/downloads/test100.zip)
    Download Softlayer, Seattle, WA, USA: 58.9MB/s
    ----------------------------------------------
    Download from Softlayer, San Jose, CA, USA (http://speedtest.sjc01.softlayer.com/downloads/test100.zip)
    Download Softlayer, San Jose, CA, USA: 91.6MB/s
    ----------------------------------------------
    Download from Softlayer, Washington, DC, USA (http://speedtest.wdc01.softlayer.com/downloads/test100.zip)
    Download Softlayer, Washington, DC, USA: 15.7MB/s
    ----------------------------------------------
    Download from VersaWeb, Las Vegas, Nevada (http://104.143.15.254/100MB.test)
    Download VersaWeb, Las Vegas, Nevada: 45.4MB/s
    ----------------------------------------------
    Download from OVH, BHS, Canada (http://bhs.proof.ovh.net/files/100Mio.dat)
    Download OVH, BHS, Canada: 14.2MB/s
    ----------------------------------------------
    Download from Vultr, Los Angeles, California (http://lax-ca-us-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, Los Angeles, California: 442MB/s
    ----------------------------------------------
    Download from Vultr, Seattle, Washington (http://wa-us-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, Seattle, Washington: 66.3MB/s
    ----------------------------------------------
    Download from Vultr, Dallas, Texas (http://tx-us-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, Dallas, Texas: 85.0MB/s
    ----------------------------------------------
    Download from Vultr, Chicago, Illinois (http://il-us-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, Chicago, Illinois: 38.5MB/s
    ----------------------------------------------
    Download from Vultr, Atlanta, Georgia (http://ga-us-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, Atlanta, Georgia: 46.7MB/s
    ----------------------------------------------
    Download from Vultr, Miami, Florida (http://fl-us-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, Miami, Florida: 37.6MB/s
    ----------------------------------------------
    Download from Vultr, New York / New Jersey (http://nj-us-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, New York / New Jersey: 27.0MB/s
    
    -------------------------------------------
    Asia bandwidth tests...
    -------------------------------------------
    
    ----------------------------------------------
    Download from Linode, Tokyo, JP (http://speedtest.tokyo.linode.com/100MB-tokyo.bin)
    Download Linode, Tokyo, JP: 20.4MB/s
    ----------------------------------------------
    Download from Softlayer, Singapore (http://speedtest.sng01.softlayer.com/downloads/test100.zip)
    Download Softlayer, Singapore: 10.2MB/s
    ----------------------------------------------
    Download from Vultr, Tokyo, Japan (http://hnd-jp-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, Tokyo, Japan: 20.6MB/s
    
    -------------------------------------------
    Europe bandwidth tests...
    -------------------------------------------
    
    ----------------------------------------------
    Download from Linode, London, UK (http://speedtest.london.linode.com/100MB-london.bin)
    Download Linode, London, UK: 15.5MB/s
    ----------------------------------------------
    Download from OVH, Paris, France (http://proof.ovh.net/files/100Mio.dat)
    Download OVH, Paris, France: 13.4MB/s
    ----------------------------------------------
    Download from SmartDC, Rotterdam, Netherlands (http://mirror.i3d.net/100mb.bin)
    Download SmartDC, Rotterdam, Netherlands: 8.93MB/s
    ----------------------------------------------
    Download from Vultr, Amsterdam, Netherlands (http://ams-nl-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, Amsterdam, Netherlands: 13.0MB/s
    ----------------------------------------------
    Download from Vultr, London, UK (http://lon-gb-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, London, UK: 15.6MB/s
    ----------------------------------------------
    Download from Vultr, Paris, France (http://par-fr-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, Paris, France: 15.5MB/s
    
    -------------------------------------------
    Australia bandwidth tests...
    -------------------------------------------
    
    ----------------------------------------------
    Download from Vultr, Sydney, Australia (http://syd-au-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, Sydney, Australia: 14.0MB/s
    

    Code (Text):
    -------------------------------------------
    Running Axel multi-threaded bandwidth benchmark...
    -------------------------------------------
    
    ----------------------------------------------
    Axel Download from Cachefly (http://cachefly.cachefly.net/100mb.test)
    Axel Download Cachefly: 184.47MB/s
    
    -------------------------------------------
    USA bandwidth tests...
    -------------------------------------------
    
    ----------------------------------------------
    Axel Download from Linode, Atlanta, GA, USA (http://speedtest.atlanta.linode.com/100MB-atlanta.bin)
    Axel Download Linode, Atlanta, GA, USA: 55.35MB/s
    ----------------------------------------------
    Axel Download from Linode, Dallas, TX, USA (http://speedtest.dallas.linode.com/100MB-dallas.bin)
    Axel Download Linode, Dallas, TX, USA: 105.50MB/s
    ----------------------------------------------
    Axel Download from Leaseweb, Manassas, VA, USA (http://mirror.us.leaseweb.net/speedtest/100mb.bin)
    Axel Download Leaseweb, Manassas, VA, USA: 39.98MB/s
    ----------------------------------------------
    Axel Download from Softlayer, Seattle, WA, USA (http://speedtest.sea01.softlayer.com/downloads/test100.zip)
    Axel Download Softlayer, Seattle, WA, USA: 49.00MB/s
    ----------------------------------------------
    Axel Download from Softlayer, San Jose, CA, USA (http://speedtest.sjc01.softlayer.com/downloads/test100.zip)
    Axel Download Softlayer, San Jose, CA, USA: 78.00MB/s
    ----------------------------------------------
    Axel Download from Softlayer, Washington, DC, USA (http://speedtest.wdc01.softlayer.com/downloads/test100.zip)
    Axel Download Softlayer, Washington, DC, USA: 18.43MB/s
    ----------------------------------------------
    Axel Download from VersaWeb, Las Vegas, Nevada (http://104.143.15.254/100MB.test)
    Axel Download VersaWeb, Las Vegas, Nevada: 89.27MB/s
    ----------------------------------------------
    Axel Download from OVH, BHS, Canada (http://bhs.proof.ovh.net/files/100Mio.dat)
    Axel Download OVH, BHS, Canada: 5.51MB/s
    ----------------------------------------------
    Axel Download from Vultr, Los Angeles, California (http://lax-ca-us-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, Los Angeles, California: 243.04MB/s
    ----------------------------------------------
    Axel Download from Vultr, Seattle, Washington (http://wa-us-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, Seattle, Washington: 82.23MB/s
    ----------------------------------------------
    Axel Download from Vultr, Dallas, Texas (http://tx-us-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, Dallas, Texas: 148.27MB/s
    ----------------------------------------------
    Axel Download from Vultr, Chicago, Illinois (http://il-us-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, Chicago, Illinois: 74.14MB/s
    ----------------------------------------------
    Axel Download from Vultr, Atlanta, Georgia (http://ga-us-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, Atlanta, Georgia: 85.74MB/s
    ----------------------------------------------
    Axel Download from Vultr, Miami, Florida (http://fl-us-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, Miami, Florida: 66.54MB/s
    ----------------------------------------------
    Axel Download from Vultr, New York / New Jersey (http://nj-us-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, New York / New Jersey: 50.70MB/s
    
    -------------------------------------------
    Asia bandwidth tests...
    -------------------------------------------
    
    ----------------------------------------------
    Axel Download from Linode, Tokyo, JP (http://speedtest.tokyo.linode.com/100MB-tokyo.bin)
    Axel Download Linode, Tokyo, JP: 12.35MB/s
    ----------------------------------------------
    Axel Download from Softlayer, Singapore (http://speedtest.sng01.softlayer.com/downloads/test100.zip)
    Axel Download Softlayer, Singapore: 7.78MB/s
    ----------------------------------------------
    Axel Download from Vultr, Tokyo, Japan (http://hnd-jp-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, Tokyo, Japan: 31.69MB/s
    
    -------------------------------------------
    Europe bandwidth tests...
    -------------------------------------------
    
    ----------------------------------------------
    Axel Download from Linode, London, UK (http://speedtest.london.linode.com/100MB-london.bin)
    Axel Download Linode, London, UK: 25.59MB/s
    ----------------------------------------------
    Axel Download from OVH, Paris, France (http://proof.ovh.net/files/100Mio.dat)
    Axel Download OVH, Paris, France: 3.83MB/s
    ----------------------------------------------
    Axel Download from SmartDC, Rotterdam, Netherlands (http://mirror.i3d.net/100mb.bin)
    Axel Download SmartDC, Rotterdam, Netherlands: 3.49MB/s
    ----------------------------------------------
    Axel Download from Vultr, Amsterdam, Netherlands (http://ams-nl-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, Amsterdam, Netherlands: 15.83MB/s
    ----------------------------------------------
    Axel Download from Vultr, London, UK (http://lon-gb-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, London, UK: 23.96MB/s
    ----------------------------------------------
    Axel Download from Vultr, Paris, France (http://par-fr-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, Paris, France: 21.42MB/s
    
    -------------------------------------------
    Australia bandwidth tests...
    -------------------------------------------
    
    ----------------------------------------------
    Axel Download from Vultr, Sydney, Australia (http://syd-au-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, Sydney, Australia: 20.79MB/s
    

    Code (Text):
    -------------------------------------------
    Running ping tests...
    -------------------------------------------
    
    ----------------------------------------------
    Pings (cachefly.cachefly.net):
    PING vip1.g5.cachefly.net (205.234.175.175) 56(84) bytes of data.
    64 bytes from vip1.G-anycast1.cachefly.net (205.234.175.175): icmp_seq=1 ttl=61 time=0.296 ms
    64 bytes from vip1.G-anycast1.cachefly.net (205.234.175.175): icmp_seq=2 ttl=61 time=0.282 ms
    64 bytes from vip1.G-anycast1.cachefly.net (205.234.175.175): icmp_seq=3 ttl=61 time=0.203 ms
    
    --- vip1.g5.cachefly.net ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2001ms
    rtt min/avg/max/mdev = 0.203/0.260/0.296/0.043 ms
    
    ----------------------------------------------
    Pings (syd-au-ping.vultr.com):
    PING syd-au-ping.vultr.com (108.61.212.117) 56(84) bytes of data.
    64 bytes from 108.61.212.117.vultr.com (108.61.212.117): icmp_seq=1 ttl=52 time=151 ms
    64 bytes from 108.61.212.117.vultr.com (108.61.212.117): icmp_seq=2 ttl=52 time=150 ms
    64 bytes from 108.61.212.117.vultr.com (108.61.212.117): icmp_seq=3 ttl=52 time=150 ms
    
    --- syd-au-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2003ms
    rtt min/avg/max/mdev = 150.780/150.927/151.192/0.486 ms
    
    ----------------------------------------------
    Pings (hnd-jp-ping.vultr.com):
    PING hnd-jp-ping.vultr.com (108.61.201.151) 56(84) bytes of data.
    64 bytes from 108.61.201.151.vultr.com (108.61.201.151): icmp_seq=1 ttl=53 time=107 ms
    64 bytes from 108.61.201.151.vultr.com (108.61.201.151): icmp_seq=2 ttl=53 time=107 ms
    64 bytes from 108.61.201.151.vultr.com (108.61.201.151): icmp_seq=3 ttl=53 time=107 ms
    
    --- hnd-jp-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2001ms
    rtt min/avg/max/mdev = 107.782/107.843/107.943/0.071 ms
    
    ----------------------------------------------
    Pings (lax-ca-us-ping.vultr.com):
    PING lax-ca-us-ping.vultr.com (108.61.219.200) 56(84) bytes of data.
    64 bytes from 108.61.219.200.vultr.com (108.61.219.200): icmp_seq=1 ttl=57 time=0.536 ms
    64 bytes from 108.61.219.200.vultr.com (108.61.219.200): icmp_seq=2 ttl=57 time=0.569 ms
    64 bytes from 108.61.219.200.vultr.com (108.61.219.200): icmp_seq=3 ttl=57 time=0.463 ms
    
    --- lax-ca-us-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2001ms
    rtt min/avg/max/mdev = 0.463/0.522/0.569/0.051 ms
    
    ----------------------------------------------
    Pings (wa-us-ping.vultr.com):
    PING wa-us-ping.vultr.com (108.61.194.105) 56(84) bytes of data.
    64 bytes from 108.61.194.105.vultr.com (108.61.194.105): icmp_seq=1 ttl=53 time=33.8 ms
    64 bytes from 108.61.194.105.vultr.com (108.61.194.105): icmp_seq=2 ttl=53 time=33.6 ms
    64 bytes from 108.61.194.105.vultr.com (108.61.194.105): icmp_seq=3 ttl=53 time=33.7 ms
    
    --- wa-us-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 33.653/33.744/33.809/0.066 ms
    
    ----------------------------------------------
    Pings (tx-us-ping.vultr.com):
    PING tx-us-ping.vultr.com (108.61.224.175) 56(84) bytes of data.
    64 bytes from 108.61.224.175.vultr.com (108.61.224.175): icmp_seq=1 ttl=54 time=27.5 ms
    64 bytes from 108.61.224.175.vultr.com (108.61.224.175): icmp_seq=2 ttl=54 time=27.6 ms
    64 bytes from 108.61.224.175.vultr.com (108.61.224.175): icmp_seq=3 ttl=54 time=27.8 ms
    
    --- tx-us-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 27.597/27.679/27.817/0.098 ms
    
    ----------------------------------------------
    Pings (il-us-ping.vultr.com):
    PING il-us-ping.vultr.com (107.191.51.12) 56(84) bytes of data.
    64 bytes from 107.191.51.12.vultr.com (107.191.51.12): icmp_seq=1 ttl=56 time=59.3 ms
    64 bytes from 107.191.51.12.vultr.com (107.191.51.12): icmp_seq=2 ttl=56 time=59.3 ms
    64 bytes from 107.191.51.12.vultr.com (107.191.51.12): icmp_seq=3 ttl=56 time=59.3 ms
    
    --- il-us-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2001ms
    rtt min/avg/max/mdev = 59.307/59.321/59.344/0.199 ms
    
    ----------------------------------------------
    Pings (ga-us-ping.vultr.com):
    PING ga-us-ping.vultr.com (108.61.193.166) 56(84) bytes of data.
    64 bytes from 108.61.193.166.vultr.com (108.61.193.166): icmp_seq=1 ttl=51 time=50.1 ms
    64 bytes from 108.61.193.166.vultr.com (108.61.193.166): icmp_seq=2 ttl=51 time=50.2 ms
    64 bytes from 108.61.193.166.vultr.com (108.61.193.166): icmp_seq=3 ttl=51 time=50.2 ms
    
    --- ga-us-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 50.183/50.216/50.246/0.260 ms
    
    ----------------------------------------------
    Pings (fl-us-ping.vultr.com):
    PING fl-us-ping.vultr.com (104.156.244.232) 56(84) bytes of data.
    64 bytes from 104.156.244.232 (104.156.244.232): icmp_seq=1 ttl=52 time=62.5 ms
    64 bytes from 104.156.244.232 (104.156.244.232): icmp_seq=2 ttl=52 time=62.4 ms
    64 bytes from 104.156.244.232 (104.156.244.232): icmp_seq=3 ttl=52 time=62.5 ms
    
    --- fl-us-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2001ms
    rtt min/avg/max/mdev = 62.487/62.541/62.588/0.291 ms
    
    ----------------------------------------------
    Pings (nj-us-ping.vultr.com):
    PING nj-us-ping.vultr.com (108.61.149.182) 56(84) bytes of data.
    64 bytes from 108.61.149.182.vultr.com (108.61.149.182): icmp_seq=1 ttl=52 time=84.5 ms
    64 bytes from 108.61.149.182.vultr.com (108.61.149.182): icmp_seq=2 ttl=52 time=84.5 ms
    64 bytes from 108.61.149.182.vultr.com (108.61.149.182): icmp_seq=3 ttl=52 time=84.7 ms
    
    --- nj-us-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2001ms
    rtt min/avg/max/mdev = 84.539/84.612/84.713/0.073 ms
    
    ----------------------------------------------
    Pings (fra-de-ping.vultr.com):
    PING fra-de-ping.vultr.com (108.61.210.117) 56(84) bytes of data.
    64 bytes from 108.61.210.117.vultr.com (108.61.210.117): icmp_seq=1 ttl=55 time=163 ms
    64 bytes from 108.61.210.117.vultr.com (108.61.210.117): icmp_seq=2 ttl=55 time=163 ms
    64 bytes from 108.61.210.117.vultr.com (108.61.210.117): icmp_seq=3 ttl=55 time=163 ms
    
    --- fra-de-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2003ms
    rtt min/avg/max/mdev = 163.749/163.817/163.895/0.335 ms
    
    ----------------------------------------------
    Pings (ams-nl-ping.vultr.com):
    PING ams-nl-ping.vultr.com (108.61.198.102) 56(84) bytes of data.
    64 bytes from 108.61.198.102.vultr.com (108.61.198.102): icmp_seq=1 ttl=51 time=169 ms
    64 bytes from 108.61.198.102.vultr.com (108.61.198.102): icmp_seq=2 ttl=51 time=165 ms
    64 bytes from 108.61.198.102.vultr.com (108.61.198.102): icmp_seq=3 ttl=51 time=170 ms
    
    --- ams-nl-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 165.045/168.531/170.664/2.485 ms
    
    ----------------------------------------------
    Pings (lon-gb-ping.vultr.com):
    PING lon-gb-ping.vultr.com (108.61.196.101) 56(84) bytes of data.
    64 bytes from 108.61.196.101.vultr.com (108.61.196.101): icmp_seq=1 ttl=51 time=133 ms
    64 bytes from 108.61.196.101.vultr.com (108.61.196.101): icmp_seq=2 ttl=51 time=133 ms
    64 bytes from 108.61.196.101.vultr.com (108.61.196.101): icmp_seq=3 ttl=51 time=133 ms
    
    --- lon-gb-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2003ms
    rtt min/avg/max/mdev = 133.801/133.813/133.831/0.013 ms
    
    ----------------------------------------------
    Pings (par-fr-ping.vultr.com):
    PING par-fr-ping.vultr.com (108.61.209.127) 56(84) bytes of data.
    64 bytes from 108.61.209.127.vultr.com (108.61.209.127): icmp_seq=1 ttl=51 time=163 ms
    64 bytes from 108.61.209.127.vultr.com (108.61.209.127): icmp_seq=2 ttl=51 time=163 ms
    64 bytes from 108.61.209.127.vultr.com (108.61.209.127): icmp_seq=3 ttl=51 time=163 ms
    
    --- par-fr-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2003ms
    rtt min/avg/max/mdev = 163.630/163.670/163.708/0.031 ms
    
    ----------------------------------------------
    Pings (VersaWeb Las Vegas):
    PING 104.143.15.254 (104.143.15.254) 56(84) bytes of data.
    64 bytes from 104.143.15.254: icmp_seq=1 ttl=54 time=7.97 ms
    64 bytes from 104.143.15.254: icmp_seq=2 ttl=54 time=7.93 ms
    64 bytes from 104.143.15.254: icmp_seq=3 ttl=54 time=7.93 ms
    
    --- 104.143.15.254 ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 7.938/7.951/7.977/0.018 ms
    
    ----------------------------------------------
    Pings (VersaWeb Seattle):
    PING 76.164.234.1 (76.164.234.1) 56(84) bytes of data.
    64 bytes from 76.164.234.1: icmp_seq=1 ttl=57 time=38.2 ms
    64 bytes from 76.164.234.1: icmp_seq=2 ttl=57 time=38.2 ms
    64 bytes from 76.164.234.1: icmp_seq=3 ttl=57 time=38.1 ms
    
    --- 76.164.234.1 ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 38.156/38.212/38.276/0.167 ms
    
    ----------------------------------------------
    Pings (OVH Canada):
    PING bhs.proof.ovh.net (192.99.19.165) 56(84) bytes of data.
    64 bytes from bhs.proof.ovh.net (192.99.19.165): icmp_seq=1 ttl=54 time=74.6 ms
    64 bytes from bhs.proof.ovh.net (192.99.19.165): icmp_seq=2 ttl=54 time=74.6 ms
    64 bytes from bhs.proof.ovh.net (192.99.19.165): icmp_seq=3 ttl=54 time=74.6 ms
    
    --- bhs.proof.ovh.net ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 74.624/74.638/74.656/0.013 ms
    
    ----------------------------------------------
    Pings (10gbps.io Frankfurt):
    PING 195.181.174.20 (195.181.174.20) 56(84) bytes of data.
    64 bytes from 195.181.174.20: icmp_seq=1 ttl=56 time=181 ms
    64 bytes from 195.181.174.20: icmp_seq=2 ttl=56 time=180 ms
    64 bytes from 195.181.174.20: icmp_seq=3 ttl=56 time=180 ms
    
    --- 195.181.174.20 ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2000ms
    rtt min/avg/max/mdev = 180.824/180.887/181.001/0.356 ms
    
    ----------------------------------------------
    Pings (10gbps.io London):
    PING 185.59.221.20 (185.59.221.20) 56(84) bytes of data.
    64 bytes from 185.59.221.20: icmp_seq=1 ttl=56 time=182 ms
    64 bytes from 185.59.221.20: icmp_seq=2 ttl=56 time=424 ms
    64 bytes from 185.59.221.20: icmp_seq=3 ttl=56 time=182 ms
    
    --- 185.59.221.20 ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 182.211/262.941/424.251/114.064 ms
    
    ----------------------------------------------
    Pings (10gbps.io Paris):
    PING 185.93.2.193 (185.93.2.193) 56(84) bytes of data.
    64 bytes from 185.93.2.193: icmp_seq=1 ttl=56 time=182 ms
    64 bytes from 185.93.2.193: icmp_seq=2 ttl=56 time=182 ms
    64 bytes from 185.93.2.193: icmp_seq=3 ttl=56 time=182 ms
    
    --- 185.93.2.193 ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2003ms
    rtt min/avg/max/mdev = 182.887/182.923/182.979/0.040 ms
    
    ----------------------------------------------
    Pings (10gbps.io Singapore):
    PING 203.174.85.38 (203.174.85.38) 56(84) bytes of data.
    64 bytes from 203.174.85.38: icmp_seq=1 ttl=57 time=176 ms
    64 bytes from 203.174.85.38: icmp_seq=2 ttl=57 time=176 ms
    64 bytes from 203.174.85.38: icmp_seq=3 ttl=57 time=176 ms
    
    --- 203.174.85.38 ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2003ms
    rtt min/avg/max/mdev = 176.685/176.729/176.765/0.486 ms
    
    ----------------------------------------------
    Pings (10gbps.io Stockholm):
    PING 185.76.9.10 (185.76.9.10) 56(84) bytes of data.
    64 bytes from 185.76.9.10: icmp_seq=1 ttl=56 time=191 ms
    64 bytes from 185.76.9.10: icmp_seq=2 ttl=56 time=191 ms
    64 bytes from 185.76.9.10: icmp_seq=3 ttl=56 time=191 ms
    
    --- 185.76.9.10 ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 191.238/191.268/191.325/0.506 ms
    
    ----------------------------------------------
    Pings (10gbps.io Moscow):
    PING 87.245.204.20 (87.245.204.20) 56(84) bytes of data.
    64 bytes from 87.245.204.20: icmp_seq=1 ttl=56 time=210 ms
    64 bytes from 87.245.204.20: icmp_seq=2 ttl=56 time=210 ms
    64 bytes from 87.245.204.20: icmp_seq=3 ttl=56 time=210 ms
    
    --- 87.245.204.20 ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2000ms
    rtt min/avg/max/mdev = 210.037/210.122/210.280/0.111 ms
    
    ----------------------------------------------
    Pings (10gbps.io Miami):
    PING 195.181.162.195 (195.181.162.195) 56(84) bytes of data.
    64 bytes from 195.181.162.195: icmp_seq=1 ttl=55 time=81.1 ms
    64 bytes from 195.181.162.195: icmp_seq=2 ttl=55 time=81.3 ms
    64 bytes from 195.181.162.195: icmp_seq=3 ttl=55 time=81.1 ms
    
    --- 195.181.162.195 ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 81.130/81.210/81.348/0.343 ms
    
    

    Code (Text):
    -------------------------------------------
    Running mtr tests...
    -------------------------------------------
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 cachefly.cachefly.net
    Start: Sun Feb 25 19:53:36 2018
    HOST: hostname         Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                        0.0%    10   20.8  19.0  11.6  21.3   2.8
      2.|-- vl101.fr3.lax.llnw.net         0.0%    10    0.2   0.2   0.2   0.3   0.0
      3.|-- ae10-352.cr0-lax1.ip4.gtt.net  0.0%    10    0.3   0.4   0.3   0.8   0.0
      4.|-- ???                           100.0    10    0.0   0.0   0.0   0.0   0.0
      5.|-- vip1.G-anycast1.cachefly.net   0.0%    10    0.2   0.3   0.2   0.5   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 speedtest.atlanta.linode.com
    Start: Sun Feb 25 19:53:51 2018
    HOST: hostname                             Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                                            0.0%    10    9.9   8.5   2.6  14.3   3.5
      2.|-- 2607:f4e8:1:110::122                               0.0%    10    0.3   1.3   0.2   9.6   2.9
      3.|-- ix-ae-20-0.tcore1.EQL-Los-Angeles.ipv6.as6453.net  0.0%    10  140.2 140.4 140.2 140.6   0.0
      4.|-- if-ae-30-2.tcore1.LVW-Los-Angeles.ipv6.as6453.net  0.0%    10  142.1 141.9 137.4 147.5   3.4
      5.|-- if-ae-34-2.tcore2.DT8-Dallas.ipv6.as6453.net       0.0%    10  136.7 136.7 136.3 137.6   0.0
      6.|-- if-ae-2-2.tcore1.DT8-Dallas.ipv6.as6453.net        0.0%    10  139.4 142.3 139.4 167.3   8.8
      7.|-- if-ae-27-2.tcore1.A56-Atlanta.ipv6.as6453.net      0.0%    10  137.3 138.3 137.2 145.8   2.6
      8.|-- 2001:5a0:a00:500::96                               0.0%    10   66.6  66.7  66.6  67.0   0.0
      9.|-- 2600:3c02:4444:1::2                                0.0%    10   66.7  66.7  66.5  67.2   0.0
     10.|-- 2600:3c02::4b                                      0.0%    10   66.4  66.1  66.0  66.4   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 speedtest.dallas.linode.com
    Start: Sun Feb 25 19:54:09 2018
    HOST: hostname        Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                       0.0%    10    2.7   6.4   1.2  11.1   3.9
      2.|-- 2607:f4e8:1:110::122          0.0%    10    0.3   0.3   0.3   0.8   0.0
      3.|-- ve6.fr4.lax.ipv6.llnw.net     0.0%    10    0.3   0.3   0.2   0.3   0.0
      4.|-- lag23.fr3.phx4.ipv6.llnw.net  0.0%    10    9.4   9.5   9.4   9.7   0.0
      5.|-- 2607:f4e8:121:4000::2         0.0%    10    9.6   9.7   9.5  11.4   0.6
      6.|-- lag38.fr4.dal.ipv6.llnw.net   0.0%    10   32.1  32.2  32.1  32.3   0.0
      7.|-- eqix-ix-da-ipv6.linode.com    0.0%    10   32.8  37.4  32.8  71.9  12.2
      8.|-- 2600:3c00:2222:11::2          0.0%    10   33.9  33.6  33.4  33.9   0.0
      9.|-- speedtest.dallas.linode.com   0.0%    10   33.1  33.0  32.8  33.3   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 mirror.us.leaseweb.net
    Start: Sun Feb 25 19:54:24 2018
    HOST: hostname                             Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                                            0.0%    10   13.0   6.9   2.1  13.0   3.7
      2.|-- 2607:f4e8:1:110::122                               0.0%    10    0.3   0.3   0.2   0.3   0.0
      3.|-- ix-ae-20-0.tcore1.EQL-Los-Angeles.ipv6.as6453.net  0.0%    10  140.5 144.4 140.3 164.3   8.6
      4.|-- if-ae-6-2.tcore2.LVW-Los-Angeles.ipv6.as6453.net   0.0%    10  140.5 141.4 140.4 148.6   2.4
      5.|-- if-ae-36-2.tcore2.AEQ-Ashburn.ipv6.as6453.net      0.0%    10  139.8 140.0 139.8 140.3   0.0
      6.|-- if-ae-2-2.tcore1.AEQ-Ashburn.ipv6.as6453.net       0.0%    10  142.8 142.9 142.4 144.8   0.6
      7.|-- if-ae-32-2.tcore1.VN5-Manassas.ipv6.as6453.net     0.0%    10  147.5 140.2 138.3 147.5   3.7
      8.|-- 2001:5a0:600:400::b2                               0.0%    10   80.4  80.5  80.4  80.6   0.0
      9.|-- 2604:9a00::108:59:15:247                           0.0%    10   79.1  79.0  78.9  79.1   0.0
     10.|-- 2604:9a00:2010:a0b8::5                             0.0%    10   79.9  80.0  79.8  80.1   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 speedtest.sea01.softlayer.com
    Start: Sun Feb 25 19:54:42 2018
    HOST: hostname                                             Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                                                            0.0%    10    8.6   7.5   2.1  10.4   2.3
      2.|-- 2607:f4e8:1:110::122                                               0.0%    10    0.3   0.4   0.2   1.4   0.0
      3.|-- ve7.fr3.lax5.ipv6.llnw.net                                         0.0%    10    0.5   2.1   0.3   7.7   2.4
      4.|-- 2001:504:13::210:131                                               0.0%    10    0.5   0.9   0.4   2.4   0.3
      5.|-- 2607:f0d0:2:2::24c                                                 0.0%    10    1.4   2.0   1.2   7.7   1.9
      6.|-- 1620.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   10.2   9.8   9.4  10.7   0.3
      7.|-- 0520.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10    9.6   9.8   9.4  10.9   0.3
      8.|-- ce20.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   27.6  27.9  27.3  28.6   0.0
      9.|-- 1630.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   26.5  27.3  26.1  33.0   2.0
     10.|-- po3.bbr01.sr01.sea01.networklayer.com                              0.0%    10   27.3  26.4  26.1  27.3   0.0
     11.|-- po1.fcr01.sr01.sea01.networklayer.com                              0.0%    10   26.6  26.7  26.5  27.5   0.0
     12.|-- softlayer.com                                                      0.0%    10   26.2  26.2  26.1  26.4   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 speedtest.sjc01.softlayer.com
    Start: Sun Feb 25 19:55:03 2018
    HOST: hostname                                             Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                                                            0.0%    10    1.8   5.4   0.8   9.7   3.4
      2.|-- 2607:f4e8:1:110::122                                               0.0%    10    0.3   0.4   0.2   1.7   0.3
      3.|-- ve7.fr3.lax5.ipv6.llnw.net                                         0.0%    10    0.6   5.0   0.3  14.7   4.9
      4.|-- 2001:504:13::210:131                                               0.0%    10    0.6   0.9   0.4   2.2   0.5
      5.|-- 2607:f0d0:2:2::24c                                                 0.0%    10    1.6   1.6   1.1   4.4   0.9
      6.|-- 1620.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   11.8  12.1  11.4  13.6   0.5
      7.|-- 9520.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   10.9  13.8  10.9  36.9   8.1
      8.|-- 1b10.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   11.2  11.3  11.1  11.8   0.0
      9.|-- 2607:f0d0:2100:1::3                                               10.0%    10   11.3  18.4  11.3  51.1  13.4
     10.|-- softlayer.com                                                      0.0%    10   11.1  11.2  11.1  11.4   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 speedtest.wdc01.softlayer.com
    Start: Sun Feb 25 19:55:20 2018
    HOST: hostname                                             Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                                                            0.0%    10    6.8   6.0   1.0   9.8   3.1
      2.|-- 2607:f4e8:1:110::122                                               0.0%    10    0.2   0.2   0.2   0.3   0.0
      3.|-- ve7.fr3.lax5.ipv6.llnw.net                                         0.0%    10    6.3   2.3   0.4   9.5   3.2
      4.|-- 2001:504:13::210:131                                               0.0%    10    1.9   1.3   0.5   7.2   2.1
      5.|-- 2607:f0d0:2:2::246                                                 0.0%    10    1.3   4.0   1.2  26.0   7.7
      6.|-- 2607:f0d0:2:2::2f2                                                 0.0%    10   31.3  32.6  31.3  41.8   3.2
      7.|-- d030.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   32.2  33.7  31.5  49.4   5.6
      8.|-- e230.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   31.7  33.0  31.6  36.5   1.6
      9.|-- ba20.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   64.0  65.4  64.0  75.6   3.5
     10.|-- ca20.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   64.5  64.5  64.0  66.9   0.7
     11.|-- 2607:f0d0:2:1::6                                                   0.0%    10   64.0  64.0  63.9  64.4   0.0
     12.|-- po2.dar01.sr01.wdc01.networklayer.com                              0.0%    10   64.2  64.2  64.1  64.3   0.0
     13.|-- po1.fcr01.sr01.wdc01.networklayer.com                              0.0%    10   65.1  65.2  64.4  70.0   1.6
     14.|-- softlayer.com                                                      0.0%    10   64.1  64.3  64.1  64.4   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 speedtest.tokyo.linode.com
    Start: Sun Feb 25 19:55:49 2018
    HOST: hostname         Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                        0.0%    10    7.2   5.8   1.2  10.8   3.2
      2.|-- 2607:f4e8:1:110::122           0.0%    10    0.3   0.4   0.3   1.0   0.0
      3.|-- ve7.fr3.lax5.ipv6.llnw.net     0.0%    10    0.4   2.1   0.3   8.0   2.8
      4.|-- tge1-5.fr3.mia1.ipv6.llnw.net  0.0%    10  106.7 106.7 106.6 106.7   0.0
      5.|-- 2001:268:fc02:34::1            0.0%    10  240.7 240.6 240.4 240.7   0.0
      6.|-- 2001:268:fa02:127::1           0.0%    10  245.1 244.3 243.4 245.1   0.3
      7.|-- 2001:268:fa00:27e::2           0.0%    10  242.5 243.9 242.5 252.0   3.0
      8.|-- 6cm-fcu204.int-gw.kddi.ne.jp   0.0%    10  244.4 247.9 243.7 256.8   4.3
      9.|-- 2001:268:f702:6d::2            0.0%    10  243.4 243.7 243.4 243.9   0.0
     10.|-- speedtest.tokyo.linode.com     0.0%    10  245.8 245.9 245.8 246.1   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 speedtest.sng01.softlayer.com
    Start: Sun Feb 25 19:56:06 2018
    HOST: hostname                                             Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                                                            0.0%    10    1.2  12.5   1.2  57.2  16.1
      2.|-- 2607:f4e8:1:110::122                                               0.0%    10    0.3   0.3   0.2   0.3   0.0
      3.|-- ve6.fr4.lax.ipv6.llnw.net                                          0.0%    10    0.3   0.7   0.3   4.1   1.1
      4.|-- lag16.fr4.nrt1.ipv6.llnw.net                                       0.0%    10  102.5  99.4  98.0 102.6   2.0
      5.|-- tge12-1.fr4.tyo3.ipv6.llnw.net                                     0.0%    10  102.4 103.4  98.4 110.5   4.4
      6.|-- 36351.tyo.equinix.com                                              0.0%    10  155.8 155.9 155.8 156.1   0.0
      7.|-- 8740.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10  158.8 158.3 156.6 159.9   0.8
      8.|-- 1940.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10  227.3 228.3 227.0 239.2   3.8
      9.|-- 9840.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10  227.2 227.3 227.0 228.5   0.3
     10.|-- 1c00.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10  227.1 227.2 226.9 228.5   0.3
     11.|-- 3.00.0000.ip4.static.sl-reverse.com                               10.0%    10  227.0 230.0 227.0 251.6   8.1
     12.|-- 2.00.0000.ip4.static.sl-reverse.com                                0.0%    10  227.0 227.0 226.9 227.2   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 speedtest.london.linode.com
    Start: Sun Feb 25 19:56:25 2018
    HOST: hostname                             Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                                            0.0%    10    7.2   7.7   3.3  11.4   2.5
      2.|-- 2607:f4e8:1:110::122                               0.0%    10   10.1  11.2   0.3  30.3  11.0
      3.|-- ix-ae-20-0.tcore1.EQL-Los-Angeles.ipv6.as6453.net  0.0%    10  140.1 146.3 140.1 178.1  13.0
      4.|-- if-ae-6-2.tcore2.LVW-Los-Angeles.ipv6.as6453.net   0.0%    10  140.4 146.8 140.4 202.6  19.6
      5.|-- if-ae-36-2.tcore2.AEQ-Ashburn.ipv6.as6453.net      0.0%    10  140.1 140.0 139.9 140.1   0.0
      6.|-- tcore4.NJY-Newark.ipv6.as6453.net                  0.0%    10  140.0 140.3 140.0 140.8   0.0
      7.|-- if-ae-4-3.thar2.NJY-Newark.ipv6.as6453.net         0.0%    10  142.4 142.6 142.3 144.2   0.5
      8.|-- if-ae-1-3.thar1.NJY-Newark.ipv6.as6453.net         0.0%    10  142.8 142.5 142.3 143.0   0.0
      9.|-- if-ae-3-3.tcore3.NJY-Newark.ipv6.as6453.net        0.0%    10  145.8 139.1 137.4 145.8   3.2
     10.|-- if-ae-9-2.tcore1.L78-London.ipv6.as6453.net        0.0%    10  141.7 142.0 141.7 143.4   0.5
     11.|-- if-ae-2-2.tcore2.L78-London.ipv6.as6453.net        0.0%    10  139.3 139.3 139.1 139.9   0.0
     12.|-- if-ae-15-2.tcore2.LDN-London.ipv6.as6453.net       0.0%    10  141.8 142.2 141.8 144.3   0.6
     13.|-- 2a01:3e0:10:2100::66                               0.0%    10  147.9 147.7 147.5 147.9   0.0
     14.|-- 2a01:7e00:7777:1::2                                0.0%    10  139.8 139.7 139.5 139.8   0.0
     15.|-- speedtest.london.linode.com                        0.0%    10  139.2 139.2 139.0 139.4   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 mirror.i3d.net
    Start: Sun Feb 25 19:56:45 2018
    HOST: hostname            Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                           0.0%    10   36.6   9.7   1.6  36.6  10.1
      2.|-- 2607:f4e8:1:110::122              0.0%    10    1.7   1.8   0.2  13.7   4.2
      3.|-- ve6.fr4.lax.ipv6.llnw.net         0.0%    10    0.3   0.4   0.2   0.9   0.0
      4.|-- lag45.fr4.sjc.ipv6.llnw.net       0.0%    10    6.4   7.0   6.4   8.3   0.6
      5.|-- lag55.fr4.ord.ipv6.llnw.net       0.0%    10   58.1  58.1  58.0  58.3   0.0
      6.|-- lag23.fr4.fra1.ipv6.llnw.net      0.0%    10  173.0 173.2 173.0 173.4   0.0
      7.|-- i3d.ipv6.plix.pl                  0.0%    10  200.6 200.8 199.9 204.8   1.4
      8.|-- 10ge.fr0-wr0.inxn-1.fra.i3d.net  10.0%    10  199.4 200.0 199.4 200.4   0.0
      9.|-- 20ge.cr0-fr0.smartdc.rtd.i3d.net  0.0%    10  212.9 213.2 212.6 214.0   0.0
     10.|-- 80ge.br2-cr0.smartdc.rtd.i3d.net  0.0%    10  212.9 213.3 212.5 214.4   0.3
     11.|-- hosted-by.i3d.net                10.0%    10  212.4 212.9 212.4 213.4   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 syd-au-ping.vultr.com
    Start: Sun Feb 25 19:57:04 2018
    HOST: hostname                        Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                                       0.0%    10   17.3  16.6  12.3  22.1   3.1
      2.|-- vl101.fr3.lax.llnw.net                        0.0%    10    0.3   0.6   0.3   2.7   0.7
      3.|-- ve7.fr3.lax5.llnw.net                         0.0%    10    7.3   3.2   0.3  10.2   3.6
      4.|-- vocus-connect-pty.as4826.any2ix.coresite.com  0.0%    10    1.0   0.8   0.7   1.0   0.0
      5.|-- bundle-150.cor01.lax01.ca.vocus.net           0.0%    10  146.1 145.8 145.8 146.1   0.0
      6.|-- 100g-0-1-0-0.cor01.sjc01.ca.vocus.net         0.0%    10  160.1 160.2 160.0 160.5   0.0
      7.|-- BE-200.cor02.syd04.nsw.VOCUS.net.au           0.0%    10  151.1 150.9 150.7 151.2   0.0
      8.|-- bundle-101.bdr01.syd03.nsw.vocus.net.au       0.0%    10  155.6 155.8 155.6 155.9   0.0
      9.|-- static-73.7.255.49.in-addr.VOCUS.net.au       0.0%    10  155.0 154.9 154.9 155.0   0.0
     10.|-- xe-1-0-1.gw102.sy3.ap.equinix.com             0.0%    10  154.9 155.0 154.9 155.2   0.0
     11.|-- 180.189.25.6                                  0.0%    10  155.1 156.2 155.0 160.5   1.8
     12.|-- vl901-ds1-b5-r1120.aus1.choopa.net            0.0%    10  152.1 165.0 152.0 281.4  40.9
     13.|-- ???                                          100.0    10    0.0   0.0   0.0   0.0   0.0
     14.|-- 108.61.212.117.vultr.com                      0.0%    10  150.9 150.9 150.8 151.3   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 hnd-jp-ping.vultr.com
    Start: Sun Feb 25 19:57:21 2018
    HOST: hostname                      Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                                     0.0%    10   18.0  18.5  12.4  23.2   3.9
      2.|-- vl101.fr3.lax.llnw.net                      0.0%    10    0.3   0.3   0.2   0.3   0.0
      3.|-- ae10-352.cr0-lax1.ip4.gtt.net               0.0%    10    0.3   1.5   0.3  12.5   3.8
      4.|-- et-0-0-67.cr4-lax2.ip4.gtt.net              0.0%    10    0.3   1.8   0.3   5.6   2.0
      5.|-- as2914.cr4-lax2.ip4.gtt.net                 0.0%    10    0.8   0.8   0.6   1.0   0.0
      6.|-- ae-3.r23.lsanca07.us.bb.gin.ntt.net         0.0%    10    0.8   2.2   0.8   7.2   2.2
      7.|-- ae-12.r31.tokyjp05.jp.bb.gin.ntt.net       40.0%    10   99.8  98.8  98.0 101.0   1.1
      8.|-- ae-3.r02.tokyjp05.jp.bb.gin.ntt.net         0.0%    10  104.2 104.2 104.1 104.2   0.0
      9.|-- ce-0-14-0-1.r02.tokyjp05.jp.ce.gin.ntt.net  0.0%    10  105.0 108.3 104.8 120.7   5.9
     10.|-- ???                                        100.0    10    0.0   0.0   0.0   0.0   0.0
     11.|-- vl516-ds1-b5-r2605.tyo1.choopa.net          0.0%    10  105.3 105.3 105.3 105.5   0.0
     12.|-- ???                                        100.0    10    0.0   0.0   0.0   0.0   0.0
     13.|-- 108.61.201.151.vultr.com                    0.0%    10  108.0 108.0 107.9 108.1   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 lax-ca-us-ping.vultr.com
    Start: Sun Feb 25 19:57:38 2018
    HOST: hostname             Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                            0.0%    10   15.7  22.9  11.6  77.4  19.5
      2.|-- vl101.fr3.lax.llnw.net             0.0%    10    0.3   0.3   0.2   0.5   0.0
      3.|-- ae10-352.cr0-lax1.ip4.gtt.net      0.0%    10    0.3   0.7   0.3   2.6   0.7
      4.|-- xe-11-2-0.cr2-lax2.ip4.gtt.net     0.0%    10    0.4   0.4   0.3   0.4   0.0
      5.|-- as20473-gw.lax20.ip4.gtt.net       0.0%    10    0.5   1.6   0.4   6.7   2.0
      6.|-- vl901-ds1-b5-c705.lax3.choopa.net  0.0%    10    1.0   3.8   1.0  28.1   8.5
      7.|-- ???                               100.0    10    0.0   0.0   0.0   0.0   0.0
      8.|-- 108.61.219.200.vultr.com           0.0%    10    0.7   0.6   0.5   0.7   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 wa-us-ping.vultr.com
    Start: Sun Feb 25 19:57:53 2018
    HOST: hostname         Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                        0.0%    10   16.4  15.7  11.7  20.5   2.9
      2.|-- vl101.fr3.lax.llnw.net         0.0%    10    0.2   0.3   0.2   0.8   0.0
      3.|-- ae10-352.cr0-lax1.ip4.gtt.net  0.0%    10    0.3   0.5   0.3   2.4   0.6
      4.|-- et-2-3-0.cr3-sea2.ip4.gtt.net  0.0%    10   35.9  35.5  35.4  35.9   0.0
      5.|-- choopa-gw.ip4.gtt.net          0.0%    10   33.8  33.8  33.7  34.1   0.0
      6.|-- ???                           100.0    10    0.0   0.0   0.0   0.0   0.0
      7.|-- 108.61.194.105.vultr.com       0.0%    10   33.8  33.8  33.8  33.9   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 tx-us-ping.vultr.com
    Start: Sun Feb 25 19:58:09 2018
    HOST: hostname              Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                             0.0%    10   20.7  18.3  15.9  22.4   1.9
      2.|-- vl101.fr3.lax.llnw.net              0.0%    10    0.3   0.3   0.2   0.3   0.0
      3.|-- ae10-352.cr0-lax1.ip4.gtt.net       0.0%    10    0.3   0.5   0.3   1.3   0.0
      4.|-- xe-2-0-1.cr0-dal2.ip4.gtt.net       0.0%    10   27.6  29.8  27.3  49.2   6.8
      5.|-- ip4.gtt.net                        30.0%    10   27.7  28.1  27.6  28.9   0.0
      6.|-- vl902-ds1-b5-02.04.dal4.choopa.net 70.0%    10  149.0 162.0 149.0 174.3  12.6
      7.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
      8.|-- 108.61.224.175.vultr.com            0.0%    10   27.6  27.9  27.6  29.6   0.6
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 il-us-ping.vultr.com
    Start: Sun Feb 25 19:58:24 2018
    HOST: hostname              Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                             0.0%    10   20.7  17.4  12.5  21.2   2.8
      2.|-- vl101.fr3.lax.llnw.net              0.0%    10    0.2   0.4   0.2   1.5   0.3
      3.|-- ae10-352.cr0-lax1.ip4.gtt.net       0.0%    10    1.0   0.5   0.3   1.1   0.0
      4.|-- xe-0-3-0.cr8-chi1.ip4.gtt.net       0.0%    10   55.4  55.5  55.4  55.8   0.0
      5.|-- ip4.gtt.net                         0.0%    10   61.1  63.2  60.7  72.6   4.3
      6.|-- vl820-ds1-b5-gj178.chi2.choopa.net  0.0%    10   59.8  59.9  59.7  60.1   0.0
      7.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
      8.|-- 107.191.51.12.vultr.com             0.0%    10   59.5  59.5  59.4  59.6   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 ga-us-ping.vultr.com
    Start: Sun Feb 25 19:58:39 2018
    HOST: hostname               Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                              0.0%    10   15.1  22.8  12.6  65.5  15.2
      2.|-- vl101.fr3.lax.llnw.net               0.0%    10    0.2   0.3   0.2   0.3   0.0
      3.|-- las-b21-link.telia.net               0.0%    10    0.9   1.0   0.7   2.1   0.3
      4.|-- dls-b21-link.telia.net               0.0%    10   30.1  32.1  29.9  45.3   4.6
      5.|-- atl-b22-link.telia.net               0.0%    10   50.4  50.6  50.4  51.2   0.0
      6.|-- atl-b21-link.telia.net               0.0%    10   50.3  52.6  50.2  73.6   7.3
      7.|-- vultr-ic-319646-atl-b21.c.telia.net  0.0%    10   53.0  52.3  50.2  56.6   2.4
      8.|-- vl901-ds1-b5-r7.atl2.choopa.net      0.0%    10   50.8  50.8  50.8  51.0   0.0
      9.|-- ???                                 100.0    10    0.0   0.0   0.0   0.0   0.0
     10.|-- 108.61.193.166.vultr.com             0.0%    10   50.3  50.3  50.2  50.4   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 fl-us-ping.vultr.com
    Start: Sun Feb 25 19:58:57 2018
    HOST: hostname                           Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                                          0.0%    10   19.4  17.0  11.5  27.7   4.3
      2.|-- vl101.fr3.lax.llnw.net                           0.0%    10    0.3   0.4   0.2   1.9   0.3
      3.|-- ix-ae-20-0.tcore1.EQL-Los-Angeles.as6453.net     0.0%    10   33.5  33.3  33.2  33.5   0.0
      4.|-- if-ae-6-2.tcore2.LVW-Los-Angeles.as6453.net      0.0%    10   34.4  38.3  33.7  51.6   6.5
      5.|-- te-0-5-0-8-pe01.losangeles.ca.ibone.comcast.net  0.0%    10    1.0   1.0   0.9   1.3   0.0
      6.|-- be-11599-cr02.losangeles.ca.ibone.comcast.net    0.0%    10    2.9   2.2   1.4   3.0   0.0
      7.|-- be-11524-cr02.dallas.tx.ibone.comcast.net        0.0%    10   34.4  33.9  32.7  34.7   0.3
      8.|-- be-12224-cr02.miami.fl.ibone.comcast.net         0.0%    10   62.8  62.7  62.1  63.5   0.0
      9.|-- be-12274-pe01.nota.fl.ibone.comcast.net          0.0%    10   61.4  61.2  61.0  61.5   0.0
     10.|-- 96-87-8-30-static.hfc.comcastbusiness.net        0.0%    10   63.2  61.9  61.5  63.2   0.3
     11.|-- ???                                             100.0    10    0.0   0.0   0.0   0.0   0.0
     12.|-- ???                                             100.0    10    0.0   0.0   0.0   0.0   0.0
     13.|-- 104.156.244.232                                  0.0%    10   62.7  62.7  62.5  63.4   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 nj-us-ping.vultr.com
    Start: Sun Feb 25 19:59:13 2018
    HOST: hostname              Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                             0.0%    10   16.8  21.8  13.9  58.4  13.2
      2.|-- vl101.fr3.lax.llnw.net              0.0%    10    1.5   0.5   0.2   1.6   0.0
      3.|-- las-b21-link.telia.net              0.0%    10    0.9   0.8   0.7   1.0   0.0
      4.|-- nyk-bb4-link.telia.net              0.0%    10   84.3  84.4  84.3  84.8   0.0
      5.|-- nyk-b2-link.telia.net               0.0%    10   84.3  85.3  84.1  95.0   3.3
      6.|-- vultr-ic-318452-nyk-b2.c.telia.net  0.0%    10   84.5  86.7  84.5  92.0   2.6
      7.|-- vl39-br1.pnj1.choopa.net            0.0%    10   88.4  88.5  84.6  95.0   3.8
      8.|-- vl139-c4-3-b5-2.pnj1.choopa.net     0.0%    10   85.3  85.2  85.1  85.4   0.0
      9.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
     10.|-- 108.61.149.182.vultr.com            0.0%    10   84.7  84.6  84.6  84.7   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 fra-de-ping.vultr.com
    Start: Sun Feb 25 19:59:29 2018
    HOST: hostname         Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                        0.0%    10   22.4  19.9  12.4  22.9   3.5
      2.|-- vl101.fr3.lax.llnw.net         0.0%    10    0.2   0.3   0.2   0.3   0.0
      3.|-- ae10-352.cr0-lax1.ip4.gtt.net  0.0%    10    0.3   0.6   0.3   2.4   0.5
      4.|-- xe-8-0-2.cr1-fra2.ip4.gtt.net  0.0%    10  163.8 163.8 163.6 163.9   0.0
      5.|-- ip4.gtt.net                    0.0%    10  164.0 164.9 163.7 173.4   2.9
      6.|-- ???                           100.0    10    0.0   0.0   0.0   0.0   0.0
      7.|-- 108.61.210.117.vultr.com       0.0%    10  163.9 163.8 163.7 164.0   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 ams-nl-ping.vultr.com
    Start: Sun Feb 25 19:59:44 2018
    HOST: hostname              Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                             0.0%    10   19.8  18.8  13.5  28.8   4.8
      2.|-- vl101.fr3.lax.llnw.net              0.0%    10    0.3   0.3   0.2   0.7   0.0
      3.|-- las-b21-link.telia.net              0.0%    10    0.9   0.8   0.7   0.9   0.0
      4.|-- nyk-bb3-link.telia.net              0.0%    10   80.6  80.5  80.2  80.6   0.0
      5.|-- ldn-bb3-link.telia.net              0.0%    10  156.0 150.9 149.6 156.0   1.9
      6.|-- adm-bb3-link.telia.net              0.0%    10  156.7 156.7 156.5 156.7   0.0
      7.|-- adm-b3-link.telia.net               0.0%    10  176.1 181.0 176.1 184.6   2.4
      8.|-- vultr-ic-313751-adm-b3.c.telia.net 10.0%    10  180.5 181.4 179.6 184.7   1.4
      9.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
     10.|-- 108.61.198.102.vultr.com            0.0%    10  170.0 167.6 161.4 170.9   3.2
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 lon-gb-ping.vultr.com
    Start: Sun Feb 25 20:00:00 2018
    HOST: hostname              Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                             0.0%    10   14.8  16.4  11.9  21.3   3.5
      2.|-- vl101.fr3.lax.llnw.net              0.0%    10    0.3   1.0   0.2   7.4   2.3
      3.|-- las-b21-link.telia.net              0.0%    10    0.9   0.8   0.8   0.9   0.0
      4.|-- ash-bb3-link.telia.net              0.0%    10   60.7  61.2  60.5  64.2   0.9
      5.|-- ldn-bb3-link.telia.net              0.0%    10  142.0 143.2 141.8 150.7   2.7
      6.|-- ldn-b9-link.telia.net               0.0%    10  135.9 136.0 135.7 137.0   0.0
      7.|-- vultr-ic-312913-ldn-b9.c.telia.net  0.0%    10  146.8 138.8 136.4 146.8   3.7
      8.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
      9.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
     10.|-- 108.61.196.101.vultr.com            0.0%    10  133.9 134.0 133.9 134.1   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 par-fr-ping.vultr.com
    Start: Sun Feb 25 20:00:16 2018
    HOST: hostname              Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                             0.0%    10   14.4  19.6  12.4  41.4   8.3
      2.|-- vl101.fr3.lax.llnw.net              0.0%    10    0.3   0.3   0.3   0.4   0.0
      3.|-- las-b21-link.telia.net              0.0%    10    0.8   0.9   0.8   1.2   0.0
      4.|-- nyk-bb3-link.telia.net              0.0%    10   83.9  83.9  83.6  84.8   0.0
      5.|-- prs-bb3-link.telia.net              0.0%    10  163.3 163.4 163.1 164.6   0.3
      6.|-- prs-b7-link.telia.net               0.0%    10  163.3 163.3 163.3 163.4   0.0
      7.|-- vultr-ic-312910-prs-b7.c.telia.net  0.0%    10  164.2 165.7 163.8 172.2   3.1
      8.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
      9.|-- 108.61.209.127.vultr.com            0.0%    10  163.7 163.7 163.6 163.9   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 VersaWeb Las Vegas
    Start: Sun Feb 25 20:00:31 2018
    HOST: hostname                  Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                                 0.0%    10   12.0  16.7  12.0  21.3   3.0
      2.|-- vl101.fr3.lax.llnw.net                  0.0%    10    0.3   0.3   0.2   0.3   0.0
      3.|-- ve6.fr4.lax.llnw.net                    0.0%    10    0.5   0.3   0.3   0.5   0.0
      4.|-- 10gigabitethernet1-3.core1.lax1.he.net  0.0%    10    9.4   2.1   0.2   9.4   3.1
      5.|-- 100ge14-1.core1.lax2.he.net             0.0%    10    6.3   4.3   0.4  11.5   4.5
      6.|-- 10ge4-3.core1.las1.he.net               0.0%    10    7.8   9.6   7.7  15.7   2.8
      7.|-- fiberhub.10ge3-2.core1.las1.he.net      0.0%    10   10.3  12.5   8.1  19.0   3.8
      8.|-- 192.228.109.86                          0.0%    10    8.2   8.1   8.1   8.2   0.0
      9.|-- lg.las1.ultravps.net                    0.0%    10    8.1   8.1   8.0   8.2   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 VersaWeb Seattle
    Start: Sun Feb 25 20:00:47 2018
    HOST: hostname                 Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                                0.0%    10   16.5  15.0  12.1  19.7   2.1
      2.|-- vl101.fr3.lax.llnw.net                 0.0%    10    0.3   0.4   0.2   1.3   0.0
      3.|-- be4695.ccr24.lax05.atlas.cogentco.com  0.0%    10    0.9   0.9   0.8   1.0   0.0
      4.|-- be2538.ccr41.lax01.atlas.cogentco.com  0.0%    10    0.7   0.8   0.6   1.1   0.0
      5.|-- be3176.ccr21.sjc01.atlas.cogentco.com  0.0%    10   11.9  11.8  11.6  12.1   0.0
      6.|-- be3178.ccr21.sfo01.atlas.cogentco.com  0.0%    10   13.0  12.9  12.8  13.1   0.0
      7.|-- be2075.ccr21.sea01.atlas.cogentco.com  0.0%    10   36.8  36.8  36.8  36.9   0.0
      8.|-- be2083.ccr21.sea02.atlas.cogentco.com  0.0%    10   30.2  30.2  30.0  30.8   0.0
      9.|-- 38.142.48.250                          0.0%    10   29.7  30.0  29.7  31.4   0.3
     10.|-- 76.164.234.1                           0.0%    10   38.4  38.3  38.3  38.4   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 OVH Canada
    Start: Sun Feb 25 20:01:04 2018
    HOST: hostname        Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                       0.0%    10    5.9  10.2   1.3  39.7  10.8
      2.|-- 2607:f4e8:1:110::122          0.0%    10    0.3   0.2   0.2   0.3   0.0
      3.|-- lag45.fr3.sjc.ipv6.llnw.net   0.0%    10    9.9  10.0   9.9  10.3   0.0
      4.|-- lag41.fr3.ord.ipv6.llnw.net   0.0%    10   64.3  64.3  64.2  64.7   0.0
      5.|-- ve6.fr4.ord.ipv6.llnw.net     0.0%    10   63.0  63.0  62.9  63.6   0.0
      6.|-- lag23.fr4.fra1.ipv6.llnw.net  0.0%    10  188.4 191.5 176.3 203.7   9.4
      7.|-- ???                          100.0    10    0.0   0.0   0.0   0.0   0.0
      8.|-- be100-1166.ams-1-a9.nl.eu     0.0%    10  188.6 189.0 188.5 189.3   0.0
      9.|-- mil-3-4m.it.eu               20.0%    10  190.2 197.8 189.3 242.5  18.4
     10.|-- po100.rbx-g1-a75.fr.eu        0.0%    10  188.9 188.7 188.1 189.1   0.0
     11.|-- vl165.rbx-g2-a75.fr.eu        0.0%    10  188.3 188.5 188.2 188.8   0.0
     12.|-- vl1305.bhs-g2-a75-lo2.qc.ca   0.0%    10  268.2 268.2 267.5 268.7   0.0
     13.|-- 2001:41d0:0:50::6:80f         0.0%    10  267.2 267.2 266.7 267.6   0.0
     14.|-- 2001:41d0:0:50::2:404d       10.0%    10  266.5 266.9 266.5 267.3   0.0
     15.|-- 2607:5300:60:44a5::1          0.0%    10  266.7 266.5 266.0 266.7   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 10gbps.io Frankfurt:
    Start: Sun Feb 25 20:01:21 2018
    HOST: hostname              Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                             0.0%    10   17.2  21.7   6.5  73.0  18.5
      2.|-- vl101.fr3.lax.llnw.net              0.0%    10    0.3   0.4   0.2   1.5   0.3
      3.|-- lag45.fr3.sjc.llnw.net              0.0%    10   10.0  10.0   9.9  10.3   0.0
      4.|-- lag41.fr3.ord.llnw.net              0.0%    10   64.3  64.3  64.2  64.9   0.0
      5.|-- lag44.fr3.lga.llnw.net              0.0%    10   80.7  80.7  80.6  81.1   0.0
      6.|-- xe-1-2-0-0.nyk10.core-backbone.com  0.0%    10   80.5  80.5  80.4  80.6   0.0
      7.|-- ae2-2025.fra30.core-backbone.com    0.0%    10  168.6 168.7 168.5 168.9   0.0
      8.|-- core-backbone.cdn77.com             0.0%    10  181.0 180.9 180.9 181.1   0.0
      9.|-- frankfurt-20.cdn77.com              0.0%    10  180.9 181.0 180.9 181.1   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 10gbps.io London:
    Start: Sun Feb 25 20:01:37 2018
    HOST: hostname              Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                             0.0%    10   17.0  18.1  14.4  21.9   2.9
      2.|-- vl101.fr3.lax.llnw.net              0.0%    10    8.1   2.5   0.2  10.4   3.8
      3.|-- lag45.fr3.sjc.llnw.net              0.0%    10   10.0  10.1   9.9  11.8   0.5
      4.|-- lag41.fr3.ord.llnw.net              0.0%    10   64.2  64.2  64.2  64.3   0.0
      5.|-- lag44.fr3.lga.llnw.net              0.0%    10   80.6  80.7  80.6  80.8   0.0
      6.|-- xe-1-2-0-0.nyk10.core-backbone.com  0.0%    10   80.4  80.4  80.3  80.4   0.0
      7.|-- ae10-2092.lon10.core-backbone.com   0.0%    10  141.8 142.0 141.7 142.6   0.0
      8.|-- core-backbone.cdn77.com             0.0%    10  181.4 181.4 181.3 181.4   0.0
      9.|-- london-20.cdn77.com                 0.0%    10  182.2 182.2 182.1 182.3   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 10gbps.io Paris:
    Start: Sun Feb 25 20:01:53 2018
    HOST: hostname                Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                               0.0%    10   19.0  19.2  12.2  44.4   9.5
      2.|-- vl101.fr3.lax.llnw.net                0.0%    10    0.3   0.3   0.2   0.8   0.0
      3.|-- ae10-352.cr0-lax1.ip4.gtt.net         0.0%    10    0.3   2.8   0.3  25.0   7.8
      4.|-- et-0-0-43.cr6-sjc1.ip4.gtt.net        0.0%    10   14.7  12.3   8.8  17.3   3.0
      5.|-- as2914.cr6-sjc1.ip4.gtt.net           0.0%    10    9.3   9.1   8.9   9.3   0.0
      6.|-- ae-1.r22.snjsca04.us.bb.gin.ntt.net   0.0%    10    9.0   9.9   8.9  16.1   2.2
      7.|-- ae-7.r23.asbnva02.us.bb.gin.ntt.net   0.0%    10   73.1  67.2  65.0  74.2   3.3
      8.|-- ae-2.r25.amstnl02.nl.bb.gin.ntt.net   0.0%    10  164.2 164.1 163.9 164.2   0.0
      9.|-- ae-3.r24.amstnl02.nl.bb.gin.ntt.net   0.0%    10  164.8 164.4 164.0 166.3   0.7
     10.|-- ae-14.r02.parsfr02.fr.bb.gin.ntt.net  0.0%    10  147.5 147.5 147.4 148.0   0.0
     11.|-- 81.25.206.50                          0.0%    10  182.7 182.7 182.7 182.8   0.0
     12.|-- unn-185-93-2-193.10gbps.io            0.0%    10  182.9 182.9 182.8 183.0   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 10gbps.io Singapore:
    Start: Sun Feb 25 20:02:10 2018
    HOST: hostname                        Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                                       0.0%    10   13.8  16.3  12.8  22.0   3.4
      2.|-- vl101.fr3.lax.llnw.net                        0.0%    10    0.2   0.2   0.2   0.3   0.0
      3.|-- ix-ae-20-0.tcore1.EQL-Los-Angeles.as6453.net  0.0%    10   34.5  33.8  33.4  35.3   0.5
      4.|-- if-ae-6-2.tcore2.LVW-Los-Angeles.as6453.net   0.0%    10  205.7 205.7 205.6 205.8   0.0
      5.|-- if-ae-7-2.tcore2.SVW-Singapore.as6453.net     0.0%    10  214.3 215.5 214.2 225.5   3.5
      6.|-- if-ae-2-2.tcore1.SVW-Singapore.as6453.net     0.0%    10  205.2 205.2 205.1 205.3   0.0
      7.|-- if-ae-11-2.thar1.SVQ-Singapore.as6453.net     0.0%    10  205.5 205.5 205.3 205.9   0.0
      8.|-- 180.87.98.50                                  0.0%    10  257.9 247.0 245.7 257.9   3.8
      9.|-- ???                                          100.0    10    0.0   0.0   0.0   0.0   0.0
     10.|-- hol14.homeoflim.com                           0.0%    10  176.8 176.8 176.7 176.9   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 10gbps.io Stockholm:
    Start: Sun Feb 25 20:02:29 2018
    HOST: hostname              Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                             0.0%    10   13.8  18.6  13.8  22.9   2.7
      2.|-- vl101.fr3.lax.llnw.net              0.0%    10    0.5   2.6   0.2   8.9   3.1
      3.|-- lag45.fr3.sjc.llnw.net              0.0%    10   10.0  10.0   9.8  10.0   0.0
      4.|-- lag41.fr3.ord.llnw.net              0.0%    10   64.2  64.2  64.1  64.5   0.0
      5.|-- lag44.fr3.lga.llnw.net              0.0%    10   80.7  80.6  80.5  80.7   0.0
      6.|-- xe-1-2-0-0.nyk10.core-backbone.com  0.0%    10   80.5  80.4  80.4  80.5   0.0
      7.|-- ae1-2081.sth10.core-backbone.com    0.0%    10  191.6 191.7 191.6 191.8   0.0
      8.|-- core-backbone.cdn77.com             0.0%    10  190.9 191.0 190.9 191.1   0.0
      9.|-- stockholm-10.cdn77.com              0.0%    10  229.1 195.1 191.3 229.1  11.9
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 10gbps.io Moscow:
    Start: Sun Feb 25 20:02:45 2018
    HOST: hostname       Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                      0.0%    10   18.3  25.1  13.4  62.1  15.5
      2.|-- vl101.fr3.lax.llnw.net       0.0%    10    0.3   0.4   0.2   1.3   0.0
      3.|-- lag45.fr3.sjc.llnw.net       0.0%    10   10.2  10.0   9.9  10.2   0.0
      4.|-- lag41.fr3.ord.llnw.net       0.0%    10   67.9  65.2  64.2  70.0   2.0
      5.|-- lag44.fr3.lga.llnw.net       0.0%    10   80.5  80.6  80.5  80.7   0.0
      6.|-- nyiix.retn.net               0.0%    10  190.4 190.4 190.2 190.4   0.0
      7.|-- ae1-3.RT.MR.MSK.RU.retn.net  0.0%    10  209.8 209.8 209.7 209.9   0.0
      8.|-- 87.245.204.20                0.0%    10  210.2 210.2 210.1 210.3   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 10gbps.io Miami:
    Start: Sun Feb 25 20:03:01 2018
    HOST: hostname               Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                              0.0%    10   21.8  22.5  11.6  76.9  19.5
      2.|-- vl101.fr3.lax.llnw.net               0.0%    10    0.3   0.4   0.2   1.9   0.5
      3.|-- ae10-352.cr0-lax1.ip4.gtt.net        0.0%    10    0.4   0.4   0.3   0.8   0.0
      4.|-- et-0-0-43.cr6-sjc1.ip4.gtt.net       0.0%    10    9.1   9.1   8.8   9.6   0.0
      5.|-- as2914.cr6-sjc1.ip4.gtt.net          0.0%    10    9.2   9.0   8.9   9.2   0.0
      6.|-- ae-1.r22.snjsca04.us.bb.gin.ntt.net  0.0%    10    9.0   9.2   9.0  10.2   0.0
      7.|-- ae-7.r23.asbnva02.us.bb.gin.ntt.net  0.0%    10   65.9  66.1  65.9  66.5   0.0
      8.|-- ae-1.r20.miamfl02.us.bb.gin.ntt.net  0.0%    10   81.4  88.7  81.4 154.2  23.0
      9.|-- ae-1.r04.miamfl02.us.bb.gin.ntt.net  0.0%    10   81.4  81.5  81.4  81.7   0.0
     10.|-- 165.254.21.26                        0.0%    10   81.2  81.2  81.2  81.3   0.0
     11.|-- unn-195-181-162-195.cdn77.com        0.0%    10   81.4  81.3  81.2  81.5   0.0
    
    

    Code (Text):
    -------------------------------------------
    Check system entropy pool availability
    -------------------------------------------
    
    entropy_avail: 4067
    entropy_avail: 4067
    entropy_avail: 4067
    entropy_avail: 4067
    

    Code (Text):
    -------------------------------------------
    OpenSSL System Benchmark
    -------------------------------------------
    
    OpenSSL 1.0.2k-fips  26 Jan 2017
    -------------------------------------------
    openssl speed -multi 8 rsa4096 rsa2048 ecdsap256 sha256 sha1 md5 rc4 aes-256-cbc aes-128-cbc
    OpenSSL 1.0.2k-fips  26 Jan 2017
    built on: reproducible build, date unspecified
    options:bn(64,64) md2(int) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
    compiler: gcc -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DKRB5_MIT -m64 -DL_ENDIAN -Wall -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches   -m64 -mtune=generic -Wa,--noexecstack -DPURIFY -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DRC4_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM
    md5             284699.40k   898011.78k  2258044.50k  3537850.03k  4124409.86k
    sha1            294420.36k   870333.63k  2106821.03k  3250504.02k  3869204.48k
    rc4            2845331.75k  4172376.00k  4627773.10k  4733641.73k  4785782.78k
    aes-128 cbc     547909.02k   579002.97k   586271.66k   589405.53k   590345.56k
    aes-256 cbc     396894.33k   413164.05k   416706.56k   418214.23k   418734.08k
    sha256          302442.80k   675528.70k  1284426.75k  1639872.51k  1784987.65k
                      sign    verify    sign/s verify/s
    rsa 2048 bits 0.000151s 0.000004s   6627.3 223809.5
    rsa 4096 bits 0.001018s 0.000016s    982.3  64387.1
                                  sign    verify    sign/s verify/s
     256 bit ecdsa (nistp256)   0.0000s   0.0000s 118525.0  51864.3
    -------------------------------------------
    openssl speed -evp aes256 -multi 8
    OpenSSL 1.0.2k-fips  26 Jan 2017
    built on: reproducible build, date unspecified
    options:bn(64,64) md2(int) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
    compiler: gcc -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DKRB5_MIT -m64 -DL_ENDIAN -Wall -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches   -m64 -mtune=generic -Wa,--noexecstack -DPURIFY -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DRC4_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM
    evp            5098231.54k  6559981.31k  6880439.89k  6972779.52k  7005604.52k
    -------------------------------------------
    openssl speed -evp aes128 -multi 8
    OpenSSL 1.0.2k-fips  26 Jan 2017
    built on: reproducible build, date unspecified
    options:bn(64,64) md2(int) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
    compiler: gcc -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DKRB5_MIT -m64 -DL_ENDIAN -Wall -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches   -m64 -mtune=generic -Wa,--noexecstack -DPURIFY -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DRC4_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM
    evp            5837247.91k  8792008.90k  9402493.10k  9585271.47k  9647740.25k
    

    Code (Text):
    -------------------------------------------
    Running mysqlslap
    -------------------------------------------
    
    mysqlslap --auto-generate-sql --auto-generate-sql-add-autoincrement --auto-generate-sql-secondary-indexes=5 --number-int-cols=5 --number-char-cols=5 --number-of-queries=25000 --auto-generate-sql-unique-query-number=40 --auto-generate-sql-unique-write-number=40 --auto-generate-sql-write-number=1000 --concurrency=64 --iterations=10 --engine=myisam
    Benchmark
            Running for engine myisam
            Average number of seconds to run all queries: 0.587 seconds
            Minimum number of seconds to run all queries: 0.576 seconds
            Maximum number of seconds to run all queries: 0.598 seconds
            Number of clients running queries: 64
            Average number of queries per client: 390
    
    Dropping the database is potentially a very bad thing to do.
    Any data stored in the database will be destroyed.
    
    Do you really want to drop the 'test' database [y/N] Database "test" dropped
    

    Code (Text):
    -------------------------------------------
    System PHP Info
    -------------------------------------------
    
    CPU: 8 x Intel(R) Xeon(R) CPU E3-1578L v5 @ 2.00GHz
                  total        used        free      shared  buff/cache   available
    Mem:          32084        1478       29998          33         607       30199
    Low:          32084        2086       29998
    High:             0           0           0
    Swap:          1949           0        1949
    ----------------------------------------------
    PHP 5.6.33 (cli) (built: Feb 25 2018 19:39:56)
    Copyright (c) 1997-2016 The PHP Group
    Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
        with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2016, by Zend Technologies
    ----------------------------------------------
    Configuration File (php.ini) Path: /usr/local/lib
    Loaded Configuration File:         /usr/local/lib/php.ini
    Scan for additional .ini files in: /etc/centminmod/php.d
    Additional .ini files parsed:      /etc/centminmod/php.d/a_customphp.ini,
    /etc/centminmod/php.d/curlcainfo.ini,
    /etc/centminmod/php.d/geoip.ini,
    /etc/centminmod/php.d/igbinary.ini,
    /etc/centminmod/php.d/imagick.ini,
    /etc/centminmod/php.d/mailparse.ini,
    /etc/centminmod/php.d/memcache.ini,
    /etc/centminmod/php.d/memcached.ini,
    /etc/centminmod/php.d/redis.ini,
    /etc/centminmod/php.d/zendopcache.ini
    
    ----------------------------------------------
    [PHP Modules]
    bcmath
    bz2
    calendar
    Core
    ctype
    curl
    date
    dom
    enchant
    ereg
    exif
    filter
    ftp
    gd
    geoip
    gettext
    gmp
    hash
    iconv
    igbinary
    imagick
    imap
    intl
    json
    ldap
    libxml
    mailparse
    mbstring
    mcrypt
    memcache
    memcached
    mhash
    mysql
    mysqli
    mysqlnd
    openssl
    pcntl
    pcre
    PDO
    pdo_mysql
    pdo_sqlite
    Phar
    posix
    pspell
    readline
    redis
    Reflection
    session
    shmop
    SimpleXML
    snmp
    soap
    sockets
    SPL
    sqlite3
    standard
    sysvmsg
    sysvsem
    sysvshm
    tidy
    tokenizer
    xml
    xmlreader
    xmlrpc
    xmlwriter
    xsl
    Zend OPcache
    zip
    zlib
    
    [Zend Modules]
    Zend OPcache
    
    ----------------------------------------------
    -------------------------------------------
    Run PHP test Zend/bench.php
    -------------------------------------------
    
    simple             0.051
    simplecall         0.081
    simpleucall        0.077
    simpleudcall       0.078
    mandel             0.131
    mandel2            0.177
    ackermann(7)       0.068
    ary(50000)         0.014
    ary2(50000)        0.013
    ary3(2000)         0.110
    fibo(30)           0.244
    hash1(50000)       0.021
    hash2(500)         0.022
    heapsort(20000)    0.049
    matrix(20)         0.061
    nestedloop(12)     0.091
    sieve(30)          0.065
    strcat(200000)     0.009
    ------------------------
    Total              1.361
    real: 1.39s user: 1.37s sys: 0.01s cpu: 99% maxmem: 31404 KB cswaits: 7
    
    simple             0.053
    simplecall         0.082
    simpleucall        0.079
    simpleudcall       0.081
    mandel             0.133
    mandel2            0.183
    ackermann(7)       0.066
    ary(50000)         0.013
    ary2(50000)        0.012
    ary3(2000)         0.110
    fibo(30)           0.245
    hash1(50000)       0.021
    hash2(500)         0.022
    heapsort(20000)    0.049
    matrix(20)         0.059
    nestedloop(12)     0.091
    sieve(30)          0.065
    strcat(200000)     0.009
    ------------------------
    Total              1.372
    real: 1.40s user: 1.38s sys: 0.01s cpu: 99% maxmem: 31408 KB cswaits: 4
    
    simple             0.051
    simplecall         0.084
    simpleucall        0.080
    simpleudcall       0.079
    mandel             0.135
    mandel2            0.224
    ackermann(7)       0.067
    ary(50000)         0.014
    ary2(50000)        0.013
    ary3(2000)         0.110
    fibo(30)           0.248
    hash1(50000)       0.021
    hash2(500)         0.023
    heapsort(20000)    0.050
    matrix(20)         0.061
    nestedloop(12)     0.091
    sieve(30)          0.066
    strcat(200000)     0.009
    ------------------------
    Total              1.425
    real: 1.45s user: 1.43s sys: 0.01s cpu: 99% maxmem: 31404 KB cswaits: 3
    
    bench.php results from 3 runs
    1.361
    1.372
    1.425
    
    bench.php avg: 1.3860
    Avg: real: 1.41s user: 1.39s sys: 0.01s cpu: 99.00% maxmem: 31405.33KB cswaits: 4.67
    created results log at /home/phpbench_logs/bench_250218-194652.log
    server PHP info log at /home/phpbench_logs/bench_phpinfo_250218-194652.log
    
    -------------------------------------------
    Run PHP test Zend/micro_bench.php
    -------------------------------------------
    
    empty_loop         0.079
    func()             0.246    0.167
    undef_func()       0.254    0.175
    int_func()         0.218    0.140
    $x = self::$x      0.170    0.091
    self::$x = 0       0.167    0.088
    isset(self::$x)    0.163    0.084
    empty(self::$x)    0.166    0.087
    $x = Foo::$x       0.162    0.083
    Foo::$x = 0        0.147    0.068
    isset(Foo::$x)     0.146    0.067
    empty(Foo::$x)     0.155    0.076
    self::f()          0.269    0.190
    Foo::f()           0.243    0.164
    $x = $this->x      0.152    0.073
    $this->x = 0       0.186    0.107
    $this->x += 2      0.159    0.080
    ++$this->x         0.143    0.064
    --$this->x         0.140    0.061
    $this->x++         0.159    0.080
    $this->x--         0.171    0.092
    isset($this->x)    0.146    0.067
    empty($this->x)    0.164    0.085
    $this->f()         0.269    0.190
    $x = Foo::TEST     0.157    0.078
    new Foo()          0.465    0.386
    $x = TEST          0.119    0.040
    $x = $_GET         0.152    0.073
    $x = $GLOBALS['v'] 0.209    0.130
    $x = $hash['v']    0.165    0.086
    $x = $str[0]       0.224    0.145
    $x = $a ?: null    0.143    0.064
    $x = $f ?: tmp     0.197    0.118
    $x = $f ? $f : $a  0.153    0.074
    $x = $f ? $f : tmp 0.207    0.128
    ------------------------
    Total              6.465
    real: 6.49s user: 6.48s sys: 0.00s cpu: 99% maxmem: 19444 KB cswaits: 6
    
    empty_loop         0.078
    func()             0.247    0.168
    undef_func()       0.253    0.175
    int_func()         0.219    0.141
    $x = self::$x      0.170    0.091
    self::$x = 0       0.167    0.089
    isset(self::$x)    0.164    0.085
    empty(self::$x)    0.168    0.089
    $x = Foo::$x       0.151    0.073
    Foo::$x = 0        0.148    0.070
    isset(Foo::$x)     0.147    0.069
    empty(Foo::$x)     0.155    0.077
    self::f()          0.280    0.202
    Foo::f()           0.239    0.160
    $x = $this->x      0.149    0.071
    $this->x = 0       0.191    0.113
    $this->x += 2      0.159    0.081
    ++$this->x         0.142    0.063
    --$this->x         0.148    0.070
    $this->x++         0.159    0.081
    $this->x--         0.164    0.086
    isset($this->x)    0.146    0.067
    empty($this->x)    0.165    0.086
    $this->f()         0.269    0.190
    $x = Foo::TEST     0.157    0.079
    new Foo()          0.461    0.382
    $x = TEST          0.125    0.047
    $x = $_GET         0.152    0.074
    $x = $GLOBALS['v'] 0.209    0.131
    $x = $hash['v']    0.165    0.086
    $x = $str[0]       0.230    0.152
    $x = $a ?: null    0.144    0.065
    $x = $f ?: tmp     0.198    0.120
    $x = $f ? $f : $a  0.153    0.075
    $x = $f ? $f : tmp 0.207    0.129
    ------------------------
    Total              6.478
    real: 6.50s user: 6.50s sys: 0.00s cpu: 99% maxmem: 19444 KB cswaits: 4
    
    empty_loop         0.079
    func()             0.247    0.168
    undef_func()       0.253    0.175
    int_func()         0.219    0.140
    $x = self::$x      0.169    0.091
    self::$x = 0       0.167    0.089
    isset(self::$x)    0.163    0.085
    empty(self::$x)    0.166    0.088
    $x = Foo::$x       0.156    0.078
    Foo::$x = 0        0.147    0.068
    isset(Foo::$x)     0.155    0.076
    empty(Foo::$x)     0.155    0.076
    self::f()          0.265    0.187
    Foo::f()           0.239    0.160
    $x = $this->x      0.150    0.071
    $this->x = 0       0.187    0.108
    $this->x += 2      0.160    0.081
    ++$this->x         0.141    0.063
    --$this->x         0.140    0.062
    $this->x++         0.159    0.081
    $this->x--         0.164    0.086
    isset($this->x)    0.145    0.067
    empty($this->x)    0.165    0.086
    $this->f()         0.268    0.190
    $x = Foo::TEST     0.157    0.078
    new Foo()          0.462    0.383
    $x = TEST          0.119    0.041
    $x = $_GET         0.152    0.074
    $x = $GLOBALS['v'] 0.205    0.127
    $x = $hash['v']    0.165    0.086
    $x = $str[0]       0.218    0.140
    $x = $a ?: null    0.143    0.065
    $x = $f ?: tmp     0.205    0.126
    $x = $f ? $f : $a  0.153    0.074
    $x = $f ? $f : tmp 0.206    0.128
    ------------------------
    Total              6.446
    real: 6.47s user: 6.47s sys: 0.00s cpu: 99% maxmem: 19444 KB cswaits: 3
    
    micro_bench.php results from 3 runs
    6.465
    6.478
    6.446
    
    micro_bench.php avg: 6.4630
    Avg: real: 6.49s user: 6.48s sys: 0.00s cpu: 99.00% maxmem: 19444.00KB cswaits: 4.33
    created results log at /home/phpbench_logs/bench_micro_250218-194652.log
    server PHP info log at /home/phpbench_logs/bench_phpinfo_250218-194652.log
    
    

    Code (Text):
    ========================================================================
       BYTE UNIX Benchmarks (Version 5.1.3)
    
       System: hostname: GNU/Linux
       OS: GNU/Linux -- 3.10.0-693.17.1.el7.x86_64 -- #1 SMP Thu Jan 25 20:13:58 UTC 2018
       Machine: x86_64 (x86_64)
       Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
       CPU 0: Intel(R) Xeon(R) CPU E3-1578L v5 @ 2.00GHz (3984.0 bogomips)
              Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
       CPU 1: Intel(R) Xeon(R) CPU E3-1578L v5 @ 2.00GHz (3984.0 bogomips)
              Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
       CPU 2: Intel(R) Xeon(R) CPU E3-1578L v5 @ 2.00GHz (3984.0 bogomips)
              Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
       CPU 3: Intel(R) Xeon(R) CPU E3-1578L v5 @ 2.00GHz (3984.0 bogomips)
              Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
       CPU 4: Intel(R) Xeon(R) CPU E3-1578L v5 @ 2.00GHz (3984.0 bogomips)
              Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
       CPU 5: Intel(R) Xeon(R) CPU E3-1578L v5 @ 2.00GHz (3984.0 bogomips)
              Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
       CPU 6: Intel(R) Xeon(R) CPU E3-1578L v5 @ 2.00GHz (3984.0 bogomips)
              Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
       CPU 7: Intel(R) Xeon(R) CPU E3-1578L v5 @ 2.00GHz (3984.0 bogomips)
              Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
       20:07:52 up 54 min,  2 users,  load average: 3.27, 3.11, 1.64; runlevel 3
    

    Code (Text):
    ------------------------------------------------------------------------
    Benchmark Run: Sun Feb 25 2018 20:07:52 - 20:32:10
    8 CPUs in system; running 1 parallel copy of tests
    
    Dhrystone 2 using register variables       42338214.0 lps   (10.0 s, 7 samples)
    Double-Precision Whetstone                     4142.7 MWIPS (11.9 s, 7 samples)
    Execl Throughput                               3565.4 lps   (30.0 s, 2 samples)
    Pipe Throughput                             1269415.3 lps   (10.0 s, 7 samples)
    Pipe-based Context Switching                 251854.6 lps   (10.0 s, 7 samples)
    Process Creation                              13937.3 lps   (30.0 s, 2 samples)
    Shell Scripts (1 concurrent)                   8575.4 lpm   (60.0 s, 2 samples)
    Shell Scripts (16 concurrent)                  2266.5 lpm   (60.0 s, 2 samples)
    Shell Scripts (8 concurrent)                   3928.1 lpm   (60.0 s, 2 samples)
    System Call Overhead                        1079320.0 lps   (10.0 s, 7 samples)
    
    System Benchmarks Partial Index              BASELINE       RESULT    INDEX
    Dhrystone 2 using register variables         116700.0   42338214.0   3628.0
    Double-Precision Whetstone                       55.0       4142.7    753.2
    Execl Throughput                                 43.0       3565.4    829.2
    Pipe Throughput                               12440.0    1269415.3   1020.4
    Pipe-based Context Switching                   4000.0     251854.6    629.6
    Process Creation                                126.0      13937.3   1106.1
    Shell Scripts (1 concurrent)                     42.4       8575.4   2022.5
    Shell Scripts (16 concurrent)                     ---       2266.5      ---
    Shell Scripts (8 concurrent)                      6.0       3928.1   6546.8
    System Call Overhead                          15000.0    1079320.0    719.5
                                                                       ========
    System Benchmarks Index Score (Partial Only)                         1354.5
    

    Code (Text):
    ------------------------------------------------------------------------
    Benchmark Run: Sun Feb 25 2018 20:32:10 - 20:56:12
    8 CPUs in system; running 8 parallel copies of tests
    
    Dhrystone 2 using register variables      196525904.0 lps   (10.0 s, 7 samples)
    Double-Precision Whetstone                    33111.4 MWIPS (10.0 s, 7 samples)
    Execl Throughput                              26435.2 lps   (30.0 s, 2 samples)
    Pipe Throughput                             7459463.6 lps   (10.0 s, 7 samples)
    Pipe-based Context Switching                1553486.6 lps   (10.0 s, 7 samples)
    Process Creation                              85442.4 lps   (30.0 s, 2 samples)
    Shell Scripts (1 concurrent)                  38681.7 lpm   (60.0 s, 2 samples)
    Shell Scripts (16 concurrent)                  2703.6 lpm   (60.1 s, 2 samples)
    Shell Scripts (8 concurrent)                   5377.3 lpm   (60.0 s, 2 samples)
    System Call Overhead                        6366871.0 lps   (10.0 s, 7 samples)
    
    System Benchmarks Partial Index              BASELINE       RESULT    INDEX
    Dhrystone 2 using register variables         116700.0  196525904.0  16840.3
    Double-Precision Whetstone                       55.0      33111.4   6020.3
    Execl Throughput                                 43.0      26435.2   6147.7
    Pipe Throughput                               12440.0    7459463.6   5996.4
    Pipe-based Context Switching                   4000.0    1553486.6   3883.7
    Process Creation                                126.0      85442.4   6781.1
    Shell Scripts (1 concurrent)                     42.4      38681.7   9123.0
    Shell Scripts (16 concurrent)                     ---       2703.6      ---
    Shell Scripts (8 concurrent)                      6.0       5377.3   8962.2
    System Call Overhead                          15000.0    6366871.0   4244.6
                                                                       ========
    System Benchmarks Index Score (Partial Only)                         6871.6
    
    

    Code (Text):
    -------------------------------------------
    centminmodbench.sh completed
    -------------------------------------------
    
    centminmodbench.sh Total Run Time: 4151.409660621 seconds
    


    Current billing cost for testing :)

    type1e-billing-01.png

    GCC 8.0.1 Compile & RPM Build Times



    Centmin Mod LEMP Stack's Nginx web server supports multiple GCC/Clang version compilers from GCC 4.8, 5.3, 6.3, 7.2 and 8.x and Clang 3.4, 4, 5, and 6. Currently testing custom GCC 8.0.1 compilations and RPM builds so let's see how the Packet.net bare metal server holds up.

    It took a total of ~2716 seconds or ~45 minutes to compile and build both binutils 2.30 + GCC 8.0.1 with Profile Guided Optimisations and build the custom CentOS 7 RPMs.

    Code (Text):
    RPMs Built
    /svr-setup/binutils-gcc8-2.30-1.el7.x86_64.rpm
    /svr-setup/gcc8-pgo-8.0-1.el7.x86_64.rpm
    
    ls -lah /svr-setup | egrep 'gcc8-pgo-8.0|binutils-gcc8-2.30'
    -rw-r--r--  1 root      root       5.2M Feb 25 22:04 binutils-gcc8-2.30-1.el7.x86_64.rpm
    -rw-r--r--  1 root      root        39M Feb 25 22:47 gcc8-pgo-8.0-1.el7.x86_64.rpm
    
    Installed Packages
    Name        : binutils-gcc8
    Arch        : x86_64
    Version     : 2.30
    Release     : 1.el7
    Size        : 48 M
    Repo        : installed
    Summary     : binutils-gcc8 for centminmod.com LEMP stack installs
    URL         : https://centminmod.com
    License     : unknown
    Description : binutils-gcc8 for centminmod.com LEMP stacks
    
    * Sun Feb 25 2018 George Liu <centminmod.com> 2.30
    - Binutils 2.30 for centminmod.com LEMP stack installs
    

    Code (Text):
    Installed Packages
    Name        : gcc8-pgo
    Arch        : x86_64
    Version     : 8.0
    Release     : 1.el7
    Size        : 287 M
    Repo        : installed
    Summary     : gcc8-pgo for centminmod.com LEMP stack installs
    URL         : https://centminmod.com
    License     : unknown
    Description : gcc8-pgo for centminmod.com LEMP stacks
    
    
    * Sun Feb 25 2018 George Liu <centminmod.com> 8
    - GCC 8 for centminmod.com LEMP stack installs
    
    *************************************************
    Compile GCC Completed
    log: /root/centminlogs/tools-gcc-install-pgo_250218-220157.log
    *************************************************
    
    Total Binutils + GCC Install Time: 2716.625791497 seconds
    
     
    Last edited: Feb 27, 2018
  3. eva2000

    eva2000 Administrator Staff Member

    35,481
    7,828
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,067
    Local Time:
    6:41 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x

    Packet.net TYPE 2A - Cavium ThunderX CN8890 @2.0Ghz



    Next up is testing the TYPE 2A bare metal server for a 96 physical core 2x Cavium ThunderX CN8890 ARMv8 64bit processor running at 2.0Ghz using spot bidding pricing at US$0.20/hr on current market rate of US$0.10/hr (on demand pricing is US$0.50/hr) for Centmin Mod 123.09beta01 install testing and centminmodbench.sh testing.

    System
    • 96 Physical Cores @ 2.0 GHz (2 × Cavium ThunderX)
    • 128 GB of DDR4 ECC RAM
    • 1x 340 GB of SSD
    • 20Gbps Bonded Network
    • CentOS 7.4 64bit
    • New Jersey
    type2a-hardware-01.png

    Cavium ThunderX CN8890 Details



    Not much details but 96 cpu cores, 48 per socket in a 2 socket - 2 Numa node configuration.

    Code (Text):
    lscpu
    Architecture:          aarch64
    Byte Order:            Little Endian
    CPU(s):                96
    On-line CPU(s) list:   0-95
    Thread(s) per core:    1
    Core(s) per socket:    48
    Socket(s):             2
    NUMA node(s):          2
    NUMA node0 CPU(s):     0-47
    NUMA node1 CPU(s):     48-95
    


    cat /proc/cpuinfo output for 96 cores
    Code (Text):
    cat /proc/cpuinfo
    processor       : 0
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 1
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 2
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 3
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 4
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 5
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 6
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 7
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 8
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 9
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 10
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 11
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 12
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 13
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 14
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 15
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 16
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 17
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 18
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 19
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 20
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 21
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 22
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 23
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 24
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 25
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 26
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 27
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 28
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 29
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 30
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 31
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 32
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 33
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 34
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 35
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 36
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 37
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 38
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 39
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 40
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 41
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 42
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 43
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 44
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 45
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 46
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 47
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 48
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 49
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 50
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 51
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 52
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 53
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 54
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 55
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 56
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 57
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 58
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 59
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 60
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 61
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 62
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 63
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 64
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 65
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 66
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 67
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 68
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 69
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 70
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 71
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 72
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 73
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 74
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 75
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 76
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 77
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 78
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 79
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 80
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 81
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 82
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 83
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 84
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 85
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 86
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 87
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 88
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 89
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 90
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 91
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 92
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 93
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 94
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1
    
    processor       : 95
    BogoMIPS        : 200.00
    Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
    CPU implementer : 0x43
    CPU architecture: 8
    CPU variant     : 0x1
    CPU part        : 0x0a1
    CPU revision    : 1

    128GB memory
    Code (Text):
    free -m
                  total        used        free      shared  buff/cache   available
    Mem:         128719         649      127363          29         706      127225
    Swap:             0           0           0
    

    disk config
    Code (Text):
    df -hT
    Filesystem     Type      Size  Used Avail Use% Mounted on
    devtmpfs       devtmpfs   63G     0   63G   0% /dev
    tmpfs          tmpfs      63G     0   63G   0% /dev/shm
    tmpfs          tmpfs      63G   30M   63G   1% /run
    tmpfs          tmpfs      63G     0   63G   0% /sys/fs/cgroup
    /dev/sda3      ext4      227G  918M  214G   1% /
    /dev/sda1      vfat      500M  368K  499M   1% /boot/efi
    tmpfs          tmpfs      13G     0   13G   0% /run/user/0
    


    Centmin Mod 123.09beta01 Install



    Centmin Mod install routine is optimised to detect Intel processors for further Intel related tuned performance. So not sure how it will do with 2x Cavium ThunderX CN8890 ARMv8 64bit based processors. This is what's great about Packet.net hourly billed bare metal - for developers like me, testing such bare metal servers to develop Centmin Mod support for these higher end processors can be done relatively cheaper than renting and committing to a full month !

    Used default curl installer command below
    Code (Text):
    yum -y update; curl -O https://centminmod.com/betainstaller.sh && chmod 0700 betainstaller.sh && bash betainstaller.sh
    

    Looks like the install failed as some YUM repos including Remi and MariaDB official YUM repo Centmin Mod sets up doesn't seem to be for ARMv8 64bit based Cavium ThunderX. In a Intel dominant server world, most YUM repos are for x86_64 architecture and not AMRv8 which would be aarch64 architecture.

    not Intel x86_64 based
    Code (Text):
    uname -m
    aarch64
    


    Will have to inspect the full initial install log to see where the compatibility issues are :D

    Edit: there is no official MariaDB ARM aarch64 supported YUM repository for CentOS MariaDB - Setting up MariaDB Repositories - MariaDB :(

    Seems like it would be cheaper to do ARMv8 development/testing working on Scaleway's ARMv8 based cloud VPS servers instead though.
     
    Last edited: Feb 27, 2018
  4. eva2000

    eva2000 Administrator Staff Member

    35,481
    7,828
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,067
    Local Time:
    6:41 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x

    Packet.net AMD EPYC 7401P c2.medium



    Packet.net has rolled out AMD EPYC 7401P under the plan name = c2.medium in Sunnyvale Califorina and Parsippany, New Jersey at US$1.00/hr ondemand pricing. AMD EPYC 7401 official product page lists base clock at 2.0Ghz, with all core turbo boost at 2.8Ghz and max turbo boost at 3Ghz. Using CentOS 7.x default 3.10 Linux Kernel may not allow AMD EPYC cpus to fully shine as alot of AMD EPYC/Ryzen performance improvements landed in upstream 4.10+ Linux Kernels.

    packetnet-amd-epyc-c2medium-01.png packetnet-amd-epyc-c2medium-02.png
    Also despite officially listing just Ubuntu as default supported OS, it looks like CentOS 7 is available

    packetnet-amd-epyc-c2medium-03.png

    System:
    • 24 Physical Cores @ 2.2 GHz (1 x AMD EPYC 7401P)
    • 64 GB of ECC RAM
    • 1 x 120 GB boot SSD and 1x 120 GB unformatted
    • 960 GB of SSD (2 x 480 GB)
    • 20 Gbps Bonded Network
    • (2 x Mellanox 10Gbps w/ LACP)
    • CentOS 7.4 64bit
    • Parsippany, New Jersey

    Packet.net AMD EPYC 7401P Details



    Web site lists 1x 120GB Boot SSD but the management area and fdisk list 2x 120GB SSD - 1x 120GB for boot and other 1x 120GB unformatted and web site lists 980GB but it's 2x480GB = 960GB and comes unformatted ready for you to format. The 120GB SSD boot disk are relatively slow so probably the intention would be instead of using default CentOS 7 profile, is to install via iPXE with custom partition configuration for CentOS 7. I can see probably using the 2x480GB SSD in raid 1 as boot. But then what to do with the 2x120GB SSDs ? Maybe better for Packet.net to just switch the offering to 4x480GB SSD and can configure 2x 480GB SSD raid 1 by default for CentOS 7 selected OS?

    Code (Text):
    free -m
                  total        used        free      shared  buff/cache   available
    Mem:          64159        1141       62809          16         208       62597
    Swap:          1949           0        1949
    

    Code (Text):
    df -hT
    Filesystem     Type      Size  Used Avail Use% Mounted on
    /dev/sdc3      ext4      108G  1.3G  101G   2% /
    devtmpfs       devtmpfs   32G     0   32G   0% /dev
    tmpfs          tmpfs      32G     0   32G   0% /dev/shm
    tmpfs          tmpfs      32G   17M   32G   1% /run
    tmpfs          tmpfs      32G     0   32G   0% /sys/fs/cgroup
    /dev/sdc1      vfat      511M  148K  511M   1% /boot/efi
    tmpfs          tmpfs     6.3G     0  6.3G   0% /run/user/0
    

    Code (Text):
    fdisk -l
    WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.
    
    Disk /dev/sdc: 120.0 GB, 120034123776 bytes, 234441648 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 4096 bytes
    I/O size (minimum/optimal): 4096 bytes / 4096 bytes
    Disk label type: gpt
    Disk identifier: F2619121-B58D-434C-9A0C-FC200530EAB9
    
    
    #         Start          End    Size  Type            Name
     1         2048      1050623    512M  EFI System      BIOS
     2      1050624      5044223    1.9G  Linux filesyste SWAP
     3      5044224    234441614  109.4G  Linux filesyste ROOT
    
    Disk /dev/sda: 480.1 GB, 480103981056 bytes, 937703088 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 4096 bytes
    I/O size (minimum/optimal): 4096 bytes / 4096 bytes
    
    
    Disk /dev/sdb: 480.1 GB, 480103981056 bytes, 937703088 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 4096 bytes
    I/O size (minimum/optimal): 4096 bytes / 4096 bytes
    
    
    Disk /dev/sdd: 120.0 GB, 120034123776 bytes, 234441648 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 4096 bytes
    I/O size (minimum/optimal): 4096 bytes / 4096 bytes
    

    Yes there's 48 of these !

    packet-amd-epyc-carbon-03.png
    Code (Text):
    lscpu
    Architecture:          x86_64
    CPU op-mode(s):        32-bit, 64-bit
    Byte Order:            Little Endian
    CPU(s):                48
    On-line CPU(s) list:   0-47
    Thread(s) per core:    2
    Core(s) per socket:    24
    Socket(s):             1
    NUMA node(s):          4
    Vendor ID:             AuthenticAMD
    CPU family:            23
    Model:                 1
    Model name:            AMD EPYC 7401P 24-Core Processor
    Stepping:              2
    CPU MHz:               1996.203
    BogoMIPS:              3992.40
    Virtualization:        AMD-V
    L1d cache:             32K
    L1i cache:             64K
    L2 cache:              512K
    L3 cache:              8192K
    NUMA node0 CPU(s):     0,4,8,12,16,20,24,28,32,36,40,44
    NUMA node1 CPU(s):     1,5,9,13,17,21,25,29,33,37,41,45
    NUMA node2 CPU(s):     2,6,10,14,18,22,26,30,34,38,42,46
    NUMA node3 CPU(s):     3,7,11,15,19,23,27,31,35,39,43,47
    Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    

    All 24 core / 48 cpu threads from /proc/cpuinfo output

    Code (Text):
    cat /proc/cpuinfo
    processor       : 0
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 0
    cpu cores       : 24
    apicid          : 0
    initial apicid  : 0
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 1
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 8
    cpu cores       : 24
    apicid          : 16
    initial apicid  : 16
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 2
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 16
    cpu cores       : 24
    apicid          : 32
    initial apicid  : 32
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 3
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 24
    cpu cores       : 24
    apicid          : 48
    initial apicid  : 48
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 4
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 4
    cpu cores       : 24
    apicid          : 8
    initial apicid  : 8
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 5
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 12
    cpu cores       : 24
    apicid          : 24
    initial apicid  : 24
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 6
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 20
    cpu cores       : 24
    apicid          : 40
    initial apicid  : 40
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 7
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 28
    cpu cores       : 24
    apicid          : 56
    initial apicid  : 56
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 8
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 1
    cpu cores       : 24
    apicid          : 2
    initial apicid  : 2
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 9
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 9
    cpu cores       : 24
    apicid          : 18
    initial apicid  : 18
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 10
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 17
    cpu cores       : 24
    apicid          : 34
    initial apicid  : 34
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 11
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 25
    cpu cores       : 24
    apicid          : 50
    initial apicid  : 50
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 12
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 5
    cpu cores       : 24
    apicid          : 10
    initial apicid  : 10
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 13
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 13
    cpu cores       : 24
    apicid          : 26
    initial apicid  : 26
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 14
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 21
    cpu cores       : 24
    apicid          : 42
    initial apicid  : 42
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 15
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 29
    cpu cores       : 24
    apicid          : 58
    initial apicid  : 58
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 16
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 2
    cpu cores       : 24
    apicid          : 4
    initial apicid  : 4
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 17
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 10
    cpu cores       : 24
    apicid          : 20
    initial apicid  : 20
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 18
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 18
    cpu cores       : 24
    apicid          : 36
    initial apicid  : 36
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 19
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 26
    cpu cores       : 24
    apicid          : 52
    initial apicid  : 52
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 20
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 6
    cpu cores       : 24
    apicid          : 12
    initial apicid  : 12
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 21
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 14
    cpu cores       : 24
    apicid          : 28
    initial apicid  : 28
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 22
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 22
    cpu cores       : 24
    apicid          : 44
    initial apicid  : 44
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 23
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 30
    cpu cores       : 24
    apicid          : 60
    initial apicid  : 60
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 24
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 0
    cpu cores       : 24
    apicid          : 1
    initial apicid  : 1
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 25
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 8
    cpu cores       : 24
    apicid          : 17
    initial apicid  : 17
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 26
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 16
    cpu cores       : 24
    apicid          : 33
    initial apicid  : 33
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 27
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 24
    cpu cores       : 24
    apicid          : 49
    initial apicid  : 49
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 28
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 4
    cpu cores       : 24
    apicid          : 9
    initial apicid  : 9
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 29
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 12
    cpu cores       : 24
    apicid          : 25
    initial apicid  : 25
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 30
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 20
    cpu cores       : 24
    apicid          : 41
    initial apicid  : 41
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 31
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 28
    cpu cores       : 24
    apicid          : 57
    initial apicid  : 57
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 32
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 1
    cpu cores       : 24
    apicid          : 3
    initial apicid  : 3
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 33
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 9
    cpu cores       : 24
    apicid          : 19
    initial apicid  : 19
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 34
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 17
    cpu cores       : 24
    apicid          : 35
    initial apicid  : 35
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 35
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 25
    cpu cores       : 24
    apicid          : 51
    initial apicid  : 51
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 36
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 5
    cpu cores       : 24
    apicid          : 11
    initial apicid  : 11
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 37
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 13
    cpu cores       : 24
    apicid          : 27
    initial apicid  : 27
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 38
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 21
    cpu cores       : 24
    apicid          : 43
    initial apicid  : 43
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 39
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 29
    cpu cores       : 24
    apicid          : 59
    initial apicid  : 59
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 40
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 2
    cpu cores       : 24
    apicid          : 5
    initial apicid  : 5
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 41
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 10
    cpu cores       : 24
    apicid          : 21
    initial apicid  : 21
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 42
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 18
    cpu cores       : 24
    apicid          : 37
    initial apicid  : 37
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 43
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 26
    cpu cores       : 24
    apicid          : 53
    initial apicid  : 53
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 44
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 6
    cpu cores       : 24
    apicid          : 13
    initial apicid  : 13
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 45
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 14
    cpu cores       : 24
    apicid          : 29
    initial apicid  : 29
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 46
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 22
    cpu cores       : 24
    apicid          : 45
    initial apicid  : 45
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    
    processor       : 47
    vendor_id       : AuthenticAMD
    cpu family      : 23
    model           : 1
    model name      : AMD EPYC 7401P 24-Core Processor
    stepping        : 2
    microcode       : 0x8001213
    cpu MHz         : 1996.203
    cache size      : 512 KB
    physical id     : 0
    siblings        : 48
    core id         : 30
    cpu cores       : 24
    apicid          : 61
    initial apicid  : 61
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    bogomips        : 3992.40
    TLB size        : 2560 4K pages
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 48 bits physical, 48 bits virtual
    power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
    


    Centmin Mod 123.09beta01 Install



    Centmin Mod install routine is optimised to detect Intel processors for further Intel related tuned performance so definitely will need to in future start optimising Centmin Mod detection routines to detect AMD EPYC and Packet.net hourly billed bare metal - for developers like me, make testing such higher end AMD EPYCprocessors relatively cheaper than renting and committing to a full month !

    Used default curl installer command below
    Code (Text):
    yum -y update; curl -O https://centminmod.com/betainstaller.sh && chmod 0700 betainstaller.sh && bash betainstaller.sh
    

    Looks like install times were let down by slower YUM related downloads (+245 seconds extra for YUM related installs) so will need to investigate while actual Nginx and PHP-FPM source compile times are relative fast but not faster than Packet.net TYPE 1E Intel Xeon E3-1578Lv5 Skylake tests. Expected as AMD EPYC 7401P is clocked lower with base clock at 2.0Ghz, with all core turbo boost at 2.8Ghz and max turbo boost at 3Ghz and using slower single 1x 120GB boot disk compared to E3-1578Lv5 setup with 1x 240GB SSD. You can check out the centminmodbench.sh disk I/O performance benchmarks further below to see how much slower the 120GB SSD boot disk is. But don't judge just Centmin Mod initial install times for overall performance. AMD EPYC 7401P's 24 core / 48 cpu threads and massive 64MB L3 Cache will enable much more multi threaded performance work benefits compared to the typical 4 core / 8 cpu thread servers. Just check out the below centminmodbench.sh OpenSSL benchmark numbers to see how far ahead AMD EPYC 7401P is.

    You can compare that to other times posted at Install - Post your Centmin Mod 123.09beta01 install time stats. Though at as Feb 24th, 2018, the 123.09beta01 installed changed some defaults to more performance at the expensive of initial install time.

    AMD EPYC 7401P Centmin Mod 123.09beta01 Install Times
    Code (Text):
    ---------------------------------------------------------------------------
    Total Curl Installer YUM or DNF Time: 340.5943 seconds
    Total YUM Time: 60.142379961 seconds
    Total YUM or DNF + Source Download Time: 86.9888
    Total Nginx First Time Install Time: 182.0339
    Total PHP First Time Install Time: 183.7518
    Download Zip From Github Time: 1.6182
    Total Time Other eg. source compiles: 418.1967
    Total Centmin Mod Install Time: 870.9712
    ---------------------------------------------------------------------------
    Total Install Time (curl yum + cm install + zip download): 1213.1837 seconds
    ---------------------------------------------------------------------------
    

    compared to Intel Xeon E3-1578Lv5 Skylake Centmin Mod 123.09beta01 Install Times
    Code (Text):
    ---------------------------------------------------------------------------
    Total Curl Installer YUM or DNF Time: 91.2912 seconds
    Total YUM Time: 64.039117358 seconds
    Total YUM or DNF + Source Download Time: 87.3015
    Total Nginx First Time Install Time: 135.2867
    Total PHP First Time Install Time: 141.0516
    Download Zip From Github Time: 1.8695
    Total Time Other eg. source compiles: 264.2750
    Total Centmin Mod Install Time: 627.9147
    ---------------------------------------------------------------------------
    Total Install Time (curl yum + cm install + zip download): 721.0754 seconds
    ---------------------------------------------------------------------------
    

    AMD EPYC 7401P Centmin Mod 123.09beta01 post install check list
    Code (Text):
    *************************************************
    * Post-Install Check List....
    *************************************************
    
    --------------------------------------------------------
    Check ccache Version:
    --------------------------------------------------------
    ccache version 3.4.1
    
    Copyright (C) 2002-2007 Andrew Tridgell
    Copyright (C) 2009-2018 Joel Rosdahl
    
    This program is free software; you can redistribute it and/or modify it under
    the terms of the GNU General Public License as published by the Free Software
    Foundation; either version 3 of the License, or (at your option) any later
    version.
    
    --------------------------------------------------------
    Check Nginx Version:
    --------------------------------------------------------
    nginx version: nginx/1.13.9
    built by gcc 7.2.1 20170829 (Red Hat 7.2.1-1) (GCC)
    built with OpenSSL 1.1.0g  2 Nov 2017
    TLS SNI support enabled
    configure arguments: --with-ld-opt='-ljemalloc -Wl,-z,relro -Wl,-rpath,/usr/local/lib' --with-cc-opt='-g -O3 -fstack-protector-strong -flto -fuse-ld=gold --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wimplicit-fallthrough=0 -fcode-hoisting -Wp,-D_FORTIFY_SOURCE=2 -gsplit-dwarf' --sbin-path=/usr/local/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --with-compat --with-http_stub_status_module --with-http_secure_link_module --with-libatomic --with-http_gzip_static_module --with-http_sub_module --with-http_addition_module --with-http_image_filter_module=dynamic --with-http_geoip_module --with-stream_geoip_module --with-stream_realip_module --with-stream_ssl_preread_module --with-threads --with-stream=dynamic --with-stream_ssl_module --with-http_realip_module --add-dynamic-module=../ngx-fancyindex-0.4.2 --add-module=../ngx_cache_purge-2.4.2 --add-module=../ngx_devel_kit-0.3.0 --add-dynamic-module=../set-misc-nginx-module-0.31 --add-dynamic-module=../echo-nginx-module-0.61 --add-module=../redis2-nginx-module-0.14 --add-module=../ngx_http_redis-0.3.7 --add-module=../memc-nginx-module-0.18 --add-module=../srcache-nginx-module-0.31 --add-dynamic-module=../headers-more-nginx-module-0.33 --with-pcre=../pcre-8.41 --with-pcre-jit --with-zlib=../zlib-cloudflare-1.3.0 --with-http_ssl_module --with-http_v2_module --with-openssl=../openssl-1.1.0g --with-openssl-opt='enable-ec_nistp_64_gcc_128'
    
    --------------------------------------------------------
    Check PHP-FPM Version:
    --------------------------------------------------------
    PHP 5.6.33 (cli) (built: Feb 26 2018 08:24:24)
    Copyright (c) 1997-2016 The PHP Group
    Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
        with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2016, by Zend Technologies
    
    --------------------------------------------------------
    Check MariaDB installed RPM Versions:
    --------------------------------------------------------
    MariaDB-devel-10.1.31-1.el7.centos.x86_64
    MariaDB-client-10.1.31-1.el7.centos.x86_64
    MariaDB-server-10.1.31-1.el7.centos.x86_64
    MariaDB-shared-10.1.31-1.el7.centos.x86_64
    MariaDB-common-10.1.31-1.el7.centos.x86_64
    MariaDB-compat-10.1.31-1.el7.centos.x86_64
    
    --------------------------------------------------------
    Check Memcached Server Version:
    --------------------------------------------------------
    memcached 1.5.5
    
    --------------------------------------------------------
    Check CSF Firewall Version:
    --------------------------------------------------------
    csf: v11.07 (generic)
    
    --------------------------------------------------------
    Check Siege Benchmark Version:
    --------------------------------------------------------
    SIEGE 4.0.4
    
    Copyright (C) 2017 by Jeffrey Fulmer, et al.
    This is free software; see the source for copying conditions.
    There is NO warranty; not even for MERCHANTABILITY or FITNESS
    FOR A PARTICULAR PURPOSE.
    
    
    --------------------------------------------------------
    Check mysqlreport version:
    --------------------------------------------------------
    mysqlreport v3.5-maria11 Jul 4 2013
    
    --------------------------------------------------------
    Check pure-ftpd Version:
    --------------------------------------------------------
    pure-ftpd v1.0.42
    
    --------------------------------------------------------
    Check YUM Repo List
    --------------------------------------------------------
    Loaded plugins: fastestmirror, priorities, versionlock
    Loading mirror speeds from cached hostfile
     * base: mirrors.centos.webair.com
     * epel: fedora-epel.mirrors.tds.net
     * extras: mirrors.centos.webair.com
     * rpmforge: mirror.team-cymru.org
     * updates: mirrors.centos.webair.com
    228 packages excluded due to repository priority protections
    repo id                        repo name                             status
    base/7/x86_64                  CentOS-7 - Base                       7,420+2,171
    centos-sclo-rh/x86_64          CentOS-7 - SCLo rh                       6,546+94
    centos-sclo-sclo/x86_64        CentOS-7 - SCLo sclo                          500
    epel/x86_64                    Extra Packages for Enterprise Linux 7  11,376+923
    extras/7/x86_64                CentOS-7 - Extras                             388
    mariadb                        MariaDB                                      15+2
    rpmforge                       RHEL 7 - RPMforge.net - dag               139+106
    updates/7/x86_64               CentOS-7 - Updates                      1,499+430
    !varnishcache_varnish41/x86_64 varnishcache_varnish41                         46
    !varnishcache_varnish41-source varnishcache_varnish41-source                   0
    repolist: 27,929
    
    --------------------------------------------------------
    mytimes check:
    /usr/bin/mytimes
    --------------------------------------------------------
    Mon Feb 26 08:29:14 UTC 2018    [UTC]
    Mon Feb 26 18:29:14 AEST 2018   [Australia/Brisbane]
    Mon Feb 26 00:29:14 PST 2018    [America/Los_Angeles]
    Mon Feb 26 02:29:14 CST 2018    [America/Chicago]
    Mon Feb 26 03:29:14 EST 2018    [America/New_York]
    Mon Feb 26 03:29:14 EST 2018    [America/Montreal]
    Mon Feb 26 08:29:14 GMT 2018    [Europe/London]
    Mon Feb 26 09:29:14 CET 2018    [Europe/Berlin]
    Mon Feb 26 15:29:14 +07 2018    [Asia/Bangkok]
    Mon Feb 26 15:29:14 +07 2018    [Asia/Ho_Chi_Minh]
    Mon Feb 26 15:29:14 WIB 2018    [Asia/Jakarta]
    Mon Feb 26 16:29:14 +08 2018    [Asia/Kuala_Lumpur]
    Mon Feb 26 16:29:14 +08 2018    [Asia/Singapore]
    --------------------------------------------------------
    


    centminmodbench.sh results



    All new VPS/Dedicated servers I get, I will run my centminmodbench.sh benchmark script at bench.centminmod.com to fully evaluate the performance of the server. The results are below.
    • The beast of the AMD EPYC 7401P processor was definitely let down by the slow 120GB SSD boot disk which only managed FIO random read/writes of less than 170MB/s and 43K/37K IOPs. Ioping measured sequential disk I/O at very low 518 IOPs and 129.6MB/s.
    • Memory bandwidth from ramdisk tests were also pretty low around 2.7GB/s for DD tests
    • Network bandwidth for wget single threaded tests were relatively slower at ~50MB/s for US East Coast. The Axel multi-threaded bandwidth tests were a bit better for US East Coast ranging from 113-168MB/s
    • UnixBench was very lower for single index = 386.5 and 48 cpus index at 7478. Maybe 3.10 Linux Kernel related ?
    • OpenSSL numbers as expected are much better with AMD EPYC 7401P hardware AES-NI assisted aes256 results being ~4.57x times faster and ecdsa sign/s being ~4.4x times faster than above Intel Xeon E3-1578Lv5 Skylake.

    Code (Text):
    -------------------------------------------
    centminmodbench.sh 0.9.5
    http://bench.centminmod.com
    written by: George Liu (eva2000)
    http://centminmod.com
    -------------------------------------------
    
    -------------------------------------------
    System Information
    -------------------------------------------
    
    3.10.0-693.17.1.el7.x86_64
    
    CentOS Linux release 7.4.1708 (Core)
    
    Centmin Mod
    ----------------------------------------------
    Architecture:          x86_64
    CPU op-mode(s):        32-bit, 64-bit
    Byte Order:            Little Endian
    CPU(s):                48
    On-line CPU(s) list:   0-47
    Thread(s) per core:    2
    Core(s) per socket:    24
    Socket(s):             1
    NUMA node(s):          4
    Vendor ID:             AuthenticAMD
    CPU family:            23
    Model:                 1
    Model name:            AMD EPYC 7401P 24-Core Processor
    Stepping:              2
    CPU MHz:               1996.203
    BogoMIPS:              3992.40
    Virtualization:        AMD-V
    L1d cache:             32K
    L1i cache:             64K
    L2 cache:              512K
    L3 cache:              8192K
    NUMA node0 CPU(s):     0,4,8,12,16,20,24,28,32,36,40,44
    NUMA node1 CPU(s):     1,5,9,13,17,21,25,29,33,37,41,45
    NUMA node2 CPU(s):     2,6,10,14,18,22,26,30,34,38,42,46
    NUMA node3 CPU(s):     3,7,11,15,19,23,27,31,35,39,43,47
    Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    
    CPU Flags
     fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate ibpb_support avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov succor smca
    
    ----------------------------------------------
    CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE
    0   0    0      0    0:0:0:0       yes
    1   1    0      1    1:1:1:1       yes
    2   2    0      2    2:2:2:2       yes
    3   3    0      3    3:3:3:3       yes
    4   0    0      4    4:4:4:4       yes
    5   1    0      5    5:5:5:5       yes
    6   2    0      6    6:6:6:6       yes
    7   3    0      7    7:7:7:7       yes
    8   0    0      8    8:8:8:0       yes
    9   1    0      9    9:9:9:1       yes
    10  2    0      10   10:10:10:2    yes
    11  3    0      11   11:11:11:3    yes
    12  0    0      12   12:12:12:4    yes
    13  1    0      13   13:13:13:5    yes
    14  2    0      14   14:14:14:6    yes
    15  3    0      15   15:15:15:7    yes
    16  0    0      16   16:16:16:0    yes
    17  1    0      17   17:17:17:1    yes
    18  2    0      18   18:18:18:2    yes
    19  3    0      19   19:19:19:3    yes
    20  0    0      20   20:20:20:4    yes
    21  1    0      21   21:21:21:5    yes
    22  2    0      22   22:22:22:6    yes
    23  3    0      23   23:23:23:7    yes
    24  0    0      0    0:0:0:0       yes
    25  1    0      1    1:1:1:1       yes
    26  2    0      2    2:2:2:2       yes
    27  3    0      3    3:3:3:3       yes
    28  0    0      4    4:4:4:4       yes
    29  1    0      5    5:5:5:5       yes
    30  2    0      6    6:6:6:6       yes
    31  3    0      7    7:7:7:7       yes
    32  0    0      8    8:8:8:0       yes
    33  1    0      9    9:9:9:1       yes
    34  2    0      10   10:10:10:2    yes
    35  3    0      11   11:11:11:3    yes
    36  0    0      12   12:12:12:4    yes
    37  1    0      13   13:13:13:5    yes
    38  2    0      14   14:14:14:6    yes
    39  3    0      15   15:15:15:7    yes
    40  0    0      16   16:16:16:0    yes
    41  1    0      17   17:17:17:1    yes
    42  2    0      18   18:18:18:2    yes
    43  3    0      19   19:19:19:3    yes
    44  0    0      20   20:20:20:4    yes
    45  1    0      21   21:21:21:5    yes
    46  2    0      22   22:22:22:6    yes
    47  3    0      23   23:23:23:7    yes
    
    ----------------------------------------------
                  total        used        free      shared  buff/cache   available
    Mem:          64159        3013       60590          30         555       60729
    Low:          64159        3568       60590
    High:             0           0           0
    Swap:          1949           0        1949
    
    ----------------------------------------------
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/sdc3       108G  6.2G   96G   7% /
    devtmpfs         32G  4.0K   32G   1% /dev
    tmpfs            32G     0   32G   0% /dev/shm
    tmpfs            32G   17M   32G   1% /run
    tmpfs            32G     0   32G   0% /sys/fs/cgroup
    /dev/sdc1       511M  148K  511M   1% /boot/efi
    tmpfs           6.3G     0  6.3G   0% /run/user/0
    tmpfs            32G  332K   32G   1% /tmp
    
    

    Code (Text):
    -------------------------------------------
    disk ioping tests
    -------------------------------------------
    
    Running IOPing I/O benchmark...
    
    IOPing I/O: ./ioping -c 10 .
    4.0 KiB from . (ext4 /dev/sdc3): request=1 time=2.2 ms
    4.0 KiB from . (ext4 /dev/sdc3): request=2 time=227 us
    4.0 KiB from . (ext4 /dev/sdc3): request=3 time=272 us
    4.0 KiB from . (ext4 /dev/sdc3): request=4 time=249 us
    4.0 KiB from . (ext4 /dev/sdc3): request=5 time=213 us
    4.0 KiB from . (ext4 /dev/sdc3): request=6 time=267 us
    4.0 KiB from . (ext4 /dev/sdc3): request=7 time=269 us
    4.0 KiB from . (ext4 /dev/sdc3): request=8 time=243 us
    4.0 KiB from . (ext4 /dev/sdc3): request=9 time=251 us
    4.0 KiB from . (ext4 /dev/sdc3): request=10 time=214 us
    
    --- . (ext4 /dev/sdc3) ioping statistics ---
    10 requests completed in 9.0 s, 2.3 k iops, 8.8 MiB/s
    min/avg/max/mdev = 213 us / 442 us / 2.2 ms / 591 us
    
    IOPing seek rate: ./ioping -RD .
    
    --- . (ext4 /dev/sdc3) ioping statistics ---
    15.9 k requests completed in 3.0 s, 5.6 k iops, 21.7 MiB/s
    min/avg/max/mdev = 150 us / 180 us / 1.9 ms / 33 us
    
    IOPing sequential: ./ioping -RL .
    
    --- . (ext4 /dev/sdc3) ioping statistics ---
    1.5 k requests completed in 3.0 s, 518 iops, 129.6 MiB/s
    min/avg/max/mdev = 1.1 ms / 1.9 ms / 4.9 ms / 353 us
    
    IOPing cached: ./ioping -RC .
    
    --- . (ext4 /dev/sdc3) ioping statistics ---
    645.6 k requests completed in 3.0 s, 228.2 k iops, 891.3 MiB/s
    min/avg/max/mdev = 3 us / 4 us / 43 us / 0 us
    
    

    Code (Text):
    -------------------------------------------
    disk DD tests
    -------------------------------------------
    
    dd if=/dev/zero of=sb-io-test bs=1M count=1k conv=fdatasync
    1024+0 records in
    1024+0 records out
    1073741824 bytes (1.1 GB) copied, 6.02867 s, 178 MB/s
    
    dd if=/dev/zero of=sb-io-test bs=64k count=16k conv=fdatasync
    16384+0 records in
    16384+0 records out
    1073741824 bytes (1.1 GB) copied, 6.12825 s, 175 MB/s
    
    dd if=/dev/zero of=sb-io-test bs=1M count=1k oflag=dsync
    1024+0 records in
    1024+0 records out
    1073741824 bytes (1.1 GB) copied, 5.65778 s, 190 MB/s
    
    dd if=/dev/zero of=sb-io-test bs=64k count=16k oflag=dsync
    16384+0 records in
    16384+0 records out
    1073741824 bytes (1.1 GB) copied, 11.2722 s, 95.3 MB/s
    

    Code (Text):
    -------------------------------------------
    disk FIO tests
    -------------------------------------------
    
    Running FIO benchmark...
    
    FIO_VERSION = fio-2.0.9
    
    FIO random reads:
    randomreads: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
    fio-2.0.9
    Starting 1 process
    randomreads: Laying out IO file(s) (1 file(s) / 1024MB)
    
    randomreads: (groupid=0, jobs=1): err= 0: pid=67001: Mon Feb 26 08:46:57 2018
      read : io=1024.3MB, bw=172760KB/s, iops=43190 , runt=  6071msec
      cpu          : usr=8.95%, sys=58.42%, ctx=23870, majf=0, minf=251
      IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
         submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
         complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
         issued    : total=r=262207/w=0/d=0, short=r=0/w=0/d=0
    
    Run status group 0 (all jobs):
       READ: io=1024.3MB, aggrb=172760KB/s, minb=172760KB/s, maxb=172760KB/s, mint=6071msec, maxt=6071msec
    
    Disk stats (read/write):
      sdc: ios=254729/2, merge=51/1, ticks=352718/22, in_queue=352676, util=98.37%
    
    FIO random writes:
    randomwrites: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
    fio-2.0.9
    Starting 1 process
    
    randomwrites: (groupid=0, jobs=1): err= 0: pid=67005: Mon Feb 26 08:47:04 2018
      write: io=1024.3MB, bw=148307KB/s, iops=37076 , runt=  7072msec
      cpu          : usr=7.00%, sys=50.12%, ctx=20903, majf=0, minf=185
      IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
         submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
         complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
         issued    : total=r=0/w=262207/d=0, short=r=0/w=0/d=0
    
    Run status group 0 (all jobs):
      WRITE: io=1024.3MB, aggrb=148307KB/s, minb=148307KB/s, maxb=148307KB/s, mint=7072msec, maxt=7072msec
    
    Disk stats (read/write):
      sdc: ios=0/255776, merge=0/55, ticks=0/417556, in_queue=417513, util=98.63%
    
    

    Code (Text):
    -------------------------------------------
    ramdisk DD tests - memory bandwidth
    -------------------------------------------
    
    dd if=/dev/zero of=sb-io-test bs=128k count=1k conv=fdatasync
    1024+0 records in
    1024+0 records out
    134217728 bytes (134 MB) copied, 0.0490407 s, 2.7 GB/s
    
    dd if=/dev/zero of=sb-io-test bs=8k count=16k conv=fdatasync
    16384+0 records in
    16384+0 records out
    134217728 bytes (134 MB) copied, 0.186969 s, 718 MB/s
    
    dd if=/dev/zero of=sb-io-test bs=128k count=1k oflag=dsync
    1024+0 records in
    1024+0 records out
    134217728 bytes (134 MB) copied, 0.0483516 s, 2.8 GB/s
    
    dd if=/dev/zero of=sb-io-test bs=8k count=16k oflag=dsync
    16384+0 records in
    16384+0 records out
    134217728 bytes (134 MB) copied, 0.186472 s, 720 MB/s
    

    Code (Text):
    -------------------------------------------
    ramdisk ioping tests - memory bandwidth
    -------------------------------------------
    
    Running IOPing I/O ramdisk benchmark...
    
    IOPing I/O: ./ioping -c 10 .
    4.0 KiB from . (tmpfs tmpfs): request=1 time=6 us
    4.0 KiB from . (tmpfs tmpfs): request=2 time=5 us
    4.0 KiB from . (tmpfs tmpfs): request=3 time=5 us
    4.0 KiB from . (tmpfs tmpfs): request=4 time=4 us
    4.0 KiB from . (tmpfs tmpfs): request=5 time=4 us
    4.0 KiB from . (tmpfs tmpfs): request=6 time=4 us
    4.0 KiB from . (tmpfs tmpfs): request=7 time=4 us
    4.0 KiB from . (tmpfs tmpfs): request=8 time=5 us
    4.0 KiB from . (tmpfs tmpfs): request=9 time=4 us
    4.0 KiB from . (tmpfs tmpfs): request=10 time=8 us
    
    --- . (tmpfs tmpfs) ioping statistics ---
    10 requests completed in 9.0 s, 204.1 k iops, 797.2 MiB/s
    min/avg/max/mdev = 4 us / 4 us / 8 us / 1 us
    
    IOPing seek rate: ./ioping -R .
    
    --- . (tmpfs tmpfs) ioping statistics ---
    247.2 k requests completed in 3.0 s, 226.6 k iops, 885.2 MiB/s
    min/avg/max/mdev = 4 us / 4 us / 18 us / 0 us
    
    IOPing sequential: ./ioping -RL .
    
    --- . (tmpfs tmpfs) ioping statistics ---
    59.2 k requests completed in 3.0 s, 35.6 k iops, 8.7 GiB/s
    min/avg/max/mdev = 26 us / 28 us / 60 us / 1 us
    
    IOPing cached: ./ioping -RC .
    
    --- . (tmpfs tmpfs) ioping statistics ---
    632.4 k requests completed in 3.0 s, 223.4 k iops, 872.6 MiB/s
    min/avg/max/mdev = 4 us / 4 us / 35 us / 0 us
    
    

    Code (Text):
    -------------------------------------------
    Running bandwidth benchmark...
    -------------------------------------------
    
    ----------------------------------------------
    Download from Cachefly (http://cachefly.cachefly.net/100mb.test)
    Download Cachefly: 52.4MB/s
    
    -------------------------------------------
    USA bandwidth tests...
    -------------------------------------------
    
    ----------------------------------------------
    Download from Linode, Atlanta, GA, USA (http://speedtest.atlanta.linode.com/100MB-atlanta.bin)
    Download Linode, Atlanta, GA, USA: 50.3MB/s
    ----------------------------------------------
    Download from Linode, Dallas, TX, USA (http://speedtest.dallas.linode.com/100MB-dallas.bin)
    Download Linode, Dallas, TX, USA: 35.2MB/s
    ----------------------------------------------
    Download from Leaseweb, Manassas, VA, USA (http://mirror.us.leaseweb.net/speedtest/100mb.bin)
    Download Leaseweb, Manassas, VA, USA: 52.6MB/s
    ----------------------------------------------
    Download from Softlayer, Seattle, WA, USA (http://speedtest.sea01.softlayer.com/downloads/test100.zip)
    Download Softlayer, Seattle, WA, USA: 25.6MB/s
    ----------------------------------------------
    Download from Softlayer, San Jose, CA, USA (http://speedtest.sjc01.softlayer.com/downloads/test100.zip)
    Download Softlayer, San Jose, CA, USA: 23.7MB/s
    ----------------------------------------------
    Download from Softlayer, Washington, DC, USA (http://speedtest.wdc01.softlayer.com/downloads/test100.zip)
    Download Softlayer, Washington, DC, USA: 20.8MB/s
    ----------------------------------------------
    Download from VersaWeb, Las Vegas, Nevada (http://104.143.15.254/100MB.test)
    Download VersaWeb, Las Vegas, Nevada: 4.96MB/s
    ----------------------------------------------
    Download from OVH, BHS, Canada (http://bhs.proof.ovh.net/files/100Mio.dat)
    Download OVH, BHS, Canada: 47.0MB/s
    ----------------------------------------------
    Download from Vultr, Los Angeles, California (http://lax-ca-us-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, Los Angeles, California: 25.9MB/s
    ----------------------------------------------
    Download from Vultr, Seattle, Washington (http://wa-us-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, Seattle, Washington: 31.6MB/s
    ----------------------------------------------
    Download from Vultr, Dallas, Texas (http://tx-us-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, Dallas, Texas: 45.8MB/s
    ----------------------------------------------
    Download from Vultr, Chicago, Illinois (http://il-us-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, Chicago, Illinois: 50.0MB/s
    ----------------------------------------------
    Download from Vultr, Atlanta, Georgia (http://ga-us-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, Atlanta, Georgia: 49.9MB/s
    ----------------------------------------------
    Download from Vultr, Miami, Florida (http://fl-us-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, Miami, Florida: 47.9MB/s
    ----------------------------------------------
    Download from Vultr, New York / New Jersey (http://nj-us-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, New York / New Jersey: 50.3MB/s
    
    -------------------------------------------
    Asia bandwidth tests...
    -------------------------------------------
    
    ----------------------------------------------
    Download from Linode, Tokyo, JP (http://speedtest.tokyo.linode.com/100MB-tokyo.bin)
    Download Linode, Tokyo, JP: 13.5MB/s
    ----------------------------------------------
    Download from Softlayer, Singapore (http://speedtest.sng01.softlayer.com/downloads/test100.zip)
    Download Softlayer, Singapore: 6.82MB/s
    ----------------------------------------------
    Download from Vultr, Tokyo, Japan (http://hnd-jp-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, Tokyo, Japan: 13.0MB/s
    
    -------------------------------------------
    Europe bandwidth tests...
    -------------------------------------------
    
    ----------------------------------------------
    Download from Linode, London, UK (http://speedtest.london.linode.com/100MB-london.bin)
    Download Linode, London, UK: 33.0MB/s
    ----------------------------------------------
    Download from OVH, Paris, France (http://proof.ovh.net/files/100Mio.dat)
    Download OVH, Paris, France: 8.11MB/s
    ----------------------------------------------
    Download from SmartDC, Rotterdam, Netherlands (http://mirror.i3d.net/100mb.bin)
    Download SmartDC, Rotterdam, Netherlands: 7.78MB/s
    ----------------------------------------------
    Download from Vultr, Amsterdam, Netherlands (http://ams-nl-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, Amsterdam, Netherlands: 29.3MB/s
    ----------------------------------------------
    Download from Vultr, London, UK (http://lon-gb-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, London, UK: 32.1MB/s
    ----------------------------------------------
    Download from Vultr, Paris, France (http://par-fr-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, Paris, France: 26.9MB/s
    
    -------------------------------------------
    Australia bandwidth tests...
    -------------------------------------------
    
    ----------------------------------------------
    Download from Vultr, Sydney, Australia (http://syd-au-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, Sydney, Australia: 9.70MB/s
    

    Code (Text):
    -------------------------------------------
    Running Axel multi-threaded bandwidth benchmark...
    -------------------------------------------
    
    ----------------------------------------------
    Axel Download from Cachefly (http://cachefly.cachefly.net/100mb.test)
    Axel Download Cachefly: 128.00MB/s
    
    -------------------------------------------
    USA bandwidth tests...
    -------------------------------------------
    
    ----------------------------------------------
    Axel Download from Linode, Atlanta, GA, USA (http://speedtest.atlanta.linode.com/100MB-atlanta.bin)
    Axel Download Linode, Atlanta, GA, USA: 113.57MB/s
    ----------------------------------------------
    Axel Download from Linode, Dallas, TX, USA (http://speedtest.dallas.linode.com/100MB-dallas.bin)
    Axel Download Linode, Dallas, TX, USA: 55.08MB/s
    ----------------------------------------------
    Axel Download from Leaseweb, Manassas, VA, USA (http://mirror.us.leaseweb.net/speedtest/100mb.bin)
    Axel Download Leaseweb, Manassas, VA, USA: 59.42MB/s
    ----------------------------------------------
    Axel Download from Softlayer, Seattle, WA, USA (http://speedtest.sea01.softlayer.com/downloads/test100.zip)
    Axel Download Softlayer, Seattle, WA, USA: 25.45MB/s
    ----------------------------------------------
    Axel Download from Softlayer, San Jose, CA, USA (http://speedtest.sjc01.softlayer.com/downloads/test100.zip)
    Axel Download Softlayer, San Jose, CA, USA: 22.61MB/s
    ----------------------------------------------
    Axel Download from Softlayer, Washington, DC, USA (http://speedtest.wdc01.softlayer.com/downloads/test100.zip)
    Axel Download Softlayer, Washington, DC, USA: 31.35MB/s
    ----------------------------------------------
    Axel Download from VersaWeb, Las Vegas, Nevada (http://104.143.15.254/100MB.test)
    Axel Download VersaWeb, Las Vegas, Nevada: 14.47MB/s
    ----------------------------------------------
    Axel Download from OVH, BHS, Canada (http://bhs.proof.ovh.net/files/100Mio.dat)
    Axel Download OVH, BHS, Canada: 94.33MB/s
    ----------------------------------------------
    Axel Download from Vultr, Los Angeles, California (http://lax-ca-us-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, Los Angeles, California: 50.08MB/s
    ----------------------------------------------
    Axel Download from Vultr, Seattle, Washington (http://wa-us-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, Seattle, Washington: 58.48MB/s
    ----------------------------------------------
    Axel Download from Vultr, Dallas, Texas (http://tx-us-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, Dallas, Texas: 65.64MB/s
    ----------------------------------------------
    Axel Download from Vultr, Chicago, Illinois (http://il-us-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, Chicago, Illinois: 126.54MB/s
    ----------------------------------------------
    Axel Download from Vultr, Atlanta, Georgia (http://ga-us-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, Atlanta, Georgia: 113.29MB/s
    ----------------------------------------------
    Axel Download from Vultr, Miami, Florida (http://fl-us-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, Miami, Florida: 106.75MB/s
    ----------------------------------------------
    Axel Download from Vultr, New York / New Jersey (http://nj-us-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, New York / New Jersey: 168.45MB/s
    
    -------------------------------------------
    Asia bandwidth tests...
    -------------------------------------------
    
    ----------------------------------------------
    Axel Download from Linode, Tokyo, JP (http://speedtest.tokyo.linode.com/100MB-tokyo.bin)
    Axel Download Linode, Tokyo, JP: 14.44MB/s
    ----------------------------------------------
    Axel Download from Softlayer, Singapore (http://speedtest.sng01.softlayer.com/downloads/test100.zip)
    Axel Download Softlayer, Singapore: 6.53MB/s
    ----------------------------------------------
    Axel Download from Vultr, Tokyo, Japan (http://hnd-jp-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, Tokyo, Japan: 18.89MB/s
    
    -------------------------------------------
    Europe bandwidth tests...
    -------------------------------------------
    
    ----------------------------------------------
    Axel Download from Linode, London, UK (http://speedtest.london.linode.com/100MB-london.bin)
    Axel Download Linode, London, UK: 32.40MB/s
    ----------------------------------------------
    Axel Download from OVH, Paris, France (http://proof.ovh.net/files/100Mio.dat)
    Axel Download OVH, Paris, France: 21.19MB/s
    ----------------------------------------------
    Axel Download from SmartDC, Rotterdam, Netherlands (http://mirror.i3d.net/100mb.bin)
    Axel Download SmartDC, Rotterdam, Netherlands: 12.86MB/s
    ----------------------------------------------
    Axel Download from Vultr, Amsterdam, Netherlands (http://ams-nl-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, Amsterdam, Netherlands: 52.78MB/s
    ----------------------------------------------
    Axel Download from Vultr, London, UK (http://lon-gb-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, London, UK: 48.78MB/s
    ----------------------------------------------
    Axel Download from Vultr, Paris, France (http://par-fr-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, Paris, France: 31.15MB/s
    
    -------------------------------------------
    Australia bandwidth tests...
    -------------------------------------------
    
    ----------------------------------------------
    Axel Download from Vultr, Sydney, Australia (http://syd-au-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, Sydney, Australia: 17.93MB/s
    

    Code (Text):
    -------------------------------------------
    Running ping tests...
    -------------------------------------------
    
    ----------------------------------------------
    Pings (cachefly.cachefly.net):
    PING vip1.g5.cachefly.net (205.234.175.175) 56(84) bytes of data.
    64 bytes from vip1.G-anycast1.cachefly.net (205.234.175.175): icmp_seq=1 ttl=58 time=0.969 ms
    64 bytes from vip1.G-anycast1.cachefly.net (205.234.175.175): icmp_seq=2 ttl=58 time=0.968 ms
    64 bytes from vip1.G-anycast1.cachefly.net (205.234.175.175): icmp_seq=3 ttl=58 time=0.991 ms
    
    --- vip1.g5.cachefly.net ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2003ms
    rtt min/avg/max/mdev = 0.968/0.976/0.991/0.010 ms
    
    ----------------------------------------------
    Pings (syd-au-ping.vultr.com):
    PING syd-au-ping.vultr.com (108.61.212.117) 56(84) bytes of data.
    64 bytes from 108.61.212.117.vultr.com (108.61.212.117): icmp_seq=1 ttl=51 time=222 ms
    64 bytes from 108.61.212.117.vultr.com (108.61.212.117): icmp_seq=2 ttl=51 time=222 ms
    64 bytes from 108.61.212.117.vultr.com (108.61.212.117): icmp_seq=3 ttl=51 time=223 ms
    
    --- syd-au-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2003ms
    rtt min/avg/max/mdev = 222.735/222.875/223.076/0.412 ms
    
    ----------------------------------------------
    Pings (hnd-jp-ping.vultr.com):
    PING hnd-jp-ping.vultr.com (108.61.201.151) 56(84) bytes of data.
    64 bytes from 108.61.201.151.vultr.com (108.61.201.151): icmp_seq=1 ttl=50 time=175 ms
    64 bytes from 108.61.201.151.vultr.com (108.61.201.151): icmp_seq=2 ttl=50 time=175 ms
    64 bytes from 108.61.201.151.vultr.com (108.61.201.151): icmp_seq=3 ttl=50 time=175 ms
    
    --- hnd-jp-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2000ms
    rtt min/avg/max/mdev = 175.640/175.652/175.663/0.484 ms
    
    ----------------------------------------------
    Pings (lax-ca-us-ping.vultr.com):
    PING lax-ca-us-ping.vultr.com (108.61.219.200) 56(84) bytes of data.
    64 bytes from 108.61.219.200.vultr.com (108.61.219.200): icmp_seq=1 ttl=53 time=87.4 ms
    64 bytes from 108.61.219.200.vultr.com (108.61.219.200): icmp_seq=2 ttl=53 time=87.4 ms
    64 bytes from 108.61.219.200.vultr.com (108.61.219.200): icmp_seq=3 ttl=53 time=87.4 ms
    
    --- lax-ca-us-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2003ms
    rtt min/avg/max/mdev = 87.433/87.456/87.471/0.242 ms
    
    ----------------------------------------------
    Pings (wa-us-ping.vultr.com):
    PING wa-us-ping.vultr.com (108.61.194.105) 56(84) bytes of data.
    64 bytes from 108.61.194.105.vultr.com (108.61.194.105): icmp_seq=1 ttl=53 time=71.4 ms
    64 bytes from 108.61.194.105.vultr.com (108.61.194.105): icmp_seq=2 ttl=53 time=71.4 ms
    64 bytes from 108.61.194.105.vultr.com (108.61.194.105): icmp_seq=3 ttl=53 time=71.5 ms
    
    --- wa-us-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 71.472/71.494/71.514/0.309 ms
    
    ----------------------------------------------
    Pings (tx-us-ping.vultr.com):
    PING tx-us-ping.vultr.com (108.61.224.175) 56(84) bytes of data.
    64 bytes from 108.61.224.175.vultr.com (108.61.224.175): icmp_seq=1 ttl=52 time=39.1 ms
    64 bytes from 108.61.224.175.vultr.com (108.61.224.175): icmp_seq=2 ttl=52 time=39.1 ms
    64 bytes from 108.61.224.175.vultr.com (108.61.224.175): icmp_seq=3 ttl=52 time=39.1 ms
    
    --- tx-us-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 39.132/39.154/39.195/0.029 ms
    
    ----------------------------------------------
    Pings (il-us-ping.vultr.com):
    PING il-us-ping.vultr.com (107.191.51.12) 56(84) bytes of data.
    64 bytes from 107.191.51.12.vultr.com (107.191.51.12): icmp_seq=1 ttl=54 time=24.9 ms
    64 bytes from 107.191.51.12.vultr.com (107.191.51.12): icmp_seq=2 ttl=54 time=24.9 ms
    64 bytes from 107.191.51.12.vultr.com (107.191.51.12): icmp_seq=3 ttl=54 time=24.9 ms
    
    --- il-us-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 24.969/24.970/24.972/0.182 ms
    
    ----------------------------------------------
    Pings (ga-us-ping.vultr.com):
    PING ga-us-ping.vultr.com (108.61.193.166) 56(84) bytes of data.
    64 bytes from 108.61.193.166.vultr.com (108.61.193.166): icmp_seq=1 ttl=53 time=18.5 ms
    64 bytes from 108.61.193.166.vultr.com (108.61.193.166): icmp_seq=2 ttl=53 time=18.6 ms
    64 bytes from 108.61.193.166.vultr.com (108.61.193.166): icmp_seq=3 ttl=53 time=18.6 ms
    
    --- ga-us-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 18.590/18.629/18.680/0.162 ms
    
    ----------------------------------------------
    Pings (fl-us-ping.vultr.com):
    PING fl-us-ping.vultr.com (104.156.244.232) 56(84) bytes of data.
    64 bytes from 104.156.244.232 (104.156.244.232): icmp_seq=1 ttl=53 time=34.0 ms
    64 bytes from 104.156.244.232 (104.156.244.232): icmp_seq=2 ttl=53 time=34.0 ms
    64 bytes from 104.156.244.232 (104.156.244.232): icmp_seq=3 ttl=53 time=34.0 ms
    
    --- fl-us-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 34.010/34.020/34.037/0.012 ms
    
    ----------------------------------------------
    Pings (nj-us-ping.vultr.com):
    PING nj-us-ping.vultr.com (108.61.149.182) 56(84) bytes of data.
    64 bytes from 108.61.149.182.vultr.com (108.61.149.182): icmp_seq=1 ttl=56 time=2.01 ms
    64 bytes from 108.61.149.182.vultr.com (108.61.149.182): icmp_seq=2 ttl=57 time=2.04 ms
    64 bytes from 108.61.149.182.vultr.com (108.61.149.182): icmp_seq=3 ttl=56 time=2.13 ms
    
    --- nj-us-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 2.019/2.065/2.132/0.071 ms
    
    ----------------------------------------------
    Pings (fra-de-ping.vultr.com):
    PING fra-de-ping.vultr.com (108.61.210.117) 56(84) bytes of data.
    64 bytes from 108.61.210.117.vultr.com (108.61.210.117): icmp_seq=1 ttl=51 time=81.5 ms
    64 bytes from 108.61.210.117.vultr.com (108.61.210.117): icmp_seq=2 ttl=51 time=81.5 ms
    64 bytes from 108.61.210.117.vultr.com (108.61.210.117): icmp_seq=3 ttl=51 time=81.6 ms
    
    --- fra-de-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 81.558/81.587/81.622/0.234 ms
    
    ----------------------------------------------
    Pings (ams-nl-ping.vultr.com):
    PING ams-nl-ping.vultr.com (108.61.198.102) 56(84) bytes of data.
    64 bytes from 108.61.198.102.vultr.com (108.61.198.102): icmp_seq=1 ttl=53 time=79.5 ms
    64 bytes from 108.61.198.102.vultr.com (108.61.198.102): icmp_seq=2 ttl=53 time=79.4 ms
    64 bytes from 108.61.198.102.vultr.com (108.61.198.102): icmp_seq=3 ttl=53 time=79.4 ms
    
    --- ams-nl-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 79.457/79.484/79.529/0.327 ms
    
    ----------------------------------------------
    Pings (lon-gb-ping.vultr.com):
    PING lon-gb-ping.vultr.com (108.61.196.101) 56(84) bytes of data.
    64 bytes from 108.61.196.101.vultr.com (108.61.196.101): icmp_seq=1 ttl=53 time=71.2 ms
    64 bytes from 108.61.196.101.vultr.com (108.61.196.101): icmp_seq=2 ttl=53 time=71.0 ms
    64 bytes from 108.61.196.101.vultr.com (108.61.196.101): icmp_seq=3 ttl=53 time=71.2 ms
    
    --- lon-gb-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 71.096/71.205/71.265/0.077 ms
    
    ----------------------------------------------
    Pings (par-fr-ping.vultr.com):
    PING par-fr-ping.vultr.com (108.61.209.127) 56(84) bytes of data.
    64 bytes from 108.61.209.127.vultr.com (108.61.209.127): icmp_seq=1 ttl=54 time=85.4 ms
    64 bytes from 108.61.209.127.vultr.com (108.61.209.127): icmp_seq=2 ttl=54 time=85.4 ms
    64 bytes from 108.61.209.127.vultr.com (108.61.209.127): icmp_seq=3 ttl=54 time=85.4 ms
    
    --- par-fr-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 85.412/85.441/85.467/0.022 ms
    
    ----------------------------------------------
    Pings (VersaWeb Las Vegas):
    PING 104.143.15.254 (104.143.15.254) 56(84) bytes of data.
    64 bytes from 104.143.15.254: icmp_seq=1 ttl=48 time=76.2 ms
    64 bytes from 104.143.15.254: icmp_seq=2 ttl=48 time=76.2 ms
    64 bytes from 104.143.15.254: icmp_seq=3 ttl=48 time=76.2 ms
    
    --- 104.143.15.254 ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 76.230/76.235/76.239/0.225 ms
    
    ----------------------------------------------
    Pings (VersaWeb Seattle):
    PING 76.164.234.1 (76.164.234.1) 56(84) bytes of data.
    64 bytes from 76.164.234.1: icmp_seq=1 ttl=54 time=66.4 ms
    64 bytes from 76.164.234.1: icmp_seq=2 ttl=54 time=66.4 ms
    64 bytes from 76.164.234.1: icmp_seq=3 ttl=54 time=66.4 ms
    
    --- 76.164.234.1 ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2003ms
    rtt min/avg/max/mdev = 66.444/66.476/66.499/0.298 ms
    
    ----------------------------------------------
    Pings (OVH Canada):
    PING bhs.proof.ovh.net (192.99.19.165) 56(84) bytes of data.
    64 bytes from bhs.proof.ovh.net (192.99.19.165): icmp_seq=1 ttl=56 time=8.91 ms
    64 bytes from bhs.proof.ovh.net (192.99.19.165): icmp_seq=2 ttl=56 time=8.87 ms
    64 bytes from bhs.proof.ovh.net (192.99.19.165): icmp_seq=3 ttl=56 time=8.88 ms
    
    --- bhs.proof.ovh.net ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2001ms
    rtt min/avg/max/mdev = 8.871/8.890/8.918/0.020 ms
    
    ----------------------------------------------
    Pings (10gbps.io Frankfurt):
    PING 195.181.174.20 (195.181.174.20) 56(84) bytes of data.
    64 bytes from 195.181.174.20: icmp_seq=1 ttl=51 time=93.1 ms
    64 bytes from 195.181.174.20: icmp_seq=2 ttl=51 time=93.1 ms
    64 bytes from 195.181.174.20: icmp_seq=3 ttl=51 time=93.1 ms
    
    --- 195.181.174.20 ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 93.110/93.143/93.180/0.028 ms
    
    ----------------------------------------------
    Pings (10gbps.io London):
    PING 185.59.221.20 (185.59.221.20) 56(84) bytes of data.
    64 bytes from 185.59.221.20: icmp_seq=1 ttl=51 time=94.5 ms
    64 bytes from 185.59.221.20: icmp_seq=2 ttl=51 time=94.6 ms
    64 bytes from 185.59.221.20: icmp_seq=3 ttl=51 time=94.5 ms
    
    --- 185.59.221.20 ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2003ms
    rtt min/avg/max/mdev = 94.562/94.591/94.618/0.355 ms
    
    ----------------------------------------------
    Pings (10gbps.io Paris):
    PING 185.93.2.193 (185.93.2.193) 56(84) bytes of data.
    64 bytes from 185.93.2.193: icmp_seq=1 ttl=51 time=101 ms
    64 bytes from 185.93.2.193: icmp_seq=2 ttl=51 time=101 ms
    64 bytes from 185.93.2.193: icmp_seq=3 ttl=51 time=101 ms
    
    --- 185.93.2.193 ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 101.089/101.165/101.214/0.054 ms
    
    ----------------------------------------------
    Pings (10gbps.io Singapore):
    PING 203.174.85.38 (203.174.85.38) 56(84) bytes of data.
    64 bytes from 203.174.85.38: icmp_seq=1 ttl=51 time=251 ms
    64 bytes from 203.174.85.38: icmp_seq=2 ttl=51 time=251 ms
    64 bytes from 203.174.85.38: icmp_seq=3 ttl=51 time=251 ms
    
    --- 203.174.85.38 ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 251.396/251.576/251.931/0.480 ms
    
    ----------------------------------------------
    Pings (10gbps.io Stockholm):
    PING 185.76.9.10 (185.76.9.10) 56(84) bytes of data.
    64 bytes from 185.76.9.10: icmp_seq=1 ttl=53 time=108 ms
    64 bytes from 185.76.9.10: icmp_seq=2 ttl=53 time=108 ms
    64 bytes from 185.76.9.10: icmp_seq=3 ttl=53 time=108 ms
    
    --- 185.76.9.10 ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 108.173/108.207/108.234/0.380 ms
    
    ----------------------------------------------
    Pings (10gbps.io Moscow):
    PING 87.245.204.20 (87.245.204.20) 56(84) bytes of data.
    64 bytes from 87.245.204.20: icmp_seq=1 ttl=59 time=113 ms
    64 bytes from 87.245.204.20: icmp_seq=2 ttl=59 time=113 ms
    64 bytes from 87.245.204.20: icmp_seq=3 ttl=59 time=113 ms
    
    --- 87.245.204.20 ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2003ms
    rtt min/avg/max/mdev = 113.478/113.496/113.525/0.275 ms
    
    ----------------------------------------------
    Pings (10gbps.io Miami):
    PING 195.181.162.195 (195.181.162.195) 56(84) bytes of data.
    64 bytes from 195.181.162.195: icmp_seq=1 ttl=55 time=55.9 ms
    64 bytes from 195.181.162.195: icmp_seq=2 ttl=55 time=55.8 ms
    64 bytes from 195.181.162.195: icmp_seq=3 ttl=55 time=55.6 ms
    
    --- 195.181.162.195 ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 55.669/55.849/55.982/0.303 ms
    
    

    Code (Text):
    -------------------------------------------
    Running mtr tests...
    -------------------------------------------
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 cachefly.cachefly.net
    Start: Mon Feb 26 08:52:11 2018
    HOST: hostname          Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                       0.0%    10    7.8  19.0   7.8  23.6   4.3
      2.|-- 147.75.36.4                   0.0%    10   78.4 155.8  12.7 1040. 318.4
      3.|-- 0.ae12.bbr1.ewr1.packet.net   0.0%    10    0.3   3.6   0.2  33.0  10.3
      4.|-- ???                          100.0    10    0.0   0.0   0.0   0.0   0.0
      5.|-- nyk-b6-link.telia.net         0.0%    10    1.0   1.1   1.0   2.0   0.0
      6.|-- nyk-b5-link.telia.net         0.0%    10    1.3   1.1   1.0   1.3   0.0
      7.|-- ???                          100.0    10    0.0   0.0   0.0   0.0   0.0
      8.|-- vip1.G-anycast1.cachefly.net  0.0%    10    1.0   1.0   1.0   1.0   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 speedtest.atlanta.linode.com
    Start: Mon Feb 26 08:52:27 2018
    HOST: hostname                  Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                               0.0%    10    8.8   6.2   1.0  11.0   3.8
      2.|-- 2604:1380:2:79::                      0.0%    10    4.6   6.2   0.9  11.2   3.3
      3.|-- 2604:1380:2:20::                      0.0%    10    0.2   0.2   0.2   0.4   0.0
      4.|-- 2001:518:6900:1::5                    0.0%    10    1.7   0.7   0.4   1.7   0.3
      5.|-- 2001:518:2001:15::1                   0.0%    10    0.3   0.8   0.2   4.8   1.2
      6.|-- 2001:518:6001:6::2                    0.0%    10    6.6   6.5   1.1  12.0   4.1
      7.|-- nyk-b3-link.telia.net                 0.0%    10    1.2   1.2   1.1   1.2   0.0
      8.|-- nyk-bb4-v6.telia.net                  0.0%    10   21.2  21.2  21.1  21.3   0.0
      9.|-- ash-bb4-v6.telia.net                  0.0%    10   21.6  22.8  21.6  32.1   3.2
     10.|-- atl-b22-v6.telia.net                  0.0%    10   32.9  25.3  24.1  32.9   2.7
     11.|-- linode-ic-321031-atl-b22.c.telia.net  0.0%    10   21.9  21.9  21.8  22.0   0.0
     12.|-- 2600:3c02:4444:4::2                   0.0%    10   21.9  22.0  21.8  22.2   0.0
     13.|-- 2600:3c02::4b                         0.0%    10   21.5  21.5  21.4  21.7   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 speedtest.dallas.linode.com
    Start: Mon Feb 26 08:52:44 2018
    HOST: hostname           Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                        0.0%    10   17.8  21.1  15.1  24.9   3.0
      2.|-- 2604:1380:2:79::               0.0%    10    2.4  10.4   1.0  61.2  18.1
      3.|-- 2604:1380:2:20::               0.0%    10    0.2   0.2   0.2   0.3   0.0
      4.|-- 2001:518:6900:1::5             0.0%    10    0.5  15.7   0.4 134.0  41.8
      5.|-- 2001:518:2001:16::1            0.0%    10    1.6   0.4   0.2   1.6   0.3
      6.|-- 2001:518:6001:7::2             0.0%    10    1.1   1.1   1.1   1.1   0.0
      7.|-- ae2-63.cr0-nyc2.ip6.gtt.net    0.0%    10   70.1  70.1  70.0  70.4   0.0
      8.|-- 2001:668:0:2:ffff:0:5995:8f3d  0.0%    10   80.3  80.3  80.2  80.5   0.0
      9.|-- 2001:668:0:3:ffff:2:0:26a      0.0%    10   42.2  42.2  42.1  42.5   0.0
     10.|-- 2600:3c00:2222:13::2           0.0%    10   42.8  42.8  42.6  42.9   0.0
     11.|-- speedtest.dallas.linode.com    0.0%    10   45.0  45.0  44.9  45.2   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 mirror.us.leaseweb.net
    Start: Mon Feb 26 08:53:00 2018
    HOST: hostname           Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                        0.0%    10   30.4  20.2  11.5  41.6  10.3
      2.|-- 2604:1380:2:78::               0.0%    10    1.1   6.5   1.1  11.4   3.6
      3.|-- 2604:1380:2:2::1               0.0%    10    6.1   5.6   0.3  40.5  12.5
      4.|-- 2001:518:6900:1::1             0.0%    10  162.9  18.6   0.4 162.9  51.0
      5.|-- 2001:518:2001:14::1            0.0%    10    0.2   0.3   0.2   1.5   0.3
      6.|-- 2001:518:6001:7::2             0.0%    10    1.1   1.1   1.0   1.1   0.0
      7.|-- ae2-63.cr0-nyc2.ip6.gtt.net    0.0%    10   70.0  70.2  70.0  71.2   0.3
      8.|-- 2001:668:0:2:ffff:0:d5fe:d6aa  0.0%    10   76.4  76.4  76.3  76.6   0.0
      9.|-- 2001:668:0:3:ffff:0:45ae:326   0.0%    10   52.2  52.1  52.0  52.2   0.0
     10.|-- 2604:9a00::3d                  0.0%    10   33.8  33.8  33.7  33.8   0.0
     11.|-- 2604:9a00:2010:a0b8::5         0.0%    10   34.0  34.0  33.9  34.1   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 speedtest.sea01.softlayer.com
    Start: Mon Feb 26 08:53:16 2018
    HOST: hostname                                               Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                                                            0.0%    10   23.2  34.2   6.8 118.8  38.0
      2.|-- 2604:1380:2:79::                                                   0.0%    10    8.4   6.3   2.4  10.6   2.7
      3.|-- 2604:1380:2:20::                                                   0.0%    10    0.3   0.2   0.2   0.3   0.0
      4.|-- de-cix.nyc.us.ibm.com                                              0.0%    10    1.1   1.4   1.1   3.2   0.5
      5.|-- 6320.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10    1.8   1.9   1.7   2.3   0.0
      6.|-- a320.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   22.8  22.9  22.7  23.2   0.0
      7.|-- 5220.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   23.5  23.5  23.0  26.8   1.1
      8.|-- ee20.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   64.5  64.5  63.9  64.9   0.0
      9.|-- 1630.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   62.5  62.6  62.5  62.7   0.0
     10.|-- po3.bbr01.sr01.sea01.networklayer.com                              0.0%    10   62.4  62.5  62.4  63.1   0.0
     11.|-- po1.fcr01.sr01.sea01.networklayer.com                              0.0%    10   63.3  65.0  63.3  74.9   3.8
     12.|-- softlayer.com                                                      0.0%    10   62.8  62.8  62.8  62.9   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 speedtest.sjc01.softlayer.com
    Start: Mon Feb 26 08:53:33 2018
    HOST: hostname                                               Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                                                            0.0%    10   20.6  26.6  12.5  57.1  13.9
      2.|-- 2604:1380:2:79::                                                   0.0%    10    6.1   8.5   1.3  41.0  11.8
      3.|-- 2604:1380:2:20::                                                   0.0%    10    0.2   1.3   0.2  10.1   3.1
      4.|-- de-cix.nyc.us.ibm.com                                              0.0%    10    1.2   1.1   1.1   1.2   0.0
      5.|-- 6320.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10    2.6   2.6   1.7   5.0   0.9
      6.|-- a320.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   22.6  22.7  22.5  22.9   0.0
      7.|-- 5220.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   22.9  22.9  22.7  23.4   0.0
      8.|-- 8320.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   45.1  44.8  44.6  45.2   0.0
      9.|-- 3620.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   67.0  67.1  66.9  67.4   0.0
     10.|-- 3520.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   79.4  77.8  69.1 133.0  19.7
     11.|-- 9700.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   66.7  67.2  66.6  69.3   0.9
     12.|-- 2607:f0d0:2100:1::3                                                0.0%    10   68.9  71.1  68.6  84.9   5.0
     13.|-- softlayer.com                                                      0.0%    10   70.9  70.9  70.9  71.0   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 speedtest.wdc01.softlayer.com
    Start: Mon Feb 26 08:53:50 2018
    HOST: hostname                                               Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                                                            0.0%    10   11.3   6.2   0.5  11.3   3.2
      2.|-- 2604:1380:2:78::                                                   0.0%    10   10.3   7.1   2.2  14.9   3.6
      3.|-- 2604:1380:2:5::1                                                   0.0%    10    0.2   0.9   0.2   7.2   2.1
      4.|-- de-cix.nyc.us.ibm.com                                              0.0%    10    1.2   1.2   1.1   1.6   0.0
      5.|-- 2320.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10    2.2   1.8   1.7   2.2   0.0
      6.|-- 2830.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10    9.9   7.9   7.1   9.9   0.9
      7.|-- e730.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10    9.1   9.7   7.4  26.8   6.0
      8.|-- bbr01.eq01.wdc02.networklayer.com                                  0.0%    10    7.1   8.3   7.1  12.5   2.0
      9.|-- po2.dar01.sr01.wdc01.networklayer.com                              0.0%    10    7.1   7.2   7.1   7.5   0.0
     10.|-- po1.fcr01.sr01.wdc01.networklayer.com                              0.0%    10    7.4   7.6   7.3   8.1   0.0
     11.|-- softlayer.com                                                      0.0%    10    7.5   7.5   7.4   7.8   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 speedtest.tokyo.linode.com
    Start: Mon Feb 26 08:54:06 2018
    HOST: hostname          Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                       0.0%    10    0.6   5.2   0.6  10.2   3.1
      2.|-- 2604:1380:2:78::              0.0%    10    0.8   9.0   0.8  31.2   8.7
      3.|-- 2604:1380:2:5::1              0.0%    10    0.2   4.4   0.2  24.3   8.7
      4.|-- 10ge3-9.core1.nyc6.he.net     0.0%    10    1.2   1.1   1.1   1.2   0.0
      5.|-- peering.kddi.com              0.0%    10   80.2  79.9  76.7  94.6   5.4
      6.|-- 6nytjbb001.int-gw.kddi.ne.jp  0.0%    10   79.9  80.3  79.8  83.0   0.7
      7.|-- 6lacGCS001.int-gw.kddi.ne.jp  0.0%    10   65.6  68.6  65.5  78.6   4.9
      8.|-- 2001:268:fb80:300::1          0.0%    10  239.5 239.2 238.3 240.0   0.0
      9.|-- 2001:268:fa00:282::2          0.0%    10  255.9 254.5 237.8 267.4   9.7
     10.|-- 6cm-fcu204.int-gw.kddi.ne.jp  0.0%    10  250.7 242.5 238.4 250.7   4.6
     11.|-- 2001:268:f702:6d::2           0.0%    10  243.7 243.0 242.9 243.7   0.0
     12.|-- speedtest.tokyo.linode.com    0.0%    10  238.6 238.6 238.5 238.7   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 speedtest.sng01.softlayer.com
    Start: Mon Feb 26 08:54:28 2018
    HOST: hostname                                               Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                                                            0.0%    10    8.4   5.1   0.6   9.4   2.8
      2.|-- 2604:1380:2:78::                                                   0.0%    10   10.7   3.9   0.6  10.7   3.0
      3.|-- 2604:1380:2:2::1                                                   0.0%    10    0.2   0.7   0.2   5.1   1.4
      4.|-- 2604:1380:2:1::                                                    0.0%    10    0.4   1.5   0.2  11.8   3.6
      5.|-- de-cix.nyc.us.ibm.com                                              0.0%    10    1.3   1.2   1.2   1.3   0.0
      6.|-- 6320.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10    1.8   2.5   1.7   6.9   1.6
      7.|-- a320.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   22.8  22.8  22.7  23.2   0.0
      8.|-- 5220.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   23.5  23.6  23.0  25.2   0.5
      9.|-- 8320.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   47.4  48.8  47.3  60.1   3.9
     10.|-- a120.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   47.8  47.7  47.3  48.2   0.0
     11.|-- c520.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   72.8  73.6  72.7  79.7   2.0
     12.|-- 8b40.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10  214.8 215.1 214.7 217.0   0.5
     13.|-- 7940.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10  240.4 240.8 240.4 243.3   0.6
     14.|-- b840.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10  242.0 240.7 240.3 242.0   0.3
     15.|-- db00.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10  240.4 240.4 240.3 240.6   0.0
     16.|-- 3.00.0000.ip4.static.sl-reverse.com                                0.0%    10  240.4 240.7 240.4 242.6   0.6
     17.|-- 2.00.0000.ip4.static.sl-reverse.com                                0.0%    10  240.3 240.4 240.3 240.7   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 speedtest.london.linode.com
    Start: Mon Feb 26 08:54:45 2018
    HOST: hostname                 Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                              0.0%    10    3.3   4.3   1.4   9.6   2.3
      2.|-- 2604:1380:2:78::                     0.0%    10    0.6   6.1   0.6  10.4   3.1
      3.|-- 2604:1380:2:2::1                     0.0%    10    0.3   2.2   0.2  19.8   6.2
      4.|-- 2001:518:6900:1::1                   0.0%    10    0.5   3.9   0.4  35.3  11.0
      5.|-- 2001:518:2001:13::1                  0.0%    10    0.2   1.6   0.2   4.7   1.7
      6.|-- 2001:518:6001:6::2                   0.0%    10    1.1   1.7   1.0   3.6   0.7
      7.|-- nyk-b3-link.telia.net                0.0%    10    1.2   1.2   1.1   1.3   0.0
      8.|-- nyk-bb4-v6.telia.net                 0.0%    10   70.7  70.8  70.7  71.1   0.0
      9.|-- 2001:2000:3019:7b::1                 0.0%    10   70.9  71.1  70.7  72.9   0.6
     10.|-- ldn-bb2-v6.telia.net                 0.0%    10   71.4  83.2  70.6 126.6  18.9
     11.|-- ldn-b4-v6.telia.net                  0.0%    10   70.6  77.0  70.6 112.0  13.1
     12.|-- linode-ic-318542-ldn-b3.c.telia.net  0.0%    10   73.5  71.8  71.4  73.5   0.5
     13.|-- 2a01:7e00:7777:4::2                  0.0%    10   71.5  71.5  71.5  71.6   0.0
     14.|-- speedtest.london.linode.com          0.0%    10   71.0  71.0  71.0  71.3   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 mirror.i3d.net
    Start: Mon Feb 26 08:55:02 2018
    HOST: hostname              Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                           0.0%    10    7.1  38.8   4.9 118.9  41.6
      2.|-- 2604:1380:2:78::                  0.0%    10    9.1   6.7   1.2  11.2   3.3
      3.|-- 2604:1380:2:2::1                  0.0%    10    0.2   2.5   0.2  22.3   6.9
      4.|-- 2001:518:6900:1::1                0.0%    10  164.9  62.8   0.4 198.8  83.8
      5.|-- 2001:518:2001:14::1               0.0%    10    0.4   0.3   0.2   0.5   0.0
      6.|-- 2001:518:1001:7::2                0.0%    10    9.3   5.1   0.9  11.7   4.5
      7.|-- 10ge.eqx-ashburn.misi3d.com       0.0%    10   10.2  12.2   6.1  16.9   3.7
      8.|-- 10ge.lr0-iadr0.th-n.lon.i3d.net   0.0%    10   82.0  83.7  77.8  89.0   3.8
      9.|-- 30ge.cr1-lr0.smartdc.rtm.i3d.net  0.0%    10   84.4  84.4  84.4  84.5   0.0
     10.|-- 80ge.br2-cr1.smartdc.rtd.i3d.net  0.0%    10   84.4  84.4  84.4  84.4   0.0
     11.|-- hosted-by.i3d.net                 0.0%    10   84.4  84.5  84.4  84.6   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 syd-au-ping.vultr.com
    Start: Mon Feb 26 08:55:21 2018
    HOST: hostname                         Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                                      0.0%    10   25.5  28.7   5.4  39.6  11.1
      2.|-- 147.75.36.4                                  0.0%    10   14.6 239.4  14.6 1098. 383.6
      3.|-- 0.ae22.bbr2.ewr1.packet.net                  0.0%    10    0.3   0.7   0.2   4.9   1.3
      4.|-- 10ge3-9.core1.nyc6.he.net                    0.0%    10    1.2   1.2   1.2   1.3   0.0
      5.|-- 100ge13-1.core1.nyc4.he.net                  0.0%    10    1.4   2.6   1.3   6.3   1.6
      6.|-- 100ge8-1.core1.sjc2.he.net                   0.0%    10   63.0  63.2  63.0  64.1   0.0
      7.|-- 10ge4-4.core1.sjc1.he.net                    0.0%    10   73.6  65.9  63.1  73.6   4.3
      8.|-- vocus.gigabitethernet2-13.core1.sjc1.he.net  0.0%    10   68.5  68.6  68.4  69.5   0.0
      9.|-- bundle-153.cor02.sjc01.ca.vocus.net          0.0%    10  218.2 218.3 218.1 218.8   0.0
     10.|-- BE-201.cor02.syd04.nsw.VOCUS.net.au          0.0%    10  218.1 218.1 218.0 218.3   0.0
     11.|-- bundle-101.bdr01.syd03.nsw.vocus.net.au      0.0%    10  218.9 218.7 218.5 219.0   0.0
     12.|-- static-73.7.255.49.in-addr.VOCUS.net.au      0.0%    10  217.9 217.9 217.8 218.0   0.0
     13.|-- xe-1-0-1.gw102.sy3.ap.equinix.com            0.0%    10  213.7 214.0 213.6 216.1   0.7
     14.|-- 180.189.25.6                                 0.0%    10  231.4 231.5 231.4 231.7   0.0
     15.|-- vl901-ds1-b5-r1120.aus1.choopa.net           0.0%    10  232.0 232.1 232.0 232.2   0.0
     16.|-- ???                                         100.0    10    0.0   0.0   0.0   0.0   0.0
     17.|-- 108.61.212.117.vultr.com                     0.0%    10  222.8 222.9 222.7 223.4   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 hnd-jp-ping.vultr.com
    Start: Mon Feb 26 08:55:42 2018
    HOST: hostname                        Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                                     0.0%    10   15.8  16.9  12.7  22.1   2.8
      2.|-- 147.75.36.4                                 0.0%    10   17.6 188.9  13.5 1013. 327.5
      3.|-- 0.ae22.bbr2.ewr1.packet.net                 0.0%    10    0.2   0.2   0.2   0.4   0.0
      4.|-- 0.xe-0-0-0.csr1.ewr2.packet.net             0.0%    10  123.0  37.3   1.5 134.1  53.3
      6.|-- 0.ae1.BR2.NYC4.ALTER.NET                    0.0%    10    2.7   2.9   2.7   3.5   0.0
      7.|-- verizon.com.customer.alter.net              0.0%    10    3.0   3.0   2.9   3.3   0.0
      8.|-- ae-3.r24.nycmny01.us.bb.gin.ntt.net        10.0%    10    3.0   3.0   3.0   3.2   0.0
      9.|-- ae-2.r20.sttlwa01.us.bb.gin.ntt.net        10.0%    10   70.3  70.5  70.2  71.9   0.4
     10.|-- ae-13.r30.tokyjp05.jp.bb.gin.ntt.net       30.0%    10  173.8 173.7 173.7 173.8   0.0
     11.|-- ae-2.r02.tokyjp05.jp.bb.gin.ntt.net         0.0%    10  181.7 181.7 181.6 181.8   0.0
     12.|-- ce-0-14-0-1.r02.tokyjp05.jp.ce.gin.ntt.net  0.0%    10  200.9 188.6 182.5 200.9   7.0
     13.|-- ???                                        100.0    10    0.0   0.0   0.0   0.0   0.0
     14.|-- vl516-ds1-b5-r2605.tyo1.choopa.net          0.0%    10  182.4 182.3 182.1 183.8   0.5
     15.|-- ???                                        100.0    10    0.0   0.0   0.0   0.0   0.0
     16.|-- 108.61.201.151.vultr.com                    0.0%    10  176.1 175.8 175.6 176.1   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 lax-ca-us-ping.vultr.com
    Start: Mon Feb 26 08:56:00 2018
    HOST: hostname                Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                             0.0%    10   13.0  14.7   8.0  18.0   3.0
      2.|-- 147.75.36.2                         0.0%    10  428.8  84.8  11.4 428.8 138.3
      3.|-- 0.ae11.bbr1.ewr1.packet.net         0.0%    10    0.2   0.2   0.2   0.2   0.0
      4.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
      5.|-- nyk-b6-link.telia.net               0.0%    10    1.0   1.0   1.0   1.0   0.0
      6.|-- nyk-bb3-link.telia.net              0.0%    10    1.0   1.1   1.0   1.3   0.0
      7.|-- las-b24-link.telia.net              0.0%    10   90.9  92.2  87.9 105.3   5.9
      8.|-- las-b4-link.telia.net               0.0%    10   87.7  91.9  87.4 127.3  12.5
      9.|-- vultr-ic-318444-las-b4.c.telia.net  0.0%    10   87.5  89.2  87.5  96.0   2.8
     10.|-- vl901-ds1-b5-c705.lax3.choopa.net   0.0%    10   88.5 103.1  88.4 219.6  41.1
     11.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
     12.|-- 108.61.219.200.vultr.com            0.0%    10   87.6  87.5  87.4  87.6   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 wa-us-ping.vultr.com
    Start: Mon Feb 26 08:56:16 2018
    HOST: hostname                  Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                               0.0%    10   15.6  21.2   6.4  67.2  16.7
      2.|-- 147.75.36.2                           0.0%    10  563.1 129.8  12.0 563.1 202.4
      3.|-- 0.ae11.bbr1.ewr1.packet.net           0.0%    10    0.2   0.2   0.1   0.2   0.0
      4.|-- 0.xe-0-0-1.csr1.ewr2.packet.net       0.0%    10    4.8  17.8   3.1 132.3  40.2
      7.|-- lag-10.ear4.Newark1.Level3.net        0.0%    10    1.1   1.1   1.0   1.6   0.0
      8.|-- ae-2-3613.edge1.Seattle3.Level3.net   0.0%    10   74.7  74.7  74.7  74.7   0.0
      9.|-- CHOOPA-LLC.edge1.Seattle3.Level3.net  0.0%    10   71.5  72.0  71.5  74.9   0.9
     10.|-- ???                                  100.0    10    0.0   0.0   0.0   0.0   0.0
     11.|-- 108.61.194.105.vultr.com              0.0%    10   71.5  71.5  71.5  71.5   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 tx-us-ping.vultr.com
    Start: Mon Feb 26 08:56:32 2018
    HOST: hostname                 Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                              0.0%    10   21.5  15.9  11.9  21.5   2.7
      2.|-- 147.75.36.2                          0.0%    10    9.1 207.2   9.1 1037. 396.7
      3.|-- 0.ae21.bbr2.ewr1.packet.net          0.0%    10    0.2   0.2   0.1   0.2   0.0
      4.|-- 0.ae0.bbr1.ewr1.packet.net           0.0%    10    0.4   1.0   0.4   6.6   1.8
      5.|-- ???                                 100.0    10    0.0   0.0   0.0   0.0   0.0
      6.|-- nyk-b6-link.telia.net                0.0%    10    1.0   1.1   1.0   1.7   0.0
      7.|-- nyk-bb3-link.telia.net              10.0%    10   38.9  42.5  38.9  65.5   8.7
      8.|-- ash-bb3-link.telia.net               0.0%    10   36.3  36.4  36.3  36.5   0.0
      9.|-- atl-b22-link.telia.net               0.0%    10   20.4  22.8  18.6  33.0   4.2
     10.|-- dls-b22-link.telia.net               0.0%    10   44.2  40.0  38.7  45.5   2.6
     11.|-- vultr-ic-318447-dls-b22.c.telia.net  0.0%    10   39.0  39.4  39.0  42.9   1.1
     12.|-- ???                                 100.0    10    0.0   0.0   0.0   0.0   0.0
     13.|-- ???                                 100.0    10    0.0   0.0   0.0   0.0   0.0
     14.|-- 108.61.224.175.vultr.com             0.0%    10   39.2  39.2  39.1  39.3   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 il-us-ping.vultr.com
    Start: Mon Feb 26 08:56:48 2018
    HOST: hostname                 Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                              0.0%    10   16.2  13.5  11.0  16.2   1.9
      2.|-- 147.75.36.4                          0.0%    10   16.0 280.8  11.6 1006. 423.6
      3.|-- 0.ae12.bbr1.ewr1.packet.net          0.0%    10    0.2   0.2   0.2   0.6   0.0
      4.|-- ???                                 100.0    10    0.0   0.0   0.0   0.0   0.0
      5.|-- nyk-b6-link.telia.net                0.0%    10    1.0   1.0   1.0   1.1   0.0
      6.|-- nyk-bb4-link.telia.net               0.0%    10   23.6  23.8  23.6  24.0   0.0
      7.|-- chi-b21-link.telia.net               0.0%    10   24.6  24.7  24.6  24.8   0.0
      8.|-- vultr-ic-318445-chi-b21.c.telia.net  0.0%    10   26.2  27.9  26.2  30.4   1.3
      9.|-- vl820-ds1-b5-gj178.chi2.choopa.net   0.0%    10   25.0  25.2  24.9  27.4   0.6
     10.|-- ???                                 100.0    10    0.0   0.0   0.0   0.0   0.0
     11.|-- 107.191.51.12.vultr.com              0.0%    10   24.9  25.0  24.9  25.0   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 ga-us-ping.vultr.com
    Start: Mon Feb 26 08:57:04 2018
    HOST: hostname                 Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                              0.0%    10   14.8  15.8  11.2  21.2   3.7
      2.|-- 147.75.36.2                          0.0%    10   18.4 190.1   9.1 934.8 368.1
      3.|-- 0.ae21.bbr2.ewr1.packet.net          0.0%    10    0.2   1.1   0.2   9.9   3.1
      4.|-- 0.ae0.bbr1.ewr1.packet.net           0.0%    10    0.4   0.5   0.4   0.9   0.0
      5.|-- ???                                 100.0    10    0.0   0.0   0.0   0.0   0.0
      6.|-- nyk-b6-link.telia.net                0.0%    10    1.0   1.1   1.0   2.0   0.0
      7.|-- nyk-bb3-link.telia.net               0.0%    10   18.8  20.0  18.6  26.7   2.4
      8.|-- ash-bb3-link.telia.net               0.0%    10    7.6   8.7   7.4  16.9   2.9
      9.|-- atl-b21-link.telia.net               0.0%    10   18.5  18.5  18.4  18.6   0.0
     10.|-- vultr-ic-319646-atl-b21.c.telia.net  0.0%    10   18.4  18.4  18.4  18.9   0.0
     11.|-- vl901-ds1-b5-r7.atl2.choopa.net      0.0%    10   26.8  19.7  18.9  26.8   2.5
     12.|-- ???                                 100.0    10    0.0   0.0   0.0   0.0   0.0
     13.|-- 108.61.193.166.vultr.com             0.0%    10   18.6  18.6  18.6  18.7   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 fl-us-ping.vultr.com
    Start: Mon Feb 26 08:57:20 2018
    HOST: hostname                Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                             0.0%    10   48.3  34.5  25.0  48.3   7.1
      2.|-- 147.75.36.2                         0.0%    10   15.2  86.5  11.4 494.7 160.4
      3.|-- 0.ae21.bbr2.ewr1.packet.net         0.0%    10    0.2   0.2   0.2   0.2   0.0
      4.|-- 0.ae0.bbr1.ewr1.packet.net          0.0%    10    0.4   0.8   0.4   2.5   0.6
      5.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
      6.|-- nyk-b6-link.telia.net               0.0%    10    1.0   1.0   1.0   1.1   0.0
      7.|-- nyk-bb3-link.telia.net              0.0%    10    1.2   1.3   1.1   2.6   0.3
      8.|-- ash-bb3-link.telia.net              0.0%    10    7.3   7.3   7.3   7.4   0.0
      9.|-- mai-b1-link.telia.net               0.0%    10   34.2  34.4  34.1  35.5   0.0
     10.|-- vultr-ic-318945-mai-b1.c.telia.net  0.0%    10   36.0  34.8  34.5  36.0   0.3
     11.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
     12.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
     13.|-- 104.156.244.232                     0.0%    10   34.1  34.1  34.0  34.1   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 nj-us-ping.vultr.com
    Start: Mon Feb 26 08:57:36 2018
    HOST: hostname                Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                             0.0%    10   21.3  17.8  11.3  38.1   8.1
      2.|-- 147.75.36.4                         0.0%    10   20.7 174.3  12.4 871.9 334.2
      3.|-- 0.ae12.bbr1.ewr1.packet.net         0.0%    10    0.2   1.7   0.2  15.3   4.7
      4.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
      5.|-- nyk-b6-link.telia.net               0.0%    10    1.0   1.2   1.0   2.8   0.5
      6.|-- nyk-bb3-link.telia.net              0.0%    10   19.5  13.9   1.2  32.2  12.3
      7.|-- nyk-b2-link.telia.net               0.0%    10    1.7   5.1   1.7  34.7  10.4
      8.|-- vultr-ic-318452-nyk-b2.c.telia.net  0.0%    10    7.6   5.9   2.0  11.6   4.1
      9.|-- vl39-br1.pnj1.choopa.net            0.0%    10    8.3   5.2   2.1  13.1   4.1
     10.|-- vl139-c4-3-b5-2.pnj1.choopa.net     0.0%    10    2.6   2.4   2.4   2.6   0.0
     11.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
     12.|-- 108.61.149.182.vultr.com            0.0%    10    2.1   2.1   2.0   2.2   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 fra-de-ping.vultr.com
    Start: Mon Feb 26 08:57:52 2018
    HOST: hostname                 Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                              0.0%    10   12.4  21.3  12.4  65.8  15.9
      2.|-- 147.75.36.2                          0.0%    10   20.9 241.4  11.4 1023. 376.2
      3.|-- 0.ae21.bbr2.ewr1.packet.net          0.0%    10    0.5   0.2   0.2   0.5   0.0
      4.|-- packet.tbr2.wbr.nac.net              0.0%    10    0.4  12.0   0.3  85.3  27.0
      5.|-- 0.lag16.tbr1.wbr.nac.net             0.0%    10    6.6   6.1   0.2  12.8   4.8
      6.|-- 0.e3-4.tbr1.tl9.nac.net              0.0%    10    1.1   1.4   1.1   4.1   0.7
      7.|-- nyk-b3-link.telia.net                0.0%    10    1.2   1.2   1.2   1.2   0.0
      8.|-- nyk-bb4-link.telia.net               0.0%    10    1.8   1.9   1.8   2.2   0.0
      9.|-- prs-bb4-link.telia.net               0.0%    10   88.1  89.3  88.1  94.4   1.9
     10.|-- prs-b2-link.telia.net                0.0%    10   72.4  72.6  72.4  73.4   0.0
     11.|-- prs-bb3-link.telia.net               0.0%    10   81.7  81.8  81.6  82.8   0.0
     12.|-- ffm-bb3-link.telia.net               0.0%    10   87.9  87.9  87.8  88.2   0.0
     13.|-- ffm-b12-link.telia.net               0.0%    10   85.3  85.3  85.3  85.4   0.0
     14.|-- vultr-ic-312911-ffm-b12.c.telia.net  0.0%    10   81.7  82.1  81.6  84.0   0.6
     15.|-- ???                                 100.0    10    0.0   0.0   0.0   0.0   0.0
     16.|-- 108.61.210.117.vultr.com             0.0%    10   81.6  81.6  81.6  81.7   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 ams-nl-ping.vultr.com
    Start: Mon Feb 26 08:58:09 2018
    HOST: hostname                Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                             0.0%    10  103.4  56.0  20.0 103.4  33.1
      2.|-- 147.75.36.4                         0.0%    10   14.9 227.2  11.5 1122. 421.2
      3.|-- 0.ae22.bbr2.ewr1.packet.net         0.0%    10    0.2   0.2   0.1   0.3   0.0
      4.|-- packet.tbr2.wbr.nac.net             0.0%    10    0.4  17.5   0.3 170.9  53.9
      5.|-- 0.lag16.tbr1.wbr.nac.net            0.0%    10    0.2   2.6   0.2  11.4   4.1
      6.|-- 0.e3-4.tbr1.tl9.nac.net             0.0%    10    1.1   1.5   1.1   4.6   0.8
      7.|-- nyk-b3-link.telia.net               0.0%    10    1.4   3.3   1.2  20.8   6.1
      8.|-- nyk-bb3-link.telia.net              0.0%    10    1.3   1.7   1.1   6.3   1.5
      9.|-- ldn-bb3-link.telia.net              0.0%    10   71.0  72.8  71.0  82.1   3.5
     10.|-- adm-bb3-link.telia.net              0.0%    10   79.1  79.2  79.0  80.3   0.3
     11.|-- adm-b3-link.telia.net               0.0%    10   79.4  81.2  79.3  95.7   5.1
     12.|-- vultr-ic-313751-adm-b3.c.telia.net  0.0%    10   79.5  79.5  79.4  79.6   0.0
     13.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
     14.|-- 108.61.198.102.vultr.com            0.0%    10   79.5  79.6  79.4  79.7   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 lon-gb-ping.vultr.com
    Start: Mon Feb 26 08:58:25 2018
    HOST: hostname                Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                             0.0%    10   16.5  31.2  16.5  40.9   7.0
      2.|-- 147.75.36.2                         0.0%    10   13.9  94.2  12.6 383.6 134.4
      3.|-- 0.ae21.bbr2.ewr1.packet.net         0.0%    10    0.2   0.3   0.2   1.3   0.0
      4.|-- packet.tbr2.wbr.nac.net             0.0%    10    0.4   6.4   0.3  60.0  18.8
      5.|-- 0.lag16.tbr1.wbr.nac.net            0.0%    10    2.4   3.7   0.2   9.1   3.2
      6.|-- 0.e3-4.tbr1.tl9.nac.net             0.0%    10    1.1   2.1   1.1   5.2   1.3
      7.|-- nyk-b3-link.telia.net               0.0%    10    1.2   1.2   1.2   1.3   0.0
      8.|-- nyk-bb3-link.telia.net              0.0%    10    1.4   1.3   1.2   1.4   0.0
      9.|-- ldn-bb3-link.telia.net              0.0%    10   71.1  71.3  71.0  73.2   0.6
     10.|-- ldn-b9-link.telia.net               0.0%    10   71.0  71.0  70.9  71.3   0.0
     11.|-- vultr-ic-312913-ldn-b9.c.telia.net  0.0%    10   72.2  72.4  71.7  75.1   0.9
     12.|-- 10.69.1.90                         20.0%    10   78.5  77.8  72.1 107.5  12.2
     13.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
     14.|-- 108.61.196.101.vultr.com            0.0%    10   71.2  71.2  71.2  71.3   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 par-fr-ping.vultr.com
    Start: Mon Feb 26 08:58:42 2018
    HOST: hostname                Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                             0.0%    10   24.0  15.6  11.9  24.0   3.3
      2.|-- 147.75.36.2                         0.0%    10   21.3 258.8  13.4 847.0 375.6
      3.|-- 0.ae21.bbr2.ewr1.packet.net         0.0%    10   26.6   2.8   0.2  26.6   8.3
      4.|-- packet.tbr2.wbr.nac.net             0.0%    10    0.4   0.5   0.3   1.1   0.0
      5.|-- 0.lag16.tbr1.wbr.nac.net            0.0%    10    0.3   0.2   0.2   0.3   0.0
      6.|-- 0.e3-4.tbr1.tl9.nac.net             0.0%    10   10.2   4.5   1.2  11.5   4.2
      7.|-- nyk-b3-link.telia.net               0.0%    10    1.2   1.3   1.2   1.8   0.0
      8.|-- nyk-bb4-link.telia.net              0.0%    10    1.8   1.9   1.8   2.1   0.0
      9.|-- prs-bb4-link.telia.net              0.0%    10   72.3  74.2  72.1  82.2   3.6
     10.|-- prs-b7-link.telia.net               0.0%    10   75.6  73.2  72.2  75.6   1.4
     11.|-- vultr-ic-312910-prs-b7.c.telia.net  0.0%    10   85.5  85.7  85.4  86.4   0.0
     12.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
     13.|-- 108.61.209.127.vultr.com            0.0%    10   85.5  85.4  85.4  85.5   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 VersaWeb Las Vegas
    Start: Mon Feb 26 08:58:57 2018
    HOST: hostname                    Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                                 0.0%    10   14.9  16.2  12.5  22.2   3.3
      2.|-- 147.75.36.2                             0.0%    10  986.3 252.3  10.3 986.3 391.4
      3.|-- 0.ae11.bbr1.ewr1.packet.net             0.0%    10    0.2   4.8   0.2  45.6  14.3
      4.|-- 10gigabitethernet1-2.core1.nyc6.he.net  0.0%    10    1.5   3.3   1.4   8.3   2.7
      5.|-- 100ge13-1.core1.nyc4.he.net             0.0%    10   12.2   3.2   1.4  12.2   3.5
      6.|-- 100ge16-1.core1.ash1.he.net             0.0%    10    6.5   6.8   6.4   9.2   0.7
      7.|-- 100ge14-2.core1.mci3.he.net             0.0%    10   30.6  32.6  30.5  41.6   3.7
      8.|-- 100ge12-1.core1.den1.he.net             0.0%    10   52.6  45.2  42.6  55.8   4.8
      9.|-- 100ge10-2.core1.slc1.he.net             0.0%    10   55.9  56.6  54.1  61.5   2.6
     10.|-- 10ge1-1.core1.las1.he.net               0.0%    10   70.3  66.6  62.9  73.3   4.6
     11.|-- 172.18.0.22                             0.0%    10   76.3  77.8  76.2  84.4   3.2
     12.|-- 192.228.109.121                         0.0%    10   76.3  76.6  76.2  79.1   0.7
     13.|-- 192.228.109.86                          0.0%    10   76.4  86.3  76.4 172.4  30.2
     14.|-- lg.las1.ultravps.net                    0.0%    10   76.2  76.2  76.2  76.3   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 VersaWeb Seattle
    Start: Mon Feb 26 08:59:15 2018
    HOST: hostname                       Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                                    0.0%    10   15.2  14.7   4.9  21.8   4.8
      2.|-- 147.75.36.4                                0.0%    10   22.2 218.7  11.8 1067. 412.9
      3.|-- 0.ae22.bbr2.ewr1.packet.net                0.0%    10    0.2   0.5   0.2   3.2   0.9
      4.|-- packet.tbr2.wbr.nac.net                    0.0%    10    0.4   0.4   0.3   0.5   0.0
      5.|-- 0.lag16.tbr1.wbr.nac.net                   0.0%    10    8.1   4.8   0.2  12.6   5.1
      6.|-- 0.e1-3.tbr1.ewr.nac.net                    0.0%    10    1.0   2.9   0.9   7.7   2.2
      7.|-- te0-0-0-24.rcr21.ewr01.atlas.cogentco.com  0.0%    10    1.6   1.5   1.2   2.6   0.3
      8.|-- be3200.rcr24.jfk01.atlas.cogentco.com      0.0%    10    1.7   1.7   1.7   1.8   0.0
      9.|-- be2897.ccr42.jfk02.atlas.cogentco.com      0.0%    10    1.8   1.8   1.5   2.0   0.0
     10.|-- be2890.ccr22.cle04.atlas.cogentco.com      0.0%    10   18.6  18.4  18.2  18.7   0.0
     11.|-- be2718.ccr42.ord01.atlas.cogentco.com      0.0%    10   20.1  19.9  19.7  20.1   0.0
     12.|-- be2832.ccr22.mci01.atlas.cogentco.com      0.0%    10   48.1  48.1  47.9  48.3   0.0
     13.|-- be3036.ccr22.den01.atlas.cogentco.com      0.0%    10   59.8  59.7  59.4  59.9   0.0
     14.|-- be3038.ccr21.slc01.atlas.cogentco.com      0.0%    10   72.4  72.5  72.1  72.7   0.0
     15.|-- be2085.ccr21.sea02.atlas.cogentco.com      0.0%    10   66.9  66.8  66.8  67.0   0.0
     16.|-- 38.142.48.250                              0.0%    10   66.4  66.4  66.4  66.5   0.0
     17.|-- 76.164.234.1                               0.0%    10   66.5  66.5  66.5  66.8   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 OVH Canada
    Start: Mon Feb 26 08:59:34 2018
    HOST: hostname    Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                 0.0%    10    1.4   4.3   1.4  11.1   2.7
      2.|-- 2604:1380:2:79::        0.0%    10   10.1   5.4   0.7  10.1   3.7
      3.|-- 2604:1380:2:20::        0.0%    10    0.2   0.2   0.2   0.2   0.0
      4.|-- de-cix.nyc.ovh.com      0.0%    10    1.8   2.0   1.8   2.3   0.0
      5.|-- 2607:5300::1c6          0.0%    10   10.8  16.0   9.8  68.4  18.4
      6.|-- 2607:5300::1c3          0.0%    10    8.8   8.9   8.8   9.0   0.0
      7.|-- 2001:41d0:0:50::6:835   0.0%    10    9.0   8.9   8.9   9.0   0.0
      8.|-- 2001:41d0:0:50::2:404d  0.0%    10    9.4  10.0   9.3  12.2   0.7
      9.|-- 2607:5300:60:44a5::1    0.0%    10    9.0   8.9   8.9   9.0   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 10gbps.io Frankfurt:
    Start: Mon Feb 26 08:59:50 2018
    HOST: hostname              Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                           0.0%    10   20.8  27.6  11.2  49.1  10.0
      2.|-- 147.75.36.2                       0.0%    10   12.1  53.2  11.4 266.6  83.1
      3.|-- 0.ae21.bbr2.ewr1.packet.net       0.0%    10    6.4   0.8   0.1   6.4   1.9
      4.|-- ae4-1.nyk10.core-backbone.com     0.0%    10    1.1   1.1   1.1   1.3   0.0
      5.|-- ae2-2025.fra30.core-backbone.com  0.0%    10   81.0  80.9  80.8  81.1   0.0
      6.|-- core-backbone.cdn77.com           0.0%    10   93.2  93.2  93.2  93.3   0.0
      7.|-- frankfurt-20.cdn77.com            0.0%    10   93.2  93.2  93.1  93.3   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 10gbps.io London:
    Start: Mon Feb 26 09:00:09 2018
    HOST: hostname               Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                            0.0%    10   12.4  15.7  11.9  23.8   3.7
      2.|-- 147.75.36.4                        0.0%    10  149.7 192.4  11.3 990.1 346.4
      3.|-- 0.ae12.bbr1.ewr1.packet.net        0.0%    10    0.2   2.3   0.2  21.2   6.6
      4.|-- 0.ae0.bbr2.ewr1.packet.net         0.0%    10    0.5   0.4   0.2   1.5   0.3
      5.|-- ae4-1.nyk10.core-backbone.com      0.0%    10    1.1   1.5   1.1   5.1   1.2
      6.|-- ae10-2092.lon10.core-backbone.com  0.0%    10   70.3  70.4  70.3  70.6   0.0
      7.|-- core-backbone.cdn77.com            0.0%    10   93.8  93.8  93.7  93.8   0.0
      8.|-- london-20.cdn77.com                0.0%    10   94.6  94.6  94.5  94.7   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 10gbps.io Paris:
    Start: Mon Feb 26 09:00:29 2018
    HOST: hostname                 Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                              0.0%    10   19.0  16.2   8.6  21.9   3.8
      2.|-- 147.75.36.2                          0.0%    10  845.2 195.9  11.4 845.2 311.7
      3.|-- 0.ae21.bbr2.ewr1.packet.net          0.0%    10    0.2   1.6   0.2  13.8   4.3
      4.|-- 0.xe-0-0-0.csr1.ewr2.packet.net      0.0%    10    3.4  12.0   3.4  62.3  17.8
      6.|-- 0.et-10-0-2.BR2.NYC4.ALTER.NET       0.0%    10    3.2   3.9   2.5  10.7   2.6
      7.|-- verizon.com.customer.alter.net       0.0%    10    2.8   2.9   2.7   3.1   0.0
      8.|-- ae-3.r24.nycmny01.us.bb.gin.ntt.net 10.0%    10    2.8   3.5   2.8   8.6   1.8
      9.|-- ae-9.r24.londen12.uk.bb.gin.ntt.net  0.0%    10   74.6  74.7  74.6  74.9   0.0
     10.|-- ae-1.r25.londen12.uk.bb.gin.ntt.net  0.0%    10   75.0  75.1  74.9  76.1   0.0
     11.|-- ae-2.r04.parsfr01.fr.bb.gin.ntt.net  0.0%    10   92.3  92.4  92.3  92.7   0.0
     12.|-- ae-5.r03.parsfr02.fr.bb.gin.ntt.net  0.0%    10   92.6  92.3  92.2  92.6   0.0
     13.|-- ae-8.r02.parsfr02.fr.bb.gin.ntt.net  0.0%    10   79.6  79.6  79.4  79.8   0.0
     14.|-- 81.25.206.50                         0.0%    10  101.0 101.1 101.0 101.3   0.0
     15.|-- unn-185-93-2-193.10gbps.io           0.0%    10  101.2 101.2 101.1 101.5   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 10gbps.io Singapore:
    Start: Mon Feb 26 09:00:47 2018
    HOST: hostname                       Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                                    0.0%    10   13.0  26.6  12.7 112.0  30.1
      2.|-- 147.75.36.2                                0.0%    10   19.8 154.7  12.2 1016. 314.2
      3.|-- 0.ae21.bbr2.ewr1.packet.net                0.0%    10    0.2   2.9   0.2  26.4   8.2
      4.|-- packet.tbr2.wbr.nac.net                    0.0%    10    0.4   0.9   0.3   5.1   1.3
      5.|-- 0.lag16.tbr1.wbr.nac.net                   0.0%    10    0.2   1.0   0.2   4.5   1.4
      6.|-- 0.e1-3.tbr1.ewr.nac.net                    0.0%    10   11.5   2.6   0.9  11.5   3.5
      7.|-- te0-0-0-24.rcr21.ewr01.atlas.cogentco.com  0.0%    10    2.5   1.4   1.1   2.5   0.3
      8.|-- be3199.rcr23.jfk01.atlas.cogentco.com      0.0%    10    1.8   1.9   1.7   2.0   0.0
      9.|-- be2896.ccr41.jfk02.atlas.cogentco.com      0.0%    10    1.6   1.7   1.5   1.9   0.0
     10.|-- be2806.ccr41.dca01.atlas.cogentco.com      0.0%    10    7.7   7.8   7.5   8.0   0.0
     11.|-- be2112.ccr41.atl01.atlas.cogentco.com      0.0%    10   19.8  19.8  19.6  20.0   0.0
     12.|-- be2687.ccr41.iah01.atlas.cogentco.com      0.0%    10   45.5  45.4  45.2  45.6   0.0
     13.|-- be2927.ccr21.elp01.atlas.cogentco.com      0.0%    10   73.3  73.5  73.3  73.7   0.0
     14.|-- be2929.ccr31.phx01.atlas.cogentco.com      0.0%    10   73.7  73.6  73.3  73.8   0.0
     15.|-- be2931.ccr41.lax01.atlas.cogentco.com      0.0%    10   73.7  73.6  73.3  73.8   0.0
     16.|-- be3271.ccr41.lax04.atlas.cogentco.com      0.0%    10   75.2  75.3  75.1  75.6   0.0
     17.|-- be3406.ccr21.sin01.atlas.cogentco.com      0.0%    10  246.7 246.9 246.7 247.9   0.3
     18.|-- 154.18.2.2                                 0.0%    10  245.0 245.0 245.0 245.2   0.0
     19.|-- hol14.homeoflim.com                        0.0%    10  251.5 251.5 251.4 251.7   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 10gbps.io Stockholm:
    Start: Mon Feb 26 09:01:04 2018
    HOST: hostname              Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                           0.0%    10  117.1  31.1  14.0 117.1  32.0
      2.|-- 147.75.36.4                       0.0%    10   15.2  88.7  12.8 699.4 214.8
      3.|-- 0.ae12.bbr1.ewr1.packet.net       0.0%    10    0.3   0.6   0.2   3.1   0.7
      4.|-- 0.ae0.bbr2.ewr1.packet.net        0.0%    10    0.2   1.3   0.2  11.6   3.5
      5.|-- ae4-1.nyk10.core-backbone.com     0.0%    10    1.1   1.7   1.1   6.6   1.6
      6.|-- ae1-2081.sth10.core-backbone.com  0.0%    10  101.4 101.5 101.3 103.2   0.3
      7.|-- core-backbone.cdn77.com           0.0%    10  101.0 101.0 101.0 101.1   0.0
      8.|-- stockholm-10.cdn77.com            0.0%    10  108.2 108.2 108.2 108.4   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 10gbps.io Moscow:
    Start: Mon Feb 26 09:01:22 2018
    HOST: hostname         Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                      0.0%    10   16.1  15.6  13.4  19.9   2.4
      2.|-- 147.75.36.2                  0.0%    10   19.9 190.8  14.3 1136. 361.8
      3.|-- 0.ae21.bbr2.ewr1.packet.net  0.0%    10    0.2   0.2   0.2   0.2   0.0
      4.|-- de-cix.nyc.retn.net          0.0%    10    1.1   1.1   1.0   1.1   0.0
      5.|-- ae1-3.RT.MR.MSK.RU.retn.net  0.0%    10  114.7 114.5 114.5 114.7   0.0
      6.|-- 87.245.204.20                0.0%    10  113.6 113.5 113.5 113.6   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 10gbps.io Miami:
    Start: Mon Feb 26 09:01:37 2018
    HOST: hostname                 Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                              0.0%    10   13.1  14.4   8.4  19.1   3.0
      2.|-- 147.75.36.4                          0.0%    10   17.8 173.9  10.4 903.0 332.7
      3.|-- 0.ae22.bbr2.ewr1.packet.net          0.0%    10    0.3   3.7   0.2  34.5  10.8
      4.|-- 0.xe-0-0-0.csr1.ewr2.packet.net      0.0%    10    2.9   6.1   1.9  10.4   3.4
      6.|-- 0.et-10-0-2.BR2.NYC4.ALTER.NET       0.0%    10    2.7   3.1   2.6   6.7   1.2
      7.|-- verizon.com.customer.alter.net       0.0%    10    2.9   2.9   2.7   3.1   0.0
      8.|-- ae-3.r25.nycmny01.us.bb.gin.ntt.net  0.0%    10    2.9   3.7   2.8   9.7   2.1
      9.|-- ae-9.r22.asbnva02.us.bb.gin.ntt.net  0.0%    10    8.4   8.4   8.3   8.6   0.0
     10.|-- ae-0.r23.asbnva02.us.bb.gin.ntt.net  0.0%    10   10.4   9.0   8.4  10.4   0.5
     11.|-- ae-1.r20.miamfl02.us.bb.gin.ntt.net  0.0%    10   65.8  45.9  38.6  65.8   9.4
     12.|-- ae-1.r04.miamfl02.us.bb.gin.ntt.net  0.0%    10   38.6  38.6  38.5  38.9   0.0
     13.|-- 165.254.21.26                        0.0%    10   37.8  37.8  37.8  37.9   0.0
     14.|-- unn-195-181-162-195.cdn77.com        0.0%    10   55.7  55.8  55.7  56.0   0.0
    
    

    Code (Text):
    -------------------------------------------
    Check system entropy pool availability
    -------------------------------------------
    
    entropy_avail: 4067
    entropy_avail: 4067
    entropy_avail: 4067
    entropy_avail: 4067
    

    Code (Text):
    -------------------------------------------
    OpenSSL System Benchmark
    -------------------------------------------
    
    OpenSSL 1.0.2k-fips  26 Jan 2017
    -------------------------------------------
    openssl speed -multi 48 rsa4096 rsa2048 ecdsap256 sha256 sha1 md5 rc4 aes-256-cbc aes-128-cbc
    808.00:111310165.33 from 17
    OpenSSL 1.0.2k-fips  26 Jan 2017
    built on: reproducible build, date unspecified
    options:bn(64,64) md2(int) rc4(8x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
    compiler: gcc -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DKRB5_MIT -m64 -DL_ENDIAN -Wall -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches   -m64 -mtune=generic -Wa,--noexecstack -DPURIFY -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DRC4_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM
    md5            1711832.20k  5308997.99k 12297699.75k 18615076.52k 21547302.91k
    sha1           2643534.69k  9116932.97k 24249715.46k 41606732.12k 51272840.53k
    rc4           14108966.49k 17533703.91k 18091231.69k 18160244.39k 18219786.24k
    aes-128 cbc    2889747.50k  3094950.14k  3190846.89k  7211115.63k  7326624.43k
    aes-256 cbc    2107697.37k  2221035.03k  2260734.21k  5312491.86k  5346129.24k
    sha256         5275014.02k 14868248.87k 33555368.62k 50065332.57k 58719780.86k
                      sign    verify    sign/s verify/s
    rsa 2048 bits 0.000042s 0.000001s  23753.6 778961.7
    rsa 4096 bits 0.000301s 0.000005s   3321.7 210777.5
                                  sign    verify    sign/s verify/s
     256 bit ecdsa (nistp256)   0.0000s   0.0000s 521739.1 223621.8
    -------------------------------------------
    openssl speed -evp aes256 -multi 48
    OpenSSL 1.0.2k-fips  26 Jan 2017
    built on: reproducible build, date unspecified
    options:bn(64,64) md2(int) rc4(8x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
    compiler: gcc -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DKRB5_MIT -m64 -DL_ENDIAN -Wall -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches   -m64 -mtune=generic -Wa,--noexecstack -DPURIFY -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DRC4_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM
    evp           22921436.58k 28635047.76k 30956237.91k 31749383.51k 32001492.83k
    -------------------------------------------
    openssl speed -evp aes128 -multi 48
    OpenSSL 1.0.2k-fips  26 Jan 2017
    built on: reproducible build, date unspecified
    options:bn(64,64) md2(int) rc4(8x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
    compiler: gcc -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DKRB5_MIT -m64 -DL_ENDIAN -Wall -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches   -m64 -mtune=generic -Wa,--noexecstack -DPURIFY -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DRC4_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM
    evp           27692549.66k 38683828.80k 41936650.41k 43406140.76k 43849318.40k
    

    Code (Text):
    -------------------------------------------
    Running mysqlslap
    -------------------------------------------
    
    mysqlslap --auto-generate-sql --auto-generate-sql-add-autoincrement --auto-generate-sql-secondary-indexes=5 --number-int-cols=5 --number-char-cols=5 --number-of-queries=25000 --auto-generate-sql-unique-query-number=40 --auto-generate-sql-unique-write-number=40 --auto-generate-sql-write-number=1000 --concurrency=64 --iterations=10 --engine=myisam
    Benchmark
            Running for engine myisam
            Average number of seconds to run all queries: 1.459 seconds
            Minimum number of seconds to run all queries: 1.405 seconds
            Maximum number of seconds to run all queries: 1.509 seconds
            Number of clients running queries: 64
            Average number of queries per client: 390
    
    Dropping the database is potentially a very bad thing to do.
    Any data stored in the database will be destroyed.
    
    Do you really want to drop the 'test' database [y/N] Database "test" dropped
    

    Code (Text):
    -------------------------------------------
    System PHP Info
    -------------------------------------------
    
    CPU: 48 x AMD EPYC 7401P 24-Core Processor
                  total        used        free      shared  buff/cache   available
    Mem:          64159        3101       60371          30         686       60633
    Low:          64159        3788       60371
    High:             0           0           0
    Swap:          1949           0        1949
    ----------------------------------------------
    PHP 5.6.33 (cli) (built: Feb 26 2018 08:24:24)
    Copyright (c) 1997-2016 The PHP Group
    Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
        with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2016, by Zend Technologies
    ----------------------------------------------
    Configuration File (php.ini) Path: /usr/local/lib
    Loaded Configuration File:         /usr/local/lib/php.ini
    Scan for additional .ini files in: /etc/centminmod/php.d
    Additional .ini files parsed:      /etc/centminmod/php.d/a_customphp.ini,
    /etc/centminmod/php.d/curlcainfo.ini,
    /etc/centminmod/php.d/geoip.ini,
    /etc/centminmod/php.d/igbinary.ini,
    /etc/centminmod/php.d/imagick.ini,
    /etc/centminmod/php.d/mailparse.ini,
    /etc/centminmod/php.d/memcache.ini,
    /etc/centminmod/php.d/memcached.ini,
    /etc/centminmod/php.d/redis.ini,
    /etc/centminmod/php.d/zendopcache.ini
    
    ----------------------------------------------
    [PHP Modules]
    bcmath
    bz2
    calendar
    Core
    ctype
    curl
    date
    dom
    enchant
    ereg
    exif
    filter
    ftp
    gd
    geoip
    gettext
    gmp
    hash
    iconv
    igbinary
    imagick
    imap
    intl
    json
    ldap
    libxml
    mailparse
    mbstring
    mcrypt
    memcache
    memcached
    mhash
    mysql
    mysqli
    mysqlnd
    openssl
    pcntl
    pcre
    PDO
    pdo_mysql
    pdo_sqlite
    Phar
    posix
    pspell
    readline
    redis
    Reflection
    session
    shmop
    SimpleXML
    snmp
    soap
    sockets
    SPL
    sqlite3
    standard
    sysvmsg
    sysvsem
    sysvshm
    tidy
    tokenizer
    xml
    xmlreader
    xmlrpc
    xmlwriter
    xsl
    Zend OPcache
    zip
    zlib
    
    [Zend Modules]
    Zend OPcache
    
    ----------------------------------------------
    -------------------------------------------
    Run PHP test Zend/bench.php
    -------------------------------------------
    
    simple             0.127
    simplecall         0.104
    simpleucall        0.110
    simpleudcall       0.115
    mandel             0.261
    mandel2            0.312
    ackermann(7)       0.116
    ary(50000)         0.028
    ary2(50000)        0.027
    ary3(2000)         0.132
    fibo(30)           0.387
    hash1(50000)       0.035
    hash2(500)         0.026
    heapsort(20000)    0.080
    matrix(20)         0.161
    nestedloop(12)     0.214
    sieve(30)          0.081
    strcat(200000)     0.012
    ------------------------
    Total              2.326
    real: 2.43s user: 2.23s sys: 0.19s cpu: 99% maxmem: 31304 KB cswaits: 8
    
    simple             0.127
    simplecall         0.104
    simpleucall        0.109
    simpleudcall       0.114
    mandel             0.315
    mandel2            0.306
    ackermann(7)       0.116
    ary(50000)         0.028
    ary2(50000)        0.027
    ary3(2000)         0.133
    fibo(30)           0.383
    hash1(50000)       0.034
    hash2(500)         0.026
    heapsort(20000)    0.080
    matrix(20)         0.165
    nestedloop(12)     0.214
    sieve(30)          0.080
    strcat(200000)     0.012
    ------------------------
    Total              2.374
    real: 2.48s user: 2.28s sys: 0.20s cpu: 99% maxmem: 31300 KB cswaits: 3
    
    simple             0.125
    simplecall         0.104
    simpleucall        0.109
    simpleudcall       0.114
    mandel             0.259
    mandel2            0.312
    ackermann(7)       0.116
    ary(50000)         0.028
    ary2(50000)        0.027
    ary3(2000)         0.132
    fibo(30)           0.380
    hash1(50000)       0.034
    hash2(500)         0.026
    heapsort(20000)    0.079
    matrix(20)         0.168
    nestedloop(12)     0.213
    sieve(30)          0.080
    strcat(200000)     0.012
    ------------------------
    Total              2.317
    real: 2.42s user: 2.24s sys: 0.18s cpu: 99% maxmem: 31300 KB cswaits: 3
    
    bench.php results from 3 runs
    2.326
    2.374
    2.317
    
    bench.php avg: 2.3390
    Avg: real: 2.44s user: 2.25s sys: 0.19s cpu: 99.00% maxmem: 31301.33KB cswaits: 4.67
    created results log at /home/phpbench_logs/bench_260218-084535.log
    server PHP info log at /home/phpbench_logs/bench_phpinfo_260218-084535.log
    
    -------------------------------------------
    Run PHP test Zend/micro_bench.php
    -------------------------------------------
    
    empty_loop         0.152
    func()             0.335    0.183
    undef_func()       0.340    0.188
    int_func()         0.283    0.132
    $x = self::$x      0.229    0.077
    self::$x = 0       0.241    0.089
    isset(self::$x)    0.239    0.087
    empty(self::$x)    0.257    0.105
    $x = Foo::$x       0.220    0.068
    Foo::$x = 0        0.228    0.076
    isset(Foo::$x)     0.216    0.064
    empty(Foo::$x)     0.227    0.075
    self::f()          0.374    0.222
    Foo::f()           0.305    0.153
    $x = $this->x      0.217    0.065
    $this->x = 0       0.279    0.128
    $this->x += 2      0.232    0.080
    ++$this->x         0.226    0.074
    --$this->x         0.233    0.081
    $this->x++         0.254    0.102
    $this->x--         0.266    0.114
    isset($this->x)    0.340    0.188
    empty($this->x)    0.246    0.094
    $this->f()         0.416    0.264
    $x = Foo::TEST     0.231    0.079
    new Foo()          0.643    0.491
    $x = TEST          0.199    0.047
    $x = $_GET         0.211    0.059
    $x = $GLOBALS['v'] 0.286    0.134
    $x = $hash['v']    0.244    0.092
    $x = $str[0]       0.319    0.167
    $x = $a ?: null    0.226    0.074
    $x = $f ?: tmp     0.304    0.152
    $x = $f ? $f : $a  0.231    0.079
    $x = $f ? $f : tmp 0.305    0.153
    ------------------------
    Total              9.553
    real: 9.66s user: 9.57s sys: 0.08s cpu: 99% maxmem: 19396 KB cswaits: 4
    
    empty_loop         0.152
    func()             0.330    0.178
    undef_func()       0.352    0.200
    int_func()         0.302    0.150
    $x = self::$x      0.258    0.106
    self::$x = 0       0.264    0.112
    isset(self::$x)    0.232    0.080
    empty(self::$x)    0.258    0.106
    $x = Foo::$x       0.228    0.076
    Foo::$x = 0        0.228    0.076
    isset(Foo::$x)     0.216    0.064
    empty(Foo::$x)     0.227    0.075
    self::f()          0.372    0.220
    Foo::f()           0.325    0.173
    $x = $this->x      0.229    0.078
    $this->x = 0       0.281    0.129
    $this->x += 2      0.231    0.079
    ++$this->x         0.235    0.083
    --$this->x         0.239    0.087
    $this->x++         0.255    0.103
    $this->x--         0.273    0.121
    isset($this->x)    0.325    0.173
    empty($this->x)    0.237    0.086
    $this->f()         0.399    0.248
    $x = Foo::TEST     0.243    0.091
    new Foo()          0.674    0.522
    $x = TEST          0.187    0.035
    $x = $_GET         0.207    0.055
    $x = $GLOBALS['v'] 0.286    0.134
    $x = $hash['v']    0.242    0.090
    $x = $str[0]       0.322    0.170
    $x = $a ?: null    0.227    0.075
    $x = $f ?: tmp     0.303    0.151
    $x = $f ? $f : $a  0.226    0.074
    $x = $f ? $f : tmp 0.304    0.152
    ------------------------
    Total              9.672
    real: 9.77s user: 9.70s sys: 0.07s cpu: 99% maxmem: 19400 KB cswaits: 3
    
    empty_loop         0.152
    func()             0.334    0.182
    undef_func()       0.357    0.204
    int_func()         0.303    0.151
    $x = self::$x      0.236    0.083
    self::$x = 0       0.238    0.085
    isset(self::$x)    0.234    0.082
    empty(self::$x)    0.234    0.082
    $x = Foo::$x       0.224    0.072
    Foo::$x = 0        0.228    0.076
    isset(Foo::$x)     0.216    0.064
    empty(Foo::$x)     0.227    0.075
    self::f()          0.365    0.213
    Foo::f()           0.327    0.175
    $x = $this->x      0.228    0.076
    $this->x = 0       0.289    0.137
    $this->x += 2      0.235    0.083
    ++$this->x         0.232    0.080
    --$this->x         0.236    0.084
    $this->x++         0.241    0.089
    $this->x--         0.257    0.105
    isset($this->x)    0.326    0.174
    empty($this->x)    0.247    0.095
    $this->f()         0.416    0.264
    $x = Foo::TEST     0.252    0.100
    new Foo()          0.686    0.534
    $x = TEST          0.197    0.045
    $x = $_GET         0.207    0.055
    $x = $GLOBALS['v'] 0.286    0.134
    $x = $hash['v']    0.241    0.089
    $x = $str[0]       0.322    0.170
    $x = $a ?: null    0.221    0.068
    $x = $f ?: tmp     0.335    0.182
    $x = $f ? $f : $a  0.237    0.085
    $x = $f ? $f : tmp 0.325    0.173
    ------------------------
    Total              9.692
    real: 9.79s user: 9.72s sys: 0.07s cpu: 99% maxmem: 19396 KB cswaits: 3
    
    micro_bench.php results from 3 runs
    9.553
    9.672
    9.692
    
    micro_bench.php avg: 9.6390
    Avg: real: 9.74s user: 9.66s sys: 0.07s cpu: 99.00% maxmem: 19397.33KB cswaits: 3.33
    created results log at /home/phpbench_logs/bench_micro_260218-084535.log
    server PHP info log at /home/phpbench_logs/bench_phpinfo_260218-084535.log
    
    

    Code (Text):
    ========================================================================
       BYTE UNIX Benchmarks (Version 5.1.3)
    
       System: hostname: GNU/Linux
       OS: GNU/Linux -- 3.10.0-693.17.1.el7.x86_64 -- #1 SMP Thu Jan 25 20:13:58 UTC 2018
       Machine: x86_64 (x86_64)
       Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
       CPU 0: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 1: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 2: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 3: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 4: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 5: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 6: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 7: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 8: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 9: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 10: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 11: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 12: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 13: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 14: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 15: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 16: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 17: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 18: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 19: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 20: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 21: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 22: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 23: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 24: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 25: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 26: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 27: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 28: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 29: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 30: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 31: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 32: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 33: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 34: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 35: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 36: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 37: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 38: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 39: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 40: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 41: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 42: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 43: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 44: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 45: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 46: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 47: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       09:06:52 up  1:06,  2 users,  load average: 13.41, 17.36, 8.27; runlevel 3
    

    Code (Text):
    ------------------------------------------------------------------------
    Benchmark Run: Mon Feb 26 2018 09:06:52 - 09:31:02
    48 CPUs in system; running 1 parallel copy of tests
    
    Dhrystone 2 using register variables       34980535.2 lps   (10.0 s, 7 samples)
    Double-Precision Whetstone                     4338.1 MWIPS (11.0 s, 7 samples)
    Execl Throughput                               1095.8 lps   (30.0 s, 2 samples)
    Pipe Throughput                              128455.4 lps   (10.0 s, 7 samples)
    Pipe-based Context Switching                  62619.1 lps   (10.0 s, 7 samples)
    Process Creation                               2893.3 lps   (30.0 s, 2 samples)
    Shell Scripts (1 concurrent)                   2991.4 lpm   (60.0 s, 2 samples)
    Shell Scripts (16 concurrent)                   893.4 lpm   (60.0 s, 2 samples)
    Shell Scripts (8 concurrent)                   1419.4 lpm   (60.0 s, 2 samples)
    System Call Overhead                          77396.8 lps   (10.0 s, 7 samples)
    
    System Benchmarks Partial Index              BASELINE       RESULT    INDEX
    Dhrystone 2 using register variables         116700.0   34980535.2   2997.5
    Double-Precision Whetstone                       55.0       4338.1    788.7
    Execl Throughput                                 43.0       1095.8    254.8
    Pipe Throughput                               12440.0     128455.4    103.3
    Pipe-based Context Switching                   4000.0      62619.1    156.5
    Process Creation                                126.0       2893.3    229.6
    Shell Scripts (1 concurrent)                     42.4       2991.4    705.5
    Shell Scripts (16 concurrent)                     ---        893.4      ---
    Shell Scripts (8 concurrent)                      6.0       1419.4   2365.7
    System Call Overhead                          15000.0      77396.8     51.6
                                                                       ========
    System Benchmarks Index Score (Partial Only)                          386.5
    

    Code (Text):
    ------------------------------------------------------------------------
    Benchmark Run: Mon Feb 26 2018 09:31:02 - 09:55:12
    48 CPUs in system; running 48 parallel copies of tests
    
    Dhrystone 2 using register variables      880376483.7 lps   (10.0 s, 7 samples)
    Double-Precision Whetstone                   165961.3 MWIPS (10.0 s, 7 samples)
    Execl Throughput                              20934.6 lps   (29.9 s, 2 samples)
    Pipe Throughput                             3310161.1 lps   (10.0 s, 7 samples)
    Pipe-based Context Switching                1285764.1 lps   (10.0 s, 7 samples)
    Process Creation                              61193.6 lps   (30.0 s, 2 samples)
    Shell Scripts (1 concurrent)                  49131.7 lpm   (60.0 s, 2 samples)
    Shell Scripts (16 concurrent)                  3363.2 lpm   (60.3 s, 2 samples)
    Shell Scripts (8 concurrent)                   6743.5 lpm   (60.1 s, 2 samples)
    System Call Overhead                        1830436.5 lps   (10.0 s, 7 samples)
    
    System Benchmarks Partial Index              BASELINE       RESULT    INDEX
    Dhrystone 2 using register variables         116700.0  880376483.7  75439.3
    Double-Precision Whetstone                       55.0     165961.3  30174.8
    Execl Throughput                                 43.0      20934.6   4868.5
    Pipe Throughput                               12440.0    3310161.1   2660.9
    Pipe-based Context Switching                   4000.0    1285764.1   3214.4
    Process Creation                                126.0      61193.6   4856.6
    Shell Scripts (1 concurrent)                     42.4      49131.7  11587.7
    Shell Scripts (16 concurrent)                     ---       3363.2      ---
    Shell Scripts (8 concurrent)                      6.0       6743.5  11239.2
    System Call Overhead                          15000.0    1830436.5   1220.3
                                                                       ========
    System Benchmarks Index Score (Partial Only)                         7478.4
    
    

    Code (Text):
    -------------------------------------------
    centminmodbench.sh completed
    -------------------------------------------
    
    centminmodbench.sh Total Run Time: 4161.857673157 seconds
    


    GCC 8.0.1 Compile & RPM Build Times



    Centmin Mod LEMP Stack's Nginx web server supports multiple GCC/Clang version compilers from GCC 4.8, 5.3, 6.3, 7.2 and 8.x and Clang 3.4, 4, 5, and 6. Currently testing custom GCC 8.0.1 compilations and RPM builds so let's see how the Packet.net bare metal server holds up.

    It took a total of ~2684 seconds or ~44.7 minutes to compile and build both binutils 2.30 + GCC 8.0.1 with Profile Guided Optimisations and build the custom CentOS 7 RPMs. Compared to above Intel Xeon E3-1578Lv5 at ~2716 seconds or ~45 minutes.

    Was expecting 48 cpu threads to be much faster, so not sure what's up. Maybe it's related to CentOS 7.4's 3.10 Linux Kernel and slower single 120GB SSD boot drive ?

    Code (Text):
    RPMs Built
    /svr-setup/binutils-gcc8-2.30-1.el7.x86_64.rpm
    /svr-setup/gcc8-pgo-8.0-1.el7.x86_64.rpm
    
    ls -lah /svr-setup | egrep 'gcc8-pgo-8.0|binutils-gcc8-2.30'
    -rw-r--r--  1 root      root       5.2M Feb 26 10:03 binutils-gcc8-2.30-1.el7.x86_64.rpm
    -rw-r--r--  1 root      root        39M Feb 26 10:44 gcc8-pgo-8.0-1.el7.x86_64.rpm
    
    Installed Packages
    Name        : binutils-gcc8
    Arch        : x86_64
    Version     : 2.30
    Release     : 1.el7
    Size        : 48 M
    Repo        : installed
    Summary     : binutils-gcc8 for centminmod.com LEMP stack installs
    URL         : https://centminmod.com
    License     : unknown
    Description : binutils-gcc8 for centminmod.com LEMP stacks
    
    
    * Mon Feb 26 2018 George Liu <centminmod.com> 2.30
    - Binutils 2.30 for centminmod.com LEMP stack installs
    

    Code (Text):
    Installed Packages
    Name        : gcc8-pgo
    Arch        : x86_64
    Version     : 8.0
    Release     : 1.el7
    Size        : 287 M
    Repo        : installed
    Summary     : gcc8-pgo for centminmod.com LEMP stack installs
    URL         : https://centminmod.com
    License     : unknown
    Description : gcc8-pgo for centminmod.com LEMP stacks
    
    
    * Mon Feb 26 2018 George Liu <centminmod.com> 8
    - GCC 8 for centminmod.com LEMP stack installs
    
    *************************************************
    Compile GCC Completed
    log: /root/centminlogs/tools-gcc-install-pgo_260218-095942.log
    *************************************************
    
    Total Binutils + GCC Install Time: 2684.645239389 seconds
    


    Other AMD EPYC 7401P Reviews



    Folks may also want to check out other AMD EPYC 7401P Reviews
     
    Last edited: Feb 27, 2018
  5. eva2000

    eva2000 Administrator Staff Member

    35,481
    7,828
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,067
    Local Time:
    6:41 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x

    Packet.net AMD EPYC 7401P With 4.15.5 Linux Kernel



    To see if CentOS 7.4's default 3.10 Linux Kernel was a factor, I updated to latest ELrepo 4.15.5-1 Kernel-ml mainline Linux Kernel package and rebooted the AMD EPYC 7401P server.
    Code (Text):
    uname -r
    4.15.5-1.el7.elrepo.x86_64
    


    Unfortunately, there were kernel related cpu lockups being reported
    Code (Text):
    48 x Dhrystone 2 using register variables  1 2 3 4 5 6 7 8 9 10
    48 x Double-Precision Whetstone  1 2 3 4 5 6 7 8 9 10
    48 x System Call Overhead  1 2 3 4 5 6 7 8 9 10
    48 x Pipe Throughput  1 2 3 4 5 6 7 8 9 10
    48 x Pipe-based Context Switching  1 2 3 4 5
    
    Message from [email protected] at Feb 26 12:39:47 ...
     kernel:watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [sshd:72234]
    
    Message from [email protected] at Feb 26 12:39:51 ...
     kernel:watchdog: BUG: soft lockup - CPU#21 stuck for 22s! [context1:72109]
    
    Message from [email protected] at Feb 26 12:40:03 ...
     kernel:watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [kworker/8:0:4089]
    
    Message from [email protected] at Feb 26 12:40:15 ...
     kernel:watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [sshd:72234]
    
    Message from [email protected] at Feb 26 12:40:19 ...
     kernel:watchdog: BUG: soft lockup - CPU#21 stuck for 22s! [context1:72109]
    
    Message from [email protected] at Feb 26 12:40:31 ...
     kernel:watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [kworker/8:0:4089]
    

    Then SSH session terminated itself. So using Serial Over SSH feature tried logging into the server and was met with a stream full of messages that bond0 NIC was down

    Code (Text):
     status down for interface enp1s0f0, disabling it in 200 ms
    ** 3 printk messages dropped **
    [ 4470.051815] bond0: link status down for interface enp1s0f0, disabling it in 200 ms
    ** 4 printk messages dropped **
    [ 4470.063819] bond0: link status down for interface enp1s0f0, disabling it in 200 ms
    ** 4 printk messages dropped **
    [ 4470.075830] bond0: link status down for interface enp1s0f0, disabling it in 200 ms
    ** 3 printk messages dropped **
    [ 4470.084888] bond0: link status down for interface enp1s0f0, disabling it in 200 ms
    ** 4 printk messages dropped **
    [ 4470.097820] bond0: link status down for interface enp1s0f0, disabling it in 200 ms
    ** 3 printk messages dropped **
    [ 4470.105818] bond0: link status down for interface enp1s0f0, disabling it in 200 ms
    ** 4 printk messages dropped **
    


    Bit of digging led me to soft lockup info at:
    On AMD EPYC 7401P there are 48 such watchdog threads
    Code (Text):
    ps uax | grep watchdog
    root         12  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/0]
    root         15  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/1]
    root         22  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/2]
    root         29  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/3]
    root         36  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/4]
    root         42  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/5]
    root         48  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/6]
    root         54  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/7]
    root         60  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/8]
    root         66  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/9]
    root         72  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/10]
    root         78  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/11]
    root         84  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/12]
    root         90  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/13]
    root         96  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/14]
    root        102  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/15]
    root        108  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/16]
    root        114  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/17]
    root        120  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/18]
    root        126  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/19]
    root        132  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/20]
    root        138  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/21]
    root        144  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/22]
    root        150  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/23]
    root        156  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/24]
    root        162  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/25]
    root        168  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/26]
    root        174  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/27]
    root        180  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/28]
    root        186  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/29]
    root        192  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/30]
    root        198  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/31]
    root        204  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/32]
    root        210  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/33]
    root        216  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/34]
    root        222  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/35]
    root        228  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/36]
    root        234  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/37]
    root        240  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/38]
    root        246  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/39]
    root        252  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/40]
    root        258  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/41]
    root        264  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/42]
    root        270  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/43]
    root        276  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/44]
    root        282  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/45]
    root        288  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/46]
    root        294  0.0  0.0      0     0 ?        S    13:01   0:00 [watchdog/47]
    root        320  0.0  0.0      0     0 ?        I<   13:01   0:00 [watchdogd]
    

    Default watchdog_thresh value is 10 so 2x10 = 20 seconds time limit
    Code (Text):
    sysctl -a 2>&1 | egrep 'watchdog|softlockup'
    kernel.nmi_watchdog = 1
    kernel.soft_watchdog = 1
    kernel.softlockup_all_cpu_backtrace = 0
    kernel.softlockup_panic = 0
    kernel.watchdog = 1
    kernel.watchdog_cpumask = 0-127
    kernel.watchdog_thresh = 10
    

    So looks like UnixBench test just pushed the softlock threshold above the 2x10 = 20 seconds, holding onto the CPU for 22 seconds
    Code (Text):
    Message from [email protected] at Feb 26 12:39:47 ...
     kernel:watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [sshd:72234]
    
    Message from [email protected] at Feb 26 12:39:51 ...
     kernel:watchdog: BUG: soft lockup - CPU#21 stuck for 22s! [context1:72109]
    

    Seeing as it's stuck with sshd it could be related to running UnixBench within a screen session and the 48x CPU section of UnixBench test ?

    So maybe adjusting kernel.watchdog_thresh from 10 to 20 would be enough ?
    Code (Text):
    echo "kernel.watchdog_thresh = 20" >> /etc/sysctl.d/101-sysctl.conf
    sysctl --system
    

    Code (Text):
    sysctl -a 2>&1 | egrep 'watchdog|softlockup'                 
    kernel.nmi_watchdog = 1
    kernel.soft_watchdog = 1
    kernel.softlockup_all_cpu_backtrace = 0
    kernel.softlockup_panic = 0
    kernel.watchdog = 1
    kernel.watchdog_cpumask = 0-127
    kernel.watchdog_thresh = 20
    


    Looks like that might have done the job and allowed UnixBench to complete. Below are results from re-run of centminmodbench.sh
    • The beast of the AMD EPYC 7401P processor was definitely let down by the slow 120GB SSD boot disk which showed no improvement
    • Memory bandwidth from ramdisk tests were also better now on the lower side of numbers
    • Network bandwidth was improved for wget single threaded tests were at ~104-307MB/s compared to ~50MB/s previously for US East Coast. The Axel multi-threaded bandwidth tests were a bit better for US East Coast ranging from 128-295MB/s compared to previously at 113-168MB/s
    • UnixBench was much better for single index = 1444.8 and 48 cpus index at 17,557.2 compared to the previously very low for single index = 386.5 and 48 cpus index at 7478.
    • OpenSSL numbers as expected were also slightly improved with AMD EPYC 7401P hardware AES-NI assisted aes256 results being ~4.6x times faster vs ~4.57x times faster on CentOS 7 3.10 Kernel and ecdsa sign/s being ~4.67x times faster vs ~4.4x times faster on CentOS 3.10 Kernel than above Intel Xeon E3-1578Lv5 Skylake.
    Code (Text):
    -------------------------------------------
    centminmodbench.sh 0.9.5
    http://bench.centminmod.com
    written by: George Liu (eva2000)
    http://centminmod.com
    -------------------------------------------
    
    -------------------------------------------
    System Information
    -------------------------------------------
    
    4.15.5-1.el7.elrepo.x86_64
    
    CentOS Linux release 7.4.1708 (Core)
    
    Centmin Mod
    ----------------------------------------------
    Architecture:          x86_64
    CPU op-mode(s):        32-bit, 64-bit
    Byte Order:            Little Endian
    CPU(s):                48
    On-line CPU(s) list:   0-47
    Thread(s) per core:    2
    Core(s) per socket:    24
    Socket(s):             1
    NUMA node(s):          4
    Vendor ID:             AuthenticAMD
    CPU family:            23
    Model:                 1
    Model name:            AMD EPYC 7401P 24-Core Processor
    Stepping:              2
    CPU MHz:               2419.247
    BogoMIPS:              3992.40
    Virtualization:        AMD-V
    L1d cache:             32K
    L1i cache:             64K
    L2 cache:              512K
    L3 cache:              8192K
    NUMA node0 CPU(s):     0,4,8,12,16,20,24,28,32,36,40,44
    NUMA node1 CPU(s):     1,5,9,13,17,21,25,29,33,37,41,45
    NUMA node2 CPU(s):     2,6,10,14,18,22,26,30,34,38,42,46
    NUMA node3 CPU(s):     3,7,11,15,19,23,27,31,35,39,43,47
    Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid amd_dcm aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb hw_pstate sme vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero irperf xsaveerptr ibpb arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov succor smca
    
    CPU Flags
     fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid amd_dcm aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb hw_pstate sme vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero irperf xsaveerptr ibpb arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov succor smca
    
    ----------------------------------------------
    CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE
    0   0    0      0    0:0:0:0       yes
    1   1    0      1    1:1:1:1       yes
    2   2    0      2    2:2:2:2       yes
    3   3    0      3    3:3:3:3       yes
    4   0    0      4    4:4:4:4       yes
    5   1    0      5    5:5:5:5       yes
    6   2    0      6    6:6:6:6       yes
    7   3    0      7    7:7:7:7       yes
    8   0    0      8    8:8:8:0       yes
    9   1    0      9    9:9:9:1       yes
    10  2    0      10   10:10:10:2    yes
    11  3    0      11   11:11:11:3    yes
    12  0    0      12   12:12:12:4    yes
    13  1    0      13   13:13:13:5    yes
    14  2    0      14   14:14:14:6    yes
    15  3    0      15   15:15:15:7    yes
    16  0    0      16   16:16:16:0    yes
    17  1    0      17   17:17:17:1    yes
    18  2    0      18   18:18:18:2    yes
    19  3    0      19   19:19:19:3    yes
    20  0    0      20   20:20:20:4    yes
    21  1    0      21   21:21:21:5    yes
    22  2    0      22   22:22:22:6    yes
    23  3    0      23   23:23:23:7    yes
    24  0    0      0    0:0:0:0       yes
    25  1    0      1    1:1:1:1       yes
    26  2    0      2    2:2:2:2       yes
    27  3    0      3    3:3:3:3       yes
    28  0    0      4    4:4:4:4       yes
    29  1    0      5    5:5:5:5       yes
    30  2    0      6    6:6:6:6       yes
    31  3    0      7    7:7:7:7       yes
    32  0    0      8    8:8:8:0       yes
    33  1    0      9    9:9:9:1       yes
    34  2    0      10   10:10:10:2    yes
    35  3    0      11   11:11:11:3    yes
    36  0    0      12   12:12:12:4    yes
    37  1    0      13   13:13:13:5    yes
    38  2    0      14   14:14:14:6    yes
    39  3    0      15   15:15:15:7    yes
    40  0    0      16   16:16:16:0    yes
    41  1    0      17   17:17:17:1    yes
    42  2    0      18   18:18:18:2    yes
    43  3    0      19   19:19:19:3    yes
    44  0    0      20   20:20:20:4    yes
    45  1    0      21   21:21:21:5    yes
    46  2    0      22   22:22:22:6    yes
    47  3    0      23   23:23:23:7    yes
    
    ----------------------------------------------
                  total        used        free      shared  buff/cache   available
    Mem:          64146        3591       59847          14         707       59881
    Low:          64146        4298       59847
    High:             0           0           0
    Swap:          1949           0        1949
    
    ----------------------------------------------
    Filesystem      Size  Used Avail Use% Mounted on
    devtmpfs         32G     0   32G   0% /dev
    tmpfs            32G     0   32G   0% /dev/shm
    tmpfs            32G  852K   32G   1% /run
    tmpfs            32G     0   32G   0% /sys/fs/cgroup
    /dev/sda3       108G   17G   86G  17% /
    /dev/sda1       511M  148K  511M   1% /boot/efi
    tmpfs           6.3G     0  6.3G   0% /run/user/0
    
    

    Code (Text):
    -------------------------------------------
    disk ioping tests
    -------------------------------------------
    
    
    Running IOPing I/O benchmark...
    
    IOPing I/O: ./ioping -c 10 .
    4.0 KiB from . (ext4 /dev/sda3): request=1 time=3.1 ms
    4.0 KiB from . (ext4 /dev/sda3): request=2 time=232 us
    4.0 KiB from . (ext4 /dev/sda3): request=3 time=265 us
    4.0 KiB from . (ext4 /dev/sda3): request=4 time=229 us
    4.0 KiB from . (ext4 /dev/sda3): request=5 time=269 us
    4.0 KiB from . (ext4 /dev/sda3): request=6 time=267 us
    4.0 KiB from . (ext4 /dev/sda3): request=7 time=272 us
    4.0 KiB from . (ext4 /dev/sda3): request=8 time=266 us
    4.0 KiB from . (ext4 /dev/sda3): request=9 time=267 us
    4.0 KiB from . (ext4 /dev/sda3): request=10 time=268 us
    
    --- . (ext4 /dev/sda3) ioping statistics ---
    10 requests completed in 9.0 s, 1.8 k iops, 7.2 MiB/s
    min/avg/max/mdev = 229 us / 542 us / 3.1 ms / 847 us
    
    IOPing seek rate: ./ioping -RD .
    
    --- . (ext4 /dev/sda3) ioping statistics ---
    18.4 k requests completed in 3.0 s, 6.3 k iops, 24.7 MiB/s
    min/avg/max/mdev = 137 us / 157 us / 1.7 ms / 23 us
    
    IOPing sequential: ./ioping -RL .
    
    --- . (ext4 /dev/sda3) ioping statistics ---
    1.6 k requests completed in 3.0 s, 526 iops, 131.6 MiB/s
    min/avg/max/mdev = 1.2 ms / 1.9 ms / 5.2 ms / 396 us
    
    IOPing cached: ./ioping -RC .
    
    --- . (ext4 /dev/sda3) ioping statistics ---
    3.4 M requests completed in 3.0 s, 1.2 M iops, 4.6 GiB/s
    min/avg/max/mdev = 0 us / 0 us / 53 us / 0 us
    
    

    Code (Text):
    -------------------------------------------
    disk DD tests
    -------------------------------------------
    
    dd if=/dev/zero of=sb-io-test bs=1M count=1k conv=fdatasync
    1024+0 records in
    1024+0 records out
    1073741824 bytes (1.1 GB) copied, 5.98212 s, 179 MB/s
    
    dd if=/dev/zero of=sb-io-test bs=64k count=16k conv=fdatasync
    16384+0 records in
    16384+0 records out
    1073741824 bytes (1.1 GB) copied, 5.9581 s, 180 MB/s
    
    dd if=/dev/zero of=sb-io-test bs=1M count=1k oflag=dsync
    1024+0 records in
    1024+0 records out
    1073741824 bytes (1.1 GB) copied, 5.6338 s, 191 MB/s
    
    dd if=/dev/zero of=sb-io-test bs=64k count=16k oflag=dsync
    16384+0 records in
    16384+0 records out
    1073741824 bytes (1.1 GB) copied, 11.0268 s, 97.4 MB/s
    

    Code (Text):
    -------------------------------------------
    disk FIO tests
    -------------------------------------------
    
    
    Running FIO benchmark...
    
    
    FIO random reads:
    randomreads: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
    fio-2.0.9
    Starting 1 process
    randomreads: Laying out IO file(s) (1 file(s) / 1024MB)
    
    randomreads: (groupid=0, jobs=1): err= 0: pid=3702: Mon Feb 26 13:35:37 2018
      read : io=1024.3MB, bw=172306KB/s, iops=43076 , runt=  6087msec
      cpu          : usr=1.96%, sys=10.53%, ctx=23835, majf=0, minf=71
      IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
         submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
         complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
         issued    : total=r=262207/w=0/d=0, short=r=0/w=0/d=0
    
    Run status group 0 (all jobs):
       READ: io=1024.3MB, aggrb=172306KB/s, minb=172306KB/s, maxb=172306KB/s, mint=6087msec, maxt=6087msec
    
    Disk stats (read/write):
      sda: ios=257601/6, merge=52/5, ticks=374401/7, in_queue=374419, util=98.39%
    
    FIO random writes:
    randomwrites: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
    fio-2.0.9
    Starting 1 process
    
    randomwrites: (groupid=0, jobs=1): err= 0: pid=3707: Mon Feb 26 13:35:44 2018
      write: io=1024.3MB, bw=147059KB/s, iops=36764 , runt=  7132msec
      cpu          : usr=2.03%, sys=10.03%, ctx=23834, majf=0, minf=6
      IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
         submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
         complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
         issued    : total=r=0/w=262207/d=0, short=r=0/w=0/d=0
    
    Run status group 0 (all jobs):
      WRITE: io=1024.3MB, aggrb=147059KB/s, minb=147059KB/s, maxb=147059KB/s, mint=7132msec, maxt=7132msec
    
    Disk stats (read/write):
      sda: ios=0/253731, merge=0/75, ticks=0/432621, in_queue=432660, util=98.63%
    
    

    Code (Text):
    -------------------------------------------
    ramdisk DD tests - memory bandwidth
    -------------------------------------------
    
    dd if=/dev/zero of=sb-io-test bs=128k count=1k conv=fdatasync
    1024+0 records in
    1024+0 records out
    134217728 bytes (134 MB) copied, 0.0389545 s, 3.4 GB/s
    
    dd if=/dev/zero of=sb-io-test bs=8k count=16k conv=fdatasync
    16384+0 records in
    16384+0 records out
    134217728 bytes (134 MB) copied, 0.0472288 s, 2.8 GB/s
    
    dd if=/dev/zero of=sb-io-test bs=128k count=1k oflag=dsync
    1024+0 records in
    1024+0 records out
    134217728 bytes (134 MB) copied, 0.0365683 s, 3.7 GB/s
    
    dd if=/dev/zero of=sb-io-test bs=8k count=16k oflag=dsync
    16384+0 records in
    16384+0 records out
    134217728 bytes (134 MB) copied, 0.0436984 s, 3.1 GB/s
    

    Code (Text):
    -------------------------------------------
    ramdisk ioping tests - memory bandwidth
    -------------------------------------------
    
    Running IOPing I/O ramdisk benchmark...
    
    IOPing I/O: ./ioping -c 10 .
    4.0 KiB from . (tmpfs tmpfs): request=1 time=2 us
    4.0 KiB from . (tmpfs tmpfs): request=2 time=3 us
    4.0 KiB from . (tmpfs tmpfs): request=3 time=2 us
    4.0 KiB from . (tmpfs tmpfs): request=4 time=2 us
    4.0 KiB from . (tmpfs tmpfs): request=5 time=2 us
    4.0 KiB from . (tmpfs tmpfs): request=6 time=3 us
    4.0 KiB from . (tmpfs tmpfs): request=7 time=2 us
    4.0 KiB from . (tmpfs tmpfs): request=8 time=2 us
    4.0 KiB from . (tmpfs tmpfs): request=9 time=2 us
    4.0 KiB from . (tmpfs tmpfs): request=10 time=2 us
    
    --- . (tmpfs tmpfs) ioping statistics ---
    10 requests completed in 9.0 s, 454.5 k iops, 1.7 GiB/s
    min/avg/max/mdev = 2 us / 2 us / 3 us / 0 us
    
    IOPing seek rate: ./ioping -R .
    
    --- . (tmpfs tmpfs) ioping statistics ---
    3.6 M requests completed in 3.0 s, 1.5 M iops, 5.8 GiB/s
    min/avg/max/mdev = 0 us / 0 us / 35 us / 0 us
    
    IOPing sequential: ./ioping -RL .
    
    --- . (tmpfs tmpfs) ioping statistics ---
    128.5 k requests completed in 3.0 s, 43.2 k iops, 10.5 GiB/s
    min/avg/max/mdev = 21 us / 23 us / 61 us / 1 us
    
    IOPing cached: ./ioping -RC .
    
    --- . (tmpfs tmpfs) ioping statistics ---
    4.2 M requests completed in 3.0 s, 1.5 M iops, 5.8 GiB/s
    min/avg/max/mdev = 0 us / 0 us / 20 us / 0 us
    
    

    Code (Text):
    -------------------------------------------
    Running bandwidth benchmark...
    -------------------------------------------
    
    ----------------------------------------------
    Download from Cachefly (http://cachefly.cachefly.net/100mb.test)
    Download Cachefly: 152MB/s
    
    -------------------------------------------
    USA bandwidth tests...
    -------------------------------------------
    
    ----------------------------------------------
    Download from Linode, Atlanta, GA, USA (http://speedtest.atlanta.linode.com/100MB-atlanta.bin)
    Download Linode, Atlanta, GA, USA: 104MB/s
    ----------------------------------------------
    Download from Linode, Dallas, TX, USA (http://speedtest.dallas.linode.com/100MB-dallas.bin)
    Download Linode, Dallas, TX, USA: 10.2MB/s
    ----------------------------------------------
    Download from Leaseweb, Manassas, VA, USA (http://mirror.us.leaseweb.net/speedtest/100mb.bin)
    Download Leaseweb, Manassas, VA, USA: 53.2MB/s
    ----------------------------------------------
    Download from Softlayer, Seattle, WA, USA (http://speedtest.sea01.softlayer.com/downloads/test100.zip)
    Download Softlayer, Seattle, WA, USA: 38.3MB/s
    ----------------------------------------------
    Download from Softlayer, San Jose, CA, USA (http://speedtest.sjc01.softlayer.com/downloads/test100.zip)
    Download Softlayer, San Jose, CA, USA: 7.68MB/s
    ----------------------------------------------
    Download from Softlayer, Washington, DC, USA (http://speedtest.wdc01.softlayer.com/downloads/test100.zip)
    Download Softlayer, Washington, DC, USA: 18.8MB/s
    ----------------------------------------------
    Download from VersaWeb, Las Vegas, Nevada (http://104.143.15.254/100MB.test)
    Download VersaWeb, Las Vegas, Nevada: 5.42MB/s
    ----------------------------------------------
    Download from OVH, BHS, Canada (http://bhs.proof.ovh.net/files/100Mio.dat)
    Download OVH, BHS, Canada: 115MB/s
    ----------------------------------------------
    Download from Vultr, Los Angeles, California (http://lax-ca-us-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, Los Angeles, California: 25.9MB/s
    ----------------------------------------------
    Download from Vultr, Seattle, Washington (http://wa-us-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, Seattle, Washington: 31.7MB/s
    ----------------------------------------------
    Download from Vultr, Dallas, Texas (http://tx-us-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, Dallas, Texas: 64.7MB/s
    ----------------------------------------------
    Download from Vultr, Chicago, Illinois (http://il-us-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, Chicago, Illinois: 97.9MB/s
    ----------------------------------------------
    Download from Vultr, Atlanta, Georgia (http://ga-us-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, Atlanta, Georgia: 119MB/s
    ----------------------------------------------
    Download from Vultr, Miami, Florida (http://fl-us-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, Miami, Florida: 68.7MB/s
    ----------------------------------------------
    Download from Vultr, New York / New Jersey (http://nj-us-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, New York / New Jersey: 307MB/s
    
    -------------------------------------------
    Asia bandwidth tests...
    -------------------------------------------
    
    ----------------------------------------------
    Download from Linode, Tokyo, JP (http://speedtest.tokyo.linode.com/100MB-tokyo.bin)
    Download Linode, Tokyo, JP: 13.2MB/s
    ----------------------------------------------
    Download from Softlayer, Singapore (http://speedtest.sng01.softlayer.com/downloads/test100.zip)
    Download Softlayer, Singapore: 8.56MB/s
    ----------------------------------------------
    Download from Vultr, Tokyo, Japan (http://hnd-jp-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, Tokyo, Japan: 13.3MB/s
    
    -------------------------------------------
    Europe bandwidth tests...
    -------------------------------------------
    
    ----------------------------------------------
    Download from Linode, London, UK (http://speedtest.london.linode.com/100MB-london.bin)
    Download Linode, London, UK: 16.7MB/s
    ----------------------------------------------
    Download from OVH, Paris, France (http://proof.ovh.net/files/100Mio.dat)
    Download OVH, Paris, France: 9.16MB/s
    ----------------------------------------------
    Download from SmartDC, Rotterdam, Netherlands (http://mirror.i3d.net/100mb.bin)
    Download SmartDC, Rotterdam, Netherlands: 14.8MB/s
    ----------------------------------------------
    Download from Vultr, Amsterdam, Netherlands (http://ams-nl-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, Amsterdam, Netherlands: 29.3MB/s
    ----------------------------------------------
    Download from Vultr, London, UK (http://lon-gb-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, London, UK: 32.3MB/s
    ----------------------------------------------
    Download from Vultr, Paris, France (http://par-fr-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, Paris, France: 31.6MB/s
    
    -------------------------------------------
    Australia bandwidth tests...
    -------------------------------------------
    
    ----------------------------------------------
    Download from Vultr, Sydney, Australia (http://syd-au-ping.vultr.com/vultr.com.100MB.bin)
    Download Vultr, Sydney, Australia: 9.75MB/s
    

    Code (Text):
    -------------------------------------------
    Running Axel multi-threaded bandwidth benchmark...
    -------------------------------------------
    
    ----------------------------------------------
    Axel Download from Cachefly (http://cachefly.cachefly.net/100mb.test)
    Axel Download Cachefly: 286.82MB/s
    
    -------------------------------------------
    USA bandwidth tests...
    -------------------------------------------
    
    ----------------------------------------------
    Axel Download from Linode, Atlanta, GA, USA (http://speedtest.atlanta.linode.com/100MB-atlanta.bin)
    Axel Download Linode, Atlanta, GA, USA: 128.62MB/s
    ----------------------------------------------
    Axel Download from Linode, Dallas, TX, USA (http://speedtest.dallas.linode.com/100MB-dallas.bin)
    Axel Download Linode, Dallas, TX, USA: 75.97MB/s
    ----------------------------------------------
    Axel Download from Leaseweb, Manassas, VA, USA (http://mirror.us.leaseweb.net/speedtest/100mb.bin)
    Axel Download Leaseweb, Manassas, VA, USA: 56.68MB/s
    ----------------------------------------------
    Axel Download from Softlayer, Seattle, WA, USA (http://speedtest.sea01.softlayer.com/downloads/test100.zip)
    Axel Download Softlayer, Seattle, WA, USA: 33.03MB/s
    ----------------------------------------------
    Axel Download from Softlayer, San Jose, CA, USA (http://speedtest.sjc01.softlayer.com/downloads/test100.zip)
    Axel Download Softlayer, San Jose, CA, USA: 25.45MB/s
    ----------------------------------------------
    Axel Download from Softlayer, Washington, DC, USA (http://speedtest.wdc01.softlayer.com/downloads/test100.zip)
    Axel Download Softlayer, Washington, DC, USA: 64.84MB/s
    ----------------------------------------------
    Axel Download from VersaWeb, Las Vegas, Nevada (http://104.143.15.254/100MB.test)
    Axel Download VersaWeb, Las Vegas, Nevada: 15.58MB/s
    ----------------------------------------------
    Axel Download from OVH, BHS, Canada (http://bhs.proof.ovh.net/files/100Mio.dat)
    Axel Download OVH, BHS, Canada: 103.93MB/s
    ----------------------------------------------
    Axel Download from Vultr, Los Angeles, California (http://lax-ca-us-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, Los Angeles, California: 37.17MB/s
    ----------------------------------------------
    Axel Download from Vultr, Seattle, Washington (http://wa-us-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, Seattle, Washington: 58.97MB/s
    ----------------------------------------------
    Axel Download from Vultr, Dallas, Texas (http://tx-us-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, Dallas, Texas: 81.73MB/s
    ----------------------------------------------
    Axel Download from Vultr, Chicago, Illinois (http://il-us-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, Chicago, Illinois: 135.76MB/s
    ----------------------------------------------
    Axel Download from Vultr, Atlanta, Georgia (http://ga-us-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, Atlanta, Georgia: 112.35MB/s
    ----------------------------------------------
    Axel Download from Vultr, Miami, Florida (http://fl-us-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, Miami, Florida: 124.39MB/s
    ----------------------------------------------
    Axel Download from Vultr, New York / New Jersey (http://nj-us-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, New York / New Jersey: 295.89MB/s
    
    -------------------------------------------
    Asia bandwidth tests...
    -------------------------------------------
    
    ----------------------------------------------
    Axel Download from Linode, Tokyo, JP (http://speedtest.tokyo.linode.com/100MB-tokyo.bin)
    Axel Download Linode, Tokyo, JP: 12.71MB/s
    ----------------------------------------------
    Axel Download from Softlayer, Singapore (http://speedtest.sng01.softlayer.com/downloads/test100.zip)
    Axel Download Softlayer, Singapore: 8.89MB/s
    ----------------------------------------------
    Axel Download from Vultr, Tokyo, Japan (http://hnd-jp-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, Tokyo, Japan: 19.12MB/s
    
    -------------------------------------------
    Europe bandwidth tests...
    -------------------------------------------
    
    ----------------------------------------------
    Axel Download from Linode, London, UK (http://speedtest.london.linode.com/100MB-london.bin)
    Axel Download Linode, London, UK: 48.96MB/s
    ----------------------------------------------
    Axel Download from OVH, Paris, France (http://proof.ovh.net/files/100Mio.dat)
    Axel Download OVH, Paris, France: 9.32MB/s
    ----------------------------------------------
    Axel Download from SmartDC, Rotterdam, Netherlands (http://mirror.i3d.net/100mb.bin)
    Axel Download SmartDC, Rotterdam, Netherlands: 13.45MB/s
    ----------------------------------------------
    Axel Download from Vultr, Amsterdam, Netherlands (http://ams-nl-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, Amsterdam, Netherlands: 44.50MB/s
    ----------------------------------------------
    Axel Download from Vultr, London, UK (http://lon-gb-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, London, UK: 60.40MB/s
    ----------------------------------------------
    Axel Download from Vultr, Paris, France (http://par-fr-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, Paris, France: 20.88MB/s
    
    -------------------------------------------
    Australia bandwidth tests...
    -------------------------------------------
    
    ----------------------------------------------
    Axel Download from Vultr, Sydney, Australia (http://syd-au-ping.vultr.com/vultr.com.100MB.bin)
    Axel Download Vultr, Sydney, Australia: 17.90MB/s
    

    Code (Text):
    -------------------------------------------
    Running ping tests...
    -------------------------------------------
    
    ----------------------------------------------
    Pings (cachefly.cachefly.net):
    PING vip1.g5.cachefly.net (205.234.175.175) 56(84) bytes of data.
    64 bytes from vip1.G-anycast1.cachefly.net (205.234.175.175): icmp_seq=1 ttl=58 time=0.970 ms
    64 bytes from vip1.G-anycast1.cachefly.net (205.234.175.175): icmp_seq=2 ttl=58 time=0.966 ms
    64 bytes from vip1.G-anycast1.cachefly.net (205.234.175.175): icmp_seq=3 ttl=58 time=0.981 ms
    
    --- vip1.g5.cachefly.net ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 0.966/0.972/0.981/0.026 ms
    
    ----------------------------------------------
    Pings (syd-au-ping.vultr.com):
    PING syd-au-ping.vultr.com (108.61.212.117) 56(84) bytes of data.
    64 bytes from 108.61.212.117.vultr.com (108.61.212.117): icmp_seq=1 ttl=51 time=222 ms
    64 bytes from 108.61.212.117.vultr.com (108.61.212.117): icmp_seq=2 ttl=51 time=222 ms
    64 bytes from 108.61.212.117.vultr.com (108.61.212.117): icmp_seq=3 ttl=51 time=222 ms
    
    --- syd-au-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 222.613/222.668/222.751/0.389 ms
    
    ----------------------------------------------
    Pings (hnd-jp-ping.vultr.com):
    PING hnd-jp-ping.vultr.com (108.61.201.151) 56(84) bytes of data.
    64 bytes from 108.61.201.151.vultr.com (108.61.201.151): icmp_seq=1 ttl=50 time=175 ms
    64 bytes from 108.61.201.151.vultr.com (108.61.201.151): icmp_seq=2 ttl=50 time=175 ms
    64 bytes from 108.61.201.151.vultr.com (108.61.201.151): icmp_seq=3 ttl=50 time=174 ms
    
    --- hnd-jp-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 174.955/175.015/175.054/0.344 ms
    
    ----------------------------------------------
    Pings (lax-ca-us-ping.vultr.com):
    PING lax-ca-us-ping.vultr.com (108.61.219.200) 56(84) bytes of data.
    64 bytes from 108.61.219.200.vultr.com (108.61.219.200): icmp_seq=1 ttl=53 time=87.4 ms
    64 bytes from 108.61.219.200.vultr.com (108.61.219.200): icmp_seq=2 ttl=53 time=87.5 ms
    64 bytes from 108.61.219.200.vultr.com (108.61.219.200): icmp_seq=3 ttl=53 time=87.4 ms
    
    --- lax-ca-us-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 87.417/87.474/87.526/0.245 ms
    
    ----------------------------------------------
    Pings (wa-us-ping.vultr.com):
    PING wa-us-ping.vultr.com (108.61.194.105) 56(84) bytes of data.
    64 bytes from 108.61.194.105.vultr.com (108.61.194.105): icmp_seq=1 ttl=53 time=71.5 ms
    64 bytes from 108.61.194.105.vultr.com (108.61.194.105): icmp_seq=2 ttl=53 time=71.6 ms
    64 bytes from 108.61.194.105.vultr.com (108.61.194.105): icmp_seq=3 ttl=53 time=71.4 ms
    
    --- wa-us-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2001ms
    rtt min/avg/max/mdev = 71.498/71.537/71.612/0.053 ms
    
    ----------------------------------------------
    Pings (tx-us-ping.vultr.com):
    PING tx-us-ping.vultr.com (108.61.224.175) 56(84) bytes of data.
    64 bytes from 108.61.224.175.vultr.com (108.61.224.175): icmp_seq=1 ttl=52 time=39.0 ms
    64 bytes from 108.61.224.175.vultr.com (108.61.224.175): icmp_seq=2 ttl=52 time=40.8 ms
    64 bytes from 108.61.224.175.vultr.com (108.61.224.175): icmp_seq=3 ttl=52 time=39.5 ms
    
    --- tx-us-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2003ms
    rtt min/avg/max/mdev = 39.088/39.845/40.893/0.798 ms
    
    ----------------------------------------------
    Pings (il-us-ping.vultr.com):
    PING il-us-ping.vultr.com (107.191.51.12) 56(84) bytes of data.
    64 bytes from 107.191.51.12.vultr.com (107.191.51.12): icmp_seq=1 ttl=54 time=25.0 ms
    64 bytes from 107.191.51.12.vultr.com (107.191.51.12): icmp_seq=2 ttl=54 time=24.9 ms
    64 bytes from 107.191.51.12.vultr.com (107.191.51.12): icmp_seq=3 ttl=54 time=24.9 ms
    
    --- il-us-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2003ms
    rtt min/avg/max/mdev = 24.957/24.979/25.007/0.130 ms
    
    ----------------------------------------------
    Pings (ga-us-ping.vultr.com):
    PING ga-us-ping.vultr.com (108.61.193.166) 56(84) bytes of data.
    64 bytes from 108.61.193.166.vultr.com (108.61.193.166): icmp_seq=1 ttl=53 time=18.6 ms
    64 bytes from 108.61.193.166.vultr.com (108.61.193.166): icmp_seq=2 ttl=53 time=18.5 ms
    64 bytes from 108.61.193.166.vultr.com (108.61.193.166): icmp_seq=3 ttl=53 time=19.0 ms
    
    --- ga-us-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2003ms
    rtt min/avg/max/mdev = 18.542/18.747/19.088/0.267 ms
    
    ----------------------------------------------
    Pings (fl-us-ping.vultr.com):
    PING fl-us-ping.vultr.com (104.156.244.232) 56(84) bytes of data.
    64 bytes from 104.156.244.232 (104.156.244.232): icmp_seq=1 ttl=53 time=34.6 ms
    64 bytes from 104.156.244.232 (104.156.244.232): icmp_seq=2 ttl=53 time=34.1 ms
    64 bytes from 104.156.244.232 (104.156.244.232): icmp_seq=3 ttl=53 time=38.4 ms
    
    --- fl-us-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 34.161/35.743/38.433/1.917 ms
    
    ----------------------------------------------
    Pings (nj-us-ping.vultr.com):
    PING nj-us-ping.vultr.com (108.61.149.182) 56(84) bytes of data.
    64 bytes from 108.61.149.182.vultr.com (108.61.149.182): icmp_seq=1 ttl=56 time=2.29 ms
    64 bytes from 108.61.149.182.vultr.com (108.61.149.182): icmp_seq=2 ttl=57 time=2.10 ms
    64 bytes from 108.61.149.182.vultr.com (108.61.149.182): icmp_seq=3 ttl=56 time=2.29 ms
    
    --- nj-us-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 2.101/2.227/2.290/0.089 ms
    
    ----------------------------------------------
    Pings (fra-de-ping.vultr.com):
    PING fra-de-ping.vultr.com (108.61.210.117) 56(84) bytes of data.
    64 bytes from 108.61.210.117.vultr.com (108.61.210.117): icmp_seq=1 ttl=51 time=82.2 ms
    64 bytes from 108.61.210.117.vultr.com (108.61.210.117): icmp_seq=2 ttl=51 time=82.0 ms
    64 bytes from 108.61.210.117.vultr.com (108.61.210.117): icmp_seq=3 ttl=51 time=82.3 ms
    
    --- fra-de-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 82.076/82.200/82.320/0.099 ms
    
    ----------------------------------------------
    Pings (ams-nl-ping.vultr.com):
    PING ams-nl-ping.vultr.com (108.61.198.102) 56(84) bytes of data.
    64 bytes from 108.61.198.102.vultr.com (108.61.198.102): icmp_seq=1 ttl=53 time=79.5 ms
    64 bytes from 108.61.198.102.vultr.com (108.61.198.102): icmp_seq=2 ttl=53 time=79.4 ms
    64 bytes from 108.61.198.102.vultr.com (108.61.198.102): icmp_seq=3 ttl=53 time=79.4 ms
    
    --- ams-nl-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2003ms
    rtt min/avg/max/mdev = 79.470/79.501/79.549/0.327 ms
    
    ----------------------------------------------
    Pings (lon-gb-ping.vultr.com):
    PING lon-gb-ping.vultr.com (108.61.196.101) 56(84) bytes of data.
    64 bytes from 108.61.196.101.vultr.com (108.61.196.101): icmp_seq=1 ttl=53 time=71.3 ms
    64 bytes from 108.61.196.101.vultr.com (108.61.196.101): icmp_seq=2 ttl=53 time=71.1 ms
    64 bytes from 108.61.196.101.vultr.com (108.61.196.101): icmp_seq=3 ttl=53 time=71.2 ms
    
    --- lon-gb-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 71.154/71.234/71.332/0.316 ms
    
    ----------------------------------------------
    Pings (par-fr-ping.vultr.com):
    PING par-fr-ping.vultr.com (108.61.209.127) 56(84) bytes of data.
    64 bytes from 108.61.209.127.vultr.com (108.61.209.127): icmp_seq=1 ttl=54 time=85.5 ms
    64 bytes from 108.61.209.127.vultr.com (108.61.209.127): icmp_seq=2 ttl=54 time=86.3 ms
    64 bytes from 108.61.209.127.vultr.com (108.61.209.127): icmp_seq=3 ttl=54 time=86.1 ms
    
    --- par-fr-ping.vultr.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 85.500/86.017/86.398/0.508 ms
    
    ----------------------------------------------
    Pings (VersaWeb Las Vegas):
    PING 104.143.15.254 (104.143.15.254) 56(84) bytes of data.
    64 bytes from 104.143.15.254: icmp_seq=1 ttl=48 time=76.4 ms
    64 bytes from 104.143.15.254: icmp_seq=2 ttl=48 time=76.5 ms
    64 bytes from 104.143.15.254: icmp_seq=3 ttl=48 time=76.2 ms
    
    --- 104.143.15.254 ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 76.285/76.410/76.545/0.336 ms
    
    ----------------------------------------------
    Pings (VersaWeb Seattle):
    PING 76.164.234.1 (76.164.234.1) 56(84) bytes of data.
    64 bytes from 76.164.234.1: icmp_seq=1 ttl=54 time=66.4 ms
    64 bytes from 76.164.234.1: icmp_seq=2 ttl=54 time=66.4 ms
    64 bytes from 76.164.234.1: icmp_seq=3 ttl=54 time=66.4 ms
    
    --- 76.164.234.1 ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 66.437/66.447/66.456/0.007 ms
    
    ----------------------------------------------
    Pings (OVH Canada):
    PING bhs.proof.ovh.net (192.99.19.165) 56(84) bytes of data.
    64 bytes from bhs.proof.ovh.net (192.99.19.165): icmp_seq=1 ttl=56 time=8.83 ms
    64 bytes from bhs.proof.ovh.net (192.99.19.165): icmp_seq=2 ttl=56 time=9.06 ms
    64 bytes from bhs.proof.ovh.net (192.99.19.165): icmp_seq=3 ttl=56 time=8.86 ms
    
    --- bhs.proof.ovh.net ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 8.830/8.920/9.063/0.102 ms
    
    ----------------------------------------------
    Pings (10gbps.io Frankfurt):
    PING 195.181.174.20 (195.181.174.20) 56(84) bytes of data.
    64 bytes from 195.181.174.20: icmp_seq=1 ttl=51 time=93.1 ms
    64 bytes from 195.181.174.20: icmp_seq=2 ttl=51 time=93.1 ms
    64 bytes from 195.181.174.20: icmp_seq=3 ttl=51 time=93.1 ms
    
    --- 195.181.174.20 ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 93.115/93.165/93.197/0.251 ms
    
    ----------------------------------------------
    Pings (10gbps.io London):
    PING 185.59.221.20 (185.59.221.20) 56(84) bytes of data.
    64 bytes from 185.59.221.20: icmp_seq=1 ttl=51 time=94.5 ms
    64 bytes from 185.59.221.20: icmp_seq=2 ttl=51 time=94.5 ms
    64 bytes from 185.59.221.20: icmp_seq=3 ttl=51 time=94.5 ms
    
    --- 185.59.221.20 ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2003ms
    rtt min/avg/max/mdev = 94.514/94.537/94.549/0.251 ms
    
    ----------------------------------------------
    Pings (10gbps.io Paris):
    PING 185.93.2.193 (185.93.2.193) 56(84) bytes of data.
    64 bytes from 185.93.2.193: icmp_seq=1 ttl=51 time=101 ms
    64 bytes from 185.93.2.193: icmp_seq=2 ttl=51 time=101 ms
    64 bytes from 185.93.2.193: icmp_seq=3 ttl=51 time=101 ms
    
    --- 185.93.2.193 ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2000ms
    rtt min/avg/max/mdev = 101.232/101.260/101.313/0.369 ms
    
    ----------------------------------------------
    Pings (10gbps.io Singapore):
    PING 203.174.85.38 (203.174.85.38) 56(84) bytes of data.
    64 bytes from 203.174.85.38: icmp_seq=1 ttl=51 time=251 ms
    64 bytes from 203.174.85.38: icmp_seq=2 ttl=51 time=251 ms
    64 bytes from 203.174.85.38: icmp_seq=3 ttl=51 time=251 ms
    
    --- 203.174.85.38 ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2003ms
    rtt min/avg/max/mdev = 251.615/251.724/251.938/0.598 ms
    
    ----------------------------------------------
    Pings (10gbps.io Stockholm):
    PING 185.76.9.10 (185.76.9.10) 56(84) bytes of data.
    64 bytes from 185.76.9.10: icmp_seq=1 ttl=53 time=108 ms
    64 bytes from 185.76.9.10: icmp_seq=2 ttl=53 time=108 ms
    64 bytes from 185.76.9.10: icmp_seq=3 ttl=53 time=108 ms
    
    --- 185.76.9.10 ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 108.212/108.255/108.277/0.030 ms
    
    ----------------------------------------------
    Pings (10gbps.io Moscow):
    PING 87.245.204.20 (87.245.204.20) 56(84) bytes of data.
    64 bytes from 87.245.204.20: icmp_seq=1 ttl=59 time=113 ms
    64 bytes from 87.245.204.20: icmp_seq=2 ttl=59 time=113 ms
    64 bytes from 87.245.204.20: icmp_seq=3 ttl=59 time=113 ms
    
    --- 87.245.204.20 ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 113.235/113.289/113.381/0.394 ms
    
    ----------------------------------------------
    Pings (10gbps.io Miami):
    PING 195.181.162.195 (195.181.162.195) 56(84) bytes of data.
    64 bytes from 195.181.162.195: icmp_seq=1 ttl=55 time=55.8 ms
    64 bytes from 195.181.162.195: icmp_seq=2 ttl=55 time=56.0 ms
    64 bytes from 195.181.162.195: icmp_seq=3 ttl=55 time=55.9 ms
    
    --- 195.181.162.195 ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 55.874/55.950/56.060/0.208 ms
    
    

    Code (Text):
    -------------------------------------------
    Running mtr tests...
    -------------------------------------------
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 cachefly.cachefly.net
    Start: Mon Feb 26 13:40:44 2018
    HOST: hostname          Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                       0.0%    10   11.8  16.0  11.8  21.7   3.9
      2.|-- 147.75.36.4                   0.0%    10   11.9  55.4  11.8 301.0  92.7
      3.|-- 0.ae12.bbr1.ewr1.packet.net   0.0%    10    0.2   0.4   0.2   2.7   0.7
      4.|-- ???                          100.0    10    0.0   0.0   0.0   0.0   0.0
      5.|-- nyk-b6-link.telia.net         0.0%    10    1.0   1.8   1.0   6.4   1.7
      6.|-- nyk-b5-link.telia.net         0.0%    10    2.8   1.5   1.0   2.8   0.7
      7.|-- ???                          100.0    10    0.0   0.0   0.0   0.0   0.0
      8.|-- vip1.G-anycast1.cachefly.net  0.0%    10    1.0   1.0   1.0   1.2   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 speedtest.atlanta.linode.com
    Start: Mon Feb 26 13:40:59 2018
    HOST: hostname                  Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                               0.0%    10   10.7  13.8   0.9  88.4  26.4
      2.|-- 2604:1380:2:79::                      0.0%    10    1.5   6.0   1.5  10.7   2.7
      3.|-- 2604:1380:2:20::                      0.0%    10    0.3   0.4   0.2   1.1   0.0
      4.|-- 2001:518:6900:1::5                    0.0%    10    0.7   2.5   0.3  20.7   6.4
      5.|-- 2001:518:2001:15::1                   0.0%    10    0.2   1.4   0.2  12.1   3.7
      6.|-- 2001:518:6001:6::2                    0.0%    10    1.1   2.1   1.0  12.0   3.4
      7.|-- nyk-b3-link.telia.net                 0.0%    10    1.2   1.4   1.1   2.8   0.5
      8.|-- nyk-bb4-v6.telia.net                  0.0%    10   21.5  21.2  21.1  21.5   0.0
      9.|-- ash-bb4-v6.telia.net                  0.0%    10   21.9  21.8  21.6  22.0   0.0
     10.|-- atl-b22-v6.telia.net                  0.0%    10   31.3  25.5  24.1  31.3   2.3
     11.|-- linode-ic-321031-atl-b22.c.telia.net  0.0%    10   21.8  22.1  21.7  23.2   0.3
     12.|-- 2600:3c02:4444:4::2                   0.0%    10   21.9  22.0  21.8  22.5   0.0
     13.|-- 2600:3c02::4b                         0.0%    10   21.5  21.5  21.4  21.6   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 speedtest.dallas.linode.com
    Start: Mon Feb 26 13:41:16 2018
    HOST: hostname           Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                        0.0%    10   19.4  19.8   9.5  29.4   6.6
      2.|-- 2604:1380:2:79::               0.0%    10    3.2   5.5   0.5  11.0   3.2
      3.|-- 2604:1380:2:20::               0.0%    10    0.2   3.3   0.2  31.2   9.8
      4.|-- 2001:518:6900:1::5             0.0%    10    0.5   4.5   0.4  40.7  12.7
      5.|-- 2001:518:2001:16::1            0.0%    10    0.2   0.5   0.2   1.9   0.3
      6.|-- 2001:518:6001:7::2             0.0%    10    1.1   1.1   1.1   1.1   0.0
      7.|-- ae2-63.cr0-nyc2.ip6.gtt.net    0.0%    10   70.1  70.1  70.0  70.2   0.0
      8.|-- 2001:668:0:2:ffff:0:5995:8f3d  0.0%    10   80.4  80.3  80.2  80.6   0.0
      9.|-- 2001:668:0:3:ffff:2:0:26a      0.0%    10   42.2  42.2  42.1  42.4   0.0
     10.|-- 2600:3c00:2222:13::2           0.0%    10   43.0  42.8  42.6  43.1   0.0
     11.|-- speedtest.dallas.linode.com    0.0%    10   44.9  44.9  44.8  45.0   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 mirror.us.leaseweb.net
    Start: Mon Feb 26 13:41:32 2018
    HOST: hostname           Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                        0.0%    10   23.8  46.5  11.8 119.5  36.9
      2.|-- 2604:1380:2:78::               0.0%    10    0.5   4.5   0.5  10.0   3.2
      3.|-- 2604:1380:2:2::1               0.0%    10    0.2   2.1   0.2  19.8   6.2
      4.|-- 2001:518:6900:1::1             0.0%    10    0.7   0.6   0.4   0.7   0.0
      5.|-- 2001:518:2001:14::1            0.0%    10    0.9   0.5   0.2   2.1   0.6
      6.|-- 2001:518:6001:7::2             0.0%    10    1.1   1.2   1.0   1.7   0.0
      7.|-- ae2-63.cr0-nyc2.ip6.gtt.net    0.0%    10   80.3  71.1  70.0  80.3   3.2
      8.|-- 2001:668:0:2:ffff:0:d5fe:d6aa  0.0%    10   76.5  77.1  76.3  82.9   2.0
      9.|-- 2001:668:0:3:ffff:0:45ae:326   0.0%    10   52.2  52.2  52.1  52.3   0.0
     10.|-- 2604:9a00::3d                  0.0%    10   33.7  33.8  33.7  34.0   0.0
     11.|-- 2604:9a00:2010:a0b8::5         0.0%    10   33.9  34.0  33.9  34.0   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 speedtest.sea01.softlayer.com
    Start: Mon Feb 26 13:41:49 2018
    HOST: hostname                                               Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                                                            0.0%    10   16.8  22.6   6.6  62.4  14.9
      2.|-- 2604:1380:2:79::                                                   0.0%    10   10.7   5.7   0.9  10.7   2.5
      3.|-- 2604:1380:2:20::                                                   0.0%    10    0.2   0.3   0.2   1.3   0.0
      4.|-- de-cix.nyc.us.ibm.com                                              0.0%    10    1.3   1.4   1.1   2.6   0.3
      5.|-- 6320.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10    2.2   2.2   1.7   4.5   0.7
      6.|-- a320.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   22.9  22.8  22.7  23.2   0.0
      7.|-- 5220.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   23.0  23.2  22.9  23.5   0.0
      8.|-- ee20.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   64.5  64.6  63.8  65.4   0.0
      9.|-- 1630.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   62.6  62.7  62.5  63.1   0.0
     10.|-- po3.bbr01.sr01.sea01.networklayer.com                              0.0%    10   62.3  62.5  62.3  63.6   0.3
     11.|-- po1.fcr01.sr01.sea01.networklayer.com                              0.0%    10   63.6  63.4  63.1  63.7   0.0
     12.|-- softlayer.com                                                      0.0%    10   62.8  62.8  62.8  62.8   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 speedtest.sjc01.softlayer.com
    Start: Mon Feb 26 13:42:06 2018
    HOST: hostname                                               Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                                                            0.0%    10   24.5  30.9  11.0 102.3  25.6
      2.|-- 2604:1380:2:79::                                                   0.0%    10    7.3   5.9   0.7  10.6   2.8
      3.|-- 2604:1380:2:20::                                                   0.0%    10    0.5   0.2   0.2   0.5   0.0
      4.|-- de-cix.nyc.us.ibm.com                                              0.0%    10    1.2   1.2   1.2   1.4   0.0
      5.|-- 6320.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10    1.9   2.0   1.7   2.3   0.0
      6.|-- a320.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   26.0  22.9  22.5  26.0   1.1
      7.|-- 5220.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   23.2  23.0  22.7  23.5   0.0
      8.|-- 8320.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   44.7  45.0  44.7  45.2   0.0
      9.|-- 3620.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   68.0  67.5  67.0  68.9   0.3
     10.|-- 3520.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   69.1  70.7  69.1  78.2   2.7
     11.|-- 9700.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   66.6  66.8  66.6  67.1   0.0
     12.|-- 2607:f0d0:2100:1::3                                               10.0%    10   68.6  69.7  68.6  71.4   1.1
     13.|-- softlayer.com                                                      0.0%    10   71.0  71.0  70.9  71.0   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 speedtest.wdc01.softlayer.com
    Start: Mon Feb 26 13:42:21 2018
    HOST: hostname                                               Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                                                           10.0%    10    1.0   6.2   0.7  15.6   5.0
      2.|-- 2604:1380:2:78::                                                   0.0%    10   11.1   6.3   0.5  11.3   3.4
      3.|-- 2604:1380:2:5::1                                                   0.0%    10    0.3   0.2   0.2   0.3   0.0
      4.|-- de-cix.nyc.us.ibm.com                                              0.0%    10    1.2   1.4   1.1   3.5   0.7
      5.|-- 2320.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10    2.0   3.2   1.7  10.6   2.9
      6.|-- 2830.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10    7.1   7.6   7.1   9.8   0.7
      7.|-- e730.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   11.4   7.9   7.4  11.4   1.2
      8.|-- bbr01.eq01.wdc02.networklayer.com                                  0.0%    10    7.2   7.3   7.0   7.8   0.0
      9.|-- po2.dar01.sr01.wdc01.networklayer.com                              0.0%    10    8.6   7.8   7.4   9.3   0.5
     10.|-- po1.fcr01.sr01.wdc01.networklayer.com                              0.0%    10    7.8   7.7   7.6   7.9   0.0
     11.|-- softlayer.com                                                      0.0%    10    7.8  12.3   7.8  20.2   4.7
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 speedtest.tokyo.linode.com
    Start: Mon Feb 26 13:42:37 2018
    HOST: hostname          Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                       0.0%    10   10.7   5.7   1.2  10.7   3.1
      2.|-- 2604:1380:2:78::              0.0%    10    6.2   6.8   1.5   9.8   2.3
      3.|-- 2604:1380:2:5::1              0.0%    10    0.3   1.4   0.2  11.9   3.6
      4.|-- 10ge3-9.core1.nyc6.he.net     0.0%    10    1.2   1.7   1.1   6.4   1.5
      5.|-- peering.kddi.com              0.0%    10   80.2  79.5  76.6  80.4   1.3
      6.|-- 6nytjbb001.int-gw.kddi.ne.jp  0.0%    10   80.0  80.4  79.8  83.0   0.9
      7.|-- 6lacGCS001.int-gw.kddi.ne.jp  0.0%    10   65.6  67.5  65.5  72.7   3.2
      8.|-- 2001:268:fb80:300::1          0.0%    10  238.2 239.5 238.2 240.4   0.3
      9.|-- 2001:268:fa00:282::2          0.0%    10  237.7 239.0 237.7 249.9   3.8
     10.|-- 6cm-fcu204.int-gw.kddi.ne.jp  0.0%    10  238.5 244.0 238.5 251.5   4.6
     11.|-- 2001:268:f702:6d::2           0.0%    10  242.9 243.0 242.9 243.7   0.0
     12.|-- speedtest.tokyo.linode.com    0.0%    10  238.6 238.8 238.6 240.0   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 speedtest.sng01.softlayer.com
    Start: Mon Feb 26 13:42:56 2018
    HOST: hostname                                               Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                                                            0.0%    10    4.0   6.2   3.1   9.5   2.1
      2.|-- 2604:1380:2:78::                                                   0.0%    10    4.5   4.0   0.2   8.7   2.4
      3.|-- 2604:1380:2:2::1                                                   0.0%    10    0.5   3.6   0.2  32.7  10.2
      4.|-- 2604:1380:2:1::                                                    0.0%    10    0.4   0.4   0.2   1.0   0.0
      5.|-- de-cix.nyc.us.ibm.com                                              0.0%    10    1.2   1.3   1.2   2.4   0.3
      6.|-- 6320.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10    1.9   2.0   1.8   2.4   0.0
      7.|-- a320.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   22.8  23.3  22.8  25.6   0.7
      8.|-- 5220.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   22.9  23.3  22.9  24.8   0.3
      9.|-- 8320.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   47.5  47.6  47.4  47.9   0.0
     10.|-- a120.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   48.0  48.3  47.4  53.7   1.9
     11.|-- c520.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10   75.2  75.1  72.8  75.7   0.6
     12.|-- 8b40.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10  215.1 215.5 214.6 217.5   0.8
     13.|-- 7940.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10  240.6 242.2 240.4 254.7   4.4
     14.|-- b840.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10  240.5 240.5 240.4 240.9   0.0
     15.|-- db00.0000.0000.0000.2000.2000.0d0f.7062.ip6.static.sl-reverse.com  0.0%    10  241.2 240.5 240.3 241.2   0.0
     16.|-- 3.00.0000.ip4.static.sl-reverse.com                               30.0%    10  240.4 241.2 240.4 245.8   1.9
     17.|-- 2.00.0000.ip4.static.sl-reverse.com                                0.0%    10  240.9 240.8 240.7 240.9   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 speedtest.london.linode.com
    Start: Mon Feb 26 13:43:12 2018
    HOST: hostname                 Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                              0.0%    10   30.4   8.0   0.3  30.4   8.5
      2.|-- 2604:1380:2:78::                     0.0%    10    5.8   6.8   2.9  11.2   2.6
      3.|-- 2604:1380:2:2::1                     0.0%    10    0.2   0.7   0.2   5.2   1.5
      4.|-- 2001:518:6900:1::1                   0.0%    10    0.6   5.7   0.4  52.3  16.4
      5.|-- 2001:518:2001:13::1                  0.0%    10    0.2   2.4   0.2  12.7   3.8
      6.|-- 2001:518:6001:6::2                   0.0%    10    5.8   6.6   1.0  12.0   4.0
      7.|-- nyk-b3-link.telia.net                0.0%    10    1.2   2.4   1.1  13.3   3.8
      8.|-- nyk-bb4-v6.telia.net                 0.0%    10   70.8  70.8  70.7  71.2   0.0
      9.|-- 2001:2000:3019:7b::1                 0.0%    10   70.8  71.8  70.8  77.3   2.0
     10.|-- ldn-bb2-v6.telia.net                 0.0%    10  109.4  79.3  70.6 109.4  13.5
     11.|-- ldn-b4-v6.telia.net                  0.0%    10   70.7  70.7  70.6  70.8   0.0
     12.|-- linode-ic-318542-ldn-b3.c.telia.net  0.0%    10   72.9  71.8  71.4  72.9   0.5
     13.|-- 2a01:7e00:7777:4::2                  0.0%    10   71.5  71.5  71.3  71.6   0.0
     14.|-- speedtest.london.linode.com          0.0%    10   71.1  71.0  71.0  71.1   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 mirror.i3d.net
    Start: Mon Feb 26 13:43:29 2018
    HOST: hostname              Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                           0.0%    10   21.4  21.8   6.3  28.7   6.5
      2.|-- 2604:1380:2:78::                  0.0%    10    4.3   3.9   0.4   9.8   3.3
      3.|-- 2604:1380:2:2::1                  0.0%    10    0.3   2.2   0.2  19.4   6.0
      4.|-- 2001:518:6900:1::1                0.0%    10    0.4   3.1   0.4  25.9   8.0
      5.|-- 2001:518:2001:14::1               0.0%    10    0.2   0.4   0.2   1.9   0.5
      6.|-- 2001:518:1001:7::2                0.0%    10    0.9   2.9   0.9   6.8   2.1
      7.|-- 10ge.eqx-ashburn.misi3d.com       0.0%    10    6.3   6.5   6.1   9.2   0.7
      8.|-- 10ge.lr0-iadr0.th-n.lon.i3d.net   0.0%    10   78.6  79.1  77.8  89.9   3.7
      9.|-- 30ge.cr1-lr0.smartdc.rtm.i3d.net  0.0%    10   84.5  84.5  84.4  85.6   0.3
     10.|-- 80ge.br2-cr1.smartdc.rtd.i3d.net  0.0%    10   84.4  96.7  84.3 146.7  25.4
     11.|-- hosted-by.i3d.net                 0.0%    10   84.4  84.8  84.4  87.8   0.8
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 syd-au-ping.vultr.com
    Start: Mon Feb 26 13:43:45 2018
    HOST: hostname                         Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                                      0.0%    10   38.6  33.9  26.8  45.1   5.7
      2.|-- 147.75.36.4                                  0.0%    10  492.6 167.4  12.0 1010. 331.5
      3.|-- 0.ae22.bbr2.ewr1.packet.net                  0.0%    10    0.3   0.3   0.2   0.4   0.0
      4.|-- 10ge3-9.core1.nyc6.he.net                    0.0%    10    6.9   2.6   1.3   8.3   2.6
      5.|-- 100ge13-1.core1.nyc4.he.net                  0.0%    10    1.4   5.8   1.4  17.9   4.8
      6.|-- 100ge8-1.core1.sjc2.he.net                   0.0%    10   63.0  63.1  62.9  63.6   0.0
      7.|-- 10ge4-4.core1.sjc1.he.net                    0.0%    10   64.1  63.7  63.1  67.1   1.1
      8.|-- vocus.gigabitethernet2-13.core1.sjc1.he.net  0.0%    10   68.4  68.5  68.4  68.7   0.0
      9.|-- bundle-153.cor02.sjc01.ca.vocus.net          0.0%    10  218.5 218.2 218.1 218.5   0.0
     10.|-- BE-201.cor02.syd04.nsw.VOCUS.net.au          0.0%    10  218.1 218.2 218.0 218.5   0.0
     11.|-- bundle-101.bdr01.syd03.nsw.vocus.net.au      0.0%    10  219.0 218.7 218.5 219.0   0.0
     12.|-- static-73.7.255.49.in-addr.VOCUS.net.au      0.0%    10  217.8 217.8 217.8 217.9   0.0
     13.|-- xe-1-0-1.gw102.sy3.ap.equinix.com            0.0%    10  213.7 213.7 213.7 213.8   0.0
     14.|-- 180.189.25.6                                30.0%    10  231.4 231.7 231.4 232.9   0.4
     15.|-- 103-43-72-34.choopa.net                      0.0%    10  232.1 232.1 232.0 232.3   0.0
     16.|-- ???                                         100.0    10    0.0   0.0   0.0   0.0   0.0
     17.|-- 108.61.212.117.vultr.com                     0.0%    10  222.8 222.8 222.6 222.9   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 hnd-jp-ping.vultr.com
    Start: Mon Feb 26 13:44:03 2018
    HOST: hostname                        Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                                     0.0%    10   11.3  14.3  11.3  19.8   2.9
      2.|-- 147.75.36.4                                 0.0%    10   12.5 202.5  12.5 1031. 334.8
      3.|-- 0.ae22.bbr2.ewr1.packet.net                 0.0%    10    0.2   0.1   0.1   0.2   0.0
      4.|-- 0.xe-0-0-0.csr1.ewr2.packet.net             0.0%    10  114.7  16.3   1.2 114.7  34.8
      6.|-- 0.et-5-0-5.BR2.NYC4.ALTER.NET               0.0%    10    1.5   3.4   1.4  20.0   5.8
      7.|-- verizon.com.customer.alter.net              0.0%    10    1.8   1.7   1.5   2.1   0.0
      8.|-- ae-3.r24.nycmny01.us.bb.gin.ntt.net         0.0%    10    3.2   2.5   2.3   3.3   0.0
      9.|-- ae-2.r20.sttlwa01.us.bb.gin.ntt.net         0.0%    10   69.6  72.8  69.5  94.4   7.9
     10.|-- ae-13.r30.tokyjp05.jp.bb.gin.ntt.net       10.0%    10  177.4 174.0 173.0 177.4   1.3
     11.|-- ae-2.r02.tokyjp05.jp.bb.gin.ntt.net         0.0%    10  181.2 181.0 181.0 181.2   0.0
     12.|-- ce-0-14-0-1.r02.tokyjp05.jp.ce.gin.ntt.net  0.0%    10  206.2 187.4 181.7 206.2   8.6
     13.|-- ???                                        100.0    10    0.0   0.0   0.0   0.0   0.0
     14.|-- vl516-ds1-b5-r2605.tyo1.choopa.net          0.0%    10  181.4 182.2 181.4 188.6   2.2
     15.|-- ???                                        100.0    10    0.0   0.0   0.0   0.0   0.0
     16.|-- 108.61.201.151.vultr.com                    0.0%    10  175.1 175.0 174.9 175.1   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 lax-ca-us-ping.vultr.com
    Start: Mon Feb 26 13:44:20 2018
    HOST: hostname                Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                             0.0%    10   20.2  23.7  11.3  90.5  23.7
      2.|-- 147.75.36.2                         0.0%    10   14.8 130.2   9.0 1052. 324.7
      3.|-- 0.ae11.bbr1.ewr1.packet.net         0.0%    10    0.2   2.9   0.1  27.0   8.5
      4.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
      5.|-- nyk-b6-link.telia.net               0.0%    10    1.0   1.3   1.0   1.9   0.0
      6.|-- nyk-bb3-link.telia.net              0.0%    10    1.2   2.3   1.1  11.6   3.2
      7.|-- las-b24-link.telia.net              0.0%    10   88.0  89.3  88.0  94.8   2.2
      8.|-- las-b4-link.telia.net               0.0%    10   87.5  87.6  87.4  88.7   0.3
      9.|-- vultr-ic-318444-las-b4.c.telia.net  0.0%    10   90.1  91.9  87.4  97.9   3.8
     10.|-- vl901-ds1-b5-c705.lax3.choopa.net   0.0%    10   88.5  88.7  88.4  89.6   0.0
     11.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
     12.|-- 108.61.219.200.vultr.com            0.0%    10   87.5  87.5  87.5  87.6   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 wa-us-ping.vultr.com
    Start: Mon Feb 26 13:44:37 2018
    HOST: hostname                  Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                               0.0%    10   11.3  16.2  11.3  21.3   3.8
      2.|-- 147.75.36.2                           0.0%    10   19.6 115.0  11.8 712.8 229.6
      3.|-- 0.ae11.bbr1.ewr1.packet.net           0.0%    10    1.4   0.3   0.1   1.4   0.3
      4.|-- 0.xe-0-0-1.csr1.ewr2.packet.net       0.0%    10    3.8   5.3   1.4  11.8   3.3
      7.|-- lag-10.ear4.Newark1.Level3.net       30.0%    10    1.2   1.1   1.0   1.2   0.0
      8.|-- ae-2-3613.edge1.Seattle3.Level3.net   0.0%    10   74.8  74.8  74.7  75.2   0.0
      9.|-- CHOOPA-LLC.edge1.Seattle3.Level3.net  0.0%    10   72.0  71.6  71.5  72.0   0.0
     10.|-- ???                                  100.0    10    0.0   0.0   0.0   0.0   0.0
     11.|-- 108.61.194.105.vultr.com              0.0%    10   71.4  71.5  71.4  71.7   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 tx-us-ping.vultr.com
    Start: Mon Feb 26 13:44:52 2018
    HOST: hostname                 Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                              0.0%    10   11.7  21.4  11.7  63.8  15.8
      2.|-- 147.75.36.2                          0.0%    10   17.2  64.0  12.1 298.2 101.5
      3.|-- 0.ae21.bbr2.ewr1.packet.net          0.0%    10    0.3   0.4   0.2   1.1   0.0
      4.|-- 0.ae0.bbr1.ewr1.packet.net           0.0%    10    3.2   1.4   0.3   7.0   2.0
      5.|-- ???                                 100.0    10    0.0   0.0   0.0   0.0   0.0
      6.|-- nyk-b6-link.telia.net                0.0%    10    1.6   1.1   1.0   1.6   0.0
      7.|-- nyk-bb3-link.telia.net               0.0%    10   39.0  39.5  38.9  42.9   1.1
      8.|-- ash-bb3-link.telia.net               0.0%    10   36.8  37.0  36.3  40.9   1.3
      9.|-- atl-b22-link.telia.net               0.0%    10   20.6  19.2  18.7  20.6   0.5
     10.|-- dls-b22-link.telia.net               0.0%    10   38.9  38.8  38.7  38.9   0.0
     11.|-- vultr-ic-318447-dls-b22.c.telia.net 10.0%    10   39.2  39.1  39.0  39.7   0.0
     12.|-- 108-61-110-6.choopa.net             90.0%    10  158.5 158.5 158.5 158.5   0.0
     13.|-- ???                                 100.0    10    0.0   0.0   0.0   0.0   0.0
     14.|-- 108.61.224.175.vultr.com             0.0%    10   42.7  39.5  39.1  42.7   1.1
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 il-us-ping.vultr.com
    Start: Mon Feb 26 13:45:09 2018
    HOST: hostname                 Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                              0.0%    10   17.3  35.6  12.4 105.2  34.5
      2.|-- 147.75.36.4                          0.0%    10  1086. 295.3  13.1 1086. 457.1
      3.|-- 0.ae12.bbr1.ewr1.packet.net          0.0%    10    0.2   4.0   0.2  37.5  11.7
      4.|-- ???                                 100.0    10    0.0   0.0   0.0   0.0   0.0
      5.|-- nyk-b6-link.telia.net                0.0%    10    1.0   1.0   1.0   1.0   0.0
      6.|-- nyk-bb4-link.telia.net               0.0%    10   24.0  23.8  23.7  24.0   0.0
      7.|-- chi-b21-link.telia.net               0.0%    10   24.6  24.6  24.6  24.7   0.0
      8.|-- vultr-ic-318445-chi-b21.c.telia.net  0.0%    10   27.5  26.8  26.5  27.5   0.0
      9.|-- vl820-ds1-b5-gj178.chi2.choopa.net   0.0%    10   25.0  25.1  24.9  25.3   0.0
     10.|-- ???                                 100.0    10    0.0   0.0   0.0   0.0   0.0
     11.|-- 107.191.51.12.vultr.com              0.0%    10   25.0  25.1  24.9  25.8   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 ga-us-ping.vultr.com
    Start: Mon Feb 26 13:45:24 2018
    HOST: hostname                 Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                              0.0%    10   14.9  15.1   7.6  25.6   5.5
      2.|-- 147.75.36.2                          0.0%    10   18.0 177.5  13.0 642.2 262.6
      3.|-- 0.ae21.bbr2.ewr1.packet.net          0.0%    10    0.3   0.3   0.2   0.7   0.0
      4.|-- 0.ae0.bbr1.ewr1.packet.net           0.0%    10   10.4   1.5   0.4  10.4   3.1
      5.|-- ???                                 100.0    10    0.0   0.0   0.0   0.0   0.0
      6.|-- nyk-b6-link.telia.net                0.0%    10    1.1   1.2   1.0   2.1   0.0
      7.|-- nyk-bb3-link.telia.net               0.0%    10   18.6  19.5  18.6  24.6   1.8
      8.|-- ash-bb3-link.telia.net               0.0%    10    7.7   8.2   7.4  13.3   1.7
      9.|-- atl-b21-link.telia.net               0.0%    10   18.4  18.6  18.4  19.3   0.0
     10.|-- vultr-ic-319646-atl-b21.c.telia.net  0.0%    10   21.7  18.7  18.3  21.7   1.1
     11.|-- 45-63-44-129.choopa.net              0.0%    10   19.0  18.9  18.8  19.0   0.0
     12.|-- ???                                 100.0    10    0.0   0.0   0.0   0.0   0.0
     13.|-- 108.61.193.166.vultr.com             0.0%    10   18.7  18.8  18.6  19.3   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 fl-us-ping.vultr.com
    Start: Mon Feb 26 13:45:40 2018
    HOST: hostname                Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                             0.0%    10   41.2  27.9  17.7  41.2   7.6
      2.|-- 147.75.36.2                         0.0%    10  389.3  96.6  12.3 389.3 131.5
      3.|-- 0.ae21.bbr2.ewr1.packet.net         0.0%    10    0.2   0.3   0.2   0.5   0.0
      4.|-- 0.ae0.bbr1.ewr1.packet.net          0.0%    10    0.9   0.9   0.4   4.6   1.3
      5.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
      6.|-- nyk-b6-link.telia.net               0.0%    10    1.1   1.1   1.0   1.9   0.0
      7.|-- nyk-bb3-link.telia.net              0.0%    10    1.2   4.8   1.1  11.8   4.5
      8.|-- ash-bb3-link.telia.net              0.0%    10    7.3   8.7   7.3  21.3   4.4
      9.|-- mai-b1-link.telia.net               0.0%    10   34.3  34.3  34.2  34.6   0.0
     10.|-- vultr-ic-318945-mai-b1.c.telia.net  0.0%    10   34.6  34.6  34.4  34.8   0.0
     11.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
     12.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
     13.|-- 104.156.244.232                     0.0%    10   34.1  34.0  34.0  34.1   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 nj-us-ping.vultr.com
    Start: Mon Feb 26 13:45:57 2018
    HOST: hostname                Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                             0.0%    10   14.1  17.5  13.6  20.4   2.5
      2.|-- 147.75.36.4                         0.0%    10   12.8  74.9  12.8 524.1 159.4
      3.|-- 0.ae12.bbr1.ewr1.packet.net         0.0%    10    0.2   0.7   0.2   5.4   1.5
      4.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
      5.|-- nyk-b6-link.telia.net               0.0%    10    1.0   1.2   1.0   2.5   0.3
      6.|-- nyk-bb3-link.telia.net              0.0%    10    5.1  11.3   1.1  19.4   6.1
      7.|-- nyk-b2-link.telia.net               0.0%    10    1.7   1.7   1.7   1.8   0.0
      8.|-- vultr-ic-318452-nyk-b2.c.telia.net  0.0%    10    2.1   2.0   1.9   2.3   0.0
      9.|-- 66-55-144-149.choopa.net            0.0%    10    3.8   2.6   2.1   4.4   0.7
     10.|-- 108-61-244-174.choopa.net           0.0%    10    2.6   2.8   2.5   3.9   0.3
     11.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
     12.|-- 108.61.149.182.vultr.com            0.0%    10    2.1   2.1   2.0   2.2   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 fra-de-ping.vultr.com
    Start: Mon Feb 26 13:46:12 2018
    HOST: hostname                 Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                              0.0%    10   14.5  18.3  11.8  31.7   5.8
      2.|-- 147.75.36.2                          0.0%    10  867.7 221.5  12.8 867.7 345.3
      3.|-- 0.ae21.bbr2.ewr1.packet.net          0.0%    10    0.3   1.7   0.2  15.0   4.6
      4.|-- packet.tbr2.wbr.nac.net              0.0%    10    0.5   0.5   0.3   0.7   0.0
      5.|-- 0.lag16.tbr1.wbr.nac.net             0.0%    10    0.2   1.0   0.2   4.5   1.5
      6.|-- 0.e3-4.tbr1.tl9.nac.net              0.0%    10    8.8   4.7   1.1  11.2   4.6
      7.|-- nyk-b3-link.telia.net                0.0%    10    1.2   1.5   1.2   2.5   0.3
      8.|-- nyk-bb4-link.telia.net               0.0%    10    1.8   1.9   1.7   2.2   0.0
      9.|-- prs-bb4-link.telia.net               0.0%    10   88.2  88.5  88.1  91.0   0.8
     10.|-- prs-b2-link.telia.net                0.0%    10   72.4  72.4  72.4  72.6   0.0
     11.|-- prs-bb3-link.telia.net               0.0%    10   81.7  81.7  81.7  81.9   0.0
     12.|-- ffm-bb3-link.telia.net               0.0%    10   87.9  88.0  87.8  88.6   0.0
     13.|-- ffm-b12-link.telia.net               0.0%    10   85.4  86.0  85.3  89.4   1.1
     14.|-- vultr-ic-312911-ffm-b12.c.telia.net  0.0%    10   82.1  93.1  81.8 189.2  33.7
     15.|-- ???                                 100.0    10    0.0   0.0   0.0   0.0   0.0
     16.|-- 108.61.210.117.vultr.com             0.0%    10   81.7  82.0  81.6  82.4   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 ams-nl-ping.vultr.com
    Start: Mon Feb 26 13:46:29 2018
    HOST: hostname                Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                             0.0%    10   35.5  30.8  18.8  39.5   5.8
      2.|-- 147.75.36.4                         0.0%    10   20.3 196.9  15.5 1123. 345.3
      3.|-- 0.ae22.bbr2.ewr1.packet.net         0.0%    10    0.3   1.4   0.2  11.4   3.5
      4.|-- packet.tbr2.wbr.nac.net             0.0%    10    0.3   6.8   0.3  53.9  16.9
      5.|-- 0.lag16.tbr1.wbr.nac.net            0.0%    10    0.2   1.6   0.2   6.4   2.2
      6.|-- 0.e3-4.tbr1.tl9.nac.net             0.0%    10    7.4   5.0   1.1  11.2   4.0
      7.|-- nyk-b3-link.telia.net               0.0%    10    1.3   1.2   1.2   1.4   0.0
      8.|-- nyk-bb3-link.telia.net              0.0%    10    1.2   1.3   1.2   1.5   0.0
      9.|-- ldn-bb3-link.telia.net              0.0%    10   71.3  71.2  71.0  72.6   0.5
     10.|-- adm-bb3-link.telia.net              0.0%    10   79.2  79.2  79.0  79.5   0.0
     11.|-- adm-b3-link.telia.net               0.0%    10   79.5  79.4  79.3  79.5   0.0
     12.|-- vultr-ic-313751-adm-b3.c.telia.net 20.0%    10   79.4  79.6  79.4  79.8   0.0
     13.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
     14.|-- 108.61.198.102.vultr.com            0.0%    10   79.6  79.5  79.4  79.7   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 lon-gb-ping.vultr.com
    Start: Mon Feb 26 13:46:45 2018
    HOST: hostname                Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                             0.0%    10   30.6  32.6  21.8  58.2  10.1
      2.|-- 147.75.36.2                         0.0%    10  928.0 252.1  11.3 928.0 383.6
      3.|-- 0.ae21.bbr2.ewr1.packet.net         0.0%    10    0.2   3.5   0.2  32.8  10.3
      4.|-- packet.tbr2.wbr.nac.net             0.0%    10  131.3  63.4   0.3 192.5  75.4
      5.|-- 0.lag16.tbr1.wbr.nac.net            0.0%    10    0.2   0.8   0.2   5.8   1.6
      6.|-- 0.e3-4.tbr1.tl9.nac.net             0.0%    10    1.1   1.6   1.1   3.6   0.7
      7.|-- nyk-b3-link.telia.net               0.0%    10    2.2   4.3   1.2  29.9   9.0
      8.|-- nyk-bb3-link.telia.net              0.0%    10    1.2   2.9   1.1  12.4   3.6
      9.|-- ldn-bb3-link.telia.net              0.0%    10   71.0  71.6  71.0  75.5   1.4
     10.|-- ldn-b9-link.telia.net               0.0%    10   71.0  71.1  70.9  72.0   0.0
     11.|-- vultr-ic-312913-ldn-b9.c.telia.net  0.0%    10   72.8  72.8  71.7  78.7   2.1
     12.|-- 10.69.1.90                         10.0%    10   72.6  93.3  72.0 176.4  41.2
     13.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
     14.|-- 108.61.196.101.vultr.com            0.0%    10   71.3  71.2  71.1  71.3   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 par-fr-ping.vultr.com
    Start: Mon Feb 26 13:47:01 2018
    HOST: hostname                Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                             0.0%    10   16.1  14.0   6.2  20.5   3.9
      2.|-- 147.75.36.2                         0.0%    10   19.4  96.7   6.5 514.7 175.1
      3.|-- 0.ae21.bbr2.ewr1.packet.net         0.0%    10    0.3   0.3   0.2   0.3   0.0
      4.|-- packet.tbr2.wbr.nac.net             0.0%    10    0.3   0.4   0.3   0.6   0.0
      5.|-- 0.lag16.tbr1.wbr.nac.net            0.0%    10    0.2   0.3   0.2   0.4   0.0
      6.|-- 0.e3-4.tbr1.tl9.nac.net             0.0%    10    1.2   1.2   1.1   1.7   0.0
      7.|-- nyk-b3-link.telia.net               0.0%    10    1.2   1.8   1.2   7.5   1.9
      8.|-- nyk-bb4-link.telia.net              0.0%    10    2.0   1.9   1.7   2.1   0.0
      9.|-- prs-bb4-link.telia.net              0.0%    10   73.9  74.3  72.2  77.5   1.9
     10.|-- prs-b7-link.telia.net               0.0%    10   72.3  72.3  72.2  72.4   0.0
     11.|-- vultr-ic-312910-prs-b7.c.telia.net 20.0%    10   85.4  86.2  85.4  91.5   2.0
     12.|-- ???                                100.0    10    0.0   0.0   0.0   0.0   0.0
     13.|-- 108.61.209.127.vultr.com            0.0%    10   85.4  85.4  85.4  85.5   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 VersaWeb Las Vegas
    Start: Mon Feb 26 13:47:17 2018
    HOST: hostname                    Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                                 0.0%    10   21.5  16.3  11.5  21.5   3.8
      2.|-- 147.75.36.2                             0.0%    10   21.4 212.5  15.2 1004. 382.3
      3.|-- 0.ae11.bbr1.ewr1.packet.net             0.0%    10    0.2   0.5   0.2   3.1   0.7
      4.|-- 10gigabitethernet1-2.core1.nyc6.he.net  0.0%    10    6.9   7.7   1.4  12.7   3.8
      5.|-- 100ge13-1.core1.nyc4.he.net             0.0%    10    5.7   4.3   1.4  11.1   2.9
      6.|-- 100ge16-1.core1.ash1.he.net             0.0%    10   14.2  19.3   6.5  92.1  25.8
      7.|-- 100ge14-2.core1.mci3.he.net             0.0%    10   32.8  32.7  30.6  39.4   2.6
      8.|-- 100ge12-1.core1.den1.he.net             0.0%    10   42.6  44.1  42.6  53.0   3.2
      9.|-- 100ge10-2.core1.slc1.he.net             0.0%    10   61.4  59.2  54.2  64.4   3.7
     10.|-- 10ge1-1.core1.las1.he.net               0.0%    10   70.3  67.6  62.9  70.5   3.3
     11.|-- 172.18.0.22                             0.0%    10   76.2  78.6  76.1  87.0   4.1
     12.|-- 192.228.109.121                         0.0%    10   76.3  77.9  76.2  86.9   3.5
     13.|-- 192.228.109.86                          0.0%    10   76.5  76.5  76.4  76.6   0.0
     14.|-- lg.las1.ultravps.net                    0.0%    10   76.2  76.2  76.1  76.3   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 VersaWeb Seattle
    Start: Mon Feb 26 13:47:35 2018
    HOST: hostname                       Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                                    0.0%    10   11.9  25.2  11.9  97.4  25.6
      2.|-- 147.75.36.4                                0.0%    10   53.8 295.1  12.8 1136. 448.1
      3.|-- 0.ae22.bbr2.ewr1.packet.net                0.0%    10    0.2   0.3   0.2   1.0   0.0
      4.|-- packet.tbr2.wbr.nac.net                    0.0%    10    0.4   2.7   0.4  12.3   4.5
      5.|-- 0.lag16.tbr1.wbr.nac.net                   0.0%    10    0.4   0.3   0.2   0.5   0.0
      6.|-- 0.e1-3.tbr1.ewr.nac.net                    0.0%    10    1.0   1.0   0.9   1.0   0.0
      7.|-- te0-0-0-24.rcr21.ewr01.atlas.cogentco.com  0.0%    10    1.3   1.4   1.3   1.6   0.0
      8.|-- be3200.rcr24.jfk01.atlas.cogentco.com      0.0%    10    1.7   1.7   1.7   1.8   0.0
      9.|-- be2897.ccr42.jfk02.atlas.cogentco.com      0.0%    10    1.5   1.7   1.5   1.8   0.0
     10.|-- be2890.ccr22.cle04.atlas.cogentco.com      0.0%    10   18.2  18.5  18.2  18.7   0.0
     11.|-- be2718.ccr42.ord01.atlas.cogentco.com      0.0%    10   19.7  19.9  19.7  20.1   0.0
     12.|-- be2832.ccr22.mci01.atlas.cogentco.com      0.0%    10   47.9  48.1  47.8  48.3   0.0
     13.|-- be3036.ccr22.den01.atlas.cogentco.com      0.0%    10   59.9  59.7  59.6  59.9   0.0
     14.|-- be3038.ccr21.slc01.atlas.cogentco.com      0.0%    10   72.5  72.4  72.1  72.6   0.0
     15.|-- be2085.ccr21.sea02.atlas.cogentco.com      0.0%    10   66.9  66.8  66.7  66.9   0.0
     16.|-- 38.142.48.250                              0.0%    10   66.4  66.5  66.4  67.8   0.3
     17.|-- 76.164.234.1                               0.0%    10   66.5  66.5  66.5  66.6   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 OVH Canada
    Start: Mon Feb 26 13:47:51 2018
    HOST: hostname    Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                 0.0%    10    7.1   7.8   2.3  11.0   2.9
      2.|-- 2604:1380:2:79::        0.0%    10    2.0   5.0   2.0  11.4   2.6
      3.|-- 2604:1380:2:20::        0.0%    10    0.3   3.8   0.2  35.6  11.2
      4.|-- de-cix.nyc.ovh.com      0.0%    10    2.0   2.0   1.7   2.2   0.0
      5.|-- 2607:5300::1c6          0.0%    10   10.5  10.5  10.0  10.8   0.0
      6.|-- 2607:5300::1c3          0.0%    10    8.9   8.9   8.9   9.2   0.0
      7.|-- 2001:41d0:0:50::6:80d   0.0%    10    8.9   8.9   8.9   9.0   0.0
      8.|-- 2001:41d0:0:50::2:404d  0.0%    10    9.9   9.6   9.3  10.2   0.0
      9.|-- 2607:5300:60:44a5::1    0.0%    10    9.1   9.1   8.9   9.5   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 10gbps.io Frankfurt:
    Start: Mon Feb 26 13:48:07 2018
    HOST: hostname              Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                           0.0%    10   34.0  32.2  15.6  38.2   6.3
      2.|-- 147.75.36.2                       0.0%    10   12.6 144.7  12.4 710.5 272.6
      3.|-- 0.ae21.bbr2.ewr1.packet.net       0.0%    10    0.4   2.3   0.2  20.9   6.5
      4.|-- ae4-1.nyk10.core-backbone.com     0.0%    10    1.2   1.1   1.1   1.3   0.0
      5.|-- ae2-2025.fra30.core-backbone.com  0.0%    10   81.0  81.7  80.7  89.0   2.5
      6.|-- core-backbone.cdn77.com           0.0%    10   93.2  93.2  93.1  93.2   0.0
      7.|-- frankfurt-20.cdn77.com            0.0%    10   93.5  93.2  93.1  93.5   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 10gbps.io London:
    Start: Mon Feb 26 13:48:23 2018
    HOST: hostname               Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                            0.0%    10   14.1  14.6   6.6  21.9   4.9
      2.|-- 147.75.36.4                        0.0%    10   29.7 227.8  12.2 1102. 412.1
      3.|-- 0.ae12.bbr1.ewr1.packet.net        0.0%    10    0.1   3.0   0.1  28.0   8.8
      4.|-- 0.ae0.bbr2.ewr1.packet.net         0.0%    10    0.4   2.0   0.1  18.6   5.8
      5.|-- ae4-1.nyk10.core-backbone.com      0.0%    10    1.1   2.8   1.1  17.7   5.2
      6.|-- ae10-2092.lon10.core-backbone.com  0.0%    10   70.6  74.1  70.3 101.7   9.8
      7.|-- core-backbone.cdn77.com            0.0%    10   93.8  93.7  93.7  93.8   0.0
      8.|-- london-20.cdn77.com                0.0%    10   94.6  94.6  94.5  94.7   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 10gbps.io Paris:
    Start: Mon Feb 26 13:48:40 2018
    HOST: hostname                 Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                              0.0%    10   15.8  27.0  14.7  63.0  17.9
      2.|-- 147.75.36.2                          0.0%    10   13.6 154.9  12.3 1068. 328.5
      3.|-- 0.ae21.bbr2.ewr1.packet.net          0.0%    10    0.3   0.3   0.2   0.4   0.0
      4.|-- 0.xe-0-0-0.csr1.ewr2.packet.net      0.0%    10    2.3  26.1   2.3  94.5  33.1
      6.|-- 0.ae8.BR2.NYC4.ALTER.NET             0.0%    10    3.1   5.3   2.7  25.9   7.2
      7.|-- verizon.com.customer.alter.net       0.0%    10    2.2   2.3   2.2   2.5   0.0
      8.|-- ae-3.r24.nycmny01.us.bb.gin.ntt.net  0.0%    10    4.0   5.0   3.0  10.8   2.9
      9.|-- ae-9.r24.londen12.uk.bb.gin.ntt.net  0.0%    10   74.8  75.4  74.8  77.6   0.7
     10.|-- ae-1.r25.londen12.uk.bb.gin.ntt.net  0.0%    10   75.1  75.5  75.1  78.7   1.0
     11.|-- ae-2.r04.parsfr01.fr.bb.gin.ntt.net  0.0%    10   92.4  92.6  92.4  92.9   0.0
     12.|-- ae-5.r03.parsfr02.fr.bb.gin.ntt.net  0.0%    10   92.4  92.5  92.4  92.9   0.0
     13.|-- ae-8.r02.parsfr02.fr.bb.gin.ntt.net  0.0%    10   80.1  79.9  79.7  80.1   0.0
     14.|-- 81.25.206.50                         0.0%    10  101.3 101.3 101.2 101.4   0.0
     15.|-- unn-185-93-2-193.10gbps.io           0.0%    10  101.3 101.4 101.3 101.5   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 10gbps.io Singapore:
    Start: Mon Feb 26 13:48:57 2018
    HOST: hostname                       Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                                    0.0%    10   12.6  16.4   9.0  30.8   6.1
      2.|-- 147.75.36.2                                0.0%    10  741.2 232.0  13.7 842.1 349.5
      3.|-- 0.ae21.bbr2.ewr1.packet.net                0.0%    10    0.2   0.3   0.2   0.4   0.0
      4.|-- packet.tbr2.wbr.nac.net                    0.0%    10    0.3  26.9   0.3 147.8  56.0
      5.|-- 0.lag16.tbr1.wbr.nac.net                   0.0%    10    0.2   0.3   0.2   1.2   0.0
      6.|-- 0.e1-3.tbr1.ewr.nac.net                    0.0%    10    3.5   5.0   0.9  11.1   3.7
      7.|-- te0-0-0-24.rcr21.ewr01.atlas.cogentco.com  0.0%    10    1.3   1.3   1.2   1.4   0.0
      8.|-- be3199.rcr23.jfk01.atlas.cogentco.com      0.0%    10    1.8   2.0   1.8   3.4   0.5
      9.|-- be2896.ccr41.jfk02.atlas.cogentco.com      0.0%    10    2.0   1.7   1.5   2.0   0.0
     10.|-- be2806.ccr41.dca01.atlas.cogentco.com      0.0%    10    8.0   7.8   7.5   8.0   0.0
     11.|-- be2112.ccr41.atl01.atlas.cogentco.com      0.0%    10   19.7  19.9  19.7  20.1   0.0
     12.|-- be2687.ccr41.iah01.atlas.cogentco.com      0.0%    10   45.3  45.4  45.2  45.9   0.0
     13.|-- be2927.ccr21.elp01.atlas.cogentco.com      0.0%    10   73.7  73.5  73.3  73.8   0.0
     14.|-- be2929.ccr31.phx01.atlas.cogentco.com      0.0%    10   73.6  73.6  73.4  73.9   0.0
     15.|-- be2931.ccr41.lax01.atlas.cogentco.com      0.0%    10   73.6  73.6  73.5  73.7   0.0
     16.|-- be3271.ccr41.lax04.atlas.cogentco.com      0.0%    10   75.4  75.5  75.4  75.7   0.0
     17.|-- be3406.ccr21.sin01.atlas.cogentco.com      0.0%    10  246.7 246.8 246.6 247.9   0.3
     18.|-- 154.18.2.2                                 0.0%    10  245.0 245.0 245.0 245.3   0.0
     19.|-- hol14.homeoflim.com                        0.0%    10  251.7 251.7 251.5 252.3   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 10gbps.io Stockholm:
    Start: Mon Feb 26 13:49:15 2018
    HOST: hostname              Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                           0.0%    10   17.0  15.5  11.2  20.0   2.9
      2.|-- 147.75.36.4                       0.0%    10  469.0 136.8   9.3 574.4 211.0
      3.|-- 0.ae12.bbr1.ewr1.packet.net       0.0%    10    0.2   0.2   0.2   0.3   0.0
      4.|-- 0.ae0.bbr2.ewr1.packet.net        0.0%    10    2.1   0.4   0.1   2.1   0.6
      5.|-- ae4-1.nyk10.core-backbone.com     0.0%    10    1.1   1.2   1.1   1.4   0.0
      6.|-- ae1-2081.sth10.core-backbone.com  0.0%    10  101.3 101.3 101.3 101.4   0.0
      7.|-- core-backbone.cdn77.com           0.0%    10  101.1 101.1 101.1 101.1   0.0
      8.|-- stockholm-10.cdn77.com            0.0%    10  108.3 108.3 108.2 108.6   0.0
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 10gbps.io Moscow:
    Start: Mon Feb 26 13:49:31 2018
    HOST: hostname         Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                      0.0%    10   19.7  17.8  12.4  22.0   3.3
      2.|-- 147.75.36.2                  0.0%    10   20.2 133.2  13.0 654.3 246.2
      3.|-- 0.ae21.bbr2.ewr1.packet.net  0.0%    10    0.3   0.3   0.2   0.4   0.0
      4.|-- de-cix.nyc.retn.net          0.0%    10    1.0   1.1   1.0   1.2   0.0
      5.|-- ae1-3.RT.MR.MSK.RU.retn.net  0.0%    10  114.3 115.8 114.3 120.7   2.4
      6.|-- 87.245.204.20                0.0%    10  113.6 113.6 113.3 115.2   0.5
    
    ----------------------------------------------
    mtr --report -w 200 --report-cycles=10 10gbps.io Miami:
    Start: Mon Feb 26 13:49:46 2018
    HOST: hostname                 Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- gateway                              0.0%    10   78.8  27.2  11.5  78.8  25.4
      2.|-- 147.75.36.4                          0.0%    10   13.0 140.0  11.4 1056. 325.9
      3.|-- 0.ae22.bbr2.ewr1.packet.net          0.0%    10    0.4   0.5   0.1   3.2   0.8
      4.|-- 0.xe-0-0-0.csr1.ewr2.packet.net      0.0%    10    3.4  12.0   2.8  62.0  17.8
      6.|-- 0.ae8.BR2.NYC4.ALTER.NET             0.0%    10    2.7   2.8   2.7   3.2   0.0
      7.|-- verizon.com.customer.alter.net       0.0%    10    2.7   2.4   2.2   2.7   0.0
      8.|-- ae-3.r25.nycmny01.us.bb.gin.ntt.net  0.0%    10    2.3   3.1   2.3   8.1   1.6
      9.|-- ae-9.r22.asbnva02.us.bb.gin.ntt.net  0.0%    10    9.5  10.5   8.7  14.2   2.0
     10.|-- ae-0.r23.asbnva02.us.bb.gin.ntt.net  0.0%    10    8.7   8.6   8.5   8.7   0.0
     11.|-- ae-1.r20.miamfl02.us.bb.gin.ntt.net  0.0%    10   38.7  38.8  38.7  39.3   0.0
     12.|-- ae-1.r04.miamfl02.us.bb.gin.ntt.net  0.0%    10   38.8  38.9  38.7  39.2   0.0
     13.|-- 165.254.21.26                        0.0%    10   38.0  38.0  37.9  38.0   0.0
     14.|-- unn-195-181-162-195.cdn77.com        0.0%    10   56.2  56.0  55.9  56.2   0.0
    
    

    Code (Text):
    -------------------------------------------
    Check system entropy pool availability
    -------------------------------------------
    
    entropy_avail: 4067
    entropy_avail: 4067
    entropy_avail: 4067
    entropy_avail: 4067
    

    Code (Text):
    -------------------------------------------
    OpenSSL System Benchmark
    -------------------------------------------
    
    OpenSSL 1.0.2k-fips  26 Jan 2017
    -------------------------------------------
    openssl speed -multi 48 rsa4096 rsa2048 ecdsap256 sha256 sha1 md5 rc4 aes-256-cbc aes-128-cbc
    :0.014341:0.000225 from 5
    613.33:112702805.33 from 17
    6974464.00:151745877.33:154208938.67 from 23
    OpenSSL 1.0.2k-fips  26 Jan 2017
    built on: reproducible build, date unspecified
    options:bn(64,64) md2(int) rc4(8x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
    compiler: gcc -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DKRB5_MIT -m64 -DL_ENDIAN -Wall -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches   -m64 -mtune=generic -Wa,--noexecstack -DPURIFY -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DRC4_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM
    md5            1713991.89k  5321480.32k 12165950.72k 18738554.88k 21742185.13k
    sha1           2632574.01k  9114193.66k 24777751.21k 42884053.67k 51963423.40k
    rc4           14219831.48k 17600669.03k 18195224.17k 18252467.96k 18290272.94k
    aes-128 cbc    2913189.89k  3122197.89k  3215172.01k  7280857.43k  7405236.09k
    aes-256 cbc    2123871.85k  2238767.57k  2278030.00k  5348877.99k  5405005.17k
    sha256         5504880.33k 15290409.75k 34443615.15k 51130241.37k 59325699.41k
                      sign    verify    sign/s verify/s
    rsa 2048 bits 0.000042s 0.000001s  24069.5 796612.1
    rsa 4096 bits 0.000299s 0.000005s   3346.8 215510.7
                                  sign    verify    sign/s verify/s
     256 bit ecdsa (nistp256)   0.0000s   0.0000s 553741.1 231184.8
    -------------------------------------------
    openssl speed -evp aes256 -multi 48
    16:64:256:1024:8192 from 31
    OpenSSL 1.0.2k-fips  26 Jan 2017
    built on: reproducible build, date unspecified
    options:bn(64,64) md2(int) rc4(8x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
    compiler: gcc -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DKRB5_MIT -m64 -DL_ENDIAN -Wall -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches   -m64 -mtune=generic -Wa,--noexecstack -DPURIFY -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DRC4_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM
    evp           23725037.36k 29072785.82k 31162345.90k 31983885.65k 32225599.49k
    -------------------------------------------
    openssl speed -evp aes128 -multi 48
    OpenSSL 1.0.2k-fips  26 Jan 2017
    built on: reproducible build, date unspecified
    options:bn(64,64) md2(int) rc4(8x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
    compiler: gcc -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DKRB5_MIT -m64 -DL_ENDIAN -Wall -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches   -m64 -mtune=generic -Wa,--noexecstack -DPURIFY -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DRC4_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM
    evp           28439367.64k 39542990.27k 42292041.47k 43768720.73k 44169289.73k
    

    Code (Text):
    -------------------------------------------
    Running mysqlslap
    -------------------------------------------
    
    mysqlslap --auto-generate-sql --auto-generate-sql-add-autoincrement --auto-generate-sql-secondary-indexes=5 --number-int-cols=5 --number-char-cols=5 --number-of-queries=25000 --auto-generate-sql-unique-query-number=40 --auto-generate-sql-unique-write-number=40 --auto-generate-sql-write-number=1000 --concurrency=64 --iterations=10 --engine=myisam
    Benchmark
            Running for engine myisam
            Average number of seconds to run all queries: 0.906 seconds
            Minimum number of seconds to run all queries: 0.889 seconds
            Maximum number of seconds to run all queries: 0.914 seconds
            Number of clients running queries: 64
            Average number of queries per client: 390
    
    Dropping the database is potentially a very bad thing to do.
    Any data stored in the database will be destroyed.
    
    Do you really want to drop the 'test' database [y/N] Database "test" dropped
    

    Code (Text):
    -------------------------------------------
    System PHP Info
    -------------------------------------------
    
    CPU: 48 x AMD EPYC 7401P 24-Core Processor
                  total        used        free      shared  buff/cache   available
    Mem:          64146        3794       59507          14         844       59658
    Low:          64146        4638       59507
    High:             0           0           0
    Swap:          1949           0        1949
    ----------------------------------------------
    PHP 5.6.33 (cli) (built: Feb 26 2018 08:24:24)
    Copyright (c) 1997-2016 The PHP Group
    Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
        with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2016, by Zend Technologies
    ----------------------------------------------
    Configuration File (php.ini) Path: /usr/local/lib
    Loaded Configuration File:         /usr/local/lib/php.ini
    Scan for additional .ini files in: /etc/centminmod/php.d
    Additional .ini files parsed:      /etc/centminmod/php.d/a_customphp.ini,
    /etc/centminmod/php.d/curlcainfo.ini,
    /etc/centminmod/php.d/geoip.ini,
    /etc/centminmod/php.d/igbinary.ini,
    /etc/centminmod/php.d/imagick.ini,
    /etc/centminmod/php.d/mailparse.ini,
    /etc/centminmod/php.d/memcache.ini,
    /etc/centminmod/php.d/memcached.ini,
    /etc/centminmod/php.d/redis.ini,
    /etc/centminmod/php.d/zendopcache.ini
    
    ----------------------------------------------
    [PHP Modules]
    bcmath
    bz2
    calendar
    Core
    ctype
    curl
    date
    dom
    enchant
    ereg
    exif
    filter
    ftp
    gd
    geoip
    gettext
    gmp
    hash
    iconv
    igbinary
    imagick
    imap
    intl
    json
    ldap
    libxml
    mailparse
    mbstring
    mcrypt
    memcache
    memcached
    mhash
    mysql
    mysqli
    mysqlnd
    openssl
    pcntl
    pcre
    PDO
    pdo_mysql
    pdo_sqlite
    Phar
    posix
    pspell
    readline
    redis
    Reflection
    session
    shmop
    SimpleXML
    snmp
    soap
    sockets
    SPL
    sqlite3
    standard
    sysvmsg
    sysvsem
    sysvshm
    tidy
    tokenizer
    xml
    xmlreader
    xmlrpc
    xmlwriter
    xsl
    Zend OPcache
    zip
    zlib
    
    [Zend Modules]
    Zend OPcache
    
    ----------------------------------------------
    -------------------------------------------
    Run PHP test Zend/bench.php
    -------------------------------------------
    
    simple             0.104
    simplecall         0.101
    simpleucall        0.107
    simpleudcall       0.111
    mandel             0.258
    mandel2            0.325
    ackermann(7)       0.115
    ary(50000)         0.015
    ary2(50000)        0.014
    ary3(2000)         0.133
    fibo(30)           0.384
    hash1(50000)       0.027
    hash2(500)         0.026
    heapsort(20000)    0.078
    matrix(20)         0.091
    nestedloop(12)     0.191
    sieve(30)          0.079
    strcat(200000)     0.011
    ------------------------
    Total              2.170
    real: 2.23s user: 2.18s sys: 0.03s cpu: 99% maxmem: 43828 KB cswaits: 8
    
    simple             0.103
    simplecall         0.102
    simpleucall        0.106
    simpleudcall       0.109
    mandel             0.256
    mandel2            0.303
    ackermann(7)       0.115
    ary(50000)         0.015
    ary2(50000)        0.014
    ary3(2000)         0.133
    fibo(30)           0.383
    hash1(50000)       0.027
    hash2(500)         0.026
    heapsort(20000)    0.078
    matrix(20)         0.087
    nestedloop(12)     0.189
    sieve(30)          0.079
    strcat(200000)     0.011
    ------------------------
    Total              2.137
    real: 2.17s user: 2.15s sys: 0.01s cpu: 99% maxmem: 44000 KB cswaits: 3
    
    simple             0.102
    simplecall         0.101
    simpleucall        0.107
    simpleudcall       0.109
    mandel             0.258
    mandel2            0.322
    ackermann(7)       0.115
    ary(50000)         0.015
    ary2(50000)        0.014
    ary3(2000)         0.132
    fibo(30)           0.388
    hash1(50000)       0.027
    hash2(500)         0.027
    heapsort(20000)    0.078
    matrix(20)         0.087
    nestedloop(12)     0.191
    sieve(30)          0.079
    strcat(200000)     0.011
    ------------------------
    Total              2.164
    real: 2.20s user: 2.18s sys: 0.02s cpu: 99% maxmem: 43992 KB cswaits: 3
    
    bench.php results from 3 runs
    2.170
    2.137
    2.164
    
    bench.php avg: 2.1570
    Avg: real: 2.20s user: 2.17s sys: 0.02s cpu: 99.00% maxmem: 43940.00KB cswaits: 4.67
    created results log at /home/phpbench_logs/bench_260218-133419.log
    server PHP info log at /home/phpbench_logs/bench_phpinfo_260218-133419.log
    
    -------------------------------------------
    Run PHP test Zend/micro_bench.php
    -------------------------------------------
    
    empty_loop         0.151
    func()             0.345    0.194
    undef_func()       0.344    0.193
    int_func()         0.284    0.133
    $x = self::$x      0.253    0.102
    self::$x = 0       0.233    0.083
    isset(self::$x)    0.243    0.093
    empty(self::$x)    0.268    0.118
    $x = Foo::$x       0.243    0.092
    Foo::$x = 0        0.225    0.074
    isset(Foo::$x)     0.214    0.063
    empty(Foo::$x)     0.226    0.075
    self::f()          0.385    0.235
    Foo::f()           0.326    0.175
    $x = $this->x      0.221    0.070
    $this->x = 0       0.282    0.131
    $this->x += 2      0.233    0.082
    ++$this->x         0.227    0.076
    --$this->x         0.234    0.083
    $this->x++         0.240    0.089
    $this->x--         0.266    0.115
    isset($this->x)    0.325    0.174
    empty($this->x)    0.246    0.095
    $this->f()         0.424    0.273
    $x = Foo::TEST     0.247    0.096
    new Foo()          0.660    0.509
    $x = TEST          0.193    0.042
    $x = $_GET         0.208    0.057
    $x = $GLOBALS['v'] 0.283    0.132
    $x = $hash['v']    0.244    0.093
    $x = $str[0]       0.320    0.169
    $x = $a ?: null    0.219    0.069
    $x = $f ?: tmp     0.302    0.151
    $x = $f ? $f : $a  0.222    0.071
    $x = $f ? $f : tmp 0.304    0.153
    ------------------------
    Total              9.641
    real: 9.69s user: 9.66s sys: 0.03s cpu: 99% maxmem: 32044 KB cswaits: 4
    
    empty_loop         0.150
    func()             0.330    0.180
    undef_func()       0.325    0.175
    int_func()         0.280    0.130
    $x = self::$x      0.256    0.105
    self::$x = 0       0.277    0.127
    isset(self::$x)    0.255    0.104
    empty(self::$x)    0.262    0.112
    $x = Foo::$x       0.245    0.095
    Foo::$x = 0        0.245    0.095
    isset(Foo::$x)     0.228    0.078
    empty(Foo::$x)     0.232    0.082
    self::f()          0.370    0.220
    Foo::f()           0.331    0.181
    $x = $this->x      0.220    0.070
    $this->x = 0       0.279    0.129
    $this->x += 2      0.232    0.082
    ++$this->x         0.247    0.097
    --$this->x         0.239    0.089
    $this->x++         0.254    0.103
    $this->x--         0.271    0.121
    isset($this->x)    0.335    0.185
    empty($this->x)    0.242    0.092
    $this->f()         0.418    0.268
    $x = Foo::TEST     0.243    0.093
    new Foo()          0.654    0.504
    $x = TEST          0.208    0.057
    $x = $_GET         0.207    0.056
    $x = $GLOBALS['v'] 0.284    0.134
    $x = $hash['v']    0.244    0.094
    $x = $str[0]       0.318    0.168
    $x = $a ?: null    0.221    0.071
    $x = $f ?: tmp     0.301    0.151
    $x = $f ? $f : $a  0.220    0.070
    $x = $f ? $f : tmp 0.303    0.153
    ------------------------
    Total              9.725
    real: 9.76s user: 9.75s sys: 0.01s cpu: 99% maxmem: 31896 KB cswaits: 3
    
    empty_loop         0.151
    func()             0.325    0.174
    undef_func()       0.330    0.179
    int_func()         0.299    0.148
    $x = self::$x      0.235    0.084
    self::$x = 0       0.257    0.106
    isset(self::$x)    0.234    0.083
    empty(self::$x)    0.250    0.100
    $x = Foo::$x       0.225    0.074
    Foo::$x = 0        0.227    0.076
    isset(Foo::$x)     0.215    0.064
    empty(Foo::$x)     0.227    0.076
    self::f()          0.369    0.219
    Foo::f()           0.326    0.175
    $x = $this->x      0.221    0.071
    $this->x = 0       0.282    0.132
    $this->x += 2      0.232    0.081
    ++$this->x         0.234    0.083
    --$this->x         0.234    0.083
    $this->x++         0.253    0.102
    $this->x--         0.268    0.117
    isset($this->x)    0.333    0.182
    empty($this->x)    0.252    0.101
    $this->f()         0.422    0.272
    $x = Foo::TEST     0.243    0.093
    new Foo()          0.666    0.515
    $x = TEST          0.209    0.058
    $x = $_GET         0.205    0.054
    $x = $GLOBALS['v'] 0.283    0.133
    $x = $hash['v']    0.242    0.092
    $x = $str[0]       0.317    0.166
    $x = $a ?: null    0.219    0.069
    $x = $f ?: tmp     0.298    0.148
    $x = $f ? $f : $a  0.236    0.085
    $x = $f ? $f : tmp 0.303    0.153
    ------------------------
    Total              9.623
    real: 9.66s user: 9.65s sys: 0.00s cpu: 99% maxmem: 31864 KB cswaits: 4
    
    micro_bench.php results from 3 runs
    9.641
    9.725
    9.623
    
    micro_bench.php avg: 9.6630
    Avg: real: 9.70s user: 9.69s sys: 0.01s cpu: 99.00% maxmem: 31934.67KB cswaits: 3.67
    created results log at /home/phpbench_logs/bench_micro_260218-133419.log
    server PHP info log at /home/phpbench_logs/bench_phpinfo_260218-133419.log
    
    

    Code (Text):
    ========================================================================
       BYTE UNIX Benchmarks (Version 5.1.3)
    
       System: hostname: GNU/Linux
       OS: GNU/Linux -- 4.15.5-1.el7.elrepo.x86_64 -- #1 SMP Thu Feb 22 22:52:54 EST 2018
       Machine: x86_64 (x86_64)
       Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
       CPU 0: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 1: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 2: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 3: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 4: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 5: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 6: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 7: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 8: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 9: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 10: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 11: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 12: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 13: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 14: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 15: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 16: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 17: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 18: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 19: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 20: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 21: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 22: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 23: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 24: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 25: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 26: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 27: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 28: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 29: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 30: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 31: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 32: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 33: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 34: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 35: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 36: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 37: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 38: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 39: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 40: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 41: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 42: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 43: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 44: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 45: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 46: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       CPU 47: AMD EPYC 7401P 24-Core Processor (3992.4 bogomips)
              Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, AMD virtualization, SYSCALL/SYSRET
       13:54:57 up 53 min,  3 users,  load average: 14.95, 17.56, 8.01; runlevel 3
    

    Code (Text):
    ------------------------------------------------------------------------
    Benchmark Run: Mon Feb 26 2018 13:54:57 - 14:19:09
    48 CPUs in system; running 1 parallel copy of tests
    
    Dhrystone 2 using register variables       35135751.8 lps   (10.0 s, 7 samples)
    Double-Precision Whetstone                     4238.7 MWIPS (11.3 s, 7 samples)
    Execl Throughput                               5577.1 lps   (30.0 s, 2 samples)
    Pipe Throughput                             1604579.9 lps   (10.0 s, 7 samples)
    Pipe-based Context Switching                 239817.5 lps   (10.0 s, 7 samples)
    Process Creation                               9672.3 lps   (30.0 s, 2 samples)
    Shell Scripts (1 concurrent)                   7283.8 lpm   (60.0 s, 2 samples)
    Shell Scripts (16 concurrent)                  2403.6 lpm   (60.0 s, 2 samples)
    Shell Scripts (8 concurrent)                   3783.5 lpm   (60.0 s, 2 samples)
    System Call Overhead                        2126181.0 lps   (10.0 s, 7 samples)
    
    System Benchmarks Partial Index              BASELINE       RESULT    INDEX
    Dhrystone 2 using register variables         116700.0   35135751.8   3010.8
    Double-Precision Whetstone                       55.0       4238.7    770.7
    Execl Throughput                                 43.0       5577.1   1297.0
    Pipe Throughput                               12440.0    1604579.9   1289.9
    Pipe-based Context Switching                   4000.0     239817.5    599.5
    Process Creation                                126.0       9672.3    767.6
    Shell Scripts (1 concurrent)                     42.4       7283.8   1717.9
    Shell Scripts (16 concurrent)                     ---       2403.6      ---
    Shell Scripts (8 concurrent)                      6.0       3783.5   6305.8
    System Call Overhead                          15000.0    2126181.0   1417.5
                                                                       ========
    System Benchmarks Index Score (Partial Only)                         1444.8
    

    Code (Text):
    ------------------------------------------------------------------------
    Benchmark Run: Mon Feb 26 2018 14:19:09 - 14:43:15
    48 CPUs in system; running 48 parallel copies of tests
    
    Dhrystone 2 using register variables      885066254.8 lps   (10.0 s, 7 samples)
    Double-Precision Whetstone                   166998.3 MWIPS (9.9 s, 7 samples)
    Execl Throughput                              26091.6 lps   (29.8 s, 2 samples)
    Pipe Throughput                            43195823.9 lps   (10.0 s, 7 samples)
    Pipe-based Context Switching                6086783.7 lps   (10.0 s, 7 samples)
    Process Creation                              80514.4 lps   (30.0 s, 2 samples)
    Shell Scripts (1 concurrent)                 108057.5 lpm   (60.0 s, 2 samples)
    Shell Scripts (16 concurrent)                  7447.9 lpm   (60.1 s, 2 samples)
    Shell Scripts (8 concurrent)                  14832.3 lpm   (60.1 s, 2 samples)
    System Call Overhead                        8000223.5 lps   (10.0 s, 7 samples)
    
    System Benchmarks Partial Index              BASELINE       RESULT    INDEX
    Dhrystone 2 using register variables         116700.0  885066254.8  75841.2
    Double-Precision Whetstone                       55.0     166998.3  30363.3
    Execl Throughput                                 43.0      26091.6   6067.8
    Pipe Throughput                               12440.0   43195823.9  34723.3
    Pipe-based Context Switching                   4000.0    6086783.7  15217.0
    Process Creation                                126.0      80514.4   6390.0
    Shell Scripts (1 concurrent)                     42.4     108057.5  25485.2
    Shell Scripts (16 concurrent)                     ---       7447.9      ---
    Shell Scripts (8 concurrent)                      6.0      14832.3  24720.5
    System Call Overhead                          15000.0    8000223.5   5333.5
                                                                       ========
    System Benchmarks Index Score (Partial Only)                        17557.2
    
    

    Code (Text):
    -------------------------------------------
    centminmodbench.sh completed
    -------------------------------------------
    
    centminmodbench.sh Total Run Time: 4118.678124307 seconds
    
     
    Last edited: Feb 27, 2018
  6. eva2000

    eva2000 Administrator Staff Member

    35,481
    7,828
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,067
    Local Time:
    6:41 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x

    Packet.net AMD EPYC 7401P + 4.15.6-1 Kernel Revisited



    Seeing as there were some improvements using ELrepo's 4.15.5-1 Linux Kernel, I did a fresh reinstall of CentOS 7.4 OS and updated to latest ELrepo mainline Linux Kernel. Seems in between the reinstall of OS, ELrepo released a newer 4.15.6-1 Linux Kernel so updated to that and did a server reboot before doing a new Centmin Mod 123.09beta01 run :)

    Code (Text):
    uname -r
    4.15.6-1.el7.elrepo.x86_64
    

    Centmin Mod 123.09beta01 install command
    Code (Text):
    yum -y update; curl -O https://centminmod.com/betainstaller.sh && chmod 0700 betainstaller.sh && bash betainstaller.sh
    

    AMD EPYC 7401P Centmin Mod 123.09beta01 Install Times with 4.15.6-1 Linux Kernel which looks much improved due YUM related downloads and also PHP-FPM compile times.
    Code (Text):
    ---------------------------------------------------------------------------
    Total Curl Installer YUM or DNF Time: 110.3599 seconds
    Total YUM Time: 36.359334737 seconds
    Total YUM or DNF + Source Download Time: 57.1042
    Total Nginx First Time Install Time: 196.2899
    Total PHP First Time Install Time: 124.1210
    Download Zip From Github Time: 1.2566
    Total Time Other eg. source compiles: 280.8104
    Total Centmin Mod Install Time: 658.3255
    ---------------------------------------------------------------------------
    Total Install Time (curl yum + cm install + zip download): 769.9420 seconds
    ---------------------------------------------------------------------------
    

    This compared to previous AMD EPYC 7401P Centmin Mod 123.09beta01 Install Times
    Code (Text):
    ---------------------------------------------------------------------------
    Total Curl Installer YUM or DNF Time: 340.5943 seconds
    Total YUM Time: 60.142379961 seconds
    Total YUM or DNF + Source Download Time: 86.9888
    Total Nginx First Time Install Time: 182.0339
    Total PHP First Time Install Time: 183.7518
    Download Zip From Github Time: 1.6182
    Total Time Other eg. source compiles: 418.1967
    Total Centmin Mod Install Time: 870.9712
    ---------------------------------------------------------------------------
    Total Install Time (curl yum + cm install + zip download): 1213.1837 seconds
    ---------------------------------------------------------------------------
    

    compared to Intel Xeon E3-1578Lv5 Skylake Centmin Mod 123.09beta01 Install Times
    Code (Text):
    ---------------------------------------------------------------------------
    Total Curl Installer YUM or DNF Time: 91.2912 seconds
    Total YUM Time: 64.039117358 seconds
    Total YUM or DNF + Source Download Time: 87.3015
    Total Nginx First Time Install Time: 135.2867
    Total PHP First Time Install Time: 141.0516
    Download Zip From Github Time: 1.8695
    Total Time Other eg. source compiles: 264.2750
    Total Centmin Mod Install Time: 627.9147
    ---------------------------------------------------------------------------
    Total Install Time (curl yum + cm install + zip download): 721.0754 seconds
    ---------------------------------------------------------------------------
    


    GCC 8.0.1 Compile & RPM Build Times



    Centmin Mod LEMP Stack's Nginx web server supports multiple GCC/Clang version compilers from GCC 4.8, 5.3, 6.3, 7.2 and 8.x and Clang 3.4, 4, 5, and 6. Currently re-testing custom GCC 8.0.1 compilations and RPM builds and it seems there are huge improvements with 4.15.6-1 kernel in place dropping binutil 2.30 + GCC 8.0.1 compile and RPM build times from ~2684 seconds to ~1939 seconds - ~38% faster so just ~32 minutes !

    Just a reminder these are Profile Guided Optimization based GCC builds so basically need to recompile twice for make. PGO boosted PHP 7 performance as would PGO built GCC 8.0.1 for resulting built binaries that utilise GCC 8.0.1 PGO enabled compiler :)
    Code (Text):
    RPMs Built
    /svr-setup/binutils-gcc8-2.30-1.el7.x86_64.rpm
    /svr-setup/gcc8-pgo-8.0-1.el7.x86_64.rpm
    
    ls -lah /svr-setup | egrep 'gcc8-pgo-8.0|binutils-gcc8-2.30'
    -rw-r--r--  1 root      root       5.2M Feb 26 15:40 binutils-gcc8-2.30-1.el7.x86_64.rpm
    -rw-r--r--  1 root      root        39M Feb 26 16:10 gcc8-pgo-8.0-1.el7.x86_64.rpm
    
    Installed Packages
    Name        : binutils-gcc8
    Arch        : x86_64
    Version     : 2.30
    Release     : 1.el7
    Size        : 48 M
    Repo        : installed
    Summary     : binutils-gcc8 for centminmod.com LEMP stack installs
    URL         : https://centminmod.com
    License     : unknown
    Description : binutils-gcc8 for centminmod.com LEMP stacks
    
    * Mon Feb 26 2018 George Liu <centminmod.com> 2.30
    - Binutils 2.30 for centminmod.com LEMP stack installs
    

    Code (Text):
    Installed Packages
    Name        : gcc8-pgo
    Arch        : x86_64
    Version     : 8.0
    Release     : 1.el7
    Size        : 287 M
    Repo        : installed
    Summary     : gcc8-pgo for centminmod.com LEMP stack installs
    URL         : https://centminmod.com
    License     : unknown
    Description : gcc8-pgo for centminmod.com LEMP stacks
    
    * Mon Feb 26 2018 George Liu <centminmod.com> 8
    - GCC 8 for centminmod.com LEMP stack installs
    
    *************************************************
    Compile GCC Completed
    log: /root/centminlogs/tools-gcc-install-pgo_260218-153809.log
    *************************************************
    
    Total Binutils + GCC Install Time: 1939.402776571 seconds
    


    CPU load average didn't go beyond 43.96 according to sar stats
    Code (Text):
    03:35:01 PM         0       655      0.64      1.82      1.37         0
    03:40:01 PM         2       670      4.25      1.83      1.38         0
    03:45:01 PM         1       673      5.15      4.89      2.80         0
    03:50:01 PM        52       767     43.96     18.94      8.48         0
    03:55:01 PM         1       662      2.99     12.93      9.42         0
    04:00:01 PM        53       771     33.53     15.89     10.89         0
    04:05:01 PM         3       674     16.43     14.97     11.63         0
    04:10:01 PM         1       658      1.39      6.78      9.03         0
    04:15:01 PM         0       649      0.08      2.52      6.55         0
    
    04:15:01 PM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
    04:20:01 PM         0       651      0.00      0.91      4.73         0
    04:25:01 PM         0       650      0.09      0.41      3.44         0
    04:30:01 PM         0       651      0.00      0.16      2.49         0
    04:35:02 PM         0       651      0.08      0.08      1.80         0
    04:40:01 PM         0       649      0.00      0.02      1.29         0
    04:45:01 PM         0       650      0.01      0.01      0.93         0
    04:50:01 PM         0       649      0.00      0.00      0.66         0
    04:55:01 PM         0       654      0.16      0.04      0.49         0
    Average:            3       604      3.94      3.11      2.37         0
    

    sar memory stats for same duration
    Code (Text):
    03:35:01 PM  62480068   3206172      4.88      3208     87172  19021956     28.10   1915948     48452         8
    03:40:01 PM  59514596   6171644      9.40     39828   1461940  20560624     30.38   3984236    877228     99612
    03:45:01 PM  57476376   8209864     12.50     94796   4544748  19185836     28.35   3981352   2638072    124488
    03:50:01 PM  50222836  15463404     23.54     96960   4959464  26218668     38.74  10941888   2929068      2788
    03:55:01 PM  56154188   9532052     14.51     97628   5092688  19995360     29.54   4950652   3026384       104
    04:00:01 PM  48303432  17382808     26.46    101408   6661720  26548720     39.23  11534824   4169488    158160
    04:05:01 PM  52561692  13124548     19.98    105004   9100884  19340412     28.57   5776880   5631816     20644
    04:10:01 PM  49631492  16054748     24.44    107720  11475204  20453276     30.22   8248624   6005844         0
    04:15:01 PM  50673364  15012876     22.86    111284  11181188  19103652     28.23   6964604   6272264        32
    
    04:15:01 PM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
    04:20:01 PM  50670508  15015732     22.86    111500  11186772  19107328     28.23   6969392   6273624        32
    04:25:01 PM  50443288  15242952     23.21    191876  11254744  19105032     28.23   7086220   6305292        24
    04:30:01 PM  50460636  15225604     23.18    192000  11254796  19082940     28.19   7086472   6305348        20
    04:35:02 PM  50478676  15207564     23.15    192152  11254784  19072260     28.18   7086352   6305320         8
    04:40:01 PM  50482276  15203964     23.15    192292  11254828  19060084     28.16   7086780   6305332        20
    04:45:01 PM  50483616  15202624     23.14    192412  11254808  19065324     28.17   7086680   6305312        36
    04:50:01 PM  50481012  15205228     23.15    192608  11255044  19061636     28.16   7090352   6302084         8
    04:55:01 PM  50463996  15222244     23.17    193336  11259452  19092144     28.21   7107500   6306344        80
    05:00:01 PM  50469136  15217104     23.17    193444  11259468  19091884     28.21   7110648   6303540         8
    Average:     57059893   8626347     13.13    126564   5686601   9141103     13.51   3164933   3884380     23466
    


    Packet.net Bill



    Troubleshooting and retesting on AMD EPYC 7401P server, prolonged the server running time so the Packet.net bill for above testing was higher than I was expecting. But the beauty of Packet.net is the hourly billing as there is no way could afford to commit for a full month for the above 3 server configurations ! I look forward to more AMD EPYC testing to properly tune Centmin Mod for AMD EPYC cpus and not just Intel cpus. All the above testing data will be thoroughly analysed and incorporated into Centmin Mod developments :)

    If you like this benchmark review and find it useful, please consider a small donation to support Centmin Mod ;)

    type1e-billing-02.png
     
    Last edited: Feb 27, 2018
  7. eva2000

    eva2000 Administrator Staff Member

    35,481
    7,828
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,067
    Local Time:
    6:41 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x

    Packet.net AMD EPYC 7401P Hardware Details



    Lets dig into the hardware specifics.

    Seems the server is a Dell PowerEdge R6415 with 01/05/2018 dated bios but version number is newer 1.1.3 ? with 8x8GB Micron ECC DDR4-2666Mhz memory and supports a max of 16x memory slots for max 2TB of memory.

    According to Dell PowerEdge R6415 downloads page, there is a newer 1.1.3 February 13, 2018 for code optimization and improved stability & performance than the existing Jan 5th, 2018 bios. But the Jan 5th 2018 bios is marked version 1.1.3 ? There's also a 1.09 bios which addresses CVE-2017-5715 Spectre vulnerability which hasn't been applied yet. So performance might change between existing bios and 1.09 and 1.1.3 bios updates.

    Dell EMC Server PowerEdge BIOS R6415/R7415 Version 1.1.3 Driver Details | Dell US

    and 1.09 January 22, 2018 bios Dell EMC Server PowerEdge BIOS R6415/R7415 Version 1.0.9 Driver Details | Dell US

    dmidecode details

    Code (Text):
    # dmidecode 3.0
    Getting SMBIOS data from sysfs.
    SMBIOS 3.0.0 present.
    
    Handle 0x0000, DMI type 0, 26 bytes
    BIOS Information
            Vendor: Dell Inc.
            Version: 1.1.3
            Release Date: 01/05/2018
            Address: 0xF0000
            Runtime Size: 64 kB
            ROM Size: 16384 kB
    

    Code (Text):
    Handle 0x0100, DMI type 1, 27 bytes
    System Information
            Manufacturer: Dell Inc.
            Product Name: PowerEdge R6415
            Version: Not Specified
            Serial Number: J1X*****
            UUID: 4C4C4544-0031-5810-8035-CAC04F434D32
            Wake-up Type: Power Switch
            SKU Number: SKU=NotProvided;ModelName=PowerEdge R6415
            Family: PowerEdge
    

    AMD EPYC 7401P
    Code (Text):
    andle 0x0400, DMI type 4, 48 bytes
    Processor Information
            Socket Designation: CPU1
            Type: Central Processor
            Family: <OUT OF SPEC>
            Manufacturer: AMD
            ID: 12 0F 80 00 FF FB 8B 17
            Version: AMD EPYC 7401P 24-Core Processor   
            Voltage: 0.8 V
            External Clock: 9600 MHz
            Max Speed: 3000 MHz
            Current Speed: 2000 MHz
            Status: Populated, Enabled
            Upgrade: <OUT OF SPEC>
            L1 Cache Handle: 0x0700
            L2 Cache Handle: 0x0701
            L3 Cache Handle: 0x0702
            Serial Number: Not Specified
            Asset Tag: Not Specified
            Part Number: Not Specified
            Core Count: 24
            Core Enabled: 24
            Thread Count: 48
            Characteristics:
                    64-bit capable
                    Multi-Core
                    Hardware Thread
                    Execute Protection
                    Enhanced Virtualization
    

    8x8GB Micron ECC DDR4-2666Mhz rated memory clocked at DDR4-2400Mhz with part number 9ASF1G72PZ-2G6D1
    Code (Text):
    Handle 0x1000, DMI type 16, 23 bytes
    Physical Memory Array
            Location: System Board Or Motherboard
            Use: System Memory
            Error Correction Type: Multi-bit ECC
            Maximum Capacity: 2 TB
            Error Information Handle: Not Provided
            Number Of Devices: 16
    
    Handle 0x1100, DMI type 17, 40 bytes
    Memory Device
            Array Handle: 0x1000
            Error Information Handle: Not Provided
            Total Width: 72 bits
            Data Width: 64 bits
            Size: 8192 MB
            Form Factor: DIMM
            Set: 1
            Locator: A1
            Bank Locator: Not Specified
            Type: DDR4
            Type Detail: Synchronous Registered (Buffered)
            Speed: 2666 MHz
            Manufacturer: 802C8632802C
            Serial Number: 191DE773
            Asset Tag: 0F1741A0
            Part Number: 9ASF1G72PZ-2G6D1
            Rank: 1
            Configured Clock Speed: 2400 MHz
            Minimum Voltage: 1.2 V
            Maximum Voltage: 1.2 V
            Configured Voltage: 1.2 V
    
    Handle 0x1101, DMI type 17, 40 bytes
    Memory Device
            Array Handle: 0x1000
            Error Information Handle: Not Provided
            Total Width: 72 bits
            Data Width: 64 bits
            Size: 8192 MB
            Form Factor: DIMM
            Set: 1
            Locator: A2
            Bank Locator: Not Specified
            Type: DDR4
            Type Detail: Synchronous Registered (Buffered)
            Speed: 2666 MHz
            Manufacturer: 802C8632802C
            Serial Number: 191DEABC
            Asset Tag: 0F1741A0
            Part Number: 9ASF1G72PZ-2G6D1
            Rank: 1
            Configured Clock Speed: 2400 MHz
            Minimum Voltage: 1.2 V
            Maximum Voltage: 1.2 V
            Configured Voltage: 1.2 V
    
    Handle 0x1102, DMI type 17, 40 bytes
    Memory Device
            Array Handle: 0x1000
            Error Information Handle: Not Provided
            Total Width: 72 bits
            Data Width: 64 bits
            Size: 8192 MB
            Form Factor: DIMM
            Set: 1
            Locator: A3
            Bank Locator: Not Specified
            Type: DDR4
            Type Detail: Synchronous Registered (Buffered)
            Speed: 2666 MHz
            Manufacturer: 802C8632802C
            Serial Number: 191DDFBD
            Asset Tag: 0F1741A0
            Part Number: 9ASF1G72PZ-2G6D1
            Rank: 1
            Configured Clock Speed: 2400 MHz
            Minimum Voltage: 1.2 V
            Maximum Voltage: 1.2 V
            Configured Voltage: 1.2 V
    
    Handle 0x1103, DMI type 17, 40 bytes
    Memory Device
            Array Handle: 0x1000
            Error Information Handle: Not Provided
            Total Width: 72 bits
            Data Width: 64 bits
            Size: 8192 MB
            Form Factor: DIMM
            Set: 1
            Locator: A4
            Bank Locator: Not Specified
            Type: DDR4
            Type Detail: Synchronous Registered (Buffered)
            Speed: 2666 MHz
            Manufacturer: 802C8632802C
            Serial Number: 191DE88E
            Asset Tag: 0F1741A0
            Part Number: 9ASF1G72PZ-2G6D1
            Rank: 1
            Configured Clock Speed: 2400 MHz
            Minimum Voltage: 1.2 V
            Maximum Voltage: 1.2 V
            Configured Voltage: 1.2 V
    
    Handle 0x1104, DMI type 17, 40 bytes
    Memory Device
            Array Handle: 0x1000
            Error Information Handle: Not Provided
            Total Width: 72 bits
            Data Width: 64 bits
            Size: 8192 MB
            Form Factor: DIMM
            Set: 1
            Locator: A5
            Bank Locator: Not Specified
            Type: DDR4
            Type Detail: Synchronous Registered (Buffered)
            Speed: 2666 MHz
            Manufacturer: 802C8632802C
            Serial Number: 191DE7E8
            Asset Tag: 0F1741A0
            Part Number: 9ASF1G72PZ-2G6D1
            Rank: 1
            Configured Clock Speed: 2400 MHz
            Minimum Voltage: 1.2 V
            Maximum Voltage: 1.2 V
            Configured Voltage: 1.2 V
    
    Handle 0x1105, DMI type 17, 40 bytes
    Memory Device
            Array Handle: 0x1000
            Error Information Handle: Not Provided
            Total Width: 72 bits
            Data Width: 64 bits
            Size: 8192 MB
            Form Factor: DIMM
            Set: 1
            Locator: A6
            Bank Locator: Not Specified
            Type: DDR4
            Type Detail: Synchronous Registered (Buffered)
            Speed: 2666 MHz
            Manufacturer: 802C8632802C
            Serial Number: 191DE7F5
            Asset Tag: 0F1741A0
            Part Number: 9ASF1G72PZ-2G6D1
            Rank: 1
            Configured Clock Speed: 2400 MHz
            Minimum Voltage: 1.2 V
            Maximum Voltage: 1.2 V
            Configured Voltage: 1.2 V
    
    Handle 0x1106, DMI type 17, 40 bytes
    Memory Device
            Array Handle: 0x1000
            Error Information Handle: Not Provided
            Total Width: 72 bits
            Data Width: 64 bits
            Size: 8192 MB
            Form Factor: DIMM
            Set: 1
            Locator: A7
            Bank Locator: Not Specified
            Type: DDR4
            Type Detail: Synchronous Registered (Buffered)
            Speed: 2666 MHz
            Manufacturer: 802C8632802C
            Serial Number: 191DE7B6
            Asset Tag: 0F1741A0
            Part Number: 9ASF1G72PZ-2G6D1
            Rank: 1
            Configured Clock Speed: 2400 MHz
            Minimum Voltage: 1.2 V
            Maximum Voltage: 1.2 V
            Configured Voltage: 1.2 V
    
    Handle 0x1107, DMI type 17, 40 bytes
    Memory Device
            Array Handle: 0x1000
            Error Information Handle: Not Provided
            Total Width: 72 bits
            Data Width: 64 bits
            Size: 8192 MB
            Form Factor: DIMM
            Set: 1
            Locator: A8
            Bank Locator: Not Specified
            Type: DDR4
            Type Detail: Synchronous Registered (Buffered)
            Speed: 2666 MHz
            Manufacturer: 802C8632802C
            Serial Number: 191DE7C1
            Asset Tag: 0F1741A0
            Part Number: 9ASF1G72PZ-2G6D1
            Rank: 1
            Configured Clock Speed: 2400 MHz
            Minimum Voltage: 1.2 V
            Maximum Voltage: 1.2 V
            Configured Voltage: 1.2 V
    
    Handle 0x1108, DMI type 17, 40 bytes
    Memory Device
            Array Handle: 0x1000
            Error Information Handle: Not Provided
            Total Width: Unknown
            Data Width: Unknown
            Size: No Module Installed
            Form Factor: Unknown
            Set: 2
            Locator: A9
            Bank Locator: Not Specified
            Type: Unknown
            Type Detail: None
            Speed: Unknown
            Manufacturer: Not Specified
            Serial Number: Not Specified
            Asset Tag: Not Specified
            Part Number: Not Specified
            Rank: Unknown
            Configured Clock Speed: Unknown
            Minimum Voltage: Unknown
            Maximum Voltage: Unknown
            Configured Voltage: Unknown
    
    Handle 0x1109, DMI type 17, 40 bytes
    Memory Device
            Array Handle: 0x1000
            Error Information Handle: Not Provided
            Total Width: Unknown
            Data Width: Unknown
            Size: No Module Installed
            Form Factor: Unknown
            Set: 2
            Locator: A10
            Bank Locator: Not Specified
            Type: Unknown
            Type Detail: None
            Speed: Unknown
            Manufacturer: Not Specified
            Serial Number: Not Specified
            Asset Tag: Not Specified
            Part Number: Not Specified
            Rank: Unknown
            Configured Clock Speed: Unknown
            Minimum Voltage: Unknown
            Maximum Voltage: Unknown
            Configured Voltage: Unknown
    
    Handle 0x110A, DMI type 17, 40 bytes
    Memory Device
            Array Handle: 0x1000
            Error Information Handle: Not Provided
            Total Width: Unknown
            Data Width: Unknown
            Size: No Module Installed
            Form Factor: Unknown
            Set: 2
            Locator: A11
            Bank Locator: Not Specified
            Type: Unknown
            Type Detail: None
            Speed: Unknown
            Manufacturer: Not Specified
            Serial Number: Not Specified
            Asset Tag: Not Specified
            Part Number: Not Specified
            Rank: Unknown
            Configured Clock Speed: Unknown
            Minimum Voltage: Unknown
            Maximum Voltage: Unknown
            Configured Voltage: Unknown
    
    Handle 0x110B, DMI type 17, 40 bytes
    Memory Device
            Array Handle: 0x1000
            Error Information Handle: Not Provided
            Total Width: Unknown
            Data Width: Unknown
            Size: No Module Installed
            Form Factor: Unknown
            Set: 2
            Locator: A12
            Bank Locator: Not Specified
            Type: Unknown
            Type Detail: None
            Speed: Unknown
            Manufacturer: Not Specified
            Serial Number: Not Specified
            Asset Tag: Not Specified
            Part Number: Not Specified
            Rank: Unknown
            Configured Clock Speed: Unknown
            Minimum Voltage: Unknown
            Maximum Voltage: Unknown
            Configured Voltage: Unknown
    
    Handle 0x110C, DMI type 17, 40 bytes
    Memory Device
            Array Handle: 0x1000
            Error Information Handle: Not Provided
            Total Width: Unknown
            Data Width: Unknown
            Size: No Module Installed
            Form Factor: Unknown
            Set: 2
            Locator: A13
            Bank Locator: Not Specified
            Type: Unknown
            Type Detail: None
            Speed: Unknown
            Manufacturer: Not Specified
            Serial Number: Not Specified
            Asset Tag: Not Specified
            Part Number: Not Specified
            Rank: Unknown
            Configured Clock Speed: Unknown
            Minimum Voltage: Unknown
            Maximum Voltage: Unknown
            Configured Voltage: Unknown
    
    Handle 0x110D, DMI type 17, 40 bytes
    Memory Device
            Array Handle: 0x1000
            Error Information Handle: Not Provided
            Total Width: Unknown
            Data Width: Unknown
            Size: No Module Installed
            Form Factor: Unknown
            Set: 2
            Locator: A14
            Bank Locator: Not Specified
            Type: Unknown
            Type Detail: None
            Speed: Unknown
            Manufacturer: Not Specified
            Serial Number: Not Specified
            Asset Tag: Not Specified
            Part Number: Not Specified
            Rank: Unknown
            Configured Clock Speed: Unknown
            Minimum Voltage: Unknown
            Maximum Voltage: Unknown
            Configured Voltage: Unknown
    
    Handle 0x110E, DMI type 17, 40 bytes
    Memory Device
            Array Handle: 0x1000
            Error Information Handle: Not Provided
            Total Width: Unknown
            Data Width: Unknown
            Size: No Module Installed
            Form Factor: Unknown
            Set: 2
            Locator: A15
            Bank Locator: Not Specified
            Type: Unknown
            Type Detail: None
            Speed: Unknown
            Manufacturer: Not Specified
            Serial Number: Not Specified
            Asset Tag: Not Specified
            Part Number: Not Specified
            Rank: Unknown
            Configured Clock Speed: Unknown
            Minimum Voltage: Unknown
            Maximum Voltage: Unknown
            Configured Voltage: Unknown
    
    Handle 0x110F, DMI type 17, 40 bytes
    Memory Device
            Array Handle: 0x1000
            Error Information Handle: Not Provided
            Total Width: Unknown
            Data Width: Unknown
            Size: No Module Installed
            Form Factor: Unknown
            Set: 2
            Locator: A16
            Bank Locator: Not Specified
            Type: Unknown
            Type Detail: None
            Speed: Unknown
            Manufacturer: Not Specified
            Serial Number: Not Specified
            Asset Tag: Not Specified
            Part Number: Not Specified
            Rank: Unknown
            Configured Clock Speed: Unknown
            Minimum Voltage: Unknown
            Maximum Voltage: Unknown
            Configured Voltage: Unknown
    

    SSD part numbers for 120GB and 480GB SSD. The 120GB SSD appears to be Intel DC S3520 Series (Downieville) M.2 SSD and 480GB SSD appears to be Intel DC S3520 Series (Downieville) 2.5" SSD.
    Code (Text):
    lshw -short -C disk
    H/W path              Device     Class          Description
    ===========================================================
    /0/12e/3.1/0/0        /dev/sdc   disk           120GB SSDSCKJB120G7R
    /0/12e/3.1/0/1        /dev/sdd   disk           120GB SSDSCKJB120G7R
    /0/135/1.1/0/0.0.0    /dev/sda   disk           480GB SSDSC2BB480G7R
    /0/135/1.1/0/0.1.0    /dev/sdb   disk           480GB SSDSC2BB480G7R
    

    Except there is no official Intel 120GB DC S3520 part as only a 150GB version exists so it could be a custom SSD manufactured by Dell. The 150GB Intel DCS3520 specs match with above shown disk speeds of around 170MB/s

    Intel® Product Specification Comparison

    epyc-ssd-compare.png

    Disk rated speeds aren't going to be able to fully utilised the 2x10Gbps bonded 20Gbps network that this AMD EPYC 7401P server sports unfortunately :(
     
    Last edited: Feb 28, 2018
  8. eva2000

    eva2000 Administrator Staff Member

    35,481
    7,828
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,067
    Local Time:
    6:41 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x

    Packet.net AMD EPYC 7401P c2.medium + Centmin Mod 123.09beta01 Install - Feb 28, 2018



    I've updated Centmin Mod 123.09beta01 code to now optimise Nginx and PHP-FPM compilation routines for AMD EPYC cpus if they are detected just like what is done when Intel cpus are detected. This will increase the initial install times for Nginx and PHP-FPM but in theory result in better performing compiled Nginx and PHP-FPM binaries.

    I have also updated CentOS 7.4 to default to 4.15.6-1 Elrepo Linux Kernel first before Centmin Mod 123.09beta01 install as well done some manual tuning to the system to see what the install time is like now.
    Code (Text):
    uname -r
    4.15.6-1.el7.elrepo.x86_64
    

    Centmin Mod 123.09beta01 install command
    Code (Text):
    yum -y update; curl -O https://centminmod.com/betainstaller.sh && chmod 0700 betainstaller.sh && bash betainstaller.sh
    

    Centmin Mod 123.09beta01 install times
    Code (Text):
    ---------------------------------------------------------------------------
    Total Curl Installer YUM or DNF Time: 128.5828 seconds
    Total YUM Time: 38.440464465 seconds
    Total YUM or DNF + Source Download Time: 59.0134
    Total Nginx First Time Install Time: 176.0012
    Total PHP First Time Install Time: 123.5181
    Download Zip From Github Time: 1.3709
    Total Time Other eg. source compiles: 341.5887
    Total Centmin Mod Install Time: 700.1213
    ---------------------------------------------------------------------------
    Total Install Time (curl yum + cm install + zip download): 830.0750 seconds
    ---------------------------------------------------------------------------
    

    Resulting Nginx compiled configuration

    AMD EPYC 7401P CPU Frequency Scaling Turbo Boost



    Interestingly, looking at turbostat output the AMD EPYC 7401P cpu frequency only fluctuated between 2344Mhz to 2550Mhz when under full 48 cpu thread cpu load. The AMD EPYC 7401P is rated for 2.0Ghz based, 2.8Ghz all cpus turbo boost and single cpu turbo boost of 3.0Ghz. So cpu frequency scaling not working properly ? Will have to investigate.

    top stats during load
    Code (Text):
    top -bn1 | head -n 52
    top - 21:46:33 up 57 min,  3 users,  load average: 48.29, 24.43, 11.38
    Tasks: 665 total,  47 running, 363 sleeping,   0 stopped,   0 zombie
    %Cpu0  :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu1  :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu2  :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu3  :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu4  : 95.8 us,  4.2 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu5  :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu6  : 20.0 us, 20.0 sy,  0.0 ni, 60.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu7  :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu8  :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu9  :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu10 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu11 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu12 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu13 : 96.0 us,  4.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu14 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu15 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu16 : 64.0 us, 16.0 sy,  0.0 ni, 20.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu17 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu18 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu19 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu20 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu21 : 96.0 us,  4.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu22 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu23 : 96.0 us,  4.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu24 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu25 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu26 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu27 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu28 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu29 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu30 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu31 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu32 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu33 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu34 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu35 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu36 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu37 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu38 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu39 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu40 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu41 : 95.8 us,  4.2 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu42 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu43 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu44 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu45 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu46 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    %Cpu47 :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    KiB Mem : 65620288 total, 48240376 free,  9984520 used,  7395392 buff/cache
    KiB Swap:  1996796 total,  1996796 free,        0 used. 54709792 avail Mem
    

    turbostat
    Code (Text):
    Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IRQ
    -       -       2321    100.00  2321    1996    248478
    0       0       2317    100.00  2317    1996    5121
    0       24      2317    100.00  2317    1996    5124
    1       8       2322    100.00  2322    1996    5146
    1       32      2322    100.00  2322    1996    5771
    2       16      2321    100.00  2321    1996    5066
    2       40      2321    100.00  2321    1996    5122
    4       4       2322    100.00  2322    1996    5020
    4       28      2322    100.00  2322    1996    5430
    5       12      2321    100.00  2321    1996    5083
    5       36      2321    100.00  2321    1996    5288
    6       20      2322    100.00  2322    1996    5043
    6       44      2322    100.00  2322    1996    5029
    8       1       2322    100.00  2322    1996    5118
    8       25      2322    100.00  2322    1996    5200
    9       9       2322    100.00  2322    1996    5547
    9       33      2322    100.00  2322    1996    5079
    10      17      2322    100.00  2322    1996    5257
    10      41      2322    100.00  2322    1996    5122
    12      5       2311    100.00  2311    1996    5049
    12      29      2311    100.00  2311    1996    5188
    13      13      2322    100.00  2322    1996    5464
    13      37      2322    100.00  2322    1996    5032
    14      21      2322    100.00  2322    1996    5055
    14      45      2322    100.00  2322    1996    5020
    16      2       2321    100.00  2321    1996    5061
    16      26      2321    100.00  2321    1996    5041
    17      10      2322    100.00  2322    1996    5641
    17      34      2322    100.00  2322    1996    5114
    18      18      2317    100.00  2317    1996    5188
    18      42      2317    100.00  2317    1996    5170
    20      6       2321    100.00  2321    1996    5094
    20      30      2321    100.00  2321    1996    5106
    21      14      2322    100.00  2322    1996    5078
    21      38      2322    100.00  2322    1996    5763
    22      22      2322    100.00  2322    1996    5244
    22      46      2322    100.00  2322    1996    5043
    24      3       2322    100.00  2322    1996    5167
    24      27      2322    100.00  2322    1996    5027
    25      11      2322    100.00  2322    1996    5168
    25      35      2322    100.00  2322    1996    5076
    26      19      2322    100.00  2322    1996    5051
    26      43      2322    100.00  2322    1996    5065
    28      7       2322    100.00  2322    1996    5079
    28      31      2322    100.00  2322    1996    5254
    29      15      2322    100.00  2322    1996    5066
    29      39      2322    100.00  2322    1996    5051
    30      23      2322    100.00  2322    1996    5045
    30      47      2322    100.00  2322    1996    5512
    

    Code (Text):
    Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IRQ
    -       -       2531    100.00  2531    1996    69073
    0       0       2532    100.00  2532    1996    1809
    0       24      2532    100.00  2532    1996    1467
    1       8       2532    100.00  2532    1996    1504
    1       32      2532    100.00  2532    1996    1501
    2       16      2531    100.00  2531    1996    1469
    2       40      2531    100.00  2531    1996    1386
    4       4       2532    100.00  2532    1996    1438
    4       28      2532    100.00  2532    1996    1276
    5       12      2531    100.00  2531    1996    1304
    5       36      2531    100.00  2531    1996    1375
    6       20      2532    100.00  2532    1996    1299
    6       44      2532    100.00  2532    1996    1301
    8       1       2532    100.00  2532    1996    1476
    8       25      2532    100.00  2532    1996    1350
    9       9       2532    100.00  2532    1996    1390
    9       33      2532    100.00  2532    1996    1201
    10      17      2532    100.00  2532    1996    1442
    10      41      2532    100.00  2532    1996    1401
    12      5       2528    100.00  2528    1996    1421
    12      29      2528    100.00  2528    1996    1290
    13      13      2532    100.00  2532    1996    1348
    13      37      2532    100.00  2532    1996    1339
    14      21      2532    100.00  2532    1996    1380
    14      45      2532    100.00  2532    1996    1377
    16      2       2531    100.00  2531    1996    1551
    16      26      2531    100.00  2531    1996    1405
    17      10      2532    100.00  2532    1996    1318
    17      34      2532    100.00  2532    1996    1345
    18      18      2530    100.00  2530    1996    1860
    18      42      2530    100.00  2530    1996    1413
    20      6       2531    100.00  2531    1996    3458
    20      30      2531    100.00  2531    1996    1371
    21      14      2532    100.00  2532    1996    1316
    21      38      2532    100.00  2532    1996    1481
    22      22      2532    100.00  2532    1996    1399
    22      46      2532    100.00  2532    1996    1373
    24      3       2532    100.00  2532    1996    1518
    24      27      2532    100.00  2532    1996    1244
    25      11      2532    100.00  2532    1996    1614
    25      35      2532    100.00  2532    1996    1172
    26      19      2532    100.00  2532    1996    1349
    26      43      2532    100.00  2532    1996    1307
    28      7       2532    100.00  2532    1996    1302
    28      31      2532    100.00  2532    1996    1256
    29      15      2532    100.00  2532    1996    1797
    29      39      2532    100.00  2532    1996    1295
    30      23      2532    100.00  2532    1996    1202
    30      47      2532    100.00  2532    1996    1183
    


    Reinstalled CentOS 7.4 with default 3.10 kernel and I see at 24 cpu thread load that cpu jumps to 2700+Mhz and 48 cpu thread drops to 2560Mhz

    Code (Text):
            Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IRQ
            -       -       1363    50.05   2724    1996    128036
            0       0       2725    100.00  2725    1996    5436
            0       24      0       0.00    2702    1996    12
            1       8       2724    100.00  2724    1996    5039
            1       32      0       0.00    2694    1996    7
            2       16      2723    100.00  2723    1996    5038
            2       40      0       0.01    2699    1996    12
            4       4       2724    100.00  2724    1996    5036
            4       28      9       0.33    2709    1996    1125
            5       12      0       0.00    2682    1996    10
            5       36      2725    100.00  2725    1996    5442
            6       20      5       0.17    2730    1996    20
            6       44      2724    100.00  2724    1996    5041
            8       1       2725    100.00  2725    1996    5036
            8       25      0       0.00    2720    1996    6
            9       9       0       0.00    2707    1996    6
            9       33      2725    100.00  2725    1996    5034
            10      17      2725    100.00  2725    1996    5036
            10      41      0       0.00    2713    1996    9
            12      5       41      1.54    2689    1996    4953
            12      29      2724    100.00  2724    1996    5036
            13      13      0       0.00    2689    1996    6
            13      37      2724    100.00  2724    1996    5036
            14      21      0       0.00    2711    1996    6
            14      45      2725    100.00  2725    1996    5036
            16      2       2724    100.00  2724    1996    5036
            16      26      0       0.00    2702    1996    6
            17      10      2725    100.00  2725    1996    5036
            17      34      0       0.00    2722    1996    6
            18      18      2722    100.00  2722    1996    5036
            18      42      0       0.00    2665    1996    8
            20      6       0       0.00    2721    1996    6
            20      30      2725    100.00  2725    1996    5036
            21      14      1       0.03    2705    1996    91
            21      38      2725    100.00  2725    1996    5036
            22      22      0       0.00    2723    1996    6
            22      46      2724    100.00  2724    1996    5036
            24      3       2724    100.00  2724    1996    5036
            24      27      0       0.00    2685    1996    6
            25      11      2725    100.00  2725    1996    5036
            25      35      0       0.00    2719    1996    6
            26      19      2724    100.00  2724    1996    5036
            26      43      0       0.00    2694    1996    6
            28      7       2725    100.00  2725    1996    5036
            28      31      0       0.00    2706    1996    6
            29      15      2725    100.00  2725    1996    5036
            29      39      0       0.00    2726    1996    6
            30      23      2725    100.00  2725    1996    5036
            30      47      6       0.21    2724    1996    33
    

    Code (Text):
    turbostat
            Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IRQ
            -       -       2561    99.96   2562    1996    245435
            0       0       2560    99.99   2560    1994    5298
            0       24      2560    99.98   2560    1994    5089
            1       8       2557    99.93   2559    1994    5192
            1       32      2562    99.94   2564    1998    5170
            2       16      2563    99.99   2563    1998    5088
            2       40      2563    100.00  2563    1998    5090
            4       4       2563    99.97   2564    1998    5090
            4       28      2563    99.96   2564    1998    5091
            5       12      2563    99.93   2565    1998    5091
            5       36      2564    99.96   2565    1998    5405
            6       20      2560    99.96   2561    1995    5092
            6       44      2560    99.96   2561    1995    5096
            8       1       2561    99.97   2562    1995    5089
            8       25      2562    100.00  2562    1995    5069
            9       9       2560    99.96   2561    1995    5086
            9       33      2560    99.93   2561    1995    5090
            10      17      2560    99.95   2562    1995    5088
            10      41      2560    99.95   2562    1995    5090
            12      5       2558    99.94   2560    1995    5119
            12      29      2558    99.93   2560    1995    5129
            13      13      2559    99.96   2560    1995    5089
            13      37      2560    99.99   2560    1995    5088
            14      21      2560    99.94   2562    1996    5167
            14      45      2561    99.98   2562    1996    5208
            16      2       2560    99.94   2562    1996    5091
            16      26      2561    99.94   2562    1996    5089
            17      10      2561    99.97   2562    1996    5090
            17      34      2561    99.95   2562    1996    5088
            18      18      2558    99.97   2558    1996    5089
            18      42      2558    99.98   2558    1996    5091
            20      6       2561    99.95   2562    1996    5089
            20      30      2561    99.94   2562    1996    5085
            21      14      2561    99.93   2562    1996    5128
            21      38      2562    99.98   2562    1996    5160
            22      22      2561    99.97   2561    1996    5089
            22      46      2560    99.93   2561    1996    5087
            24      3       2560    99.98   2561    1996    5152
            24      27      2559    99.93   2561    1996    5121
            25      11      2561    99.97   2562    1996    5087
            25      35      2561    99.95   2562    1996    5089
            26      19      2560    99.98   2560    1996    5088
            26      43      2559    99.95   2560    1996    5089
            28      7       2563    99.99   2563    1996    5087
            28      31      2563    99.99   2563    1996    5089
            29      15      2563    99.99   2563    1996    5088
            29      39      2561    99.93   2563    1996    5088
            30      23      2563    100.00  2563    1996    5087
            30      47      2563    100.00  2563    1996    5090
    

    and 1 cpu thread load hits the max 3.0Ghz turbo speed
    Code (Text):
            Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IRQ
            -       -       65      2.16    2994    1996    7555
            0       0       3       0.11    2893    1996    516
            0       24      0       0.00    2717    1996    7
            1       8       0       0.01    2653    1996    16
            1       32      0       0.00    2690    1996    7
            2       16      0       0.01    2802    1996    13
            2       40      0       0.00    2567    1996    12
            4       4       91      3.03    2993    1996    269
            4       28      0       0.00    2630    1996    7
            5       12      0       0.00    2648    1996    11
            5       36      2       0.07    2891    1996    506
            6       20      0       0.00    2602    1996    7
            6       44      3       0.11    2745    1996    812
            8       1       0       0.00    2746    1996    7
            8       25      0       0.00    2722    1996    7
            9       9       0       0.00    2724    1996    7
            9       33      0       0.01    2703    1996    16
            10      17      0       0.00    2712    1996    7
            10      41      0       0.00    2714    1996    7
            12      5       0       0.00    2540    1996    8
            12      29      0       0.00    2594    1996    7
            13      13      0       0.00    2523    1996    7
            13      37      0       0.00    2574    1996    7
            14      21      0       0.00    2528    1996    7
            14      45      0       0.00    2590    1996    7
            16      2       0       0.00    2752    1996    5
            16      26      0       0.00    2707    1996    7
            17      10      0       0.00    2687    1996    7
            17      34      0       0.00    2716    1996    7
            18      18      0       0.00    2667    1996    7
            18      42      4       0.14    2959    1996    21
            20      6       0       0.00    2579    1996    7
            20      30      0       0.00    2648    1996    7
            21      14      1       0.02    2590    1996    66
            21      38      0       0.01    2667    1996    31
            22      22      0       0.00    2576    1996    7
            22      46      0       0.00    2643    1996    7
            24      3       0       0.00    2782    1996    5
            24      27      0       0.00    2716    1996    7
            25      11      0       0.00    2783    1996    8
            25      35      0       0.00    2684    1996    8
            26      19      0       0.00    2586    1996    7
            26      43      0       0.00    2651    1996    7
            28      7       0       0.00    2593    1996    7
            28      31      0       0.00    2653    1996    7
            29      15      0       0.00    2886    1996    7
            29      39      2994    100.00  2994    1996    5012
            30      23      0       0.00    2581    1996    7
            30      47      3       0.11    2970    1996    17
    

    So looks like it's something to do with either my 4.15 Kernel update or additional tweaks I did manually.

    Looks like it was due to my own tweaking of kernel boot command paramters - the culprit was setting idle=poll ! But that only gets me to 2.55-2.65Ghz max frequency and not 2.80Ghz rated max for AMD 7401P.

    Removing that idle=poll and reboot server shows turbostat

    at idle
    Code (Text):
    Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IRQ     C1      C2      C1%     C2%
    -       -       0       0.01    2797    1996    978     374     588     1.65    98.34
    0       0       2       0.06    2763    1996    182     10      167     0.11    99.84
    0       24      0       0.00    2822    1996    12      7       4       0.09    99.91
    1       8       1       0.02    2595    1996    110     11      96      3.15    96.83
    1       32      0       0.00    2735    1996    12      7       4       0.08    99.91
    2       16      0       0.00    2637    1996    14      10      5       0.10    99.89
    2       40      0       0.00    2722    1996    12      8       4       0.10    99.89
    4       4       0       0.00    2686    1996    14      8       4       0.12    99.88
    4       28      1       0.02    2906    1996    110     20      86      24.98   75.00
    5       12      0       0.00    2732    1996    12      6       5       0.08    99.92
    5       36      0       0.01    2758    1996    69      18      60      47.45   52.54
    6       20      0       0.00    2659    1996    17      13      4       0.11    99.88
    6       44      0       0.00    2743    1996    12      5       6       0.07    99.92
    8       1       0       0.00    2775    1996    12      8       4       0.11    99.89
    8       25      0       0.00    2874    1996    11      7       3       0.09    99.91
    9       9       0       0.00    2770    1996    11      7       4       0.09    99.91
    9       33      0       0.00    2880    1996    11      7       3       0.09    99.91
    10      17      0       0.00    2672    1996    13      5       7       0.07    99.93
    10      41      0       0.00    2907    1996    10      5       4       0.07    99.93
    12      5       0       0.00    2634    1996    12      9       3       0.12    99.88
    12      29      0       0.00    2662    1996    11      7       3       0.08    99.92
    13      13      0       0.00    2581    1996    11      8       4       0.10    99.90
    13      37      0       0.00    2737    1996    9       5       3       0.08    99.92
    14      21      0       0.00    2660    1996    12      8       4       0.09    99.90
    14      45      0       0.00    2743    1996    9       5       3       0.08    99.92
    16      2       0       0.01    2794    1996    15      10      5       0.11    99.89
    16      26      0       0.00    2790    1996    10      6       3       0.07    99.93
    17      10      0       0.00    2728    1996    11      5       4       0.07    99.93
    17      34      0       0.00    2851    1996    10      6       3       0.07    99.93
    18      18      0       0.00    2658    1996    12      5       5       0.07    99.93
    18      42      0       0.00    2899    1996    9       4       4       0.05    99.95
    20      6       0       0.00    2636    1996    12      5       4       0.08    99.92
    20      30      0       0.00    2728    1996    10      4       4       0.04    99.96
    21      14      0       0.01    2733    1996    15      6       9       0.08    99.92
    21      38      0       0.00    2794    1996    8       4       3       0.06    99.94
    22      22      0       0.00    2621    1996    10      7       3       0.07    99.92
    22      46      0       0.00    2735    1996    8       5       2       0.08    99.92
    24      3       0       0.00    2800    1996    10      8       2       0.09    99.91
    24      27      0       0.00    2714    1996    14      5       8       0.07    99.92
    25      11      0       0.00    2760    1996    9       6       3       0.07    99.93
    25      35      0       0.00    2841    1996    9       6       2       0.07    99.93
    26      19      0       0.00    2780    1996    11      6       4       0.07    99.93
    26      43      0       0.00    2860    1996    8       4       3       0.05    99.95
    28      7       0       0.00    2659    1996    10      7       3       0.06    99.94
    28      31      0       0.01    2729    1996    11      4       5       0.03    99.97
    29      15      0       0.00    2723    1996    13      10      4       0.14    99.86
    29      39      0       0.00    2810    1996    15      29      2       0.14    99.86
    30      23      0       0.01    2741    1996    14      15      4       0.13    99.86
    30      47      2       0.08    2964    1996    16      3       9       0.05    99.87
    

    1 cpu load @ 2994Mhz
    Code (Text):
    Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IRQ     C1      C2      C1%     C2%
    -       -       63      2.09    2993    1996    6182    454     681     2.26    95.64
    0       0       2       0.07    2675    1996    191     12      173     0.16    99.77
    0       24      0       0.00    2770    1996    18      9       8       0.12    99.87
    1       8       1       0.02    2592    1996    113     23      91      9.89    90.09
    1       32      0       0.00    2831    1996    16      9       7       0.12    99.88
    2       16      0       0.00    2552    1996    17      10      7       0.14    99.86
    2       40      0       0.00    2698    1996    16      10      5       0.14    99.86
    4       4       0       0.00    2662    1996    16      11      5       0.15    99.84
    4       28      1       0.02    2853    1996    123     26      85      47.88   52.10
    5       12      0       0.00    2764    1996    17      10      6       0.13    99.87
    5       36      0       0.01    2719    1996    74      19      64      46.07   53.92
    6       20      0       0.00    2596    1996    17      11      6       0.15    99.85
    6       44      0       0.00    2707    1996    16      8       7       0.11    99.89
    8       1       200     6.69    2994    1996    354     9       7       0.13    93.19
    8       25      4       0.12    2991    1996    25      10      7       0.13    99.75
    9       9       0       0.01    2725    1996    23      10      12      0.14    99.85
    9       33      0       0.00    2837    1996    15      8       6       0.10    99.90
    10      17      0       0.00    2662    1996    17      7       9       0.08    99.92
    10      41      0       0.00    2822    1996    15      7       7       0.10    99.90
    12      5       0       0.00    2658    1996    16      12      4       0.15    99.85
    12      29      0       0.00    2694    1996    14      8       5       0.12    99.89
    13      13      0       0.00    2654    1996    15      13      4       0.15    99.85
    13      37      0       0.00    2674    1996    14      7       6       0.09    99.91
    14      21      0       0.00    2544    1996    15      6       8       0.09    99.91
    14      45      0       0.00    2726    1996    14      6       7       0.07    99.93
    16      2       2789    93.17   2994    1996    4676    0       2       0.00    6.83
    16      26      0       0.00    2958    1996    15      7       6       0.10    99.89
    17      10      0       0.00    2701    1996    15      11      5       0.14    99.86
    17      34      0       0.00    2831    1996    13      7       5       0.10    99.90
    18      18      0       0.00    2648    1996    15      7       7       0.09    99.90
    18      42      0       0.00    2789    1996    13      7       5       0.09    99.90
    20      6       0       0.00    2626    1996    15      8       6       0.11    99.89
    20      30      0       0.00    2742    1996    13      7       5       0.09    99.91
    21      14      0       0.01    2661    1996    19      6       11      0.08    99.91
    21      38      0       0.00    2696    1996    13      7       5       0.09    99.91
    22      22      0       0.00    2614    1996    14      8       6       0.10    99.89
    22      46      0       0.00    2697    1996    13      7       5       0.08    99.91
    24      3       3       0.09    2958    1996    27      14      8       0.21    99.70
    24      27      0       0.00    2666    1996    17      6       9       0.10    99.90
    25      11      0       0.00    2744    1996    14      9       5       0.12    99.88
    25      35      0       0.00    2840    1996    12      6       5       0.08    99.92
    26      19      0       0.00    2714    1996    14      7       6       0.09    99.90
    26      43      0       0.00    2830    1996    12      6       5       0.07    99.92
    28      7       0       0.00    2667    1996    13      8       4       0.11    99.89
    28      31      0       0.00    2733    1996    12      6       5       0.07    99.93
    29      15      0       0.01    2713    1996    13      22      4       0.10    99.89
    29      39      0       0.01    2780    1996    12      9       3       0.10    99.89
    30      23      0       0.01    2684    1996    14      15      5       0.10    99.89
    30      47      3       0.10    2914    1996    17      3       8       0.03    99.87
    

    all 24 cpu threads load [email protected] 2.63Ghz
    Code (Text):
    Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IRQ     C1      C2      C1%     C2%
    -       -       1327    50.48   2629    1996    126226  2260    2936    8.60    40.92
    0       0       34      1.28    2628    1996    4229    1686    2506    42.57   56.21
    0       24      2629    99.99   2629    1996    5052    10      0       0.01    0.00
    1       8       0       0.02    2624    1996    104     17      88      14.97   85.02
    1       32      2629    99.99   2629    1996    5053    10      0       0.01    0.00
    2       16      306     11.66   2628    1996    16      6       6       0.09    88.25
    2       40      2629    99.99   2629    1996    5051    14      0       0.01    0.00
    4       4       2629    99.99   2629    1996    5052    15      0       0.01    0.00
    4       28      1       0.03    2627    1996    150     38      102     58.41   41.56
    5       12      0       0.01    2629    1996    35      8       24      1.22    98.77
    5       36      2629    99.98   2629    1996    5053    10      0       0.02    0.00
    6       20      2629    99.99   2629    1996    5054    10      0       0.01    0.00
    6       44      0       0.01    2627    1996    72      20      63      60.97   39.01
    8       1       2629    99.99   2629    1996    5053    14      0       0.01    0.00
    8       25      0       0.00    2622    1996    14      5       6       0.06    99.94
    9       9       2629    99.98   2629    1996    5053    10      0       0.02    0.00
    9       33      0       0.00    2632    1996    17      8       6       77.33   22.67
    10      17      2629    99.99   2629    1996    5054    10      0       0.01    0.00
    10      41      0       0.00    2634    1996    16      6       7       0.08    99.92
    12      5       2629    99.99   2629    1996    5053    10      0       0.01    0.00
    12      29      0       0.00    2630    1996    15      6       6       0.08    99.92
    13      13      2629    99.99   2629    1996    5054    10      0       0.01    0.00
    13      37      0       0.00    2629    1996    14      5       6       0.06    99.94
    14      21      2629    99.99   2629    1996    5054    10      0       0.01    0.00
    14      45      0       0.00    2622    1996    17      7       7       77.78   22.21
    16      2       2629    99.98   2629    1996    5051    10      0       0.02    0.00
    16      26      0       0.00    2622    1996    16      8       5       0.12    99.88
    17      10      268     10.19   2629    1996    21      9       7       0.13    89.68
    17      34      2629    99.99   2629    1996    5054    15      0       0.01    0.00
    18      18      2629    99.99   2629    1996    5053    14      0       0.01    0.00
    18      42      0       0.00    2619    1996    14      5       6       0.06    99.94
    20      6       2629    99.99   2629    1996    5053    14      0       0.01    0.00
    20      30      0       0.01    2624    1996    22      6       12      0.10    99.90
    21      14      2629    99.99   2629    1996    5054    14      0       0.01    0.00
    21      38      0       0.00    2621    1996    16      6       7       0.08    99.92
    22      22      2629    99.99   2629    1996    5051    10      0       0.01    0.00
    22      46      0       0.00    2617    1996    15      6       6       0.08    99.92
    24      3       2629    99.99   2629    1996    5053    10      0       0.01    0.00
    24      27      0       0.00    2622    1996    18      6       9       0.10    99.90
    25      11      3       0.13    2620    1996    38      14      15      0.23    99.64
    25      35      2629    99.99   2629    1996    5054    14      0       0.01    0.00
    26      19      2629    99.99   2629    1996    5054    10      0       0.01    0.00
    26      43      0       0.00    2631    1996    20      8       9       0.12    99.88
    28      7       2629    99.99   2629    1996    5053    10      0       0.01    0.00
    28      31      0       0.01    2624    1996    24      90      3       0.21    99.78
    29      15      2629    99.99   2629    1996    5054    10      0       0.01    0.00
    29      39      0       0.02    2624    1996    39      19      17      77.45   22.53
    30      23      2629    100.00  2629    1996    5035    0       0       0.00    0.00
    30      47      3       0.12    2623    1996    29      7       13      0.11    99.78
    

    all 48 cpu threads load [email protected]
    Code (Text):
    Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IRQ     C1      C2      C1%     C2%
    -       -       2536    99.98   2537    1996    242691  491     0       0.02    0.00
    0       0       2537    100.00  2537    1996    5109    0       0       0.00    0.00
    0       24      2536    99.97   2537    1996    5050    10      0       0.03    0.00
    1       8       2536    99.98   2537    1996    5064    13      0       0.02    0.00
    1       32      2536    99.97   2537    1996    5049    10      0       0.03    0.00
    2       16      2536    99.98   2537    1996    5055    11      0       0.02    0.00
    2       40      2536    99.97   2537    1996    5053    10      0       0.02    0.00
    4       4       2536    99.97   2537    1996    5054    10      0       0.03    0.00
    4       28      2536    99.98   2537    1996    5195    11      0       0.02    0.00
    5       12      2536    99.98   2537    1996    5051    10      0       0.02    0.00
    5       36      2536    99.97   2537    1996    5049    10      0       0.03    0.00
    6       20      2536    99.98   2537    1996    5051    14      0       0.02    0.00
    6       44      2536    99.97   2537    1996    5052    10      0       0.03    0.00
    8       1       2536    99.98   2537    1996    5052    15      0       0.02    0.00
    8       25      2536    99.98   2537    1996    5052    10      0       0.02    0.00
    9       9       2536    99.98   2537    1996    5052    10      0       0.02    0.00
    9       33      2536    99.98   2537    1996    5053    10      0       0.02    0.00
    10      17      2536    99.97   2537    1996    5049    10      0       0.03    0.00
    10      41      2536    99.97   2537    1996    5052    10      0       0.03    0.00
    12      5       2536    99.97   2537    1996    5053    10      0       0.03    0.00
    12      29      2536    99.98   2537    1996    5052    10      0       0.02    0.00
    13      13      2536    99.98   2537    1996    5051    10      0       0.02    0.00
    13      37      2536    99.97   2537    1996    5052    10      0       0.03    0.00
    14      21      2536    99.98   2537    1996    5051    10      0       0.02    0.00
    14      45      2536    99.97   2537    1996    5052    10      0       0.03    0.00
    16      2       2536    99.97   2537    1996    5053    10      0       0.03    0.00
    16      26      2536    99.98   2537    1996    5051    11      0       0.02    0.00
    17      10      2536    99.97   2537    1996    5051    10      0       0.03    0.00
    17      34      2536    99.97   2537    1996    5053    10      0       0.03    0.00
    18      18      2536    99.98   2537    1996    5051    10      0       0.02    0.00
    18      42      2536    99.97   2537    1996    5052    11      0       0.03    0.00
    20      6       2536    99.98   2537    1996    5053    10      0       0.02    0.00
    20      30      2536    99.98   2537    1996    5051    10      0       0.02    0.00
    21      14      2536    99.98   2537    1996    5051    10      0       0.02    0.00
    21      38      2536    99.97   2537    1996    5050    10      0       0.03    0.00
    22      22      2536    99.98   2537    1996    5054    10      0       0.02    0.00
    22      46      2536    99.98   2537    1996    5050    10      0       0.02    0.00
    24      3       2536    99.97   2537    1996    5050    10      0       0.03    0.00
    24      27      2536    99.97   2537    1996    5050    10      0       0.03    0.00
    25      11      2536    99.97   2537    1996    5052    10      0       0.03    0.00
    25      35      2536    99.98   2537    1996    5053    10      0       0.02    0.00
    26      19      2536    99.98   2537    1996    5051    10      0       0.02    0.00
    26      43      2536    99.98   2537    1996    5052    10      0       0.02    0.00
    28      7       2536    99.97   2537    1996    5053    11      0       0.03    0.00
    28      31      2536    99.98   2537    1996    5051    10      0       0.02    0.00
    29      15      2536    99.98   2537    1996    5051    10      0       0.02    0.00
    29      39      2536    99.98   2537    1996    5052    10      0       0.02    0.00
    30      23      2536    99.97   2537    1996    5052    11      0       0.03    0.00
    30      47      2536    99.98   2537    1996    5051    13      0       0.02    0.00
    


    So again it was my own manual tweaks I needed to remove for another kernel command boot parameter skew_tick=1 LOL

    24 cpu thread load now gives 2760Mhz clock speed
    Code (Text):
    Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IRQ     C1      C2      C1%     C2%
    -       -       1007    36.46   2761    2000    95200   5272    2920    26.23   37.44
    0       0       1978    71.50   2767    2004    3714    81      24      18.70   9.92
    0       24      0       0.01    2751    2004    92      83      17      20.88   79.52
    1       8       577     20.81   2774    2004    405     69      19      28.89   50.61
    1       32      1982    71.63   2766    2004    3712    83      26      21.03   7.45
    2       16      1987    71.84   2765    2004    3709    86      1       28.27   0.00
    2       40      1       0.03    2722    2004    89      44      45      15.83   84.51
    4       4       1987    71.85   2766    2003    3717    65      33      18.54   9.71
    4       28      1       0.05    2759    2003    192     80      100     72.80   27.50
    5       12      1986    71.82   2765    2003    3711    82      4       28.27   0.00
    5       36      1       0.04    2766    2003    271     70      205     10.00   90.29
    6       20      1985    71.81   2765    2003    3718    89      3       28.27   0.00
    6       44      1       0.03    2753    2003    205     97      105     45.95   54.34
    8       1       1985    71.80   2765    2003    3708    86      5       28.28   0.00
    8       25      0       0.01    2743    2002    106     86      16      20.98   79.30
    9       9       0       0.01    2739    2002    86      74      11      20.88   79.42
    9       33      1985    71.82   2764    2002    3726    88      15      20.94   7.33
    10      17      1984    71.78   2764    2002    3709    76      11      28.29   0.02
    10      41      0       0.02    2772    2002    117     86      23      21.08   79.18
    12      5       24      0.86    2738    2002    3381    1391    1969    55.93   43.51
    12      29      1983    71.78   2762    2001    3709    88      2       28.28   0.00
    13      13      1971    71.38   2762    2001    3691    84      1       28.68   0.00
    13      37      0       0.01    2765    2001    93      82      13      20.97   79.26
    14      21      285     10.19   2796    2000    602     72      7       41.08   48.93
    14      45      1981    71.75   2762    2000    3703    78      6       28.30   0.01
    16      2       1934    70.09   2760    2000    3712    128     50      29.02   0.95
    16      26      0       0.01    2751    2000    91      78      13      20.93   79.25
    17      10      1709    62.01   2755    2000    3248    99      12      28.57   9.49
    17      34      0       0.01    2694    2000    99      83      16      21.03   79.14
    18      18      1979    71.75   2758    1999    3711    90      1       28.30   0.00
    18      42      0       0.01    2720    1999    98      82      16      21.01   79.14
    20      6       1871    67.87   2757    1999    3518    82      2       32.17   0.00
    20      30      0       0.01    2776    1999    98      80      17      21.00   79.14
    21      14      1979    71.73   2759    1999    3709    83      2       28.30   0.00
    21      38      0       0.01    2769    1999    96      81      15      21.00   79.12
    22      22      1979    71.73   2759    1998    3710    84      1       28.30   0.00
    22      46      0       0.01    2776    1998    95      79      16      20.96   79.14
    24      3       1978    71.72   2758    1998    3709    82      2       28.31   0.00
    24      27      0       0.01    2733    1998    94      81      13      21.00   79.08
    25      11      1978    71.71   2758    1998    3710    87      1       28.31   0.00
    25      35      0       0.01    2787    1998    93      78      15      20.95   79.12
    26      19      2301    83.32   2762    1997    3712    20      4       16.69   0.00
    26      43      0       0.01    2752    1997    92      80      13      20.97   79.08
    28      7       1977    71.70   2757    1997    3709    82      2       28.31   0.00
    28      31      0       0.02    2791    1997    92      81      12      20.98   79.04
    29      15      1976    71.69   2756    1997    3710    87      3       21.89   6.42
    29      39      3       0.11    2790    1997    118     163     16      20.88   79.04
    30      23      1975    71.68   2756    1996    3710    85      1       28.32   0.00
    30      47      4       0.13    2787    1996    100     77      16      20.88   79.00
    

    48 cpu threads gives 2580-2603Mhz cpu clock speed - is this the max or should it be 2.8Ghz ?
    Code (Text):
    Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IRQ     C1      C2      C1%     C2%
    -       -       1773    68.42   2591    1988    167510  772     730     0.55    30.90
    0       0       1773    68.72   2580    1979    3506    26      27      0.18    30.84
    0       24      1722    66.84   2577    1980    3414    14      14      0.13    32.76
    1       8       1754    68.00   2579    1980    3473    18      12      0.11    31.63
    1       32      1710    66.36   2577    1980    3397    9       22      0.08    33.28
    2       16      1731    67.15   2578    1981    3421    12      7       0.09    32.50
    2       40      1698    65.90   2578    1981    3362    9       11      0.08    33.76
    4       4       1772    68.61   2583    1981    3539    16      52      5.42    25.74
    4       28      1809    69.87   2589    1982    3636    9       48      4.26    25.66
    5       12      1798    69.50   2587    1982    3545    8       15      0.07    30.21
    5       36      1794    69.38   2586    1983    3571    13      45      0.11    30.30
    6       20      1769    68.45   2584    1983    3489    16      6       0.09    31.25
    6       44      1808    69.81   2590    1983    3641    29      36      10.42   19.57
    8       1       1757    67.98   2584    1984    3476    17      19      0.11    31.70
    8       25      1793    69.28   2588    1984    3528    12      12      0.09    30.44
    9       9       1756    67.93   2585    1984    3461    9       12      0.07    31.81
    9       33      1791    69.18   2589    1985    3522    11      10      0.09    30.56
    10      17      1793    69.25   2589    1985    3537    18      16      0.22    30.36
    10      41      1770    68.42   2587    1985    3487    11      13      0.13    31.28
    12      5       1814    70.02   2591    1986    3568    16      15      0.16    29.67
    12      29      1783    68.91   2587    1986    3512    21      11      0.21    30.72
    13      13      1770    68.39   2589    1987    3478    19      6       0.16    31.30
    13      37      1782    68.80   2590    1987    3504    12      12      0.14    30.92
    14      21      1765    68.17   2589    1987    3473    18      9       0.18    31.52
    14      45      1780    68.70   2591    1988    3501    10      15      0.11    31.05
    16      2       1745    67.44   2588    1988    3448    25      20      0.25    32.18
    16      26      1740    67.24   2588    1988    3424    10      13      0.11    32.52
    17      10      1806    69.55   2597    1989    3533    20      5       0.15    30.19
    17      34      1716    66.30   2588    1989    3380    9       17      0.11    33.47
    18      18      1810    69.73   2596    1989    3543    20      7       0.12    30.04
    18      42      1707    65.98   2587    1990    3365    15      15      0.19    33.73
    20      6       1778    68.57   2592    1990    3484    20      6       0.15    31.18
    20      30      1747    67.46   2590    1990    3432    11      11      0.13    32.32
    21      14      1760    67.88   2592    1991    3453    21      9       0.19    31.84
    21      38      1786    68.82   2596    1991    3500    11      13      0.12    30.97
    22      22      1772    68.31   2594    1991    3474    15      10      0.15    31.47
    22      46      1807    69.49   2600    1992    3531    15      11      0.14    30.30
    24      3       1777    68.47   2596    1992    3482    14      13      0.12    31.34
    24      27      1749    67.44   2593    1993    3427    10      11      0.10    32.40
    25      11      1766    68.04   2596    1993    3462    17      13      0.19    31.72
    25      35      1713    66.05   2594    1993    3372    15      21      0.21    33.69
    26      19      1777    68.41   2597    1994    3477    20      10      0.19    31.36
    26      43      1819    69.88   2604    1994    3549    13      14      0.11    29.98
    28      7       1815    69.72   2604    1994    3551    21      14      0.24    30.01
    28      31      1803    69.31   2602    1995    3521    21      10      0.17    30.50
    29      15      1799    69.14   2602    1995    3509    20      6       0.16    30.68
    29      39      1801    69.20   2602    1995    3533    49      12      0.16    30.62
    30      23      1802    69.27   2602    1996    3516    19      8       0.14    30.58
    30      47      1793    68.87   2603    1996    3503    8       16      0.08    31.05
    


    So lets try a new Centmin Mod 123.09beta01 install
    Code (Text):
    yum -y update; curl -O https://centminmod.com/betainstaller.sh && chmod 0700 betainstaller.sh && bash betainstaller.sh
    

    Centmin Mod 123.09beta01 install times at ~733 seconds which for AMD cpu compiler flag optimised build is decent considering non AMD cpu compiler flag optimised build previous took ~769 seconds
    Code (Text):
    ---------------------------------------------------------------------------
    Total Curl Installer YUM or DNF Time: 152.7325 seconds
    Total YUM Time: 35.876224841 seconds
    Total YUM or DNF + Source Download Time: 54.1354
    Total Nginx First Time Install Time: 138.7361
    Total PHP First Time Install Time: 103.9416
    Download Zip From Github Time: 1.0804
    Total Time Other eg. source compiles: 282.6103
    Total Centmin Mod Install Time: 579.4234
    ---------------------------------------------------------------------------
    Total Install Time (curl yum + cm install + zip download): 733.2363 seconds
    ---------------------------------------------------------------------------
    

    Looking good, checking cpu frequency during Nginx compilation stage of Centmin Mod installation shows ~2780Mhz cpu frequency :)
    Code (Text):
    grep MHz /proc/cpuinfo
    cpu MHz         : 2773.788
    cpu MHz         : 2778.998
    cpu MHz         : 2745.497
    cpu MHz         : 2765.689
    cpu MHz         : 2770.491
    cpu MHz         : 2724.596
    cpu MHz         : 2772.604
    cpu MHz         : 2784.042
    cpu MHz         : 2782.290
    cpu MHz         : 2780.214
    cpu MHz         : 2779.382
    cpu MHz         : 2783.001
    cpu MHz         : 2768.134
    cpu MHz         : 2767.165
    cpu MHz         : 2780.245
    cpu MHz         : 2791.233
    cpu MHz         : 2754.883
    cpu MHz         : 2782.203
    cpu MHz         : 2747.773
    cpu MHz         : 2772.235
    cpu MHz         : 2769.830
    cpu MHz         : 2778.107
    cpu MHz         : 2781.585
    cpu MHz         : 2777.746
    cpu MHz         : 2778.892
    cpu MHz         : 2779.206
    cpu MHz         : 2755.335
    cpu MHz         : 2772.050
    cpu MHz         : 2780.281
    cpu MHz         : 2750.842
    cpu MHz         : 2774.551
    cpu MHz         : 2782.898
    cpu MHz         : 2784.215
    cpu MHz         : 2784.794
    cpu MHz         : 2780.141
    cpu MHz         : 2784.718
    cpu MHz         : 2774.572
    cpu MHz         : 2773.928
    cpu MHz         : 2777.358
    cpu MHz         : 2792.444
    cpu MHz         : 2752.807
    cpu MHz         : 2785.860
    cpu MHz         : 2750.918
    cpu MHz         : 2773.167
    cpu MHz         : 2774.841
    cpu MHz         : 2776.860
    cpu MHz         : 2786.369
    cpu MHz         : 2775.252
    


    GCC 8.0.1 Compile & RPM Build Times



    Centmin Mod LEMP Stack's Nginx web server supports multiple GCC/Clang version compilers from GCC 4.8, 5.3, 6.3, 7.2 and 8.x and Clang 3.4, 4, 5, and 6. Retesting binutils 2.30 + GCC 8.0.1 with Profile Guided Optimizations (PGO) custom RPM build times. Took ~2074 seconds to build RPMs.
    Code (Text):
    /opt/gcc-8-20180225/bin/gcc -v
    Using built-in specs.
    COLLECT_GCC=/opt/gcc-8-20180225/bin/gcc
    COLLECT_LTO_WRAPPER=/opt/gcc-8-20180225/libexec/gcc/x86_64-redhat-linux/8/lto-wrapper
    Target: x86_64-redhat-linux
    Configured with: ../configure --prefix=/opt/gcc-8-20180225 --disable-multilib --enable-bootstrap --enable-plugin --with-gcc-major-version-only --enable-shared --disable-nls --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-install-libiberty --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-linker-hash-style=gnu --enable-languages=c,c++ --enable-initfini-array --disable-libgcj --enable-gnu-indirect-function --with-tune=generic --build=x86_64-redhat-linux --enable-lto --enable-gold
    Thread model: posix
    gcc version 8.0.1 20180225 (experimental) (GCC)
    
    RPMs Built
    /svr-setup/binutils-gcc8-2.30-1.el7.x86_64.rpm
    /svr-setup/gcc8-pgo-8.0-1.el7.x86_64.rpm
    
    ls -lah /svr-setup | egrep 'gcc8-pgo-8.0|binutils-gcc8-2.30'
    -rw-r--r--  1 root      root       5.2M Feb 28 00:46 binutils-gcc8-2.30-1.el7.x86_64.rpm
    -rw-r--r--  1 root      root        39M Feb 28 01:17 gcc8-pgo-8.0-1.el7.x86_64.rpm
    
    Installed Packages
    Name        : binutils-gcc8
    Arch        : x86_64
    Version     : 2.30
    Release     : 1.el7
    Size        : 48 M
    Repo        : installed
    Summary     : binutils-gcc8 for centminmod.com LEMP stack installs
    URL         : https://centminmod.com
    License     : unknown
    Description : binutils-gcc8 for centminmod.com LEMP stacks
    
    * Wed Feb 28 2018 George Liu <centminmod.com> 2.30
    - Binutils 2.30 for centminmod.com LEMP stack installs
    

    Code (Text):
    Installed Packages
    Name        : gcc8-pgo
    Arch        : x86_64
    Version     : 8.0
    Release     : 1.el7
    Size        : 287 M
    Repo        : installed
    Summary     : gcc8-pgo for centminmod.com LEMP stack installs
    URL         : https://centminmod.com
    License     : unknown
    Description : gcc8-pgo for centminmod.com LEMP stacks
    
    * Wed Feb 28 2018 George Liu <centminmod.com> 8
    - GCC 8 for centminmod.com LEMP stack installs
    
    *************************************************
    Compile GCC Completed
    log: /root/centminlogs/tools-gcc-install-pgo_280218-004337.log
    *************************************************
    
    Total Binutils + GCC Install Time: 2074.655782081 seconds
    


    AMD EPYC 7401P Turbo Frequencies



    Looking at EPYC 7401P - AMD the Turbo frequencies are as follows
     
    Last edited: Feb 28, 2018
  9. eva2000

    eva2000 Administrator Staff Member

    35,481
    7,828
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,067
    Local Time:
    6:41 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x

    Packet.net AMD EPYC 7401P - Dell PowerEdge R6415 Bios



    A quick peak into Dell PowerEdge R6415 Bios for the AMD EPYC 7401P server :)

    packetnet-epyc-bios-setup-01.png packetnet-epyc-bios-setup-02.png packetnet-epyc-bios-setup-03.png packetnet-epyc-bios-setup-04.png packetnet-epyc-bios-setup-05.png packetnet-epyc-bios-setup-06.png packetnet-epyc-bios-setup-07.png packetnet-epyc-bios-setup-08.png packetnet-epyc-bios-setup-09.png packetnet-epyc-bios-setup-10.png
     
..
Thread Status:
Not open for further replies.