Want to subscribe to topics you're interested in?
Become a Member

Sysadmin Video Server at 2gbps Tips

Discussion in 'System Administration' started by upgrade81, Mar 30, 2020.

  1. upgrade81

    upgrade81 Premium Member Premium Member

    260
    15
    18
    Sep 5, 2016
    Italy
    Ratings:
    +26
    Local Time:
    8:41 AM
    1.17
    10.3
    Hello everyone!

    I manage a VM 8core amd ryzer 2000ghz SSD which acts as a server for various videos delivery (mp4).
    The problem is that when it reaches around 2gbps / s it starts to struggle.
    A lot of cpu time is spent by ksoftirq / 7, and even connecting to ssh is sometimes difficult. (Timeout)

    Does anyone have experience with high traffic servers for mp4 video? tips?
    Thank you

    Code:
    netstat -n | grep :443 |wc -l
    7021
    Code:
    cat /proc/interrupts
               CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       
      1:          0          0          0          9          0          0          0          0   IO-APIC   1-edge      i8042
      6:          0          0          0          0          0          3          0          0   IO-APIC   6-edge      floppy
      8:          0          0          0          0          0          0          0          0   IO-APIC   8-edge      rtc0
      9:          0          0          0          0          0          0          0          0   IO-APIC   9-fasteoi   acpi
     11:          0         32          0          0          0          0          0          0   IO-APIC  11-fasteoi   uhci_hcd:usb1, virtio2, snd_intel8x0
     12:          0          0         15          0          0          0          0          0   IO-APIC  12-edge      i8042
     14:          0          0          0          0          0          0          0          0   IO-APIC  14-edge      ata_piix
     15:          0          0          0          0          0          0          0          0   IO-APIC  15-edge      ata_piix
     24:          0          0          0          0          0          0          0          0   PCI-MSI 81920-edge      virtio1-config
     25:          0          0          0          0          0          0          0  119535236   PCI-MSI 81921-edge      virtio1-req.0
     26:          0          0          0          0          0          0          0          0   PCI-MSI 49152-edge      virtio0-config
     27:          0          0       8839          0          0          0          0 2458104622   PCI-MSI 49153-edge      virtio0-input.0
     28:          0          0 2087572167      20510          0          0          0          0   PCI-MSI 49154-edge      virtio0-output.0
    NMI:          0          0          0          0          0          0          0          0   Non-maskable interrupts
    LOC:  242498412  244547692 2083283479  240495539  240852602  240205421  239301178 2908770224   Local timer interrupts
    SPU:          0          0          0          0          0          0          0          0   Spurious interrupts
    PMI:          0          0          0          0          0          0          0          0   Performance monitoring interrupts
    IWI:          0          0          0          0          0          0          0       5360   IRQ work interrupts
    RTR:          0          0          0          0          0          0          0          0   APIC ICR read retries
    RES:  133106492  126266239  101753325  114185677  114145843  114039358  114441769    6096856   Rescheduling interrupts
    CAL:   15560856   15550235   16693640   15495830   15521587   15539931   15513590      82570   Function call interrupts
    TLB:     104490     117434     117286     131875     127052     124583     143121      82673   TLB shootdowns
    TRM:          0          0          0          0          0          0          0          0   Thermal event interrupts
    THR:          0          0          0          0          0          0          0          0   Threshold APIC interrupts
    DFR:          0          0          0          0          0          0          0          0   Deferred Error APIC interrupts
    MCE:          0          0          0          0          0          0          0          0   Machine check exceptions
    MCP:       1275       1275       1275       1275       1275       1275       1275       1275   Machine check polls
    HYP:          0          0          0          0          0          0          0          0   Hypervisor callback interrupts
    HRE:          0          0          0          0          0          0          0          0   Hyper-V reenlightenment interrupts
    HVS:          0          0          0          0          0          0          0          0   Hyper-V stimer0 interrupts
    ERR:          0
    MIS:          0
    PIN:          0          0          0          0          0          0          0          0   Posted-interrupt notification event
    NPI:          0          0          0          0          0          0          0          0   Nested posted-interrupt event
    PIW:          0          0          0          0          0          0          0          0   Posted-interrupt wakeup event
    SecureCRT_tZYTx6DPeO.png
     
  2. eva2000

    eva2000 Administrator Staff Member

    44,426
    10,144
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,696
    Local Time:
    4:41 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Have a read of Sysadmin - Help with a solution for video streaming :( and how to use Nginx mp4/flv sliced byte range caching. But that only helps for caching videos but ultimately it comes down to optimizing video file size and having enough network bandwidth to serve the videos. 2Gbps might not be enough, probably need 10+ Gbps.

    Your VPS has 2Gbp network support ? as most only have 1Gbps.
     
  3. upgrade81

    upgrade81 Premium Member Premium Member

    260
    15
    18
    Sep 5, 2016
    Italy
    Ratings:
    +26
    Local Time:
    8:41 AM
    1.17
    10.3
    Hi yes I read it and I'm using it.

    Si is a friend of mine who runs the data center and he personalized the offer.
    The host server has a 10gbps network card and the VM has no bandwidth restrictions.

    Do you have possible solutions? Could it be overhead of the virtual network adapter that cannot handle all packets?
     
  4. rdan

    rdan Well-Known Member

    4,940
    1,179
    113
    May 25, 2014
    Ratings:
    +1,787
    Local Time:
    2:41 PM
    Mainline
    10.2
    Why not use Cloudflare? Or any cheap CDN.
     
  5. eva2000

    eva2000 Administrator Staff Member

    44,426
    10,144
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,696
    Local Time:
    4:41 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    section 2.8 Self-Serve Subscription Agreement | Cloudflare may prohibit it if it's a excessive portion of overall cached traffic

    But it would have to be a very very high portion for Cloudflare to even notice or be concerned with.

    Yeah you'll need to further optimize your OS and network settings including network adapter which may help. This would be outside of Centmin Mod support as Centmin Mod is provided as is. So something you'd need to look at yourself or hire someone to look into for you.

    Though with KVM VPS, you'd be limited by how the KVM host node configures each VPS i.e. most KVM only have a single receive (RX) and transmit (TX) queue for virtio-net driver unless your web host configures KVM VPS for multi-queue based virtio-net driver so to have multiple RX/TX queues Features/MQ virtio net - Fedora Project Wiki

    The gist is if you want better performance generally use a dedicated server instead of VPS for this.
     
  6. upgrade81

    upgrade81 Premium Member Premium Member

    260
    15
    18
    Sep 5, 2016
    Italy
    Ratings:
    +26
    Local Time:
    8:41 AM
    1.17
    10.3
    I immediately check how the host is configured.

    Thank you very much.
     
  7. Rake-GH

    Rake-GH Active Member

    125
    56
    28
    Jul 29, 2019
    USA
    Ratings:
    +92
    Local Time:
    2:41 AM
    default
    default
    I know of 2 options that are worth looking into

    1) Vimeo can host your videos and it's 50$/month

    upload_2020-4-1_13-40-14.png

    2) CloudFlare video streaming:
    upload_2020-4-1_13-33-27.png

    If I get banned from youtube again I think I'm gonna use Cloudflare
     
  8. upgrade81

    upgrade81 Premium Member Premium Member

    260
    15
    18
    Sep 5, 2016
    Italy
    Ratings:
    +26
    Local Time:
    8:41 AM
    1.17
    10.3
    the server does 260TB of traffic per month, both are too expensive, unfortunately.
     
  9. eva2000

    eva2000 Administrator Staff Member

    44,426
    10,144
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,696
    Local Time:
    4:41 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    They're relatively inexpensive when you start needing 10-100Gbps network connections. Try getting a quote for a dedicated server with 10Gbps, 20Gbps, 40Gbps and 100Gbps network adaptor connections and dedicated 10-100Gbps bandwidth lines. You'll see the price is alot more.

    But really depends on your needs. Seems for you it's >2Gbps at least.