Sysadmin Video Server at 2gbps Tips

upgrade81 · Mar 30, 2020

Hello everyone!

I manage a VM 8core amd ryzer 2000ghz SSD which acts as a server for various videos delivery (mp4).
The problem is that when it reaches around 2gbps / s it starts to struggle.
A lot of cpu time is spent by ksoftirq / 7, and even connecting to ssh is sometimes difficult. (Timeout)

Does anyone have experience with high traffic servers for mp4 video? tips?
Thank you

Code:

netstat -n | grep :443 |wc -l
7021

Code:

cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       
  1:          0          0          0          9          0          0          0          0   IO-APIC   1-edge      i8042
  6:          0          0          0          0          0          3          0          0   IO-APIC   6-edge      floppy
  8:          0          0          0          0          0          0          0          0   IO-APIC   8-edge      rtc0
  9:          0          0          0          0          0          0          0          0   IO-APIC   9-fasteoi   acpi
 11:          0         32          0          0          0          0          0          0   IO-APIC  11-fasteoi   uhci_hcd:usb1, virtio2, snd_intel8x0
 12:          0          0         15          0          0          0          0          0   IO-APIC  12-edge      i8042
 14:          0          0          0          0          0          0          0          0   IO-APIC  14-edge      ata_piix
 15:          0          0          0          0          0          0          0          0   IO-APIC  15-edge      ata_piix
 24:          0          0          0          0          0          0          0          0   PCI-MSI 81920-edge      virtio1-config
 25:          0          0          0          0          0          0          0  119535236   PCI-MSI 81921-edge      virtio1-req.0
 26:          0          0          0          0          0          0          0          0   PCI-MSI 49152-edge      virtio0-config
 27:          0          0       8839          0          0          0          0 2458104622   PCI-MSI 49153-edge      virtio0-input.0
 28:          0          0 2087572167      20510          0          0          0          0   PCI-MSI 49154-edge      virtio0-output.0
NMI:          0          0          0          0          0          0          0          0   Non-maskable interrupts
LOC:  242498412  244547692 2083283479  240495539  240852602  240205421  239301178 2908770224   Local timer interrupts
SPU:          0          0          0          0          0          0          0          0   Spurious interrupts
PMI:          0          0          0          0          0          0          0          0   Performance monitoring interrupts
IWI:          0          0          0          0          0          0          0       5360   IRQ work interrupts
RTR:          0          0          0          0          0          0          0          0   APIC ICR read retries
RES:  133106492  126266239  101753325  114185677  114145843  114039358  114441769    6096856   Rescheduling interrupts
CAL:   15560856   15550235   16693640   15495830   15521587   15539931   15513590      82570   Function call interrupts
TLB:     104490     117434     117286     131875     127052     124583     143121      82673   TLB shootdowns
TRM:          0          0          0          0          0          0          0          0   Thermal event interrupts
THR:          0          0          0          0          0          0          0          0   Threshold APIC interrupts
DFR:          0          0          0          0          0          0          0          0   Deferred Error APIC interrupts
MCE:          0          0          0          0          0          0          0          0   Machine check exceptions
MCP:       1275       1275       1275       1275       1275       1275       1275       1275   Machine check polls
HYP:          0          0          0          0          0          0          0          0   Hypervisor callback interrupts
HRE:          0          0          0          0          0          0          0          0   Hyper-V reenlightenment interrupts
HVS:          0          0          0          0          0          0          0          0   Hyper-V stimer0 interrupts
ERR:          0
MIS:          0
PIN:          0          0          0          0          0          0          0          0   Posted-interrupt notification event
NPI:          0          0          0          0          0          0          0          0   Nested posted-interrupt event
PIW:          0          0          0          0          0          0          0          0   Posted-interrupt wakeup event

eva2000 · Mar 30, 2020

Have a read of Sysadmin - Help with a solution for video streaming :( and how to use Nginx mp4/flv sliced byte range caching. But that only helps for caching videos but ultimately it comes down to optimizing video file size and having enough network bandwidth to serve the videos. 2Gbps might not be enough, probably need 10+ Gbps.

Your VPS has 2Gbp network support ? as most only have 1Gbps.

upgrade81 · Mar 30, 2020

Hi yes I read it and I'm using it.

Si is a friend of mine who runs the data center and he personalized the offer.
The host server has a 10gbps network card and the VM has no bandwidth restrictions.

Do you have possible solutions? Could it be overhead of the virtual network adapter that cannot handle all packets?

rdan · Mar 30, 2020

Why not use Cloudflare? Or any cheap CDN.

eva2000 · Mar 30, 2020

rdan said: ↑

Why not use Cloudflare? Or any cheap CDN.
Click to expand...

section 2.8 Self-Serve Subscription Agreement | Cloudflare may prohibit it if it's a excessive portion of overall cached traffic

2.8 Limitation on Non-HTML Caching
The Service is offered primarily as a platform to cache and serve web pages and websites. Unless explicitly included as a part of a Paid Service purchased by you, you agree to use the Service solely for the purpose of serving web pages as viewed through a web browser or other application and the Hypertext Markup Language (HTML) protocol or other equivalent technology. Use of the Service for the storage or caching of video (unless purchased separately as a Paid Service) or a disproportionate percentage of pictures, audio files, or other non-HTML content, is prohibited.
Click to expand...

But it would have to be a very very high portion for Cloudflare to even notice or be concerned with.

upgrade81 said: ↑

Hi yes I read it and I'm using it.

Si is a friend of mine who runs the data center and he personalized the offer.
The host server has a 10gbps network card and the VM has no bandwidth restrictions.

Do you have possible solutions? Could it be overhead of the virtual network adapter that cannot handle all packets?
Click to expand...

Yeah you'll need to further optimize your OS and network settings including network adapter which may help. This would be outside of Centmin Mod support as Centmin Mod is provided as is. So something you'd need to look at yourself or hire someone to look into for you.

Though with KVM VPS, you'd be limited by how the KVM host node configures each VPS i.e. most KVM only have a single receive (RX) and transmit (TX) queue for virtio-net driver unless your web host configures KVM VPS for multi-queue based virtio-net driver so to have multiple RX/TX queues Features/MQ virtio net - Fedora Project Wiki

Today's high-end server have more processors, guests running on them tend have an increasing number of vcpus. The scale of the protocol stack in guest in restricted because of the single queue virtio-net:

The network performance does not scale as the number of vcpus increasing: Guest can not transmit or retrieve packets in parallel as virtio-net have only one TX and RX, virtio-net drivers must be synchronized before sending and receiving packets. Even through there's software technology to spread the loads into different processor such as RFS, such kind of method is only for transmission and is really expensive in guest as they depends on IPI which may brings extra overhead in virtualized environment.

Multiqueue nic were more common used and is well supported by linux kernel, but current virtual nic can not utilize the multi queue support: the tap and virtio-net backend must serialize the co-current transmission/receiving request comes from different cpus.

In order the remove those bottlenecks, we must allow the paralleled packet processing by introducing multi queue support for both back-end and guest drivers. Ideally, we may let the packet handing be done by processors in parallel without interleaving and scale the network performance as the number of vcpus increasing.

The following parts were changed to parallize the packet processing:

tuntap: convert the driver to multiqueue by allowing multiple socket/fd to be attached to the device, each socket/fd exposed by the device could be treated as a queue.

qemu:

net: Add multiple queue infrastructure to qemu

let qemu can create multiple vhost threads for a virtio-net device

userspace multiple queue virtio-net

guest driver: let the driver can use multiple virtqueues to do packet sending/receiving.

Click to expand...

The gist is if you want better performance generally use a dedicated server instead of VPS for this.

upgrade81 · Mar 30, 2020

I immediately check how the host is configured.

Thank you very much.

Rake-GH · Apr 2, 2020

I know of 2 options that are worth looking into

1) Vimeo can host your videos and it's 50$/month

2) CloudFlare video streaming:

If I get banned from youtube again I think I'm gonna use Cloudflare

upgrade81 · Apr 2, 2020

Rake-GH said: ↑

I know of 2 options that are worth looking into

1) Vimeo can host your videos and it's 50$/month

View attachment 7322

2) CloudFlare video streaming:
View attachment 7321

If I get banned from youtube again I think I'm gonna use Cloudflare
Click to expand...

the server does 260TB of traffic per month, both are too expensive, unfortunately.

eva2000 · Apr 2, 2020

upgrade81 said: ↑

the server does 260TB of traffic per month, both are too expensive, unfortunately.
Click to expand...

They're relatively inexpensive when you start needing 10-100Gbps network connections. Try getting a quote for a dedicated server with 10Gbps, 20Gbps, 40Gbps and 100Gbps network adaptor connections and dedicated 10-100Gbps bandwidth lines. You'll see the price is alot more.

But really depends on your needs. Seems for you it's >2Gbps at least.

Log in / Register

Forums

Welcome to Centmin Mod Community

Sysadmin Video Server at 2gbps Tips

upgrade81 Member

eva2000 Administrator Staff Member

upgrade81 Member

rdan Well-Known Member

eva2000 Administrator Staff Member

upgrade81 Member

Rake-GH Active Member

upgrade81 Member

eva2000 Administrator Staff Member

.

Log in / Register

Useful Searches

Welcome to Centmin Mod Community

Sysadmin Video Server at 2gbps Tips

upgrade81 Member

eva2000 Administrator Staff Member

upgrade81 Member

rdan Well-Known Member

eva2000 Administrator Staff Member

upgrade81 Member

Rake-GH Active Member

upgrade81 Member

eva2000 Administrator Staff Member

.