/ Register

Welcome to Centmin Mod Community
Register Now

Sysadmin Help with a solution for video streaming :(

Discussion in 'System Administration' started by KinderOvO, Sep 1, 2018.

Previous Thread Next Thread
Loading...
Page 2 of 3
  1. Sep 4, 2018 #21
    eva2000

    eva2000 Administrator Staff Member

    46,214
    10,506
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,289
    Local Time:
    11:36 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    This thread has renewed my interestlearning for video encoding/streaming again - it's not something I dabble in much due to how expensive it can get at scale though. Just updated Centmin Mod 123.09beta01's addons/ffmpeg.sh installer with optional AV1 encoder support too Beta Branch - update addons/ffmpeg.sh add AV1 support. From what i read much slower encoding speeds than VP9 webm and H.264/265 but much smaller file sizes due to better compression etc. Some results on AV1 site AV1 Test Results.

    AV1 beats x264 and libvpx-vp9 in practical use case

    So if you're currently streaming H.264 mp4 videos, AV1 encoded could reduce size by upto 50% !

    Enjoy!
     
  2. Sep 4, 2018 #22
    gabel

    gabel Premium Member Premium Member

    66
    36
    18
    Jan 7, 2016
    Ratings:
    +59
    Local Time:
    1:36 PM
    1.9.9
    MariaDB 10
    kvs makes use of limit_rate_after , here's a post where they explain how this can be done using their script (could be applied to other scripts as well )

    Code:
    https://www.kernel-video-sharing.com/forum/forum/educational-support/educational-series/34-how-to-save-bandwidth-with-kvs-tube-script
     
    Last edited: Sep 4, 2018
  3. Sep 5, 2018 #23
    eva2000

    eva2000 Administrator Staff Member

    46,214
    10,506
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,289
    Local Time:
    11:36 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Thanks @gabel definitely interesting and smart approach :)

    Module ngx_http_core_module

     
  4. Sep 20, 2018 #24
    KinderOvO

    KinderOvO Member

    92
    1
    8
    Jan 13, 2015
    Ratings:
    +1
    Local Time:
    11:36 PM
    1.4
    10
    Total DISK READ : 474.02 M/s | Total DISK WRITE : 3.3 M/s
    Actual DISK READ: 477.67 M/s | Actual DISK WRITE: 1.15 M/s

    Is my iotop result

    Thanks all for attention to help-me

    I am currently using 6 gbps full

    I plan to upgrade to 20gbps.

    Being that I disc rate are already high, all my discs are ssds which generates a very high cost.

    I need a lot of space, because I use videos in 3 formats, 720p, 480p and 360p

    because many visitors use 3G and a slow connection.

    What would be the best solution, since nginx is not caching the videos.

    Is there any way to cache the videos and decrease disk usage.

    I currently use 4 x 2TB SSD
    all the disks are already with 70% of use, already I am thinking how I will do to 20gbps, since with 6gbps the reading rate is very high.
     
  5. Sep 20, 2018 #25
    eva2000

    eva2000 Administrator Staff Member

    46,214
    10,506
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,289
    Local Time:
    11:36 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Already mentioned in this thread with links to using nginx sliced byte range caching to cache videos :) For disk space usage, re-encode videos for smaller size I guess
     
  6. Sep 20, 2018 #26
    KinderOvO

    KinderOvO Member

    92
    1
    8
    Jan 13, 2015
    Ratings:
    +1
    Local Time:
    11:36 PM
    1.4
    10
    Do I need to format my current server and reinstall centmin? Could you share your vhost? and config for mp4 / webm files.
     
  7. Sep 20, 2018 #27
    eva2000

    eva2000 Administrator Staff Member

    46,214
    10,506
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,289
    Local Time:
    11:36 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
  8. Sep 20, 2018 #28
    KinderOvO

    KinderOvO Member

    92
    1
    8
    Jan 13, 2015
    Ratings:
    +1
    Local Time:
    11:36 PM
    1.4
    10
    Got it, have you ever used HLS? would it have an advantage? because I currently have to generate 3 videos, 720p, 480p, 320p - because I have a lot of visitors from areas with slower internet.

    It would be the same process, I looked into generating videos with codec AV1 for hls, but I did not find anything of the type.
     
  9. Oct 1, 2018 #29
    KinderOvO

    KinderOvO Member

    92
    1
    8
    Jan 13, 2015
    Ratings:
    +1
    Local Time:
    11:36 PM
    1.4
    10
  10. Oct 1, 2018 #30
    eva2000

    eva2000 Administrator Staff Member

    46,214
    10,506
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,289
    Local Time:
    11:36 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
  11. Oct 1, 2018 #31
    KinderOvO

    KinderOvO Member

    92
    1
    8
    Jan 13, 2015
    Ratings:
    +1
    Local Time:
    11:36 PM
    1.4
    10
    My problem is this directive
    limit_traffic_rate rate 1500k;

    nginx not start after add.
     
  12. Oct 1, 2018 #32
    eva2000

    eva2000 Administrator Staff Member

    46,214
    10,506
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,289
    Local Time:
    11:36 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    check nginx config test output
    Code (Text):
    nginx -t
     
  13. Oct 1, 2018 #33
    KinderOvO

    KinderOvO Member

    92
    1
    8
    Jan 13, 2015
    Ratings:
    +1
    Local Time:
    11:36 PM
    1.4
    10
    no cache miss or hit :(
     

    Attached Files:

  14. Oct 1, 2018 #34
    eva2000

    eva2000 Administrator Staff Member

    46,214
    10,506
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,289
    Local Time:
    11:36 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Keep trying and testing. It worked for me Sysadmin - Help with a solution for video streaming :(. No free support provided from me - this is usually something my clients pay me to consult for. This is something you need to figure out on your own.

    nginx-sliced-byte-range-caching-011018-00.png

    added conditional headers to track which sliced byte chunks are sliced = 1 and sliced-cache = HIT or MISS. It will only be from cache if you skip back or forth to point in a video's playback the first time and then cached (cache HIT) for subsequent requests for however long you configured cache expiry and cache inactive time. After cache size reached or expiry/inactive time is reached, the cache is cleared so again first time request is cache = MISS

    nginx-sliced-byte-range-caching-011018-01.png

    and sliced = 1 but sliced-cache = MISS

    nginx-sliced-byte-range-caching-011018-02.png
     
    Last edited: Oct 1, 2018
  15. Oct 1, 2018 #35
    KinderOvO

    KinderOvO Member

    92
    1
    8
    Jan 13, 2015
    Ratings:
    +1
    Local Time:
    11:36 PM
    1.4
    10
    Thanks for the attention, I will try something different because it did not work for me, it is not caching the .mp4 files.
     
  16. Oct 4, 2018 #36
    KinderOvO

    KinderOvO Member

    92
    1
    8
    Jan 13, 2015
    Ratings:
    +1
    Local Time:
    11:36 PM
    1.4
    10
    Currently
    Network: 698.22 MBit/s
    Iotop: Total DISK READ : 175.79 M/s | Total DISK WRITE : 65.83 K/s
    Cpu: Average: 1.72 1.66 1.57

    When it arrives at 3gbps of use, it is super slow, load of the cpu passes of the 33%, iotop in the heights.

    So my doubt and the next, I can never have a server to stream 5gbps? because currently my hardware is good in my point of view, but I already have these problems using little traffic.
    2x2620v4 + 64GB DDR4 + 4x2TB SSD
     
  17. Oct 14, 2018 #37
    Kintaro

    Kintaro Member

    79
    5
    8
    Dec 2, 2016
    Italy
    Ratings:
    +19
    Local Time:
    2:36 PM
    1.15.x
    MariaDB 10
    Just use Pied Piper :whistle:


    p.s. sorry but I couldn't resist.
     
  18. Oct 24, 2018 #38
    KinderOvO

    KinderOvO Member

    92
    1
    8
    Jan 13, 2015
    Ratings:
    +1
    Local Time:
    11:36 PM
    1.4
    10
  19. Oct 24, 2018 #39
    eva2000

    eva2000 Administrator Staff Member

    46,214
    10,506
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,289
    Local Time:
    11:36 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
  20. Nov 20, 2019 #40
    buik

    buik 'The only thing we have to fear is fear itself ' Premium Member

    1,347
    372
    83
    Apr 29, 2016
    Flanders
    Ratings:
    +1,126
    Local Time:
    2:36 PM
    @KinderOvO year plus ago when you initially launched you question about:
    'Help with a solution for video streaming'. How is it going and what solution are you using now?
    '
     
Previous Thread Next Thread
Loading...
Page 2 of 3

.