Join the community today
Register Now

Images on cloud/cdn ?

Discussion in 'System Administration' started by pamamolf, Sep 8, 2015.

Tags:
  1. pamamolf

    pamamolf Well-Known Member

    2,721
    242
    63
    May 31, 2014
    Ratings:
    +433
    Local Time:
    1:16 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    Hi

    Is the best solution for performance most to move all images to a cloud or CDN when there is no disk space and can't get bigger ssd's disks?

    Can you please recommend me a nice and fast cloud or cdn service?

    Thank you
     
  2. eva2000

    eva2000 Administrator Staff Member

    30,156
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    8:16 AM
    Nginx 1.13.x
    MariaDB 5.5
    Depends 2 types of CDN methods - pull based or push based. Most CDNs are predominantly pull based, so you need the origin - local server to serve the original image/static files anyway. So that doesn't suit your needs if your concern is running out of local disk space.

    The push based CDN is where you upload images/static files to the CDN cloud storage i.e for Amazon Cloudfront you'd upload images to an Amazon S3 bucket and setup Amazon Cloudfront to point to that S3 bucket which acts as the origin. Still you'd need to have the original source images and static files at one time or another to be locally stored before being uploaded to cloud /CDN storage like an S3 bucket. So either way you cut it, you need ample disk space on local server especially if you take into account growing disk log storage and the fact you need reserve disk free space for backups/restores of your data.

    Both @RoldanLT and myself have used KeyCDN but only used for pull based although support push based CDN usage as well. However, my experience using it on this forum has been mixed.. always get a slight but very noticeable lag on initial page loads/navigation compared to non-CDN. Then there's Amazon Cloudfront haven't used it this forum, but I do use it on my Wordpress 7 test site :) KeyCDN easier to setup your own SSL certificate for https based CDN usage. Amazon Cloudfront is alot tricker for SSL CDN usage as they don't have a web interface for it they use their AWS CLI command line tools AFAIK so haven't tried it. There's also MaxCDN I use, expensive for SSL https based CDN usage at US$250/month for std SSL and US$600/month for wildcard SSL and most expensive with monthly contracts but performance probably the best or on par with Amazon Cloudfront for non-https at least.

    Also I use CDN77 as well :)

    I'm also developing my own private SPDY/HTTP/2 based CDN/proxy for centminmod.com usage as well using Centmin Mod LEMP stack's Nginx to power it so I have full control :) CentminMod.com deployment in geodns cluster is the first part of it :)
     
    Last edited: Sep 8, 2015
    • Informative Informative x 1
  3. RoldanLT

    RoldanLT Well-Known Member

    3,901
    949
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,298
    Local Time:
    6:16 AM
    1.11
    10.2
    If your script/web app is capable of uploading your static files on different server, then that would be easy.
    I mean attachment uploaded by users, avatars.
     
  4. pamamolf

    pamamolf Well-Known Member

    2,721
    242
    63
    May 31, 2014
    Ratings:
    +433
    Local Time:
    1:16 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    Yes my script is able to use direct cdn uploads from users so when they upload an avatar it can store straight to the cdn.....

    At the moment i do not use https but i will next year so it must be easy to configure it on CDN later....

    So a push based cdn is what i need if i read correct your info :)

    I will try the Amazon Cloudfront and the KeyCDN :)
     
  5. RoldanLT

    RoldanLT Well-Known Member

    3,901
    949
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,298
    Local Time:
    6:16 AM
    1.11
    10.2
    For cheaper way, you can use separate VPS on SSD with only Nginx on it.
    yum install nginx -y
     
    • Like Like x 1
  6. pamamolf

    pamamolf Well-Known Member

    2,721
    242
    63
    May 31, 2014
    Ratings:
    +433
    Local Time:
    1:16 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    Creating my own cdn?

    Don't know how to configure it :(
     
  7. RoldanLT

    RoldanLT Well-Known Member

    3,901
    949
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,298
    Local Time:
    6:16 AM
    1.11
    10.2
    Not exactly your own CDN, but plain server only use for serving static files.
     
  8. pamamolf

    pamamolf Well-Known Member

    2,721
    242
    63
    May 31, 2014
    Ratings:
    +433
    Local Time:
    1:16 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    But maybe i can set it up from my script as i can do it with the above cdn's by just changing the path for remote images :)
     
  9. RoldanLT

    RoldanLT Well-Known Member

    3,901
    949
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,298
    Local Time:
    6:16 AM
    1.11
    10.2
    But costly.
     
  10. pamamolf

    pamamolf Well-Known Member

    2,721
    242
    63
    May 31, 2014
    Ratings:
    +433
    Local Time:
    1:16 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    George do you use for your cluster MariaDB galera cluster and GlusterFS ?
     
  11. eva2000

    eva2000 Administrator Staff Member

    30,156
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    8:16 AM
    Nginx 1.13.x
    MariaDB 5.5
    GlusterFS is only one I have experience with out of the 2. For centminmod.com it's just a Git backed repository which allows easier update/syncing for files. For CDN it will be pull/proxy based so no syncing needed.
     
  12. pamamolf

    pamamolf Well-Known Member

    2,721
    242
    63
    May 31, 2014
    Ratings:
    +433
    Local Time:
    1:16 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    Not so clear answer on what you use for your cluster for syncing files and database clustering :)
     
  13. RoldanLT

    RoldanLT Well-Known Member

    3,901
    949
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,298
    Local Time:
    6:16 AM
    1.11
    10.2
    WoW, so I just know some technology today:D
    That GIT can be use for Cluster Server :D
    Didn't think of that is possible :D
    Thanks Eva!
     
  14. eva2000

    eva2000 Administrator Staff Member

    30,156
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    8:16 AM
    Nginx 1.13.x
    MariaDB 5.5
    GlusterFS/NFS and also csync2 and lsyncd and MySQL replication or Tungsten Replicator Replication
    Well it's easy for files only which aren't user generated like centminmod.com - use gitlab.com and not github.com has they have large file size support and gitlab.com offers free private and public git repos GitLab.com | GitLab
     
    Last edited: Sep 8, 2015
    • Informative Informative x 2
  15. pamamolf

    pamamolf Well-Known Member

    2,721
    242
    63
    May 31, 2014
    Ratings:
    +433
    Local Time:
    1:16 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    Add the power to Centminmod lol :)

    Is it very important to use ssd's on the remote images server as they will be limited by the Network speed transfer?
     
  16. eva2000

    eva2000 Administrator Staff Member

    30,156
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    8:16 AM
    Nginx 1.13.x
    MariaDB 5.5
    SSDs help in terms of disk latency access times. But you'd be limited by both network speed and number of cpu cores/threads you have available if you're hosting it yourself for CDN. Reason why I am looking into 10Gbps network connections too :D
     
  17. pamamolf

    pamamolf Well-Known Member

    2,721
    242
    63
    May 31, 2014
    Ratings:
    +433
    Local Time:
    1:16 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    I am thinking the scenario that RoldanLT recommend me to get a dedicated server and just host there the images....

    Ssd's are more expensive and i am not so sure if they really help for such scenario....
     
  18. eva2000

    eva2000 Administrator Staff Member

    30,156
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    8:16 AM
    Nginx 1.13.x
    MariaDB 5.5
    Yeah or just get a better server :)
     
  19. pamamolf

    pamamolf Well-Known Member

    2,721
    242
    63
    May 31, 2014
    Ratings:
    +433
    Local Time:
    1:16 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    Getting a better server and pay more than 100 euro extra just for bigger ssd's as now the server is more than enough maybe is not the best solution.... :)
     
  20. RoldanLT

    RoldanLT Well-Known Member

    3,901
    949
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,298
    Local Time:
    6:16 AM
    1.11
    10.2