/ Register

Learn about Centmin Mod LEMP Stack today
Become a Member

Best solution for hosting images and expand? Use Amazon S3 or custom build like Glusterfs ?

Discussion in 'System Administration' started by pamamolf, Dec 22, 2018.

Previous Thread Next Thread
Loading...
  1. Dec 22, 2018 #1
    pamamolf

    pamamolf Premium Member Premium Member

    3,156
    304
    83
    May 31, 2014
    Ratings:
    +547
    Local Time:
    11:08 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    Hello

    On my site i have around 1TB of images and i am using Cloudflare in front but it seems that size increasing very fast and may be not a great solution to continue using one server with many disks for that ....

    Images increasing around 300Gb per year ... My monthly traffic for images (includes some static files also like css, html) from Cloudflare is around 10TB per month...

    I was thinking to use Amazon S3 but it may be expensive so i am checking also some other solutions like Glusterfs .....

    What's your opinion?

    Thank you
     
  2. Dec 22, 2018 #2
    eva2000

    eva2000 Administrator Staff Member

    37,710
    8,254
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,710
    Local Time:
    7:08 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Find a solutions provider who is a Cloudflare Bandwidth Alliance partner to get free bandwidth between the provider and Cloudflare or reduced cost bandwidth https://www.cloudflare.com/bandwidth-alliance/. Not all listed Bandwidth Alliance partners currently have actively deployed free or reduced cost bandwidth pricing yet so you need to confirm with Cloudflare and the partner first i.e. Linode and DigitalOcean currently are partners but free/reduced bandwidth with Cloudflare is not active yet - they're still working out the details. Amazon S3 isn't a Cloudflare Bandwidth Alliance partnet though. But BackBlaze is https://www.cloudflare.com/bandwidth-alliance/backblaze/

    FYI, S3 has difference storage classes so storage cost can be between US$0.01/GB to US$0.023/GB https://aws.amazon.com/s3/pricing/
     
    • Informative Informative x 1
  3. Dec 26, 2018 #3
    minodd

    minodd New Member

    10
    1
    3
    Nov 29, 2018
    Ratings:
    +2
    Local Time:
    5:08 PM
    Nginx 1.15.6
    MariaDB 10
    The disadvantage with BackBlaze B2 is, I can't find a solution to prevent hotlinking. Yes, we can use hotlink protection on the Cloudflare side. However, it's really easy to find out the actual B2 address of the file. Backblaze provide nothing like referer settings to stop hotlinking.
     
    • Informative Informative x 1
  4. Dec 26, 2018 #4
    pamamolf

    pamamolf Premium Member Premium Member

    3,156
    304
    83
    May 31, 2014
    Ratings:
    +547
    Local Time:
    11:08 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    I am checking also Wasabi as it is S3 compatible and it may be easier and it is a bit cheaper....
     
  5. Dec 26, 2018 #5
    JJC84

    JJC84 Active Member

    182
    76
    28
    Jan 31, 2018
    Ratings:
    +114
    Local Time:
    3:08 AM
    1.15.x
    10.1.x
    You might want to try IBM Cloud or Azure as they're similarly priced to Amazon S3 with configurable permissions and are partnered with Cloudflare. But that is just off the top of my head. I'll see if I can get some concrete numbers for you here in the near future.
     
    • Informative Informative x 2
  6. Dec 26, 2018 #6
    eva2000

    eva2000 Administrator Staff Member

    37,710
    8,254
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,710
    Local Time:
    7:08 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Interesting didn't know that
    Ah haven't looked at Wasabi yet so thanks
    Yeah Azure looks more attractive bandwidth pricing wise with Cloudflare Bandwidth Alliance now :)
     
  7. Dec 28, 2018 #7
    eva2000

    eva2000 Administrator Staff Member

    37,710
    8,254
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,710
    Local Time:
    7:08 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    they're an active Cloudflare Bandwidth Alliance member too
     
  8. Dec 28, 2018 #8
    pamamolf

    pamamolf Premium Member Premium Member

    3,156
    304
    83
    May 31, 2014
    Ratings:
    +547
    Local Time:
    11:08 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    Wasabi seems to be compatible with S3 and i tested at my IPB forum as it is compatible also with S3 but it didn't work ...

    Maybe i am doing something wrong or it just don't work...
     
  9. Dec 28, 2018 #9
    Kintaro

    Kintaro Member

    59
    2
    8
    Dec 2, 2016
    Ratings:
    +11
    Local Time:
    10:08 AM
    1.9.10
    MariaDB 10
    I used Wasabi trial and I only had to change the target server from S3's to Wasabi's and worked like a charm.

    The only problem with Wasabi is that the minimum payment is 5$/month (and they servers are USA only for now)
     
    • Informative Informative x 1
  10. Dec 28, 2018 #10
    eva2000

    eva2000 Administrator Staff Member

    37,710
    8,254
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,710
    Local Time:
    7:08 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    what do you mean didn't work ?

    seems @Kintaro got it to work though :)
     
  11. Dec 29, 2018 #11
    pamamolf

    pamamolf Premium Member Premium Member

    3,156
    304
    83
    May 31, 2014
    Ratings:
    +547
    Local Time:
    11:08 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    Ok it was a configuration issue :)

    It works now for me also...

    European location will be added at the end of January :)
     
    • Like Like x 1
  12. Dec 29, 2018 #12
    minodd

    minodd New Member

    10
    1
    3
    Nov 29, 2018
    Ratings:
    +2
    Local Time:
    5:08 PM
    Nginx 1.15.6
    MariaDB 10
    Cool, I guess I'll give it a try as well!
     
  13. Dec 30, 2018 #13
    bassie

    bassie Well-Known Member

    1,045
    255
    83
    Apr 29, 2016
    Ratings:
    +748
    Local Time:
    10:08 AM
    These are concrete solutions to the current problem. The short term.
    What will you do in the longer term?

    Your forum images grow almost 30% per year.

    What I want to advise you is making a good plan.
    Look at the longer term.

    Carefully test more then one solution and implement a solution that can grow with your site, is stable within your environment ( apart from all the beautiful talk from suppliers) and perhaps even more important.

    You forum should still be profitable, including all servers, services, new service to be added and other cost.

    Is it about a money site (forum)?
    If that does not apply. There are other solutions. For example to archive old images.
     
    Last edited: Dec 30, 2018
    • Informative Informative x 1
  14. Jan 17, 2019 at 9:32 PM #14
    Kintaro

    Kintaro Member

    59
    2
    8
    Dec 2, 2016
    Ratings:
    +11
    Local Time:
    10:08 AM
    1.9.10
    MariaDB 10
    I add that is cheaper if you are spending more then 5$ every month (and I think many are) because they charge at least 5$/month
     
    • Informative Informative x 1
Previous Thread Next Thread
Loading...
..

.