/ Register

Want more timely Centmin Mod News Updates?
Become a Member

Sysadmin Best solution for hosting images and expand? Use Amazon S3 or custom build like Glusterfs ?

Discussion in 'System Administration' started by pamamolf, Dec 22, 2018.

Previous Thread Next Thread
Loading...
  1. Dec 22, 2018 #1
    pamamolf

    pamamolf Premium Member Premium Member

    3,656
    353
    83
    May 31, 2014
    Ratings:
    +679
    Local Time:
    4:47 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    Hello

    On my site i have around 1TB of images and i am using Cloudflare in front but it seems that size increasing very fast and may be not a great solution to continue using one server with many disks for that ....

    Images increasing around 300Gb per year ... My monthly traffic for images (includes some static files also like css, html) from Cloudflare is around 10TB per month...

    I was thinking to use Amazon S3 but it may be expensive so i am checking also some other solutions like Glusterfs .....

    What's your opinion?

    Thank you
     
  2. Dec 22, 2018 #2
    eva2000

    eva2000 Administrator Staff Member

    42,743
    9,667
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,920
    Local Time:
    12:47 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Find a solutions provider who is a Cloudflare Bandwidth Alliance partner to get free bandwidth between the provider and Cloudflare or reduced cost bandwidth https://www.cloudflare.com/bandwidth-alliance/. Not all listed Bandwidth Alliance partners currently have actively deployed free or reduced cost bandwidth pricing yet so you need to confirm with Cloudflare and the partner first i.e. Linode and DigitalOcean currently are partners but free/reduced bandwidth with Cloudflare is not active yet - they're still working out the details. Amazon S3 isn't a Cloudflare Bandwidth Alliance partnet though. But BackBlaze is https://www.cloudflare.com/bandwidth-alliance/backblaze/

    FYI, S3 has difference storage classes so storage cost can be between US$0.01/GB to US$0.023/GB https://aws.amazon.com/s3/pricing/
     
    • Informative Informative x 1
  3. Dec 26, 2018 #3
    minodd

    minodd New Member

    13
    1
    3
    Nov 29, 2018
    Ratings:
    +2
    Local Time:
    10:47 PM
    Nginx 1.15.6
    MariaDB 10
    The disadvantage with BackBlaze B2 is, I can't find a solution to prevent hotlinking. Yes, we can use hotlink protection on the Cloudflare side. However, it's really easy to find out the actual B2 address of the file. Backblaze provide nothing like referer settings to stop hotlinking.
     
    • Informative Informative x 1
  4. Dec 26, 2018 #4
    pamamolf

    pamamolf Premium Member Premium Member

    3,656
    353
    83
    May 31, 2014
    Ratings:
    +679
    Local Time:
    4:47 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    I am checking also Wasabi as it is S3 compatible and it may be easier and it is a bit cheaper....
     
  5. Dec 26, 2018 #5
    JJC84

    JJC84 Ad astra per aspera Premium Member

    247
    109
    43
    Jan 31, 2018
    San Antonio, Texas
    Ratings:
    +167
    Local Time:
    8:47 AM
    1.15.x
    10.x.x
    You might want to try IBM Cloud or Azure as they're similarly priced to Amazon S3 with configurable permissions and are partnered with Cloudflare. But that is just off the top of my head. I'll see if I can get some concrete numbers for you here in the near future.
     
    • Informative Informative x 2
  6. Dec 26, 2018 #6
    eva2000

    eva2000 Administrator Staff Member

    42,743
    9,667
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,920
    Local Time:
    12:47 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Interesting didn't know that
    Ah haven't looked at Wasabi yet so thanks
    Yeah Azure looks more attractive bandwidth pricing wise with Cloudflare Bandwidth Alliance now :)
     
  7. Dec 28, 2018 #7
    eva2000

    eva2000 Administrator Staff Member

    42,743
    9,667
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,920
    Local Time:
    12:47 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    they're an active Cloudflare Bandwidth Alliance member too
     
  8. Dec 28, 2018 #8
    pamamolf

    pamamolf Premium Member Premium Member

    3,656
    353
    83
    May 31, 2014
    Ratings:
    +679
    Local Time:
    4:47 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    Wasabi seems to be compatible with S3 and i tested at my IPB forum as it is compatible also with S3 but it didn't work ...

    Maybe i am doing something wrong or it just don't work...
     
  9. Dec 28, 2018 #9
    Kintaro

    Kintaro Member

    63
    3
    8
    Dec 2, 2016
    Italy
    Ratings:
    +14
    Local Time:
    3:47 PM
    1.15.x
    MariaDB 10
    I used Wasabi trial and I only had to change the target server from S3's to Wasabi's and worked like a charm.

    The only problem with Wasabi is that the minimum payment is 5$/month (and they servers are USA only for now)
     
    • Informative Informative x 1
  10. Dec 28, 2018 #10
    eva2000

    eva2000 Administrator Staff Member

    42,743
    9,667
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,920
    Local Time:
    12:47 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    what do you mean didn't work ?

    seems @Kintaro got it to work though :)
     
  11. Dec 29, 2018 #11
    pamamolf

    pamamolf Premium Member Premium Member

    3,656
    353
    83
    May 31, 2014
    Ratings:
    +679
    Local Time:
    4:47 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    Ok it was a configuration issue :)

    It works now for me also...

    European location will be added at the end of January :)
     
    • Like Like x 1
  12. Dec 29, 2018 #12
    minodd

    minodd New Member

    13
    1
    3
    Nov 29, 2018
    Ratings:
    +2
    Local Time:
    10:47 PM
    Nginx 1.15.6
    MariaDB 10
    Cool, I guess I'll give it a try as well!
     
  13. Dec 30, 2018 #13
    buik

    buik Well-Known Member

    1,225
    329
    83
    Apr 29, 2016
    Ratings:
    +978
    Local Time:
    3:47 PM
    These are concrete solutions to the current problem. The short term.
    What will you do in the longer term?

    Your forum images grow almost 30% per year.

    What I want to advise you is making a good plan.
    Look at the longer term.

    Carefully test more then one solution and implement a solution that can grow with your site, is stable within your environment ( apart from all the beautiful talk from suppliers) and perhaps even more important.

    You forum should still be profitable, including all servers, services, new service to be added and other cost.

    Is it about a money site (forum)?
    If that does not apply. There are other solutions. For example to archive old images.
     
    Last edited: Dec 30, 2018
    • Informative Informative x 1
  14. Jan 17, 2019 #14
    Kintaro

    Kintaro Member

    63
    3
    8
    Dec 2, 2016
    Italy
    Ratings:
    +14
    Local Time:
    3:47 PM
    1.15.x
    MariaDB 10
    I add that is cheaper if you are spending more then 5$ every month (and I think many are) because they charge at least 5$/month
     
    • Informative Informative x 1
  15. May 5, 2019 #15
    Revenge

    Revenge Active Member

    443
    92
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +333
    Local Time:
    2:47 PM
    1.9.x
    10.1.x
    Wasabi only charge for storage space or also for bandwidth usage?

    For example, if i use 1 TB of storage that will generate about 20 TB of bandwidth per month, i will still only pay the 6$/Month?
     
    • Agree Agree x 1
  16. May 5, 2019 #16
    eva2000

    eva2000 Administrator Staff Member

    42,743
    9,667
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,920
    Local Time:
    12:47 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    https://wasabi.com/pricing/ they have cost estimate calculator on their site ;)

    and https://wasabi.com/pricing/pricing-faqs
     
  17. May 5, 2019 #17
    pamamolf

    pamamolf Premium Member Premium Member

    3,656
    353
    83
    May 31, 2014
    Ratings:
    +679
    Local Time:
    4:47 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    They do not charge for bandwidth usage. Yes only 6$ :)
     
  18. Aug 30, 2019 #18
    Kuro

    Kuro Member

    80
    10
    8
    Feb 8, 2018
    Ratings:
    +17
    Local Time:
    9:47 PM
    1.15.10
    10.3
    I don't think so. :unsure:
    upload_2019-8-30_10-33-53.png
     
    Last edited: Aug 30, 2019
    • Informative Informative x 1
  19. Aug 30, 2019 #19
    pamamolf

    pamamolf Premium Member Premium Member

    3,656
    353
    83
    May 31, 2014
    Ratings:
    +679
    Local Time:
    4:47 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    If you use Cloudflare with it, it is free as they have a special agreement...
     
    • Agree Agree x 2
Previous Thread Next Thread
Loading...

.