Welcome to Centmin Mod Community
Register Now

Sysadmin Is there a Newb based tutorial on clustering servers?

Discussion in 'System Administration' started by ljseals, Dec 29, 2016.

  1. ljseals

    ljseals Member

    88
    22
    8
    Dec 20, 2016
    Ratings:
    +43
    Local Time:
    8:27 AM
    Greetings,
    I am a newb! However, with the help of the Lord I was able to successfully install Centmin mod. I thank the Lord for touching each one of you hearts to help me. However, I am wondering is there a newb based tutorial on how to setup clusters of servers, not only for geo-location but mainly for backup. As an online eCommerce store, I believe that it will be critical to have this setup or similar setup.

    Specifically, the ability to have a master slave and a separate server to mitigate problems down the line. I ask because I believe that this would be a cost effective solution but do not quite yet understand how this would work in practice.

    As a startup, I want to start off my business the right way but as of right now money is sparse, anyone know of some good tutorials for newbs?
     
    • Like Like x 1
  2. ljseals

    ljseals Member

    88
    22
    8
    Dec 20, 2016
    Ratings:
    +43
    Local Time:
    8:27 AM
    However, if that cannot be accomplished how can I use centmin with a failover ip based solution. I pray for the Lord's help in this regard.
     
  3. eva2000

    eva2000 Administrator Staff Member

    29,763
    6,721
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,035
    Local Time:
    11:27 PM
    Nginx 1.13.x
    MariaDB 5.5
    Centmin Mod is provide as is, so short of scripted related bugs or issues, any further optimisation to the web stack components - nginx, php-fpm, mariadb mysql, csf firewall etc or web app specific configurations are left to the Centmin Mod user to deal with. So I do not provide any free support for such.

    However, Centmin Mod users are free to help each other out and ask questions or give answers on this community forum. My hopes are that this community forum evolves so that more veteran long time Centmin Mod users help new Centmin Mod users out :)

    Unfortunately, noob and cluster high availability setup don't go together. Clustered setup is advanced configuration that requires practice and reading documentation to understand all moving parts (google-fu ;)) and then understanding what can go wrong operational wise in clusters for web, php-fpm and mysql segments and how to fix such issues. I suggest you start with single setup and get familiar with it all first and practice on test servers for clustering
     
  4. ljseals

    ljseals Member

    88
    22
    8
    Dec 20, 2016
    Ratings:
    +43
    Local Time:
    8:27 AM
    Thanks, but I believe that through the Lord's help that I can do it. Any resources, I guess basically needing two servers not for load balancing as much as for backup just in case one server goes down as the software I use is known to glitch.
     
  5. eva2000

    eva2000 Administrator Staff Member

    29,763
    6,721
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,035
    Local Time:
    11:27 PM
    Nginx 1.13.x
    MariaDB 5.5
    i wouldn't use software that is known to glitch especially for ecommerce site !

    terminology wise you mean if site is down, the backup copy comes online so the site isn't down ? that is still load balancing with high availability and is a complicated beast to configure even for a seasoned system admin professional. As you're still new to linux, i'd start smaller with single server and test vps server + test ecommerce site until you are familiar with it all both linux/hosting and system admin wise and ecommerce software wise.
     
  6. ljseals

    ljseals Member

    88
    22
    8
    Dec 20, 2016
    Ratings:
    +43
    Local Time:
    8:27 AM
    Well for the price it has very top level modules that I need to effectively lists products. For instance, they have a scanner module that I can scanner the barcodes and it will do a webscape of amazon and other websites. That alone it worth it. Not to mention that I have a significant amount of money invested in it already.

    The problem with most software it that it is hosted by the company. This is an open source software that I can host on my on website and it is pci compliant. Seeing that I have had problems in the past with people overseeing my data, or should I say limiting access to my data, I would rather do it all in house. That is why I prayed about it and that is why, I am here.

    X-cart is having a lot of bug issues because they are trying to fit a bunch of modules to work together with one another with a seamless install for the end user. However, up until this time I have been developing my site and honestly it has been frustrating the amount of error and crashes that has occurred. So much so that I want to protect the database even from myself entering products because it can glitch and cause the all the products to be lost.

    The staff is helpful and shows a lot of love to me and other customers so I guess it also has to do with loyalty as they will go out of their way to help. At this point, I have to start over because there were so many glitches and the database of products and category items that I have are corrupted so when I place them inside of the cart they are not corrected structured in the program thus giving even more errors.

    The software is very good but they just need to work out the kinks, it will be fine over time but in the meantime it is crucial to backup things that will save me from entering products and categories over again from scratch. That is frustrating; especially when you have the items in the excel format but know that they will not transpose to x-cart correctly.

    So while I know what you will think, possibly what you will say but for the price, I can not beat it anywhere. This type of software would cost me multitudes more at other vendors and I have very little money. I love your website as I can read and take action. It has a database of actionable steps anyone can follow in order to have a fast and functioning server. I do not mind reading. Honestly, I read and the steps you take are actionable and you present them in an honest way.

    For example, this website is great in the search engines because it is optimized to meet google standards. That is why I can create a forum talking about X-Cart or server and it jumps on the first page of Google. So I am not a fool, I see what you are doing and I take actionable steps to recreate it (What you are doing); while looking at my competitors who seemingly have little to no knowledge that this even exists in terms of server/website optimization. I thank you for the website but will have to implement a strategy before I enter 1000's of products in the database. I hope you understand. God bless you!
     
  7. SFLC

    SFLC Active Member

    224
    59
    28
    Dec 4, 2016
    The Canadas
    Ratings:
    +112
    Local Time:
    3:27 PM
    1
    10
    @ljseals I'm no expert on this subject, however, if I understood what you said, maybe you should try dns round robin, as for backup solutions, I believe there's a few posts on that subject on this forum, so you'll find those of help. For load balancing, that's fairly complicated and depends on many factors that would be different from person to person.

    Depending on the specs of your server environment, and the amount of traffic / footprint it leaves on your setup, which I'm deducing from your posts may be limited, it may be easier for you to upgrade to a more powerful machine than to cluster a few lower performance ones. It will of course cost more, however, will be a lot easier from a short-term perspective. For example, I can go buy 500 playstations, spend the next 6 months figuring out how to cluster them, or I can just go rent a cray.

    Also, theres no shame in starting small, Amazon didn't start off with a multi-million dollar setup, theres no need to have machines capable of handling millions of customers when you only get a few thousand.

    Anyways, thats my 2 cents.
     
    • Like Like x 1
  8. ljseals

    ljseals Member

    88
    22
    8
    Dec 20, 2016
    Ratings:
    +43
    Local Time:
    8:27 AM
    Thanks @SFLC what I expect is to have a fully backed solution that will be dependable. I start small as I am still developing the site but will increase as traffic does. I expect to have tremendous amount of traffic to my site but will upgrade accordingly. I guess I am not making it clear that I am not concerned as much about load balancing as much as not having a fatal error crash my site.

    Just imagined you have a website with over 30,000 products that is also used as you retail pos system. If it goes down you pretty much have lost your business. So while I cannot say conclusively what setup will be the best for me at this time. I believe that it is most important to have the framework now established so I do not have problems in the future. Thus I would like to have it set up now and if traffic surpasses the threshold then it would be easy for me to upgrade to a new server; however, if I go barebones now without even looking at the possibility of backing up my servers and website at an enterprise level, I might as well not go into business.

    Thus, I need to setup a fool-proof setup that will be future upgrade-able that I can do in house. My thoughts are:
    • Daily backup of mysql database (Taken off site and sent to two hubic accounts)
    • Daily backup of mysql database and files by X-Cart software (Taken off site and sent to two hubic accounts)
    • Two servers with master/master or master/slave that has two hour delay just in case first server goes down and also for team to upload products without touching live site.
    • Dev store to upgrade the modules and make sure things are working and any html or customizations needed to the site go without fail.

    I believe that this setup will be beneficial and help to start and grow my business to be the top in its category. I may have to look at best practices with an eCommerce store but as a store who sells one off products and has a very high turnover the store will have constant updates of products and inventory. Thanks for you reply. God bless you!
     
  9. SFLC

    SFLC Active Member

    224
    59
    28
    Dec 4, 2016
    The Canadas
    Ratings:
    +112
    Local Time:
    3:27 PM
    1
    10
    Your plan makes perfect sense @ljseals

    1 and 2, can be resolved by using google cloud services, your sql db will be remote and you can set up your backup options to run automatically, also this alleviates a lot of server load.

    3, I believe you can use rsync to sync your servers, im no expert on this but you can always google this.

    4, I think the aforementioned should take care of this.

    Also, you can setup the google sdk on centos then install gfuse, and mount a google cloud bucket as a "local" directory, which you can use for anything you'd like, backup storage for your html files would be for your case.

    I wish you the best of luck, I think you have a solid plan and sound judgment, I have no doubt your undertaking will be a success.
     
    • Like Like x 1
  10. ljseals

    ljseals Member

    88
    22
    8
    Dec 20, 2016
    Ratings:
    +43
    Local Time:
    8:27 AM
    For anyone inquiring, I have researched several options and by going with CRM (sales and customer data), PIM (Product information management), ERP (Enterprise resource planning) and it will allow me to have my product database and other things on another server without directly affecting the main site.

    The names of the programs are:
    Magento Magento Connect - The Magento Extension Marketplace
    OROCRM https://www.orocrm.com
    Akeneo Akeneo the Open Source PIM
    and maybe an erp...

    So instead of having two servers with replicating data, I will have a database of my products on another server that updates via api. The above software is open source. Although, I am having issues with Centmin Mod with the CRM and PIM, I believe that the Lord will take care of it. This is just an update if anyone else were unfamiliar with these programs as I was. God bless you!
     
    • Informative Informative x 2
  11. eva2000

    eva2000 Administrator Staff Member

    29,763
    6,721
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,035
    Local Time:
    11:27 PM
    Nginx 1.13.x
    MariaDB 5.5
  12. eva2000

    eva2000 Administrator Staff Member

    29,763
    6,721
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,035
    Local Time:
    11:27 PM
    Nginx 1.13.x
    MariaDB 5.5
  13. ljseals

    ljseals Member

    88
    22
    8
    Dec 20, 2016
    Ratings:
    +43
    Local Time:
    8:27 AM
    Well, I have it installed it just not loading properly. Just have to troubleshoot it.. Takes some time. Thanks...:)
     
  14. Colin

    Colin Premium Member Premium Member

    111
    33
    28
    Oct 7, 2015
    Sheffield UK
    Ratings:
    +83
    Local Time:
    2:27 PM
    1.11.#
    MariaDB 10.1.#
    The missing logic/calculations is downtime versus revenue.

    Thus if your ecommerce store/business services are down, whats the mean time from disaster to recovery?

    If it's a day, what value is a day to your trading position? If it's hundreds or if it's thousands.

    If you assume one box might fail for 1 hour in every month, you can cost if it's worth protecting. If you split to 2 boxes and again assume each box will fail, you have just doubled your failure risk.

    I worked for a company that had a ecommerce platform on 2 servers with a 12hour recovery. That platform generated around 11million pounds revenue a year, or roughly £1,255 an hour avg. A 12hour outage then would potentially see lost revenue on avg numbers of £15,060. Personally I don't think it was 12hours, more like 24 as the dev's didn't have a 24/7 support contract.

    They were lucky in that it didn't go down; while I was there, however there was no appetite to mitigate.

    Another site was in travel, and they did considerably more revenue and most of the servers were being run ragged as it was using ruby on rails ;) So we had lots, all physical, pre-cloud embracing, so cater for the peaks... The only problem is we had big spofs ( Single Points Of Failure ) in the database. The cost to remedy that wasn't acceptable. The dba quit.

    So with that in mind, the holy grail post 5 nines, grab a coffee > http://www.edgeblog.net/2007/in-search-of-five-9s/

    Beware this is a near bottomless rabbit hole...one of my ex CTO's had a huge bladed HA solution in each dc, over 3 dc's and it still failed thanks to routine maintenance & a digger big bucket.

    A minimal solution, you still need backup etc, would be 3 servers[A,B,C], running maria cluster for data on a,b,c. Onto two of the servers, a,b load your nginx /web assets. The third, c I'd leave out of the public, that would be your db/dumper/splitbrainer etc. The two web facing, a,b, can either be done with DNS IP, not quick or put a floating ip + loadbalancer x2 in place...and now you're well on your way to the bottomless rabbit hole.

    I'd also if possible split the functions to different web pairs, to the crm, erp etc. Especially if they have different workload requirements. So measure your segments and apply accordingly.


    The loadbalancer is a handy concept to build, as you can then do blue/green deployments to a, if it's ok, roll to b. If it fails rollback a. i.e. you always have one up! Ideal world I'd not have the web surface on the db surface, but then you need another two boxes. If handling cards, you need to thin about PCI compliance or it's equivalent and that's 'private' network security too.

    The two hard things; well known, the third, scaling beyond one server ;)

    Now jump on the 8hour flight and get someone to turn something off ala chaos monkey.

    Get it in writing from your directors or yourself, that solutionX is enough and you're happy the cost v downtime calc's, then print it out and put in the red DR Folder. :)

    Good luck!

    Rackspace Magento Article: http://c744563d32d0468a7cf1-2fe04d8...entoEcommerce ArchitecturesWhitepaper2013.pdf
     
    Last edited: Feb 21, 2017
    • Informative Informative x 2
  15. ljseals

    ljseals Member

    88
    22
    8
    Dec 20, 2016
    Ratings:
    +43
    Local Time:
    8:27 AM
    Thanks @Colin,

    I will take that into consideration but you must understand that my depth of understanding of system admin is very limited. I took a class in high school I believe in DOS, maybe in 1996 or so. Through the Lord's help I can catch on quick but I am not able to setup a complex setup at this time.

    As far as downtime versus revenue, I have made a backup script that backs up the database and the the file structure on a daily, weekly, and monthly basis. I will consider writing a script that will be hourly that will delete every hour but I now only have a script to backup the site and database manually when I make changes. I have a simple script that does not work but will send the files to two separate hubics at different intervals. It works manually but the problem with the script is either the cron is wrong or I have a simple error but I have not found it.
    Script
    Code:
    #!/bin/bash
    /usr/bin/rclone copy /home/nginx/domains/domain/backup hubic1:default/Backups
    Cron
    Code:
    00 6 * * * /bash /root/tools/rclonecopya.sh && curl -fsS --retry 3 https://hchk.io/217dbf3b-cc62-43f9-b63c-xxxxxxxxxxxx > /dev/null
    
    The pim and crm will be needed to take risks off the main site as I will have many barcodes and single products with high turnover. I will just have these programs backed up with the database only.

    If I need to restore the website from backup it will take me about 15 minutes but if I have to renew the OS it would take considerable longer but around 1 hour.

    What I am facing now is that the pim and crm seemingly is not compatible as Akeneo needs a lamp server and I have been trying to hack it to make it work on nginx; unsuccessfully. There is a docker image GitHub - theiconic/docker-nginx-akeneo: Akeneo PIM docker using Nginx but I do not have the ability to contain that in centmin mod at this time.

    I have time but I do not have the money and I have had very bad experiences with outsourcing. Some of these programs can be utilized on webhosting platforms that automatically install but being at the hands of a middle man host is something I am not likely to go back to.

    So due to my knowledge as a system admin I will have to learn to do these things but having three servers would have multiple points of failure, if I am not able to manage them. So at this point, I have to keep it simple but needing an enterprise solution if that makes sense. God bless you!
     
    Last edited: Feb 21, 2017
    • Informative Informative x 1
  16. Colin

    Colin Premium Member Premium Member

    111
    33
    28
    Oct 7, 2015
    Sheffield UK
    Ratings:
    +83
    Local Time:
    2:27 PM
    1.11.#
    MariaDB 10.1.#
    I do, did and have. It's as much a heads up, attention this way be dragons. I, well not quite, I have one client that uses a DOS epos system my dad wrote and I did a bit on it too. I dread the day he calls... multiuser dos, multi windowed. DOS rocks.

    The thing about complexity is to take it one step at a time, but know where you're going. I don't and hence I shy away from it until I have to! I describe myself as a dangerous sys admin, I know enough to mess it all up and not quite enough to make it right. I tend to operate on philosophy of if the box dies, throw it away over extended hours of head scratching. Tests your backup too ;) Test that before you throw it away mind ;) Can't recall who said it, the last good backup is the last one you restored from. I keep meaning to make a test system restore from the live every 24hours...

    Docker is an even deeper rabbit hole. But it comes with a handy roadmap. So open the dockerfile, there is your map to making it work on nginx/centminmod. It will show the package, and caching it's expecting, e.g. APC. Note the image is alpinelinux, so apk/paths won't work on centos... then; hope I'm not teaching you to suck eggs, use the nginx conf and apply the 'centminmod' changes, e.g. paths.

    I'd do that line by line over trying to get docker in production; logging, monitoring, data persistence, networking and more. Conceptually docker and all the other orchestration tools looks great. I've been using it for a year or more on my dev env, but I've not gone to production with it and I might not ever. Solaris had a similar thing and it scrambled my brain more than once or twice.

    Correct. Keep it simple :D Don't have test on production either. I know it's more cost, but test is test, you're trying to break it purposefully! or worse test a firewall change and kill production. I'd argue having test/practise environments on a different host/account, in case an automated api call goes wrong in testing.

    Good luck with it!

    Thanks to for the CRM link.
    I've been hunting for something halfway decent, that I could host versus a 50+ per month paas... there are only two of us, but spreadsheets and whiteboards aren't doing it.
     
    • Informative Informative x 1