Handling Centmin Mod High Availability issues on AWS (and other CSPs)

I am reposting the ff as per an admin suggestion:
*****************

Hello People. New guy here.

I keep on hitting the same challenges which I suspect you guys share. A client requires an HA (say WP for discussion here) on AWS ( I am sure the discussion is also relevant to GCP, Azure and the rest). They also want highly performant stacks.

---

I build them suitable AWS infrastructure/platform for their solution and use premade stacks (say LEMP ) or even with baked-in apps (LEMP + app). All is good. The problem is their LEMP stacks are not as performant as CENTMIN MOD. I have been spying on you guys the last couple of years and testing your stack which keeps outperforming the others but it's not suitable for HA use.

I am done spying on the sideline and I finally decided to come here and bring this particular aspect to your attention. Slight changes in the configuration can make CENTMIN MOD be suitable for cloud applications. It's not like it cannot be used on the cloud, but making it work is a PITA

There are few points to keep in mind. The LEMP resides on VMs/EC2 in an autoscaling group that feeds off a load balancer. EC2s pop in and out of existence according to demand. SSLs typically get terminated at the LB. EC2s remain in private networks (VPC) with their SG accepting inbound only from the LB. Hence, the SSL, firewall etc configurations are not as relevant (except for the SSL, to forward the header to the app downstream so that it knows the users are requesting over https: or else, the app will construct a response URL over HTTP and the end-users' browser throw a feat and claim there is unsafe script.

Focusing on the autoscaling group, as demand increases, new EC2 is spun. Installing the CENTMIN MOD at boot is out of the question as it takes too long. Hence, the only viable solution I see (assuming immediacy is a must) is to install the stack, configure whatever app is necessary, then bake a golden EC2 Image (AMI). Then spin that on and configure the fewest details during boot. The problem with such setup, however, is that some machine info is hardwired during setup.

And that is what I want to bring to your attention. Why not make it configurable with init? Thus far I notice host names, host IP are hardwired say for WP configuration. The WP on CENTMIN MOD is super fast but unfortunately not easy to make HA. True it's possible to write a script at init and alter the configurations but not clear how many of these configurations need to change as I am not aware of what other info is hardwired.

As a general principle, the stack - I understand it should be optimized for a specific type of machine - but this does not mean it needs to be aware of the particular identity of the machine or the env. As the cliche goes, the servers need to be the cattle, not the pet, and the stack can be performant for a particular breed of machines (the cattle) than a specific machine we tend to care for (the pet).

If only you guys can do that. I am glad to provide free consulting with regard to the AWS env ( I have spent 8K + hours on it and while the ecosystem is too complex for one brain to master, I am quite familiar with it) as I am interested to make it work on AWS hassle-free. Like I said the same concept extends to other CSPs. So any CENTMIN MOD expert interested in this endeavor?

Thanks for reading

 

Thanks, @eva2000

packer.io probably is a better abstraction for working with diff CSPs as far as configuring or creating the image. I am not sure though if it brings anything to the table when it comes to dealing with the issues pertinent to the HA & Scaling challenges. Here I identify what I think are the core issues that need attention.

There are many concerns that need to be addressed. Some we can tackle as a generic issue while others can be handled by end-users per their specific use cases and requirements.

I suggest we create a base image ( I can create few images for popular varieties on AWS) with an optimized LEMP stack as a community AMI. End users can then tailor this AMI to their own use case and create their own golden Image to suit their purposes.

Here are the core issues that need attention when it comes to HA & elasticity issues.

1. The stack is optimized for specific instance types. The most important thing to note there that it is optimized for a specific variety. However, it need not make any assumptions about the hostname or any peculiar info.
   
   
2. There is a need for a shared file system or EC2s that pop in and out have no way to share their state. On AWS, a viable shared file system is EFS (it's NFS: running your own NFS solution is a headache). I note here three things.
   1. One is that not all apps can work this way but many popular ones can.
   2. Another is that Nginx and as much of the stack as possible need to run on the instance. (first off, EFS is just a file server and nothing can execute on it so everything has to be fetched to the instance: and that is slow. Caching can mitigate some of these issues but that is tangential). So the reason I bring EFS up is that Nginx, while it runs on the instance, needs to account for EFS mounting-the mounting can be configured with fstab but the Nnginx needs to be configured to work with mounted folders. This perhaps is an app-level concern but just throwing it out there.
   3. Some prefer to use their own sync/replication solutions. It can certainly be the only way for apps that can't work on NFS but these are the minority of use cases and I am ignoring the problem at all. (Something that can be addressed when the need arise).
   4. For specific use cases, S3 can be used. This is also something that can be addressed at a later point.
      
3. There is also a need for a shared database. On AWS, the RDBMS service is RDS. Some apps are a better fit for DynamoDB. A similar argument can be made as above when it comes to the RDBMS. Some create their own replication solution but there already is a baked-in solution in RDS. I am not aware of what kind of optimization you do for MySQL. If any, it would be redundant. The issue to address here is to make the stack configurable such that connection params (that allow the instances to connect to RDS) can be accepted. I believe the stack still needs the mysql driver for connection.
   
   
4. SSL is typically terminated (some compliance regimens may require secure backend transport: but rare) at the load balancer. Note that the use of an LB is imperative for Autoscaling Groups (an essential feature for HA + dynamic scaling). In which case, the Nginx and whatever it proxies downstream need to be aware that they are sitting behind an SSL terminated LB in order to construct the proper URL. Or else, whatever the app extracts from the header would be HTTP (not HTTPS). Nowadays, almost all client-facing requests come over https (and the LB is configured to redirect all HTTP to HTTPS. Nginx can handle this easily. Generally, well-developed apps can also handle this. They would have a setting file to configure to let them construct their URLs properly. The Nginx would just need to proxy the proper header (X_Forwarded_Proto..)
   
   
5. The Security Group (SG) and Network Access Control List (NACL) are tasked with controlling traffic from and into the instance and the subnet the instances reside. The CSF configuration, I suspect, becomes redundant. All blacklisting/whitelisting with specific ports, IP, etc need to be handled by the SG & NACL. In fact, the best practice setup is to terminate front facing traffic at the LB and the instances remain in private networks behind the LB, only allowing traffic from the SG of the LB at specific ports. Everything else is basically blocked. Having said this, if there is any service that actually whitelist/blacklist based on DNS, then that setting can be useful as SG/NACL function at the network layer (won't resolve DNS).
   
   
6. Memcache /Redis settings. In the typical production-setup, the cache doesn't exist in the RAM of the instance but rather, on its own dedicated specialized server called ElastiCache on AWS. These are RAM optimized instances with endpoints (addressable by DNS) , super-fast (ping @< ms) serving Memcache/Redis. They can also be made redundant and more available. Scaling is also possible. Most importantly, they have shared caches so diff number of instances can share them.
   
   
7. Other issues I failed to mention? These were what came to the top of my mind.

Summary:
1) optimize for specific machine type
2) account for shared file systems (EFS mount points)
3) account for shared database system (RDS endpoints)
4) proxy header info for proper URL construction downstream
5) modify or nullify CSF (use SG and NACL: instead - to be configured by end-user
6) account for caching endpoint (than embed cache)

Note: I can handle all the entailed AWS related tasks. I just need help with the Centmin Mod side.

 

Thanks again @eva2000.

Few points:

1. Please understand that I am not asking that you support AWS. I gather that you got an open-source LEMP stack that runs in any (CSP & web hosting providers) env. Asking you to tailor it to a specific vendor's env, especially a walled garden like AWS and one that is not particularly the destination for your target end-users, would not make sense. It would be wrong as well. I am actually the one who is trying to do that. Which is why I am here trying to get some help and direction.
   

1. Speaking of your target market, I realize that you got budget-conscious end-users in mind. It is not only the big corps who are on the cloud though. Turns out that a lot of small to medium businesses (including moms & pops stores) are moving to the cloud lately. They may not have the dedicated crew to build and maintain their own env, but still, they find it worth the headache to move to the cloud as many are following the trends and have started running an aggressive social media & email marketing. And traditional web hosting providers don't cut it as they experience a highly variable, sometimes global traffic. They also seek modern business services such as bot chats, notifications so forth. CSPs provide more suitable env in this regard though they come at a premium. Yes, bandwidth seems expensive but customers get reliable bandwidth. They also achieve high availability and high elasticity in addition to running a more fault-tolerant system. DR is a breath. They get to pay only for what they use. There are hundreds of advanced business services they can subscribe to on an on-demand basis as well. Most small businesses, in fact, save a lot of money by going to the cloud assuming they can also build a business to justify their expansion as they increase their capacity and capability significantly. Also, whether the cloud is desirable env or not is a matter of perspective: Do they see IT as an essential resource or a strategic capability? I can go on and on, and make arguments for the benefits (and of course there is a cost and complexity trade-off).
   
2. Having said that, I am not arguing the plus and minus of going to the cloud. But I want to point out that those your target can also benefit immensely from the Centmin Mod in the cloud env (it's possible to use it but making it work in the HA & elastic env is not trivial as I discussed in the original post). I also noted that, for open-sourced LEMP stacks, I have found Centmin Mod to be among the best performers (not that I am the expert in the matter). So I thought I will tailor it to AWS. But then why AWS only? It is just one vendor. There are many. As I said, I intend to bake in some images for a few popular machines on AWS. Others can take it to GCP, Azure etc. I can host the images on the market as a community image (open + free) or you guys can do it as it's a trivial matter. Others can do the same for other CSPs. My intention is to understand Centmin Mod enough, so as to have few cloud configurable components (not for AWS, but for a generic CSP). On the cloud, most services come as an endpoint. I identified the NFS, the cache, the database endpoints as the main areas of interest. Additionally, I identified the SSL & FW issue. Yes, all these can be disabled or edited or scripted somehow. But I figured it would be awesome if they can be configurable.
   
3. Speaking of supporting EC2, I don't particularly seek that; besides, the stack actually works on EC2. It just needs some configurations, which like you said have a scriptable resolution. I already have done it manually ( and it's a trivial matter to script the solutions) but haven't used it in production b/c I wasn't sure that I have identified all the necessary configurations. I also could have invested time and accommodate all but not sure how it would have survived updates. So, I did find the effort worthy. Here is actually how I get to do it:
   1. Centmin Mod works comfortably on EC2. It just hardwires host, which I edit
   1. Mounting at boot is trivial. As I said, this is an app-level issue: for WP it's where centmin.sh installs the root. Maybe it's possible to edit that. I simply copy the public folder, mount EFS on the public folder and put the copied files in it. I basically pull the rug under its feet.
   2. The same goes for RDS. For WP, for instance, it's trivial to edit the wp-config.php. I turn off the CSF and use SGs & NACLs. SSL can be fixed for WP at wp-config.php.
   3. I haven't tried the caching though. I instead used whatever was installed. Which leads me to the crux of what I am trying to say

1. Here is what I am trying to get to: How do we make Centmin Mod more generically cloud-friendly?
   1. What are the settings that particularly make installation & optimization coupled to the particular machine? (mind you, I am not talking about types like CPU count, RAM or even OS, etc. Let's assume we got a machine image with a given configuration of OS, RAM, CPU, etc. I already realize that the installation hardcode the host IP --this, of course, can be replaced with grep or other scripting solution, but what other info does it hardcode? This is one thing I am not sure about.
   2. For WP, how about making the root folder configurable for NFS endpoint? Same things for the endpoints for database and caching
   1. How can I prevent SSL & CSF configurations from running in the first place (than turns off after the fact?). I say this because part of these configurations cost time and can cost minutes.

Now, I am not too self centered or too naïve to ask that you make a new version to accommodate these changes. But rather a pointer to guide me in say writing a remediating script that can be added to the installation so as say to skip on CSF & SSL, account for endpoints, and inject particular server info. Hopefully, I can make it generic enough that others can extend it to other env.



Does it make sense now? Sorry if I was being vague. I guess you question is helping me articulate it better : )

 

Thanks for the response @eva2000.

- I will look into the centminfinish function. I also am aware that SSL configuration can be skipped. I was just throwing it in there along with the CSF, which can also be turned off. What is ideal, is to actually not have the components run in the first place as the idea is to save the time of installation. So something of an equivalent to a centminstart (though I realize this may not exist) would have been nicer. But this is just a minor issue.

- A major issue is identifying how the configuration/installation/optimization gets 'coupled' to a particular VM. As I said, I have identified the host IP as one. What else should I look into? Is there documentation I can look into? This way, I can bake an image for a given instance configuration (given CPU, OS, RAM, etc. ). I can do this for some popular instance types ( say t3.small, t3.medium, t3Large, etc.). An end-user can pick a machine type and create a launch configuration (or template) that an autoscaling group can use to spin up instances as necessary. Every time an instance is launched, it only needs to account for these (yet unidentified and what I am after) peculiarities. If all works out well, it's also possible to create a job that can update these images on a regular basis churning out updated images.

- Another issue is the endpoints (cache & RDS). Once again, these can be done at the app-level. However, I am curious to know if it's possible to somehow integrate this into the stack. First off, this also cut out time. Secondly, it's also a matter of integration. As I said, running a cache in ElastiCache is much more efficient and cost-effective. I am not sure what can be said with regard to the MySQL/MariaDB.

Speaking of time, I keep focusing on it for a couple of reasons. For starter, for specific applications, the very latest stack can be built on the fly at launch (without waiting for an upgrade). This only make sense if the installation and optimization can run quickly. Secondly, the job I refer to can be run in Lambda if the process takes less than 15 minutes. That would significantly simplify things. Related to the time is the pursuit of minimalism (for security and complexity reasons than size).

What do you think? Thanks for your time.

 

Thanks @eva2000. Sorry I was a bit busy.

As far as my interest goes, I will look into the stack more carefully on my next project now you tell me that there is nothing in particular that tie the installation to a particular IP.

As far as my general idea is concerned, however, you make me realize a serious flaw in my thinking. I guess I am basically trying to bake an image which only those who have fundamental grasp of the Cloud env & ops can use and operate. These are probably the very people who don't need it. Duh.

At any rate, I really appreciate the good work you do, your time & patience.

Let me know if there is anything I can help (I mean for free, not looking for a business) with if you guys ever embark on an AWS project. I believe my account here is tied to my email.

Thanks!