Welcome to Centmin Mod Community
Register Now

Handling Centmin Mod High Availability issues on AWS (and other CSPs)

Discussion in 'System Administration' started by gooseegg, Nov 5, 2019.

  1. gooseegg

    gooseegg New Member

    7
    0
    1
    Nov 1, 2019
    Ratings:
    +0
    Local Time:
    3:21 PM
    1.17
    10
    I am reposting the ff as per an admin suggestion:
    *****************

    Hello People. New guy here.

    I keep on hitting the same challenges which I suspect you guys share. A client requires an HA (say WP for discussion here) on AWS ( I am sure the discussion is also relevant to GCP, Azure and the rest). They also want highly performant stacks.

    I build them suitable AWS infrastructure/platform for their solution and use premade stacks (say LEMP ) or even with baked-in apps (LEMP + app). All is good. The problem is their LEMP stacks are not as performant as CENTMIN MOD. I have been spying on you guys the last couple of years and testing your stack which keeps outperforming the others but it's not suitable for HA use.

    I am done spying on the sideline and I finally decided to come here and bring this particular aspect to your attention. Slight changes in the configuration can make CENTMIN MOD be suitable for cloud applications. It's not like it cannot be used on the cloud, but making it work is a PITA

    There are few points to keep in mind. The LEMP resides on VMs/EC2 in an autoscaling group that feeds off a load balancer. EC2s pop in and out of existence according to demand. SSLs typically get terminated at the LB. EC2s remain in private networks (VPC) with their SG accepting inbound only from the LB. Hence, the SSL, firewall etc configurations are not as relevant (except for the SSL, to forward the header to the app downstream so that it knows the users are requesting over https: or else, the app will construct a response URL over HTTP and the end-users' browser throw a feat and claim there is unsafe script.

    Focusing on the autoscaling group, as demand increases, new EC2 is spun. Installing the CENTMIN MOD at boot is out of the question as it takes too long. Hence, the only viable solution I see (assuming immediacy is a must) is to install the stack, configure whatever app is necessary, then bake a golden EC2 Image (AMI). Then spin that on and configure the fewest details during boot. The problem with such setup, however, is that some machine info is hardwired during setup.

    And that is what I want to bring to your attention. Why not make it configurable with init? Thus far I notice host names, host IP are hardwired say for WP configuration. The WP on CENTMIN MOD is super fast but unfortunately not easy to make HA. True it's possible to write a script at init and alter the configurations but not clear how many of these configurations need to change as I am not aware of what other info is hardwired.

    As a general principle, the stack - I understand it should be optimized for a specific type of machine - but this does not mean it needs to be aware of the particular identity of the machine or the env. As the cliche goes, the servers need to be the cattle, not the pet, and the stack can be performant for a particular breed of machines (the cattle) than a specific machine we tend to care for (the pet).

    If only you guys can do that. I am glad to provide free consulting with regard to the AWS env ( I have spent 8K + hours on it and while the ecosystem is too complex for one brain to master, I am quite familiar with it) as I am interested to make it work on AWS hassle-free. Like I said the same concept extends to other CSPs. So any CENTMIN MOD expert interested in this endeavor?

    Thanks for reading :)
     
  2. eva2000

    eva2000 Administrator Staff Member

    42,079
    9,499
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,615
    Local Time:
    7:21 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    quoting my other reply

    and post here DigitalOcean - DigitalOcean Marketplace For 1 Click Applications
    So basically when I built my Packer.io Centmin Mod pre-installed image on DigitalOcean images for DigitalOcean 1-click Marketplace compatibility, I configured packer json config file with shell and file provisioners which launches /var/lib/cloud/scripts/per-instance/01-setup-first-login.sh script which configures DigitalOcean's first droplet launch by DO user to launch /opt/centminmod/first-login.sh script will does all the outlined above stuff to properly setup and regenerate passwords/logs, hostnames and re-auto optimise a Droplet based on it's cpu/memory/disk I/O specs.
    Code (Text):
      "provisioners": [
        {
          "type": "shell",
          "inline": [
            "mkdir -p /opt/centminmod"
          ]
        },
        {
          "type": "file",
          "source": "scripts/first-login.sh",
          "destination": "/opt/centminmod/"
        },
        {
          "type": "file",
          "source": "scripts/01-setup-first-login.sh",
          "destination": "/var/lib/cloud/scripts/per-instance/"
        },
        {
          "type": "shell",
          "inline": [
            "chmod +x /opt/centminmod/first-login.sh",
            "chmod +x /var/lib/cloud/scripts/per-instance/01-setup-first-login.sh"
          ]
        },
    

    I haven't used Packer.io for AWS EC2 but it supports it Amazon AMI - Builders - Packer by HashiCorp to build AWS AMI images. Not sure if you've used Packer.io for AWS EC2 ?

    Preview of what first login prompts would look like eventually
    Code (Text):
    ===============================================================================
    * Getting Started Guide - https://centminmod.com/getstarted.html
    * Centmin Mod FAQ - https://centminmod.com/faq.html
    * Centmin Mod Config Files - https://centminmod.com/configfiles.html
    * Change Log - https://centminmod.com/changelog.html
    * Community Forums https://community.centminmod.com  [ << Register ]
    ===============================================================================
    
    
    Below are a number of tasks required to initially setup your server
    
    
    --------------------------------------------------------------------
    Setup Server Administration Email
    Emails will be used for future notification alert features
    --------------------------------------------------------------------
    Hit Enter To Skip...
    Will be prompted everytime run centmin.sh if both emails not entered
    --------------------------------------------------------------------
    enter primary email: [email protected]
    enter secondary email: [email protected]
    --------------------------------------------------------------------
    
    Primary: 1
    setup at /etc/centminmod/email-primary.ini
    
      [email protected]
    
    Secondary: 2
    setup at /etc/centminmod/email-secondary.ini
    
      [email protected]
    
    --------------------------------------------------------------------
    Setup main hostname as per Getting Started Guide Step 1
    https://centminmod.com/getstarted.html
    
    Main hostname is not same as desired site domain name but
    used for where server statistics files get hosted as outlined
    here https://community.centminmod.com/threads/1513/
    
    It's usually something like host.domain.com
    --------------------------------------------------------------------
    
    Enter desired hostname for this VPS: host.do-test.com
    
    
    --------------------------------------------------------------------
    Whitelist IP in CSF Firewall
    --------------------------------------------------------------------
    Adding 45.xxx.xxx.xxx to csf.allow and iptables ACCEPT...
    csf: IPSET adding [45.xxx.xxx.xxx] to set [chain_ALLOW]
    
    
    --------------------------------------------------------------------
    Ensure centmin mod up to date
    --------------------------------------------------------------------
    Saved working directory and index state WIP on 123.09beta01: 303fa79 smarter MARCH_TARGETNATIVE='n' routine in 123.09beta01
    HEAD is now at 303fa79 smarter MARCH_TARGETNATIVE='n' routine in 123.09beta01
    Updating 303fa79..c173666
    Fast-forward
     centmin.sh                |   7 ++++++-
     example/custom_config.inc |   1 +
     inc/shortcuts_install.inc |  22 +++++++++++++---------
     inc/sshd.inc              |   3 +++
     tools/php-systemd.sh      | 190 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
     5 files changed, 213 insertions(+), 10 deletions(-)
     create mode 100644 tools/php-systemd.sh
    
    --------------------------------------------------------------------
    regenerate /etc/ssl/private/pure-ftpd-dhparams.pem
    --------------------------------------------------------------------
    --------------------------------------------------------------------
    regenerating pure-ftpd self-signed ssl certificate
    --------------------------------------------------------------------
    Generating a 1024 bit RSA private key
    ................++++++
    ..............++++++
    writing new private key to '/etc/pki/pure-ftpd/pure-ftpd.pem'
    -----
    --------------------------------------------------------------------
    Memcached Server Admin Login File: /usr/local/nginx/html/memcache_e6b9b3b7c4fe33d9.php
    Memcached Server Admin Login: /memcache_e6b9b3b7c4fe33d9.php
    new memcached username: memadmin2nOJDiSQRZ8
    new memcached password: qx71dmaSIKthkshpBNL7NUVOlA
    --------------------------------------------------------------------
    
    --------------------------------------------------------------------
    PHP Info Login File: /usr/local/nginx/html/37fb8314_phpi.php
    PHP Info Login: /37fb8314_phpi.php
    PHP Info Login username: phpiadminjGxbutW8tzUteE8
    PHP Info Login password: R3rLh54nQ3loFkl3UVr6lbdpQ
    --------------------------------------------------------------------
    
    --------------------------------------------------------------------
    Generate mysql root password
    --------------------------------------------------------------------
    Previous MySQL root password:
    
    [client]
    user=root
    password=Jba538IU41gyfVKdAhb8nxTEXFt
    
    mysqladmin -u root -pJba538IU41gyfVKdAhb8nxTEXFt password NmtATq5mcrFGiROovi6AjXilIVe5
    
    --------------------------------------------------------------------
    New MySQL root user password: NmtATq5mcrFGiROovi6AjXilIVe5
    --------------------------------------------------------------------
    
    --------------------------------------------------------------------
    /root/.my.cnf updated
    --------------------------------------------------------------------
    
    [client]
    user=root
    password=NmtATq5mcrFGiROovi6AjXilIVe5
    
     
  3. gooseegg

    gooseegg New Member

    7
    0
    1
    Nov 1, 2019
    Ratings:
    +0
    Local Time:
    3:21 PM
    1.17
    10
    Thanks, @eva2000

    packer.io probably is a better abstraction for working with diff CSPs as far as configuring or creating the image. I am not sure though if it brings anything to the table when it comes to dealing with the issues pertinent to the HA & Scaling challenges. Here I identify what I think are the core issues that need attention.

    There are many concerns that need to be addressed. Some we can tackle as a generic issue while others can be handled by end-users per their specific use cases and requirements.

    I suggest we create a base image ( I can create few images for popular varieties on AWS) with an optimized LEMP stack as a community AMI. End users can then tailor this AMI to their own use case and create their own golden Image to suit their purposes.

    Here are the core issues that need attention when it comes to HA & elasticity issues.
    1. The stack is optimized for specific instance types. The most important thing to note there that it is optimized for a specific variety. However, it need not make any assumptions about the hostname or any peculiar info.

    2. There is a need for a shared file system or EC2s that pop in and out have no way to share their state. On AWS, a viable shared file system is EFS (it's NFS: running your own NFS solution is a headache). I note here three things.
      1. One is that not all apps can work this way but many popular ones can.
      2. Another is that Nginx and as much of the stack as possible need to run on the instance. (first off, EFS is just a file server and nothing can execute on it so everything has to be fetched to the instance: and that is slow. Caching can mitigate some of these issues but that is tangential). So the reason I bring EFS up is that Nginx, while it runs on the instance, needs to account for EFS mounting-the mounting can be configured with fstab but the Nnginx needs to be configured to work with mounted folders. This perhaps is an app-level concern but just throwing it out there.
      3. Some prefer to use their own sync/replication solutions. It can certainly be the only way for apps that can't work on NFS but these are the minority of use cases and I am ignoring the problem at all. (Something that can be addressed when the need arise).
      4. For specific use cases, S3 can be used. This is also something that can be addressed at a later point.
    3. There is also a need for a shared database. On AWS, the RDBMS service is RDS. Some apps are a better fit for DynamoDB. A similar argument can be made as above when it comes to the RDBMS. Some create their own replication solution but there already is a baked-in solution in RDS. I am not aware of what kind of optimization you do for MySQL. If any, it would be redundant. The issue to address here is to make the stack configurable such that connection params (that allow the instances to connect to RDS) can be accepted. I believe the stack still needs the mysql driver for connection.

    4. SSL is typically terminated (some compliance regimens may require secure backend transport: but rare) at the load balancer. Note that the use of an LB is imperative for Autoscaling Groups (an essential feature for HA + dynamic scaling). In which case, the Nginx and whatever it proxies downstream need to be aware that they are sitting behind an SSL terminated LB in order to construct the proper URL. Or else, whatever the app extracts from the header would be HTTP (not HTTPS). Nowadays, almost all client-facing requests come over https (and the LB is configured to redirect all HTTP to HTTPS. Nginx can handle this easily. Generally, well-developed apps can also handle this. They would have a setting file to configure to let them construct their URLs properly. The Nginx would just need to proxy the proper header (X_Forwarded_Proto..)

    5. The Security Group (SG) and Network Access Control List (NACL) are tasked with controlling traffic from and into the instance and the subnet the instances reside. The CSF configuration, I suspect, becomes redundant. All blacklisting/whitelisting with specific ports, IP, etc need to be handled by the SG & NACL. In fact, the best practice setup is to terminate front facing traffic at the LB and the instances remain in private networks behind the LB, only allowing traffic from the SG of the LB at specific ports. Everything else is basically blocked. Having said this, if there is any service that actually whitelist/blacklist based on DNS, then that setting can be useful as SG/NACL function at the network layer (won't resolve DNS).

    6. Memcache /Redis settings. In the typical production-setup, the cache doesn't exist in the RAM of the instance but rather, on its own dedicated specialized server called ElastiCache on AWS. These are RAM optimized instances with endpoints (addressable by DNS) , super-fast (ping @< ms) serving Memcache/Redis. They can also be made redundant and more available. Scaling is also possible. Most importantly, they have shared caches so diff number of instances can share them.

    7. Other issues I failed to mention? These were what came to the top of my mind.

    Summary:
    1) optimize for specific machine type
    2) account for shared file systems (EFS mount points)
    3) account for shared database system (RDS endpoints)
    4) proxy header info for proper URL construction downstream
    5) modify or nullify CSF (use SG and NACL: instead - to be configured by end-user
    6) account for caching endpoint (than embed cache)

    Note: I can handle all the entailed AWS related tasks. I just need help with the Centmin Mod side.
     
    Last edited: Nov 6, 2019
  4. eva2000

    eva2000 Administrator Staff Member

    42,079
    9,499
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,615
    Local Time:
    7:21 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Thanks for sharing your thoughts
    What exactly do you need on Centmin Mod side ? Most of what you outline on AWS level can already be done on that end via scripting/AWS APIs even with Centmin Mod. As Centmin Mod just a LEMP stack bundle of open source software.

    But yes it can't be done in Centmin Mod's existing automated routines via centmin.sh. And I wouldn't have the spare time bandwidth to cater to this if it means developing and maintaining and providing support for the actual AWS AMI image with Centmin Mod installed on it and/or if you want an easy automated way of doing all the AWS side stuff from within Centmin Mod centmin/routines given all the other Centmin Mod developments I have going outlined on dev dashboard at centminmod/centminmod (whistlist does have AWS EC2 support later on though).

    Also not many Centmin Mod users are the type that can afford to be paying AWS EC2 related bandwidth costs at US$90-120+ per terabyte (for <10TB that's US$900-1,200/month bandwidth bill just itself excluding EC2 and other AWS services in play). If they can afford that, they can easily hire me to do custom paid Centmin Mod development work for whatever they need :)

    But it can be configured manually or via own scripting to do so. For example EFS/NFS just use mount/label names that Centmin Mod would look for like /home/nginx/domains, /etc/centminmod, /root/centminlogs etc and Nginx proxy header info just create a Nginx vhost template that you can replace Centmin Mod Nginx vhosts with manually/scripted with the desired proxy related headers and nginx backend listen ports etc. A full overview of Centmin Mod file/directory structure is outlined at Centmin Mod Configuration Files - CentminMod.com LEMP Nginx web stack for CentOS which will make it clearer for your distributed file system configuration/scripting.

    Examples - though I am working on better custom templates eventually :)
    Also these 2 migration guides give you a better overview of structure and moving pieces in Centmin Mod LEMP stack so you know what needs to configured in HA/distributed file systems etc.
    1. Moving Centmin Mod site data to another Centmin Mod server
    2. Moving cPanel/WHM site data to Centmin Mod server
    Also working on a Centmin Mod data/server transfer tool to allow easier migration of full Centmin Mod sites to new Centmin Mod servers Beta Branch - revise centmin.sh menu option 21 in 123.09beta01 and posted a demo output of what directories/files get backed up which may help you better understand Centmin Mod's directory/file structure centmin.sh menu option 21 data transfer's backup routine verbose example output demo

    Same with Security Group (SG) and Network Access Control List (NACL), AWS awscli/API can handle all that. Similar to Vultr Firewall API example I guess - Vultr - Vultr Firewall Replicate CSF Firewall Inbound Rule Sets. CSF Firewall you can just disable and for reference for your AWS level firewall config CSF - Centmin Mod LEMP stack CSF Firewall default port listing

    CentOS 7
    Code (Text):
    systemctl stop csf lfd
    systemctl disable csf lfd
    


    But yes remote mysql connections isn't automatically supported in Centmin Mod for addons/mysqladmin_shell.sh creation of database names/user/passwords/grant privileges. I did just add to my to list for remote mysql support in addons/mysqladmin_shell.sh though centminmod/centminmod :) PHP already has everything it needs to connect to remote MySQL servers as long as firewall/ACLs you configure on AWS level end for AWS database servers are setup. I did briefly test AWS RDS connection setup with Centmin Mod LEMP stack way back and that worked fine Centmin firewall not allow AWS RDS connection

    And well if you disable CSF Firewall, you wouldn't have to do the CSF Firewall steps anyway.

    Creating an officially supported Centmin Mod AWS AMI image would be something very far down but still on my wish list to do if I have the free time/funds to do such. I only work on Centmin Mod in my free time and I also only ever release Centmin Mod related features/products that I am willing to provide basic support for and free support for developing and maintaining an official Centmin Mod AWS AMI image and any issues that arise, wouldn't be high on my list unfortunately. But right now my focus would be on getting all the development list of things to do outlined at centminmod/centminmod done first.

    However, I hope the info I provided helps you better understand Centmin Mod's moving pieces.
     
    Last edited: Nov 6, 2019
  5. gooseegg

    gooseegg New Member

    7
    0
    1
    Nov 1, 2019
    Ratings:
    +0
    Local Time:
    3:21 PM
    1.17
    10
    Thanks again @eva2000.

    Few points:
    1. Please understand that I am not asking that you support AWS. I gather that you got an open-source LEMP stack that runs in any (CSP & web hosting providers) env. Asking you to tailor it to a specific vendor's env, especially a walled garden like AWS and one that is not particularly the destination for your target end-users, would not make sense. It would be wrong as well. I am actually the one who is trying to do that. Which is why I am here trying to get some help and direction.
    1. Speaking of your target market, I realize that you got budget-conscious end-users in mind. It is not only the big corps who are on the cloud though. Turns out that a lot of small to medium businesses (including moms & pops stores) are moving to the cloud lately. They may not have the dedicated crew to build and maintain their own env, but still, they find it worth the headache to move to the cloud as many are following the trends and have started running an aggressive social media & email marketing. And traditional web hosting providers don't cut it as they experience a highly variable, sometimes global traffic. They also seek modern business services such as bot chats, notifications so forth. CSPs provide more suitable env in this regard though they come at a premium. Yes, bandwidth seems expensive but customers get reliable bandwidth. They also achieve high availability and high elasticity in addition to running a more fault-tolerant system. DR is a breath. They get to pay only for what they use. There are hundreds of advanced business services they can subscribe to on an on-demand basis as well. Most small businesses, in fact, save a lot of money by going to the cloud assuming they can also build a business to justify their expansion as they increase their capacity and capability significantly. Also, whether the cloud is desirable env or not is a matter of perspective: Do they see IT as an essential resource or a strategic capability? I can go on and on, and make arguments for the benefits (and of course there is a cost and complexity trade-off).
    2. Having said that, I am not arguing the plus and minus of going to the cloud. But I want to point out that those your target can also benefit immensely from the Centmin Mod in the cloud env (it's possible to use it but making it work in the HA & elastic env is not trivial as I discussed in the original post). I also noted that, for open-sourced LEMP stacks, I have found Centmin Mod to be among the best performers (not that I am the expert in the matter). So I thought I will tailor it to AWS. But then why AWS only? It is just one vendor. There are many. As I said, I intend to bake in some images for a few popular machines on AWS. Others can take it to GCP, Azure etc. I can host the images on the market as a community image (open + free) or you guys can do it as it's a trivial matter. Others can do the same for other CSPs. My intention is to understand Centmin Mod enough, so as to have few cloud configurable components (not for AWS, but for a generic CSP). On the cloud, most services come as an endpoint. I identified the NFS, the cache, the database endpoints as the main areas of interest. Additionally, I identified the SSL & FW issue. Yes, all these can be disabled or edited or scripted somehow. But I figured it would be awesome if they can be configurable.
    3. Speaking of supporting EC2, I don't particularly seek that; besides, the stack actually works on EC2. It just needs some configurations, which like you said have a scriptable resolution. I already have done it manually ( and it's a trivial matter to script the solutions) but haven't used it in production b/c I wasn't sure that I have identified all the necessary configurations. I also could have invested time and accommodate all but not sure how it would have survived updates. So, I did find the effort worthy. Here is actually how I get to do it:
      1. Centmin Mod works comfortably on EC2. It just hardwires host, which I edit
      1. Mounting at boot is trivial. As I said, this is an app-level issue: for WP it's where centmin.sh installs the root. Maybe it's possible to edit that. I simply copy the public folder, mount EFS on the public folder and put the copied files in it. I basically pull the rug under its feet.
      2. The same goes for RDS. For WP, for instance, it's trivial to edit the wp-config.php. I turn off the CSF and use SGs & NACLs. SSL can be fixed for WP at wp-config.php.
      3. I haven't tried the caching though. I instead used whatever was installed. Which leads me to the crux of what I am trying to say


    1. Here is what I am trying to get to: How do we make Centmin Mod more generically cloud-friendly?
      1. What are the settings that particularly make installation & optimization coupled to the particular machine? (mind you, I am not talking about types like CPU count, RAM or even OS, etc. Let's assume we got a machine image with a given configuration of OS, RAM, CPU, etc. I already realize that the installation hardcode the host IP --this, of course, can be replaced with grep or other scripting solution, but what other info does it hardcode? This is one thing I am not sure about.
      2. For WP, how about making the root folder configurable for NFS endpoint? Same things for the endpoints for database and caching
      1. How can I prevent SSL & CSF configurations from running in the first place (than turns off after the fact?). I say this because part of these configurations cost time and can cost minutes.
    Now, I am not too self centered or too naïve to ask that you make a new version to accommodate these changes. But rather a pointer to guide me in say writing a remediating script that can be added to the installation so as say to skip on CSF & SSL, account for endpoints, and inject particular server info. Hopefully, I can make it generic enough that others can extend it to other env.



    Does it make sense now? Sorry if I was being vague. I guess you question is helping me articulate it better : )
     
  6. eva2000

    eva2000 Administrator Staff Member

    42,079
    9,499
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,615
    Local Time:
    7:21 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Ah your reply got caught in forum's automated moderation queue, so had to approve it :)

    Yes adding a script or function to centmin mod install's initial install ending to trigger a custom script to customise the environment wouldn't be that hard to do - I believe some private centmin mod github branches have such test code already being tested in the past. Probably add it into inc/centminfinish.inc include file's centminfinish function to be triggered after initial centmin mod install is done.

    SSL account end points are disabled by default when you create nginx vhosts unless you manually choose HTTPS/SSL. So all nginx vhosts are created as non-HTTPS by default so very easily can setup a front end AWS Load Balancer to just talk with non-HTTPS nginx vhost backend AFAIK and allow AWS Load Balancer to terminate HTTPS/SSL connections.

    would be much easier to just create specific NFS mounts for specific nginx vhosts like Wordpress at /home/nginx/domains/wp.domain.com/public it would be transparent to Centmin Mod won't require any changes to Centmin Mod itself.
     
  7. gooseegg

    gooseegg New Member

    7
    0
    1
    Nov 1, 2019
    Ratings:
    +0
    Local Time:
    3:21 PM
    1.17
    10
    Thanks for the response @eva2000.

    - I will look into the centminfinish function. I also am aware that SSL configuration can be skipped. I was just throwing it in there along with the CSF, which can also be turned off. What is ideal, is to actually not have the components run in the first place as the idea is to save the time of installation. So something of an equivalent to a centminstart (though I realize this may not exist) would have been nicer. But this is just a minor issue.

    - A major issue is identifying how the configuration/installation/optimization gets 'coupled' to a particular VM. As I said, I have identified the host IP as one. What else should I look into? Is there documentation I can look into? This way, I can bake an image for a given instance configuration (given CPU, OS, RAM, etc. ). I can do this for some popular instance types ( say t3.small, t3.medium, t3Large, etc.). An end-user can pick a machine type and create a launch configuration (or template) that an autoscaling group can use to spin up instances as necessary. Every time an instance is launched, it only needs to account for these (yet unidentified and what I am after) peculiarities. If all works out well, it's also possible to create a job that can update these images on a regular basis churning out updated images.

    - Another issue is the endpoints (cache & RDS). Once again, these can be done at the app-level. However, I am curious to know if it's possible to somehow integrate this into the stack. First off, this also cut out time. Secondly, it's also a matter of integration. As I said, running a cache in ElastiCache is much more efficient and cost-effective. I am not sure what can be said with regard to the MySQL/MariaDB.

    Speaking of time, I keep focusing on it for a couple of reasons. For starter, for specific applications, the very latest stack can be built on the fly at launch (without waiting for an upgrade). This only make sense if the installation and optimization can run quickly. Secondly, the job I refer to can be run in Lambda if the process takes less than 15 minutes. That would significantly simplify things. Related to the time is the pursuit of minimalism (for security and complexity reasons than size).

    What do you think? Thanks for your time.
     
  8. eva2000

    eva2000 Administrator Staff Member

    42,079
    9,499
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,615
    Local Time:
    7:21 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Nothing in Centmin Mod is actually linked to server IP address itself. Besides your domain DNS A/AAAA records from 3rd party pointing to your servers. Nginx will use and resolve to whatever IP address is the main IP of the server it's on by default as Nginx vhosts don't specify an IP address in their listen directives it's just the listen port being 80 or 443. PHP-FPM server listens to 127.0.0.1 and MySQL to localhost.

    Wordpress auto installer via centmin.sh menu option 22 and nginx vhost creation via centmin.sh menu option 2 or nv command all dynamically get the IP address references from what IP is detected on server. So if you bake an an image and launch that image and create an nginx vhost or wordpress site via centmin.sh menu option 2, 22 or nv command, the IP obtained is the IP of the server it's running on. Unless you are experiencing something different ?

    Usually easier to re-enable something that was installed, then to install it again considering the auto tuning that goes into CSF Firewall setup. For instance if someone wants to use CSF Firewall instead of AWS level firewall, it's just a matter of restarting CSF Firewall service and instantaneously they have working CSF Firewall config.

    Not everyone will want to use web provider level firewalls with AWS, Google Cloud, Vultr etc. I could add a persistent config variable to allow CSF Firewall to not be installed initially. But very hesitant for folks who may opt to do this just to make life easier without a firewall or folks who have a time gap where they're unprotected between disabling CSF Firewall and the time it takes them to implement their own firewall choices/alternatives.

    And if they in fact properly configure their own firewall for Centmin Mod needs i.e. accidentally they configure and leak/expose ports they shouldn't have because they are not familiar with the services running on their Centmin Mod server. CSF Firewall out of box provides full protection for Centmin Mod LEMP stack's services and ensures what is install, runs properly. Analogy is if CSF Firewall like AWS S3 buckets by default are set to private and not public accessible on creation unless otherwise configured. Then let users decide if they want to publicly expose an AWS S3 bucket to the internet (turn off CSF Firewall). As opposed to just let folks create AWS S3 buckets with default public access out of the box (turn off CSF Firewall) and assume they'll know enough to set AWS S3 buckets to private themselves later on.

    You'd have to elaborate on that. Centmin Mod's own automation as far as MariaDB MySQL database creation/user creation is with addons/mysqladmin_shell.sh and centmin.sh menu option 22 wordpress installer where the assumption is local MariaDB MySQL database server is to be used.

    Using RDS isn't as simple as pointing to RDS remote instance as it depends on end user being able to properly configure the RDS instances VPC Security Group's inbound TCP rules on AWS side and traditional remote MySQL & RDS servers require properly grant privileges for remote MySQL server connections. And if CSF Firewall is running would require proper remote connection firewall whitelisting or end user's alternative firewall/network configuration. Without that proper configuration for RDS or MySQL remote servers from Centmin Mod end user side, Centmin Mod will fail to allow web apps to connect to the remote MySQL/RDS servers and loose that smooth automated process of create a site/database seamlessly. Then these users will be coming back to me asking for free support for configuring advance setups on AWS/Cloud provide side which is outside of Centmin Mod's current provided configuration. For me it comes back to whatever I offer in Centmin Mod, I have to be willing to back that with some form of support as well. With that being said, I have added to my 123.09beta01 to do list adding eventually remote MySQL server detection support to addons/mysqladmin_shell.sh at centminmod/centminmod. But it will probably be just a setting where you can define remote MySQL ip address/port and user/pass. Nothing will be done for remote MySQL server configuration/setup at AWS/Cloud provider level as that would be left to end user to do.

    from previous post quoted starters to see where everything is
     
  9. gooseegg

    gooseegg New Member

    7
    0
    1
    Nov 1, 2019
    Ratings:
    +0
    Local Time:
    3:21 PM
    1.17
    10
    Thanks @eva2000. Sorry I was a bit busy.

    As far as my interest goes, I will look into the stack more carefully on my next project now you tell me that there is nothing in particular that tie the installation to a particular IP.

    As far as my general idea is concerned, however, you make me realize a serious flaw in my thinking. I guess I am basically trying to bake an image which only those who have fundamental grasp of the Cloud env & ops can use and operate. These are probably the very people who don't need it. Duh.

    At any rate, I really appreciate the good work you do, your time & patience.

    Let me know if there is anything I can help (I mean for free, not looking for a business) with if you guys ever embark on an AWS project. I believe my account here is tied to my email.

    Thanks!
     
  10. JJC84

    JJC84 Premium Member Premium Member

    238
    102
    43
    Jan 31, 2018
    San Antonio, Texas
    Ratings:
    +158
    Local Time:
    3:21 PM
    1.15.x
    10.x.x
    My take on implementing highly available fault tolerant WordPress infrastructures on AWS (a domain that I have a particular interest in obtaining a certain level of mastery within over the last 1.5-2 years spurring me to obtain the Solutions Architect Associate level certification) is quite a bit different.

    It really depends on your use case which I am not going to make any assumptions about but I ended up fairly quickly throwing out the notion of using monolithic EC2 stacks behind a load balancer in distinct favor of orchestrated containerized N-tiered and microservice (big microservices in this case) architectures instead.

    A rough outline of where I ended up was docker containers scheduled by Kubernetes using Helm charts and the Portworx plugin which is a tremendous boon when having to use containers which are stateless to encapsulate stateful services in the relational databases which are inherent to WordPress and generally all other CMS platforms.

    Running WordPress properly (if I am even running it properly because who knows? I just do my best) in this way is a bit tricky for a few reasons such as how to approach disaster recovery as well as how to keep your data, plugins, uploads etc... from sharding across multiple container instances or running into issues with stuck EBS volumes, etc...

    I am going to read the thread over again a few more times and then hopefully come back with more specified insight.

    Can I ask what is your use case? How much traffic do you have? Is using something like EC2 instances which are much more wasteful in resources and as a result monetary expenditures favored due to the amount of instances being fairly small or do you need something in the higher spectrum of things tending towards enterprise loads? I am in the total dark as to how I can help or if it is even necessary for me to unless I know more about what it is that you are trying to accomplish when you say that you need to setup WordPress to be highly available on AWS.

    I am not totally sure if I explained that properly or not but I really like this thread and am enjoying reading the conversation so far.