Join the community today
Register Now

Install Unable to SSH after Centminmod install On Google Cloud server

Discussion in 'Install & Upgrades or Pre-Install Questions' started by Lav, Apr 10, 2020.

  1. Lav

    Lav Member

    49
    1
    8
    Feb 23, 2020
    Ratings:
    +1
    Local Time:
    10:59 PM
    1.17.8
    10.3
    Please fill in any relevant information that applies to you:
    • CentOS Version: CentOS 7
    • Centmin Mod Version Installed: 123.09beta01
    • Nginx Version Installed: 1.17.9
    • PHP Version Installed: 7.3
    • MariaDB MySQL Version Installed: 10.3.xx
    I don't know what's going wrong? I have installed this stack several times before but from yesterday after fresh installation on a new server I cannot connect via SSH to my server.
    I am not doing anything differently. I am installing it the way I have installed my all previous centmin stacks as a 'root' user but now after complete installation when I reboot the server I am unable to log back in via SSH and it simply says connection refused after trying to connect for more than 5 or 10 mins everytime.
    What's wrong here?
     
  2. eva2000

    eva2000 Administrator Staff Member

    45,448
    10,313
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,997
    Local Time:
    3:29 AM
    Nginx 1.19.x
    MariaDB 5.5/10.x
  3. Lav

    Lav Member

    49
    1
    8
    Feb 23, 2020
    Ratings:
    +1
    Local Time:
    10:59 PM
    1.17.8
    10.3
    I am hosting this on Google Cloud. Does this helps?
    No idea at this moment about alternative method for gcp, will take a look into it but why now? It never happened before then why its happening now? Any idea?
     
  4. eva2000

    eva2000 Administrator Staff Member

    45,448
    10,313
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,997
    Local Time:
    3:29 AM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Google Cloud has their own firewall you need to configure and Centmin Mod doesn't natively work with Google Cloud has it forces sudo user usage so Centmin Mod initial install isn't properly installed as it requires full root non-sudo user. So not sure how you got Centmin Mod to work previously on Google Cloud as the sudo issue could of prevented some Centmin Mod installed items to not install properly or not install/configure Centmin Mod and server correctly. I assume if you previously had it working, you already properly configured Google Cloudflare's native firewall to prevent it blocking access to your servers ? See what Centmin Mod ports are needed at CSF - Centmin Mod LEMP stack CSF Firewall default port listing

    Centmin Mod 123.09beta01 has made progress so that sudo user is supported only after initial Centmin Mod install but not for initial install. But I also made some recent undocumented improvements so you could technically install if you prefix every centmin mod related command with sudo. You can see changes I made in Dec 2019 on official Github repo search at centminmod/centminmod. This change outlined at Beta Branch - update installers to setup /etc/sudoers.d/addpaths earlier in 123.09beta01 is one that is key to making sudo work for initial Centmin Mod installs if all commands are prefixed with sudo.

    So for Google Cloud server to install Centmin Mod it would become
    Code (Text):
    sudo yum -y update; sudo curl -O https://centminmod.com/betainstaller73.sh && sudo chmod 0700 betainstaller73.sh && sudo bash betainstaller73.sh
    

    note every command is prefixed with sudo

    It's undocumented feature as virtually no testing is done by me for Google Cloud support as I don't have a Google Cloud account to test with due to the cost and free time constraints. I did make these sudo changes as other Centmin Mod users on Google Cloud were having issues and confirm sudo prefix changes worked for them. Just no testing done by me.
     
  5. Lav

    Lav Member

    49
    1
    8
    Feb 23, 2020
    Ratings:
    +1
    Local Time:
    10:59 PM
    1.17.8
    10.3
    I know about GCP Firewall and I have knowledge about those ports and I think everything is fine on this side. I have installed centminmod more than 50 times on my servers without any issue of this kind so I don't see these happened by chance. I am doing nothing different this time too but still its not working from yesterday so I think there are some changes have been made on the centminmod stack that's why it's not working.
    I install centminmod as root user after disabling selinux permanently but anyways I have just made two servers and installing centminmod on both as root user with one has sudo command and the other one has the normal command. Will inform you in just 15 minutes of how both went.
    Thanks for the help!
     
  6. eva2000

    eva2000 Administrator Staff Member

    45,448
    10,313
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,997
    Local Time:
    3:29 AM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Like how you think/test :)

    Remember to grab initial install logs BEFORE rebooting for them too. You can save logs locally and do a diff compare of initial install logs to see what differences there are between them :)
     
  7. Lav

    Lav Member

    49
    1
    8
    Feb 23, 2020
    Ratings:
    +1
    Local Time:
    10:59 PM
    1.17.8
    10.3
    I used the sudo command which you posted above to install centminmod but it didn't worked. After rebooting the system I don't know what goes wrong so that SSH stops working and neither ftpd login works. I am unable to copy the files here cause ftpd is not working and I dont know what other method I can use to extract that file from the server.
    This has never happened before and now I have no idea how to sort it out. If you can help me in extracting those log files here then it would be really helpful.
     
  8. Lav

    Lav Member

    49
    1
    8
    Feb 23, 2020
    Ratings:
    +1
    Local Time:
    10:59 PM
    1.17.8
    10.3
    I think I got the problem. During setting up the VM, there is a new option of secure boot, vTPM and integrity monitoring. Never saw this before.
    vTPM and integrity monitoring is checked by default and I think the problem is with
    Virtual Trusted Platform Module (vTPM) cause it uses boring ssl to verify keys whereas after installation of centminmod, it installs open ssl. I think that is causing the problem.
    This is the official documentation Shielded VM  |  Documentation  |  Google Cloud
    Do I need to uncheck it or do I have any other option cause it may be useful for security.
     
  9. eva2000

    eva2000 Administrator Staff Member

    45,448
    10,313
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,997
    Local Time:
    3:29 AM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    If there's option to disable vTPM, then try a fresh install with vTPM disabled to confirm
     
  10. eva2000

    eva2000 Administrator Staff Member

    45,448
    10,313
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,997
    Local Time:
    3:29 AM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    You mean you loose SSH/sftp/ftp connectivity after Centmin Mod install even before rebooting server ? I suggested to get the logs after initial install but BEFORE rebooting so you still have SSH/sftp/ftp access before the reboot
     
  11. Lav

    Lav Member

    49
    1
    8
    Feb 23, 2020
    Ratings:
    +1
    Local Time:
    10:59 PM
    1.17.8
    10.3
    I tried everything which my mind can think of as of now.
    Here is what I found.
    If centminmod installed via putty then I can keep accessing SSH via putty but the stack must be installed via putty and I have no reason to understand how and why?
    If the stack is installed via the native GCP SSH then I loose access after the first reboot after installation of stack and cannot access it then even by trying to connect it via putty.
    But stack installed via putty keeps its access whereas native GCP SSH cannot connect to that VM instance.
    Do you have any idea about why and how?
     
  12. eva2000

    eva2000 Administrator Staff Member

    45,448
    10,313
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,997
    Local Time:
    3:29 AM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Only thing I can think of it CSF Firewall installed by Centmin Mod on initial install will whitelist the IP address detected on SSH client login. If you use Putty, you connect directly and your ISP IP is logged and detected and whitelisted. If you connect via GCP SSH, your ISP IP isn't detected and whatever internal IP used by GCP SSH to connect your server is detected and that internal IP is whitelisted.

    In GCP SSH and Putty SSH, what output do you get for command that reveals system detected SSH client IP ? It should show your ISP IP address you are connecting from and is the IP Centmin Mod detects and whitelists on initial install for CSF Firewall usage
    Code (Text):
    echo "$SSH_CLIENT"
    

    If after install, you whitelist your ISP IP
    Code (Text):
    csf -a YOUR_ISP_IP

    then reboot, do you get SSH access ?

    I generally recommend you use Putty or your own SSH client rather than GCP SSH or consoles to install Centmin Mod because of this
     
  13. Lav

    Lav Member

    49
    1
    8
    Feb 23, 2020
    Ratings:
    +1
    Local Time:
    10:59 PM
    1.17.8
    10.3
    I tried by whitelisting the ip address in csf but it didn't worked. As of now the only way which I have found to SSH is the private key which I have entered during making a new VM before installation of centminmod. Once the installation is complete, this private key remains the only way to SSH and I cannot add another private key or can't do SSH via gcp cloud console cause it gives the error of
    Code:
    ssh-permission-denied-publickey
    . This has never happened before. I tried to google the error and I found some including changing the file permission to 700 of
    Code:
    /root/.ssh
    but cannot find this file on centminmod due to different file structure.
    https://stackoverflow.com/questions/36300446/ssh-permission-denied-publickey-gssapi-with-mic
    This link will make you understand my problem. How to solve this?
     
  14. eva2000

    eva2000 Administrator Staff Member

    45,448
    10,313
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,997
    Local Time:
    3:29 AM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    You mean this ?
    Code (Text):
    chmod 700 /root/.ssh
    chmod 600 /root/.ssh/authorized_keys
    

    That is normally done by end users or when you use ssh-copy-id command line to copy a generated public SSH key to remote server or with your web host provider their automation usually creates those directories and authorized_keys file. I know upcloud, digitalocean, vultr all automatically do this when you choose to add a SSH key to the VPS server you create.

    You're saying Google Cloud server doesn't create this directory and authorized_keys file when you setup SSH key logins ? or sets it up in a different path/file ?
     
  15. eva2000

    eva2000 Administrator Staff Member

    45,448
    10,313
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,997
    Local Time:
    3:29 AM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    from Google Groups

    and Managing SSH keys in metadata | Compute Engine Documentation
     
  16. Lav

    Lav Member

    49
    1
    8
    Feb 23, 2020
    Ratings:
    +1
    Local Time:
    10:59 PM
    1.17.8
    10.3
    On a fresh install of CentOS 7
    Code:
    [root@centminmod ~]# ls -lah /root/.ssh
    total 4.0K
    drwx------. 2 root root  29 Apr 11 11:34 .
    dr-xr-x---. 3 root root 103 Apr 11 11:34 ..
    -rw-------. 1 root root 922 Apr 11 11:34 authorized_keys
    
    stat -c "%a %n" /root/.ssh
    700 /root/.ssh
    
    stat -c "%a %n" /root/.ssh/authorized_keys
    600 /root/.ssh/authorized_keys
    But after installation of 'centminmod', there is no directory of
    Code:
    /root/.ssh
    . I tried creating it myself and also created authorized_keys inside .ssh folder using 'touch' command but it didn't worked.
    Interestingly, today I made some new VM instance from the snapshots of centminmod stack which I have from last month project and now I am not able to connect to those too but from the last six months I have made several VM from snapshots of centminmod stack but never faced this issue.
    I have currently two running instance on Ubuntu and they are running very fine. This is only happening on CentOS 7 after installation of centminmod.
    The only way to connect to these instance is the putty key installed at the fresh installation of this VM. Only that key is working and apart from that neither able to add another key or access it via gcp ssh window.
    Any idea why this is happening cause this is the first time I am facing this issue.
     
  17. eva2000

    eva2000 Administrator Staff Member

    45,448
    10,313
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,997
    Local Time:
    3:29 AM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Strange, Centmin Mod 123.09beta01 has no code or routines that even touch /root/.ssh directory so the directory shouldn't be removed by Centmin Mod.

    I wonder if this is related to installing via sudo user on Google Cloud and not full root user? But either way Centmin Mod doesn't even touch /root/.ssh so it shouldn't.

    From SSH from the browser  |  Compute Engine Documentation  |  Google Cloud
    Try these commands as both sudo user itself and via sudo command and see what output it gives
    Code (Text):
    echo "$HOME/.ssh/authorized_keys"
    sudo echo "$HOME/.ssh/authorized_keys"
    

    It could be SSH key directory setup by Google Cloud is at /home/sudousername/.ssh and not /root/.ssh directory

    This is where me not having experience with Google Cloud shows heh. Maybe @Chris would have a clue as he's using Centmin Mod on Google Cloud :)
     
  18. eva2000

    eva2000 Administrator Staff Member

    45,448
    10,313
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,997
    Local Time:
    3:29 AM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Oh you may need 2 sets of SSH keys Connecting to instances using advanced methods

     
  19. Lav

    Lav Member

    49
    1
    8
    Feb 23, 2020
    Ratings:
    +1
    Local Time:
    10:59 PM
    1.17.8
    10.3
    I managed to SSH into my server by using a different key by following this guide https://stackoverflow.com/a/59009648/13039298 and then manually adding public key in to authorized key file after creating .ssh and authorized key file with proper file permission. Yeah it's not like before where I had the liberty to connect to my server from anywhere but I think it happened for good cause if my server can only be accessed via a single private key then it makes it more secure but yeah a little bit risky.
    How you look at this and is it a good idea to keep a single source for SSH connection?
    Does it have any negative effect?
     
  20. eva2000

    eva2000 Administrator Staff Member

    45,448
    10,313
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,997
    Local Time:
    3:29 AM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    From Install - Unable to SSH after Centminmod install On Google Cloud server, I believe that is how you're meant to do it, have the SSH key setup by Google Cloud for access VM internally and a separate SSH key setup to access VM server outside of Google Cloud i.e. with putty or SSH clients. With both SSH keys, it would mean you can access VM from either SSH connection source and not be limited to 1. Or are you saying once you setup separate SSH key for 3rd party SSH client use, you can no long use Google Cloud internal connection method to VM using the SSH key setup via Google Cloud ? You should be able to use both now?

    From official Centmin Mod Getting Started Guide step 4 for CSF Firewall Getting Started Guide - CentminMod.com LEMP Nginx web stack for CentOS, I suggest setting up server access via other servers you may have access too in case you loose access from a specific IP too.

    and for Dynmaic IP ISP users