Join the community today
Become a Member

Amazon AWS Problems restoring from image in AWS

Discussion in 'Virtual Private Server (VPS) hosting' started by fly, Aug 14, 2020.

  1. eva2000

    eva2000 Administrator Staff Member

    46,861
    10,629
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,498
    Local Time:
    7:40 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    what updates where they for ?
    Code (Text):
    yum history list

    then for the latest transaction id XX do
    Code (Text):
    yum history info XX

    No idea, didn't spend too much time investigating. I used official CentOS 7.8 AMI image for AWS Ohio us-east2 from CentOS AWS AMI Cloud Images on t3a.small EC2 instances
    Code (Text):
    CentOS Linux 7 us-east-2 x86_64 ami-0a75b786d9a7f8144
    

    for both original image and AWS Backup restored EC2 instances so both had same hardware.

     
  2. fly

    fly Premium Member Premium Member

    49
    7
    8
    Jul 27, 2019
    Ratings:
    +12
    Local Time:
    5:40 AM
    They were Centmin code updates, not yum updates. IIRC, there were two file changes. Once I updated CM, combined with your posted fixes, everything works.

    I've now confirmed that image restores also work on my own production instances. #dogfooding

    Any thoughts yet on moving this fix into your codebase? Or is this a bug that should actually be filed with the CentOS AWS team somehow? There's probably a really good chance that there are folks out there that might run into this issue in a real DR scenario. While not the end of the world, its likely to add a bunch of time to recovery.

    I guess more to the point, how do we move forward and how can I help?
     
  3. eva2000

    eva2000 Administrator Staff Member

    46,861
    10,629
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,498
    Local Time:
    7:40 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Interesting none of the 123.09beta01 updates in the past 2 weeks listed at centminmod/centminmod would be remotely related to this issue though !

    Good to know :)

    Yeah I'm thinking of how I can incorporate this fix/workaround just for Amazon EC2 instance based servers though I don't test Centmin Mod 123.09beta01 on Amazon EC2 or Google Cloud so there's probably other issues untested for anyway.

    I could first check to see if server is running on Amazon network
    Code (Text):
    curl -4s https://ipinfo.io | jq -r '.org | ascii_downcase'
    as16509 amazon.com, inc.
    

    and if so trigger the workarounds.

    Though just tested with t3a.nano EC2 instance without Centmin Mod installed and see cloud-init YUM package and service are enabled and installed - that might explain why non-Centmin Mod based CentOS 7 AMI OS image based AWS Backup and restore worked and why Centmin Mod with cloud-init YUM package installed also allowed AWS Backup restores to work.
    Code (Text):
    yum -q list cloud-init | tr -s ' ' | column -t
    Installed          Packages
    cloud-init.x86_64  18.5-6.el7.centos    installed
    Available          Packages
    cloud-init.x86_64  18.5-6.el7.centos.5  updates
    

    Code (Text):
    systemctl status cloud-{init-local,init,config,final} | grep Active
       Active: active (exited) since Tue 2020-08-25 14:29:37 UTC; 24min ago
       Active: active (exited) since Tue 2020-08-25 14:29:47 UTC; 24min ago
       Active: active (exited) since Tue 2020-08-25 14:30:02 UTC; 23min ago
       Active: active (exited) since Tue 2020-08-25 14:30:06 UTC; 23min ago
    

    Code (Text):
    systemctl is-enabled cloud-{init-local,init,config,final}
    enabled
    enabled
    enabled
    enabled
    

    journalctl boot log grep for cloud-init
    Code (Text):
    journalctl -b --no-pager | grep 'cloud-init' | grep -v 'ci-info' | sed -e "s|$(hostname)|hostname|g"
    Aug 25 14:28:44 localhost.localdomain systemd[1]: Starting Initial cloud-init job (pre-networking)...
    Aug 25 14:28:46 localhost.localdomain cloud-init[607]: Cloud-init v. 18.5 running 'init-local' at Tue, 25 Aug 2020 14:28:46 +0000. Up 8.87 seconds.
    Aug 25 14:29:37 hostname systemd[1]: Started Initial cloud-init job (pre-networking).
    Aug 25 14:29:39 hostname systemd[1]: Starting Initial cloud-init job (metadata service crawler)...
    Aug 25 14:29:40 hostname cloud-init[903]: Cloud-init v. 18.5 running 'init' at Tue, 25 Aug 2020 14:29:40 +0000. Up 62.69 seconds.
    Aug 25 14:29:47 hostname systemd[1]: Started Initial cloud-init job (metadata service crawler).
    Aug 25 14:30:00 hostname cloud-init[1083]: Cloud-init v. 18.5 running 'modules:config' at Tue, 25 Aug 2020 14:29:59 +0000. Up 81.70 seconds.
    Aug 25 14:30:05 hostname cloud-init[1143]: Cloud-init v. 18.5 running 'modules:final' at Tue, 25 Aug 2020 14:30:04 +0000. Up 87.08 seconds.
    Aug 25 14:30:06 hostname cloud-init[1143]: Cloud-init v. 18.5 finished at Tue, 25 Aug 2020 14:30:06 +0000. Datasource DataSourceEc2Local.  Up 88.90 seconds
    

    Maybe not all CentOS 7 AMI images have cloud-init installed as the AMI images are AWS EC2 region specific ? Or maybe Centmin Mod install somehow removed cloud-init ? Will have to check for that. You can check for me on your Centmin Mod installed EC2 instances if any output comes from
    Code (Text):
    yum history list cloud-init
    

    if any YUM transaction ids come up i.e. XX, run
    Code (Text):
    yum history info XX
    

    If yum history info doesn't result any related to yum remove/erase of cloud-init package, then cloud-init wasn't removed and the CentOS 7 AMI OS image didn't have cloud-init in the first place.
     
  4. fly

    fly Premium Member Premium Member

    49
    7
    8
    Jul 27, 2019
    Ratings:
    +12
    Local Time:
    5:40 AM
    Odd. Before I started testing all this my codebase was up to date. After you posted the fix, I ran the commands, snapped, image, restore and it didn't work. When I went back to the test instance, I noticed the message about Centmin updates. I did them and went back through the fix again. I suppose it's possible I missed a command the first time.
    While the lack of evidence isn't evidence, I've been personally running it for at least a couple of years. I also have clients running it as well. So I'd say it works pretty well.

    I think you're on to something with Centmin somehow removing cloud-init. I'm not *exactly* sure the totality of what it does during instance initialization, but it's not trivial. In fact, cloud-init can be used to install Centmin during bootup.

    Initially thinking it had something to do with CSF, I set the userdata to disable CSF (which uses cloud-init to do so). It didn't work, and in hindsight, I should have explored that a bit more.
     
  5. fly

    fly Premium Member Premium Member

    49
    7
    8
    Jul 27, 2019
    Ratings:
    +12
    Local Time:
    5:40 AM
    I've confirmed on @Dryline's infrastructure that cloud-init also was not installed. So you must be on to something about Centmin somehow uninstalling it.

    Either way, your fixes worked again. Thanks!
     
  6. eva2000

    eva2000 Administrator Staff Member

    46,861
    10,629
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,498
    Local Time:
    7:40 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    ;)
     
  7. fly

    fly Premium Member Premium Member

    49
    7
    8
    Jul 27, 2019
    Ratings:
    +12
    Local Time:
    5:40 AM
    Whoops, missed that. Here it is!
    Code:
    [15:06][root@centos]# yum history info 65
    Loaded plugins: fastestmirror, priorities, versionlock
    Transaction ID : 65
    Begin time     : Wed Apr  1 15:45:33 2020
    Begin rpmdb    : 791:db13924d8ce8ac8d0b163b1de8ffcaddb709c72d
    End time       :            15:45:34 2020 (1 seconds)
    End rpmdb      : 788:d59cee45b8e48950aa0789ef54c5ca7410d75e62
    User           : Cloud User <centos>
    Return-Code    : Success
    Command Line   : -q -y remove python-urllib3
    Transaction performed with:
        Updated       rpm-4.11.3-40.el7.x86_64                      @base
        Updated       yum-3.4.3-163.el7.centos.noarch               @base
        Updated       yum-plugin-fastestmirror-1.1.31-52.el7.noarch @base
    Packages Altered:
        Erase cloud-init-18.5-3.el7.centos.x86_64  @base
        Erase python-requests-2.6.0-9.el7_7.noarch @updates
        Erase python-urllib3-1.10.2-7.el7.noarch   @base
    history info
     
  8. Dryline

    Dryline New Member

    7
    3
    3
    Apr 11, 2019
    Colorado
    Ratings:
    +5
    Local Time:
    3:40 AM
    1.15.9
    On behalf of the Stormtrack community I want to thank you for your time and effort @eva2000 in resolving this issue. You and @fly saved the day and we appreciate it. :)
     
  9. eva2000

    eva2000 Administrator Staff Member

    46,861
    10,629
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,498
    Local Time:
    7:40 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Looks like centmin.sh python update routines removed cloud-init when python-urllib3 was removed (in prep for reinstall). So will need to make sure that cloud-init is reinstalled if it was already installed prior.

    You're welcome and great to hear :) If you like, don't forget Ways To Support Centmin Mod ;) :D
     
  10. Dryline

    Dryline New Member

    7
    3
    3
    Apr 11, 2019
    Colorado
    Ratings:
    +5
    Local Time:
    3:40 AM
    1.15.9
    Actually I sent a donation to you a couple of days ago. The email on my PayPal is different than on my account here. My last name is the same as the color of the sky if that helps you tie it together. Thanks again for solving the issue we had. We have enjoyed using your product over the last couple of years and will continue to do so going forward.
     
  11. eva2000

    eva2000 Administrator Staff Member

    46,861
    10,629
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,498
    Local Time:
    7:40 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Awesome, much appreciated :D
     
  12. fly

    fly Premium Member Premium Member

    49
    7
    8
    Jul 27, 2019
    Ratings:
    +12
    Local Time:
    5:40 AM
    I was wondering if you had thought any more about how to roll a fix into Centmin Mod, or if you have already.

    And thank you again!
     
  13. eva2000

    eva2000 Administrator Staff Member

    46,861
    10,629
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,498
    Local Time:
    7:40 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
  14. fly

    fly Premium Member Premium Member

    49
    7
    8
    Jul 27, 2019
    Ratings:
    +12
    Local Time:
    5:40 AM
  15. eva2000

    eva2000 Administrator Staff Member

    46,861
    10,629
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,498
    Local Time:
    7:40 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
  16. fly

    fly Premium Member Premium Member

    49
    7
    8
    Jul 27, 2019
    Ratings:
    +12
    Local Time:
    5:40 AM
    I ran into my first issue with this. This is the error I'm seeing in the EC2 system log:
    Code:
    [   30.146592] cloud-init[595]: Traceback (most recent call last):
    [   30.149869] cloud-init[595]: File "/usr/bin/cloud-init", line 9, in <module>
    [   30.155197] cloud-init[595]: load_entry_point('cloud-init==18.5', 'console_scripts', 'cloud-init')()
    [   30.160468] cloud-init[595]: File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
    [   30.205245] cloud-init[595]: return get_distribution(dist).load_entry_point(group, name)
    [   30.209864] cloud-init[595]: File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2852, in load_entry_point
    [   30.215741] cloud-init[595]: return ep.load()
    [   30.218421] cloud-init[595]: File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2443, in load
    [   30.222985] cloud-init[595]: return self.resolve()
    [   30.225465] cloud-init[595]: File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2449, in resolve
    [   30.230243] cloud-init[595]: module = __import__(self.module_name, fromlist=['__name__'], level=0)
    [   30.234383] cloud-init[595]: File "/usr/lib/python2.7/site-packages/cloudinit/cmd/main.py", line 26, in <module>
    [   30.239522] cloud-init[595]: from cloudinit import netinfo
    [   30.242918] cloud-init[595]: File "/usr/lib/python2.7/site-packages/cloudinit/netinfo.py", line 15, in <module>
    [   30.252793] cloud-init[595]: from cloudinit.net.network_state import net_prefix_to_ipv4_mask
    [   30.257702] cloud-init[595]: File "/usr/lib/python2.7/site-packages/cloudinit/net/__init__.py", line 13, in <module>
    [   30.325209] cloud-init[595]: from cloudinit.net.network_state import mask_to_net_prefix
    [   30.329107] cloud-init[595]: File "/usr/lib/python2.7/site-packages/cloudinit/net/network_state.py", line 15, in <module>
    [   30.334797] cloud-init[595]: from cloudinit import util
    [   30.337440] cloud-init[595]: File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 49, in <module>
    [   30.341584] cloud-init[595]: from cloudinit import url_helper
    [   30.344612] cloud-init[595]: File "/usr/lib/python2.7/site-packages/cloudinit/url_helper.py", line 13, in <module>
    [   30.348798] cloud-init[595]: import requests
    [   30.351125] cloud-init[595]: File "/usr/lib/python2.7/site-packages/requests/__init__.py", line 54, in <module>
    [   30.484705] cloud-init[595]: pyopenssl.inject_into_urllib3()
    [   30.488513] cloud-init[595]: AttributeError: 'module' object has no attribute 'inject_into_urllib3'
    [[1;31mFAILED[0m] Failed to start Initial cloud-init job (pre-networking).
    Any idea what that might be?
     
    Last edited: Sep 24, 2020
  17. eva2000

    eva2000 Administrator Staff Member

    46,861
    10,629
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,498
    Local Time:
    7:40 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    what's the context around that error, were you reboot server, restoring image backup and/or how long has the EC2 server been running until the first instance of this error was logged ?
     
  18. fly

    fly Premium Member Premium Member

    49
    7
    8
    Jul 27, 2019
    Ratings:
    +12
    Local Time:
    5:40 AM
    This was on a test restore of a fixed instance. That was the error as it was booting up. As somewhat expected, cloud-init isn't working, so I am unable to connect to the instance.
     
  19. eva2000

    eva2000 Administrator Staff Member

    46,861
    10,629
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,498
    Local Time:
    7:40 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Ok going to have to investigate some more
     
  20. Brijesh Joshi

    Brijesh Joshi New Member

    3
    1
    3
    Sep 25, 2020
    India
    Ratings:
    +1
    Local Time:
    3:10 PM
    A corrupt Windows registry can cause an EC2 Windows instance to freeze, slow down, restart randomly, or become unusable.

    Even without performing other backup utilities or configurations beforehand, Windows regularly and automatically makes its own backup of the registry in the following location:

    C:\Windows\System32\config\RegBack

    EC2Rescue can restore these previously created backups from the RegBack directory to the current registry of the EC2 instance.