Amazon AWS Problems restoring from image in AWS

They were Centmin code updates, not yum updates. IIRC, there were two file changes. Once I updated CM, combined with your posted fixes, everything works.

I've now confirmed that image restores also work on my own production instances. #dogfooding

Any thoughts yet on moving this fix into your codebase? Or is this a bug that should actually be filed with the CentOS AWS team somehow? There's probably a really good chance that there are folks out there that might run into this issue in a real DR scenario. While not the end of the world, its likely to add a bunch of time to recovery.

I guess more to the point, how do we move forward and how can I help?

 

Odd. Before I started testing all this my codebase was up to date. After you posted the fix, I ran the commands, snapped, image, restore and it didn't work. When I went back to the test instance, I noticed the message about Centmin updates. I did them and went back through the fix again. I suppose it's possible I missed a command the first time.

While the lack of evidence isn't evidence, I've been personally running it for at least a couple of years. I also have clients running it as well. So I'd say it works pretty well.

I think you're on to something with Centmin somehow removing cloud-init. I'm not *exactly* sure the totality of what it does during instance initialization, but it's not trivial. In fact, cloud-init can be used to install Centmin during bootup.

Initially thinking it had something to do with CSF, I set the userdata to disable CSF (which uses cloud-init to do so). It didn't work, and in hindsight, I should have explored that a bit more.

 

I've confirmed on @Dryline's infrastructure that cloud-init also was not installed. So you must be on to something about Centmin somehow uninstalling it.

Either way, your fixes worked again. Thanks!

 

Whoops, missed that. Here it is!

Code:

[15:06][root@centos]# yum history info 65
Loaded plugins: fastestmirror, priorities, versionlock
Transaction ID : 65
Begin time     : Wed Apr  1 15:45:33 2020
Begin rpmdb    : 791:db13924d8ce8ac8d0b163b1de8ffcaddb709c72d
End time       :            15:45:34 2020 (1 seconds)
End rpmdb      : 788:d59cee45b8e48950aa0789ef54c5ca7410d75e62
User           : Cloud User <centos>
Return-Code    : Success
Command Line   : -q -y remove python-urllib3
Transaction performed with:
    Updated       rpm-4.11.3-40.el7.x86_64                      @base
    Updated       yum-3.4.3-163.el7.centos.noarch               @base
    Updated       yum-plugin-fastestmirror-1.1.31-52.el7.noarch @base
Packages Altered:
    Erase cloud-init-18.5-3.el7.centos.x86_64  @base
    Erase python-requests-2.6.0-9.el7_7.noarch @updates
    Erase python-urllib3-1.10.2-7.el7.noarch   @base
history info

 

I was wondering if you had thought any more about how to roll a fix into Centmin Mod, or if you have already.

And thank you again!

 

Excellent! That said, I have not. I have a few more clients to fix this on and wondered if a cmupdate would actually fix it, or if I should run the commands upthread.

 

I ran into my first issue with this. This is the error I'm seeing in the EC2 system log:

Code:

[   30.146592] cloud-init[595]: Traceback (most recent call last):
[   30.149869] cloud-init[595]: File "/usr/bin/cloud-init", line 9, in <module>
[   30.155197] cloud-init[595]: load_entry_point('cloud-init==18.5', 'console_scripts', 'cloud-init')()
[   30.160468] cloud-init[595]: File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
[   30.205245] cloud-init[595]: return get_distribution(dist).load_entry_point(group, name)
[   30.209864] cloud-init[595]: File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2852, in load_entry_point
[   30.215741] cloud-init[595]: return ep.load()
[   30.218421] cloud-init[595]: File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2443, in load
[   30.222985] cloud-init[595]: return self.resolve()
[   30.225465] cloud-init[595]: File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2449, in resolve
[   30.230243] cloud-init[595]: module = __import__(self.module_name, fromlist=['__name__'], level=0)
[   30.234383] cloud-init[595]: File "/usr/lib/python2.7/site-packages/cloudinit/cmd/main.py", line 26, in <module>
[   30.239522] cloud-init[595]: from cloudinit import netinfo
[   30.242918] cloud-init[595]: File "/usr/lib/python2.7/site-packages/cloudinit/netinfo.py", line 15, in <module>
[   30.252793] cloud-init[595]: from cloudinit.net.network_state import net_prefix_to_ipv4_mask
[   30.257702] cloud-init[595]: File "/usr/lib/python2.7/site-packages/cloudinit/net/__init__.py", line 13, in <module>
[   30.325209] cloud-init[595]: from cloudinit.net.network_state import mask_to_net_prefix
[   30.329107] cloud-init[595]: File "/usr/lib/python2.7/site-packages/cloudinit/net/network_state.py", line 15, in <module>
[   30.334797] cloud-init[595]: from cloudinit import util
[   30.337440] cloud-init[595]: File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 49, in <module>
[   30.341584] cloud-init[595]: from cloudinit import url_helper
[   30.344612] cloud-init[595]: File "/usr/lib/python2.7/site-packages/cloudinit/url_helper.py", line 13, in <module>
[   30.348798] cloud-init[595]: import requests
[   30.351125] cloud-init[595]: File "/usr/lib/python2.7/site-packages/requests/__init__.py", line 54, in <module>
[   30.484705] cloud-init[595]: pyopenssl.inject_into_urllib3()
[   30.488513] cloud-init[595]: AttributeError: 'module' object has no attribute 'inject_into_urllib3'
[[1;31mFAILED[0m] Failed to start Initial cloud-init job (pre-networking).

Any idea what that might be?

 

This was on a test restore of a fixed instance. That was the error as it was booting up. As somewhat expected, cloud-init isn't working, so I am unable to connect to the instance.