Linode Connectivity Issues - Frankfurt

Oct 4, 02:52 UTC
Resolved - Being that we have not experienced additional connectivity issues affecting our Frankfurt data center, this matter is now resolved.

If you are still experiencing connectivity issues, please reach out to our Customer Support Team for assistance.

Oct 4, 01:57 UTC
Monitoring - The connectivity issues from the scheduled network maintenance in our Frankfurt data center has been corrected. Our team is continuing to monitor the situation to ensure connectivity remains stable.

If you are still experiencing connectivity issues, please reach out to our Customer Support Team for assistance.

Oct 3, 23:55 UTC
Investigating - We are aware of connectivity issues affecting Linodes in our Frankfurt data center as a result of the scheduled maintenance and are currently investigating. We will continue to provide additional updates as this incident develops.

Oct 2, 03:32 UTC
Monitoring - After many hours of troubleshooting with Cisco, it appears we have hit hardware and software failures on both internet facing edge routers simultaneously. The "A" side router had a module degrade silently, meaning that the module did not fully fail and caused strange forwarding behavior without notifying the operating system of the failure. Once the "A" side router was reloaded, the module failure became apparent during the boot process. Both redundant routers also experienced the same undocumented forwarding table bug that caused the blackholing of some customer traffic. At this time we have been able to normalize connectivity. A separate maintenance will be required to fully resolve this incident, at which point a full post mortem will be made available.

Oct 2, 02:16 UTC
Update - Our Network Operations team is continuing to work with Cisco on a resolution to these issues. We'll continue to provide updates as the situation develops.

---

Oct 2, 01:01 UTC
Update - Our engineers are still troubleshooting with Cisco. Cisco believes a software bug is causing the reachability problems we are seeing. A workaround is currently being developed. More updates to follow.

Oct 1, 23:52 UTC
Update - Both of our Frankfurt internet edge routers are experiencing issues with their forwarding tables. This is causing sporadic blackholing of customer traffic. Our engineers have reached out to the router vendor and are engaged with their engineering team. More updates will be provided as information become available.

Oct 1, 23:26 UTC
Update - We are currently still investigating the connectivity issues affecting Linodes in our Frankfurt data center. We will continue to provide additional updates as this incident develops.

Oct 1, 14:14 UTC
Investigating - We are aware of connectivity issues affecting Linodes in our Frankfurt data center and are currently investigating. We will continue to provide additional updates as this incident develops.

Continue reading...