Learn about Centmin Mod LEMP Stack today
Register Now

Linode Connectivity Issues - Frankfurt

Discussion in 'VPS Provider Network Status' started by eva2000, Nov 1, 2017.

  1. eva2000

    eva2000 Administrator Staff Member

    30,634
    6,862
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,308
    Local Time:
    1:42 PM
    Nginx 1.13.x
    MariaDB 5.5
    Oct 4, 02:52 UTC
    Resolved - Being that we have not experienced additional connectivity issues affecting our Frankfurt data center, this matter is now resolved.

    If you are still experiencing connectivity issues, please reach out to our Customer Support Team for assistance.

    Oct 4, 01:57 UTC
    Monitoring - The connectivity issues from the scheduled network maintenance in our Frankfurt data center has been corrected. Our team is continuing to monitor the situation to ensure connectivity remains stable.

    If you are still experiencing connectivity issues, please reach out to our Customer Support Team for assistance.

    Oct 3, 23:55 UTC
    Investigating - We are aware of connectivity issues affecting Linodes in our Frankfurt data center as a result of the scheduled maintenance and are currently investigating. We will continue to provide additional updates as this incident develops.

    Oct 2, 03:32 UTC
    Monitoring - After many hours of troubleshooting with Cisco, it appears we have hit hardware and software failures on both internet facing edge routers simultaneously. The "A" side router had a module degrade silently, meaning that the module did not fully fail and caused strange forwarding behavior without notifying the operating system of the failure. Once the "A" side router was reloaded, the module failure became apparent during the boot process. Both redundant routers also experienced the same undocumented forwarding table bug that caused the blackholing of some customer traffic. At this time we have been able to normalize connectivity. A separate maintenance will be required to fully resolve this incident, at which point a full post mortem will be made available.

    Oct 2, 02:16 UTC
    Update - Our Network Operations team is continuing to work with Cisco on a resolution to these issues. We'll continue to provide updates as the situation develops.

    Oct 2, 01:01 UTC
    Update - Our engineers are still troubleshooting with Cisco. Cisco believes a software bug is causing the reachability problems we are seeing. A workaround is currently being developed. More updates to follow.

    Oct 1, 23:52 UTC
    Update - Both of our Frankfurt internet edge routers are experiencing issues with their forwarding tables. This is causing sporadic blackholing of customer traffic. Our engineers have reached out to the router vendor and are engaged with their engineering team. More updates will be provided as information become available.

    Oct 1, 23:26 UTC
    Update - We are currently still investigating the connectivity issues affecting Linodes in our Frankfurt data center. We will continue to provide additional updates as this incident develops.

    Oct 1, 14:14 UTC
    Investigating - We are aware of connectivity issues affecting Linodes in our Frankfurt data center and are currently investigating. We will continue to provide additional updates as this incident develops.

    Continue reading...