[AusNOG] Cloudflare offline

[Brad Peczka]

Just another reminder to everyone to ensure you've always got an Out-of-Band option for places you can't physically get to.

A console server and a switched PDU operating through 3G or a DSL line should be the bare minimum, depending on how important your kit is.

Regards,
-Brad.

________________________________________
From: ausnog-bounces at lists.ausnog.net [ausnog-bounces at lists.ausnog.net] On Behalf Of Damian Guppy [the.damo at gmail.com]
Sent: Sunday, 3 March 2013 10:08 PM
To: Jared Hirst
Cc: ausnog at lists.ausnog.net
Subject: Re: [AusNOG] Cloudflare offline

They have now put up an incident report, cause was a combination of a bad rule was applied to all edge routers across all 23 global datacenters using flowspec and a bug in Junos caused the routers to have a memory leak and crash when they processed the rule, to top things off their automated recovery tools couldnt reboot/recover the vast majority of the routers automatically, and the ones they could got flooded with all the traffic the rest of them would normally handle. They ended up having to get people onsite at all datacenters to physically hard reboot the routers.

Poor guys

http://blog.cloudflare.com/todays-outage-post-mortem-82515

--Damian


On Sun, Mar 3, 2013 at 6:38 PM, Jared Hirst <jared.hirst at serversaustralia.com.au<mailto:jared.hirst at serversaustralia.com.au>> wrote:
30 minutes so far offline ☹

Looks to be with the DNS cluster and route withdraws.

http://www.webhostingtalk.com/showthread.php?t=1243320&page=3

From: ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net> [mailto:ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>] On Behalf Of Ross [Eve IT]
Sent: Sunday, March 03, 2013 9:36 PM
To: ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>
Subject: [AusNOG] Cloudflare offline

@CloudFlare
We are experiencing a system wide issue. Our team is investigating. Will continue to update with information.

--
Ross.

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
http://lists.ausnog.net/mailman/listinfo/ausnog