[AusNOG] Cloudflare offline

[Damian Guppy]

They have now put up an incident report, cause was a combination of a bad
rule was applied to all edge routers across all 23 global datacenters using
flowspec and a bug in Junos caused the routers to have a memory leak and
crash when they processed the rule, to top things off their automated
recovery tools couldnt reboot/recover the vast majority of the routers
automatically, and the ones they could got flooded with all the traffic the
rest of them would normally handle. They ended up having to get people
onsite at all datacenters to physically hard reboot the routers.

Poor guys

http://blog.cloudflare.com/todays-outage-post-mortem-82515

--Damian


On Sun, Mar 3, 2013 at 6:38 PM, Jared Hirst <
jared.hirst at serversaustralia.com.au> wrote:

> 30 minutes so far offline L
>
>
>
> Looks to be with the DNS cluster and route withdraws.
>
>
>
> http://www.webhostingtalk.com/showthread.php?t=1243320&page=3
>
>
>
> *From:* ausnog-bounces at lists.ausnog.net [mailto:
> ausnog-bounces at lists.ausnog.net] *On Behalf Of *Ross [Eve IT]
> *Sent:* Sunday, March 03, 2013 9:36 PM
> *To:* ausnog at lists.ausnog.net
> *Subject:* [AusNOG] Cloudflare offline
>
>
>
> @CloudFlare
>
> We are experiencing a system wide issue. Our team is investigating. Will
> continue to update with information.
>
>
>
> --
>
> Ross.
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20130303/1ad6ea3d/attachment.html>