<div dir="ltr"><div style>They have now put up an incident report, cause was a combination of a bad rule was applied to all edge routers across all 23 global datacenters using flowspec and a bug in Junos caused the routers to have a memory leak and crash when they processed the rule, to top things off their automated recovery tools couldnt reboot/recover the vast majority of the routers automatically, and the ones they could got flooded with all the traffic the rest of them would normally handle. They ended up having to get people onsite at all datacenters to physically hard reboot the routers.</div>

<div style><br></div><div style>Poor guys</div><div><br></div><a href="http://blog.cloudflare.com/todays-outage-post-mortem-82515">http://blog.cloudflare.com/todays-outage-post-mortem-82515</a><br><div><br></div><div style>

--Damian</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Sun, Mar 3, 2013 at 6:38 PM, Jared Hirst <span dir="ltr"><<a href="mailto:jared.hirst@serversaustralia.com.au" target="_blank">jared.hirst@serversaustralia.com.au</a>></span> wrote:<br>

<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">30 minutes so far offline </span><span style="font-size:11.0pt;font-family:Wingdings;color:#1f497d">L</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"></span></p>


<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Looks to be with the DNS cluster and route withdraws.</span></p>


<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><a href="http://www.webhostingtalk.com/showthread.php?t=1243320&page=3" target="_blank">http://www.webhostingtalk.com/showthread.php?t=1243320&page=3</a></span></p>


<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p><div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in"><p class="MsoNormal">


<b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> <a href="mailto:ausnog-bounces@lists.ausnog.net" target="_blank">ausnog-bounces@lists.ausnog.net</a> [mailto:<a href="mailto:ausnog-bounces@lists.ausnog.net" target="_blank">ausnog-bounces@lists.ausnog.net</a>] <b>On Behalf Of </b>Ross [Eve IT]<br>


<b>Sent:</b> Sunday, March 03, 2013 9:36 PM<br><b>To:</b> <a href="mailto:ausnog@lists.ausnog.net" target="_blank">ausnog@lists.ausnog.net</a><br><b>Subject:</b> [AusNOG] Cloudflare offline</span></p></div><div><div class="h5">

<p class="MsoNormal"> </p><div>
<p class="MsoNormal">@CloudFlare</p></div><div><p class="MsoNormal">We are experiencing a system wide issue. Our team is investigating. Will continue to update with information.</p></div><div><p class="MsoNormal"> </p></div>


<div><p class="MsoNormal">--</p></div><div><p class="MsoNormal">Ross.</p></div></div></div></div></div>
<br>_______________________________________________<br>
AusNOG mailing list<br>
<a href="mailto:AusNOG@lists.ausnog.net">AusNOG@lists.ausnog.net</a><br>
<a href="http://lists.ausnog.net/mailman/listinfo/ausnog" target="_blank">http://lists.ausnog.net/mailman/listinfo/ausnog</a><br>
<br></blockquote></div><br></div>