<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.apple-style-span
{mso-style-name:apple-style-span;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>It takes the referred to level of noise to take the focus away from Juniper’s gross unreliability!<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> ausnog-bounces@lists.ausnog.net [mailto:ausnog-bounces@lists.ausnog.net] <b>On Behalf Of </b>Ben Dale<br><b>Sent:</b> Tuesday, March 05, 2013 9:42 AM<br><b>To:</b> ausnog@lists.ausnog.net<br><b>Subject:</b> Re: [AusNOG] Cloudflare offline<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Just to close this one off for everyone who hit me up offline, it looks like they hit an existing PR 734453.<o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>After upgrading to the release suggested, lab tests confirm the issue is no longer happening (though abnormally high values are still accepted).<o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>You may now return to your regular arguments over historical decisions.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><div><p class=MsoNormal><span style='font-size:13.5pt;font-family:"Helvetica","sans-serif";color:black'>Cheers,<br><br>Ben<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><div><div><p class=MsoNormal>On 04/03/2013, at 4:35 PM, Ben Dale <<a href="mailto:bdale@comlinx.com.au">bdale@comlinx.com.au</a>> wrote:<o:p></o:p></p></div><p class=MsoNormal><br><br><o:p></o:p></p><div><div><p class=MsoNormal><o:p> </o:p></p></div><div><div><p class=MsoNormal>On 04/03/2013, at 12:08 AM, Damian Guppy <<a href="mailto:the.damo@gmail.com">the.damo@gmail.com</a>> wrote:<o:p></o:p></p></div><p class=MsoNormal><br><br><o:p></o:p></p><div><div><p class=MsoNormal>They have now put up an incident report, cause was a combination of a bad rule was applied to all edge routers across all 23 global datacenters using flowspec and a bug in Junos caused the routers to have a memory leak and crash when they processed the rule, to top things off their automated recovery tools couldnt reboot/recover the vast majority of the routers automatically, and the ones they could got flooded with all the traffic the rest of them would normally handle. They ended up having to get people onsite at all datacenters to physically hard reboot the routers.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Poor guys<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><p class=MsoNormal><a href="http://blog.cloudflare.com/todays-outage-post-mortem-82515">http://blog.cloudflare.com/todays-outage-post-mortem-82515</a><o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>--Damian<o:p></o:p></p></div></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><div><p class=MsoNormal>Bug looks to be pretty easy to reproduce too (in an arbitrary version):<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>bdale@mx80-bng1> show route table inetflow.0<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><div><p class=MsoNormal>inetflow.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)<o:p></o:p></p></div><div><p class=MsoNormal>Restart Complete<o:p></o:p></p></div><div><p class=MsoNormal>+ = Active Route, - = Last Active, * = Both<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>173.2.3.4,*,port=53,len=99971,=99985/term:N/A <o:p></o:p></p></div><div><p class=MsoNormal> *[Flow/5] 00:01:37<o:p></o:p></p></div><div><p class=MsoNormal> Fictitious<o:p></o:p></p></div></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>bdale@mx80-bng1> show chassis routing-engine | match Mem<o:p></o:p></p></div><div><p class=MsoNormal> Memory utilization 37 percent<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>... after taking a swig of beverage <o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><div><p class=MsoNormal>bdale@mx80-bng1> show chassis routing-engine | match Mem<o:p></o:p></p></div><div><p class=MsoNormal> Memory utilization 97 percent<o:p></o:p></p></div></div><div><div><p class=MsoNormal>bdale@mx80-bng1> show chassis routing-engine | match Mem<o:p></o:p></p></div><div><p class=MsoNormal> Memory utilization 99 percent<o:p></o:p></p></div></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>It also pegs the CPU up to maximum during this time.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Anyone using Flowspec out there might want to take a good hard look at your validation until this is addressed (a commit script would do the trick). Especially so, those receiving Flowspec via BGP from external sources eg: Team Cymru (should be just prefixes), Arbor (Roland may have more insight on sizing validation) etc as there appears to be no way to filter/validate specific rules (just sources you learnt them from).<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>I've tried a few other "illegal" values (eg: 65537, 65555) for packet length, but nothing kicks it off like the Cloudflare sizes (the rate memory is consumed *may* be proportional to the size of the packet described). Removing the route prior to topping out doesn't reclaim the memory either : (<o:p></o:p></p></div></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Ben<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div></div></div><p class=MsoNormal>_______________________________________________<br>AusNOG mailing list<br><a href="mailto:AusNOG@lists.ausnog.net">AusNOG@lists.ausnog.net</a><br><a href="http://lists.ausnog.net/mailman/listinfo/ausnog">http://lists.ausnog.net/mailman/listinfo/ausnog</a><o:p></o:p></p></div><p class=MsoNormal><o:p> </o:p></p></div></div></body></html>