[AusNOG] Cloudflare offline

[Tony]

Yes, when it was pointed out by Jonathon I thought the same thing. From the outage report they put up:

"our attack profiler output the fact that the attack packets were between 99,971 and 99,985 bytes long"

There is no physical way that is possible with a 16-bit field, so the "profiler" has obviously spat out the wrong information, possibly due to some other sort of mangled header.

The person then took that output and created a rule based on it which was pushed out to the routers.
The routers probably barfed because the rule was out of bounds and never expected a figure that large. It is still a bug there though as the router OS should have parsed it and rejected the rule based on illegal values.

All very interesting and as per other posts good on them for admitting what the problem was and helping others to avoid it.



>________________________________
> From: Martin - StudioCoast <martin.sinclair at studiocoast.com.au>
>To: ausnog at lists.ausnog.net 
>Sent: Monday, 4 March 2013 12:53 PM
>Subject: Re: [AusNOG] Cloudflare offline
> 
>
>The problem is it's impossible to forgeaUDP header to be greater than 65535 no matter what you put in the header. The only exception is some exotic IPv6 jumbograms which I doubt are very widely supported (or routed).
>Sounds like more of a bug in thecloudflare profile systems than anything.
>
>Martin
> 
>
>
On 4/03/2013 9:36 AM, Mark Tees wrote:
>
>The mentioned on their blog that config walk likely wrong as it was typed out manually. The rule should have been 99971-99985 if it was legitimate. But that was never legitimate. Any attack signature that had a packet size larger than what is real shouldn't have made it in. A guess would be that maybe their detection system picked up packets with forged headers. On 04/03/2013, at 9:40 AM, Jonathan Thorpe wrote: 
>>There appears to be something a little odd about the rule they've installed.  +    route 173.X.X.X/32-DNS-DROP {
+        match {
+            destination 173.X.X.X/32;
+            port 53;
+            packet-length [ 99971 99985 ];
+        }
+        then discard;
+    } Specifically, they mention "In this case, our attack profiler output the fact that the attack packets were between 99,971 and 99,985 bytes long". Unless there's a typo in the report, the packet-length statement above only includes the two listed. The other thing that looks weird is that IPv4 supports a maximum packet length header of 16-bits (65,535), so how are packet lengths above this possible? Kind Regards,
Jonathan From: ausnog-bounces at lists.ausnog.net [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Jeffrey Sims
Sent: Monday, 4 March 2013 8:55 AM
To: Don Gould
Cc: ausnog at lists.ausnog.net Subject: Re: [AusNOG] Cloudflare offline +1 for a great approach by Cloudflare! Agreed, now people know there is a potential bug in JunOS that eegits of the internet might try again (ie: mimik the packet size again to trigger the bug conditions to crash the network).  Collaboration is the key! On Mon, Mar 4, 2013 at 8:31 AM, Don Gould <don at bowenvale.co.nz> wrote: On 4/03/2013 10:21 a.m., Jethro Carr wrote:
On Mon, 2013-03-04 at 04:37 +1100, Emily Ozols wrote:
So... Who's getting fired? <bit of snipping...> the key difference
between a competent engineer and an incompetent one is whether they
followed processes and handled the disaster in a professional manner. I think a key difference is when people just fess up to a mistake so everyone can learn from it! :) Nothing worse than wondering what caused the stuff up and if you could have been part of the cause. Even worse is making a stuff up that you might not have if you'd just known that someone else had done exactly the same thing the week before but didn't share because they were fearful of not being able to feed their kids the next week. +1 for a great approach by Cloudflare! D -- 
Don Gould
31 Acheson Ave
Mairehau
Christchurch, New Zealand
Ph: + 64 3 348 7235
Mobile: + 64 21 114 0699 _______________________________________________
AusNOG mailing list AusNOG at lists.ausnog.net http://lists.ausnog.net/mailman/listinfo/ausnog _______________________________________________
AusNOG mailing list AusNOG at lists.ausnog.net http://lists.ausnog.net/mailman/listinfo/ausnog
>>_______________________________________________
AusNOG mailing list AusNOG at lists.ausnog.net http://lists.ausnog.net/mailman/listinfo/ausnog
>
>_______________________________________________
>AusNOG mailing list
>AusNOG at lists.ausnog.net
>http://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>