[AusNOG] census issues tonight

[James Braunegg]

Dear Andy

If you look on public peering exchanges around the world for an abnormal increase of traffic during last night, you don't see any such increase, nor any evidence.

I.e. When CloudFlare reported a large 300gbit+ scale NTP reflection attack back in 2014 peering's graphs around the world showed a spike in traffic.

If large amounts of traffic was heading towards Australia say via Vocus, Nextgen or on Telstra network last night which was so large and uncontrollable you would have expected to see packet loss on international links, however nobody saw anything.

So was this a super large DDoS attack, I doubt it, was it even a DDoS attack ... maybe it was maybe it wasn't... despite what the media are saying.

However remember if it was an attack it didn't have to be super large to cause an effect to the platform if the platform was only designed for 10gbit of capacity.

It could have also been an application based attack say TCP traffic instead of the typical flood your network UDP style attack, it could have also been legitimate Australian users just overloading the system, (VIPDoS / Click Frenzy)  I could also sit here all day and assuming what it was or was not... The only ones who would actually know what occurred would be if you're a network operator within Vocus, Nextgen or Telstra.

As for what was observed.. that's easy.

Packet Loss, high pings, route's both internationally and domestically changing, the first 20 IP addresses of 150.207.169.0/24 null routed via /32 routes both originating from NextGen and from Vocus followed by traffic shifting to Telstra network and then the network returning to normal but the application remaining offline.

Kindest Regards

James Braunegg
P:  1300 769 972  |  M:  0488 997 207 |  D:  (03) 9751 7616
E:   james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>  |  ABN:  12 109 977 666
W:  www.micron21.com/ddos-protection<http://www.micron21.com/ddos-protection>   T: @micron21

Follow us on Twitter<http://www.twitter.com/micron21> for important service and system updates.

[M21.jpg]

This message is intended for the addressee named above. It may contain privileged or confidential information. If you are not the intended recipient of this message you must not use, copy, distribute or disclose it to anyone other than the addressee. If you have received this message in error please return the message to the sender by replying to it and then delete the message from your computer.

From: Andy Taylor [mailto:andy at coastalaudio.com.au]
Sent: Wednesday, 10 August 2016 9:11 AM
To: 'Nathan Brookfield' <Nathan.Brookfield at simtronic.com.au>; 'James Hodgkinson' <yaleman at ricetek.net>; James Braunegg <james.braunegg at micron21.com>; 'Tim Raphael' <raphael.timothy at gmail.com>; ausnog at lists.ausnog.net
Subject: RE: [AusNOG] census issues tonight

So...what evidence (if any) is there to suggest that this was a DDoS attack originating from O/S...?
I'm assuming that ASD are onto it...and that they are aware of proxy chaining...?
Was Tomcat properly hardened, was there any form of DDoS mitigation in place?
One would think that, given the current animosity between Australia and China (Olympics and Spratleys)...
Not only would load testing have been in place, but a comprehensive PenTest would have been conducted...?

Andy Taylor
Technical Director

0424 656 973

[ca_logo]

www.coastalaudio.com.au<http://www.coastalaudio.com.au/>



From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Nathan Brookfield
Sent: Wednesday, 10 August 2016 8:35 AM
To: James Hodgkinson <yaleman at ricetek.net<mailto:yaleman at ricetek.net>>; James Braunegg <james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>>; Tim Raphael <raphael.timothy at gmail.com<mailto:raphael.timothy at gmail.com>>; ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] census issues tonight


Didn't have to do with Geolocation at all, just don't advertise the prefix down any international paths.  If your not on a domestic carrier, no site, that simple.



I'm sure there is a small percentage that would have had issues who use Proxies, VPN's etc but would be a small minority compared to the amount of people who were inconvenienced.



Kindest Regards,

Nathan Brookfield (VK2NAB)



Chief Executive Officer

Simtronic Technologies Pty Ltd



Local: (02) 4749 4949 | Fax: (02) 4749 4950 | Direct: (02) 4749 4951

Web: http://www.simtronic.com.au<http://www.simtronic.com.au/> | E-mail: nathan.brookfield at simtronic.com.au<mailto:nathan.brookfield at simtronic.com.au>



________________________________
From: James Hodgkinson <yaleman at ricetek.net<mailto:yaleman at ricetek.net>>
Sent: Wednesday, 10 August 2016 8:11 AM
To: James Braunegg; Nathan Brookfield; Tim Raphael; ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] census issues tonight


"I still don't understand why the /24 was even advertised internationally at all...  doesn't make sense for a website that only Australians should have been accessing from within Australia."

Because geolocation is poor guesswork at the best of times? We run into issues with it all the time. I'm surprised they had their "CDN" nodes in the same range (and certificate, seriously, lazy much?)

James

On Tue, 9 Aug 2016, at 23:44, James Braunegg wrote:

You don't reroute traffic.... From having an application fail...



You might reroute traffic from network congestion.... But you only black hole /32 routes when you want to stop unwanted traffic with a sledge hammer....



I still don't understand why the /24 was even advertised internationally at all...  doesn't make sense for a website that only Australians should have been accessing from within Australia.



Kindest Regards



James Braunegg
P:  1300 769 972  | M:  0488 997 207 | D:  (03) 9751 7616

E: james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>  | ABN:  12 109 977 666
W:  www.micron21.com/ddos-protection<http://www.micron21.com/ddos-protection> T: @micron21



Follow us on Twitter<http://www.twitter.com/micron21> for important service and system updates.
[M21.jpg]

This message is intended for the addressee named above. It may contain privileged or confidential information. If you are not the intended recipient of this message you must not use, copy, distribute or disclose it to anyone other than the addressee. If you have received this message in error please return the message to the sender by replying to it and then delete the message from your computer.



From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Nathan Brookfield
Sent: Tuesday, 9 August 2016 11:39 PM
To: Tim Raphael <raphael.timothy at gmail.com<mailto:raphael.timothy at gmail.com>>; ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] census issues tonight



If you add that to Vocus blackholing some /32's intermittently, I think we can safely assume that they were a tasty target [:(]  What an absolute disaster.....



Kindest Regards,

Nathan Brookfield (VK2NAB)



Chief Executive Officer

Simtronic Technologies Pty Ltd



Local: (02) 4749 4949 | Fax: (02) 4749 4950 | Direct: (02) 4749 4951

Web: http://www.simtronic.com.au<http://www.simtronic.com.au/> | E-mail: nathan.brookfield at simtronic.com.au<mailto:nathan.brookfield at simtronic.com.au>



________________________________


From: AusNOG <ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>> on behalf of Tim Raphael <raphael.timothy at gmail.com<mailto:raphael.timothy at gmail.com>>
Sent: Tuesday, 9 August 2016 11:25 PM
To: ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] census issues tonight



So what do we think?



DDoS? (as in illegitimate traffic)



I think the evidence might be there given we saw some Telstra route changes that look an awful lot like scrubbing near the customer tail...





- Tim





On 9 Aug 2016, at 9:23 PM, Michael Schipp <michaelsc at mellanox.com<mailto:michaelsc at mellanox.com>> wrote:



So they now pay me $180?



From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Damian Guppy
Sent: Tuesday, 9 August 2016 11:11 PM
To: Joshua D'Alton <joshua at railgun.com.au<mailto:joshua at railgun.com.au>>
Cc: ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] census issues tonight



Well, looks like they have given up...



https://twitter.com/ABSCensus/status/762996836357419008



"ABS & Census website are unavailable. The service won't be restored tonight. We will update you in AM. We apologise for the inconvenience."







On Tue, Aug 9, 2016 at 9:05 PM, Joshua D'Alton <joshua at railgun.com.au<mailto:joshua at railgun.com.au>> wrote:

Surely those 11 would be the load balancing stack in front, then again 24 million requests lasting 10 minutes over 72hours is only ~1k requests per second, probably easily manageable by 11 servers with a bunch of big SQL boxes behind it.



Tracing to the IPs gives different results.



.1 includes ibm.

.2 goes straight.

.3 includes ibm

.4 goes straight

.5 includes ibm but fails to reach target

.6... *



On Tue, Aug 9, 2016 at 10:50 PM, James Braunegg <james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>> wrote:

Total of 11 servers used for the application stack....



https://twitter.com/oliyoung/status/761028000821288961



https://pbs.twimg.com/media/Co-2565VMAAZeBq.jpg



Thanks Tom M for the links



Kindest Regards



James Braunegg
P:  1300 769 972<tel:1300%20769%20972>  |  M:  0488 997 207<tel:0488%20997%20207> |  D:  (03) 9751 7616<tel:%2803%29%209751%207616>

E:   james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>  |  ABN:  12 109 977 666
W:  www.micron21.com/ddos-protection<http://www.micron21.com/ddos-protection>   T: @micron21



Follow us on Twitter<http://www.twitter.com/micron21> for important service and system updates.

<image002.jpg>

This message is intended for the addressee named above. It may contain privileged or confidential information. If you are not the intended recipient of this message you must not use, copy, distribute or disclose it to anyone other than the addressee. If you have received this message in error please return the message to the sender by replying to it and then delete the message from your computer.



From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>] On Behalf Of Troy Cowin
Sent: Tuesday, 9 August 2016 10:48 PM
To: Colin Stubbs <colin.stubbs at equatetechnologies.com.au<mailto:colin.stubbs at equatetechnologies.com.au>>; ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>

Subject: Re: [AusNOG] census issues tonight



And the rest...



http://i.imgur.com/N7tlah4.png





From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Colin Stubbs
Sent: Tuesday, 9 August 2016 8:35 PM
To: Glenn Powell <glenn at glennbridge.com.au<mailto:glenn at glennbridge.com.au>>
Cc: AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
Subject: Re: [AusNOG] census issues tonight



$54,367.50 to be precise...

https://www.tenders.gov.au/?event=public.cn.view&CNUUID=069BE127-081D-5AF9-EB096752848D31A8



On 9 Aug 2016 8:14 PM, "Glenn Powell" <glenn at glennbridge.com.au<mailto:glenn at glennbridge.com.au>> wrote:

They spent $50k on a 3rd party to do load testing, I wonder what the success criteria of that testing looked like?





On 9 Aug 2016, at 8:09 PM, Brent Paddon <brent.paddon at gmail.com<mailto:brent.paddon at gmail.com>> wrote:



Only cost $9.6M to host the site...



Brent



On Tue, Aug 9, 2016 at 8:07 PM, James Braunegg <james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>> wrote:

Dead as a doornail .... from Telstra...



How sad... Now What...



Kindest Regards



James Braunegg
P:  1300 769 972<tel:1300%20769%20972>  |  M:  0488 997 207<tel:0488%20997%20207> |  D:  (03) 9751 7616<tel:%2803%29%209751%207616>

E:   james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>  |  ABN:  12 109 977 666
W:  www.micron21.com/ddos-protection<http://www.micron21.com/ddos-protection>   T: @micron21



Follow us on Twitter<http://www.twitter.com/micron21> for important service and system updates.

<image001.jpg>

This message is intended for the addressee named above. It may contain privileged or confidential information. If you are not the intended recipient of this message you must not use, copy, distribute or disclose it to anyone other than the addressee. If you have received this message in error please return the message to the sender by replying to it and then delete the message from your computer.



From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>] On Behalf Of John Lindsay
Sent: Tuesday, 9 August 2016 7:52 PM
To: Joseph Goldman <joe at apcs.com.au<mailto:joe at apcs.com.au>>
Cc: ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] census issues tonight



It's turned to glue for me.



Was great an hour or so ago.



John Lindsay

johnslindsay at mac.com<mailto:johnslindsay at mac.com>

+61403577711<tel:%2B61403577711>



On 9 Aug 2016, at 6:51 PM, Joseph Goldman <joe at apcs.com.au<mailto:joe at apcs.com.au>> wrote:



Just completed it then was super snappy for me, so assuming they either ramped up servers or people gave up lol.

On 09/08/16 18:56, Joshua D'Alton wrote:

Their phone number too just rings out with "we're too busy, please call back". Probably due to aforementioned issues.. and then the few million old people who are trying to do it over the phone. Or people who need to call because they were doing it and their session died and their 9 digit code no longer works :/



On Tue, Aug 9, 2016 at 6:33 PM, Troy Cowin <troy at perthsystems.com.au<mailto:troy at perthsystems.com.au>> wrote:

Yep lots of various errors even earlier this morning - took about 5 minutes of refreshing to actually get into the site. Once in though it worked ok, hate to see how it will handle the load now its peak hour.





From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>] On Behalf Of Ben Cornish
Sent: Tuesday, 9 August 2016 4:28 PM
To: ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>
Subject: [AusNOG] census issues tonight



Not surprising - but anyone else seeing issues with the Census site tonight ?



Seems Soft layer is serving up parts of this behind the scenes..

Im seeing  504 errors from soft layer for large chunks of IP ranges - but not all.



Anyone else seeing this ?









Ben Cornish
Chief Operating Officer





Over the Wire Holdings Ltd (ASX:OTW)

GPO Box 1807 Brisbane, QLD, 4001, Australia

Level 1, 24 Little Edward Street, Spring Hill, QLD 4000
t    1300 689 689<tel:1300%20689%20689>

m  0417 617 204<tel:0417%20617%20204>

e   Ben.Cornish at overthewire.com.au<mailto:Ben.Cornish at overthewire.com.au>     www.overthewire.com.au<http://www.overthewire.com.au/>





_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
http://lists.ausnog.net/mailman/listinfo/ausnog





_______________________________________________

AusNOG mailing list

AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>

http://lists.ausnog.net/mailman/listinfo/ausnog



_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
http://lists.ausnog.net/mailman/listinfo/ausnog



_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
http://lists.ausnog.net/mailman/listinfo/ausnog



_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
http://lists.ausnog.net/mailman/listinfo/ausnog



_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
http://lists.ausnog.net/mailman/listinfo/ausnog

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
http://lists.ausnog.net/mailman/listinfo/ausnog



_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
http://lists.ausnog.net/mailman/listinfo/ausnog



_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
http://lists.ausnog.net/mailman/listinfo/ausnog


_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
http://lists.ausnog.net/mailman/listinfo/ausnog

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20160810/8eb25aae/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 2683 bytes
Desc: image001.jpg
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20160810/8eb25aae/attachment-0001.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 16869 bytes
Desc: image002.png
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20160810/8eb25aae/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 506 bytes
Desc: image003.png
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20160810/8eb25aae/attachment-0003.png>