[AusNOG] census issues tonight

Wed Aug 10 08:35:10 EST 2016

Didn't have to do with Geolocation at all, just don't advertise the prefix down any international paths.  If your not on a domestic carrier, no site, that simple.

I'm sure there is a small percentage that would have had issues who use Proxies, VPN's etc but would be a small minority compared to the amount of people who were inconvenienced.

Kindest Regards,

Nathan Brookfield (VK2NAB)

Chief Executive Officer

Simtronic Technologies Pty Ltd

Local: (02) 4749 4949 | Fax: (02) 4749 4950 | Direct: (02) 4749 4951

Web: http://www.simtronic.com.au<http://www.simtronic.com.au/> | E-mail: nathan.brookfield at simtronic.com.au<mailto:nathan.brookfield at simtronic.com.au>

________________________________
From: James Hodgkinson <yaleman at ricetek.net>
Sent: Wednesday, 10 August 2016 8:11 AM
To: James Braunegg; Nathan Brookfield; Tim Raphael; ausnog at lists.ausnog.net
Subject: Re: [AusNOG] census issues tonight

"I still don’t understand why the /24 was even advertised internationally at all…  doesn’t make sense for a website that only Australians should have been accessing from within Australia."

Because geolocation is poor guesswork at the best of times? We run into issues with it all the time. I'm surprised they had their "CDN" nodes in the same range (and certificate, seriously, lazy much?)

James

On Tue, 9 Aug 2016, at 23:44, James Braunegg wrote:

You don’t reroute traffic…. From having an application fail…

You might reroute traffic from network congestion…. But you only black hole /32 routes when you want to stop unwanted traffic with a sledge hammer….

I still don’t understand why the /24 was even advertised internationally at all…  doesn’t make sense for a website that only Australians should have been accessing from within Australia.

Kindest Regards

James Braunegg
P:  1300 769 972  | M:  0488 997 207 | D:  (03) 9751 7616

E: james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>  | ABN:  12 109 977 666
W:  www.micron21.com/ddos-protection<http://www.micron21.com/ddos-protection> T: @micron21

Follow us on Twitter<http://www.twitter.com/micron21> for important service and system updates.
[M21.jpg]

This message is intended for the addressee named above. It may contain privileged or confidential information. If you are not the intended recipient of this message you must not use, copy, distribute or disclose it to anyone other than the addressee. If you have received this message in error please return the message to the sender by replying to it and then delete the message from your computer.

From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Nathan Brookfield
Sent: Tuesday, 9 August 2016 11:39 PM
To: Tim Raphael <raphael.timothy at gmail.com>; ausnog at lists.ausnog.net
Subject: Re: [AusNOG] census issues tonight

If you add that to Vocus blackholing some /32's intermittently, I think we can safely assume that they were a tasty target [:(]  What an absolute disaster.....

Kindest Regards,

Nathan Brookfield (VK2NAB)

Chief Executive Officer

Simtronic Technologies Pty Ltd

Local: (02) 4749 4949 | Fax: (02) 4749 4950 | Direct: (02) 4749 4951

Web: http://www.simtronic.com.au<http://www.simtronic.com.au/> | E-mail: nathan.brookfield at simtronic.com.au<mailto:nathan.brookfield at simtronic.com.au>

________________________________

From: AusNOG <ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>> on behalf of Tim Raphael <raphael.timothy at gmail.com<mailto:raphael.timothy at gmail.com>>
Sent: Tuesday, 9 August 2016 11:25 PM
To: ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] census issues tonight

So what do we think?

DDoS? (as in illegitimate traffic)

I think the evidence might be there given we saw some Telstra route changes that look an awful lot like scrubbing near the customer tail...

- Tim

On 9 Aug 2016, at 9:23 PM, Michael Schipp <michaelsc at mellanox.com<mailto:michaelsc at mellanox.com>> wrote:

So they now pay me $180?

From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Damian Guppy
Sent: Tuesday, 9 August 2016 11:11 PM
To: Joshua D'Alton <joshua at railgun.com.au<mailto:joshua at railgun.com.au>>
Cc: ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] census issues tonight

Well, looks like they have given up...

https://twitter.com/ABSCensus/status/762996836357419008

"ABS & Census website are unavailable. The service won't be restored tonight. We will update you in AM. We apologise for the inconvenience."

On Tue, Aug 9, 2016 at 9:05 PM, Joshua D'Alton <joshua at railgun.com.au<mailto:joshua at railgun.com.au>> wrote:

Surely those 11 would be the load balancing stack in front, then again 24 million requests lasting 10 minutes over 72hours is only ~1k requests per second, probably easily manageable by 11 servers with a bunch of big SQL boxes behind it.

Tracing to the IPs gives different results.

.1 includes ibm.

.2 goes straight.

.3 includes ibm

.4 goes straight

.5 includes ibm but fails to reach target

.6... *

On Tue, Aug 9, 2016 at 10:50 PM, James Braunegg <james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>> wrote:

Total of 11 servers used for the application stack….

https://twitter.com/oliyoung/status/761028000821288961

https://pbs.twimg.com/media/Co-2565VMAAZeBq.jpg

Thanks Tom M for the links

Kindest Regards

James Braunegg
P:  1300 769 972<tel:1300%20769%20972>  |  M:  0488 997 207<tel:0488%20997%20207> |  D:  (03) 9751 7616<tel:%2803%29%209751%207616>

E:   james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>  |  ABN:  12 109 977 666
W:  www.micron21.com/ddos-protection<http://www.micron21.com/ddos-protection>   T: @micron21

Follow us on Twitter<http://www.twitter.com/micron21> for important service and system updates.

<image002.jpg>

This message is intended for the addressee named above. It may contain privileged or confidential information. If you are not the intended recipient of this message you must not use, copy, distribute or disclose it to anyone other than the addressee. If you have received this message in error please return the message to the sender by replying to it and then delete the message from your computer.

From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>] On Behalf Of Troy Cowin
Sent: Tuesday, 9 August 2016 10:48 PM
To: Colin Stubbs <colin.stubbs at equatetechnologies.com.au<mailto:colin.stubbs at equatetechnologies.com.au>>; ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>

Subject: Re: [AusNOG] census issues tonight

And the rest…

http://i.imgur.com/N7tlah4.png

From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Colin Stubbs
Sent: Tuesday, 9 August 2016 8:35 PM
To: Glenn Powell <glenn at glennbridge.com.au<mailto:glenn at glennbridge.com.au>>
Cc: AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
Subject: Re: [AusNOG] census issues tonight

$54,367.50 to be precise...

https://www.tenders.gov.au/?event=public.cn.view&CNUUID=069BE127-081D-5AF9-EB096752848D31A8

On 9 Aug 2016 8:14 PM, "Glenn Powell" <glenn at glennbridge.com.au<mailto:glenn at glennbridge.com.au>> wrote:

They spent $50k on a 3rd party to do load testing, I wonder what the success criteria of that testing looked like?

On 9 Aug 2016, at 8:09 PM, Brent Paddon <brent.paddon at gmail.com<mailto:brent.paddon at gmail.com>> wrote:

Only cost $9.6M to host the site...

Brent

On Tue, Aug 9, 2016 at 8:07 PM, James Braunegg <james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>> wrote:

Dead as a doornail …. from Telstra…

How sad… Now What…

Kindest Regards

James Braunegg
P:  1300 769 972<tel:1300%20769%20972>  |  M:  0488 997 207<tel:0488%20997%20207> |  D:  (03) 9751 7616<tel:%2803%29%209751%207616>

E:   james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>  |  ABN:  12 109 977 666
W:  www.micron21.com/ddos-protection<http://www.micron21.com/ddos-protection>   T: @micron21

Follow us on Twitter<http://www.twitter.com/micron21> for important service and system updates.

<image001.jpg>

This message is intended for the addressee named above. It may contain privileged or confidential information. If you are not the intended recipient of this message you must not use, copy, distribute or disclose it to anyone other than the addressee. If you have received this message in error please return the message to the sender by replying to it and then delete the message from your computer.

From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>] On Behalf Of John Lindsay
Sent: Tuesday, 9 August 2016 7:52 PM
To: Joseph Goldman <joe at apcs.com.au<mailto:joe at apcs.com.au>>
Cc: ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] census issues tonight

It’s turned to glue for me.

Was great an hour or so ago.

John Lindsay

johnslindsay at mac.com<mailto:johnslindsay at mac.com>

+61403577711<tel:%2B61403577711>

On 9 Aug 2016, at 6:51 PM, Joseph Goldman <joe at apcs.com.au<mailto:joe at apcs.com.au>> wrote:

Just completed it then was super snappy for me, so assuming they either ramped up servers or people gave up lol.

On 09/08/16 18:56, Joshua D'Alton wrote:

Their phone number too just rings out with "we're too busy, please call back". Probably due to aforementioned issues.. and then the few million old people who are trying to do it over the phone. Or people who need to call because they were doing it and their session died and their 9 digit code no longer works :/

On Tue, Aug 9, 2016 at 6:33 PM, Troy Cowin <troy at perthsystems.com.au<mailto:troy at perthsystems.com.au>> wrote:

Yep lots of various errors even earlier this morning – took about 5 minutes of refreshing to actually get into the site. Once in though it worked ok, hate to see how it will handle the load now its peak hour.

From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>] On Behalf Of Ben Cornish
Sent: Tuesday, 9 August 2016 4:28 PM
To: ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>
Subject: [AusNOG] census issues tonight

Not surprising – but anyone else seeing issues with the Census site tonight ?

Seems Soft layer is serving up parts of this behind the scenes..

Im seeing  504 errors from soft layer for large chunks of IP ranges – but not all.

Anyone else seeing this ?

Ben Cornish
Chief Operating Officer

Over the Wire Holdings Ltd (ASX:OTW)

GPO Box 1807 Brisbane, QLD, 4001, Australia

Level 1, 24 Little Edward Street, Spring Hill, QLD 4000
t    1300 689 689<tel:1300%20689%20689>

m  0417 617 204<tel:0417%20617%20204>

e   Ben.Cornish at overthewire.com.au<mailto:Ben.Cornish at overthewire.com.au>     www.overthewire.com.au<http://www.overthewire.com.au/>

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
http://lists.ausnog.net/mailman/listinfo/ausnog

_______________________________________________

AusNOG mailing list

AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>

http://lists.ausnog.net/mailman/listinfo/ausnog

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
http://lists.ausnog.net/mailman/listinfo/ausnog

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
http://lists.ausnog.net/mailman/listinfo/ausnog

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
http://lists.ausnog.net/mailman/listinfo/ausnog

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
http://lists.ausnog.net/mailman/listinfo/ausnog

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
http://lists.ausnog.net/mailman/listinfo/ausnog

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
http://lists.ausnog.net/mailman/listinfo/ausnog

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
http://lists.ausnog.net/mailman/listinfo/ausnog

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
http://lists.ausnog.net/mailman/listinfo/ausnog

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20160809/b5e26fc4/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 2683 bytes
Desc: image001.jpg
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20160809/b5e26fc4/attachment-0001.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 506 bytes
Desc: image002.png
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20160809/b5e26fc4/attachment-0001.png>