[AusNOG] census issues tonight

Tim Raphael raphael.timothy at gmail.com
Tue Aug 9 23:50:39 EST 2016 

Exactly - that plus the mysterious disappearance of hop in the path just before the customer tail for some percentage of traffic.

I would imagine there might be some services that require international connectivity - perhaps even offshore engineers / TAC / IBM support for 24/7 monitoring etc.

- Tim


> On 9 Aug 2016, at 9:44 PM, James Braunegg <james.braunegg at micron21.com> wrote:
> 
> You don’t reroute traffic…. From having an application fail…
>  
> You might reroute traffic from network congestion…. But you only black hole /32 routes when you want to stop unwanted traffic with a sledge hammer….
>  
> I still don’t understand why the /24 was even advertised internationally at all…  doesn’t make sense for a website that only Australians should have been accessing from within Australia.
>  
> Kindest Regards
>  
> James Braunegg
> P:  1300 769 972  |  M:  0488 997 207 |  D:  (03) 9751 7616
> E:   james.braunegg at micron21.com <mailto:james.braunegg at micron21.com>  |  ABN:  12 109 977 666   
> W:  www.micron21.com/ddos-protection <http://www.micron21.com/ddos-protection>   T: @micron21
>  
> Follow us on Twitter <http://www.twitter.com/micron21> for important service and system updates.
> 
> <image001.jpg>
> 
> This message is intended for the addressee named above. It may contain privileged or confidential information. If you are not the intended recipient of this message you must not use, copy, distribute or disclose it to anyone other than the addressee. If you have received this message in error please return the message to the sender by replying to it and then delete the message from your computer.
>  
> From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net <mailto:ausnog-bounces at lists.ausnog.net>] On Behalf Of Nathan Brookfield
> Sent: Tuesday, 9 August 2016 11:39 PM
> To: Tim Raphael <raphael.timothy at gmail.com <mailto:raphael.timothy at gmail.com>>; ausnog at lists.ausnog.net <mailto:ausnog at lists.ausnog.net>
> Subject: Re: [AusNOG] census issues tonight
>  
> If you add that to Vocus blackholing some /32's intermittently, I think we can safely assume that they were a tasty target <image002.png> What an absolute disaster.....
>  
> Kindest Regards,
> Nathan Brookfield (VK2NAB)
>  
> Chief Executive Officer
> Simtronic Technologies Pty Ltd
>  
> Local: (02) 4749 4949 | Fax: (02) 4749 4950 | Direct: (02) 4749 4951
> Web: http://www.simtronic.com.au <http://www.simtronic.com.au/> | E-mail: nathan.brookfield at simtronic.com.au <mailto:nathan.brookfield at simtronic.com.au>
>  
> From: AusNOG <ausnog-bounces at lists.ausnog.net <mailto:ausnog-bounces at lists.ausnog.net>> on behalf of Tim Raphael <raphael.timothy at gmail.com <mailto:raphael.timothy at gmail.com>>
> Sent: Tuesday, 9 August 2016 11:25 PM
> To: ausnog at lists.ausnog.net <mailto:ausnog at lists.ausnog.net>
> Subject: Re: [AusNOG] census issues tonight
>  
> So what do we think?
>  
> DDoS? (as in illegitimate traffic)
>  
> I think the evidence might be there given we saw some Telstra route changes that look an awful lot like scrubbing near the customer tail...
>  
>  
> - Tim
>  
>  
> On 9 Aug 2016, at 9:23 PM, Michael Schipp <michaelsc at mellanox.com <mailto:michaelsc at mellanox.com>> wrote:
>  
> So they now pay me $180?
>  
> From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net <mailto:ausnog-bounces at lists.ausnog.net>] On Behalf Of Damian Guppy
> Sent: Tuesday, 9 August 2016 11:11 PM
> To: Joshua D'Alton <joshua at railgun.com.au <mailto:joshua at railgun.com.au>>
> Cc: ausnog at lists.ausnog.net <mailto:ausnog at lists.ausnog.net>
> Subject: Re: [AusNOG] census issues tonight
>  
> Well, looks like they have given up...
>  
> https://twitter.com/ABSCensus/status/762996836357419008 <https://twitter.com/ABSCensus/status/762996836357419008>
>  
> "ABS & Census website are unavailable. The service won't be restored tonight. We will update you in AM. We apologise for the inconvenience."
>  
>  
>  
> On Tue, Aug 9, 2016 at 9:05 PM, Joshua D'Alton <joshua at railgun.com.au <mailto:joshua at railgun.com.au>> wrote:
> Surely those 11 would be the load balancing stack in front, then again 24 million requests lasting 10 minutes over 72hours is only ~1k requests per second, probably easily manageable by 11 servers with a bunch of big SQL boxes behind it.
>  
> Tracing to the IPs gives different results.
>  
> .1 includes ibm.
> .2 goes straight.
> .3 includes ibm
> .4 goes straight
> .5 includes ibm but fails to reach target
> .6... *
>  
> On Tue, Aug 9, 2016 at 10:50 PM, James Braunegg <james.braunegg at micron21.com <mailto:james.braunegg at micron21.com>> wrote:
> Total of 11 servers used for the application stack….
>  
> https://twitter.com/oliyoung/status/761028000821288961 <https://twitter.com/oliyoung/status/761028000821288961>
>  
> https://pbs.twimg.com/media/Co-2565VMAAZeBq.jpg <https://pbs.twimg.com/media/Co-2565VMAAZeBq.jpg>
>  
> Thanks Tom M for the links
>  
> Kindest Regards
>  
> James Braunegg
> P:  1300 769 972 <tel:1300%20769%20972>  |  M:  0488 997 207 <tel:0488%20997%20207> |  D:  (03) 9751 7616 <tel:%2803%29%209751%207616>
> E:   james.braunegg at micron21.com <mailto:james.braunegg at micron21.com>  |  ABN:  12 109 977 666   
> W:  www.micron21.com/ddos-protection <http://www.micron21.com/ddos-protection>   T: @micron21
>  
> Follow us on Twitter <http://www.twitter.com/micron21> for important service and system updates.
> 
> <image002.jpg>
> 
> This message is intended for the addressee named above. It may contain privileged or confidential information. If you are not the intended recipient of this message you must not use, copy, distribute or disclose it to anyone other than the addressee. If you have received this message in error please return the message to the sender by replying to it and then delete the message from your computer.
>  
> From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net <mailto:ausnog-bounces at lists.ausnog.net>] On Behalf Of Troy Cowin
> Sent: Tuesday, 9 August 2016 10:48 PM
> To: Colin Stubbs <colin.stubbs at equatetechnologies.com.au <mailto:colin.stubbs at equatetechnologies.com.au>>; ausnog at lists.ausnog.net <mailto:ausnog at lists.ausnog.net>
> 
> Subject: Re: [AusNOG] census issues tonight
>  
> And the rest…
>  
> http://i.imgur.com/N7tlah4.png <http://i.imgur.com/N7tlah4.png>
>  
>  
> From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net <mailto:ausnog-bounces at lists.ausnog.net>] On Behalf Of Colin Stubbs
> Sent: Tuesday, 9 August 2016 8:35 PM
> To: Glenn Powell <glenn at glennbridge.com.au <mailto:glenn at glennbridge.com.au>>
> Cc: AusNOG at lists.ausnog.net <mailto:AusNOG at lists.ausnog.net>
> Subject: Re: [AusNOG] census issues tonight
>  
> $54,367.50 to be precise...
> https://www.tenders.gov.au/?event=public.cn.view&CNUUID=069BE127-081D-5AF9-EB096752848D31A8 <https://www.tenders.gov.au/?event=public.cn.view&CNUUID=069BE127-081D-5AF9-EB096752848D31A8>
>  
> On 9 Aug 2016 8:14 PM, "Glenn Powell" <glenn at glennbridge.com.au <mailto:glenn at glennbridge.com.au>> wrote:
> They spent $50k on a 3rd party to do load testing, I wonder what the success criteria of that testing looked like?
>  
>  
> On 9 Aug 2016, at 8:09 PM, Brent Paddon <brent.paddon at gmail.com <mailto:brent.paddon at gmail.com>> wrote:
>  
> Only cost $9.6M to host the site... 
>  
> Brent
>  
> On Tue, Aug 9, 2016 at 8:07 PM, James Braunegg <james.braunegg at micron21.com <mailto:james.braunegg at micron21.com>> wrote:
> Dead as a doornail …. from Telstra…
>  
> How sad… Now What…
>  
> Kindest Regards
>  
> James Braunegg
> P:  1300 769 972 <tel:1300%20769%20972>  |  M:  0488 997 207 <tel:0488%20997%20207> |  D:  (03) 9751 7616 <tel:%2803%29%209751%207616>
> E:   james.braunegg at micron21.com <mailto:james.braunegg at micron21.com>  |  ABN:  12 109 977 666   
> W:  www.micron21.com/ddos-protection <http://www.micron21.com/ddos-protection>   T: @micron21
>  
> Follow us on Twitter <http://www.twitter.com/micron21> for important service and system updates.
> 
> <image001.jpg>
> 
> This message is intended for the addressee named above. It may contain privileged or confidential information. If you are not the intended recipient of this message you must not use, copy, distribute or disclose it to anyone other than the addressee. If you have received this message in error please return the message to the sender by replying to it and then delete the message from your computer.
>  
> From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net <mailto:ausnog-bounces at lists.ausnog.net>] On Behalf Of John Lindsay
> Sent: Tuesday, 9 August 2016 7:52 PM
> To: Joseph Goldman <joe at apcs.com.au <mailto:joe at apcs.com.au>>
> Cc: ausnog at lists.ausnog.net <mailto:ausnog at lists.ausnog.net>
> Subject: Re: [AusNOG] census issues tonight
>  
> It’s turned to glue for me.
>  
> Was great an hour or so ago.
>  
> John Lindsay
> johnslindsay at mac.com <mailto:johnslindsay at mac.com>
> +61403577711 <tel:%2B61403577711>
>  
> On 9 Aug 2016, at 6:51 PM, Joseph Goldman <joe at apcs.com.au <mailto:joe at apcs.com.au>> wrote:
>  
> Just completed it then was super snappy for me, so assuming they either ramped up servers or people gave up lol.
> 
> On 09/08/16 18:56, Joshua D'Alton wrote:
> Their phone number too just rings out with "we're too busy, please call back". Probably due to aforementioned issues.. and then the few million old people who are trying to do it over the phone. Or people who need to call because they were doing it and their session died and their 9 digit code no longer works :/
>  
> On Tue, Aug 9, 2016 at 6:33 PM, Troy Cowin <troy at perthsystems.com.au <mailto:troy at perthsystems.com.au>> wrote:
> Yep lots of various errors even earlier this morning – took about 5 minutes of refreshing to actually get into the site. Once in though it worked ok, hate to see how it will handle the load now its peak hour. <>
>  
>  
> From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net <mailto:ausnog-bounces at lists.ausnog.net>] On Behalf Of Ben Cornish
> Sent: Tuesday, 9 August 2016 4:28 PM
> To: ausnog at lists.ausnog.net <mailto:ausnog at lists.ausnog.net>
> Subject: [AusNOG] census issues tonight
>  
> Not surprising – but anyone else seeing issues with the Census site tonight ?
>  
> Seems Soft layer is serving up parts of this behind the scenes..
> Im seeing  504 errors from soft layer for large chunks of IP ranges – but not all.
>  
> Anyone else seeing this ?
>  
>  
>  
>  
> Ben Cornish
> Chief Operating Officer
>  
>  
> Over the Wire Holdings Ltd (ASX:OTW)
> GPO Box 1807 Brisbane, QLD, 4001, Australia
> Level 1, 24 Little Edward Street, Spring Hill, QLD 4000
> t    1300 689 689 <tel:1300%20689%20689>
> m  0417 617 204 <tel:0417%20617%20204>
> e   Ben.Cornish at overthewire.com.au <mailto:Ben.Cornish at overthewire.com.au>     www.overthewire.com.au <http://www.overthewire.com.au/>
>  
>  
> 
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net <mailto:AusNOG at lists.ausnog.net>
> http://lists.ausnog.net/mailman/listinfo/ausnog <http://lists.ausnog.net/mailman/listinfo/ausnog>
>  
>  
> 
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net <mailto:AusNOG at lists.ausnog.net>
> http://lists.ausnog.net/mailman/listinfo/ausnog <http://lists.ausnog.net/mailman/listinfo/ausnog>
>  
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net <mailto:AusNOG at lists.ausnog.net>
> http://lists.ausnog.net/mailman/listinfo/ausnog <http://lists.ausnog.net/mailman/listinfo/ausnog>
>  
> 
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net <mailto:AusNOG at lists.ausnog.net>
> http://lists.ausnog.net/mailman/listinfo/ausnog <http://lists.ausnog.net/mailman/listinfo/ausnog>
>  
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net <mailto:AusNOG at lists.ausnog.net>
> http://lists.ausnog.net/mailman/listinfo/ausnog <http://lists.ausnog.net/mailman/listinfo/ausnog>
>  
> 
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net <mailto:AusNOG at lists.ausnog.net>
> http://lists.ausnog.net/mailman/listinfo/ausnog <http://lists.ausnog.net/mailman/listinfo/ausnog>
> 
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net <mailto:AusNOG at lists.ausnog.net>
> http://lists.ausnog.net/mailman/listinfo/ausnog <http://lists.ausnog.net/mailman/listinfo/ausnog>
>  
> 
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net <mailto:AusNOG at lists.ausnog.net>
> http://lists.ausnog.net/mailman/listinfo/ausnog <http://lists.ausnog.net/mailman/listinfo/ausnog>
>  
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net <mailto:AusNOG at lists.ausnog.net>
> http://lists.ausnog.net/mailman/listinfo/ausnog <http://lists.ausnog.net/mailman/listinfo/ausnog>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20160809/77d4207b/attachment.html>

More information about the AusNOG mailing list