[AusNOG] census issues tonight

James Braunegg james.braunegg at micron21.com
Tue Aug 9 20:56:23 EST 2016


I’d love to be the person auditing the Census environment tomorrow on what went wrong….

Things break…. Sure….. Now build it better and try again… Just like Click Frenzy the first time….

Kindest Regards

James Braunegg
P:  1300 769 972  |  M:  0488 997 207 |  D:  (03) 9751 7616
E:   james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>  |  ABN:  12 109 977 666
W:  www.micron21.com/ddos-protection<http://www.micron21.com/ddos-protection>   T: @micron21

Follow us on Twitter<http://www.twitter.com/micron21> for important service and system updates.

[M21.jpg]

This message is intended for the addressee named above. It may contain privileged or confidential information. If you are not the intended recipient of this message you must not use, copy, distribute or disclose it to anyone other than the addressee. If you have received this message in error please return the message to the sender by replying to it and then delete the message from your computer.

From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of John Lindsay
Sent: Tuesday, 9 August 2016 8:54 PM
To: Sean Finn <sean at wikwok.com>
Cc: ausnog at lists.ausnog.net
Subject: Re: [AusNOG] census issues tonight

Well the threat of being fined if you don’t do it is causing lots of people who don’t listen very carefully to get obsessed over doing this right now.

Morning radio is going to be so funny. Can’t wait for the interviews with Michael McCormack. There’s a minister who won’t be allowed to quit and won’t be trying for deputy PM for a while longer.

John Lindsay
johnslindsay at mac.com<mailto:johnslindsay at mac.com>
+61403577711

On 9 Aug 2016, at 8:19 PM, Sean Finn <sean at wikwok.com<mailto:sean at wikwok.com>> wrote:

Has anyone got any bandwidth graphs that illustratre why theres more traffic going to the Census site than the Click Frenzy fail?

On 9 Aug 2016 8:30 PM, "Glenn Powell" <glenn at glennbridge.com.au<mailto:glenn at glennbridge.com.au>> wrote:
TPG routes are now heading via Telstra to IBM rather than via Nextgen which is the path from 30+ mins ago.


On 9 Aug 2016, at 8:28 PM, James Braunegg <james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>> wrote:

Looks like something funny is going on...

Vocus has null routed the 1st 20 IP's internationally ie  150.207.169.1 to 150.207.169.20  but still advertising 150.207.169.0/24<http://150.207.169.0/24> to the world for any IP address above 150.207.169.20


BGP routing table entry for 150.207.169.1/32<http://150.207.169.1/32>, version 2931238604

Paths: (1 available, best #1, table Default-IP-Routing-Table)

  4826

    10.255.255.255 (metric 10118050) from 38.28.1.83 (38.28.1.83)

      Origin incomplete, localpref 150, valid, internal, best

      Community: 174:990 174:20912 174:21001 174:22013

      Originator: 66.28.1.228, Cluster list: 38.28.1.83, 38.28.1.67, 38.28.1.65


BGP routing table entry for 150.207.169.0/24<http://150.207.169.0/24>, version 2931901711
Paths: (1 available, best #1, table Default-IP-Routing-Table)
Flag: 0x920
  4637 1221 37935
    134.159.95.224 (metric 10118050) from 38.28.1.83 (38.28.1.83)
      Origin IGP, metric 4294967294, localpref 100, valid, internal, best
      Community: 174:11100 174:20666 174:21100 174:22012
      Originator: 38.28.1.92, Cluster list: 38.28.1.83, 38.28.1.67, 38.28.1.162


Surely you would just drop the whole /24 internationally  if you did not want international traffic to hit the Census site.

Kindest Regards
James Braunegg
P:  1300 769 972  |  M:  0488 997 207 |  D:  (03) 9751 7616
E:   james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>  |  ABN:  12 109 977 666
W:  www.micron21.com/ddos-protection<http://www.micron21.com/ddos-protection>   T: @micron21

Follow us on Twitter<http://www.twitter.com/micron21> for important service and system updates.

<image001.jpg>

This message is intended for the addressee named above. It may contain privileged or confidential information. If you are not the intended recipient of this message you must not use, copy, distribute or disclose it to anyone other than the addressee. If you have received this message in error please return the message to the sender by replying to it and then delete the message from your computer.

-----Original Message-----
From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Brad Peczka
Sent: Tuesday, 9 August 2016 8:22 PM
To: Glenn Powell <glenn at glennbridge.com.au<mailto:glenn at glennbridge.com.au>>; ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] census issues tonight

I'm seeing responses from 150.207.169.1-5, but nothing else in that block.

I wonder if their compute stack just noped itself out of existence?

Regards,
-Brad.
________________________________________
From: AusNOG [ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>] On Behalf Of Glenn Powell [glenn at glennbridge.com.au<mailto:glenn at glennbridge.com.au>]
Sent: Tuesday, 9 August 2016 6:13 PM
To: ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] census issues tonight

They spent $50k on a 3rd party to do load testing, I wonder what the success criteria of that testing looked like?


On 9 Aug 2016, at 8:09 PM, Brent Paddon <brent.paddon at gmail.com<mailto:brent.paddon at gmail.com<mailto:brent.paddon at gmail.com%3cmailto:brent.paddon at gmail.com>>> wrote:

Only cost $9.6M to host the site...

Brent

On Tue, Aug 9, 2016 at 8:07 PM, James Braunegg <james.braunegg at micron21.com<mailto:james.braunegg at micron21.com<mailto:james.braunegg at micron21.com%3cmailto:james.braunegg at micron21.com>>> wrote:
Dead as a doornail …. from Telstra…

How sad… Now What…

Kindest Regards

James Braunegg
P:  1300 769 972  |  M:  0488 997 207 |  D:  (03) 9751 7616
E:   james.braunegg at micron21.com<mailto:james.braunegg at micron21.com<mailto:james.braunegg at micron21.com%3cmailto:james.braunegg at micron21.com>>  |  ABN:  12 109 977 666
W:  www.micron21.com/ddos-protection<http://www.micron21.com/ddos-protection<http://www.micron21.com/ddos-protection%3chttp:/www.micron21.com/ddos-protection>>   T: @micron21

Follow us on Twitter<http://www.twitter.com/micron21> for important service and system updates.

<image001.jpg>

This message is intended for the addressee named above. It may contain privileged or confidential information. If you are not the intended recipient of this message you must not use, copy, distribute or disclose it to anyone other than the addressee. If you have received this message in error please return the message to the sender by replying to it and then delete the message from your computer.

From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net><mailto:ausnog-bounces at lists.ausnog.net%3cmailto:ausnog-bounces at lists.ausnog.net%3e>] On Behalf Of John Lindsay
Sent: Tuesday, 9 August 2016 7:52 PM
To: Joseph Goldman <joe at apcs.com.au<mailto:joe at apcs.com.au<mailto:joe at apcs.com.au%3cmailto:joe at apcs.com.au>>>
Cc: ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net%3cmailto:ausnog at lists.ausnog.net>>
Subject: Re: [AusNOG] census issues tonight

It’s turned to glue for me.

Was great an hour or so ago.

John Lindsay
johnslindsay at mac.com<mailto:johnslindsay at mac.com<mailto:johnslindsay at mac.com%3cmailto:johnslindsay at mac.com>>
+61403577711<tel:%2B61403577711><tel:%2B61403577711>

On 9 Aug 2016, at 6:51 PM, Joseph Goldman <joe at apcs.com.au<mailto:joe at apcs.com.au<mailto:joe at apcs.com.au%3cmailto:joe at apcs.com.au>>> wrote:

Just completed it then was super snappy for me, so assuming they either ramped up servers or people gave up lol.
On 09/08/16 18:56, Joshua D'Alton wrote:
Their phone number too just rings out with "we're too busy, please call back". Probably due to aforementioned issues.. and then the few million old people who are trying to do it over the phone. Or people who need to call because they were doing it and their session died and their 9 digit code no longer works :/

On Tue, Aug 9, 2016 at 6:33 PM, Troy Cowin <troy at perthsystems.com.au<mailto:troy at perthsystems.com.au<mailto:troy at perthsystems.com.au%3cmailto:troy at perthsystems.com.au>>> wrote:
Yep lots of various errors even earlier this morning – took about 5 minutes of refreshing to actually get into the site. Once in though it worked ok, hate to see how it will handle the load now its peak hour.


From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net><mailto:ausnog-bounces at lists.ausnog.net%3cmailto:ausnog-bounces at lists.ausnog.net%3e>] On Behalf Of Ben Cornish
Sent: Tuesday, 9 August 2016 4:28 PM
To: ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net%3cmailto:ausnog at lists.ausnog.net>>
Subject: [AusNOG] census issues tonight

Not surprising – but anyone else seeing issues with the Census site tonight ?

Seems Soft layer is serving up parts of this behind the scenes..
Im seeing  504 errors from soft layer for large chunks of IP ranges – but not all.

Anyone else seeing this ?




Ben Cornish
Chief Operating Officer


Over the Wire Holdings Ltd (ASX:OTW)
GPO Box 1807 Brisbane, QLD, 4001, Australia Level 1, 24 Little Edward Street, Spring Hill, QLD 4000
t    1300 689 689
m  0417 617 204
e   Ben.Cornish at overthewire.com.au<mailto:Ben.Cornish at overthewire.com.au<mailto:Ben.Cornish at overthewire.com.au%3cmailto:Ben.Cornish at overthewire.com.au>>    www.overthewire.com.au<http://www.overthewire.com.au/<http://www.overthewire.com.au%3chttp/www.overthewire.com.au/>>



_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net%3cmailto:AusNOG at lists.ausnog.net>>
http://lists.ausnog.net/mailman/listinfo/ausnog





_______________________________________________

AusNOG mailing list

AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net%3cmailto:AusNOG at lists.ausnog.net>>

http://lists.ausnog.net/mailman/listinfo/ausnog

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net%3cmailto:AusNOG at lists.ausnog.net>>
http://lists.ausnog.net/mailman/listinfo/ausnog


_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net%3cmailto:AusNOG at lists.ausnog.net>>
http://lists.ausnog.net/mailman/listinfo/ausnog


_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net%3cmailto:AusNOG at lists.ausnog.net>>
http://lists.ausnog.net/mailman/listinfo/ausnog
_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
http://lists.ausnog.net/mailman/listinfo/ausnog


_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
http://lists.ausnog.net/mailman/listinfo/ausnog
_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
http://lists.ausnog.net/mailman/listinfo/ausnog

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20160809/1ab6d366/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 2683 bytes
Desc: image001.jpg
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20160809/1ab6d366/attachment-0001.jpg>


More information about the AusNOG mailing list