[AusNOG] census issues tonight

Fri Aug 12 09:28:45 EST 2016

Dear Matthew

There is no concrete evidence that it was a DDoS attack or that it was not a DDoS Attack.... However the facts point towards it being more than just a capacity issue.

All I know is what I observed (Happy to be corrected if I got anything wrong)

1.       The day before the Census no /32 null routes were present internationally.

2.       At some point during the Census night /32 null routes appeared for the first 21 IP addresses within the 150.207.169.0/24 internationally - originating from both nextgen and Vocus  (They forget to null route the entire /24)

3.       Nextgen was adverting routes to NTT internationally, and Vocus was adverting routes to Xo HE and Cogent

4.       During the Census night within Australia we saw ICMP packet loss towards first 21 IP addresses within 150.207.169.0/24 which responded to ICMP

5.       During the Census after the site was down we saw the /32 null routes removed for the first 21 IP addresses coming in via Vocus in the USA, but an ACL to block all traffic towards 150.207.169.0/24 on Vocus interface in LA.

6.       Removing the /32 null routes and putting a ACL to block the entire /24 within Vocus LA network would tend to suggest Vocus had network capacity to absorb any unwanted traffic, without affecting other Vocus customers or they wanted to see what was actually occurring (packet capture the attack traffic)

7.       During the Census after the site was down we saw routes change and international traffic start to come in via Telstra Global directly (not via Vocus or Nextgen) with no null routes in place (other than on NTT's network which was still receiving null routes from Nextgen)

8.       During the Census after the site was down after traffic was rerouted to come in via Telstra we saw the last hop towards IBM disappear (which would support the fact a IBM router failed or did not have enough capacity) and Telstra network was connected directly to the Census infrastructure.

9.       During the Census after Telstra took control of international and domestic routing and the IBM router was bypassed ICMP pings returned to normal. The site was now reachable and loaded but the application was turned off, with a message saying sorry.

10.   During the Census night looking at the SSL certificate showed 11 back end servers or 11 backend load balancers.

11.   Then the entire nation paused for 24 hours or so....

12.   Nextgen have stopped adverting routes to NTT internationally (and now only send routes to domestic peers local peering exchanges)

13.   Vocus is only advertising routes to XO, Cogent, HE internationally, but is also null routing these internationally, (as /32 and /25 routes)

14.   The side effect of Vocus null routing internationally is causing international DNS servers from not resolving DNS because the name servers for census.abs.gov.au all reside within the same IP range 150.207.169.0/24

15.   Not having global accessible DNS name servers for the domain census.abs.gov.au is stopping Australians accessing the site who are using international DNS servers, instead of local DNS servers.

16.    Effectively now 150.207.169.0/24 is only available within Australia (Why was this not done in the first place ?)

17.   ICMP has been turned off for all servers / load balancers and router interfaces as local traffic gets "close" 150.207.169.0/24

18.   Looking at the SSL certificate for https://census.abs.gov.au it still shows 11 back end servers or 11 load balancers connections.

19.   A 3rd party review of the events By Patrick Gray - which falls into line with what I saw.... Http://Risky.biz/censusfail/

Questions... Draw / Make your own Answers

History of DDoS attacks show that 95% of most DDoS attacks originate from outside of Australia, so if it was not some form of DDoS attack (Layer 3, 4 or 7 attack) why make a lot of changes to a live production environment on Census night which only effect international traffic ?

The IBM router could have failed or could have run out of capacity which is why it was bypassed

Packet loss towards 150.207.169.0/24 from within Australia seen on Cenus night was either due to the faulty router dropping traffic or maxing out due to not enough capacity due to many users connecting or unwanted international traffic reaching the site, or a combination of everything.

If it was a server capacity issue not having enough compute capacity, why has the SSL certificate not changed ?

If it was not a DDoS attack (other than common sense)  why put in place international null routing  for the first 21 IP addresses during Census night?

If it was not a network capacity issue (other than common sense) why now are international null routes in place for the entire /24 when before the event no null routes were in place.

Kindest Regards

James Braunegg
P:  1300 769 972  |  M:  0488 997 207 |  D:  (03) 9751 7616
E:   james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>  |  ABN:  12 109 977 666
W:  www.micron21.com/ddos-protection<http://www.micron21.com/ddos-protection>   T: @micron21

Follow us on Twitter<http://www.twitter.com/micron21> for important service and system updates.

[M21.jpg]

This message is intended for the addressee named above. It may contain privileged or confidential information. If you are not the intended recipient of this message you must not use, copy, distribute or disclose it to anyone other than the addressee. If you have received this message in error please return the message to the sender by replying to it and then delete the message from your computer.

From: Matthew Matters [mailto:MMatters at ausnetservers.net.au]
Sent: Friday, 12 August 2016 7:31 AM
To: Luke Fong <luke at lateralplains.com>; James Braunegg <james.braunegg at micron21.com>; johnstsquare at tpg.com.au; ausnog at lists.ausnog.net
Subject: Re: [AusNOG] census issues tonight

Correct me if im wrong but CENSUS wasn't DDOS in the traditional sense it DDOSED by legitimate traffic wasn't it?

The pipe was overloaded by everyone trying to do their form.

I agree heads should roll because they should of had a pipe that was GB/s + to cope with the amount of users that they wanted to lodge.

Regards,

Matthew Matters  Managing Director / CEO of Aus Net Servers Australia Pty Ltd
Management Department  |  Small Business Hosting Sales & Services  |  Aus Net Servers Australia Pty Ltd
P  03 5326 0050  |  M  0402 711 135  |  E  mmatters at ausnetservers.net.au<https://www.ansamail.com.au/owa/redir.aspx?C=106998d21b72497a98b44c48875775bb&URL=mailto%3ammatters%40ausnetservers.net.au> |  W  www.ausnetservers.net.au<http://www.ausnetservers.net.au>

WE HAVE MOVED: Our new office is now located at 11/1 Eastwood Street Ballarat Central Victoria 3350.
________________________________
From: AusNOG <ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>> on behalf of Luke Fong <luke at lateralplains.com<mailto:luke at lateralplains.com>>
Sent: Thursday, August 11, 2016 10:43 PM
To: 'James Braunegg'; johnstsquare at tpg.com.au<mailto:johnstsquare at tpg.com.au>; ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] census issues tonight

Hey James,

I think you forgot to mention a company that can help also...

www.micron21.com/ddos-protection<http://www.micron21.com/ddos-protection>

;)

Cheers
L

[luke2k15-2]
Kind Regards,
Luke Fong
Operations Manager
Lateral Plains Pty Ltd
PO Box 549
Ballarat ,Vic 3353
Tel : 03 5317 7123

From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of James Braunegg
Sent: Thursday, 11 August 2016 10:36 PM
To: johnstsquare at tpg.com.au<mailto:johnstsquare at tpg.com.au>; ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] census issues tonight

They are not Geo Blocking anything....

International DNS is not working as direct result of all 4 name servers for census.abs.gov.au being hosted within 150.207.169.0/24 which is effectively black holed internationally via /32 and /25 routes so no international DNS server can resolve the domain.

They could have at least hosted DNS on someone else's network or used route 53 so DNS would resolve correctly for anyone not using an Australian DNS server.

auolpr00dn03d.abs.gov.au.   ['150.207.169.20']   [TTL=10800]
auolpr00dn02d.abs.gov.au.   ['150.207.169.7']   [TTL=10800]
auolpr00dn04d.abs.gov.au.   ['150.207.169.21']   [TTL=10800]
auolpr00dn01d.abs.gov.au.   ['150.207.169.6']   [TTL=10800]

Basically to complete the Census you have to use an Australian based connection with an Australian DNS server.

On a side note I think the ABS will be placing an order with NSFOCUS or Arbor for some Layer 3 to 7 DDoS scrubbing hardware ... which clearly they don't have, as if they had the correct DDoS scrubbing technology in place they would not need to use the black holing technique which is currently in place.

Kindest Regards

James Braunegg
P:  1300 769 972  |  M:  0488 997 207 |  D:  (03) 9751 7616
E:   james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>  |  ABN:  12 109 977 666
W:  www.micron21.com/ddos-protection<http://www.micron21.com/ddos-protection>   T: @micron21

Follow us on Twitter<http://www.twitter.com/micron21> for important service and system updates.

[M21.jpg]

This message is intended for the addressee named above. It may contain privileged or confidential information. If you are not the intended recipient of this message you must not use, copy, distribute or disclose it to anyone other than the addressee. If you have received this message in error please return the message to the sender by replying to it and then delete the message from your computer.

From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of johnstsquare at tpg.com.au<mailto:johnstsquare at tpg.com.au>
Sent: Thursday, 11 August 2016 9:40 PM
To: ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>
Subject: Re: [AusNOG] census issues tonight

+1
The same with 8.8.4.4 and OpenDNS public resolvers

ABS is using geo-blocking with layer-3 IP ACL on the routers upstream from their DNS servers.  VPN users terminating outside of Australia (yay HBO Go and Amazon video), or employees of MNCs with resolvers outside of Australia.

Because it was a layer-3 block, they just dropped the traffic and the user's resolver would keep sending DNS queries.  As a result, there were numerous resolvers sending a flood of requests to census.abs.gov.au<http://census.abs.gov.au/> DNS servers which looked like a small amplification attack.

Additionally they have taken the wrong move of increasing TTL's to try to reduce load on their DNS. This makes it hard to move to a cloud DDOS provider as the dns will take 24hrs to propagate.  14400 seconds.
www.census.abs.gov.au<http://www.census.abs.gov.au>. 14400 IN A 150.207.169.5

------

$ dig +trace www.census.abs.gov.au<http://www.census.abs.gov.au> @61.88.88.88

; <<>> DiG 9.8.3-P1 <<>> +trace www.census.abs.gov.au<http://www.census.abs.gov.au> @61.88.88.88

;; global options: +cmd

. 333196 IN NS j.root-servers.net.

. 333196 IN NS k.root-servers.net.

. 333196 IN NS l.root-servers.net.

. 333196 IN NS m.root-servers.net.

. 333196 IN NS a.root-servers.net.

. 333196 IN NS b.root-servers.net.

. 333196 IN NS c.root-servers.net.

. 333196 IN NS d.root-servers.net.

. 333196 IN NS e.root-servers.net.

. 333196 IN NS f.root-servers.net.

. 333196 IN NS g.root-servers.net.

.. 333196 IN NS h.root-servers.net.

. 333196 IN NS i.root-servers.net.

;; Received 228 bytes from 61.88.88.88#53(61.88.88.88) in 152 ms

au. 172800 IN NS a.au.

au. 172800 IN NS b.au.

au. 172800 IN NS u.au.

au. 172800 IN NS v.au.

au. 172800 IN NS w.au.

au. 172800 IN NS x.au.

au. 172800 IN NS y.au.

au. 172800 IN NS z.au.

;; Received 491 bytes from 199.7.83.42#53(199.7.83.42) in 114 ms

gov.au. 86400 IN NS w.au.

gov.au. 86400 IN NS x.au.

gov.au. 86400 IN NS z.au.

gov.au. 86400 IN NS y.au.

;; Received 279 bytes from 58.65.253.73#53(58.65.253.73) in 146 ms

abs.gov.au. 14400 IN NS ns1.telstra.net.

abs.gov.au. 14400 IN NS ns1.abs.gov.au.

;; Received 102 bytes from 37.209.198.5#53(37.209.198.5) in 62 ms

census.abs.gov.au. 10800 IN NS auolpr00dn01d.abs.gov.au.

census.abs.gov.au. 10800 IN NS auolpr00dn02d.abs.gov.au.

census.abs.gov.au. 10800 IN NS auolpr00dn04d.abs.gov.au.

census.abs.gov.au. 10800 IN NS auolpr00dn03d.abs.gov.au.

;; Received 215 bytes from 139.130.4.5#53(139.130.4.5) in 48 ms

www.census.abs.gov..au<http://www.census.abs.gov..au>. 14400 IN A 150.207.169.5

www.census.abs.gov.au<http://www.census.abs.gov.au>. 14400 IN A 150.207.169.8

census.abs.gov.au. 86400 IN NS auolpr00dn01d.abs.gov.au.

census.abs.gov.au. 86400 IN NS auolpr00dn03d.abs.gov.au.

census.abs.gov.au. 86400 IN NS auolpr00dn02d.abs.gov.au.

census.abs.gov.au. 86400 IN NS auolpr00dn04d.abs.gov.au.

;; Received 183 bytes from 150.207.169.7#53(150.207.169.7) in 9 ms

From: Chris Lee <chris at datachaos.com.au<mailto:chris at datachaos.com.au>>
Date: Thursday, August 11, 2016 at 7:09 PM
To: "ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>" <ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>>
Subject: Re: [AusNOG] census issues tonight

Online so long as you don't use Google DNS for lookups...

; <<>> DiG 9.10.4-P1 <<>> @8.8..8.8<https://urldefense.proofpoint.com/v2/url?u=http-3A__8.8.8.8&d=DQMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=wJDREqbOvAj7uAMLV05riA&m=fv569LrIV-cypFQUVYMlmz69TV4_76PM3m30R6LCx-0&s=lVEBnodBT1tutMSqVpjploWPMSXH5ioOE1oO1a3y_hQ&e=> census.abs.gov.au<https://urldefense.proofpoint.com/v2/url?u=http-3A__census.abs.gov.au&d=DQMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=wJDREqbOvAj7uAMLV05riA&m=fv569LrIV-cypFQUVYMlmz69TV4_76PM3m30R6LCx-0&s=SjWtw_nm3J4SjJxIdaGpWiN25_EK69qzsxCpYdAIe_o&e=>
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 45286
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;census.abs.gov.au<https://urldefense.proofpoint.com/v2/url?u=http-3A__census.abs.gov.au&d=DQMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=wJDREqbOvAj7uAMLV05riA&m=fv569LrIV-cypFQUVYMlmz69TV4_76PM3m30R6LCx-0&s=SjWtw_nm3J4SjJxIdaGpWiN25_EK69qzsxCpYdAIe_o&e=>.             IN      A

;; Query time: 33 msec
;; SERVER: 8.8.8.8#53(8..8.8.8)
;; WHEN: Thu Aug 11 19:06:03 AEST 2016
;; MSG SIZE  rcvd: 46

WE HAVE MOVED: Our new office is now located at 11/1 Eastwood Street Ballarat Central Victoria 3350.

[Follow us]

[Facebook]<https://www.facebook.com/ANSASERVERS>

[Twitter]<http://twitter.com/#!/ANSASERVERS>

[Google+]<https://plus.google.com/101907839864050850442/>

The information transmitted in this e-mail is for the exclusive use of the intended addressee and may contain confidential and/or privileged material. Any review, re-transmission, dissemination or other use of it, or the taking of any action in reliance upon this information by persons and/or entities other than the intended recipient is prohibited. If you received this in error, please inform the sender and/or addressee immediately and delete the material. If you have been sent this email and it is not addressed to you please forward the email as is to hostmaster at ausnetservers.net.au<mailto:hostmaster at ausnetservers.net.au> and delete all local and inta-local copies including backups from your system. E-mails may not be secure, may contain computer viruses and may be corrupted in transmission. Please carefully check this e-mail (and any attachment) accordingly. No warranties are given and no liability is accepted for any loss or damage caused by such matters.

This email has been scanned before transmission with business grade antivirus and antispam software but as mentioned above no warranties can be given that the email has not been contaminated after transmission.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20160811/74d37087/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 2683 bytes
Desc: image001.jpg
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20160811/74d37087/attachment-0001.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 20987 bytes
Desc: image002.png
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20160811/74d37087/attachment-0001.png>