[AusNOG] QLD Optus Downtime?

[Mat Walker]

Optus update:


-----

Progress Update:

10:15 Multiple data/voice services reported as down in Queensland 11:00 Router is being investigated by the vendor at Roachdale Exchange 11:30 Majority of services have now returned.  Impacted router was isolated.

-----

we will be leaving our work around in place until the dust settles and we've seen some stability for a good length of time.

From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Mat Walker
Sent: Thursday, 25 July 2013 11:09 AM
To: 'Andrew McLeod'; ausnog at lists.ausnog.net
Subject: Re: [AusNOG] QLD Optus Downtime?

>From Optus;


System ID: 395745

Outage ID: 11621035

Outage Type: Unplanned

Start Date/Time: 25/07/2013 10:56:10

State:

QLD

Service Type: Major Outage - Voice & Data



Description:

Multiple data/voice services reported as down in Queensland



Progress Update:

10:15 Multiple data/voice services reported as down in Queensland 11:00 Roter is being investigated by the vendor at Roachdale Exchange


We've shut down our primary QLD Optus BGP peer and are now preferring most traffic via our Sydney Optus & NTT peers.

On the grapevine we hear that they have a Juniper router that has crapped itself, is "with the vendor" but is propagating to other parts of their network, as we're seeing mass BGP peer updates / resets on our legacy Optus Cisco BGP peers.

From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Andrew McLeod
Sent: Thursday, 25 July 2013 11:05 AM
To: ausnog at lists.ausnog.net
Subject: Re: [AusNOG] QLD Optus Downtime?

We identified significant congestion with connectivity at approx. 10:05 BNE and subsequently stopped our own BGP announcements into that backbone to maintain customer connectivity (ie, we're only announcing to our other upstreams).

A good example of congestion from Telstra Melbourne back to Brisbane (believed via Optus) can be seen here:

https://www.telstra.net/cgi-bin/trace?destination=202.183.111.54

(You may need to refresh a couple times to see the intermittent extent of the issue at hop 8):

1  gigabitethernet3-3.exi2.melbourne.telstra.net (203.50.77.53)  0.318 ms  0.217 ms  0.245 ms
2  tengige0-8-0-5.win-core1.melbourne.telstra.net (203.50.80.129)  1.994 ms  3.236 ms  3.991 ms
3  bundle-ether12.ken-core4.sydney.telstra.net (203.50.11.12)  13.111 ms  15.104 ms  15.984 ms
4  tengigabitethernet7-1.ult2.sydney.telstra.net (203.50.20.51)  13.112 ms  12.981 ms  12.986 ms
5  nttaus1.lnk.telstra.net (139.130.74.6)  13.362 ms  13.232 ms  13.361 ms
6  202.68.65.21 (202.68.65.21)  13.362 ms  13.358 ms  13.486 ms
7  ge-1-0-9-74.a00.sydnau02.au.ra.gin.ntt.net (202.68.66.230)  13.612 ms  13.604 ms  13.486 ms
8  * w27-iirihs-imbi.es-core-bo1.iseek.com.au (220.157.95.211)  2139.356 ms  385.883 ms

Kind regards,
[email_signature]

From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of David George
Sent: Thursday, 25 July 2013 10:48 AM
To: Mischa King; ausnog at lists.ausnog.net
Subject: Re: [AusNOG] QLD Optus Downtime?

We are as well, I've got no real info at this point.
25 mins ago(ish), our site activity has dropped significantly and some of our remote sites are completely offline.


From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Mischa King
Sent: Thursday, 25 July 2013 10:44 AM
To: ausnog at lists.ausnog.net<mailto:ausnog at lists.ausnog.net>
Subject: [AusNOG] QLD Optus Downtime?

Anyone else having any problems in qld?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20130725/aa820a4f/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 36430 bytes
Desc: image001.png
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20130725/aa820a4f/attachment.png>