[AusNOG] So who broke the internet now ?

Tue Mar 16 15:26:51 EST 2010

Wow, so 1 maintenance activity by a single provider changing a network
interface caused all that uproar and poor international performance
between multiple service providers and their customers.

Can that be right ?

Regards, 

Darren Moss
General Manager
Australia and New Zealand
[p] 1300 131 083 [f] 03 9017 2287
[e] Darren.Moss at em3.com.au [w] www.em3.com.au <http://www.em3.com.au/> 

em3 People and Technology | Managed Technology Experts
postal: PO Box 2333, Moorabbin VIC 3189

New Zealand Airedale Street, Auckland City
postal: PO Box 39573, Howick 2045
[p] 09 887 0550 [f] 09 887 0273

________________________________

From: ausnog-bounces at lists.ausnog.net
[mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of James Braunegg
Sent: 2010-03-16 3:20 pm
To: Damien Coxall
Cc: ausnog at ausnog.net
Subject: Re: [AusNOG] So who broke the internet now ?

Found the source of the issue last night... Next Gen were doing some
upgrades... which went a little wrong... 

Dear Customer,

Please be advised the following Emergency Planned Outage Works by
Nextgen Network has been scheduled.

This work is expected to result in an outage to your service/s listed
below.

Start Date:       16/03/10            Start Time:      00:01 (AEDT)   

Finish Date:     16/03/10            Finish Time:    02:00 (AEDT)  

Description of Works: 

Nextgen has organized an emergency planned outage to replace the faulty
card on one of the core device.  

Your service listed above will experience an outage of 30 minutes during
the specified window.  

Nextgen apologizes for any inconvenience caused.

If you require any further assistance please do not hesitate to call
Nextgen SMC on 1800 099 299.

Regards

Kindest Regards

James Braunegg
W:  1300 769 972  |  M:  0488 997 207 |  D:  (03) 9751 7616

E:   james.braunegg at micron21.com  |  ABN:  85 126 029 203   

This message is intended for the addressee named above. It may contain
privileged or confidential information. If you are not the intended
recipient of this message you must not use, copy, distribute or disclose
it to anyone other than the addressee. If you have received this message
in error please return the message to the sender by replying to it and
then delete the message from your computer.

From: Damien Coxall [mailto:damien at dreamboat.id.au] 
Sent: Tuesday, March 16, 2010 1:49 PM
To: James Braunegg
Subject: Re: [AusNOG] So who broke the internet now ?

Hi James,

There was nothing unusual in my monitoring overnight, either with
domestic / international transit or overseas transmission.

Both Netspace and Primus live downstream of Optus and from the looks of
things so do you, so I can only guess the problem lay there somewhere,
even though I saw no issue with any of my data interconnects with them.

Damien,
Vocus.

On Tue, Mar 16, 2010 at 1:20 AM, James Braunegg <
james.braunegg at micron21.com> wrote:

Hey all, did anyone else see some international and some domestic ping
times increase by 10x around 12:12am this morning? Lasting for around 15
minutes ?

We saw some of our outbound transit to "parts of the internet slow down"
we routed around the issues to resolve it.

Also noticed people are complaining from many ISP's on whirlpool..... 

>From our monitoring BGP was flapping around like a pig... its all seems
fine now but mainly interested to know what happen...

My guess some major link went down... traffic went via redundant paths
which couldn't take all the traffic hence the increased ping times...

So who's cable broke ?

Netspace issues

http://forums.whirlpool.net.au/forum-replies.cfm?t=1404190
<http://forums.whirlpool.net.au/forum-replies.cfm?t=1404190> 

Optus issues

http://forums.whirlpool.net.au/forum-replies.cfm?t=1404247
<http://forums.whirlpool.net.au/forum-replies.cfm?t=1404247> 

Primus issues

http://forums.whirlpool.net.au/forum-replies.cfm?t=1404283
<http://forums.whirlpool.net.au/forum-replies.cfm?t=1404283> 

Kindest Regards

James Braunegg
W:  1300 769 972  |  M:  0488 997 207 |  D:  (03) 9751 7616

E:   james.braunegg at micron21.com <mailto:james.braunegg at micron21.com>
|  ABN:  85 126 029 203   

This message is intended for the addressee named above. It may contain
privileged or confidential information. If you are not the intended
recipient of this message you must not use, copy, distribute or disclose
it to anyone other than the addressee. If you have received this message
in error please return the message to the sender by replying to it and
then delete the message from your computer.

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
http://lists.ausnog.net/mailman/listinfo/ausnog

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20100316/dd4fcf15/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 2683 bytes
Desc: image001.jpg
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20100316/dd4fcf15/attachment.jpg>