[AusNOG] So who broke the internet now ?

Thu Mar 18 10:45:47 EST 2010

That's what I thought.

(it was a loaded question)

Regards, 

Darren Moss
General Manager
Australia and New Zealand
[p] 1300 131 083 [f] 03 9017 2287
[e] Darren.Moss at em3.com.au [w] www.em3.com.au <http://www.em3.com.au/> 

em3 People and Technology | Managed Technology Experts
postal: PO Box 2333, Moorabbin VIC 3189

New Zealand Airedale Street, Auckland City
postal: PO Box 39573, Howick 2045
[p] 09 887 0550 [f] 09 887 0273

________________________________

From: Colwell, Scott [mailto:scott.colwell at nn.com.au] 
Sent: 2010-03-18 10:20 am
To: Darren Moss; James Braunegg; Damien Coxall
Cc: ausnog at ausnog.net
Subject: RE: [AusNOG] So who broke the internet now ?

No.  Changing a line card on an SDH add drop mux doesn't normally even
impact a single
ISP.

Scott

________________________________

	From: ausnog-bounces at lists.ausnog.net
[mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Darren Moss
	Sent: Tuesday, 16 March 2010 3:27 PM
	To: James Braunegg; Damien Coxall
	Cc: ausnog at ausnog.net
	Subject: Re: [AusNOG] So who broke the internet now ?

	Wow, so 1 maintenance activity by a single provider changing a
network interface caused all that uproar and poor international
performance between multiple service providers and their customers.

	Can that be right ?

	Regards, 

	Darren Moss
	General Manager
	Australia and New Zealand
	[p] 1300 131 083 [f] 03 9017 2287
	[e] Darren.Moss at em3.com.au [w] www.em3.com.au
<http://www.em3.com.au/> 

	em3 People and Technology | Managed Technology Experts
	postal: PO Box 2333, Moorabbin VIC 3189

	New Zealand Airedale Street, Auckland City
	postal: PO Box 39573, Howick 2045
	[p] 09 887 0550 [f] 09 887 0273

________________________________

	From: ausnog-bounces at lists.ausnog.net
[mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of James Braunegg
	Sent: 2010-03-16 3:20 pm
	To: Damien Coxall
	Cc: ausnog at ausnog.net
	Subject: Re: [AusNOG] So who broke the internet now ?

	Found the source of the issue last night... Next Gen were doing
some upgrades... which went a little wrong... 

	Dear Customer,

	Please be advised the following Emergency Planned Outage Works
by Nextgen Network has been scheduled.

	This work is expected to result in an outage to your service/s
listed below.

	Start Date:       16/03/10            Start Time:      00:01
(AEDT)   

	Finish Date:     16/03/10            Finish Time:    02:00
(AEDT)  

	Description of Works: 

	Nextgen has organized an emergency planned outage to replace the
faulty card on one of the core device.  

	Your service listed above will experience an outage of 30
minutes during the specified window.  

	Nextgen apologizes for any inconvenience caused.

	If you require any further assistance please do not hesitate to
call Nextgen SMC on 1800 099 299.

	Regards

	Kindest Regards

	James Braunegg
	W:  1300 769 972  |  M:  0488 997 207 |  D:  (03) 9751 7616

	E:   james.braunegg at micron21.com  |  ABN:  85 126 029 203   

	This message is intended for the addressee named above. It may
contain privileged or confidential information. If you are not the
intended recipient of this message you must not use, copy, distribute or
disclose it to anyone other than the addressee. If you have received
this message in error please return the message to the sender by
replying to it and then delete the message from your computer.

	From: Damien Coxall [mailto:damien at dreamboat.id.au] 
	Sent: Tuesday, March 16, 2010 1:49 PM
	To: James Braunegg
	Subject: Re: [AusNOG] So who broke the internet now ?

	Hi James,

	There was nothing unusual in my monitoring overnight, either
with domestic / international transit or overseas transmission.

	Both Netspace and Primus live downstream of Optus and from the
looks of things so do you, so I can only guess the problem lay there
somewhere, even though I saw no issue with any of my data interconnects
with them.

	Damien,
	Vocus.

	On Tue, Mar 16, 2010 at 1:20 AM, James Braunegg <
james.braunegg at micron21.com> wrote:

	Hey all, did anyone else see some international and some
domestic ping times increase by 10x around 12:12am this morning? Lasting
for around 15 minutes ?

	We saw some of our outbound transit to "parts of the internet
slow down" we routed around the issues to resolve it.

	Also noticed people are complaining from many ISP's on
whirlpool..... 

	From our monitoring BGP was flapping around like a pig... its
all seems fine now but mainly interested to know what happen...

	My guess some major link went down... traffic went via redundant
paths which couldn't take all the traffic hence the increased ping
times...

	So who's cable broke ?

	Netspace issues

	http://forums.whirlpool.net.au/forum-replies.cfm?t=1404190
<http://forums.whirlpool.net.au/forum-replies.cfm?t=1404190> 

	Optus issues

	http://forums.whirlpool.net.au/forum-replies.cfm?t=1404247
<http://forums.whirlpool.net.au/forum-replies.cfm?t=1404247> 

	Primus issues

	http://forums.whirlpool.net.au/forum-replies.cfm?t=1404283
<http://forums.whirlpool.net.au/forum-replies.cfm?t=1404283> 

	Kindest Regards

	James Braunegg
	W:  1300 769 972  |  M:  0488 997 207 |  D:  (03) 9751 7616

	E:   james.braunegg at micron21.com
<mailto:james.braunegg at micron21.com>   |  ABN:  85 126 029 203   

	This message is intended for the addressee named above. It may
contain privileged or confidential information. If you are not the
intended recipient of this message you must not use, copy, distribute or
disclose it to anyone other than the addressee. If you have received
this message in error please return the message to the sender by
replying to it and then delete the message from your computer.

	_______________________________________________
	AusNOG mailing list
	AusNOG at lists.ausnog.net
	http://lists.ausnog.net/mailman/listinfo/ausnog

**********************************************************************

This email and any files transmitted with it are confidential and

intended solely for the use of the individual or entity to whom they

are addressed. If you have received this email in error please notify

the system manager.

**********************************************************************

_____________________________________________________________________ 
This e-mail has been scanned for viruses by MCI's Internet Managed 
Scanning Services - powered by MessageLabs. For further information 
visit http://www.mci.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20100318/76cc4f3d/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 2683 bytes
Desc: image001.jpg
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20100318/76cc4f3d/attachment.jpg>