[AusNOG] Vocus international service outage

Wed Aug 27 06:59:45 EST 2014

Still waiting to hear about the outage from a couple of days before the last
one as well. Don't forget that one.

From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Luke
Mackinnon
Sent: Wednesday, 27 August 2014 2:02 a.m.
To: Wolfgang Nagele (AusRegistry)
Cc: Ausnog at ausnog.net
Subject: Re: [AusNOG] Vocus international service outage

Hi Wolfgang, 

We expect to publish an interim RFO tomorrow. The final update to the NOC
notice was sent, 2014/08/23 23:50 UTC, please let me know if you did not
receive this. 

Given the complexity and scale of the event, in terms of number of devices
impacted we are still working on the report. We are keen to make sure we are
able to share as much detail as possible.  Unfortunately we will not have
the definitive conclusions from the main equipment vendor involved as they
are still analysing the information we provided and the hardware we returned
to them yesterday.

Once you receive the interim RFO, if you have any questions please don't
hesitate to contact the NOC or myself. 

Kind regards,

Luke MacKinnon | COO

M: +61 417735700   D: +61 2 8999 8289   E:
<mailto:%20luke.mackinnon at vocus.com.au> luke.mackinnon at vocus.com.au

P: 1300 88 99 88 or +61 2 8999 8999   W:  <http://www.vocus.com.au/>
vocus.com.au   

A: Vocus House, Level 1, 189 Miller Street, North Sydney, NSW 2060,
Australia

On 26 Aug 2014, at 9:15 pm, Wolfgang Nagele (AusRegistry)
<wolfgang.nagele at ausregistry.com.au
<mailto:wolfgang.nagele at ausregistry.com.au> > wrote:

Hi James,

Still waiting for the RfO on this whole thing with the details. Neither seen
one here nor as a follow-up to customer notifications.

Cheers,

Wolfgang

On 8/24/14, 12:38 AM, "Wolfgang Nagele (AusRegistry)"
<wolfgang.nagele at ausregistry.com.au
<mailto:wolfgang.nagele at ausregistry.com.au> > wrote:

Hi James,

Hmm - can understand that but would have expected that there is sufficient
redundancy in the LA landing of your network. I would have expected that a
re-route and taking SJC largely out of the equation would be possible.
Surprised to say the least .

Cheers,

Wolfgang

On 8/24/14, 12:10 AM, "James Spenceley" <james at iroute.org
<mailto:james at iroute.org> > wrote:

Early mail is a power surge in a US DC has damaged both core routers.
Wouldn't surprise me if transport from other providers out of that building
will be having similar issues. 

Circuits are being moved directly to borders as we speak.

Sent from my iPhone

On 24 Aug 2014, at 0:00, Jared Hirst <jared.hirst at serversaustralia.com.au
<mailto:jared.hirst at serversaustralia.com.au> > wrote:

WOW.... It has taken 2 hours to get remote hands to the DC with what seems
to be a device with no redundancy?

2014/08/23 13:55
UTC

Engineers are currently awaiting remote support in the US. Links will be
physically moved from the failed device in order to restore services on an
alternate device.

On Sat, Aug 23, 2014 at 11:29 PM, Andrew Yager <andrew at rwts.com.au
<mailto:andrew at rwts.com.au> > wrote:

[hijacking the thread.]

They say there is a big rewrite coming on the way rpd and sampled interact
in 14.2; and the slow convergance issues have been fixed in more releases
than I care to remember right now. but people say they are pretty good in
the 12.3r6 train. Our MX80's are slated for upgrade to that at some stage in
the next few months. 

Some noise about this on j-nsp again today.

Andrew

On 23 August 2014 23:23, Jonathan Thorpe <jthorpe at conexim.com.au> wrote:

True, but they otherwise work exceptionally well.

I'm not sure what kind of PowerPC CPU is doing all the work on an MX80's RE,
but I do sometimes wonder if the CPU on a <$40 Raspberry Pi might be more up
to the job :-P

From: Tony Wicks [mailto:tony at wicks.co.nz <mailto:tony at wicks.co.nz> ] 
Sent: Saturday, 23 August 2014 11:13 PM
To: Jonathan Thorpe
Cc: 'Ausnog at ausnog.net <mailto:Ausnog at ausnog.net> '
Subject: RE: [AusNOG] Vocus international service outage

Well, if you buy the big chassis boxes...

From: AusNOG [ <mailto:ausnog-bounces at lists.ausnog.net>
mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Jonathan Thorpe
Sent: Sunday, 24 August 2014 1:10 a.m.
To: Andrew Yager; Jared Hirst
Cc:  <mailto:Ausnog at ausnog.net> Ausnog at ausnog.net
Subject: Re: [AusNOG] Vocus international service outage

Glad I'm not the only one holding my breath on our MXs :)

From: AusNOG [ <mailto:ausnog-bounces at lists.ausnog.net>
mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Andrew Yager
Sent: Saturday, 23 August 2014 10:55 PM
To: Jared Hirst
Cc:  <mailto:Ausnog at ausnog.net> Ausnog at ausnog.net
Subject: Re: [AusNOG] Vocus international service outage

We've done the same (about 20 minutes ago).

Right now I hate how long Juniper MX's take to stabilise their routing table
with sampling on.

Andrew

On 23 August 2014 22:41, Jared Hirst <jared.hirst at serversaustralia.com.au
<mailto:jared.hirst at serversaustralia.com.au> > wrote:

We have just turned Vocus off. Using other providers for now, as the
flapping is causing it to go up and down.

On Sat, Aug 23, 2014 at 10:37 PM, Daniel Watson <Daniel at glovine.com.au
<mailto:Daniel at glovine.com.au> > wrote:

Indeed seeing some big drops in gaming traffic at present, normally we see
above 60mbit on weekends at evenings, but not even seeing 40mbit at present
:S

Regards,

Daniel Watson

Network Administrator / Network Operations Manager

E  <mailto:Daniel at GloVine.com.au> Daniel at GloVine.com.au

W  <http://www.glovine.com.au/> www.GloVine.com.au

From: AusNOG [mailto: <mailto:ausnog-bounces at lists.ausnog.net>
ausnog-bounces at lists.ausnog.net] On Behalf Of Jared Hirst
Sent: Saturday, 23 August 2014 10:35 PM

To: Andrew Cox
Cc:  <mailto:Ausnog at ausnog.net> Ausnog at ausnog.net
Subject: Re: [AusNOG] Vocus international service outage

Yeah we are seeing this! Everything running via them is flapping, they claim
to have 'routed around it' but that's not the case. Very frustrating as it's
been an hour and no one there seems to know whats going on....

On Sat, Aug 23, 2014 at 10:29 PM, Andrew Cox <
<mailto:andrew.cox at bigair.net.au> andrew.cox at bigair.net.au> wrote:

Hey All,

Just saw the dashboards light up with connectivity issues internationally
for Vocus services and thought I'd make others aware.

Vocus outage report is saying: "core network link between 59 Doody Street,
Alexandria and 55 South Market Street, San Jose has failed" which hopefully
isn't a Southern Cross fault!

Anyone else seeing this or have more info?

Cheers,

Andrew

_______________________________________________
AusNOG mailing list
 <mailto:AusNOG at lists.ausnog.net> AusNOG at lists.ausnog.net
 <http://lists.ausnog.net/mailman/listinfo/ausnog>
http://lists.ausnog.net/mailman/listinfo/ausnog

-- 

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net <mailto:AusNOG at lists.ausnog.net> 
http://lists.ausnog.net/mailman/listinfo/ausnog

-- 
Andrew Yager, Managing Director   MACS (Snr) CP BCompSc MCP
Real World Technology Solutions Pty Ltd - IT people you can trust
ph: 1300 798 718 <tel:1300%20798%20718>  or (02) 9037 0500
<tel:%2802%29%209037%200500> 
fax: (02) 9037 0591 <tel:%2802%29%209037%200591> 
http://www.rwts.com.au/

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net <mailto:AusNOG at lists.ausnog.net> 
http://lists.ausnog.net/mailman/listinfo/ausnog

-- 
Andrew Yager, Managing Director   MACS (Snr) CP BCompSc MCP
Real World Technology Solutions Pty Ltd - IT people you can trust
ph: 1300 798 718 <tel:1300%20798%20718>  or (02) 9037 0500
<tel:%2802%29%209037%200500> 
fax: (02) 9037 0591 <tel:%2802%29%209037%200591> 
http://www.rwts.com.au/

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net <mailto:AusNOG at lists.ausnog.net> 
http://lists.ausnog.net/mailman/listinfo/ausnog

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net <mailto:AusNOG at lists.ausnog.net> 
http://lists.ausnog.net/mailman/listinfo/ausnog

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net <mailto:AusNOG at lists.ausnog.net> 
http://lists.ausnog.net/mailman/listinfo/ausnog

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20140827/c123ca05/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 3129 bytes
Desc: not available
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20140827/c123ca05/attachment.png>