[AusNOG] Ozservers

[Robert Hudson]

>From OzServers:
--
7:35AM - 11:40AM AEST

Issue Summary:

At around 7:35AM our primary transit provider suffered from issues with the
networking equipment providing connectivity to 96 Lytton Road.
http://status.vocus.com.au/view-incident.aspx?IncidentID=198

Normally in this situation BGP should have flipped routing through our
alternative path via NextDC B1 and bypassed the affected upstream router.
Due to erroneous configuration this failed to occur as it should and the
secondary router would instead become unresponsive whenever the BGP flapped
over.

We have a contingency plan in which we can bypass Vocus entirely by
utilizing on-demand transit from another provider in the event of
catastrophic upstream failures but this is delivered via MegaPort to our
NextDC POP.

Effectively the network fell victim to a third party issue being compounded
by an issue within our own infrastructure, an issue which also precluded
the rollout of our failure plan.

Resolution:

The network was stabilized by our network contractor correcting the issue
with the router at NextDC POP and allowing traffic to route around the
affected Vocus equipment. When we have confirmation that our upstream has
corrected their issues routing will be returned to its normal paths.

Corrective Measures:

* Work with network contractor to re-implement and test BGP failover
between 96 Lytton and NextDC B1
* Ensure support staff more quickly update social media platforms to keep
clients in the loop. With our support being provided over our primary links
it can result in clients being left in the dark when catastrophic network
failure occurs.
* Investigate ways in which additional transit providers can be introduced
to the network blend in a timely manner upon catastrophic failure of
primary transit.


-----------------------------------------------
Oz Servers Pty Ltd
Phone: 1300 138 969
Web: http://www.ozservers.com.au

On 1 August 2016 at 10:44, Darryl Ware <dware at brizemedia.com> wrote:

> This is related to the Vocus outage in QLD.
>
> Cheers,
> Darryl
>
>
> > On 1 Aug 2016, at 8:34 AM, Jonathan Eastgate <
> jonathan.eastgate at simpro.co> wrote:
> >
> > Hi Matt.
> >
> > Confirmed - we also have a small amount of equipment in at OzServers -
> and all offline.
> >
> > Phone number rings and rings and then diverts to a non working number.
> >
> > Cheers!
> >
> >
> > Jonathan J. Eastgate
> > Chief Technology Officer | simPRO Software Group
> > Ph: 1300 139 467    +61 7 3147 8777
> >
> >
> >
> > Keep up to date with simPRO at: http://simprogroup.com/blog
> >
> > The contents of this email are subject to our email disclaimer.
> >
> >
> > On Mon, Aug 1, 2016 at 8:12 AM, Matt Osborne <protocolture at gmail.com>
> wrote:
> > Hello,
> >
> > Anyone else seeing issues at OzServers in Brisbane?
> >
> > Their phone is ringing out and we cannot see any of our kit over there.
> >
> > If you have a mobile number for someone answering over there please let
> me know off list.
> >
> > Regards,
> >
> > Matt Osborne.
> >
> >
> > _______________________________________________
> > AusNOG mailing list
> > AusNOG at lists.ausnog.net
> > http://lists.ausnog.net/mailman/listinfo/ausnog
> >
> >
> >
> > --_______________________________________________
> > AusNOG mailing list
> > AusNOG at lists.ausnog.net
> > http://lists.ausnog.net/mailman/listinfo/ausnog
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20160801/8111b8d2/attachment.html>