<div dir="ltr">From OzServers:<div>--</div><div><span style="font-family:verdana;font-size:13.3333px">7:35AM - 11:40AM AEST</span><br style="font-family:verdana;font-size:13.3333px"><br style="font-family:verdana;font-size:13.3333px"><span style="font-family:verdana;font-size:13.3333px">Issue Summary:</span><br style="font-family:verdana;font-size:13.3333px"><br style="font-family:verdana;font-size:13.3333px"><span style="font-family:verdana;font-size:13.3333px">At around 7:35AM our primary transit provider suffered from issues with the networking equipment providing connectivity to 96 Lytton Road.</span><a href="http://status.vocus.com.au/view-incident.aspx?IncidentID=198" target="_blank" style="font-family:verdana;font-size:13.3333px">http://status.vocus.com.au/view-incident.aspx?IncidentID=198</a><br style="font-family:verdana;font-size:13.3333px"><br style="font-family:verdana;font-size:13.3333px"><span style="font-family:verdana;font-size:13.3333px">Normally in this situation BGP should have flipped routing through our alternative path via NextDC B1 and bypassed the affected upstream router. Due to erroneous configuration this failed to occur as it should and the secondary router would instead become unresponsive whenever the BGP flapped over.</span><br style="font-family:verdana;font-size:13.3333px"><br style="font-family:verdana;font-size:13.3333px"><span style="font-family:verdana;font-size:13.3333px">We have a contingency plan in which we can bypass Vocus entirely by utilizing on-demand transit from another provider in the event of catastrophic upstream failures but this is delivered via MegaPort to our NextDC POP.</span><br style="font-family:verdana;font-size:13.3333px"><br style="font-family:verdana;font-size:13.3333px"><span style="font-family:verdana;font-size:13.3333px">Effectively the network fell victim to a third party issue being compounded by an issue within our own infrastructure, an issue which also precluded the rollout of our failure plan.</span><br style="font-family:verdana;font-size:13.3333px"><br style="font-family:verdana;font-size:13.3333px"><span style="font-family:verdana;font-size:13.3333px">Resolution:</span><br style="font-family:verdana;font-size:13.3333px"><br style="font-family:verdana;font-size:13.3333px"><span style="font-family:verdana;font-size:13.3333px">The network was stabilized by our network contractor correcting the issue with the router at NextDC POP and allowing traffic to route around the affected Vocus equipment. When we have confirmation that our upstream has corrected their issues routing will be returned to its normal paths.</span><br style="font-family:verdana;font-size:13.3333px"><br style="font-family:verdana;font-size:13.3333px"><span style="font-family:verdana;font-size:13.3333px">Corrective Measures:</span><br style="font-family:verdana;font-size:13.3333px"><br style="font-family:verdana;font-size:13.3333px"><span style="font-family:verdana;font-size:13.3333px">* Work with network contractor to re-implement and test BGP failover between 96 Lytton and NextDC B1</span><br style="font-family:verdana;font-size:13.3333px"><span style="font-family:verdana;font-size:13.3333px">* Ensure support staff more quickly update social media platforms to keep clients in the loop. With our support being provided over our primary links it can result in clients being left in the dark when catastrophic network failure occurs.</span><br style="font-family:verdana;font-size:13.3333px"><span style="font-family:verdana;font-size:13.3333px">* Investigate ways in which additional transit providers can be introduced to the network blend in a timely manner upon catastrophic failure of primary transit.</span><br style="font-family:verdana;font-size:13.3333px"><br style="font-family:verdana;font-size:13.3333px"><br style="font-family:verdana;font-size:13.3333px"><span style="font-family:verdana;font-size:13.3333px">------------------------------</span><span style="font-family:verdana;font-size:13.3333px">-----------------</span><br style="font-family:verdana;font-size:13.3333px"><span style="font-family:verdana;font-size:13.3333px">Oz Servers Pty Ltd</span><br style="font-family:verdana;font-size:13.3333px"><span style="font-family:verdana;font-size:13.3333px">Phone: 1300 138 969</span><br style="font-family:verdana;font-size:13.3333px"><span style="font-family:verdana;font-size:13.3333px">Web: </span><a href="http://www.ozservers.com.au/" target="_blank" style="font-family:verdana;font-size:13.3333px">http://www.ozservers.com.au</a><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 1 August 2016 at 10:44, Darryl Ware <span dir="ltr"><<a href="mailto:dware@brizemedia.com" target="_blank">dware@brizemedia.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">This is related to the Vocus outage in QLD.<br>
<br>
Cheers,<br>
Darryl<br>
<span class=""><br>
<br>
> On 1 Aug 2016, at 8:34 AM, Jonathan Eastgate <<a href="mailto:jonathan.eastgate@simpro.co">jonathan.eastgate@simpro.co</a>> wrote:<br>
><br>
> Hi Matt.<br>
><br>
> Confirmed - we also have a small amount of equipment in at OzServers - and all offline.<br>
><br>
> Phone number rings and rings and then diverts to a non working number.<br>
><br>
> Cheers!<br>
><br>
><br>
> Jonathan J. Eastgate<br>
> Chief Technology Officer | simPRO Software Group<br>
> Ph: 1300 139 467 +61 7 3147 8777<br>
><br>
><br>
><br>
</span><span class="">> Keep up to date with simPRO at: <a href="http://simprogroup.com/blog" rel="noreferrer" target="_blank">http://simprogroup.com/blog</a><br>
><br>
> The contents of this email are subject to our email disclaimer.<br>
><br>
><br>
> On Mon, Aug 1, 2016 at 8:12 AM, Matt Osborne <<a href="mailto:protocolture@gmail.com">protocolture@gmail.com</a>> wrote:<br>
> Hello,<br>
><br>
> Anyone else seeing issues at OzServers in Brisbane?<br>
><br>
> Their phone is ringing out and we cannot see any of our kit over there.<br>
><br>
> If you have a mobile number for someone answering over there please let me know off list.<br>
><br>
> Regards,<br>
><br>
> Matt Osborne.<br>
><br>
><br>
> _______________________________________________<br>
> AusNOG mailing list<br>
> <a href="mailto:AusNOG@lists.ausnog.net">AusNOG@lists.ausnog.net</a><br>
> <a href="http://lists.ausnog.net/mailman/listinfo/ausnog" rel="noreferrer" target="_blank">http://lists.ausnog.net/mailman/listinfo/ausnog</a><br>
><br>
><br>
><br>
</span>> --_______________________________________________<br>
<div class="HOEnZb"><div class="h5">> AusNOG mailing list<br>
> <a href="mailto:AusNOG@lists.ausnog.net">AusNOG@lists.ausnog.net</a><br>
> <a href="http://lists.ausnog.net/mailman/listinfo/ausnog" rel="noreferrer" target="_blank">http://lists.ausnog.net/mailman/listinfo/ausnog</a><br>
<br>
_______________________________________________<br>
AusNOG mailing list<br>
<a href="mailto:AusNOG@lists.ausnog.net">AusNOG@lists.ausnog.net</a><br>
<a href="http://lists.ausnog.net/mailman/listinfo/ausnog" rel="noreferrer" target="_blank">http://lists.ausnog.net/mailman/listinfo/ausnog</a><br>
</div></div></blockquote></div><br></div>