[AusNOG] Ozservers

[Matt Osborne]

Thanks everyone.

On Mon, Aug 1, 2016 at 6:48 PM, Robert Hudson <hudrob at gmail.com> wrote:

> From OzServers:
> --
> 7:35AM - 11:40AM AEST
>
> Issue Summary:
>
> At around 7:35AM our primary transit provider suffered from issues with
> the networking equipment providing connectivity to 96 Lytton Road.
> http://status.vocus.com.au/view-incident.aspx?IncidentID=198
>
> Normally in this situation BGP should have flipped routing through our
> alternative path via NextDC B1 and bypassed the affected upstream router.
> Due to erroneous configuration this failed to occur as it should and the
> secondary router would instead become unresponsive whenever the BGP flapped
> over.
>
> We have a contingency plan in which we can bypass Vocus entirely by
> utilizing on-demand transit from another provider in the event of
> catastrophic upstream failures but this is delivered via MegaPort to our
> NextDC POP.
>
> Effectively the network fell victim to a third party issue being
> compounded by an issue within our own infrastructure, an issue which also
> precluded the rollout of our failure plan.
>
> Resolution:
>
> The network was stabilized by our network contractor correcting the issue
> with the router at NextDC POP and allowing traffic to route around the
> affected Vocus equipment. When we have confirmation that our upstream has
> corrected their issues routing will be returned to its normal paths.
>
> Corrective Measures:
>
> * Work with network contractor to re-implement and test BGP failover
> between 96 Lytton and NextDC B1
> * Ensure support staff more quickly update social media platforms to keep
> clients in the loop. With our support being provided over our primary links
> it can result in clients being left in the dark when catastrophic network
> failure occurs.
> * Investigate ways in which additional transit providers can be introduced
> to the network blend in a timely manner upon catastrophic failure of
> primary transit.
>
>
> -----------------------------------------------
> Oz Servers Pty Ltd
> Phone: 1300 138 969
> Web: http://www.ozservers.com.au
>
> On 1 August 2016 at 10:44, Darryl Ware <dware at brizemedia.com> wrote:
>
>> This is related to the Vocus outage in QLD.
>>
>> Cheers,
>> Darryl
>>
>>
>> > On 1 Aug 2016, at 8:34 AM, Jonathan Eastgate <
>> jonathan.eastgate at simpro.co> wrote:
>> >
>> > Hi Matt.
>> >
>> > Confirmed - we also have a small amount of equipment in at OzServers -
>> and all offline.
>> >
>> > Phone number rings and rings and then diverts to a non working number.
>> >
>> > Cheers!
>> >
>> >
>> > Jonathan J. Eastgate
>> > Chief Technology Officer | simPRO Software Group
>> > Ph: 1300 139 467    +61 7 3147 8777
>> >
>> >
>> >
>> > Keep up to date with simPRO at: http://simprogroup.com/blog
>> >
>> > The contents of this email are subject to our email disclaimer.
>> >
>> >
>> > On Mon, Aug 1, 2016 at 8:12 AM, Matt Osborne <protocolture at gmail.com>
>> wrote:
>> > Hello,
>> >
>> > Anyone else seeing issues at OzServers in Brisbane?
>> >
>> > Their phone is ringing out and we cannot see any of our kit over there.
>> >
>> > If you have a mobile number for someone answering over there please let
>> me know off list.
>> >
>> > Regards,
>> >
>> > Matt Osborne.
>> >
>> >
>> > _______________________________________________
>> > AusNOG mailing list
>> > AusNOG at lists.ausnog.net
>> > http://lists.ausnog.net/mailman/listinfo/ausnog
>> >
>> >
>> >
>> > --_______________________________________________
>> > AusNOG mailing list
>> > AusNOG at lists.ausnog.net
>> > http://lists.ausnog.net/mailman/listinfo/ausnog
>>
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> http://lists.ausnog.net/mailman/listinfo/ausnog
>>
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20160802/b5df9ec1/attachment.html>