[AusNOG] Zettagrid Sydney Offline

[Paul Brooks]

On 24/06/2015 3:09 PM, Shane Short wrote:
> What about on different continents? There could be some kind of catastrophic act of
> god that wipes out both facilities? Oh.. separate hemispheres too, to protect you
> against freak storm events..

Yep - thats exactly why Google and AWS zones are scattered across the planet. Not so
much from a facility point of view, but it helps having a 'local' node for local
customers to operate from if you've lost connectivity between nodes, such as common
cable cut scenario (undersea earthquake off Phillipines took out 6 different submarine
cables simultaneously) type event.


>
> At some point you play the odds. How many times has the aforementioned "global
> fuckup" happened?
As it happened it wasn't service-affecting - it took out the '-2' router in each pair
in each city, the other one carried on, and nobody noticed.
Did cause some white knuckles though.







>
> Paul Brooks wrote:
>> On 24/06/2015 2:25 PM, Tim Raphael wrote:
>>>
>>> On Wed, Jun 24, 2015 at 12:09 PM, Mark Newton <newton at atdot.dotat.org
>>> <mailto:newton at atdot.dotat.org>> wrote:
>>>
>>>
>>>
>>>     If you aren't provider-diverse, you aren't diverse.
>>>
>>>       - mark
>>>
>>>
>>> And ideally location diverse too for the purposes of DR. The latency between
>>> Sydney and Melbourne for a lot of our customers is so negligible at the
>>> application layer that they buy two vDCs, one in each of our Availability Zones
>>> and connect them together via our MPLS services. Each zone in our case is
>>> completely independent at the infrastructure level.
>>
>> Are the zones monitored by independent NOCs, and operated by completely diverse NOC
>> engineers so the same finger-fumble or automated provisioning tool can't be applied
>> to configs in both locations?
>>
>> I've seen someone press 'commit' to a router config script with a typo  that
>> proceeded to take down approx. 45 routers across 20-something countries. Each
>> country was completely independent at the infrastructure level then, too.
>>
>> +1 What Mark said above. If your multiple diverse services stem ultimately from the
>> same boardroom table, you aren't diverse.
>>
>> Paul.
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> http://lists.ausnog.net/mailman/listinfo/ausnog

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20150624/93481582/attachment.html>