[AusNOG] Zettagrid Sydney Offline

Wed Jun 24 15:34:06 EST 2015

Perhaps the most important thing, and the thing that hasn't been mentioned, is to *test* your redundancy measures. Otherwise you're assuming your vendor's gear is always working as advertised, which of course it always does.
"Assumption is the mother of all f*** ups" - Under Siege 2: Dark Territory. 
It is worth having a read about how Netflix have automated their failure testing via an army of "monkeys"
The Netflix Tech Blog: The Netflix Simian Army

|   |
|   |   |   |   |   |
| The Netflix Tech Blog: The Netflix Simian ArmyWe’ve talked a bit in the past about our move to the cloud and John shared some of our lessons learned in going through that transition in a previous post.  |
|  |
| View on techblog.netflix.com | Preview by Yahoo |
|  |
|   |

      From: Shane Short <shane at short.id.au>
 To: Paul Brooks <pbrooks-ausnog at layer10.com.au> 
Cc: ausnog at lists.ausnog.net 
 Sent: Wednesday, 24 June 2015, 15:09
 Subject: Re: [AusNOG] Zettagrid Sydney Offline

What about on different continents? There could be some kind of catastrophic act of god that wipes out both facilities? Oh.. separate hemispheres too, to protect you against freak storm events..

Maybe we should co-lo on the moon too, incase there's some kind of catastrophic earth event.. but even then we still have single point of flares (SPoF) to contend with.. mmm

At some point you play the odds. How many times has the aforementioned "global fuckup" happened?

Paul Brooks wrote:

On 24/06/2015 2:25 PM, Tim Raphael wrote:

 On Wed, Jun 24, 2015 at 12:09 PM, Mark Newton <newton at atdot.dotat.org> wrote:

 If you aren't provider-diverse, you aren't diverse.

   - mark

  And ideally location diverse too for the purposes of DR. The latency between Sydney and Melbourne for a lot of our customers is so negligible at the application layer that they buy two vDCs, one in each of our Availability Zones and connect them together via our MPLS services. Each zone in our case is completely independent at the infrastructure level.   

 Are the zones monitored by independent NOCs, and operated by completely diverse NOC engineers so the same finger-fumble or automated provisioning tool can't be applied to configs in both locations?

 I've seen someone press 'commit' to a router config script with a typo  that proceeded to take down approx. 45 routers across 20-something countries. Each country was completely independent at the infrastructure level then, too.

 +1 What Mark said above. If your multiple diverse services stem ultimately from the same boardroom table, you aren't diverse.

 Paul.
 _______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
http://lists.ausnog.net/mailman/listinfo/ausnog

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
http://lists.ausnog.net/mailman/listinfo/ausnog

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20150624/c05e4183/attachment.html>