<html><body><div style="color:#000; background-color:#fff; font-family:Helvetica Neue-Light, Helvetica Neue Light, Helvetica Neue, Helvetica, Arial, Lucida Grande, Sans-Serif;font-size:16px"><div id="yui_3_16_0_1_1435121480886_6764"><span id="yui_3_16_0_1_1435121480886_6835">Perhaps the most important thing, and the thing that hasn't been mentioned, is to *test* your redundancy measures. Otherwise you're assuming your vendor's gear is always working as advertised, which of course it always does.</span></div><div id="yui_3_16_0_1_1435121480886_6764"><span><br></span></div><div id="yui_3_16_0_1_1435121480886_6764">"Assumption is the mother of all f*** ups" - Under Siege 2: Dark Territory. </div><div id="yui_3_16_0_1_1435121480886_6764"><br></div><div id="yui_3_16_0_1_1435121480886_6764">It is worth having a read about how Netflix have automated their failure testing via an army of "monkeys"</div><div id="yui_3_16_0_1_1435121480886_6764"><br></div><div id="yui_3_16_0_1_1435121480886_6764" dir="ltr"><a href="http://techblog.netflix.com/2011/07/netflix-simian-army.html" id="yui_3_16_0_1_1435121480886_10027">The Netflix Tech Blog: The Netflix Simian Army</a><br></div><div style="width:450px; font-family: 'Georgia', 'Times', 'Times New Roman', 'serif';margin-top:5px; margin-bottom: 5px; background-color: #ffffff;" contenteditable="false" id="enhancrCard_0" class="link-enhancr-attachment link-enhancr-element "><table class="link-enhancr-element" cellspacing="0" cellpadding="0" border="0" style="width:450px; height:auto; position: relative; display: block;" id="yui_3_16_0_1_1435121480886_10049"><tbody id="yui_3_16_0_1_1435121480886_10048"><tr class="link-enhancr-element" id="yui_3_16_0_1_1435121480886_10047"><td class="link-enhancr-element" colspan="7" style="height: 1px; background-color: #e5e5e5; font-size: 1px; border-collapse: collapse;" id="yui_3_16_0_1_1435121480886_10046"><div class="link-enhancr-element" style="height: 1px; background-color: #e5e5e5; font-size: 1px; line-height:0px;" id="yui_3_16_0_1_1435121480886_10045"> </div></td></tr><tr class="link-enhancr-element" id="yui_3_16_0_1_1435121480886_10052"><td rowspan="5" class="link-enhancr-element" style="width: 1px; background-color: #e5e5e5; font-size: 1pt; border-collapse: collapse;"><div class="link-enhancr-element" style="width: 1px; background-color: #e5e5e5; font-size: 1pt;"> </div></td><td rowspan="5" class="link-enhancr-element" style="width: 14px; background-color: #ffffff; font-size: 0pt; border-collapse: collapse;"><div class="link-enhancr-element" style="width: 14px; background-color: #ffffff; font-size: 14pt;"> </div></td><td colspan="2" class="link-enhancr-element" style="height: 6px; background-color: #ffffff; font-size: 0pt; border-collapse: collapse;" id="yui_3_16_0_1_1435121480886_10051"><div class="link-enhancr-element" style="height: 6px; background-color: #ffffff; font-size: 6pt;" id="yui_3_16_0_1_1435121480886_10050"> </div></td><td rowspan="5" class="link-enhancr-element" style="width: 20px; background-color: #ffffff; font-size: 0pt; border-collapse: collapse;" id="yui_3_16_0_1_1435121480886_10061"><div class="link-enhancr-element" style="width: 20px; background-color: #ffffff; font-size: 20pt;"> </div></td><td class="link-enhancr-element" rowspan="5" width="1" style="width: 1px; background-color: #e5e5e5; font-size: 1pt; border-collapse: collapse;"><div class="link-enhancr-element" style="width: 1px; background-color: #e5e5e5; font-size: 1pt;"> </div></td></tr><tr id="yui_3_16_0_1_1435121480886_10058"><td class="link-enhancr-element" colspan="2" style="width: 100%; vertical-align: middle; font-family: 'Georgia', 'Times', 'Times New Roman', 'serif';" id="yui_3_16_0_1_1435121480886_10057"><div class="link-enhancr-text-part link-enhancr-element" style="line-height:16.5px; background-color: #ffffff; width: 414px;" id="yui_3_16_0_1_1435121480886_10056"><div class="link-enhancr-element" style="word-wrap: break-word; word-break: break-all;" id="yui_3_16_0_1_1435121480886_10055"><span class="link-enhancr-element icon icon-shrink link-enhancr-toggle"></span><span class="link-enhancr-element icon icon-close link-enhancr-delete" id="yui_3_16_0_1_1435121480886_10060"></span><a href="http://techblog.netflix.com/2011/07/netflix-simian-army.html" class="link-enhancr-card-urlWrapper link-enhancr-element" style="text-decoration: none !important; color: #000000 !important; line-height: 100%; font-size: 18px; display: block;" id="yui_3_16_0_1_1435121480886_10054"><span class="link-enhancr-element link-enhancr-card-title" style="margin: 0; font-weight: normal;margin-bottom: 3px; font-size: 18px; line-height: 21px; max-height: 43px; color: #000000; overflow: hidden !important; display: inline-block;" id="yui_3_16_0_1_1435121480886_10053">The Netflix Tech Blog: The Netflix Simian Army</span></a><div style="font-size: 13px; line-height: 20px; color: #999999; max-height: 81px; font-family: 'Georgia', 'Times', 'Times New Roman', 'serif';overflow: hidden;" class="link-enhancr-card-description link-enhancr-element" id="yui_3_16_0_1_1435121480886_10059">We’ve talked a bit in the past about our move to the cloud and John shared some of our lessons learned in going through that transition in a previous post. </div></div></div></td></tr><tr id="yui_3_16_0_1_1435121480886_10064"><td colspan="2" class="link-enhancr-element" style="height: 6px; background-color: #ffffff; font-size: 0pt; border-collapse: collapse;" id="yui_3_16_0_1_1435121480886_10063"><div class="link-enhancr-element" style="height: 6px; background-color: #ffffff; font-size: 6pt;" id="yui_3_16_0_1_1435121480886_10062"></div></td></tr><tr><td class="link-enhancr-element" style="vertical-align: middle; font-family: 'Arial', 'Helvetica Neue', 'Helvetica', 'sans-serif';"><div class="link-enhancr-element" style="font-size: 0pt;"><a href="http://techblog.netflix.com/2011/07/netflix-simian-army.html" class="link-enhancr-card-url link-enhancr-element" style="color: black; text-decoration: none !important;cursor:pointer !important;" target="_blank"><span class="link-enhancr-element link-enhancr-view-on" style="display: inline-block; line-height: 11px; max-width: 314px; min-width: 254px; overflow: hidden; max-height: 13px; word-break: break-all;" id="yui_3_16_0_1_1435121480886_10042"><span class="link-enhancr-element link-enhancr-mobile-no-resize" style="vertical-align:middle; font-size: 9px; line-height: 11px; color: #999999; -moz-text-size-adjust: none; -ms-text-size-adjust: none; -webkit-text-size-adjust:none; text-size-adjust:none;">View on <span style="font-weight: bold" class="link-enhancr-view-on-domain">techblog.netflix.com</span></span></span></a></div></td><td class="link-enhancr-element" style="vertical-align: middle; width: 100px; font-family: 'Arial', 'Helvetica Neue', 'Helvetica', 'sans-serif';"><div class="link-enhancr-element link-enhancr-preview-wrapper" style="max-width: 100px; min-width: 80px; overflow: hidden; text-align: right; line-height: 11px; max-height: 13px; font-size: 0pt;"><span class="link-enhancr-element link-enhancr-preview-by link-enhancr-mobile-no-resize" style="vertical-align:middle; font-size: 9px; line-height: 11px; color: #999999; -moz-text-size-adjust: none; -ms-text-size-adjust: none; -webkit-text-size-adjust:none; text-size-adjust:none;">Preview by Yahoo</span></div></td></tr><tr><td colspan="2" class="link-enhancr-element" style="height: 9px; background-color: #ffffff; font-size: 0pt; border-collapse: collapse;"><div class="link-enhancr-element" style="height: 9px; background-color: #ffffff; font-size: 9pt;"></div></td></tr><tr class="link-enhancr-element"><td class="link-enhancr-element" colspan="7" style="height: 1px; background-color: #e5e5e5; font-size: 1px; border-collapse: collapse;"><div class="link-enhancr-element" style="height: 1px; background-color: #e5e5e5; font-size: 1px; line-height:0px"> </div></td></tr></tbody></table></div><div id="yui_3_16_0_1_1435121480886_6764"> </div><br> <div style="font-family: Helvetica Neue-Light, Helvetica Neue Light, Helvetica Neue, Helvetica, Arial, Lucida Grande, Sans-Serif; font-size: 16px;" id="yui_3_16_0_1_1435121480886_9950"> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, Sans-Serif; font-size: 16px;" id="yui_3_16_0_1_1435121480886_9949"> <div dir="ltr" id="yui_3_16_0_1_1435121480886_9948"> <hr size="1" id="yui_3_16_0_1_1435121480886_9947"> <font size="2" face="Arial" id="yui_3_16_0_1_1435121480886_10065"> <b><span style="font-weight:bold;">From:</span></b> Shane Short <shane@short.id.au><br> <b><span style="font-weight: bold;">To:</span></b> Paul Brooks <pbrooks-ausnog@layer10.com.au> <br><b><span style="font-weight: bold;">Cc:</span></b> ausnog@lists.ausnog.net <br> <b><span style="font-weight: bold;">Sent:</span></b> Wednesday, 24 June 2015, 15:09<br> <b><span style="font-weight: bold;">Subject:</span></b> Re: [AusNOG] Zettagrid Sydney Offline<br> </font> </div> <div class="y_msg_container" id="yui_3_16_0_1_1435121480886_10067"><br><div id="yiv2311385199"><div id="yui_3_16_0_1_1435121480886_10066">What about on different
continents? There could be some kind of catastrophic act of god that
wipes out both facilities? Oh.. separate hemispheres too, to protect you
against freak storm events..<br clear="none">
<br clear="none">
Maybe we should co-lo on the moon too, incase there's some kind of
catastrophic earth event.. but even then we still have single point of
flares (SPoF) to contend with.. mmm<br clear="none">
<br clear="none">
At some point you play the odds. How many times has the aforementioned
"global fuckup" happened?<br clear="none">
<div class="qtdSeparateBR"><br><br></div><div class="yiv2311385199yqt2389310787" id="yiv2311385199yqtfd45693"><br clear="none">
Paul Brooks wrote:
<blockquote type="cite">
</blockquote></div></div><div class="yiv2311385199yqt2389310787" id="yiv2311385199yqtfd52315"><div><div class="yiv2311385199moz-cite-prefix">On 24/06/2015 2:25 PM, Tim Raphael
wrote:<br clear="none">
</div>
<blockquote type="cite"><div dir="ltr">
<div class="yiv2311385199gmail_extra"><br clear="none">
<div class="yiv2311385199gmail_quote">On Wed, Jun 24, 2015 at 12:09 PM,
Mark Newton <span dir="ltr"><<a rel="nofollow" shape="rect" ymailto="mailto:newton@atdot.dotat.org" target="_blank" href="mailto:newton@atdot.dotat.org">newton@atdot.dotat.org</a>></span>
wrote:<br clear="none">
<blockquote class="yiv2311385199gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><br clear="none">
<br clear="none">
If you aren't provider-diverse, you aren't diverse.<br clear="none">
<span class="yiv2311385199HOEnZb"><font color="#888888"><br clear="none">
- mark<br clear="none">
</font></span>
<div class="yiv2311385199HOEnZb">
<div class="yiv2311385199h5"><br clear="none">
</div>
</div></blockquote>
<div><br clear="none">
</div>
<div>And ideally location diverse too for the purposes of
DR. The latency between Sydney and Melbourne for a lot of
our customers is so negligible at the application layer
that they buy two vDCs, one in each of our Availability
Zones and connect them together via our MPLS services.
Each zone in our case is completely independent at the
infrastructure level.</div>
</div>
</div>
</div></blockquote>
<br clear="none">
Are the zones monitored by independent NOCs, and operated by
completely diverse NOC engineers so the same finger-fumble or
automated provisioning tool can't be applied to configs in both
locations?<br clear="none">
<br clear="none">
I've seen someone press 'commit' to a router config script with a
typo that proceeded to take down approx. 45 routers across
20-something countries. Each country was completely independent at
the infrastructure level then, too.<br clear="none">
<br clear="none">
+1 What Mark said above. If your multiple diverse services stem
ultimately from the same boardroom table, you aren't diverse.<br clear="none">
<br clear="none">
Paul.<br clear="none">
<pre>_______________________________________________
AusNOG mailing list
<a rel="nofollow" shape="rect" class="yiv2311385199moz-txt-link-abbreviated" ymailto="mailto:AusNOG@lists.ausnog.net" target="_blank" href="mailto:AusNOG@lists.ausnog.net">AusNOG@lists.ausnog.net</a>
<a rel="nofollow" shape="rect" class="yiv2311385199moz-txt-link-freetext" target="_blank" href="http://lists.ausnog.net/mailman/listinfo/ausnog">http://lists.ausnog.net/mailman/listinfo/ausnog</a>
</pre>
</div></div></div><br><div class="yqt2389310787" id="yqtfd80950">_______________________________________________<br clear="none">AusNOG mailing list<br clear="none"><a shape="rect" ymailto="mailto:AusNOG@lists.ausnog.net" href="mailto:AusNOG@lists.ausnog.net">AusNOG@lists.ausnog.net</a><br clear="none"><a shape="rect" href="http://lists.ausnog.net/mailman/listinfo/ausnog" target="_blank">http://lists.ausnog.net/mailman/listinfo/ausnog</a><br clear="none"></div><br><br></div> </div> </div> </div></body></html>