<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">To my mind the win by these guys is not
that they were or were prepared for every occurrence or that they
did what needed to be done to keep it on line. All admirable. But
the real win here was that they keep the communication lines open
with their customers.<br>
<br>
There was no ducking and weaving the problem. The end result was
they communicated the situation and that allowed some customers to
help (in the way of providing the bucket brigade). The lesson we
can all learn here and i have said it before. Communicate with
your customers. Dont Lie. Dont cover up. Unless you have been
grossly negligent being transparent and giving the customers good
timely information they will normaly forgive you. <br>
<br>
Matt.<br>
<br>
<br>
On 2/11/12 2:18 PM, Christopher Pollock wrote:<br>
</div>
<blockquote
cite="mid:CAK6biYicoO7-Bac+wEWVv2end9cqJVQSEMYoQ0jyzXGue6p8YQ@mail.gmail.com"
type="cite">A++ post, would agree again.
<div><br>
</div>
<div>It's supremely easy to sit back and criticise people for not
doing blank to prevent an outage or datacentre failure. You
don't have to peruse Whirlpool for long after any outage to hear
people 'why didnt they do this or that'. And in a perfect world,
our HA plans would all go exactly to plan and every scenario
could be prepared for and anything in the world could happen and
everything would be fine anyway.</div>
<div><br>
</div>
<div>The reality of it is, if you've spent any real amount of time
building & managing datacentres or maintaining
highly-available services, you know that a HA plan is really
only as good as the circumstances you can feasibly spend money
preparing for. Sometimes it means running to Mitre 10 to buy
their entire stock of fans. Other times it means carrying
hundreds of litres of diesel up a dark staircase.</div>
<div><br>
</div>
<div>There is basically no amount of money you can spend that
guarantees your stuff won't fall over, so my hat is off to the
people who got hit by a bad situation and put some hard work
into staying online.</div>
<div><br>
</div>
<div>--<br>
Christopher Pollock,<br>
io Networks Pty Ltd.
<div>e. <a moz-do-not-send="true"
href="mailto:chris@ionetworks.com.au" target="_blank">chris@ionetworks.com.au</a><br>
p. 1300 1 2 4 8 16</div>
<div>d. 07 3188 7588</div>
<div>
m. 0410 747 765</div>
<div>skype: christopherpollock</div>
<div><a moz-do-not-send="true"
href="http://twitter.com/chrisionetworks" target="_blank">twitter.com/chrisionetworks</a></div>
<div>
<div><a moz-do-not-send="true"
href="http://www.ionetworks.com.au" target="_blank">http://www.ionetworks.com.au</a><br>
In-house, Outsourced.</div>
</div>
<br>
<br>
<br>
<div class="gmail_quote">On Fri, Nov 2, 2012 at 7:01 AM, Mark
Newton <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:newton@atdot.dotat.org" target="_blank">newton@atdot.dotat.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">That's all very well and
good, but it seems to me that they've just suffered a 100
year storm and they stayed up.
<div><br>
</div>
<div>I'm sure you can armchair quarterback 'til the cows
come home, but you're basically criticizing a success.</div>
<div><br>
<div>
<div class="im">
<div>On 02/11/2012, at 6:35 AM, Martin Hepworth <<a
moz-do-not-send="true"
href="mailto:maxsec@gmail.com" target="_blank">maxsec@gmail.com</a>>
wrote:</div>
<br>
<blockquote type="cite">That still leaves them with
a single data centre, your DR should be in a
separate physical location at least 45 miles away
</blockquote>
<div><br>
</div>
</div>
<div>... if you're a bank. Which they aren't.</div>
<div><br>
</div>
<div>Note also that you're talking about DR, but what
they've actually demonstrated is HA. The article
already said that they had a DR plan to relocate
services to another datacentre; they didn't need to
invoke it because they didn't have a disaster, <i>because
they stayed up.</i></div>
<div><i><br>
</i></div>
<div>If they actually went 100% down, then brought
themselves up at another datacentre 6 hours later,
you'd be praising them for having a well thought out
old-school DR plan, right?</div>
<div><br>
</div>
<div>(some enterprises have DR plans which take
anything up to a week to execute. DR != HA.)</div>
<div class="im">
<div><br>
</div>
<blockquote type="cite">
<div>This goes back to old school infosec on risk
and costs to business of outages. </div>
</blockquote>
<div><br>
</div>
</div>
<div>Yes, and also the cost of infrastructure. One
doesn't protect one's fruit bowl with a $50,000
safe.</div>
<div class="im"><br>
<blockquote type="cite">
<div>Problem with alot of the new facilities being
build on Saas/cloudy offerings is that theyve
grown so fast theyve nit done some of thr basics
and rely on luck to get out of problems!</div>
</blockquote>
<div><br>
</div>
</div>
The problem that SaaS/cloud offerings have is that
they're reliant on a software substrate that's rarely
been tested in true adversity, and therefore rely
quite a bit on trust.</div>
<div><br>
</div>
<div>When the chips are down, is your cloud provider as
good as they say they are? </div>
<span class="HOEnZb"><font color="#888888">
<div><br>
</div>
<div> - mark</div>
</font></span></div>
<div><br>
</div>
</div>
<br>
_______________________________________________<br>
AusNOG mailing list<br>
<a moz-do-not-send="true"
href="mailto:AusNOG@lists.ausnog.net">AusNOG@lists.ausnog.net</a><br>
<a moz-do-not-send="true"
href="http://lists.ausnog.net/mailman/listinfo/ausnog"
target="_blank">http://lists.ausnog.net/mailman/listinfo/ausnog</a><br>
<br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
AusNOG mailing list
<a class="moz-txt-link-abbreviated" href="mailto:AusNOG@lists.ausnog.net">AusNOG@lists.ausnog.net</a>
<a class="moz-txt-link-freetext" href="http://lists.ausnog.net/mailman/listinfo/ausnog">http://lists.ausnog.net/mailman/listinfo/ausnog</a>
</pre>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
/* Matt Perkins
Direct 1300 137 379 Spectrum Networks Ptd. Ltd.
Office 1300 133 299 <a class="moz-txt-link-abbreviated" href="mailto:matt@spectrum.com.au">matt@spectrum.com.au</a>
Fax 1300 133 255 Level 6, 350 George Street Sydney 2000
SIP <a class="moz-txt-link-abbreviated" href="mailto:1300137379@sip.spectrum.com.au">1300137379@sip.spectrum.com.au</a>
PGP/GNUPG Public Key can be found at <a class="moz-txt-link-freetext" href="http://pgp.mit.edu">http://pgp.mit.edu</a>
*/
</pre>
</body>
</html>