[AusNOG] After Sandy Knocks Out Power, ... (huffingtonpost.com)

[Matt Perkins]

To my mind the win by these guys is not that they were or were prepared 
for every occurrence or that they did what needed to be done to keep it 
on line. All admirable. But the real win here was that they keep the 
communication lines open with their customers.

  There was no ducking and weaving the problem. The end result was they 
communicated the situation and that allowed some customers to help (in 
the way of providing the bucket brigade).  The lesson we can all learn 
here and i have said it before. Communicate with your customers. Dont 
Lie. Dont cover up. Unless you have been grossly negligent being 
transparent and giving the customers good timely information they will 
normaly forgive you.

Matt.


  On 2/11/12 2:18 PM, Christopher Pollock wrote:
> A++ post, would agree again.
>
> It's supremely easy to sit back and criticise people for not doing 
> blank to prevent an outage or datacentre failure.  You don't have to 
> peruse Whirlpool for long after any outage to hear people 'why didnt 
> they do this or that'. And in a perfect world, our HA plans would all 
> go exactly to plan and every scenario could be prepared for and 
> anything in the world could happen and everything would be fine anyway.
>
> The reality of it is, if you've spent any real amount of time building 
> & managing datacentres or maintaining highly-available services, you 
> know that a HA plan is really only as good as the circumstances you 
> can feasibly spend money preparing for.  Sometimes it means running to 
> Mitre 10 to buy their entire stock of fans.  Other times it means 
> carrying hundreds of litres of diesel up a dark staircase.
>
> There is basically no amount of money you can spend that guarantees 
> your stuff won't fall over, so my hat is off to the people who got hit 
> by a bad situation and put some hard work into staying online.
>
> --
> Christopher Pollock,
> io Networks Pty Ltd.
> e. chris at ionetworks.com.au <mailto:chris at ionetworks.com.au>
> p. 1300 1 2 4 8 16
> d. 07 3188 7588
> m. 0410 747 765
> skype: christopherpollock
> twitter.com/chrisionetworks <http://twitter.com/chrisionetworks>
> http://www.ionetworks.com.au
> In-house, Outsourced.
>
>
>
> On Fri, Nov 2, 2012 at 7:01 AM, Mark Newton <newton at atdot.dotat.org 
> <mailto:newton at atdot.dotat.org>> wrote:
>
>     That's all very well and good, but it seems to me that they've
>     just suffered a 100 year storm and they stayed up.
>
>     I'm sure you can armchair quarterback 'til the cows come home, but
>     you're basically criticizing a success.
>
>     On 02/11/2012, at 6:35 AM, Martin Hepworth <maxsec at gmail.com
>     <mailto:maxsec at gmail.com>> wrote:
>
>>     That still leaves them with a single data centre, your DR should
>>     be in a separate physical location at least 45 miles away 
>
>     ... if you're a bank.  Which they aren't.
>
>     Note also that you're talking about DR, but what they've actually
>     demonstrated is HA.  The article already said that they had a DR
>     plan to relocate services to another datacentre; they didn't need
>     to invoke it because they didn't have a disaster, /because they
>     stayed up./
>     /
>     /
>     If they actually went 100% down, then brought themselves up at
>     another datacentre 6 hours later, you'd be praising them for
>     having a well thought out old-school DR plan, right?
>
>     (some enterprises have DR plans which take anything up to a week
>     to execute.  DR != HA.)
>
>>     This goes back to old school infosec on risk and costs to
>>     business of outages.
>
>     Yes, and also the cost of infrastructure.  One doesn't protect
>     one's fruit bowl with a $50,000 safe.
>
>>     Problem with alot of the new facilities being build on
>>     Saas/cloudy offerings is that theyve grown so fast theyve nit
>>     done some of thr basics and rely on luck to get out of problems!
>
>     The problem that SaaS/cloud offerings have is that they're reliant
>     on a software substrate that's rarely been tested in true
>     adversity, and therefore rely quite a bit on trust.
>
>     When the chips are down, is your cloud provider as good as they
>     say they are?
>
>       - mark
>
>
>     _______________________________________________
>     AusNOG mailing list
>     AusNOG at lists.ausnog.net <mailto:AusNOG at lists.ausnog.net>
>     http://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog


-- 
/* Matt Perkins
         Direct 1300 137 379     Spectrum Networks Ptd. Ltd.
         Office 1300 133 299     matt at spectrum.com.au
         Fax    1300 133 255     Level 6, 350 George Street Sydney 2000
         SIP 1300137379 at sip.spectrum.com.au
         PGP/GNUPG Public Key can be found at  http://pgp.mit.edu
*/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20121102/d724c9c7/attachment.html>