[AusNOG] Zettagrid Sydney Offline

[Shane Short]

+1

Failures happen, even in the most redundant well architected 
infrastructure. If it didn't, we wouldn't need to be on call, because we 
*all* do things properly and perfectly, right?

I've had instances where you set up things like a SAN with redundant 
storage controllers, only to have the failover either not work properly, 
or not work at all. I highly suspect that's what's happened here, given 
the information in the updates.

You can't just shuck the responsibility of redundancy to your compute 
provider... as much as they're clearly trying to do the right thing, 
*you* should be planning for the worst, while hoping for the best. *You* 
should have a DR plan, backups and failure options. If you're so heavily 
impacted by a compute outage, I suggest *you* aren't doing something 
right either.

So much for DR plans....
-Shane

Brad Peczka wrote:
> If your entire business is down because of a provider outage, then you really need to reassess your hosting strategy.
>
> Outsourcing your hosting outsources responsibility for the hardware, but not your responsibility to your business and customers in the event of an outage. At the very least have a backup provider, or have physical iron that you control and can use to provide core services until your primary compute is back online - and then, learn to do risk assessments and structure your environment accordingly.
>
> Regards,
> -Brad.
> ________________________________________
> From: AusNOG [ausnog-bounces at lists.ausnog.net] On Behalf Of Nathan Brookfield [Nathan.Brookfield at simtronic.com.au]
> Sent: Wednesday, 24 June 2015 9:27 AM
> To: 'Joseph Goldman'; ausnog at lists.ausnog.net
> Subject: Re: [AusNOG] Zettagrid Sydney Offline
>
> If you’re paying a small fortune for virtual server hosting and your entire business or a large amount of customers has been down since 0715 and you are not being provided an ETA then I think you would want answers as well.
>
> Kindest Regards,
> Nathan Brookfield (VK2NAB)
>
> Chief Executive Officer
> Simtronic Technologies Pty Ltd
>
> Local: (02) 4749 4949 | Fax: (02) 4749 4950 | Direct: (02) 4749 4951
> Web: http://www.simtronic.com.au<http://www.simtronic.com.au/>  | E-mail: nathan.brookfield at simtronic.com.au<mailto:nathan.brookfield at simtronic.com.au>
>
> From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Joseph Goldman
> Sent: Wednesday, 24 June 2015 11:24 AM
> To: ausnog at lists.ausnog.net
> Subject: Re: [AusNOG] Zettagrid Sydney Offline
>
> As far as customer updates go mid-crisis, I believe the status page is more than detailed - an RFO should be expected after the fact, but mid-outage I think as much info as they've given you is as much info as you could really expect!
> On 24/06/15 11:11, James Cunningham wrote:
> Hi All,
>
> Does anyone have any information on the massive outage that Zettagrid are having in Sydney at the moment? No answer from their support phone number, and emails are going unanswered.
>
> Apparantly all of their VMs in their Sydney data centre are offline because of hardware failure on their SAN.
>
> http://status.zettagrid.com/
>
> We have about 20 customer servers there, and they are all offline, and our helpdesk can't handle all the screaming customers we have at the moment.
> So much for redundancy....
> James
>
>
>
>
>
> _______________________________________________
>
> AusNOG mailing list
>
> AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
>
> http://lists.ausnog.net/mailman/listinfo/ausnog
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog