[AusNOG] Outage at iPrimus Datacentre - 55 King St, Melbourne

Sun Feb 1 23:52:43 EST 2009

Hi All,

>From my last post you can see I'm a bit annoyed by the outage at Primus
;)

We have only just settled customer penalty payments from the last
outage.

When we removed our app servers today, I have just been told this was
pre-organised by my people with NOC staff.

So, I'd like to clarify, we removed app servers with pre-arranged
approval from the Primus DC NOC team.

I don't have any issues with the NOC team (who have always been good to
us over the years) or the poor people in attendance on the day.

It was a debacle and I genuinely feel for the people and businesses
affected.

Back in my box now.

Regards, 

Darren Moss
General Manager, Director 
[p] 1300 131 083 x105 [f] 03 9532 6100 [m] 0421 042 999
[e] Darren.Moss at em3.com.au [w] www.em3.com.au <http://www.em3.com.au/> 
[h] www.em3.com.au/TechnicalSupport
<http://www.em3.com.au/TechnicalSupport.html>  

Reach me by dialing Extension 105.

em3 People and Technology | Managed Technology Experts
postal: PO Box 2333, Moorabbin VIC 3189

New Zealand Airedale Street, Auckland City
postal: PO Box 39573, Howick 2045 NZ
[p] (09) 92 555 26 [m] 021 841 541

Managed IT Services : Specialist Application Hosting: Hosted Microsoft
Exchange Server : Disaster Recovery
Blackberry and iPhone email : Spam Filtering and Virus Control :
Security & Load Testing : Technical Support

Find out more about our Business Technology services at 
http://services.em3.com.au

This communication may contain confidential or privileged information
intended solely for the individual or entity above. If you are not the
intended recipient you must not use, interfere with, disclose, copy or
retain this email and you should notify the sender immediately by return
email or by contacting our office on +61 1300 131 083. Opinions
expressed here are not necessarily those of em3 People and Technology
Pty Ltd.

From: ausnog-bounces at lists.ausnog.net
[mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Darren Moss
Sent: Sunday, 1 February 2009 10:35 PM
To: ausnog at ausnog.net
Subject: Re: [AusNOG] Outage at iPrimus Datacentre - 55 King St,
Melbourne

What an absolute disgrace. 

I was onsite today and the place was a disaster. No lights, no air-con,
security doors wide open.

No checking of IDs, process out the door.

My staff and I removed application servers and walked straight past the
NOC staff - who did not even ask what we were doing.

Yes, that's right, lugging HP Proliant boxes and rail kits under our
arms. Can you believe it ???

Thank goodness we removed the bulk of our application servers from there
in 2008.

The rest will be removed this week.

Regards, 

Darren Moss
General Manager, Director 
[p] 1300 131 083 x105 [f] 03 9532 6100 [m] 0421 042 999
[e] Darren.Moss at em3.com.au [w] www.em3.com.au <http://www.em3.com.au/> 
[h] www.em3.com.au/TechnicalSupport
<http://www.em3.com.au/TechnicalSupport.html>  

Reach me by dialing Extension 105.

em3 People and Technology | Managed Technology Experts
postal: PO Box 2333, Moorabbin VIC 3189

New Zealand Airedale Street, Auckland City
postal: PO Box 39573, Howick 2045 NZ
[p] (09) 92 555 26 [m] 021 841 541

Managed IT Services : Specialist Application Hosting: Hosted Microsoft
Exchange Server : Disaster Recovery
Blackberry and iPhone email : Spam Filtering and Virus Control :
Security & Load Testing : Technical Support

Find out more about our Business Technology services at 
http://services.em3.com.au

This communication may contain confidential or privileged information
intended solely for the individual or entity above. If you are not the
intended recipient you must not use, interfere with, disclose, copy or
retain this email and you should notify the sender immediately by return
email or by contacting our office on +61 1300 131 083. Opinions
expressed here are not necessarily those of em3 People and Technology
Pty Ltd.

Primus staff have just confirmed that the facility is back on mains
power, and power has been restored to all customer areas.

No word on what happened to the generator (generators?) yet.

From: Skeeve Stevens [mailto:skeeve at skeeve.org] 
Sent: Sunday, 1 February 2009 6:05 PM
To: 'Bevan Slattery'; Campbell, Alex; ausnog at ausnog.net
Subject: RE: [AusNOG] Outage at iPrimus Datacentre - 55 King St,
Melbourne

Any word as to what happened to the generators?

...Skeeve

--

Skeeve Stevens, CEO/Technical Director

eintellego Pty Ltd - The Networking Specialists

skeeve at eintellego.net / www.eintellego.net

Phone: 1300 753 383, Fax: (+612) 8572 9954

Cell +61 (0)414 753 383 / skype://skeeve

--

NOC, NOC, who's there?

Disclaimer: Limits of Liability and Disclaimer: This message is for the
named person's use only. It may contain sensitive and private
proprietary or legally privileged information. You must not, directly or
indirectly, use, disclose, distribute, print, or copy any part of this
message if you are not the intended recipient. eintellego Pty Ltd and
each legal entity in the Tefilah Pty Ltd group of companies reserve the
right to monitor all e-mail communications through its networks.  Any
views expressed in this message are those of the individual sender,
except where the message states otherwise and the sender is authorised
to state them to be the views of any such entity. Any reference to
costs, fee quotations, contractual transactions and variations to
contract terms is subject to separate confirmation in writing signed by
an authorised representative of eintellego. Whilst all efforts are made
to safeguard inbound and outbound e-mails, we cannot guarantee that
attachments are virus-free or compatible with your systems and do not
accept any liability in respect of viruses or computer problems
experienced.

From: ausnog-bounces at lists.ausnog.net
[mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Bevan Slattery
Sent: Sunday, 1 February 2009 5:58 PM
To: Campbell, Alex; skeeve at eintellego.net; ausnog at ausnog.net
Subject: Re: [AusNOG] Outage at iPrimus Datacentre - 55 King St,
Melbourne
Importance: High

Further update from Primus. Failure of substation.  City Power (I think)
have been working on the issue.  Problem has been rectified and are
about to switch back to mains as I type.

Cheers

[b]

________________________________

From: ausnog-bounces at lists.ausnog.net
[mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Campbell, Alex
Sent: Sunday, February 01, 2009 4:37 PM
To: skeeve at eintellego.net; ausnog at ausnog.net
Subject: Re: [AusNOG] Outage at iPrimus Datacentre - 55 King St,
Melbourne

We lost power at 2:40pm and saw both A & B feeds come back up in the
expansion room at about 4:02pm. At this stage some of the older areas
haven't had power restored yet.

Not getting any info from DC staff on cause or likelihood of continued
power issues tonight.

Alex
(who doesn't mind hiding out from the Melbourne heat in the aircon at
Primus right now)

-------------------------- 
Sent using BlackBerry 

________________________________

From: ausnog-bounces at lists.ausnog.net 
To: ausnog at ausnog.net 
Cc: nznog at list.waikato.ac.nz 
Sent: Sun Feb 01 15:12:59 2009
Subject: [AusNOG] Outage at iPrimus Datacentre - 55 King St, Melbourne 

Hey all,

Little detail at present... but the iPrimus DC in 55 King St Melbourne
seems to have had a complete failure.  Power, UPS and Generator have
apparently all failed.

Anyone else have more detail, feel free to follow up.

...Skeeve

--

Skeeve Stevens, CEO/Technical Director

eintellego Pty Ltd - The Networking Specialists

skeeve at eintellego.net / www.eintellego.net

Phone: 1300 753 383, Fax: (+612) 8572 9954

Cell +61 (0)414 753 383 / skype://skeeve

--

NOC, NOC, who's there?

Disclaimer: Limits of Liability and Disclaimer: This message is for the
named person's use only. It may contain sensitive and private
proprietary or legally privileged information. You must not, directly or
indirectly, use, disclose, distribute, print, or copy any part of this
message if you are not the intended recipient. eintellego Pty Ltd and
each legal entity in the Tefilah Pty Ltd group of companies reserve the
right to monitor all e-mail communications through its networks.  Any
views expressed in this message are those of the individual sender,
except where the message states otherwise and the sender is authorised
to state them to be the views of any such entity. Any reference to
costs, fee quotations, contractual transactions and variations to
contract terms is subject to separate confirmation in writing signed by
an authorised representative of eintellego. Whilst all efforts are made
to safeguard inbound and outbound e-mails, we cannot guarantee that
attachments are virus-free or compatible with your systems and do not
accept any liability in respect of viruses or computer problems
experienced.

______________________________________________________________________
This email has been scanned by MessageLabs anti-virus and spam system
www.em3.com.au
______________________________________________________________________

________________________________

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
http://lists.ausnog.net/mailman/listinfo/ausnog

______________________________________________________________________
This email has been scanned by MessageLabs anti-virus and spam system
www.em3.com.au
______________________________________________________________________

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20090201/fc7d4e06/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.gif
Type: image/gif
Size: 8972 bytes
Desc: image001.gif
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20090201/fc7d4e06/attachment.gif>