<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>The questions left to be answered for Vocus is why did it take 12
hours to restore one rail. Where was the static bypass and failing
that where was the maintenance bypass. Im guessing that's what
it's on maintenance bypass now and the slow action may have
something to do with the public holiday in Vic.. <br>
</p>
<p><br>
</p>
<p>I shouldn't speculate. But it is fun. <br>
</p>
<p><br>
</p>
<p> Something seriously let go in that UPS. If i had to put my
electrical engineers hat on I would say their load capacitor
maintenance was not done and let go. Or someone was working
upstream and broke the natural/earth bond. But there's a lot of
parts in a system like that , most modern DC UPS setups are fairly
robust. I once had 3 strings of batteries catch fire and melt
down and the UPS didn't miss a beat other then spewing a lot of
alarms. We moved strings around on the fly and nobody even
noticed. <br>
</p>
<p><br>
</p>
<p>The other things that seems odd is why did an A failure trip B.
Sounds like a single supply is at surge/break capacity. Also a
worry. <br>
</p>
<p><br>
</p>
<p>But in the end we all have redundancy around 530 Collins right ?
Right ? <br>
</p>
<p><br>
</p>
<p>Matt</p>
<p><br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 14/3/2023 11:50 am, Christopher
Hawker wrote:<br>
</div>
<blockquote type="cite"
cite="mid:SYBP282MB0538A5F3C118B6D6A4ED85F5AFBE9@SYBP282MB0538.AUSP282.PROD.OUTLOOK.COM">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<style type="text/css" style="display:none;">P {margin-top:0;margin-bottom:0;}</style>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);" class="elementToProof">
SY1 apparently experienced a full power outage back in July
2009, according to AusNOG Archives. Some reported full outages,
some partial. Was not a DC fault though, looks like a much
larger grid issue where a fault at the Bayswater Power Station
in the Hunter Valley (which affected generators in QLD and VIC)
caused about 1000MW to be dropped from the grid, resulting in
rolling blackouts across QLD, NSW, TAS, SA and VIC.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);" class="elementToProof ContentPasted0">
<a
href="https://lists.ausnog.net/pipermail/ausnog/2009-July/003222.html"
id="LPlnk264879" moz-do-not-send="true"
class="moz-txt-link-freetext">https://lists.ausnog.net/pipermail/ausnog/2009-July/003222.html</a><br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);" class="elementToProof ContentPasted0">
CH.</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt"
face="Calibri, sans-serif" color="#000000"><b>From:</b> AusNOG
<a class="moz-txt-link-rfc2396E" href="mailto:ausnog-bounces@lists.ausnog.net"><ausnog-bounces@lists.ausnog.net></a> on behalf of Nathan
Brookfield <a class="moz-txt-link-rfc2396E" href="mailto:Nathan.Brookfield@iperium.com.au"><Nathan.Brookfield@iperium.com.au></a><br>
<b>Sent:</b> Tuesday, March 14, 2023 11:40 AM<br>
<b>To:</b> Matt Perkins <a class="moz-txt-link-rfc2396E" href="mailto:matt@spectrum.com.au"><matt@spectrum.com.au></a>;
<a class="moz-txt-link-abbreviated" href="mailto:ausnog@lists.ausnog.net">ausnog@lists.ausnog.net</a> <a class="moz-txt-link-rfc2396E" href="mailto:ausnog@lists.ausnog.net"><ausnog@lists.ausnog.net></a><br>
<b>Subject:</b> Re: [AusNOG] Vocus Datacentre outage.</font>
<div> </div>
</div>
<style>@font-face
{font-family:Helvetica}@font-face
{font-family:"Cordia New"}@font-face
{font-family:"Cambria Math"}@font-face
{font-family:Calibri}@font-face
{font-family:Consolas}p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif}a:link, span.x_MsoHyperlink
{color:blue;
text-decoration:underline}pre
{margin:0cm;
font-size:10.0pt;
font-family:"Courier New"}span.x_HTMLPreformattedChar
{font-family:Consolas}span.x_gmailsignatureprefix
{}span.x_EmailStyle23
{font-family:"Calibri",sans-serif;
color:windowtext}.x_MsoChpDefault
{font-size:10.0pt}div.x_WordSection1
{}</style>
<div link="blue" vlink="purple" style="word-wrap:break-word"
lang="EN-AU">
<div class="x_WordSection1">
<p class="x_MsoNormal"><span style="">Have 100% had A and B
fail separately at SY1 but never both, thankfully!!!</span></p>
<p class="x_MsoNormal"><span style=""> </span></p>
<p class="x_MsoNormal"><span style="">Could have been part of
the facility though so don’t quote me.</span></p>
<p class="x_MsoNormal"><span style=""> </span></p>
<p class="x_MsoNormal"><span style="">UK on the other hand,
fail!</span></p>
<p class="x_MsoNormal"><span style=""> </span></p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt;
padding:3.0pt 0cm 0cm 0cm">
<p class="x_MsoNormal"><b><span lang="EN-US">From:</span></b><span
lang="EN-US"> Matt Perkins
<a class="moz-txt-link-rfc2396E" href="mailto:matt@spectrum.com.au"><matt@spectrum.com.au></a>
<br>
<b>Sent:</b> Tuesday, 14 March 2023 11:30 AM<br>
<b>To:</b> Nathan Brookfield
<a class="moz-txt-link-rfc2396E" href="mailto:Nathan.Brookfield@iperium.com.au"><Nathan.Brookfield@iperium.com.au></a>;
<a class="moz-txt-link-abbreviated" href="mailto:ausnog@lists.ausnog.net">ausnog@lists.ausnog.net</a><br>
<b>Subject:</b> Re: [AusNOG] Vocus Datacentre outage.</span></p>
</div>
</div>
<p class="x_MsoNormal"> </p>
<p>SY1 had A or B fail a few years back if my memory serves me
correctly. But I was referring London and Manchester which
both had total outages in recent years.
</p>
<p>Matt.</p>
<p> </p>
<div>
<p class="x_MsoNormal">On 14/3/2023 11:26 am, Nathan
Brookfield wrote:</p>
</div>
<blockquote style="margin-top:5.0pt; margin-bottom:5.0pt">
<p class="x_MsoNormal"><span style="">I’ve never had an A+B
power rail failure at Equinix in 18 years I’ve been in
there facilities, TOUCH WOOD!!!</span></p>
<p class="x_MsoNormal"><span style=""> </span></p>
<div>
<p class="x_MsoNormal" style="margin-bottom:12.0pt"><b><span
style="font-size:9.0pt;
font-family:"Helvetica",sans-serif;
color:#00233A">Nathan Brookfield | VK2NAB<br>
</span></b><span style="font-size:9.0pt;
font-family:"Helvetica",sans-serif;
color:#00233A">General Manager</span><span
style="font-size:10.0pt;
font-family:"Helvetica",sans-serif"><br>
<br>
</span><b><span style="font-size:9.0pt;
font-family:"Helvetica",sans-serif;
color:#FF5400">p</span></b><span
style="font-size:9.0pt;
font-family:"Helvetica",sans-serif;
color:#00233A">: 1300 592 330 |
</span><b><span style="font-size:9.0pt;
font-family:"Helvetica",sans-serif;
color:#FF5400">m</span></b><span
style="font-size:9.0pt;
font-family:"Helvetica",sans-serif;
color:#00233A">: 0412 266 008</span><span
style="font-size:10.0pt;
font-family:"Helvetica",sans-serif"><br>
</span><b><span style="font-size:9.0pt;
font-family:"Helvetica",sans-serif;
color:#FF5400">e</span></b><span
style="font-size:9.0pt;
font-family:"Helvetica",sans-serif;
color:#00233A">:
<a href="mailto:Nathan.brookfield@iperium.com.au"
moz-do-not-send="true" class="moz-txt-link-freetext">Nathan.brookfield@iperium.com.au</a>
| </span><b><span style="font-size:9.0pt;
font-family:"Helvetica",sans-serif;
color:#FF5400">w</span></b><span
style="font-size:10.0pt;
font-family:"Helvetica",sans-serif;
color:#00233A">: iperium.com.au</span><span
style="font-family:"Helvetica",sans-serif;
color:#00233A"><br>
<br>
</span><span style="font-size:10.0pt;
font-family:"Helvetica",sans-serif;
color:#00233A">Suite 4.02, 189 Kent Street Sydney NSW
2000</span></p>
<p class="x_MsoNormal" style="margin-bottom:12.0pt"><span
style="font-size:9.0pt;
font-family:"Arial",sans-serif;
color:#464646"><img id="x_Picture_x0020_2"
style="width:2.0833in; height:.6041in"
data-outlook-trace="F:1|T:1"
src="cid:part1.7yw1h804.sOMiKwBy@spectrum.com.au"
class="" width="200" height="58" border="0"></span><br>
<b><span
style="font-family:"Helvetica",sans-serif;
color:#00233A">Your Telco Team</span></b></p>
</div>
<p class="x_MsoNormal" style="margin-bottom:12.0pt"><span
style="font-size:9.0pt; color:darkgray">DISCLAIMER: This
document is intended solely for the named addressee.
This electronic communication, which includes any files
or attachments thereto, contains proprietary or
confidential information and may be privileged and
otherwise protected under copyright or other applicable
intellectual property laws. The use, disclosure, copying
or distribution of any of the information contained in
this document, by any person other than the addressee,
is strictly prohibited. If you received this document in
error, please contact the sender immediately and delete
all the material from any computer. Confidentiality and
legal privilege are not waived or lost by reason of
mistaken delivery to you. Any views or opinions
presented are solely those of the author and do not
necessarily represent those of Iperium. <br>
<br>
WARNING: Computer viruses can be transmitted via email.
The recipient should check this email and any
attachments for the presence of viruses. Iperium accepts
no liability for any damage caused by any virus
transmitted by this email.</span></p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt;
padding:3.0pt 0cm 0cm 0cm">
<p class="x_MsoNormal"><b><span lang="EN-US">From:</span></b><span
lang="EN-US"> AusNOG
<a href="mailto:ausnog-bounces@lists.ausnog.net"
moz-do-not-send="true"><ausnog-bounces@lists.ausnog.net></a>
<b>On Behalf Of </b>Matt Perkins<br>
<b>Sent:</b> Tuesday, 14 March 2023 11:18 AM<br>
<b>To:</b> <a href="mailto:ausnog@lists.ausnog.net"
moz-do-not-send="true"
class="moz-txt-link-freetext">ausnog@lists.ausnog.net</a><br>
<b>Subject:</b> Re: [AusNOG] Vocus Datacentre
outage.</span></p>
</div>
</div>
<p class="x_MsoNormal"> </p>
<p>Yea because Eqinix has never had a power outage. :) </p>
<p>I think it's a little premature to judge at the moment.
Doesn't sound good based on the current info but we have
all had bad days im sure.
</p>
<p>Matt</p>
<p> </p>
<div>
<p class="x_MsoNormal">On 14/3/2023 11:06 am, Bradley Amm
wrote:</p>
</div>
<blockquote style="margin-top:5.0pt; margin-bottom:5.0pt">
<p class="x_MsoNormal" style="margin-bottom:12.0pt">Friends
like let friends put gear in a Vocus dtacenter. Only
NextDC or Equinix :) </p>
<div>
<p class="x_MsoNormal">Sent from my iPhone</p>
</div>
<div>
<p class="x_MsoNormal"><br>
<br>
<br>
</p>
<blockquote style="margin-top:5.0pt;
margin-bottom:5.0pt">
<p class="x_MsoNormal" style="margin-bottom:12.0pt">On
14 Mar 2023, at 7:08 am, DaZZa
<a href="mailto:dazzagibbs@gmail.com"
moz-do-not-send="true"><dazzagibbs@gmail.com></a>
wrote:</p>
</blockquote>
</div>
<blockquote style="margin-top:5.0pt; margin-bottom:5.0pt">
<div>
<p class="x_MsoNormal"> </p>
<div>
<div>
<p class="x_MsoNormal">Doesn't say a whole lot
about Vocus' data center design methodology, nor
their periodic contingency testing.</p>
</div>
<div>
<p class="x_MsoNormal"> </p>
</div>
<div>
<p class="x_MsoNormal">They're apparently
replacing the entire UPS on Friday - guess we
can all expect more outages then when they stuff
that up.</p>
</div>
<div>
<p class="x_MsoNormal"> </p>
</div>
<div>
<p class="x_MsoNormal">D</p>
</div>
</div>
<p class="x_MsoNormal"> </p>
<div>
<div>
<p class="x_MsoNormal">On Mon, 13 Mar 2023 at
18:30, Chad Kelly <<a
href="mailto:chad@cpkws.com.au"
moz-do-not-send="true"
class="moz-txt-link-freetext">chad@cpkws.com.au</a>>
wrote:</p>
</div>
<blockquote style="border:none; border-left:solid
#CCCCCC 1.0pt; padding:0cm 0cm 0cm 6.0pt;
margin-left:4.8pt; margin-top:5.0pt;
margin-right:0cm; margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="x_MsoNormal" style=""><span
lang="EN-US">Thought I would create a
new thread on this, as the issue wasn’t
related to AussieBB.
</span></p>
<p class="x_MsoNormal" style=""><span
lang="EN-US">It is looking like this
Melbourne outage is similar to the
Sydney outage back in 2018, in that it
appears to have been yet again caused by
an issue with their UPS systems yet
again.</span></p>
<p class="x_MsoNormal" style=""><span
lang="EN-US">I’ll leave this hear <a
href="https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">
https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/</a>
and this article on today’s outage as
well.
</span></p>
<p class="x_MsoNormal" style=""><span
lang="EN-US"><a
href="https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987</a></span></p>
<p class="x_MsoNormal" style=""><span
lang="EN-US"> </span></p>
<p class="x_MsoNormal" style=""><span
lang="EN-US">Regards Chad. </span></p>
<p class="x_MsoNormal" style=""><span
lang="EN-US"> </span></p>
<p class="x_MsoNormal" style=""><span
lang="EN-US"> </span></p>
<p class="x_MsoNormal" style=""><span
lang="EN-US"> </span></p>
<p class="x_MsoNormal" style="">Chad Kelly </p>
<p class="x_MsoNormal" style="">Manager </p>
<p class="x_MsoNormal" style="">CPK Web
Services </p>
<p class="x_MsoNormal" style="">Phone 03
52730246</p>
<p class="x_MsoNormal" style="">Web <a
href="https://www.cpkws.com.au"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">
https://www.cpkws.com.au</a></p>
<div>
<p class="x_MsoNormal"><image001.jpg></p>
</div>
<p class="x_MsoNormal" style=""><span
lang="EN-US"> </span></p>
</div>
</div>
<p class="x_MsoNormal">_______________________________________________<br>
AusNOG mailing list<br>
<a href="mailto:AusNOG@lists.ausnog.net"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">AusNOG@lists.ausnog.net</a><br>
<a
href="https://lists.ausnog.net/mailman/listinfo/ausnog"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://lists.ausnog.net/mailman/listinfo/ausnog</a></p>
</div>
</blockquote>
</div>
<p class="x_MsoNormal"><br clear="all">
<br>
<span class="x_gmailsignatureprefix">-- </span></p>
<div>
<p class="x_MsoNormal">veg·e·tar·i·an:<br>
Ancient tribal slang for the village idiot who
can't hunt, fish or ride</p>
</div>
<p class="x_MsoNormal">_______________________________________________<br>
AusNOG mailing list<br>
<a href="mailto:AusNOG@lists.ausnog.net"
moz-do-not-send="true"
class="moz-txt-link-freetext">AusNOG@lists.ausnog.net</a><br>
<a
href="https://lists.ausnog.net/mailman/listinfo/ausnog"
moz-do-not-send="true"
class="moz-txt-link-freetext">https://lists.ausnog.net/mailman/listinfo/ausnog</a></p>
</div>
</blockquote>
<p class="x_MsoNormal"><br>
<br>
<br>
</p>
<pre>_______________________________________________</pre>
<pre>AusNOG mailing list</pre>
<pre><a href="mailto:AusNOG@lists.ausnog.net" moz-do-not-send="true" class="moz-txt-link-freetext">AusNOG@lists.ausnog.net</a></pre>
<pre><a href="https://lists.ausnog.net/mailman/listinfo/ausnog" moz-do-not-send="true" class="moz-txt-link-freetext">https://lists.ausnog.net/mailman/listinfo/ausnog</a></pre>
</blockquote>
<pre>-- </pre>
<pre>** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL </pre>
<pre> </pre>
<pre>/* Matt Perkins</pre>
<pre> Direct 02 8916 8101 Spectrum Networks Ptd. Ltd.</pre>
<pre> Office 1300 133 299 <a href="mailto:matt@spectrum.com.au" moz-do-not-send="true" class="moz-txt-link-freetext">matt@spectrum.com.au</a></pre>
<pre> ABN 66 090 112 913 Level 6, 350 George Street Sydney 2000</pre>
<pre> </pre>
<pre>*/</pre>
</blockquote>
<pre>-- </pre>
<pre>** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL </pre>
<pre> </pre>
<pre>/* Matt Perkins</pre>
<pre> Direct 02 8916 8101 Spectrum Networks Ptd. Ltd.</pre>
<pre> Office 1300 133 299 <a href="mailto:matt@spectrum.com.au" moz-do-not-send="true" class="moz-txt-link-freetext">matt@spectrum.com.au</a></pre>
<pre> ABN 66 090 112 913 Level 6, 350 George Street Sydney 2000</pre>
<pre> </pre>
<pre>*/</pre>
</div>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL
/* Matt Perkins
Direct 02 8916 8101 Spectrum Networks Ptd. Ltd.
Office 1300 133 299 <a class="moz-txt-link-abbreviated" href="mailto:matt@spectrum.com.au">matt@spectrum.com.au</a>
ABN 66 090 112 913 Level 6, 350 George Street Sydney 2000
*/</pre>
</body>
</html>