<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
p.xmsonormal, li.xmsonormal, div.xmsonormal
{mso-style-name:x_msonormal;
margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.xgmailsignatureprefix
{mso-style-name:x_gmailsignatureprefix;}
span.EmailStyle27
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;
font-weight:normal;
font-style:normal;
text-decoration:none none;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-AU link=blue vlink=purple style='word-wrap:break-word'><div class=WordSection1><p class=MsoNormal><span style='mso-fareast-language:EN-US'>Something that I have to keep reminding beancounters with A/B planning is don’t go above 40% load on either side.<o:p></o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'>Its very difficult to convince a CFO who’s entire life is spent with a spreadsheet open in front of him is that N+1 means typically Double, and that Kw != Kva.<o:p></o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span lang=EN-US>From:</span></b><span lang=EN-US> AusNOG <ausnog-bounces@lists.ausnog.net> <b>On Behalf Of </b>Matt Perkins<br><b>Sent:</b> Tuesday, March 14, 2023 9:18 AM<br><b>To:</b> Christopher Hawker <chris@thesysadmin.dev>; Nathan Brookfield <Nathan.Brookfield@iperium.com.au>; ausnog@lists.ausnog.net<br><b>Subject:</b> Re: [AusNOG] Vocus Datacentre outage.<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p>The questions left to be answered for Vocus is why did it take 12 hours to restore one rail. Where was the static bypass and failing that where was the maintenance bypass. Im guessing that's what it's on maintenance bypass now and the slow action may have something to do with the public holiday in Vic.. <o:p></o:p></p><p><o:p> </o:p></p><p>I shouldn't speculate. But it is fun. <o:p></o:p></p><p><o:p> </o:p></p><p> Something seriously let go in that UPS. If i had to put my electrical engineers hat on I would say their load capacitor maintenance was not done and let go. Or someone was working upstream and broke the natural/earth bond. But there's a lot of parts in a system like that , most modern DC UPS setups are fairly robust. I once had 3 strings of batteries catch fire and melt down and the UPS didn't miss a beat other then spewing a lot of alarms. We moved strings around on the fly and nobody even noticed. <o:p></o:p></p><p><o:p> </o:p></p><p>The other things that seems odd is why did an A failure trip B. Sounds like a single supply is at surge/break capacity. Also a worry. <o:p></o:p></p><p><o:p> </o:p></p><p>But in the end we all have redundancy around 530 Collins right ? Right ? <o:p></o:p></p><p><o:p> </o:p></p><p>Matt<o:p></o:p></p><p><o:p> </o:p></p><p><o:p> </o:p></p><div><p class=MsoNormal>On 14/3/2023 11:50 am, Christopher Hawker wrote:<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><p class=MsoNormal><span style='font-size:12.0pt;color:black'>SY1 apparently experienced a full power outage back in July 2009, according to AusNOG Archives. Some reported full outages, some partial. Was not a DC fault though, looks like a much larger grid issue where a fault at the Bayswater Power Station in the Hunter Valley (which affected generators in QLD and VIC) caused about 1000MW to be dropped from the grid, resulting in rolling blackouts across QLD, NSW, TAS, SA and VIC.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;color:black'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;color:black'><a href="https://lists.ausnog.net/pipermail/ausnog/2009-July/003222.html">https://lists.ausnog.net/pipermail/ausnog/2009-July/003222.html</a><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;color:black'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;color:black'>CH.<o:p></o:p></span></p></div><div class=MsoNormal align=center style='text-align:center'><hr size=2 width="98%" align=center></div><div id=divRplyFwdMsg><p class=MsoNormal><b><span style='color:black'>From:</span></b><span style='color:black'> AusNOG <a href="mailto:ausnog-bounces@lists.ausnog.net"><ausnog-bounces@lists.ausnog.net></a> on behalf of Nathan Brookfield <a href="mailto:Nathan.Brookfield@iperium.com.au"><Nathan.Brookfield@iperium.com.au></a><br><b>Sent:</b> Tuesday, March 14, 2023 11:40 AM<br><b>To:</b> Matt Perkins <a href="mailto:matt@spectrum.com.au"><matt@spectrum.com.au></a>; <a href="mailto:ausnog@lists.ausnog.net">ausnog@lists.ausnog.net</a> <a href="mailto:ausnog@lists.ausnog.net"><ausnog@lists.ausnog.net></a><br><b>Subject:</b> Re: [AusNOG] Vocus Datacentre outage.</span> <o:p></o:p></p><div><p class=MsoNormal> <o:p></o:p></p></div></div><div><div><p class=xmsonormal>Have 100% had A and B fail separately at SY1 but never both, thankfully!!!<o:p></o:p></p><p class=xmsonormal> <o:p></o:p></p><p class=xmsonormal>Could have been part of the facility though so don’t quote me.<o:p></o:p></p><p class=xmsonormal> <o:p></o:p></p><p class=xmsonormal>UK on the other hand, fail!<o:p></o:p></p><p class=xmsonormal> <o:p></o:p></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=xmsonormal><b><span lang=EN-US>From:</span></b><span lang=EN-US> Matt Perkins <a href="mailto:matt@spectrum.com.au"><matt@spectrum.com.au></a> <br><b>Sent:</b> Tuesday, 14 March 2023 11:30 AM<br><b>To:</b> Nathan Brookfield <a href="mailto:Nathan.Brookfield@iperium.com.au"><Nathan.Brookfield@iperium.com.au></a>; <a href="mailto:ausnog@lists.ausnog.net">ausnog@lists.ausnog.net</a><br><b>Subject:</b> Re: [AusNOG] Vocus Datacentre outage.</span><o:p></o:p></p></div></div><p class=xmsonormal> <o:p></o:p></p><p>SY1 had A or B fail a few years back if my memory serves me correctly. But I was referring London and Manchester which both had total outages in recent years. <o:p></o:p></p><p>Matt.<o:p></o:p></p><p> <o:p></o:p></p><div><p class=xmsonormal>On 14/3/2023 11:26 am, Nathan Brookfield wrote:<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=xmsonormal>I’ve never had an A+B power rail failure at Equinix in 18 years I’ve been in there facilities, TOUCH WOOD!!!<o:p></o:p></p><p class=xmsonormal> <o:p></o:p></p><div><p class=xmsonormal style='margin-bottom:12.0pt'><b><span style='font-size:9.0pt;font-family:"Helvetica",sans-serif;color:#00233A'>Nathan Brookfield | VK2NAB<br></span></b><span style='font-size:9.0pt;font-family:"Helvetica",sans-serif;color:#00233A'>General Manager</span><span style='font-size:10.0pt;font-family:"Helvetica",sans-serif'><br><br></span><b><span style='font-size:9.0pt;font-family:"Helvetica",sans-serif;color:#FF5400'>p</span></b><span style='font-size:9.0pt;font-family:"Helvetica",sans-serif;color:#00233A'>: 1300 592 330 | </span><b><span style='font-size:9.0pt;font-family:"Helvetica",sans-serif;color:#FF5400'>m</span></b><span style='font-size:9.0pt;font-family:"Helvetica",sans-serif;color:#00233A'>: 0412 266 008</span><span style='font-size:10.0pt;font-family:"Helvetica",sans-serif'><br></span><b><span style='font-size:9.0pt;font-family:"Helvetica",sans-serif;color:#FF5400'>e</span></b><span style='font-size:9.0pt;font-family:"Helvetica",sans-serif;color:#00233A'>: <a href="mailto:Nathan.brookfield@iperium.com.au">Nathan.brookfield@iperium.com.au</a> | </span><b><span style='font-size:9.0pt;font-family:"Helvetica",sans-serif;color:#FF5400'>w</span></b><span style='font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#00233A'>: iperium.com.au</span><span style='font-family:"Helvetica",sans-serif;color:#00233A'><br><br></span><span style='font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#00233A'>Suite 4.02, 189 Kent Street Sydney NSW 2000</span><o:p></o:p></p><p class=xmsonormal style='margin-bottom:12.0pt'><span style='font-size:9.0pt;font-family:"Arial",sans-serif;color:#464646'><img border=0 width=200 height=58 style='width:2.0833in;height:.6041in' id="x_Picture_x0020_2" src="cid:image001.png@01D9565E.996332F0"></span><br><b><span style='font-family:"Helvetica",sans-serif;color:#00233A'>Your Telco Team</span></b><o:p></o:p></p></div><p class=xmsonormal style='margin-bottom:12.0pt'><span style='font-size:9.0pt;color:darkgray'>DISCLAIMER: This document is intended solely for the named addressee. This electronic communication, which includes any files or attachments thereto, contains proprietary or confidential information and may be privileged and otherwise protected under copyright or other applicable intellectual property laws. The use, disclosure, copying or distribution of any of the information contained in this document, by any person other than the addressee, is strictly prohibited. If you received this document in error, please contact the sender immediately and delete all the material from any computer. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you. Any views or opinions presented are solely those of the author and do not necessarily represent those of Iperium. <br><br>WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. Iperium accepts no liability for any damage caused by any virus transmitted by this email.</span><o:p></o:p></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=xmsonormal><b><span lang=EN-US>From:</span></b><span lang=EN-US> AusNOG <a href="mailto:ausnog-bounces@lists.ausnog.net"><ausnog-bounces@lists.ausnog.net></a> <b>On Behalf Of </b>Matt Perkins<br><b>Sent:</b> Tuesday, 14 March 2023 11:18 AM<br><b>To:</b> <a href="mailto:ausnog@lists.ausnog.net">ausnog@lists.ausnog.net</a><br><b>Subject:</b> Re: [AusNOG] Vocus Datacentre outage.</span><o:p></o:p></p></div></div><p class=xmsonormal> <o:p></o:p></p><p>Yea because Eqinix has never had a power outage. :) <o:p></o:p></p><p>I think it's a little premature to judge at the moment. Doesn't sound good based on the current info but we have all had bad days im sure. <o:p></o:p></p><p>Matt<o:p></o:p></p><p> <o:p></o:p></p><div><p class=xmsonormal>On 14/3/2023 11:06 am, Bradley Amm wrote:<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=xmsonormal style='margin-bottom:12.0pt'>Friends like let friends put gear in a Vocus dtacenter. Only NextDC or Equinix :) <o:p></o:p></p><div><p class=xmsonormal>Sent from my iPhone<o:p></o:p></p></div><div><p class=xmsonormal style='margin-bottom:12.0pt'><br><br><o:p></o:p></p><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=xmsonormal style='margin-bottom:12.0pt'>On 14 Mar 2023, at 7:08 am, DaZZa <a href="mailto:dazzagibbs@gmail.com"><dazzagibbs@gmail.com></a> wrote:<o:p></o:p></p></blockquote></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><p class=xmsonormal> <o:p></o:p></p><div><div><p class=xmsonormal>Doesn't say a whole lot about Vocus' data center design methodology, nor their periodic contingency testing.<o:p></o:p></p></div><div><p class=xmsonormal> <o:p></o:p></p></div><div><p class=xmsonormal>They're apparently replacing the entire UPS on Friday - guess we can all expect more outages then when they stuff that up.<o:p></o:p></p></div><div><p class=xmsonormal> <o:p></o:p></p></div><div><p class=xmsonormal>D<o:p></o:p></p></div></div><p class=xmsonormal> <o:p></o:p></p><div><div><p class=xmsonormal>On Mon, 13 Mar 2023 at 18:30, Chad Kelly <<a href="mailto:chad@cpkws.com.au">chad@cpkws.com.au</a>> wrote:<o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt'><div><div><div><p class=xmsonormal><span lang=EN-US>Thought I would create a new thread on this, as the issue wasn’t related to AussieBB. </span><o:p></o:p></p><p class=xmsonormal><span lang=EN-US>It is looking like this Melbourne outage is similar to the Sydney outage back in 2018, in that it appears to have been yet again caused by an issue with their UPS systems yet again.</span><o:p></o:p></p><p class=xmsonormal><span lang=EN-US>I’ll leave this hear <a href="https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/" target="_blank">https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/</a> and this article on today’s outage as well. </span><o:p></o:p></p><p class=xmsonormal><span lang=EN-US><a href="https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987" target="_blank">https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987</a></span><o:p></o:p></p><p class=xmsonormal><span lang=EN-US> </span><o:p></o:p></p><p class=xmsonormal><span lang=EN-US>Regards Chad. </span><o:p></o:p></p><p class=xmsonormal><span lang=EN-US> </span><o:p></o:p></p><p class=xmsonormal><span lang=EN-US> </span><o:p></o:p></p><p class=xmsonormal><span lang=EN-US> </span><o:p></o:p></p><p class=xmsonormal>Chad Kelly <o:p></o:p></p><p class=xmsonormal>Manager <o:p></o:p></p><p class=xmsonormal>CPK Web Services <o:p></o:p></p><p class=xmsonormal>Phone 03 52730246<o:p></o:p></p><p class=xmsonormal>Web <a href="https://www.cpkws.com.au" target="_blank">https://www.cpkws.com.au</a><o:p></o:p></p><div><p class=xmsonormal><image001.jpg><o:p></o:p></p></div><p class=xmsonormal><span lang=EN-US> </span><o:p></o:p></p></div></div><p class=xmsonormal>_______________________________________________<br>AusNOG mailing list<br><a href="mailto:AusNOG@lists.ausnog.net" target="_blank">AusNOG@lists.ausnog.net</a><br><a href="https://lists.ausnog.net/mailman/listinfo/ausnog" target="_blank">https://lists.ausnog.net/mailman/listinfo/ausnog</a><o:p></o:p></p></div></blockquote></div><p class=xmsonormal><br clear=all><br><span class=xgmailsignatureprefix>-- </span><o:p></o:p></p><div><p class=xmsonormal>veg·e·tar·i·an:<br>Ancient tribal slang for the village idiot who can't hunt, fish or ride<o:p></o:p></p></div><p class=xmsonormal>_______________________________________________<br>AusNOG mailing list<br><a href="mailto:AusNOG@lists.ausnog.net">AusNOG@lists.ausnog.net</a><br><a href="https://lists.ausnog.net/mailman/listinfo/ausnog">https://lists.ausnog.net/mailman/listinfo/ausnog</a><o:p></o:p></p></div></blockquote><p class=xmsonormal style='margin-bottom:12.0pt'><br><br><o:p></o:p></p><pre>_______________________________________________<o:p></o:p></pre><pre>AusNOG mailing list<o:p></o:p></pre><pre><a href="mailto:AusNOG@lists.ausnog.net">AusNOG@lists.ausnog.net</a><o:p></o:p></pre><pre><a href="https://lists.ausnog.net/mailman/listinfo/ausnog">https://lists.ausnog.net/mailman/listinfo/ausnog</a><o:p></o:p></pre></blockquote><pre>-- <o:p></o:p></pre><pre>** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL <o:p></o:p></pre><pre> <o:p></o:p></pre><pre>/* Matt Perkins<o:p></o:p></pre><pre> Direct 02 8916 8101 Spectrum Networks Ptd. Ltd.<o:p></o:p></pre><pre> Office 1300 133 299 <a href="mailto:matt@spectrum.com.au">matt@spectrum.com.au</a><o:p></o:p></pre><pre> ABN 66 090 112 913 Level 6, 350 George Street Sydney 2000<o:p></o:p></pre><pre> <o:p></o:p></pre><pre>*/<o:p></o:p></pre></blockquote><pre>-- <o:p></o:p></pre><pre>** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL <o:p></o:p></pre><pre> <o:p></o:p></pre><pre>/* Matt Perkins<o:p></o:p></pre><pre> Direct 02 8916 8101 Spectrum Networks Ptd. Ltd.<o:p></o:p></pre><pre> Office 1300 133 299 <a href="mailto:matt@spectrum.com.au">matt@spectrum.com.au</a><o:p></o:p></pre><pre> ABN 66 090 112 913 Level 6, 350 George Street Sydney 2000<o:p></o:p></pre><pre> <o:p></o:p></pre><pre>*/<o:p></o:p></pre></div></div></blockquote><pre>-- <o:p></o:p></pre><pre>** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL <o:p></o:p></pre><pre><o:p> </o:p></pre><pre>/* Matt Perkins<o:p></o:p></pre><pre> Direct 02 8916 8101 Spectrum Networks Ptd. Ltd.<o:p></o:p></pre><pre> Office 1300 133 299 <a href="mailto:matt@spectrum.com.au">matt@spectrum.com.au</a><o:p></o:p></pre><pre> ABN 66 090 112 913 Level 6, 350 George Street Sydney 2000<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>*/<o:p></o:p></pre></div></body></html>