<div dir="ltr"><div>Has anyone heard any details about the planned UPS maintenance that's scheduled for tomorrow at VDC-MEL03? Only info I've seen is what's in the last outage update, which is just that UPS-A is being replaced, nothing on whether this is service impacting or anything like that.</div><br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Thanks,<div>Andrew Welch</div><div><a href="mailto:mail@andrewwelch.net" target="_blank">mail@andrewwelch.net</a></div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 14 Mar 2023 at 13:15, Jared Hirst | Servers Australia <<a href="mailto:jared.hirst@serversaustralia.com.au">jared.hirst@serversaustralia.com.au</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div>Good times back then Joel! It was a UPS issue back there too if I remember, and the A and B feeds were all fed into the one UPS….</div><div><br></div><div><span>A </span><span>tonne of people shifted within that week, for me it was the way that it was palmed off, as you say, others might have issues but they own those issues and rectify them…</span></div><div><br></div><div><br></div><div></div><br><div><table cellspacing="0" cellpadding="0" border="0" style="font-family:Helvetica,"Open Sans",Arial,sans-serif"> <tbody> <tr> <td style="padding-left:10px" valign="top"><br> <img src="https://info.serversaustralia.com.au/hubfs/Brand-2018/logo-font-sau.gif" width="215"><span style="font-size:16px;color:rgb(7,105,186)"><br>Jared Hirst</span><br> <span style="font-size:14px;color:rgb(7,105,186)">Chief Executive Officer</span><br> <span style="font-size:12px;color:rgb(7,105,186)">P: </span><span style="font-size:12px;color:rgb(68,68,68)">(02) 8115 8801 </span><br><span style="font-size:12px;color:rgb(68,68,68);line-height:16px">4 Amy Close, Wyong, NSW 2259</span><span style="font-size:12px;color:rgb(68,68,68)"><br>Need assistance? We are here 24/7 +61 2 8115 8888 </span></td> </tr> </tbody></table></div></div><img src="https://app.frontapp.com/api/1/noauth/companies/servers_australia_pty_ltd/seen/msg_sanx3yd/han_62fwax/e0d3086e.gif" alt="Sent from Front" aria-hidden="true" style="width: 1px; height: 1px;"><br><blockquote type="cite">On March 14, 2023 at 12:26 PM GMT+11 <a href="mailto:joel@zonenetworks.com.au" rel="noopener noreferrer" target="_blank">joel@zonenetworks.com.au</a> wrote:<br><br><div id="m_1647106079795142179fae_sanx3yd-3n6ef0"><div>
<div>
</div>
<div lang="EN-AU">
<div>
<p><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">In EQX defence Sy1 didn’t go dark… it was multiple (3-5) rows of rack that lost power in 2009.<span></span></span></p>
<p><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span> </span></span></p>
<p><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">We were one of the customers heavily impacted in 2009 but understood the reason why it happened and EQX fixed it all up.<span></span></span></p>
<p><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span> </span></span></p>
<p><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Vocus Doody st different story and now melb<span></span></span></p>
<p><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span> </span></span></p>
<p><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">We pulled out of Doody after the drama we faced there.<span></span></span></p>
<p><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span> </span></span></p>
<p><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Joel.<span></span></span></p>
<p><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span> </span></span></p>
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0cm 0cm">
<p><b><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif">From:</span></b><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif"> AusNOG <<a href="mailto:ausnog-bounces@lists.ausnog.net" rel="noopener noreferrer" target="_blank">ausnog-bounces@lists.ausnog.net</a>>
<b>On Behalf Of </b>Nathan Brookfield<br>
<b>Sent:</b> Tuesday, 14 March 2023 11:54 AM<br>
<b>To:</b> Christopher Hawker <<a href="mailto:chris@thesysadmin.dev" rel="noopener noreferrer" target="_blank">chris@thesysadmin.dev</a>><br>
<b>Cc:</b> <a href="mailto:ausnog@lists.ausnog.net" rel="noopener noreferrer" target="_blank">ausnog@lists.ausnog.net</a><br>
<b>Subject:</b> Re: [AusNOG] Vocus Datacentre outage.<span></span></span></p>
</div>
</div>
<p><span> </span></p>
<p>I knew I didn’t remember it for a reason <span style="font-family:"Segoe UI Symbol",sans-serif">
😂</span> <span></span></p>
<div>
<p><span> </span></p>
</div>
<div>
<div>
<div>
<table border="0" cellspacing="0" cellpadding="0" width="0" style="width:225pt;background:rgb(223,223,225);border-collapse:collapse">
<tbody>
<tr>
<td style="padding:0cm">
<table border="0" cellspacing="0" cellpadding="0" width="0" style="width:225pt;background:rgb(223,223,225)">
<tbody>
<tr>
<td style="padding:6pt 0cm">
<div style="margin-left:12pt;margin-right:12pt;max-width:100%;overflow:hidden">
<p><b><span style="font-size:9pt;font-family:Helvetica,sans-serif"><a href="https://lists.ausnog.net/pipermail/ausnog/2009-July/003239.html" rel="noopener noreferrer" target="_blank"><span style="color:black;text-decoration:none">[AusNOG] Equinix Outage</span></a><span></span></span></b></p>
<p><b><span style="font-size:8.5pt;font-family:Helvetica,sans-serif"><a href="https://lists.ausnog.net/pipermail/ausnog/2009-July/003239.html" rel="noopener noreferrer" target="_blank"><span style="color:rgb(162,162,169);text-decoration:none">lists.ausnog.net</span></a><span></span></span></b></p>
</div>
</td>
<td width="60" style="width:27pt;padding:0cm 9pt 0cm 0cm">
<p><span style="font-family:Helvetica,sans-serif"><a href="https://lists.ausnog.net/pipermail/ausnog/2009-July/003239.html" rel="noopener noreferrer" target="_blank"><span style="font-family:Calibri,sans-serif"><img border="0" width="48" height="48" style="width: 0.5in; height: 0.5in;" id="m_1647106079795142179_x0000_i1025"></span></a><span></span></span></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<p><span> </span></p>
<div>
<p>On 14 Mar 2023, at 10:50, Christopher Hawker <<a href="mailto:chris@thesysadmin.dev" rel="noopener noreferrer" target="_blank">chris@thesysadmin.dev</a>> wrote:<span></span></p>
</div>
<div>
<p><span> </span></p>
</div>
<div>
<p><span style="font-family:Tahoma,sans-serif"></span> <span>
</span></p>
<div>
<p style="background:white"><span style="font-family:Calibri,sans-serif;color:black">SY1 apparently experienced a full power outage back in July 2009, according to AusNOG Archives. Some reported full outages, some partial. Was not a DC fault
though, looks like a much larger grid issue where a fault at the Bayswater Power Station in the Hunter Valley (which affected generators in QLD and VIC) caused about 1000MW to be dropped from the grid, resulting in rolling blackouts across QLD, NSW, TAS, SA
and VIC.<span></span></span></p>
</div>
<div>
<p style="background:white"><span style="font-family:Calibri,sans-serif;color:black"><span> </span></span></p>
</div>
<div>
<p style="background:white"><span style="font-family:Calibri,sans-serif;color:black"><a href="https://lists.ausnog.net/pipermail/ausnog/2009-July/003222.html" id="m_1647106079795142179LPlnk264879" rel="noopener noreferrer" target="_blank">https://lists.ausnog.net/pipermail/ausnog/2009-July/003222.html</a><span></span></span></p>
</div>
<div>
<p style="background:white"><span style="font-family:Calibri,sans-serif;color:black"><span> </span></span></p>
</div>
<div>
<p style="background:white"><span style="font-family:Calibri,sans-serif;color:black">CH.<span></span></span></p>
</div>
<div align="center" style="text-align:center">
<hr width="98%" align="center">
</div>
<div id="m_1647106079795142179divRplyFwdMsg">
<p><b><span style="font-size:11pt;font-family:Calibri,sans-serif;color:black">From:</span></b><span style="font-size:11pt;font-family:Calibri,sans-serif;color:black"> AusNOG <<a href="mailto:ausnog-bounces@lists.ausnog.net" rel="noopener noreferrer" target="_blank">ausnog-bounces@lists.ausnog.net</a>>
on behalf of Nathan Brookfield <<a href="mailto:Nathan.Brookfield@iperium.com.au" rel="noopener noreferrer" target="_blank">Nathan.Brookfield@iperium.com.au</a>><br>
<b>Sent:</b> Tuesday, March 14, 2023 11:40 AM<br>
<b>To:</b> Matt Perkins <<a href="mailto:matt@spectrum.com.au" rel="noopener noreferrer" target="_blank">matt@spectrum.com.au</a>>;
<a href="mailto:ausnog@lists.ausnog.net" rel="noopener noreferrer" target="_blank">ausnog@lists.ausnog.net</a> <<a href="mailto:ausnog@lists.ausnog.net" rel="noopener noreferrer" target="_blank">ausnog@lists.ausnog.net</a>><br>
<b>Subject:</b> Re: [AusNOG] Vocus Datacentre outage.</span> <span></span></p>
<div>
<p> <span></span></p>
</div>
</div>
<div>
<div>
<p>Have 100% had A and B fail separately at SY1 but never both, thankfully!!!<span></span></p>
<p> <span></span></p>
<p>Could have been part of the facility though so don’t quote me.<span></span></p>
<p> <span></span></p>
<p>UK on the other hand, fail!<span></span></p>
<p> <span></span></p>
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0cm 0cm">
<p><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Matt Perkins <<a href="mailto:matt@spectrum.com.au" rel="noopener noreferrer" target="_blank">matt@spectrum.com.au</a>>
<br>
<b>Sent:</b> Tuesday, 14 March 2023 11:30 AM<br>
<b>To:</b> Nathan Brookfield <<a href="mailto:Nathan.Brookfield@iperium.com.au" rel="noopener noreferrer" target="_blank">Nathan.Brookfield@iperium.com.au</a>>;
<a href="mailto:ausnog@lists.ausnog.net" rel="noopener noreferrer" target="_blank">ausnog@lists.ausnog.net</a><br>
<b>Subject:</b> Re: [AusNOG] Vocus Datacentre outage.</span><span></span></p>
</div>
</div>
<p> <span></span></p>
<p>SY1 had A or B fail a few years back if my memory serves me correctly. But I was referring London and Manchester which both had total outages in recent years.
<span></span></p>
<p>Matt.<span></span></p>
<p> <span></span></p>
<div>
<p>On 14/3/2023 11:26 am, Nathan Brookfield wrote:<span></span></p>
</div>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<p>I’ve never had an A+B power rail failure at Equinix in 18 years I’ve been in there facilities, TOUCH WOOD!!!<span></span></p>
<p> <span></span></p>
<div>
<p style="margin-bottom:12pt"><b><span style="font-size:9pt;font-family:Helvetica,sans-serif;color:rgb(0,35,58)">Nathan Brookfield | VK2NAB<br>
</span></b><span style="font-size:9pt;font-family:Helvetica,sans-serif;color:rgb(0,35,58)">General Manager</span><span style="font-size:10pt;font-family:Helvetica,sans-serif"><br>
<br>
</span><b><span style="font-size:9pt;font-family:Helvetica,sans-serif;color:rgb(255,84,0)">p</span></b><span style="font-size:9pt;font-family:Helvetica,sans-serif;color:rgb(0,35,58)">: 1300 592 330 |
</span><b><span style="font-size:9pt;font-family:Helvetica,sans-serif;color:rgb(255,84,0)">m</span></b><span style="font-size:9pt;font-family:Helvetica,sans-serif;color:rgb(0,35,58)">: 0412 266 008</span><span style="font-size:10pt;font-family:Helvetica,sans-serif"><br>
</span><b><span style="font-size:9pt;font-family:Helvetica,sans-serif;color:rgb(255,84,0)">e</span></b><span style="font-size:9pt;font-family:Helvetica,sans-serif;color:rgb(0,35,58)">:
<a href="mailto:Nathan.brookfield@iperium.com.au" rel="noopener noreferrer" target="_blank">Nathan.brookfield@iperium.com.au</a> | </span><b><span style="font-size:9pt;font-family:Helvetica,sans-serif;color:rgb(255,84,0)">w</span></b><span style="font-size:10pt;font-family:Helvetica,sans-serif;color:rgb(0,35,58)">: <a href="http://iperium.com.au/" rel="noopener noreferrer" target="_blank">iperium.com.au</a></span><span style="font-family:Helvetica,sans-serif;color:rgb(0,35,58)"><br>
<br>
</span><span style="font-size:10pt;font-family:Helvetica,sans-serif;color:rgb(0,35,58)">Suite 4.02, 189 Kent Street Sydney NSW 2000</span><span></span></p>
<div>
<p><span style="font-size:9pt;font-family:Arial,sans-serif;color:rgb(70,70,70)"><image001.png><span></span></span></p>
</div>
<p><br>
<b><span style="font-family:Helvetica,sans-serif;color:rgb(0,35,58)">Your Telco Team</span></b>
<span></span></p>
</div>
<p style="margin-bottom:12pt"><span style="font-size:9pt;color:darkgray">DISCLAIMER: This document is intended solely for the named addressee. This electronic communication, which includes any files or attachments thereto, contains proprietary
or confidential information and may be privileged and otherwise protected under copyright or other applicable intellectual property laws. The use, disclosure, copying or distribution of any of the information contained in this document, by any person other
than the addressee, is strictly prohibited. If you received this document in error, please contact the sender immediately and delete all the material from any computer. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery
to you. Any views or opinions presented are solely those of the author and do not necessarily represent those of Iperium. <br>
<br>
WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. Iperium accepts no liability for any damage caused by any virus transmitted by this email.</span><span></span></p>
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0cm 0cm">
<p><b><span lang="EN-US">From:</span></b><span lang="EN-US"> AusNOG
<a href="mailto:ausnog-bounces@lists.ausnog.net" rel="noopener noreferrer" target="_blank"><ausnog-bounces@lists.ausnog.net></a>
<b>On Behalf Of </b>Matt Perkins<br>
<b>Sent:</b> Tuesday, 14 March 2023 11:18 AM<br>
<b>To:</b> <a href="mailto:ausnog@lists.ausnog.net" rel="noopener noreferrer" target="_blank">ausnog@lists.ausnog.net</a><br>
<b>Subject:</b> Re: [AusNOG] Vocus Datacentre outage.</span><span></span></p>
</div>
</div>
<p> <span></span></p>
<p>Yea because Eqinix has never had a power outage. :) <span></span></p>
<p>I think it's a little premature to judge at the moment. Doesn't sound good based on the current info but we have all had bad days im sure.
<span></span></p>
<p>Matt<span></span></p>
<p> <span></span></p>
<div>
<p>On 14/3/2023 11:06 am, Bradley Amm wrote:<span></span></p>
</div>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<p style="margin-bottom:12pt">Friends like let friends put gear in a Vocus dtacenter. Only NextDC or Equinix :) <span></span></p>
<div>
<p>Sent from my iPhone<span></span></p>
</div>
<div>
<p style="margin-bottom:12pt"><br>
<br>
<span></span></p>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<p style="margin-bottom:12pt">On 14 Mar 2023, at 7:08 am, DaZZa
<a href="mailto:dazzagibbs@gmail.com" rel="noopener noreferrer" target="_blank"><dazzagibbs@gmail.com></a> wrote:<span></span></p>
</blockquote>
</div>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<div>
<p> <span></span></p>
<div>
<div>
<p>Doesn't say a whole lot about Vocus' data center design methodology, nor their periodic contingency testing.<span></span></p>
</div>
<div>
<p> <span></span></p>
</div>
<div>
<p>They're apparently replacing the entire UPS on Friday - guess we can all expect more outages then when they stuff that up.<span></span></p>
</div>
<div>
<p> <span></span></p>
</div>
<div>
<p>D<span></span></p>
</div>
</div>
<p> <span></span></p>
<div>
<div>
<p>On Mon, 13 Mar 2023 at 18:30, Chad Kelly <<a href="mailto:chad@cpkws.com.au" rel="noopener noreferrer" target="_blank">chad@cpkws.com.au</a>> wrote:<span></span></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt">
<div>
<div>
<div>
<p><span lang="EN-US">Thought I would create a new thread on this, as the issue wasn’t related to AussieBB.
</span><span></span></p>
<p><span lang="EN-US">It is looking like this Melbourne outage is similar to the Sydney outage back in 2018, in that it appears to have been yet again caused by an issue with their UPS systems yet again.</span><span></span></p>
<p><span lang="EN-US">I’ll leave this hear <a href="https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/" rel="noopener noreferrer" target="_blank">
https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/</a> and this article on today’s outage as well.
</span><span></span></p>
<p><span lang="EN-US"><a href="https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987" rel="noopener noreferrer" target="_blank">https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987</a></span><span></span></p>
<p><span lang="EN-US"> </span><span></span></p>
<p><span lang="EN-US">Regards Chad. </span><span></span></p>
<p><span lang="EN-US"> </span><span></span></p>
<p><span lang="EN-US"> </span><span></span></p>
<p><span lang="EN-US"> </span><span></span></p>
<p>Chad Kelly <span></span></p>
<p>Manager <span></span></p>
<p>CPK Web Services <span></span></p>
<p>Phone 03 52730246<span></span></p>
<p>Web <a href="https://www.cpkws.com.au/" rel="noopener noreferrer" target="_blank">https://www.cpkws.com.au</a><span></span></p>
<div>
<p><image001.jpg><span></span></p>
</div>
<p><span lang="EN-US"> </span><span></span></p>
</div>
</div>
<p>_______________________________________________<br>
AusNOG mailing list<br>
<a href="mailto:AusNOG@lists.ausnog.net" rel="noopener noreferrer" target="_blank">AusNOG@lists.ausnog.net</a><br>
<a href="https://lists.ausnog.net/mailman/listinfo/ausnog" rel="noopener noreferrer" target="_blank">https://lists.ausnog.net/mailman/listinfo/ausnog</a><span></span></p>
</div>
</blockquote>
</div>
<p><br>
<br>
<span>-- </span><span></span></p>
<div>
<p>veg·e·tar·i·an:<br>
Ancient tribal slang for the village idiot who can't hunt, fish or ride<span></span></p>
</div>
<p>_______________________________________________<br>
AusNOG mailing list<br>
<a href="mailto:AusNOG@lists.ausnog.net" rel="noopener noreferrer" target="_blank">AusNOG@lists.ausnog.net</a><br>
<a href="https://lists.ausnog.net/mailman/listinfo/ausnog" rel="noopener noreferrer" target="_blank">https://lists.ausnog.net/mailman/listinfo/ausnog</a><span></span></p>
</div>
</blockquote>
<p style="margin-bottom:12pt"><br>
<br>
<span></span></p>
<pre>_______________________________________________<span></span></pre>
<pre>AusNOG mailing list<span></span></pre>
<pre><a href="mailto:AusNOG@lists.ausnog.net" rel="noopener noreferrer" target="_blank">AusNOG@lists.ausnog.net</a><span></span></pre>
<pre><a href="https://lists.ausnog.net/mailman/listinfo/ausnog" rel="noopener noreferrer" target="_blank">https://lists.ausnog.net/mailman/listinfo/ausnog</a><span></span></pre>
</blockquote>
<pre>-- <span></span></pre>
<pre>** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL <span></span></pre>
<pre> <span></span></pre>
<pre>/* Matt Perkins<span></span></pre>
<pre> Direct 02 8916 8101 Spectrum Networks Ptd. Ltd.<span></span></pre>
<pre> Office 1300 133 299 <a href="mailto:matt@spectrum.com.au" rel="noopener noreferrer" target="_blank">matt@spectrum.com.au</a><span></span></pre>
<pre> ABN 66 090 112 913 Level 6, 350 George Street Sydney 2000<span></span></pre>
<pre> <span></span></pre>
<pre>*/<span></span></pre>
</blockquote>
<pre>-- <span></span></pre>
<pre>** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL <span></span></pre>
<pre> <span></span></pre>
<pre>/* Matt Perkins<span></span></pre>
<pre> Direct 02 8916 8101 Spectrum Networks Ptd. Ltd.<span></span></pre>
<pre> Office 1300 133 299 <a href="mailto:matt@spectrum.com.au" rel="noopener noreferrer" target="_blank">matt@spectrum.com.au</a><span></span></pre>
<pre> ABN 66 090 112 913 Level 6, 350 George Street Sydney 2000<span></span></pre>
<pre> <span></span></pre>
<pre>*/<span></span></pre>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div></blockquote>_______________________________________________<br>
AusNOG mailing list<br>
<a href="mailto:AusNOG@lists.ausnog.net" target="_blank">AusNOG@lists.ausnog.net</a><br>
<a href="https://lists.ausnog.net/mailman/listinfo/ausnog" rel="noreferrer" target="_blank">https://lists.ausnog.net/mailman/listinfo/ausnog</a><br>
</blockquote></div>