[AusNOG] NextDC Melbourne - Scheduled Power Maintenance - 15th August

Thu Aug 10 10:13:33 EST 2017

If you have single fed equipment in your rack that needs planning before it
can be shut down, do you not run a totally forseeable and preventable risk
then should that one feed trip?

This is what ATS are for.

Sam

On Thursday, 10 August 2017, Shaun McGuane <shaun at rackcentral.com> wrote:

> HI Simon,
>
>
>
> Thank you for your input to the list.
>
>
>
> While you may have provided notifications well in advance, your schedule
> provided has been changed multiple times and this creates confusion.
>
> Without asking for further clarification last night I would never have
> received the following updated schedule from Craig Armstrong today.
>
>
>
> Data Hall 2 – B feed (west/blue PDU side) Tuesday 15th Aug, A feed has
> been pushed back to later date around November, you will receive official
> notice once new date confirmed.
>
> Data Hall 3 –A and B feed has been pushed back to later date around
> October/November, you will receive official notice once new date confirmed.
>
> Data Hall 4 –A and B feed has been pushed back to later date around
> October/November, you will receive official notice once new date confirmed.
>
>
>
> From the latest update today, It appears that most of the works are being
> postponed for the time being
>
>
>
> Not all devices within racks support 2 feeds and planning is involved with
> these devices, which is why it is crucial to receive accurate information
> about any pending outages or upgrades
> to the NextDC DataCentres which affect services.
>
>
>
> Additionally, I have not received these updates via the normal NextDC
> E-Mail communication method, so I am sure other customers will be wondering
> whats happening too.
>
>
>
> Happy to have a conversation offline about this feel free to give me a
> call on my mobile.
>
>
>
> Regards
>
> Shaun
>
>
>
>
>
>
>
> *From:* AusNOG [mailto:ausnog-bounces at lists.ausnog.net
> <javascript:_e(%7B%7D,'cvml','ausnog-bounces at lists.ausnog.net');>] *On
> Behalf Of *Simon Cooper
> *Sent:* Thursday, 10 August 2017 8:39 AM
> *To:* ausnog at lists.ausnog.net
> <javascript:_e(%7B%7D,'cvml','ausnog at lists.ausnog.net');>
> *Subject:* Re: [AusNOG] NextDC Melbourne - Scheduled Power Maintenance -
> 15th August
>
>
>
> Thanks guys, appreciate the feedback.
>
>
>
> At NEXTDC we work very hard to keep critical infrastructure maintenance
> that impacts racks or the environment within the data hall to an absolute
> minimum.
>
>
>
> However, some works are required to happen from time to time by law and
> one of those activities is a de-energisation and inspection of each of our
> major switchboards.  This is all about ensuring that the electrical system
> is safe, and is an unavoidable piece of work. Rightly so.
>
>
>
> Every rack in a NEXTDC data centre has 2N power feeds offered to it, and
> we highly recommend that they are used for every device in the rack.  This
> occasional maintenance is one of the many reasons we provide this
> capability.  Any device using the 2N supplies (aka A & B feeds) will simply
> see its power swing from one feed to the other.  Any rack that is drawing
> no more that its contracted power level will have no issue when all devices
> move from A to B or vice versa.
>
>
>
> Our notifications were made well in advance, as we want to give customers
> time to consider the management of these works and we don't assume that
> every rack is perfectly deployed:  We are here 24/7 to assist customers
> with rack inspections or specific circuit down tests or maybe a controlled
> move prior to the day (e.g. for devices that are single fed).
>
>
>
> Also, as we work though those discussions with customers we understand the
> timing in relation to their businesses, and we work very hard to coordinate
> schedules despite each switch board supporting many different customers and
> customer instances across the data centre.  Occasionally that means we have
> to make a difficult decisions which may result in schedule changes.  When
> that does happen, we do not compromise our notification periods.
>
>
>
> I'll be happy to discuss these works with anyone - probably best offline
> or on the phone - but trust me when I say we are doing our absolute best to
> keep your data centre operating safely and reliably for many years to come.
>
>
>
> Thanks,
>
> Simon
>
> COO, NEXTDC
>
>
>
> -----Original Message-----
> *From:* nathan at nightsys.net
> <javascript:_e(%7B%7D,'cvml','nathan at nightsys.net');>
> *Sent:* Thu, 10 Aug 2017 05:39:02 +1000
> *To:* shaun at rackcentral.com
> <javascript:_e(%7B%7D,'cvml','shaun at rackcentral.com');>
> *Subject:* Re: [AusNOG] NextDC Melbourne - Scheduled Power Maintenance -
> 15th August
>
> This seems reasonable? As long as no single Data Hall has both A and B
> feeds removed simultaneously, that seems normal. Doing B on Hall 2 and A on
> Hall 3 on the same day does seem slightly adventurous, but not entirely
> unreasonable (mostly in terms of if something goes bad in both, staff get
> stretched thin remediating issues).
>
>
>
> A and B feeds are designed to provide redundancy to equipment, allowing
> one feed to fail and systems to continue on the other. Anything mission
> critical should be able to draw from either feed as required (dual PSU),
> and if there is a power deficit on one feed (causing circuits to trip),
> that would imply the circuits are incorrectly sized?
>
>
>
> In addition, anything mission critical with a single power supply would be
> on an ATS you would assume, or a pair of equivalent devices installed with
> one on each rail?
>
>
>
> Things break in datacentres, planned maintenance happens sometimes, better
> than no maintenance at all :) The fact they are notifying you is better
> than not, and it being scheduled is probably still better than unscheduled
> :) (easier to ensure sufficient staffing to keep a close eye on things)
>
>
>
> Nathan.
>
>
>
> On Wed, Aug 9, 2017 at 3:43 PM, Shaun McGuane <shaun at rackcentral.com
> <javascript:_e(%7B%7D,'cvml','shaun at rackcentral.com');>> wrote:
>
> Here is the entire scheduled maintenance windows for all the halls we
> occupy.
>
>
>
> DataHall 2
>
> A Feed - Tuesday 8th August 2017 07:00 UTC +10 until 16:00 UTC +10.
> B Feed - Tuesday 15th August 2017 07:00 UTC +10 until 16:00 UTC +10.
>
>
>
> DataHall 3
>
> A Feed - Tuesday 15th August 2017 07:00 UTC +10 until 16:00 UTC +10
> B Feed - Monday 14th August 2017 07:00 UTC +10 until 16:00 UTC +10
>
>
>
> DataHall 4
>
> B Feed - Tuesday 15th August 2017 07:00 UTC +10 until 16:00 UTC +10.
>
>
>
> ·         So looks like no power on 3 data halls B feeds on the same day.
>
> ·         Data Hall 2 was scheduled for A rail on the 8th of August but
> has been re-scheduled with no new date of works.
>
> ·         Data Hall 3 was scheduled for A rail on the
>
>
>
> Problem is that I have 3x separate dates on 3 separate e-mails for Data
> Hall 3 and I have requested a revised clarification
>
> Just now on the above as there has already been re-schedules for Data Hall
> 2 and 3.
>
>
>
> This feels like it’s going sideways and it has not even started.
>
>
>
> Regards
>
> Shaun
>
>
>
>
>
> *From:* AusNOG [mailto:ausnog-bounces at lists.ausnog.net
> <javascript:_e(%7B%7D,'cvml','ausnog-bounces at lists.ausnog.net');>] *On
> Behalf Of *Murat Sener
> *Sent:* Wednesday, 9 August 2017 3:20 PM
> *To:* Matthew VK3EVL <hitman at itglowz.com
> <javascript:_e(%7B%7D,'cvml','hitman at itglowz.com');>>;
> ausnog at lists.ausnog.net
> <javascript:_e(%7B%7D,'cvml','ausnog at lists.ausnog.net');>
>
>
> *Subject:* Re: [AusNOG] NextDC Melbourne - Scheduled Power Maintenance -
> 15th August
>
>
>
> We have been told only Feed B in Data hall 2 will be effected as per
> original schedule and Feed A is yet to be rescheduled.
>
>
>
>
>
>
>
> *Murat Sener*     Datacentre Manager
>
> *PREMIER* Technology Solutions
> *P. *1300 76 76 48    *F. *1300 76 76 49    *M. *0450 604 350
>
>
>
> *How am I doing? Let us know!*
> SIMPLY CLICK ON A FACE TO GIVE FEEDBACK
>
> *GREAT*
>
> *AVERAGE*
>
> *BAD*
>
>
> <https://www.crewhu.com/survey.html?crewHuID=qm0Hxbzstp&employeeID=rLeIz2SkIa&type=signat&rating=5>
>
>
> <https://www.crewhu.com/survey.html?crewHuID=qm0Hxbzstp&employeeID=rLeIz2SkIa&type=signat&rating=0>
>
>
> <https://www.crewhu.com/survey.html?crewHuID=qm0Hxbzstp&employeeID=rLeIz2SkIa&type=signat&rating=-5>
>
>
>
> *From:* AusNOG [mailto:ausnog-bounces at lists.ausnog.net
> <javascript:_e(%7B%7D,'cvml','ausnog-bounces at lists.ausnog.net');>] *On
> Behalf Of *Matthew VK3EVL
> *Sent:* Wednesday, 9 August 2017 3:13 PM
> *To:* ausnog at lists.ausnog.net
> <javascript:_e(%7B%7D,'cvml','ausnog at lists.ausnog.net');>
> *Subject:* Re: [AusNOG] NextDC Melbourne - Scheduled Power Maintenance -
> 15th August
>
>
>
> What about both feeds happening on the same day at the same time?
>
>
>
> On 9/08/2017 2:55 PM, Shaun McGuane wrote:
>
> HI Damien,
>
>
>
> My apologies that’s correct – but isn’t anyone concerned about the works
> being completed during business hours?
>
> What about increased loading of the active rail once one of them is
> switched off for these works?
>
>
>
> Regards
>
> Shaun
>
>
>
>
>
> *From:* Damien Gardner Jnr [mailto:rendrag at rendrag.net
> <javascript:_e(%7B%7D,'cvml','rendrag at rendrag.net');>]
> *Sent:* Wednesday, 9 August 2017 2:39 PM
> *To:* <ausnog at lists.ausnog.net>
> <javascript:_e(%7B%7D,'cvml','ausnog at lists.ausnog.net');>
> <ausnog at lists.ausnog.net>
> <javascript:_e(%7B%7D,'cvml','ausnog at lists.ausnog.net');>; Shaun McGuane
> <shaun at rackcentral.com>
> <javascript:_e(%7B%7D,'cvml','shaun at rackcentral.com');>
> *Subject:* Re: [AusNOG] NextDC Melbourne - Scheduled Power Maintenance -
> 15th August
>
>
>
> It says 7am -> 4pm in what you quoted? (UTC+10 is AEST)
>
>
>
> On Wed, 9 Aug 2017 at 2:36 pm, Shaun McGuane <shaun at rackcentral.com
> <javascript:_e(%7B%7D,'cvml','shaun at rackcentral.com');>> wrote:
>
> HI Noggers,
>
>
>
> Has anyone seen this scheduled maintenance notification happening next
> week?
>
> They maybe upgrading feeds or boards throughout the datacentre – has
> everyone seen this notification of works?
>
> It appears to be happening starting 5PM through to 2AM AEST for all
> Datahalls on the same date.
>
>
>
>
>
> *WHAT:*
> NEXTDC M1 Melbourne data centre, with the assistance of specialist
> contractors, will conduct offline preventative electrical maintenance work
> on high-voltage and low-voltage main isolators. This will affect A and B
> feeds to the racks in Data Hall 2/3/4 on the 15th of August.
>
> *WHEN:*
> A Feed - Tuesday 15th August 2017 07:00 UTC +10 until 16:00 UTC +10.
> B Feed - Tuesday 15th August 2017 07:00 UTC +10 until 16:00 UTC +10.
>
> *HOW AM I AFFECTED?*
> Due to the nature of the work a single feed will be removed as per the
> dates and times above. We recommend that customers balance the power
> between both power rails to minimise impact during this period. If you have
> single feed devices we recommend you:
>
>
>
> Regards
>
> Shaun
>
>
>
>
>
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> <javascript:_e(%7B%7D,'cvml','AusNOG at lists.ausnog.net');>
> http://lists.ausnog.net/mailman/listinfo/ausnog
>
> --
>
> Damien Gardner Jnr
> VK2TDG. Dip EE. GradIEAust
> rendrag at rendrag.net <javascript:_e(%7B%7D,'cvml','rendrag at rendrag.net');>
> -  http://www.rendrag.net/
> --
> We rode on the winds of the rising storm,
>  We ran to the sounds of thunder.
> We danced among the lightning bolts,
>  and tore the world asunder
>
>
>
> _______________________________________________
>
> AusNOG mailing list
>
> AusNOG at lists.ausnog.net <javascript:_e(%7B%7D,'cvml','AusNOG at lists.ausnog.net');>
>
> http://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
>
> MESSAGE PROTECTED BY *PREMIER TECHNOLOGY SOLUTIONS*
> <http://www.premiertech.com.au> - POWERED BY *MAILGUARD*
> <http://www.mailguard.com.au/mg>.
> Report this message as spam
> <https://console.mailguard.com.au/ras/1RqQHQIE5J/6DWUZjiMDnlts7JYyCLNB3/0>
>
>
>
>
> MESSAGE PROTECTED BY *PREMIER TECHNOLOGY SOLUTIONS*
> <http://www.premiertech.com.au> - POWERED BY *MAILGUARD*
> <http://www.mailguard.com.au/mg>.
>
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> <javascript:_e(%7B%7D,'cvml','AusNOG at lists.ausnog.net');>
> http://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20170810/dde2c775/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 3738 bytes
Desc: not available
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20170810/dde2c775/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 3778 bytes
Desc: not available
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20170810/dde2c775/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 4714 bytes
Desc: not available
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20170810/dde2c775/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 3771 bytes
Desc: not available
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20170810/dde2c775/attachment-0007.png>