[AusNOG] DC undervoltage issues

[James Braunegg]

Dear Joshua

With reference to the power authorities contacting power consumers they will call, email and SMS anyway possible for you to "shed" load off the grid as they need the capacity, if you don't reduce load they charge you a super high price for the period as agreed within a contract.

I believe they also give typically 24 hours' notice so you know it's going to occur.

Kindest Regards

James Braunegg
P:  1300 769 972  |  M:  0488 997 207 |  D:  (03) 9751 7616
E:   james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>  |  ABN:  12 109 977 666
W:  www.micron21.com/ddos-protection<http://www.micron21.com/ddos-protection>   T: @micron21


[Description: Description: Description: Description: M21.jpg]
This message is intended for the addressee named above. It may contain privileged or confidential information. If you are not the intended recipient of this message you must not use, copy, distribute or disclose it to anyone other than the addressee. If you have received this message in error please return the message to the sender by replying to it and then delete the message from your computer.

From: Joshua D'Alton [mailto:joshua at railgun.com.au]
Sent: Wednesday, January 15, 2014 11:18 PM
To: James Braunegg
Cc: AusNOG at lists.ausnog.net
Subject: Re: [AusNOG] DC undervoltage issues

Yep, will be in my email tomorrow. Understand about 2N thanks.

SMART on HDDs don't show much increase, then again it is cool isle so I wouldn't expect much increase. Understand about cheep power and generators (as an aside Bev didn't include details, how does this work practically? Does the energy provider call up the DC and say oh hey, as per agreement go onto generators for X hours thanks, your next bill will be credited ABC ?).

I forgot to mention both DCs are operated by the same company, and both are rural, not in areas with fires at the moment, but I haven't traced electricity supply to see if transmission is.

Thanks.

On Wed, Jan 15, 2014 at 11:04 PM, James Braunegg <james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>> wrote:
Dear Joshua

My first question back to the datacentre would be are they running the UPS's in double conversion mode and what brand / make / model is the UPS technology, without this information it would be very hard to tell what the issue is.

If the power infrastructure is true 2N no system should ever be above 45% to 48% maximum at full load (assuming no failure on any component) so being a hot day should never push things "over the edge"

I doubt the cooling system is UPS'ed and if it was it would be running it would be more than likely running of different UPS not the same UPS for IT load. Typically mechanical cooling are backed up by diesel generation but not UPS power as you can be without cooling for a few seconds whilst diesel generators start.

If your monitoring temperature in the rack have you noticed any increase in temperatures during the day ?

You might also find they might be running on generator power (you can get very cheap power if you agree to stop using mains power in periods of high demand on the grid) however again in double conversion mode you should not see any differences in power supply coming to your rack.

What is very strange is your seeing the same issue across to physical different sites, what brand / model UPS's are you using within your rack ?

Kindest Regards


James Braunegg
P:  1300 769 972  |  M:  0488 997 207 |  D:  (03) 9751 7616
E:   james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>  |  ABN:  12 109 977 666<tel:12%20109%20977%20666>
W:  www.micron21.com/ddos-protection<http://www.micron21.com/ddos-protection>   T: @micron21


[Description: Description: Description: Description: M21.jpg]
This message is intended for the addressee named above. It may contain privileged or confidential information. If you are not the intended recipient of this message you must not use, copy, distribute or disclose it to anyone other than the addressee. If you have received this message in error please return the message to the sender by replying to it and then delete the message from your computer.

From: Joshua D'Alton [mailto:joshua at railgun.com.au<mailto:joshua at railgun.com.au>]
Sent: Wednesday, January 15, 2014 10:47 PM
To: James Braunegg
Cc: AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
Subject: Re: [AusNOG] DC undervoltage issues

Thanks for the answers, mostly what I surmised but good to have some other opinions.

James, so formal!  216V I saw briefly, its since increased to around 225 somewhat stable now. Across a single phase. UPS doesn't have freq information in the GUI, I'd have to wait for the weekly report, but I wouldn't imagine any freq changes. Power factor is also missing from the GUI. One DC says 100% isolated 2N from supplier to rack feeds, the other don't have the docs with me but IIRC N+1. Nothing upstream, the feeds come into the racks into a powerbar on either side of the rack, which then the UPS are plugged into, and then the servers into the UPSs.

Both DCs are somewhat remote, I suspect that is partly responsible. Cheers.

On Wed, Jan 15, 2014 at 10:33 PM, James Braunegg <james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>> wrote:
Dear Joshua

Thanks for your post, and questions regarding Datacentre voltages.

What type of voltage drops are you seeing and is this across a single phase or three phase circuit ?

Have you notice the Hz frequency change, and are you monitoring power factor ?

What is upstream from the power within your rack ?

Whilst you might have A and B power feeds it  could be supplied via a STS (static transfer switch) via a N+1 (x) UPS deployment or is it a 2N UPS setup where both feed A and feed B are 100% isolated.

Most if not all double conversion battery based UPS's will have phase voltage monitoring and below "x" threshold on any incoming phase from its upstream power source will transfer load to say batteries to remove any issues with zero interruption.

Upstream from the UPS infrastructure will be ATSs which also will have phase voltage monitoring and below "x" threshold on any phase from the incoming supply will trigger the ATS to obtain power from its alternative source typically diesel generation power.

As for power problems within Victoria I have not seen any issues with our mains supply its been perfect, nor noticed any issues at our pop sites.

Kindest Regards


James Braunegg
P:  1300 769 972  |  M:  0488 997 207 |  D:  (03) 9751 7616
E:   james.braunegg at micron21.com<mailto:james.braunegg at micron21.com>  |  ABN:  12 109 977 666<tel:12%20109%20977%20666>
W:  www.micron21.com/ip-transit<http://www.micron21.com/ip-transit>   T: @micron21


[Description: Description: Description: Description: M21.jpg]
This message is intended for the addressee named above. It may contain privileged or confidential information. If you are not the intended recipient of this message you must not use, copy, distribute or disclose it to anyone other than the addressee. If you have received this message in error please return the message to the sender by replying to it and then delete the message from your computer.

From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net<mailto:ausnog-bounces at lists.ausnog.net>] On Behalf Of Joshua D'Alton
Sent: Wednesday, January 15, 2014 10:15 PM

To: AusNOG at lists.ausnog.net<mailto:AusNOG at lists.ausnog.net>
Subject: [AusNOG] DC undervoltage issues

Evening noggers, hope you're all surviving the heat, especially in VIC!

Speaking of, in 2 DCs were seeing borderline undervoltage on our UPSs (yes we run them even with A+B power because sensitive equipment), I've read that there are power issues in VIC at the moment, but I thought they were limited to residential zones (with rolling black/brownouts)?

Anyone else seeing this, or have comments?

As a side question, does anyone (perhaps Bevan?) know if DCs will manually switch to genny/backup power in this sort of scenario, if so details would be nice.

(with regards to that, I looked at one DC contract and it doesn't mention anything about the specifics of power delivered, just that it is A+B with X battery backup and Y generator capacity, and Z SLA, doesn't mention what sorts of voltages would break said SLA...)

Anyhoo, cheers!

PS, also seeing strange intermittent issues with Telstra and Pacnet connecting to local google and others, not sure why the power quality would affect that, but..


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20140115/07ace9ec/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 2683 bytes
Desc: image001.jpg
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20140115/07ace9ec/attachment.jpg>