[AusNOG] Powering a Cisco Nexus 7018
Lincoln Dale
ltd at cisco.com
Tue Apr 12 15:46:05 EST 2011
you could say i have some clue w.r.t. the Nexus switches as i may have had something to do with its development...
first up, some background:
PSUs & power consumption:
the N7K system PSUs are sized around investment protection & worst case scenarios.
by investment protection, what we mean is that we can power/cool in excess of 800W of 'stuff' per slot and the system as a whole can remain operational even with high ambient temperatures.
Cooling-wise, that 18 slot chassis has in excess of 3KW worth of fans for airflow. that doesn't mean it _consumes_ 3KW for the fans, but under the right [wrong] conditions it might.
Module power consumption wise reality is that Cisco does not ship any I/O modules which consume the maximum power budget/slot. and even if we did, modules themselves have variable power consumption depending on traffic levels, port activity, transceiver type, features turned on etc.
PSU breaker sizing:
if you have 6KW PSUs you have 2 x 3KW inputs and you need 20A breakers for these based on australian standards.
if you have 7.5KW PSUs then you have 2 x 4KW inputs and you need 30A breakers for those. (and again for Australian standard reasons, the power cord will be 'fixed' on these).
how many PSUs you need depends on what you have in the system. the system is sized such that you can have PSU failure (lose a PSU) or grid failure (lose 50% of inputs) and the system can still operate.
what this means for all practical purposes is that there is twice as many PSUs in the system than what it actually requires.
you can tell the system your desire PSU redundancy wise (combined/grid-redundant/PSU-redundant) and that will change its calculations on whether it thinks it has sufficient power to power up modules based on the available inputs.
not that it matters in Australia, but you can mix/match 110V/220V AC and/or DC inputs/voltages and have different inputs on different phases, no restrictions.
the system is even smart enough to proportionally load-share across different input types.
the recommendation would be to connect the two inputs on a single PSU to different 'grids', i.e. if you have two grids (call them A/B) then have A/B going to each of PSU1, each of A/B going to PSU2, each of A/B going to PSU3 etc.
this way grid failure or PSU failure at most takes out 50%.
you then size the system with appropriate # of PSUs to ensure it can power the I/O modules installed in a worst-case scenario with that 50% number.
Actual power consumption:
assuming you have them, just do "show environment power" and it will give you both the budgeted worst-case power consumption and what it _actually_ is.
we have power monitoring on the PSU inputs so the system can tell you in real-time what its consuming. newer modules have this on the per-module consumption too.
to give you an idea, here is a system i have in a lab. you can see the difference between 'budgeted' (3430W) and 'actual' (1333W):
~ lincolndale$ ssh ltd at ltd-n7010-1 "show env power"
Power Supply:
Voltage: 50 Volts
Power Actual Total
Supply Model Output Capacity Status
(Watts ) (Watts )
------- ------------------- ----------- ----------- --------------
1 N7K-AC-6.0KW 678 W 3000 W Ok
2 N7K-AC-6.0KW 655 W 3000 W Ok
3 N7K-AC-6.0KW 0 W 0 W Ok
Actual Power
Module Model Draw Allocated Status
(Watts ) (Watts )
------- ------------------- ----------- ----------- --------------
1 N7K-M148GT-11 N/A 400 W Powered-Up
2 N7K-M132XP-12 N/A 750 W Powered-Up
5 N7K-SUP1 N/A 210 W Powered-Up
6 N7K-SUP1 N/A 210 W Powered-Up
Xb1 N7K-C7010-FAB-1 N/A 60 W Powered-Up
Xb2 N7K-C7010-FAB-1 N/A 60 W Powered-Dn
Xb3 N7K-C7010-FAB-1 N/A 60 W Powered-Up
Xb4 N7K-C7010-FAB-1 N/A 60 W Powered-Dn
Xb5 N7K-C7010-FAB-1 N/A 60 W Powered-Dn
fan1 N7K-C7010-FAN-S 76 W 720 W Powered-Up
fan2 N7K-C7010-FAN-S 76 W 720 W Powered-Up
fan3 N7K-C7010-FAN-F 8 W 120 W Powered-Up
fan4 N7K-C7010-FAN-F 8 W 120 W Powered-Up
N/A - Per module power not available
Power Usage Summary:
--------------------
Power Supply redundancy mode (configured) PS-Redundant
Power Supply redundancy mode (operational) Non-Redundant
Total Power Capacity (based on configured mode) 6000 W
Total Power of all Inputs (cumulative) 6000 W
Total Power Output (actual draw) 1333 W
Total Power Allocated (budget) 3430 W
Total Power Available for additional modules 2570 W
~ lincolndale$
while i don't work on Nexus products any more, more than happy to answer any queries you have on them.
cheers,
lincoln.
Lincoln Dale | Cisco Systems | Distinguished Engineer, Data Center Switching Services CTO Architecture & Strategy Team | ltd at cisco.com
On 12/04/2011, at 2:39 PM, Damien Morris wrote:
> Hi list,
>
> Has anyone had experience with these (or similar) brutes of switches..? Specifically, running and distributing power into them.
>
> They have several PSUs per chassis with multiple inputs per PSU, despite not needing anywhere near that much juice unless really fully loaded (I'd imagine).
>
> Do people generally run a couple (2-4 from different feeds) 20-30A circuits per chassis and distribute it themselves? Using what kinda PDU gear?
>
> All feedback/advice/war stories welcome, looking to avoid having to force salespeople onto pedal powered generators.
>
> Thanks,
> Damien.
More information about the AusNOG
mailing list