Thursday, March 1, 2012

UCS 2.0 Port Channel to Fabric/Port Channel to IOM

Hi Guys

With the introduction of the new Cisco UCS 2.0 Software and Hardware updates we can now enjoy a very useful feature that allows us to Port Channel from the IOM modules to the Fabric Interconnect.

In this article I will explain how to configure the Port Channels between the IOM to the Fabric Interconnects (It is super easy)

(See: for a great summary of all the changes)

Why is this important? You might not know but previously, the way the uplinks worked between the Fabric Interconnect to the IOM Modules was that depending on the number of links you had (1, 2 or 4 where the supported number of uplinks), the bandwidth available between the IOM moduels to the Fabric interconnects was statically pinned between the servers, So for example:

If you only had one Uplink, all the server slots (1 to 8) would share a single uplink (1:8 Oversubscription)
If you had 2 Uplinks, all Odd number server slots would go out one link, all even number would go out the other Uplink.
If you had 4 Uplinks (the maximum possible with the old IOM modules) you would have the bandwidth split between pairs of servers, (e.g. 1 and 3, 2 and 4, 5 and 7, 6 and 8)

Now, this lead to all sorts of clever things, let's say for example you had 4 blades, one of which you wanted to have tons of bandwidth, What you might do is put all your normal blades in the even number slots, and stick your high-bandwidth server in the odd number slot, just to ensure it had all the bandwidth and was not oversubscribed with another server.

If however, you have UCS 2.0, Fabric Interconnects 62XX series, and the new 22XX IOM modules, you can implement a port channel between the fabric interconnect to the IOM, this allows all the servers equal access to the uplinks (and yes, QOS applies)

Here is how to do it

Cable your chassis to your fabric interconnects in the normal way, in our example let us say we went 4 Uplinks to each Fabric Interconnect)

Next,  Go to your equipment Tab, click on the root equipment object, then click on policies
You will find the Discovery policy, set it to how many links you have (In the screenshot ,1 link is shown. This probably would not make a very good Etherchannel! So make sure you set this to however many links your running into the chassis) and importantly, make sure you set the Preference to Port Channel like the screenshot below

Next, you just configure your ports facing the IOM as server ports like you have always done (Make sure you make as many ports server ports as how many links you said there would be as per the screenshot baove, so for example, if you set the above to 4, make sure you specify 4 server ports per UCS fabric interconnect :))

To do this, go to equipment tab, fabric interconnects -> Fabric Interconnect (A and B, you will repeat for each of them) and then fixed modules -> Unconfigured ethernet ports, right click the ports that are cabled between your IOM and your FI and click "Configure as Server Port"

Wow, that was easy! Now let's just verify it worked,
Go to LAN tab -> Internal Lan, click on Internal Fabric A, Port Channels -> Port Channel (number)
You will see the uplinks from the IOM have formed a port channel

Easy as that :)

I hope this helped someone out there, my next blog post will be boot from iSCSI help


  1. The other advantage over 1.4 is the following:
    Lets say you have configured your chassis discovery with 4 links and cabled correctly, and then for some reason one of the cables are cut/disconnected. It would then disconnect the uplinks on 2 blades to that specific FI (hopefully you have vnic failover configured) until the cable is reconnected, or until the policy of the number of uplinks on discovery is changed to 2 and the chassis is re-discovered.

  2. Hi guys,

    So if understand well there is no longer the issue of having half of the blades pinned to one port of the IO module and half on the other if you had only 2 cables on the IO module?

    Before without port channeling you had to reacknowldge the chassis in case of a cable failure in order to resume to the nearing commitment ratio, how does it behave now if you add a new cable or lose one?