Monday, June 17, 2013

CCIE DC: Advanced Fabric Path and vPC+

Hi Guys

Many things scare a newbie CCIE DC candidate, and one of those things that scared me was VPC+, VPC+ and Enhanced VPC?? i said to myself, How could it possibly get any more complicated?

Fortunately VPC+ (as well as eVPC which I have covered previously) is quite simple, here is a sample configuration

So first of all, you configure your fabricpath interfaces and VLAN like you always have:

vlan 10
  mode fabricpath
 


interface Ethernet1/1
  switchport mode fabricpath


Configure your VPC as normal with a special diffirence:


vpc domain 1
  peer-keepalive destination 10.0.8.212
  fabricpath switch-id 100!

The fabricpath switch-id command manually specifies a switch-id to be used and shared amongst the devices, this MUST MATCH on both vPC Peers

next you configure your peer link, and you must configure it as a fabricpath port:


interface Ethernet1/9 -- Member interface
  switchport mode fabricpath
  channel-group 1 mode active









interface port-channel1
  switchport mode fabricpath
  spanning-tree port type network
  speed 10000
  vpc peer-link
 

Peer Link shown above.

IMPORTANT NOTE: 
This also means that you can't use non-FabricPath VLAN's for your vPC member ports, they will fail to come up

Finally here is the VPC Member port:



interface port-channel2
  switchport access vlan 10
  spanning-tree port type edge
  speed 1000
  vpc 2


With this config everything shows as it should

N5K-p6-1(config-if)# show vpc
Legend:
                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                   : 1
vPC+ switch id                  : 100
Peer status                     : peer adjacency formed ok
vPC keep-alive status           : peer is alive
vPC fabricpath status           : peer is reachable through fabricpath
Configuration consistency status: success
Per-vlan consistency status     : success
Type-2 consistency status       : success
vPC role                        : primary
Number of vPCs configured       : 1
Peer Gateway                    : Disabled
Dual-active excluded VLANs      : -
Graceful Consistency Check      : Enabled

vPC Peer-link status
---------------------------------------------------------------------
id   Port   Status Active vlans
--   ----   ------ --------------------------------------------------
1    Po1    up     10

vPC status
---------------------------------------------------------------------------
id     Port        Status Consistency Reason       Active vlans vPC+ Attrib
--     ----------  ------ ----------- ------       ------------ -----------
2      Po2         up     success     success      10           DF: Partial


If you try and add a non-fabricpath VLAN, like in our example VLAN 1 below, like with an FCoE VLAN it will allow it but it won't be etherchanneled, if you don't have a FabricPath enabled VLAN at all...


interface port-channel2
 switchport access vlan 1 - (Not enabled for FabricPath this VLAN)
  spanning-tree port type edge trunk
  speed 1000
  vpc 2
!



The Port channel will come up on the primary vPC, but the VPC will not come up:





N5K-p6-1(config-if)# show vpc
Legend:
                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                   : 1
vPC+ switch id                  : 100
Peer status                     : peer adjacency formed ok
vPC keep-alive status           : peer is alive
vPC fabricpath status           : peer is reachable through fabricpath
Configuration consistency status: success
Per-vlan consistency status     : success
Type-2 consistency status       : success
vPC role                        : primary
Number of vPCs configured       : 1
Peer Gateway                    : Disabled
Dual-active excluded VLANs      : -
Graceful Consistency Check      : Enabled

vPC Peer-link status
---------------------------------------------------------------------
id   Port   Status Active vlans
--   ----   ------ --------------------------------------------------
1    Po1    up     10

vPC status
---------------------------------------------------------------------------
id     Port        Status Consistency Reason       Active vlans vPC+ Attrib
--     ----------  ------ ----------- ------       ------------ -----------
2      Po2         down   success     success      -            DF: Partial




This is because no non-fabricpath VLAN will go over that peerlink, because the peerlink is now a fabricpath port! (and CE VLAN's DON'T go over fabricpath ports!)





Fabric Path + MTU:

Did you know that Fabric Path itself, if you looked at it with a wire capturing utility, is NOT ethernet, it's ethernet encapsulated inside fabricpath, fabricpath adds an overhead of approx 15 bytes, but takes into account all of that for you, what it does NOT take into account however is jumbo MTU's, you need to specifically configure them if you want them/need them.


Here's proof:

N5K-p6-1(config-if)# show fabricpath isis interface  brief
Fabricpath IS-IS domain: default
Interface    Type  Idx State        Circuit   MTU  Metric  Priority  Adjs/AdjsUp
--------------------------------------------------------------------------------
port-channel1 P2P   1     Up/Ready   0x01/L1   1500 20      64          1/1
Ethernet1/1  P2P   2     Up/Ready   0x01/L1   1500 40      64          1/1
Ethernet1/2  P2P   3     Up/Ready   0x01/L1   1500 40      64          1/1
Ethernet1/3  P2P   4     Up/Ready   0x01/L1   1500 40      64          1/1
Ethernet1/4  P2P   5     Up/Ready   0x01/L1   1500 40      64          1/1



More proof? Check this out:






Here is all my relevant config:

 interface Vlan10
  no shutdown
  mtu 9216
  ip address 10.1.1.1/24


The adapter on the server has been set correctly also.

But if i configure it correctly on both ends

N7K-6-1(config-if)# show fabricpath isis interface brief
Fabricpath IS-IS domain: default
Interface    Type  Idx State        Circuit   MTU  Metric  Priority  Adjs/AdjsUp
--------------------------------------------------------------------------------
Ethernet1/21 P2P   1     Up/Ready   0x01/L1   1500 40      64          1/1
Ethernet1/22 P2P   7     Up/Ready   0x01/L1   1500 40      64          1/1
Ethernet1/23 P2P   3     Up/Ready   0x01/L1   1500 40      64          1/1
Ethernet1/24 P2P   8     Up/Ready   0x01/L1   1500 40      64          1/1
Ethernet1/25 P2P   4     Up/Ready   0x01/L1   1500 40      64          1/1
Ethernet1/26 P2P   5     Up/Ready   0x01/L1   1500 40      64          1/1
Ethernet1/27 P2P   2     Up/Ready   0x01/L1   1500 40      64          1/1
Ethernet1/28 P2P   6     Up/Ready   0x01/L1   1500 40      64          1/1

N7K-6-1(config-if)# int eth1/21 - 28
N7K-6-1(config-if-range)# mtu ?
1500-9216  Enter MTU

N7K-6-1(config-if-range)# mtu 9216


 And:


N5K-p6-2(config-if)# show fabricpath isis interface brief
Fabricpath IS-IS domain: default
Interface    Type  Idx State        Circuit   MTU  Metric  Priority  Adjs/AdjsUp
--------------------------------------------------------------------------------
port-channel1 P2P   1     Up/Ready   0x01/L1   1500 20      64          1/1
Ethernet1/5  P2P   2     Up/Ready   0x01/L1   1500 40      64          1/1
Ethernet1/6  P2P   3     Up/Ready   0x01/L1   1500 40      64          1/1
Ethernet1/7  P2P   4     Up/Ready   0x01/L1   1500 40      64          1/1
Ethernet1/8  P2P   5     Up/Ready   0x01/L1   1500 40      64          1/1


So i changed the QoS policy on the N5k:



N5K-p6-1(config-pmap-nq)# show run | sect policy-map
policy-map type network-qos jumbomtu
  class type network-qos class-fcoe
    pause no-drop
    mtu 2158
  class type network-qos class-default
    mtu 9216
    multicast-optimize

N5K-p6-1(config)# system qos
N5K-p6-1(config-sys-qos)# service-policy type network-qos jumbomtu

Yet this did not seem to affect the output from the show fabricpath isis interface, and i couldn't set the MTU directly on the interface

however after doing both these steps, i could ping across at the maximum MTU of 8972


C:\Documents and Settings\student>ping 10.1.1.1 -l 8972 -f
Pinging 10.1.1.1 with 8972 bytes of data:

Reply from 10.1.1.1: bytes=8972 time=1ms TTL=255
Reply from 10.1.1.1: bytes=8972 time=1ms TTL=255
Reply from 10.1.1.1: bytes=8972 time=1ms TTL=255

Ping statistics for 10.1.1.1:



So interestingly, you MUST set this on the fabricpath interface on your 7k:

Check out this ping when i took it off:

N7K-6-1# conf t
Enter configuration commands, one per line.  End with CNTL/Z.
N7K-6-1(config)# int eth1/21 - 28
N7K-6-1(config-if-range)# no mtu



C:\Documents and Settings\student>ping 10.1.1.1 -l 8972 -f
Pinging 10.1.1.1 with 8972 bytes of data:


Request Timed out.


When i enabled it again:

C:\Documents and Settings\student>ping 10.1.1.1 -l 8972 -f

Pinging 10.1.1.1 with 8972 bytes of data:

Reply from 10.1.1.1: bytes=8972 time=1ms TTL=255
Reply from 10.1.1.1: bytes=8972 time=1ms TTL=255


So these appear to be the steps you need to take, but I will get into advanced MTU a bit later, just know that if you want to enable it with fabricpath, you gotta specify it on a per-interface basis on your 7k, but on a 5k it just needs to be in your class-default QoS Policy (or whatever traffic class your traffic is falling under)


4 comments:

  1. Peter, I am wondeing why you use "spanning-tree port type network" int the fabricpath enabled vpc peer-link.

    ReplyDelete
    Replies
    1. It's there by default after configuring the Peer Link, but you could easily remove it and it would have no effect.

      When you configure the port-channel with "vpc peer-link", it will automatically enable BA on the link ("spanning-tree port type network"), but once you configure "switchport mode fabricpath" on the Peer-Link to enable vPC+, BA is rendered useless as, like everyone knows, FabricPath is NOT Ethernet and does NOT us STP.

      Delete
  2. it's there by "default" but most people verify that it is there. It provides bridge assurance.

    ReplyDelete