Sunday, December 20, 2009

CEF - The Final Frontier

Hi Guys,

Wow been a long time since I have done an update: Sorry about that guys I have been concentrating pretty heavily on two things:

1. Ironport Exam (passed it, got my CICSP woo!)2. Working on something called AXL which is basically a SOAP object for CCM that lets you talk directly to the database: Stay tuned for more info in another blog

3. Weighing up if i should finish my final CCSP Exam so i can get my damn CCSP (Its the NAC exam which is annoying because it is quite difficult to get real-world NAC experience)

Anyway, I think it's important I talk about something pretty low-level in Cisco routers called CEF.

CEF stands for Cisco Express forwarding and is a special algorithm that utilizes the Cisco Hardware ASICS for routing, saving CPU cycles.
(I think the algorithm is called a B-Tree algorithm, if you know the name please post in the comments!)

Has my bland description of CEF bored you yet? It shouldnt: CEF is exceptionally important in how a router makes routing decisions. The Common belief is that a show ip route will tell you exactly where a packet is to be routed, this is not strictly true, although the Cisco router does use the routing table, it only copies this information to the CEF table, which you can see for yourself with show ip cef:

PeterCCIE18371#show ip cef
Prefix Next Hop Interface
0.0.0.0/0 attached Virtual-Access2
attached Dialer1
0.0.0.0/32 receive
10.0.0.0/24 attached FastEthernet0/1
10.0.0.0/32 receive
10.0.0.1/32 10.0.0.1 FastEthernet0/1
10.0.0.2/32 10.0.0.2 FastEthernet0/1
10.0.0.3/32 receive

99.999 percent of the time the routing-table and CEF table will match (except in the instance of policy-maps, which is why the CEF table can be extra useful: you can see what routing decisions where made AFTER your policy-map was applied which you can't always do in the routing table)

However, the CEF table is also extremely useful for debugging hard to troubleshoot problems, For example, a customer of mine had a 837 router that could not be pinged or reached by users on an SSLVPN sitting on an ASA in another part of the customers network.

A policy-map was being used for a nonat statement and had not been updated with the addresses of the ASA SSL VPN Users, so I modified that access-list Everything appeared totally OK according to packet-tracer on the ASA and other debugging tools. After going through things a few times I double checked the cef output

Sure enough, CEF had not updated to reflect the changes in the policy-map and still had a cached entry for that particular IP address, turning CEF off and back on again resolved the issue.

So Top Tip readers: when you look at a problem and your POSITIVE everything is configured the way it should be, don't trust the routing table to tell you what decision the router will be making, check the cef table with show ip cef or show ip cef

Now quickly to talk about another interesting issue: Load Balancing.

As you are probably aware, a Cisco Router can load balance outgoing routes to two different hops. This works as long as your two routes have the same administrative distance and same cost or metric in your routing protocol (except in the case of the EIGRP Variance command)

So, assuming your using static routes for some odd reason and have the administrative distance for both of those routes set to the same value your routing table might look a little something like this:

PeterCCIE18371#show ip route 10.10.10.10
Routing entry for 10.10.10.10/32
Known via "static", distance 1, metric 0
Routing Descriptor Blocks:
10.0.0.71
Route metric is 0, traffic share count is 1
* 10.0.0.70
Route metric is 0, traffic share count is 1

Why has 10.0.0.70 got a little * next to it you might ask? I am glad you ask: This is your indication to let you know what hop the router last took for traffic to that destination, if you see this little asterisk moving between your two next-hops your probably load-balancing reasonably effectively.

The question is, what algorithm is the Cisco router using for this load balancing? How can we determine this? Remember the routing table only tells us about potential routing decisions, it is not the definitive authority on how we will route our packets.. Do you remember what is?

Thats Right CEF

So, if we ask CEF how it is load balancing this route?

PeterCCIE18371#show ip cef 10.10.10.10
10.10.10.10/32, version 7386, epoch 0, per-destination sharing
0 packets, 0 bytes
Flow: Origin AS 0, Peer AS 0, mask 32
via 10.0.0.71, 0 dependencies, recursive
traffic share 1
next hop 10.0.0.71, FastEthernet0/1 via 10.0.0.0/24
valid glean adjacency
via 10.0.0.70, 0 dependencies, recursive
traffic share 1
next hop 10.0.0.70, FastEthernet0/1 via 10.0.0.0/24
valid glean adjacency
0 packets, 0 bytes switched through the prefix
tmstats: external 0 packets, 0 bytes
internal 0 packets, 0 bytes



So now we have a bit more information to work with: The router is using per-destination sharing. While fine for most situations it might not work in your particular requirements, for example what if 10.10.10.10 is another router you have a tunnel to? You have two paths to it but because the destination is always going to be 10.10.10.10 the load-balancing algorithm is never going to utilize the second route!

You have two options, you could turn off ip cef and cisco fast switching completely on the interface with the no ip route-cache and no ip cef interface config and global config commands (this may be your only option on certain platforms) OR you could enable per-packet sharing with the following interface commands:

interface FastEthernet0/1
ip address X.X.X.X secondary
ip address 10.0.0.3 255.255.255.0
ip access-group block_others in
ip accounting output-packets
ip load-sharing per-packet
ip nbar protocol-discovery
ip nat inside
duplex auto
speed auto
fair-queue
end

You must enable this command on EVERY interface that is involved in any load sharing, so for example if you had two interfaces you where load-balancing across you must enable load-sharing per-packet on both interfaces. you can confirm with show ip interface

if we return to our CEF entry, we see the following:

PeterCCIE18371#show ip cef 10.10.10.10
10.10.10.10/32, version 7386, epoch 0, per-packet sharing
0 packets, 0 bytes
Flow: Origin AS 0, Peer AS 0, mask 32
via 10.0.0.71, 0 dependencies, recursive
traffic share 1, current path
next hop 10.0.0.71, FastEthernet0/1 via 10.0.0.0/24
valid glean adjacency
via 10.0.0.70, 0 dependencies, recursive
traffic share 1
next hop 10.0.0.70, FastEthernet0/1 via 10.0.0.0/24
valid glean adjacency
0 packets, 0 bytes switched through the prefix
tmstats: external 0 packets, 0 bytes
internal 0 packets, 0 bytes


There you have it folks, CEF and Load-balancing! Until next time.

Pete

No comments:

Post a Comment