When a FabricPath edge switch needs to send a frame to a remote MAC address, it performs a MAC address table lookup and finds an entry of the form SWID.SSID.LID. The SWID represents the switch-ID of the remote FabricPath edge switch, the SSID represents the sub-switch ID (which is only used in vPC+) and the LID represents the outbound port on the remote edge switch. However, the method by which these LIDs are derived doesn’t seem to be very well documented and this had been bugging me for a while. So I decided to dig in and see if I could find out a bit more about the way LIDs are used on the Nexus switches.
I found some hints in the excellent Cisco Live Presentation BRKDCT-3313 – Fabric Path operation and troubleshooting (2013 London). Inside this presentation I found a somewhat cryptic sentence that states “for N7K the LID is the port index of the ingress interface, for N5K LID most of the time will be 0”. Let’s see what we can make of that.
The acronym LID stands for “Local ID” and, as the name implies, it has local significance to the switch that a particular MAC address resides on. As such, it is up to the implementation to determine how to derive a unique LID to represent its ports. Apparently, the Nexus 5000 and Nexus 7000 engineering teams did not talk to each other to agree on some consistent method of assigning the LIDs, but each created their own platform-specific implementation.
For the N5K I couldn’t really find more than that the LID will usually be 0, but there may be some exceptions. For the N7K, the Cisco Live presentation states that the LID maps to the “port index” of the ingress interface. (The interface represented by the LID is an ingress interface from the perspective of the edge switch that inserts the LID into the outer source address. For the switch sending to the MAC address it represents the egress port at the destination edge switch).
So I decided to get into the lab and see if I could find some commands that would help me establish the relation between the LID and the outbound interface on the edge switch. I created a very simple FabricPath network and performed a couple of pings to generate some MAC address table entries.
Let’s have a look at a specific entry in the MAC address table of a Nexus 7000:
N7K-1-pod5# show mac address-table dynamic vlan 100 Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vPC Peer-Link, (T) - True, (F) - False VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID ---------+-----------------+--------+---------+------+----+------------------ * 100 0005.73e9.8c81 dynamic 960 F F Eth3/15 100 0005.73e9.fcfc dynamic 960 F F 16.0.14 100 00c0.dd18.6ce0 dynamic 420 F F 16.0.14 100 00c0.dd18.6ce1 dynamic 0 F F 16.0.14 * 100 00c0.dd18.6e08 dynamic 0 F F Eth3/15 * 100 00c0.dd18.6e09 dynamic 0 F F Eth3/15
So for example, let’s zoom in on the MAC address 0005.73e9.fcfc. According the table, frames for this destination should be sent to SWID.SSID.LID “16.0.14”. From the SWID part, we can see that the MAC address resides on the switch with ID “16”. To find the corresponding switch hostname we can use the following command:
System-ID Primary Secondary Reachable Bcast-Priority Ftag-Root Capable Hostname MT-0 b414.89dc.7a44 16 [C] 0[C] Yes 64 [S] Y N7K-2-pod6 f025.72a8.bf44* 15 [C] 0[C] Yes 64 [S] Y N7K-1-pod5
So we jump to switch N7K-2-pod6 and perform another MAC address table lookup:
N7K-2-pod6# show mac address-table address 0005.73e9.fcfc Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vPC Peer-Link, (T) - True, (F) - False VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID ---------+-----------------+--------+---------+------+----+------------------ * 100 0005.73e9.fcfc dynamic 450 F F Eth3/15
Now we know that the outbound interface for the MAC address on the destination edge switch is Ethernet 3/15. So how can we map the LID “14” to this interface?
The Cisco Live presentation states that the LID corresponds to the “port index” for the interface. So how can we find the port index? The port index is an internal identifier for the interface, also referred to as the LTL and there are some show commands to determine these LTLs. For example, if we wanted to know the LTL for interface E3/15, we could issue the following command:
N7K-2-pod6# show system internal pixm info interface e 3/15 LTL TYPE LTL ======================== PHY_PORT 0xe FLOOD_W_FPOE 0x8031 FLOOD_W_FPOE 0x8035
Here we find that the LTL for the interface is 0xe, which equals 14 in decimal. This shows that the LID is actually the decimal representation of the LTL. (FabricPath switch-IDs, subswitch-IDs and Local IDs are represented in decimal by default).
This lookup can also be performed in reverse. If we take the LID and convert it to its hexadecimal representation of 0xe, we can find the corresponding interface as follows:
N7K-2-pod6# show system internal pixm info ltl 0xe Member info ------------------ Type LTL --------------------------------- PHY_PORT Eth3/15 FLOOD_W_FPOE 0x8035 FLOOD_W_FPOE 0x8031
So through use of these two commands, we can map a FabricPath LID to an interface and vice versa on a Nexus 7000.