When a FabricPath edge switch needs to send a frame to a remote MAC address, it performs a MAC address table lookup and finds an entry of the form SWID.SSID.LID. The SWID represents the switch-ID of the remote FabricPath edge switch, the SSID represents the sub-switch ID (which is only used in vPC+) and the LID represents the outbound port on the remote edge switch. However, the method by which these LIDs are derived doesn’t seem to be very well documented and this had been bugging me for a while. So I decided to dig in and see if I could find out a bit more about the way LIDs are used on the Nexus switches.
I found some hints in the excellent Cisco Live Presentation BRKDCT-3313 – Fabric Path operation and troubleshooting (2013 London). Inside this presentation I found a somewhat cryptic sentence that states “for N7K the LID is the port index of the ingress interface, for N5K LID most of the time will be 0”. Let’s see what we can make of that.
The acronym LID stands for “Local ID” and, as the name implies, it has local significance to the switch that a particular MAC address resides on. As such, it is up to the implementation to determine how to derive a unique LID to represent its ports. Apparently, the Nexus 5000 and Nexus 7000 engineering teams did not talk to each other to agree on some consistent method of assigning the LIDs, but each created their own platform-specific implementation.
For the N5K I couldn’t really find more than that the LID will usually be 0, but there may be some exceptions. For the N7K, the Cisco Live presentation states that the LID maps to the “port index” of the ingress interface. (The interface represented by the LID is an ingress interface from the perspective of the edge switch that inserts the LID into the outer source address. For the switch sending to the MAC address it represents the egress port at the destination edge switch).
So I decided to get into the lab and see if I could find some commands that would help me establish the relation between the LID and the outbound interface on the edge switch. I created a very simple FabricPath network and performed a couple of pings to generate some MAC address table entries.
Let’s have a look at a specific entry in the MAC address table of a Nexus 7000:
N7K-1-pod5# show mac address-table dynamic vlan 100 Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vPC Peer-Link, (T) - True, (F) - False VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID ---------+-----------------+--------+---------+------+----+------------------ * 100 0005.73e9.8c81 dynamic 960 F F Eth3/15 100 0005.73e9.fcfc dynamic 960 F F 16.0.14 100 00c0.dd18.6ce0 dynamic 420 F F 16.0.14 100 00c0.dd18.6ce1 dynamic 0 F F 16.0.14 * 100 00c0.dd18.6e08 dynamic 0 F F Eth3/15 * 100 00c0.dd18.6e09 dynamic 0 F F Eth3/15
So for example, let’s zoom in on the MAC address 0005.73e9.fcfc. According the table, frames for this destination should be sent to SWID.SSID.LID “16.0.14”. From the SWID part, we can see that the MAC address resides on the switch with ID “16”. To find the corresponding switch hostname we can use the following command:
System-ID Primary Secondary Reachable Bcast-Priority Ftag-Root Capable Hostname MT-0 b414.89dc.7a44 16 [C] 0[C] Yes 64 [S] Y N7K-2-pod6 f025.72a8.bf44* 15 [C] 0[C] Yes 64 [S] Y N7K-1-pod5
So we jump to switch N7K-2-pod6 and perform another MAC address table lookup:
N7K-2-pod6# show mac address-table address 0005.73e9.fcfc Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vPC Peer-Link, (T) - True, (F) - False VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID ---------+-----------------+--------+---------+------+----+------------------ * 100 0005.73e9.fcfc dynamic 450 F F Eth3/15
Now we know that the outbound interface for the MAC address on the destination edge switch is Ethernet 3/15. So how can we map the LID “14” to this interface?
The Cisco Live presentation states that the LID corresponds to the “port index” for the interface. So how can we find the port index? The port index is an internal identifier for the interface, also referred to as the LTL and there are some show commands to determine these LTLs. For example, if we wanted to know the LTL for interface E3/15, we could issue the following command:
N7K-2-pod6# show system internal pixm info interface e 3/15 LTL TYPE LTL ======================== PHY_PORT 0xe FLOOD_W_FPOE 0x8031 FLOOD_W_FPOE 0x8035
Here we find that the LTL for the interface is 0xe, which equals 14 in decimal. This shows that the LID is actually the decimal representation of the LTL. (FabricPath switch-IDs, subswitch-IDs and Local IDs are represented in decimal by default).
This lookup can also be performed in reverse. If we take the LID and convert it to its hexadecimal representation of 0xe, we can find the corresponding interface as follows:
N7K-2-pod6# show system internal pixm info ltl 0xe Member info ------------------ Type LTL --------------------------------- PHY_PORT Eth3/15 FLOOD_W_FPOE 0x8035 FLOOD_W_FPOE 0x8031
So through use of these two commands, we can map a FabricPath LID to an interface and vice versa on a Nexus 7000.
Thank you for this very good documentation. I have known what the SWID and the SSID was for a long time but not the LID.
I actually doesn’t see why you would need to translate the LID to a physical port since you can SSH to the Nexus device that owns the corresponding MAC-address and simply use the command #show mac address-table address xxxx.xxxx.xxxx to see the local port that the MAC-address is assigned to.
CCNA, CCNP Routing
I agree that there’s no real need to be able to make this mapping between LID and port. As you said, you can always just do a MAC lookup on the remote device.
I just didn’t like that I was seeing the LID in the FabricPath tables and I couldn’t easily see how it was derived.
So there is no real practical application to this, other than getting a better understanding of the FabricPath implementation.
Thank you for the post.
I have a question. You wrote
“We can see that the MAC address resides on the switch with ID “16″. To find the corresponding switch hostname we can use the following command:”
Could you please post the command?
My apologies. It looks like I missed a line while copying and pasting.
The command is “show fabricpath isis switch-id”.
Dear thanks so much for this excellent deep inside…. Imho the LID is a key to understand the data-plane operations but i am not 100% sure…
I think that without LID the “leaf” switch should “tag” the classical ethernet frame with the source vlan tag and encapsulate 802.1q frame in the fabricpath header… and in this case the receiving host should be on the same source vlan id…..
But with LID the fabric path core become vlan unaware because the port id is encoded in fp frame … and indeed receiving leaf does’not make “lookup mac address” but only to “route” the eth native frame on the LID port… is this correct interpretation ?
I do not think that the 802.1q tag really matters here. The decision to send it tagged or untagged to the final destination depends on the CE port being a trunk or an access port.
As far as I understand, the main function of the LID is to optimize the lookup on the last hop switch, because a MAC lookup is not required, but the frame can be switched to the CE port based on the LID.
However, in my opinion this should not bypass the VLAN check. If the VLAN tag in the encapsulated frame does not match the access VLAN or trunk allowed VLAN list of the destination CE port, the frame should be dropped.
Thanks for the elaborate post on the Farbicpath LID. I could not find the command “show system internal pixm info” on Nexus 5500 series switches, to find out the interface correponding to the LID. Is the CLI different in case of Nexus 5K. If so, could you please post it?
Thanks and Regards,
The “show system internal pix info” command is indeed specific to the Nexus 7000 and I do not know the equivalent command on the Nexus 5000. However, I also think it is not that relevant. As far as I know the Nexus 5000 FabricPath implementation sets LID to 0, so there is no inherent mapping to a port on this platform.