Nexus 5000 Non-Routable VDC Mode

While I was working on a Nexus lab, I ran into an interesting issue that had me scratching my head for a while before I managed to figure out what was going on. You gotta love those unintentional troubleshooting exercises! This was the situation:

I had two Nexus 5000 switches with an identical configuration and identical software version, but an interesting difference in behavior. On both switches I had configured a VLAN interface, primarily as a testing point that I could ping to from some other switches. When I actually started pinging to see if the IP addresses on these SVIs were reachable, I noticed that I could ping one of the Nexus 5000 switches, but not the other. While performing the usual Layer 1 and Layer 2 connectivity checks I noticed that the SVI on the Nexus 5000 that I could not ping was down! The relevant configuration on this switch was as follows:

feature interface-vlan
vlan 1,75
interface Vlan75
  no shutdown
  ip address 10.37.37.77/24

However, the interface itself showed the following:

N5K-1# show int vlan 75
Vlan75 is down, line protocol is down
  Hardware is EtherSVI, address is  0005.73c1.9981
  Internet Address is 10.37.37.77/24
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec

The most common causes for this would be either a missing VLAN, or not having any spanning-tree forwarding ports for the VLAN. However, I had already performed my spanning tree checks and I knew that I had some ports in forwarding state for VLAN 75.

I compared the configuration of the misbehaving switch to the configuration on the other Nexus 5000, but it was identical except for the IP addresses. However, on the other switch the SVI had come up without a problem. A key hint about what was going on was shown in the output of the show interface brief command:

N5K-1# show int vlan 75 brief

-------------------------------------------------------------------------------
Interface Secondary VLAN(Type)                    Status Reason
-------------------------------------------------------------------------------
Vlan75    --                                      down   Non-routable VDC mode

Interestingly, this command gave me a reason why the interface was down, which was not shown in the regular show interface command. Apparently, the SVI was down because the switch was in a “non-routable VDC mode”. This still didn’t really get me anywhere, but at least I had a hint now. A bit of googling turned up the following link on the Cisco support forums:

https://supportforums.cisco.com/thread/2131130

Unfortunately, this post was about the Nexus 7000, not the Nexus 5000, and had to do with the lack of M1 linecards to provide the routing function for a VDC. However, it did yield another piece of the puzzle. By adding the management command to the SVI, I managed to get the interface up:

N5K-1(config)# int vlan 75
N5K-1(config-if)# management
N5K-1(config-if)# show int vlan 75 brief

-------------------------------------------------------------------------------
Interface Secondary VLAN(Type)                    Status Reason
-------------------------------------------------------------------------------
Vlan75    --                                      up     --

So I had found the workaround that would allow me to continue with my lab scenario, but I didn’t find this very satisfying. I could still not explain why one of these two Nexus 5000 switches required the obscure management command, which I had never needed before for this type of configuration, while the other worked as expected. So I started digging for differences between the two switches. As mentioned, the NX-OS version was exactly the same and also the hardware was identical. Both switches were 5548Ps without a Layer 3 module or expansion module installed.

Finally I found the difference. Although neither switch had a Layer 3 module, the misbehaving switch actually had a Layer 3 Base Services license (LAN_BASE_SERVICES_PKG) installed:

N5K-1# show license usage
Feature                      Ins  Lic   Status Expiry Date Comments
                                 Count
--------------------------------------------------------------------------------
FCOE_NPV_PKG                  Yes   -   Unused Never       -
FM_SERVER_PKG                 No    -   Unused             -
ENTERPRISE_PKG                Yes   -   Unused Never       -
FC_FEATURES_PKG               Yes   -   Unused Never       -
VMFEX_FEATURE_PKG             No    -   Unused             -
ENHANCED_LAYER2_PKG           Yes   -   Unused Never       -
LAN_BASE_SERVICES_PKG         Yes   -   Unused Never       -
LAN_ENTERPRISE_SERVICES_PKG   Yes   -   Unused Never       -
--------------------------------------------------------------------------------

The other switch did not have the Layer 3 Base Services package installed:

N5K-2(config-vlan)# show license usage
Feature                      Ins  Lic   Status Expiry Date Comments
                                 Count
--------------------------------------------------------------------------------
FCOE_NPV_PKG                  No    -   Unused             -
FM_SERVER_PKG                 No    -   Unused             -
ENTERPRISE_PKG                No    -   Unused             -
FC_FEATURES_PKG               No    -   Unused             Grace 119D 22H
VMFEX_FEATURE_PKG             No    -   Unused             -
ENHANCED_LAYER2_PKG           Yes   -   Unused Never       -
LAN_BASE_SERVICES_PKG         No    -   Unused             -
LAN_ENTERPRISE_SERVICES_PKG   Yes   -   Unused Never       -
--------------------------------------------------------------------------------

So apparently, my misbehaving switch must have had a Layer 3 daughter card  at some point and somebody had installed the Layer 3 Base license for it, but currently the switch did not have the Layer engine installed anymore. So I had a hunch that this might be the cause of my issue. To confirm this hypothesis I uninstalled the license:

N5K-1# clear license MDS20111121084524878.lic
Clearing license MDS20111121084524878.lic:
SERVER this_host ANY
VENDOR cisco
INCREMENT LAN_BASE_SERVICES_PKG cisco 1.0 permanent uncounted \
        VENDOR_STRING=MDS_SWIFTN55-BAS1K9= \
        HOSTID=VDH=SSI14410JQG \
        NOTICE="201111210845248781 \
        " SIGN=70188100AC20

Do you want to continue? (y/n) y
Clearing license ......2012 Oct  2 17:13:20 N5K-p5-1 %LICMGR-1-LOG_LIC_LICENSE_EXPIRED: Evaluation license expired for feature LAN_BASE_SERVICES_PKG.
done
N5K-1# 2012 Oct  2 17:13:20 N5K-1 %PFMA-2-MOD_REMOVE: Module 3 removed (Serial number JAF1440AMME)

Then I removed the management command from the SVI to see what this would do:

N5K-1(config)# int vlan 75
N5K-1(config-if)# no management
N5K-1(config-if)# show int vlan 75 brief

-------------------------------------------------------------------------------
Interface Secondary VLAN(Type)                    Status Reason
-------------------------------------------------------------------------------
Vlan75    --                                      up     --

And as expected, the interface stayed up! Just to be absolutely sure, I reinstalled the license:

N5K-1# install license MDS20111121084524878.lic
Installing license ......Enable Layer 3.
Error: while enabling/disabling L3: btcm, err: enabling/disabling feature is in progress (0x40aa0004)

Please reload the switch if the switch has previously gone through a non-disruptive NX-OS upgrade.
done
N5K-1# show int vlan 75 brief

-------------------------------------------------------------------------------
Interface Secondary VLAN(Type)                    Status Reason
-------------------------------------------------------------------------------
Vlan75    --                                      down   Non-routable VDC mode

And our “Non-routable VDC mode” error has reappeared.

In hindsight, this behavior has some obvious parallels with the Nexus 7000 case that I found on the Cisco Support Community. We have a VDC (in case of the Nexus 5000 this is the only VDC), which is functioning in Layer 3 mode, but which misses the actual forwarding hardware that would allow it to route packets. The lack of forwarding capability then forces the switch to disable its SVIs. I presume that this is done to prevent the switch from becoming a routing black hole if the forwarding hardware would die or get removed (for example when performing OIR on a Nexus 5596UP). To force the interface up, even if there is no routing capability, you have to mark it as a management interface. (I haven’t had time to test this yet, but I would guess that this also exempts the interface from the Layer 3 switching function and causes it to act as a stub host on the segment, rather than a router interface.)

So what can we conclude from this behavior?

Simply installing a Layer 3 Base Services license changes the primary capability of the switch. Instead of behaving like a pure Layer 2 switch, the switch now considers itself to be a Layer 3 switch, even if there is no Layer 3 forwarding engine installed. However, by lack of a forwarding engine it will disable all SVIs to prevent possible traffic black-holing. To force an SVI up in this mode, you can mark the SVI as a management interface through use of the management command.

Although not intended, I got a very interesting troubleshooting exercise thrown into my lab and I had a lot of fun figuring out what caused the SVI on my Nexus 5000 to stay down. It is really a corner case (why would you install a LAN Base license on a switch that doesn’t have a Layer 3 module?), but I still thought it would be worth documenting, just in case somebody bumps into this same error message. And, if I am ever in need of a really devious troubleshooting exercise for one of my Nexus classes, I now have a very interesting issue to throw at my students!

 

 

20 thoughts on “Nexus 5000 Non-Routable VDC Mode

  1. After moving routing onto different routers on our network we spent several hours trying to figure out why our management vlans were not coming up.

    Thank you very much for this post it was instrumental in solving our problem!

  2. Pingback: Cisco Nexus 5000 Series vPC Configuration Example #1 | Paul Gerard Porter

  3. Very interesting, what he says is logical, if the nexus does not have the L3 module, so you need the L3 license?

    Highly good contribution thanks.

    • I am glad you liked the post. So indeed, if you do not have the Layer 3 module, you do not need the LAN Base (= Basic Layer 3) license. And actually, as this issue showed, having the license installed without the module can cause problems.

      Tom

  4. Nailed my question exactly, down to why it was throwing that message. (Switch I didn’t expect to have a license installed on it did!)

    Thanks.

  5. Thank you so much for this post, just ran into the exact same situation ! I believe that Cisco didn’t expected someone to remove a layer3 daughter card back to a layer2 once it is installed… 😉

  6. When I deleted the “LAN_BASE_SERVICES_PKG”,resulting “ENTERPRISE_PKG and FC_FEATURES_PKG went in to unstall mark.

    Before deletion :
    sh license usage
    Feature Ins Lic Status Expiry Date Comments
    Count
    ——————————————————————————–
    FCOE_NPV_PKG No – Unused –
    FM_SERVER_PKG Yes – Unused 25 Dec 2014 –
    ENTERPRISE_PKG Yes – Unused Never –
    FC_FEATURES_PKG Yes – Unused Never –
    VMFEX_FEATURE_PKG No – Unused –
    ENHANCED_LAYER2_PKG No – Unused –
    LAN_BASE_SERVICES_PKG Yes – In use Never –
    LAN_ENTERPRISE_SERVICES_PKG No – Unused –
    ——————————————————————————–

    After Deletion:
    sh license usage
    Feature Ins Lic Status Expiry Date Comments
    Count
    ——————————————————————————–
    FCOE_NPV_PKG No – Unused –
    FM_SERVER_PKG Yes – Unused 25 Dec 2014 –
    ENTERPRISE_PKG No – Unused –
    FC_FEATURES_PKG No – Unused –
    VMFEX_FEATURE_PKG No – Unused –
    ENHANCED_LAYER2_PKG No – Unused –
    LAN_BASE_SERVICES_PKG No – Unused –
    LAN_ENTERPRISE_SERVICES_PKG No – Unused –
    ——————————————————————————–

    Can you share any specific reson and next step i have to do?

    • Hi,

      Sorry I didn’t respond earlier. It seems like these messages ended up in my spam folder.

      Did you maybe have a license bundle on the switch? I have seen this before, where multiple licenses were bundled in a single license file. Because uninstallation of licenses works at the file level, this means that when you uninstall the L3 license you will also uninstall the other licenses that are bundled in the same file.

      If this is the case, you may want to contact TAC and see if there is a way to split the license into multiple files, so you can disable the L3 license without disabling the others.

      Regards,

      Tom

  7. Hi Tom,

    I had the same issue with my both NX55. I don’t have any L3 daughter card installed, but I have LAN_BASE_SERVICE_PKG running in my NX.
    SZANDCSF01# sh license usage
    Feature Ins Lic Status Expiry Date Comments
    Count
    ——————————————————————————–
    FCOE_NPV_PKG No – Unused –
    FM_SERVER_PKG No – Unused –
    ENTERPRISE_PKG Yes – Unused Never –
    FC_FEATURES_PKG Yes – In use Never –
    VMFEX_FEATURE_PKG No – Unused –
    ENHANCED_LAYER2_PKG No – Unused Grace 119D 23H
    LAN_BASE_SERVICES_PKG Yes – In use Never –
    LAN_ENTERPRISE_SERVICES_PKG No – Unused –
    ——————————————————————————–
    But what makes me don’t understand is, from 3 Vlans I created, only 1 Vlan is not UP, the other 2 are UP and I can ping both Vlan.
    SZANDCSF01# sh int vlan 716 brief

    ——————————————————————————-
    Interface Secondary VLAN(Type) Status Reason
    ——————————————————————————-
    Vlan716 — up —
    SZANDCSF01# sh int vlan 717 brief

    ——————————————————————————-
    Interface Secondary VLAN(Type) Status Reason
    ——————————————————————————-
    Vlan717 — up —
    SZANDCSF01# sh int vlan 718 brief

    ——————————————————————————-
    Interface Secondary VLAN(Type) Status Reason
    ——————————————————————————-
    Vlan718 — down VLAN is down

    All Vlans are having same configuration:
    interface Vlan716
    no shutdown
    description VLAN-SVR716
    management
    no ip redirects
    ip address 172.27.16.2/24

    interface Vlan717
    no shutdown
    description VLAN-SVR717
    management
    no ip redirects
    ip address 172.27.17.2/24

    interface Vlan718
    no shutdown
    description VLAN-SVR718
    management
    no ip redirects
    ip address 172.27.18.2/23

    Do you have any advises for me on this?

    Thank you.

    • Hi Pramudia,

      Sorry for not answering earlier. It seems like your message got stuck in my spam folder.

      That is strange. When you do a “show int brief” does it also list “non-routable VDC mode” as the reason for the VLAN interface being down?
      Maybe you are hitting a different issue here…

      And did you also try uninstalling the L3 Base License?

      Regards,

      Tom

  8. Thank you very much for spending the time to share your findings.
    Your tips helped me in my case, also.

    For me the key was to notice the “down : Non-routable VDC mode” message.
    Then by googling I found your post and also a Cisco ticket, that says the same as you, but in a far less explanatory way.

    https://tools.cisco.com/bugsearch/bug/CSCty42451

    I see posts spanning in several years…

    It is possible to buy a Nexus with L2 board and L3 license (L3 base license is free).
    Perhaps that’s the reason that many people have this problem.

    BR,

  9. I ran into the exact same issue! I suspected it was related to the recent application of routing license (LAN_BASE_SERVICES_PKG), which is what I did right before the problem surfaced. I was able to solve it with the management command.

    interface Vlan90
    description Mgmt Interface (NETDEV1 Vlan)
    no shutdown
    management
    !
    N5K# sh license usage
    Feature Ins Lic Status Expiry Date Comments
    Count
    ——————————————————————————–
    FCOE_NPV_PKG No – Unused –
    FM_SERVER_PKG No – Unused –
    ENTERPRISE_PKG No – Unused –
    FC_FEATURES_PKG No – Unused –
    VMFEX_FEATURE_PKG No – Unused –
    ENHANCED_LAYER2_PKG No – Unused –
    NETWORK_SERVICES_PKG No – Unused –
    LAN_BASE_SERVICES_PKG Yes – In use never –
    LAN_ENTERPRISE_SERVICES_PKG No – Unused –
    ——————————————————————————–

Leave a Reply

Your email address will not be published. Required fields are marked *