Cisco Switch Troubleshooting Basics and Steps listPosted on: April 12, 2020, by : Admin
Configuring a switch is usually an smooth task, but when things go wrong you need to know what to do. Indeed, in this article, we jump-start you to interchange troubleshooting on Cisco gadgets. With this step-by-step guide you will study the troubleshooting commands, and a way to use them. Moreover, you’ll learn how to find gadgets in a network, which include where they’re connected, IP and MAC address. It is time to start!
This article comes with a Cisco Packet Tracer lab. To gain the maximum from what you research, you ought to strive the lab while analyzing the article. This way, we will guide you through the transfer troubleshooting steps and fixes.
This lab leverages the topology from the previous article about VLANs configuration. In fact, the majority of issues you are going to have with switches involve VLANs. However, we made some changes to enhance switch troubleshooting. First, we used a Layer 3 switch as “Top Switch”. This switch can perform routing, and allows inter-VLAN communication (that was not possible previously).
This is the topology we are going to work in. In the cloud, we have three switches, but we cannot see how they interconnect devices. We need to find out with troubleshooting.
After that, we hidden all the switches. This way, we cannot simply see where the cable goes. Instead, we must check the configuration on the switches, doing real switch troubleshooting. We will have to jump on the core device (Top Switch L3) and then telnet into other switches.
It can be a hardware fault, someone plugging a PC in the wrong port, or anything else you need to fix. Tickets are raised by users complaining that something do no work, then you need to identify the problem and solve it. For this article, you have three tickets to solve:
- Users in the Sales office are complaining they cannot print
- A guy from the Engineering department complains he has no network access
- A Junior IT Support technician reported strange alerts on RightSwitch when connected via console
To access the peripheral switches (LeftSwitch, CenterSwitch and RightSwitch) contained in the cloud, refer to the following table.
Access credential Device IP address VTY password Enable secret TopSwitch 10.80.0.1 – ictshore Left Switch 10.80.0.2 ictshore ictshore CenterSwitch 10.80.0.3 ictshore ictshore Righ tSwitch 10.80.0.4 ictshore ictshore List of steps
Since this is a troubleshooting article, we cannot define a specific steps list that will work 100% of the times. However, the following list is the one that will get you to the problem faster. Moreover, we arranged it to be optimized for switch troubleshooting.
- Define the problem
- Find device(s) having issues
- Check VLANs configuration
- Verify trunk ports configuration
- Check access ports configuration
- Troubleshoot client issues
The first step is going to be “Define the problem” every time. However, based on what the problem really is, later steps may be followed in a different order. We are going to follow these steps for both tickets, but before we do it we should talk about troubleshooting approaches.
Troubleshooting Approaches and Tips
Some people thing troubleshooting is like an art. Instead, troubleshooting is a simple task that can be written in form of procedure. Troubleshooting can be stressful, and you might end up performing the same checks again and again. This is frustrating, as you convince yourself that there is no reason for the problem, and it shouldn’t be happening. Nonetheless, it is happening. You need to have a plan. We can talk a lot about troubleshooting approaches, but for now we will give you just a few tips.
- First, state the problem clearly, talking with the user if necessary.
- Use the divide-and-conquer approach. This mean you start by pinging between the two devices that cannot communicate. If the ping is successful, check for application problems at upper OSI layers. Instead, if the ping fails, do a traceroute to find where in the network the problem is. Then, check data-link layer and physical connections. This approach is opposed to the bottom-up approach (starting from the cabling) and the top-down approach (starting from the application). With this approach, you start in the middle of the OSI layer.
- Write down everything you do, so you won’t be repeating steps. This is useful for complex issues or infrastructures.
- If you still cannot resolve the issue, don’t panic. Google is your friend, check for users having similar issues.
These concepts are fundamentals. They will help you a lot in switch troubleshooting. We are going to see how right below when working on the tickets.
We have over 15,628 DOWNLOAD of this E-Book on our site from different individual in IT Field and Discipline. Discover how we can help you too.
Fill the form below and Get this E-Book for FREE.
Define The Problem
The first ticket we are going to face is for the PCs in the Sales department that cannot print. This definition is way far from a punctual problem statement. After calling users, we clarify that they cannot access printers, servers and file share on the network. Based on that, we know that it is unlikely to be a print problem. Both users have the same problem, but the two PCs can communicate with one another.
It’s time for you to gather technical details. You ask the users to open the command prompt (from the Start or with Win+R,then cmd for Windows users) and type ipconfig /all. We need to gather four items: IP address, default gateway, subnet mask and Physical address (mac address). The two devices having problem are Laptop0 and Laptop1, so here is an example of Laptop 0 output.
This is the output from Laptptop0
After talking with the user, here is what we have learnt.
- User 1 is using a laptop with IP 10.100.2.10 and subnet mask of 255.255.255.0. Its default gateway is 10.100.20.1 and the MAC address is 00-D0-FF-66-C6-A1.
- User 2 is also using a laptop, with IP 10.100.2.11 and subnet mask of 255.255.255.0. Its default gateway is still 10.100.20.1 and the MAC address is 00-E0-8F-E1-5D-8B.
From that, we know that IP configuration on the two laptops is correct. This is likely to be a network issue, and we need to find where in the network the issue is.
Gathering More Information
When we face this kind of problem, we need to find out where the isolation is. In other words, we know that these two PCs are somehow isolated, but we need to know if they can reach at least their gateway, or if there is a problem with L2 segmentation. So, we jump on the Top Switch L3, which is the default gateway, and we try to ping these two IP addresses. Both pings will fail.
Both laptops are not reachable from their default gateway (Top Switch L3).
Verify ARP cache
Since pings are failing, and we are in the same subnet, we need to verify if data-link layer is working as expected. To verify data-link layer, we start by issuing show ip arp just after those pings. With this command, we are checking if our switch was able to resolve the IP addresses to MAC addresses.
This is the content of the ARP table.
The command show ip arp presents you the content of the ARP table. In other words, you can find all the bindings between IP addresses and MAC addresses the switch know. It may contain hundreds, or even thousands of entries in a real-world switch performing routing. This command presents you several columns, here’s their meaning.
- Protocol– Internet stands for IP, identifies the protocol that triggered the ARP request/response process
- Address– IP address
- Age– Time in minutes since the entry is in the table
- Hardware Addr– MAC address associated to the IP address in the same row
- Type– ARP encapsulation type, generally “ARPA”, way out of scope for this article
- Interface– On which interface the switch learnt the MAC address
Entries that has no age (but a dash instead) are the ones associated switch’s IP addresses. Moreover, note that this table is populated only from ARP requests the switch made. As as a result, ARP requests made from other devices and passing through the switch will be ignored.
Checking the previous output, we didn’t find any MAC address but the ones of the local switch. This mean that ARP requests have been failing, so we have a problem at the data-link layer or even below.
Verify MAC address table
The MAC address table on a switch remembers the association between a MAC address and a physical port of the switch. The switch will populate that table and keep it up-to-date every time it receives an ethernet frame. In other words, to see the MAC address of these laptops on the Top switch, laptops must generate traffic and that traffic must reach the Top switch. Even ARP responses are traffic, but we don’t know if our requests made it to the client. In case they didn’t, the device won’t have generated any traffic. Just to be sure, jump on Laptop0 and ping the default gateway, 10.100.2.1.
Ping fails as expected.
Even if ping fails, our PC has generated some traffic, so we can check the ARP table on the Top switch to see if they made it. We do it by typing show mac address-table from the privileged exec prompt.
This is the content of the MAC address table.
We are looking for the MAC address of the Laptop0, which is 00d0.ff66.c6a1(in Cisco notation). Nevertheless, we cannot find this MAC address here. Now we are sure that communication with the device doing routing is broken. We need to find if we can reach the peripheral switch, at least. Since we don’t know to where in the network Laptop0 is, we need to check all the switches. Our objective is to find a switch having that MAC address in the MAC address table.
Checking MAC address table on peripheral switches
To continue with our switch troubleshooting, we are going to connect to LeftSwitch. To do it, remain on TopSwitchL3 in privileged exec, then type telnet 10.80.0.2 and enter the password as required. Then, once again, issue show mac address-table and check the results.
This time we found the device we were looking for (highlighted).
Here we are! We found our device, it is connected to the FastEthernet 0/10port on Left Switch. At this point, we know that the connection between Laptop0 and Left Switch is working fine, but there might be some problems between LeftSwitch and top switch. Before we check that, try to discover where Laptop1 is connected on your own. Just for the sake of learning, here’s the explanation of show mac address-table output.
- Vlan– the VLAN associated to the MAC address, this comes from the VLAN associated to the port
- Mac Address– the MAC address itself
- Type– How the MAC address was learnt, dynamic means from an ethernet frame, static means it was hard coded
- Port– Physical port this MAC address was seen on
Tip: if you don’t see the MAC address even there, it means the MAC address table aged out. Try again the ping from the Laptop and, as soon as it finishes, immediately check the MAC address table.
This problem involves two devices on the same VLAN, so it could be potentially a problem of that VLAN. We issue show vlan brief to check if the VLAN exists, and we find that VLAN 102 is the one for Laptops. Consequently, we need to check if this VLAN goes on the trunk toward Top Switch L3. We do that with show interfaces trunk.
VLAN 102 is missing from the trunk to Top Switch L3.
This must be the problem. Therefore, all we need to do is enter in configuration mode, go under the interface GigabitEthernet 0/1 and add the missing VLANs. Below, the needed commands.
interface GigabitEthernet 0/1 switchport trunk allowed vlan add 102 end
Once you typed that, try to ping the default gateway from both Laptop1 and Laptop2. This problem prevented all the traffic from devices on Left Switch to reach the Top Switch, the routing core. As a result, all devices inside VLAN 102 on Left Switches, was isolated from everything else. Review this configuration, and when you feel confident with what we did move to the next ticket.
Define The Problem
The second ticket in our Switch Troubleshooting is the one from the Engineering guy, the one complaining to have no network access. This time, we know the problem is on Laptop2, and only on that device. All laptops are working fine, and other devices connected to the same switch are working fine too. Therefore, we know that this problem is very limited, and can be associated to either PC settings or access port settings.
Gathering Additional Information
Again, we need to know the whereabouts of the PC. So, log into Laptop2 and, from the prompt, type ipconfig /all. You will find that this PC has the IP address 10.100.2.12 associated with a /24 subnet mask, the default gateway is the 10.100.2.1 and the MAC address is 0002.154c.552c. Since we know the problem is likely to be on the device, it’s time to find it.
Finding The Device In The Network
This time we will go much faster. First thing, from the command prompt of Laptop2, issue a ping toward 10.100.2.1 to generate traffic. After that, immediately check the MAC address table of TopSwitchL3 with show mac address-table command. We are looking for 0002.154c.552c. Apparently, we found it.
We can see the MAC address of the laptop on Top Switch L3.
At a first glance, the laptop might seem connected to FastEthernet 0/2 on Top Switch L3. The problem is clear from there, the laptop is in VLAN 301 (servers)instead of VLAN 102 (laptops). However, we are doing switch troubleshooting, so we must look closely. Thus, we see than this port is associated to multiple MAC addresses in different VLANs. We should check whether it is a trunk using our friend show interfaces trunk.
This interface is a trunk, likely to be connected to another Switch.
To help you, FastEthernet 0/2 is the interface connecting Center Switch. Since we saw the MAC address of the laptop coming from that interface, the laptop must be connected to that switch. We can reach it with telnet 10.80.0.3 and check its MAC address table.
Here we are. This time we truly found the device.
Interface FastEhternet 0/10 is an access port in VLAN 301. To put it in VLAN 102, as it should be, use the following commands.
interface FastEtherent 0/10 switchport access vlan 102 end
Done. The problem is fixed, and pinging again can prove it.
Define The Problem
For the third ticket of our switch troubleshooting, a Junior technician told us about strange messages on Right Switch. So, we know where the problem is and we just need to check those messages. Log into Right Switch using telnet 10.80.0.4 to check those messages. However, since messages normally appear in console only, and we don’t have console access, we need to find an alternative way.
To see the log messages in VTY lines, type terminal monitor, and to disable that use terminal no monitor. If you wait for about 30 seconds with the terminal monitor on, you will see a CDP error message as in picture.
Here is the CDP error we were looking for.
Fixing Native VLAN Mismatch
Let’s analyze the syntax. Here’s the whole message.
%CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on GigabitEthernet0/1 (1), with TopSwitchL3 FastEtherent0/3 (99).
This message is extremely clear, but analyzing all its sections will make that even clearer.
- %CDP– Cisco Discovery Protocol, is the protocol that noticed the error
- 4– Severity, indicates how sever this error is, where 0 is the most sever and 7 is debugging. 4 means warning.
- NATIVE_VLAN_MISMATCH– The code of the error
- Native VLAN mismatch discovered– Brief description of the error, it indicates that the native VLAN on one side of a trunk is different from the one set by the other switch
- on GigabitEthernet0/1– interface of the local switch
- (1)– Native VLAN of the local switch
- with TopSwitchL3– Hostname of the remote switch
- FastEthernet0/3– interface of the remote switch
- (99)– Native VLAN of the remote switch
What this message is trying to say is that the native VLAN on this switch differs with the one on the other side of the trunk. Specifically, Top Switch L3 is using VLAN 99 – correct – while Right Switch is using VLAN 1 – wrong. To fix this, simply add the commands below.
interface GigabitEthernet0/1 switchport trunk native vlan 99 end
Then wait for about a minute, you won’t see any more message like that.