Troubleshooting Packet Loss between Devices

ping -n 20 8.8.8.8

Determining Where Packet Loss Is Occurring over Routed Links

If packet loss is seen, the next step is to identify where the packet loss begins to occur. 'tracert' can be used to check each layer 3 device along the path to the destination:

1. Open a command prompt on a client PC via the Start Menu and search for "cmd"
2. use ping command:

tracert  -d  8.8.8.8

This will perform a trace route to 8.8.8.8 and present each hop as an IP address. Substitute 8.8.8.8 with whatever address must be tested to.

3. Wait for the trace to complete; press CTRL/CMD+C if multiple lines ending with "Request timed out" are encountered.

 

A lack of response will be represented by an asterisk (*), potentially indicating packet loss or that the device is configured to not respond. The test may need to be completed multiple times to identify where loss is occurring. If packet loss is frequently encountered after a particular hop, the issue is most likely with that device or between it and the previous hop. This screenshot illustrates a tracert clear of packet loss. The only device to not respond (hop 11) is likely configured to do so, as there is no packet loss after it.

 

 

In this next screen shot, packet loss is regularly encountered beginning with hop 2. This indicates there may be an issue with the ISP gateway or the link(s) between the client gateway and ISP gateway. It is recommended to test from multiple clients at different locations in the network to help rule out specific client issues and develop commonality between clients experiencing the problem.

 

 

As a more robust test, the tool MTR can be used to perform a continuous series of traces and present a % of loss at each hop in the path to more clearly identify where the loss is occurring. Output for the above scenario would appear similar to:

 

 

Determining Packet Loss on WAN Uplink

If packet loss is observed on the WAN uplink, the next step is to determine if the loss is on the WAN Appliance or on the ISP side.

You can determine which interface is experiencing loss by taking packet captures on the LAN and Internet interfaces of the WAN Appliance.

1. Run constant pings from a PC to a public IP address.
2. Take simultaneous packet captures on the LAN and WAN of the security appliance.
3. Filter the traffic with source and destination IP address and ICMP.
4. Check that the ICMP requests are appropriately forwarded from the LAN to the WAN of the WAN Appliance.
5. If traffic is forwarded appropriately from LAN to WAN, the WAN Appliance is likely not the cause of the issue so you will need to troubleshoot the packet loss further on the ISP side.

Determining Where Packet Loss is Occurring in a Wireless/Switched Network

Tracert only provides information for layer 3 devices in the path, such as routers. However, in the case where packet loss is occurring at the first hop and must pass through a wireless access point and switch to get there, additional testing is required to isolate the problem. In this case, testing will need to be done multiple times, while getting progressively closer to the layer 3 devices. The following steps are illustrated in the image below:

1. Ping the access point to test wireless quality. If using a Cisco Meraki AP, ping my.meraki.com.
   If loss begins occurring here, refer to the article on troubleshooting wireless performance.
2. Ping a client connected to the same VLAN (if configured) on the switch that the wireless client is connected to. If multiple switches exist along the path, repeat this step as needed.
   If loss begins occurring here, the issue is most likely:
   - Duplex/speed settings mismatch on the link between the AP and the switch or the switch and the wired client
   - Bad cable between the AP and switch or the switch and the wired client
3. Connect a client directly to the router/firewall on the same VLAN as the wireless client and ping it from the wireless client.
   If loss begins occurring here, the issue is most likely:
   - Duplex/speed settings mismatch on the link between the switch and the router/firewall or the router/firewall and the wired client
   - Bad cable between the AP and switch or the router/firewall and the wired client

If testing with Cisco Meraki devices, it is also possible to ping the first MS switch or WAN appliance in the path. The IP address of the switch can be found by navigating to switch.meraki.com and the WAN appliance at wired.meraki.com.

 

Duplex Mismatch

This occurs when two ends of a link are using different speed/duplex settings, such as 100Mbps/half-duplex and 1000Mbps/full-duplex. When this occurs, some or all traffic will be lost on the link.

To correct this, ensure both sides of the link have identical settings. Ideally, both ends of the connection should be set to "Auto" for both speed and duplex. If a speed or duplex setting must be manually set on one end, ensure that it has been set to the same value(s) on the other end as well.

 

Link Congestion (Too Much Traffic)

This occurs when more traffic is attempting to go over a network link than it can support, such as 60 Mbps of traffic passing over the same 20 Mbps link. This creates a bottleneck, resulting in some traffic being dropped. 

There are multiple ways to solve this, including:

• Increasing the capacity of the link being overwhelmed to allow for all traffic
• Apply traffic-shaping rules on MX security appliances or MR access points to limit the volume of traffic, particularly focusing on decreasing the volume of undesirable traffic
• Apply traffic-shaping rules on MX security appliances or MR access points to prioritize more important traffic

Firewall Blocking Certain Traffic

Even if packet loss isn't occurring for all types of traffic, an upstream firewall may be filtering certain types of traffic. This can result in some websites loading and others failing, or some services being accessible while others are not.

If a firewall exists between two devices/locations experiencing these symptoms, ensure that the firewall is not blocking the traffic that is experiencing the problem.

 

 

Bad Cable or Loose Connection

A cable that has been poorly/incorrectly terminated or damaged can result in an incomplete or an inaccurate electrical signal passing between devices.

Swapping out a cable or performing a cable test on the one in question can help eliminate this as a possibility. Similarly, a cable that has not been fully seated in the port, or has been seated in a port with dust or other nonconductive debris on the pins, can result in an incomplete electrical signal. Be sure to keep all ports free of dust or buildup and ensure cables are securely connected.