Skip to main content

 

Cisco Meraki Documentation

Troubleshooting Offline Devices Using Packet Captures

Overview

This article explains how to collect and analyze packet captures to troubleshoot unreachable or offline devices. 

This guide applies to the following scenarios:

  • You have the topology below and want to understand why the device is offline when the last known upstream neighbor is an online Cisco Meraki device. A red cross marks the offline device. A blue arrow and circle mark the last known upstream neighbor.

clipboard_e67785f9ff4b646d4a79b5d6b56f83c22.png

  • The offline device does not have a Meraki neighbor, but the default gateway is a managed MX WAN appliance.

clipboard_e9d1f705be149b638dcb52fec4308520f.png

You can take a packet capture from the upstream neighbor or the MX to investigate why the offline device is having problems. The next section explains how to take a packet capture.

Troubleshooting offline device issues

Follow these steps to collect packet capture to investigate an offline device:
Step 1. Identify the upstream neighbor of the offline device. If the upstream neighbor is also offline, move the troubleshooting process to that device.

If the upstream neighbor is a Meraki switch, take a port mirror capture. A port mirror capture can surface extra information not visible in a dashboard capture. For steps, refer to Packet Captures and Port Mirroring on the MS Switch.

If you cannot collect a port mirror capture at this time, continue to step 2.

Note: If your upstream neighbor is a non-Meraki device but your offline device is an MS running MS16+, you can take the capture using the Local Status Page (LSP), for more details please review the document Cisco Meraki Local Status Page: MS Switches.

If the upstream neighbor is a non-Meraki device and the offline device cannot capture using the local status page, refer to the device vendor's documentation. 


Step 2. Prepare to take a packet capture on the upstream neighbor.

Go to Network-wide > Monitor > Packet Capture / Intelligent capture. 

From the drop-down select the upstream neighbor category:

Upstream Neighbor Device Model         Select
MX for security appliances
MS or CS for switches
MR or CW for access points
MG for cellular gateways
CG  for campus gateways
  • From the list of devices, select the identified upstream neighbor. 
  • For the interface, select the interface where the offline device connects. 
  • For output, select one of the following: 
    • Save to cloud: Use this option for Intelligent Capture and analyze the capture directly in the dashboard. 
    • Download .pcap file: Use this option for a standard packet capture. You need Wireshark to open and analyze the file.
  • For duration, select 300 seconds.

intelligent capture window image

Step 3. If possible, disconnect the network cable between the offline device and the upstream neighbor, then start the packet capture for approximately 5 minutes.

Step 4. Reconnect the offline device cable immediately after starting the packet capture. 

Step 5. Wait for the capture to finish.

For more details about the packet capture feature, refer to Packet Capture Overview.

Troubleshooting packet capture review issue

Steps to troubleshoot packet capture review are:

Step 1. Find and note the offline device's MAC address. The MAC address appears under the device's name in the dashboard. If the offline device is an MX WAN appliance and you want to capture traffic from one of its WAN interfaces, refer to Calculating MX WAN MAC Addresses.

clipboard_e9200b3117b7d8a512be613f2fb4587f5.png

Step 2. Open the .pcap file.

  • For .pcaps downloaded to your computer, use Wireshark.
  • For Intelligent Capture files saved to the cloud, go to Network-wide > Monitor > Intelligent Capture > Store captures, and open it directly on the dashboard.

Step 3. Filter the capture to focus on the offline device traffic.

Use this filter:

cdp.deviceid == aaaabbbbcccc or lldp.chassis.id.mac == aaaabbbbcccc or (eth.addr == aaaabbbbcccc and (arp||dhcp||dns||tcp.port == 443))

Where aaaabbbbcccc is your offline device's MAC address, all lower case and no separation between octets. Example:

clipboard_ec6edcbdf5f864b51d6d3b56feee35b00.png

The following sections describe the packets you should see to confirm the offline device is present and attempting to connect to the Meraki cloud. 

CDP/LLDP packets

The presence of CDP or LLDP packets sent by the offline device confirms the device is present, powered on, connected to the expected port, and establishing a Layer 2 network connection. 

This is how a CDP packet sent by a device with MAC address 0c:8d:db:0b:f9:07 looks like:

clipboard_e9387cd678fe3e3b41e90090b5657323d.png

Example of an LLDP packet:

clipboard_e7d8095c789fca0c6f169d95ece578e0f.png

If you don't see CDP or LLDP packets sent by the offline device, most likely the device is either disconnected, powered off or has some other hardware problem. Go on-site to do a physical check of the offline device.

DHCP packets

If the device is configured to use a static IP address, you might not see DHCP packets, since a static IP address falls back to DHCP only when it cannot reach the dashboard. You can expect to see the full DORA (Discover, Offer, Request, Acknowledge) exchange.

clipboard_ec16b9bfa08928d7edfc80ee50d81da50.png

If you do not see Offers, verify the port connected to the offline device has the correct VLAN (Virtual Local Area Network) configuration. 

If you see Nacks or another response instead of Offers or Acks, review the DHCP server logs to determine why the server is denying an IP address to the offline device. 

ARP packets

After the device receives an IP address, it sends an ARP (Address Resolution Protocol) request for the default gateway's IP address. Expect a reply from the gateway's IP address containing its MAC address information. 

clipboard_ee7e7b453c0717751a92d84f082ed12fc.png

If you see a request but not a reply, verify the DHCP server provides the correct default gateway information, and confirm the offline device can reach the default gateway. 

DNS packets

After the device confirms it can reach the default gateway, it sends a DNS query to resolve the dashboard server's domain name. Expect to see DNS queries with this format:

<region>[...].meraki.com

The region refers to where the offline device's network is hosted. For example, a device associated with a network hosted in the United States sends DNS requests for us[...].meraki.com

The exact domain name varies, but always follows the <region>[...].meraki.com format. Refer to the Firewall info section for organization-specific domain name / IP addresses.

clipboard_ea9dbccd064940ab29a17af201a1f521d.png

If you don't see DNS responses, verify the DNS server is reachable and can resolve meraki.com domains. 

TCP packets

Packet captures taken from the Meraki dashboard do not capture TCP traffic between the device and the cloud IP addresses. You should see everything else, including traffic to cloud destination IPs. 

TCP packets - 3-way-handshake

If the device resolves the dashboard servers, it opens a TCP session with the IP address from the DNS response, in this case, 209.206.52.53. The device sends a TCP SYN packet. You should then see a SYN-ACK packet from the dashboard's IP address, followed by an ACK from the device.

clipboard_e4072a3e6770644bde98086108820d0e2.png

If you only see SYN packets sent by the offline divice and never see an ACK coming from a Meraki public IP. An intermediate device between the offline device and the ISP is likely blocking the traffic. The traffic can be blocked from the device to the cloud, or from the cloud to the device. 

TCP packets - mTLS authentication

If the device was able to comple the TCP 3-way-handshake, it moves on to the mTLS (Mutual Transport Layer Security) process. This example shows the expected packet flow.

clipboard_e134d4a74b8f2d909f4fba46514179c8f.png

If you see all these packets and the device is still having problems to communicate with the cloud, ensure there certificate sent by the Meraki cloud was signed by Meraki. To review that, open the certificate packet sent by the cloud (The destination IP should be the offline device):

clipboard_e9c1ddc0dc9918c09553e3dc84df0f96f.png

Open the packet's section Transport Layer Security > TLSv1.2 > Handshake Protocol: Certificate > Certificates. Expand the three certs and confirm the issuer looks similar to this:

clipboard_e47bbd5b3c52e61fbb820a73829413d20.png

You see all these packets, and the device still cannot communicate with the cloud. The offline device's traffic is likely going through SSL (Secure Sockets Layer) inspection, which breaks authentication with the Meraki cloud and prevents the device from going online. 

Firewall information page

Upstream firewall requirements vary by product, firmware, region, and enabled services. For the most accurate per-organization firewall information, go to Help > Firewall info in the dashboard.

Firewall info.png

Record the destination IP addresses and ports for the following entries. Use this information when reviewing and filtering packet captures to troubleshoot cloud connectivity issues:

  • Meraki cloud communication
  • Backup Meraki cloud communication
  • Backup configuration downloads
  • NTP (Network Time Protocol) time synchronization
  • Uplink connection monitor

For more details, refer to Upstream Firewall Rules for Cloud Connectivity.