Cisco Meraki VPN peers can use Automatic NAT Traversal to establish a secure IPsec tunnel through a firewall or NAT. When ACLs on an upstream firewall block source ports or more likely the case destination UDP ports in the range 32768-61000 on outbound traffic, a peer will not be able to punch a hole in the firewall and establish a tunnel with other remote peers.
In the example below, the firewall blocks peer1 from sending outbound UDP packets in the necessary destination port range. This prevents a hole punch. Now when peer2 tries to send inbound packets its packets are dropped on the outside interface of the firewall because they do not match and existing outbound session. In this instance the tunnel will not be established between peer1 and peer2.
Failed connectivity tests or a VPN status of 'disconnected' indicates a tunnel failure between peers in Dashboard.
In a Site-to-site VPN if two peers are unable to establish a VPN connection they will show as disconnected on each others' VPN status pages. In this case, a packet capture should be taken on the primary Internet interface of both peers in a connection to analyze which firewall is blocking IPsec communication.
With an AP to MX concentrator connection type, use the Test connectivity button on the AP network. Running the test will report which APs "failed to connect to the concentrator." Please note if the issue is on the concentrator side, it is likely that all APs will fail the test. A packet capture should be taken on the wired interface of each AP that failed to connect and concentrator. Another capture should be taken from the primary Internet interface of the MX. These captures can be analyzed to determine which sites firewall is blocking outbound IPsec communication.
AP to VM concentrator connections cannot be debugged in this manner because a packet capture cannot be taken from a VM. Use the Test connectivity button on the AP network or the SSID status Live tool to determine which APs "failed to connect to the concentrator." SSID status will be reported as disconnected for each peer that fails to establish a tunnel to the concentrator. This will isolate the problem to specific APs. Next, get packet captures off the wired interface of each AP that failed for analysis. Please note if the issue is on the concentrator side, it is likely that all APs will fail the test. This generally indicates the problem is with the firewall upstream from the VM.
Packet captures can be taken from Dashboard and downloaded as a .pcap file for analysis and filtering in Wireshark packet analyzer. They are invaluable for troubleshooting connections between hosts and isolating connectivity issues.
In the example below there is an AP to VM concentrator tunnel that will not establish. We take packet captures from different points in the path to help determine which firewall is blocking the peer-to-peer communication.
The first capture, shown below, was taken from the wired interface of AP 10.0.8.99. We can see the AP attempting to punch a hole in its local upstream firewall by sending packets to 220.127.116.11, which is the outside IP address of the NAT that the VM concentrator sits behind. Notice the the AP is sending traffic to the concentrator but there is no return traffic in the capture from the VM behind the NAT.
AP 10.0.8.99:45540 -> VM 18.104.22.168:53654
A second capture, shown below, was taken from the inside interface of the MX firewall upstream from the VM concentrator. The VM concentrator uses IP address 10.0.50.246 on the LAN. We can see the VM sending packets to 22.214.171.124 which is the outside IP address of the NAT that the AP sits behind in an attempt to punch a hole in its local upstream firewall. Notice the VM is sending traffic to the AP but no return traffic is present from the AP behind the NAT.
VM 10.0.50.246:53654 -> AP 126.96.36.199:45540
A third capture was then taken, this time from the outside interface of the MX firewall upstream from the VM. We can the see the VM's traffic has been translated to 188.8.131.52, which is the firewall's outside IP address, and that it is being forwarded onto the Internet. This indicates the firewall is not blocking outbound IPsec traffic in the VM concentrator site. However, we do not see any traffic originating from 184.108.40.206, the IP address of the NAT device the AP sits behind. From this we can conclude that the firewall upstream from the AP is blocking outbound IPsec traffic within the UDP port range 32768-61000.
VM 220.127.116.11:53654 -> AP 18.104.22.168:45540
To confirm we take a final capture from the outside interface of the MX firewall upstream from the AP, shown below. This capture shows packets originating from the VM at 22.214.171.124 and arriving at the AP firewall's outside interface at 126.96.36.199. We still do not see any traffic originating from the AP being sent from the outside interface. This indicates the AP firewall is in fact blocking outbound IPsec traffic on the inside interface, specifically destination UDP port range 32768-61000.
VM 188.8.131.52:53654 -> AP 184.108.40.206:45540
Once we reconfigure the firewall upstream from the AP to allow outbound destination port range 32768-61000, peers are able to form a tunnel. Although the first 4 captures are filtered by UDP ports 53654 and 45540, once the firewall is opened two-way traffic can occur on any dynamically chosen ports as shown below on a packet capture taken from the wired interface of the AP. Now the AP is registered with and using with port 41091 for VPN communication.
AP 10.0.8.99:41091 -> VM 220.127.116.11:53654
AP 10.0.8.99:41091 <- VM 18.104.22.168:53654
Below are two examples of ACLs that could be used to allow peer-to-peer communication between Cisco Meraki VPN peers. For the second option, X.X.X.X/32 represents the IP address of the Cisco Meraki device.
allow inside to outside, protocol: udp, source ip: any, src port: any, dst ip: any, dst port: 32768-61000 allow outside to inside established (may not be necessary with stateful firewalls)
allow inside to outside, protocol: udp, source ip: X.X.X.X/32, src port: 32768-61000, dst ip: any, dst port: 32768-61000 allow outside to inside established (may not be necessary with stateful firewalls)
In summary, Cisco Meraki VPN peers must be able to use high number UDP ports to communicate with each other. Security systems such as firewalls that disallow this traffic may prevent successful traffic flow over the VPN. Please follow the diagnostic and troubleshooting steps above to resolve such issues.