MX Routing Behavior
Overview
This article discusses route configuration and the interaction of differing routes on the MX. This article is intended to serve as a reference for deeper understanding of how routing decisions are made on MX Security Appliances.
This article may be especially useful for:
- Understanding the underlying mechanics of MPLS failover to Auto VPN
- Learning how VPN routing decisions in a DC-DC Failover configuration are made
- Understanding how the MX will behave in more complex routing configurations that leverage multiple types of routes or overlapping routes
Please note that this article assumes familiarity with configuring MX Security Appliances.
Types of Routes
MX Security Appliances support the configuration of several different types of routes, as detailed below.
Directly Connected
Directly connected routes are subnets defined in the Security & SD-WAN > Configure > Addressing & VLANs page of Dashboard. This is true of configurations leveraging a number of VLANs defined on an MX, or those that utilize a single subnet configuration with VLANs disabled.
Static routes are not directly connected.
Client VPN
The client VPN subnet is configured under the Security & SD-WAN > Configure > Client VPN page of Dashboard. There is only ever a single client VPN subnet on an individual MX network.
While client VPN utilizes the IPsec protocol to form a secure tunnel with the end device, the client VPN subnet is treated differently from routes to non-Meraki VPN peers.
Static Routes
Static routes are configured on the Security & SD-WAN > Configure > Addressing & VLANs page of Dashboard. Configuration of static routes is only possible while the MX is operating in Routed mode.
Static routes are used to communicate with subnets or VLANs that are not defined or "owned" by the MX, but are reachable through another layer 3 device on the network. Static routes require a next hop IP address be specified within the scope of a configured VLAN or subnet to be able to successfully route traffic to another layer 3 device.
Static routes are used to define subnets accessible through the LAN of the MX.
Auto VPN
Cisco Meraki's Auto VPN can be configured on the Security & SD-WAN > Configure > Site-to-site VPN page of Dashboard. Auto VPN is a layer 3, IPsec-based site-to-site VPN. The mechanics are outlined in this white paper.
An MX Security Appliance configured to participate in an Auto VPN topology will automatically create routes for subnets included in the Auto VPN topology.
Non-Meraki VPN Peers (Other IPsec)
Non-Meraki VPN peers are configured on the Security & SD-WAN > Configure > Site-to-site VPN page of Dashboard. These VPN peers are connected to using IPsec. If an MX is configured to establish a VPN with a non-Meraki VPN peer, the MX will also have routes to the private subnets defined for that VPN peer. If a full tunnel is required, both peers must configure a private subnet of 0.0.0.0/0 as its private subnet for the IPsec SAs to be created successfully.
MX cannot route between two Non-Meraki VPN peers.
NAT
The MX Security Appliance can be configured to operate in Routed mode, from the Security & SD-WAN > Configure > Addressing & VLANs page of Dashboard. When in Routed mode, the MX can be configured with multiple LAN subnets and static routes. If a route for a particular subnet does not exist on the MX then in this mode, the traffic is NATed and sent out across an Ethernet connection into one of the Internet interfaces, or out a cellular connection.
Route Priority
Each type of route configured on the MX has a specific priority in comparison with other types of routes. The priority is as follows:
- Directly Connected
- Client VPN
- Static Routes
- Auto VPN Routes
- Non-Meraki VPN Peers
- BGP learned Routes
- NAT*
*If no routes are defined, then the traffic is NATed and sent out an active Internet interface. This only occurs while the MX is configured in Routed mode.
Note: For BGP route selection please refer to our BGP documentation.
When routing decisions are made, traffic destined for an address for which multiple routes exist will be routed in the order of priority above. For example, if a route for 10.0.0.0/8 is defined as both a static route and as an Auto VPN route, traffic destined for that subnet is routed using the static route, as static routes take precedence over Auto VPN routes.
Auto VPN routes take precedence over Non-Meraki VPN Peers, BGP learned routes, and NAT, even when the Auto VPN routes are down.
Non-Meraki VPN routes are considered "always active" and will not automatically fail over when the peer connection is down. For instance, a Non-Meraki VPN peer route for 0.0.0.0/0 will always take priority over the NAT default route, regardless of the Non-Meraki VPN tunnel connection state.
Overlapping Routes
Route priority dictates how traffic is routed when multiple routes exist to the same subnet. However, overlapping routes that are not identical are also present in many deployments. In this case, the most specific route will be used.
Multiple Ports With the Same MAC Address is Seen on Those Ports
Designs which require devices which leverage the same MAC address across these ports should be configured to have each interface configured to use a unique MAC address or only have one connection to the MX. If this design consideration is not heeded, there is a possibility that traffic destined to the device on a different VLAN maybe sent out the incorrect port.
Advanced Configurations
Static Route Tracking
Static routes can be configured with three different availability settings:
- Always active
- Active while the next hop responds to ping
- Active while host responds to ping
If the static route is always active, the configured static route will always be maintained in the routing table.
When a route is configured as "Active while the next hop responds to ping", or "Active while a host responds to ping", the MX tracks the route. If the MX stops receiving ping responses for a period of time, the route will be removed from the routing table. The route is re-added when responses are received again.
Note: If a route is configured as "Active while host responds to ping", the configured host IP must lie within the subnet configured for the static route
Note: If a static route is configured with the "Active while host responds to ping" option, both the next-hop and the specified host need to be able to respond to pings. If either IP address stops responding to pings, the route will be marked as down and will be removed from the MX's route table.
The route tracking mechanism only has local significance, which means, that the status of the route doesn't influence whether the route is advertised or not and it won't affect the routing on the remote end of the VPN.
Datacenter Redundancy (DC-DC Failover)
A DC-DC failover configuration is defined by the following characteristics:
- Multiple MXs are configured in VPN concentrator mode on the Security & SD-WAN > Configure > Addressing & VLANs page of Dashboard
- The VPN concentrator mode MXs are configured to advertise the same upstream subnet(s) via the Security & SD-WAN > Configure > Site-to-site VPN page in Dashboard
- The VPN concentrator mode MXs are configured as Auto VPN hubs for spoke MXs if a hub & spoke topology is being used
In this configuration, multiple routes exist to the same subnet and the routes are the same type (Auto VPN). Since route priority and matching on the most specific route cannot be used to determine which Auto VPN route should be used, the concentrator priority is used as a tie breaker.
In a DC-DC failover configuration, Auto VPN traffic to subnets advertised from multiple concentrators is routed to the highest priority VPN concentrator that is reachable. VPN Concentrator priority is defined on the Security & SD-WAN > Configure > Site-to-site VPN page in Dashboard.
It is important to note that concentrator priorities are used only by appliances in Hub (Mesh) mode. An appliance in Hub-and-Spoke mode will ignore the concentrator priorities and will use its hub priorities instead.
When an MX Security Appliance is configured to connect to multiple VPN concentrators advertising identical subnets, the routes to those subnets become tracked. Hello messages are periodically sent across the tunnel to monitor connectivity. If the tunnel to the highest priority hub goes down, the route is removed from the route table and traffic is routed to the next highest priority hub that is reachable.
Failover Times
| Service Name | Failover Time | Failback Time |
| Auto VPN Tunnels | 30-40 seconds | 30-40 seconds |
| DC-DC Failover | 20-30 seconds | 20-30 seconds |
| Static Route Tracking | 15-20 seconds | 9-12 seconds |
| WAN connectivity | 300 seconds or less | 15-30 seconds |
Example Scenarios and Traffic Flow Details
MPLS Failover to Auto VPN
This article describes the configuration and failover behavior for a configuration that leverages MPLS failover to Auto VPN tunnels.
DC-DC Failover
The following section provides an explanation of how VPN routing functions in a configuration where multiple VPN hubs are configured in a DC-DC failover topology.
Consider the following example:
- One branch location
- The branch acting as a spoke to 4 VPN hubs
- One VPN hub, DC1 hosting a /8 subnet, and a unique /24
- One VPN hub, DC2, is hosting a unique /16 that overlaps with DC1
- Two VPN hubs, DC3 and DC4, are configured to advertise the same /24. DC3 has a higher hub priority than DC4.
In this configuration, the branch MX will receive routes for all subnets at each of the 4 datacenters, even those that overlap. The following table represents the routes presented to the branch MX in this configuration:
| Subnet | Next Hop Concentrator | Hub Priority |
| 10.0.0.0/8 | DC1 | 1 |
| 192.168.100.0/24 | DC1 | 1 |
| 10.1.0.0/16 | DC2 | 2 |
| 172.16.1.0/24 | DC3 | 3 |
| 172.16.1.0/24 | DC4 | 4 |
How is traffic routed given the above configuration?
- If traffic is sent to 10.100.12.44
- The branch MX sends this traffic to DC1. This subnet is only contained within the 10.0.0.0/8 subnet advertised via this DC.
- If traffic is sent to 192.168.100.30
- The branch MX sends this traffic to DC1. This subnet is only advertised by DC1.
- If traffic is sent to 10.1.0.98
- The branch MX sends this traffic to DC2. This subnet matches on both the 10.0.0.0/8 subnet advertised by DC1 and the 10.1.0.0/16 subnet advertised by DC2. While DC1 has a higher hub priority, the MX prefers the most specific route and sends the traffic to DC2. If the branch MX does not have connectivity to DC2, traffic will not be routed to DC1. This is because only identical subnets are tracked for failover.**
- If traffic is sent to 172.16.1.214
- The branch MX sends this traffic to DC3. DC3 has a higher priority than DC4, so traffic for the subnet advertised from both concentrators is sent to DC3. If a VPN connection cannot be established to DC3, then this traffic is routed to DC4.
** If you require the spoke MX to be able to failover to a hub with a less specific subnet, please contact Cisco Meraki Support.
Multiple Overlapping Routes
Consider the following example:
- One branch
- A static route for the 172.16.0.0/12 subnet to MPLS
- Static routes to other downstream subnets of 172.16.30.0/24, 192.168.20.0/25
- Auto VPN tunnels to two Datacenters, DC1 and DC2
- DC1 is configured to advertise subnets of 10.1.10.0/24 and 172.16.10.0/24
- DC2 is configured to advertise subnets of 10.1.10.0/24 and 172.16.0.0/12
- An IPsec VPN connection to a non-Meraki VPN peer is also present
- The non-Meraki VPN peer has 172.16.45.0/24, 192.168.20.0/25, and 192.168.0.0/16
The following table represents the routes presented to the branch MX in this configuration:
| Subnet | Route Type | Hub Priority |
| 10.1.10.0/24 (DC1) | Auto VPN | 1 |
| 10.1.10.0/24 (DC2) | Auto VPN | 2 |
| 172.16.0.0/12 | Static | N/A - Not an Auto VPN route |
| 172.16.0.0/12 (DC2) | Auto VPN | 2 |
| 172.16.10.0/24 (DC1) | Auto VPN | 1 |
| 172.16.30.0/24 | Static | N/A - Not an Auto VPN route |
| 172.16.45.0/24 | Non-Meraki VPN Peer | N/A - Not an Auto VPN route |
| 192.168.0.0/16 | Non-Meraki VPN Peer | N/A - Not an Auto VPN route |
| 192.168.20.0/25 | Static | N/A - Not an Auto VPN route |
| 192.168.20.0/25 | Non-Meraki VPN Peer | N/A - Not an Auto VPN route |
How is traffic routed given the above configuration?
- If traffic is sent to 10.1.10.1
- The MX sends this traffic to DC1. DC1 has a higher priority than DC2, so traffic for the subnet advertised from both concentrators is sent to DC1. If a VPN connection cannot be established to DC1, then this traffic is routed to DC2.
- If traffic is sent to 172.16.0.1
- The MX has both a static and Auto VPN route for this subnet. Since static routes have a higher priority than Auto VPN routes, traffic will be sent using the static route.
If route tracking has been configured on the static route, when the MX stops receiving ping responses for the static route it will be removed from the routing table. In this scenario, the only remaining route is the Auto VPN route and traffic will be sent to DC2. - If traffic is sent to 172.16.10.1
- The MX has multiple routes for 172.16.0.0/12, but a more specific Auto VPN route for the 172.16.10.0/24 subnet is available. Traffic will be sent to DC1 using the more specific route.
- If traffic is sent to 172.16.30.1
- The MX has multiple routes for 172.16.0.0/12, but a more specific static route for the 172.16.30.0/24 subnet is available. Traffic will be sent using the more specific static route.
If route tracking has been configured on the static route, when the MX stops receiving ping responses for the static route it will be removed from the routing table. In this scenario, traffic will be routed using the available 172.16.0.0/12 routes. As static routes have a higher priority than Auto VPN routes, traffic will be sent using the static route.
If the static route for 172.16.0.0/12 has also been removed via route tracking, then the Auto VPN route will be used and traffic is sent to DC2. - If traffic is sent to 172.16.45.1
- The MX has multiple routes for 172.16.0.0/12, but a more specific route for the 172.16.45.0/24 subnet is available via the non-Meraki VPN peer. Traffic will be sent using the more specific route from the non-Meraki VPN peer.
- If traffic is sent to 192.168.0.1
- The MX has a route available for 192.168.0.0/16 via a non-Meraki VPN peer. Traffic will be sent to the non-Meraki VPN peer.
- If traffic is sent to 192.168.20.1
- The MX has multiple routes for 192.168.20.0/25, as well as a route for 192.168.0.0/16. The more specific 192.168.20.0/25 routes will be used over the /16 route available from the non-Meraki VPN peer. As static routes have a higher priority than routes from non-Meraki VPN peers, traffic will be sent using the static route.
If route tracking has been configured on the static route, when the MX stops receiving ping responses for the static route it will be removed from the routing table. In this scenario, traffic would be sent to the remaining 192.168.20.0/25 route available via the non-Meraki VPN peer.
Additional Considerations
When configuring routing on an MX Security Appliance it is important to take note of the following considerations.
Auto VPN and Non-Meraki VPN peers
An MX Security Appliance can establish tunnels to both Auto VPN and Non-Meraki VPN peers. The MX will send traffic to those VPN peers using the principles discussed above. However, an MX that builds tunnels to both Auto VPN and Non-Meraki VPN peers, will only route traffic between the Non-Meraki VPN peers and other Auto VPN peers if BGP routing over IPsec VPN is in use.
Spoke-to-Spoke Communication
All MX Security Appliances joined to an Auto VPN topology will receive routes for all other subnets joined to the Auto VPN topology. In a hub and spoke topology, a spoke MX will receive routes to communicate with other spoke sites with a next hop of their configured VPN hub(s). The VPN hubs will then route between spoke sites.
To prevent certain traffic from entering the VPN, site-to-site VPN firewall rules can be configured on the Security & SD-WAN > Configure > Site-to-site VPN page in Dashboard.
Use of Default Routes and Exit Hubs
Hubs and spokes utilizing full-tunnel (exit hub or default route) Auto VPN configurations will not fail over to a secondary default route without manual intervention if Auto VPN connectivity is disrupted between the peers (with the exception of DC-DC failover.) For example, if a spoke is experiencing connectivity problems to a hub that is configured as the default route traffic can be lost as the security appliance will not failover to the unencrypted WAN.
Routed Mode and Auto VPN
You can only advertise the same subnet from more than one appliance if all appliances advertising that subnet are in Passthrough or VPN Concentrator mode. All subnets advertised from an appliance in Routed mode must be unique within the Auto VPN topology.
Private Subnet Configuration on Multiple Non-Meraki VPN Peers
When two or more non-Meraki VPN peers are configured identical or overlapping private subnets, you will be prompted with a message alerting you of the overlapping or identical subnets that must be confirmed before saving changes. The messages describe the following routing behavior:
- When two subnets overlap, IP traffic will be routed to the most specific subnet.
- When two subnets directly conflict, IP routing behavior will be undefined. This can be avoided by configuring the Availability column with different network tags to ensure that no security appliances have peers with conflicting IP subnets.
An example screenshot and confirmation message can be found below:
Behavior of Management Traffic
The following services and features have their traffic always ingress or egress the primary WAN/Internet uplink:
- Management connection to the Meraki Cloud on TCP ports 80, 443 and UDP port 7351 - Additional information can be found here and here.
- Meraki Cloud Authentication ( 18.107.9 and above / 18.208 and above)
- Security Lists - Lists used for IDS/IPS signatures.
- Content Filtering - Top Sites lists and Full Lists lookups (pre-MX17)
- AutoVPN, Non-Meraki VPN, and Client VPN tunnels.
- Content Filtering (Talos MX 17.10.5 and above / 18.107.3 and above)
The following services/tools will specifically adhere to the route priority and not necessarily ingress or egress the primary WAN/Internet uplink:
- Advanced Malware Protection (AMP) - Registration on TCP Port 443 (Specific FQDN(s) can be located under Help -> Firewall info on the dashboard)
- Advanced Malware Protection (AMP) - File disposition lookups used for Advanced Malware Protection on TCP port 443 (Specific FQDN(s) can be located under Help -> Firewall info on the dashboard)
- Meraki Cloud Authentication ( below 18.107.9 and 18.208)
- Meraki services on TCP ports 80, 443 and 7734 not covered above.
- Ping and Dashboard Throughput Live Tools.
- Content Filtering (Talos / 17.10.4 and below / 18.107.2 and below)
- ThousandEyes Agent traffic defined here.
- List Updates for the following services: Content Filtering , IDS/IPS Rule Updates and Geo-IP Lists for Layer 7 Country-Based Firewall rules.
- Redirected or relayed DNS queries when Umbrella DNS integration is enabled or "Use Umbrella" is configured under DHCP settings.
Note: The above behavior is specific to traffic initiated by the MX appliance itself, any Meraki devices connected downstream will always adhere to the route priority of the MX.