MX Warm Spare - High-Availability Pair

Use Case and Benefits

In most customer deployments, network downtime has a direct impact on the business and should be avoided at all costs. Warm spare functionality prevents the network from having a single point of failure, allowing for fast, automatic recovery in the event of device failure. This functionality not only reduces the negative impact on end-user services but also offers significant benefits:

1. Reduced Network Downtime: In the event of hardware failure, network downtime will be greatly reduced or eliminated entirely, depending on the architecture being used.

2. No Manual Intervention Required: There is no need for manual intervention by the network administration team to facilitate recovery from a hardware failure.

3. Zero-Downtime MX Upgrades: When MX appliances are configured to operate in HA (High Availability), the dashboard will automatically take steps to minimize downtime during upgrades. This is achieved through the automated process detailed in the "Appliance Network with Two MXs in an HA Configuration" section of the "Best Practices for Meraki Firmware" document.

Underlying Concepts and Technologies

Network Setup 

Each concentrator has its own IP address to exchange management traffic with the Meraki cloud controller. However, the concentrators also share a virtual IP address that is used for non-management communication.

WAN Virtual IPs 

VIP addresses are shared by both the primary and warm spare appliance. Inbound and outbound traffic use this address to maintain the same IP address during a failover and reduce disruption. The virtual IPs are configured on the Security & SD-WAN > Monitor > Appliance status page, under the Spare section in the upper-left corner of the page. If two uplinks are configured, a VIP can be configured for each uplink. Each VIP must be in the same subnet as the IP addresses of both appliances for the uplink it is configured for, and it must be unique. In particular, it cannot be the same as either the primary or the warm spare's IP address.

LAN IP addresses are configured based on the appliance IPs in any configured VLANs. No virtual IPs are required on the LAN.

When using features such as port forwarding and NAT rules, services that direct traffic to the HA pair should be configured with the virtual IP address of the HA pair, not the individual WAN IP addresses of the primary and spare MXs.

Additionally, for DDNS to work with virtual IPs, the IP address of the primary uplink and the virtual IP address need to resolve to the same upstream public IP.

Virtual MAC addresses

When using an MX in HA mode:

• LAN side will use a virtual MAC address for all configured VLANs instead of the device MAC address.
• If WAN interfaces are configured to use virtual uplink IPs, the WAN MAC address will also use a different virtual MAC address. This virtual MAC address is based on the last three octets from the primary MX; the first three octets of the virtual MAC will always be "cc:03:d9".

• For MXs in HA mode that are configured to use WAN uplink IPs instead of virtual IPs, the MXs will use the physical MAC address of the respective WAN interface.
• Using the warm spare "swap button" on MX that are configured to use virtual IPs will change the shared virtual WAN and LAN MAC addresses, which will most likely cause connectivity disruption. It is not recommended to use it in production hours.
• Connection monitoring and management traffic will still use WAN interface MAC, even if VIP is configured. 

Failure Detection 

There are two failure detection methods for router mode warm spare.

WAN failover: WAN monitoring is performed using the same internet connectivity tests that are used for uplink failover. For more data on these checks, see the Connection Monitoring for WAN Failover article. If the primary appliance does not have a valid internet connection based on these tests, it will stop sending VRRP heartbeats, which will result in a failover. When uplink connectivity on the original primary appliance is restored and the warm spare begins receiving VRRP heartbeats again, it will relinquish the active role back to the primary appliance.

LAN failover: The two appliances share health information over the network via the VRRP protocol. These VRRP heartbeats occur at layer two and are performed on all configured VLANs. When no advertisements reach the spare on all VLANs, it will trigger a failover. When the warm spare begins receiving VRRP heartbeats again, it will relinquish the active role back to the primary appliance.

DHCP Synchronization 

The MXs in a routed mode high-availability pair exchange DHCP state information over the LAN on UDP port 3483. This prevents a DHCP IP address from being handed out to a client after a failover if it has already been assigned to another client prior to the failover.

Requirements and Best Practices 

When configuring routed HA, it is critical that both MXs have a reliable connection to each other on the LAN, so the heartbeats of the primary MX can be seen reliably by the spare. To ensure this connection is reliable:

• The two MXs should be connected to each other through a downstream switch (or ideally, multiple switches) on the LAN to allow for passing VRRP heartbeats.
  • There should be no more than one additional hop between them, and they must be able to communicate on all VLANs.
  • Make sure Spanning-Tree Protocol (STP) is enabled on the downstream switching infrastructure, as a properly-configured HA topology will introduce a loop on the network.
• When first configuring routed HA, the spare should be added and configured in the dashboard before the device is physically deployed, so it will immediately fetch its configuration and behave appropriately.
• Ensure that both MX's have their own uplink IP address for dashboard connectivity as mentioned here.
  • If a virtual IP is being used, an additional IP address is needed, and all three IP's must be in the same subnet.

Recommended Topologies

There are two physical architectures available for routed warm spare deployments.

Troubleshooting Routed Warm Spare

If there is a problem with the routed HA configuration, there may be various symptoms that will affect the network, and it may not be obvious that the root cause is routed HA. This section outlines what issues with HA typically look like, as well as recommended troubleshooting steps.

Dual Active Issue

The most common sign of a problem with routed HA is a dual active scenario where both the primary and spare MX report in the dashboard as being active. This can be observed in the dashboard under Security & SD-WAN > Monitor > Appliance status and by comparing the current state of each appliance.

This will occur if the primary MX is online and sending heartbeats that aren't seen by the spare, resulting in the spare thinking that the primary is down. If both the primary and spare are in the active state, this will cause various issues with the network, affecting DHCP, routing, VPN, etc.