Border Gateway Protocol (BGP)

Last updated
Save as PDF

Overview

All MX security appliances support the ability to communicate AutoVPN route information using BGP. This guide introduces key concepts, how BGP is implemented on MX appliances, and how to configure BGP.

Mixed operation of MX14 and MX15/16 networks is NOT supported.
BGP is not supported for template networks.
IPv6 eBGP is currently only supported in One-Armed Concentrator mode.
eBGP is supported for passthrough-mode hubs and routed-mode hubs running MX 18.207 or higher firmware.
- When establishing an eBGP peering on the WAN in routed mode, NAT must be disabled on that uplink (see NAT Exceptions with Manual Inbound Firewall) otherwise the subnets advertised by the MX will be behind NAT and unreachable via eBGP subnets upstream.
eBGP is currently not supported on an uplink using PPPoE encapsulation.

Key Concepts

Before deploying BGP, it is important to understand several key concepts.

Deployment Mode

All MXs can be configured in either Routed or VPN concentrator mode. There are important considerations for both modes. For more detailed information on concentrator modes, click here.

One-Armed Concentrator

In this mode, the MX is configured with a single Ethernet connection to the upstream network. All traffic will be sent and received on this interface. This is the recommended configuration for MX appliances serving as VPN termination points into the datacenter.

Routed Mode

iBGP establishes relationships over AutoVPN and will establish and exchange routes between AutoVPN peers.

eBGP peer relationships are supported on both the WAN and LAN of the MX, peering is bi-directional and routes will be automatically redistributed into AutoVPN.

VPN Topology

There are several options available for the structure of the VPN deployment.

Hub and Spoke

In a hub and spoke configuration, the MX security appliances at the branches and remote offices connect directly to specific MX appliances and will not form tunnels to other MX or Z-series devices in the organization. Communication between branch sites or remote offices is available through the configured VPN hubs. This is the recommended VPN topology for most deployments.

VPN Mesh

In a mesh configuration, an MX appliance at the branch or remote office is configured to connect directly to any other MXs in the organization that are also in mesh mode, as well as any spoke MXs that are configured to use it as a hub.

Datacenter Redundancy (DC-DC Failover)

Deploying one or more MXs to act as VPN concentrators in additional datacenters provides greater redundancy for critical network services. In a dual- or multi-datacenter configuration, identical subnets can be advertised from each datacenter with a VPN concentrator mode MX.

In a DC-DC failover design, a spoke site will form VPN tunnels to all VPN hubs that are configured for that site. For subnets that are unique to a particular hub, traffic will be routed directly to that hub so long as tunnels between the spoke and hub are established successfully. For subnets that are advertised from multiple hubs, spokes sites will send traffic to the highest priority hub that is reachable.

For information on configuring DC-DC Failover, please see this article.

Warm Spare (High Availability) for VPN concentrators

When configured for high availability (HA), one MX serves as the primary unit and the other MX operates in a spare mode. All traffic flows through the primary MX, while the spare operates as an added layer of redundancy in the event of failure.

Failover between MXs in an HA configuration leverages VRRP heartbeat packets. These heartbeat packets are sent from the Primary MX to the Spare MX via the singular uplink for MXs operating in VPN concentrator mode in order to indicate that the Primary is online and functioning properly. As long as the Spare is receiving these heartbeat packets, it functions in the passive state. If the Spare stops receiving these heartbeat packets, it will assume that the Primary is offline and will transition into the active state. In order to receive heartbeats in a one-armed concentrator configuration, both VPN concentrator MXs should have uplinks on the same subnet within the datacenter.

For Routed mode configurations, both concentrators must be able to communicate using the LAN ports. More information on Routed mode warm spare can be found here.

Only one MX license is required for the HA pair, as only a single device is in full operation at any given time.

BGP Terminology

Border Gateway Protocol (BGP) is a highly scalable dynamic routing protocol that is used to exchange routing information between and within autonomous systems (AS).

Autonomous System (AS)

An autonomous system is a network, or group of networks, under a common administration and with common routing policies.

AS Path

The different autonomous systems traffic passes through to arrive at a specific router is its AS path. The AS path is a property of each route.

iBGP

If BGP is used to exchange routes within an autonomous system, then the protocol is referred to as Interior BGP (iBGP).

eBGP

When BGP is used between different autonomous systems, then the protocol is referred to as External BGP (eBGP).

Implementation

MX appliances use iBGP to exchange route information over Meraki AutoVPN. MXs deployed as one-armed VPN concentrator use eBGP to exchange and learn routes within the datacenter. Learned routes are redistributed to AutoVPN spokes using iBGP.

iBGP

BGP is configured on the Security & SD-WAN > Configure > Routing settings page. It is a requirement that the MX be participating in AutoVPN for BGP to be enabled. When BGP is toggled to enabled, the VPN BGP AS (this is an organization-wide setting) and iBGP Holdtimer can be set.

Screenshot of iBGP configuration. BGP is set to enabled, the BGP VPN AS number is set to 64513, and the iBGP holdtimer is 240.

This AS number will be used for iBGP. Configuring this AS number will automatically set all other MXs in the organization to use the same AS number. Any change to the BGP VPN AS number applies to all MXs in the dashboard organization and will require them to fetch a new configuration.

iBGP sessions are automatically established between MX appliances over their Meraki AutoVPN tunnels.

eBGP

The Security & SD-WAN > Configure > Routing settings page, can be used to configure eBGP neighbors when BGP is toggled to enabled.

To configure one, click "Add a BGP neighbor," then configure the Neighbor IP and Remote AS and Source Interface.

Screenshot of the full dashboard BGP configuration options, including eBGP neighbors.

The eBGP neighbor must be configured to peer with the MX using the correlating interface IP and the organization-wide VPN BGP AS number.

Passthrough Mode eBGP multi-hop consideration

For passthrough mode eBGP multi-hop, this option is configured per neighbor. This value can be adjusted to peer the concentrator with something multiple hops away in the data center or cloud. If multihop is used AND the eBGP peer is also advertising the IP route that the MX is using to connect to the eBGP peer, 10.100.0.0/24 in the above example. Then this route MUST be added to the list of 'Local Networks' in the 'VPN settings' section above the 'BGP settings' section of the 'Site-to-site VPN' page, as shown below:

Screenshot showing the VPN settings > Local networks config for an MX in passthrough mode. Since multi-hop is used and the eBGP peer is advertising the 10.100.0.0/24 route that the MX is using the connect the eBGP peer, this same route is added on this page.

This is to avoid the more specific path to the eBGP peer being learned by the MX and then being lost in the event of the peer relationship dropping.

If the MX is deployed in a warm spare (high availability) configuration, the Virtual IP (VIP) will be used instead of the WAN IP address.

Default Update and Timeout Timers

The MX attempts to negotiate its hold time/update time at 240 seconds and its keep-alive time at 80 seconds. These timers get negotiated down to the lowest value between the pair, depending on what the other side of the eBGP pair has set. If the non-MX side of the pair is set for 180/60, the MX will negotiate down to 180/60. If the non-MX side is set for 300/100, the non-MX side will negotiate down to 240/80.

This value can be adjusted for faster failovers. For the iBGP VPN hold time the default as specified is 240. The configuration option on the concentrator site to site VPN page can be adjusted and this value applies to all Auto VPN peers we have iBGP connections to. For the eBGP hold timer this is per neighbor and can be more aggressive since this is monitoring a local link. When adjusting either value it is important to take into account load on the MX concentrator.

Platform Sizing Considerations

For Z3/c platforms we recommend only advertising a handful of aggregates or a default route to the datacenter. For the rest of the MX Security & SD-WAN appliances we recommend sizing the BGP topology to 1500 prefixes from the DC. This is due to the fact that smaller appliances may also be running other features such as Advanced Security and Meraki Insight etc.

Route Table Integrity

In order to maintain integrity of the route table for all MXs in the SD-WAN fabric, Meraki has implemented protection both inbound and outbound from the MX. To protect the integrity of the route tables inbound, there is a configurable Receive limit. In addition to protecting the integrity inbound, there is also an AS Path ACL that is placed outbound for all eBGP peers. This AS Path ACL ensures that the Meraki ASN is always the originating ASN. This ensures that Merakis SD WAN fabric will never be transit between two DCs. By default this is on and can be disabled by clicking the checkbox under Allow transit.

BGP Path Selection Attributes

eBGP attributes AS Path Prepend, MED and Weight require MX 19.1.4 or greater firmware.

Meraki provides the AS Path Prepend, Multi-Exit discriminator (MED) and Weight attributes to influence route propagation and path selection between autonomous systems, these optional attributes are applied on a per neighbor basis.

AS Path Prepending is used to influence traffic inbound by artificially extending the as AS path length. When routes are advertised to a peer, additional AS numbers are appended to the existing AS path. As BGP prefers shorter AS paths when calculating BGP best path algorithm, AS Path Prepending is able to influence advertised routes appearing appear less or more desirable to a peer. The AS Path Prepend list supports up to 10, 4 byte unsigned, space separated AS numbers.

Multi-Exit discriminator (MED) is used to influence traffic inbound by signaling a suggested entry point to a peer, When advertising routes to a peer an additional attribute value is included with the routes advertised. The med value is a 4 byte unsigned number, a lower MED value is preferred to a higher one.

Weight is a Cisco propriety metric used locally influence inbound route priority. As weight is locally significant, it is not advertised to external peers. As each MX can assign it's own weight to a peer, it gives the ability to influence the sites routing path to a common peer but not the entire AS. Weight is a value from 0 to 49 with a higher weight being a more preferred path.

Neighbor Security

Meraki offers two forms of neighbor security, TTL Security and MD5 password.

TTL Security

TTL security is a technique that verifies the legitimacy of BGP updates by using the Time-to-Live (TTL) field in IP packets. Administrators can modify the TTL value in BGP packets, this way they verify if the packet has passed through the expected number of routers. The MX will drop BGP updates with unexpected TTL values in an attempt to help prevent against certain attacks.

MD5 Password

Message Digest5 (MD5) password in BGP authentication provides an additional layer of security by ensuring that only authorized peers can establish BGP sessions, reducing the risk of unauthorized route updates. With MD5 hashing, BGP routers exchange a hash value derived from a shared key during the session establishment process. This hash value is used to verify the authenticity of subsequent BGP messages exchanged between the routers.

Route Advertisement Behavior

Routes are advertised to eBGP and iBGP peers in the following conditions:

A routed mode eBGP participant will learn routes advertised to it by other AutoVPN peers and re-advertise these iBGP learned routes to eBGP peers
A routed mode eBGP participant will learn routes advertised to it by eBGP peers and re-advertise these eBGP learned routes to other AutoVPN peers via iBGP
A VPN concentrator will advertise local networks which are not directly connected and are configured on the site-to-site VPN settings page via iBGP. These local networks will not be advertised via eBGP to external peers.
A VPN concentrator will advertise local Client VPN/AnyConnect VPN networks via iBGP and eBGP to external peers as long as the subnets are VPN Mode 'Enabled' on the Site-to-Site VPN page.

Note: On routed mode MXs, the MX will only advertise those routes to its eBGP peer that are participating in AutoVPN. If a particular route on the Security & SD-WAN > Site to Site VPN page for any network has VPN mode set to “Disabled” this route will not be propagated via BGP. Static route redistribution is currently not supported on the MX platform.

The following scenarios will illustrate how routes are learned and advertised through both Internal and External BGP.

Note: On MX-Z devices, traffic for the following services and tools will adhere to the route priority outlined in our MX Routing Behavior article

Advanced Malware Protection registration
Meraki Cloud Authentication
Meraki Cloud Communication on TCP ports 80, 443, and 7734
Ping and Dashboard Throughput Live Tools
List Updates for the following services
- Content Filtering
- IDS/IPS Rule Updates
- Geo-IP Lists for Layer 7 Country-Based Firewall Rules

Furthermore, if an MX is configured for eBGP and receives a route that overlaps with our cloud connectivity network ranges, the MX’s cloud management traffic will follow that BGP route, so it is imperative that the MX, as well as it’s eBGP peer, have connectivity to everything listed on the Help > Firewall Info page in this scenario.

Scenario 1: One-Armed Concentrator mode - Single iBGP and eBGP Peers

Consider the following topology:

An MX configured as a one-armed VPN concentrator
Another datacenter router with an established eBGP session with the one-armed concentrator
A VPN spoke connected to the one-armed concentrator over AutoVPN. The two have an established iBGP session.
10.255.0.0/24 is a subnet that exists in the datacenter and is advertised over eBGP
172.17.7.0/24 is a subnet that exists in a remote branch and is advertised over iBGP

Diagram of the topology described above (One-armed concentrator MX > BGP Peer > Edge Firewall > Internet > VPN Spoke).

In this scenario:

The One-armed Concentrator MX will learn 172.17.7.0/24 via iBGP from the VPN Spoke MX
The One-armed Concentrator MX will learn 10.255.0.0/24 via eBGP from the BGP Peer
The VPN Spoke MX will learn of 10.255.0.0/24 via iBGP from the One-armed Concentrator MX
The BGP Peer will learn 172.17.7.0/24 via eBGP from the One-armed Concentrator MX

Scenario 2: One-Armed Concentrator mode - Multiple eBGP Peers

Consider the following topology:

An MX configured as a one-armed VPN concentrator
Two datacenter routers, "BGP Peer A" and "BGP Peer B" with established eBGP sessions with the one-armed VPN concentrator.
A VPN spoke connected to the one-armed concentrator over AutoVPN. The two have an established iBGP session.
10.255.0.0/24 is a subnet that exists in the datacenter and is advertised over eBGP through BGP peer A
192.168.4.0/24 is a subnet that exists in the datacenter and is advertised over eBGP through BGP peer B
172.17.7.0/24 is a subnet that exists in a remote branch and is advertised over iBGP

Diagram of the topology described above. One-armed concentrator is connected to two separate eBGP peers instead of one.

In this scenario:

The One-armed Concentrator MX will learn 172.17.7.0/24 via iBGP from the VPN Spoke MX
The One-armed Concentrator MX will learn 10.255.0.0/24 via eBGP from the BGP Peer A
The One-armed Concentrator MX will learn 192.168.4.0/24 via eBGP from BGP Peer B
The VPN Spoke MX will learn of 10.255.0.0/24 and 192.168.4.0/24 via iBGP from the One-armed Concentrator MX
Both BGP Peer A and BGP Peer B will learn 172.17.7.0/24 via eBGP from the One-armed Concentrator MX
BGP Peer A will also learn 192.168.4.0/24 from its eBGP session with the One-armed Concentrator MX
BGP Peer B will also learn 10.255.0.0/24 from its eBGP session with the One-armed Concentrator MX

Scenario 3: One-Armed Concentrator mode - Datacenter Redundancy (DC-DC Failover)

Consider the following topology:

A VPN Spoke with a local subnet of 172.17.7.0/24
Two datacenters set up with a DC-DC Failover configuration:
- The VPN Spoke MX establishes VPN to a one-armed VPN Concentrator in each datacenter
- The same subnet is advertised to the spoke through both datacenters
- Communication to the datacenters is an active/passive failover configuration.
  - The active hub is defined based on it's configured hub priority. The one-armed concentrator in the primary datacenter has the highest priority in this scenario.
    - If the VPN Spoke MX were configured in a VPN mesh configuration, the organization-wide VPN concentrator priority would act as the hub priority in this case.
The primary datacenter has subnets of 10.0.0.0/8 and 192.168.80.0/24
The primary datacenter has an ASN of 1111
The one-armed VPN concentrator in the Primary datacenter has an active eBGP session with BGP Peer A
The secondary datacenter has subnets of 10.0.0.0/8 and 192.168.1.0/24
The secondary datacenter has an ASN of 2222
The one-armed VPN concentrator in the Secondary datacenter has an active eBGP session with BGP Peer B
The VPN Spoke MX is connected to the One-armed VPN Concentrator MXs in both datacenters. There are active iBGP sessions between the spoke MX and both concentrators.
The organization-wide AutoVPN ASN is 8888

Diagram of the topology described above. This time, there are two one-armed concentrator MXs deployed in a DC-DC failover setup.

In this scenario:

Primary DC

The One-armed Concentrator MX will learn 10.0.0.0/8 and 192.168.80.0/24 via eBGP from BGP Peer A.
The One-armed Concentrator MX will learn 172.17.7.0/24 via iBGP from the VPN Spoke MX.
BGP Peer A will learn 172.17.7.0/24 via eBGP from the One-armed Concentrator MX

Secondary DC

The One-armed Concentrator MX will learn 10.0.0.0/8 and 192.168.1.0/24 via eBGP from BGP Peer B.
The One-armed Concentrator MX will learn 172.17.7.0/24 via iBGP from the VPN Spoke MX.
- Routes learned from the VPN Spoke MX by the One-armed Concentrator MX in the secondary DC will have an additional ASN (8888) pre-pended
BGP Peer B will learn 172.17.7.0/24 via eBGP from the One-armed Concentrator MX
- Routes that are learned via iBGP by the One-armed VPN Concentrator in the secondary DC and re-advertised via eBGP will have an AS Path of 8888 8888

VPN Spoke

The VPN Spoke MX will learn 10.0.0.0/8 and 192.168.80.0/24 from the primary One-armed Concentrator MX
The VPN Spoke MX will learn 10.0.0.0/8 and 192.168.1.0/24 from the secondary One-armed Concentrator MX
The 10.0.0/8 route was learned from two different sources. Since the One-armed Concentrator MX in the secondary datacenter has a lower hub priority, the 10.0.0.0/8 route learned from the primary is preferred
This path preference is achieved via AS path pre-pending, this means that:
- Routes from the primary One-armed Concentrator MX will have an AS path of 8888
- Routes from the secondary One-armed Concentrator MX will have a less desirable AS path of 8888 8888 and the primary route will be preferred
- This pattern continues for any additional hubs

Scenario 4: Routed mode - eBGP peer on WAN of spoke

Consider the following topology:

VPN Spoke MX is configured in No-NAT routed mode.
VPN Spoke MX site is a site with both direct internet access and private network backhaul.
VPN Spoke MX and DC edge both have an eBGP peering with a service provider.

Diagram of the topology described above. VPN spoke and one armed concentrators are both connected to a private WAN network to facilitate eBGP peering.

In this scenario:

VPN Spoke MX will learn 10.0.0.0/8 via iBGP from One Armed Concentrator MX
VPN Spoke MX will learn 10.0.0.0/8 via eBGP from eBGP peer AS 51010
One Armed Concentrator MX will learn 172.16.23.0/24 via iBGP from VPN Spoke MX

NOTE: If an uplink goes into a failed state because its Internet connectivity checks for WAN failover stop passing, MX devices will continue to use any eBGP-learned routes over the uplink in question if they still have connectivity to the peer advertising them. This is done under the assumption that they are still valid routes if the peer has not withdrawn them.

Scenario 5: Routed mode - eBGP peer on LAN

Consider the following topology:

DC Edge MX has 3x VPN spoke MXs
DC Edge MX has an eBGP peering with downstream network.

Diagram of BGP scenario 5 topology. Both spoke and hub MXs are in routed mode. The hub learns routes from eBGP peer, which are injected over AutoVPN via iBGP

In this scenario:

DC Edge MX will learn 10.0.0.0/8 via eBGP from eBGP peer AS 51010
eBGP peer AS 51010 will learn 172.16.20.0/24, 172.16.21.0/24 and 172.16.22.0/24 via eBGP from DC Edge MX
VPN Spoke MXs will learn 10.0.0.0/8 via iBGP from DC Edge MX