Skip to main content
Cisco Meraki

Border Gateway Protocol (BGP)

  1. Last updated
  2. Save as PDF

Overview

All MX security appliances support the ability to communicate AutoVPN route information using BGP. This guide introduces key concepts, how BGP is implemented on MX appliances, and how to configure BGP.

 

Mixed operation of MX14 and MX15/16 networks is NOT supported 

Key Concepts

Before deploying BGP, it is important to understand several key concepts.

Concentrator Mode

All MXs can be configured in either NAT or VPN concentrator mode. There are important considerations for both modes. For more detailed information on concentrator modes, click here.

One-Armed Concentrator

In this mode, the MX is configured with a single Ethernet connection to the upstream network. All traffic will be sent and received on this interface. This is the recommended configuration for MX appliances serving as VPN termination points into the datacenter.

NAT Mode

  • iBGP establishes relationships over AutoVPN and will establish and exchange routes between:
    • A BGP peer acting as a One-Armed Concentrator in the DC and-
    • A NAT mode MX.
  • eBGP peer relationships are not supported on NAT Mode MX devices. eBGP is only supported on one-armed (pass-through) concentrators.

VPN Topology

There are several options available for the structure of the VPN deployment.

Hub and Spoke

In a hub and spoke configuration, the MX security appliances at the branches and remote offices connect directly to specific MX appliances and will not form tunnels to other MX or Z-series devices in the organization. Communication between branch sites or remote offices is available through the configured VPN hubs. This is the recommended VPN topology for most deployments.

VPN Mesh

In a mesh configuration, an MX appliance at the branch or remote office is configured to connect directly to any other MXs in the organization that are also in mesh mode, as well as any spoke MXs that are configured to use it as a hub.

Datacenter Redundancy (DC-DC Failover)

Deploying one or more MXs to act as VPN concentrators in additional datacenters provides greater redundancy for critical network services. In a dual- or multi-datacenter configuration, identical subnets can be advertised from each datacenter with a VPN concentrator mode MX. 

 

In a DC-DC failover design, a spoke site will form VPN tunnels to all VPN hubs that are configured for that site. For subnets that are unique to a particular hub, traffic will be routed directly to that hub so long as tunnels between the spoke and hub are established successfully. For subnets that are advertised from multiple hubs, spokes sites will send traffic to the highest priority hub that is reachable.

 

For information on configuring DC-DC Failover, please see this article.

Warm Spare (High Availability) for VPN concentrators

When configured for high availability (HA), one MX serves as the primary unit and the other MX operates in a spare mode. All traffic flows through the primary MX, while the spare operates as an added layer of redundancy in the event of failure.

 

Failover between MXs in an HA configuration leverages VRRP heartbeat packets. These heartbeat packets are sent from the Primary MX to the Spare MX via the singular uplink for MXs operating in VPN concentrator mode in order to indicate that the Primary is online and functioning properly. As long as the Spare is receiving these heartbeat packets, it functions in the passive state. If the Spare stops receiving these heartbeat packets, it will assume that the Primary is offline and will transition into the active state. In order to receive heartbeats in a one-armed concentrator configuration, both VPN concentrator MXs should have uplinks on the same subnet within the datacenter.

 

For Routed mode configurations, both concentrators must be able to communicate using the LAN ports. More information on Routed mode warm spare can be found here.

 

Only one MX license is required for the HA pair, as only a single device is in full operation at any given time.

BGP Terminology

Border Gateway Protocol (BGP) is a highly scalable dynamic routing protocol that is used to exchange routing information between and within autonomous systems (AS).

Autonomous System (AS)

An autonomous system is a network, or group of networks, under a common administration and with common routing policies.

AS Path

The different autonomous systems traffic passes through to arrive at a specific router is its AS path. The AS path is a property of each route.

iBGP

If BGP is used to exchange routes within an autonomous system, then the protocol is referred to as Interior BGP (iBGP).

eBGP

When BGP is used between different autonomous systems, then the protocol is referred to as External BGP (eBGP).

Implementation

MX appliances use iBGP to exchange route information over Meraki AutoVPN. MXs deployed as one-armed VPN concentrator use eBGP to exchange and learn routes within the datacenter. Learned routes are redistributed to AutoVPN spokes using iBGP.

iBGP

On the Security & SD-WAN > Configure > Site-to-site VPN settings page, BGP configuration is available for one-armed VPN concentrator MXs. When BGP is toggled to enabled, the VPN BGP AS  (this is an organization-wide setting) and iBGP Holdtimer can be set.

Screen Shot 2021-01-12 at 2.16.05 PM.png

This AS number will be used for iBGP. Configuring this AS number will automatically set all other MXs in the organization to use the same AS number.

 

iBGP sessions are automatically established between MX appliances over their Meraki AutoVPN tunnels.

eBGP

On the Security & SD-WAN > Configure > Site-to-site VPN settings page, BGP configuration is available for one-armed VPN concentrator Hub MXs. When BGP is toggled to enabled, BGP neighbors can be configured

 

To configure an eBGP neighbor, click "Add a BGP neighbor." Then, configure the Neighbor IP and Remote AS.

Screen Shot 2021-01-12 at 2.16.52 PM.png

The BGP neighbor must be configured to peer with the one-armed VPN concentrator using its WAN1 IP and the organization-wide VPN BGP AS number.

MXs WAN 1 IP address:
clipboard_e54833f0e4fc0bf601df128b99f98fe88.png

For eBGP multi-hop, this option is configured per neighbor. This value can be adjusted to peer the concentrator with something multiple hops away in the data center or cloud.  If multihop is used AND the eBGP peer is also advertising the IP route that the MX is using to connect to the eBGP peer, 10.101.0.0/24 in the above example.  Then this route MUST be added to the list of 'Local Networks' in the 'VPN settings' section above the 'BGP settings' section of the 'Site-to-site VPN' page, as shown below:

Screenshot 2020-12-08 at 12.33.08.png

  This is to avoid the more specific path to the eBGP peer being learned by the MX and then being lost in the event of the peer relationship dropping.

If the one-armed VPN concentrator is deployed in a warm spare (high availability) configuration, the virtual IP (vIP) will be used instead of the WAN1 IP address.

Default Update and Timeout Timers

The MX attempts to negotiate its hold time/update time at 240 seconds and its keep-alive time at 80 seconds. These timers get negotiated down to the lowest value between the pair, depending on what the other side of the eBGP pair has set. If the non-MX side of the pair is set for 180/60, the MX will negotiate down to 180/60. If the non-MX side is set for 300/100, the non-MX side will negotiate down to 240/80.

 

This value can be adjusted for faster failovers. For the iBGP VPN hold time the default as specified is 240. The configuration option on the concentrator site to site VPN page can be adjusted and this value applies to all Auto VPN peers we have iBGP connections to. For the eBGP hold timer this is per neighbor and can be more aggressive since this is monitoring a local link. When adjusting either value it is important to take into account load on the MX concentrator. 

Platform Sizing Considerations

For Z3/c platforms we recommend only advertising a handful of aggregates or a default route to the datacenter. For the rest of the MX Security & SD-WAN appliances we recommend sizing the BGP topology to 1500 prefixes from the DC. This is due to the fact that smaller appliances may also be running other features such as Advanced Security and Meraki Insight etc.

Route Table Integrity

In order to maintain integrity of the route table for all MXs in the SD-WAN fabric, Meraki has implemented protection both inbound and outbound from the vpn concentrator. To protect the integrity of the route tables inbound, there is a configurable Receive limit. In addition to protecting the integrity inbound, there is also an AS Path ACL that is placed outbound for all eBGP peers. This AS Path ACL ensures that the Meraki ASN is always the originating ASN. This ensures that Merakis SD WAN fabric will never be transit between two DCs. By default this is on and can be disabled by clicking the checkbox under Allow transit. 

 

Screen Shot 2020-08-17 at 11.38.21 AM.png

Route Advertisement Behavior

Routes are advertised to eBGP and iBGP peers in the following conditions:

  • A VPN spoke will learn routes advertised to it by other AutoVPN peers.
  • A one-armed VPN concentrator will learn routes advertised to it by other AutoVPN peers and re-advertise these iBGP learned routes to eBGP peers
  • A one-armed VPN concentrator will learn routes advertised to it by its eBGP peers and re-advertise these eBGP learned routes to other AutoVPN peers via iBGP
  • A one-armed VPN concentrator will advertise local networks which are not directly connected and are configured on the site-to-site VPN settings page via iBGP, but not via eBGP to external peers

 

The following scenarios will illustrate how routes are learned and advertised through both Internal and External BGP.

Note: On MX-Z devices, traffic for the following services and tools will adhere to the route priority outlined in our MX Routing Behavior article

  • Advanced Malware Protection registration

  • Meraki Cloud Authentication

  • Meraki Cloud Communication on TCP ports 80, 443, and 7734

  • Ping and Dashboard Throughput Live Tools

  • List Updates for the following services

    • Content Filtering

    • IDS/IPS Rule Updates

    • Geo-IP Lists for Layer 7 Country-Based Firewall Rules


Furthermore, if an MX is configured for eBGP and receives a route that overlaps with our cloud connectivity network ranges, the MX’s cloud management traffic will follow that BGP route, so it is imperative that the MX, as well as it’s eBGP peer, have connectivity to everything listed on the Help > Firewall Info page in this scenario.

Scenario 1: Single iBGP and eBGP Peers

Consider the following topology:

 

  • An MX configured as a one-armed VPN concentrator
  • Another datacenter router with an established eBGP session with the one-armed concentrator
  • A VPN spoke connected to the one-armed concentrator over AutoVPN. The two have an established iBGP session.
  • 10.255.0.0/24 is a subnet that exists in the datacenter and is advertised over eBGP
  • 172.17.7.0/24 is a subnet that exists in a remote branch and is advertised over iBGP

Scenario 1.png

In this scenario:

  • The One-armed Concentrator MX will learn 172.17.7.0/24 via iBGP from the VPN Spoke MX
  • The One-armed Concentrator MX will learn 10.255.0.0/24 via eBGP from the BGP Peer
  • The VPN Spoke MX will learn of 10.255.0.0/24 via iBGP from the One-armed Concentrator MX
  • The BGP Peer will learn 172.17.7.0/24 via eBGP from the One-armed Concentrator MX

Scenario 2: Multiple eBGP Peers

Consider the following topology:

 

  • An MX configured as a one-armed VPN concentrator
  • Two datacenter routers, "BGP Peer A" and "BGP Peer B" with established eBGP sessions with the one-armed VPN concentrator.
  • A VPN spoke connected to the one-armed concentrator over AutoVPN. The two have an established iBGP session.
  • 10.255.0.0/24 is a subnet that exists in the datacenter and is advertised over eBGP through BGP peer A
  • 192.168.4.0/24 is a subnet that exists in the datacenter and is advertised over eBGP through BGP peer B
  • 172.17.7.0/24 is a subnet that exists in a remote branch and is advertised over iBGP

Scenario 2.png

In this scenario:

  • The One-armed Concentrator MX will learn 172.17.7.0/24 via iBGP from the VPN Spoke MX
  • The One-armed Concentrator MX will learn 10.255.0.0/24 via eBGP from the BGP Peer A
  • The One-armed Concentrator MX will learn 192.168.4.0/24 via eBGP from BGP Peer B
  • The VPN Spoke MX will learn of 10.255.0.0/24 and 192.168.4.0/24 via iBGP from the One-armed Concentrator MX
  • Both BGP Peer A and BGP Peer B will learn 172.17.7.0/24 via eBGP from the One-armed Concentrator MX
  • BGP Peer A will also learn 192.168.4.0/24 from its eBGP session with the One-armed Concentrator MX
  • BGP Peer B will also learn 10.255.0.0/24 from its eBGP session with the One-armed Concentrator MX

Scenario 3: Datacenter Redundancy (DC-DC Failover)

Consider the following topology:

 

  • A VPN Spoke with a local subnet of 172.17.7.0/24
  • Two datacenters set up with a DC-DC Failover configuration:
    • The VPN Spoke MX establishes VPN to a one-armed VPN Concentrator in each datacenter  
    • The same subnet is advertised to the spoke through both datacenters
    • Communication to the datacenters is an active/passive failover configuration.
      • The active hub is defined based on it's configured hub priority. The one-armed concentrator in the primary datacenter has the highest priority in this scenario.
        • If the VPN Spoke MX were configured in a VPN mesh configuration, the organization-wide VPN concentrator priority would act as the hub priority in this case.
  • The primary datacenter has subnets of 10.0.0.0/8 and 192.168.80.0/24
  • The primary datacenter has an ASN of 1111
  • The one-armed VPN concentrator in the Primary datacenter has an active eBGP session with BGP Peer A
  • The secondary datacenter has subnets of 10.0.0.0/8 and 192.168.1.0/24
  • The secondary datacenter has an ASN of 2222
  • The one-armed VPN concentrator in the Secondary datacenter has an active eBGP session with BGP Peer B
  • The VPN Spoke MX is connected to the One-armed VPN Concentrator MXs in both datacenters. There are active iBGP sessions between the spoke MX and both concentrators.
  • The organization-wide AutoVPN ASN is 8888

Scenario 3(1).png

In this scenario:

 

Primary DC

  • The One-armed Concentrator MX will learn 10.0.0.0/8 and 192.168.80.0/24 via eBGP from BGP Peer A.
  • The One-armed Concentrator MX will learn 172.17.7.0/24 via iBGP from the VPN Spoke MX.
  • BGP Peer A will learn 172.17.7.0/24 via eBGP from the One-armed Concentrator MX

 

Secondary DC

  • The One-armed Concentrator MX will learn 10.0.0.0/8 and 192.168.1.0/24 via eBGP from BGP Peer B.
  • The One-armed Concentrator MX will learn 172.17.7.0/24 via iBGP from the VPN Spoke MX.
    • Routes learned from the VPN Spoke MX by the One-armed Concentrator MX in the secondary DC will have an additional ASN (8888) pre-pended
  • BGP Peer B will learn 172.17.7.0/24 via eBGP from the One-armed Concentrator MX
    • Routes that are learned via iBGP by the One-armed VPN Concentrator in the secondary DC and re-advertised via eBGP will have an AS Path of 8888 8888

 

VPN Spoke

  • The VPN Spoke MX will learn 10.0.0.0/8 and 192.168.80.0/24 from the primary One-armed Concentrator MX
  • The VPN Spoke MX will learn 10.0.0.0/8 and 192.168.1.0/24 from the secondary One-armed Concentrator MX
  • The 10.0.0/8 route was learned from two different sources. Since the One-armed Concentrator MX in the secondary datacenter has a lower hub priority, the 10.0.0.0/8 route learned from the primary is preferred
  • This path preference is achieved via AS path pre-pending, this means that:
    • Routes from the primary One-armed Concentrator MX will have an AS path of 8888 
    • Routes from the secondary One-armed Concentrator MX will have a less desirable AS path of 8888 8888 and the primary route will be preferred
    • This pattern continues for any additional hubs
  1. Back to top
  • Was this article helpful?

Recommended articles

  1. Article type
    Topic
    Stage
    Final
  2. Tags
    1. bgp
    2. eBGP
    3. ibgp
    4. is_translated