Best Practices for Meraki Firmware
Introduction to Meraki Firmware
Cisco Meraki has always prided itself on delivering powerful networking and IT solutions in a simple, easy to manage fashion. This extends to firmware management on Meraki devices. Traditionally, firmware management is a tedious, time-consuming, and risky procedure met with dread and loathing by the network administrator tasked with carrying out the upgrades, but Meraki works to limit this burden. Complexity has long plagued firmware management practices throughout the industry, spawning horror stories about experiences such as upgrades that went sideways because of a corrupted USB drive or late nights in data centers manually provisioning the new code.
Why the Cloud Makes Us Different
Meraki tackles the complex firmware issue by leveraging the power of Meraki’s cloud-based dashboard to allow for easy deployment and firmware scheduling. The dashboard provides unique insights into new features as they become available in new firmware releases.
From a security perspective, the benefits of the cloud are unparalleled. Today, new security vulnerabilities are constantly announced, and network infrastructure is not immune to exploits. To contain threats at this scale, flexibility and rapid software remediation is paramount. At Meraki, we have the power to immediately react to discovered exploits, patch the vulnerability, and make this firmware immediately available for customers to leverage.
Simplified Firmware Updates
In the early days of Meraki, the only firmware configuration required was to specify a convenient maintenance window for your network, such as late at night on a weekend, for example. As Meraki has grown alongside its customer base, we have incorporated tighter controls over firmware for customers who desire these while still maintaining the simplicity of cloud-based delivery. Customers can now manage firmware for each network in their organization by selecting which firmware runs on which network.
Meraki differentiates itself through its firmware delivery using the Meraki cloud platform, by providing an exceptionally swift and reliable way to deliver firmware upgrades. The results are evident in our users’ impressive firmware adoption rates. Even given the options for finer controls, the vast majority of our users adopt and run on our latest firmware builds almost immediately after stable release candidates are available. Our extensive testing and our beta adoption process ensures that we deliver reliable builds at a regular cadence, delivering up-to-date security and stability.
Meraki Firmware Conventions
Meraki firmware nomenclature is the same across products and consists of a major, minor, maintenance, and hotfix number as part of the name.
Meraki's version numbering scheme is designed to help users understand the different levels of updates to their firmware. Major versions are typically more significant than minor versions, and they should be carefully considered before upgrading. Minor versions and Maintenance versions are typically less significant, and they can be upgraded more easily.
The firmware version is named using the format given below:
<Product Name> <Major Version>.<Minor Version>. <Maintenance Version>. <Hotfix Version (if applicable)>.
Firmware Version Type | Description |
Major | A new major firmware version is released with the launch of new products, technologies and/or major features. |
Minor | A new minor firmware version is released throughout the lifespan of a major firmware version and may contain new features along with bug fixes. |
Maintenance | A new Maintenance firmware version is released throughout the lifespan of a minor firmware version and contains bug fix updates to each minor release version. |
Hotfix | A new hotfix firmware version contains urgent fixes for a minor firmware release version that cannot wait for, or be accommodated by the next Maintenance firmware version. |
MX appliances are using an updated version of this convention that will be further supported through the Dashboard UI and API in future updates.
- Major Versions (major changes, representing a family of feature releases)
- Minor Versions (represents the feature release within the major version and comes with new features)
- Point Release (takes minor version definition)
The user interface currently represents the minor and point release numbers together. For example, MX 18.1.05 will be represented as MX 18.105. Thus, MX 18.201 would represent a different minor version and feature set than any MX 18.1XX releases.
Meraki firmware release cycle consists of three stages during the firmware rollout process namely beta, release candidate (RC) and stable firmware. This cycle is covered in more detail in the Meraki Firmware Development Lifecycle section of this document.
Automated Firmware Upgrades
Meraki’s goal is to make networking simple and one of the ways that we do this is by automating firmware upgrades. In order to support the process of firmware maturity and to provide the most stable experience to customers, Meraki will schedule firmware upgrades for networks that meet the criteria for a firmware upgrade. All qualified networks receive automated upgrades. While this upgrade method does not require any additional input from the administrator, it may not be appropriate as a complete firmware management process, depending on the needs of your network.
When new firmware becomes available it will immediately be available on dashboard for an administrator to upgrade to. Though it will eventually be pushed to qualified networks via the automated upgrade process, the automated upgrade process does not happen immediately after release and is rolled out over time. The automated process can sometimes take weeks to occur on all networks, depending on certain factors.
Some factors that may affect the automated deployment time period include: potential conflicts between new and old firmware builds, the number of devices receiving the new build, or special configurations on critical devices or networks that require caution for upgrades. The primary considerations for Meraki when deploying firmware upgrades is to preserve maximum security, uptime, and compatibility. If any of these factors are at risk, Meraki may choose to wait to deploy until those risks have been resolved.
The firmware that is selected via the automatic upgrade process can be one of three release types; Stable, Stable Release Candidate, or Beta. When an automated firmware upgrade is released by Meraki, networks that are scheduled for automated upgrades will be moved to the latest version. Periodically, automated upgrades may occur for firmware versions that are beta, stable release candidate, or stable. Customers will be notified via email when these upgrades are scheduled.
Automatic upgrades for beta firmware releases will only be scheduled for customers that have enabled the 'Try beta firmware' option in Network-wide > Configure > General or who are already running an older beta firmware release.
While automated firmware upgrades are pushed out to all qualified networks over time, due to the potential delays mentioned above, a more manual process may be required for some organizations. If a network needs a more timely upgrade pattern, it is best for the organization administrators to schedule upgrade times manually on the Organization > Firmware Upgrades page in the dashboard.
Administrators and network alert recipients will be notified when an automated firmware upgrade is scheduled. By default, these upgrades are scheduled 1 to 2 weeks from the date of notification. Additionally, a notification banner within dashboard will be present for organization administrators after the upgrade has been scheduled. Networks that do not contain devices or where all devices are dormant will have upgrades scheduled immediately.
This firmware upgrade process cannot be opted out of as it is a core service provided by Meraki however the upgrade(s) may always be rescheduled.
Automated firmware upgrade decisions are made on a per-network basis. As a result, if an upgrade is to be deployed it may or may not be deployed to all networks in the organization with that device type.
Automated firmware upgrades do not occur on a fixed timetable. As a result, a network running older beta firmware may not be immediately upgraded to recently released beta firmware.
Some networks might not get a firmware upgrade scheduled due to various reasons. We recommend network administrators check all of their Dashboard networks periodically for available firmware upgrades and upgrade them manually to the latest firmware versions in such scenarios.
General Firmware Best Practices
Meraki was built on the promise of making management of devices intuitive, and this extends to Meraki firmware management. Thanks to the power of the Meraki dashboard, we are able to create and release high quality firmware that allows access to cutting-edge features and high quality, secure software. Out of the box, we recommend you let the simple, automatic and seamless updates work to your advantage. By default, your devices will be scheduled for updates when new firmware becomes available — firmware that has been robustly validated and tested before being deployed.
Meraki’s default firmware settings include:
-
no automatic beta firmware deployments
-
a default upgrade window
-
a default upgrade choice of Wednesdays
On average, Meraki deploys a new firmware version once a quarter for each product family, and this cadence ensures you get access to new features and functionalities as they become available, minimizing major changes between firmware versions to ensure high quality software.
Once you are scheduled for an automatic update, Meraki will notify you 2 weeks in advance of the scheduled upgrade and, within this two week time window, you have the ability to reschedule to a day and time of your choosing. We recommend selecting a time that is most convenient to your business needs, and if you want to, you can set this time as your default upgrade window under your general network settings.
If you want to take advantage of the most advanced and newest features, we recommend that you enable the “Try beta firmware” toggle. This will give you early access to the latest Meraki firmware after it has finished the full internal automated and manual testing process in our firmware development cycle. If you are running beta firmware, you get earlier access to new features, as well as the opportunity to provide feedback on these features before they become generally available! If you have any issues on the new beta firmware you can always roll back to the previous stable version, or the previously installed version if you roll back within 14 days.
Templates
With configuration templates it is possible to push a standard configuration against multiple sites at the same time. When a network is attached to a template, the firmware is controlled by the template. It is not possible to configure a network to use a different version of firmware than what the template is configured for. In certain cases Meraki Support is able to upgrade individual devices, but this should not be relied upon as this prevents normal upgrades in the future.
Best Practice for Multi-Branch Deployments
Now that we understand how the Meraki firmware system works, let's talk about how you can leverage this to confidently manage firmware on your network.
As shown in the diagram above, firmware should be rolled out in stages when managing a large-scale network. This approach allows you to test new features and verify stability in your production environment before rolling out new features globally.
Test Network(s) (Beta)
It is recommended to have designated network(s) to test beta firmware when released. Test networks can be a lab network or production network that is smaller but that also has enough devices to test new features. If there are production networks of different sizes in your organization, it is best to have an additional beta network of each size.
Why you Should Test Beta Firmware
Although all Meraki beta firmware undergoes rigorous testing as described in the beta release process, we recommend testing the new beta code in your designated test networks. This ensures the firmware is tested based on the needs of your unique environment and works without issues for real users.
Beta Feedback Mechanisms
If bugs are encountered during beta firmware rollout, you should contact Meraki Support to ensure the issue is documented internally, using our defined process. Opening a case will ensure the appropriate details are collected and presented to Meraki engineering teams for resolution. As part of resolution, you may be provided with a hotfix (also commonly referred to as a patch fix) to verify resolution. Once a fix is confirmed, it will be rolled into a new beta version after going through our firmware release process so customers can continue testing.
Release Candidate Network(s) (Validate)
Once beta firmware is tested, you can choose to wait until the major version reaches release candidate (RC) status or roll out beta firmware to the remaining networks if you are satisfied with the validated beta firmware. If you have a policy to only use stable firmware in production, then you can move onto the next step in the process, which is to roll out the RC firmware to designated RC networks. As mentioned in the firmware rollout process, RC is very close to stable and hence can be rolled out to a larger pool of networks in the production environment. During this phase, customers do not need to have an extensive test plan because, at this point, all new features have been tested and the focus is on widely rolling out the firmware through the network. If you are encountering problems with stable firmware,we recommend upgrading to the next release candidate to see if the problems continue. The release candidate will include many fixes that might resolve the problem.
Stable Release Network(s) (Full Deployment)
Once a firmware is marked as stable, customers can roll out firmware to all the remaining networks either using the firmware upgrades tool or, optionally, using the automatic upgrade process to roll out firmware. If you have followed our firmware best practice for validating and testing the current Stable Release, you can deploy with confidence that it will work well in your unique environment.
In the scenario where you find the new beta or release candidate firmware is functioning as required and you would like to use this version on your entire deployment, go ahead and deploy this version across your entire deployment - we strive to deliver high quality firmware at all stages of our development process. If you do run into issues after the deployment, you can always easily roll back to the previous major stable firmware version.
Best Practice for MR Firmware
As our wireless portfolio grows, Meraki continues to focus on delivering the high performance and high availability network that modern deployments require. To achieve this goal we focus on minimizing downtime during an upgrade, maintaining scheduling flexibility, and preserving the accuracy of your upgrade maintenance window. Most Meraki access points (APs) will reboot in less than 1 minute after an update, ensuring minimal disruption to the end user even if they need to do a firmware upgrade during working hours.
We are constantly working on improving the firmware upgrade experience and further minimizing network downtime. Please refer to the Access Point Firmware Upgrade Strategy for more details.
Additionally, when you are running a Meraki wireless network, it is important to keep a few things in mind to ensure you have a great Wi-Fi firmware deployment experience. First, make sure you keep all of your APs on a single firmware version. Many Wi-Fi features are depending on the same expected behavior among the access points. For example, if you are using L3 roaming, some different versions of firmware may not be compatible with each other for L3 roaming features in particular.
Second, when upgrading a wireless network, client devices with older drivers may have issues with new features. We test against over 100 unique client devices (including many different laptops, smartphones and legacy wireless devices with unique wireless chipsets) in our labs before shipping any wireless firmware, but it's a good idea to have a single test AP to validate clients that might be unique to your business environment. These may include a custom point of sale (POS) system or barcode scanner that is critical to your business. Again, testing these potentially unique clients on the latest Meraki beta wireless firmware is a best practice as it ensures that Meraki can be notified of any potential interoperability issue early in the development cycle.
In addition to these basic best practices, Meraki APs also include features unique to the product line that make large scale firmware updates better.
Best Practice for Large Scale Wireless Networks
Traditionally, when running large scale campus wireless networks, upgrading wireless firmware has been considered risky. Thanks to the agile and cloud-based firmware development process used by Meraki engineers, there are a few things you can do to make these deployments less risky. Even in the largest networks, the best practice with Meraki is to designate an isolated area of your network to test and validate the newest Meraki firmware. With a designated Meraki MR test area, you can get access to validate all Meraki wireless firmware in your physical environment. This allows you to engage with Meraki engineers directly, earlier in the software development process, so you can help provide feedback on new features and identify any potential issue that may affect your deployment.
In the example above we have designated the cafeteria and a large meeting area as our Meraki test area. This was done by moving the selected APs into their own dashboard network so they could be assigned a (beta) firmware version, separate from the main network(s). This also allows the APs to be rolled back to a stable version quickly, if needed, by simply moving the APs back to the main production dashboard network.
This part of our deployment is an ideal choice for a few reasons:
- The area includes six Meraki access points, which ensures we have a reasonable number of access points to test on
- The area provides us with a diverse group of client devices, as people will bring many different smartphones and laptops to this area
- Almost all employees frequent this area of the building at some point during the day
- Because this is not a business-critical area, the impact of a potential wireless issue will be more manageable to the users
Once you have validated and are comfortable with the current firmware in the test environment, you can confidently deploy the update to the rest of your network. It is important to note that, in this example, you may occasionally have some roaming issues as users navigate in and out of the designated test area, because the deployed firmware versions may be different, and roaming may not yet be seamless between the two versions.
Best Practice for MS Firmware
When you move farther up the networking stack to switching there are additional things you need to take into consideration. It’s always important to consider the topology of your switches as, when you drive closer to the network core and away from the access layer, the risk during a firmware upgrade increases. Because of this, in a larger switch-based network you should always start the upgrade closest to the access layer. Two unique aspects of managing Meraki switch firmware is that we support both:
-
Staged upgrades to allow you to upgrade in logical increments. For instance, starting from low-risk locations at the access layer and moving onto the higher risk core, and...
-
Automatic updates for switch stacks
When upgrading Meraki switches it is important that you allocate enough time in your upgrade window for each group or phase to ensure a smooth transition. Each upgrade cycle needs enough time to download the new version to the switches, perform the upgrade, allow the network to reconverge around protocols such as spanning tree and OSPF that may be configured in your network, and some extra time to potentially roll back if any issue is uncovered after the upgrade.
The high-level process for a switch upgrade involves the following:
-
The switch downloads the new firmware (time varies depending on your connection)
-
The switch starts a countdown of 20 minutes to allow any other switches downstream to finish their download
-
The switch reboots with its new firmware (about a minute)
-
Network protocols reconverge (varies depending on configuration)
Meraki MS devices use a “safe configuration” mechanism, which allows them to revert to the last good (“safe”) configuration in the event that a configuration change causes the device to go offline or reboot. During routine operation, if a device remains functional for a certain amount of time (30 minutes in most circumstances, or 2 hours on the MS after a firmware upgrade), a configuration is deemed safe.
When a device comes online for the first time or immediately after a factory reset, a new safe configuration file is generated since one doesn’t exist previously. Multiple reboots in quick succession during initial bringup may result in a loss of this configuration and failure to come online. In such events, a factory reset will be required to recover.
It is recommended to leave the device online for 2 hours for the configuration to be marked safe after the first boot or a factory reset.
Staged Upgrades
To make managing complex switched networks simpler, Meraki supports automatic staged firmware updates. This allows you to easily designate groups of switches into different upgrade stages. When you are scheduling your upgrades you can easily (as in the example below) mark multiple stages of upgrades. In this case, we started with the access layer switches in Stage 1 and gradually upgraded toward the core in Stage 3. Once you start the staged upgrade, the Stage 1 switches will complete the entire upgrade cycle before the Stage 2 upgrades start. This cycle will repeat until all the switches are upgraded in all three stages.
Upgrading a Switch Stack
In addition to supporting staged upgrades, Meraki also simplifies managing a switch stack. As part of our upgrade toolset, we automatically handle the upgrade of the entire switch stack. This upgrade manages the upload of firmware to each switch and takes care of each reboot within the switch stack. If you are upgrading switch stacks within your staged upgrade, Meraki will automatically upgrade the switch stack as part the staged upgrades. The upgrade process for a stack follows the same high-level process outlined previously, with each stack member rebooting close to the same time and the stack then automatically re-forming as the members come online
Best Practice for MX Firmware
Unlike many other products offered by Meraki, MX appliances and Z-Series devices have a one-dashboard-network per-site model. This provides very granular control of how upgrades can be managed across the deployment.
All firmware upgrades will require that the MX appliance reboots, so it is important to ensure that an appropriate maintenance window has been put in place, as the MX upgrade process will take down the entire local network in most scenarios. Given the central/upstream nature of MX devices, it is also recommended to allow for sufficient time to monitor and test after the upgrade completes to ensure the maintenance window completes successfully.
When managing a deployment with many MXs, the following are useful best practices that can help make firmware transitions and management simpler.
Appliance Network with Two MXs in an HA Configuration
When MX appliances configured to operate in High Availability (HA) (either in NAT/routed mode or when operating as one-armed VPN concentrators), the dashboard will automatically take steps to minimize downtime when upgrades are performed to ensure a zero-downtime MX upgrade. This is achieved through the following automated process:
-
The Primary MX downloads firmware
-
The Primary MX stops advertising VRRP
-
The Secondary MX becomes active
-
The Primary MX reboots
-
The Primary MX comes online again
-
The Primary MX starts advertising VRRP again
-
The Primary MX becomes active again
-
The Secondary MX downloads firmware (approximately 15 minutes after the original upgrade is scheduled)
-
The Secondary MX stops advertising VRRP
-
The Secondary MX reboots and comes back online
Deployments using AutoVPN
When upgrading a VPN concentrator, it is important to plan for a maintenance window that allows for the upgrades to complete and for verifications to be performed that ensure connectivity is fully re-established and network systems are healthy.
It is highly recommended that customers plan for maintenance windows in accordance with the scale and complexity of the deployment where the upgrades are being performed. For example, more time should be allotted for upgrading a VPN concentrator supporting 1000 spoke sites and leveraging a dynamic routing connection between the concentrator and datacenter, than for a VPN concentrator with only 10 spoke sites.
If beta firmware is being tested on a VPN concentrator, it is best to plan for time in the maintenance window to allow for the upgrade to complete and validate the operational state after the upgrade has been completed. It is also recommended to allocate an additional window of time for rolling back to the previous build, in case you run into unmanageable issues.
When concentrators are configured in HA, they will follow the steps mentioned above. VPN tunnels will begin establishing to the spare appliance while the primary is upgrading. However, the primary appliances typically complete the upgrades fast enough that spoke sites have minimal interactions with the spare concentrator.
In general, even with equipment in HA, it is best to always be prepared for some amount of downtime and impact for spoke sites. In almost all cases these are simply a matter of seconds as spoke sites fail between concentrator pairs, but the impact can become more noticeable if there are WAN connectivity problems between the data center and spoke locations.
Meraki Firmware Development Lifecycle
One of the key advantages of being a cloud managed device company is that Meraki is able to leverage full internal automated testing, while also being able to utilize our cloud to monitor key device performance metrics across our entire installed user base. To ensure robust and reliable firmware development, Meraki follows a consistent software release process to validate and deploy consistent and reliable firmware. Meraki's firmware development process has four stages: alpha, beta, stable release candidate (RC), and stable. Every firmware version is created and released with the goal of graduating to stable. If a particular build fails to pass our key metrics at any stage of the development process, a new build is created and the process begins anew. The following sections go over each of the stages in more detail.
Alpha Pre-Release
With all new Meraki firmware including both major and minor releases, we start out every new build by running it through our full alpha testing process. Before any release hits our users’ hands, we validate the release by running it through our ever-expanding testing suites, and check for regressions or new features that are not performing as expected. Each product line has automated and manual testing specific to the product, that are designed to ensure Meraki minimizes the chance of regressions as we continue to create and expand on our software feature set.
As part of our core philosophy, after a new build has successfully passed the testing phase, we deploy the new firmware release on our own personal and engineering networks. We believe it is important that we deploy and run our own firmware before any of our customers deploy our firmware. During this process we will run this firmware in our real world deployments for one or more weeks before we consider releasing the build as a new beta version.
If a build successfully passes all of our release criteria, we will start to make the new build available to our customer base. If any issues are discovered that need to be resolved, we will start the process over once the issue has been addressed before moving the release forward. In some more rare cases, we will move forward with a build with a known regression, due to complexity or timing of the fix, and in this scenario we will note the regression in the release notes for that version.
Beta Release
Firmware is made available for production use at first under "Beta." Often customers will run beta firmware in their production network to take advantage of new features and bug fixes. Beta firmware has already gone through internal regression, stability, and performance testing to limit risks when applied to production networks. Customers that opt into beta firmware via the Try beta firmware configuration option on dashboard will be automatically notified and scheduled to upgrade to these versions as they are released. These upgrades can be canceled, modified, and reverted using the firmware upgrades tool in the dashboard. Customers can also manually upgrade their networks at any time to beta firmware by using the firmware upgrade tool. Beta firmware can be considered analogous to “Early Deployment” firmware seen in other products in the industry.
The latest beta firmware is fully supported by our Support and Engineering teams. Older betas are supported with best effort; an upgrade to the latest beta will ensure full support.
Stable Release Candidate
As a new firmware version matures from beta, it has the opportunity to graduate into a stable release candidate. A formal review of the beta firmware’s success is conducted by our software and product teams. Key performance indicators (KPIs) for quantifying firmware quality are analyzed including open support cases & engineering issues, firmware adoption, and stability metrics. After the formal review, a beta may be reclassified as a "Stable Release Candidate." At this point the firmware version will be indicated as such in the firmware upgrade tool. Once a new stable release candidate is available, Engineering will begin scheduling a limited set of customers for upgrade. These upgrades can be canceled, modified, or reverted using the firmware upgrade tool as well.
The latest stable release candidate firmware is fully supported by our Support and Engineering teams. Older stable release candidates are supported with best effort; an upgrade to the latest beta, stable release candidate, or stable will ensure full support.
Stable Release
A stable release candidate matures into a stable version over time as it is slowly rolled out to devices globally. When the Meraki install-base hits a specified threshold for a major version (roughly 20% of nodes), that firmware revision will be promoted to stable, pending a final formal review. For point releases, the determination will be made on a case-by-case basis.
Again, the same KPIs are analyzed as used in the stable release candidate review. Upon completion of these processes the firmware can be promoted to "Stable." After promotion, stable versions can be applied by any customer via the firmware upgrade tool on dashboard. The latest stable version is also the version that is used for all newly created dashboard networks for a particular device.
Firmware Upgrade Tool
To make all of the best practices above simple to manage, you can use the Meraki firmware upgrade tool. We have built this tool to allow organizations to easily manage all Meraki firmware across the product portfolio in a single dashboard. As with all of our cloud features, we are continuing to build more functionality in the firmware upgrade tool to increase usability and simplify firmware management.
On the Overview tab, customers find a variety of information, such as a list of recent upgrades in the dashboard organization, pending upgrades that have been automatically or manually scheduled, the ability to cancel or reschedule these upgrades as well as a list of firmware versions that are available in beta, stable release candidate, or stable form for a given Meraki product.
Included with the available beta, stable release candidate, and stable firmware versions available in dashboard is a list of changelog notes. These notes allow customers to be fully aware of any new features, bug fixes, and existing known issues found between their existing firmware in use and the version planned for upgrade. Customers leveraging configuration templates may also enjoy the benefits of the firmware upgrade tool.
Requesting Specific Versions of Firmware
If there is a specific version of firmware that is needed for reasons of compatibility, then it can be requested from Meraki Support. Please note that any problems that are encountered while running versions of firmware that are not either stable or release candidate will not be supported and Meraki Support may need to recommend upgrading to the latest version of firmware in order to continue troubleshooting.
Locked Firmware
In general, it is discouraged to upgrade firmware on specific devices rather than upgrading the entire network. However, during the course of troubleshooting, Meraki Support may find it necessary to try a particular version of firmware on a specific device. For devices that have their firmware set manually by Meraki Support, you’ll see the message: Firmware version locked, please contact Support.
Addition or removal of locked firmware cannot be scheduled, please call Meraki Support to have this completed. The device will then automatically downgrade/upgrade to match the network firmware.
Additional Firmware Documentation
In addition to this best practices document please reference our other documentation to help you best deploy your Meraki products: