Meraki and ThousandEyes Integration Troubleshooting
Overview
This document provides guidance on how to troubleshoot the MX and ThousandEyes integration. A general introduction and information on how the ThousandEyes (TE for short) and Meraki integrations work can be found in the Meraki MX ThousandEyes Configuration Guide document.
If you are interested in knowing the status of ThousandEyes Services please check their Status Page.
Architecture Review
As mentioned in the configuration article, for the MX to run the ThousandEyes application inside our firmware, it requires a special version of the ThousandEyes Enterprise Agent, or EA for short.
After the MX is configured from Dashboard to activate the integration, it will download the agent image and configuration from our Meraki backend, specifically from the Container Registry. Once completed, the MX will send the agent health information and test results back to the TE Dashboard for consumption and reporting. During regular operation, the agent will check in with the ThousandEyes Dashboard every minute to upload new test results as a keep-alive mechanism.
It’s important to highlight that the Enterprise Agent uses the MX internal networking stack (198.51.x.x) and does not interfere with or modify the data plane traffic. It only interacts with the data traffic unless necessary, for example, when performing a test on a LAN-side server or contacting the TE dashboard.
If a ThousandEyes agent is removed from the MX via Dashboard, the container will be shut down and deleted. This process also deletes the configuration from Dashboard.
Account Linkage
The account linkage process aims to tie your Meraki Dashboard with a ThousandEyes account group. From a logical perspective, a TE’s account group is similar to a Meraki Organization. After completing this process, both Dashboards will become aware of each other.
Linking the Meraki and the ThousandEyes Dashboard requires authenticating against ThousandEye’s platform. As such, you are required to log into your TE account while onboarding. The account linkage process ends when the Meraki Dashboard is instructed to generate a configuration telling the MX to download the Enterprise Agent image.
Please refer to the “Agent Guided Configuration and Deployment” section under the configuration document for a detailed description of the UI steps needed to perform the account linkage process.
Linkage steps
- Go to Insight > Active Monitoring
- If the Organization is not linked you will be presented with a Marketing page explaining ThousandEyes overview. To continue, click "Try it".
- Click on “Log in” if you have an existing ThousandEyes account. If not, you can create a new one via a Free Trial.
- You will see a new window appear by clicking any of the options.
- Fill out the username and password field using ThousandEyes credentials to complete the account linkage. After completing the association, a green “Connected” icon will appear.
- Select the Application to monitor. This page leverages a ThousandEyes feature called Verified Test Templates. With test Templates, multiple individual tests are bundled together under a primary template. Be aware that some templates, require a tenant/subdomain to be added. After finished, click next.
- Select the network where the agent will be installed. Preexisting networks can be found using their tags or names under the search field.
- Once the selection is complete, the process will finish after clicking “Start monitoring”.
- The onboarding experience ends once you are presented with the Agent List page, where you can disable or erase the TE agents.
Troubleshooting
For any troubleshooting efforts during the account linkage, please use the browser Developer Tools and HAR files, as they will capture any interaction between the user and or Dashboard.
Issue: Authentication Page requiring cookies
The web page required to authenticate the user against the ThousandEyes Platform does require cookies to be enabled. Safari is known to be prone to this issue. Example image below:
Workaround: Please allow cookies for the site or use a different browser.
Issue: The network selection table is incomplete
As a reminder, any network with an MX67+ AND running MX 18.104+ firmware is eligible for the integration. If the network selection table does not include any network or the list does not match the expected number of networks, please check the following:
- Start the onboarding process with the developer tools opened on the browser.
- When you reach the network selection screen, click on the Network tab of the developer tools. Check for the API:
- After clicking on this call, please review the Response field. It should include a list of eligible networks to be onboarded:
Next Steps: Confirm that the networks to activate do comply with the prerequisites specified above. If they do, but the list is incomplete, please submit a support case and include the resulting HAR file and a video recording.
Issue: An error occurred during authorization. Please refresh the page and try again. If the issue continues, please contact support for assistance.
After initiating the activation process, this error banner appears under the “Authorize ThousandEyes” section.
Example pic below:
This is triggered by the Organization having only one Full-Org admin.
Workaround: Please create an additional full-org admin so you have at least two in your Organization.
Issue: "Hmm. Dud link." message during account linkage
While attempting to complete the account linkage process, a screen shows a "Hmm. Dud link." message. This screen does not let the onboarding process continue. Example below:
Mismatching regions between the ThousandEyes and Meraki Dashboards trigger this screen. As noted in the Account Requirements section in the Configuration document, both accounts must belong to the same geographical region for the account linkage process to succeed. This enforcement is required to comply with international data regulations on information privacy.
Next Steps: Ensure that both Dashboards are in the same region. In the case of the Meraki Dashboard this can be achieved by creating a new Organization in the correct region and transferring devices. In the case of migrating or moving your ThousandEyes account, please contact your ThousandEyes Account Representative.
Issue: After linking my accounts, I get assigned to an incorrect ThousandEyes Account
After choosing the Log In option on the onboarding process and entering your ThousandEyes credentials your Meraki Dashboard does not get linked to the expected ThousandEyes Account Group.
Next Steps: The Meraki Dashboard only has visibility of a single ThousandEyes Organization at a time and its underlying Account Groups. If you have multiple ThousandEyes Organizations (Multiple Customers), please ensure that during the initial linkage, your Login Account Group setting on your ThousandEyes User Profile is set to correct ThousandEyes Organization and Account Group. More information can be found in the ThousandEyes Documentation.
Issue: A ThousandEyes Organization already exists for this Account. Please contact your company's ThousandEyes Platform Administrator or ThousandEyes Support and share this ID.
After selecting the option to Create a new account during the onboarding process, the Meraki Dashboard will return this error if a ThousandEyes Account already exists for you and it's already claimed by your ThousandEyes Platform Administrator.
Next Steps: Please contact ThousandEyes Support and share the ID.
Issue: An error occurred while retrieving your Account information. Before attempting again, please contact support for further assistance.
After selecting the option to Create a new account during the onboarding process, the Meraki Dashboard will return this error if your Meraki Account does not have enough information to create a new ThousandEyes account. Example screen below:
Next Steps: Please reach out to Meraki Support.
Issue: A ThousandEyes Organization already exists for this user. Please contact your company's ThousandEyes Platform Administrator or ThousandEyes Support.
After selecting the option to Create a new account during the onboarding process, the Meraki Dashboard will return this error if the email address used by the Meraki Administrator who is trying to create the new ThousandEyes Trial is already tied with another ThousandEyes Organization. Example below:
Next Steps: Please contact ThousandEyes Support, or try to create a new account with a new email address that does not belong to an existing ThousandEyes Organization/Account Group.
Issue: Invalid account token
The Meraki Dashboard performs a user token verification check when retrieving data from the ThousandEyes platform and and if it we receive an indication that the token is no longer valid a token refresh is requested.
Next Steps: Please continue and refresh your token.
Traffic Pattern
Test Traffic
After a test is configured on the ThousandEyes Dashboard, the traffic destined for the test destination server or device will follow the MX routing table. As with other applications inside the MX, if the Agent sends test traffic across a VPN tunnel, the MX will use its source IP associated with the highest-numbered VLAN participating in the VPN.
To verify that the agent is sending the test traffic and is going over the desired path, you can rely on the routing table of the MX and packet captures from the expected interfaces. The example below is from an AutoVPN tunnel and a DNS test to the 192.168.11.28 Server:
The Agent will be internally assigned an IP address from the range 198.51.100.x, typically 198.51.100.2, with a default gateway of 198.51.100.1. This IP address is not visible on the Meraki Dashboard but might show in the ThousandEyes Dashboard or in the traceroute tool. This is expected.
ThousandEyes Dashboard Traffic
After the agent is activated and running, it will reach out to the ThousandEyes dashboard for regular checking every minute or less via TCP port 443. The entire destination URLs and IP addresses for TE’s infrastructure can be found in the Firewall Configuration for Enterprise Agents document, but most agents will use the URLs below:
| Americas | European Union |
Troubleshooting
Issue: The MX sends test traffic sourced from the 6.0.0.0/0 IP Address range
When an MX is configured as a VPN Concentrator, it uses this IP address to send traffic over the AutoVPN tunnel.
Workaround: Change the MX mode to NAT.
Behavior during firmware upgrades and downgrades
- You can downgrade your MX firmware to an unsupported version. This downgrading process removes the Agent from MX but stores the network ThousandEyes configuration.
- If you upgrade a device previously enabled for ThousandEyes, the device will automatically re-download the ThousandEyes Agent.
FAQ
Can the agent be deployed via API?
At this time, agent provisioning is not available via Dashboard API.
Can the ThousandEyes Agents be deployed using Templates?
The ThousandEyes agents can only be deployed using the network selection step under the activation wizard.
Are event logs available?
Change and Event logs will be available at GA for Agent provisioning and deletion.
How much monitoring traffic is used by the MX and the ThousandEyes integration?
In addition to the normal amount of traffic used for management purposes as part of the Meraki Cloud connectivity, the ThousandEyes Agent generates additional traffic depending on the number of tests and test frequency configured. More information can be found here.
Why is my agent utilization high if their tests are failing?
Agent utilization depends on how long tests run inside the agent itself. If network conditions change so the tests take longer to complete, the agent utilization will increase. A failing test is considered an adverse network condition. Notes on agent utilization here.
How does the MX route the test traffic sourced from the Agent?
This traffic will follow the MX routing table, but if the traffic destination is over an AutoVPN tunnel, it will be sourced from the MX IP address of the highest number VLAN allowed over the tunnel.
For example, if the MX has a default route pointing to the WAN interface and the Agent needs to send traffic to an Internet-hosted application, the traffic will be NATed to the MX WAN IP address. But if there’s an overlapping default route (for example, a static route via the LAN), the Agent will use it as its next hop for the internet-bound traffic. For more information on the MX route priority, please refer to the MX Routing Behavior KB.
Is IPv6 Supported?
Yes. IPv6 testing will be available for this integration.
Can a Meraki Dashboard be associated with multiple ThousandEyes?
No. A Meraki Dashboard organization can only be associated with one ThousandEyes account.
How long does it take before I start seeing reports on my ThousandEyes dashboard?
After the agents are configured on the Merak/ThousandEyes Dashboard, it might take up to 15 minutes for the reports to appear.
Why are some Public DNS servers rejecting my lookup tests?
Neither ThousandEyes nor Meraki recommends using public DNS servers as a destination for DNS tests. These servers might reject our testing requests as a DDoS mitigation measure if the frequency of tests is too high (less than 2 minutes).
Does the ThousandEyes agent impact the MX performance?
When no tests are performed, there’s no impact on the MX performance. When an agent runs, resource utilization does not impact the MX published throughput and performance metrics.
Which DNS Server does the agent use for internal queries?
The agent uses whichever DNS servers are present under the uplink configuration for the MX. As of today, this setting cannot be modified.
What happens if the MX agent download gets interrupted due to upstream packet loss?
The MX will keep retrying every minute until it successfully downloads the container image from Dashboard. A backoff mechanism will be implemented to avoid a DDoS-like behavior in a future implementation.
Is there a ThousandEyes status page?
Yes, It’s located at https://status.thousandeyes.com. Customers can subscribe to alerts or notifications.
Why is my agent showing a 198.51.100.2 IP address?
The MX will assign the agent the IP address 198.51.100.2 with a default gateway of 198.51.100.1. This internal network lives inside the MX and is not routable outside the agent environment. Effectively, the MX will perform a NAT of this IP address to allow for external connectivity. This IP address is not visible on the Dashboard but might show in the TE Dashboard or the traceroute tool.
Do I open a case with Meraki Support or ThousandEyes Support?
If you need assistance with a Meraki Dashboard configuration, please contact Meraki Support. For any ThousandEyes configuration, please contact ThousandEyes’ support.
Why is the SSL time for MX-based agents tests higher than other platforms?
ThousandEyes Enterprise Agents running on the MX Platform with configured HTTP Servers test will experience higher SSL Time due to processing power limitations on the lower-end models.
To avoid false positives, we recommend using dynamic alert conditions based on the standard deviation of the SSL time. For more information on Dynamic Baselines, please consult the ThousandEyes configuration guide here.