Home > Wireless LAN > Monitoring and Reporting > Scanning API

Scanning API

Introduction

Thanks to widely available 'smart devices equipped with WiFi and BLE, Cisco Meraki's wireless access points can detect and provide location analytics to report on user foot traffic behavior. This can be especially useful in multi-site retail or enterprise deployments where admins or departments beyond IT wish to learn more about trends and user engagement. Coupled with traditional reporting from the WiFi network on client devices, applications and websites, Cisco Meraki provides a holistic view of online and offline user traffic.

Leveraging our globally distributed datacenter architecture, Cisco Meraki has built an end-to-end system that can aggregate data from thousands of endpoints for effective collection, analysis, and presentation of this data on the Meraki Dashboard. Comparisons can be run between different sites and time periods, and Cisco Meraki's network tagging functionality allows for an unlimited variation of comparisons by creating batches of networks that can be grouped together based on district, region, or any other preference. In addition to the built-in location analytics view, the Location API enables Cisco Meraki customers to detect and aggregate real-time data for custom applications.

The location API delivers data in real-time from the Meraki cloud and can be used to detect WiFi (associated and non-associated) and Bluetooth Low Energy (BLE) devices in real-time. The elements are exported via an HTTP POST of JSON data to a specified destination server. The raw data is aggregated from all access points within a network on the Meraki cloud, and sent directly from the cloud to an organization's data warehouse or business intelligence center. The JSON posts occur frequently, typically batched every minute for each AP.

Using the physical placement of the access points from the Map & Floorplan on the Dashboard, the Meraki cloud estimates the location of the client. The geo-location coordinates (latitude, longitude) and X,Y location data accuracy can vary based on a number of factors and should be considered a best effort estimate. AP placement, environmental conditions, and client device orientation can influence X,Y estimation; experimentation can help improve the accuracy of results or determine a maximum acceptable uncertainty for data points.

Scanning Data Elements

The Scanning API version 2.0 data architecture device classification and location information. Using the physical placement of the access points from the Map & Floorplan on the Dashboard, the cloud estimates the location of the client.

 

Data Elements

Name
Format
Description
apMac string MAC address of the observing AP
apTags [string] JSON array of all tags applied to the AP in dashboard
apFloors [string] JSON array of all floorplan names on which this AP appears
clientMac string Device MAC
ipv4 string Client IPv4 address and hostname, in "hostname/address" format; only "/address" if no hostname, null if not available
ipv6 string Client IPv6 address and hostname, in "hostname/address" format; only "/address" if no hostname, null if not available
seenTime ISO 8601 date string Observation time in UTC; example: "1970-01-01T00:00:00Z"
seenEpoch integer Observation time in seconds since the UNIX epoch
ssid string Client SSID name; null if the device is not connected
rssi integer Device RSSI as seen by AP
manufacturer string Device manufacturer; null if manufacturer could not be determined
os string Device operating system; null if the OS could not be determined
location location Device geolocation; null if location could not be determined
lat decimal Device latitude in degrees N of the equator
lng decimal Device longitude in degrees E of the prime meridian
unc decimal Uncertainty in meters
x [decimal] JSON array of x offsets (in meters) from lower-left corner of each floorplan
y [decimal] JSON array of y offsets (in meteres) from lower-left corner of each floorplan

HTTP POST body format

{
   "version":"2.0",
   "secret":<string>,
   "type":<event type>,
   "data":<event-specific data>
}

 

Event Specific Data Format

{
  "apMac": <string>,
  "apTags": [<string, ...],
  "apFloors": [<string>, ...],
  "observations": [
    {
      "clientMac": <string>,
      "ipv4": <string>,
      "ipv6": <string>,
      "seenTime": <string>,
      "seenEpoch": <integer>,
      "ssid": <string>,
      "rssi": <integer>,
      "manufacturer": <string>,
      "os": <string>,
      "location": {
        "lat": <decimal>,
        "lng": <decimal>,
        "unc": <decimal>,
        "x": [<decimal>, ...],
        "y": [<decimal>, ...]
      },
    },...
  ]
}

 

Enable Scanning API

The Location API is configured in the Meraki Dashboard on the Network Wide > General settings page in a few simple steps:

  1. Turn on the API by selecting Scanning API enabled in the dropdown box.

  2. Specify a post URL and the authentication secret (the secret is used by your HTTP server to validate that the JSON posts are coming from the Meraki cloud)
  3. Specify which Scanning API version your HTTP server is prepared to receive and process.
  4. Configure and host your HTTP server to receive JSON objects.
  5. Upon the first connection, the Meraki cloud will perform a single HTTP GET; the server must return the organization-specific validator string as a response, which will verify the organization's identity as the Cisco Meraki customer. The Meraki cloud will then begin performing JSON posts. 

 

 

 

Protocol flow between Meraki cloud and third-party server:

 

Bluetooth Scanning API

Meraki APs with an integrated Bluetooth Low Energy (BLE) radio can detect and locate nearby Bluetooth Low Energy devices. This data is then provided via API to third party applications. Examples of such devices include battery based beacons, Apple iBeacons, fitness monitors, and remote sensors. 

Using the physical placement of the access points from the Map & Floorplan on the Dashboard, the Meraki cloud estimates the location of the client. The geo-location coordinates (latitude, longitude) and X,Y location data accuracy can vary based on a number of factors and should be considered a best effort estimate. AP placement, environmental conditions, and client device orientation can influence X,Y estimation; experimentation can help improve the accuracy of results or determine a maximum acceptable uncertainty for data points.

 

To enable BLE devices to be located, enable the BLE scanning radio on the access points. BLE Scanning is enabled in the Wireless >  Bluetooth Settings > Scanning settings page by selecting "Scanning enabled" in the Scanning section, as shown in Figure 3 below:

Untitled:Users:dmlandry:Google Drive:Desktop:Screen Shot 2015-03-08 at 14.33.16.png

Figure 3: Enabling BLE scanningBLE Scanning API

 

Bluetooth API Data Elements

Name
Format
Description
apMac string MAC address of the observing AP
apTags [string] JSON array of all tags applied to the AP in dashboard
apFloors [string] JSON array of all floorplan names on which this AP appears
clientMac string Device MAC
seenTime ISO 8601 date string Observation time in UTC; example: "1970-01-01T00:00:00Z"
seenEpoch integer Observation time in seconds since the UNIX epoch
rssi integer Device RSSI as seen by AP
location location Device geolocation; null if location could not be determined
lat decimal Device latitude in degrees N of the equator
lng decimal Device longitude in degrees E of the prime meridian
unc decimal Uncertainty in meters
x [decimal] JSON array of x offsets (in meters) from lower-left corner of each floorplan
y [decimal] JSON array of y offsets (in meteres) from lower-left corner of each floorplan

Enable the Scanning API with BLE Scanning, and the Location API will include both WiFi and Bluetooth devices seen by the access points in a single data feed. The event type BluetoothDevicesSeen is used to identify the observations from the Bluetooth radio. Below are the JSON formats used by the Location API for Bluetooth devices.

 

HTTP POST body format

{
   "version":"2.0",
   "secret":<string>,
   "type":"BluetoothDevicesSeen",
   "data":<event-specific data>
}

 

Bluetooth API Data Format

{
  "apMac": <string>,
  "apFloors": [<string>, ...],
  "apTags": [<string, ...],
  "observations": [
    {
      "location": {
        "lat": <decimal>,
        "lng": <decimal>,
        "unc": <decimal>,
        "x": [<decimal>, ...],
        "y": [<decimal>, ...]
      },
      "seenTime": <string>,
      "clientMac": <string>,
      "seenEpoch": <integer>,
      "rssi": <integer>,
    },...
  ]
}

Location and Privacy

Meraki understands that some end users may be concerned about the collection and use of location information. In an effort to address these concerns, Meraki developed location services with privacy in mind, including a number of security mechanisms to eliminate uniquely identifiable elements from the data that it collects. Meraki also recommends that its customers and partners implement a number of privacy-friendly features. 

 

Meraki uses probe requests, data frames, and bluetooth beacon frames to locate and store client location. Because the location data contain raw MAC addresses, Meraki implemented a number of security mechanisms to anonymize the data in an irreversible fashion. Once uploaded, the Meraki cloud anonymizes or produces a hash of MAC addresses so that they are not identifiable. The Meraki cloud only stores the hashed version of the MAC address.

 

The hash function is as follows:

hash(mac bytes, org secret) =

SHA1(mac bytes ++ org secret).takeRight(4)

 

where:

++ indicates concatenation;

takeRight(4) returns the least significant 4 bytes of the SHA1; and

org secret is a per-customer salt.

 

Example:

client MAC is 11:22:33:44:55:66

org secret is t3lrdd

 

least significant 4 bytes of SHA1(112233445566t3Irdd) = 0x0e456406

 

SHA1 is a widely known one-way cryptographic function. Using SHA1 hashes in this manner is the current industry standard. In order to provide an additional layer of security beyond SHA1 hashing, Meraki's hash function truncates the hash to 4 bytes. This produces an information theoretic loss, as the domain of the function is larger than the range: a 6-byte MAC allows (2^48) possibilities whereas a 4-byte hash allows (2^32) possibilities. This results in 65,000 possible (org + MAC) combinations for each one 4-byte hashed MAC address. Therefore, given a MAC that has been salted, hashed, and truncated with the unique Meraki algorithm, it would be mathematically impossible to know with a reasonable degree of certainty what the original client MAC address was.

 

The hash function leads to information theoretic loss, and the original MAC address of client can never be recovered.

 

Cisco Meraki includes a customer-specific org-secret in the hash function. As a result, Cisco Meraki does not have any visibility into client behavior across our customers networks worldwide. And, of course, no Cisco Meraki customer can see the analytics of another customer's organization or where foot traffic goes after leaving the presence of its own WiFi / BLE network.

 

Finally, Cisco Meraki's website offers a global opt-out feature that allows users to submit the MAC addresses of their devices, after which the Meraki cloud will no longer detect their MAC addresses either for its built-in Location Analytics views or for real-time export via the Location API. Cisco Meraki also recommends that retailers and others using the Location API post notices on the availability of this global opt-out in prominent locations, preferably in the storefront or at building entrances where location detection is taking place. 

 

Privacy for Location API

Cisco Meraki's Location API, outlined above, exports raw MAC addresses to a specified third-party server. There are a number of privacy protection mechanisms that we have implemented, including:

 

No customer identity tie-in mechanisms

Cisco Meraki does not directly provide any way of tying MAC addresses to customer identity. These systems must be separately built and hosted by a customer, partner or service provider.

 

No customer contact mechanisms

Cisco Meraki does not provide any mechanism by which the location API data can be used to contact customers in any way. For real-time user engagement, Cisco Meraki customers must build and maintain their own platform for contacting customers.

 

Recommended best practices

Cisco Meraki recommends a number of best practices for users of its API, including:

  • Opt-in Cisco Meraki customers should make it explicitly clear at the time of identity tie-in (e.g., via a splash page or through a mobile app) that user-provided information may be linked to a device's MAC address for more extensive engagement.
  • On-premise notification As with the use of the built-in Location Analytics, notice should be prominently displayed in areas using of the Location API data.
  • Opt-out In addition to providing an opt-in policy, Cisco Meraki customers should make their own customers aware of Cisco Meraki's global opt-out policy (allowing an opt-out by MAC address) and provide an intuitive means of accessing Cisco Meraki's opt-out page. Cisco Meraki's global opt-out is available at https://account.meraki.com/optout.

Troubleshooting

In this article, "endpoint" will refer to the server receiving the Scanning API POSTs. Depending on configuration, this may include a web engine such as Apache or Nginx in addition to a backend application such as a rails service.

Scanning API Data is not received

Issues with Scanning API data not arriving can be separated into two types. Either the endpoint sees incoming TCP connections from Dashboard, or it does not:

No Connections from Dashboard

In the event that the endpoint sees no communication from Dashboard, the first step is to ensure that there is basic network connectivity between the endpoint and Dashboard. Ensure that there is no filtering occurring by investigating logs from any and all firewalls that exist upstream of the endpoint. If there are no firewalls present, or the firewalls are relatively permissive, you should be able to telnet to the endpoint from an arbitrary external address.

A lack of data can also occur over short periods (15 to 20 minutes) if an endpoint's responses are slow, as outlined below. If endpoint logs do not indicate long responses, errors, connectivity to the endpoint can be confirmed from other sources, and no traffic filtering is occurring, please contact Cisco Meraki support for further assistance.

Connections from Dashboard

If TCP connections are seen from Dashboard but no Scanning API data is received, consider the following possible causes:

  • Considerations with HTTPS

When using HTTPS, the endpoint must have a valid certificate signed by a valid public Certificate Authority. If a certificate is not signed by a recognized CA, is expired, revoked or otherwise invalid, a session cannot be established to allow the incoming POST.

In some circumstances, a signed certificate can be valid but still unrecognized if signed by an intermediary CA unknown to Dashboard. For this reason, we recommend including the full CA chain when using HTTPS.

  • Endpoint is taking too long to respond

If an endpoint is taking an exceptionally long time - 500ms or longer - to respond to a POST, that endpoint's information will be dropped without being sent. It is recommended that applications separate processing of incoming Scanning API data from data routing for this reason.

  • Endpoint is returning an error

Check the application and service logs to ensure that the endpoint application is not reporting errors. Please refer to your endpoint application/service vendor for documentation and troubleshooting assistance.

  • URL fails to validate

During URL validation, the endpoint MUST return a 200 OK response with a body containing only the validator string. If the response is something other than a 200 OK, or the body is not an exact match to the validator string, validation will fail and the endpoint URI cannot be saved as an entry. Accessing the POST URL directly using a browser such as Google Chrome or Firefox, or generating a GET request with curl, are ways to check if the endpoint is responding correctly.

Malformed Data

Certain information is included if and only if a client associates to the network. This includes manufacturer, OS, SSID and IP addressing. Certain information, such as location values or AP tags, are included only if available. Other information should always be included and always have a value. If it does not, please contact Meraki Support.

You must to post a comment.
Last modified
00:33, 30 Jan 2017

Tags

This page has no custom tags.

Classifications

This page has no classifications.

Article ID

ID: 5481

Contact Support

Most questions can be answered by reviewing our documentation, but if you need more help, Cisco Meraki Support is ready to work with you.

Open a Case