North American Object Storage Service Impact (8-4-2017)

Overview

On August 3rd, 2017, our engineering team made a configuration change that applied an erroneous policy to our North American object storage service and caused certain data uploaded prior to 11:20AM Pacific time on August 3 to be deleted. The issue has since been remediated and is no longer occurring. In the majority of cases, this issue will not impact network operations, but will be an inconvenience as some data may have been deleted. This issue is limited to user uploaded data. Network configuration data is not impacted and no customer data was compromised.

Our engineering team is actively working to investigate what data we can recover, as well as what tools we can build to help our customers specifically identify what files have been deleted from their Organization. Meraki is endeavoring to make these tools available as soon as possible in order to help our customers save time and effort. 

Impacted customers will see a notification banner in the dashboard, with a link to organization-specific details.

Update - 8/15/2017

Recovery Tools - The bulk file upload tool and the floor plan replacement tool have been released and are now available for impacted customers in their dashboard. 

• The single file upload tool will allow you to upload a single floor plan image to replace a specific missing floor plan and retain your AP placements, with the option to then resize or rotate as usual.
• The bulk file upload tool allows users to select multiple files on their local machine, and to upload them to the dashboard to see if they match any deleted files. Where there's a match, the uploaded files can be restored. 

Further details on these tools can be found here:
https://documentation.meraki.com/zGe...e_Upload_Tools

Update - 8/11/2017

Data Recovery - File recovery efforts are ongoing. Where successful, files will be automatically replaced in their proper location unless customers have already taken the step of uploading their own replacement files. File recovery status for each impacted Organization is available in the Organization Remediation page. 
Recovery Tools - We're finalizing work on two tools: a single-file upload tool and a bulk upload tool.

• For floor plans, we are providing a tool that will allow you to upload a single floor plan image to replace a specific missing floor plan and retain your AP placements, with the option to then resize or rotate as usual.
• The bulk upload tool will allow the selection of multiple files of any kind and automatically identify perfect matches, to seamlessly replace missing assets. If you have already manually replaced some of your assets, we won't overwrite them.

We expect these tools to be available early next week (week of August 14th).
Next Update - We will continue providing updates on this page as progress is made.

Update - 8/9/2017

Data Recovery - Our engineering team continues the recovery effort to retrieve assets. The likelihood of successful file recovery will vary depending on asset type, and at this time, we do not expect to be able to recover most assets. We are working diligently to recover as many customer uploaded files as possible. As these are recovered they will be automatically replaced in their proper location, unless customers have already taken the step of uploading their own replacement asset.
Recovery Tools - We're building two tools: a single-file upload tool and a bulk upload tool.

• The single file upload tool will allow you to upload a single floor plan image to replace a specific missing floor plan and retain your AP placements, with the option to then resize or rotate as usual.
• The bulk upload tool will allow you to choose a folder of files and automatically identify perfect matches to seamlessly replace missing assets. If you have already manually replaced some of your assets, we won't overwrite them.

Further details of these tools will be communicated as soon as possible.
Next Update - We will continue providing progress updates on this page. The next update will be published no later than end of day Friday.

Update - 8/7/2017

Data Recovery - Our engineering team has determined that, in some cases, assets may be recoverable through our data cache, and a recovery effort is currently underway.

Recovery Tools - We’ve heard from many of you that you have retained copies of your original files. We’re building a bulk upload tool that will automate re-uploading those files into their appropriate location, based on the checksums we have in our database. Our aim for this tool is to help you recover as many files as possible. 

Next Update - We’ll continue providing progress updates on this page. The next update will be published no later than end of day Wednesday.

Areas of Impact

Revision History

August 5th, 12:59PM PDT: Added notice for Maps & floor plans recommending that all users wait before deleting or re-uploading any floor plans. 

August 5th, 6:38PM PDT: Added additional information about ongoing Engineering work that is being done to make recovery easier for users. Added additional information about remediation steps that should or should not be taken at this time. 

August 7th, 12:07PM PDT: Clarification to Areas of Impact and Overview.

August 7th, 6:02PM PDT: Updated Overview section.

August 7th, 7:13PM PDT: Created Update section and posted update with new information.

August 9th, 5:17PM PDT: Situation update.

August 9th, 9:10PM PDT: Added link to Dashboard Organization Remediation page.

August 10th, 10:05AM PDT: Added additional guidance on data recovery efforts.

August 11th, 5:03PM PDT: Situation update.