How can we help?

Feb 28 2017: Outage on AWS S3 Server

12:40 EST

A major Internet storage service (Amazon S3) came offline at 12:40 PM EST. All of our PDF reports and logos are affected and cannot be uploaded or downloaded. While reports may still be run and sent, they will not be saved and they will not include any logos until this issue is resolved by Amazon. 

Amazon Web Services is experiencing issues on us-east-1 with high error rates. Many internet services have been affected by this outage: Amazon’s Twitch, Atlassian’s Bitbucket, Buffer, Business Insider, Chef, Citrix, Codecademy, Coursera, Cracked, Docker, Expedia, Expensify, FiftyThree, Giphy, Greenhouse, Heroku, Home Chef, iFixit, IFTTT, Imgur, isitdownrightnow.com, Lonely Planet, Mailchimp, Medium, Microsoft’s HockeyApp, News Corp, NinjaCat, Pantheon, Quora, Razer, Signal, Slack, Sprout Social, Travis CI, Trello, Twilio, Unbounce, the U.S. Securities and Exchange Commission (SEC), and Zendesk, among other services.

12:52 EST - Update from Amazon:

We are investigating increased error rates in the US-EAST-1 Region.

13:16 EST - Update from Amazon:

We've identified the issue as high error rates with S3 in US-EAST-1, which is also impacting applications and services dependent on S3. We are actively working on correcting the issue.

14:35 EST - Update from Amazon:

The dashboard has recovered. You will see status updates for individual services at: https://status.aws.amazon.com/

14:46 EST - Update from NinjaCat:
We are currently investigating the status of all of our services that are impacted by the above outage. We will post more information as it becomes available.
Our areas of concern for the above outage include but are not limited to the following:
  • Agency logos missing from client dashboards
  • Running, Viewing, Saving and adding notes in PDF reports.
  • Downloading client reports or viewing historical PDF reports.
  • Many services, including client-facing dashboards, are very, very slow to load.
  • Call-tracking services that use AWS S3 servers, such as saving and downloading call recordings.  (However, NinjaTrack calls are still being received and routed correctly).
14:57 EST - Update from Amazon:
We continue to experience high error rates with S3 in US-East-1, which is impacting some other AWS services.

For S3, we believe we understand root cause and are working hard at repairing. Future updates across all services will be on dashboard.

15:50 EST - Update from NinjaCat:

Amazon has not yet fully resolved the issue or offered an ETA of its resolution, but files in our system are starting to come back consistently.  
17:00 EST - Update from NinjaCat:
While much of our functionality is coming back and we are able to read data from Amazon's servers, we are still unable to write data back to them so reports that are being run are showing the status of "failed". However, reports that are set to automatic delivery and have an email address configured under the report settings are successfully being emailed. 
17:08 EST - Update from Amazon:
We are fully recovered for operations for adding new objects in S3, which was our last operation showing a high error rate. The Amazon S3 service is operating normally.



Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request


Please sign in to leave a comment.