This week started on the wrong foot with a 1 hour and 45 minute outage early Monday morning, that affected many of you. I’d like to apologize for this, and shed some light on what happened.
It began as a network outage at our hosting provider’s data center, which was resolved fairly quickly. Unfortunately, Tracker’s load balancers did not recover correctly, as they should have automatically, and to complete the perfect storm, the alerting mechanism that’s supposed to notify us and the hosting provider’s operations team did not work either due to misconfiguration.
The load balancer configuration issue has been corrected, and we’re confident that this problem will not occur again. Tracker’s production environment is designed to recover automatically from network and other types of outages, and our hosting provider monitors and responds to problems on a 24/7 basis.
To help you determine if Tracker is experiencing an outage in the future, we’ve enabled a public status page for Tracker, via Pingdom, one of the site monitoring services that we rely on.
To keep up with the latest updates, please follow @pivotaltracker. And to get help, visit https://www.pivotaltracker.com/help or email tracker@pivotallabs.com.
About the Author
