Yesterday around 8:50AM CST we began experiencing periods of service interruption that affected a large number of users. The service interruption unfortunately grew into a larger problem and ultimately led to a wide spread interruption for all users.
Immediately upon our systems noticing potential problems we began to investigate the issue along with our core providers. We strive to maintain redundancy with all points of our infrastructure and redundancy was in place with the affected systems. Unfortunately the perfect storm occurred that caused the issue to replicate across systems causing the ultimate failure for a redundancy plan.
As the interruption grew we determined it necessary to run some processes that would reset some of our data systems. This also required some scans to run for data integrity. We chose to go this route to ensure data reliability and to prevent any data loss from before the service interruptions.
During and after the service interruption we began to work with our provider and on internal plans to better mitigate such scenarios in the future. While we do not release any technical specs (for security reasons) I can say that we have started implementing additional layers of redundancy and balancing as of yesterday evening and have several further projects being actively worked on.
We appreciate all the support from our users during this service interruption.
Monitoring service outages & availability
We have a service status page setup at http://status.activecampaign.com that allows you to view any current service outages or issues we are currently experiencing. When an issue occurs we tend to update that frequently so you can remain up to date with the progress of resolving whatever issue is at hand.