All Systems Operational

About This Site

SNworks monitors all of its critical services both internally and through 3rd party services. If there is a disruption in service, incident notes will be updated here.

Uptime over the past 90 days. View historical uptime.
CEO Operational
90 days ago
99.99 % uptime
Today
API ? Operational
90 days ago
100.0 % uptime
Today
Application ? Operational
90 days ago
99.97 % uptime
Today
Front End Sites ? Operational
90 days ago
100.0 % uptime
Today
Guides Application ? Operational
Search Service ? Operational
Staging Server ? Operational
Databases ? Operational
Mongo Alpha ? Operational
3rd Party APIs Operational
Bitbucket SSH ? Operational
Bitbucket Website and API ? Operational
Disqus Admin Operational
Disqus API Operational
Disqus Embed Operational
GitHub ? Operational
Google Apps Drive Operational
Google Apps Calendar Operational
imgix API Service Operational
imgix Rendering Infrastructure Operational
AWS - Amazon Web Services Operational
AWS Elastic Cloud Compute Service Operational
AWS Relational Database Service Operational
Atlassian Bitbucket Pipelines Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
had a major outage
had a partial outage

No incidents or maintenance related to this downtime.

System Metrics Month Week Day
CEO Application
Fetching
Past Incidents
Apr 21, 2019

No incidents reported today.

Apr 20, 2019

No incidents reported.

Apr 19, 2019

No incidents reported.

Apr 18, 2019
CEO Outage
Resolved - CEO Application cluster was unavailable for about 10 minutes this evening. The source of the outage has been identified and corrected.
Apr 18, 21:30 EDT
Several sites are under heavy load
Resolved - This incident has been resolved.
Apr 18, 10:54 EDT
Monitoring - Database balancer decided it no longer wanted to balance things and was shunting all traffic to a single instance. Contacting the service provider to determine the cause, but continuing to monitor the issue.
Apr 18, 10:19 EDT
Identified - Several sites are under a very sudden burst of heavy load. Backup databases are starting as quickly as possible and response times should return to normal shortly.
Apr 18, 10:10 EDT
Apr 17, 2019
Search Index Delayed
Resolved - This incident has been resolved.
Apr 17, 09:33 EDT
Monitoring - Search indexing has returned to normal, we are monitoring performance.
Apr 17, 09:08 EDT
Identified - Search index updates are delayed while we investigate an issue with search database connectivity. New content will be indexed, but the indexing rate will be slowed.
Apr 16, 22:33 EDT
Database Updates
Completed - Over the past few weeks we've been investigating why pagination has become slower and last night the dam finally broke. An update to one of the back end databases crashed a search indexer which caused a connection dog-pile on the back end database cluster which in turned caused dropped connections and timeouts. We were able to determine the cause (the crashed index) and mitigate it by shunting traffic to a secondary cluster for the night.

This did not impact the front end sites, as those run on a completely separate hardware and software cluster.

This morning we replaced the search indexer, updated the database indexes to mitigate the pagination slowdowns and tweaked the connection handling settings.

We will continue to monitor the system throughout the day and evening.
Apr 17, 09:33 EDT
Verifying - Database indexes have been applied and updated and we are verifying the updates.
Apr 17, 09:09 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Apr 17, 08:00 EDT
Scheduled - We will be updating data base indexes Wednesday morning between 8am and 10am EDT. You may not be able to access CEO intermittently during this time. This will not impact front end web sites.
Apr 16, 23:56 EDT
Apr 15, 2019

No incidents reported.

Apr 14, 2019

No incidents reported.

Apr 13, 2019

No incidents reported.

Apr 12, 2019

No incidents reported.

Apr 11, 2019

No incidents reported.

Apr 10, 2019

No incidents reported.

Apr 9, 2019

No incidents reported.

Apr 8, 2019

No incidents reported.

Apr 7, 2019

No incidents reported.

← Incident History Powered by Statuspage