Over the past few weeks we've been investigating why pagination has become slower and last night the dam finally broke. An update to one of the back end databases crashed a search indexer which caused a connection dog-pile on the back end database cluster which in turned caused dropped connections and timeouts. We were able to determine the cause (the crashed index) and mitigate it by shunting traffic to a secondary cluster for the night.
This did not impact the front end sites, as those run on a completely separate hardware and software cluster.
This morning we replaced the search indexer, updated the database indexes to mitigate the pagination slowdowns and tweaked the connection handling settings.
We will continue to monitor the system throughout the day and evening.
Posted 5 months ago. Apr 17, 2019 - 09:33 EDT
Database indexes have been applied and updated and we are verifying the updates.
Posted 5 months ago. Apr 17, 2019 - 09:09 EDT
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted 5 months ago. Apr 17, 2019 - 08:00 EDT
We will be updating data base indexes Wednesday morning between 8am and 10am EDT. You may not be able to access CEO intermittently during this time. This will not impact front end web sites.
Posted 5 months ago. Apr 16, 2019 - 23:56 EDT
This scheduled maintenance affected: CEO (API, Application).