SNworks attempted a system upgrade early this morning. While the database updates occurred without incident the switchover to the new server node did not.
We quickly realized that were were not able to get the new servers responding properly and made the decision to roll back the upgrade. SNworks maintains parallel infrastructure during upgrades, so it was a simple matter to switch the network end points back.
Unfortunately, due to the amount of time that passed, our caches were cold and the entire system needed to warm them back up again. Couple this with the dramatic increase in traffic we’ve seen over the last week and you have a recipe for failure.
We were able to quickly expand our server fleet to take some of the excess load and things returned to normal. Total downtime is estimated to be less than 20 minutes.
We sincerely apologize for any trouble this issue caused this morning and are holding this upgrade back until we can determine exactly what happened.