Hey everyone, quick update -
OTT had an outage on the forums that lasted at least 4 hours this morning. Visitors to the forums saw a series of database errors instead of the typical forum and post listing they expect to see. We eventually narrowed it down to a series of changes the hosting provider automatically installed which broke the connection to the database that contains the forum user profile, forum settings, posts and threads. Normally automatic updates are backward compatible and do not introduce disruption.
While I am happy that we were able to fix the issue, we did have a failure detecting it fast enough. Some of the automated systems setup to detect when the forum is down did not alert as expected which prolonged the outage period. We're going to update those monitors so that next time we are able to detect and mitigate much faster.
Anyway, just wanted to give the community a heads up that OTT is still alive and well and that next time we'll get better and faster at detecting and fixing issues like this. Thank you for hanging in there. Go team orange!
OTT had an outage on the forums that lasted at least 4 hours this morning. Visitors to the forums saw a series of database errors instead of the typical forum and post listing they expect to see. We eventually narrowed it down to a series of changes the hosting provider automatically installed which broke the connection to the database that contains the forum user profile, forum settings, posts and threads. Normally automatic updates are backward compatible and do not introduce disruption.
While I am happy that we were able to fix the issue, we did have a failure detecting it fast enough. Some of the automated systems setup to detect when the forum is down did not alert as expected which prolonged the outage period. We're going to update those monitors so that next time we are able to detect and mitigate much faster.
Anyway, just wanted to give the community a heads up that OTT is still alive and well and that next time we'll get better and faster at detecting and fixing issues like this. Thank you for hanging in there. Go team orange!