Forums are back!

Hi all. Forums are back after about ~6h of downtime. Under 1 minute of data was lost in the process.

Thanks for being patient today. A post-mortem will be available on the blog once I get a little rest!

4 Likes

Gerred, nice job and a big thank you for all your hard work

1 Like

At ~3AM PST this morning the instance the forum software was moved onto ran out of disk space due to logs not properly being archived. The CloudWatch alert on the instance didn’t notify us, and the server was down for 3 hours while the Ionic team was asleep.

This has since been fixed. Disk space has been dramatically increased, and the monitoring/alerting is being reworked in order to prevent this from happening again. The outage was only on the application server, so no data loss occurred.

Thanks for your patience - at this time, it looks like all issues are resolved. I’ll be keeping a close eye on the forums over the next few days.