All services offline Sunday evening

Post by **Steve Sokolowski** » Fri Jan 12, 2018 8:35 am

We're sorry to announce that all services will be offline for several hours during the evening of January 14 or the morning of January 15. We aren't willing to promise a specific time. The purpose of the downtime is to apply the Meltdown security vulnerability patches. Since these are dist-upgrades, they will require reboots. It is well known that these patches were rushed and not well-tested, so there is a risk of a system becoming corrupted and needing to be restored from backup, and that would require additional time.

I just upgraded 8 virtual machines in our development environment and didn't have any problems. We're going to allow the development environment to run for two days before moving on to upgrading production machines. There are about 40 virtual machines in the production environment, some of which will have a complex shutdown and reboot procedure. We'll try to keep the critical parts of the system online for as much of the period as possible (for example, the website might be offline but mining will still continue.) However, when the hypervisors are being updated, everything will be offline.

There is no ETA for when Debian will resolve the Spectre vulnerability. We were waiting on both to be available to make one upgrade, but we need to balance the downtime with the inability to install any new software on the virtual machines in the interim. With more and more forks and new coins occurring, we decided that it's time to take the hit and have Chris install the updates.

There will be four additional periods of downtime coming in the next few months: to resolve Spectre, to upgrade the BIOSes to fully protect against Meltdown, to upgrade to Postgres 10, and to switch to the Comcast Enterprise connection.