Summary of maintenance on Memorial Day weekend
Posted: Sun May 21, 2017 8:46 am
We previously announced that Chris would be visiting the datacenter over the Memorial Day weekend to upgrade servers. The four goals of these upgrades are to:
- Improve the reliability of the forums by resolving the issues where the forums server disconnects from the Internet for ten minutes at a time
- Increase coin daemon server capacity to support SHA-256 mining
- Increase the number of workers the mining server can support
- Increase the resiliency of the system to power issues
- Chris will replace the failing network card in the forums server with a new card that will not disconnect the system periodically. This will result in 0.5-1 hour of downtime for the forums only.
- Chris will assemble and install a new piece of dedicated hardware that will only host the mining server. The system will have the fastest singlethreaded processor available today, which will double the performance of the mining server without any code changes. We expect we will be able to support 6000 workers after this upgrade, from the 3200 that I tested up to this point. When combined with software changes Chris plans to put out tonight, we can probably get the total up to 8000 concurrent workers. There will be 30 minutes of mining downtime while the virtual machine is copied to the new server and brought online.
- Chris will install a surplus HP server he purchased that just came off lease from a large corporation, which we estimate can support about 150 additional coin daemons on its 32 cores. No downtime is expected.
- Chris will install eight solid state disks in the existing and new coin daemon servers. Each of the four servers will have two additional SSDs installed. The servers are disk limited, and these extra disks will provide for lower orphan rates, more up-to-date block explorers, and support for the 175 SHA-256 coins. During this period, each server will be offline for an hour, during which the coins on that server will be unavailable for mining and payouts. Mining will continue, but profitability will fall for a few percent during that hour.
- Chris will install an additional uninterruptible power supply. The power outage last weekend caused a surge when external power was restored and the generator turned off. He will balance the servers between the two UPSs, making it less likely that one of them will be overloaded. Because all servers have two power cords, there will be no impact from this maintenance.