Upcoming performance improvements maintenance
Posted: Tue Oct 12, 2021 4:28 pm
Over the past week, we've spent a significant amount of effort improving the system's capacity. Even though we increased the capacity of the system in 2018 to support 250,000 users, it wasn't until Salad joined that it was discovered that the reliability of the live data is not at the same level as that of the mining servers and money auditing. The system has proven itself to work as designed by not losing any money or data, but simply recovering later when there aren't enough CPU cycles available isn't the level of service we want to deliver.
One of the immediate changes you should notice after these upgrades have been completed is that the data at the top of the forums pages will load almost instantly, instead after the delay caused by the current 100% CPU load on the server. Miner dashboard updates and balance updates should occur at more regular intervals, instead of appearing to occur in batches where there are few updates and then the system catches up later.
Here are the upcoming releases:
Chris and I are continuing to work 12-16 hours per day, 7 days per week, until the live data has at least four times the capacity needed to serve today's customers. We apologize for these frequent upgrades all of a sudden; however, we are spending as much money as possible right now on increasing capacity to make Prohashing an even better experience for all of you and for Salad's customers!
One of the immediate changes you should notice after these upgrades have been completed is that the data at the top of the forums pages will load almost instantly, instead after the delay caused by the current 100% CPU load on the server. Miner dashboard updates and balance updates should occur at more regular intervals, instead of appearing to occur in batches where there are few updates and then the system catches up later.
Here are the upcoming releases:
- On October 13, we will release the Crossbar server upgrade that will enable parallel proxy workers. We need to both change the configuration and also upgrade the version of Crossbar from 19.1 to 21.3. Miners won't be disconnected, and shares will be queued in memory if necessary, but there could be an interruption to live data and a delay before earnings are recorded while this upgrade is performed.
- On October 14, we will restart all mining servers to apply changes that will reduce orphan rates and increase profitability. Currently, the mining servers run queries like "select username, api_key from person" every minute. These queries worked well in 2013, but now can take many seconds to compute the results. The 100% CPU time prevents found blocks from being submitted to the network. The upgraded servers will only query for changes, reducing the rows processed every minute from 500,000 to 0 or 1. This upgrade is expected to disconnect miners for less than 10 seconds, unless a revert is necessary.
- On the weekend of October 16-17, we may upgrade Crossbar a second time to add another type of worker to parallelize additional types of requests, should the first upgrade on October 13 prove that we only added enough capacity for the some of the request types.
- On October 21, Comcast will be upgrading our dedicated bandwidth to give us 8 times more capacity, from 500Mbps to 2Gbps. This procedure requires multiple people in different locations to adjust the ISP's routers, fiber wavelengths, and other configuration at the same time Chris manually changes cables. Chris is working on a plan to try to minimize downtime, but it's difficult to foresee a circumstance where everything goes so smoothly that it can be done with less than 15 minutes of downtime.
Chris and I are continuing to work 12-16 hours per day, 7 days per week, until the live data has at least four times the capacity needed to serve today's customers. We apologize for these frequent upgrades all of a sudden; however, we are spending as much money as possible right now on increasing capacity to make Prohashing an even better experience for all of you and for Salad's customers!