Status as of Saturday, October 20, 2018
Posted: Sat Oct 20, 2018 11:28 am
Good morning! Our focus on stability and reliability continues.
- Two days ago, bitcoin mining was disabled after a lost block was discovered. It turns out that the cause of the lost block is a serious race condition that affects all coins and, once the release is issued, will significantly improve profitability. In the currently deployed code, when a share is received, the order of operations is to create the potential block's header and store it in a member variable, then allow a different thread to calculate the hash, and then possibly submit the header from the member variable to the coin daemon. However, because the header is stored in a member variable, it could be changed to a non-block if the CPU returns from the threads in a different order than it went into them. That would cause the non-block to be submitted to the daemon and lost, and the valid block would not be submitted.
- This bug is likely responsible for losses of about $3m in revenue over the history of the pool. When released, the fix will improve profitability by 3-5% across all coins. At the same time as this release, Chris will also add a new column to store debug data for when blocks are lost, which will assist us in the next phase of stability improvements - that of figuring out why some specific coins lose blocks and go into error frequently. The system will go offline later today for a few minutes to execute the release.
- Next on the list is a memory leak in the mining server. The leak seems to cause one mining server to restart every 6 hours, so miners are disconnected once every two days, on average. I'm wondering whether these restarts are responsible for some of the "I can't connect" tickets we often receive, but this is a hard problem that will take days or weeks to resolve.
- Some customers are encountering difficulty with saving payout addresses. The cause of the issue is another side effect of stability improvements. We improved the code that governs when payout addresses can be saved to ensure that customers have to enter them immediately upon adding a coin. But some addresses were missing because some customers had mined static coins and had never been paid those coins. Chris ran a query to add those addresses. If you are having difficulty saving, you'll need to enter payout addresses for all coins, or forfeit the coins you don't wish to be paid.
- There are new features under development, but it will be some time before they're released. Vance is continuing to work on Monero mining, and Constance is starting with allowing miner grouping in the user interface.