Brief update as of Thursday, November 10

[Steve Sokolowski]

Today, we discovered, fixed, and released a major issue that had been plauging the system for at least a year.

The cause of a problem was a race condition in our "orphan mining" feature. Under normal circumstances, when one of our blocks is orphaned, we attempt to revert the main chain by taking that block's transactions and mining them into our next block that builds upon the orphaned block. This approach increases profit for us, because we earn twice the block reward if the new block is found, while still maintaining the integrity of the network. The same transactions are included, just one block later.

To accomplish this, we need to ask the daemon for the difficulty of the next block following ours. However, we discovered that a condition could occur where another pool finds a second block before while the coin daemon is calculating the difficulty. This problem has become worse recently because we are out of capacity and are waiting to transfer additional servers we have on hand to the hosting site on November 21. When the slow servers delayed the difficulty response, the pool would correctly discard the orphaned block information and mine the current block, but then the difficulty from the last block would be received and overwrite the correct difficulty. Some coins change their difficulties every block, meaning that some of our work requests would not match the daemon's difficulty.

If the difficulty had increased, that means that miners would see "lost" blocks that aren't actually lost; the pool simply tried to submit blocks that are too easy. That reduced earnings and resulted in poor coin selection. If the difficulty decreased, then money would be wasted, because the pool wouldn't submit blocks that were actually be valid, believing the difficulty was actually higher than it was. In both cases, rejected blocks would lead to static coin miners being disconnected.

All of these issues have been fixed. Over the next month, as orphan rates readjust, I expect that profitability will increase by 3% or so. Actual profitability is likely to decline due to the A4s coming online, but the decline will be reduced by the resolution of this bug.