Chris got one of the errors!

[Steve Sokolowski]

I'm happy to announce that we finally got one of the bugs that's been causing constant problems for the past few months. This one took about 100 hours to figure out.

The cause of the problem was that when there is a network disconnect, block notifications incoming from the data get queued in an array, and then are resent upon reconnect. However, the array never gets cleared after the blocks are resent. Therefore, at the next disconnect, more blocks pile up in this array, and both the new blocks and the ones that were never cleared get resent. This continues for days until every disconnect causes tens of thousands of old blocks to be resent.

The result is that the disconnects start to occur more frequently over time, because of the huge amount of traffic destabilizing the network, and that old blocks also get resent after the new ones, causing the mining servers to record stale shares until the next new block is received normally. Eventually, the share inserters fail to be able to keep up with all the rejected shares.

Chris will be doing an emergency release in about 30 minutes to resolve this issue. It should not be necessary to restart the mining servers or the website. Tomorrow, we may need to take the system offline for a brief period to restore all the previous settings we erroneously changed because we thought that those changes would fix the problem, but info will be provided about that then.