Major mining server update released & issue last night

[Chris Sokolowski]

This morning, we released a major update of the mining server. This update vastly improves the performance of the server. There are no changes to the mining interface or the website. The biggest change you will see is that now the server will rarely, if ever, get behind in inserting shares, ensuring that all statistics remain up to date at all times.

Some of the code optimizations include:

• Simultaneously inserting multiple shares into the database
• Aggregating balance changes of multiple shares into one update rather than updating miner balances after every share
• Offloading database processing code from the mining server's main thread
• Changing the way the mining server receives block templates from a pull method to a push method.

I will still occasionally need to perform database backup or share archival, and that will cause the database to get behind for up to an hour, but I will make sure to do this in the very early morning so it will affect the least number of users.

In an unrelated issue, the database experienced an issue this morning that caused all shares to be lost from 12:00AM to 10:36AM. The cause was that an ID column overflowed a 32-bit integer. Steve thought he had corrected all the columns the last time this happened (http://forums.prohashing.com/viewtopic.php?f=4&t=749) but he accidentally forgot one column. Since the mining server was still accepting shares for that time, I will be crediting users for those 10 1/2 hours at the same rate as their average of yesterday February 20. I will update this post once I have made the changes.

Thanks for continuing to mine with us.

[Chris Sokolowski]

I want to update everyone regarding the issues we're been experiencing witht eh mining server over the past few days. The update was a complete rewrite of many of the mining server's functions, and as an unfortunate side-effect, it introduced new bugs.

The server has crashed a few times over the past week due to database resource overload. This issue should now be resolved. We have a very powerful database server with a 2TB RAID 10 of SSDs, 8 CPU cores, and 42GB of memory, but the previous mining server was barely using a fraction of the hardware's power. I had previously optimized the database to work with the old mining server code, which favored large amounts of disk reads, so I created a 40 GB disk cache. Although that disk cache limited the working memory of the database to 2GB, the database never needed that much memory. The new mining server has the opposite characteristics - it uses much more database working memory and many fewer disk reads reducing the need for disk cache. I have had to find the right balance through trial and error, and the database has crashed each time it exceed the working memory limit. I now have database working memory set to 20 GB, and the mining server is using about 4 GB of that memory, leaving room for growth.

We are aware of the lingering issues including the sporadic "miner has no coin assigned" and "miner is using an incorrect algorithm" rejected shares. Steve will have a fix for the latter error by the end of the weekend, and we are spending the weekend diagnosing the former error. There will likely continue to be server restarts each night to release bugfixes as we want to release bugfixes as soon as they are available.

[GenTarkin]

No worries! You guys rock!!! I greatly admire people who are transparent and honest about stuff thats going on! Thanks for your hard work!