Page 1 of 1

System issue

Posted: Wed Feb 03, 2016 9:20 am
by Steve Sokolowski
Hi,

There's some sort of system issue that I'm told took the mining server offline. Unfortunately, I haven't been able to get in contact with Chris right now.

I'll keep trying to wake him up. Look here for more information as it becomes available.

Re: System issue

Posted: Wed Feb 03, 2016 12:11 pm
by Steve Sokolowski
The system has been brought back online by Chris. He will first execute yesterday's payouts, and will then determine how many shares, if any, were lost, and credit balances. Then, he'll try to figure out what happened.

He should be able to make a post here this afternoon with more info.

Re: System issue

Posted: Wed Feb 03, 2016 9:24 pm
by Chris Sokolowski
Thanks everyone for your patience during the unexpected downtime. The mining server crashed around 2:00 AM EST, but even before that, it was queueing shares for hours before that. The last share inserted was at 12:45 AM EST, meaning the mining server was 75 minutes behind at the time of the crash.

The cause of the crash was that the mining server was overloaded. Every time that one of the mining-enabled coins has a new block, the mining server re-analyzes the profitability of that coin, computes the optimal assignment of workers to all coins, and then re-assigns workers. As we have been adding coins over the past few weeks, the frequency at which one of the mining coins has a new block has been increasing, and the number of workers to assign has also increased. Last night, the server began receiving notifications of new blocks faster than it could compute the optimal assignments of miners, and as a result it crashed.

Steve has prepared a mining server update the reduces load by ignoring the step of analyzing worker assignments when we receive a new block on a coin that is less profitable than Litecoin. This single change should reduce the number of worker assignment computations by 95%. I will be testing and releasing this code tonight, which may result in a 30 second disconnect.

As compensation for the crash and missing shares, I have increased user balances by adding 1.75 hours of mining time to all users balances, which would be the earnings as if the mining server had crashed at 2:30 AM. All users mining between 9:45 PM and 12:45 AM EST yesterday received extra money proportional to their hashing power during that time.

As always, please fee free to comment if you have any questions. Thanks again for your support.

Re: System issue

Posted: Fri Feb 05, 2016 2:09 am
by Chris Sokolowski
It looks like the server still isn't performing up to our expectations. The 35 GH/s spike earlier today caused the server to get overloaded, and while no shares were lost, the server did get behind in inserting shares. I am fairly confident this is not a database issue but rather a mining server issue. We are working to profile performance and resolve bottlenecks. Over the next few days, you can expect a 30 second restart of the mining server every morning to deploy optimizations.