Problems with share recording and live data

[Steve Sokolowski]

There are currently problems with share recording and live data. The cause of the problems is unknown, and they started occurring without any releases or action on our part, making them difficult to troubleshoot.

I'll continue to look into the issue. Right now, there is no ETA on when they will be resolved. Mining continues as normal and Chris will make share corrections after the issues have been resolved.

[CSZiggy]

why bother, no one cares.

[Steve Sokolowski]

Is WAMP down or just not showing on the main website?

Can bots still connect and mine the top coin off of it?

That data isn't available to the public and anyone trying to connect to it will find that it isn't useful for their purposes.

I'm having difficulty investigating because these things always tend to happen when we are the least equipped to fix them - for example, during the holidays when we do not have access to the development environment.

[CSZiggy]

why bother, no one cares.

[jpuche27]

Ok. What should we do? Stay with you or we have to change to another pool??

Will you recognize the profit lost?

[Steve Sokolowski]

OK, the issue appears to be resolved. The cause of the problem was that the system was waiting for daemons to respond to requests to obtain data about found blocks. I reduced the amount of wait time so that the system will continue inserting shares even if the blocks are not available. Unfortunately, I was not able to determine why the daemons were performing so poorly and why the problem happened suddenly without our having made any changes.

After the holiday weekend, I'll add code to, once the system is behind by a large amount, skip all the blocks and just insert shares as quickly as possible to get back to normal.

Chris will work on the share corrections later today. Yesterday's profitability is listed below litecoin, which indicates that the issue probably began yesterday for one of the share inserters and went unnoticed. Therefore, he will likely credit customers for both yesterday and today when hashrates are low. The corrections will be made in about 12 hours.

[Steve Sokolowski]

OK, I think I figured this one out. The system is running out of memory and dropping shares as a result.

What probably happened is that while more miners have joined us over time, Chris never increased the amount of memory and there were too many shares to be processed. When he gets back from lunch, I'll ask him to increase the amount of memory available to the share inserters.

[Steve Sokolowski]

For those who are interested, this is a case of unintended consequences. I determined that the cause of the out of memory issues is that we previously estimated the profitability from the master mining server's profitability. When we started making exact estimates from all the mining servers, the number of profitability updates climbed by a factor of 8.

Because the old profitability data is cleared after 15 minutes, and not by the number of records in memory, the updates were eight times as frequent, and the system ran out of memory despite all the code working great.

[jpuche27]

Hello steve. Can you tell me if the problem is fixed? Because i didnt get my money from yesterday. So i need to know a answer from you. Thanks

[Steve Sokolowski]

Hello steve. Can you tell me if the problem is fixed? Because i didnt get my money from yesterday. So i need to know a answer from you. Thanks

My understanding before I went to sleep was that yes, the problem should have been resolved overnight. About $10,000 was credited to customers.