Multiple issues; delays in ticket resolution

[Steve Sokolowski]

I'm posting an announcement here to make customers aware of multiple issues that are currently going on.

First, there is a problem related to shares that are worth too much, which I think is related to static coin mining. This is the first task I'm investigating today and all other issues are of lower priority. I hope to have a fix for this issue later this morning, after which time I'll do a mining server release.

Second, there will need to be share corrections for the results of the first problem, once it is fixed. Chris will probably work on those this afternoon.

Third, there is an issue with equihash mining not coming online. I hope to investigate that later today and figure out what's wrong. You can see that the hashrates for equihash are not appearing.

Fourth, the mining server installation will still go on as scheduled today to reduce load on the existing servers. That will happen tonight. Something changed this morning that has caused some of the servers to have significantly increased load, and that's causing rejected shares again.

Fifth, if there is time, I need to get back to the "low luck miners" investigation. Some customers were able to reproduce the issue, but I haven't yet had time to analyze the test data due to the other issues. Once the other issues are resolved, I'll be able to make some progress on this problem.

Finally, the number of open support tickets has increased to more than 100 and continues to rise. Unfortunately, I don't have an ETA for when we will be able to address the tickets, given how these other problems have all come up at the same time. I apologize to those awaiting support, and we will try to get back to you as soon as possible.

[mikiko]

Great, congratulate, your services really on high level.
I will wait about 1 week and than I revalue next steps...

[Steve Sokolowski]

I've taken equihash mining offline temporarily until I can determine why the exchanges for those coins keep going into error.

[Steve Sokolowski]

There appears to be something wrong with network connectivity today that's causing problems getting pricing data from exchanges. That's causing some coin errors. I'll investigate this issue later on in the day; since we'll be getting a new connection tomorrow, any work spent on this will be wasted anyway.

[Steve Sokolowski]

I think I figured out the cause of the weird balances. It turns out that the primary mining server, which handles the coin pricing, is under heavy load, while the other servers are not. That caused the pricing to become out of date, while shares are continuing being processed by the other servers. Therefore, the pricing data would be out of date for the shares being processed normally. I added a delay to the share processing to wait until current pricing data is available before writing shares.

It's not clear why the primary server started having high load last night. The most likely cause is that the IP addresses connecting to the server are imbalanced again. The new hardware will cut the load on the primary server by half automatically, so we'll see what happens when it is installed and then make a decision whether it's necessary to delay working on the "low luck" mining to implement a better load balancing algorithm or not.

[Steve Sokolowski]

I think I figured out the cause of the weird balances. It turns out that the primary mining server, which handles the coin pricing, is under heavy load, while the other servers are not. That caused the pricing to become out of date, while shares are continuing being processed by the other servers. Therefore, the pricing data would be out of date for the shares being processed normally. I added a delay to the share processing to wait until pricing data current as of the share submission time is available before writing shares.

It's not clear why the primary server started having high load last night. The most likely cause is that the IP addresses connecting to the server are imbalanced again. The new hardware will cut the load on the primary server by half automatically, so we'll see what happens when it is installed and then make a decision whether it's necessary to delay working on the "low luck" mining to implement a better load balancing algorithm immediately or whether it can wait until the future.

[Steve Sokolowski]

OK, I figured out why the server load was so high in the first place. It turns out that there were some miners that were misconfigured, attempting to reconnect hundreds of times per second. By chance, these static coin miners were from IP addresses allocated to the primary mining server.

I configured fail2ban to send icmp-port-unreachable messages to these rigs, instead of using a more CPU-intensive process of validating their password arguments. Once I did that, CPU usage of the primary server is now averaging 40-60%.

Now I'm going to continue to investigate other issues and will post updates here as we have them.

[Steve Sokolowski]

I think I figured out the equihash problem. For some reason, the zcash daemon was responding slowly. I restarted it, and zcash isn't going into error anymore.

I don't know what caused that issue, but it's fixed, so I'll put it on the list to investigate later.

[Steve Sokolowski]

We're working on the balances now. Chris is going to correct the negative balances and the 2-hr period last night where Compucoins messed up the earnings. We'll get the payouts out within an hour or two.

[Steve Sokolowski]

Chris has completed the share corrections due to the Verge hack and payouts will return to the normal schedule shortly.

In the end, he reverted the changes to all but 100 of the affected accounts, because the total deviation in those accounts was only expected to be $100. Enjoy the free money, if you earned any.

The other 100 accounts had significant deviations, so they were corrected to LTC + 15%, as previously listed.

Thanks for your patience while we made the balance corrections! At this point, all known issues that came up today have been resolved. Now, Chris will be starting on the new mining server. I expect to get through all the support tickets tomorrow morning, so that all the easy ones are resolved and every ticket at least has been acknowledged.