Multiple issues; delays in ticket resolution

News updates about the Prohashing pool
Forum rules
The News forum is only for updates about the Prohashing pool.

Replies to posts in this forum should be related to the news being announced. If you need support on another issue, please post in the forum related to that topic or seek one of the official support options listed in the top right corner of the forums page or on prohashing.com/about.

For the full list of PROHASHING forums rules, please visit https://prohashing.com/help/prohashing- ... rms-forums.
User avatar
Steve Sokolowski
Posts: 4585
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Multiple issues; delays in ticket resolution

Post by Steve Sokolowski » Thu Apr 05, 2018 8:11 am

I'm posting an announcement here to make customers aware of multiple issues that are currently going on.

First, there is a problem related to shares that are worth too much, which I think is related to static coin mining. This is the first task I'm investigating today and all other issues are of lower priority. I hope to have a fix for this issue later this morning, after which time I'll do a mining server release.

Second, there will need to be share corrections for the results of the first problem, once it is fixed. Chris will probably work on those this afternoon.

Third, there is an issue with equihash mining not coming online. I hope to investigate that later today and figure out what's wrong. You can see that the hashrates for equihash are not appearing.

Fourth, the mining server installation will still go on as scheduled today to reduce load on the existing servers. That will happen tonight. Something changed this morning that has caused some of the servers to have significantly increased load, and that's causing rejected shares again.

Fifth, if there is time, I need to get back to the "low luck miners" investigation. Some customers were able to reproduce the issue, but I haven't yet had time to analyze the test data due to the other issues. Once the other issues are resolved, I'll be able to make some progress on this problem.

Finally, the number of open support tickets has increased to more than 100 and continues to rise. Unfortunately, I don't have an ETA for when we will be able to address the tickets, given how these other problems have all come up at the same time. I apologize to those awaiting support, and we will try to get back to you as soon as possible.
mikiko
Posts: 6
Joined: Tue Apr 03, 2018 8:49 am

Re: Multiple issues; delays in ticket resolution

Post by mikiko » Thu Apr 05, 2018 8:41 am

Great, congratulate, your services really on high level. :oops:
I will wait about 1 week and than I revalue next steps... :D
User avatar
Steve Sokolowski
Posts: 4585
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Re: Multiple issues; delays in ticket resolution

Post by Steve Sokolowski » Thu Apr 05, 2018 9:02 am

I've taken equihash mining offline temporarily until I can determine why the exchanges for those coins keep going into error.
Last edited by Steve Sokolowski on Thu Apr 05, 2018 9:24 am, edited 1 time in total.
User avatar
Steve Sokolowski
Posts: 4585
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Re: Multiple issues; delays in ticket resolution

Post by Steve Sokolowski » Thu Apr 05, 2018 9:19 am

There appears to be something wrong with network connectivity today that's causing problems getting pricing data from exchanges. That's causing some coin errors. I'll investigate this issue later on in the day; since we'll be getting a new connection tomorrow, any work spent on this will be wasted anyway.
User avatar
Steve Sokolowski
Posts: 4585
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Re: Multiple issues; delays in ticket resolution

Post by Steve Sokolowski » Thu Apr 05, 2018 11:17 am

I think I figured out the cause of the weird balances. It turns out that the primary mining server, which handles the coin pricing, is under heavy load, while the other servers are not. That caused the pricing to become out of date, while shares are continuing being processed by the other servers. Therefore, the pricing data would be out of date for the shares being processed normally. I added a delay to the share processing to wait until current pricing data is available before writing shares.

It's not clear why the primary server started having high load last night. The most likely cause is that the IP addresses connecting to the server are imbalanced again. The new hardware will cut the load on the primary server by half automatically, so we'll see what happens when it is installed and then make a decision whether it's necessary to delay working on the "low luck" mining to implement a better load balancing algorithm or not.
User avatar
Steve Sokolowski
Posts: 4585
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Re: Multiple issues; delays in ticket resolution

Post by Steve Sokolowski » Thu Apr 05, 2018 11:18 am

I think I figured out the cause of the weird balances. It turns out that the primary mining server, which handles the coin pricing, is under heavy load, while the other servers are not. That caused the pricing to become out of date, while shares are continuing being processed by the other servers. Therefore, the pricing data would be out of date for the shares being processed normally. I added a delay to the share processing to wait until pricing data current as of the share submission time is available before writing shares.

It's not clear why the primary server started having high load last night. The most likely cause is that the IP addresses connecting to the server are imbalanced again. The new hardware will cut the load on the primary server by half automatically, so we'll see what happens when it is installed and then make a decision whether it's necessary to delay working on the "low luck" mining to implement a better load balancing algorithm immediately or whether it can wait until the future.
User avatar
Steve Sokolowski
Posts: 4585
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Re: Multiple issues; delays in ticket resolution

Post by Steve Sokolowski » Thu Apr 05, 2018 12:04 pm

OK, I figured out why the server load was so high in the first place. It turns out that there were some miners that were misconfigured, attempting to reconnect hundreds of times per second. By chance, these static coin miners were from IP addresses allocated to the primary mining server.

I configured fail2ban to send icmp-port-unreachable messages to these rigs, instead of using a more CPU-intensive process of validating their password arguments. Once I did that, CPU usage of the primary server is now averaging 40-60%.

Now I'm going to continue to investigate other issues and will post updates here as we have them.
User avatar
Steve Sokolowski
Posts: 4585
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Re: Multiple issues; delays in ticket resolution

Post by Steve Sokolowski » Thu Apr 05, 2018 2:24 pm

I think I figured out the equihash problem. For some reason, the zcash daemon was responding slowly. I restarted it, and zcash isn't going into error anymore.

I don't know what caused that issue, but it's fixed, so I'll put it on the list to investigate later.
User avatar
Steve Sokolowski
Posts: 4585
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Re: Multiple issues; delays in ticket resolution

Post by Steve Sokolowski » Thu Apr 05, 2018 3:23 pm

We're working on the balances now. Chris is going to correct the negative balances and the 2-hr period last night where Compucoins messed up the earnings. We'll get the payouts out within an hour or two.
User avatar
Steve Sokolowski
Posts: 4585
Joined: Wed Aug 27, 2014 3:27 pm
Location: State College, PA

Re: Multiple issues; delays in ticket resolution

Post by Steve Sokolowski » Thu Apr 05, 2018 6:05 pm

Chris has completed the share corrections due to the Verge hack and payouts will return to the normal schedule shortly.

In the end, he reverted the changes to all but 100 of the affected accounts, because the total deviation in those accounts was only expected to be $100. Enjoy the free money, if you earned any.

The other 100 accounts had significant deviations, so they were corrected to LTC + 15%, as previously listed.

Thanks for your patience while we made the balance corrections! At this point, all known issues that came up today have been resolved. Now, Chris will be starting on the new mining server. I expect to get through all the support tickets tomorrow morning, so that all the easy ones are resolved and every ticket at least has been acknowledged.
Locked