Mining server memory issues

[Steve Sokolowski]

Good morning!

Recently, we've been receiving tickets from customers who state that they are "unable to connect" to the mining servers. We originally thought that these customers had Internet connectivity issues. While some do, we also discovered that the mining servers have been running out of memory and restarting, usually around every four hours. For five minutes after the restarts, some customers are unable to connect because the mining servers haven't yet received coin prices.

There has always been a memory leak in the mining servers, but we ignored it both because the servers would restart quickly, and because the restarts were occurring every few days. It's not clear why the problem has gotten worse recently.

We are addressing this issue in two ways. First, I'm working to reduce memory usage in the mining servers in the hopes of either fixing the leak, or if that is not possible to do in a reasonable timeframe, at least decreasing memory usage so that the servers restart less frequently. One task I already finished, and which will be active the next time each server restarts, is cleaning up old ethash DAGs sooner. Chris will also spend $500 to upgrade the memory in the mining servers from 32 to 128GB.

We anticipate that this problem will continue to improve over the course of this week, and hope that the new memory will arrive within two weeks. By June 21, we expect that at the least, we should be able to make the restarts occur 12 times less frequently.

[-MaVerick-]

Can't go wrong with going 4 times the RAM!

[BMChris]

I love how transparent the Dev team are at Prohashing, it's great and really helps to build trust.

[Steve Sokolowski]

There seems to be some confusion about the scope of this issue.

The memory issue only interrupts miners who are doing Proswitching FPPS mining for about 2 minutes. Solo or PPLNS miners may see longer delays, but only if they are mining a small coin network. The anchor coins, like bitcoin or litecoin, would only be unavailable for 2 minutes.

If you are encountering an issue where your connectivity is unavailable for 30 minutes or an hour, you are having an unrelated issue. We aren't aware of any widespread issues that would cause disconnects as lengthy as that. Please submit a support ticket in that case, and we'll be glad to help.

[Steve Sokolowski]

Update: yesterday's improvements appear to have mitigated this problem enough that it is now occurring about half as often - about every 5 hours for half of customers and every 20 hours for the other half. I plan to continue making improvements today, and the memory is in the mail for delivery next week.

[Steve Sokolowski]

Chris found that the memory will arrive on June 14.

We expect there to be 15 minutes of mining server downtime around midnight EDT on June 15 to install this memory. No other services will be affected.

[Steve Sokolowski]

The shipment estimate has been delayed until June 15 now. Unfortunately, there is no vendor who will ship memory overnight, at any cost. However, we're looking at whether we can pay a delivery driver to drive 400 miles and deliver the parts.

At the same time, when a server runs out of memory and restarts, I'm working on a way to make the mining servers get prices from another mining server, rather than waiting for them to be provided by exchanges. That will minimize the interruptions when a restart occurs.

[Steve Sokolowski]

I've reduced the time it takes for a mining server to start up from 10 minutes to 10 seconds. This change will be available tonight.

[Banished_Privateer]

That's like 60x improvement