Mining server memory issues
Posted: Tue Jun 08, 2021 9:01 am
Good morning!
Recently, we've been receiving tickets from customers who state that they are "unable to connect" to the mining servers. We originally thought that these customers had Internet connectivity issues. While some do, we also discovered that the mining servers have been running out of memory and restarting, usually around every four hours. For five minutes after the restarts, some customers are unable to connect because the mining servers haven't yet received coin prices.
There has always been a memory leak in the mining servers, but we ignored it both because the servers would restart quickly, and because the restarts were occurring every few days. It's not clear why the problem has gotten worse recently.
We are addressing this issue in two ways. First, I'm working to reduce memory usage in the mining servers in the hopes of either fixing the leak, or if that is not possible to do in a reasonable timeframe, at least decreasing memory usage so that the servers restart less frequently. One task I already finished, and which will be active the next time each server restarts, is cleaning up old ethash DAGs sooner. Chris will also spend $500 to upgrade the memory in the mining servers from 32 to 128GB.
We anticipate that this problem will continue to improve over the course of this week, and hope that the new memory will arrive within two weeks. By June 21, we expect that at the least, we should be able to make the restarts occur 12 times less frequently.
Recently, we've been receiving tickets from customers who state that they are "unable to connect" to the mining servers. We originally thought that these customers had Internet connectivity issues. While some do, we also discovered that the mining servers have been running out of memory and restarting, usually around every four hours. For five minutes after the restarts, some customers are unable to connect because the mining servers haven't yet received coin prices.
There has always been a memory leak in the mining servers, but we ignored it both because the servers would restart quickly, and because the restarts were occurring every few days. It's not clear why the problem has gotten worse recently.
We are addressing this issue in two ways. First, I'm working to reduce memory usage in the mining servers in the hopes of either fixing the leak, or if that is not possible to do in a reasonable timeframe, at least decreasing memory usage so that the servers restart less frequently. One task I already finished, and which will be active the next time each server restarts, is cleaning up old ethash DAGs sooner. Chris will also spend $500 to upgrade the memory in the mining servers from 32 to 128GB.
We anticipate that this problem will continue to improve over the course of this week, and hope that the new memory will arrive within two weeks. By June 21, we expect that at the least, we should be able to make the restarts occur 12 times less frequently.