Resolved profitability issues
Posted: Sun May 22, 2016 4:06 pm
After 100 hours of research, we have finally been able to determine why the pool's profitability had turned lower in recent times.
Chris had been running queries for about a week attempting to correlate low profits with a number of factors. He focused on what had changed at the time profits had turned lower, which was some time last month. We had released multiple features, and therefore Chris examined every one of them in turn. With nothing else seeming to have changed, it appeared that the reintroduction of the "r=" parameter to set a static work restart interval or the orphan mining bonuses were responsible. Over the past few weeks, we reverted each of the changes, but the issue persisted.
Yesterday, we considered shutting down until we could offer better service to our customers, but Chris had a great idea. While he had tried to correlate profit by coin, and by time, one of the correlations Chris had not thought to research was whether certain miners contributed less profit than others. It turns out that there were some miners who have luck that is significantly lower than average for the most difficult coins. There is a clear separation between these miners and other miners, and when low-luck miners are excluded, the difference in profitability is eliminated entirely.
After determining that some miners had worse luck than others, I spent a long time developing an algorithm that can filter these miners who are not contributing to the pool, and I'm now confident that they can be detected with a false-positive rate of less than 1 in 5000. By taking steps to address these miners, we can restore the pool to normal profitability.
We aren't exactly sure what would cause miners to have luck at finding litecoins that is near zero. If the issue is a firmware bug, the bug is pretty strange because only certain shares are missing, and the shares tend to be the ones that find blocks. If someone wrote software to purposely introduce bad luck, that also makes little sense because they gain no benefit and stand to lose money. While we can't make sense of the reason for this problem, it does seem to occur at cloud mining services, suggesting that people are renting this bad hashrate and aren't aware that they aren't getting what they're paying for. Perhaps the cloud hashing services themselves have bugs in their proxy software, because it again doesn't make sense why someone would set such miners to a cloud hashing service.
To eliminate the impact from these miners, we will be making an immediate change, and then will implement a long-term solution within two weeks. Beginning tomorrow, the query I described above will be run every morning, and when it detects one of these misconfigured miners, it will forfeit earnings for that miner from the previous day. This possibility of forfeiture has been included in the documentation so that everyone is aware of it. We recognize that forfeiture may occur in error, but at a rate where normal miners would expect to lose a day of earnings no more than once every 15 years. On the other hand, miners who exhibit this issue will find that they forfeit profits every day until they fix the issue. As always, Chris will be available to answer questions and assist users who are targeted by the algorithm by explaining the problem and assisting in its correction.
In the long term, we can completely eliminate this problem by assigning these miners to lower-difficulty coins. Because the miners only seem to lose the highest difficulty shares, they can be be assigned to the most profitable coins that are low difficulty, where they will lose almost zero earnings due to luck. Once implemented, these miners will actually assist the rest of the pool, becuase the rest of the pool with normal luck miners will be able to mine more difficult coins and have fewer work restarts, earning them greater profits instead. These suboptimal miners would be unwanted at other pools, so we can advertise that we welcome people that have this bug. Once we implement this change, we will eliminate the forfeiture because it will no longer be necessary.
We expect profitability to return to normal by tomorrow. Thanks to everyone for your patience, and we look forward to releasing the permanent fix within the next few weeks!
Chris had been running queries for about a week attempting to correlate low profits with a number of factors. He focused on what had changed at the time profits had turned lower, which was some time last month. We had released multiple features, and therefore Chris examined every one of them in turn. With nothing else seeming to have changed, it appeared that the reintroduction of the "r=" parameter to set a static work restart interval or the orphan mining bonuses were responsible. Over the past few weeks, we reverted each of the changes, but the issue persisted.
Yesterday, we considered shutting down until we could offer better service to our customers, but Chris had a great idea. While he had tried to correlate profit by coin, and by time, one of the correlations Chris had not thought to research was whether certain miners contributed less profit than others. It turns out that there were some miners who have luck that is significantly lower than average for the most difficult coins. There is a clear separation between these miners and other miners, and when low-luck miners are excluded, the difference in profitability is eliminated entirely.
After determining that some miners had worse luck than others, I spent a long time developing an algorithm that can filter these miners who are not contributing to the pool, and I'm now confident that they can be detected with a false-positive rate of less than 1 in 5000. By taking steps to address these miners, we can restore the pool to normal profitability.
We aren't exactly sure what would cause miners to have luck at finding litecoins that is near zero. If the issue is a firmware bug, the bug is pretty strange because only certain shares are missing, and the shares tend to be the ones that find blocks. If someone wrote software to purposely introduce bad luck, that also makes little sense because they gain no benefit and stand to lose money. While we can't make sense of the reason for this problem, it does seem to occur at cloud mining services, suggesting that people are renting this bad hashrate and aren't aware that they aren't getting what they're paying for. Perhaps the cloud hashing services themselves have bugs in their proxy software, because it again doesn't make sense why someone would set such miners to a cloud hashing service.
To eliminate the impact from these miners, we will be making an immediate change, and then will implement a long-term solution within two weeks. Beginning tomorrow, the query I described above will be run every morning, and when it detects one of these misconfigured miners, it will forfeit earnings for that miner from the previous day. This possibility of forfeiture has been included in the documentation so that everyone is aware of it. We recognize that forfeiture may occur in error, but at a rate where normal miners would expect to lose a day of earnings no more than once every 15 years. On the other hand, miners who exhibit this issue will find that they forfeit profits every day until they fix the issue. As always, Chris will be available to answer questions and assist users who are targeted by the algorithm by explaining the problem and assisting in its correction.
In the long term, we can completely eliminate this problem by assigning these miners to lower-difficulty coins. Because the miners only seem to lose the highest difficulty shares, they can be be assigned to the most profitable coins that are low difficulty, where they will lose almost zero earnings due to luck. Once implemented, these miners will actually assist the rest of the pool, becuase the rest of the pool with normal luck miners will be able to mine more difficult coins and have fewer work restarts, earning them greater profits instead. These suboptimal miners would be unwanted at other pools, so we can advertise that we welcome people that have this bug. Once we implement this change, we will eliminate the forfeiture because it will no longer be necessary.
We expect profitability to return to normal by tomorrow. Thanks to everyone for your patience, and we look forward to releasing the permanent fix within the next few weeks!
