Earnings errors resolved
Posted: Mon Mar 04, 2019 8:33 am
There were two earnings errors on March 2.
The first error involved an incorrect Dash price that was received at 5:30pm EST on March 2. The price caused miners who were mining Dash during that time to be credited 1000x normal earnings. After hours of investigation, Chris was unable to determine the root cause of the problem. However, he was able to determine the correct earnings and adjusted balances. For that minute when the price was incorrect, customers received our normal 15% bonus - but since the problem only occurred for one minute, the impact of the bonus was minimal. Customers who were overpaid as a result of the error have negative balances and have the good fortune of being able to earn back these debts at the now lower value, given that coin prices have declined overnight.
Because Chris was not able to determine the cause of the problem, he instead added yet another failsafe. Now, if pool earnings increase more than 33% between yesterday and today, payouts will be automatically delayed for manual intervention.
A separate issue affected one customer who sent a lot of ethash hashrate to the pool to statically mine Ethereum. The hashrate appears to have been rented from Nicehash in response to a configuration error where we incorrectly believed the block reward for ETH was 3 instead of 2. While we originally intended to pay at the incorrect rate of 3, we found that there are no blocks recorded as having been found during this time. The error is not a database problem, because the blockchain also does not contain any records of Ethereum blocks having been mined by us. There was an ETH block found last night by someone else as well. We're still trying to figure out what the bug is that affected this one customer; that is why the earnings for March 2 appear to be higher than most recent days.
The first error involved an incorrect Dash price that was received at 5:30pm EST on March 2. The price caused miners who were mining Dash during that time to be credited 1000x normal earnings. After hours of investigation, Chris was unable to determine the root cause of the problem. However, he was able to determine the correct earnings and adjusted balances. For that minute when the price was incorrect, customers received our normal 15% bonus - but since the problem only occurred for one minute, the impact of the bonus was minimal. Customers who were overpaid as a result of the error have negative balances and have the good fortune of being able to earn back these debts at the now lower value, given that coin prices have declined overnight.
Because Chris was not able to determine the cause of the problem, he instead added yet another failsafe. Now, if pool earnings increase more than 33% between yesterday and today, payouts will be automatically delayed for manual intervention.
A separate issue affected one customer who sent a lot of ethash hashrate to the pool to statically mine Ethereum. The hashrate appears to have been rented from Nicehash in response to a configuration error where we incorrectly believed the block reward for ETH was 3 instead of 2. While we originally intended to pay at the incorrect rate of 3, we found that there are no blocks recorded as having been found during this time. The error is not a database problem, because the blockchain also does not contain any records of Ethereum blocks having been mined by us. There was an ETH block found last night by someone else as well. We're still trying to figure out what the bug is that affected this one customer; that is why the earnings for March 2 appear to be higher than most recent days.