Some shares lost yesterday; downtime expected today
Posted: Tue Mar 26, 2019 8:20 am
Unfortunately, some shares were not recorded yesterday. The cause of the problem was determined to be that one miner was earning Stronghandcoins, which have an extremely low value. 100m of these coins are worth $17.
When we designed the system in 2013, there were no coins had a market capitalization of more than 82 million (the most was litecoin.) The database columns were sized so that the running earnings for a single day were recorded with eight digits before the decimal point, and 24 after. For the first time yesterday, someone earned more than 100m of a single type of coin in a single day. As designed, the share inserters attempted to insert as many shares as possible. Most batches of 2500 shares were inserted successfully, but batches including shares that would cause that customer's balance to exceed 100m Stronghandcoins were not inserted.
The daily balances tables are causing the problem, so the issue went away at midnight. It is likely to recur this evening when the daily balances get high again if nothing is done. It appears as if someone changed the all-time balance tables several years ago to mitigate this problem, so it must have been encountered in 2016 and at the time it must not have been thought that so many coins would be earned in a single day.
Resolving this issue will require us to modify the size of the database columns, an operation that causes Postgresql to rewrite all of the rows in the table. These tables are usually constantly locked for writes during normal operation, so we'll likely have to take the system offline to alter the tables. There are millions of rows in these tables, so the downtime could take a while.
After the columns have been modified, Chris will then determine if any data exists that would allow him to issue share corrections for shares that were not inserted yesterday evening. He'll post more in about six hours about the downtime.
When we designed the system in 2013, there were no coins had a market capitalization of more than 82 million (the most was litecoin.) The database columns were sized so that the running earnings for a single day were recorded with eight digits before the decimal point, and 24 after. For the first time yesterday, someone earned more than 100m of a single type of coin in a single day. As designed, the share inserters attempted to insert as many shares as possible. Most batches of 2500 shares were inserted successfully, but batches including shares that would cause that customer's balance to exceed 100m Stronghandcoins were not inserted.
The daily balances tables are causing the problem, so the issue went away at midnight. It is likely to recur this evening when the daily balances get high again if nothing is done. It appears as if someone changed the all-time balance tables several years ago to mitigate this problem, so it must have been encountered in 2016 and at the time it must not have been thought that so many coins would be earned in a single day.
Resolving this issue will require us to modify the size of the database columns, an operation that causes Postgresql to rewrite all of the rows in the table. These tables are usually constantly locked for writes during normal operation, so we'll likely have to take the system offline to alter the tables. There are millions of rows in these tables, so the downtime could take a while.
After the columns have been modified, Chris will then determine if any data exists that would allow him to issue share corrections for shares that were not inserted yesterday evening. He'll post more in about six hours about the downtime.