Some shares lost yesterday; downtime expected today

[Steve Sokolowski]

Unfortunately, some shares were not recorded yesterday. The cause of the problem was determined to be that one miner was earning Stronghandcoins, which have an extremely low value. 100m of these coins are worth $17.

When we designed the system in 2013, there were no coins had a market capitalization of more than 82 million (the most was litecoin.) The database columns were sized so that the running earnings for a single day were recorded with eight digits before the decimal point, and 24 after. For the first time yesterday, someone earned more than 100m of a single type of coin in a single day. As designed, the share inserters attempted to insert as many shares as possible. Most batches of 2500 shares were inserted successfully, but batches including shares that would cause that customer's balance to exceed 100m Stronghandcoins were not inserted.

The daily balances tables are causing the problem, so the issue went away at midnight. It is likely to recur this evening when the daily balances get high again if nothing is done. It appears as if someone changed the all-time balance tables several years ago to mitigate this problem, so it must have been encountered in 2016 and at the time it must not have been thought that so many coins would be earned in a single day.

Resolving this issue will require us to modify the size of the database columns, an operation that causes Postgresql to rewrite all of the rows in the table. These tables are usually constantly locked for writes during normal operation, so we'll likely have to take the system offline to alter the tables. There are millions of rows in these tables, so the downtime could take a while.

After the columns have been modified, Chris will then determine if any data exists that would allow him to issue share corrections for shares that were not inserted yesterday evening. He'll post more in about six hours about the downtime.

[srossman]

Thank you for the update Steve. I appreciate your quick response as always!

[Steve Sokolowski]

Thank you for the update Steve. I appreciate your quick response as always!

It turns out that the column was in status_daily_balance, which aggregates the coins that all miners have mined. As a result, it only has one row per coin per day, and we were able to rewrite all the rows in just one second. There was no service interruption and the bug was resolved.

[srossman]

great news! Glad to hear it wasn't a complicated fix for you guys.
Thanks again and have a great night.

[Steve Sokolowski]

great news! Glad to hear it wasn't a complicated fix for you guys.
Thanks again and have a great night.

Chris completed the share corrections for this event, which credited about $973 to customers. As far as we're aware, everything relating to this issue is now closed.