Why payouts didn't execute today

Post by **Steve Sokolowski** » Fri Dec 11, 2015 9:05 pm

Some customers contacted Chris earlier today asking why our bitcoin payouts didn't execute. He was on his way out to racquetball, so he didn't have a chance to respond. Before he left, he checked the database to see payout rows like this...

coin, transaction_hash, to_address, balance, transaction_fee
"Bitcoin";"728833ea5c23a44b030a04d3990eea1a93cd09ea43d04fe4828a4206bb2ec6c9";"1Hy3F8qm2p6PvkkVCka9JTVkhyg3Fh1NfF";0.00627600;0.00000441;"2015-12-11 05:19:10.166497-05"
"Feathercoin";"2776636e05b7bd92143b0ef1056cd470f81ad1a733ed96da4b0b686551e97b41";"6vuwV1qu9CeBLjdB479Sj2Zc2HRA4MgZNU";15.95541913;0.00000000;"2015-12-11 05:19:05.469972-05"
"Nocturnacoin";"adf6c4268f2ee09c3c8e7c8a7e891893642de2622fcb76220d34144c763890d7";"NS55nJccqQyonEc9KmwjCVCqVXsmfskuvv";0.05694630;0.00010000;"2015-12-11 05:19:06.041053-05"
"Canada_ecoin";"3a7e1c6961796e8d8756a61308c58ebcf43c7316781f61e956ae19fc21ed4efb";"CToPV1ujSP8NM8vpU21axgxNCt1s5FEcPp";23.03846880;0.00000000;"2015-12-11 05:19:06.936859-05"
"Canada_ecoin";"3a7e1c6961796e8d8756a61308c58ebcf43c7316781f61e956ae19fc21ed4efb";"CNb82S9FMi7Fwj6bfN7Yyt7uxkiHHf8nSR";59.60269247;0.00000000;"2015-12-11 05:19:06.936859-05"
"Peercoin";"baade9b9488a6f74f5e5a77fa4f7b914325436c341ce8f4a008ba9bc453f2a20";"PGG9MzZbeRY6d8EBqsaNthWewRMdeD56HJ";2.41281157;0.01000042;"2015-12-11 05:19:08.425991-05"
"Bitcoin";"728833ea5c23a44b030a04d3990eea1a93cd09ea43d04fe4828a4206bb2ec6c9";"1NdK7YYCqCzKkrV8xNRwvTXZVwGPYCDRWm";0.06968752;0.00000441;"2015-12-11 05:19:10.166497-05"
"Bitcoin";"728833ea5c23a44b030a04d3990eea1a93cd09ea43d04fe4828a4206bb2ec6c9";"1HhddeyYn4VppKNn4D9KSPDopv1fmGFxtW";0.00463742;0.00000441;"2015-12-11 05:19:10.166497-05"
"Bitcoin";"728833ea5c23a44b030a04d3990eea1a93cd09ea43d04fe4828a4206bb2ec6c9";"17SU8HrRBCwQ1noZiRmVWga8LUWoc4NdbG";0.05423821;0.00000441;"2015-12-11 05:19:10.166497-05"
"Bitcoin";"728833ea5c23a44b030a04d3990eea1a93cd09ea43d04fe4828a4206bb2ec6c9";"184S4X8NHYciEok2DhGDrffuEuQ5Xcd1NN";0.00614451;0.00000441;"2015-12-11 05:19:10.166497-05"
"Bitcoin";"728833ea5c23a44b030a04d3990eea1a93cd09ea43d04fe4828a4206bb2ec6c9";"19M744KQe7aK7xQNjgLyejsBiUT2nA7dyT";0.04127872;0.00000441;"2015-12-11 05:19:10.166497-05"
"Bitcoin";"728833ea5c23a44b030a04d3990eea1a93cd09ea43d04fe4828a4206bb2ec6c9";"1CcFxcJLFzrRLop98b7aMhCGh3i2r1G3VU";0.01698659;0.00000441;"2015-12-11 05:19:10.166497-05"
"Bitcoin";"728833ea5c23a44b030a04d3990eea1a93cd09ea43d04fe4828a4206bb2ec6c9";"1FaoTg3davZRGqewQYUWrdf31trUy4A8Bc";0.05091361;0.00000441;"2015-12-11 05:19:10.166497-05"
"Bitcoin";"728833ea5c23a44b030a04d3990eea1a93cd09ea43d04fe4828a4206bb2ec6c9";"1M3xBMb2YbyHyne3oWGasB22SwQv1xvTi";0.01478484;0.00000441;"2015-12-11 05:19:10.166497-05"
"Bitcoin";"728833ea5c23a44b030a04d3990eea1a93cd09ea43d04fe4828a4206bb2ec6c9";"19VHsnnkgzv9V45bi8kknZxCUDALWL2ecd";0.00467031;0.00000441;"2015-12-11 05:19:10.166497-05"
"Bitcoin";"728833ea5c23a44b030a04d3990eea1a93cd09ea43d04fe4828a4206bb2ec6c9";"1MiyfboP7LcVfqNsmJZsjegopHFEjYLXvQ";0.02211267;0.00000441;"2015-12-11 05:19:10.166497-05"
"Bitcoin";"728833ea5c23a44b030a04d3990eea1a93cd09ea43d04fe4828a4206bb2ec6c9";"15irgJ6f3i11ubooBhiU1MHWthsmbWwf4h";0.03159965;0.00000441;"2015-12-11 05:19:10.166497-05"
"Bitcoin";"728833ea5c23a44b030a04d3990eea1a93cd09ea43d04fe4828a4206bb2ec6c9";"18cxXbTLZSPquyzW5JzbKCN5XNEwMZc25o";0.02964991;0.00000441;"2015-12-11 05:19:10.166497-05"
"Bitcoin";"728833ea5c23a44b030a04d3990eea1a93cd09ea43d04fe4828a4206bb2ec6c9";"1NY3VTSNEj94pXWKpL3F5irBbTD7kYWtRG";0.00960952;0.00000441;"2015-12-11 05:19:10.166497-05"
"Bitcoin";"728833ea5c23a44b030a04d3990eea1a93cd09ea43d04fe4828a4206bb2ec6c9";"1AugwJJCJiySy9A3WmhK7MNVkEdiDSnxWY";0.09510808;0.00000441;"2015-12-11 05:19:10.166497-05"
"Bitcoin";"728833ea5c23a44b030a04d3990eea1a93cd09ea43d04fe4828a4206bb2ec6c9";"16NzUm5uxRStL22KibCE8NmxerjuK6NT8m";0.03195432;0.00000441;"2015-12-11 05:19:10.166497-05"

...and so on. From these rows, it sure looks like everything paid out. And there are pretty big numbers there: 0.09 BTC, 0.03 BTC, 0.05 BTC, and some amounts in other coins. Then Chris checked the block explorers for a random selection of coins. Sure enough, they had all paid out, except for bitcoin. The bitcoin transaction at 3 block explorers was nowhere to be found.

Perplexed, Chris spent half an hour trying to figure out what had happened. The debug log showed the API response from bitcoin, and it appeared correct. The code to execute payouts starts a transaction, attempts to insert rows, rolls back if balances would be negative, executes payouts, and then either commits or rolls back the transaction, so it's almost impossible that the database could be out of sync. The payout code had purposely had not been changed for 400 days after it had been shown to be reliable.

Chris was faced with a decision. The payout code had never presented a bug in the entire history of the pool, so either there was some new issue that had just appeared this morning, or something had happened to the transaction. The transaction was not confirmed, so the money wasn't gone. He decided to delete the rows from the table so that two days' balances would be paid at once tomorrow. Before he did that, however, he ran out of time and went to play racquetball, deciding he would do it when he returned.

When he got back, he checked once more before executing the query to revert, and discovered that the transaction was suddenly on the blockchain at https://blockchain.info/tx/728833ea5c23 ... 06bb2ec6c9. It turns out that the fee for the next block is too high, so the code requested that bitcoin-xt estimate a fee so that the transaction would be included within six blocks (12 cents). That would normally be the end of the issue, except that a huge surge of transactions occurred shortly thereafter and it actually took 80 blocks to confirm this transaction.

This is financial software. One would expect that when a confirmation time of 6 blocks is requested, the software would be reliable enough to be able to return a fee value that might need 5 blocks, or 9 blocks, or even 15 blocks. But this financial network is in such a crisis now that requesting 6 blocks actually means that 80 blocks could be required to service the transaction. This isn't Hearn's fault for writing a bad program or the Core developers' fault for creating a bad fee estimation algorithm. It wouldn't even be that bad if we knew definitively that it would take 80 blocks. The issue is that it's now officially impossible to depend upon the bitcoin network anymore to know when or if your payment will be transacted because the congestion is so bad that even minor spikes in volume create dramatic changes in network conditions. To whom is it acceptable that one could wait either 60 minutes or 14 hours, chosen at random?

It's ludicrous that people are actually writing posts on reddit claiming that there is no crisis. People were criticizing my post yesterday on the grounds that I somehow overstated the seriousness of the situation. Do these people actually use the bitcoin network to send money everyday?