Prohashing Mining Pool Forums

Currently, share processing is delayed, and we're investigating the cause of the problem.

No money has been lost; there is just a delay in crediting balances as the database computes the shares. I'll keep everyone updated as we find out more.

We determined the cause of this problem.

It turns out that there is a bug in the PostgreSQL 12 to 13 upgrader, which is maintained by the Postgres developers. PostgreSQL 13 provides support for a faster index format, and part of the upgrade process is to create those new indexes.

For an unknown reason, the upgrader failed to create the newer indexes on some tables. Chris assumed, as was written in the documentation, that failure would simply mean that the old indexes would be used. Instead, what actually happened is that there are now multiple indexes that are all updated on each INSERT ON CONFLICT statement, while only one of them is actually needed for reads, as the bug in the upgrader did not delete the corrupt indexes. That causes four disk writes for some tables that only needed two before, halving the database's performance.

Chris is too tired to continue working on the issue, and it looks like the database has almost caught up. Tonight, he will contact the Postgres developers and attempt to figure out how to correct this problem in a live system that has 2TB of data.

Chris may have to back burner adding Coinbase payout for Tezos, assuming this is the same Chris. Think getting the indexes working properly is a bit more important.