Chris correcting balances again

[Steve Sokolowski]

Chris will be correcting balances again this evening, for a period of time this morning when shares were not recorded.

Our fix of increasing memory didn't work. It turns out that upon further investigation, the increase in memory usage is due to the payout software locking the database tables, as a failsafe against unforeseen bugs that would cause balances to go below zero.

The current procedure is to begin the transaction, see if a constraint is violated, and if not, then execute the payout, and then add the fees and commit. If a payout fails, then we rollback, and if a constraint is violated, the payout never happens.

The problem is that with so many more customers, some coins now take minutes to build all the payout transactions, and during that time shares can't be recorded because an exclusive lock is held by the payout software. These shares are stored in memory to wait until the database is available, which doesn't happen for hours, and the memory gets exhausted before then. We're working on a fix for this problem and hope to have it deployed by the weekend.

We're finding that all of these issues are software, not hardware, problems, which means that given enough time it should be possible to increase stability of the system. If there were not a bubble going on right now, we might consider hiring someone to accelerate the process, but most people here can recall what it was like in 2014 and how most businesses barely survived. I've been pessimistic lately about these price rises and am very afraid of the fallout after the next crash, so we're methodically resolving these issues one at a time by ourselves until we have a better idea of what the bottom will be.

[tmopar]

Steve, could you add a single character column to the shares table that is P for paid out, I for in progress, and blank for common entries and to maintain backwards compatibility.

If you added this column, just prior to a pay out event, you could set all shares which as of that moment are not P and are blank to I for in progress, then instead of a lock, you could just do a query filtering for these specific records thus no exclusivity and new fresh shares can continue to be inserted in the meantime into the same table as is without modifications.

Let me know what you think.
Tmopar

[GregoryGHarding]

what if you allow shares to be recorded to a secondary database only used for when payouts is running then if an no errors occur during payouts then commit the new db shares to the existing db instead of recording to memory.

[sirslayerjr]

im missing some of my balance .. almost 100 usd of doge and litecoins

[Steve Sokolowski]

Steve, could you add a single character column to the shares table that is P for paid out, I for in progress, and blank for common entries and to maintain backwards compatibility.

If you added this column, just prior to a pay out event, you could set all shares which as of that moment are not P and are blank to I for in progress, then instead of a lock, you could just do a query filtering for these specific records thus no exclusivity and new fresh shares can continue to be inserted in the meantime into the same table as is without modifications.

Let me know what you think.
Tmopar

Actually, we have a different solution that I finished coding already. The solution is to try and then commit all the database rows first. Then, we execute the payouts. If there is a failure, then we have the ids of the database rows stored in memory and delete them to reverse the changes.

Once tested, this way will eliminate the need to hold locks for longer than a few milliseconds at a time, and it's far less complex than adding new columns or creating a secondary database. I'm hoping that I can do the testing today.

[tmopar]

I hope the solution you have came up with will work and if so I will not continue to argue with success.

I would however argue that tt is a true statement that this new method depends on ram and there is still a finite amount of ram involved and if anything goes wrong either with the DB load or with the payout app, the lock is still in place and causing the aforementioned problems.

A single character is just a byte, with lots of over head, cumulative at the FS,DB level and VM levels, maybe 5 or 10 bytes per record I will grant you.

I still see a major decisive advantage in that it would preclude the need for locking whatsoever in this context. As such and this cannot be understated _shares are guaranteed to continue to flow as otherwise would normally_. Also the memory need not be tapped at all in doing so categorically removing from the real of possibility the problems listed in the op:

"..during that time shares can't be recorded because an exclusive lock is held by the payout software. These shares are stored in memory to wait until the database is available, which doesn't happen for hours, and the memory gets exhausted before then. "

The overhead to implement would merely be a single ~100 byte update query tied into the payout app, to basically prime and set all current unused shares to "ready for payout mode".

There would be a single modification to the where clause of the payout app's main workload query (~30 bytes), to focus for these freshly minted and unpaid records having an "I" in the new column.

Let me know what you think,
Tmopar

[GregoryGHarding]

adding a new column into an already established database would require sweeping all of the code associated with that database for possible compatibility issues. that's why i voiced a secondary database mirrored off the construction of the first then commit after successful payout run, that way no recording to ram necessary at all. but chris is more familiar with his systems then we are so i can only assume he knows what the best route is. good luck with testing the change.