Status as of Friday, June 9, 2017
Posted: Fri Jun 09, 2017 8:40 am
Here's a brief update:
- We seem to have stabilized the system through temporarily increasing difficulties to reduce the number of shares being inserted.
- But some of the efforts have resulted in permanent improvements. Chris finally figured out the "open file descriptors" problem with WAMP - that turned out to be caused by too many connections, due to an explosion in API usage recently. The API as implementation on the website should also be more reliable now, keeping the live data from disappearing.
- Chris also figured out that another cause of the crashes in the middle of the night (after fixing the payouts, which had been the previous cause) was the vacuuming. There is an "nThreads" parameter, which was set to 2, and now he set it to 1. That seems to have stopped the server from getting behind, while still keeping the disks from overloading.
- Our goal this weekend will be to eliminate data like block hash of the block the miner is building upon, the IP of the user who submitted a share, and other data that is unnecessary to store in the database for shares. A block hash has takes up 64 bytes, whereas other data in a share row like double precision difficulty takes up only 8 bytes. Four weeks from now, once the data containing these columns has fallen off the back of the archive process, we will delete the columns that now contain only nulls and VACUUM FULL the table, reducing its size dramatically and allowing us to move it back to the faster disks. Plus, since we can ignore this data now, there is less CPU load writing it to memory and creating Strings, and after an initial risk there will be less stuff around to cause bugs. We'll be investigating what other data we can ignore and delete without reducing the featureset of the system.
- Chris corrected balances last night. There haven't been any interruptions so far today. If there are, Chris will correct balances again at midnight before payouts.
- Two factor authentication was delayed by these issues. The code is there but has a security vulnerability that is yet to be fixed. I'm not going to devote effort to fixing it and releasing it until we can do the data reduction mentioned above.
- We ask for patience with payout issues. We'll be glad to assist if it's been 24 hours since your payout was due. However, we don't guarantee a payout time of day because the bitcoin network is overloaded and unreliable. Our efforts to move the system to litecoin as a primary coin will result in more dependable payout times once the system has more capacity.