Status as of Wednesday, April 8
Posted: Wed Apr 08, 2015 9:15 am
Unfortunately, yesterday's update failed, and we reverted to the previous version.
The cause of the failure was that the server was reaching 100% CPU load for an unknown reason. Whatever was causing so much load delayed the processing of shares, which in turn caused those shares to be more likely to be rejected. It also caused work restart measurements to be extremely inaccurate because the overloaded server could not respond to the tests in a timely manner.
On the good side, we were able to successfully confirm that the website and the database are now in their final states. If you look at the site, you'll see all the documentation for work restarts, and you'll see that each share has new fields for "work restart time" and "average work restart penalty," among others. With the old code, these values are all the default of zero penalty for now. We'll be able to switch to the new code with a simple 1-minute restart.
Fixing this issue is going to be difficult because it does not appear on Dev. We think it is more likely to be caused by having more coins enabled on Prod, rather than having more miners, since Chris tested with many miners on Dev. If that is the case, then he can start a copy of the server and see if it encounters the same problems and we can work from there.
Normally, we would start immediately, but we are at an all-stop right now because of taxes. Last year, it only took me 10 hours to do taxes. This year, with the business, Chris is already at 25 hours and we have no estimate as to when he will finish. The complexity of the tax code dwarfs by far that of any coin daemon. Even before we started computing the business expenses, we already had 150 pages of stock sales to send to them.
At this point, we need to devote all of our available time to taxes until they are finished, putting fixing this release on hold. We hope that we will be done with the taxes within a week. Chris will continue to respond to customer service requests about the problems with the old code until he can get around to testing this new code. I apologize about the delay, and we hope to get back to moving forward as soon as the taxes are finished.
The cause of the failure was that the server was reaching 100% CPU load for an unknown reason. Whatever was causing so much load delayed the processing of shares, which in turn caused those shares to be more likely to be rejected. It also caused work restart measurements to be extremely inaccurate because the overloaded server could not respond to the tests in a timely manner.
On the good side, we were able to successfully confirm that the website and the database are now in their final states. If you look at the site, you'll see all the documentation for work restarts, and you'll see that each share has new fields for "work restart time" and "average work restart penalty," among others. With the old code, these values are all the default of zero penalty for now. We'll be able to switch to the new code with a simple 1-minute restart.
Fixing this issue is going to be difficult because it does not appear on Dev. We think it is more likely to be caused by having more coins enabled on Prod, rather than having more miners, since Chris tested with many miners on Dev. If that is the case, then he can start a copy of the server and see if it encounters the same problems and we can work from there.
Normally, we would start immediately, but we are at an all-stop right now because of taxes. Last year, it only took me 10 hours to do taxes. This year, with the business, Chris is already at 25 hours and we have no estimate as to when he will finish. The complexity of the tax code dwarfs by far that of any coin daemon. Even before we started computing the business expenses, we already had 150 pages of stock sales to send to them.
At this point, we need to devote all of our available time to taxes until they are finished, putting fixing this release on hold. We hope that we will be done with the taxes within a week. Chris will continue to respond to customer service requests about the problems with the old code until he can get around to testing this new code. I apologize about the delay, and we hope to get back to moving forward as soon as the taxes are finished.
See, I was going more for the "ironic meme" thing there, but it looks like I missed. My apologies. Just for the record, I'm a Democrat and a _huge_ fan of Obamacare and the unprecedented amount of good it has done, both in terms of the economy in general and in terms of 'hey, less dying is kinda the whole reason for countries to exist in the first place'.