Status as of Thursday, January 18, 2018
Posted: Thu Jan 18, 2018 8:57 am
Good morning!
- The website is offline right now because, for an unknown reason, it suddenly started requiring a lot of database load. The share inserters got behind by an hour, so I shut the website down to allow them to catch up. After they have caught up, I'll restart the website and try to figure out what is causing the problem. My first thought is that the Meltdown security vulnerability fix caused the issue, because NVMe disk writes are by far the most affected task by the slowdowns, almost 30% slower than before.
- Yesterday, we discovered that there were about 1m calls per hour to /user/checkPassword on the website, which was causing excessive CPU load on the website. I spent the day adding code to write IP addresses making these calls to a file, and then iptables blocks these IP addresses before Java needs to load the Spring framework. That reduced CPU load from 1200% to about 110% by the end of the day, after 35,000 IP addresses were blocked.
- There are a few coins that will be discontinued when Debian 7 reaches end of life in May. Chris will be announcing these shortly. We don't have the source code for those coins, and therefore can't recompile them for Debian 9. We don't know if there are any copies of the source code for those coins remaining in the world at all. With Debian 7 likely to start having security vulnerabilities after May, we will need to shut down this last Debian 7 server, and the coins along with it.
- We found out that there is a form, W8-BEN, that international customers need to complete, basically certifying that they are not US customers. We'll release a file uploader for people to upload those forms later today. All that international customers will need to do is print the form, sign it, and scan or take an image of it. We aren't required to perform identity verification on these forms. US customers and customers who earn less than $600 are not affected.
- Chris is going to begin installing the SHA-256 coins next week. With the number of bugs having declined significantly, and this recent crash providing the industry some breathing room to improve systems, Chris thinks he's gotten ahead of the support tickets. We can't release SHA-256 mining, however, until the Enterprise internet connection is available, and that won't be available until March.
- Chris also installed rippled on the development and production servers, and I plan to enable the coin for payouts soon. However, since it will take a while to drive to the NFC championship game this weekend, don't expect Ripple payouts for at least 10 days.
- The Verge daemon finally finished reindexing after 3 days. This one is on the Verge developers - they released a client that deletes the blockchain at startup without prompting the user. We successfully processed the Verge payouts, but it's going to take a few hours of my time today to respond to all the tickets opened about this issue, so we appreciate your patience.