Status as of Friday, September 14, 2018

[Steve Sokolowski]

Good morning!

• Later today, I'll be posting a discussion of the results of the most recent poll about important pool features. Keep an eye out for that!
• We are continuing to focus a lot of effort into the new monitoring system. There are now 216 services monitored, up from 71 a few days ago. Yesterday I was able to add things like checking to see whether there has been a sharp decline in the pool's reserve. I also added support for Postgres databases, to be notified when database load becomes excessive. Today, I'll be adding monitoring of the forums MariaDB database, along with RAID array monitoring. I expect a lot of effort to be focused on expanding the monitored services for the next week or two.
• Last night, it appears that there were some customers who experienced network connectivity issues, as indicated in the forums. However, given that the monitoring system did not detect any issues, we can be confident for the first time that these issues were not widespread. We can't help you if you don't submit a support ticket with additional information, so please do that if you encounter problems in the future.
• Work continues on why the Ethereum client is unable to maintain a connection to the network. We were able to determine that one daemon server appears to lock up due to a dramatic increase in disk writes, and during that time, the connection to many coins is lost. The RAID card's battery backup on that server had failed, which temporarily disabled the write-back cache, so we recharged it. There have not been any further issues since then. Chris spent the night trying to add RAID monitoring so that we'll be notified when the battery backups have failed like this, but the code only appears to work on Red Hat and is difficult to compile for Debian.
• Constance has completed code that will iterate through all the 1099-MISC data that was submitted prior to the USPS address checking having been implemented. The code will delete invalid addresses and request that affected customers resubmit corrected data. Because there is a limit to the number of concurrent queries the USPS allows, we will slowly request these corrections over the next month, with the intention of all 1099-MISC and W8-BEN data being as accurate as possible before November 1.