Status as of Thursday, January 18, 2018

[Steve Sokolowski]

Good morning!

• The website is offline right now because, for an unknown reason, it suddenly started requiring a lot of database load. The share inserters got behind by an hour, so I shut the website down to allow them to catch up. After they have caught up, I'll restart the website and try to figure out what is causing the problem. My first thought is that the Meltdown security vulnerability fix caused the issue, because NVMe disk writes are by far the most affected task by the slowdowns, almost 30% slower than before.
• Yesterday, we discovered that there were about 1m calls per hour to /user/checkPassword on the website, which was causing excessive CPU load on the website. I spent the day adding code to write IP addresses making these calls to a file, and then iptables blocks these IP addresses before Java needs to load the Spring framework. That reduced CPU load from 1200% to about 110% by the end of the day, after 35,000 IP addresses were blocked.
• There are a few coins that will be discontinued when Debian 7 reaches end of life in May. Chris will be announcing these shortly. We don't have the source code for those coins, and therefore can't recompile them for Debian 9. We don't know if there are any copies of the source code for those coins remaining in the world at all. With Debian 7 likely to start having security vulnerabilities after May, we will need to shut down this last Debian 7 server, and the coins along with it.
• We found out that there is a form, W8-BEN, that international customers need to complete, basically certifying that they are not US customers. We'll release a file uploader for people to upload those forms later today. All that international customers will need to do is print the form, sign it, and scan or take an image of it. We aren't required to perform identity verification on these forms. US customers and customers who earn less than $600 are not affected.
• Chris is going to begin installing the SHA-256 coins next week. With the number of bugs having declined significantly, and this recent crash providing the industry some breathing room to improve systems, Chris thinks he's gotten ahead of the support tickets. We can't release SHA-256 mining, however, until the Enterprise internet connection is available, and that won't be available until March.
• Chris also installed rippled on the development and production servers, and I plan to enable the coin for payouts soon. However, since it will take a while to drive to the NFC championship game this weekend, don't expect Ripple payouts for at least 10 days.
• The Verge daemon finally finished reindexing after 3 days. This one is on the Verge developers - they released a client that deletes the blockchain at startup without prompting the user. We successfully processed the Verge payouts, but it's going to take a few hours of my time today to respond to all the tickets opened about this issue, so we appreciate your patience.

[pavvappav]

FYI - The W8-BEN is only valid for customers coming from the countries listed on this page where there is a tax treaty between the US and the foreign nation:
https://www.irs.gov/businesses/internat ... ies-a-to-z

The W8-BEN also requires that the individual filling out the form certify that their earned income complies with income covered under the treaty between the US and their country. A link to each treaty is available from the country page linked above.

[AppleMiner]

So for all the foreign customers who have already submitted (NOT US CITIZEN), will all of those be reset so they have to pick again and upload a form or since they already submitted for this year are they good to go?

[Steve Sokolowski]

So for all the foreign customers who have already submitted (NOT US CITIZEN), will all of those be reset so they have to pick again and upload a form or since they already submitted for this year are they good to go?

We'll delete the submissions for foreign customers a few at a time, so that Constance isn't overloaded with support tickets all at once. The fortunate part is that we don't need to perform identity verification with collecting passports or anything like that, so it should be simple for people to sign the forms and upload.

[spauk]

Yesterday, we discovered that there were about 1m calls per hour to /user/checkPassword on the website, which was causing excessive CPU load on the website. I spent the day adding code to write IP addresses making these calls to a file, and then iptables blocks these IP addresses before Java needs to load the Spring framework. That reduced CPU load from 1200% to about 110% by the end of the day, after 35,000 IP addresses were blocked.

if someone was trying to bruteforce hack into user accounts, is there a way for you to see which accounts they were trying to break into, and notify the users to update password or something, in case any attempts did get through? or is there nothing to worry about?

[Steve Sokolowski]

I think this sort of thing is pretty standard for almost every site on the Internet, and most likely these are bots that randomly scan every IP and start guessing weak passwords for common usernames when they find a webserver.

There's no reason to suspect anything is unusual. Other than a huge number of people being banned, the impact is minimal now that CPU usage is lower. I think the last report of someone who said her payout addresses were changed was about a week ago, despite there being 6000 active accounts.

[nemesis-t-warlock]

Are you also going to be adding a requirement to collect email addresses so you can inform people more effectively than just through the forum or when they realise they aren't being paid? Maybe they could also subscribe to the updates you do without having to monitor the forum.

[Steve Sokolowski]

Are you also going to be adding a requirement to collect email addresses so you can inform people more effectively than just through the forum or when they realise they aren't being paid? Maybe they could also subscribe to the updates you do without having to monitor the forum.

There is no legal requirement to collect E-Mail addresses, so we don't. Our plan is always to do the absolute minimum required by law and to use collected information as minimally as possible, which is why we are simply storing all this data on disconnected disks that won't be looked at for a year.

In the future, we might change the "password reset E-Mail" address to an "account E-Mail address," but that would still be opt-in and people who choose not to be notified wouldn't be able to receive messages.

[Eyedol-X]

Anxious to get going on Ripple -- when will we see BTG?

[nemesis-t-warlock]

There is no legal requirement to collect E-Mail addresses, so we don't. Our plan is always to do the absolute minimum required by law and to use collected information as minimally as possible, which is why we are simply storing all this data on disconnected disks that won't be looked at for a year.

In the future, we might change the "password reset E-Mail" address to an "account E-Mail address," but that would still be opt-in and people who choose not to be notified wouldn't be able to receive messages.

Understood, but just from the ability to inform people of importance announcements this must be beneficial. Pretty much every other pool does this and you have much greater complexity that calls for communication options. Sure, it isn't legally required but that's not really the point since you are doing things that are legally required with no reliable way of informing people if those legal requirements change except to stop paying them.