Page 2 of 8

Re: Update on the pool downtime

Posted: Sun Jun 18, 2017 3:29 am
by AvPro
Speculating, but perhaps an entirely new set of IP address are needed. As the IP set you owned was public prior to cloudflare being set up, can the attacker bypass CF and go straight to your IP seeing as they know it?

Re: Update on the pool downtime

Posted: Sun Jun 18, 2017 3:33 am
by sirslayerjr
goto dslreports.com goto forum and verizon.. this site has the best support to get issues resolved quickly

Re: Update on the pool downtime

Posted: Sun Jun 18, 2017 3:35 am
by sirslayerjr
Chris states

Furthermore, all 5 IP addresses can connect to each other through their public IP addresses, but they can't connect to anything beyond Verizon's first router.

that means they just need to reset the verizon router

Re: Update on the pool downtime

Posted: Sun Jun 18, 2017 4:53 am
by sirslayerjr
or its the switch/router behind the verizon. from my experience, verizon routers are very picky on whats behind its modem/router . i used and old ass 3com switch and netgear n600 dual band . which worked good behind the verizon

Re: Update on the pool downtime

Posted: Sun Jun 18, 2017 7:07 am
by Steve Sokolowski
sirslayerjr wrote:or its the switch/router behind the verizon. from my experience, verizon routers are very picky on whats behind its modem/router . i used and old ass 3com switch and netgear n600 dual band . which worked good behind the verizon
The problem is back within Verizon's network. We don't have access to those routers.

One temporary solution I suggested to Chris is to obtain an IP address through Verizon Wireless for the trader, which does not need to be dynamic, and to reassign the working IP addresses to the server.

No matter what is done, it will be at least 12 hours until service is restored, but a more likely timeframe for complete stability is 36 hours.

SHA-256 mining has been delayed until July 23 because the attacks took up the time we needed to implement it.

Re: Update on the pool downtime

Posted: Sun Jun 18, 2017 7:26 am
by FRISKIE
Hey Steve - improved DDoS protection will be worth the wait.

Quick question though - how will this affect payouts?

Re: Update on the pool downtime

Posted: Sun Jun 18, 2017 8:01 am
by Steve Sokolowski
FRISKIE wrote:Hey Steve - improved DDoS protection will be worth the wait.

Quick question though - how will this affect payouts?
No money was lost, but the servers are inaccessible.

All site operations are offline for an indefinite period of time. If Chris can connect, he will pay all accounts down to zero. If it turns out that the only solution is to physically move the servers, then it could be a few days before he can gain access to the system again.

This issue is likely going to require a dramatic re-architecting of the system. The biggest problem we have right now is that we can't figure out what actually is happening because the customer service representatives at Verizon are saying their system is working. We gave up on Cloudflare because everyone there acted like a robot and couldn't tell us whether they could protect non-HTTP traffic.

I think that the first solution we'll try is simply increasing the connection speed to 1 Gb, and seeing if that is enough to allow us to determine what type of traffic is coming through and filter it. At the very least, we could get the site into a half-working state that doesn't trigger whatever happens with Verizon, and then just let the attacker run out of money.

If that doesn't work, then I think that the next step is to figure out if there is a VPS server that can get more bandwidth and filter connections as a proxy. This method will reduce profitability by about 1% due to increased orphan rates. The third choice is to physically move the servers to a datacenter that has enough bandwidth, but I don't think that any such datacenters that charge a low enough price exist where the pool could remain profitable.

Re: Update on the pool downtime

Posted: Sun Jun 18, 2017 8:14 am
by VanessaEzekowitz
Steve, how big of a machine/how much performance do you actually need? I've been a satisfied customer of OVH for a few years now, and their "So You Start" line of dedicated servers is decently-powered, while still being inexpensive. The downside (might not actually affect you) is that network is capped at 250 Mbps (total bandwidth/traffic is unmetered though).

They have decent DDoS protection. Last time one happened to my server, if I recall correctly, it was about 300 Gbps and lasted for a few hours, but it only caused a minor disruption in services/accessibility (for maybe one minute) before OVH's mitigation systems absorbed it.

Looks like you can forget Cloudflare - if I read their help center right, they do not protect port 3333, let alone dealing with stratum+tcp: https://support.cloudflare.com/hc/en-us ... work-with-

Re: Update on the pool downtime

Posted: Sun Jun 18, 2017 8:38 am
by FRISKIE
Hey Steve - thanks for addressing the payout issue clearly, it's going to become a top concern as the day goes forward.

You guys do what you have to do, I'll still be here, as will most of the rest of the community. These are the growing pains that plague online businesses that managed to catch attention of these cowards who don't have the balls to resolve grievances through discussion.

Re: Update on the pool downtime

Posted: Sun Jun 18, 2017 8:39 am
by Steve Sokolowski
VanessaEzekowitz wrote:Steve, how big of a machine/how much performance do you actually need? I've been a satisfied customer of OVH for a few years now, and their "So You Start" line of dedicated servers is decently-powered, while still being inexpensive. The downside (might not actually affect you) is that network is capped at 250 Mbps (total bandwidth/traffic is unmetered though).

They have decent DDoS protection. Last time one happened to my server, if I recall correctly, it was about 300 Gbps and lasted for a few hours, but it only caused a minor disruption in services/accessibility (for maybe one minute) before OVH's mitigation systems absorbed it.

Looks like you can forget Cloudflare - if I read their help center right, they do not protect port 3333, let alone dealing with stratum+tcp: https://support.cloudflare.com/hc/en-us ... work-with-
I found a place called Javapipe that seems to have a solution.

The main limitation is that we can't reconfigure the system because we have to maintain physical control of the hardware. Therefore, it seems to me that the best choice is to have a remote host filter traffic that goes to Verizon.

I found a place called Javapipe that appears to do just that, and the price is low enough that the pool would still be worth running while using it. I don't think that 300Gbps attacks are very common, and if they are, they have to be extraordinarily expensive because ISPs would quickly notice that hundreds of their customers suddenly saturated their upload capacity all at the same time and shut off those bots, permanently reducing the capabilities of the botnet.

When Chris wakes up in 30m, I'm going to sign up for them and we'll try setting up a VPN, where this remote server takes all the connections after whatever filtering they do, and simply retransmits the packets to the other machines.