Update on the pool downtime

[Chris Sokolowski]

Hi Everyone,

We sincerely apologize for the downtime. If you are not already aware, we have been the victims of denial of service attacks. I have been working all morning trying to restore service, so far unsuccessfully.

We just signed up for a CloudFlare account and routed the site through it, and that service is working fine. However, the issue that prevents full restoration of service is in the hands of our internet service provider, Verizon. We own 5 IP addresses, and two of those IP addresses appear to be blocked from within Verizon's network. We cannot run the pool without at least 4 IP addresses.

I have talked with Verizon's customer today, and the two techs to which I talked cannot find any issues with their service, which I find to be highly implausible. I have taken our working servers and did nothing but switch their IP address to the "blocked" address, and they are no longer connectable externally and have no ability to connect to connect to the internet from within the servers themselves. Similarly, I have taken the servers that were on the "blocked" addresses and changed their IP addresses to the working addresses, and they become connectable and can reach the internet from within the servers themselves. Furthermore, all 5 IP addresses can connect to each other through their public IP addresses, but they can't connect to anything beyond Verizon's first router. I don't see any reason why this behavior would be due to anything but the ISP itself. I am planning to wait a few hours and call again when there is a shift change so that a different customer service representative can actually fix the issue or at least provide us with a completely new set of IP addresses.

I want to assure everyone that there was no intrusion into our system and all money is safe; this is simply a denial of service attack. I will be providing updates as I have them. In the meantime, if anyone has alternative suggestions on what could be causing this "blocked" IP addresses, I would appreciate the comments.

[GregoryGHarding]

Chris, do you have Verizon DDoS Shield+ Protection with your service? if so they may be re-routing your traffic to a "mitigation facility".

Mitigation Activation. When mitigation is initiated, both legitimate traffic and DDoS attack traffic will be redirected to pre-deployed mitigation facilities either by: (a) Customer’s redirection, if mutually agreed by the parties, or (b) Verizon upon receipt of Customer’s notification. In order to receive DDoS Shield, Customer must have a public Internet circuit and publicly rerouteable IP address space via Border Gateway Protocol (“BGP”), of at least a Classless Inter-Domain Routing (“CIDR”) /24 for IPv4 or /64 for IPv6 or larger for either. All equipment associated with DDoS Shield is housed within Verizon facilities and remains the property of Verizon.

--http://www.verizonenterprise.com/extern ... R17_mk.htm

im not in the states so i'm not familiar with verizon or business class service, but i figured any input is good input,

[mjgraham]

I did have an issue once with ARP entries not changing or they were static so even though the IP changed it was using the old MAC address and wouldn't work that way.

[tmopar]

Strange.. hash flare ads are appearing in the forums now... ????

[GregoryGHarding]

Strange.. hash flare ads are appearing in the forums now... ????

my referal signature

[vinylwasp]

Hi Everyone,

We just signed up for a CloudFlare account and routed the site through it, and that service is working fine.

Chris, I can't connect to the main site through CF, it's saying the remote site is down. (Sydney POP)

Just be aware that when you switch CloudFlare on, all www connections appear to suddenly be coming from a small number of CF addresses. If you have other defensive technologies in place (such as a Verizon DDoS service) they may interpret this as a DDoS attack. You need to whitelist the public CF addresses (on their site) in all your other tech apart from your firewalls.

You should then configure your firewalls to block everything to port 80 and 443 from anywhere but CF for the IP's you're protecting.This creates a trusted access router for CF and blocks everything else. Ask your carrier to do this upstream if you can. Doing this gets tricky if you're multi-hosting on a single IP and relying on headers, but it sounds like you're not.

HTH.

[GregoryGHarding]

the main site remains down until chris can sort out the blocked address issues

[vinylwasp]

Thanks Gregory, wasn't sure what services were included in Chris's comment above. I'll just have to be patient then. Working with those big carriers can be sooo painful.
Cheers

[vinylwasp]

DNS Propogation is incomplete too, though I'm resolving correctly here.

Check it here: https://www.whatsmydns.net/#A/prohashing.com

[sirslayerjr]

I use to be on verizon fios.. here in california.. that they got there shit tight and im kind a surprise youre having issues??!!