DDoS

[Edge100x]

A different group is taking credit for the Steam problems.

Tonight isn't a good night for DDoS attacks, overall. Not because we're seeing them here (it has actually been more quiet than other nights), but because of these attacks on EA and Steam. They'll be rapidly trying to learn the lessons that we've already learned regarding these large attacks, and once they figure things out, scrambling to try to follow the same steps that we've taken. Both of these attacking groups hit our customers months ago, as well as just recently, and we intimately understand the issues.

I've reached out with my usual offer to help EA/DICE in troubleshooting their attack.

[rustydusty1717]

Whether the information is correct or not, this was just posted:

(link removed by staff)

[kraze]

Whether the information is correct or not, this was just posted:

(link removed by staff)

As much as this information may need to be out there, we just can't have that posted here. I had to remove it.

[TacTicToe]

Not sure if it is related or not, but our website seems to be down now. I just sent in a ticket about it. Steam is still down. I hate these bastards.

[Edge100x]

TacTicToe, I didn't see any downtime for hosted website machines tonight, or any notable attacks on customers here, so that was likely unrelated.

[null]

Edge, you guys have a lot of POPs, even spanning two continents. Have you ever considered using anycast for DDoS mitigation? (That is, announce your space out of every POP, perform filtering at every POP, then tunnel to the actual server's location.)

This would work really well for stateless filtering (e.g. blocking amplification attacks). Sure, L7 attacks are still possible, but not particularly common against game servers.

[rustydusty1717]

My apologies, check their Twitter for all the info:

https://twitter.com/DerpTrolling

[Edge100x]

Edge, you guys have a lot of POPs, even spanning two continents. Have you ever considered using anycast for DDoS mitigation?

Yes, and I've done it before in Seattle, but it doesn't work well as a general rule. Game servers are very latency-sensitive and the exact location matters to customers. Connections are long-running and routing changes frequently enough that clients would be frequently disconnected. We also don't have our own backbone with fiber connecting our facilities.

Anycast and DNS steering do work well for services that are transaction-based, such as DNS servers and websites.

[null]

Edge, you guys have a lot of POPs, even spanning two continents. Have you ever considered using anycast for DDoS mitigation?

Yes, and I've done it before in Seattle, but it doesn't work well as a general rule. Game servers are very latency-sensitive and the exact location matters to customers. Connections are long-running and routing changes frequently enough that clients would be frequently disconnected. We also don't have our own backbone with fiber connecting our facilities.

I think you have enough POPs, at least in the US, to minimize added latency. For example, I'm on Qwest/CenturyLink in AZ; all of my traffic that gets handed off to another carrier will go to LA. NFO also happens to have a POP in LA, so filtering there and passing off good traffic to the destination (say, NY) won't result in any real amount of added latency unless NFO's LA->NY path is worse than the one Qwest would take (it's most likely similar).

While private transport between DCs would be nice, it's not an absolute requirement.

I'm unsure why clients would be disconnected frequently? The idea would be to only perform stateless filtering (with router ACLs; for example, dropping UDP traffic from source port 53, and only allowing it to your own DNS servers), and tunnel everything else to the appropriate DC without even inspecting the payload.

In the future, such a service could also be expanded to include some level of stateless application-layer filtering for the most common games. For example, many application protocols will have a CRC field in their packet header (the presumption here is UDP), and this can be validated in real-time easily. Random traffic from generic tools won't have a valid CRC because such tools are designed for brute force and without specific knowledge of any particular application.

Even stateful filtering could be done, but it's a lot more work. The state table would have to be replicated in real-time between all DCs to prevent the disconnections you speak of. To ensure it doesn't become another DoS vector, state table entries can only be created after a source address is validated (stateless, of course).

The idea is to combat the most common attack tools that any 12-year-old kid with mommy's credit card can obtain. The kids we're talking about aren't intelligent enough to analyze network protocols and develop their own tools. Attacks that consist entirely of random data without any regard for the application protocol, aren't difficult to mitigate provided enough bandwidth is available. Look at that idiot's Twitter account - he's talking about booters FFS.

I have actually developed a Linux kernel module to perform this filtering for a certain game. This kind of filtering can be done at high PPS rates (GigE wire-rate with 64B packets) in kernel mode, provided the software is designed intelligently (for example, cache misses absolutely have to be minimized at this scale). I haven't yet played with any of the user-mode network stacks (e.g. Intel DPDK or PF_RING), but they look promising as well and should even open the door for doing this at 10GigE wire-rate. In fact, I just might have to order a few X520s to play around with and see what kind of PPS I can sustain.

[Edge100x]

I had an edited version of that which fleshed things out a bit more and removed the bit about disconnections, because I recognized that it would not be a major factor; it was lost when I had to scramble to look at the problems in NYC. The edited version also pointed out that trying to do anycast would add significant complexity and remove many performance benefits to our services, that customers run a variety of other applications and not just game servers, and that paying twice for a significant portion of our bandwidth would add a significant cost to our services that we would need to pass on to customers.

Most of the attacks we've had affect us recently (this week) have not been against us, but other customers of Internap at the same facility. An anycast setup would not help customers at locations in that case (and would actually make things worse). The problems have mostly not related to packet rate, and filters would not help; Internap's upstreams are being flooded out.

If you'd like to talk more privately, where I can explain what's going on in some more detail, I will do my best to fit that in. This thread is not a good place for details.

[fantasygirl]

....This thread is not a good place for details.

agree!

[null]

I had an edited version of that which fleshed things out a bit more and removed the bit about disconnections, because I recognized that it would not be a major factor; it was lost when I had to scramble to look at the problems in NYC. The edited version also pointed out that trying to do anycast would add significant complexity and remove many performance benefits to our services, that customers run a variety of other applications and not just game servers, and that paying twice for a significant portion of our bandwidth would add a significant cost to our services that we would need to pass on to customers.

Yeah, it's a complex topic and definitely a tradeoff like you said.

Most of the attacks we've had affect us recently (this week) have not been against us, but other customers of Internap at that facility. An anycast setup would not help customers at locations in that case (and would actually make things worse). The problems have mostly not related to packet rate, and filters would not help; Internap's upstreams are being flooded out.

Ah, I was under the impression at least some of these attacks were targeted at NFO customers.

(Somewhat off-topic and just a note: have you looked at the economics of purchasing transit from a few carriers and using Noction, vs. the current Internap? It would solve the problem of Internap's undersized transit capacity, but I'm just not certain if it's any more cost-effective at NFO's scale to "do it yourself.")

If you'd like to talk more privately, where I can explain what's going on in some more detail, I will do my best to fit that in. This thread is not a good place for details.

Thanks. I should clarify that I'm not (at this time) an NFO customer, though I've seen the attack activity from servers I frequent and that's what's brought me here. But, if you're willing, I'd love to chat about this and mitigation in general, preferably after all this has died down (I realize you have a lot on your plate right now).

Thanks again for taking the time to discuss this on here. Hopefully in the near future (and if you're willing) we'll be able to continue it off-forum. You can contact me at the email address on my forum account, or if you'd prefer I contact you I'm happy to do that as well. (Just please only after the current situation is taken care of - I understand you're busy and don't want to disrupt you.)

[Edge100x]

null, I won't provide further details here, but yes, we've considered the options that you've mentioned so far and had to rule them out, for the most part, although I never remove anything completely from the table.

I realize that you're trying to help, and I thank you for that, but it is hard to see the full picture from outside. I am on the inside, knowledgeable about these and previous attacks, and spending most of my time working on them (and have been for a long time now).

[bhzdclan]

down again

[Edge100x]

bhzdclan, each location is separate. Right now Internap's having problems in Atlanta, if that's what you're referring to.