DDoS
- Edge100x
- Founder
- Posts: 12960
- https://www.youtube.com/channel/UC40BgXanDqOYoVCYFDSTfHA
- Joined: Thu Apr 18, 2002 11:04 pm
- Location: Seattle
- Contact:
Re: DDoS
A different group is taking credit for the Steam problems.
Tonight isn't a good night for DDoS attacks, overall. Not because we're seeing them here (it has actually been more quiet than other nights), but because of these attacks on EA and Steam. They'll be rapidly trying to learn the lessons that we've already learned regarding these large attacks, and once they figure things out, scrambling to try to follow the same steps that we've taken. Both of these attacking groups hit our customers months ago, as well as just recently, and we intimately understand the issues.
I've reached out with my usual offer to help EA/DICE in troubleshooting their attack.
Tonight isn't a good night for DDoS attacks, overall. Not because we're seeing them here (it has actually been more quiet than other nights), but because of these attacks on EA and Steam. They'll be rapidly trying to learn the lessons that we've already learned regarding these large attacks, and once they figure things out, scrambling to try to follow the same steps that we've taken. Both of these attacking groups hit our customers months ago, as well as just recently, and we intimately understand the issues.
I've reached out with my usual offer to help EA/DICE in troubleshooting their attack.
-
- This is my homepage
- Posts: 642
- Joined: Sun Sep 20, 2009 6:15 pm
Re: DDoS
Whether the information is correct or not, this was just posted:
(link removed by staff)
(link removed by staff)
Re: DDoS
As much as this information may need to be out there, we just can't have that posted here. I had to remove it.rustydusty1717 wrote:Whether the information is correct or not, this was just posted:
(link removed by staff)
@Kraze^NFo> Juski has a very valid point
@Juski> Got my new signature, thanks!
@Kraze^NFo> Out of context!
@Juski> Doesn't matter!
@Juski> You said I had a valid point! You can't take it back now! It's out there!
@Juski> Got my new signature, thanks!
@Kraze^NFo> Out of context!
@Juski> Doesn't matter!
@Juski> You said I had a valid point! You can't take it back now! It's out there!
Re: DDoS
Not sure if it is related or not, but our website seems to be down now. I just sent in a ticket about it. Steam is still down. I hate these bastards.
Re: DDoS
TacTicToe, I didn't see any downtime for hosted website machines tonight, or any notable attacks on customers here, so that was likely unrelated.
Re: DDoS
Edge, you guys have a lot of POPs, even spanning two continents. Have you ever considered using anycast for DDoS mitigation? (That is, announce your space out of every POP, perform filtering at every POP, then tunnel to the actual server's location.)
This would work really well for stateless filtering (e.g. blocking amplification attacks). Sure, L7 attacks are still possible, but not particularly common against game servers.
This would work really well for stateless filtering (e.g. blocking amplification attacks). Sure, L7 attacks are still possible, but not particularly common against game servers.
-
- This is my homepage
- Posts: 642
- Joined: Sun Sep 20, 2009 6:15 pm
Re: DDoS
Yes, and I've done it before in Seattle, but it doesn't work well as a general rule. Game servers are very latency-sensitive and the exact location matters to customers. Connections are long-running and routing changes frequently enough that clients would be frequently disconnected. We also don't have our own backbone with fiber connecting our facilities.null wrote:Edge, you guys have a lot of POPs, even spanning two continents. Have you ever considered using anycast for DDoS mitigation?
Anycast and DNS steering do work well for services that are transaction-based, such as DNS servers and websites.
Re: DDoS
I think you have enough POPs, at least in the US, to minimize added latency. For example, I'm on Qwest/CenturyLink in AZ; all of my traffic that gets handed off to another carrier will go to LA. NFO also happens to have a POP in LA, so filtering there and passing off good traffic to the destination (say, NY) won't result in any real amount of added latency unless NFO's LA->NY path is worse than the one Qwest would take (it's most likely similar).Edge100x wrote:Yes, and I've done it before in Seattle, but it doesn't work well as a general rule. Game servers are very latency-sensitive and the exact location matters to customers. Connections are long-running and routing changes frequently enough that clients would be frequently disconnected. We also don't have our own backbone with fiber connecting our facilities.null wrote:Edge, you guys have a lot of POPs, even spanning two continents. Have you ever considered using anycast for DDoS mitigation?
While private transport between DCs would be nice, it's not an absolute requirement.
I'm unsure why clients would be disconnected frequently? The idea would be to only perform stateless filtering (with router ACLs; for example, dropping UDP traffic from source port 53, and only allowing it to your own DNS servers), and tunnel everything else to the appropriate DC without even inspecting the payload.
In the future, such a service could also be expanded to include some level of stateless application-layer filtering for the most common games. For example, many application protocols will have a CRC field in their packet header (the presumption here is UDP), and this can be validated in real-time easily. Random traffic from generic tools won't have a valid CRC because such tools are designed for brute force and without specific knowledge of any particular application.
Even stateful filtering could be done, but it's a lot more work. The state table would have to be replicated in real-time between all DCs to prevent the disconnections you speak of. To ensure it doesn't become another DoS vector, state table entries can only be created after a source address is validated (stateless, of course).
The idea is to combat the most common attack tools that any 12-year-old kid with mommy's credit card can obtain. The kids we're talking about aren't intelligent enough to analyze network protocols and develop their own tools. Attacks that consist entirely of random data without any regard for the application protocol, aren't difficult to mitigate provided enough bandwidth is available. Look at that idiot's Twitter account - he's talking about booters FFS.
I have actually developed a Linux kernel module to perform this filtering for a certain game. This kind of filtering can be done at high PPS rates (GigE wire-rate with 64B packets) in kernel mode, provided the software is designed intelligently (for example, cache misses absolutely have to be minimized at this scale). I haven't yet played with any of the user-mode network stacks (e.g. Intel DPDK or PF_RING), but they look promising as well and should even open the door for doing this at 10GigE wire-rate. In fact, I just might have to order a few X520s to play around with and see what kind of PPS I can sustain.
Re: DDoS
I had an edited version of that which fleshed things out a bit more and removed the bit about disconnections, because I recognized that it would not be a major factor; it was lost when I had to scramble to look at the problems in NYC. The edited version also pointed out that trying to do anycast would add significant complexity and remove many performance benefits to our services, that customers run a variety of other applications and not just game servers, and that paying twice for a significant portion of our bandwidth would add a significant cost to our services that we would need to pass on to customers.
Most of the attacks we've had affect us recently (this week) have not been against us, but other customers of Internap at the same facility. An anycast setup would not help customers at locations in that case (and would actually make things worse). The problems have mostly not related to packet rate, and filters would not help; Internap's upstreams are being flooded out.
If you'd like to talk more privately, where I can explain what's going on in some more detail, I will do my best to fit that in. This thread is not a good place for details.
Most of the attacks we've had affect us recently (this week) have not been against us, but other customers of Internap at the same facility. An anycast setup would not help customers at locations in that case (and would actually make things worse). The problems have mostly not related to packet rate, and filters would not help; Internap's upstreams are being flooded out.
If you'd like to talk more privately, where I can explain what's going on in some more detail, I will do my best to fit that in. This thread is not a good place for details.
-
- New to forums
- Posts: 7
- Joined: Wed Jan 01, 2014 12:33 pm
Re: DDoS
agree!Edge100x wrote:....This thread is not a good place for details.
Re: DDoS
Yeah, it's a complex topic and definitely a tradeoff like you said.Edge100x wrote:I had an edited version of that which fleshed things out a bit more and removed the bit about disconnections, because I recognized that it would not be a major factor; it was lost when I had to scramble to look at the problems in NYC. The edited version also pointed out that trying to do anycast would add significant complexity and remove many performance benefits to our services, that customers run a variety of other applications and not just game servers, and that paying twice for a significant portion of our bandwidth would add a significant cost to our services that we would need to pass on to customers.
Ah, I was under the impression at least some of these attacks were targeted at NFO customers.Edge100x wrote:Most of the attacks we've had affect us recently (this week) have not been against us, but other customers of Internap at that facility. An anycast setup would not help customers at locations in that case (and would actually make things worse). The problems have mostly not related to packet rate, and filters would not help; Internap's upstreams are being flooded out.
(Somewhat off-topic and just a note: have you looked at the economics of purchasing transit from a few carriers and using Noction, vs. the current Internap? It would solve the problem of Internap's undersized transit capacity, but I'm just not certain if it's any more cost-effective at NFO's scale to "do it yourself.")
Thanks. I should clarify that I'm not (at this time) an NFO customer, though I've seen the attack activity from servers I frequent and that's what's brought me here. But, if you're willing, I'd love to chat about this and mitigation in general, preferably after all this has died down (I realize you have a lot on your plate right now).Edge100x wrote:If you'd like to talk more privately, where I can explain what's going on in some more detail, I will do my best to fit that in. This thread is not a good place for details.
Thanks again for taking the time to discuss this on here. Hopefully in the near future (and if you're willing) we'll be able to continue it off-forum. You can contact me at the email address on my forum account, or if you'd prefer I contact you I'm happy to do that as well. (Just please only after the current situation is taken care of - I understand you're busy and don't want to disrupt you.)
Re: DDoS
null, I won't provide further details here, but yes, we've considered the options that you've mentioned so far and had to rule them out, for the most part, although I never remove anything completely from the table.
I realize that you're trying to help, and I thank you for that, but it is hard to see the full picture from outside. I am on the inside, knowledgeable about these and previous attacks, and spending most of my time working on them (and have been for a long time now).
I realize that you're trying to help, and I thank you for that, but it is hard to see the full picture from outside. I am on the inside, knowledgeable about these and previous attacks, and spending most of my time working on them (and have been for a long time now).
Re: DDoS
bhzdclan, each location is separate. Right now Internap's having problems in Atlanta, if that's what you're referring to.