Chicago capacity upgrades?

[jimmy99]

NFO chicago capacity seems to be rather stagnant despite bandwidth and transit becoming cheaper. Many providers have substantially boosted links, notably to help withstand ddos attacks. with the recent surge of botnet floods, using upstream ACL's to drop a port is no longer viable. With home connections, and many other data centers offering higher and higher capacity links, these attacks have also grown in size, a 100 or even 600 gig flood isn't strange anymore. is there any plans for Chicago's capacity to be expanded anytime soon? even an extra 100-200 gigs could be a substantial upgrade in staggering these larger and nonstop floods.

[Edge100x]

I'm not sure where you're getting your information, but not much has changed, actually, on the attack side.

We have not been observing botnet-based attacks getting noticeably larger or more destructive recently.

Reflection attacks, while still not to historic levels, have ticked up in size a bit recently. Unfortunately, the capacity problems with these are normally upstream and our own capacity upgrades would not help with them. This is also often true of the largest botnet-based attacks -- an upstream will be internally saturated, and upgrading our links doesn't help in that case. We could add links to different upstream providers, but there are significant diminishing returns to doing so, partially because the inbound traffic from attacks isn't spread evenly among upstreams.

Unfortunately, no, bandwidth hasn't gotten much less expensive in the last few years, and cross-connects are pricier than they previously were (in some facilities, more than triple the cost), so the overall cost to upgrade is higher than it was before.

An extra 100-200 Gbps would not be a substantial upgrade. Usually if an attack is too large for us or our upstreams to handle right now, it's *much* too large, not just a little.

Demand and revenues also make it difficult to support upgrades.

In short, we're always reconsidering our bandwidth mixes and looking options, but the facts that upstreams are often the bottleneck (within their networks, which haven't been significantly upgraded), and the cost of upgrading is so high (while revenues are not going up) make upgrading complicated.

(I wouldn't put too much stock in the numbers listed at other providers as well, for the reasons I gave, on top of the fact that DDoS resistance in general is often greatly exaggerated elsewhere.)

[jimmy99]

Hi edge, as for where I got the info on the attack sizes it was horizoniq (inap) ddos report. And other sources. as for the upstream part, don't most providers with protection these days have an upstream they use to absorb attacks? a cheap upstream normally (historically cogent was used). many sources are reporting higher botnet based floods such as mirai variants, while Amps, while larger, isn't much of an issue because of how simple they are to filter with basic ACL'S. Does this mean NFO can't do much about larger botnet floods that rely on raw attack power vs a basic amplification method such as Chargen? as far as i know RFC 5575 with a supported upstream could easily rectify the situation, as NFO maintains a list of botnet ips, and rfc 5575 allows those to be pushed upstream to the providers routers, nipping it in the bud. a bit of a hardcore method, but with judicial use would be powerful.

[Edge100x]

Hi edge, as for where I got the info on the attack sizes it was horizoniq (inap) ddos report.

Wouldn't put much stock in it, then! INAP (and its successors) have always been completely, horribly, bad at understanding and mitigating DDoS attacks. They have always struggled to understand what I told them and never implemented effective solutions on their end. They also don't have effective monitoring.

And other sources. as for the upstream part, don't most providers with protection these days have an upstream they use to absorb attacks? a cheap upstream normally (historically cogent was used).

No, it doesn't work like that, unfortunately. You can't choose the upstream that an attack comes in under. Providers have upstreams, and can try to traffic shape them for various targets (performance, traffic loads, etc), but those adjustments apply to all inbound traffic, not just attack traffic. And, since you can't 100% control inbound traffic choices, if you advertise prefixes over an upstream at all, you might see an attack that overloads that specific upstream.

Cogent wouldn't be a particularly good attack sink, since they use Cisco equipment (which has poor egress ACL performance) and commonly have intra-city network bottlenecks. But, really, no single NSP would actually make a good attack sink right now.

many sources are reporting higher botnet based floods such as mirai variants

I can speak to what I see here. We see a wide variety of attacks. Previous years were worse, so far.

But it's the summer, and many skids are at home and researching vulnerabilities to expand their botnets, so I do expect a bit of a spike.

while Amps, while larger, isn't much of an issue because of how simple they are to filter with basic ACL'S.

I wish it were that simple. But internal upstream bottlenecks are a real thing, and upstreams aren't very good about backbone-level ACLs. Many have removed or curtailed them since the big DNS reflection spike in 2014, which has resulted in saturation. Lessons learned, lessons then forgotten..

Cisco devices, and lower-end devices from any manufacturer, also have performance problems with egress ACLs.

Chargen isn't really a problem. NTP isn't much of a problem, either. It's back to DNS nowadays. Possibly because DNS can't be blocked without also cutting out some legitimate traffic.

Direct attacks from spoofed IPs also cause problems, of course.

..RFC 5575...

This was obsoleted by RFC8955 and its updates (9117, 9184). I know of one major NSP that supports BGP flowspec, and it's HE, which we don't purchase from due to their pricing. BGP flowspec is also pretty limited, since it handles only basic ACLs. I'd like to try it out sometime for blocking high-bandwidth botnet IPs, which would help with certain repeat attacks, but right now, it's one of those things that's good in theory but hasn't yet worked out in practice.

Most NSPs are more interested in selling their own in-house (well, usually vendor-supported) scrubbing services than trying to support external DDoS mitigation solutions. They make more money that way.

[jimmy99]

Wouldn't put much stock in it, then! INAP (and its successors) have always been completely, horribly, bad at understanding and mitigating DDoS attacks. They have always struggled to understand what I told them and never implemented effective solutions on their end. They also don't have effective monitoring.

I see, i thought they improved over the years! seems they put more work into the graphical design of their website than the actual protection.

No, it doesn't work like that, unfortunately. You can't choose the upstream that an attack comes in under. Providers have upstreams, and can try to traffic shape them for various targets (performance, traffic loads, etc), but those adjustments apply to all inbound traffic, not just attack traffic. And, since you can't 100% control inbound traffic choices, if you advertise prefixes over an upstream at all, you might see an attack that overloads that specific upstream.

Cogent wouldn't be a particularly good attack sink, since they use Cisco equipment (which has poor egress ACL performance) and commonly have intra-city network bottlenecks. But, really, no single NSP would actually make a good attack sink right now.

Makes sense, didn't realize cogent had issues with ACL's

I can speak to what I see here. We see a wide variety of attacks. Previous years were worse, so far.

But it's the summer, and many skids are at home and researching vulnerabilities to expand their botnets, so I do expect a bit of a spike.

Yes, i think 2016/17 and 2014 were by far the most annoying years for me dealing with ddos. DNS+NTP era smashing everything with dumb amounts of bandwith. while 2016/17 were memcache and mirai smashing things, the mirai devs truly unleashed Pandora's box, their botnet source even to this day is still being utilized for high bandwith floods. i hope this dies down, I'd rather they abuse reflectors than botnets, the botnets are seriously annoying especially since they can have their traffic manipulated easier to bypass filters compared to an amp, which the best you can do is use nonstandard ports.

I wish it were that simple. But internal upstream bottlenecks are a real thing, and upstreams aren't very good about backbone-level ACLs. Many have removed or curtailed them since the big DNS reflection spike in 2014, which has resulted in saturation. Lessons learned, lessons then forgotten..

Cisco devices, and lower-end devices from any manufacturer, also have performance problems with egress ACLs.

Chargen isn't really a problem. NTP isn't much of a problem, either. It's back to DNS nowadays. Possibly because DNS can't be blocked without also cutting out some legitimate traffic.

Direct attacks from spoofed IPs also cause problems, of course.

how dumb, i thought they converted to BGP flowspec instead to make it easier and more uniformed for customers to use. also, about the DNS resurgence, what's up with that? shouldn't the reflectors have been significantly reduced over the years? are there new reflectors coming online? seems Amps are pretty useless on NFO, never had an issue of amps doing anything lol good work on that front.

This was obsoleted by RFC8955 and its updates (9117, 9184). I know of one major NSP that supports BGP flowspec, and it's HE, which we don't purchase from due to their pricing. BGP flowspec is also pretty limited, since it handles only basic ACLs. I'd like to try it out sometime for blocking high-bandwidth botnet IPs, which would help with certain repeat attacks, but right now, it's one of those things that's good in theory but hasn't yet worked out in practice.

Most NSPs are more interested in selling their own in-house (well, usually vendor-supported) scrubbing services than trying to support external DDoS mitigation solutions. They make more money that way.

Yes, i hope you also do try it out sometime! if it can be implemented capacity locally wont be as much of a problem! that alongside syncookies/syn_proxy would be great to see on nfo, syn floods remain the most popular flood universally because of their exhaustive capabilities. and yeah HE is the only one i know of that does it as well, other than luman used to do it, not sure if they still do, GTT from what I've seen offers it, but its hard to get, you need contact with the right person, otherwise they act like it doesn't exist. Regarding spoofed attacks, i think providers have gotten a lot better at blocking spoofed traffic compared to the mid 2010s which is good to hear any purely spoofed floods tend to be smaller in nature, it's much more useful if they amplify it 2-5 gigs of spoofed isn't doing to be dropping any routers these days unless its hitting a stateful firewall with tiny capacities. anyway thanks for the great convo edge!

[Edge100x]

Recently, and since we wrote here, we've been seeing a big spike in large reflection attacks that AS1299/Arelion (formerly Telia Carrier) and AS174/Cogent really struggle with, partially because they use Cisco gear (which doesn't have good ACL performance) and partially due to internal capacity issues. Arelion, in particular, seems to have real problems right now. The traffic gets filtered on their side during a large reflection/amplification attack, but their hardware problem causes heavy packet loss anyway.

Spoofed traffic is unfortunately still very much a thing. There are still many rogue providers that don't clamp down on it .

[joey44]

Wouldn't having an additional 200 or more gigs added using HE help? as jimmy said it seems it supports BGP flow spec. HE is a tier 1 too, their equipment is used to handle this type of stuff. upstreams dont work fast so nagging arelion might not work very well.

[Edge100x]

HE only supports flat-rate 100G ports, so they're prohibitively expensive.

HE also has its pros and cons. Trying to force all traffic inbound through HE would have performance consequences of its own..

[sarahkitty]

RIP the chicago datacenter. My TF2 server's got opps. Best of luck to all the NFOservers IT people probably going through hell right now. Inshallah upstream links will be upgraded

[rustydusty1717]

I have a VDS in Chicago and rarely see any DDoS attacks?

[Edge100x]

rustydusty1717, that's good! Most customers don't see DDoS attacks, but we have extensive measures in place to help those who do.

Recently, there has been a wave of attacks against popular game servers, and more specifically reflection attacks against them. We've found that our upstreams have been the weak link in our defense against these attacks, and not our own networks; filters on the upstream side are working to prevent the traffic from reaching us, but at the same time, we still see packet loss during the attacks. This has impacted other customers in Chicago, most notably, to the point that we posted an event to customer control panels that talks about it. Some clients are being impacted more than others, depending on how individual users route to the location.

[bobby77]

rustydusty1717, you may not see attacks, but I can assure you for us hosting game servers on dedicated servers it's not a matter of "if" you will get a ddos attacks, it's a matter of when. these attacks have been nonstop this summer, just constant bombardment from kids that don't have to go to school.