Server rentals :: NFOservers.com

When an IP address is "null-routed", all traffic is locally blocked to that IP address, and the instruction to block all traffic is sent to upstream providers so that they do the same (whenever possible) through "Remotely Triggered Black Hole" (RTBH) advertisements. This prevents traffic to that IP address from touching NFO's network, and it prevents it from touching networks of our providers (and/or causes it to be dropped at their network edge).

Null-routes are very big deal and we take them seriously. They are only used here as an emergency stopgap measure during a flood-type Distributed Denial of Service attack (DDoS) that causes NFO's links to an upstream to be saturated, causes one or more of an upstream's links to its upstream providers (or its own internal ones) to be saturated, or causes NFO's router to be overloaded. In such events, the attack leads to packets being dropped for all customers at the same location, and the null-route is the only way to immediately stop the collateral damage. We have a system here that detects the packet loss and automatically implements null-routes, allowing them to occur quickly and limit the damage as much as possible.

Events that require null-routes are relatively rare. For most attacks, NFO's links to upstreams are large enough that we can absorb the attack traffic and filter it for our customers, avoiding the need for such a drastic event. When an emergency null-route is required, it is put in place for a limited time.

Every automatic null-route is fully investigated by NFO after it has been put in place to confirm that it was truly necessary and to determine what we can do to help the customer avoid them in the future. Steps that we frequently take include:

• Contacting our upstreams to determine if one of its links were saturated. If so, we may change the way that our prefixes are being advertised to distribute the traffic differently across their providers. We will also encourage upstreams (such as INAP) to upgrade their infrastructures.
• Determining whether an upstream ACL (filter) would be helpful. If an upstream was not having problems -- that is, if only NFO's connections was saturated -- and it is a simple attack that can be targeted with a basic ACL, it may be possible for an upstream to apply a filter to block the traffic before it reaches our network. In this case, we will ask for this and then remove the null-route once it has been applied.
• Contacting compromised hosts and reflectors. On behalf of our customers, we contact those who are involved with the attacks so that they can clean up their networks and systems.
• Notifying law enforcement contacts.
• Blocking botnet IP addresses at our router. For attacks that cause our router CPU capacity to be exhausted, blocking botnet IP addresses locally may help with further attacks.
• Infrastructure upgrades. We are always evaluating router hardware upgrades and connection upgrades on our end. However, we are limited in what we can do because the bottleneck is frequently with INAP and its upstreams, meaning that INAP has to upgrade before we can.

Because the purpose of a null-route is to prevent collateral damage to other customers, we can't allow the target of an attack that required a null-route to use multiple IP addresses in such a way that the targeted server is brought back online at an alternate IP address (in our experience, this almost always leads to the other IP address(es) also being attacked and causes significant additional damage). If we see this type of behavior, we will put a stop to it.

With a new wave of extremely large (100+ Gbps) attacks, we've been forced to issue more null-routes than usual recently -- sometimes several per day. Though we discuss how we handle null-routes in detail in the first post above, we still frequently hear specific questions about null-routes that we will answer here, for easy reference.

Can't you just block my attack?

We do filter attacks. We block attacks internally, the instant they occur. But, that doesn't prevent our upstream links from being saturated by the attacks -- it only stops our internal links from being saturated. With most null-routes, the attack saturated our upstream links, instead.

At all our locations, we exclusively buy from upstreams that can help filter certain types of attacks on their end, as well. We leverage these filters whenever possible.

The attack that I saw looks small! Why did you need to null-route?

We have a highly-tuned internal mitigation system that prevents most attack traffic from reaching target customers. Because customers see little of the attack traffic, attacks can look small. Most machines also have 1 Gbps network adapters, so the tens of Gbps that these attacks generate simply can't be seen.

If we issue a null-route, it means that the attack saturated our links to upstreams and/or caused problems for one of our upstream providers -- which means that it was extremely large. Automatic safeguards prevent null-routes from being issued in other circumstances, and every single null-route is manually reviewed to make sure that it was truly necessary.

Can you remove our null-route early?

In most cases, if a null-route was issued, it was necessary, and can't be removed early. When we remove a null-route before the posted period has passed, it exposes the location to a much higher risk of further impact.

There are occasional cases where we can remove null-routes early, such as when we are able to work with an upstream to implement a new filter that would help with that specific type of attack, or when a false positive is determined. In those cases, we remove the null-route without requiring that the customer contact us. This means that you should not contact us to ask for a null-route to be removed.

The null-route event says that it will stay in place for some number of hours. I don't believe that the attack would have lasted more than a few minutes. Why does the null need to last so long?

A null-route is a huge deal -- an emergency, stopgap, last-resort measure that we take to protect our network, and other customers, from the serious impact of a major attack. With that in mind, we base the amount of time that we null-route each target IP address on a number of factors, including:

- The likelihood of additional attacks (and additional collateral damage to other customers)
- How impactful the attack was or could be (if repeated)
- The rate at which similar attacks and null-routes are occurring at the same location
- The length of time that it takes our system to send out emails to other involved hosts
- The amount of time it takes the most responsive hosts to investigate and clean the compromised devices that we report to them

If there is a belief that the attacker wouldn't have continued attacking for very long, that's good, but that is not an important factor in this equation.

We manually evaluate every null-route action and remove a null-route early when we determine that it makes sense to do so -- based on our calculation of these factors for that specific attack. For instance, if an attack only barely saturated connections to our upstreams and caused only light packet loss, we may remove the null-route before the communicated amount of time, because having a small percentage of the compromised devices removed from the botnet would make additional attacks small enough that we could mitigate them effectively.

How big was a specific attack? What type of traffic was involved?

For security reasons, we do not generally release more information than what we state publicly about all attacks (including in these KB articles and in the network capacity numbers given at https://www.nfoservers.com/networklocations.php) and what is posted to the control panel.

Additional information beyond that is very valuable to attackers, who use it to tune their attacks and market their illegal services.

The attacker told me something about an attack that triggered a null-route, or warned me that more attacks are coming, or told me that I had to do something to prevent more attacks. What should I do?

You should stop communicating with the attacker, and make sure that anyone associated with your service also does not communicate with the attacker. Universally, communicating with the attacker leads to further attacks.

I am panicking because I received attacks over the last couple of days that triggered null-routes. Will these continue to happen forever?

No. Attack sizes rarely stay large for very long. Botnets often shrink in size over time as the devices that attackers use are fixed, rebooted, or taken offline -- or are compromised by different groups. Spoofed traffic sources (used for very large reflection attacks) are typically noticed and taken offline, and reflection attacks are also generally easier to filter than botnet attacks.

It seems like it's really easy to trigger a null-route (with just a few minutes of traffic) that takes down my service (for hours)! Is that true?

No. Attacks that trigger null-routes are exceptional. Run-of-the-mill attackers can't run this level of attack, and as we mention in another answer, attacks of this size can't keep recurring indefinitely.

In terms of the length of an attack, if an attacker can run an attack for a few minutes, then that attacker could also run it for much longer -- hours, if necessary.

Will I receive a credit or discount for time that my service is null-routed?

We are not able to provide credits to the victims of attacks. We are not to blame for these attacks, and we do not receive a credit from our upstreams for them, so a credit would not make sense.

We also put a large amount of time and money into defending targeted customers against attacks, and we take the most damage from them, but we do not issue customers a bill for handling them.

I want to go after the attacker! What can I do?

You should report the attack to the law enforcement authority that handles cybercrime within your jurisdiction. In the US, that is generally the FBI, and the crime can be reported through https://www.ic3.gov.

Make sure that your report to law enforcement includes any information that you have on who might have launched the attack, and any evidence that you can present of that. For instance, if the attacker told you that he or she was going to attack, you can pass along that conversation, along with all contact details that you have for the attacker.

Make sure to also mention NFOservers as your host, and let your law enforcement contact know that we have more data that we can provide about the attack size and devices that were involved. We already work with law enforcement sources, including the FBI, to provide targeted information on large-scale DDoS events, and we can loop in your new contact.

You should not retaliate for an attack by launching an attack of your own. Not only is this illegal, but it is also an escalation that will increase the likelihood that you will receive further attacks yourself. You may also target the wrong person, or cause serious collateral damage to an ISP or other host.

Can't NFO just upgrade its network to handle the type of large-scale attack that led to my null-route?

Sometimes, we have the option of upgrading our network, at a significant cost. We have to weigh this large cost against how many customers would be helped by the upgrade, and we have to determine if the attacks will continue after the upgrade is done (upgrades typically take months to perform, due to waiting for upstreams to provision the new circuits). When it makes sense, we proceed with such upgrades. We have upgraded our Chicago location repeatedly, for instance, and continue to add to its capacity.

At other times, dependent on the attack sizes and location, we can't upgrade our upstream links because the upstreams themselves are being saturated. We must wait for upstreams to upgrade first, if so. We start those conversations when we see such a situation occur.

I think that another host can do better in handling attacks that cause null-routes with NFO. Why should I stay with NFO?

In most cases, it is not true that another host could do better than what we can do in our Chicago location. We have advanced mitigation systems at all locations that are generally seen as the best available, and our Chicago location, specifically, has a larger capacity and better upstream filters than most providers can dedicate to mitigation, even the largest ones.

We have heard reports that some hosts do not issue null-routes and instead simply let large attacks come through and bring down the target service -- meaning that when the attacker stops running the attack, the service recovers. This is not a positive thing, as it means that other customers connected to the same network see major connection interruptions from their "noisy neighbors". Null-routes, issued appropriately, are an important component in ensuring a stable, high-performance network.