Cannot Access server when proxy is between cloudflare and NFO VPS

[laterbreh]

Hello,

Tech support suggested I post this here...

I have configured my box to be identical from my previous server. On my previous configuration its: type in url > Cloudflare > DDOS Proxy > Server. No problems, can access instantly.

Here the above is not true. I can only access the server if cloudflare is set up in this way: Cloudflare > NFO Server

I have tried several different DDOS proxies they all work instantly if i point to my other server, but not to this one. UFW is off, iptables clean flushed and reset, and firewall has no rules on NFO side. I'm baffled as to why this is not working.

Does anyone have any ideas?

[Edge100x]

It is possible that the DDoS proxy provider is mis-identifying the CloudFlare traffic as attack traffic, since it will come from only a limited number of IP addresses. But, it's difficult to say. You should consider contacting their support about this.

I really wouldn't recommend the setup that you're proposing. Because CloudFlare is already a proxy, it is unlikely that you would gain protection, and likely that you would actually lose protection (because you add another point of failure), by putting another layer in there. It's also highly unlikely that your 3rd party proxy would offer better DDoS mitigation than we do. (Realistically, you probably don't even need CloudFlare, if you have some basic scripts set up to block simple attacks.)

[laterbreh]

It is possible that the DDoS proxy provider is mis-identifying the CloudFlare traffic as attack traffic, since it will come from only a limited number of IP addresses. But, it's difficult to say. You should consider contacting their support about this.

I really wouldn't recommend the setup that you're proposing. Because CloudFlare is already a proxy, it is unlikely that you would gain protection, and likely that you would actually lose protection (because you add another point of failure), by putting another layer in there. It's also highly unlikely that your 3rd party proxy would offer better DDoS mitigation than we do. (Realistically, you probably don't even need CloudFlare, if you have some basic scripts set up to block simple attacks.)

Hi, Thank you for your response.

We are using the free cloudflare package to gain some of their features and tools and to quickly change DNSs when necessary when we deploy new web apps. Using the same proxy on our other webserver we have zero issues with the setup. Furthermore you can easily obtain the ip address of a host behind cloudflare, so we use a another proxy that just simply forwards all the traffic. Further as i said setting cloudflare to point directly at the server works just fine. But as soon as I add the proxy between cloudflare and the server, i can no longer access my webapp.

There has to be something different here as I mirrored all of my nginx configurations etc including the same release and version of my OS.


I'm truly at a loss here. I hope that I can find a solution!

Thank you.

[laterbreh]

Okay, I have some more information... its very peculiar... if i turn off my webapp with my current setup (Cloudflare > Proxy > NFO Host), I am isntantly greeted with the nginx 404 i can spam refresh and the 404 page is served.

But if I turn on my webapp with the configureation i dont see the 404 from nginx the browser just indefinitely tries to refresh.

[laterbreh]

After 3 reformats of the machine, i started digging further into the other server... On a hunch I went to /etc/hosts and there WAS something I was missing that my other server had that this VPS didnt come with out of the box:

127.0.0.1 localhost.locadomain localhost

once I changed the stock configuration to this... for some reason it works. I'm in disbelief that this could have caused several hours of frustration, but I hope that this may help someone in the future.

[laterbreh]

After 3 reformats of the machine, i started digging further into the other server... On a hunch I went to /etc/hosts and there WAS something I was missing that my other server had that this VPS didnt come with out of the box:

127.0.0.1 localhost.locadomain localhost

once I changed the stock configuration to this... for some reason it works. I'm in disbelief that this could have caused several hours of frustration, but I hope that this may help someone in the future.

Spoke too soon. For 10 minutes it ran perfectly. Was able to refresh and interact with the app without problems. Now its back to the same thing... just loading...... the page and then Error.
Changed config from Cloudflare > Proxy > NFO Server to Cloudflare > NFO Server, NOW the problem continues to persist.
The only way I can access the app now is directly via IP:Port of the NFO VPS.

Tried with NGINX and Apache. Both have worked on my other servers with this same app.

I am at a complete loss for words. No changes, and everything stops working. Ive never EVER had this kind of problems setting up a server (As Ive done this same thing multiple times on many other hosts).

I use NFO for my gameservers so I thought I can run some production servers here.

I guess all I can say is thankfully the first 48 hours are free. Since no one in the support department has ANY idea why the server just chooses to break without any interaction from me, I guess Ill be taking my business elsewhere.

Thank you for your time.

[laterbreh]

Also I forgot to insert this bit of information: Followed the same setup process on my linode vps, digital ocean, ramnode, and AWS instance. All worked flawlessly on the first attempt. Same process here yields infinite load to a timeout.

[kraze]

It sounds like you identified your own issue. You mentioned above that if you disable your web app your unique setup works(Cloudflare > Proxy > NFO Host) , but with it enabled it does not. It sounds like you may need to dig into the configuration of the web app.

Overall, there wouldn't be anything we'd be doing that would stop this from working. It's also important to note here that the service itself appears to be working fine, but your special configuration to the service isn't.

Issues like this are fairly difficult to resolve in any format due to the complex nature of them. There's only so much information you can pass along and only so much we'll be able to see. I'd recommend you strip down your configuration and start putting it in piece by piece and testing along the way. You should also try bypassing complete parts of your system to see if they work(does the web app work without the mitigation provider..etc).

[laterbreh]

It sounds like you identified your own issue. You mentioned above that if you disable your web app your unique setup works(Cloudflare > Proxy > NFO Host) , but with it enabled it does not. It sounds like you may need to dig into the configuration of the web app.

Overall, there wouldn't be anything we'd be doing that would stop this from working. It's also important to note here that the service itself appears to be working fine, but your special configuration to the service isn't.

Issues like this are fairly difficult to resolve in any format due to the complex nature of them. There's only so much information you can pass along and only so much we'll be able to see. I'd recommend you strip down your configuration and start putting it in piece by piece and testing along the way. You should also try bypassing complete parts of your system to see if they work(does the web app work without the mitigation provider..etc).

I believe you miss-understood, Yes, it worked for 10 minutes but then it stopped. Now I can only access it via cloudflare > Nfohost. I have identical configurations on several other VPSs that I have tested and they work. Also to add, I dont know how my setup is unique... You can easily acquire the ip address of a server using cloudflare... Would you prefer that your DDOS filters take 100GBPS attacks directly? Or would you rather have my proxies take care of it? Also your reputation of insta-null-routing over tiny ddos attacks is well known over the internet. So i bring my own proxies to put in-front of my servers. So this "unique" situation appears to work across the board on other hosts running the same OS, with the same configuration, yet yours doesnt.

So, does anyone have a real suggestion? Or are you going to assume that its "not NFO its you"? Must be me since the several other vps ive done this on have 0 issues.

Like ive said before, I would love to bring all of my business here. But if I cant get a simple forwarding proxy to work properly then ill just take my business elsewhere.

I find it thoroughly disappointing that a company that I hold in such high regard and speak volumes of as a provider is essentially washing their hands of the issue.

GG NFO

[rlm850]

GG NFO

Cloudfare has nothing to do with NFO, nor does NFO rely nor need on cloudfare (for such a large, well-known GSP, do you see them hiding their web servers behind cloudfare [which is normally what people that use cloudfare wish to obtain])?

Cloudfare is a 3rd party app, and you are on an unmanaged VDS. No matter what host you go to, unless they have some direct tie with cloudfare in which they father the price for support in , then you will receive help on it.

This almost follows the same concept of jailbreaking an iPhone. Why would a company support you with running something that could potentionally decline the quality of your service/product? NFO offers by far the best DDOS protection you will ever find -- and it's free. You could pay other hosts $20+ for DDoS filtering (not true mitigation) but you will never price match NFO, and that's why they have an SLA (which many companies have, but don't actually stand up to).

Targeting NFO about this issue is an issue itself. It is not their fault they cannot assist you with something they do not assist with, unless they personally know and are willing to give you some basic help with it. That's why the forums are here - so you could then ask around and hope another customer knows the answer. Not bash NFO..

--EDIT--

I don't know where you ever heard that NFO immediately null routes IPs based on small attacks.. I have been back and forth with NFO over 6 years. I have watched them gradually upgrade every single aspect of their network and hardware. Not that I suggest it, but you could attack any server you choose hosted on NFO. It will not move, it will not spike - it will not go anywhere. You know how many messages I've gotten from friends saying they were going to knock this or that offline because of this or that, then try, and I'm still looking at the server populated to the max?

[laterbreh]

GG NFO

Cloudfare has nothing to do with NFO, nor does NFO rely nor need on cloudfare (for such a large, well-known GSP, do you see them hiding their web servers behind cloudfare [which is normally what people that use cloudfare wish to obtain])?

Cloudfare is a 3rd party app, and you are on an unmanaged VDS. No matter what host you go to, unless they have some direct tie with cloudfare in which they father the price for support in , then you will receive help on it.

This almost follows the same concept of jailbreaking an iPhone. Why would a company support you with running something that could potentionally decline the quality of your service/product? NFO offers by far the best DDOS protection you will ever find -- and it's free. You could pay other hosts $20+ for DDoS filtering (not true mitigation) but you will never price match NFO, and that's why they have an SLA (which many companies have, but don't actually stand up to).

Targeting NFO about this issue is an issue itself. It is not their fault they cannot assist you with something they do not assist with, unless they personally know and are willing to give you some basic help with it. That's why the forums are here - so you could then ask around and hope another customer knows the answer. Not bash NFO..

Did you read the thread... like at all or were you looking for a thread to just vent on?

I don't expect support on a VPS and I've never demanded it. Cloudflare in a free capacity has large utility for someone with many webhosts and DNSs (I think that is obvious which leads me to believe you truly have no idea whats going on). I am not new at this. Furthermore, I know for a fact that your server will be null routed the second youre hit with a baby 20GBPS attack. My proxies can handle 100Gpbs+ attacks I've tested them personally. I literally only use cloudflare as my DNS, and I point it to my DDOS Proxies that I pay separately for which forward all traffic. I am the perfect customer: I bring my own protection so I dont have to rely on someone elses capacity to handle a potential threat. Great.

But to tell me that after doing the same process on many hosts from a slew of personal notes that dictate step by step to facilitate consistent deployment across hosts "just doesn't work here" is ridiculous.

Honestly, from the response you gave you sound like someone that will barely be able to understand any of the vocabulary here (I'm honestly surprised you know what an SLA is), so I don't know why I even bothered addressing you.

I've closed my account.

Thank you all for your time.

[rlm850]

Did you read the thread... like at all or were you looking for a thread to just vent on?

Vent on.. I just woke up from a 5 hour sleep/34 hour wake because I had to go to the bathroom, and I seen you were requesting help.

I don't expect support on a VPS and I've never demanded it. Cloudflare in a free capacity has large utility for someone with many webhosts and DNSs (I think that is obvious which leads me to believe you truly have no idea whats going on). I am not new at this.

A VDS, while similar to a VPS, is nothing like one. You should not compare. You are twisting your own words around, targeting the host (NFO), while they have no requirement to assist you with anything 3rd party (unless they are personally familiar with it, at their own level, and take time out of their work time to share this information with you). I have been in web design for 12 years and running machines for 8, I've been doing datacom for 2. I am not new to this industry either, and I offered you the advice I had, here, on a forum that you requested assistance.

Furthermore, I know for a fact that your server will be null routed the second youre hit with a baby 20GBPS attack. My proxies can handle 100Gpbs+ attacks I've tested them personally. I literally only use cloudflare as my DNS, and I point it to my DDOS Proxies that I pay separately for which forward all traffic. I am the perfect customer: I bring my own protection so I dont have to rely on someone elses capacity to handle a potential threat. Great.

Awesome. I'm proud of you for taking such a leap as a customer for any host. But this is not any host.. you will be degrading the true DDoS mitigation that you receive here. Like I said -- I'm not suggesting you attack a server, but I am stating that it is not going down. Most stress testing services and individuals that offer these services through their own network know this, and they will either kick you off that service, or they will state that it's simply just not going to work, and save their resources. I have heard countless amounts of whining about how they simply just cannot bring the server down. It will not work. But, since you wish to be smart: you're*

But to tell me that after doing the same process on many hosts from a slew of personal notes that dictate step by step to facilitate consistent deployment across hosts "just doesn't work here" is ridiculous.

You are welcome to do as you please with any service you have here (so long as it's bound by the ToS). Anything will work here unless otherwise stated (e.g. no minecraft on managed windows machines, onlix linux). But you cannot force support out of a company nor individuals that do not directly offer support for that application/service. Go jailbreak an iPhone and walk in to an Apple store telling them you can't get your phone to communicate with the GSM server now.

Honestly, from the response you gave you sound like someone that will barely be able to understand any of the vocabulary here (I'm honestly surprised you know what an SLA is), so I don't know why I even bothered addressing you.

I've closed my account.

Thank you all for your time.

You did not address me. I am a happy customer here who has seen no degrading of my service, posting on NFOs public forum, where anyone can assist anyone, or simply share their ideas, and I was willing to share mine. You did not have to reply back, and could have ignored.

I wish you luck wherever you may go (which will probably be back here).

[Edge100x]

This thread seems to have gone off the rails a bit. It was originally about troubleshooting, and that's where it would be best to keep it. There is no reason for vitriol when investigating a technical concern.

To clarify some points:

- I can't think of any obvious reason why you'd have problems between your DDoS proxy and our service. We have only very specific filters on our overall circuits that block common types of attacks such as NTP monlist attacks, and these do not affect normal traffic.
- You have indicated that your problem relates to your own application, DDoS proxy, and/or CloudFlare, and that turning off one or more of these causes the problem to go away. This means that you should concentrate your troubleshooting energy on those factors first. You need to ask yourself questions like "What is different about the traffic when the proxy sends it versus when my client sends it?" and "What is different about my application that might cause this to stop working after a short period?" You will want to use tools such as tcpdump to look at the actual traffic, and your webserver logs to see what errors it might be spitting out.
- Other customers have not reported this type of problem, so it is likely the cause relates to your specific software configuration. We have many of customers that run through CloudFlare just fine, and many customers have also experimented with 3rd party mitigation providers (most of which later disable them).
- Adding a 3rd party mitigation provider into the mix is generally a bad idea. Only a handful of those claiming external mitigation exceed our capabilities, and any others will just add an extra point of failure and decrease performance. As a general rule of thumb, if you're not paying thousands of dollars a month for a scrubbing service, it is not going to do anything to help you beyond our standard offering. (CloudFlare is a narrow exception to this, for some types of attacks.) All that said, you're welcome to use a 3rd party scrubber if you wish. We do not block them and we do not mind.
- Even though we don't market ourselves as a DDoS mitigation host, we are well known for having some of the best protection. We consistently see attackers use us as a benchmark -- when they think that they can take out an NFO-hosted customer, they consider that a very big deal and trumpet it. Such claims don't stick for long because their users quickly find that their attacks are mitigated in a variety of ways here (viewtopic.php?f=25&t=4931).
- We do not null-route small attacks. A small attack would be something like 5 Gbps or under. A 20 Gbps attack is considered moderate-sized by current standards and we don't have to null many of those, either -- thanks to smart upstream filters and increased capacity where needed. We regularly see many large attacks (such as 100 Gbps attacks) mitigated just fine, thanks to those upstream filters.
- If you expect to be attacked and need the strongest protection, consider our Seattle or Chicago locations first. Both currently have 50 Gbps of upstream connectivity, and strong upstream filtering. (Of course, further upgrades will occur everywhere over time, including those locations.)
- While I can understand your desire to do so, it is not OK to "test" the DDoS mitigation capabilities of a host by attacking it, whether that host is CloudFlare, us, or some other provider.
- These forums can be read and viewed by anyone. Those who work here will have "Staff" or "Founder" as a title. So far, only Kraze and I have been involved with this thread, and our responses have been professional.

[laterbreh]

This thread seems to have gone off the rails a bit. It was originally about troubleshooting, and that's where it would be best to keep it. There is no reason for vitriol when investigating a technical concern.

To clarify some points:

- I can't think of any obvious reason why you'd have problems between your DDoS proxy and our service. We have only very specific filters on our overall circuits that block common types of attacks such as NTP monlist attacks, and these do not affect normal traffic.
- You have indicated that your problem relates to your own application, DDoS proxy, and/or CloudFlare, and that turning off one or more of these causes the problem to go away. This means that you should concentrate your troubleshooting energy on those factors first. You need to ask yourself questions like "What is different about the traffic when the proxy sends it versus when my client sends it?" and "What is different about my application that might cause this to stop working after a short period?" You will want to use tools such as tcpdump to look at the actual traffic, and your webserver logs to see what errors it might be spitting out.
- Other customers have not reported this type of problem, so it is likely the cause relates to your specific software configuration. We have many of customers that run through CloudFlare just fine, and many customers have also experimented with 3rd party mitigation providers (most of which later disable them).
- Adding a 3rd party mitigation provider into the mix is generally a bad idea. Only a handful of those claiming external mitigation exceed our capabilities, and any others will just add an extra point of failure and decrease performance. As a general rule of thumb, if you're not paying thousands of dollars a month for a scrubbing service, it is not going to do anything to help you beyond our standard offering. (CloudFlare is a narrow exception to this, for some types of attacks.) All that said, you're welcome to use a 3rd party scrubber if you wish. We do not block them and we do not mind.
- Even though we don't market ourselves as a DDoS mitigation host, we are well known for having some of the best protection. We consistently see attackers use us as a benchmark -- when they think that they can take out an NFO-hosted customer, they consider that a very big deal and trumpet it. Such claims don't stick for long because their users quickly find that their attacks are mitigated in a variety of ways here (viewtopic.php?f=25&t=4931).
- We do not null-route small attacks. A small attack would be something like 5 Gbps or under. A 20 Gbps attack is considered moderate-sized by current standards and we don't have to null many of those, either -- thanks to smart upstream filters and increased capacity where needed. We regularly see many large attacks (such as 100 Gbps attacks) mitigated just fine, thanks to those upstream filters.
- If you expect to be attacked and need the strongest protection, consider our Seattle or Chicago locations first. Both currently have 50 Gbps of upstream connectivity, and strong upstream filtering. (Of course, further upgrades will occur everywhere over time, including those locations.)
- It is not OK to "test" the DDoS mitigation capabilities of a host by attacking it, whether that host is CloudFlare, us, or some other provider.
- These forums can be read and viewed by anyone. Those who work here will have "Staff" or "Founder" as a title. So far, only Kraze and I have been involved with this thread, and our responses have been professional.

Edge, I thank you for your response.
I was actually in the process of researching more thoroughly into your filtering capacities and your response warrants me to consider not using an external service. Paranoia of DDOS can drive a person crazy.

@rlm850 I appologize for my remarks, again frustration can lead to misplaced anger. My Appologies.

Just about every host we are on insisted that we needed external mitigation or our services would be nullrouted or terminated.

I apologize for becoming testy and possibly inciting angst in the thread. "Simple" configurations like this that end up not working can drive a person up a wall.

I will re-open an account and give it a shot on one of the nodes your mentioned that have the highest capacity as my web-apps are prone to attacks. Its probably best that at this point I get some sleep.

Thank you for your assurance!

[rlm850]

Just about every host we are on insisted that we needed external mitigation or our services would be nullrouted or terminated.

We all stated from the get-go that you must need 0 changes made to any machine here to provide more security. They are more secure than you think, with backbones of 20gbps and up (Denver is the smallest). The only reason I've seen a server terminated at NFO is for fraud. Why would they terminate a customer with a network related issue, when they need this issue to learn and make the network stronger? That gets none of us anywhere.