Server keeps freezing, hanging up

[jakallan3]

We have two three-core VPS's running eight Halo servers and a TeamSpeak 3 on one while another is running six Halo servers. We can't seem to determine why our Halo servers will suddenly stop responding and crash.

It's difficult to explain. We'll notice one of our Halo servers are down, remote control the VPS, and find the Halo dedicated console window to have halted its activity. It's completely accessible and usable, but it's no longer printing data to its window and any commands I use (i.e. sv_players) turn up nothing. No one can join the server at all until we exit the window and restart it.

On the more crowded VPS we have a CPU average of about 80% when everything is busy and 60% on the other. We don't believe its the CPU or any computer hardware causing the problem, but we're open to any and all suggestions.

We don't know what to try in order to fix this. If we can't find an explanation soon we'll have to leave NFO servers and I really enjoy the services offered here ... please help I'll provide any information necessary in order to help anyone help fix this problem.

[soja]

I haven't had much experience with halo servers, so please excuse me if I am asking obvious questions here.

Are there any third party addons that can be the cause?

With that many active servers, I assume you have been around for a while, have you changed anything recently?

[jakallan3]

Yes, we're running Gandanur on most of them and Phasor on one or two of them. We don't believe this is the problem because they run fine for other people who are using them with NFO. In fact, NFO offers Gandanur as an auto-install plugin for their Halo servers.

[Edge100x]

I had one customer mention attacks against his Halo server recently with similar symptoms. Has anyone threatened an attack against your server?

It might be worth running a continuous traffic capture so that you can later go back and pull out the packets sent to the server right before it froze. A single attack packet may be to blame for this.

[jakallan3]

You may be right Edge. In fact, as soon as I read your post I connected to my VPS just to find my FTP servers under a brute-force attack from an IP in China (proxy?).

In any case, without a doubt this is more than likely a malicious individual looking to cause trouble. How do I stop this? Is there software for capturing traffic? Help!

[Edge100x]

I recommend running "windump" and having it write out to a file using a command like this:

Code: Select all

windump -n -X -s 1500 -w capture.pcap dst x.x.x.x

Where "x.x.x.x" is the IP to monitor. After your next freeze, hit Control-C to stop the capture, then run a command like this:

Code: Select all

windump -n -r capture.pcap > capture.txt

Then, open capture.txt in Notepad and look for suspicious traffic near the end.

I recommend restarting the first command fairly regularly if the server is not going down -- such as hourly -- to reduce the size of the output file.

[jakallan3]

Where do I place windump? I tried what you said, I couldn't make it work. Maybe I did something wrong?

[jakallan3]

Another questions (since I can't edit my above post):

What sort of protection is offered with unmanaged VPS's? Any sort of firewalls or extra security to protect from outside attacks?

[Edge100x]

You can place it anywhere, and you just need to run it from the command line using the commands that I gave. One easy place to put it would be "c:\".

You have access to a Firewall page in your control panel that contains powerful traffic filtering tools. We need to know the nature of the attack before I can help you to create a filter rule. There is, unfortunately, no way to automate that process in your case, because you're likely seeing an application-specific DoS that is not known.

[jakallan3]

Is it perhaps due to Windows Server 2008 R2 running 64 bit?

I'm totally running out of ideas on this. I don't know anyone else using NFO servers and running other Halo PC/CE servers. Is there anyone out there that can help?

[jakallan3]

Hate to double post but I can't edit my above post.

Is there a Windows Server 2008 32 bit version you guys have for auto-install?

[jakallan3]

Halo servers just crashed again. Apologies for the triple post.

We had 3 Halo servers on one VPS and 1 Halo server on another crash at the exact same time.

[Edge100x]

It is very unlikely that crashes would be due to a 32-bit versus 64-bit OS -- that's something I've never seen. We do not offer a 32-bit version of Windows 2008.

Were you able to make the packet capture?

[jakallan3]

No, I haven't done it yet. I'm too frusterated trying to make it work.

I found a log of all the Halo problems in my Windows Server 2008 R2 error report thing ... 86 failures!!!!

Here is an excerpt from it:

Description
Faulting Application Path: C:\Users\Administrator\Desktop\#6\#6.exe

Problem signature
Problem Event Name: APPCRASH
Application Name: #6.exe
Application Version: 1.0.9.620
Application Timestamp: 4a9036e9
Fault Module Name: #6.exe
Fault Module Version: 1.0.9.620
Fault Module Timestamp: 4a9036e9
Exception Code: c0000005
Exception Offset: 0009332c
OS Version: 6.1.7601.2.1.0.272.7
Locale ID: 1033
Additional Information 1: 2a21
Additional Information 2: 2a21931d819eec1dc737e8c7119af640
Additional Information 3: a1b9
Additional Information 4: a1b95af070e8db562e650e5964e352ab

Here's another, same problem:

Description
Faulting Application Path: C:\Users\Administrator\Desktop\#11\#11.exe

Problem signature
Problem Event Name: APPCRASH
Application Name: #11.exe
Application Version: 1.0.9.620
Application Timestamp: 4a9036e9
Fault Module Name: #11.exe
Fault Module Version: 1.0.9.620
Fault Module Timestamp: 4a9036e9
Exception Code: c0000005
Exception Offset: 0009332c
OS Version: 6.1.7601.2.1.0.272.7
Locale ID: 1033
Additional Information 1: 815f
Additional Information 2: 815f3196296a684315f92e08f3a36539
Additional Information 3: 90c8
Additional Information 4: 90c86d5b3002a0c50d21df6382f06b42