Ubuntu 12.04 TF2 Gameservers Randomly Crashing

[moxie2020]

I am getting random crashes that do not relaunch the server. I get what appears to be a generic dump message and there are no errors in any log.

I am trying to use -debug but it is not giving me any output. I tried to manually start gdb and attach a session and I get the following error:

Could not attach to process. If your uid matches the uid of the target
process, check the setting of /proc/sys/kernel/yama/ptrace_scope, or try
again as the root user. For more details, see /etc/sysctl.d/10-ptrace.conf
ptrace: Operation not permitted.

I looked at the ptrace_scope file and its blank.

[Edge100x]

Do you allow a large enough (such as unlimited) core size? If you don't, then you won't get a proper "core" file saved, and "-debug" will have nothing to go on.

Make sure that you're testing your servers without plugins enabled, as well.

[moxie2020]

I have not specified a core size.
I am testing servers now with sourcemod installed without any plugins.

[moxie2020]

Do you allow a large enough (such as unlimited) core size? If you don't, then you won't get a proper "core" file saved, and "-debug" will have nothing to go on.

Make sure that you're testing your servers without plugins enabled, as well.

I am being told to use -condebug instead of -debug for tf2 by support. I thought debug gave me an actual crash report and condebug is just a console output.

Either way I need to figure out how to get an error report.
I am using a forked server setup as well so I was hoping to be able to specify an filename.

[Edge100x]

I am being told to use -condebug instead of -debug for tf2 by support. I thought debug gave me an actual crash report and condebug is just a console output.

We're supporting you in this thread. Do you mean Steam support?

You are correct that -condebug just writes the console output to a file. -debug tells the srcds_run script to run "gdb" on the "core" file after a server crashes and print out the backtrace and a few other things (which you can also do yourself, but this can be more convenient).

Set the core size to unlimited with "ulimit -c unlimited" before you run your server. If you do not do this, srcds_run sets it to 2000, and you will not get a valid "core" file.

When I say to test without plugins, I mean without Sourcemod, too . Pure vanilla will tell you whether your problem is with the game.

[moxie2020]

I got a response from helpdesk telling me to use -condebug instead of -debug.

The servers do not crash vanilla but do crash with sourcemod on a clean install.

I got a couple of -debug reports before but I am being told by folks in allied modders: "Your stack trace is too late, it's already inside the exception handler (which is then crashing again due to Valve miss-implementing it), you'll need to run the server inside a debugger to catch the exception before Breakpad handles it. -asherkin"

I got some support from alliedmodders and was shown how to run a session in gdb. Odd thing is these servers have not crashed in 2 days while other sites with otherwise identical setups are crashing left and right.

Here is a link to that entire conversation:
https://forums.alliedmods.net/showthread.php?t=186071

So anyway. My servers constantly crash with sourcemod without errors unless I turn off -autoupdate and run them in gdb apparently.

[Edge100x]

It's strange that it would work fine in gdb but not under normal circumstances. I imagine that it's possible that Sourcemod has a locking/threading bug and gdb slows things down just enough that it doesn't come up, but I can't think of too much else that could cause this behavior.

Is this the very latest snapshot of Sourcemod? Have you tried the other branch (1.5 versus 1.4)?

[moxie2020]

Ok gdb finally crashed as well here is the info I gathered:

(gdb) bt
#0 0xf7fdb430 in __kernel_vsyscall ()
#1 0xf7d011ef in raise () from /lib/i386-linux-gnu/libc.so.6
#2 0xf7d04835 in abort () from /lib/i386-linux-gnu/libc.so.6
#3 0xf7d3c2fa in ?? () from /lib/i386-linux-gnu/libc.so.6
#4 0xf7d46e42 in ?? () from /lib/i386-linux-gnu/libc.so.6
#5 0xf7d48fbf in ?? () from /lib/i386-linux-gnu/libc.so.6
#6 0xf7d49737 in ?? () from /lib/i386-linux-gnu/libc.so.6
#7 0xf7d4b347 in realloc () from /lib/i386-linux-gnu/libc.so.6
#8 0xf7f99ae4 in CStdMemAlloc::Realloc(void*, unsigned int) () from bin/libtier0.so
#9 0xf44afc74 in CUtlVector<CNavArea*, CUtlMemory<CNavArea*, int> >::InsertBefore(int, CNavArea* const&) ()
from /home/pro/orangebox/tf/bin/server.so
#10 0xf4804eb0 in CTFPlayer::OnNavAreaChanged(CNavArea*, CNavArea*) () from /home/pro/orangebox/tf/bin/server.so
#11 0xf472dbe7 in NextBotPlayer<CTFPlayer>::OnNavAreaChanged(CNavArea*, CNavArea*) ()
from /home/pro/orangebox/tf/bin/server.so
#12 0xf435a8d2 in CBaseCombatCharacter::UpdateLastKnownArea() () from /home/pro/orangebox/tf/bin/server.so
#13 0xf459646a in CBasePlayer::PreThink() () from /home/pro/orangebox/tf/bin/server.so
#14 0xf4803f82 in CTFPlayer::PreThink() () from /home/pro/orangebox/tf/bin/server.so
#15 0xf45ac1c2 in CPlayerMove::RunPreThink(CBasePlayer*) () from /home/pro/orangebox/tf/bin/server.so
#16 0xf45ac7d2 in CPlayerMove::RunCommand(CBasePlayer*, CUserCmd*, IMoveHelper*) ()
from /home/pro/orangebox/tf/bin/server.so
#17 0xf458c3c4 in CBasePlayer::PlayerRunCommand(CUserCmd*, IMoveHelper*) ()
from /home/pro/orangebox/tf/bin/server.so
#18 0xf4804486 in CTFPlayer::PlayerRunCommand(CUserCmd*, IMoveHelper*) () from /home/pro/orangebox/tf/bin/server.so
#19 0xee3c65d8 in __SourceHook_MFHCls_PlayerRunCmdHook::Func(CUserCmd*, IMoveHelper*) ()
from /home/pro/orangebox/tf/addons/sourcemod_goldrush/extensions/sdktools.ext.2.ep2v.so
#20 0xf45a6f9a in CBasePlayer::PhysicsSimulate() () from /home/pro/orangebox/tf/bin/server.so
#21 0xf472ea4a in NextBotPlayer<CTFPlayer>::PhysicsSimulate() () from /home/pro/orangebox/tf/bin/server.so
#22 0xf4728070 in CTFBot::PhysicsSimulate() () from /home/pro/orangebox/tf/bin/server.so
#23 0xf4565c5b in Physics_SimulateEntity(CBaseEntity*) () from /home/pro/orangebox/tf/bin/server.so
#24 0xf4566024 in Physics_RunThinkFunctions(bool) () from /home/pro/orangebox/tf/bin/server.so
#25 0xf444fdc0 in CServerGameDLL::GameFrame(bool) () from /home/pro/orangebox/tf/bin/server.so
#26 0xeef392e2 in __SourceHook_FHCls_IServerGameDLLGameFramefalse::Func(bool) ()
from /home/pro/orangebox/tf/addons/sourcemod_goldrush/bin/sourcemod.2.ep2v.so
#27 0xf7496c6c in CServerPlugin::GameFrame(bool) () from /home/pro/orangebox/bin/engine.so
#28 0xf7491950 in SV_Think(bool) () from /home/pro/orangebox/bin/engine.so
#29 0xf7492b60 in SV_Frame(bool) () from /home/pro/orangebox/bin/engine.so
#30 0xf73e9e0c in _Host_RunFrame_Server(bool) () from /home/pro/orangebox/bin/engine.so
#31 0xf73eab8a in _Host_RunFrame(float) () from /home/pro/orangebox/bin/engine.so
#32 0xf73eb4cf in Host_RunFrame(float) () from /home/pro/orangebox/bin/engine.so
#33 0xf73f6852 in CHostState::FrameUpdate(float) () from /home/pro/orangebox/bin/engine.so
#34 0xf73f6b98 in HostState_Frame(float) () from /home/pro/orangebox/bin/engine.so
#35 0xf74af260 in CEngine::Frame() () from /home/pro/orangebox/bin/engine.so
#36 0xf74ab22a in CDedicatedServerAPI::RunFrame() () from /home/pro/orangebox/bin/engine.so
#37 0xf78691be in RunServer() () from bin/dedicated.so
#38 0xf7868ec8 in CDedicatedExports::RunServer() () from bin/dedicated.so
#39 0xf74abb26 in CModAppSystemGroup::Main() () from /home/pro/orangebox/bin/engine.so
#40 0xf74c6fc8 in CAppSystemGroup::Run() () from /home/pro/orangebox/bin/engine.so
#41 0xf74ac92a in CDedicatedServerAPI::ModInit(ModInfo_t&) () from /home/pro/orangebox/bin/engine.so
#42 0xf7869396 in CDedicatedAppSystemGroup::Main() () from bin/dedicated.so
#43 0xf786cea8 in CAppSystemGroup::Run() () from bin/dedicated.so
#44 0xf786eb7c in CSteamApplication::Main() () from bin/dedicated.so
#45 0xf786cea8 in CAppSystemGroup::Run() () from bin/dedicated.so
#46 0xf78697fd in main () from bin/dedicated.so
#47 0xf786acd5 in DedicatedMain () from bin/dedicated.so
#48 0x080488ad in main ()
(gdb)

Alliedmodders is telling me its a memory allocation issue and suggesting possibly moving to a 32 bit OS to solve the problem.

I haven't tried 1.5 yet.

[Edge100x]

Since the application runs as 32-bit either way, I doubt that switching the "bit-ness" of the OS would make any difference in working around the bug.

The backtrace suggests that either invalid data is being passed to realloc(), or the memory reallocation is failing for another reason. I also see that bots are involved (along with Sourcemod). Does this happen when you have bots disabled?

Is this on a VDS -- and are you just running out of memory? TF2 has at least one known memory leak (with SourceTV).

[moxie2020]

Since the application runs as 32-bit either way, I doubt that switching the "bit-ness" of the OS would make any difference in working around the bug.

The backtrace suggests that either invalid data is being passed to realloc(), or the memory reallocation is failing for another reason. I also see that bots are involved (along with Sourcemod). Does this happen when you have bots disabled?

Is this on a VDS -- and are you just running out of memory? TF2 has at least one known memory leak (with SourceTV).

It is on a VDS. I do have another 64 bit fully dedicated server set up identically that was donated to us and it has the same issues.
I have -nohltv set and have sourceTV and Replays disabled.
I do not show any signs of running out of memory especially since most of the servers are empty.

I set up a 32bit Ubuntu server just in case on the dedicated server. So far so good. 13 servers already at 8 hours and counting without a crash which is already better than any other location has done.

I am not counting any unhatched chickens but if this does solve the problem is there a way to copy the image onto a VDS?

[moxie2020]

I am also running bots with sourcemod. I have not tried it without bots but its a moot point if that is the issue because I have found it impossible to fill a server with no bot place holders.

I will try a few servers with this setup just to eliminate it as a possibility.

[Edge100x]

Switching a VDS to 32-bit Ubuntu may require that you reinstall it from an ISO. But, you could try rsync'ing the system over before doing that, similar to what we describe here: http://www.nfoservers.com/forums/viewto ... =46&t=5045

[moxie2020]

Did you change the kernel?


Quote:
cat /usr/src/linux/.config | grep HZ
You need 1000 HZ, or the server will crash with the world.


I was given the advice above. Right now everything is default which is 250 HZ.

Do you think this might make any difference?

[Edge100x]

The kernel hz wouldn't make a difference, no.

[moxie2020]

Switching a VDS to 32-bit Ubuntu may require that you reinstall it from an ISO. But, you could try rsync'ing the system over before doing that, similar to what we describe here: http://www.nfoservers.com/forums/viewto ... =46&t=5045

Will rsyncing work if they are different bit types? If not I am fine with doing a reinstall but not sure how to do that remotely.