CRASHING: Segmentation Fault & No symbol table info availabl

[p3dsman]

I am running a HL2DM VDS, Debian Linux. As of recently with no changes to the server, I have started to experience random crashes of only two types showing in the debug.log:

1. Segmentation Fault
2. No symbol table info available.

This happens WITH or WITHOUT any plugins running...very strange crashes, and I have also been told this is NOT happening on Windows based servers.


Any ideas?? thanks in advance...

* I should note that these issues have only been since the end of October updates that Valve has released. Otherwise I could leave the server going for weeks and no issues.

[Edge100x]

Could you post the full debug.log output and the last hundred or so lines of the server console from right before the crash?

[soja]

Disable sourcemod and metamod.

If this is the problem, get the latest betas for both here:

SM: http://www.sourcemod.net/snapshots.php (as of writing hg3683 is latest)

MM: http://sourcemm.net/snapshots (as of writing hg811 is latest)

[rustydusty1717]

Eww seg fault. I get those all the time on my linux machine, although I though it was strictly HL1 engine issues.

https://forums.alliedmods.net/showthread.php?t=91189

[Edge100x]

A segmentation fault is a very generic crash that most correspond to, which is why more information is needed (if it shows in the log).

[p3dsman]

A segmentation fault is a very generic crash that most correspond to, which is why more information is needed (if it shows in the log).

souja and rustydusty1717: thanks for your help. unfortunately, this happens WITH or WITHOUT serverside plugins. I am using only Eventscripts with one script that I wrote for the server. Even if I disable Eventscripts and run vanilla so to speak, the crashes STILL happen.

This is a fresh debug.log, not even 30 minutes ago. I also have a core dump if needed. Game Server and VDS are all current with updating...this is driving me crazy. I have even re-installed the OS, changed from Ubuntu to Debian and a few other Linux Distros to no avail. It happens so randomly, even with w FRESH server install. I notice it more on map changes, but earlier it also crashed mid-game. The more players, the more often the crash it seems. Hope I can get this going right again. I haven't changed anything in the last 3 months on my server, updates is the only thing that has changed. I don't know if I am just not meeting a requirement of libraries with the SRCDS Server engine or what.

Code: Select all

----------------------------------------------
CRASH: Sat Nov 24 11:20:23 CST 2012
Start Line: ./srcds_linux -game hl2mp -console -debug +ip 66.151.244.149 +port 27015 +maxplayers 18 +map dm_killbox_kbh_2_fix2 -autoupdate -secure
[New Thread 5534]
[New Thread 5540]
[New Thread 5536]
[New Thread 5576]
[New Thread 5538]
#0  0xfd80400a in ?? ()
No symbol table info available.
eax            0xf0	240
ecx            0x0	0
edx            0x2	2
ebx            0xeba7c47c	-341326724
esp            0xeb99fe80	0xeb99fe80
ebp            0xeb99feb8	0xeb99feb8
esi            0x0	0
edi            0x0	0
eip            0xfd80400a	0xfd80400a
eflags         0x210202	[ IF RF ID ]
cs             0x23	35
ss             0x2b	43
ds             0x2b	43
es             0x2b	43
fs             0x0	0
gs             0x63	99
End of Source crash report
----------------------------------------------

[Edge100x]

What are the last lines in the console from before this happens?

[p3dsman]

What are the last lines in the console from before this happens?

Thats the funny thing, it doesn't show any extended info or anything else that would even give a hint. It does however say the "no symbol table available". From what I have read, this has something to do with threads and the libgc++ library. It's a bit beyond my understanding, and a bit overkill just for whats going on. I would switch to a Windows OS and try that, but my current VDS subscription allows for Linux only. I prefer Linux actually but cannot see paying extra to sort of reinvent the wheel so to speak.

[Edge100x]

What I mean is the actual console and not the debug.log -- so it will show people playing in the server, map changes, and so on. The last few lines will tell us more about what was happening in-game that might have triggered the crash.

This debug.log is unfortunately not useful.

[p3dsman]

What I mean is the actual console and not the debug.log -- so it will show people playing in the server, map changes, and so on. The last few lines will tell us more about what was happening in-game that might have triggered the crash.

This debug.log is unfortunately not useful.

I have Putty open running a screen log, as soon as it crashes again, I will post back.

[Edge100x]

You can also capture this log to disk with the -condebug option for the game server.

[p3dsman]

You can also capture this log to disk with the -condebug option for the game server.

This JUST happened. Had 16 players in my server and all is normal, I was playing too....all was perfect, then game just "froze" out of nowhere clientside...the server timed out in HLSW, and this was in the console:

Code: Select all

L 11/25/2012 - 19:02:44: "nelknurt<267><STEAM_0:1:42425085><Unassigned>" killed "BIG Brian<271><STEAM_0:0:5420422><Unassigned>" with "357"
L 11/25/2012 - 19:02:44: "Crash3143<274><STEAM_0:0:47681903><Unassigned>" killed "3Q * cAsPeR *<255><STEAM_0:0:29088320><Unassigned>" with "pistol"
L 11/25/2012 - 19:02:44: "Commander Shepard ME3<257><STEAM_0:0:6249702><Unassigned>" killed "Crash3143<274><STEAM_0:0:47681903><Unassigned>" with "ar2"
L 11/25/2012 - 19:02:45: "NYANQQ<268><STEAM_0:0:37596397><Unassigned>" killed "Sgt VEGA<273><STEAM_0:0:9848886><Unassigned>" with "shotgun"
L 11/25/2012 - 19:02:46: "cjfurlan<234><STEAM_0:1:13770854><Unassigned>" committed suicide with "world"
PreMinidumpCallback: updating dump comment
Uploading dump (in-process) [proxy '']
/tmp/dumps/crash_20121125190246_1.dmp
Segmentation fault (core dumped)
BFD: Warning: /srcds/scorps_mausoleum/orangebox/core is truncated: expected core file size >= 403517440, found: 1114112.
Cannot access memory at address 0xf77aa8f0
Cannot access memory at address 0xeb9d9ebc
/srcds/scorps_mausoleum/orangebox/debug.cmds:4: Error in sourced command file:
Cannot access memory at address 0xf77aa8f0
email debug.log to linux@valvesoftware.com
Sun Nov 25 19:02:46 CST 2012: Server restart in 10 seconds

[Edge100x]

Ok, that is a bit more useful, as it tells us that it happened right after someone committed suicide -- it could be that the bug is related to some unusual method of triggering that, such as a particular type of fall. It's worth checking out whether you can repeat this yourself, by connecting and killing yourself via falls and other methods. Are you running a standard map?

It's also possible that it's not suicide-related. As you continue to monitor for crashes and look at the output, it should become clear if that is the case.

I recommend entering this before starting your server, to get a better memory dump and crash output (that you could provide to Valve along with the reproduction steps):

Code: Select all

ulimit -c unlimited

[p3dsman]

I have run this server bone stock, maps and no plugins. it still crashes. fresh OS, fresh Game Server, all 100% stock. I don't know what else to do. I don't want this to ruin the server I have going. On a limb here, but could the physical memory in the machine possibly be having problems? I shouldn't be having issues like this bone stock on an 8 year old game...any other ideas??


It did good all day and out of nowhere it crashes here, more complete randomness:

Code: Select all

EVERYTHING as normal...everyone playing...

L 11/26/2012 - 20:24:10: "♠♥Rhelgerman♣♦<286><STEAM_0:1:20006982><>" connected, address "68.203.195.12:27005"
Client "♠♥Rhelgerman♣♦" connected (68.203.195.12:27005).
L 11/26/2012 - 20:24:10: "Sprinkles<281><STEAM_0:1:4007113><Unassigned>" killed "EmptyGOFIO!<237><STEAM_0:1:19832344><Unassigned>" with "combine_ball"
Clamping SetLocalVelocity(-447.484741,-2420.703613,-4.292568) on weapon_shotgun
DataTable warning: player: Out-of-range value (17.386415) in SendPropFloat 'm_flPlaybackRate', clamping.
DataTable warning: player: Out-of-range value (17.336452) in SendPropFloat 'm_flPlaybackRate', clamping.
DataTable warning: player: Out-of-range value (17.013977) in SendPropFloat 'm_flPlaybackRate', clamping.
L 11/26/2012 - 20:24:10: "♠♥Rhelgerman♣♦<286><STEAM_0:1:20006982><>" STEAM USERID validated
L 11/26/2012 - 20:24:12: "=BWS=Romell<282><STEAM_0:0:20395960><Unassigned>" killed "Windowlicker III<278><STEAM_0:1:3683393><Unassigned>" with "stunstick"
L 11/26/2012 - 20:24:15: "EmptyGOFIO!<237><STEAM_0:1:19832344><Unassigned>" killed "Meanea<265><STEAM_0:1:11595717><Unassigned>" with "ar2"
L 11/26/2012 - 20:24:16: "VoN<280><STEAM_0:1:19422210><Unassigned>" killed "Wolfy <3<284><STEAM_0:0:29998988><Unassigned>" with "ar2"
L 11/26/2012 - 20:24:18: "(ØČĐ)™ RIP<267><STEAM_0:1:3731803><Unassigned>" killed "Spartans<275><STEAM_0:1:38580364><Unassigned>" with "ar2"
L 11/26/2012 - 20:24:19: "Catsaremygods<225><STEAM_0:1:55926195><Unassigned>" killed 
"Meanea<265><STEAM_0:1:11595717><Unassigned>" with "physics"


IT CRASHED HERE, but this is a new one and no other info available...

ConVarRef mat_dxlevel doesn't point to an existing ConVar
Game_srv.so loaded for "Half-Life 2 Deathmatch"

[p3dsman]

AND Here is my debug.log if it helps...its much more extensive.

Code: Select all

----------------------------------------------
CRASH: Mon Nov 26 20:24:19 CST 2012
Start Line: ./srcds_linux -game hl2mp -console -secure -debug +ip 66.151.244.149 +port 27015 +maxplayers 18 +map dm_killbox_kbh_2_fix2 -autoupdate -condebug
[New Thread 2005]
[New Thread 2007]
[New Thread 2008]
[New Thread 2003]
[New Thread 2004]
Core was generated by `./srcds_linux -game hl2mp -console -secure -debug +ip 66.151.244.149 +port 2701'.
Program terminated with signal 11, Segmentation fault.
#0  0x00000000 in ?? ()
#0  0x00000000 in ?? ()
No symbol table info available.
eax            0x1	1
ecx            0x0	0
edx            0x0	0
ebx            0xeb98f19c	-342298212
esp            0xeb98f0b8	0xeb98f0b8
ebp            0x0	0x0
esi            0x0	0
edi            0xf761fff4	-144572428
eip            0x0	0
eflags         0x210296	[ PF AF SF IF RF ID ]
cs             0x23	35
ss             0x2b	43
ds             0x2b	43
es             0x2b	43
fs             0x0	0
gs             0x63	99
From        To          Syms Read   Shared Object Library
0xf7718460  0xf7732c18  Yes (*)     /lib32/libm.so.6
0xf7710a40  0xf77119b8  Yes (*)     /lib32/libdl.so.2
0xf7683914  0xf76c9c78  Yes         bin/libstdc++.so.6
0xf7628340  0xf7634748  Yes (*)     /lib32/libpthread.so.0
0xf74f3a80  0xf75ebaec  Yes (*)     /lib32/libc.so.6
0xf7742830  0xf7759a8f  Yes (*)     /lib/ld-linux.so.2
0xf74c8e00  0xf74d80fc  Yes         bin/libgcc_s.so.1
0xf74a57c0  0xf74b810c  Yes (*)     bin/libtier0_srv.so
0xf74948c0  0xf74986d8  Yes (*)     /lib32/librt.so.1
0xf73b2120  0xf73cf93c  Yes (*)     bin/libvstdlib_srv.so
0xf6f116c0  0xf6f76758  Yes (*)     bin/dedicated_srv.so
0xf6ee1600  0xf6eebf04  Yes (*)     bin/libsteam_api.so
0xf6b43f00  0xf6ccc550  Yes (*)     /srcds/scorps_mausoleum/orangebox/bin/engine_srv.so
0xf664e3e0  0xf6669494  Yes (*)     /srcds/scorps_mausoleum/orangebox/bin/soundemittersystem_srv.so
0xf6532170  0xf65df454  Yes (*)     /srcds/scorps_mausoleum/orangebox/bin/materialsystem_srv.so
0xf50a3060  0xf5130b68  Yes (*)     /srcds/scorps_mausoleum/orangebox/bin/studiorender_srv.so
0xf4f33420  0xf50280c0  Yes (*)     /srcds/scorps_mausoleum/orangebox/bin/vphysics_srv.so
0xf4ea8840  0xf4ee88d8  Yes (*)     /srcds/scorps_mausoleum/orangebox/bin/datacache_srv.so
0xf4e7f240  0xf4e89754  Yes (*)     /srcds/scorps_mausoleum/orangebox/bin/shaderapiempty_srv.so
0xefbc6be0  0xf0331470  Yes (*)     /srcds/scorps_mausoleum/orangebox/hl2mp/bin/server_srv.so
0xef8c5a40  0xef8d11e4  Yes (*)     /srcds/scorps_mausoleum/orangebox/bin/scenefilecache_srv.so
0xebc306c0  0xec5570bc  Yes (*)     bin/steamclient.so
0xebaf6e80  0xebb0cc78  Yes (*)     bin/libtier0_s.so
0xeba79d90  0xebac46c0  Yes (*)     bin/libvstdlib_s.so
0xeb9db490  0xeba4aa2c  Yes (*)     bin/crashhandler.so
0xeb84ca90  0xeb874cb4  Yes (*)     /srcds/scorps_mausoleum/orangebox/hl2mp/addons/sc_protect.so
0xeb7f4490  0xeb822104  Yes (*)     /srcds/scorps_mausoleum/orangebox/hl2mp/addons/plr.so
0xeb7969e0  0xeb7b55c8  Yes (*)     bin/libtier0.so
0xeb6d6b10  0xeb7000d4  Yes (*)     bin/libvstdlib.so
0xeb12d7c0  0xeb26ce10  Yes         /srcds/scorps_mausoleum/orangebox/hl2mp/addons/mattie_eventscripts.so
0xeaea7c80  0xeaf947c4  Yes         /srcds/scorps_mausoleum/orangebox/hl2mp/addons/eventscripts/_engines/python/Lib/plat-linux2/libpython2.5.so.1.0
0xeae7ea00  0xeae7f2a8  Yes (*)     /lib32/libutil.so.1
0xea240f70  0xea279db4  Yes         /srcds/scorps_mausoleum/orangebox/hl2mp/addons/eventscripts/_engines/python/Lib/plat-linux2/psyco/_psyco.so
0xe8236a68  0xe8239ed8  Yes         /srcds/scorps_mausoleum/orangebox/hl2mp/addons/eventscripts/_engines/python/Lib/plat-linux2/array.so
0xf773bd58  0xf773cd64  Yes         /srcds/scorps_mausoleum/orangebox/hl2mp/addons/eventscripts/_engines/python/Lib/plat-linux2/math.so
0xe82310fc  0xe82320a0  Yes         /srcds/scorps_mausoleum/orangebox/hl2mp/addons/eventscripts/_engines/python/Lib/plat-linux2/time.so
0xe81686ac  0xe816b0ec  Yes         /srcds/scorps_mausoleum/orangebox/hl2mp/addons/eventscripts/_engines/python/Lib/plat-linux2/_struct.so
0xe80dfe20  0xe80e2264  Yes         /srcds/scorps_mausoleum/orangebox/hl2mp/addons/eventscripts/_engines/python/Lib/plat-linux2/strop.so
0xe809a478  0xe809c698  Yes         /srcds/scorps_mausoleum/orangebox/hl2mp/addons/eventscripts/_engines/python/Lib/plat-linux2/collections.so
0xeae837bc  0xeae84960  Yes         /srcds/scorps_mausoleum/orangebox/hl2mp/addons/eventscripts/_engines/python/Lib/plat-linux2/_md5.so
0xe8095794  0xe8097160  Yes         /srcds/scorps_mausoleum/orangebox/hl2mp/addons/eventscripts/_engines/python/Lib/plat-linux2/_sha.so
0xe8090790  0xe809387c  Yes         /srcds/scorps_mausoleum/orangebox/hl2mp/addons/eventscripts/_engines/python/Lib/plat-linux2/_sha256.so
0xe807d790  0xe808dfa4  Yes         /srcds/scorps_mausoleum/orangebox/hl2mp/addons/eventscripts/_engines/python/Lib/plat-linux2/_sha512.so
0xe8039c9c  0xe803aa94  Yes         /srcds/scorps_mausoleum/orangebox/hl2mp/addons/eventscripts/_engines/python/Lib/plat-linux2/_random.so
0xe7f8cefc  0xe7f92334  Yes         /srcds/scorps_mausoleum/orangebox/hl2mp/addons/eventscripts/_engines/python/Lib/plat-linux2/_socket.so
0xe7f88934  0xe7f89a50  Yes         /srcds/scorps_mausoleum/orangebox/hl2mp/addons/eventscripts/_engines/python/Lib/plat-linux2/fcntl.so
0xe7f866c8  0xe7f86dd8  Yes         /srcds/scorps_mausoleum/orangebox/hl2mp/addons/eventscripts/_engines/python/Lib/plat-linux2/_bisect.so
0xe74a9480  0xe772ef58  Yes (*)     bin/libsteam.so
0xe70d14b0  0xe71b2a04  Yes         bin/libsteamvalidateuseridtickets.so
0xe7a75a90  0xe7a7d0d8  Yes (*)     /lib32/libnss_files.so.2
0xe7a6ebe0  0xe7a71be8  Yes (*)     /lib32/libnss_dns.so.2
0xe7a5c630  0xe7a688f8  Yes (*)     /lib32/libresolv.so.2
0xe7a4b6f0  0xe7a54fd0  Yes         /srcds/scorps_mausoleum/orangebox/hl2mp/addons/eventscripts/_engines/python/Lib/plat-linux2/datetime.so
(*): Shared library is missing debugging information.
End of Source crash report
----------------------------------------------