Managed Linux VDS Issues & Crashes

[ren3gade]

I am currently on the 6 core VDS - Managed Panel - Linux package.
I started with the 3 core, went to the 4 core, and now on the 6 core due to my CPU usage skyrocketing over 50% when I have players on my servers, as well as my memory starting to dip a tad too low.

I currently run 6 gameservers on my VDS. (2 TF2, 3 CS:GO, 1 GMOD) All of my servers are modded some in one way or another. But they are all modded differently, and all run different addons.

I have been having "Unexpected Server Stop" problems with ALL of my game servers installed. My GMOD server, my 2 TF2 Servers, and my 3 CSGO Servers. They all randomly crash usually 1-5 times per day and in my events log it always says "Unexpected Server Stop". So I have scanned all sourcemod error logs, all condebug logs, and the event logs. I can't see anything on the addons/mods side that is causing the crashes.

The only thing that stands out to me is these error logs that pop up everytime one of the servers crash:

Code: Select all

./exec_sngjb.run: line 324: 26751 Segmentation fault  (core dumped) $HL_CMD

warning: Could not load shared library symbols for linux-gate.so.1.
Do you need "set solib-search-path" or "set sysroot"?
email debug.log to linux@valvesoftware.com
Mon Jan  6 00:21:53 CST 2014: Server Quit

Code: Select all

./exec_sngjb.run: line 324: 21164 Segmentation fault  (core dumped) $HL_CMD

warning: Could not load shared library symbols for linux-gate.so.1.
Do you need "set solib-search-path" or "set sysroot"?
debug.cmds:5: Error in sourced command file:
No function contains program counter for selected frame.
email debug.log to linux@valvesoftware.com
Sun Jan  5 14:28:01 CST 2014: Server Quit

But I was told from support that those errors in the events log are normal. So I have ignored them.

Has anyone else ever experienced issues like this on a managed VDS - Linux?

Now don't get me wrong, I know NFO is an amazing host, in my opinion the best game server host open. But I have been trying to solve this with them the past like 2 weeks, and have gotten no where near fixing this issue, other than being told it's probably an addon/mod error/issue. So 2 nights ago, I started wiping my game servers, and reconfiguring them all from scratch to make sure there are no errors or problems, and all the addons are installed cleanly. And the one server I finished, is still crashing 2-3 times per day randomly. BUT, I am going to continue to wipe them all and redo them all to make sure they are all configured as cleanly as possible and are error free. I am just here asking if anyone else has ever experienced issues like this.

Thanks guys

[Edge100x]

Wiping and redoing them is not usually an effective troubleshooting step, because it just brings the server back to the way it was.

Have you troubleshooted by removing plugins/configuration adjustments one-by-one? Or, alternately, by starting with nothing and seeing if the server still crashes, then adding back in the customizations one-by-one? These are usually the best approaches to investigate random crashes and other seemingly intractable problems. I mentioned them to you in a support request a few times, I believe.

[ren3gade]

Wiping and redoing them is not usually an effective troubleshooting step, because it just brings the server back to the way it was.

Have you troubleshooted by removing plugins/configuration adjustments one-by-one? Or, alternately, by starting with nothing and seeing if the server still crashes, then adding back in the customizations one-by-one? These are usually the best approaches to investigate random crashes and other seemingly intractable problems. I mentioned them to you in a support request a few times, I believe.

Yes I understand that.
Each server that I re-do.. I add one thing, I restart, I check for errors in logs, and make sure there are no problems. If there are no errors and it works fine, then I move on to the next one.

The servers don't crash every 20 minutes. And they don't crash when the server is empty it seems. It's always when we have players on the server. And our servers require certain addons/gamemodes to even make it the server it is, so we can't fill up our servers when they are just reset to a stock server and wait 24 hours to see if it crashes at all.

We have a heavily modded GMOD server that is under construction, and it has a handful of script errors. But it is empty and never has players on it cuz it is new and locked up. It runs almost all the same scripts as our other popular GMOD TTT server, but this locked up one that never has players, has never crashed. But our GMOD TTT who has players on it, and is running basically the same scripts, crashes.

So I can't think of any other way to try and "test/troubleshoot" the problem. As our type of servers don't just allow us to remove our whole mod that makes the whole server for a day, and see if that fixes it.

On our TF2 Jailbreak server: the only sourcemod errors we have are sourcebans database errors, but the server crashes 3-6 times per day randomly. Now I haven't wiped, and re-done/troubleshooted this server at all yet. But I didn't think sourcebans not being able to write to a database would cause the server to just crash. But who knows. On servers in the past when I was young/didn't know much about servers years ago, I would just throw tons of crap on the server and even used yucky event scripts, and I wouldnt rarely have server crashes. But who knows, maybe I just am getting unlucky and am having a tiny little problem on each different server.

[ren3gade]

I'm curious.. is there a lot of users who use the Linux Gentoo Managed VDS to host their gameservers on?

I am wondering if other OS's would be more stable, run smoother, or use up less resources? Such as Windows, or another Linux distro.

[Edge100x]

Yes, there are many users with Managed Linux servers. Windows is much more common, but we haven't seen problems with Linux, and we have been using it for high-end services for many years.

[Madhavok]

I have a managed linux VDS, and I don't have any issues, and I didn't have any issues with Windows when I had that. I didn't notice a difference, except linux has prioritization of servers with people in them and empty servers. Performance wise, I didn't see anything else. I picked linux for the ability to run a 128 tick match GO server.

[ren3gade]

Fair enough. I suppose I will continue to wipe, troubleshoot, reconfigure, addon by addon.. and see how they run when I'm all finished!

[soja]

Do the crashes happen on any particular map/event?

Our servers used to crash a lot on player suicide, but that was due to a plugin which we removed. It took a while to figure out, but we got it.

Which server crashes most often, and what is the IP? The best bet would be to play in the server until it crashes, and see if you can narrow down what happened while playing.

[ren3gade]

Do the crashes happen on any particular map/event?

Our servers used to crash a lot on player suicide, but that was due to a plugin which we removed. It took a while to figure out, but we got it.

Which server crashes most often, and what is the IP? The best bet would be to play in the server until it crashes, and see if you can narrow down what happened while playing.

Team Fortress 2 Jailbreak: 72.5.195.58:27015
Team Fortress 2 Deathrun: 74.91.115.7:27015

Those 2 seem to be the ones crashing often. (2-5 times per day)

After wiping and redoing my CSGO Idle and CSGO Aim DM.. those 2 seem to be running smooth so far.

I wiped and redid my TF2 DR 2 days ago, and as soon as I was done with it, I had like 4 server crashes that same day. These are the only 2 errors i get in my SM error logs on TF2 Deathrun:

Code: Select all

SM] Displaying call stack trace for plugin "deathrun.smx":
L 01/05/2014 - 17:22:59: [SM]   [0]  Line 260, /home/groups/alliedmodders/forums/files/2/1/5/9/5/5/115483.attach::moveplayers2()
L 01/05/2014 - 17:49:29: Error log file session closed.
L 01/05/2014 - 18:10:49: SourceMod error session started
L 01/05/2014 - 18:10:49: Info (map "dr_mario_b4") (file "errors_20140105.log")
L 01/05/2014 - 18:10:49: [SM] Plugin encountered error 15: Array index is out of bounds
L 01/05/2014 - 18:10:49: [SM] Displaying call stack trace for plugin "deathrun.smx":
L 01/05/2014 - 18:10:49: [SM]   [0]  Line 260, /home/groups/alliedmodders/forums/files/2/1/5/9/5/5/115483.attach::moveplayers2()
L 01/05/2014 - 18:28:32: Error log file session closed.
L 01/05/2014 - 18:28:42: SourceMod error session started
L 01/05/2014 - 18:28:42: Info (map "dr_horrors_prefinal") (file "errors_20140105.log")
L 01/05/2014 - 18:28:42: [SM] Plugin encountered error 15: Array index is out of bounds
L 01/05/2014 - 18:28:42: [SM] Displaying call stack trace for plugin "deathrun.smx":
L 01/05/2014 - 18:28:42: [SM]   [0]  Line 260, /home/groups/alliedmodders/forums/files/2/1/5/9/5/5/115483.attach::moveplayers2()
L 01/05/2014 - 18:30:42: Error log file session closed.
L 01/05/2014 - 18:30:52: SourceMod error session started
L 01/05/2014 - 18:30:52: Info (map "dr_industrial_v2b") (file "errors_20140105.log")
L 01/05/2014 - 18:30:52: [SM] Plugin encountered error 15: Array index is out of bounds
L 01/05/2014 - 18:30:52: [SM] Displaying call stack trace for plugin "deathrun.smx":
L 01/05/2014 - 18:30:52: [SM]   [0]  Line 260, /home/groups/alliedmodders/forums/files/2/1/5/9/5/5/115483.attach::moveplayers2()
L 01/05/2014 - 18:56:33: Error log file session closed.
L 01/05/2014 - 18:56:44: SourceMod error session started
L 01/05/2014 - 18:56:44: Info (map "dr_woods_b5") (file "errors_20140105.log")
L 01/05/2014 - 18:56:44: [SM] Plugin encountered error 15: Array index is out of bounds
L 01/05/2014 - 18:56:44: [SM] Displaying call stack trace for plugin "deathrun.smx":
L 01/05/2014 - 18:56:44: [SM]   [0]  Line 260, /home/groups/alliedmodders/forums/files/2/1/5/9/5/5/115483.attach::moveplayers2()
L 01/05/2014 - 19:07:15: Error log file session closed.
L 01/05/2014 - 19:07:24: SourceMod error session started
L 01/05/2014 - 19:07:24: Info (map "dr_prison_escape_v7") (file "errors_20140105.log")
L 01/05/2014 - 19:07:24: [SM] Plugin encountered error 15: Array index is out of bounds
L 01/05/2014 - 19:07:24: [SM] Displaying call stack trace for plugin "deathrun.smx":
L 01/05/2014 - 19:07:24: [SM]   [0]  Line 260, /home/groups/alliedmodders/forums/files/2/1/5/9/5/5/115483.attach::moveplayers2()
L 01/05/2014 - 19:18:33: Error log file session closed.
L 01/05/2014 - 19:18:44: SourceMod error session started
L 01/05/2014 - 19:18:44: Info (map "dr_minecraft_b10") (file "errors_20140105.log")
L 01/05/2014 - 19:18:44: [SM] Plugin encountered error 15: Array index is out of bounds
L 01/05/2014 - 19:18:44: [SM] Displaying call stack trace for plugin "deathrun.smx":
L 01/05/2014 - 19:18:44: [SM]   [0]  Line 260, /home/groups/alliedmodders/forums/files/2/1/5/9/5/5/115483.attach::moveplayers2()
L 01/05/2014 - 19:18:44: [SM] Native "ChangeClientTeam" reported: Client 19 is not in game
L 01/05/2014 - 19:18:44: [SM] Displaying call stack trace for plugin "deathrun.smx":
L 01/05/2014 - 19:18:44: [SM]   [0]  Line 186, /home/groups/alliedmodders/forums/files/2/1/5/9/5/5/115483.attach::SwitchTeam()
L 01/05/2014 - 19:42:40: [SM] Native "ChangeClientTeam" reported: Client 10 is not in game
L 01/05/2014 - 19:42:40: [SM] Displaying call stack trace for plugin "deathrun.smx":
L 01/05/2014 - 19:42:40: [SM]   [0]  Line 186, /home/groups/alliedmodders/forums/files/2/1/5/9/5/5/115483.attach::SwitchTeam()
L 01/05/2014 - 19:44:18: Error log file session closed.
L 01/05/2014 - 19:44:28: SourceMod error session started
L 01/05/2014 - 19:44:28: Info (map "dr_badwater") (file "errors_20140105.log")
L 01/05/2014 - 19:44:28: [SM] Plugin encountered error 15: Array index is out of bounds
L 01/05/2014 - 19:44:28: [SM] Displaying call stack trace for plugin "deathrun.smx":
L 01/05/2014 - 19:44:28: [SM]   [0]  Line 260, /home/groups/alliedmodders/forums/files/2/1/5/9/5/5/115483.attach::moveplayers2()
L 01/05/2014 - 20:12:25: Error log file session closed.
L 01/05/2014 - 20:12:35: SourceMod error session started
L 01/05/2014 - 20:12:35: Info (map "dr_cocainum_b4") (file "errors_20140105.log")
L 01/05/2014 - 20:12:35: [SM] Plugin encountered error 15: Array index is out of bounds
L 01/05/2014 - 20:12:35: [SM] Displaying call stack trace for plugin "deathrun.smx":
L 01/05/2014 - 20:12:35: [SM]   [0]  Line 260, /home/groups/alliedmodders/forums/files/2/1/5/9/5/5/115483.attach::moveplayers2()
L 01/05/2014 - 20:26:10: SourceMod error session started
L 01/05/2014 - 20:26:10: Info (map "dr_minecraft_b10") (file "errors_20140105.log")
L 01/05/2014 - 20:26:10: [SM] Unable to load extension "vanillaweps.ext": /servers/sngdr/tf/addons/sourcemod/extensions/vanillaweps.ext.so: cannot open shared object file: No such file or directory
L 01/05/2014 - 20:26:10: [SM] Unable to load plugin "tNoUnlocksPls.vanillaweps.smx": Required extension "UnlockBlock" file("vanillaweps.ext") not running
L 01/05/2014 - 20:26:10: [store.smx] Gifts! isn't installed or failed to load. Gifts support will be disabled. Please install Gifts. (http://forums.alliedmods.net/showthread.php?t=175185)
L 01/05/2014 - 20:26:24: [SM] Plugin encountered error 15: Array index is out of bounds
L 01/05/2014 - 20:26:24: [SM] Displaying call stack trace for plugin "deathrun.smx":
L 01/05/2014 - 20:26:24: [SM]   [0]  Line 260, /home/groups/alliedmodders/forums/files/2/1/5/9/5/5/115483.attach::moveplayers2()
L 01/05/2014 - 20:53:13: Error log file session closed.
L 01/05/2014 - 20:53:13: SourceMod error session started
L 01/05/2014 - 20:53:13: Info (map "dr_sawmill_final") (file "errors_20140105.log")
L 01/05/2014 - 20:53:13: [SM] Unable to load plugin "tNoUnlocksPls.vanillaweps.smx": Required extension "UnlockBlock" file("vanillaweps.ext") not running
L 01/05/2014 - 20:53:24: [SM] Plugin encountered error 15: Array index is out of bounds
L 01/05/2014 - 20:53:24: [SM] Displaying call stack trace for plugin "deathrun.smx":
L 01/05/2014 - 20:53:24: [SM]   [0]  Line 260, /home/groups/alliedmodders/forums/files/2/1/5/9/5/5/115483.attach::moveplayers2()
L 01/05/2014 - 20:55:29: SourceMod error session started
L 01/05/2014 - 20:55:29: Info (map "dr_minecraft_b10") (file "errors_20140105.log")
L 01/05/2014 - 20:55:29: [SM] Unable to load extension "vanillaweps.ext": /servers/sngdr/tf/addons/sourcemod/extensions/vanillaweps.ext.so: cannot open shared object file: No such file or directory
L 01/05/2014 - 20:55:29: [SM] Unable to load plugin "tNoUnlocksPls.vanillaweps.smx": Required extension "UnlockBlock" file("vanillaweps.ext") not running
L 01/05/2014 - 20:55:30: [store.smx] Gifts! isn't installed or failed to load. Gifts support will be disabled. Please install Gifts. (http://forums.alliedmods.net/showthread.php?t=175185)
L 01/05/2014 - 20:55:42: [SM] Plugin encountered error 15: Array index is out of bounds
L 01/05/2014 - 20:55:42: [SM] Displaying call stack trace for plugin "deathrun.smx":
L 01/05/2014 - 20:55:42: [SM]   [0]  Line 260, /home/groups/alliedmodders/forums/files/2/1/5/9/5/5/115483.attach::moveplayers2()
L 01/05/2014 - 21:26:11: Error log file session closed.

I contacted the TF2 DR addon coder about the error. (everyone on alliedmods said they are getting this index out of bounds error and whatnot) But he isn't messaging me back. https://forums.alliedmods.net/showthread.php?t=201623

As for the TF2 Jailbreak. I just wiped, and redid that one tonight. I will keep you posted on how well it runs. It currently has zero sourcemod errors.

[ren3gade]

I've had some people watching when the servers crash. so far this is all we've gotten..


1.) When the map was about to change from a rtv. It crashed.
JB just crashed with the following last 4 lines:
L 01/08/2014 - 18:25:22: Started map "jb_legojail_b1" (CRC "be92efd3b0c94a6bd7f2b4d9b1b4bec5")
L 01/08/2014 - 18:25:22: -------- Mapchange to jb_legojail_b1 --------
L 01/08/2014 - 18:25:22: Error log file session closed.
./exec_sngjb.run: line 324: 6460 Segmentation fault (core dumped) $HL_CMD
./exec_sngjb.run: line 324: 6460 Segmentation fault (core dumped) $HL_CMD (1 crash)
./exec_sngjb.run: line 324: 31702 Segmentation fault (core dumped) $HL_CMD (another crash)
./exec_sngaimdm.run: line 318: 22175 Segmentation fault (core dumped) $HL_CMD (Another crash)
./exec_sngdr.run: line 324: 7275 Segmentation fault (core dumped) $HL_CMD (another crash)

2.) Crashed once when an admin did @slay all.

3.) Another crashed happened during a map change. The server crashed before it could load the new map.

4.) " I've also seen the /explode @all crash the server. So Admin commands that lead to slaying everyone at a time can lead to a crash, but this isn't always. When me and EGG King where on (EU Time), there was no admins on, so something else triggered the crash."

So it seems rtv's/mapchanges are causing crashes? But I'm not sure why. sv_allowvotes is 0.
TF2 JB Plugins:
http://gyazo.com/5a4d734e0d6f710cf644c6aaa74130f1
http://gyazo.com/3df0d628468bbb84206a703ec01ef732
TF2 JB Disabled Plugins:
http://gyazo.com/b263221445d4c00192404e0cb28fd1f4

TF2 DR Plugins:
http://gyazo.com/3cfd50182449b3158745ea3e8bde529b
http://gyazo.com/822a5bbb3450c501d4240d6145cd9004
TF2 DR Disabled Plugins:
http://gyazo.com/ff6a0019686c25aee6fc05f4466eac7c

[ren3gade]

Anyone got any ideas of what I can do, from the post I made above this?

[Dinnanid]

The VDS I help manage is having the same issue however only on the one server that we had before we transfered to the VDS. I am wondering if that could be the issue. We have not noticed any patterns in the crashing and the server runs Garrysmod TTT.

[stickz]

You should ask Edge to look into this issue further. It's highly like this could be related to the Linux Kernel the Managed system uses. It's definitely not related to any error sessions on sourcemod.

I started experiencing this issue on 4.2.8 and 4.3.5 kernels on unmanged myself. This never appeared before in years of running earlier kernel versions.

I am also curious as to the cause of this issue. Google is not very helpful; but, ironically one of the pages it directs to is this thread because it's heavily related to this topic. The little information I did find... this is something stored in your gameserver process exposed by the Linux Kernel, used for system calls. It's not an actual binary within the Linux operating system.

Here are a few of my crash dumps from both this provider and anther provider.

Ubuntu 14.04 with 4.2.8 Kernel
https://crash.limetech.org/jbx7hczjazb3

Ubuntu 15.10 with 4.3.5 Kernel
https://crash.limetech.org/wkx4c2gjlyon

[stickz]

Furthermore, I highly recommend using the accelerator extension for your sourcemod server. It will provide much more detailed crash reports like the ones I posted above. https://forums.alliedmods.net/showthread.php?t=277703