Linux Distro Performance Differences

[Velsk]

I've been running a 24 player TF2 server on a single core VDS running Debian 6.0 for the past couple of weeks. I chose Debian since I am most familiar with Ubuntu. However, the server is full pretty much all day long, and when it's full top shows the CPU usage fluctuating between 75%-95% depending on the map and state of the game. The server framerate drops < 100fps quite frequently in these conditions. There are small noticeable hiccups in game that I would like to alleviate.

Due to this I began looking into compiling a custom kernel to see if I can get the server running a little smoother. However, kernel.org is currently down, so that is not a possibility at the moment.

Browsing through the Knowledge-base a bit I saw it mentioned that Gentoo was the distro of choice for performance with Linux. However, this post was from 2009. Is this still the case, and realistically how much of a difference is there between the two with vanilla kernels?

[Edge100x]

Debian/Ubuntu has come quite a way in the last couple of years and now performs pretty well by default, if SELinux is not enabled.

I'm a little surprised that you're seeing such dips with only a 24-slot server. Are you running particularly intensive maps, or particularly intensive plugins?

Orangebox is lightly multithreaded, and having the OS using another core is nice, so upgrading to a 2-core configuration might be something you can consider to eke out a bit more performance.

Have you also thought about trying Windows? Orangebox performs much better on Windows, in its current state.

Have you also tried limiting the FPS with fps_max 100 (or so)? This might help to smooth things out a little by reducing overhead. The game runs at 66 tick internally, and the only benefit to a higher FPS is in helping ticks to fire at the right times.

[Velsk]

I'd like to stick with Linux mainly because this is kind of a hobby project, and I find tinkering around in Linux much more fun than Windows.

I was thinking about upgrading to a 2 core VDS, but I wanted to do so mainly to set up a second server at some point. Perhaps this is the route I will go, but first I'd like to figure out how to squeeze out as much as I can from this one core.

I'll try limiting the fps to 100 and see how that goes. Thanks for the suggestion.

I'm running the default map rotation with a few pruned out and the following SourceMod plugins:

Code: Select all

[SM] Listing 28 plugins:
  01 "SuperLogs: TF2" (2.0.28) by Thomas "CmptrWz" Berezansky & psychonic
  02 "HLstatsX CE Ingame Plugin" (1.6.14) by psychonic
  03 "MapChooser" (1.3.8) by AlliedModders LLC
  04 "RCON Lock" (0.6.7) by devicenull
  05 "Basic Info Triggers" (1.3.8) by AlliedModders LLC
  06 "Admin Menu" (1.3.8) by AlliedModders LLC
  07 "Basic Votes" (1.3.8) by AlliedModders LLC
  08 "[TF2] gScramble Team Manager" (3.0.8) by Goerge
  09 "Player Commands" (1.3.8) by AlliedModders LLC
  10 "Sound Commands" (1.3.8) by AlliedModders LLC
  11 "AFK Manager" (3.4.3) by Rothgar
  12 "Basic Commands" (1.3.8) by AlliedModders LLC
  13 "Basic Comm Control" (1.3.8) by AlliedModders LLC
  14 "Simple AllTalk Manager" (1.3.0) by Simple Plugins
  15 "Forlix FloodCheck" (1.71) by Forlix (Dominik Friedrichs)
  16 "Anti-Flood" (1.3.8) by AlliedModders LLC
  17 "Basic Ban Commands" (1.3.8) by AlliedModders LLC
  18 "Fun Commands" (1.3.8) by AlliedModders LLC
  19 "Fun Votes" (1.3.8) by AlliedModders LLC
  20 "Nextmap" (1.3.8) by AlliedModders LLC
  21 "Win panel for losing team" (1.3) by Reflex
  22 "Admin File Reader" (1.3.8) by AlliedModders LLC
  23 "Client Preferences" (1.3.8) by AlliedModders LLC
  24 "Rock The Vote" (1.3.8) by AlliedModders LLC
  25 "Auto change map" (1.3) by Mleczam
  26 "Admin Help" (1.3.8) by AlliedModders LLC
  27 "Basic Chat" (1.3.8) by AlliedModders LLC
  28 "Reserved Slots" (1.3.8) by AlliedModders LLC

[Edge100x]

Those don't look too unusual/crazy. Please let me know if the fps_max setting helps.

Also make sure that sv_minrate is set to a high value like 100000, and sv_maxrate is set to 0. Not having these is the most common performance mistake we see made.

[Velsk]

fps_max alone didn't help.

I forgot to mention that I'm running the TF2 replay system with offsite storage. So just to test things out I disabled it last night. Along with fps_max 100 the CPU utilization is now averaging between 65-75%. The framerate seems a bit more stable, but I have still seen the CPU usage spike to the 90s once or twice. Does this sound normal to you, or still too high?

Also, are you sure those rate settings are correct? I've never been on a server that forces a min rate so high. I've typically seen between 1500-5000, with a max between 20000-60000. These wouldn't effect CPU utilization though, would they?

[Edge100x]

The new CPU usage sounds much more reasonable.

The rates I gave are the ones I recommend using, because clients usually set a much-too-low "rate" setting. There is no harm to using a high "rate" unless the client is on a very slow connection such as dialup or you need to restrict server bandwidth usage, and a high rate will help to prevent choke. Choke tends to be more prevalent when CPU usage is high because the server can't as accurately measure data rates.

[Velsk]

Do you think there would be any noticeable performance difference in switching to Gentoo over Debian?

What about running a custom kernel? I was looking into following the advice on this wiki. Are there any things to be aware of when doing this with a VDS?

[Edge100x]

You could certainly try a custom kernel with different settings if you wish. The biggest performance increase would likely come from removing optional firewall (iptables) components, as they drag the system down a bit. Switching to a different Linux distro shouldn't help performance versus compiling your own kernel, as the kernel is the key component here.

My own extensive performance testing doesn't agree with much of what that guide says. For instance, using an -rt kernel actually decreased performance in my testing, because it significantly increased overhead. I also found that using an "idler" application in the background did not help, and actually reduced performance by restricting the amount that the processor could turbo. You should also never run a server as a real-time process if you have only a single core (because if it freezes, your entire VDS will become unresponsive), though it's useful for multi-core systems.

[Velsk]

Well, I ended up recompiling the kernel with processor type set to Xeon, timer frequency to 100Hz. It seems like my average cpu usage is down slightly, but not much. I didn't use the rt kernel patch, and am not using the idler application.

Regardless, I am still unable to get a stable framerate on the server. Even if I set fps_max to 66.66 there are regular framerate drops on the server when it is full. This happens no matter what I set the min/max rates to and only when the server is close to full. When the framerate drops it results in a slight stutter seen on all clients.

Is a single core Linux VPS simply insufficient for running a 24 man TF2 server, or am I missing something?

[Edge100x]

A single core VDS should generally be sufficient for a single server of any type. OB does run best on Windows, though, and OB is lightly multithreaded, so another core might help.

Something must be amiss here, but I'm not sure what.

[Velsk]

Using htop to view the CPU utilization of each thread of srcds shows that nearly 100% of the CPU utilization resides in one thread. So I don't see how this would resolve the issue.

At that point, if two cores on a VDS were required for stable FPS it would be more cost efficient to get a game server running on a dedicated server.

Anyway, if it helps at all here is my server.cfg:

Code: Select all

// General server settings
hostname "Not Real"
sv_password ""
sv_lan 0
sv_region 1

// Disable built in voting
sv_allow_votes 0
sv_vote_issue_nextlevel_allowed 0

// Rcon settings
rcon_password "notreal"
sv_rcon_banpenalty 15 // # minutes to ban users who fail rcon authentication
sv_rcon_log 1 // Enable rcon logging
sv_rcon_maxfailures 3 // Max # times a user can fail rcon authentication before being banned
sv_rcon_minfailures 5 // # of times a user can fail rcon authentication in sv_rcon_minfailuretime before being banned
sv_rcon_minfailuretime 10 // # of seconds to track failed rcon authentications


// Game settings
mp_timelimit 30
mp_maxrounds 4
mp_winlimit 0
mp_allowspectators 1
mp_autoteambalance 1

// Server logging
log on
logaddress_add 127.0.1.1:27500
sv_log_onefile 1
sv_logbans 1
sv_logblocks 0
sv_logecho 0
sv_logfile 1
sv_logflush 0
sv_logsdir "logs"

// Voice settings
sv_voiceenable 1
sv_alltalk 1

// Bandwidth rates/settings
sv_allowupload 1
sv_allowdownload 1
sv_minrate 40000
sv_maxrate 100000
sv_minupdaterate 33
sv_maxupdaterate 66
sv_mincmdrate 33
sv_maxcmdrate 66

// Run in pure mode
sv_pure 1
sv_pure_kick_clients 1

Running with the following command line flags:

Code: Select all

-autoupdate -game tf +map ctf_2fort +fps_max 100

[Edge100x]

A dedicated server at the price point of a 2-core VDS would not be as fast. Machines that inexpensive are unfortunately quite old and slow.

You've changed that quite a bit from our default configuration, which I wasn't expecting; I thought you'd be using our baseline. That means we will have to backtrack a bit here. Have you tried going back to our included one as a test? Have you experimented with disabling the various threading cvars (sv_parallel_packentities, sv_parallel_sendsnapshot, net_queued_packet_thread)?

You should definitely add "net_splitpacket_maxrate 100000" to the configuration, as well. Not having that can cause skipping in-game (showing as gaps in the net_graph).

[Velsk]

I downloaded all the server files myself on the VDS so I didn't have the baseline config. Do you have a link to the baseline config you're referring to?

I also caved in and upgraded to the 2 core VDS plan, which bumped my processor speed up a bit as well. Out of curiosity, what's the basis for determining what kind of machine you're put on when you get with the VDS plans?

[Velsk]

The second core did indeed seem to stabilize the framerate. CPU utilization looks like it has reduced as well, but this might just be due to the way htop is averaging these values. Either that or the increased CPU speed.

I still noticed some gaps in the net_graph, as you mentioned, and I tried "net_splitpacket_maxrate 100000" which seemed to do the trick.

Thank you for all your help. I really appreciate it.

Now I will have to see if I can run two servers with no issues in the coming days.

[Edge100x]

I don't have a link, but I can dig it up and post the config, if you'd like. The cvars I listed are most of the relevant ones for you, I think.

Where a server goes mostly depends on space availability. If a machine can't fit an upgraded server (since it needs another CPU core and more memory than the machine has available), the VDS is automatically moved to a different node as part of the upgrade. Some nodes are faster than others, as we are continually purchasing new machines. We do our best to satisfy customer requests to be put onto certain nodes, if a particular speed of processor is desired.