Terrible performance on the E5-2690 v2

[henk717]

Hello server fanatics,

When i signed up for NFo i got assigned a server with a Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz CPU.
From day one the server has been underperforming immensely maximizing the CPU usage with a mere 10 players on Half-Life 2 Deathmatch coop maps (Also without addons loaded the issue persists). This is particulairly surpricing since our old server at a competitor used to run a weaker Intel Xeon X5650 and we never faced CPU issues on that machine with our setup.

When support was unable to provide me with a solution for my current setup i was moved to a Intel(R) Xeon(R) CPU E5-2697 v3 in my first month at NFo. This had decreased the CPU load to an acceptable degree and the server had been performing good on the same slot count as our old host (Although not managing to go much higher then that which still might indicate the OS is not performing correctly). Finding this a good solution i capped the slots occordingly and was satisfied with NFo's service from that moment.

This was until today when the Linux servers where moved away from the node we where happily on back to the Intel(R) Xeon(R) CPU E5-2690 v2. There are currently not enough players to load test but the CPU was already at 20% with merely 1 player joining which is the same amount as it was when i first signed up indicating the server will not make it past 9 slots without performance degradation.

I have already opened a support ticket but staff member Vanderburg recommended me to make this public on the forum so the other community members or John can help me out since apparently it is impossible to move my server back to a CPU it performs well on. Vanderburg stated that he finds it unlikely that i will receive any performance gain from a 400mhz slower CPU that is a generation newer and did not offer me a solution up to this point. I agree with Vanderburg that it seems unlikely that a 400mhz slower CPU would cause performance gain, but since this was the case i could assume that the E5-2690 is not well optimized to run Half-Life 2 servers where the

Here is a summary of the CPU's behavior on the server :
- The game engine can only use 1 core, increasing cores will have no effect.
- The CPU load is mostly caused by the players and maps , removing plugins only has a very small effect as they are well optimized.
- The machine is running 66tick with a high rate value to ensure all entities arrive to the clients without lag, this can not be changed without sacrafising performance further. Rate changing experiments did not resolve the issue when attempted anyway but merely lessened the impact slightly.
- The machine runs the NFo installation of Ubuntu Server 14.04 LTS without any kernel or network stack modifications.
- The server process is forced to CPU1 so it does not continiously shift between CPU0 and CPU1. This was originally not the case and did not cause this issue. This trick while making sure other processes do not interfere with the main process is also no solution to the problem.
- The CPU load increases as more players join, this does not increase the amount of NPC's we have active on the map. It is possible the load is caused by the networking elements of the engine given the relatively high amounts of data compared to non cooperative servers.
- Using the same plugin setup we did not face issues on a managed server from the competitive host using a Intel Xeon X5650. Because i had no insight in there setup i am unaware what the differences are. I managed to fetch the CPU name from a crash dump.

Any attempt at modifying cvars has had no effect up to this point, the issue seems to be beyond the HL2DM configuration.

Please let me know if someone has an idea how to solve the issue as this severely impacts our server performance making it unplayable beyond 10 players.

Many thanks,
Henky!!

[Edge100x]

It sounds like you have an extremely intensive server configuration! I haven't heard of an HL2MP server using so much CPU power. It would certainly be worth looking into what you can do to reconfigure it. Using a lower tickrate might help, for instance. You could also try a newer OS, as a later kernel version will have better PVHVM drivers; that would help a bit when it comes to load-balancing. You should avoid locking the process to a single core, and you should make sure that your configuration enables any variables that increase threading, such as sv_parallel_sendsnapshot.

I can verify that the the E5-2690 v2 and E5-2697 v3 have similar per-core performance, from a hardware perspective. Both are far, far faster than an X5650, even considering virtualization overhead.

[henk717]

A complete rebuild of the server won't be possible without massive downtime, an OS switch here for is not an option i'd like to take up to this point. sv_parallel_sendsnapshot has been experimented with without much impact.

The E5-2690 v2 and E5-2697 v3 supposedly having similar performance still leds me to believe E5-2690 v2 is not properly optimized for the source engine as the E5-2697 v3 has a very noticable performance gain. We are able to increase the slot count by at least 5 slots and had it on 18 succesfully on most maps. The intense maps run well up to 15 slots. The E5-2690 v2 by comparison will have maxed out its core around 10 slots on these maps meaning we can't hit the slot count we used to. The old unknown configuration of the old host ran the intensive maps fine with the 16 slot limit i payed for on a X5650 with the same cvar settings.

On your advice i have removed the CPU lock script, as it was not originally in place when we where on the E5-2690 v2 this wont resolve the issue.

Are you aware of any Kernel related or networking related issue that can drastically increase the CPU usage for a process? If so, any tips on how to remedy the situation?

[Edge100x]

A complete rebuild of the server won't be possible without massive downtime, an OS switch here for is not an option i'd like to take up to this point. sv_parallel_sendsnapshot has been experimented with without much impact.

These were reasonable next steps. I'm not sure what else to recommend that you do.

The E5-2690 v2 and E5-2697 v3 supposedly having similar performance still leds me to believe E5-2690 v2 is not properly optimized for the source engine as the E5-2697 v3 has a very noticable performance gain.

My description of their performance as being comparable is based on my own testing with various types of game servers, including Source servers. I'm not repeating something that someone else said.

I am not aware of any efforts by Intel to architect processors to optimize them to run Source engine game servers and it seems unlikely that they would do so. On Valve's end, the same code runs on both in the same way; Valve doesn't include special optimizations for different processor types.

Both the E5-2690 v2 and E5-2697 v3 are relatively modern, non-EOL processors that perform well. But, over the next few years, they will inevitably start to be phased out on our network as newer machines are rolled out -- mostly because the newer machines have more cores and allow us to run more customers under the same power envelope as the older ones. Intel's strategy with server processors right now is to add more cores, rather than increasing the performance of each core, so we don't expect much of a performance change for single-threaded applications (presumably some of their enterprise customers are requesting that strategy).

The old unknown configuration of the old host ran the intensive maps fine with the 16 slot limit i payed for on a X5650 with the same cvar settings.

Either your old host was actually using a much faster processor than the X5650, or you had a different software configuration there. We have never used that processor here, but we did use the X5660, which was the same generation and at a higher clock speed (2.8 GHz instead of 2.66 GHz); we phased the X5660 out network-wide quite some time ago because it fell behind the performance of newer processors (the E5-2690 v1 was the next generation and had better per-core performance).

You could always go back to your old host if you wish.

Are you aware of any Kernel related or networking related issue that can drastically increase the CPU usage for a process?

If there is a kernel or networking issue, you would see that in the system or interrupt CPU usage (as opposed to the process CPU usage), so you should look for that.

If so, any tips on how to remedy the situation?

If there were a kernel problem, you would want to try updating to a newer kernel, as I recommended previously. In general, newer kernels will have better performance.

You could move your server into our shared (standalone) server system.

You could try Windows, since we know that Source engine performance is better on Windows -- but you suggested that you didn't want to reinstall anything, and Windows would also cost more.

[henk717]

Further performance testing so far has indicated the impact of the CPU switch not to be as big as the gain we had from the switch once it happened. Idle CPU usage seems higher but the decrease in performance seemed less then expected.

Going back to the old host is certainly not going to happen, don't get me wrong this post was only made after encouragement of the support staff and overall i have been very satisfied with the service here, its just an oddity i can't quite place which is why i asked help diagnosing why the CPU usage is seemingly high. We are aware it is a CPU demanding server however this persists even on lighter maps and with the addons disabled.

Switching to Windows while a lot easier then switching to another linux distribution in terms of setup would be impossible, even if the performance on Windows is more optimized our plugins are coded for Linux and can not be adapted due to vast differences on the windows platform and the way the engine operates. The bugs on Windows in the past also have not been ideal for cooperative servers this is more streamlined now but in the general sense cooperative servers are better run on Linux where Puzzle servers for instance run better on the Windows platform. This incompatibility was the main reason to move away from a managed solution and get a VPS.

I will do more testing in the weekends as this is when the server will reach its max player capacity more easily.

A potential cause i could think off is the lack of proper xen drivers on the guest. I noticed the panel does not have the ISO and could not find a resource online to check inside the VM if these drivers are present. If this can be checked and it turns out this is not installed it might help the overall performance if i install the additions so the server can better adapt to the Xen infrastructure.

[Edge100x]

In terms of drivers, the Linux kernel itself includes them. Newer kernels will typically have updated, more optimized drivers. The drivers mostly come into play for I/O, though, and wouldn't affect the CPU usage of an application very much.

[Nlbn]

Registered because this forum appears in Google results and I didn't find any results for my processor when I was buying it...

Intel Xeon 2690 v2

LinX (210.2 GFlops), CPU-Z (3302), CINEBENCH (13.48), Performance test (13728), and RealTemp temperature (80 max).

[Edge100x]

Nibn, that isn't a pleasant necro, because by bumping a super-old post with a performance concern to the forefront, you may confuse current customers and make them think that they have to worry about performance. Your post doesn't relate to the original concern and isn't about a service here, which is also confusing.

You will need to make your post somewhere else.