Server spike issues on VDS

[Mr.Tickles]

Greetings Everyone,
Going to go ahead and revive my thread as I am encountering a similar issue again.
Anywhere between fifteen minutes to an hour each server will experience a brief half a second stutter that everyone on the server experiences.

My setup has changed a bit since I originally opened this thread a few months ago so here is a update on what the specs of my Unmanaged VDS is and what I am running on it.

Specs:
6 HT CPU cores
9GB of Ram
Windows Server 2012

Servers and other items running.
24 player slot tf2 server X 5 (these are almost always near full)
One 18 player slot private tf2 server (almost always empty as it is used for scrims and lobbies)
Teamspeak 3 server (averages 10-30 users at a time)
Community website webserver (just a Mybb forum nothing fancy)
Three or four Mysql databases for various plugins we run on our servers.

What I have tried so far:

I have assigned each of the servers their own CPU cores as I have heard that TF2 servers like other source games has trouble running on multi hyper threaded cores. Did this ages ago but double checked it recently to make sure it remained unchanged.

Made sure the EXE's for each server is running with high priority (not real time or normal)

Configured each of the TF2 servers to the same config that Soja provided with me in this thread earlier (which has made the servers far more stable). Double checked to make sure everything was correct.

I had my dev check all plugins to ensure they were all up to date which seems to be the case.

Checked my resources for CPU and Memory. Currently running at about 48% CPU and 68% on Memory.


Any ideas? Need further info? Let me know either way. Thanks in advance!
~Tickles

[kraze]

Your issue could be with disk usage, depending on how much your databases are being accessed it could present an issue and it would explain the small stutters you see every 15 few minutes or so.

If you're able to I'd turn them off for an hour or so and monitor the servers to see if the skipping stops. if it does the fix it fairly easy and cheap. You can add 5GB of SSD space to your VDS for $3 via our order page.

Your CPU usage is also at the point where we don't recommend you let it get much higher. After 50% you could start to see performance artifacts in game.

[Mr.Tickles]

I am going to have to look into the database side of things for sure. Will do some testing later.

As for the CPU usage I have been able to run upwards of 70% before without issue but I will defer to your knowledge of server capabilities. Any recommendations on how to lower CPU usage without effecting our current setup?

[Mr.Tickles]

Here is a small update on the situation. Kraze you very well may be right. However, it most likely is not the mysql databases but the fact that I run replays on all five of my TF2 servers. The accrue about 1gb of data a day so I imagine they are the culprits that are exceeding my write speed.

My next plan of action is to get a 5gb SSD upgrade this weekend and reroute the replays to be stored there and see if this resolves it.

Thanks for getting me pointed in the right direction.

[kraze]

No problem, please do let us know what your testing turns up.

[Mr.Tickles]

Well after making the SSD upgrade and moving both the mysql databases and our replays to the SSD it has helped out quite a bit but has not fully resolved the issue.

Been watching it for sometime and may have found some things worthy of note.

1. It appears that the C drive is still occasionally exceeding its write speed. Much less often now but still seems to do so.

Theory: I am wondering if it is exceeding write speed due to a temporary write to the C drive from the servers to the C drive before being moved to the F drive (SSD) thus still causing it to freeze up gameplay.


2. I have six cores and six servers. I read that each core should be able to maintain a 32 slot TF2 slot server each. I run 24 slot servers. However I did notice that CPU 0 and CPU 1 have close to max use on both (around 94%).

Theory: The OS of course is taking up a good amount of CPU 0 and 1 but I imagine that a larger part may be the mysql databases. My thought might be to add to more cores and dedicate the first two to OS and mysql and the others to one TF2 server a piece.

3. I have to manually set the affinity for each server every restart. Anyone have a way to permanently set the affinity in windows server 2012?


I would like some advice and opinions on these observations. Especially number 1 as I am at a loss on what to do about this.

[kraze]

1. It appears that the C drive is still occasionally exceeding its write speed. Much less often now but still seems to do so.

Theory: I am wondering if it is exceeding write speed due to a temporary write to the C drive from the servers to the C drive before being moved to the F drive (SSD) thus still causing it to freeze up gameplay.

It is possible during certain times you're seeing some disk lag but It's hard to say. With replays disabled did your issues go away?

2. I have six cores and six servers. I read that each core should be able to maintain a 32 slot TF2 slot server each. I run 24 slot servers. However I did notice that CPU 0 and CPU 1 have close to max use on both (around 94%).

Theory: The OS of course is taking up a good amount of CPU 0 and 1 but I imagine that a larger part may be the mysql databases. My thought might be to add to more cores and dedicate the first two to OS and mysql and the others to one TF2 server a piece.

3. I have to manually set the affinity for each server every restart. Anyone have a way to permanently set the affinity in windows server 2012?

Last time I checked the OS isn't limited to CPU 0 or 1, it would use what's available at the time. It is possible it is using those cores more frequently, though, but there isn't a real way to confirm that.

I don't recommend setting server affinities, Windows does a decent job of balancing processes around cores to keep performance solid and stopping it from doing so could cause a core to get routinely pegged at 100%. This also isn't telling anything else to touch that core, but telling that process to stay on that core.

Adding additional cores could definitely allow the server to stretch a bit, this would likely give more processes its own core, such as MySQL. If the lag is due to that then it could surely help. You may also try just turning off replays on say two servers to see if the issues stops, if it does then it's likely disk usage.

I'm sure you've seen by now that the real answer here is trial and error, I wish I had more insight but these types of issues require a lot of it.

[Mr.Tickles]

It is possible during certain times you're seeing some disk lag but It's hard to say. With replays disabled did your issues go away?

Sadly no. Your original thoughts on the mysql databases using the larger amount of write speed appears to be correct. I have done some "pruning" and got rid of a few of our extra that were not really needed or used but that does not seem to have helped that much.

Last time I checked the OS isn't limited to CPU 0 or 1, it would use what's available at the time. It is possible it is using those cores more frequently, though, but there isn't a real way to confirm that.

I thought so as well. That is why it surprised me to find that the first two cpu's were being favored by the bulk of OS and other processes. If nothing else do you think that getting an additional two cores would help distribute the load a bit better? Because as of this moment it appears that every few minutes or so those two cpu's spike and any servers tied to those cpu's take the brunt of it.

I don't recommend setting server affinities, Windows does a decent job of balancing processes around cores to keep performance solid and stopping it from doing so could cause a core to get routinely pegged at 100%. This also isn't telling anything else to touch that core, but telling that process to stay on that core.

As I understand it source servers such as TF2, CSS:GO, and etc have issues when run on multiple cores. I have seen multiple sources (including NFO knowledge base) that recommend assigning each to its own core.

I most definitely understand that it does not prevent other process's from using those cores but from what I have seen it seems that any cores above cpu 1 are hanging out at about 54%-65% when full.

Correct me if I am wrong on this though. After all this is my first extended period as a TF2 server host. Especially the scale that we have extended to at this point.

[kraze]

I thought so as well. That is why it surprised me to find that the first two cpu's were being favored by the bulk of OS and other processes. If nothing else do you think that getting an additional two cores would help distribute the load a bit better? Because as of this moment it appears that every few minutes or so those two cpu's spike and any servers tied to those cpu's take the brunt of it.

Yes, it is possible. The more cores you have the more processes will be balanced out. While Windows will always do its own thing in the background, it could definitely help. This won't help if you have a server using 90% CPU usage, but if you have two processes sharing a core it would likely move one of those off.

As I understand it source servers such as TF2, CSS:GO, and etc have issues when run on multiple cores. I have seen multiple sources (including NFO knowledge base) that recommend assigning each to its own core.

I most definitely understand that it does not prevent other process's from using those cores but from what I have seen it seems that any cores above cpu 1 are hanging out at about 54%-65% when full.

Correct me if I am wrong on this though. After all this is my first extended period as a TF2 server host. Especially the scale that we have extended to at this point.

They wouldn't have issues, no. While it's true that servers don't normally take advantage of multiple cores that's not the issue here.

We're pretty unanimous here when it comes to assigning affinities and rarely recommend customers do it, can you point me to the KB article that says this?