Server lagging(timeouts)

User avatar
soja
This is my homepage
This is my homepage
Posts: 2389
https://www.youtube.com/channel/UC40BgXanDqOYoVCYFDSTfHA
Joined: Fri May 18, 2012 3:20 pm

Re: Server lagging(timeouts)

Post by soja »

Image

The server was not running at perfect FPS. var is(in lamense terms, as i understand it) the deviation from the average fps, this shows a var of 19, when this should not go over 7-8 for a normally operating server.

The snapshot is low because the server was hopping between single digit FPS and the normal fps, why it got captured at 15 instead of the other fps rates, i have no idea.

We run our sv_maxcmdrate and sv_maxupdaterates at 34.

Our net_splitpacket_maxrate is set quite high, but to my understanding, it will compress the data that it is currently sending out, meaning there will be no difference if it is set to 500,000, or 800,000(ours isn't that high), because the engine could never send that much data.

We cannot use the default settings, if we did our server would be unplayable(specifically using sv_parallel_sendsnapshot set to 0).

Notice in the red box, that is what we see when we experience the lag, the server fps hits single digits at the same time as those red triangles pop up(if you go into a server and shut it down, you will see this but as a solid line, not triangles)

Our current rates are this:

sv_minrate 75000
sv_maxrate 0
net_splitpacket_maxrate 201000

sv_maxupdaterate 34
sv_minupdaterate 33
sv_maxcmdrate 34
sv_mincmdrate 33

sv_parallel_sendsnapshot 1

sv_parallel_packentities 1
sv_max_queries_sec_global 3000
Not a NFO employee
User avatar
soja
This is my homepage
This is my homepage
Posts: 2389
Joined: Fri May 18, 2012 3:20 pm

Re: Server lagging(timeouts)

Post by soja »

Taken from the valve explanation of net_graph for orangebox, describing what those triangles are.

When net_graphshowinterp is 1, this area shows for each client frame how much interpolation was needed. If there is a large gap between packets (packet loss, server framerate too low, etc.), then the client will have insufficient data for interpolation and will start to extrapolate.


(My net_graphshowinterp is 1)
Not a NFO employee
User avatar
soja
This is my homepage
This is my homepage
Posts: 2389
Joined: Fri May 18, 2012 3:20 pm

Re: Server lagging(timeouts)

Post by soja »

Sorry, my client is running at a rate of 999,999.
Not a NFO employee
User avatar
Edge100x
Founder
Founder
Posts: 12948
Joined: Thu Apr 18, 2002 11:04 pm
Location: Seattle
Contact:

Re: Server lagging(timeouts)

Post by Edge100x »

Ah, so you aren't running with sv_parallel_sendsnapshot 0 and sv_parallel_packentities 0. I would expect to see problems even on normal servers with those set to 1. For your testing, I do recommend using 0. Even on very large, very busy servers, we've found it to work best.

Definitely try a lower net_splitpacket_maxrate such as 50000, if just briefly, as that won't hurt. We have found 50000 to be the best value after testing higher and lower ones.

It seems that your updaterate/cmdrate settings are being ignored by the server, or you're defining them in a different file, because your screenshot shows that the negotiated values are at 101/100.

The FPS rate that I'm referring to is in the lower-left corner -- it's labelled "sv". This is the realized FPS and tickrate on the server. The "/s" number on the "in" line is the received snapshots per second, which can be different. I've only previously seen it be different when choke or packet loss is involved, so the fact that it does not match the FPS is intriguing, and likely does relate to what you're seeing AFA the interpolation.

In any case, if you believe you're running the same settings here as on other test servers that work well, then some difference is being missed. In terms of trying to find a workaround, what OS are you running, and what OS are the other servers you're testing with running?
User avatar
soja
This is my homepage
This is my homepage
Posts: 2389
Joined: Fri May 18, 2012 3:20 pm

Re: Server lagging(timeouts)

Post by soja »

As i have said before, with those set to 0, we can't play on the server smoothly, this is a 50 slot ZE(zombie escape) server. Typical servers have people die, and are separated around the map, with our mod, there are always 40+ people alive and playing, and are usually in the same area on the map, which leads to significantly higher resource usage compared to other server types(may be equivalent to a GG:DM server). The server fps under sv: with these set to 0 gets in the TEENS, i will change it and get a screenshot to show you, we would have to run a 30 slot server to keep good fps with these set to the recommended value.

I will try a lower splitpacket when we get back to those map(s).

We have our min and max cmdrate/updaterate forced by the server at 33 or 34, so my client temporarily uses those until i connect to a 100-tick server(such as our scrim server).
Image

I know what server fps is and where it is displayed, as i said in my earlier posts, that is why we had to run the snapshot commands at 1 instead of 0, to stabilize fps.

My thinking is that because the server fps was dipping between single digits and 66.7, the updates were moving up and down alot, and in that particular screencap, it was at 15, moving back up to 34.

We have tested on 1 other server so far(for stargate) running centos 5, i will try windows and centos again on this map.

I am downloading the server files for local testing on a linux box i have here.
Not a NFO employee
User avatar
Edge100x
Founder
Founder
Posts: 12948
Joined: Thu Apr 18, 2002 11:04 pm
Location: Seattle
Contact:

Re: Server lagging(timeouts)

Post by Edge100x »

Interesting that the net_graph output shows the incorrect negotiated updaterate and cmdrate. I guess they must have introduced that bug recently.

What OS are you using here? I mean in terms of Windows versus Linux (the specific Linux distribution does not matter to this). If external Linux servers don't see a problem, and you're running Windows, then switching to Linux may be advisable.
User avatar
soja
This is my homepage
This is my homepage
Posts: 2389
Joined: Fri May 18, 2012 3:20 pm

Re: Server lagging(timeouts)

Post by soja »

Our NFO box is linux(gentoo)
Not a NFO employee
User avatar
Edge100x
Founder
Founder
Posts: 12948
Joined: Thu Apr 18, 2002 11:04 pm
Location: Seattle
Contact:

Re: Server lagging(timeouts)

Post by Edge100x »

Ah. Most of my testing in terms of the threading variables has been on Windows, and the code behaves differently on the two in that regard, which may explain your different experiences with those.

You're certain that the other server was using the exact same configuration (same CS:S version, etc)?
User avatar
soja
This is my homepage
This is my homepage
Posts: 2389
Joined: Fri May 18, 2012 3:20 pm

Re: Server lagging(timeouts)

Post by soja »

Not a NFO employee
User avatar
Edge100x
Founder
Founder
Posts: 12948
Joined: Thu Apr 18, 2002 11:04 pm
Location: Seattle
Contact:

Re: Server lagging(timeouts)

Post by Edge100x »

It's good to see that you have the same realized updaterate with both during that test, though that could change at other times. When you repeated the steps to reproduce your map problem, did it occur with them set to 0?
User avatar
soja
This is my homepage
This is my homepage
Posts: 2389
Joined: Fri May 18, 2012 3:20 pm

Re: Server lagging(timeouts)

Post by soja »

Edge100x wrote:Ah. Most of my testing in terms of the threading variables has been on Windows, and the code behaves differently on the two in that regard, which may explain your different experiences with those.

You're certain that the other server was using the exact same configuration (same CS:S version, etc)?
I am not 100% sure it was exactly identical, it was our old server(vps) that we used before we came to you guys, i did have to update it right before we tried testing on it(CS:S updated after we got here), and we used the latest versions of the things we had(the zombie mod) and we copied the /cstrike/cfg folder and /cstrike/adddons/sourcemod/configs folder.
Not a NFO employee
User avatar
soja
This is my homepage
This is my homepage
Posts: 2389
Joined: Fri May 18, 2012 3:20 pm

Re: Server lagging(timeouts)

Post by soja »

Edge100x wrote:It's good to see that you have the same realized updaterate with both during that test, though that could change at other times. When you repeated the steps to reproduce your map problem, did it occur with them set to 0?
We have not tried that yet, that will be next.

Id prefer not to force a map change with 40+ people in our server, so when we get to it we get to it, although i will nominate it as often as i can :)
Not a NFO employee
User avatar
Edge100x
Founder
Founder
Posts: 12948
Joined: Thu Apr 18, 2002 11:04 pm
Location: Seattle
Contact:

Re: Server lagging(timeouts)

Post by Edge100x »

If the CS:S update came in between, that certainly could relate. Valve made a bunch of changes with it, not all of which were communicated to admins. Here is the list of changes that they did post:

Counter-Strike: Source
- Fixed a client crash related to dynamic model loading
- Fixed dedicated server not receiving restart requests when updates are released
- Fixed IP bans not applying to RCON access
- Fixed server crash exploit related to audio buffer overruns
- Fixed server crash caused by using changelevel2
- Updated console logging system to minimize opening and closing of files, greatly improving performance under certain conditions
- Additional improvements for client/server stability
- Fixed a problem that allowed malicious clients to disable the "ping" and "status" commands for other connected clients
- Fatal engine errors are now written to the server log file
- Added sv_lowedict_threshold and sv_lowedict_action which allow the engine to take action before running out of free edicts
- Possible values for sv_lowedict_action are: 0 - no action, 1 - warn to log file, 2 - attempt to restart the game, if applicable, 3 - restart the map, 4 - go to the next map in the map cycle
- Added changelevel_next server command which changes to the next map in the map cycle
- Added mp_restartgame_immediate ConVar which has the same effect as mp_restartgame without a delay

One possibility is that the engine logs the error that you're seeing now in this particular instance, while it didn't before, and that extra logging is hurting performance. Or, that it is handling the error condition differently in some other way.
User avatar
soja
This is my homepage
This is my homepage
Posts: 2389
Joined: Fri May 18, 2012 3:20 pm

Re: Server lagging(timeouts)

Post by soja »

I managed to capture a screenshot during one of the lag spikes on ze_stargate_escape_v8.

I have always noticed that when a server is using 100% cpu, lerp turns yellow. I know that is not the official cause of lerp changing colors, but that is what i have observed while running srcds based servers.

Notice the server is also at 2fps during these spikes, perhaps something in either our plugins or configs are causing this huge spike.

http://serveredirect.com/upload/uploade ... 393185.jpg


EDIT: this was with sv_parallel_sendsnapshot 0; net_splitpacket_maxrate 50000; log off
Not a NFO employee
User avatar
Edge100x
Founder
Founder
Posts: 12948
Joined: Thu Apr 18, 2002 11:04 pm
Location: Seattle
Contact:

Re: Server lagging(timeouts)

Post by Edge100x »

You're definitely seeing a problem there with the server unable to keep up. Now it's a matter of figuring out how to prevent it (if it's even possible to prevent).

Are you using the console tool in the control panel at all? One thing to note is that you need to stay away from that tool if you are seeing a lot of log output. (Most customers only use the rcon tool, and you shouldn't need to touch the console one, but I want to make sure.)

(If you're curious, we talk about yellow lerp more here: http://www.nfoservers.com/forums/viewto ... =25&t=4028)
Post Reply