Server lagging(timeouts)

[soja]

Our ze server lags when specific things happen on the map ze_stargate_escape_v8.

We have copied the exact config/plugin/etc structure to our old host and it didn't seem to lag.

When it happens on this server, the console is spammed with "DataTable warning: (class player): Out-of-range value (-1500.000000) in SendPropFloat 'm_vecBaseVelocity', clamping." which i have read is harmless, but that is the only thing that i can find. When this happens the server has small timeouts where it is completely unresponsive for a very short amount of time, and then is fine once again. On this map players go through a teleporter that then places the player in a tube, with trigger_push, after that they go into another teleporter which places them in the final room. The server lags only when there are 4+ players in this tube, which is when the timeout and error happens. The more players in the tube, the laggier it is and the more timeouts we have.

We have decompiled the map and it doesn't look like there is anything on the maps end to cause this, we have joined other ze servers running the map and they have no problems like this.

Is there some reason to the cause of this?

(We are on a managed dedicated server that is underloaded, if it matters.)


Thanks

[kraze]

Usually we do not recommend taking files from other host and using them here as it can cause issues. Have you attempted doing a complete re-install of the mod and only copying the basics such a a config file only?

Also is it specifically only this map " ze_stargate_escape_v8."?

[soja]

Usually we do not recommend taking files from other host and using them here as it can cause issues. Have you attempted doing a complete re-install of the mod and only copying the basics such a a config file only?

Also is it specifically only this map " ze_stargate_escape_v8."?

Yes, it only happens on that map, and only when people are using the tube. We have noclip enabled, so players shouldn't be pushing others outside of it.

We did update to the latest sourcemod snapshot recently, but that neither fixed nor caused this issue i don't think, i will double check.

[kraze]

Have you checked your log files to see if see if anything more is displayed there?

[soja]

My mistake, we updated the zombie reloaded plugin 2-3 days ago, but that was after the problem happened.

I was watching console when it happened, and it was spamming the error in my OP, i can't exactly count how many times it was displayed, but it was a lot, and i think it was per-person.

[soja]

Checked logs, and nothing, not even the error mentioned in the OP.

[Edge100x]

If you're on a dedicated machine, we don't have to worry about this being hardware, at least -- it's purely about your server setup.

Have you tried different values for sv_maxrate, sv_minrate, and net_splitpacket_maxrate? Try 100000 for net_splitpacket_maxrate, for instance. Are you seeing choke or packet loss in net_graph 3 output, as well? What does the graph look like there?

Are you running the same OS as the other servers that you have tested this configuration on? Pluguins can run differently on different OSes.

[soja]

If you're on a dedicated machine, we don't have to worry about this being hardware, at least -- it's purely about your server setup.

Have you tried different values for sv_maxrate, sv_minrate, and net_splitpacket_maxrate? Try 100000 for net_splitpacket_maxrate, for instance. Are you seeing choke or packet loss in net_graph 3 output, as well? What does the graph look like there?

Are you running the same OS as the other servers that you have tested this configuration on? Pluguins can run differently on different OSes.

It isn't that kind of lag, it is literally server timing out(as if it crashed) for less than a second, and then responds again. I know it isn't hardware, I was thinking the server was exporting logs or something like that, which included the spam i as getting in the console, for each player, and that is why it lags(trying to keep the logs exported while running the server). We do have hlstatsx logging, ill try disabling logging next time we are on that map, to see if that is somehow the cause.

We are not seeing choke or packet loss in net graph, we have small red triangles that indicate a timeout(similar to if we shut down the server, where you see at the bottom the red that appears) and it comes back. It is not per client, or for a client in the tube, it is server-wide(players at spawn experience the lag to, me as a spectator anywhere on the map, i see it too).

Our net_splitpacket_maxrate and sv_min/maxrate are fine, we have tuned them to our liking.

Would sv_parallel_sendsnapshot being set to 0 cause this, or to 1? We have been fighting to get it set to 1, because the control panel puts it in as 0, because it provides more stable fps.(we had to make a seperate cfg for all of our rates, to override the defaults that keep being reset by the panel).

We were on Centos 5.x on our other server, and gentoo here.

[Edge100x]

Having the wrong net_splitpacket_maxrate value can lead to exactly this symptom, which is why I brought that up. I do not know anything else that would cause this on a normal server, software-wise.

You should not change sv_parallel_sendsnapshot from our value, since it's known to work best. You should also not try to override the system if it is fixing your sv_minrate/sv_maxrate for you (assuming that you have the option on the file manager page set to off); it only does this if you choose truly terrible values that are known to cause problems (like an sv_minrate of 0 -- far too low -- or an sv_maxrate of 10000 -- far too low).

Seriously, we set our defaults for good reason! They are finely-tuned and known to offer the highest performance, providing the least choke and loss. The only reason that a client would need to muck with his personal rate is if he's on a very slow connection such as a modem link.

[soja]

Having the wrong net_splitpacket_maxrate value can lead to exactly this symptom, which is why I brought that up. I do not know anything else that would cause this on a normal server, software-wise.

You should not change sv_parallel_sendsnapshot from our value, since it's known to work best. You should also not try to override the system if it is fixing your sv_minrate/sv_maxrate for you (assuming that you have the option on the file manager page set to off); it only does this if you choose truly terrible values that are known to cause problems (like an sv_minrate of 0 -- far too low -- or an sv_maxrate of 10000 -- far too low).

Seriously, we set our defaults for good reason! They are finely-tuned and known to offer the highest performance, providing the least choke and loss. The only reason that a client would need to muck with his personal rate is if he's on a very slow connection such as a modem link.

I know what you are talking about, we have been messing with the splitpacket command for awhile now(before nfo), and we were pleased to see you set it to 50,000(increases cpu usage, and no other host does this i have seen), but that is slightly lower than what we needed. We focus on server fps when determining if we have optimal performance, with the settings we have now(including but not limited to sv_minrate; sv_maxrate; net_splitpacket_maxrate; sv_maxupdaterate; sv_maxcmdrate; sv_parallel_sendsnapshot) we are able to keep a consistent 60+ server fps with 50 players defending a holding point.

I have not been able to find much information on "sv_parallel_sendsnapshot", and what it does, can you clarify? Our server FPS dips into the teens with it off, and 60+ with it on.

The only commands i can think of that could cause this, while looking at our rates cfg are:

sv_parallel_sendsnapshot 1
sv_parallel_packentities 1
sv_max_queries_sec_global 3000

Would any of these perhaps be the cause? If so i will put the map back in rotation and we can check with them toggled off/modified.


Thank you

[Edge100x]

Yes, setting those to 1 will cause problems like the ones you are describing (this is why we started adjusting them, long ago). You should use our defaults, instead:

sv_parallel_packentities 0
sv_parallel_sendsnapshot 0

You could also try "-threads 1" on the server command line, if these don't seem to be enough.

[soja]

Yes, setting those to 1 will cause problems like the ones you are describing (this is why we started adjusting them, long ago). You should use our defaults, instead:

sv_parallel_packentities 0
sv_parallel_sendsnapshot 0

You could also try "-threads 1" on the server command line, if these don't seem to be enough.

we will try these when we go to that map again.

Can you elaborate on what "sv_parallel_sendsnapshot" does? entering it into the console with no variable doesn't give output like others do.

Thanks

[Edge100x]

Valve has not described it that I have seen, but from the name, it sounds as though it causes the engine to spawn one or more threads to send its generated snapshots to each client, theoretically allowing the server to move on to handling the next tick at the same time. In practice, we have not found it to improve performance.

[soja]

We just had the same lag spike at the same point in the map.

I managed to capture the net graph while it was happening.

The server was freezing and everyone felt the lag.

http://serveredirect.com/upload/uploade ... 581026.jpg

http://serveredirect.com/upload/uploade ... 585598.jpg

http://serveredirect.com/upload/uploade ... 591771.jpg


Any more ideas?

[Edge100x]

Those do not show a clear problem. They do tell us that the server was humming right along with a perfect FPS/tickrate, and that you didn't see any loss or choke.

Your received snapshot rate from the server seems pretty low (~15), but since there's no choke, it's not from the server holding back snapshots. What do you normally see there?

Some of the packets are a little large, which makes me wonder about whether you're running into problems with net_splitpacket_maxrate. I know we saw issues with <50000 and >50000, but I have never seen complaints about 50000 before, though, so that may not be it.

To be clear, you're using all of our recommended default settings on the server right now, correct? What "rate" setting are you using on the client?

As you mentioned originally, it could be a map design problem, and potentially one that only comes up on specific OSes. I haven't seen a map problem like this before, but you're running an unusual map and configuration, and the engine is undergoing frequent changes.