L4D2 - Timeout and Disconnect message during map changes

[Patriot]

Hello,
I apologize in advance for the long post

System Details:
6-core VDS with 6Gb RAM and Windows Server 2008 R2 x64, CPU usage never above 25%, memory usage never above 50%. Website and MySQL DBs both hosted at NFO in the same data center as the VDS. No additional software on the VDS other than a control panel to manage the game servers. Command line for all l4d2 servers includes “-threads 1”.

The Issue:
During many map changes on the versus and co-op servers, players will get "timeout" and "attempting to reconnect" messages on-screen once the map change begins. The screen will not refresh for 20+ seconds, and then most players are loaded into the next map. Some players get disconnected. This affects all connected players regardless of their physical location and routing to the server.

History:
It happened once before across all of the l4d2 servers and only ended with a full wipe and reload of Windows 2008 R2 x64. At that time, another customer on the same physical machine was doing heavy disk I/O for 20 minutes of every hour. Since that could include 2-3 map changes on my game servers, it may have been a contributing factor. That disk activity has since been stopped.

Current State:
Everything has been running smoothly since the OS reinstall and now suddenly the timeout/disconnect issue appears again. Nothing has been changed in the game server configurations as was the case previously. The game server configurations are stable and not subject to experimentation with plugins. I do all plugin/configuration testing on a separate test server.

There are only two plugins installed that use MySQL DBs:
1. SourceBans with a total of 143 bans, so a tiny DB
2. l4d2 custom stats, which was just wiped clean so also a tiny DB. This plugin updates the players' stats in-game with chat messages, but that has never been a performance issue.

After unloading and moving both of those plugins to the disabled folder, the issue seems to have resolved, but there has not been enough player activity to be 100% sure.

Coincidentally, my WordPress website at NFO, which also uses a MySQL DB, has always been a little slow to load. At one point I moved the entire website/MySQL db to another hosting provider without making any other changes and the loading was noticeably faster. I asked the support team about moving my website/MySQL DB to another machine as a test, but that is not an option.

Attempt to diagnose:
Using the Performance Monitor, I looked at the network activity while a couple of the servers were occupied and observed the following:

1. With only 8-10 players connected, Under Network Activity there are literally hundreds of IPs and ISP addresses showing as connected to srcds.exe, and all of them are getting various levels of network traffic. These connections remain even when the servers are empty.

Q: Is this just showing all of the connections to the physical machine hosting my VDS or is something else happening?

2. Before disabling the stats and SourceBans plugins, the TCP Connections section showed a connection to the MySQL DB on port 3306, as expected, but with a latency of 230ms, which seems very high. That is the same latency to my PC over RDP, and it is a bit laggy.

Q: Is Performance Monitor accurately reporting the latency to the MySQL DB, and if so, is it an expected value or is there an issue?

Is there something else I should look at or other steps to take to identify the cause of this issue?

Any ideas or solutions you can offer are greatly appreciated.

Best regards,

Patriot

[Edge100x]

From what you're describing, levels are just taking a long time to load. Is this also what you're seeing server-side, in the server logs?

If so, the first thing that I would look at is memory usage and I/O speed. Do you have these servers sharing a single install, for efficiency? How much memory do you have available when the problem is occurring?

[Patriot]

From what you're describing, levels are just taking a long time to load. Is this also what you're seeing server-side, in the server logs?

No, and this does not seem like a load time issue. Load times vary greatly in l4d2 as determined by the map file size and the slowest loading connected player. Many times I've experienced very long load times when a very slow loading player is connected. Even then, no network disconnect occurs.

Instead, the network disconnect happens almost immediately when the map load starts, not after a long loading period. And HLSW confirms the network disconnect for 20+ seconds when the timeout message appears in-game.

If so, the first thing that I would look at is memory usage and I/O speed. Do you have these servers sharing a single install, for efficiency? How much memory do you have available when the problem is occurring?

There is 6Gb or RAM on the VDS and usage has never been more than 50%. I'm not sure if you mean memory, disk, or network I/O.

Each server instance uses a separate game installation, so they are not sharing a single installation as in a forked server setup.

Right now, with the MySQL db plugins unloaded, a full server did not have any timeout/disconnect issues. So this again leads me to think there is an issue with the connection to the MySQL dbs.

Can you also comment on the 230 ms latency between srcds and the MySQL DB as reported in Performance Monitor?

Thank you,
Patriot

[Edge100x]

It could easily be a plugin issue, yes. It was important to look at loading times, as well, since those would potentially be easier to resolve.

I have not seen latencies shown in performance monitor before, so I don't have insights into how accurate it would be. What does a tracert show? And, how long do those queries take to execute locally on the webserver? If you noticed that the same queries also run very slowly through the website, that could be a red flag that the database is not optimized properly.

[Patriot]

After hours of testing yesterday I thought I had the issue traced to the MySQL db connections, but then the timeout happened again even with both of the MySQL-dependent plugins uninstalled from the test server.

However, I can recreate the in-game timeout behavior exactly by alt-tabing out of the game, shutting down the game server, and then returning to the game. The timeout "attempting to reconnect" message is exactly the same. That indicates that the game server is losing network connection with the players when the timeout happens during map change.

Late last night I did an OS reinstall, which seemed to fix the disconnect problem when it originally appeared, and rebuilt a couple of the servers. The first map change on the co-op server threw the network timeout error.

Note: for others that are using an SSD, it also gets wiped clean during an OS reinstall, not just the disk with the OS...

[Edge100x]

Years ago, we had a problem with map changes timing clients out because of Valve stats uploads on L4D servers. I haven't seen this recently, but you might try the plugin that stopped them back then: https://forums.alliedmods.net/showthread.php?t=85395

If the servers don't do it stock, but do it after you reload you config, you may need to go through everything you run line-by-line, though .

[Patriot]

The timeouts are still happening with varying frequency on all four servers.

Now, the whole VDS went offline and crashed all of the game servers, which were mostly full.

In Event Viewer there are 5 new warning messages that started right when the VDS went offline:

Event 129, XenVBD
Reset to device, \Device\RaidPort0, was issued.

Code: Select all

Log Name:      System
Source:        XenVbd
Date:          9/20/2012 6:57:33 PM
Event ID:      129
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      **********
Description:
Reset to device, \Device\RaidPort0, was issued.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="XenVbd" />
    <EventID Qualifiers="32772">129</EventID>
    <Level>3</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-09-20T23:57:33.359375000Z" />
    <EventRecordID>6839</EventRecordID>
    <Channel>System</Channel>
    <Computer>patriotgamesvds</Computer>
    <Security />
  </System>
  <EventData>
    <Data>\Device\RaidPort0</Data>
    <Binary>0F001800010000000000000081000480040000000000000000000000000000000000000000000000000000000000000000000000810004800000000000000000</Binary>
  </EventData>
</Event>

Is there a hardware problem with the physical machine?

Patriot

[.=QUACK=.Major.Pain]

We see the same issue.

I run 2 servers, and one runs fine, but the other does as you describe.
Sort of locks up at times.

We use a control panel the restarts the server if it happens.
I get emailed everytime it has to restart, and it happens 10-12 times a day.

Weird that only one server does it, and still does it after several verifying the files.

[Edge100x]

Patriot, please open a support request if you'd like me to investigate the physical machine.

That error suggests a problem with the Xen drivers, but it may be difficult to determine what triggered it. Possibilities include heavy disk or network access, suddenly-low memory, or unusual activity. We have a new version of the Xen drivers that you could try to see if it helps with crashes. I am not certain that this would relate to your L4D2 problem.

[Patriot]

Patriot, please open a support request if you'd like me to investigate the physical machine.

That error suggests a problem with the Xen drivers, but it may be difficult to determine what triggered it. Possibilities include heavy disk or network access, suddenly-low memory, or unusual activity. We have a new version of the Xen drivers that you could try to see if it helps with crashes. I am not certain that this would relate to your L4D2 problem.

Already done.

Thanks,

Patriot

[Patriot]

We see the same issue.

I run 2 servers, and one runs fine, but the other does as you describe.
Sort of locks up at times.

We use a control panel the restarts the server if it happens.
I get emailed everytime it has to restart, and it happens 10-12 times a day.

Weird that only one server does it, and still does it after several verifying the files.

I think we both use the EzUGCC game panel, which works very well. A couple of things to think about with your freezes:

1. check under that server's main left4dead2 folder for a "dumps" folder. If it is there and full of .dmp files (e.g crash_srcds.exe_20120921011316_1.dmp) then srcds.exe is crashing. More often than not the crashes are caused by either outdated gamedata files, bad or incorrect cvars in the server.cfg (or other .cfgs if you run them) or a SourceMod plugin.

2. If you use it, check the Sourcemod logs folder for any plugin or extension related errors

3. In ugccmon.cfg, located in C:\Windows\System32 folder, make sure your [serverCheck] setting for that server monitor is not too low (default is 30 seconds!). Otherwise, if the server is slow to start or has other issues, you can get into a monitor-driven restart loop.

4. In the UGCC server panel, verify that that server's the command line is good and under the executable tab that process priority is correct.

You probably know all of this but I thought it might spark an idea to help with the debugging.

The issue my servers are having is timeouts during map changes, not server crashes requiring a restart.

[.=QUACK=.Major.Pain]

Correct. Using EzUGCC.

I have 2 dump files but they are from July 27.
Nothing recent.

We don't use Sourcemod/metamod.
Just running plain Jane servers for custom maps.

Today was a good day after yesterdays update.
Only 1 crash today. They might have fixed an issue.