VDS + TF2 + 32 Players -- sv_fps 66.7 Anyone?

[Python]

Has anyone here been able to achieve a solid sv_fps of 66.7 on a full server (32 players) for Team Fortress 2 on their VDS.

I'm starting to believe a VDS can't achieve this vs a Dedicated server. I've done everything i could think of to try to achieve a lag free server but have had no success.

Tried:
- Disabled Soucemod so no mods/plugins are loaded
- Re installed the OS (Ubuntu 10.10 64bit). Base install only SRCDS running.
- Re installed SRCDS (This was done after the OS rebuild)
- Tried assigning to srcds to a single core and also assigned it to all cores to see if that made a difference.

I've had no luck in achieving something solid under decent load. CPU Utilization peaks and stabilizes at 100% for the SRCDS process/instance when the server is 32/32.

NFO has upgraded me to their fastest cpu:

yyy@xxx:~$ cat /proc/cpuinfo | grep CPU
model name : Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz
model name : Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz
model name : Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz

Still same results.

I'm interested in other clients running a VDS + tf2 32players.

If anyone is interested in checking out the server to with net_Graph 4 the ip address is:
70.42.74.139:27015

[Edge100x]

What maps and mode are you running on the server? Are there any bots?

Does it run at 66 FPS at lower player counts? What FPS do you tend to see at 32?

Are you using our configs? Have you also tried a managed VDS?

[Python]

What maps and mode are you running on the server? Are there any bots?

Does it run at 66 FPS at lower player counts? What FPS do you tend to see at 32?

Are you using our configs? Have you also tried a managed VDS?

Hi Edge,

The server is a mix between Payload and CP. Payload defiantly takes a high hit on the server FPS. When you get into some more tight congestion with lots going on i've had the sv_fps drop down into the yellows (20's)

No bots, server is 100% real players.


The server runs at 66 FPS up till it gets around 28-30 players. Worse if its a payload map.
This is currently on pl_goldrush.

14:49:50 stats
14:49:50 CPU In (KB/s) Out (KB/s) Uptime Map changes FPS Players Connects
0.00 142.91 606.49 1838 80 51.72 32 1571
14:49:51 stats
14:49:51 CPU In (KB/s) Out (KB/s) Uptime Map changes FPS Players Connects
0.00 142.52 595.94 1838 80 56.64 32 1571
14:49:52 stats
14:49:52 CPU In (KB/s) Out (KB/s) Uptime Map changes FPS Players Connects
0.00 141.55 608.42 1838 80 46.96 32 1571
14:49:52 stats
14:49:53 CPU In (KB/s) Out (KB/s) Uptime Map changes FPS Players Connects
0.00 143.23 602.29 1838 80 28.12 32 1571
14:49:53 stats
14:49:53 CPU In (KB/s) Out (KB/s) Uptime Map changes FPS Players Connects
0.00 145.05 606.44 1838 80 50.51 32 1571
14:49:54 stats
14:49:54 CPU In (KB/s) Out (KB/s) Uptime Map changes FPS Players Connects
0.00 143.86 594.68 1838 80 46.33 32 1571
14:49:55 stats
14:49:55 CPU In (KB/s) Out (KB/s) Uptime Map changes FPS Players Connects
0.00 141.77 590.01 1838 80 45.49 32 1571
14:49:58 stats
14:49:58 CPU In (KB/s) Out (KB/s) Uptime Map changes FPS Players Connects
0.00 139.95 620.74 1838 80 46.81 32 1571
14:49:59 stats
14:49:59 CPU In (KB/s) Out (KB/s) Uptime Map changes FPS Players Connects
0.00 140.36 626.88 1838 80 49.43 32 1571
14:49:59 stats
14:49:59 CPU In (KB/s) Out (KB/s) Uptime Map changes FPS Players Connects
0.00 142.02 629.27 1838 80 37.28 32 1571
14:50:00 stats
14:50:00 CPU In (KB/s) Out (KB/s) Uptime Map changes FPS Players Connects
0.00 142.35 633.72 1838 80 45.11 32 1571
14:50:00 stats
14:50:01 CPU In (KB/s) Out (KB/s) Uptime Map changes FPS Players Connects
0.00 144.39 631.85 1838 80 54.49 32 1571
14:50:01 stats
14:50:01 CPU In (KB/s) Out (KB/s) Uptime Map changes FPS Players Connects
0.00 145.10 629.21 1838 80 60.82 32 1571
14:50:02 stats
14:50:02 CPU In (KB/s) Out (KB/s) Uptime Map changes FPS Players Connects
0.00 144.38 659.13 1838 80 57.43 32 1571
14:50:03 stats
14:50:03 CPU In (KB/s) Out (KB/s) Uptime Map changes FPS Players Connects
0.00 143.95 676.57 1838 80 63.29 32 1571
14:50:03 stats
14:50:03 CPU In (KB/s) Out (KB/s) Uptime Map changes FPS Players Connects
0.00 143.99 671.93 1838 80 50.95 32 1571

Where can i get your configs from?
I haven't tried a managed VDS - Whats the difference?

[Python]

I've had a ticket open regarding this: #336210

In the ticket i've tried/mentioned everything in my original post.
It's just something that kinda bothers me when i venture outside of my server and check out other 32p servers and see the sv_fps rock solid at 66.7 with absolutely no fluctuations.

I've pretty much come to the conclusion that its VDS overhead that is causing the issue(s) unless you have some good ideas/suggestions/anything you are able to do on your end Edge.

Thanks for your time,
Python

[Edge100x]

I'd recommend that you make sure it's not restricted to a core and that you experiment with the different threading variables. Some customers have noticed improvements from these. Threading variables include:

-threads x (on the command line)
sv_parallel_packentities 0/1
sv_parallel_sendsnapshot 0/1
net_queued_packet_thread 0/1

We can provide you with our server.cfg if you contact us directly, but mostly I was thinking of those variables, in relation to it.

A managed VDS uses our setup/control panel/software back-end: http://www.nfoservers.com/forums/viewto ... =51&t=4559. The kernel is one that we've optimized a bit more than default ones, and it runs slightly different on Xen, as well.

Have you also tried Windows? When you talk about most servers running better, you are mostly talking about Windows servers, as these are the most common type. Most of our TF2 servers here run on Windows.

[Python]

I'd recommend that you make sure it's not restricted to a core and that you experiment with the different threading variables. Some customers have noticed improvements from these. Threading variables include:

-threads x (on the command line)
sv_parallel_packentities 0/1
sv_parallel_sendsnapshot 0/1
net_queued_packet_thread 0/1

We can provide you with our server.cfg if you contact us directly, but mostly I was thinking of those variables, in relation to it.

A managed VDS uses our setup/control panel/software back-end: http://www.nfoservers.com/forums/viewto ... =51&t=4559. The kernel is one that we've optimized a bit more than default ones, and it runs slightly different on Xen, as well.

Have you also tried Windows? When you talk about most servers running better, you are mostly talking about Windows servers, as these are the most common type. Most of our TF2 servers here run on Windows.

I've fooled around with all those settings and notice no impact.
Current settings are as followed:

net_queued_packet_thread 1
sv_parallel_packentities 0
sv_parallel_sendsnapshot 0
net_splitpacket_maxrate 50000
net_maxcleartime 0.01
sv_max_queries_sec_global 100000


I've changed them 1, default settings, etc no improvements. At the time of my ticket that i mentioned earlier the settings were all 1 and maxcleartime and max queries were at default settings.

I haven't tried windows due to the fact that servers can be better managed/configured/automated with linux. I prefer to have all my servers configured with nemrun to watch/fetch/automate updates instantly - usually minutes before the patch is officially announced. Automation is a huge feature for me which is why i went with a VDS.

As for other servers, i look at the competition on ahead of me on gametracker (http://www.gametracker.com/search/tf2/?) and before the NY Sandy outtage i was #3 [Now 9th]. For instance if i were to hop into skial's Dustbowl LA server (#1) my fps there doesn't flinch away from 66.7. I'm going to assume Bottiger runs his servers on Linux due to the fact that he's on the hlds_linux mailing list.

EDIT: yes the server mentioned above is Linux. I added it to HLSW and it indicates v1.2.3.7 (Linux, Dedicated, Protocol 17)

[Edge100x]

You're on very fast hardware and I don't really think that the virtualization overhead would be enough to cause this.

Are the PV-on-HVM drivers properly loading? Have you tried a newer Ubuntu version, such as 12.04?

[Python]

You're on very fast hardware and I don't really think that the virtualization overhead would be enough to cause this.

I fully assumed that and was being baffled at the performance i'm achieving. That's why i decided to bring this to the forums to see if anyone is having any similar issues with their VDS's.


I haven't tried 12.04. Defiantly an option.

I'm going to assume i have no PV-on-HVM driver?
I didn't install anything after doing the base installation from your control panel ~1.5week ago when i rebuilt the OS.

I'm looking at this documentation:
http://wiki.xen.org/wiki/Xen_Linux_PV_o ... HVM_guests

And when i perform the step title:
Verify Xen PVHVM drivers are working in the Linux HVM guest kernel

I get the following results:
xxx@yyy:/etc$ dmesg | egrep -i 'xen|front'
[ 0.000000] ACPI: RSDP 00000000000ea020 00024 (v02 Xen)
[ 0.000000] ACPI: XSDT 00000000fc00eaf0 0003C (v01 Xen HVM 00000000 HVML 00000000)
[ 0.000000] ACPI: FACP 00000000fc00e8d0 000F4 (v04 Xen HVM 00000000 HVML 00000000)
[ 0.000000] ACPI: DSDT 00000000fc002c40 0BC09 (v02 Xen HVM 00000000 INTL 20090123)
[ 0.000000] ACPI: APIC 00000000fc00e9d0 000D8 (v02 Xen HVM 00000000 HVML 00000000)
[ 0.000000] ACPI: HPET 00000000fc00eab0 00038 (v01 Xen HVM 00000000 HVML 00000000)


I also don't have a folder under /etc/xen

xxx@yyy:/etc$ ls /etc/xen*
ls: cannot access /etc/xen*: No such file or directory

[Edge100x]

I don't think that the default Ubuntu 10.10 kernel includes PV-on-HVM drivers. But, 12.04 definitely does.

Which reminds me that I need to add 12.10 to the list.

[Python]

What would your recommendations be.
Upgrade to 12.04 ? 12.10 ? or install the drivers on my current build of 10.10?

[Python]

I think i'm leaning towards a OS re install.

I can't find any documentation/resources on PV-on-HVM for Ubuntu 10.10

Do you have any documentation for this?

[Edge100x]

Ubuntu upgrades can be hairy and others have had difficulty going from a regular to PV-on-HVM kernel when they've tried to switch to 12.04. I would recommend a clean reinstall.

[soja]

Focus on the sv_parallel_sendsnapshot command. On high-traffic servers this can absolutely tank your server fps. On my 60 slot CS:S server this set to 1 provides fluid and smooth gameplay at spawns with 50+ people. Set to 0 causes us to hover between 25-35.

I have had first hand experience with vds overhead, and it is a killer to high traffic servers (the 60 slot I mentioned before used to be on a vds, but even on a 1270 @ 3.4Ghz it was not performing to standards). I honestly thought a 32 slot TF2 would run fine on it though, might make me re-think purchasing one for a TF2 venture my community is doing.

[Edge100x]

soja, I don't think that the problem that Python is seeing is widespread. I suspect that the step of switching to an improved kernel will help his situation, as it removes a big chunk of any virtualization overhead. Windows servers also have similar optimizations because we install drivers for it. I'll continue to follow up with him in this thread.

[soja]

okay thanks, i'm interested to see how this turns out