VDS + TF2 + 32 Players -- sv_fps 66.7 Anyone?

[Python]

Python, I'm going to try switching you to the new Xen. You can expect to see that tomorrow morning, if all goes as planned with further testing today.

Thanks John i look forward to testing this with you. I'm assuming the xen host will require a reboot which will take down my server breifely. That's fine just let me know.

[soja]

Yeah we were seeing the same thing without the splitpacket command(imagine doubling the players and then seeing spikes ), I would run it at a reasonable amount, for 32 slot TF2 50k seems fine, and will provide a seamless play experience for anyone without net_graph on. If you want the best net_graph possible with around the same play experience, you can max it out. We run 175k for our ze and 75-100k for our TF2 servers.

EDIT: john, is openvz's overhead less or more than xen? We ran our 50 slot server on a 1230v1 on openvz on a lightly loaded shared machine before we came here and we had no performance issues like we experienced on xen here.

[Edge100x]

soja, it depends on the configuration of both. Xen is low-overhead when done correctly, and should be comparable. It also has far fewer limitations than openvz (openvz requires running the same kernel for all VPSes, for instance, and guests have to use the same type of file system), making it better for general-purpose virtualization.

[soja]

thanks john

[soja]

This is the performance of our CS:S server at spawn with 46 players

Code: Select all

16:09:36 stats
16:09:36 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   141.24     1543.19     15753   381          71.52    45       17066   
16:09:36 stats
16:09:36 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   138.96     1529.41     15753   381          73.68    45       17066   
16:09:39 stats
16:09:39 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   137.48     1496.01     15753   381          80.67    45       17066   
16:09:39 stats
16:09:39 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   134.92     1500.15     15753   381          66.65    45       17066   
16:09:40 stats
16:09:40 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   134.13     1493.98     15753   381          66.69    45       17066   
16:09:41 stats
16:09:41 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   133.66     1451.86     15753   381          66.32    45       17066   
16:09:44 stats
16:09:44 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   135.44     1516.23     15753   381          66.64    45       17066   
16:09:45 stats
16:09:45 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   135.43     1513.14     15753   381          66.68    46       17067   
16:09:46 stats
16:09:46 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   135.75     1494.61     15753   381          62.62    46       17067   
16:09:46 stats
16:09:47 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   134.80     1452.66     15753   381          66.67    46       17067   
16:09:47 stats
16:09:48 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   136.85     1383.54     15753   381          66.64    46       17067   
16:09:48 stats
16:09:48 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   138.16     1322.58     15753   381          66.78    46       17067   
16:09:49 stats
16:09:50 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   138.24     1300.16     15753   381          66.66    46       17067   
16:09:50 stats
16:09:50 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   138.04     1279.37     15753   381          66.81    46       17067   
16:09:51 stats
16:09:51 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   139.05     1251.08     15753   381          66.58    46       17067   
16:09:52 stats
16:09:52 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   142.11     1241.72     15753   381          66.66    46       17067   

idk what it is but it seems like your vds overhead is crazy. Either that or my machines turbo boost is on and yours isn't :/

this is on a 1270v1 at 28% CPU usage

[Python]

No change with the xen upgrade. This is on pl_badwater:

Code: Select all

09:03:45 stats
09:03:45 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   132.78     885.93      410     14           36.35    32       122     
09:03:46 stats
09:03:46 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   133.22     871.86      410     14           42.94    32       122     
09:03:46 stats
09:03:46 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   134.41     835.37      410     14           44.90    32       122     
09:03:47 stats
09:03:47 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   136.75     827.14      410     14           49.31    32       122     
09:03:47 stats
09:03:47 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   139.02     826.08      410     14           57.09    32       122     
09:03:48 stats
09:03:48 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   139.85     832.72      410     14           55.12    32       122     
09:03:48 stats
09:03:48 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   140.54     844.49      410     14           60.76    32       122     
09:03:49 stats
09:03:49 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   139.20     858.85      410     14           54.87    32       122     
09:03:49 stats
09:03:50 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   136.63     839.98      410     14           54.08    32       122     
09:03:50 stats
09:03:50 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   137.19     820.56      410     14           35.35    32       122     
09:03:50 stats
09:03:50 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   136.34     812.81      410     14           41.24    32       122     
09:03:51 stats
09:03:51 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   137.05     808.41      410     14           46.32    32       122     
09:03:51 stats
09:03:52 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   138.41     801.61      410     14           52.19    32       122     
09:03:52 stats
09:03:52 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   138.75     796.39      410     14           41.15    32       122     
09:03:52 stats
09:03:53 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   139.38     787.45      410     14           34.33    32       122     
09:03:53 stats
09:03:53 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   139.37     782.27      410     14           33.14    32       122     
09:03:54 stats
09:03:54 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   139.66     769.44      410     14           45.49    32       122     
09:03:54 stats
09:03:54 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   135.92     796.54      410     14           58.06    32       122     
09:03:55 stats
09:03:55 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   132.53     873.49      410     14           31.32    32       122     
09:03:55 stats
09:03:55 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   131.51     900.69      410     14           43.31    32       122     
09:03:56 stats
09:03:56 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   130.59     899.50      410     14           42.86    32       122     
09:03:56 stats
09:03:57 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   128.40     885.29      410     14           31.14    32       122     
09:03:57 stats
09:03:57 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   127.89     872.49      410     14           50.34    32       122     
09:03:57 stats
09:03:58 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   127.49     860.49      410     14           49.87    32       122     
09:03:58 stats
09:03:58 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   126.06     853.67      410     14           43.01    32       122     
09:03:59 stats
09:03:59 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   124.14     836.40      410     14           49.87    32       122     
09:03:59 stats
09:03:59 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   123.35     821.86      410     14           38.90    32       122     
09:04:00 stats
09:04:00 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   124.48     804.04      410     14           46.23    32       122     
09:04:00 stats
09:04:01 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   127.23     790.02      410     14           45.07    32       122     
09:04:01 stats
09:04:01 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   127.94     800.75      410     14           30.65    32       122     
09:04:02 stats
09:04:02 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   128.14     828.18      410     14           50.12    32       122     
09:04:02 stats
09:04:02 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   128.08     864.84      410     14           44.70    32       122     
09:04:03 stats
09:04:03 CPU    In (KB/s)  Out (KB/s)  Uptime  Map changes  FPS      Players  Connects
         0.00   127.15     894.25      410     14           57.41    32       122   

[Python]

I've created ticket #394405 to follow up with this issue.

[Edge100x]

I'm not sure what else to recommend trying while continuing to run Linux, then, if you're already running a properly PV-on-HVM setup and plugin-free server with -threads 1, sv_parallel_sendsnapshot 1, and proper rate settings, etc.

I also don't have a lot of managed Linux clients (whether on a VDS or not) with busy 32-slot servers to compare you to here, to be honest, so I'm not sure what the performance baseline for a server like yours should be. We have a few clients with busy servers on managed Windows VDSes and they have stable tickrates that remain in the 60-70 range, though.

If this was a virtualization-related issue, I was actually expecting that Xen upgrade to help quite a bit. I'd also expect for both Windows and Linux servers to be adversely affected.

Historically, GoldSrc, Source, and Orangebox games have performed much worse on Linux than Windows, but I had thought that Valve improved the situation with their recent updates. It is possible that they haven't.

(You posted the bit about the support request as I was typing this. There's nothing different that I'd say there.)

[Python]

I'm defeated and out of all possible ideas.
I've spent extensive time and resources into this issue and haven't made a lick of difference.

If anyone else would like to chime in and give suggestions i'll be happy to try them.

[soja]

I'm not sure what else to recommend trying while continuing to run Linux, then, if you're already running a properly PV-on-HVM setup and plugin-free server with -threads 1, sv_parallel_sendsnapshot 1, and proper rate settings, etc.

I also don't have a lot of managed Linux clients (whether on a VDS or not) with busy 32-slot servers to compare you to here, to be honest, so I'm not sure what the performance baseline for a server like yours should be. We have a few clients with busy servers on managed Windows VDSes and they have stable tickrates that remain in the 60-70 range, though.

If this was a virtualization-related issue, I was actually expecting that Xen upgrade to help quite a bit. I'd also expect for both Windows and Linux servers to be adversely affected.

Historically, GoldSrc, Source, and Orangebox games have performed much worse on Linux than Windows, but I had thought that Valve improved the situation with their recent updates. It is possible that they haven't.

(You posted the bit about the support request as I was typing this. There's nothing different that I'd say there.)

John, i would be willing to do a stress test on a 32 slot TF2 vanilla server(with sm/mm installed) with my community if you would like on a vds trial on a managed linux install on the 2.9Ghz system.

Let me know if you're interested.

[Edge100x]

As a follow-up to this, soja tested a full 32-slot TF2 server on a managed Linux VDS and it performed at a solid 65+ FPS, so this turned out to not be a Linux-specific or virtualization-specific problem. It's still not clear what is causing Python's dips, but switching to a managed Linux VDS may help him.

[Python]

Do you have:

1. Server.cfg used
2. Command Line Start parameters.


I'd like to test with identical settings. I don't think sd_doomsday was a good map to test, would of made more sense to compair apples to apples (eg. goldrush) since i had stats outputs from there.

Most CP maps run almost 66.7, the payload maps take the largest hit on the server. Anyways id like to try to clone my server settings to be identical to what you tested with. I'll also try running sd_doomsday with my current settings to see how it compares.


Py

[soja]

Server cfg was stock nfo, but we used more intense settings to stress the server as much as possible:

net_splitpacket_maxrate 450000 (450k, anything over 150k is the same usually)
net_maxcleartime .01
sv_parallel_sendsnapshot 1
sv_minrate 100000 (100k)
sv_maxrate 0

sv_minupdaterate 66
sv_maxupdaterate 67
sv_mincmdrate 66
sv_maxcmdrate 67

[soja]

Here is the stock NFO command line as well(maxplayers custom edited by me)

Code: Select all

-game tf -ip 74.91.119.52 +maxplayers 31 -debug -console -nocrashdialog -norestart +host_timer_spin_ms 0

Here is a copy of the server.cfg on our dustbowl server:

Code: Select all

//Default Settings
//Server's Hostname
hostname "[GFLClan.com]*DustBowl*|Classic Respawn|Chicago"
//Server's password
sv_password ""
//Rcon password
rcon_password "supersecretpw"

//bot settings
tf_bot_quota_mode fill
tf_bot_quota 6
tf_bot_join_after_player 1


//Server settings
sv_pure 0
sv_consistency 0
sv_alltalk 1

dynamicslots_threshold 24
dynamicslots_high 30
dynamicslots_low 24

//Populate Settings
//S1(NC)
sv_region 0

// Allow clients to upload customizations files
sv_allowupload 1

// Allow clients to download files
sv_allowdownload 1

//Download Settings
// Download Settings //
// Maximum allowed file size for uploading in MB
net_maxfilesize 64
sv_downloadurl "http://gflfastdl.site.nfoservers.com/fastdl/tf/"

//Server Match Making
tf_server_identity_account_id xxxxxx
tf_server_identity_token "xxxxxxxxxxxxxxx"

//Other Common Convars
//fast respawn ***OFF***
sm_respawn_time_enabled "1"     
sm_respawn_time_blue "5.0"      
sm_respawn_time_red "5.0"

//logging
log 1
sv_logbans 0 
sv_logecho 1 
sv_logfile 1
sv_log_onefile 0
logaddress_delall                       // Removes all previous log destinations
logaddress_add 199.191.57.242:27503        // Sends logs to your HLstatsX CE daemon

//Voting Commands
sv_vote_issue_changelevel_allowed 0
sv_vote_issue_kick_allowed 0
sv_vote_issue_nextlevel_allowed 0
mp_idledealmethod 0
mp_timelimit 1440

sv_minrate 100000
sv_maxrate 0
sv_maxupdaterate 120
sv_minupdaterate 10
sv_max_queries_sec 3
sv_max_queries_sec_global 100000
sv_stats 0
sv_parallel_packentities 0
sv_parallel_sendsnapshot 0
net_splitpacket_maxrate 100000
fps_max 0

exec rates

and our rates.cfg(overwrites nfos settings):

Code: Select all

sv_maxupdaterate 67
sv_minupdaterate 67
sv_maxcmdrate 67
sv_mincmdrate 67
net_maxcleartime .01
net_splitpacket_maxrate 150000
sv_parallel_sendsnapshot 1
sv_parallel_packentities 1