Running AA3 Issues

[.=QUACK=.Major.Pain]

We have an AA3 server running on our VDS, and we seem to see lag spikes at times. Looking at our processes, it only runs at 37-39% CPU.

Should I run the process at High or Realtime?
This is our primary game.

Also, restarting the VDS, will this set the process back to Normal?

Any other ideas to improve the game and reduce the lag.

Currently we are running on a 3-core VDS package with 9 servers.

I guess the next question would be, since 9 is the upper limit, does this still apply if all the other game servers have no one on them?

[kraze]

Well even though the other servers have no one in it they are still there. They do take up a little bit of usage but not a lot. Setting the process to high would help a little. After restarting the VDS everything is going to get reset as far as that. There are some scripts and what not you can use telling it to start in high priority if needed.

[Edge100x]

Is your AA3 server using an entire core itself, at times? To see, look under the "Processes" page of the task manager.

[.=QUACK=.Major.Pain]

Not sure how to tell.

Looking at the processes page, it says:

AA3game.exe*32
CPU 37-38%
Memory 590,000K

Empty server.


Restarted the game server

AA3game.exe*32
CPU 4-5%
Memory 230,000K

Server is currently empty.


How do I check if it's using the entire core?

[Edge100x]

You'd need to look at the CPU and memory usage when the server has players in it who are experiencing the issue, and watch it over time to see if it changes.

I haven't played around with AA3 much, but 37% CPU and 590 MB of memory when empty seems like quite a lot.

[.=QUACK=.Major.Pain]

It was poorly made and a resources hog.

I don't know why it sat so high.
That's what it was last night too with about 10 people on the server.
After they left, it stayed at those numbers.

Restarting the game server, it dropped right down.

[.=QUACK=.Major.Pain]

We have been trying the server again tonight, and seen some high pings and lag/skipping.

With 8 players on server:
CPU 4-6% before round starts
CPU 20-22% during game play
CPU 34-37% spikes for whatever reason

Other servers total 10-15% CPU
We dropped from 9 server to 7 to test the difference.
Gonna shut a few more off and test it.

memory 167,000 - 171,000 range

Pings seem to be 38-70 with 3-4 people
68-156 5-6 people
78-290 7-8 people

Our guys mentioned lag spikes and skipping.

I think we may upgrade to the 4 core package, but thought I would see what you think first.

[kraze]

I know of an old issue with AA3 where I believe after 16 slots the game just was coded so bad that it just could not send and receive data properly, causing for bad game play for users. With a more recent up date I think they made you able to have a good 20.

Back in October "Network optimizations to allow up to 20+ players depending on the server’s configuration. "

Are any of your servers by chance bigger then 16-20, and are they full when seeing issues?

Try to get the server going good again and take some more notes. Mark how many people exactly are in the server and what there average ping,cpu,ram is. Then wait till you have 4 more and do it again. After getting some data you will be able to examine it and see when and where the problem starts. Also keep an eye on certain events in the game. Start of the round,end of the round? Right after a new map or before? Making a detailed list like this will truly help you to understand things better.

Another option you can try right off the bat is to run the AA3 server in high priority.

[.=QUACK=.Major.Pain]

Already running at high priority.
I found errors coming up in the logs, and this is when the lag spikes.
The more consecutive error, the longer the lag spike.

I turned off more game servers and now running AA3, TS3 and L4D2 and things appear to run better.

Upgrade looks necessary.

[kraze]

That may be the easiest route to go at this point.

[TimeX]

Out of curiosity, what errors were you seeing in the logs around the time of the spikes?

[.=QUACK=.Major.Pain]

ScriptWarning: AA_DroppedObjective Alley_day_cloudy.TheWorld:PersistentLevel.AA_DroppedObjective_4 (Function AA3Baseline.AA_DroppedObjective:GiveTo:00C6) Accessed None 'PlayerReplicationInfo'
ScriptWarning: AA_DroppedObjective Alley_day_cloudy.TheWorld:PersistentLevel.AA_DroppedObjective_4 (Function AA3Baseline.AA_DroppedObjective:GiveTo:00CF) Accessed None
ScriptWarning: AA_DroppedObjective Alley_day_cloudy.TheWorld:PersistentLevel.AA_DroppedObjective_4 (Function AA3Baseline.AA_DroppedObjective:GiveTo:0161) Accessed None 'PlayerReplicationInfo'
ScriptWarning: AA_DroppedObjective Alley_day_cloudy.TheWorld:PersistentLevel.AA_DroppedObjective_4 (Function AA3Baseline.AA_DroppedObjective:GiveTo:016A) Accessed None
ScriptWarning: AA_DroppedObjective Alley_day_cloudy.TheWorld:PersistentLevel.AA_DroppedObjective_4 (Function AA3Baseline.AA_DroppedObjective:GiveTo:01B4) Accessed None 'Controller'
ScriptWarning: AA_DroppedObjective Alley_day_cloudy.TheWorld:PersistentLevel.AA_DroppedObjective_4 (Function AA3Baseline.AA_DroppedObjective:GiveTo:00C6) Accessed None 'PlayerReplicationInfo'
ScriptWarning: AA_DroppedObjective Alley_day_cloudy.TheWorld:PersistentLevel.AA_DroppedObjective_4 (Function AA3Baseline.AA_DroppedObjective:GiveTo:00CF) Accessed None
ScriptWarning: AA_DroppedObjective Alley_day_cloudy.TheWorld:PersistentLevel.AA_DroppedObjective_4 (Function AA3Baseline.AA_DroppedObjective:GiveTo:0161) Accessed None 'PlayerReplicationInfo'
ScriptWarning: AA_DroppedObjective Alley_day_cloudy.TheWorld:PersistentLevel.AA_DroppedObjective_4 (Function AA3Baseline.AA_DroppedObjective:GiveTo:016A) Accessed None
ScriptWarning: AA_DroppedObjective Alley_day_cloudy.TheWorld:PersistentLevel.AA_DroppedObjective_4 (Function AA3Baseline.AA_DroppedObjective:GiveTo:01B4) Accessed None 'Controller'
ScriptWarning: AA_DroppedObjective Alley_day_cloudy.TheWorld:PersistentLevel.AA_DroppedObjective_4 (Function AA3Baseline.AA_DroppedObjective:GiveTo:00C6) Accessed None 'PlayerReplicationInfo'
ScriptWarning: AA_DroppedObjective Alley_day_cloudy.TheWorld:PersistentLevel.AA_DroppedObjective_4 (Function AA3Baseline.AA_DroppedObjective:GiveTo:00CF) Accessed None
ScriptWarning: AA_DroppedObjective Alley_day_cloudy.TheWorld:PersistentLevel.AA_DroppedObjective_4 (Function AA3Baseline.AA_DroppedObjective:GiveTo:0161) Accessed None 'PlayerReplicationInfo'
ScriptWarning: AA_DroppedObjective Alley_day_cloudy.TheWorld:PersistentLevel.AA_DroppedObjective_4 (Function AA3Baseline.AA_DroppedObjective:GiveTo:016A) Accessed None
ScriptWarning: AA_DroppedObjective Alley_day_cloudy.TheWorld:PersistentLevel.AA_DroppedObjective_4 (Function AA3Baseline.AA_DroppedObjective:GiveTo:01B4) Accessed None 'Controller'
ScriptWarning: AA_DroppedObjective Alley_day_cloudy.TheWorld:PersistentLevel.AA_DroppedObjective_4 (Function AA3Baseline.AA_DroppedObjective:GiveTo:00C6) Accessed None 'PlayerReplicationInfo'
ScriptWarning: AA_DroppedObjective Alley_day_cloudy.TheWorld:PersistentLevel.AA_DroppedObjective_4 (Function AA3Baseline.AA_DroppedObjective:GiveTo:00CF) Accessed None
ScriptWarning: AA_DroppedObjective Alley_day_cloudy.TheWorld:PersistentLevel.AA_DroppedObjective_4 (Function AA3Baseline.AA_DroppedObjective:GiveTo:0161) Accessed None 'PlayerReplicationInfo'
ScriptWarning: AA_DroppedObjective Alley_day_cloudy.TheWorld:PersistentLevel.AA_DroppedObjective_4 (Function AA3Baseline.AA_DroppedObjective:GiveTo:016A) Accessed None
ScriptWarning: AA_DroppedObjective Alley_day_cloudy.TheWorld:PersistentLevel.AA_DroppedObjective_4 (Function AA3Baseline.AA_DroppedObjective:GiveTo:01B4) Accessed None 'Controller'
ScriptWarning: AA_Objective_Briefcase_US Alley_day_cloudy.TheWorld:PersistentLevel.AA_Objective_Briefcase_US_53 (Function AA3Baseline.AA_Objective_Briefcase_US:AttachToActor:00CF) Accessed None 'Materials'


ScriptWarning: AA_MCExtractionObjectiveMarker Alley_inf_vip_large_dev.TheWorld:PersistentLevel.AA_MCExtractionObjectiveMarker_0 (Function AA3Baseline.AA_MCObjectiveMarker:SetObjectiveCompletedPawn:0057) Accessed None 'aapCompletedBy'
ScriptWarning: AA_MCExtractionObjectiveMarker Alley_inf_vip_large_dev.TheWorld:PersistentLevel.AA_MCExtractionObjectiveMarker_0 (Function AA3Baseline.AA_MCObjectiveMarker:SetObjectiveCompletedPawn:0060) Accessed None


ScriptWarning: AA_PickupFactory_Objective Alley_inf_co_large_dev.TheWorld:PersistentLevel.AA_PickupFactory_Objective_0 (Function AA3Baseline.AA_PickupFactory_Objective:NotifyUnHideSAIIcon:0007) Accessed None 'MyObjectiveMarker'
ScriptWarning: AA_PickupFactory_Objective Alley_inf_co_large_dev.TheWorld:PersistentLevel.AA_PickupFactory_Objective_0 (Function AA3Baseline.AA_PickupFactory_Objective:InitializePickup:015E) Accessed None 'MyObjectiveMarker'


PerfWarning: InstanceGeom: 8.862622 ms - 936 Convex, Terrain 8.874875 ms, PerTriSM 4.759550 ms, InitArticulated 0.000000 ms, BodyInst Alloc 17.788924 ms, InitBody 93.523778 ms
PerfWarning: InitBody Avg 1.#INF00 ms per frame over 0 frames
PerfWarning: NxActor Creation: 46.457138 ms


Warning: Failed to load 'Class None.': Failed to find object 'Class None.'


Warning: Failed to find streaming level object associated with 'SeqAct_MultiLevelStreaming Alley_day_cloudy.TheWorld:PersistentLevel.Main_Sequence.SeqAct_MultiLevelStreaming_0'


ScriptLog: INFORMATION: Using first valid group found, not percentage due to blocking of spawn group by other team
Warning: Obj AA_MCFireteamSpawnMarker_6 has no handler for AA_MCFireteamSpawnGroupManagerSeqAct_1
Warning: Obj AA_MCFireteamSpawnMarker_5 has no handler for AA_MCFireteamSpawnGroupManagerSeqAct_1
Warning: Obj AA_MCFireteamSpawnMarker_4 has no handler for AA_MCFireteamSpawnGroupManagerSeqAct_1
ScriptLog: INFORMATION: Preparing to start match.
ScriptLog: INFORMATION: Using first valid group found, not percentage due to blocking of spawn group by other team
Warning: Obj AA_MCFireteamSpawnMarker_0 has no handler for AA_MCFireteamSpawnGroupManagerSeqAct_0
Warning: Obj AA_MCFireteamSpawnMarker_1 has no handler for AA_MCFireteamSpawnGroupManagerSeqAct_0
Warning: Obj AA_MCFireteamSpawnMarker_3 has no handler for AA_MCFireteamSpawnGroupManagerSeqAct_0


ScriptWarning: AA_BulletWeapon_M4A1 Alley_day_cloudy.TheWorld:PersistentLevel.AA_BulletWeapon_M4A1_6 (Function AA3Baseline.AA_BulletWeapon:CalcWeaponFire:026A) Accessed array 'AA_BulletWeapon_M4A1_6.HitRoundType' out of bounds (0/0)
ScriptWarning: AA_BulletWeapon_M4A1 Alley_day_cloudy.TheWorld:PersistentLevel.AA_BulletWeapon_M4A1_6 (Function AA3Baseline.AA_BulletWeapon:CalcWeaponFire:026A) Accessed null class context 'HitRoundType'
ScriptWarning: AA_BulletWeapon_M4A1 Alley_day_cloudy.TheWorld:PersistentLevel.AA_BulletWeapon_M4A1_6 (Function AA3Baseline.AA_BulletWeapon:CalcWeaponFire:0273) Accessed null class context
ScriptWarning: AA_BulletWeapon_M4A1 Alley_day_cloudy.TheWorld:PersistentLevel.AA_BulletWeapon_M4A1_6 (Function AA3Baseline.AA_BulletWeapon:CalcWeaponFire:026A) Accessed array 'AA_BulletWeapon_M4A1_6.HitRoundType' out of bounds (0/0)
ScriptWarning: AA_BulletWeapon_M4A1 Alley_day_cloudy.TheWorld:PersistentLevel.AA_BulletWeapon_M4A1_6 (Function AA3Baseline.AA_BulletWeapon:CalcWeaponFire:026A) Accessed null class context 'HitRoundType'
ScriptWarning: AA_BulletWeapon_M4A1 Alley_day_cloudy.TheWorld:PersistentLevel.AA_BulletWeapon_M4A1_6 (Function AA3Baseline.AA_BulletWeapon:CalcWeaponFire:0273) Accessed null class context


I posted on the AA forum, and some of our guys are beta testers, so I asked them to check with the dev and beta forum.

[kraze]

I am not 100% sure to be honest. I do see some warnings in there. However I know a lot of games that spew warnings and so on and it means nothing. I would wait and see what the devs have to say. Also for a faster fix just shutoff more servers or upgrade.

[.=QUACK=.Major.Pain]

Well we have been running the server tonight with only AA3, L4D2, TF2 and TS3, and there were 15-17 people on the server and it was running great. The CPU was from 6-38 at times, and memory at 213,000.

At 10:30pm est, the pings went up to 300-400 and stayed there for about 5 minutes.
The CPU and memory were still in the same range, so something had to be up with the internet connection.

Not sure what the server internet speed is supposed to be, but speedtest says it's 27Mbps down and 39Mbps up.

[kraze]

Our connection except for Denver are 1 gbps, and if you are in Dallas or Chicago it is connected to a 10gbps backbone.

However speed test sites are not always the best to display results because most are not capable of displaying the speeds we offer.

Also testing the connection with nothing running would be a good idea.