Arma3 Major Performance Drops

[dburke87]

I'm running several (two live and a test server) ArmA3 servers off a full-box dedi, and we bounce between having a total of 30 players on to above 110. Up until I made some changes recently, the server performed fine, even with one server at 80/80, and the other at ~40/80. Now, I'm seeing some very strange and sudden drops in performance. The server will almost always recover from this, but where our servers used to trend in a sort of sloping straight line, now its FPS/CPS flow in peaks and valleys.

I'm attaching a picture of the server monitor I use, and am also providing my arma.rpt file. I'm finding some strange messages in it, such as:

" found in (408,297)."
"20:44:11 Message not sent - error 0, message ID = ffffffff, to 1014259370 (James)"
"20:44:11 NetServer::SendMsg: cannot find channel #1014259370, users.card=37"
"Server: Object 16:65 not found (message 112)"

The object not found messages have always occurred, but now I'm seeing even more of them. This particular case occurred with around 40 players, where we used to have no such errors at 80/80 players. Any guidance on what some of these messages mean and what would cause them would be greatly appreciated.

[kraze]

What does CPU/ram usage look like on the machine? Do you notice these performance issues when the servers start filling up or does it happen randomly?

[dburke87]

RAM usage has never been an issue, even close. CPU usage looks normal to me. In other words, CPU usage sits right where it always has, even when we had no dips in performance. I assign the first server to cores 0-3, and the second to cores 4-7. The first core in each block sits around 50-75% usage at higher capacity, and the others sit between 0-35%. Overall usage is around 50%.

It also does occur randomly, whether we're at 20 players or 80. I mapped out blocks in 1 minute intervals, 5 minutes before a dip in performance, during the dip, and 5 minutes after, and logged every function call during this period. Nothing stood out except for those "object not found" and "found in" messages.

I also noticed somewhat severe spikes in bandwidth usage during these dips in performance. We usually sit at ~10-20 Mbps, and I was seeing spikes of up to 250Mbps. This was just pointed out to me right after the first post in this thread. DDoS attacks?

[dburke87]

Maybe this is too much information, but this is my log of all server functions. Blocks are before, during, and after, corresponding to the dip in performance. NFC represents the "Client: Object not found messages" and NFS is the same but for the server.

[kraze]

It is very possible you are seeing that. There is no other reason you should see spikes like that. Next time it happens run a WireShark capture and post the results the pastebin and then here with a link.

I also don't recommend you assign affinities to servers. Windows does a decent job of bouncing processes around cores and when you interrupt that it can cause issues, especially in your case where pure raw power isn't likely your issue.

[dburke87]

I actually have a few saved WireShark captures already form these "attacks". I'll post a pastebin as soon as I can. Unfortunately, I made a rookie mistake when trying to block a certain IP. I turned on Windows Firewall before applying any rules. Now I can't even remote in to my server... Fun times.

[Caliban55]

As to the "object not found" messages: There can various causes for them, some examples are ammo objects, projectiles, etc.. I don't know of any reference available to sort them out, except for BIS themselves. They are usually not critical, or influence the server performance and can also be just notifications that the engine uses.

Point is, you can't really do anything about this (not much at least as long as you don't want to examine each code structure), so my recommendation is to ignore them.

I have seen an increase with the latest 1.14 version also, but as I said, it is not critical.

"Message not sent": Can mean that the client disconnected already when the server tried to sent a particular string, his synchronization is not correct, the player object could not be determined correctly, more messages where sent in a particular server cylcle than the server could handle and the send part has been dropped, etc. .

As to the spikes: Can be DDos, or players creating high number of objects at the same time, or players hacking and mass spawning in objects. Take a look in your BattlEye filters and limit those calls.

[dburke87]

Yeah, I hate those pesky messages even though they may not represent any problems, however, the "found in xxx" messages are new to my reports. I did check through BattlEye filters, and saw no signs of publicVariable tampering, mass spawning of objects, or foul play of any kind. If not DDoS, could it be an issue with JIP players and the server broadcasting a large amount of information to them? Maybe publicVariables, setVariable with global flagged as true, etc.? Thanks for all the replies.

[dburke87]

I have also isolated a correlation between the sudden, massive performance drops and the spikes in network traffic. The spikes again can reach between 100Mbps to 250Mbps, but cannot be narrowed down to a single source IP. Whether completely unlikely or not, is it at all possible that these performance drops and network traffic spikes are related to a JIP player joining, or does this almost absolutely point to some form of a DDoS attack?

[Caliban55]

I also noticed the "found ..." since the latest version, but as there is not much to do about this, just ignore .

JIP clients and massive ammount of network send variables can cause an issue. This will depend though on how many players connect over a certain time intervall. For example, if you have 30 clients joining within 5 seconds, every publicVariable value and globally flagged variable has to be synced, together with the weather now; on top of this, clients who do not have the mission downloaded yet will also allocate resources. You could try moving those code completley on the client side and not on the server.

Especially with the MP framework, the persistent option should be used very carefully (try to avoid it in general, there are probaly better work-arounds depending on the situation).

BEC had a join queu system a few versions back, but that got disabled I think; with this you could have tested limiting the number of clients who can join over a certain intervall.