Five Minutes of Bad Lag! - LA Servers?

[TimSad]

Okay, I am posting this to see if anyone else having their servers hosted in LA or perhaps anywhere else is seeing this. Around 9 times within the past week (maybe more) I have seen all 4 of my LA hosted Battlefield 3 servers do this terrible lag all at the same time. It seems to last for a good 5 minutes then finally subsides but does a good job at completely emptying out my servers. It can be described as follows: A rubber-bandy, occasionally freezing for quite some seconds then unfreezing, tugging sort of lag.

This has caused my servers a good deal of grief and I'm kinda hoping that I'm not the only one in this boat here. If you guys just simply own servers in LA but do not play much, maybe you can find out if this has happened by going over your chat logs and see where the word "lag" has been used the most and look up the same times on your other possible LA servers and see if there were many player complaints of lag on those servers too.

I've had a support ticket going with NFO for a couple weeks on this matter but nothing has been found on their end so there is simply nothing they can do about it. I think we've closed out then re-opened the ticket a couple times even since the issue keeps arising. Hah!

I'm honestly not sure what my next step would be to make this pesky 5 minute lag - as I'm calling it - go away. I post this in hopes for a response from any other BF3 server admins that this may be happening to or that may have some valuable input for me. Thanks, all!


Tim (TimSad)

[soja]

Do you run bf3-only on your server? Do you have a minecraft on there? On magnetic drives when minecraft backs up every hour it does a good job of sucking the entire disk up until its done(causing our CS:S servers to freeze simply because it couldn't write logs to disk).

If not this, then can you run a continuous ping or run winMTR while it happens to see if its network related?

the cmd line would be ping 1.2.3.4 -t (the -t flag makes it run until you ctrl+c)

WinMTR can be downloaded here: http://winmtr.net/download-winmtr/

[Edge100x]

Is this happening at specific times of day? If it's during the regular morning maintenance, that could relate. Otherwise, a routing problem may be the most likely cause, either between certain clients and the server or between the server and Blaze. Other possibilities include an application-specific attack against your servers, issue with Blaze, issue with PB, or issue with a 3rd-party tool. Do you have screen shots turned on, or are you having PB connect to an external service?

[TimSad]

It's happening at really random times. I don't have auto PB screenshots turned on but I am streaming with PBBans.com. If this proves to be something that I really can't do anything about, I guess I can accept that. But if it were to start happening let's say multiple times a day, then it would be hard to ignore.

[TimeX]

If you are streaming to PBBans, I would recommend disabling that for a time to see if it helps. That would require disabling of the stream to PBBans, and from them, also. BF3 and other recent EA games don't like receiving a lot of commands in a short period of time, and the streams can cause that even without there being a bandwidth issue.

[.=QUACK=.Major.Pain]

You may see this when multiple people join the server at pretty much the same time.
It can also happen with one person joining, if you have added extra checks to your server for new players joining.

We seen this happen with players joining our Bad Company 2 server when we had it.
We added extra custom checks for joining players, and when you saw a lag spike, we could also see someone joining the server.

Watch if your seeing this same thing. Might give you a clue that it's caused by checking joining players.

[TimSad]

Just happened again and pretty much completely emptied out all of my servers. I had PBBans streaming disabled for about a week and it had still happened a few times. This is bad health for my servers.

[.=QUACK=.Major.Pain]

Guessing your running on a VDS?

[TimSad]

Negative.

[Edge100x]

TimSad, I can't check anything about your servers (or your personal routing) specifically for you here in the forum. I'd be happy to look at any factors that might be visible on our end, such as DDoSes and potential machine problems, but you'd need to submit a support request.

None of your recent complaints have corresponded to general problems in LA -- we have not been having overall problems or complaints specific to that location.

Customers have also brought this thread to our attention, in relation to other crashes they've seen: http://battlelog.battlefield.com/bf3/fo ... 9995733/1/

[TimSad]

I'll keep digging and try to find out what I can about these horrendous patches of lag. It's probably something somewhere along the network that we can't really do anything about though, eh?

[Edge100x]

If the lag is a network problem, it should be visible in MTR output, and we can look at it further. Sometimes such problems can be routed around by InterNAP.

[TimSad]

Ahhh, okay.

[.=QUACK=.Major.Pain]

Just a little something that may or may not apply to you.

Not sure your location and route to the server, but there has been an issue with Level3 having extreme packet loss as traffic is routed through them from east to west coast and vice versa.

We notified them, our host notified them, and our datacenter in Seattle notified them with copies of tracerts and other testing results to show that their datacenter was causing extreme lag with the extreme packet loss.

Their answer was that they were aware of it and that it was going on for some time, and they didn't intend on resolving the issue. Basically telling everyone to just deal with it.

Not sure what else can be done, and not sure if your in the same situation.

[TimSad]

I haven't seen it in quite some time but it has made a return. The same lag is just now occurring throughout all of my servers. Players are gradually leaving the servers... Wish there was a way to find out what is causing this.