Did all BF4 servers in the world get rebooted last night?

Post Reply
dyno
A semi-regular
A semi-regular
Posts: 15
Joined: Tue Sep 11, 2012 6:57 pm

Did all BF4 servers in the world get rebooted last night?

Post by dyno » Tue Nov 25, 2014 4:55 pm

Trying to do some troubleshooting and it appears that all BF4 servers worldwide rebooted last night / early this morning. Gametracker shows correlated dips across the board. Was this the case? Did other admins see their servers restart last night?

Looking at it from the outside it appears that individual servers on different GSPs were rebooted depending on the time zone they were in so the forced restart happened very early in the morning. I noticed this on several European servers and then we also had our US based servers restarted extremely late (early morning). If this is the case and where we have multiple GSPs restarting their clients servers late a night, on the same night, was it that DICE sent out a directive for all GSPs to force restart their clients BF4 servers?

Should this forced restart have been known ahead of time by NFO, in the future could we (the customers renting individual servers) be notified about a mass / forced restart ahead of time via the control panel ?

User avatar
soja
This is my homepage
This is my homepage
Posts: 2389
Joined: Fri May 18, 2012 3:20 pm

Re: Did all BF4 servers in the world get rebooted last night

Post by soja » Tue Nov 25, 2014 5:05 pm

If NFO rebooted your server there should be an entry in the "Events Log" tab. Otherwise it could of been a blaze blackout. NFO staff should be able to confirm/deny this.
Not a NFO employee

User avatar
kraze
Staff
Staff
Posts: 4357
Joined: Fri Sep 17, 2010 9:06 am
Location: California

Re: Did all BF4 servers in the world get rebooted last night

Post by kraze » Tue Nov 25, 2014 5:10 pm

We didn't do any mass restarts. We would also notify customers before doing any type of global or mass change. It is very likely that Blaze just had an issue. Due to the design of Blaze if the server loses connection it will drop all players.
@Kraze^NFo> Juski has a very valid point
@Juski> Got my new signature, thanks!
@Kraze^NFo> Out of context!
@Juski> Doesn't matter!
@Juski> You said I had a valid point! You can't take it back now! It's out there!

barhund
This is my homepage
This is my homepage
Posts: 124
Joined: Fri May 16, 2014 7:37 am

Re: Did all BF4 servers in the world get rebooted last night

Post by barhund » Tue Nov 25, 2014 5:37 pm

We've been seeing a issues with blaze lately, far more than normal.
Image

dyno
A semi-regular
A semi-regular
Posts: 15
Joined: Tue Sep 11, 2012 6:57 pm

Re: Did all BF4 servers in the world get rebooted last night

Post by dyno » Tue Nov 25, 2014 6:15 pm

Thank you for your replies.

This instance does not appear to be simply a typical blaze event.
I'm curious if it's possible for DICE to send a command out to restart servers?

This server had uptime less than 24 hours, and was restarted.

Code: Select all

[03:56:17] [FailLog] OnServerInfo fServerUptime = 17069, serverInfo.ServerUptime = 6
[03:56:17] [FailLog] New map/mode: Siege of Shanghai/Rush
[03:56:17] [MULTIbalancer]:3 DETECTED GAME SERVER CRASH (recorded uptime longer than latest serverInfo uptime)
[03:56:17] [FailLog] Type:GAME_SERVER_RESTART, UTC:20141125_09:56:17, Server:"NFOservers.com Server", Map:Siege of Shanghai, Mode:Rush, Round:1/2, Players:64/0/0, Uptime:04:44:29...... CHICAGO MACHINE
This server had a similar restart (uptime greater than 24 hours):

Code: Select all

[05:04:28] [FailLog] OnServerInfo fServerUptime = 291090, serverInfo.ServerUptime = 4
[05:04:28] [FailLog] New map/mode: Siege of Shanghai/Rush
[05:04:28] [MULTIbalancer]:3 DETECTED GAME SERVER CRASH (recorded uptime longer than latest serverInfo uptime)
[05:04:28] [FailLog] Type:GAME_SERVER_RESTART, UTC:20141125_11:04:28, Server:" NFOservers.com Server", Map:Siege of Shanghai, Mode:Rush, Round:1/2, Players:64/3/0, Uptime:3.08:51:30, Details:" DALLAS MACHINE
And another with a similar effect:

Code: Select all

[05:05:40] [FailLog] OnServerInfo fServerUptime = 41779, serverInfo.ServerUptime = 18
[05:05:40] [FailLog] New map/mode: Siege of Shanghai/Conquest Large
[05:05:40] [MULTIbalancer]:3 DETECTED GAME SERVER CRASH (recorded uptime longer than latest serverInfo uptime)
[05:05:40] [FailLog] Type:GAME_SERVER_RESTART, UTC:20141125_11:05:40, Server:"NFOservers.com Server", Map:Siege of Shanghai, Mode:Conquest Large, Round:1/1, Players:64/53/0, Uptime:11:36:19, Details: DALLAS MACHINE
When we've lost connection to Blaze in the past the servers were not rebooted in this manner. I would think we'd be seeing the error message BLAZE_DISCONNECT instead of GAME SERVER CRASH which is indicating that the machine was actually restarted instead of just a blaze disconnect.

User avatar
kraze
Staff
Staff
Posts: 4357
Joined: Fri Sep 17, 2010 9:06 am
Location: California

Re: Did all BF4 servers in the world get rebooted last night

Post by kraze » Tue Nov 25, 2014 6:26 pm

I would not expect Procon logs to be able to determine a Blaze crash or not. Though, we're more than happy to investigate specific claims. Just open a request asking for us to check.
@Kraze^NFo> Juski has a very valid point
@Juski> Got my new signature, thanks!
@Kraze^NFo> Out of context!
@Juski> Doesn't matter!
@Juski> You said I had a valid point! You can't take it back now! It's out there!

dyno
A semi-regular
A semi-regular
Posts: 15
Joined: Tue Sep 11, 2012 6:57 pm

Re: Did all BF4 servers in the world get rebooted last night

Post by dyno » Tue Nov 25, 2014 6:52 pm

The plugin for ProCon, FailLog (at myrcon), checks server population and up time compared to previous polling data. If it sees a big change it guesses what happened. Did 50 people leave in 30 seconds? Blaze. Did the server uptime change from last polling data? Server restart.

This is from our daily restarts. It sees the same result when we were forcing a daily restart. It knew that uptime changed without seeing a mass disconnect, therefor restart not blaze.

Code: Select all

[18:24:10] [FailLog] OnServerInfo fServerUptime = 87344, serverInfo.ServerUptime = 1
[18:24:10] [FailLog] New map/mode: Operation Whiteout/Rush
[18:24:10] [MULTIbalancer]:3 DETECTED GAME SERVER CRASH (recorded uptime longer than latest serverInfo uptime)
[18:24:10] [FailLog] Type:GAME_SERVER_RESTART, UTC:20141121_00:24:10, Server:" NFOServers.com Some Server", Map:Operation Whiteout, Mode:Rush, Round:1/2, Players:64/50/0, Uptime:1.00:15:44, Details
[18:24:10] [MULTIbalancer]:3 Revalidating all players and teams: Crash Login 
I was hoping to see if other admins from different regions saw that their servers were also rebooted last night. Not so much that there is something wrong with a single server which needs investigating at a local level. Server Admins can check by looking at procon to see what the current uptime of your server is. If I'm correct, all of us should have a starting uptime (that is when the machine was last rebooted) of sometime between 0300 and 0500 LOCAL.

I would have to imagine you'd see mass player / connection drops in a traffic during the morning period. Also our servers now share nearly the same uptime so just looking at several machines in different data centers around the world should show the picture. Just looking at GT listing there are very visible drops for the majority of servers at nearly the exact same times. Using default time zone on GT: If your server is in the USA it rebooted at ~5AM, according to the graph on GT. If your server is in the EU it rebooted at ~9PM US time. that is pretty much across the board.

My suspicions are that DICE issued a force reboot as a ninja fix without letting anyone know. That they waited until the night, when traffic would be lowest, for each local server and then forced servers to reboot. Just not sure why they wouldn't let anyone know.

User avatar
Ruger
A regular
A regular
Posts: 35
Joined: Mon Oct 14, 2013 10:53 am
Location: Texas
Contact:

Re: Did all BF4 servers in the world get rebooted last night

Post by Ruger » Tue Nov 25, 2014 7:39 pm

Both of our server went down for 30 minutes early this morning and they are on different boxes. Both are showing Machine uptime: 0 days, 16 hours, 35 minutes, 0 seconds and Machine uptime: 0 days, 16 hours, 22 minutes, 52 seconds.

dyno
A semi-regular
A semi-regular
Posts: 15
Joined: Tue Sep 11, 2012 6:57 pm

Re: Did all BF4 servers in the world get rebooted last night

Post by dyno » Tue Nov 25, 2014 9:46 pm

Ruger wrote: Machine uptime: 0 days, 16 hours, 22 minutes, 52 seconds.
Yup. Didn't think to look at machine up time on our box... Good call! Ours is matching up as well. Perhaps globally all servers restarted for Patch Tuesday since they're hosted on Windows machines? And perhaps all important patches would be applied depending on local time that would explain why the machines were all taken down. Guess that means a potential unintended BF4 'feature' is un-related.

Case closed. // chief wiggum

User avatar
kraze
Staff
Staff
Posts: 4357
Joined: Fri Sep 17, 2010 9:06 am
Location: California

Re: Did all BF4 servers in the world get rebooted last night

Post by kraze » Tue Nov 25, 2014 9:58 pm

Digging deeper here it appears this was due to Windows updates being applied globally. Not directly an event triggered by us but done at a set time each morning(if applicable). You can see a list of what occurs in the early morning via the events log in your control panel by clicking on the other maintenance link.
@Kraze^NFo> Juski has a very valid point
@Juski> Got my new signature, thanks!
@Kraze^NFo> Out of context!
@Juski> Doesn't matter!
@Juski> You said I had a valid point! You can't take it back now! It's out there!

User avatar
Edge100x
Founder
Founder
Posts: 12219
Joined: Thu Apr 18, 2002 11:04 pm
Location: Seattle
Contact:

Re: Did all BF4 servers in the world get rebooted last night

Post by Edge100x » Tue Nov 25, 2014 11:28 pm

Yes, the reboots on Tuesday were due to Windows patches being applied. Some more machines will likely reboot on Wednesday morning, as often Windows requires multiple reboots to apply multiple sets of patches (annoyingly).

Post Reply