Did all BF4 servers in the world get rebooted last night?
-
- A semi-regular
- Posts: 15
- https://www.youtube.com/channel/UC40BgXanDqOYoVCYFDSTfHA
- Joined: Tue Sep 11, 2012 6:57 pm
Did all BF4 servers in the world get rebooted last night?
Trying to do some troubleshooting and it appears that all BF4 servers worldwide rebooted last night / early this morning. Gametracker shows correlated dips across the board. Was this the case? Did other admins see their servers restart last night?
Looking at it from the outside it appears that individual servers on different GSPs were rebooted depending on the time zone they were in so the forced restart happened very early in the morning. I noticed this on several European servers and then we also had our US based servers restarted extremely late (early morning). If this is the case and where we have multiple GSPs restarting their clients servers late a night, on the same night, was it that DICE sent out a directive for all GSPs to force restart their clients BF4 servers?
Should this forced restart have been known ahead of time by NFO, in the future could we (the customers renting individual servers) be notified about a mass / forced restart ahead of time via the control panel ?
Looking at it from the outside it appears that individual servers on different GSPs were rebooted depending on the time zone they were in so the forced restart happened very early in the morning. I noticed this on several European servers and then we also had our US based servers restarted extremely late (early morning). If this is the case and where we have multiple GSPs restarting their clients servers late a night, on the same night, was it that DICE sent out a directive for all GSPs to force restart their clients BF4 servers?
Should this forced restart have been known ahead of time by NFO, in the future could we (the customers renting individual servers) be notified about a mass / forced restart ahead of time via the control panel ?
Re: Did all BF4 servers in the world get rebooted last night
If NFO rebooted your server there should be an entry in the "Events Log" tab. Otherwise it could of been a blaze blackout. NFO staff should be able to confirm/deny this.
Not a NFO employee
Re: Did all BF4 servers in the world get rebooted last night
We didn't do any mass restarts. We would also notify customers before doing any type of global or mass change. It is very likely that Blaze just had an issue. Due to the design of Blaze if the server loses connection it will drop all players.
@Kraze^NFo> Juski has a very valid point
@Juski> Got my new signature, thanks!
@Kraze^NFo> Out of context!
@Juski> Doesn't matter!
@Juski> You said I had a valid point! You can't take it back now! It's out there!
@Juski> Got my new signature, thanks!
@Kraze^NFo> Out of context!
@Juski> Doesn't matter!
@Juski> You said I had a valid point! You can't take it back now! It's out there!
Re: Did all BF4 servers in the world get rebooted last night
We've been seeing a issues with blaze lately, far more than normal.
Re: Did all BF4 servers in the world get rebooted last night
Thank you for your replies.
This instance does not appear to be simply a typical blaze event.
I'm curious if it's possible for DICE to send a command out to restart servers?
This server had uptime less than 24 hours, and was restarted.
This server had a similar restart (uptime greater than 24 hours):
And another with a similar effect:
When we've lost connection to Blaze in the past the servers were not rebooted in this manner. I would think we'd be seeing the error message BLAZE_DISCONNECT instead of GAME SERVER CRASH which is indicating that the machine was actually restarted instead of just a blaze disconnect.
This instance does not appear to be simply a typical blaze event.
I'm curious if it's possible for DICE to send a command out to restart servers?
This server had uptime less than 24 hours, and was restarted.
Code: Select all
[03:56:17] [FailLog] OnServerInfo fServerUptime = 17069, serverInfo.ServerUptime = 6
[03:56:17] [FailLog] New map/mode: Siege of Shanghai/Rush
[03:56:17] [MULTIbalancer]:3 DETECTED GAME SERVER CRASH (recorded uptime longer than latest serverInfo uptime)
[03:56:17] [FailLog] Type:GAME_SERVER_RESTART, UTC:20141125_09:56:17, Server:"NFOservers.com Server", Map:Siege of Shanghai, Mode:Rush, Round:1/2, Players:64/0/0, Uptime:04:44:29...... CHICAGO MACHINE
Code: Select all
[05:04:28] [FailLog] OnServerInfo fServerUptime = 291090, serverInfo.ServerUptime = 4
[05:04:28] [FailLog] New map/mode: Siege of Shanghai/Rush
[05:04:28] [MULTIbalancer]:3 DETECTED GAME SERVER CRASH (recorded uptime longer than latest serverInfo uptime)
[05:04:28] [FailLog] Type:GAME_SERVER_RESTART, UTC:20141125_11:04:28, Server:" NFOservers.com Server", Map:Siege of Shanghai, Mode:Rush, Round:1/2, Players:64/3/0, Uptime:3.08:51:30, Details:" DALLAS MACHINE
Code: Select all
[05:05:40] [FailLog] OnServerInfo fServerUptime = 41779, serverInfo.ServerUptime = 18
[05:05:40] [FailLog] New map/mode: Siege of Shanghai/Conquest Large
[05:05:40] [MULTIbalancer]:3 DETECTED GAME SERVER CRASH (recorded uptime longer than latest serverInfo uptime)
[05:05:40] [FailLog] Type:GAME_SERVER_RESTART, UTC:20141125_11:05:40, Server:"NFOservers.com Server", Map:Siege of Shanghai, Mode:Conquest Large, Round:1/1, Players:64/53/0, Uptime:11:36:19, Details: DALLAS MACHINE
Re: Did all BF4 servers in the world get rebooted last night
I would not expect Procon logs to be able to determine a Blaze crash or not. Though, we're more than happy to investigate specific claims. Just open a request asking for us to check.
@Kraze^NFo> Juski has a very valid point
@Juski> Got my new signature, thanks!
@Kraze^NFo> Out of context!
@Juski> Doesn't matter!
@Juski> You said I had a valid point! You can't take it back now! It's out there!
@Juski> Got my new signature, thanks!
@Kraze^NFo> Out of context!
@Juski> Doesn't matter!
@Juski> You said I had a valid point! You can't take it back now! It's out there!
Re: Did all BF4 servers in the world get rebooted last night
The plugin for ProCon, FailLog (at myrcon), checks server population and up time compared to previous polling data. If it sees a big change it guesses what happened. Did 50 people leave in 30 seconds? Blaze. Did the server uptime change from last polling data? Server restart.
This is from our daily restarts. It sees the same result when we were forcing a daily restart. It knew that uptime changed without seeing a mass disconnect, therefor restart not blaze.
I was hoping to see if other admins from different regions saw that their servers were also rebooted last night. Not so much that there is something wrong with a single server which needs investigating at a local level. Server Admins can check by looking at procon to see what the current uptime of your server is. If I'm correct, all of us should have a starting uptime (that is when the machine was last rebooted) of sometime between 0300 and 0500 LOCAL.
I would have to imagine you'd see mass player / connection drops in a traffic during the morning period. Also our servers now share nearly the same uptime so just looking at several machines in different data centers around the world should show the picture. Just looking at GT listing there are very visible drops for the majority of servers at nearly the exact same times. Using default time zone on GT: If your server is in the USA it rebooted at ~5AM, according to the graph on GT. If your server is in the EU it rebooted at ~9PM US time. that is pretty much across the board.
My suspicions are that DICE issued a force reboot as a ninja fix without letting anyone know. That they waited until the night, when traffic would be lowest, for each local server and then forced servers to reboot. Just not sure why they wouldn't let anyone know.
This is from our daily restarts. It sees the same result when we were forcing a daily restart. It knew that uptime changed without seeing a mass disconnect, therefor restart not blaze.
Code: Select all
[18:24:10] [FailLog] OnServerInfo fServerUptime = 87344, serverInfo.ServerUptime = 1
[18:24:10] [FailLog] New map/mode: Operation Whiteout/Rush
[18:24:10] [MULTIbalancer]:3 DETECTED GAME SERVER CRASH (recorded uptime longer than latest serverInfo uptime)
[18:24:10] [FailLog] Type:GAME_SERVER_RESTART, UTC:20141121_00:24:10, Server:" NFOServers.com Some Server", Map:Operation Whiteout, Mode:Rush, Round:1/2, Players:64/50/0, Uptime:1.00:15:44, Details
[18:24:10] [MULTIbalancer]:3 Revalidating all players and teams: Crash Login
I would have to imagine you'd see mass player / connection drops in a traffic during the morning period. Also our servers now share nearly the same uptime so just looking at several machines in different data centers around the world should show the picture. Just looking at GT listing there are very visible drops for the majority of servers at nearly the exact same times. Using default time zone on GT: If your server is in the USA it rebooted at ~5AM, according to the graph on GT. If your server is in the EU it rebooted at ~9PM US time. that is pretty much across the board.
My suspicions are that DICE issued a force reboot as a ninja fix without letting anyone know. That they waited until the night, when traffic would be lowest, for each local server and then forced servers to reboot. Just not sure why they wouldn't let anyone know.
Re: Did all BF4 servers in the world get rebooted last night
Both of our server went down for 30 minutes early this morning and they are on different boxes. Both are showing Machine uptime: 0 days, 16 hours, 35 minutes, 0 seconds and Machine uptime: 0 days, 16 hours, 22 minutes, 52 seconds.
Re: Did all BF4 servers in the world get rebooted last night
Yup. Didn't think to look at machine up time on our box... Good call! Ours is matching up as well. Perhaps globally all servers restarted for Patch Tuesday since they're hosted on Windows machines? And perhaps all important patches would be applied depending on local time that would explain why the machines were all taken down. Guess that means a potential unintended BF4 'feature' is un-related.Ruger wrote: Machine uptime: 0 days, 16 hours, 22 minutes, 52 seconds.
Case closed. // chief wiggum
Re: Did all BF4 servers in the world get rebooted last night
Digging deeper here it appears this was due to Windows updates being applied globally. Not directly an event triggered by us but done at a set time each morning(if applicable). You can see a list of what occurs in the early morning via the events log in your control panel by clicking on the other maintenance link.
@Kraze^NFo> Juski has a very valid point
@Juski> Got my new signature, thanks!
@Kraze^NFo> Out of context!
@Juski> Doesn't matter!
@Juski> You said I had a valid point! You can't take it back now! It's out there!
@Juski> Got my new signature, thanks!
@Kraze^NFo> Out of context!
@Juski> Doesn't matter!
@Juski> You said I had a valid point! You can't take it back now! It's out there!
Re: Did all BF4 servers in the world get rebooted last night
Yes, the reboots on Tuesday were due to Windows patches being applied. Some more machines will likely reboot on Wednesday morning, as often Windows requires multiple reboots to apply multiple sets of patches (annoyingly).