Minecraft process killed on VDS

[MarS]

Im running a minecraft server on a single core vps with about 4 people online and it keeps shutting down every 15 minutes or so. Im running ubuntu 10.4 and when i use the /top command to view resource usage, the entry for java shows 87.5%. I was wondering if there was a system in place or something that automatically kills a process that is making the vps use more than its allocated resources. This would explain the random shutdowns with the message "Killed" returned in the console. Has anyone else had this problem?

Thanks

[Edge100x]

We don't have a system on our end, but if your Linux instance detects an OOM (Out Of Memory) condition, it will start killing off processes until it has enough free memory to operate. It sounds like you may be running out of memory and running into this situation.

You can check to see whether this is happening by looking at "dmesg" output or the content of your "/var/log/messages" file. If you confirm that the Java process is using too much memory, you can start your Minecraft server with a different allowed maximum using the "-Xmx" command line option -- "-Xmx768M" would limit it to using 768 MB.

[MarS]

That very much sounds like the problem. So if i limit it to 768 would i be technically starving it since it wants to use more than 1gb? would it then start using disk as memory?

Here is the log produced by dmesg:

[10684.610289] rsyslogd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
[10684.610292] rsyslogd cpuset=/ mems_allowed=0
[10684.610295] Pid: 739, comm: rsyslogd Not tainted 2.6.32-24-server #43-Ubuntu
[10684.610297] Call Trace:
[10684.610303] [<ffffffff810b36bd>] ? cpuset_print_task_mems_allowed+0x9d/0xb0
[10684.610307] [<ffffffff810f6ee4>] oom_kill_process+0xd4/0x2f0
[10684.610310] [<ffffffff810f74a0>] ? select_bad_process+0xd0/0x110
[10684.610312] [<ffffffff810f7538>] __out_of_memory+0x58/0xc0
[10684.610314] [<ffffffff810f76ce>] out_of_memory+0x12e/0x1a0
[10684.610317] [<ffffffff8155aaee>] ? _spin_lock+0xe/0x20
[10684.610320] [<ffffffff810fa751>] __alloc_pages_slowpath+0x511/0x580
[10684.610322] [<ffffffff810fa931>] __alloc_pages_nodemask+0x171/0x180
[10684.610327] [<ffffffff8112d817>] alloc_pages_current+0x87/0xd0
[10684.610329] [<ffffffff810f46b7>] __page_cache_alloc+0x67/0x70
[10684.610332] [<ffffffff810fe289>] __do_page_cache_readahead+0xc9/0x210
[10684.610334] [<ffffffff810fe3f1>] ra_submit+0x21/0x30
[10684.610336] [<ffffffff810f5f7e>] filemap_fault+0x3fe/0x450
[10684.610339] [<ffffffff811127f4>] __do_fault+0x54/0x500
[10684.610342] [<ffffffff81115d38>] handle_mm_fault+0x1a8/0x3c0
[10684.610345] [<ffffffff8155d665>] do_page_fault+0x125/0x3b0
[10684.610347] [<ffffffff8155afb5>] page_fault+0x25/0x30
[10684.610348] Mem-Info:
[10684.610349] Node 0 DMA per-cpu:
[10684.610351] CPU 0: hi: 0, btch: 1 usd: 0
[10684.610353] Node 0 DMA32 per-cpu:
[10684.610354] CPU 0: hi: 186, btch: 31 usd: 65
[10684.610358] active_anon:121531 inactive_anon:121590 isolated_anon:0
[10684.610358] active_file:129 inactive_file:245 isolated_file:0
[10684.610359] unevictable:0 dirty:0 writeback:3 unstable:0
[10684.610360] free:1996 slab_reclaimable:702 slab_unreclaimable:1856
[10684.610360] mapped:19 shmem:59 pagetables:832 bounce:0
[10684.610362] Node 0 DMA free:4032kB min:60kB low:72kB high:88kB active_anon:5916kB inactive_anon:5900kB active_file:16kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15356kB mlocked:0kB dirty:0kB writeback:0kB mapped:8kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:28kB kernel_stack:0kB pagetables:20kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[10684.610369] lowmem_reserve[]: 0 994 994 994
[10684.610371] Node 0 DMA32 free:3952kB min:4000kB low:5000kB high:6000kB active_anon:480208kB inactive_anon:480460kB active_file:500kB inactive_file:980kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1018080kB mlocked:0kB dirty:0kB writeback:12kB mapped:68kB shmem:236kB slab_reclaimable:2808kB slab_unreclaimable:7396kB kernel_stack:1096kB pagetables:3308kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1469 all_unreclaimable? yes
[10684.610378] lowmem_reserve[]: 0 0 0 0
[10684.610380] Node 0 DMA: 2*4kB 6*8kB 9*16kB 8*32kB 8*64kB 4*128kB 2*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 4040kB
[10684.610388] Node 0 DMA32: 476*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3952kB
[10684.610395] 440 total pagecache pages
[10684.610397] 0 pages in swap cache
[10684.610399] Swap cache stats: add 0, delete 0, find 0/0
[10684.610401] Free swap = 0kB
[10684.610402] Total swap = 0kB
[10684.612399] 262144 pages RAM
[10684.612401] 6651 pages reserved
[10684.612402] 551 pages shared
[10684.612403] 252812 pages non-shared
[10684.612406] Out of memory: kill process 768 (avahi-daemon) score 12724 or a child
[10684.616269] Killed process 770 (avahi-daemon)


and from my messages log (just a part, around the time of the crashes)

Jan 16 02:34:43 minecraft kernel: [29957.473153] [<ffffffff810f7538>] __out_of_memory+0x58/0xc0
Jan 16 02:34:43 minecraft kernel: [29957.473156] [<ffffffff810f76ce>] out_of_memory+0x12e/0x1a0
Jan 16 02:34:43 minecraft kernel: [29957.473159] [<ffffffff8155aaee>] ? _spin_lock+0xe/0x20
Jan 16 02:34:43 minecraft kernel: [29957.473161] [<ffffffff810fa751>] __alloc_pages_slowpath+0x511/0x580
Jan 16 02:34:43 minecraft kernel: [29957.473164] [<ffffffff810fa931>] __alloc_pages_nodemask+0x171/0x180
Jan 16 02:34:43 minecraft kernel: [29957.473168] [<ffffffff8112d817>] alloc_pages_current+0x87/0xd0
Jan 16 02:34:43 minecraft kernel: [29957.473170] [<ffffffff810f46b7>] __page_cache_alloc+0x67/0x70
Jan 16 02:34:43 minecraft kernel: [29957.473173] [<ffffffff810fe289>] __do_page_cache_readahead+0xc9/0x210
Jan 16 02:34:43 minecraft kernel: [29957.473176] [<ffffffff810fe3f1>] ra_submit+0x21/0x30
Jan 16 02:34:43 minecraft kernel: [29957.473178] [<ffffffff810f5f7e>] filemap_fault+0x3fe/0x450
Jan 16 02:34:43 minecraft kernel: [29957.473181] [<ffffffff811127f4>] __do_fault+0x54/0x500
Jan 16 02:34:43 minecraft kernel: [29957.473183] [<ffffffff81115d38>] handle_mm_fault+0x1a8/0x3c0
Jan 16 02:34:43 minecraft kernel: [29957.473186] [<ffffffff8155d665>] do_page_fault+0x125/0x3b0
Jan 16 02:34:43 minecraft kernel: [29957.473188] [<ffffffff8155afb5>] page_fault+0x25/0x30
Jan 16 02:34:43 minecraft kernel: [29957.473190] Mem-Info:
Jan 16 02:34:43 minecraft kernel: [29957.473191] Node 0 DMA per-cpu:
Jan 16 02:34:43 minecraft kernel: [29957.473193] CPU 0: hi: 0, btch: 1 usd: 0
Jan 16 02:34:43 minecraft kernel: [29957.473194] Node 0 DMA32 per-cpu:
Jan 16 02:34:43 minecraft kernel: [29957.473196] CPU 0: hi: 186, btch: 31 usd: 134
Jan 16 02:34:43 minecraft kernel: [29957.473200] active_anon:184603 inactive_anon:58427 isolated_anon:0
Jan 16 02:34:43 minecraft kernel: [29957.473201] active_file:118 inactive_file:71 isolated_file:0
Jan 16 02:34:43 minecraft kernel: [29957.473202] unevictable:0 dirty:0 writeback:0 unstable:0
Jan 16 02:34:43 minecraft kernel: [29957.473203] free:2008 slab_reclaimable:882 slab_unreclaimable:1940
Jan 16 02:34:43 minecraft kernel: [29957.473204] mapped:5 shmem:60 pagetables:1041 bounce:0
Jan 16 02:34:43 minecraft kernel: [29957.473207] Node 0 DMA free:4036kB min:60kB low:72kB high:88kB active_anon:9252kB inactive_anon:2560kB active_file:40kB inactive_file:0kB unevictable:0kB isola$
Jan 16 02:34:43 minecraft kernel: [29957.473217] lowmem_reserve[]: 0 994 994 994
Jan 16 02:34:43 minecraft kernel: [29957.473220] Node 0 DMA32 free:3996kB min:4000kB low:5000kB high:6000kB active_anon:729160kB inactive_anon:231148kB active_file:432kB inactive_file:284kB unevic$
Jan 16 02:34:43 minecraft kernel: [29957.473227] lowmem_reserve[]: 0 0 0 0
Jan 16 02:34:43 minecraft kernel: [29957.473229] Node 0 DMA: 9*4kB 8*8kB 6*16kB 4*32kB 2*64kB 2*128kB 5*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 4036kB
Jan 16 02:34:43 minecraft kernel: [29957.473235] Node 0 DMA32: 105*4kB 43*8kB 32*16kB 13*32kB 4*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3996kB
Jan 16 02:34:43 minecraft kernel: [29957.473241] 249 total pagecache pages
Jan 16 02:34:43 minecraft kernel: [29957.473242] 0 pages in swap cache
Jan 16 02:34:43 minecraft kernel: [29957.473244] Swap cache stats: add 0, delete 0, find 0/0
Jan 16 02:34:43 minecraft kernel: [29957.473245] Free swap = 0kB
Jan 16 02:34:43 minecraft kernel: [29957.473246] Total swap = 0kB
Jan 16 02:34:43 minecraft kernel: [29957.475528] 262144 pages RAM
Jan 16 02:34:43 minecraft kernel: [29957.475530] 6651 pages reserved
Jan 16 02:34:43 minecraft kernel: [29957.475531] 454 pages shared
Jan 16 02:34:43 minecraft kernel: [29957.475531] 252862 pages non-shared
Jan 16 04:35:08 minecraft kernel: [37182.287116] java invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
Jan 16 04:35:08 minecraft kernel: [37182.287120] java cpuset=/ mems_allowed=0
Jan 16 04:35:08 minecraft kernel: [37182.287124] Pid: 9153, comm: java Not tainted 2.6.32-24-server #43-Ubuntu
Jan 16 04:35:08 minecraft kernel: [37182.287126] Call Trace:
Jan 16 04:35:08 minecraft kernel: [37182.287133] [<ffffffff810b36bd>] ? cpuset_print_task_mems_allowed+0x9d/0xb0
Jan 16 04:35:08 minecraft kernel: [37182.287138] [<ffffffff810f6ee4>] oom_kill_process+0xd4/0x2f0
Jan 16 04:35:08 minecraft kernel: [37182.287142] [<ffffffff810f74a0>] ? select_bad_process+0xd0/0x110
Jan 16 04:35:08 minecraft kernel: [37182.287145] [<ffffffff810f7538>] __out_of_memory+0x58/0xc0
Jan 16 04:35:08 minecraft kernel: [37182.287148] [<ffffffff810f76ce>] out_of_memory+0x12e/0x1a0

there are several invocations to the oom_killer throughout the log buti didnt copy all. Some say to be invoked by java, others by rsyslogd. Is it safe to assume this IS a memory problem? lol

[Edge100x]

Yes, you are definitely running out of memory. I would recommend adjusting your Minecraft command line as I described as the first step here; this will cause Minecraft to use less memory for its internal caching and also lead Java to run garbage collections more often. What command line are you currently using?

Also, are you using swap (what shows up with a "free -m")? Swap normally allows the OS to avoid OOM conditions, though it can't prevent all of them.

[MarS]

doh... i was wondering why it wasnt using disk as memory before killing stuff.

total used free shared buffers cached
Mem: 998 959 38 0 28 61
-/+ buffers/cache: 868 129
Swap: 0 0 0

as for the command line, im using:

java -Xms768M -Xmx768M -jar craftbukkit.jar nogui

within a "screen -dmS mc"

[MarS]

by the way i just added the 768 after your recommendation, it was at 1024 during the crashes

[Edge100x]

That combination explains it, then.

I'd recommend adding some swap, as well as the change that you've already made. Try commands like this:

Code: Select all

dd if=/dev/zero of=/swap.img bs=10M count=100
mkswap /swap.img
swapon /swap.img

[MarS]

thanks, i did all of that. It now reads:

Mem: 998 984 13 0 27 100
-/+ buffers/cache: 855 142
Swap: 999 0 999


Also, my server hasnt crashed anymore. However my cpu is now spiking a bit so i will need to upgrade to have some overhead i think. Thanks!

[Edge100x]

CPU usage will spike when it has to load chunks from disk, so you are right that an upgrade might help (it would give you more RAM, allowing you to raise the maximum again and allow the server to keep more data loaded). I'd only recommend it if you're noticing problems while playing, though.

[MarS]

Well right now the java machine sits at 82% ram usage, so i think im pushing the limits, especially when my slot max is 20. It funny however how for $15 less i am getting superior performance than my other by the slot provider

[Edge100x]

Each client in-game can use up to 30-something megabytes of memory due to view distance and angle, so you're looking at a tight fit with 768M -- that doesn't leave much left for caching. A RAM upgrade should definitely smooth things out.

I'm glad to hear that it's performing well for you! Many of the per-slot providers don't really know much about how to run Minecraft servers, I suspect, especially the more cookie-cutter TCAdmin-based ones; they don't realize that it has very different requirements than other games. I have been working on customizing our normal system to offer it here, and I'm taking my time because I want to make sure that our servers offer the highest level of performance.

[MarS]

yep thanks for the help, and if you know any tweaks in linux for the jvm then im all ears hehe, im not the most experienced with this OS, but google is my friend.

One last question though if i may..
Ive hosted many many servers on many different providers and ive always noticed that dallas is kinda sketchy on connectivity. I dont know if the area's networks or something are overloaded or what, but i was wondering if you had any insight on this because i was thinking of switching the vps to chicago or somewhere else. I hear the network is nice over there, but maybe its all hearsay. What do you think, would i be better off over there than here?
I used to stick with Dallas since I'm in Tx and ping mattered to me in fps'es, but not in this game

[Edge100x]

You might try different JVMs for Minecraft; that's one small way that it can be optimized. If you upgrade your RAM, you might also try keeping your world on a ramdisk (if it is under a couple hundred MB), as that would make loading and saving it a bit faster.

Our experience with InterNAP in Dallas has been very solid; I have not heard or seen any performance issues there. I'd say that you're very safe in staying .

[MarS]

will do, thank you. By the way feel free to change my thread's title to something more descriptive for future users who might come across the same ram issue Maybe something like "Minecraft process killed on VDS". Again, thank you!

[Edge100x]

Good call -- will do. You're welcome