Memory and CPU usage issues

[automatik]

I have a 2x Core VDS and am experiencing a few issues.

After about 12 hours or so no matter if there has been players on the server or not I end up with 90%+ CPU usage and 1.9gb of 2gb used and I have to clear the memory caches to watch it slowly climb back up after a while.

I am running the NFO Gentoo distro.

2x 32 slot DOD:S Pubs both at tic 0 and fpsmax 0

[Edge100x]

Could you post your output from "free -m" and "top" when you are seeing this? It is easy to misread these, and I suspect that you might be accidentally including cache in your estimate of available memory.

[automatik]

well its slowly climbing back up but here it is with 1 32slot pub running with 12 slots used.

total used free shared buffers cached
Mem: 2024 940 1084 0 15 160
-/+ buffers/cache: 764 1260
Swap: 1023 2 1021


Tasks: 52 total, 1 running, 51 sleeping, 0 stopped, 0 zombie
Cpu(s): 6.8%us, 0.8%sy, 0.0%ni, 86.3%id, 0.0%wa, 2.2%hi, 3.8%si, 0.0%st
Mem: 2073564k total, 976128k used, 1097436k free, 15620k buffers
Swap: 1048568k total, 2120k used, 1046448k free, 164844k cached

[automatik]

I also use 'htop' as my primary for viewing cpu and mem usage

[Edge100x]

It looks like you have 1260 MB of RAM available and your CPU usage is 13.7% right now, then, which is not bad, although the RAM usage does seem a little high (hopefully just because you're running a lot of plugins -- without the rest of "top", I can't say for certain where it's being used).

[automatik]

top - 02:09:58 up 3 days, 9:33, 1 user, load average: 0.11, 0.03, 0.01
Tasks: 52 total, 1 running, 51 sleeping, 0 stopped, 0 zombie
Cpu(s): 44.0%us, 0.3%sy, 0.0%ni, 43.8%id, 0.0%wa, 6.2%hi, 5.7%si, 0.0%st
Mem: 2073564k total, 989148k used, 1084416k free, 16332k buffers
Swap: 1048568k total, 2120k used, 1046448k free, 171980k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
29040 srv101 -21 0 879m 781m 21m S 99 38.6 197:40.10 srcds_i486
1 root 20 0 1716 552 528 S 0 0.0 0:01.55 init
2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root RT -5 0 0 0 S 0 0.0 0:00.09 migration/0
4 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0
5 root RT -5 0 0 0 S 0 0.0 0:00.06 migration/1
6 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1
7 root 15 -5 0 0 0 S 0 0.0 0:00.02 events/0
8 root 15 -5 0 0 0 S 0 0.0 0:00.00 events/1
9 root 15 -5 0 0 0 S 0 0.0 0:00.08 khelper
14 root 15 -5 0 0 0 S 0 0.0 0:00.00 async/mgr
122 root 15 -5 0 0 0 S 0 0.0 0:00.01 kblockd/0
123 root 15 -5 0 0 0 S 0 0.0 0:00.22 kblockd/1
125 root 15 -5 0 0 0 S 0 0.0 0:00.00 kacpid
126 root 15 -5 0 0 0 S 0 0.0 0:00.00 kacpi_notify
127 root 15 -5 0 0 0 S 0 0.0 0:00.00 kacpi_hotplug
173 root 15 -5 0 0 0 S 0 0.0 0:00.00 ata/0
174 root 15 -5 0 0 0 S 0 0.0 0:00.01 ata/1
175 root 15 -5 0 0 0 S 0 0.0 0:00.00 ata_aux
179 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksuspend_usbd
183 root 15 -5 0 0 0 S 0 0.0 0:00.00 khubd
186 root 15 -5 0 0 0 S 0 0.0 0:00.00 kseriod
250 root 20 0 0 0 0 S 0 0.0 0:00.00 pdflush
251 root 20 0 0 0 0 S 0 0.0 0:00.08 pdflush
252 root 15 -5 0 0 0 S 0 0.0 0:00.30 kswapd0
253 root 15 -5 0 0 0 S 0 0.0 0:00.00 aio/0
254 root 15 -5 0 0 0 S 0 0.0 0:00.00 aio/1
960 root 15 -5 0 0 0 S 0 0.0 0:00.00 scsi_eh_0
963 root 15 -5 0 0 0 S 0 0.0 0:00.00 scsi_eh_1
1016 root 15 -5 0 0 0 S 0 0.0 0:00.00 kpsmoused
1046 root 15 -5 0 0 0 S 0 0.0 0:00.00 usbhid_resumer
1050 root 15 -5 0 0 0 S 0 0.0 0:00.00 rpciod/0
1051 root 15 -5 0 0 0 S 0 0.0 0:00.00 rpciod/1
1064 root 15 -5 0 0 0 S 0 0.0 0:04.87 kjournald2
1156 root 16 -4 2128 528 524 S 0 0.0 0:00.03 udevd
3274 root 20 0 4976 340 324 S 0 0.0 0:00.00 syslog-ng
3275 root 20 0 5092 1340 1224 S 0 0.1 0:01.12 syslog-ng
3401 root 20 0 4384 728 632 S 0 0.0 0:00.91 sshd
3462 root 20 0 2000 660 616 S 0 0.0 0:00.13 cron
3529 root 20 0 1760 616 612 S 0 0.0 0:00.00 agetty
3530 root 20 0 1760 616 612 S 0 0.0 0:00.00 agetty
3531 root 20 0 1760 616 612 S 0 0.0 0:00.00 agetty
3532 root 20 0 1760 616 612 S 0 0.0 0:00.00 agetty
3533 root 20 0 1760 616 612 S 0 0.0 0:00.00 agetty
3534 root 20 0 1760 616 612 S 0 0.0 0:00.00 agetty
3636 proftpd 20 0 10376 948 604 S 0 0.0 0:01.88 proftpd
3696 srv101 20 0 3232 1436 740 S 0 0.1 0:00.09 screen
3697 srv101 20 0 3212 1604 1320 S 0 0.1 0:00.00 bash
3701 srv101 20 0 2992 1248 1020 S 0 0.1 0:00.00 srcds_run
29499 root 20 0 7452 2588 2004 S 0 0.1 0:00.17 sshd
29504 root 20 0 3080 1628 1324 S 0 0.1 0:00.18 bash
29679 root 20 0 2388 1116 876 R 0 0.1 0:00.99 top

[Edge100x]

29040 srv101 -21 0 879m 781m 21m S 99 38.6 197:40.10 srcds_i486

Yikes, so the usage is all from the game server! A 781 MB RSS is insane for any type of game server. Does it use this much right after the server is started, as well? If you run without plugins temporarily, does it use this much? My immediate thought is that a plugin might be leaking memory.

It's also using an entire CPU core -- is it very busy right now?

[automatik]

14/32 slots used right now. I have not tried it without certain plugins just yet. When i get time tomorrow I will

[Edge100x]

Please do let me know the result! Also, if you're running a SourceTV, I recommend turning that off. I have seen major problems with SourceTVs eating up memory on Linux servers in the past.

[automatik]

Yes there is a sourcetv running as well..

[automatik]

with no plugins and a restarted game server

top - 12:45:25 up 3 days, 20:08, 1 user, load average: 0.03, 0.07, 0.02
Tasks: 52 total, 1 running, 51 sleeping, 0 stopped, 0 zombie
Cpu0 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2073564k total, 869968k used, 1203596k free, 14712k buffers
Swap: 1048568k total, 2120k used, 1046448k free, 751220k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31370 root 20 0 2388 1116 876 R 1 0.1 0:00.39 top
1 root 20 0 1716 552 528 S 0 0.0 0:01.56 init
2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root RT -5 0 0 0 S 0 0.0 0:00.09 migration/0
4 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0
5 root RT -5 0 0 0 S 0 0.0 0:00.06 migration/1
6 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1
7 root 15 -5 0 0 0 S 0 0.0 0:00.02 events/0
8 root 15 -5 0 0 0 S 0 0.0 0:00.00 events/1
9 root 15 -5 0 0 0 S 0 0.0 0:00.08 khelper
14 root 15 -5 0 0 0 S 0 0.0 0:00.00 async/mgr
122 root 15 -5 0 0 0 S 0 0.0 0:00.01 kblockd/0
123 root 15 -5 0 0 0 S 0 0.0 0:00.22 kblockd/1
125 root 15 -5 0 0 0 S 0 0.0 0:00.00 kacpid
126 root 15 -5 0 0 0 S 0 0.0 0:00.00 kacpi_notify
127 root 15 -5 0 0 0 S 0 0.0 0:00.00 kacpi_hotplug
173 root 15 -5 0 0 0 S 0 0.0 0:00.00 ata/0
174 root 15 -5 0 0 0 S 0 0.0 0:00.01 ata/1
175 root 15 -5 0 0 0 S 0 0.0 0:00.00 ata_aux
179 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksuspend_usbd
183 root 15 -5 0 0 0 S 0 0.0 0:00.00 khubd
186 root 15 -5 0 0 0 S 0 0.0 0:00.00 kseriod
250 root 20 0 0 0 0 S 0 0.0 0:00.00 pdflush
251 root 20 0 0 0 0 S 0 0.0 0:00.09 pdflush
252 root 15 -5 0 0 0 S 0 0.0 0:00.30 kswapd0
253 root 15 -5 0 0 0 S 0 0.0 0:00.00 aio/0
254 root 15 -5 0 0 0 S 0 0.0 0:00.00 aio/1
960 root 15 -5 0 0 0 S 0 0.0 0:00.00 scsi_eh_0
963 root 15 -5 0 0 0 S 0 0.0 0:00.00 scsi_eh_1
1016 root 15 -5 0 0 0 S 0 0.0 0:00.00 kpsmoused
1046 root 15 -5 0 0 0 S 0 0.0 0:00.00 usbhid_resumer
1050 root 15 -5 0 0 0 S 0 0.0 0:00.00 rpciod/0
1051 root 15 -5 0 0 0 S 0 0.0 0:00.00 rpciod/1
1064 root 15 -5 0 0 0 S 0 0.0 0:05.09 kjournald2
1156 root 16 -4 2128 528 524 S 0 0.0 0:00.03 udevd
3274 root 20 0 4976 340 324 S 0 0.0 0:00.00 syslog-ng
3275 root 20 0 5092 1344 1224 S 0 0.1 0:01.21 syslog-ng
3401 root 20 0 4384 728 632 S 0 0.0 0:00.93 sshd
3462 root 20 0 2000 660 616 S 0 0.0 0:00.14 cron
3529 root 20 0 1760 616 612 S 0 0.0 0:00.00 agetty
3530 root 20 0 1760 616 612 S 0 0.0 0:00.00 agetty
3531 root 20 0 1760 616 612 S 0 0.0 0:00.00 agetty
3532 root 20 0 1760 616 612 S 0 0.0 0:00.00 agetty
3533 root 20 0 1760 616 612 S 0 0.0 0:00.00 agetty
3534 root 20 0 1760 616 612 S 0 0.0 0:00.00 agetty
3636 proftpd 20 0 10376 948 604 S 0 0.0 0:02.11 proftpd
31298 root 20 0 7452 2552 1984 S 0 0.1 0:00.02 sshd
31303 root 20 0 2952 1620 1320 S 0 0.1 0:00.05 bash
31507 srv101 20 0 3232 1432 736 S 0 0.1 0:00.00 screen
31508 srv101 20 0 3212 1604 1320 S 0 0.1 0:00.01 bash
31512 srv101 20 0 2732 1232 1012 S 0 0.1 0:00.00 srcds_run
31517 srv101 20 0 199m 85m 19m S 0 4.2 0:02.25 srcds_i486

[Edge100x]

That's much more normal there (200 MB RAM, 0% CPU), but we'll have to see how it changes with people in the server.

[automatik]

So here is what I have noticed.

After every map change the servers RAM usage will go up anywhere from 0-10mb and then the "total" RAM usage of the box will go up 20-50mb. And it will just keep climbing and climbing until it maxes. Right now to help alleviate the problem I set a cronjob to clear memory cache every 60 minutes

[Edge100x]

You don't need to worry about memory usage by the file cache; more is actually better there, because having the files cached in RAM will make later level loads faster. Ideally, all of your free RAM will be used by the file cache -- there's no disadvantage to it, because the OS will instantly use it for any program that needs it, with no performance loss. You should definitely not clear that cache manually.

The memory used by the game server itself is the only concern here. If it is consistently going up as the server's uptime increases, you're looking at a likely memory leak situation.