Before actually asking, just to be clear: yes, I know about disk cache, and no, it is not my case :) Sorry, for this preamble :)
I'm using CentOS 5. Every application in the system is swapping heavily, and the system is very slow. When I do free -m, here is what I got:
total used free shared buffers cached
Mem: 3952 3929 22 0 1 18
-/+ buffers/cache: 3909 42
Swap: 16383 46 16337
So, I actually have only 42 Mb to use! As far as I understand, -/+ buffers/cache actually doesn't count the disk cache, so I indeed only have 42 Mb, right? I thought, I might be wrong, so I tried to switch off the disk caching and it had no effect - the picture remained the same.
So, I decided to find out who is using all my RAM, and I used top for that. But, apparently, it reports that no process is using my RAM. The only process in my top is MySQL, but it is using 0.1% of RAM and 400Mb of swap. Same picture when I try to run other services or applications - all go in swap, top shows that MEM is not used (0.1% maximum for any process).
top - 15:09:00 up 2:09, 2 users, load average: 0.02, 0.16, 0.11
Tasks: 112 total, 1 running, 111 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 4046868k total, 4001368k used, 45500k free, 748k buffers
Swap: 16777208k total, 68840k used, 16708368k free, 16632k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ SWAP COMMAND
3214 ntp 15 0 23412 5044 3916 S 0.0 0.1 0:00.00 17m ntpd
2319 root 5 -10 12648 4460 3184 S 0.0 0.1 0:00.00 8188 iscsid
2168 root RT 0 22120 3692 2848 S 0.0 0.1 0:00.00 17m multipathd
5113 mysql 18 0 474m 2356 856 S 0.0 0.1 0:00.11 472m mysqld
4106 root 34 19 251m 1944 1360 S 0.0 0.0 0:00.11 249m yum-updatesd
4109 root 15 0 90152 1904 1772 S 0.0 0.0 0:00.18 86m sshd
5175 root 15 0 90156 1896 1772 S 0.0 0.0 0:00.02 86m sshd
Restart doesn't help, and, by they way is very slow, which I wouldn't normally expect on this machine (4 cores, 4Gb RAM, RAID1).
So, with that - I'm pretty sure that this is not a disk cache, who is using the RAM, because normally it should have been reduced and let other processes to use RAM, rather then go to swap.
So, finally, the question is - if someone has any ideas how to find out what process is actually using the memory so heavily?
