We're in the middle of a project to move our infrastructure from a co-lo situation into Amazon EC2 and we've noticed some weird memory characteristics of the processes in our setup. Without going into too much detail about the specifics of our processes, we've noticed that on our EC2 instances "top" will show processes using a lot of swap space -- in fact, much greater than the amount of available swap or (if you add it all up) more than the available disk.
Here's a sample top output:
Mem: 7136868k total, 5272300k used, 1864568k free, 256876k buffers
Swap: 1048572k total, 0k used, 1048572k free, 2526504k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ SWAP COMMAND
4121 jboss 20 0 5913m 603m 14m S 0.7 8.7 3:59.90 5.2g java
22730 root 20 0 2394m 4012 1976 S 2.0 0.1 4:20.57 2.3g PassengerHelper
20564 rails 20 0 2539m 220m 9828 S 0.3 3.2 0:23.58 2.3g java
1423 nscd 20 0 877m 1464 972 S 0.0 0.0 0:03.89 876m nscd
You can see, for instance, that jboss is reportedly using 5.2 gigs of swap space which is definitely impossible since there's only 1G allocated and none is being used (probably because there's still 1.8G of RAM free).
And here's the results of uname -a:
Linux xxx.yyy.zzz 2.6.35.14-106.53.amzn1.x86_64 #1 SMP Fri Jan 6 16:20:10 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
We're running an AMI based off of the default Amazon Linux AMI (Amazon Linux AMI release 2011.09, so some RHEL5 and RHEL 6) with not too many customizations and definitely no kernel-level customizations.
Something here tells me that on this particular kernel/distribution, the reporting of swap or maybe even total memory usage isn't what it appears to be...
Any help would be appreciated!
