After a server bounce happens, and after around 40-45 days time period, we receive continuous “Committed Virtual Memory” alerts which indicates the usage of swap space in the magnitude of 4GB
This also causes the application to perform very slowly and experience a number of stalled transactions.
Server Setup:
4 Tomcat Servers (version 7.0.22) that are load balanced (not clustered) by 2 Apache Servers. And the Apache servers themselves supply static content and routing to these 4 tomcat servers.
Java Runtime Version: java version "1.6.0_30" Java(TM) SE Runtime
Environment (build 1.6.0_30-b12) Java HotSpot(TM) 64-Bit Server VM
(build 20.5-b03, mixed mode
Memory Startup Parameters:
MEMORY_OPTIONS="-Xms1024m -Xmx1024m -Xss192k -XX:MaxGCPauseMillis=500 -XX:+HeapDumpOnOutOfMemoryError -XX:MaxPermSize=256m -XX:+CMSClassUnloadingEnabled"
Monitoring – Wily monitoring is available in all the production servers that monitors key server parameters and sends out configurable alert emails based on pre defined settings.
Note: Each of the servers also has two other separate tomcat domains that run different applications
Investigated area:
There is no Heap Memory Leak and the GC is running fine without any issues over any period of time
The current busy thread count corresponds directly to the application usage – weekends and nights have lesser no. of threads compared to business hours
ThreadLocal uses a WeakReference internally. If the ThreadLocal is not strongly referenced, it will be garbage-collected, even though various threads have values stored via that ThreadLocal.
Additionally, ThreadLocal values are actually stored in the Thread; if a thread dies, all of the values associated with that thread through a ThreadLocal are collected.
If you have a ThreadLocal as a final class member, that's a strong reference, and it cannot be collected until the class is unloaded. But this is how any class member works, and isn't considered a memory leak.
The cited problem only comes into play when the value stored in a ThreadLocal strongly references that ThreadLocal—sort of a circular reference. In this case, the value (a SimpleDateFormat), has no backwards reference to the ThreadLocal. There's no memory leak in this code.
Can anyone please let me know what could be the cause of this and what to be monitored?
