We have an application running on WebSphere (unfortunately it is 6.1 which is no longer supported, it has not yet been migrated in production to a later version) which becomes entirely unresponsive because of hung threads.
As far as I can tell we entirely exhaust one of the thread pools.
I have activated hung thread detection and I get a core/thread dump when hung threads are detected.
The server can run for several days without problems but has crashed twice this week.
When load the core/thread dump in "IBM Thread and Monitor Dump Analyzer for Java", it tells me that there are a certain number of hung threads (this time it was 2, last time 11) and multiple (usually around 40) threads "waiting on condition" and some running threads. I believe one of the thread pool has around that size (50).
Now what I see in there are threads waiting for locks, having locks or in wait.
Most of them show a stack track which always ends like this:
at java/lang/Object.wait(Native Method)
at java/lang/Object.wait(Object.java:231)
Now, how can I track this down to either a server configuration problem, application issue, WebSphere problem or something else? How is this supposed to help me track down the problem when almost everything in there refers to IBM code?
I cannot ask IBM's help as 6.1 is now an unsupported version of WebSphere and while work has been done to make it work under WebSphere 7 we are not yet ready to switch to it in Production yet.
