Strange enduser experience with Liferay, Glassfish and Apache on RedHat
- by Pete Helgren
Tried multiple forums to get to the bottom of this. I hope I can get some direction here:
Here is the stack I am working with:
Red Hat Enterprise Linux Server release 5.6 (Tikanga)
Liferay 6.0.6 on Glassfish 3.0.1 MySQL 5.0.77
Apache 2.2.3
The Liferay portal provides a variety of portlets to end users. Static content (web pages), static resources (primarily pdf and mp3 files 1mb - 80mb in size), File upload and download capabilities (primarily 40-60mb mp3 files) and online streaming of those MP3 files.
Here is the strange end user experiences:
Under normal load: (20-30) users uploading, downloading or streaming files and 20-30 accessing static content (some of it downloads), we see the following:
1) Clicking a link triggers the download of a portion of an MP3 (the portion is a few seconds long).
2) Clicking on a link tiggers the download of the page content rather than rendering.
3) Clicking a link causes the page to dump binary data to the end user rather than the expected content.
4) Clicking a link returns the text of a javascript file rather than rendering the page.
Each occurrence is totally random (or appears so). Sometimes it works, sometimes it doesn't. It seems to have no relation to browser or client OS. The strange events seem to occur much more frequently when using an SSL connection rather than regular http.
Apache serves as a proxy server only (reverse). It basically passes all the requests through to Glassfish. There isn't any static content proxy served by Apache.
We rebuilt the entire stack from scratch and redeployed the portlet wars and still have the same issues. Liferay is running as a single server (not clustered). We disabled mod_cache in Apache. The problems are more frequent as the server load grows. This morning the load is pretty light and we are seeing few problems but the use of the site will grow, particularly tonight around 9pm CST through Wednesday morning. You could try the site (http://preview.bsfinternational.org) during those times and I would expect that you might experience one of the weirdnesses as you randomly click links on the site (https is invoked only when signed in). Again, https seems to exacerbate the issue.
This seems very much like a caching issue but I don't know where in the stack to start peeling the onion. Apache? Liferay? Glassfish? MySQL? Maybe even Redhat? We are stumped and most forums we have posted to (LifeRay and Glassfish) have returned very few suggestions. I just need an idea of where to start looking. I understand that we could have a portlet
EDIT: Opening the files in a Hex editor that appear to be pages that download rather than render, we see that the first 4000 characters are "junk" and then the "HTTP/1.1 ...." 'normal' header is seen. So something is dumping a jumble of characters up to offset 4000 (when viewing it in a Hex editor). Perhaps a clue?
Ideas?
