Jun 182012
 

Tomcat servers are often sat behind Apache, with traffic passed through using mod_proxy, but they can be used directly. I am a n00b at this, a part-time convert to Linux from Windows via VMware; I think more and more people are dipping their toe in the Linux water for apps like Confluence, WordPress and so on so I offer my pain in case it helps anyone else.

My recent experience with Confluence leads me to conclude that running Tomcat without Apache, at least for this application, is a bad idea. This is the first time I had seen it done that way, I have always had Tomcat behind Apache before now, and looking at this experience I can see why.

There is no good single source for scaling documents, reference architectures and so on for Confluence; for an application targeted at the enterprise it seems to lack a fair bit of the collateral that enterprise ops teams like to see.

What we found was that running Confluence out of the box (on RHEL 6.2 x86/64) resulted in instability, with the application frequently crashing. Clustering support is – ahem - rudimentary, so we have a monitor in place that watches the processes and restarts if necessary, or moves to a hot standby if there are too many failures.

Out of the box, logs are written into the application directory tree, by default  the service runs as root, and the app serves directly using Tomcat. I don’t know to what extent this is naive choices by the team here, but that’s how I found it.

There was no documentation to warn about the stream handle count (ulimit) and that bit us hard on the arse fairly quickly.

Debugging was tedious in the extreme. Tomcat logs were inadequate and times for garbage collection and other metrics were from process load not in usable time format; also log rollover seems to have to be coded in the app, not Tomcat. But the biggest problem was that the app would randomly just crash under load.

On the advice of Adaptavist (who seem pretty good, very responsive support) we put Apache in front of Tomcat and passed the traffic through using mod_proxy. It took about an hour to get this set up, and another couple of hours to get the Confluence service running as a dedicated service account and writing logs into the /var/log/ tree, which is on a separate partition for the usual reasons.

The result was surprisingly unambiguous. Not only do the Apache logs give better diagnostics, but the traffic to Tomcat seems to be better regulated – the Tomcat logs indicate a less spiky load. And the biggest surprise was that the periodic crashing vanished completely. With the exception of a single database disconnect, the app has been completely stable since adding Apache.

So, I reckon if you are using a Tomcat-based app and you are not a Linux/Unix guru you would be well advised to use Apache and mod_proxy in front of it, to give more informative logging and better control over the application’s workload.

 Leave a Reply

(required)

(required)

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>