ActiveMQ's "Too many open files" error

Intro

Last week I had to spend some time figuring out why a few of our applications were crashing over the weekend and even some nights during the week. Our status monitor was throwing up failures nonstop, and the only remedy we could find was to restart the applications and let them live another day.

After a couple days of this, we think we've finally found out what was happening.

What's happening?

Our applications use ActiveMQ to communicate between each other, and as part of our monitoring process, we check the connection between our apps and ActiveMQ.
The common way we do this is to send a message to a queue on ActiveMQ in transactional mode, then roll it back. If no exceptions are thrown during the process, we consider the ActiveMQ connection to be up.

The problem is, ActiveMQ was sometimes keeping sockets open when the status monitor would send a message and rollback. As a result of this, the "open file" count for the ActiveMQ process slowly kept climbing higher and higher, until the ActiveMQ log became filled with messages like this:

Database ~/activemq/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '~/activemq/data/kahadb/lock' could not be locked. | org.apache.activemq.store.kahadb.MessageDatabase

Could not accept connection : java.net.SocketException: Too many open files | org.apache.activemq.broker.TransportConnector | ActiveMQ Transport Server: tcp://0.0.0.0:61616

Let's fix it

We had to find out why out open-file count was always increasing! After a long day of trying to reproduce the error in a test environment, followed by a few attempted solutions, we discovered the problem.

When we wired up the connection to ActiveMQ in our applications, we were just using a simple ActiveMQConnectionFactory to create the connections. For some reason, when we rolled back messages sent to ActiveMQ, the socket on the broker's end would sometimes stay open.

We quickly discovered that by using a PooledConnectionFactory in place of the ActiveMQConnectionFactory, the sockets were being properly released when messages were rolled back! Success!

Conclusion

As always, the best way to figure out what's going on is to reproduce the error in a safe, closed environment (be it 'Dev' or 'Test' or 'Staging' or whatever you like). Once you can get it to happen reliably, you are able to try any number of solutions and can verify the results consistently.

Hopefully if you encounter an error like this, you'll be able to find the problem and remedy it quickly!

Event Espresso

I recently had to set up an online booking system for a WordPress site. After looking around for some Paypal plugins, I eventually stumbled upon Event Espresso. I installed the free version and quickly found that it had everything I needed.

This WordPress plugin lets you set up registration pages for any number of events, classes, seminars or whatever else you want. You set up your events in the Event Espresso configuration, and it generates a registration form for you. You just need to put a link to this registration form somewhere on your site, and when customers fill out the form, they'll be prompted to pay for the event through Paypal. All you need to do to integrate Paypal is provide the email address where the money should be paid.

After a user has submitted their registration and paid the fee, Event Espresso stores their information in your WordPress database. You are able to see who has registered, their information, and even their payment status.

Event Espresso provides a wide range of options which you can use to customize your event registration:

• Each event can contain a writeup of the event details, with full HTML support. If you provide the address for an event, a Google Maps link is even included on the registration page
• You can customize the fields that show up on the registration form, and mark them as required/optional. Event Espresso will even validate fields, such as ensuring a valid e-mail address is entered
• There is an option to send the user a personalized/custom confirmation email to say thanks, or to give them more information
• With the paid version, you can even send newsletters to everyone who has registered for an event

This plugin made event registration a piece of cake, and their website has a huge amount of help and resources to answer any questions or get you through any problem.

How to style an html file input to show only a button

A few days ago, I was trying to take an html input element, and replace it with a simple button that would prompt the user for a file when clicked on. One of the requirements was that the style had to work across FireFox, Internet Explorer (7+) and Safari.

At first, I tried hiding the file input, then displaying a button that would redirect clicks to the file input. This worked on a few browsers, but others would block this technique due to security concerns.

After a quick Google search, I found the following solutions, but they all failed for their own reasons:

• http://www.quirksmode.org/dom/inputfile.html : This provided a styling trick for the input button and text field. I had only wanted a button

• http://shauninman.com/archive/2007/09/10/styling_file_inputs_with_css_and_the_dom : This involved making the html input invisible, and overlaying it on a button-like div with a background. This worked for the most part, but javascript was used to re-position the file input on the page so the user would always click on the input. Unfortunately, in IE9, the input would follow the mouse outside the div, so if you clicked somewhere completely different, the file input would receive the click and prompt for a file - not good.

• http://www.appelsiini.net/projects/filestyle : Same problem as above.

I liked Shaun Inman's idea of overlaying a transparent file input on a div with a button-esque background, but the solution for ensuring the file input was clicked didn't satisfy my cross-browser compatibility requirements.

Instead of moving the file input around the page to ensure it got clicked at any location in the div, I decided to try making the file input take up the whole div. This was done by giving the button-div a fixed size, and adding the style "overflow:hidden". This way, the file input can be as big as we need, while not spilling out into the rest of the page.

Here's an example:

HTML

<div class="inputWrapper">
    <input class="fileInput" type="file" name="file1"/>
</div>

CSS

.inputWrapper {
    height: 32px;
    width: 64px;
    overflow: hidden;
    position: relative;
    cursor: pointer;
    /*Using a background color, but you can use a background image to represent a button*/
    background-color: #DDF;
}
.fileInput {
    cursor: pointer;
    height: 100%;
    position:absolute;
    top: 0;
    right: 0;
    z-index: 99;
    /*This makes the button huge. If you want a bigger button, increase the font size*/
    font-size:50px;
    /*Opacity settings for all browsers*/
    opacity: 0;
    -moz-opacity: 0;
    filter:progid:DXImageTransform.Microsoft.Alpha(opacity=0)
}

Demo:

Upload!

All you need to do to make the button look nice is apply a background image to the div, and add a 'pressed' background image to the :hover or :active state to give it the impression of being clicked!