Alef Arendsen

A couple of weeks ago I posted about a problem with ThreadLocals I had to hunt down. Thanks to YourKit it took me about half an hour to figure out where the problem was. Unfortunately it wasn’t something I could fix easily since it the problem was part of third party library.

But I’m not going to talk about that again. Instead, I just wanted to say thanks to the guys at YourKit for an truely excellent product. It’s quite cheap (about €400,=) and really takes an excellent approach to profiling. The way snapshots in the application can be compared is really handy and if you’re into integration with an IDE (Eclipse, IDEA), you’re done in under two seconds. Setting it up to profile application servers is also a piece of cake. YourKit modifies the startup scripts for several app. servers.

The Joel Test (by Joel Spolsky) includes this one line about ‘using the best tools money can buy’. Well, this is certainly the best tool for me right now! Pretty convenient that it’s cheap: allows me to drink an extra beer .

Some more pointers: On demand profiling, which allows you to profile application running in a production environment as well! And also the Snapshot comparison features are pretty cool!

Yeah! Today I heard the new house is going to be finished in about 3 weeks. On Friday the 11th I get to inspect it for the first time to see if everything has been done correctly. The week after that I get the keys, so they have a week to fix the mistakes they made (and there’ll always be something). I dropped by today and got totally freaked out when I saw they had torn the complete terrace apart! They had taken all the bricks out of there (because those where the wrong ones I heard later). Well, it’s gonna be fixed before I move in, so that’s good.

After I get the keys to the house I have to paint the walls (which is probably going to take me about a week). After that I need to get somebody to do the floor and after that the kitchen needs to be installed. So although I’m getting my keys in about 3 weeks, I’ll probably only move in the second week of April or something. Well, at least I’ll have something to spend time on (as if I didn’t have enough things on my mind ).

A couple of days ago I posted a piece on ThreadLocals, using a rather provoking title (believe me, it wasn’t meant to be that provoking ). This post digs a little deeper and explain why OutOfMemoryErrors occur. I probably could have figured this out by Googling a bit, but I was kind of curious to dig into it myself first.

The main problem lies in the implementation of the ThreadLocal class itself. It uses a map containing weak references, storing the ThreadLocal instance as the key and the object as the value. The value is where the problem is. Consider the following class:

package jteam.threadlocal;
 
public class Garbage {
 
    static ThreadLocal tl = new ThreadLocal();
 
    public Garbage() {
        tl.set(new Garbage(false));
    }
 
    public Garbage(boolean f) {
    }
}

The static ThreadLocal object is filled in the constructor with an instance of the class itself. When thinking about this structure, one can come up with the following rather interesting conclusion: the class has a static field, the ThreadLocal, which is used as the key in the WeakHashMap. The value itself is the Garbage object which in its turn has a direct reference to the class it belongs to (the Garbage class), hence an indirect (and strong) reference to the ThreadLocal field. Now if we refer to the JavaDOC of the java.util.WeakHashMap, it clearly tells us that values in a WeakHashMap should never, ever maintain a strong (direct or indirect) reference to the key, because “that will prevent the keys from being discarded” in this case resulting in:

• The ThreadLocal not being discarded
• The Garbage object not being discarded
• The Garbase class not being discarded
• The classloader that loaded the Garbage class not being discarded (the class has a reference to the classloader it was loaded by)
• Unless of course the current Thread dies (which does not often in a server environment)

Okay, an interesting situation nonetheless. Below code fragments can be used by the way to simulate the problem. Compile the GarbageProducer and the GarbageClassLoader into a different classes directory then the Garbage object (don’t ask me why, but I think there’s some additional classloader issues if you put them in the same directory ). The second direectory should be called classes2 (as you can see in the code). Run this with -verbose:gc turned on and you’ll see your VM running out of memory immediately!

Once again: I’m not saying ThreadLocals are useless. I just think that there should be a bit more warnings in the JavaDOCs .

The main class:

package jteam.threadlocal;
 
import java.net.URL;
import java.io.File;
 
public class GarbageProducer {
 
    URL u;
 
    public GarbageProducer() throws Exception {
        File f = new File(System.getProperty(
            "user.dir"), "classes2");
        u = f.toURL();
        System.out.println(u);
    }
 
    public void run() throws Exception {
        for (int i = 0; i < 20000; i++) {
            test();
            System.gc();
        }
    }
 
    public void test()
    throws Exception {
 
        GarbageClassLoader loader = 
                new GarbageClassLoader(
                    new URL[] {u},
                Thread.currentThread().
                    getContextClassLoader());
        Class clzz = loader.loadClass(
            "jteam.threadlocal.Garbage");
        Object o = clzz.newInstance();
 
    }
 
    public static void main(String args[])
    throws Exception {
        new GarbageProducer().run();
    }
}

The classloader (you don’t have to use a custom one, using the URLClassLoader does the same, I just used it for debugging purposes):

package jteam.threadlocal;
 
import java.net.URLClassLoader;
import java.net.URL;
 
public class GarbageClassLoader
extends URLClassLoader {
 
    public GarbageClassLoader(URL[] urls,
            ClassLoader classLoader) {
        super(urls, classLoader);
    }
 
    public Class loadClass(String s)
    throws ClassNotFoundException {
        return super.loadClass(s);
    }
 
    protected synchronized Class loadClass(
        String s, boolean b)
    throws ClassNotFoundException {
        return super.loadClass(s, b);
    }
}

Today I upgraded to WordPress 1.5, which turned out to be a 5 minute process! There has really been a huge turn for the better the last one or two years when it comes to ‘professional open source’ (as joker Marc calls it). Apart from the issue of the lack of a company behind many open source projects (of course I have to mention Interface21 here and there’s also other companies like the JBoss group backing a couple of projects) there used to be many issues with the release cycle of certain open source technologies. I’m really glad to see that changing.

Top priority should of course always be the technological bits and pieces but without decent documentation, regular releases but also howtos and upgrades guides accompanying those releases if necessary are key to the success of some open source projects.

WordPress team: good job!!!

P.s. I updated my template as well by the way. I used an existing template but changed some things around. I didn’t do any major CSS wizardry, but there might be some issues with different browsers (I can tell it works with Safari on my Mac). If there are issues, I’d appreciate it if you left a comment!

I recently started reading The 7 habits of highly effective people by Stephen R. Covey. I just started reading about proactivity and so far, it’s an excellent book. Highly recommended!!

Nice and all that you can tie resources and stuff to a Thread and than deeper down in your call stack retrieve them again, but obviously, people are using them for the wrong purposes without thinking about potential problems that might cause! One of the consequences of incorrectly using ThreadLocals is that it prevents webapplications from being redeployable. Okay, in a production environment, you probably don’t do hot-deployments, but still.

Today I chased down a memory/redeployment issue in Tomcat for a client of mine. The usual stuff: after a bit of profiling it showed that the classloader of an undeployed application couldn’t be cleaned up, due to the use of ThreadLocals. In org.apache.axis.Service a Call object is created and kept around for whatever reasons in a ThreadLocal. The createCall() method returns the call immediately and doesn’t bother about the ThreadLocal anymore. Without inspecting the code and thorougly profiling the application you don’t have a clue of what’s going on. We’ll probably create a patch ourselves to solve this specific problem.

This is about the twentieth time I’ve seen clients with problems that had to deal with the use of ThreadLocals. This clearly indicates that somewhere there should be warnings that the evil ThreadLocal should be used with caution, or at least only when you know you have to unset it again in a multi-threaded server environment.

By the way, it’s not like I think ThreadLocals should be removed from Java. They’re an excellent way to implement the passing of context information (as Ted Neward dubbed it the context pattern once I think) and it’s heavily used (in a proper way!) in the Spring Framework. You should 1) know what you’re doing and 2) clean your own mess up and not leave it for somebody else to call ThreadLocal.set(null);!

Matt asked if anybody had done any projects with BPEL. Well here are some experiences.

I recently did a project where I had to present a couple of technologies that would be the basis for a new provisioning system for a large company. The company already implemented about 10 to 20 SOAP services (either based on Java or simply wrapping a Perl system) so one of the requirements was for it to be working together with SOAP as easy as possible.

I ended up with jBPM and BPEL and we finally went the WS-* route and decided BPEL was going to be it. We had the choice between Oracle Business Process Manager (based on Collaxa, taken over by Orace I think last June–could also have been June 2003) and ActiveBPEL (by ActiveEndpoints–some of the guys there originally worked for SilverStream). The latter is a new product and still has some bugs and glitches, but I think it’s getting there. Oracle is pricy and they couldn’t really convince me of them not dropping the compatibility with other application server than just Oracle. So we went for ActiveBPEL. I won’t talk about the product itself right now too much. All I’d like to say is that apart from the glitches and tiny bugs they ActiveBPEL has a pretty good designer application. It’s really easy to get a simple flow up-and-running. If you need a tool to do BPEL modeling, definitely check this one out! By the way, the ActiveBPEL engine itself (which does exactly what it’s supposed to do and performs quite well) is open source (GPL licensed) and there’s a commercial version as well (also one based on JBoss but who needs that ), the designer costs a bit of money (can’t remember really, but it’s nothing compared to what Oracle charges).

About BPEL itself….. errr….. well, let’s say that I think it’s all rather complex! You have to think really carefully how to design your system and services and there’s a lot of SOAP going on there. Then theres the usual complexity you’re getting when using SOAP. Then there is for example the partner links which you have to add to your descriptors. Axis doesn’t support those (AFAIK) so I created a special Servlet filter to add them to the WSDL (so now I have a http://server.com/services/Service?bpel URL filtering the ?wsdl variant.

As far as XP or agile is concerned, BPEL doesn’t exactly facilitate iterative development. You’ll have to do a lot of upfront planning. Dragging and dropping a process together is not something you’re going to do in like five minutes (yes of course, if you have a small example with a service returning Hello World, not if you have 10 to 20 services and a process that has a lot of asynchronous activities, parallel processing and fail over).

Then the issue of the specification in general. I don’t think it’s mature enough yet. A lot of the things you’d expect from a business process management solution aren’t there yet. The first version of the spec doesn’t have retries or failover yet (it’s probably going to be included in the second version of the spec). Some vendors offer features like this (but those are proprietary) and you can of course get something in place that does this yourself. Then there’s the issue of an action manager or a task manager (whatever you call it). Business process management always involves notification, rescheduling, the creation of tasks, etcetera. BPEL doesn’t provide something here. Oracle has its proprietary addition (I think it’s called task manager).

Last but not least there’s the issue of the incomplete set of standard related to web services. There’s a lot of things that still need to be resolved and then there are no really mature tools that provide implementations for all the standards (security, atomic transaction, business transaction, federation, etcetera).

I had an interesting chat with Simon Phipps back in September or October about BPEL and other integration standards. He mentioned UBL and was questioning whether we should focus on standardizing the infrastructural issues (security, transport, etcetera) or the language we are talking. Why not leaving the transport and security issues as they are and standardize on what we actually want to say to eachother. UBL (I think it’s ebXML related) is an example of this.

Maybe BPEL can be useful if you already have a dozen SOAP services and need to create several business processes and customize small bits of them easily. Don’t think however that it’s the best thing that has happened to mankind since the invention of matches–BPEL sure is a lot more complex !

I’ve gotta find myself a new pet project. There are basically just two criteria for the proect: 1) I have to be able to finish it in about four to six weeks and 2) I can only spend about 10 hours a week on it. I have the following things I can choose from:

1. Upgrade the Spring nightly build

Upgrade the automated nightly build/test environment for the Spring Framework. I’d like to include better reporting (somewhat like the continuous build running on the SourceForge machines right now) some advanced server testing, using different JDKs, different databases and different application servers. Switching of application servers is already in there, but offering support for different JDKs and for example PostgreSQL, MySQL and Oracle would be really cool.

I’ve also been toying with the idea of getting this into place for several other open source projects. In on of our projects we are using Groovy. In one of the latest releases we had the application fully tested on our staging server but right after deploying it on the live server the JDK crashed. Fortunately sombody else had already noticed that Groovy didn’t work that well in combination with a certain JRockit release. The latest JRockit update fixed this and after upgrading to live server with that latest release the problem was fixed (that was the one and only difference between the live server and the staging machine by the way–remember Murphy’s Law?). Anyway, I think it’d be great to get some infrastructure like this in place for some of the key open source projects.

2. Get myself familiar with Ruby

Well, you can’t exactly call this a pet project, but it’s something I’ve been meaning to do for a long time. I can’t really find a use case though right now and I think a pet project or something like it should benefit somehow, not only in the long run. The onyl reasonable thing I can think of is the idea I’ve been having about a Ruby-based build process reusing Ant to do the actual work. Probably it won’t be feasible to put this to work also in the near future since I have a really large existing investment in our current build environment.

3. Get myself familiar with Java Server Faces

I don’t know about this one. I like getting into it, but I don’t know I want to learn more frontend stuff. I’ve been doing SiteMesh lately, been doing a lot of writing for the Spring book on the MVC parts of Spring. I’ve also developed a couple of frontends for application the last half year so I’m not sure.

4. Sit back and relax

Well no!

There’s probably more ideas popping up during the weekend. Opinions are always welcome of course!

Hmmm, rather annoying: when using Subversion and Eclipse (no Eclipse plugin), the default settings seem to copy all the .svn folders to my output dir. To fix this, go to Window > Preferences > Java > Compiler > Building and change the Filtered Resources pattern to include .svn/. This will leave all the .svn folders where they are.

We’re in the process of migrating the JTeam corporate site to a new server. In the meantime, this site will pop up every now and then. The migration should be finished before 10pm (CET).

Thanks for your patience.