. .
JAN 2012

Hacking Around Grails' Broken Snapshot Dependency Mechanism

Even though Grails (up onto, and including Grails 2) support snapshot dependencies, the feature is flawed in a way that makes it unusable for us. This will be fixed in Grails 3, but we couldn't wait that long, so we ended up hacking together a workaround. This article describes why and how we did it.

So what is the problem?

In short, Grails doesn't handle snapshot dependencies.


Coming from a fairly heavy Maven background, we got the feeling that Grails' build system is a castle built on the mud which is Apache Ivy. Although Ivy and Grails might work well for monolithic projects, as soon as you start to depend on having a modular build with changing components, or snapshot dependencies, you start running into trouble.


If you're unfamiliar with snapshots, here's a quick explanation, copied from Sonatype's Maven book:

Why would you use this? SNAPSHOT versions are used for projects under active development. If your project depends on a software component that is under active development, you can depend on a SNAPSHOT release, and Maven will periodically attempt to download the latest snapshot from a repository when you run a build. Similarly, if the next release of your system is going to have a version "1.4", your project would have a version "1.4-SNAPSHOT" until it was formally released.


Now Grails and Ivy kind of support snapshot dependencies, although the feature has been struck with some annoying bugs. 

And even if you do upgrade to the latest version of Grails, or apply various workarounds - there is still a fundamental limitation in the feature, described in the documentation.  


In short, as soon as you start mixing local maven repository (in your ~/.m2/repository) with a remote repository, you run into trouble, and have to start doing stuff like manually deleting files out of your Ivy cache, or local maven repo. 


This would be very painful for us, and anyone else that rely on snapshot dependencies in a Grails project. Surely somebody must be trying to find a solution?


The good news is that the answer is yes - Hans Dockter, of Gradle fame, found and reported this weakness in Ivy some time ago - and they also built a solution for it into Gradle (which also uses Ivy).

Furthermore, Gradle has an overall improved dependency mechanism nearing completion at the time of writing.


Now you might know that it's on the roadmap of Grails to switch to using Gradle for building, but unfortunately, this is not scheduled before Grails 3.0, arriving no sooner than 2013.


We cannot wait that long to get our snapshots working, so we had to come up with something different.

What can we do?


These were the alternatives we could come up with:

  • Avoid snapshot dependencies altogether 

This would mean that every time we made a change in a dependency, we would have to make a release, and bump the version number in our BuildConfigs. We have a very nice system for making releases by the push of a button, but it would be a hassle with maintaining version numbers. Eventually we would make mistakes and build something with outdated dependencies.

  • Continuously delete snapshots out of the local Ivy cache

This was the first workaround we tried. Eventually our Grails build scripts got littered with delete statements into ~/.ivy2/cache/. This was unintuitive, and over time got really annoying to work with. We would sometimes forget the cleanup steps, and scratch our chins and wonder why the wrong snapshot would be in use, after first having spent an hour figuring out that the problem was the snapshot dependency. 

  • Hack together a workaround that makes Grails work properly with snapshots

This is what we ended up doing. And we called it...

Libify (our Grails-snapshot-fix hack)


We start with a simple idea: A Grails project has a 'lib' folder where you can copy in dependencies. Instead of declaring our snapshot dependencies in the normal BuildConfig.groovy configuration, we'll download them into the lib folder using a proper tool that understands snapshot dependencies: The Maven Ant Tasks (http://maven.apache.org/ant-tasks/index.html).


We trigger this by hooking into one of the early Grails build events (http://grails.org/doc/latest/guide/commandLine.html#events):

eventResolveDependenciesEnd = {
    // copySnapshotDepsIntoLibFolder()
}

 

We have to enter which dependencies we have somewhere. We created a file called libify-dependencies.groovy that looked like this:

[
        [groupId: 'de.viaboxx', artifactId: 'agnes', version: '2.11-SNAPSHOT', type: 'jar', scope: 'compile'],
        [groupId: 'de.viaboxx', artifactId: 'elliot', version: '1.1-SNAPSHOT', type: 'jar', scope: 'compile'],
]

 

This is basically just a Groovy map that we can evaluate into a list of dependencies, like this:

    def libifyDependenciesFile = new File('libify-dependencies.groovy')
    def shell = new GroovyShell()
    def deps = shell.evaluate(libifyDependenciesFile)

 

We can now iterate the deps, using the Maven ant tasks to create a fileset:

 

mvn.dependencies(filesetId: 'artifacts', versionsId: 'dependency.versions') { 
    dependencies.each { dependency(it) } 
}

and copy these into the lib directory:

    ant.copy(todir: 'lib', overwrite: true) {
        fileset(refid: 'artifacts')
        mapper(type: 'flatten')
    }

That's basically the gist of it. Now there are some technical issues that we work around:

  • The dependencies drag in all transitive dependencies, including the non-snapshot ones we already have declared in the BuildConfig. 
    • Solution: Copy the artifacts into a temporary directory: target/libify-temp, from where we then can copy only the artifacts declared in libify over to the lib folder.
  • The classpath used for compiling the project is evaluated *before* the event hooks in, so we get compilation problems the first time we build before the lib folder has been populated.
    • Solution: Use the rootLoader.addURL(url) method to add the libify dependencies to the classpath on the fly, after they have been resolved

As we went along, we figured that manipulating the 'lib' folder through the Grails lifecycle is a bad idea, as we get all sorts of nasty locking on files when building on Windows. We therefore abandoned the practice of using the lib folder altogether, instead just referencing the dependencies as they are in the temp folder.


This presents us with a new problem: We need to make sure our dependencies get included in the final resulting WAR file. We achieve this by overriding Grails' dependency resolution at the end of the libify script:

    buildConfig.grails.war.dependencies = {   antBuilder ->
        // add libified files to war/WEB-INF/lib
        fileset(dir: libifyTempDir, includes: libifyDeps)
        fileset(dir: "${basedir}/lib") {
            include(name: "*.jar")
        }
        grailsSettings.runtimeDependencies?.each { File f ->
            fileset(dir: f.parent, includes: f.name)
        }
    }

Now we've cleaned up our _Events.groovy script a little, and put it in a Gist on GitHub:

https://gist.github.com/1633380

It's not really something you should just drop into your scripts folder, but rather copy in some of it, and adapt the rest to fit your project. 

For instance, we found we had to make some nasty reading of bounded variables to find relative paths of "parent plugins" (many projects share one libify-dependencies.groovy). If you also need this, get in touch and we'll share how we do it.

Note that we do not include version-numbers in our JAR-file names (it's a long story). You could easily adapt the script to use JAR-files with version numbers in them if you want.

Any minuses?

The main annoyance is the IDE integration. Usually IntelliJ IDEA will import and build a Grails project just fine automatically. But with this custom location of dependencies, we have to manually add the target/libify-temp directory into our classpath configuration in the IDE. It's a hassle, but we can live with it. Maybe we'll make a script that fixes the IDE's configuration files at some point.

This workaround is prone to breakage when we upgrade to Grails 2, or any other version. However, we will deal with this at the time of upgrade, and when we finally convert to Grails 3 we can remove Libify entirely.

Final words

You will probably (not) appreciate the hacky nature of this solution, but it works for us right now, and enables us to continue using snapshots the way we are used to, until Grails 3 comes along.
As a final disclaimer: We actually didn't start off with Libify for this purpose. We just wanted to strip certain dependencies of the version number in their file name before including them in the WAR. Only later did we start "abusing" the script for getting snapshot dependencies.