Perl tests in Hudson via JUnit

Perl tests in Hudson

I'm willing to automate the unit testing of my Perl modules, by plugging them into Hudson. Hudson is a continuous integration server, it "monitors executions of repeated jobs, such as building a software project or jobs run by cron". It works well, is extensible, and looks good.

So, The easiest way to do that is have Hudson to execute the module test suite, and look at the output. Actually, not really the output, but a file containing the output of the tests, in JUnit format.

Now, by default, Perl unit tests doesn't output JUnit. You know, that kind of stuff :

That's the standard output of tests run through TAP, the Test Anything Protocol, which is great, and (imho) better than JUnit.

Anyway, that output is generated because the tests are run through TAP::Harness, that outputs the result in TAP format on the console.

So first of all, let's see how to transform the tests output into JUnit. My modules use Module::Build (and also Dist::Zilla, but that's an other story) as building and releasing system. When using Module::Build, running the test suite is easy :

Solution 1 : use TAP::Formatter::JUnit

After some research and looking in the source code, I discovered that you can pass a lot of options to TAP::Harness, via Module::Build. Especially, you can specify a formatter to TAP::Harness when running the test suite. TAP formatters are modules that all inherit of TAP::Formatter::Base.

For instance, if instead of outputing the result on the console, you want it to be stored in a file, use TAP::Formatter::File. As you may have guessed now, there is a formatter that outputs JUnit : TAP::Formatter::Junit. That's exactly what we want. By issuing the following command line, the output of the test suite is in JUnit format.

Now we just need to tell Hudson to run it and look at the output

Solution 2 : use TAP::Harness::JUnit

For some reasons, I had some trouble with TAP::Formatter::Junit on windows, mainly because there is no ActiveState PPM for this package. So I decided to give a go with TAP::Harness::JUnit

Its documentation is self-explanatory : it's a child class of TAP::Harness, that accepts an xml file name as argument, and procudes directly a JUnit output. This module being directly available in ActiveState Perl, it seemed to be aa good approach.

So the goal is to replace the call to TAP::Harness into TAP::Harness::JUnit, when doing Build test. And actually, it would be better to have an additional action to Build, so that we could do Build test, and Build hudson_test.

Luckily, Module::Build makes it easy to add an action. By looking at the ACTION_test method in Module::Build::Base, it's easy to see what needs to be changed. See this code snippet. What we want is a copy of that code that would run TAP::Harness::JUnit instead of TAP::Harness.

After some work, here is what I eneded with. This code is to be added into your Build.PL.

Now, build and run your tests in hudson mode

Configure Hudson

That's the easiest part : Setup a new project, add as execution line "perl Build.PL & Build hudson_test" (if you are on a windows hudson node). And point the project to the output file called "hudson_test_output.xml". That's it !

Comments

[Mark]

Thanks for documenting this. I recently tried to use Hudson to track Perl automated testing and never got it quite working. Docs like this would have been helpful.

I eventually gave up and tried installing Smolder, a competing product available from CPAN. Using it's built-in web server and the SQLite backend, it was easy to get up and running, and I knew as a Perl project that I could hack on it if it didn't do quite what I wanted.

[Michael Peters]

As the author of Smolder, I'm curious as to what features Hudson provides that you use that Smolder doesn't. I know Hudson does the actual job management (which smolder doesn't it leaves that up to you) but out of curiousity, is there anything else?

[niceperl]

Great work!

I've used Hudson & Sonar to make IC in my JAva projects with good results (easy, functional, ...).
Integrating Perl in this environment is a brilliant idea

Thank you

[Graham TerMarsch]

As the author of TAP::Formatter::JUnit, I'm curious to hear about whether you encountered other issues that prevented you from using it, or was it just "lack of PPM" that did it in for you?

Heck... lets face it... I'm curious to hear from anyone using TAP::Formatter::JUnit, even if just to hear what you're using it for and where its working (or not working) for you.

Thanks for the write-up Damien, always nice to see people trying it out. :)

[Johannes Plunien]

Nice blog post! :)

Do you have any tips how to integrate TAP::Harness::JUnit with Module::Install?

Thanks,
plu

[dams]

@ Michael Peters :

The main reason for not using Smolder, is that I didn't have choice about installing and using Hudson :) It was already there, and part of the job was to plug my tests to it, not come with a different continuous integration platform.

The other reason may be that I didn't know much about Smolder. I think it would benefit from some kind of public homepage and better introduction, if you want to advertise it. Nevertheless I'll try to give it a go for my personal knowledge :)

[dams]

@ Graham TerMarsch :

First of all, It worked well on Mac OS X. Then I saw that there were no PPM available for windows. Nevertheless, I tried to include TAP::Formatter::JUnit locally to my modules, in an include directory, and have that directory pushed to @INC somehow. It worked well on Mac OS X again (after uninstalling the system version of T::F::JUnit), but I didnt manage to make it work on windows... It complained with Cant load T::H::JUnit (useing the module worked however).

Instead of spending more time in what was becoming darker and darker hacking, I tried the other solution, which worked, and I kinda liked the idea of having an additional action for Module::Build

dams

[Justin Mason]

Here's a Solution 3: http://taint.org/2008/03/26/124602a.html . We use this on the SpamAssassin Hudson: http://hudson.zones.apache.org/hudson/job/SpamAssassin-trunk/

NAME

tap-to-junit-xml – convert perl-style TAP test output to JUnit-style XML

SYNOPSIS

tap-to-junit-xml "test suite name" [ outputprefix ] tap_output.log

DESCRIPTION

Parse test suite output in TAP (Test Anything Protocol) format, and produce XML output in a similar format to that produced by the junit ant task. This is useful for consumption by continuous-integration systems like Hudson.

[Cheap Computers Canada]

Using the different codes you can certainly pass on different options.

[Jeff Lavallee]

An alternative to creating a Module::Build subclass is just to run the tests using prove:

prove --harness=TAP::Harness::JUnit

This has been working quite well for me in Hudson!

[dams]

@Jeff Lavallee :
You are right ! I should probably have mentioned that before expanding the explanation to adding a new Build.PL action. However in my particular need, I needed the command line to be as simple as it could be, as it had to be launched by other people, not familiar with Perl... Thats why I went on with adding a Build.PL rule