Friday, December 28, 2007

Using TestNG in Scala: By Example

Here is a very, very simple example of some TestNG stuff that I wrote the other day in Scala (it can also be found here):

package com.joshcough.cpu.electric;

import org.testng.annotations._
import org.testng.Assert_

package com.joshcough.cpu.electric;

import org.testng.annotations._
import org.testng.Assert._

class GeneratorTest {

def expectNewGeneratorToBeOn() = {
val gen: Generator = new Generator

def expectTurnedOffGeneratorIsOff() = {
val gen: Generator = new Generator

This might not be the prettiest code in the world, but:
  • It works (which is more than I could accomplish with others)
  • I can run it in my IDE
  • I can run it in Ant
  • I only have to do exactly what I ever did with Java
  • Its still way better than JUnit

There were a few things I had to do to make this work:
  • When compiling in Ant, I had to add this to my scalac call:
    • target="jvm-1.5" ...
  • When compiling in Eclipse I had to do this:
    • Project -> Properties -> Scala Compiler Properties -> target = jvm-1.5

I could give a more detailed example, but I haven't tried it yet. I'm pretty sure that things like @BeforeMethod, @AfterClass, @DataProvider and what not all just work the same. I'll try to come up with a better example though.


  1. Hi, great that you found out that TestNG plays nicely with Scala and thanks for sharing this insight.

    The only thing that annoys me is that I didn't get it to run in Eclipse. I set target to jvm-1.5 in the project preferences and created a suite xml, but when I run the test suite from eclipse, the test runner reports that it is skipping my classes because it didn't find any TestNG annotations. When I open the generated class files, the TestNG annotations are there. Is there anything else that you did to get it to run?

  2. Phouk,

    I had to jump through some hoops to get this work work. I wrote this up on Artima. Here is the link:

    "You're right about how I'm running TestNG with Scala. As I said in my most recent blog, its a pain. I didn't really explain, but I will now.

    First, I cant just right click the Scala file and do Run As -> TestNG test. Cedric said Alex might get back to me on seeing if we could add that support.

    What I was able to accomplish was running it through a TestNG XML file, right clicking on it and Run As -> TestNG Test. The xml file is simple, as you can see here:

    But, I was only able to do this after calling my Ant target to compile all my code and tests. But even then there is some weird interaction that I still need to figure out. The eclipse Scala plugin is compiling the files to the bin directory. I'm compiling them into target. There are classpath issues and each time before I run the tests I have to do Project -> Clean. Such a pain. Most of the time I get frustrated and just run them though Ant, which I could also do with specs, so I almost moved to specs.

    But, every once in a while when I have a failing tests, its so nice to be able to run them in the IDE and really dig in and figure out whats going on. Its not nearly as nice doing that though Ant."

  3. This comment has been removed by the author.

  4. Let me expand on hoq I do this:

    1) I first do Project -> Clean. This removes all the classfiles in the bin directory and scalac will not recompile them right away. I wish it would, but it currently doesn't compile anything until the file is explicitly saved.

    2) Compile my classes using Ant into the target directory.

    3) Also using Ant, build a jar file containing all my compiled classes and put it into the target directory. I call this jar Test.jar.

    4) I created a TestNG XML file for my suite. You can put the names of whatever packages or classes you want to run in it.

    5) I right clicked the file in Eclipse and did Run As -> TestNG Suite. THE FIRST TIME THIS WILL FAIL. THE CLASSPATH IS BAD.

    6) I then had to open up the Run Dialog and find the run configuration for my XML suite. I had to add the target directory to the classpath of the run configuration.

    7) Click apply, but don't run it through the open Run Dialog screen, because for some reason if fails.

    8) Get out of that screen and right click on the XML suite again, and do Run As -> TestNG Suite.


    This is such a pain, but when you really want to see things nicely, its worth it. The set up mostly has to be done once. You'll still have to clean your project and compile through Ant each time. Most of the time, its easier just to run through Ant, but like I said, sometimes its worth it.

  5. Last night I integrated TestNG support into ScalaTest, which is a new Scala unit test framework I'm working on with Bill Venners. We're planning a release with that support very soon.

    With ScalaTest you can write TestNG code the way you do today, and run your tests through the ScalaTest GUI. You can also write regular ScalaTest tests and run them together with the TestNG tests.

    As the Scala Eclipse environment improves, I'm sure ScalaTest will get good support there too. I don't see TestNG getting any real backing for Scala and that's unfortunate.

    To get ScalaTest visit

  6. Thanks for your answer. The ScalaTest solution is nice, but being able to run the tests in the TestNG runner would even be nicer - it makes a nicer argument for adoption if you can say, "Look, we can keep using our test framework and run the tests the same way we do with Java". I looked into it a bit more, and changing value for the target dropdown in the Eclipse Scala project preferences doesn't seem to make any differences. Even setting it to cldc or msil, the output file always looks the same to me (although I didn't diff it).

    As an experiment, I created a java project pointing to the bin directory of the scala project as part of the classpath. When I do the following (MyEnumTest is my compiled, annotated scala test class):

    Class cls = MyEnumTest.class;
    √Ānnotation[] declaredAnnotations = cls.getDeclaredAnnotations();
    Annotation[] annotations = cls.getAnnotations();

    ... both declaredAnnotations and annotations are empty. But, when I drag the class file to the Eclipse editor, I see:

    // Compiled from MyEnumTest.scala (version 1.1 : 45.3, no super bit)
    public class test.MyEnumTest implements scala.ScalaObject {

    ... i.e., there's something about the annotation there.

    All in all, I'm pretty confused. From the behaviour, I would say the target option in the plugin is not evaluated correctly, and the annotations are not written. But then, why do I see @org.testng.annotations.Test in the classfile? But if this is the correct (complete) way to write out the annotation, why doesn't the reflection api pick it up?

    ?!? Ideas, anybody? ?!?

  7. I'm seeing pretty much the same behavior, if I'm understanding you correctly. It is quite confusing. Unfortunately we appear to be the only ones doing TestNG in Scala, so I'm not sure if anyone is going to have ideas.

    I like your idea of running ScalaTest tests in the TestNG UI. I talked a bit on the Artima post about having a pluggable xUnit UI, but no one responded on that. I agree that its really important to be able to run your unit tests in the IDE the same way you do today. I personally think that the Scala unit test framework that gets IDE integration first is going to become the market leader.

    Once a ScalaTest IDE plugin comes, the change from TestNG to ScalaTest should be easy.

  8. Hi again,

    at least I understand the cause of the problem a bit better now. The class file compiled from scala is written out with major.minor bytecode version number 45.3 (corresponding to Java 1.1 compatibility) instead of 49.0 (corresponding to Java 5.0 compatibility), regardless of the setting in the plugin preferences. When I go in with a hex editor and change these two numbers, cls.get(Declared)Annotations picks up the annotations correctly. In other words, the annotations are there, but they are not picked up when loading the class because the class file format version number is wrong.

    After manually hacking the version numbers with the hex editor, I can also run this test via xml test suite directly from the eclipse TestNG plugin. So these two numbers are the only thing standing in the way of being able to use TestNG as is directly in Scala and the Eclipse TestNG plugin.

    Re: "I like your idea of running ScalaTest tests in the TestNG UI." - Actually, what I really want is write and run TestNG tests in scala, not run ScalaTest in the TestNG UI. I think being able to use an established, feature-rich test framework like TestNG would be a strong selling point for Scala, especially for mixed Scala/Java environments.

    I'll try to find out on the mailing lists if anybody knows anything about this bug and if/when it's going to be fixed.

  9. Wow! That's a really impressive find, my friend.

    The Scala Eclipse plugin is open source, as far as I know. Maybe you can take a look at how its using the compiler. When you use scalac via Ant this problem doesn't happen. If you can't look into it let me know and eventually I'll try, but I have a lot on my plate.

    I also think its important to be able to use TestNG in Scala, its what I started with initially. In fact, I've done quite a bit of work with it. I think though, that a framework that helps you write tests in a more Scalalike/functional way AND lets you still write tests the way you want AND still lets you run the old unit tests you've already written is more important. This is why I made the jump to start helping with ScalaTest.

  10. Phouk (Falk Bruegmann)January 28, 2008 12:19 PM

    I posted a message to - we'll see what they say.

    About working on the plugin: Probably not so easy at the moment. From reading the mailing archives of that same scala-tools list, I gather that the work on the plugin involves heavy hacking of the Scala compiler to add incremental behaviour to it, and that the plugin is undergoing major changes at the moment. But better look for yourself...

    About ScalaTest: Right, I see your point now. Certainly would be great to have that option as well.

  11. Phouk (Falk Bruegmann)January 28, 2008 9:12 PM

    Hi Jack, now I tried with ant. When setting the "target" attribute to "jvm-1.5" and setting the "force" attribute to "always", ant is working reliably for me. (The setup is with a separate eclipse project as described above.)

    I guess this is a workable workaround until the new, hopefully fixed scala plugin is out (which might be a while).

  12. [added the comment left on TheServerSide on January 11, 2008]

    Hi Jack,

    Maybe you would like review the library I created: specs

    It uses Scala features to make your tests more readable (in my mind,...). It also integrates Scalacheck which is an incredible tool to add to your unit testing toolbox.

    With specs, you can run the tests as JUnit tests so there is a minimal integration with the IDE.

    With some issues though:
    -the current Eclipse plugin doesn't support Scala 2.6.1 (this causes some RuntimeExceptions)
    -you have to create a separate Java project to run the tests (see the wiki)
    -you can't jump to the code from a test

    Following your post, I'll investigate the possibility of piggy-backing on TestNG runners to see if I can provide a better integration to the IDEs.

    Please have a look at specs, I would be interested by your feedback.