ScalaTest and TestNG

THIS IS AN INCOMPLETE DRAFT, POSTED FOR REVIEW. I UNDERSTAND SOME SECTIONS NEED WORK AND SOME SECTIONS ARE EMPTY OR MISSING ENTIRELY, AND THAT THE FORMAT MIGHT BE MESSED UP.

ScalaTest has two important goals.

1. Allow tests to be written in Scala easily, and concisely.
   
2. Allow Java developers to transition to ScalaTest with minimal effort.

With these two goals in mind I'm happy to announce ScalaTest's integration with TestNG. This integration offers two features that meet the goals of ScalaTest.

1. TestNG tests can be written in Scala, and run in both ScalaTest and TestNG runners.
   
2. Existing TestNG tests can be run in ScalaTest.

By being able to write new TestNG tests in Scala, a developer doesn't have the overhead of learning a new test framework and a new language at once. And more importantly, by being able to run existing test suites in ScalaTest developers can feel confident that all their code is working without the overhead of running two test frameworks at once.

This article will show you how to use ScalaTest to do both, with the help of an example (available for download from the ScalaTest Subversion repository). The example has been tested against Scala 2.7.0, and TestNG 5.7. To use the example you'll also need to download the latest version of ScalaTest.

After downloading everything, you'll need to place the jars into the lib directories of the example. Place the TestNG jar into the java/lib, and place the Scala related jars into the scala/lib folder. You should end up with the following directory structure (as shown here in Eclipse).

About the Example

The example contains what we'll refer to as "existing" code (Java), and "new" code (Scala). The thinking here is that you are working on an existing project in Java, you have a Java code base complete with unit tests (you do have unit tests, don't you?), and that you're interested in exploring Scala. If you're Java code isn't covered with tests the content here is still relevant; you can learn how to write TestNG tests in Scala.

The existing code lives in the java folder where you'll find:

• VolumeKnob.java - An interface for volume knobs
• BoringVolumeKnob.java - A boring implementation of VolumeKnob
• BoringVolumeKnobTest.java - A boring test for BoringVolumeKnob
• volume-tests.xml - A TestNG XML suite to run BoringVolumeKnobTest
• build.xml - Ant file that builds the code and runs the tests
  

The new code is in the scala folder, and it builds upon the existing Java code. In the scala folder you'll find:

• AwesomeVolumeKnob.java - A totally awesome implementation of VolumeKnob
• AwesomeVolumeKnobTest.java - An awesome test for AwesomeVolumeKnob
• build.xml - Ant file that builds the code and has targets to
  
• Run just the existing Java tests in ScalaTest
  
• Run just the Scala tests in ScalaTest
• Run both the Java and Scala tests in ScalaTest
  

Quick Look At What Needs To Be Tested

In the next section we're going to learn how to write TestNG tests in ScalaTest. But before we do, lets take a quick look at what we're going to test. Recall in the java folder the interface VolumeKnob. It's very simple:

public interface VolumeKnob {
    public abstract int currentVolume();
    public abstract int maxVolume();
    public abstract void turnUp();
    public abstract void turnDown();
}

VolumeKnob has two implementations that need testing. The first is a rather boring Java implementation - BoringVolumeKnob. It's too boring to show here, and it can't be turned up beyond 10. It has a corresponding TestNG test class also written in Java - BoringVolumeKnobTest. That won't be shown here either, since we'll assume you know TestNG.

There is also a Scala implementation of that interface, AwesomeVolumeKnob. AwesomeVolumeKnob's have three amazing qualities:

• They always go to at least 11 (of course)
• They can never be turned down
• They can always be turned up, regardless of the max volume.

import org.scalatest.legacy.VolumeKnob

class AwesomeVolumeKnob( val maxVolume: int ) extends VolumeKnob {
  if( maxVolume < 11 ) 
    throw new IllegalArgumentException("...These go to eleven.");
  
  var currentVolume = maxVolume;
  
  def turnDown = 
    throw new IllegalAccessError("AwesomeVolumeKnobs cannot be turned down");
  
  // AwesomeVolumeKnobs don't care about max volume
  def turnUp = currentVolume = currentVolume + 1;
}

AwesomeVolumeKnob is accompanied by AwesomeVolumeKnobTest, which is a TestNG test written in Scala. We'll cover that next.

Writing TestNG tests in ScalaTest

Because Scala allows you to use Java's annotations, TestNG tests can be written in Scala at least as easily as they can in Java. Here is a quick example:

import org.scalatest.legacy.VolumeKnob

class AwesomeVolumeKnobTest{
  @Test
  def awesomeVolumeKnobsCanBeTurnedUpReallyHigh(){
    val v = new AwesomeVolumeKnob(10000)
    for( i <- 1 to 1000 ) v.turnUp
  }
}

There's really nothing to it. This class can be compiled by the Scala compiler and run in any TestNG runner.

To enable your test to be run in ScalaTest, simply extend the ScalaTest trait org.scalatest.testng.TestNGSuite. That's it. Here is the full implementation of AwesomeVolumeKnobTest in all its glory.

import org.scalatest.testng.TestNGSuite
import org.testng.annotations._

class AwesomeVolumeKnobTest extends TestNGSuite{

  @Test{ val description=
         "create AVK's with max volume < 11 and ensure IllegalArg is thrown" 
         val dataProvider="low volumes", 
         val expectedExceptions = Array( classOf[IllegalArgumentException] )}
  def awesomeVolumeKnobsAlwaysGoToAtLeastEleven(maxVolume: int){
    new AwesomeVolumeKnob(maxVolume);  
  }  
  
  @Test{ val description=
         "try to turn down some AVK's and ensure IllegalAccess is thrown"
         val dataProvider="high volumes", 
         val expectedExceptions = Array( classOf[IllegalAccessError] )}
  def awesomeVolumeKnobsCanNeverBeTurnedDown(maxVolume: int){
    new AwesomeVolumeKnob(maxVolume).turnDown();
  } 
  
  @Test{ val description="crank it up" }
  def awesomeVolumeKnobsCanBeTurnedUpReallyHigh(){
    val v = new AwesomeVolumeKnob(10000)
    for( i <- 1 to 1000 ) v.turnUp
  }
  
  @DataProvider{val name="high volumes"}
  def goodVolumes = 
    Array(v(11), v(20),v(30),v(40),v(50),v(60),v(70),v(80),v(90),v(100))
  
  @DataProvider{val name="low volumes"}
  def lowVolumes = 
    Array(v(1), v(2),v(3),v(4),v(5),v(6),v(7),v(8),v(9),v(10))
  
  def v( i: Integer ) = Array(i)
}

There are a few things to notice with this implementation.

Running Scala TestNG Tests in ScalaTest

Running TestNGSuite's in ScalaTest is no different than running any other ScalaTest Suite - simply use the Ant task. In scala/build.xml in the example, you'll find a target called "test-scala-only":

 <target name="test-scala-only" depends="compile">
  <taskdef name="scalatest" classname="org.scalatest.tools.ScalaTestTask" 
   classpathref="test.classpath"/>
  
  <scalatest>
   <runpath>
    <pathelement path="test.classpath"/>
    <pathelement location="${test.jar.file}"/>
   </runpath>
   
   <suite classname="org.scalatest.testng.AwesomeVolumeKnobTest"/>
   
   <reporter type="stdout" />
   <reporter type="graphic" />
   
  </scalatest>
 </target>

In this case we have just one Suite to run:

<suite classname="org.scalatest.testng.AwesomeVolumeKnobTest"/>

Running this task brings up the ScalaTest UI:

Running the Java Tests

If you're a TestNG user, you're likely to be familiar with running test suites from the IDE. There are a couple of different ways to do it, and whichever you choose, you're sure to end up like something like this (as shown in Eclipse):

While this is great, and familiar, and comfortable, it would be a pain to have to run your Java tests through TestNG's UI and then have to switch over to ScalaTests UI to run your Scala tests.

Running Java Tests in ScalaTest

ScalaTest provides a simple way to run TestNG xml suites in its Ant task. In scala/build.xml in the example, you'll find a target called "test-java-only":

 <target name="test-java-only" depends="compile">
  <taskdef name="scalatest" classname="org.scalatest.tools.ScalaTestTask" 
   classpathref="test.classpath"/>
  
  <scalatest>
   <runpath>
    <pathelement path="test.classpath"/>
    <pathelement location="${test.jar.file}"/>
   </runpath>
   
   <testNGSuites>
    <pathelement location="${java.dir}/src/test/volume-tests.xml"/>
   </testNGSuites>
   
   <reporter type="stdout" />
   <reporter type="graphic" />
   
  </scalatest>
 </target>

Inside the testNGSuites block, simply put the location of the xml suite.

  <testNGSuites>
    <pathelement location="${java.dir}/src/test/volume-tests.xml"/>
  </testNGSuites>

While in this case there's one xml suite only, ScalaTest supports multiple xml suites. All suites get run in the same TestNG instance. Running ScalaTest via Ant with the graphic reporter option brings up the ScalaTest UI:

As you can see, ScalaTest reported the exact same results as the TestNG Eclipse plugin. (But...notice that our green bar is much brighter!)

Running All The Tests Together

Finally, for the moment of truth...though being so easy, it's likely a bit of a letdown. To run all the tests together, simply use both options in the Ant task, as in the "test" target in scala/build.xml.

 <target name="test-java-only" depends="compile">
  <taskdef name="scalatest" classname="org.scalatest.tools.ScalaTestTask" 
   classpathref="test.classpath"/>
  
  <scalatest>
   <runpath>
    <pathelement path="test.classpath"/>
    <pathelement location="${test.jar.file}"/>
   </runpath>
   
   <suite classname="org.scalatest.testng.AwesomeVolumeKnobTest"/>

   <testNGSuites>
    <pathelement location="${java.dir}/src/test/volume-tests.xml"/>
   </testNGSuites>
   
   <reporter type="stdout" />
   <reporter type="graphic" />
   
  </scalatest>
 </target>

Running this task brings up the ScalaTest UI:

Problems

Summary