SBT and Test Arguments

If you want to pass arguments to tests in SBT, you've come to the right place. First know this - it only works in certain situations, it's a bit crufty, and its subject to change. But I'll try to keep this page up to date if it does change.

Another important note, if you want to pass arguments at the sbt command line, you can only do it for test-quick, and test-only (I think). It doesn't work for the default "test" command. This is a bit unfortunate, but there is a workaround (not a very good one, but at least it exists).

A final note: these examples are for ScalaTest and ScalaCheck. I don't yet have examples for Specs, but Eric has implemented argument handling (in 1.6.2.1-SNAPSHOT for scala 2.7.7 and 1.6.4-SNAPSHOT for scala 2.8.0.Beta1).

Passing Args to ScalaTest from the Command Line

Let's say you have this little ScalaTest class that uses both params and tags:

import org.scalatest.fixture.FixtureFunSuite
import org.scalatest.Tag

class WickedCoolTest extends FixtureFunSuite{
 type FixtureParam = Map[String,Any]
 override def withFixture(test: OneArgTest) {
   test(test.configMap)
 }
 test("1", Tag("dood1")){ conf => println("dood1: " + conf) }
 test("2", Tag("dood2")){ conf => println("dood2: " + conf) }
 test("3", Tag("dood3")){ conf => println("dood3: " + conf) }
}

To run all tests in in this class use:

test-only WickedCoolTest

To run only tests tagged with "dood1" use:

test-only WickedCoolTest -- -n dood1

To run tests tagged with dood1 or dood2, use:

test-only WickedCoolTest -- -n "dood1 dood2"

To run all tests except for tests tagged with dood2, run:

test-only WickedCoolTest -- -l dood2

To run all tests except for tests tagged with dood2 dood3, run:

test-only WickedCoolTest -- -l "dood2 dood3"

To pass configuration parameters to a test, use:

test-only WickedCoolTest -- -Danswer=42

To pass many config params, add more -D's:

test-only WickedCoolTest -- -Danswer=42 -DrealAnswer=54

Tags and parameters can be used in combination, like so:

test-only WickedCoolTest -- -n "dood dood2" -Dhey=you -Dm=f

This runs only tests tagged with dood or dood2 and produces the following output:

dood: Map(hey -> you, m -> f)
dood2: Map(hey -> you, m -> f)

Passing Args to ScalaCheck from the Command Line

I'll spare you the entire example and just get to the point here.

To set the minimum number of successful tests, use '-s'

> test-only *IntParallelArrayCheck -- -s 5000
...
[info] == scala.collection.parallel.mutable.IntParallelArrayCheck ==
[info] OK, passed 5000 tests.

The rest of the examples are very similar. Consult ScalaCheck itself for further documentation.

• -s (minSuccessfulTests): Number of tests that must succeed in order to pass a property
  
• -d (maxDiscardedTests): Number of tests that can be discarded before ScalaCheck stops testing a property
  
• -n (minSize): Minimum data generation size
  
• -x (maxSize): Maximum data generation size
  
• -w (workers): Number of threads to execute in parallel for testing
  
• -z (wrkSize): Amount of work each thread should do at a time
  

Default Arguments

Maybe you want to pass default arguments to your test framework of choice. That is, you want to pass the same arguments every time you run your tests (and still be able to pass more dynamically, if you wish). You can do this too, by adding elements to 'testOptions'. Doing so will take effect when you run sbt test, sbt test-only, sbt test-quick, etc.

Let's say you want the minimum number of ScalaCheck passing tests to be 5000 every time you run ScalaCheck. Here is how you do that:

override def testOptions = 
  super.testOptions ++ 
  Seq(TestArgument(TestFrameworks.ScalaCheck, "-s", "5000")) 

Or maybe you only ever want to run your fast ScalaTest tests.

override def testOptions = 
  super.testOptions ++ 
  Seq(TestArgument(TestFrameworks.ScalaTest, "-n", "fast")) 

Custom Test Tasks

Maybe you have some test that you run all the time. At the sbt command line, instead of saying > test-quick blah.blah.Blah, you just want to say: > blah.

You can do that too, and you can pass args to it dynamically, and you get the default arguments as well. You'll have to take the following code and put it into your sbt file. It's ugly, I know, but the results are nice.

lazy val blah = singleTestTask("blah.blah.Blah")
 
private def singleTestTask(className: String) = task { args =>
  defaultTestTask(TestFilter(_ == className) :: 
   testOptions.toList ::: ScalaTestArgs(args))
}

private def newScalaTestArg(l: String*) = 
  TestArgument(TestFrameworks.ScalaTest, l:_*)

private def ScalaTestArgs(args: Seq[String]): List[TestArgument] = {
  def KVArgs(args: Seq[String]): TestArgument = 
    newScalaTestArg(args.map("-D" + _):_*)
  def tagsFromArgs(tags: Seq[String]): List[TestArgument] = {
    if (tags.isEmpty) Nil else 
      List(newScalaTestArg("-n", tags.mkString(" ")))
  }
  val (kvs, tags) = args.partition(_.contains("="))
  KVArgs(kvs.toSeq) :: tagsFromArgs(tags.toSeq)
}

Subclasses

Finally, maybe you want to set up tasks like 'test-fast' and 'test-slow' which only run your fast and slow tests. Let's imagine that you've created a trait called blah.blah.SlowTest and all of your slow tests extend that trait. Tests that don't extend SlowTest are considered to be in the fast group. With the code below, at the sbt command line you'll be able to say > test-fast, and > test-slow.

Again, it's a bit ugly to put this stuff in your sbt project file, but it works for now, until I come up with a better plan :)

lazy val testSlow = 
  runSubclassesOf("org.nlogo.util.SlowTest")
lazy val testFast = 
  runEverythingButSubclassesOf("org.nlogo.util.SlowTest")

private def runSubclassesOf(className: String) = {
  val subclass: Boolean => Boolean = x => x
  subclassTest(className, subclass)
}
  
private def runEverythingButSubclassesOf(className: String) = {
  val notSubclass: Boolean => Boolean = x => ! x
  subclassTest(className, notSubclass)
}

private def subclassTest(className: String, 
                         subclassCheck: Boolean => Boolean) = task { 
  args =>
  lazy val jars = 
    testClasspath.get.toList.map(_.asURL).toArray[java.net.URL]
  lazy val loader = 
    new java.net.URLClassLoader(jars,buildScalaInstance.loader)
  def clazz(name: String) = Class.forName(name, false, loader)
  lazy val superClass = clazz(className)
  def filter = 
    TestFilter(c => 
      subclassCheck(superClass.isAssignableFrom(clazz(c))))
  defaultTestTask
    (filter :: testOptions.toList ::: ScalaTestArgs(args))
}

Yeah...

Let me know if you have any issues with any of this, I'll be happy to help. I put it together pretty quickly. If there are any glaring errors or omissions, please tell.