Tests and Booleans and Matchers

I was working with the new ScalaTest matchers API recently when I came across something that made me think (I usually don't think). I had something simple like:

Nil.empty must be(true)

This seems harmless enough, but when I made it fail intentionally the error message I got was something reasonably similar to:

Expected true but got false.

This obviously isn't a very good error message, and I was thinking at first the maybe we should provide the user API to provide their own error message. Something like so:

Nil.empty must be(true, "expected Nil to be empty, but it wasn't!")

This also seems harmless enough. However, I talked this over with Bill Venners and in this case there is a way to do it using symbols and reflection that gives better error messages:

Nil must be('empty)

If this fails, it will give a nice error message like:

Nil was not empty

This is ok, but type safety people will scream, and they'd probably be right. And, does this work in all cases and can we always get good error messages? I think no. Let me explain why.

What if we have an object that has a method that takes an argument and returns a boolean like this theoretical surroundedBy method that could exist on String:

def surroundedBy( s: String ) = startsWith(s) && endsWith(s)

We can't use the symbol form here because the method takes a parameter. So what can we do? If we go back to the original form then we are back to our original bad error message.

"htth".surroundedBy( "h" ) must be (true)
"http".surroundedBy( "h" ) must be (false)

Do we have any other options? Yes (with caveats as well). But first lets think about what we'd really like to write here. If we were writing pure English, it would be this:

"htth" must be surrounded by "h"
"http" must not be surrounded by "h"

Matchers

ScalaTest 0.9.5 came out with a new Matchers API that I'll touch on briefly here, and point you to documentation for more. Using matchers, we can get our client code pretty close to this. We'll have to define our custom matcher first:

case class surroundedBy(r:String) extends BeMatcher[String] {
  def apply(l: String) =
    MatchResult(
      l.surroundedBy(r),
      l + " was not surrounded by " + r,
      l + " was surrounded by " + r
    )
}

And our client test code turns out to be pretty close to the English:

"htth" must be(surroundedBy("h"))
"http" must not be surroundedBy("h")

I like this a lot, but, only if its going to be used a lot. Otherwise, if it's a one off test, I could just kick it back way old school and use an assert with an error message:

assert( "htth".surroundedBy("h") === true,
     "expected http to be surrounded by h, but it wasnt." )
assert( "http".surroundedBy("h") === false,
     "expected http not to be surrounded by h, but it was!!!" )

This is less readable code, but it is less code - about 6 lines less when you include the custom matcher. You get more or less the same error message in this case as well. It is quite ugly though. In this case I'd much rather have an API that just gives me what I originally wanted:

"htth".surroundedBy("h") must be(true,
 "expected htth to be surrounded by h, but it wasnt.")

or better:

"htth" must be('surroundedBy, "h")

I could see this one existing easily. It's really nice, but not type safe.

Anyway, maybe there already are better alternatives like the ones I've explained I simply don't know about. Maybe I haven't done my homework and I should be scolded. I really hope not, and I'll happily write up (and more importantly - use) anything I find that helps make this code more readable. Until alternatives surface, I guess I'll somewhat reluctantly stay with these rules:

When testing methods that return booleans you can

• Use the symbol style ('empty) if the method takes no arguments and you aren't terrible concerned about type safety.


• Use old school assertions when the method takes arguments but you won't have a lot of repetitive tests for the method.


• Use a custom matcher otherwise

You'll also find that ScalaTest (and Specs which is unfortunately not covered here) provides many matchers that can use for many common types.

In my next post I'll provide a much deeper overview of how to write your own custom matchers in ScalaTest, starting by explaining what that surroundedBy case class means.

In the meantime, you can look at ScalaTest Matchers and Specs Matchers.

Here's all the custom matcher code:

package org.scalatest.examples
 
import org.scalatest.matchers.MustMatchers._  
import org.scalatest.matchers._


class MatcherTest extends FunSuite with MustMatchers{
 
  implicit def surroundable[T](l:String) = new {
    def surroundedBy(r:String) = l.startsWith(r) && l.endsWith(r)
  }
  
  case class surroundedBy(r:String) extends BeMatcher[String] {
    def apply(l: String) = 
      MatchResult(
        l.surroundedBy(r), 
        l + " was not surrounded by " + r, 
        l + " was surrounded by " + r
      )
  }
  
  test("weve got the place surrounded"){
    "htth" must be(surroundedBy("h"))
    "http" must not be surroundedBy("t")
  }
}