Clickcheck and Peckcheck

Libraries for automatic specification-based testing. They're both incomplete compared to the original QuickCheck (the Lisp comes closer than the Python).

Peckcheck

This is like Python's unittest module, except that test methods may take arguments. You must declare each argument with a default value which names a test-data generator. The peckcheck module will then peck at your test methods with a bunch of generated values. Sample usage:
    from peckcheck import TestCase, an_int, main

    class TestArithmetic(TestCase):
        def testAddCommutes(self, x=an_int, y=an_int):
            assert x + y == y + x
        def testAddAssociates(self, x=an_int, y=an_int, z=an_int):
            assert x + (y + z) == (x + y) + z

    if __name__ == '__main__':
        main()

You can create a test-data generator of your own by defining a function with one parameter, the size bound. For example:

    def a_weekday(size):
        import random
        return random.choice(['Sun','Mon','Tue','Wed','Thu','Fri','Sat'])

Clickcheck

This also offers random testing like the above, but within its own testing framework, because the existing Lisp testing libraries weren't quite to my taste. One aim of this framework is to be easy to get started with -- here, we load the package and use it to test that 2+3=5:
  CL-USER> (load "clickcheck.lisp")
  CL-USER> (use-package :click-check)
  CL-USER> (click-check (is= (+ 2 3) 5))
  Starting tests with seed 
  #S(RANDOM-STATE
     #*0110000000100101111101000101000111001100101100111001010111011101)
  .
  1 test submitted; all passed.
  T

Why display the seed? For repeatability when we run random test cases:

  CL-USER> (click-check (for-all ((n an-integer))
                          (is= (* 2 n) (+ n n))))
  Starting tests with seed 
  #S(RANDOM-STATE
     #*0110000000100101111101000101000111001100101100111001010111011101)
  ...............................................................................
  .....................
  1 test submitted; all passed.
  T

The line of dots shows a bunch of successful test cases, one dot for each. The summary line "1 test submitted" means the same single test was done on all those random values of N. We could have done more tests:

  CL-USER> (click-check (for-all ((n an-integer))
                          (is= (* 2 n) (+ n n))
                          (is= (* 3 n) (+ n n n)))
                        (for-all ((m an-integer) (n an-integer))
                          (is= (+ m n) (+ n m))))
  Starting tests with seed 
  #S(RANDOM-STATE
     #*1101011110111110101110111111111010110000111011101101010000100110)
  ...............................................................................
  ...............................................................................
  ...............................................................................
  ...............................................................
  3 tests submitted; all passed.
  T

In general, (CLICK-CHECK expression...) evaluates the argument expressions and reports on any tests performed in that dynamic extent. This means you can put the tests off in functions or files or wherever; typical usage is then like (click-check (my-test-suite)) or (click-check (load "tests1.lisp") (load "tests2.lisp")).

Let's see what failure looks like:

  CL-USER> (click-check (for-all ((n an-integer))
                          (test (evenp n))))
  Starting tests with seed 
  #S(RANDOM-STATE
     #*1011111110011110110110101000010101100100101110101010000011111001)
  .X
  FAIL (TEST (EVENP N))
    for ((N -3))
    1/2 counterexamples.
  1 test submitted; 1 FAILED.
  NIL

Now along with the dot for a passing test case, there's an X for a failing one. Testing continues after a failure unless *BREAK-ON-FAILURE* is true; the reason this test run stopped early was because FOR-ALL exits early once all the tests in its body have failed, on the grounds that finding more counterexamples is probably a waste of effort.

You can define test-data generators for your own types of data:

  CL-USER> (defparameter a-color 
             (lambda () 
               (pick-weighted 
                 (1 'red)
                 (2 'green)
                 (1 'blue))))
  A-COLOR
  CL-USER> (click-check (for-all ((c a-color))
                          (test (symbolp c))))
  Starting tests with seed 
  #S(RANDOM-STATE
     #*1111010101010001100010110010000111111000110000001110000100100111)
  ...............................................................................
  .....................
  1 test submitted; all passed.
  T

So far we've been running the tests all together in one sequence of expressions, with any side effects in one test potentially interfering with the next. Much of the time this is fine, since Lisp encourages a mostly-functional style; but for the sake of other styles we can isolate tests using WRAP-EACH:

  CL-USER> (defun x-tests ()
             (wrap-each (let ((x (set-up-an-x)))
                          (unwind-protect WRAPPEE
                            (tear-down-an-x x)))
               (test (foo x))
               (test (bar x))))
which works as if we'd written
  CL-USER> (defun x-tests ()
             (let ((x (set-up-an-x)))
               (unwind-protect (test (foo x))
                 (tear-down-an-x x)))
             (let ((x (set-up-an-x)))
               (unwind-protect (test (bar x))
                 (tear-down-an-x x))))

That's the gist of this package; the manual and examples cover more features, conveniences, and details.

Related work

History

Around 1998 I was trying out test-first development in Scheme and felt that my first attempts took too much work. One obvious improvement was to give type declarations for my functions and generate tests of type-correctness from them automatically. Since Scheme has no static type system, you can use the same framework for tests that don't correspond to ordinary types, using 'types' like prime numbers. From there it's a small step to fancier specification-based tests like
(for-all ((x <set>) (y <set>))
  (set-equal (union x y) (union y x)))

That was as far as I got; QuickCheck had the same basic idea and added conditional laws, histograms of test data distribution, and randomly generated functions. Plus, they published! Meanwhile I just drifted away from this out of too much sloth to maintain fine-grained test suites in my free time. Now that Extreme Programming has gotten so popular I figure it's past time to dust this stuff off and give it another go, trying to build on the work others have done.


Home   |   © 1994-2004 by Darius Bacon