Unit Testing

Testing Base Julia

Julia is under rapid development and has an extensive test suite to verify functionality across multiple platforms. If you build Julia from source, you can run this test suite with make test. In a binary install, you can run the test suite using Base.runtests().

Base.runtests — Function.

runtests([tests=["all"] [, numcores=ceil(Int, Sys.CPU_CORES / 2) ]])

Run the Julia unit tests listed in tests, which can be either a string or an array of strings, using numcores processors. (not exported)

source

Basic Unit Tests

The Base.Test module provides simple unit testing functionality. Unit testing is a way to see if your code is correct by checking that the results are what you expect. It can be helpful to ensure your code still works after you make changes, and can be used when developing as a way of specifying the behaviors your code should have when complete.

Simple unit testing can be performed with the @test() and @test_throws() macros:

Base.Test.@test — Macro.

@test ex
@test f(args...) key=val ...

Tests that the expression ex evaluates to true. Returns a PassResult if it does, a FailResult if it is false, and an ErrorResult if it could not be evaluated.

The @test f(args...) key=val... form is equivalent to writing @test f(args..., key=val...) which can be useful when the expression is a call using infix syntax such as approximate comparisons:

@test a ≈ b atol=ε

This is equivalent to the uglier test @test ≈(a, b, atol=ε). It is an error to supply more than one expression unless the first is a call expression and the rest are assignments (k=v).

source

Base.Test.@test_throws — Macro.

@test_throws extype ex

Tests that the expression ex throws an exception of type extype. Note that @test_throws does not support a trailing keyword form.

source

For example, suppose we want to check our new function foo(x) works as expected:

julia> using Base.Test

julia> foo(x) = length(x)^2
foo (generic function with 1 method)

If the condition is true, a Pass is returned:

julia> @test foo("bar") == 9
Test Passed
  Expression: foo("bar") == 9
   Evaluated: 9 == 9

julia> @test foo("fizz") >= 10
Test Passed
  Expression: foo("fizz") >= 10
   Evaluated: 16 >= 10

If the condition is false, then a Fail is returned and an exception is thrown:

julia> @test foo("f") == 20
Test Failed
  Expression: foo("f") == 20
   Evaluated: 1 == 20
ERROR: There was an error during testing
 in record at test.jl:268
 in do_test at test.jl:191

If the condition could not be evaluated because an exception was thrown, which occurs in this case because length() is not defined for symbols, an Error object is returned and an exception is thrown:

julia> @test foo(:cat) == 1
Error During Test
  Test threw an exception of type MethodError
  Expression: foo(:cat) == 1
  MethodError: `length` has no method matching length(::Symbol)
   in foo at none:1
   in anonymous at test.jl:159
   in do_test at test.jl:180
ERROR: There was an error during testing
 in record at test.jl:268
 in do_test at test.jl:191

If we expect that evaluating an expression should throw an exception, then we can use @test_throws() to check that this occurs:

julia> @test_throws MethodError foo(:cat)
Test Passed
  Expression: foo(:cat)
   Evaluated: MethodError

Working with Test Sets

Typically a large number of tests are used to make sure functions work correctly over a range of inputs. In the event a test fails, the default behavior is to throw an exception immediately. However, it is normally preferable to run the rest of the tests first to get a better picture of how many errors there are in the code being tested.

The @testset() macro can be used to group tests into sets. All the tests in a test set will be run, and at the end of the test set a summary will be printed. If any of the tests failed, or could not be evaluated due to an error, the test set will then throw a TestSetException.

Base.Test.@testset — Macro.

@testset [CustomTestSet] [option=val  ...] ["description"] begin ... end
@testset [CustomTestSet] [option=val  ...] ["description $v"] for v in (...) ... end
@testset [CustomTestSet] [option=val  ...] ["description $v, $w"] for v in (...), w in (...) ... end

Starts a new test set, or multiple test sets if a for loop is provided.

If no custom testset type is given it defaults to creating a DefaultTestSet. DefaultTestSet records all the results and, if there are any Fails or Errors, throws an exception at the end of the top-level (non-nested) test set, along with a summary of the test results.

Any custom testset type (subtype of AbstractTestSet) can be given and it will also be used for any nested @testset invocations. The given options are only applied to the test set where they are given. The default test set type does not take any options.

The description string accepts interpolation from the loop indices. If no description is provided, one is constructed based on the variables.

By default the @testset macro will return the testset object itself, though this behavior can be customized in other testset types. If a for loop is used then the macro collects and returns a list of the return values of the finish method, which by default will return a list of the testset objects used in each iteration.

source

We can put our tests for the foo(x) function in a test set:

julia> @testset "Foo Tests" begin
           @test foo("a")   == 1
           @test foo("ab")  == 4
           @test foo("abc") == 9
       end
Test Summary: | Pass  Total
Foo Tests     |    3      3

Test sets can also be nested:

julia> @testset "Foo Tests" begin
           @testset "Animals" begin
               @test foo("cat") == 9
               @test foo("dog") == foo("cat")
           end
           @testset "Arrays $i" for i in 1:3
               @test foo(zeros(i)) == i^2
               @test foo(ones(i)) == i^2
           end
       end
Test Summary: | Pass  Total
Foo Tests     |    8      8

In the event that a nested test set has no failures, as happened here, it will be hidden in the summary. If we do have a test failure, only the details for the failed test sets will be shown:

julia> @testset "Foo Tests" begin
           @testset "Animals" begin
               @testset "Felines" begin
                   @test foo("cat") == 9
               end
               @testset "Canines" begin
                   @test foo("dog") == 9
               end
           end
           @testset "Arrays" begin
               @test foo(zeros(2)) == 4
               @test foo(ones(4)) == 15
           end
       end

Arrays: Test Failed
  Expression: foo(ones(4)) == 15
   Evaluated: 16 == 15
 in record at test.jl:297
 in do_test at test.jl:191
Test Summary: | Pass  Fail  Total
Foo Tests     |    3     1      4
  Animals     |    2            2
  Arrays      |    1     1      2
ERROR: Some tests did not pass: 3 passed, 1 failed, 0 errored, 0 broken.
 in finish at test.jl:362

Other Test Macros

As calculations on floating-point values can be imprecise, you can perform approximate equality checks using either @test a ≈ b (where ≈, typed via tab completion of \approx, is the isapprox() function) or use isapprox() directly.

julia> @test 1 ≈ 0.999999999

julia> @test 1 ≈ 0.999999
ERROR: test failed: 1 isapprox 0.999999
 in expression: 1 ≈ 0.999999
 in error at error.jl:21
 in default_handler at test.jl:30
 in do_test at test.jl:53

Base.Test.@inferred — Macro.

@inferred f(x)

Tests that the call expression f(x) returns a value of the same type inferred by the compiler. It is useful to check for type stability.

f(x) can be any call expression. Returns the result of f(x) if the types match, and an ErrorResult if it finds different types.

julia> using Base.Test

julia> f(a,b,c) = b > 1 ? 1 : 1.0
f (generic function with 1 method)

julia> typeof(f(1,2,3))
Int64

julia> @code_warntype f(1,2,3)
Variables:
  #self#::#f
  a::Int64
  b::Int64
  c::Int64

Body:
  begin
      unless (Base.slt_int)(1, b::Int64)::Bool goto 3
      return 1
      3:
      return 1.0
  end::UNION{FLOAT64, INT64}

julia> @inferred f(1,2,3)
ERROR: return type Int64 does not match inferred return type Union{Float64, Int64}
Stacktrace:
 [1] error(::String) at ./error.jl:21

julia> @inferred max(1,2)
2

source

Base.Test.@test_warn — Macro.

@test_warn msg expr

Test whether evaluating expr results in STDERR output that contains the msg string or matches the msg regular expression. If msg is a boolean function, tests whether msg(output) returns true. If msg is a tuple or array, checks that the error output contains/matches each item in msg. Returns the result of evaluating expr.

See also @test_nowarn to check for the absence of error output.

source

Base.Test.@test_nowarn — Macro.

@test_nowarn expr

Test whether evaluating expr results in empty STDERR output (no warnings or other messages). Returns the result of evaluating expr.

source

Broken Tests

If a test fails consistently it can be changed to use the @test_broken() macro. This will denote the test as Broken if the test continues to fail and alerts the user via an Error if the test succeeds.

Base.Test.@test_broken — Macro.

@test_broken ex
@test_broken f(args...) key=val ...

Indicates a test that should pass but currently consistently fails. Tests that the expression ex evaluates to false or causes an exception. Returns a BrokenResult if it does, or an ErrorResult if the expression evaluates to true.

The @test_broken f(args...) key=val... form works as for the @test macro.

source

@test_skip() is also available to skip a test without evaluation, but counting the skipped test in the test set reporting. The test will not run but gives a BrokenResult.

Base.Test.@test_skip — Macro.

@test_skip ex
@test_skip f(args...) key=val ...

Marks a test that should not be executed but should be included in test summary reporting as Broken. This can be useful for tests that intermittently fail, or tests of not-yet-implemented functionality.

The @test_skip f(args...) key=val... form works as for the @test macro.

source

Creating Custom `AbstractTestSet` Types

Packages can create their own AbstractTestSet subtypes by implementing the record and finish methods. The subtype should have a one-argument constructor taking a description string, with any options passed in as keyword arguments.

Base.Test.record — Function.

record(ts::AbstractTestSet, res::Result)

Record a result to a testset. This function is called by the @testset infrastructure each time a contained @test macro completes, and is given the test result (which could be an Error). This will also be called with an Error if an exception is thrown inside the test block but outside of a @test context.

source

Base.Test.finish — Function.

finish(ts::AbstractTestSet)

Do any final processing necessary for the given testset. This is called by the @testset infrastructure after a test block executes. One common use for this function is to record the testset to the parent's results list, using get_testset.

source

Base.Test takes responsibility for maintaining a stack of nested testsets as they are executed, but any result accumulation is the responsibility of the AbstractTestSet subtype. You can access this stack with the get_testset and get_testset_depth methods. Note that these functions are not exported.

Base.Test.get_testset — Function.

get_testset()

Retrieve the active test set from the task's local storage. If no test set is active, use the fallback default test set.

source

Base.Test.get_testset_depth — Function.

get_testset_depth()

Returns the number of active test sets, not including the defaut test set

source

Base.Test also makes sure that nested @testset invocations use the same AbstractTestSet subtype as their parent unless it is set explicitly. It does not propagate any properties of the testset. Option inheritance behavior can be implemented by packages using the stack infrastructure that Base.Test provides.

Defining a basic AbstractTestSet subtype might look like:

import Base.Test: record, finish
using Base.Test: AbstractTestSet, Result, Pass, Fail, Error
using Base.Test: get_testset_depth, get_testset
struct CustomTestSet <: Base.Test.AbstractTestSet
    description::AbstractString
    foo::Int
    results::Vector
    # constructor takes a description string and options keyword arguments
    CustomTestSet(desc; foo=1) = new(desc, foo, [])
end

record(ts::CustomTestSet, child::AbstractTestSet) = push!(ts.results, child)
record(ts::CustomTestSet, res::Result) = push!(ts.results, res)
function finish(ts::CustomTestSet)
    # just record if we're not the top-level parent
    if get_testset_depth() > 0
        record(get_testset(), ts)
    end
    ts
end

And using that testset looks like:

@testset CustomTestSet foo=4 "custom testset inner 2" begin
    # this testset should inherit the type, but not the argument.
    @testset "custom testset inner" begin
        @test true
    end
end

Unit Testing

Testing Base Julia

Basic Unit Tests

Working with Test Sets

Other Test Macros

Broken Tests

Creating Custom AbstractTestSet Types

Creating Custom `AbstractTestSet` Types