Test.Tasty.Runners

API for test runners

Working with the test tree

data TestTree

The main data structure defining a test suite.

It consists of individual test cases and properties, organized in named groups which form a tree-like hierarchy.

There is no generic way to create a test case. Instead, every test provider (tasty-hunit, tasty-smallcheck etc.) provides a function to turn a test case into a TestTree.

Groups can be created using testGroup.

Constructors

forall t.IsTest t => SingleTest TestName t

A single test of some particular type

TestGroup TestName [TestTree]

Assemble a number of tests into a cohesive group

PlusTestOptions (OptionSet -> OptionSet) TestTree

Add some options to child tests

forall a. WithResource (ResourceSpec a) (IO a -> TestTree)

Acquire the resource before the tests in the inner tree start and release it after they finish. The tree gets an IO action which yields the resource, although the resource is shared across all the tests.

AskOptions (OptionSet -> TestTree)

Ask for the options and customize the tests based on them

After DependencyType Expr TestTree

Only run after all tests that match a given pattern finish (and, depending on the DependencyType, succeed)

foldTestTree

Arguments

:: forall b. Monoid b

=> TreeFold b

the algebra (i.e. how to fold a tree)

-> OptionSet

initial options

-> TestTree

the tree to fold

-> b

Fold a test tree into a single value.

The fold result type should be a monoid. This is used to fold multiple results in a test group. In particular, empty groups get folded into mempty.

Apart from pure convenience, this function also does the following useful things:

1. Keeping track of the current options (which may change due to PlusTestOptions nodes)
2. Filtering out the tests which do not match the patterns

Thus, it is preferred to an explicit recursive traversal of the tree.

Note: right now, the patterns are looked up only once, and won't be affected by the subsequent option changes. This shouldn't be a problem in practice; OTOH, this behaviour may be changed later.

data TreeFold b

An algebra for folding a TestTree.

Instead of constructing fresh records, build upon trivialFold instead. This way your code won't break when new nodes/fields are indroduced.

Constructors

TreeFold

Fields

• foldSingle :: forall t. IsTest t => OptionSet -> TestName -> t -> b
• foldGroup :: OptionSet -> TestName -> b -> b
• foldResource :: forall a. OptionSet -> ResourceSpec a -> (IO a -> b) -> b
• foldAfter :: OptionSet -> DependencyType -> Expr -> b -> b

trivialFold :: Monoid b => TreeFold b

trivialFold can serve as the basis for custom folds. Just override the fields you need.

Here's what it does:

• single tests are mapped to mempty (you probably do want to override that)
• test groups are returned unmodified
• for a resource, an IO action that throws an exception is passed (you want to override this for runners/ingredients that execute tests)

data ResourceSpec a

ResourceSpec describes how to acquire a resource (the first field) and how to release it (the second field).

Constructors

ResourceSpec (IO a) (a -> IO ())

newtype Traversal f

Monoid generated by *>

Constructors

Traversal

Fields

• getTraversal :: f ()

Instances

Applicative f => Monoid (Traversal f)

Defined in Test.Tasty.Runners.Reducers

Methods

mempty :: Traversal f

mappend :: Traversal f -> Traversal f -> Traversal f

mconcat :: [Traversal f] -> Traversal f

Applicative f => Semigroup (Traversal f)

Defined in Test.Tasty.Runners.Reducers

Methods

(<>) :: Traversal f -> Traversal f -> Traversal f

sconcat :: NonEmpty (Traversal f) -> Traversal f

stimes :: Integral b => b -> Traversal f -> Traversal f

newtype Ap f a

Monoid generated by liftA2 (<>)

Starting from GHC 8.6, a similar type is available from Data.Monoid. This type is nevertheless kept for compatibility.

Constructors

Ap

Fields

• getApp :: f a

Instances

Applicative f => Applicative (Ap f)

Defined in Test.Tasty.Runners.Reducers

Methods

pure :: a -> Ap f a

(<*>) :: Ap f (a -> b) -> Ap f a -> Ap f b

liftA2 :: (a -> b -> c) -> Ap f a -> Ap f b -> Ap f c

(*>) :: Ap f a -> Ap f b -> Ap f b

(<*) :: Ap f a -> Ap f b -> Ap f a

Functor f => Functor (Ap f)

Defined in Test.Tasty.Runners.Reducers

Methods

fmap :: (a -> b) -> Ap f a -> Ap f b

(<$) :: a -> Ap f b -> Ap f a

Monad f => Monad (Ap f)

Defined in Test.Tasty.Runners.Reducers

Methods

(>>=) :: Ap f a -> (a -> Ap f b) -> Ap f b

(>>) :: Ap f a -> Ap f b -> Ap f b

return :: a -> Ap f a

(Applicative f, Monoid a) => Monoid (Ap f a)

Defined in Test.Tasty.Runners.Reducers

Methods

mempty :: Ap f a

mappend :: Ap f a -> Ap f a -> Ap f a

mconcat :: [Ap f a] -> Ap f a

(Applicative f, Monoid a) => Semigroup (Ap f a)

Defined in Test.Tasty.Runners.Reducers

Methods

(<>) :: Ap f a -> Ap f a -> Ap f a

sconcat :: NonEmpty (Ap f a) -> Ap f a

stimes :: Integral b => b -> Ap f a -> Ap f a

Ingredients

data Ingredient

Ingredients make your test suite tasty.

Ingredients represent different actions that you can perform on your test suite. One obvious ingredient that you want to include is one that runs tests and reports the progress and results.

Another standard ingredient is one that simply prints the names of all tests.

Similar to test providers (see IsTest), every ingredient may specify which options it cares about, so that those options are presented to the user if the ingredient is included in the test suite.

An ingredient can choose, typically based on the OptionSet, whether to run. That's what the Maybe is for. The first ingredient that agreed to run does its work, and the remaining ingredients are ignored. Thus, the order in which you arrange the ingredients may matter.

Usually, the ingredient which runs the tests is unconditional and thus should be placed last in the list. Other ingredients usually run only if explicitly requested via an option. Their relative order thus doesn't matter.

That's all you need to know from an (advanced) user perspective. Read on if you want to create a new ingredient.

There are two kinds of ingredients.

The first kind is TestReporter. If the ingredient that agrees to run is a TestReporter, then tasty will automatically launch the tests and pass a StatusMap to the ingredient. All the ingredient needs to do then is to process the test results and probably report them to the user in some way (hence the name).

TestManager is the second kind of ingredient. It is typically used for test management purposes (such as listing the test names), although it can also be used for running tests (but, unlike TestReporter, it has to launch the tests manually if it wants them to be run). It is therefore more general than TestReporter. TestReporter is provided just for convenience.

The function's result should indicate whether all the tests passed.

In the TestManager case, it's up to the ingredient author to decide what the result should be. When no tests are run, the result should probably be True. Sometimes, even if some tests run and fail, it still makes sense to return True.

Constructors

TestReporter [OptionDescription] (OptionSet -> TestTree -> Maybe (StatusMap -> IO (Time -> IO Bool)))

For the explanation on how the callback works, see the documentation for launchTestTree.

TestManager [OptionDescription] (OptionSet -> TestTree -> Maybe (IO Bool))

type Time = Double

Time in seconds. Used to measure how long the tests took to run.

tryIngredients :: [Ingredient] -> OptionSet -> TestTree -> Maybe (IO Bool)

Run the first Ingredient that agrees to be run.

If no one accepts the task, return Nothing. This is usually a sign of misconfiguration.

ingredientOptions :: Ingredient -> [OptionDescription]

Return the options which are relevant for the given ingredient.

Note that this isn't the same as simply pattern-matching on Ingredient. E.g. options for a TestReporter automatically include NumThreads.

ingredientsOptions :: [Ingredient] -> [OptionDescription]

Like ingredientOption, but folds over multiple ingredients.

Standard console ingredients

NOTE: the exports in this section are deprecated and will be removed in the future. Please import Test.Tasty.Ingredients.Basic if you need them.

Console test reporter

consoleTestReporter :: Ingredient

A simple console UI

Tests list

listingTests :: Ingredient

The ingredient that provides the test listing functionality

newtype ListTests

This option, when set to True, specifies that we should run in the «list tests» mode

Constructors

ListTests Bool

Instances

Eq ListTests

Defined in Test.Tasty.Ingredients.ListTests

Methods

(==) :: ListTests -> ListTests -> Bool

(/=) :: ListTests -> ListTests -> Bool

Ord ListTests

Defined in Test.Tasty.Ingredients.ListTests

Methods

compare :: ListTests -> ListTests -> Ordering

(<) :: ListTests -> ListTests -> Bool

(<=) :: ListTests -> ListTests -> Bool

(>) :: ListTests -> ListTests -> Bool

(>=) :: ListTests -> ListTests -> Bool

max :: ListTests -> ListTests -> ListTests

min :: ListTests -> ListTests -> ListTests

IsOption ListTests

Defined in Test.Tasty.Ingredients.ListTests

Methods

defaultValue :: ListTests

parseValue :: String -> Maybe ListTests

optionName :: Tagged ListTests String

optionHelp :: Tagged ListTests String

showDefaultValue :: ListTests -> Maybe String

optionCLParser :: Parser ListTests

testsNames :: OptionSet -> TestTree -> [TestName]

Obtain the list of all tests in the suite

Command line handling

parseOptions :: [Ingredient] -> TestTree -> IO OptionSet

Parse the command-line and environment options passed to tasty.

Useful if you need to get the options before defaultMain is called.

Once within the test tree, askOption should be used instead.

The arguments to this function should be the same as for defaultMainWithIngredients. If you don't use any custom ingredients, pass defaultIngredients.

optionParser :: [OptionDescription] -> ([String], Parser OptionSet)

Generate a command line parser from a list of option descriptions, alongside any related warning messages.

Since: 1.3

suiteOptionParser :: [Ingredient] -> TestTree -> ([String], Parser OptionSet)

The command line parser for the test suite, alongside any related warnings.

Since: 1.3

defaultMainWithIngredients :: [Ingredient] -> TestTree -> IO ()

Parse the command line arguments and run the tests using the provided ingredient list.

When the tests finish, this function calls exitWith with the exit code that indicates whether any tests have failed. See defaultMain for details.

Running tests

data Status

Current status of a test

Constructors

NotStarted

test has not started running yet

Executing Progress

test is being run

Done Result

test finished with a given result

Instances

Show Status

Defined in Test.Tasty.Run

Methods

showsPrec :: Int -> Status -> ShowS

show :: Status -> String

showList :: [Status] -> ShowS

data Result

A test result

Constructors

Result

Fields

• resultOutcome :: Outcome

  Did the test fail? If so, why?

• resultDescription :: String

  resultDescription may contain some details about the test. For a passed test it's ok to leave it empty. Providers like SmallCheck and QuickCheck use it to provide information about how many tests were generated.

  For a failed test, resultDescription should typically provide more information about the failure.

• resultShortDescription :: String

  The short description printed in the test run summary, usually OK or FAIL.

• resultTime :: Time

  How long it took to run the test, in seconds.

• resultDetailsPrinter :: ResultDetailsPrinter

  An action that prints additional information about a test.

  This is similar to resultDescription except it can produce colorful/formatted output; see Test.Tasty.Providers.ConsoleFormat.

  This can be used instead of or in addition to resultDescription.

  Usually this is set to noResultDetails, which does nothing.

  Since: 1.3.1

Instances

Show Result

Defined in Test.Tasty.Core

Methods

showsPrec :: Int -> Result -> ShowS

show :: Result -> String

showList :: [Result] -> ShowS

data Outcome

Outcome of a test run

Note: this is isomorphic to Maybe FailureReason. You can use the generic-maybe package to exploit that.

Constructors

Success

test succeeded

Failure FailureReason

test failed because of the FailureReason

Instances

Generic Outcome

Defined in Test.Tasty.Core

Associated Types

type Rep Outcome :: Type -> Type

Methods

from :: Outcome -> Rep Outcome x

to :: Rep Outcome x -> Outcome

Show Outcome

Defined in Test.Tasty.Core

Methods

showsPrec :: Int -> Outcome -> ShowS

show :: Outcome -> String

showList :: [Outcome] -> ShowS

type Rep Outcome

Defined in Test.Tasty.Core

type Rep Outcome = D1 ('MetaData "Outcome" "Test.Tasty.Core" "tasty-1.4.2.2-inplace" 'False) (C1 ('MetaCons "Success" 'PrefixI 'False) (U1 :: Type -> Type) :+: C1 ('MetaCons "Failure" 'PrefixI 'False) (S1 ('MetaSel ('Nothing :: Maybe Symbol) 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedLazy) (Rec0 FailureReason)))

data FailureReason

If a test failed, FailureReason describes why

Constructors

TestFailed

test provider indicated failure of the code to test, either because the tested code returned wrong results, or raised an exception

TestThrewException SomeException

the test code itself raised an exception. Typical cases include missing example input or output files.

Usually, providers do not have to implement this, as their run method may simply raise an exception.

TestTimedOut Integer

test didn't complete in allotted time

TestDepFailed

a dependency of this test failed, so this test was skipped.

Instances

Show FailureReason

Defined in Test.Tasty.Core

Methods

showsPrec :: Int -> FailureReason -> ShowS

show :: FailureReason -> String

showList :: [FailureReason] -> ShowS

resultSuccessful :: Result -> Bool

True for a passed test, False for a failed one.

data Progress

Test progress information.

This may be used by a runner to provide some feedback to the user while a long-running test is executing.

Constructors

Progress

Fields

• progressText :: String

  textual information about the test's progress

• progressPercent :: Float

  progressPercent should be a value between 0 and 1. If it's impossible to compute the estimate, use 0.

Instances

Show Progress

Defined in Test.Tasty.Core

Methods

showsPrec :: Int -> Progress -> ShowS

show :: Progress -> String

showList :: [Progress] -> ShowS

type StatusMap = IntMap (TVar Status)

Mapping from test numbers (starting from 0) to their status variables.

This is what an ingredient uses to analyse and display progress, and to detect when tests finish.

launchTestTree

Arguments

:: OptionSet

-> TestTree

-> (StatusMap -> IO (Time -> IO a))

A callback. First, it receives the StatusMap through which it can observe the execution of tests in real time. Typically (but not necessarily), it waits until all the tests are finished.

After this callback returns, the test-running threads (if any) are terminated and all resources acquired by tests are released.

The callback must return another callback (of type Time -> IO a) which additionally can report and/or record the total time taken by the test suite. This time includes the time taken to run all resource initializers and finalizers, which is why it is more accurate than what could be measured from inside the first callback.

-> IO a

Start running the tests (in background, in parallel) and pass control to the callback.

Once the callback returns, stop running the tests.

The number of test running threads is determined by the NumThreads option.

newtype NumThreads

Number of parallel threads to use for running tests.

Note that this is not included in coreOptions. Instead, it's automatically included in the options for any TestReporter ingredient by ingredientOptions, because the way test reporters are handled already involves parallelism. Other ingredients may also choose to include this option.

Constructors

NumThreads

Fields

• getNumThreads :: Int

Instances

Num NumThreads

Defined in Test.Tasty.Options.Core

Methods

(+) :: NumThreads -> NumThreads -> NumThreads

(-) :: NumThreads -> NumThreads -> NumThreads

(*) :: NumThreads -> NumThreads -> NumThreads

negate :: NumThreads -> NumThreads

abs :: NumThreads -> NumThreads

signum :: NumThreads -> NumThreads

fromInteger :: Integer -> NumThreads

Eq NumThreads

Defined in Test.Tasty.Options.Core

Methods

(==) :: NumThreads -> NumThreads -> Bool

(/=) :: NumThreads -> NumThreads -> Bool

Ord NumThreads

Defined in Test.Tasty.Options.Core

Methods

compare :: NumThreads -> NumThreads -> Ordering

(<) :: NumThreads -> NumThreads -> Bool

(<=) :: NumThreads -> NumThreads -> Bool

(>) :: NumThreads -> NumThreads -> Bool

(>=) :: NumThreads -> NumThreads -> Bool

max :: NumThreads -> NumThreads -> NumThreads

min :: NumThreads -> NumThreads -> NumThreads

IsOption NumThreads

Defined in Test.Tasty.Options.Core

Methods

defaultValue :: NumThreads

parseValue :: String -> Maybe NumThreads

optionName :: Tagged NumThreads String

optionHelp :: Tagged NumThreads String

showDefaultValue :: NumThreads -> Maybe String

optionCLParser :: Parser NumThreads

data DependencyException

Exceptions related to dependencies between tests.

Constructors

DependencyLoop

Test dependencies form a loop. In other words, test A cannot start until test B finishes, and test B cannot start until test A finishes.

Instances

Exception DependencyException

Defined in Test.Tasty.Run

Methods

toException :: DependencyException -> SomeException

fromException :: SomeException -> Maybe DependencyException

displayException :: DependencyException -> String

Show DependencyException

Defined in Test.Tasty.Run

Methods

showsPrec :: Int -> DependencyException -> ShowS

show :: DependencyException -> String

showList :: [DependencyException] -> ShowS

Options

suiteOptions :: [Ingredient] -> TestTree -> [OptionDescription]

All the options relevant for this test suite. This includes the options for the test tree and ingredients, and the core options.

coreOptions :: [OptionDescription]

The list of all core options, i.e. the options not specific to any provider or ingredient, but to tasty itself. Currently contains TestPattern and Timeout.

Patterns

newtype TestPattern

Constructors

TestPattern (Maybe Expr)

Instances

Show TestPattern

Defined in Test.Tasty.Patterns

Methods

showsPrec :: Int -> TestPattern -> ShowS

show :: TestPattern -> String

showList :: [TestPattern] -> ShowS

Eq TestPattern

Defined in Test.Tasty.Patterns

Methods

(==) :: TestPattern -> TestPattern -> Bool

(/=) :: TestPattern -> TestPattern -> Bool

IsOption TestPattern

Defined in Test.Tasty.Patterns

Methods

defaultValue :: TestPattern

parseValue :: String -> Maybe TestPattern

optionName :: Tagged TestPattern String

optionHelp :: Tagged TestPattern String

showDefaultValue :: TestPattern -> Maybe String

optionCLParser :: Parser TestPattern

parseExpr :: String -> Maybe Expr

parseTestPattern :: String -> Maybe TestPattern

noPattern :: TestPattern

type Path = Seq String

exprMatches :: Expr -> Path -> Bool

testPatternMatches :: TestPattern -> Path -> Bool

Utilities

formatMessage :: String -> IO String

Catch possible exceptions that may arise when evaluating a string. For normal (total) strings, this is a no-op.

This function should be used to display messages generated by the test suite (such as test result descriptions).

See e.g. https://github.com/UnkindPartition/tasty/issues/25

forceElements :: [a] -> ()

Force elements of a list (https://ro-che.info/articles/2015-05-28-force-list)

installSignalHandlers :: IO ()

Install signal handlers so that e.g. the cursor is restored if the test suite is killed by SIGTERM. Upon a signal, a SignalException will be thrown to the thread that has executed this action.

This function is called automatically from the defaultMain* family of functions. You only need to call it explicitly if you call tryIngredients yourself.

This function does nothing on non-UNIX systems or when compiled with GHC older than 7.6.

newtype SignalException

This exception is thrown when the program receives a signal, assuming installSignalHandlers was called.

The CInt field contains the signal number, as in Signal. We don't use that type synonym, however, because it's not available on non-UNIXes.

Constructors

SignalException CInt

Instances

Exception SignalException

Defined in Test.Tasty.Runners.Utils

Methods

toException :: SignalException -> SomeException

fromException :: SomeException -> Maybe SignalException

displayException :: SignalException -> String

Show SignalException

Defined in Test.Tasty.Runners.Utils

Methods

showsPrec :: Int -> SignalException -> ShowS

show :: SignalException -> String

showList :: [SignalException] -> ShowS

timed :: IO a -> IO (Time, a)

Measure the time taken by an IO action to run

getTime :: IO Time

Get monotonic time

Warning: This is not the system time, but a monotonically increasing time that facilitates reliable measurement of time differences.