lix/tests/functional2/lang/README.md

Lang Tests

Writing lang tests

The lang tests are special in that in addition to the usual Python tests, it also has a framework for automatically creating tests purely from resource files.

Each folder in lang is a test, although it may contain multiple subtests. If a folder is a Python module (i.e. has an __init__.py), it will be treated as a Python test instead.

There are two different ways of creating Lang tests. The "simple" way with no test.toml, which does more automagic and requires less setup but also is less powerful, and the "complex" way, which allows full control over the test runners via a test.toml.

Lang test runners

All lang tests focus on testing either the parsing or evaluation of Nix code, which yields for available runners:

• eval-okay: run eval, expecting an exit code of 0
• eval-fail: run eval, expecting an exit code of 1
• parse-okay: run parser, expecting an exit code of 0, stdout will be converted from json to yaml
• parse-fail: run parser, expecting an exit code of 1

The runners will test a given input file, and assert that the Nix stdout and stderr match the golden files. Typical names for these are eval-fail.err.exp or parse-okay.out.exp, indicating which runner they are referring to and whether they contain stdout or stderr. If an *.out.exp or *.err.exp is not present for a test, the Nix output will be expected to be empty (i.e. no file is equivalent to empty file).

Without test.toml

A test without test.toml simply contains the following files:

• in.nix: The input Nix code
• RUNNER_NAME.out.exp and/or RUNNER_NAME.err.exp: Contain the expected output for stdout and stderr respectively
  • The output of multiple runners may be provided, in which all of them will be run against the input file individually.

When creating a test, simply touch the desired exp files and use --accept-tests to fill them.

It is possible to use different in-files within the same folder: to do so, suffix them with -number-or-descriptive-string resulting in in-number-or-descriptive-string.nix. Add the same suffix to the expected output file, e.g. eval-okay-number-or-descriptive-string.out.exp The tests will be discovered by searching the expected output files for the corresponding in files.

With test.toml

A test.toml allows for the following additional test configuration features:

• Specifying additional CLI arguments to Nix
• Running the same runner (e.g. eval-okay) with different CLI flags
• Providing additional Nix files, e.g. for import tests

The test.toml has the following structure:

[[test]]
runner = "RUNNER_NAME"

[[test]]
name = "another-test"
runner = "eval-fail"
flags = ["-some", "-new", "flag"]

[[test]]
name = "third-runner"
runner = "eval-okay"
extra-files = ["./other_nix_file.nix"]

• The top-level element is a list called test. One can add a new element by labeling a section [[test]]
• runner must be one of "eval-okay", "eval-fail", "parse-okay", "parse-fail"
• name defaults to runner plus the suffix of the given in file. Must be set manually if multiple test use the same runner and files.
• matrix optional boolean, which indicates if the test will be run on a single file or multiple. defaults to False (single file)
• in optional argument to specify on what files to run.
  • For non-matrix tests, it must be a single file name and defaults to "in.nix"
  • For matrix tests, it must be a list of file names and defaults to all available in files.
• flags optionally specifies a list of additional CLI arguments to be passed to Nix
• extra-files optionally describes a list of (relative) file paths for additional files to be copied into the test directory before execution.
• global-assets optionally describes a list of global assets like config.nix to be copied into the test directory before execution.

As before, the test folder contains an in.nix together with the expected output files. For these, the naming scheme is CUSTOM_NAME.out.exp/CUSTOM_NAME.err.exp, where CUSTOM_NAME stands for the name given in the test.toml.

Example:

[[test]]
runner = "parse-okay"
flags = ["--no-warning"]

[[test]]
# We must set this name manually to avoid collision
name = "parse-okay-with-warning"
runner = "parse-okay"

[[test]]
runner = "eval-fail"

Which implies the following files to exist:

in.nix
test.toml
parse-okay.out.exp
parse-okay-with-warning.out.exp
parse-okay-with-warning.err.exp
eval-fail.err.exp

When creating a test, simply writing the in.nix and test.toml is sufficient, all .exp files can be automatically generated with --accept-tests.

Here too, it is possible to work with multiple input files, though it works slightly differently to without a test toml:

• for non-matrix tests, set the in parameter to the name of the according in file, equivalent to tests without a toml.
• for matrix tests, either not set the in parameter (this will test the runner on all present in files) or set it to a list of in file names.

Additional Notes

• The file lib.nix is a general library file available to all test, and for that copied into each lang test's directory automatically.
  • There is no need to declare and copy it for each individual test that needs it, import ./lib.nix will always work out of the box.
• In the test.toml, it is currently not supported to pass paths with subdirectories into the extra-files attribute. If that functionality is required, use a pytest tests instead.
  • It is possible to call the according test runner function directly to avoid boilerplate
• If additional functionalities are required, placing a .py file in the directory tells the framework to ignore it. One can then write pytest tests as usual
• The test suit will fail, if any files are unused. This is done to avoid unrecognized tests due to bad naming.