blokyk
f91bdc9367
frames manually added with `addErrorContext` are generally a lot more useful/informative to the average user than other frames, esp. in the module system, which can create much better error messages than we can. however, before this change, frames from addErrorContext were truncated by default if they weren't in the first 3 frames, so they were basically useless (with `--show-trace`, you're dredging through 250 frames of module shenanigans just to spot one singular line). this change also removes the frames for _the call to_ `addErrorContext`, which is just pure noise. the way this change is done hopefully leaves a bit of space for future similar changes to error printing, by introducing a new `TraceKind` enum that can be used to categorize traces (i haven't done that in this CL because that would be a pretty herculean task, given all the calls to `BaseError::addTrace` in the codebase, and we probably want to be careful about what categories we choose). the actual printing code could definitely be improved tho... (e.g. by iterating twice through the trace stack instead to first pick out the most important traces and _then_ printing less "important" traces if there's space left) Change-Id: I52acc52f231991a9f2309d9cecae362397c6888c
Lang Tests
Writing lang tests
The lang tests are special in that in addition to the usual Python tests, it also has a framework for automatically creating tests purely from resource files.
Each folder in lang is a test, although it may contain multiple subtests.
If a folder is a Python module (i.e. has an __init__.py), it will be treated as a Python test instead.
There are two different ways of creating Lang tests.
The "simple" way with no test.toml, which does more automagic and requires less setup but also is less powerful,
and the "complex" way, which allows full control over the test runners via a test.toml.
Lang test runners
All lang tests focus on testing either the parsing or evaluation of Nix code, which yields for available runners:
eval-okay: run eval, expecting an exit code of 0eval-fail: run eval, expecting an exit code of 1parse-okay: run parser, expecting an exit code of 0, stdout will be converted from json to yamlparse-fail: run parser, expecting an exit code of 1
The runners will test a given input file, and assert that the Nix stdout and stderr match the golden files.
Typical names for these are eval-fail.err.exp or parse-okay.out.exp, indicating which runner they are referring to and whether they contain stdout or stderr.
If an *.out.exp or *.err.exp is not present for a test, the Nix output will be expected to be empty (i.e. no file is equivalent to empty file).
Without test.toml
A test without test.toml simply contains the following files:
in.nix: The input Nix codeRUNNER_NAME.out.expand/orRUNNER_NAME.err.exp: Contain the expected output for stdout and stderr respectively- The output of multiple runners may be provided, in which all of them will be run against the input file individually.
When creating a test, simply touch the desired exp files and use --accept-tests to fill them.
It is possible to use different in-files within the same folder:
to do so, suffix them with -number-or-descriptive-string resulting in in-number-or-descriptive-string.nix.
Add the same suffix to the expected output file, e.g. eval-okay-number-or-descriptive-string.out.exp
The tests will be discovered by searching the expected output files for the corresponding in files.
With test.toml
A test.toml allows for the following additional test configuration features:
- Specifying additional CLI arguments to Nix
- Running the same runner (e.g.
eval-okay) with different CLI flags - Providing additional Nix files, e.g. for
importtests
The test.toml has the following structure:
[[test]]
runner = "RUNNER_NAME"
[[test]]
name = "another-test"
runner = "eval-fail"
flags = ["-some", "-new", "flag"]
[[test]]
name = "third-runner"
runner = "eval-okay"
extra-files = ["./other_nix_file.nix"]
- The top-level element is a list called
test. One can add a new element by labeling a section[[test]] runnermust be one of"eval-okay","eval-fail","parse-okay","parse-fail"namedefaults torunnerplus the suffix of the given in file. Must be set manually if multiple test use the same runner and files.matrixoptional boolean, which indicates if the test will be run on a single file or multiple. defaults toFalse(single file)inoptional argument to specify on what files to run.- For non-matrix tests, it must be a single file name and defaults to
"in.nix" - For matrix tests, it must be a list of file names and defaults to all available in files.
- For non-matrix tests, it must be a single file name and defaults to
flagsoptionally specifies a list of additional CLI arguments to be passed to Nixextra-filesoptionally describes a list of (relative) file paths for additional files to be copied into the test directory before execution.global-assetsoptionally describes a list of global assets likeconfig.nixto be copied into the test directory before execution.
As before, the test folder contains an in.nix together with the expected output files.
For these, the naming scheme is
CUSTOM_NAME.out.exp/CUSTOM_NAME.err.exp, where CUSTOM_NAME stands for the name given in the test.toml.
Example:
[[test]]
runner = "parse-okay"
flags = ["--no-warning"]
[[test]]
# We must set this name manually to avoid collision
name = "parse-okay-with-warning"
runner = "parse-okay"
[[test]]
runner = "eval-fail"
Which implies the following files to exist:
in.nix
test.toml
parse-okay.out.exp
parse-okay-with-warning.out.exp
parse-okay-with-warning.err.exp
eval-fail.err.exp
When creating a test, simply writing the in.nix and test.toml is sufficient, all .exp files can be automatically generated with --accept-tests.
Here too, it is possible to work with multiple input files, though it works slightly differently to without a test toml:
- for non-matrix tests, set the
inparameter to the name of the according in file, equivalent to tests without a toml. - for matrix tests, either not set the
inparameter (this will test the runner on all present in files) or set it to a list of in file names.
Additional Notes
- The file
lib.nixis a general library file available to all test, and for that copied into each lang test's directory automatically.- There is no need to declare and copy it for each individual test that needs it,
import ./lib.nixwill always work out of the box.
- There is no need to declare and copy it for each individual test that needs it,
- In the
test.toml, it is currently not supported to pass paths with subdirectories into theextra-filesattribute. If that functionality is required, use a pytest tests instead.- It is possible to call the according test runner function directly to avoid boilerplate
- If additional functionalities are required, placing a
.pyfile in the directory tells the framework to ignore it. One can then write pytest tests as usual - The test suit will fail, if any files are unused. This is done to avoid unrecognized tests due to bad naming.