122 lines
5.9 KiB
Markdown
122 lines
5.9 KiB
Markdown
# Lang Tests
|
|
|
|
## Writing lang tests
|
|
|
|
The `lang` tests are special in that in addition to the usual Python tests, it also has a framework for automatically creating tests purely from resource files.
|
|
|
|
Each folder in `lang` is a test, although it may contain multiple subtests.
|
|
If a folder is a Python module (i.e. has an `__init__.py`), it will be treated as a Python test instead.
|
|
|
|
There are two different ways of creating Lang tests.
|
|
The "simple" way with no `test.toml`, which does more automagic and requires less setup but also is less powerful,
|
|
and the "complex" way, which allows full control over the test runners via a `test.toml`.
|
|
|
|
### Lang test runners
|
|
|
|
All lang tests focus on testing either the parsing or evaluation of Nix code, which yields for available runners:
|
|
|
|
- `eval-okay`: run eval, expecting an exit code of 0
|
|
- `eval-fail`: run eval, expecting an exit code of 1
|
|
- `parse-okay`: run parser, expecting an exit code of 0, stdout will be converted from json to yaml
|
|
- `parse-fail`: run parser, expecting an exit code of 1
|
|
|
|
The runners will test a given input file, and assert that the Nix stdout and stderr match the golden files.
|
|
Typical names for these are `eval-fail.err.exp` or `parse-okay.out.exp`, indicating which runner they are referring to and whether they contain stdout or stderr.
|
|
If an `*.out.exp` or `*.err.exp` is not present for a test, the Nix output will be *expected* to be empty (i.e. no file is equivalent to empty file).
|
|
|
|
### Without `test.toml`
|
|
|
|
A test without `test.toml` simply contains the following files:
|
|
|
|
- `in.nix`: The input Nix code
|
|
- `RUNNER_NAME.out.exp` and/or `RUNNER_NAME.err.exp`: Contain the expected output for stdout and stderr respectively
|
|
- The output of multiple runners may be provided, in which all of them will be run against the input file individually.
|
|
|
|
When creating a test, simply `touch` the desired `exp` files and use `--accept-tests` to fill them.
|
|
|
|
It is possible to use different `in`-files within the same folder:
|
|
to do so, suffix them with `-number-or-descriptive-string` resulting in `in-number-or-descriptive-string.nix`.
|
|
Add the same suffix to the expected output file, e.g. `eval-okay-number-or-descriptive-string.out.exp`
|
|
The tests will be discovered by searching the expected output files for the corresponding in files.
|
|
|
|
### With `test.toml`
|
|
|
|
A `test.toml` allows for the following additional test configuration features:
|
|
|
|
- Specifying additional CLI arguments to Nix
|
|
- Running the same runner (e.g. `eval-okay`) with different CLI flags
|
|
- Providing additional Nix files, e.g. for `import` tests
|
|
|
|
The `test.toml` has the following structure:
|
|
|
|
```toml
|
|
[[test]]
|
|
runner = "RUNNER_NAME"
|
|
|
|
[[test]]
|
|
name = "another-test"
|
|
runner = "eval-fail"
|
|
flags = ["-some", "-new", "flag"]
|
|
|
|
[[test]]
|
|
name = "third-runner"
|
|
runner = "eval-okay"
|
|
extra-files = ["./other_nix_file.nix"]
|
|
```
|
|
- The top-level element is a list called `test`. One can add a new element by labeling a section `[[test]]`
|
|
- `runner` must be one of `"eval-okay"`, `"eval-fail"`, `"parse-okay"`, `"parse-fail"`
|
|
- `name` defaults to `runner` plus the suffix of the given in file. Must be set manually if multiple test use the same runner and files.
|
|
- `matrix` optional boolean, which indicates if the test will be run on a single file or multiple. defaults to `False` (single file)
|
|
- `in` optional argument to specify on what files to run.
|
|
- For non-matrix tests, it must be a single file name and defaults to `"in.nix"`
|
|
- For matrix tests, it must be a list of file names and defaults to all available in files.
|
|
- `flags` optionally specifies a list of additional CLI arguments to be passed to Nix
|
|
- `extra-files` optionally describes a list of (relative) file paths for additional files to be copied into the test directory before execution.
|
|
- `global-assets` optionally describes a list of global assets like `config.nix` to be copied into the test directory before execution.
|
|
|
|
As before, the test folder contains an `in.nix` together with the expected output files.
|
|
For these, the naming scheme is
|
|
`CUSTOM_NAME.out.exp`/`CUSTOM_NAME.err.exp`, where `CUSTOM_NAME` stands for the name given in the `test.toml`.
|
|
|
|
**Example:**
|
|
|
|
```toml
|
|
[[test]]
|
|
runner = "parse-okay"
|
|
flags = ["--no-warning"]
|
|
|
|
[[test]]
|
|
# We must set this name manually to avoid collision
|
|
name = "parse-okay-with-warning"
|
|
runner = "parse-okay"
|
|
|
|
[[test]]
|
|
runner = "eval-fail"
|
|
```
|
|
|
|
Which implies the following files to exist:
|
|
|
|
```
|
|
in.nix
|
|
test.toml
|
|
parse-okay.out.exp
|
|
parse-okay-with-warning.out.exp
|
|
parse-okay-with-warning.err.exp
|
|
eval-fail.err.exp
|
|
```
|
|
|
|
When creating a test, simply writing the `in.nix` and `test.toml` is sufficient, all `.exp` files can be automatically generated with `--accept-tests`.
|
|
|
|
|
|
Here too, it is possible to work with multiple input files, though it works slightly differently to without a test toml:
|
|
- for non-matrix tests, set the `in` parameter to the name of the according in file, equivalent to tests without a toml.
|
|
- for matrix tests, either not set the `in` parameter (this will test the runner on *all* present in files) or set it to a list of in file names.
|
|
|
|
### Additional Notes
|
|
- The file [`lib.nix`](./lang/lib.nix) is a general library file available to all test, and for that copied into each lang test's directory automatically.
|
|
- There is no need to declare and copy it for each individual test that needs it, `import ./lib.nix` will always work out of the box.
|
|
- In the `test.toml`, it is currently not supported to pass paths with subdirectories into the `extra-files` attribute. If that functionality is required, use a [pytest tests](#writing-python-tests) instead.
|
|
- It is possible to call the according test runner function directly to avoid boilerplate
|
|
- If additional functionalities are required, placing a `.py` file in the directory tells the framework to ignore it. One can then write [pytest tests](#writing-python-tests) as usual
|
|
- The test suit will fail, if any files are unused. This is done to avoid unrecognized tests due to bad naming.
|