guiwf/lix
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

tests/functional2/lang: improve toml design

Browse Source 
Redesigns the test.toml to use a list instead of a directory
additionally it is now possible to do toml and matrix tests on singular
files as well as on a subset of files.

Fixes: #851

Change-Id: If8635109c6274f406ad68fe35315b9125f45f67d
This commit is contained in:
Commentator2.0
2025-06-20 11:36:13 +02:00
parent f2eb920e46
commit d1afc83676
7 changed files with 637 additions and 112 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/functional2/DEVELOPMENT.md
+1 -1
View File
@@ -1,4 +1,4 @@
# DEVELOOPMENT
# DEVELOPMENT
## Goals
- Eliminate implicit dependencies on files in the test directory as well as the requirement to copy the test files to the build directory as is currently hacked in the other functional test suite.
tests/functional2/README.md
+32 -19
View File
@@ -92,7 +92,7 @@ Conversely, this convention makes it easy to see which test an asset belongs to.
For a more exhaustive documentation, check the [pytest documentation](https://docs.pytest.org/en/stable/example/parametrize.html).
When parametrizing a test, the test is called once per provided argument.
When a test is parameterized multiple times, every combination of parameters will be run in a matrix.
When a test is parametrized multiple times, every combination of parameters will be run in a matrix.
If that is not intended, one can provide multiple tuple arguments within a single parametrization:
```python
@@ -204,7 +204,7 @@ from functional2.testlib.fixtures.file_helper import AssetSymlink
@pytest.mark.parametrize(
"files",
# Setup the symlink so that the golden value will update
# Set up the symlink so that the golden value will update
[ { "out": AssetSymlink("assets/test_example/out.exp"), } ],
indirect=True,
)
@@ -242,7 +242,7 @@ Calls to the logger are captured separately by pytest and results in more beauti
The `lang` tests are special in that in addition to the usual Python tests, it also has a framework for automatically creating tests purely from resource files.
Each folder in `lang` is a test, although it may contain multiple sub-tests.
Each folder in `lang` is a test, although it may contain multiple subtests.
If a folder is a Python module (i.e. has an `__init__.py`), it will be treated as a Python test instead.
There are two different ways of creating Lang tests.
@@ -260,7 +260,7 @@ All lang tests focus on testing either the parsing or evaluation of Nix code, wh
The runners will test a given input file, and assert that the Nix stdout and stderr match the golden files.
Typical names for these are `eval-fail.err.exp` or `parse-okay.out.exp`, indicating which runner they are referring to and whether they contain stdout or stderr.
If an `out.exp` or `err.exp` is not present for a test, the Nix output will be *expected* to be empty (i.e. no file is equivalent to empty file).
If an `*.out.exp` or `*.err.exp` is not present for a test, the Nix output will be *expected* to be empty (i.e. no file is equivalent to empty file).
### Without `test.toml`
@@ -272,6 +272,11 @@ A test without `test.toml` simply contains the following files:
When creating a test, simply `touch` the desired `exp` files and use `--accept-tests` to fill them.
It is possible to use different `in`-files within the same folder:
to do so, suffix them with `-number-or-descriptive-string` resulting in `in-number-or-descriptive-string.nix`.
Add the same suffix to the expected output file, e.g. `eval-okay-number-or-descriptive-string.out.exp`
The tests will be discovered by searching the expected output files for the corresponding in files.
### With `test.toml`
A `test.toml` allows for the following additional test configuration features:
@@ -283,21 +288,26 @@ A `test.toml` allows for the following additional test configuration features:
The `test.toml` has the following structure:
```toml
[custom-runner-name]
[[test]]
runner = "RUNNER_NAME"
[another-runner]
[[test]]
name = "another-test"
runner = "eval-fail"
flags = ["-some", "-new", "flag"]
[third-runner]
[[test]]
name = "third-runner"
runner = "eval-okay"
extra-files = ["./other_nix_file.nix"]
```
- The top-level names are the names for the test and may be chosen freely. The reason this exists is that when running the same runner multiple times, its name is not a unique identifier for the test anymore.
- It is okay to simply write `[eval-okay] runner = "eval-okay"` for tests that don't make use of this.
- The top-level element is a list called `test`. One can add a new element by labeling a section `[[test]]`
- `runner` must be one of `"eval-okay"`, `"eval-fail"`, `"parse-okay"`, `"parse-fail"`
- `name` defaults to `runner` plus the suffix of the given in file. Must be set manually if multiple test use the same runner and files.
- `matrix` optional boolean, which indicates if the test will be run on a single file or multiple. defaults to `False` (single file)
- `in` optional argument to specify on what files to run.
- For non-matrix tests, it must be a single file name and defaults to `"in.nix"`
- For matrix tests, it must be a list of file names and defaults to all available in files.
- `flags` optionally specifies a list of additional CLI arguments to be passed to Nix
- `extra-files` optionally describes a list of (relative) file paths for additional files to be copied into the test directory before execution.
@@ -308,14 +318,16 @@ For these, the naming scheme is
**Example:**
```toml
[parse-okay]
[[test]]
runner = "parse-okay"
flags = ["--no-warning"]
[parse-okay-with-warning]
[[test]]
# We must set this name manually to avoid collision
name = "parse-okay-with-warning"
runner = "parse-okay"
[eval-fail]
[[test]]
runner = "eval-fail"
```
@@ -332,13 +344,14 @@ eval-fail.err.exp
When creating a test, simply writing the `in.nix` and `test.toml` is sufficient, all `.exp` files can be automatically generated with `--accept-tests`.
### Additional Notes
- Instead of providing a single `in.nix` file, multiple `in-$name.nix` files can be provided instead (where `$name` may be a descriptive name or simply a counting number).
- When using a `test.toml`, all in files will be tested with all runners in a Matrix
- When not using the toml, the runner for each in file will be determined by the `RUNNER_NAME-$name.out.exp` file name, e.g. `eval-fail-1.err.exp` for `in-1.nix` as input and `eval-fail` as runner
Here too, it is possible to work with multiple input files, though it works slightly differently to without a test toml:
- for non-matrix tests, set the `in` parameter to the name of the according in file, equivalent to tests without a toml.
- for matrix tests, either not set the `in` parameter (this will test the runner on *all* present in files) or set it to a list of in file names.
### Additional Notes
- The file [`lib.nix`](./lang/lib.nix) is a general library file available to all test, and for that copied into each lang test's directory automatically.
- There is no need to declare and copy it for each individual test that needs it, `import ./lib.nix` will always work out of the box.
- In the `test.toml`, it is currently not supported to pass paths with subdirectories into the `extra-files` attribute. If that functionality is required, use a [pytest tests](#writing-othercustom-tests) instead.
- In the `test.toml`, it is currently not supported to pass paths with subdirectories into the `extra-files` attribute. If that functionality is required, use a [pytest tests](#writing-python-tests) instead.
- It is possible to call the according test runner function directly to avoid boilerplate
- If additional functionalities are required, placing a `.py` file in the directory tells the framework to ignore it. One can then write [pytest tests](#writing-othercustom-tests) as usual
- If additional functionalities are required, placing a `.py` file in the directory tells the framework to ignore it. One can then write [pytest tests](#writing-python-tests) as usual
tests/functional2/lang/import/test.toml
+1 -1
View File
@@ -1,3 +1,3 @@
[eval-okay]
[[test]]
runner = "eval-okay"
extra-files = ["imported.nix", "imported2.nix"]
tests/functional2/lang/lang_util.py
+202 -76
View File
@@ -1,16 +1,17 @@
import dataclasses
import logging
import re
from enum import StrEnum
from functools import cache
from pathlib import Path
from typing import Any
from typing import Any, NamedTuple, ClassVar
from collections.abc import Generator
import toml
from functional2.testlib.fixtures.file_helper import AssetSymlink, CopyFile, FileDeclaration
from functional2.testlib.utils import is_value_of_type, test_base_folder
from toml import TomlDecodeError
from functional2.testlib.fixtures.file_helper import FileDeclaration, CopyFile, AssetSymlink
from functional2.testlib.utils import test_base_folder, is_value_of_type
LANG_TEST_ID_PATTERN = "{folder_name}:{test_name}"
@@ -44,39 +45,68 @@ Base message for invalid runner, to use across collection
"""
SUFFIX_REGEX = r"-[\w-]+?"
NAMING_PATTERN_LANG_TEST = re.compile(
NAME_PATTERN_GENERIC_EXP = re.compile(
rf"{LangTestRunner.as_regex_selector()}(?P<suffix>{SUFFIX_REGEX})?"
)
NAME_PATTERN_IN_FILE = rf"in({SUFFIX_REGEX})?.nix"
class InFile(NamedTuple):
name: str
suffix: str
@classmethod
def parse(cls, name_str: str) -> "InFile":
"""
Parses the given name into a InFile object
:param name_str: string to parse
:raises ValueError: When The given string was invalid
:return: InFile named tuple, on successful parse
"""
match_ = re.fullmatch(NAME_PATTERN_IN_FILE, name_str)
if match_ is None:
msg = f"invalid in-file name {name_str!r}"
raise ValueError(msg)
return cls(match_.group(0), match_.group(1) or "")
class LangTest:
ids: ClassVar[set[str]] = set()
"""
Set of all existing Ids
"""
def __init__(
self,
test_name: str,
name: str,
folder_name: str,
runner: LangTestRunner,
in_file: InFile,
flags: list[str] | None = None,
extra_files: list[str] | None = None,
suffix: str = "",
):
"""
Internal class to represent a lang test
:param test_name: name of the explicit test (e.g. "eval-depr" or "eval-allow-depr")
:param name: name of the explicit test (e.g. "eval-depr" or "eval-allow-depr")
:param folder_name: the folder / group of tests this one originates from (e.g. "nul_bytes")
:param runner: which runner to run this test on (e.g. EVAL_FAIL or PARSE_OKAY)
:param in_file: what the input file is
:param flags: additional flags provided for nix
:param extra_files: any additional files which should be copied into the tests directory
:param suffix: suffix of the in file
:raises ValueError(id, msg): when the id constructed from the parameters is not unique
"""
self.test_name = test_name
self.full_name = LANG_TEST_ID_PATTERN.format(
folder_name=folder_name, test_name=f"{test_name}{suffix}"
)
self.runner = runner
self.flags = flags or []
self.folder = folder_name
self.extra_files = extra_files or []
self.suffix = suffix
self.in_file_name = in_file.name
self.suffix = in_file.suffix
self.test_name = f"{name}{self.suffix}"
self.id = LANG_TEST_ID_PATTERN.format(folder_name=self.folder, test_name=self.test_name)
if self.id in LangTest.ids:
msg = f"id {self.id!r} is not unique. Please set the 'name' attribute manually"
raise ValueError(self.id, msg)
LangTest.ids.add(self.id)
def _get_files(self) -> FileDeclaration:
"""
@@ -84,10 +114,10 @@ class LangTest:
:return: FileDeclaration object containing all files required for the test
"""
files = {
"in.nix": CopyFile(f"{self.folder}/in{self.suffix}.nix"),
"in.nix": CopyFile(f"{self.folder}/{self.in_file_name}"),
"lib.nix": CopyFile("lib.nix"),
"out.exp": AssetSymlink(f"{self.folder}/{self.test_name}{self.suffix}.out.exp"),
"err.exp": AssetSymlink(f"{self.folder}/{self.test_name}{self.suffix}.err.exp"),
"out.exp": AssetSymlink(f"{self.folder}/{self.test_name}.out.exp"),
"err.exp": AssetSymlink(f"{self.folder}/{self.test_name}.err.exp"),
}
for file in self.extra_files:
# Make sure to add the extra-files requested by the test.toml
@@ -99,7 +129,7 @@ class LangTest:
Converts the LangTest to the parameters required for parametrization of the test runners
:return: a Tuple of the FileDeclaration (used by the `files` fixture), list of flags and a unique id
"""
return self._get_files(), self.flags, self.full_name
return self._get_files(), self.flags, self.id
class InvalidLangTest:
@@ -125,6 +155,138 @@ def _group_lang_tests(tests: list[LangTest]) -> dict[LangTestRunner, list[LangTe
return grouped_tests
@dataclasses.dataclass
class LangTestDefinition:
"""
A parsed object of a singular `test` section from the `test.toml` file
"""
runner: LangTestRunner
name: str
flags: list[str]
extra_files: list[str]
matrix: bool
in_: list[InFile]
@classmethod
def parse(cls, dict_: dict[str, Any], in_file_names: list[str]) -> "LangTestDefinition":
"""
Parses the given dict to a LangTestDefinition object.
:param dict_: content of a singular test section within the `test.toml` file
:param in_file_names: a list of all existing in files, used as a default for matrix tests
:return: LangTestDefinition object containing the parsed information
:raises ValueError(test_name, [*reasons]): when any information was invalid
"""
issues = []
runner_name = dict_.pop("runner", "")
try:
runner = LangTestRunner(runner_name)
except ValueError:
issues.append(INVALID_TESTER_NAME % runner_name)
runner = None
extra_files = dict_.pop("extra-files", [])
if not is_value_of_type(extra_files, list[str]):
issues.append(
f"invalid value type for 'extra_files': {extra_files}, expected a list of strings"
)
flags = dict_.pop("flags", [])
if not is_value_of_type(flags, list[str]):
issues.append(f"invalid value type for 'flags': {flags}, expected a list of strings")
test_name = dict_.pop("name", runner_name)
is_matrix = dict_.pop("matrix", False)
if not is_value_of_type(is_matrix, bool):
issues.append(f"invalid type for 'matrix': {is_matrix}, expected a boolean")
in_file_def = dict_.pop("in", None)
if is_matrix:
in_correct_type = in_file_def is None or is_value_of_type(in_file_def, list[str])
in_file_names = in_file_def or in_file_names
else:
in_correct_type = is_value_of_type(in_file_def, str | None)
in_file_names = [in_file_def] if in_file_def is not None else ["in.nix"]
in_files: list[InFile] = []
if not in_correct_type or len(in_file_names) == 0:
issues.append(
f"invalid type for 'in': {in_file_def!r}, expected a{' list of' if is_matrix else ''} string"
)
else:
# Only check naming if the type is actually correct
for i, name in enumerate(in_file_names):
try:
in_files.append(InFile.parse(name))
except ValueError as e:
issues.append(f"{e.args[0]} at position {i} for 'in'")
if dict_:
issues.append(f"unexpected arguments: {list(dict_.keys())!r}")
if issues:
raise ValueError(test_name, issues)
return cls(
runner=runner,
name=test_name,
flags=flags,
extra_files=extra_files,
matrix=is_matrix,
in_=in_files,
)
def build(self, folder: str) -> tuple[list[LangTest], list[InvalidLangTest]]:
"""
Builds this LangTestDefinition into LangTests.
:param folder: test group folder name this Definition was created from
:return: list of valid LangTests and a list of InvalidLangTests (created when the id wasn't unique)
"""
tests = []
invalid = []
for file in self.in_:
try:
tests.append(
LangTest(self.name, folder, self.runner, file, self.flags, self.extra_files)
)
except ValueError as e:
id_, *reasons = e.args
invalid.append(InvalidLangTest(id_, reasons))
return tests, invalid
def parse_toml(
toml_content: dict[str, Any], in_files: list[str]
) -> Generator[LangTestDefinition | ValueError, None, None]:
"""
Parses the given toml content to LangTestDefinitions.
:param toml_content: content of the `test.toml` file
:param in_files: a list of all in files, used as a default for matrix tests
:returns: Generator yielding `LangTestDefinition`s for successful parses and ValueError(test_name, [*reasons]) for parse failures
"""
test_attr_name = "test"
issues = []
test_dicts = toml_content.pop(test_attr_name, [])
if not test_dicts:
issues.append(f"key {test_attr_name!r} not found or empty")
test_dicts = []
elif not is_value_of_type(test_dicts, list[dict[str, Any]]):
issues.append(f"Invalid type for {test_attr_name!r}. Expected an Array of Tables")
test_dicts = []
if len(toml_content) > 0:
issues.append(
f"unexpected key(s) {list(toml_content.keys())}; expected only {test_attr_name!r}"
)
for declaration in test_dicts:
try:
yield LangTestDefinition.parse(declaration, in_files)
except ValueError as e:
yield e
if issues:
yield ValueError("", issues)
def _collect_toml_test_group(folder: Path) -> tuple[list[LangTest], list[InvalidLangTest]]:
"""
Collects all tests, declared by a `test.toml` file within the given folder
@@ -132,68 +294,28 @@ def _collect_toml_test_group(folder: Path) -> tuple[list[LangTest], list[Invalid
:return: a list of valid test configurations and a list of invalid test configurations
"""
parent_name = folder.name
test_declaration = folder / "test.toml"
try:
infos: dict[str, Any] = toml.load(test_declaration)
except TomlDecodeError as e:
return [], [InvalidLangTest(parent_name, [f"couldn't parse toml: {e!r}"])]
invalid_tests: list[InvalidLangTest] = []
tests: list[LangTest] = []
# files starting with "_" will be ignored, e.g. "__pycache__"
all_files = {f.name for f in folder.iterdir() if not f.name.startswith("_")}
in_files = [f for f in all_files if re.fullmatch(rf"in({SUFFIX_REGEX})?\.nix", f) is not None]
# suffixes of the in files, e.g.
# in.nix => ''
# in-1.nix => '-1'
# in-some-test.nix => '-some-test'
# etc
in_suffixes = [
suffix.group(1) or ""
for suffix in [
re.fullmatch(rf"in({SUFFIX_REGEX})?\.nix", file.name) for file in folder.iterdir()
]
if suffix is not None
]
try:
infos: dict[str, Any] = toml.load(folder / "test.toml")
except TomlDecodeError as e:
return [], [InvalidLangTest(parent_name, [f"couldn't parse toml: {e!r}"])]
for test_name, definition in infos.items():
test_errors: list[str] = []
for test in parse_toml(infos, in_files):
test_name = test.name if isinstance(test, LangTestDefinition) else test.args[0]
full_name = LANG_TEST_ID_PATTERN.format(folder_name=parent_name, test_name=test_name)
if not isinstance(definition, dict):
invalid_tests.append(
InvalidLangTest(
full_name, [f"invalid value for {test_name!r}; only tests are expected"]
)
)
if isinstance(test, ValueError):
invalid_tests.append(InvalidLangTest(full_name, test.args[1]))
continue
flags = definition.pop("flags", [])
if not is_value_of_type(flags, list[str]):
test_errors.append(
f"invalid value type for 'flags': {flags}, expected a list of strings"
)
runner_name = definition.pop("runner", None)
try:
runner = LangTestRunner(runner_name)
except ValueError:
test_errors.append(INVALID_TESTER_NAME % runner_name)
runner = None
extra_files = definition.pop("extra-files", [])
if not is_value_of_type(extra_files, list[str]):
test_errors.append(
f"invalid value type for 'extra_files': {extra_files}, expected a list of strings"
)
if len(definition) > 0:
test_errors.append(f"unexpected arguments: {list(definition.keys())!r}")
if test_errors:
invalid_tests.append(InvalidLangTest(full_name, test_errors))
continue
# Add a test for each in file
tests += [
LangTest(test_name, folder.name, runner, flags, extra_files, in_suffix)
for in_suffix in in_suffixes
]
new_tests, new_invalids = test.build(parent_name)
tests += new_tests
invalid_tests += new_invalids
return tests, invalid_tests
@@ -218,7 +340,7 @@ def _collect_generic_test_group(folder: Path) -> tuple[list[LangTest], list[Inva
# `"parse-fail-some-name.err.exp".split(".")[0]` => "parse-fail-some-name"
test_name = file.name.rsplit(".", 2)[0]
full_name = LANG_TEST_ID_PATTERN.format(folder_name=parent_name, test_name=test_name)
match = re.fullmatch(NAMING_PATTERN_LANG_TEST, test_name)
match = re.fullmatch(NAME_PATTERN_GENERIC_EXP, test_name)
if match is None:
if re.match(LangTestRunner.as_regex_selector(), test_name) is None:
reason = INVALID_TESTER_NAME % test_name
@@ -228,7 +350,11 @@ def _collect_generic_test_group(folder: Path) -> tuple[list[LangTest], list[Inva
continue
runner_name, suffix = match.groups()
runner = LangTestRunner(runner_name)
tests.append(LangTest(runner_name, folder.name, runner, suffix=suffix or ""))
suffix = suffix or ""
tests.append(
LangTest(runner_name, folder.name, runner, InFile(f"in{suffix}.nix", suffix))
)
return tests, invalid_tests
@@ -257,7 +383,7 @@ def _collect_all_tests() -> tuple[list[LangTest], list[InvalidLangTest]]:
node: Path
# skip files, as test groups are folders and custom tests will be collected by pytest
# these are files like this `lang_util.py` or the `lib.nix` etc
if node.is_file() or node.name == "assets":
if node.is_file() or node.name == "assets" or node.name.startswith("_"):
continue
# ignore test groups, which have a py file, as those are set up fully custom
tests/functional2/lang/line_endings/test.toml
+4 -2
View File
@@ -1,6 +1,8 @@
[depr]
[[test]]
name = "depr"
runner = "parse-fail"
[allow-depr]
[[test]]
name = "allow-depr"
runner = "parse-okay"
flags = ["--extra-deprecated-features", "cr-line-endings"]
tests/functional2/lang/nul_bytes/test.toml
+8 -4
View File
@@ -1,13 +1,17 @@
[eval-depr]
[[test]]
name = "eval-depr"
runner = "eval-fail"
[eval-allow-depr]
[[test]]
name = "eval-allow-depr"
runner = "eval-okay"
flags = ["--extra-deprecated-features", "nul-bytes"]
[parse-depr]
[[test]]
name = "parse-depr"
runner = "parse-fail"
[parse-allow-depr]
[[test]]
name = "parse-allow-depr"
runner = "parse-okay"
flags = ["--extra-deprecated-features", "nul-bytes"]
tests/functional2/lang/test_lang_infra.py
+389 -9
View File
@@ -49,9 +49,10 @@ def test_detects_generic_lang_test(pytest_command: Command):
"my_name.out.exp": File("{ }\n"),
"test.toml": File(
dedent("""
[my_name]
runner = "eval-okay"
""")
[[test]]
name = "my_name"
runner = "eval-okay"
""")
),
}
}
@@ -335,7 +336,8 @@ def test_generic_bad_runner_name(pytest_command: Command):
"my_name.out.exp": File("{ }\n"),
"test.toml": File(
dedent("""
[my_name]
[[test]]
name = "my_name"
runner = "plushies"
""")
),
@@ -371,7 +373,8 @@ def test_toml_bad_runner_name(pytest_command: Command):
"my_name.out.exp": File("{ }\n"),
"test.toml": File(
dedent("""
[my_name]
[[test]]
name = "my_name"
runner = "eval-okay"
cuddles = true
""")
@@ -408,10 +411,12 @@ def test_toml_too_many_args(pytest_command: Command):
"test.toml": File(
dedent("""
invalid_test = "eval-okay"
[my_name]
[[test]]
name = "my_name"
runner = "eval-okay"
flags = 1
[second]
[[test]]
name = "second"
runner = "eval-okay"
extra-files = [false, true, true]
""")
@@ -431,7 +436,7 @@ def test_toml_invalid_argument_types(pytest_command: Command):
result = pytest_command.run().expect(1)
err = result.stdout_plain
assert "test_invalid_configuration" in err
assert "invalid value for 'invalid_test'; only tests are expected" in err
assert "unexpected key(s) ['invalid_test']; expected only 'test'" in err
assert "invalid value type for 'flags'" in err
assert "invalid value type for 'extra_files':" in err
@@ -449,7 +454,8 @@ def test_toml_invalid_argument_types(pytest_command: Command):
"my_name.out.exp": File("{ }\n"),
"test.toml": File(
dedent("""
[my_name]
[[test]]
name = "my_name"
runner = "eval-okay"
cuddles = True
""")
@@ -500,3 +506,377 @@ def test_updates_expected_output(
snapshot("out.exp")
== (files / "functional2/lang/update_test/eval-okay.out.exp").read_text()
)
@pytest.mark.parametrize(
("files", "pytest_command"),
[
(
get_functional2_lang_files(
{
"functional2": {
"lang": {
"single-multi": {
"in.nix": File("{}"),
"in-1.nix": File("{}"),
"eval-fail.err.exp": File(""),
"eval-okay.out.exp": AssetSymlink(
"assets/test_lang_infra/runner_eo.out.exp"
),
"test.toml": File(
dedent("""
[[test]]
runner = "eval-okay"
in = ["in.nix", "in-1.nix"]
""")
),
}
}
}
}
),
[],
)
],
indirect=True,
)
@pytest.mark.usefixtures("files")
def test_toml_single_only_str(pytest_command: Command):
res = pytest_command.run().expect(1)
err = res.stdout_plain
assert "test_invalid_configuration[single-multi:eval-okay-reasons0]" in err
assert "invalid type for 'in': ['in.nix', 'in-1.nix'], expected a string" in err
@pytest.mark.parametrize(
("files", "pytest_command"),
[
(
get_functional2_lang_files(
{
"functional2": {
"lang": {
"multi-single": {
"in.nix": File("{}"),
"eval-okay.out.exp": AssetSymlink(
"assets/test_lang_infra/runner_eo.out.exp"
),
"test.toml": File(
dedent("""
[[test]]
matrix = true
runner = "eval-okay"
in = "in.nix"
""")
),
}
}
}
}
),
[],
)
],
indirect=True,
)
@pytest.mark.usefixtures("files")
def test_toml_matrix_only_list_str(pytest_command: Command):
res = pytest_command.run().expect(1)
err = res.stdout_plain
assert "test_invalid_configuration[multi-single:eval-okay-reasons0]" in err
assert "invalid type for 'in': 'in.nix', expected a list of string" in err
@pytest.mark.parametrize(
("files", "pytest_command"),
[
(
get_functional2_lang_files(
{
"functional2": {
"lang": {
"matrix-all": {
"in.nix": File("{}"),
"in-1.nix": File("{}"),
"eval-okay-1.out.exp": AssetSymlink(
"assets/test_lang_infra/runner_eo.out.exp"
),
"eval-okay.out.exp": AssetSymlink(
"assets/test_lang_infra/runner_eo.out.exp"
),
"test.toml": File(
dedent("""
[[test]]
matrix = true
runner = "eval-okay"
""")
),
}
}
}
}
),
[],
)
],
indirect=True,
)
@pytest.mark.usefixtures("files")
def test_toml_matrix_uses_all_files(pytest_command: Command):
res = pytest_command.run().ok()
out = res.stdout_plain
assert "test_eval[matrix-all:eval-okay] PASSED" in out
assert "test_eval[matrix-all:eval-okay-1] PASSED" in out
@pytest.mark.parametrize(
("files", "pytest_command"),
[
(
get_functional2_lang_files(
{
"functional2": {
"lang": {
"mixed-matrix": {
"in.nix": File("{}"),
"in-1.nix": File("{}"),
"eval-okay-1.out.exp": AssetSymlink(
"assets/test_lang_infra/runner_eo.out.exp"
),
"eval-okay.out.exp": AssetSymlink(
"assets/test_lang_infra/runner_eo.out.exp"
),
"non-matrix-1.out.exp": AssetSymlink(
"assets/test_lang_infra/runner_eo.out.exp"
),
"test.toml": File(
dedent("""
[[test]]
matrix = true
runner = "eval-okay"
[[test]]
name = "non-matrix"
in = "in-1.nix"
runner = "eval-okay"
""")
),
}
}
}
}
),
[],
)
],
indirect=True,
)
@pytest.mark.usefixtures("files")
def test_toml_mixing_matrix_single(pytest_command: Command):
res = pytest_command.run().ok()
out = res.stdout_plain
assert "test_eval[mixed-matrix:eval-okay] PASSED" in out
assert "test_eval[mixed-matrix:eval-okay-1] PASSED" in out
assert "test_eval[mixed-matrix:non-matrix-1] PASSED" in out
@pytest.mark.parametrize(
("files", "pytest_command"),
[
(
get_functional2_lang_files(
{
"functional2": {
"lang": {
"non-unique": {
"in.nix": File("{}"),
"in-1.nix": File("{}"),
"test.toml": File(
dedent("""
[[test]]
matrix = true
runner = "eval-okay"
[[test]]
in = "in-1.nix"
runner = "eval-okay"
""")
),
}
}
}
}
),
[],
)
],
indirect=True,
)
@pytest.mark.usefixtures("files")
def test_toml_non_unique_name(pytest_command: Command):
res = pytest_command.run().expect(1)
err = res.stdout_plain
assert "test_invalid_configuration[non-unique:eval-okay-1-reasons0]" in err
assert (
"id 'non-unique:eval-okay-1' is not unique. Please set the 'name' attribute manually" in err
)
@pytest.mark.parametrize(
("files", "pytest_command"),
[
(
get_functional2_lang_files(
{
"functional2": {
"lang": {
"subset": {
"in.nix": File("{}"),
"in-1.nix": File("{}"),
"in-hello.nix": File("{}"),
"eval-okay-1.out.exp": AssetSymlink(
"assets/test_lang_infra/runner_eo.out.exp"
),
"matrix.out.exp": AssetSymlink(
"assets/test_lang_infra/runner_eo.out.exp"
),
"matrix-hello.out.exp": AssetSymlink(
"assets/test_lang_infra/runner_eo.out.exp"
),
"test.toml": File(
dedent("""
[[test]]
name = "matrix"
matrix = true
in = ["in.nix", "in-hello.nix"]
runner = "eval-okay"
[[test]]
runner = "eval-okay"
in = "in-1.nix"
""")
),
}
}
}
}
),
[],
)
],
indirect=True,
)
@pytest.mark.usefixtures("files")
def test_toml_matrix_subset(pytest_command: Command):
res = pytest_command.run().ok()
out = res.stdout_plain
assert "test_eval[subset:matrix] PASSED" in out
assert "test_eval[subset:matrix-hello] PASSED" in out
assert "test_eval[subset:matrix-1] PASSED" not in out
assert "test_eval[subset:eval-okay-1] PASSED" in out
assert "test_eval[subset:eval-okay] PASSED" not in out
@pytest.mark.parametrize(
("files", "pytest_command"),
[
(
get_functional2_lang_files(
{
"functional2": {
"lang": {
"in-naming": {
"in-some-name.nix": File("{}"),
"eval-okay-some-name.out.exp": AssetSymlink(
"assets/test_lang_infra/runner_eo.out.exp"
),
"test.toml": File(
dedent("""
[[test]]
matrix = true
in = ["in-some-name", "-some-name.nix", "some-name.nix"]
runner = "eval-okay"
""")
),
}
}
}
}
),
[],
)
],
indirect=True,
)
@pytest.mark.usefixtures("files")
def test_toml_bad_in_naming(pytest_command: Command):
res = pytest_command.run().expect(1)
out = res.stdout_plain
assert "test_invalid_configuration[in-naming:eval-okay-reasons0]" in out
for msg in [
"invalid in-file name 'in-some-name' at position 0 for 'in'",
"invalid in-file name '-some-name.nix' at position 1 for 'in'",
"invalid in-file name 'some-name.nix' at position 2 for 'in'",
]:
assert msg in out
@pytest.mark.parametrize(
("files", "pytest_command"),
[
(
get_functional2_lang_files(
{
"functional2": {
"lang": {
"toml-missing": {
"eval-okay.out.exp": AssetSymlink(
"assets/test_lang_infra/runner_eo.out.exp"
),
"test.toml": File(
dedent("""
[[test]]
runner = "eval-okay"
""")
),
}
}
}
}
),
[],
)
],
indirect=True,
)
@pytest.mark.usefixtures("files")
def test_toml_missing_in_file(pytest_command: Command):
res = pytest_command.run().expect(1)
out = res.stdout_plain
assert "ERROR lang/test_lang.py::test_eval[toml-missing:eval-okay] - FileNotFound" in out
@pytest.mark.parametrize(
("files", "pytest_command"),
[
(
get_functional2_lang_files(
{
"functional2": {
"lang": {
"generic-missing": {
"eval-okay.out.exp": AssetSymlink(
"assets/test_lang_infra/runner_eo.out.exp"
)
}
}
}
}
),
[],
)
],
indirect=True,
)
@pytest.mark.usefixtures("files")
def test_generic_missing_in_file(pytest_command: Command):
res = pytest_command.run().expect(1)
out = res.stdout_plain
assert "ERROR lang/test_lang.py::test_eval[generic-missing:eval-okay] - FileNotFound" in out