Goals:
- Distribute reviews to people who can do the reviews
- Not prevent anything from getting done
- Allow giving away more commit access
Anti-goals:
- Silo people into particular areas
- Discourage contributing to any area
This was drafted by glancing at git logs. It is not likely to be very
accurate; the goal here is that we figure out a way to distribute
reviews to the right people.
Change-Id: I8be44bf7fdeca23da8099124eec7bc3a30e34627
Added an additional check that all files present within a folder must be
used/referenced. Otherwise an InvalidLangTest will be created.
This ensures that there weren't any mishaps while migrating tests
resulting in files being ignored and hence some tests not being run.
Fixes: #852
Change-Id: Ie096c5670bc20325ba72c7d6ce33c06667c66ab1
Redesigns the test.toml to use a list instead of a directory
additionally it is now possible to do toml and matrix tests on singular
files as well as on a subset of files.
Fixes: #851
Change-Id: If8635109c6274f406ad68fe35315b9125f45f67d
Currently when a lang test fails, (or any snapshot assertion for that
matter) the error message is rather bulky.
This is due to both sides being printed fully, using escaped newlines
(i.e. everything is one line)
This is awful to read and check what the actual difference is. Also
there is no indication that one can update the golden files using the
cli flag.
This commit changes the error message when comparing snapshots against
something
a list of lines is shown, where the output differed. An additional note
about how to update the files automatically was added too
Change-Id: Ibedcf48018c27f924b807fbd42362fb608d27441
We upgrade to 25.05 release, which contains the curl commit
https://github.com/curl/curl/commit/5fbd78eb2dc4afbd8884e8eed27147fc3d4318f6
done in
https://github.com/NixOS/nixpkgs/pull/396200#issuecomment-2795944006.
This fixes HTTP transfers generating arbitrary errors and possibly
failing unusually.
Users who are already depending on 25.05-small or a recent unstable
already had the fix.
Special mention to the Linux kernel who gave me the opportunity to get
on a 24 hours bisection side quest to fix the local release engineering
test.
Special thanks to everyone who had to endure me ranting.
Change-Id: I866caf65d5ea103f1fa5eccd57df8031c9eacda0
Co-authored-by: eldritch horrors <pennae@lix.systems>
Co-authored-by: helle <helle@h3l.li>
Signed-off-by: Raito Bezarius <raito@lix.systems>
Most of these are simple fixes and clarifications. One set of fixes will
come in the commit that actually upgrades nixpkgs and hence ruff as it
will otherwise cause errors here.
Change-Id: Ie857da0f6cf728478700ec2d24cf518f8c7b7815
Currently, the typecheck for the config values is only done
half-heartedly only checking if something is either a list or non-list
item, but not checking what type the list items are
this commit fixes the typecheck and adds test for proper serialization
Change-Id: Ifd93842b19b1dd870bdb3af0c000243b4380e7aa
The error message used to only contain the last key of the merge failure
this commit changes the message to contain the full path to the merge
conflict, resolving ambiguity
Change-Id: I9848a559b1b888e50a548eef8609bf34506040de
currently, there is a small helper funciton in lang_util to check if
something is of a list type generic
to improve re-usability, this function is moved to utils and improved to
be also check for nested iterables and such
Change-Id: I92984daa4c4decf13d340a2ea5e52f724cee800e
All changes are uniform and done with the same script, so checking only
some should suffice. For that reason, any tests involving multiple files
or custom CLI flags are not included in this commit.
Change-Id: Ib2d0e08937b56e241d99771a58aad34ed3ad308a
The current `RelativeTo` design is both more complex and more confusing
than necessary. Its four variants are now reduced to only two. They are
now also represented as different classes, to better communicate the
difference in semantics and also intent.
Change-Id: Ia60fc7a2dfa0f62bdef90dde347fd8603fd3fbf9
By default, xfail tests will always "pass" when the test fails,
disrecsarding any restrictions put on them via their parameters.
By enabling the `xfail_strict` option, xfails won't pass anymore when
the failstate is different from what is described in their parameters.
Change-Id: Ifea6e27d716d91f60210e6ba24175074fa39c304
Add Documentation for usage and development within functional2
including common fixtures and where to find them
This is done to make the migration from functional easier and give devs
a reference for how one writes tests
Change-Id: I6ee73e654d245fd4ad43e495d1172e406313cb23
This creates a framework similar to the old lang.sh from functional.
Some notable changes:
- instead of having a .flags file, a test.toml can declare flags
- additionally the test.toml can also declare extra files and multiple
runners for the given input file.
- there won't be any old tests hanging around anymore which weren't
deleted properly in the installation
- all files for a single test are defined decleratively and there won't
be any residues
Tests can be placed within the functional2/lang folder
most migrations should be rather clean
Implements: #825
Change-Id: I5f9149903ec5b078008969a4ae77305417c11475
Currently, tests are marked as "passed" when golden files are updated.
With this change, the tests are marked as skipped instead.
Additionally finally introduces tests to check if the snapshot behaves
as expected
Change-Id: I438eed70e0b94d561e99cc1e0363092809da827e
Add utils for general-use functions and paths
Additionally introduces a pytest_command fixture, which creates a
testing environment for pytest within the tmp_path. This allows for
encapsulated testing of our frameworks (i.e. snapshot, lang etc)
Change-Id: Ic0a5bc4bfc0b0bfbac15bc51dd4a94fae6ee6f26
allow to pass absolute paths or similar Path entries to declaration of
files instead of just string paths relative to the requesting file
Change-Id: I616da6abbb73d1d63ead370e9ae37a401d85f42d
The custom subcommand test fixture used to replace the environment
with PATH prepended with the directory of the subcommand.
This caused the tests to fail on darwin with auto-allocate-uids
enabled, as the dynamic users aren't added to the user database
inside the sandbox, as opposed to linux.
Other environments were unaffected because the build user is a real user
with a database entry and HOME set.
Update the environment instead, also preserving hermetic env
created earlier by NixCommand constructor.
Change-Id: I7e59fd69ff13d1d395316d857b63a356e1648159
with_env now overrides the environment, similar to with_stdin
an additional function update_env was created to mirror the prior
functionality of with_env, updating the env
This was changed as previously it was impossible to delete variables
from the env
replaced the code of .ok() with a call to .expect, to remove the code
duplication
Change-Id: I83933893c7f2ccfdc7bd4933b7592b475c435e76
Currently the Command and CommandResult classes are mixed into the nix
fixture file.
This commit moves them out into their own lib file, to make it more
obvious that they can be used standalone for other applications too
Additionally improved documentation of said classes
and bumped log level of stdout and err on unexpected exitcodes, as it is
within an error context
Change-Id: If2d554acde86fd54f2445fc46453f06923af5fe9
Due to nix-store making its paths read-only, pytest was unable to remove
the test files and hence the entire temporary directory, screaming all
over the place in stderr about that, getting worse for each test run.
By making the nix fixture first yield nix and then, after the test
finished running, changing the file permissions to include read on all
files and directories within the temp folder, pytest is able to properly
remove old test runs again
Additionally added more clear instructions for file deletion to the
pytest configuration
Change-Id: Ia7e3d195665968ac80a57d0e525691b28be7f503
Add a snapshot fixture, which allows comparing and updating strings
against external files
resolves#595
Change-Id: I518f594c601eb7805c6492c0352fca753fda04c9
Ruff is used to enforce our code-style for the python parts of the
reposity, similar to clang-tidy for the cpp parts.
This includes a pre-commit hook to format code before it is committed
When "unfixable" - i.e. no autoformatting is available - the commit is
rejected
resolves#812
Change-Id: I6830c2fc29ae86337ec18f2b0e3565fac66c5523
Use logger in favor over print statment.
This is explicitly supported and encuraged by pytest, which also allows
for capturing logs separate from stdout calls, which is handy for when
e.g. lix code calls out to stdout to keep those differentiated from test
output
Change-Id: Ib88565a1663da3b77ca6b95f8edf644eafb4a99d
separating the nix fixture from the __init__.py file to increase
readability and overview over the existing fixtures
Change-Id: I7a86cb729942e83a95b9eabbb09563822f3f9e54
This removes the bash dependency issue that the previous test had, the
tests are translated roughly 1 to 1 and do not address any of the TOD
list items.
Closes#800, also superceeds I81817b9fe4a5eeb019f7e04b5a60e091bbde26b2
Change-Id: Ie440cd0a602d4ca4a48edcaf517b7358ec572710
Documenting fetchTree is an exercise in frustration because of the sheer
amount of stringly typed everything in it. I do not know which fields
exist without reading the entirety of libfetchers. However, we can write
something slightly perfunctory but at least perhaps somewhat helpful
documentation-wise.
Fixes: https://git.lix.systems/lix-project/lix/issues/609
Change-Id: I991391b53fcd69172dbc7efb9d384e62bc847b91
This is a useful piece of functionality to being able to eat URL
hyperlinks, for instance, which is a bug that Lix has while dealing with
terminal output today.
Change-Id: I77b2de107b2525cad7ea5dea28bfba2cc78b9e6d
This commit makes Lix include the summarized content of the value being
indexed when it is bad.
lix/lix2 » nix eval --expr '{x.y = 2;}' 'x.y.z'
error: the value being indexed in the selection path 'x.y.z' at 'x.y' should be a set but is an integer: 2
lix/lix2 » nix eval --expr '{x.y = { a = 3; };}' 'x.y.z'
error: attribute 'z' in selection path 'x.y.z' not found inside path 'x.y', whose contents are: { a = 3; }
Did you mean a?
lix/lix2 » nix eval --expr '{x.y = { a = 3; };}' 'x.y.1'
error: the expression selected by the selection path 'x.y.1' should be a list but is a set: { a = 3; }
Change-Id: I3202aba0e437e00b4c6d3ee287a2d9a7c6892dbf
I want this for being able to write reasonable expect-test style tests
for oneliners. We will still probably want something like insta for more
complicated test cases where you actually *want* the output in a
different file, but for now this will do.
cc: https://git.lix.systems/lix-project/lix/issues/595
Change-Id: I6ddc42963cc49177762cfca206fe9a9efe1ae65d
I don't know what the heck the xonsh module is doing but its obviously
crimes so it has to go. It was never intended to be running here anyway.
Fixes: https://git.lix.systems/lix-project/lix/issues/593
Change-Id: I1877698469392f85884945aaa60987c68c4e0ebc
There is absolutely no good reason these should show up in NARs besides
misconfigured systems and as long as the case hack exists, unpacking
such a NAR will cause its repacking to be wrong on systems with case
hack enabled.
This should not have any security impact on Lix to fix, but it was one
of the vectors for CVE-2024-45593:
https://github.com/NixOS/nix/security/advisories/GHSA-h4vv-h3jq-v493
Change-Id: I85b6075aacc069ee7039240b0f525804a2d8edcb
This also rewrites a lot of the command handling in the fixtures
library, since we want to more precisely control which way that the nix
store is set up in the tests, rather than the previous method of
renaming /nix/store to some temp dir (which allows builds but does not
allow any /nix/store paths or stability across runs, which is a
significant issue for snapshot testing).
It uses a builder to reduce the amount of state carelessly thrown
around.
The evil NARs are inspired by CVE-2024-45593
(https://github.com/NixOS/nix/security/advisories/GHSA-h4vv-h3jq-v493).
No bugs were found in this endeavor.
Change-Id: Iee41b055fa96529c5a3c761f680ed1d0667ba5da
This is capped at 12 because 3.7 seconds of startup is painful enough
and 5.5 seconds with 24 was more annoying.
Change-Id: I327db40fd98deaa5330cd9cf6de99fb07b2c1cb0
I am tired of bad shell scripts, let me write bad python quickly
instead. It's definitely, $100%, better.
This is not planned as an immediate replacement of the old test suite,
but we::jade would not oppose tests getting ported.
What is here is a mere starting point and there is a lot more
functionality that we need.
Fixes: https://git.lix.systems/lix-project/lix/issues/488
Change-Id: If762efce69030bb667491b263b874c36024bf7b6
