So far it was impossible to put absolute paths into files, as one
wouldn't know whre the test would be placed.
This commit adds a new Fileish variant called `EnvTemplate` which uses
the given string as a template and replaces the `@ENV_VARIABLE_NAME@`
placeholders with the according value of the environment variable.
This way one can use `@HOME@` or `@TEST_ROOT@` to build absolute paths
Change-Id: I425cb6408dceb8d7f26d136ace4ac98b1ca31ec3
Calling `nix.nix` with an empty argument list, would crash on darwin due
to a Index Out Of Range Error, as the build="auto" functionallity would
try to access the first element of the provided arguments (second
element at the place of call) resulting in a crash
Change-Id: Ia678109808b3f75a30182114baa421cedaa8759b
If BUILD_TEST_SHELL is not set or empty, None or the empty string respectively
should not make it into the PATH. Ensure this property.
Change-Id: I4ce9b0c06c407b465308b63b9cb64e7d6a6a6964
These are a footgun and are not acceptable in functional2 due to thread
safety, effects on other tests, etc.
Co-authored-by: Commentator2.0 <lix@crystal-cavern.systems>
Change-Id: I8d7285061eaa9bab27edd52f3646024c8cf605e5
This reverts commit 42691f0d94.
Reason for revert: darwin builds time out now, apparently due to newly appearing infinite loops in the test suite
Change-Id: I81a09efd53636a287f3095ac87ed8854b6e57922
The functional2 test suite was broken in the development shell for two mostly
separate reasons leading to "no such file or directory" errors:
* The `BUILD_TEST_SHELL` (already containing the correct path) would always be
set from the Meson option `build_test_shell`, even to the empty string if
that option is not set. Skip the overwrite in this case to make the
environment variable work again.
* Sandboxed builds would fail to find their builder `/bin/sh`, since in the
development shell busybox is not found, so no sandbox shell gets configured.
Use the shell from `BUILD_TEST_SHELL` instead. (How this does not break the
old functional tests remains a mystery.)
Change-Id: I6a6a696424e8caaef3f9b68e3738bfd58ea0b056
Currenlty one is required to always write the bulky `mark.parametrize`
with indirect and things
This commit adds a custom decorator for usage of files, which hides the
parametrization complexity from the user.
Change-Id: I526e016d12006669dc302dfc5af619735399c503
So far, the environment used by `command` was completely leaky and the
one used by `nix` was very leaky despite it trying to be a "hermetic"
environment.
This commit moves the hermaticity to `command` and changes its
implementation to be not leak anything.
To achieve this, the following changes were also nessecary:
- the `files` and `snapshot` fixture now use the folder `test-home`
within the tmp_path directory by default, as the `HOME` environment
variable is set to there. (extraction not possible due to dependencies
of command etc also using this directory)
Fixes: #847, #848
Change-Id: I55f86ee0e1615e73fcf442ee2f28f3b89893bbb4
When testing specific internal functionallity while needing things from
the testlib, so far, the tests for the testlib have always been copied
too.
To reduce the amount of additional program required when later making
the env of the pytest_command declarative, and to not test the same
tests a multitude of times (and potentially reaching infinite recursion)
those tests will no longer be copied
Change-Id: I36ec3824a21ed30f9b8ff19948031d1edbf6c76c
Snix's nar-bridge[1] stores NARs under a different URL, i.e.
`nar/snix-castore/<hash>.nar` rather than `nar/<filehash>.nar`. Right
now, when copying into such a store via the http binary-cache, we'd end
up with wrong cache entries that point to the wrong NAR URL.
On Hydra, this is a fatal error, i.e. builds that depend on previously
built paths (that were written to the cache before by the queue runner)
would be aborted because of that.
This patch removes the caching since we'd have to re-fetch the narinfo
to do taht and this can also happen the next time, the narinfo is
queried. Also, removes the negative cache entry indicating that the
store-path doesn't exist in the store.
We don't have any coverage for http-stores so far, so I wrote a small
testcase for the "default" case and the nar-bridge case in functional2
since it has a very nice fixture for an HTTP server ready. I'm aware
that there's a CL for a nicer cache server[2], but given I'm down a
pretty deep rabbit hole by playing around with Snix, I decided to not
finish the CL and write something small for the tests in here. This can
be replaced by the fixtures from that CL later on as well.
[1] https://snix.dev/docs/components/overview/#nar-bridge
[2] https://gerrit.lix.systems/c/lix/+/2431/1
Change-Id: I4fcdf47a6bf9c3c8fbeb235eeca7a48914a4d693
Currently when a lang test fails, (or any snapshot assertion for that
matter) the error message is rather bulky.
This is due to both sides being printed fully, using escaped newlines
(i.e. everything is one line)
This is awful to read and check what the actual difference is. Also
there is no indication that one can update the golden files using the
cli flag.
This commit changes the error message when comparing snapshots against
something
a list of lines is shown, where the output differed. An additional note
about how to update the files automatically was added too
Change-Id: Ibedcf48018c27f924b807fbd42362fb608d27441
We upgrade to 25.05 release, which contains the curl commit
https://github.com/curl/curl/commit/5fbd78eb2dc4afbd8884e8eed27147fc3d4318f6
done in
https://github.com/NixOS/nixpkgs/pull/396200#issuecomment-2795944006.
This fixes HTTP transfers generating arbitrary errors and possibly
failing unusually.
Users who are already depending on 25.05-small or a recent unstable
already had the fix.
Special mention to the Linux kernel who gave me the opportunity to get
on a 24 hours bisection side quest to fix the local release engineering
test.
Special thanks to everyone who had to endure me ranting.
Change-Id: I866caf65d5ea103f1fa5eccd57df8031c9eacda0
Co-authored-by: eldritch horrors <pennae@lix.systems>
Co-authored-by: helle <helle@h3l.li>
Signed-off-by: Raito Bezarius <raito@lix.systems>
Most of these are simple fixes and clarifications. One set of fixes will
come in the commit that actually upgrades nixpkgs and hence ruff as it
will otherwise cause errors here.
Change-Id: Ie857da0f6cf728478700ec2d24cf518f8c7b7815
Currently, the typecheck for the config values is only done
half-heartedly only checking if something is either a list or non-list
item, but not checking what type the list items are
this commit fixes the typecheck and adds test for proper serialization
Change-Id: Ifd93842b19b1dd870bdb3af0c000243b4380e7aa
The error message used to only contain the last key of the merge failure
this commit changes the message to contain the full path to the merge
conflict, resolving ambiguity
Change-Id: I9848a559b1b888e50a548eef8609bf34506040de
currently, there is a small helper funciton in lang_util to check if
something is of a list type generic
to improve re-usability, this function is moved to utils and improved to
be also check for nested iterables and such
Change-Id: I92984daa4c4decf13d340a2ea5e52f724cee800e
The current `RelativeTo` design is both more complex and more confusing
than necessary. Its four variants are now reduced to only two. They are
now also represented as different classes, to better communicate the
difference in semantics and also intent.
Change-Id: Ia60fc7a2dfa0f62bdef90dde347fd8603fd3fbf9
Add Documentation for usage and development within functional2
including common fixtures and where to find them
This is done to make the migration from functional easier and give devs
a reference for how one writes tests
Change-Id: I6ee73e654d245fd4ad43e495d1172e406313cb23
This creates a framework similar to the old lang.sh from functional.
Some notable changes:
- instead of having a .flags file, a test.toml can declare flags
- additionally the test.toml can also declare extra files and multiple
runners for the given input file.
- there won't be any old tests hanging around anymore which weren't
deleted properly in the installation
- all files for a single test are defined decleratively and there won't
be any residues
Tests can be placed within the functional2/lang folder
most migrations should be rather clean
Implements: #825
Change-Id: I5f9149903ec5b078008969a4ae77305417c11475
Currently, tests are marked as "passed" when golden files are updated.
With this change, the tests are marked as skipped instead.
Additionally finally introduces tests to check if the snapshot behaves
as expected
Change-Id: I438eed70e0b94d561e99cc1e0363092809da827e
Add utils for general-use functions and paths
Additionally introduces a pytest_command fixture, which creates a
testing environment for pytest within the tmp_path. This allows for
encapsulated testing of our frameworks (i.e. snapshot, lang etc)
Change-Id: Ic0a5bc4bfc0b0bfbac15bc51dd4a94fae6ee6f26
allow to pass absolute paths or similar Path entries to declaration of
files instead of just string paths relative to the requesting file
Change-Id: I616da6abbb73d1d63ead370e9ae37a401d85f42d
with_env now overrides the environment, similar to with_stdin
an additional function update_env was created to mirror the prior
functionality of with_env, updating the env
This was changed as previously it was impossible to delete variables
from the env
replaced the code of .ok() with a call to .expect, to remove the code
duplication
Change-Id: I83933893c7f2ccfdc7bd4933b7592b475c435e76
Currently the Command and CommandResult classes are mixed into the nix
fixture file.
This commit moves them out into their own lib file, to make it more
obvious that they can be used standalone for other applications too
Additionally improved documentation of said classes
and bumped log level of stdout and err on unexpected exitcodes, as it is
within an error context
Change-Id: If2d554acde86fd54f2445fc46453f06923af5fe9
Due to nix-store making its paths read-only, pytest was unable to remove
the test files and hence the entire temporary directory, screaming all
over the place in stderr about that, getting worse for each test run.
By making the nix fixture first yield nix and then, after the test
finished running, changing the file permissions to include read on all
files and directories within the temp folder, pytest is able to properly
remove old test runs again
Additionally added more clear instructions for file deletion to the
pytest configuration
Change-Id: Ia7e3d195665968ac80a57d0e525691b28be7f503
Add a snapshot fixture, which allows comparing and updating strings
against external files
resolves#595
Change-Id: I518f594c601eb7805c6492c0352fca753fda04c9
Use logger in favor over print statment.
This is explicitly supported and encuraged by pytest, which also allows
for capturing logs separate from stdout calls, which is handy for when
e.g. lix code calls out to stdout to keep those differentiated from test
output
Change-Id: Ib88565a1663da3b77ca6b95f8edf644eafb4a99d
separating the nix fixture from the __init__.py file to increase
readability and overview over the existing fixtures
Change-Id: I7a86cb729942e83a95b9eabbb09563822f3f9e54
This is a useful piece of functionality to being able to eat URL
hyperlinks, for instance, which is a bug that Lix has while dealing with
terminal output today.
Change-Id: I77b2de107b2525cad7ea5dea28bfba2cc78b9e6d
I want this for being able to write reasonable expect-test style tests
for oneliners. We will still probably want something like insta for more
complicated test cases where you actually *want* the output in a
different file, but for now this will do.
cc: https://git.lix.systems/lix-project/lix/issues/595
Change-Id: I6ddc42963cc49177762cfca206fe9a9efe1ae65d
This also rewrites a lot of the command handling in the fixtures
library, since we want to more precisely control which way that the nix
store is set up in the tests, rather than the previous method of
renaming /nix/store to some temp dir (which allows builds but does not
allow any /nix/store paths or stability across runs, which is a
significant issue for snapshot testing).
It uses a builder to reduce the amount of state carelessly thrown
around.
The evil NARs are inspired by CVE-2024-45593
(https://github.com/NixOS/nix/security/advisories/GHSA-h4vv-h3jq-v493).
No bugs were found in this endeavor.
Change-Id: Iee41b055fa96529c5a3c761f680ed1d0667ba5da
I am tired of bad shell scripts, let me write bad python quickly
instead. It's definitely, $100%, better.
This is not planned as an immediate replacement of the old test suite,
but we::jade would not oppose tests getting ported.
What is here is a mere starting point and there is a lot more
functionality that we need.
Fixes: https://git.lix.systems/lix-project/lix/issues/488
Change-Id: If762efce69030bb667491b263b874c36024bf7b6
Commentator2.0