Some source trees might not be representable inside of the NAR listing
format v1 as file paths (on Linux) are not guaranteed to be valid UTF-8.
When something like this happens on a large-scale build farm, a
mysterious "queued" but impossible to process job appears, this is
because we cannot write the NAR listing and serialization always fails.
Why did this work before? nlohmann was introduced _after_ such paths
were ingested, see: 09f00dd4d0.
What happened for such previously mis-serialized NAR listings?
```
curl -v 'https://cache.nixos.org/nz8p9hn00r6z7s57581c1hiv39pa1ia6.ls' |
brotli -d | jq .
```
This fixes the build of `sub-batch`
(https://github.com/kl/sub-batch/tree/master/tests/rename_invalid_utf8)
on ForkOS infrastructure.
Many thanks to Puck for the assistance on holding `rr` right on this one
and finding the history of these changes.
Change-Id: I2c2fbac70818e02810f9fd236c3a248187bf5fe7
Signed-off-by: Raito Bezarius <raito@lix.systems>
Instead of allocating a new Value and copy the symbol string
representation inside of it, we can pass along the underlying Value,
which avoids (garbage collected) allocations.
This results in:
* a ~8 % reduction for `gc.totalBytes` over
`nixos.ec2.closures.x86_64-linux` for NixOS 24.11. (920MiB → 842MiB)
* a slight reduction in CPU time due to less allocations being performed
at all
Change-Id: I097f586dbc98f889fbc62d0a5f80c9d76ddedfd2
Signed-off-by: Raito Bezarius <raito@lix.systems>
The backing storage for symbols becomes a class storing a Value and a
string.
The Value is itself a string which contents points to the owned string.
Recovering a `SymbolStr` is still possible.
Change-Id: I171151abc3c0a513f2150c4b54edd61dea256cce
Signed-off-by: Raito Bezarius <raito@lix.systems>
The symbol table will contain types that encloses a Value, thus, it
needs to depend upon the Value header, whereas the Value header depends
on `Symbol` for typedefs.
We move the typedefs in the place where they are used.
Change-Id: Ic533e5aad927b9bc4a9d1723430e90e86a4b5466
Signed-off-by: Raito Bezarius <raito@lix.systems>
This simplifies many call-sites where construction can take place
automatically.
Change-Id: I87f697d55375676345b388024eb8df900bf808de
Co-authored-by: Tom Hubrecht <github@mail.hubrecht.ovh>
Signed-off-by: Raito Bezarius <raito@lix.systems>
This fixes Meson's "Project does not target a minimum version but uses
feature introduced in '1.1': meson.options file" warning.
Silly Meson.
I also added a note in the top-level meson.build to indicate
`meson_version` is specified in more than one place.
Change-Id: I2c04278bb46a562a1c96cd2e5e4d9ce59ce8e125
Lix has a style guide:
https://wiki.lix.systems/books/lix-contributors/page/code but
contributors like me have been unable to enforce it, which is sad.
To avoid further violations of that style guide, we enable a pre-commit
hook for clang formatting of the changed lines.
Change-Id: I217452efa3ac8bd66b4d3a08a6fe9a241207790b
Signed-off-by: Raito Bezarius <raito@lix.systems>
This is useful to reformat only changed hunks of a file via
`clang-format`.
Change-Id: I9aa8526d75fd2301113ee57f3a2e595f3b03504f
Signed-off-by: Raito Bezarius <raito@lix.systems>
Currently, the typecheck for the config values is only done
half-heartedly only checking if something is either a list or non-list
item, but not checking what type the list items are
this commit fixes the typecheck and adds test for proper serialization
Change-Id: Ifd93842b19b1dd870bdb3af0c000243b4380e7aa
The error message used to only contain the last key of the merge failure
this commit changes the message to contain the full path to the merge
conflict, resolving ambiguity
Change-Id: I9848a559b1b888e50a548eef8609bf34506040de
currently, there is a small helper funciton in lang_util to check if
something is of a list type generic
to improve re-usability, this function is moved to utils and improved to
be also check for nested iterables and such
Change-Id: I92984daa4c4decf13d340a2ea5e52f724cee800e
this could've just ignored exceptions thrown by the remote. in the
current implementation there's no way such an exception could have
propagated to the client though, so there's no change in behavior.
Change-Id: Ide03bda1cb0ad7fb5f27b4ee5d16efd6c2b635ba
mostly to make moving this to async writes easier. this won't have a
performance impact because it's only a single packet, that's written
to a BufferedSink, but the connection sink only gets a single write.
Change-Id: I9a5f1afe7d3e25f5f4502ef9520ff2f2529431ba
the test is for the map that usually wraps it though because it's the
bit we're interested in replacing, and it has custom serializer code.
Change-Id: If77a236dfca738b646ed2b7a5c65515dad6b7295
the old protocols are largely untested, mostly unused, and have design
problems that make the RPC transition a lot harder, if not impossible.
in theory we could ship a transparent protocol-converting proxy that'd
isolate the daemon itself from old protocol versions, but that's a lot
of code to maintain for presumably little gain or even no gain at all.
Change-Id: I4c3f3bb34d39044f6aeb07c10caaf13b8340a220
All changes are uniform and done with the same script, so checking only
some should suffice. For that reason, any tests involving multiple files
or custom CLI flags are not included in this commit.
Change-Id: Ib2d0e08937b56e241d99771a58aad34ed3ad308a
The current `RelativeTo` design is both more complex and more confusing
than necessary. Its four variants are now reduced to only two. They are
now also represented as different classes, to better communicate the
difference in semantics and also intent.
Change-Id: Ia60fc7a2dfa0f62bdef90dde347fd8603fd3fbf9
Previously, paths not being deleted by gcDeleteSpecific would result in
(a) hardlinks not being cleaned up, and
(b) statistics not being reported correctly.
By throwing the error later, we fix both of these problems.
Change-Id: I8019f3e10d9f22e81ea87bb26b77f04ebc888a19
By default, xfail tests will always "pass" when the test fails,
disrecsarding any restrictions put on them via their parameters.
By enabling the `xfail_strict` option, xfails won't pass anymore when
the failstate is different from what is described in their parameters.
Change-Id: Ifea6e27d716d91f60210e6ba24175074fa39c304
Remote binary caches support `write-nar-listing` options where they create a `HASH.ls` file for quick indexing without having to download the nar.
This commit makes experimental `nix store ls` attempt to read these files instead of downloading the full nar.
The difference is very obvious with large packages like stellarium:
nix store ls --store "https://cache.nixos.org" /nix/store/ijpvwgs9zamqaax5dy2cd0kxgz7lr7an-stellarium-25.1 -R
Change-Id: I6a37e0788b3a91c319331a8de69c51daf3efa955
Add Documentation for usage and development within functional2
including common fixtures and where to find them
This is done to make the migration from functional easier and give devs
a reference for how one writes tests
Change-Id: I6ee73e654d245fd4ad43e495d1172e406313cb23
This creates a framework similar to the old lang.sh from functional.
Some notable changes:
- instead of having a .flags file, a test.toml can declare flags
- additionally the test.toml can also declare extra files and multiple
runners for the given input file.
- there won't be any old tests hanging around anymore which weren't
deleted properly in the installation
- all files for a single test are defined decleratively and there won't
be any residues
Tests can be placed within the functional2/lang folder
most migrations should be rather clean
Implements: #825
Change-Id: I5f9149903ec5b078008969a4ae77305417c11475
Currently, tests are marked as "passed" when golden files are updated.
With this change, the tests are marked as skipped instead.
Additionally finally introduces tests to check if the snapshot behaves
as expected
Change-Id: I438eed70e0b94d561e99cc1e0363092809da827e
Add utils for general-use functions and paths
Additionally introduces a pytest_command fixture, which creates a
testing environment for pytest within the tmp_path. This allows for
encapsulated testing of our frameworks (i.e. snapshot, lang etc)
Change-Id: Ic0a5bc4bfc0b0bfbac15bc51dd4a94fae6ee6f26
allow to pass absolute paths or similar Path entries to declaration of
files instead of just string paths relative to the requesting file
Change-Id: I616da6abbb73d1d63ead370e9ae37a401d85f42d
Due to how meson works with the current justfile options, it is not
possible to pass additional arguments into the functional2 test
suit/pytest.
Due to that, it isn't possilbe to narrow down what tests to execute or
add output options or similar.
This commit adds an additional recipe, calling pytest directly ensuring
arguments are handed through
Change-Id: I3748d1cd5fddc16b11fff11c0f1a77195e37c837
The first auto-GC request would not be registered as a waiter due to a logic
error. As a result, if that request was synchronous (as happens during
evaluation) it would be stuck forever waiting on a promise that will never be
fulfilled.
Register also the first request properly so that it is notified and unblocked
again when the GC has finished. Also add a test verifying that auto-GC
triggering during evaluation will not get stuck.
Fixes: https://git.lix.systems/lix-project/lix/issues/844
Change-Id: I157afdc737415261e48d6d01d46c586a2927a1ad
The custom subcommand test fixture used to replace the environment
with PATH prepended with the directory of the subcommand.
This caused the tests to fail on darwin with auto-allocate-uids
enabled, as the dynamic users aren't added to the user database
inside the sandbox, as opposed to linux.
Other environments were unaffected because the build user is a real user
with a database entry and HOME set.
Update the environment instead, also preserving hermetic env
created earlier by NixCommand constructor.
Change-Id: I7e59fd69ff13d1d395316d857b63a356e1648159
The experimental `nix eval` command already supports a `--raw` flag.
This commit implements the same flag for the stable nix-instantiate command.
Until now instructions and scripts that didn't want to rely on experimental
features had to use workarounds such as:
nix-instantiate --eval <something> | tr -d \"
(which also undesirably also removes double quotation marks within the string), or
nix-instantiate --eval <something> | jq -j
(which undesirably depends on another package).
Co-authored-by: Raito Bezarius <raito@lix.systems>
Co-authored-by: Silvan Mosberger <silvan.mosberger@tweag.io>
Change-Id: Iced9a80ee7edd60af2385c5193485f1774175339
`fetchGit` has been modified a long time ago to use fetchTree, however,
we don't care about `lastModified` because we are not in a flake
context, this hack introduces a `git-locked` type of input that only
cares about `narHash` being present. This is needed to avoid fetching
the remote repo each time `fetchGit` is evaluated whith the result
present in the store.
Change-Id: I521c6fcccf8cf12945594f205d7fd4c8c2cf89e9
The coerce integer feature was not rebased before merge and we do not
have a merge queue, hence, after merge, the HEAD was in a broken state.
We take a commitment to invest into a merge queue now and do a fixup
here.
Change-Id: Ied9410690b542359859ab5f597f22ebceb857305
Signed-off-by: Raito Bezarius <raito@lix.systems>
When assigning an a value to NIX_GET_COMPLETIONS that could not be
parsed as an integer lix would just crash, as the value was directly
passed to stoi, without handling the return value.
This change switches the parsing to use string2Int and throws an
exception if the return value is empty.
The behaviour of lix is slightly changed through, as the value of the
variable was previously parsed to an int and then assigned to a variable
of size_t.
This change in behaviour can only be observed in cases where the
value of NIX_GET_COMPLETIONS is chosen so when it overflows it would
be valid index of the provided arguments again.
Through this change the variable is parsed as a size_t and negative
values are rejected.
Change-Id: Idf7c5740274c6e07d5bb13d7e2ed32764bfc27f8
When using completion, the number of the word for which the shell
requests completion is provided in the environment variable
`NIX_GET_COMPLETIONS`. When the number smaller than 1 is or larger
than the number of arguments nix coredumps as a assert is violated.
This change removes the assert and instead throws an exception informing
the user that their autocomplete is most likely misconfigured.
Change-Id: I821719e470e576b6f63c06beb097338b53d183e0
Raito Bezarius