meson fixed the bugs that prevented using the sanitizer args, but using
such a fixed meson will provide broken flags to the subprojects. *sigh*
Change-Id: I0ab54351b19a3879e9a97daed1621dc5f730d376
it's used very rarely, on ssh it's pretty much broken, and on ssh-ng
it's standing squarely in the way of the rpc transition. we have not
found significant use of this feature in public configs (with all of
*two* repositories using this store parameter), so why even keep it?
Change-Id: Ifcafdf794d815001a1a55a771e5823010e5b174b
there's no reason a store should not open a connection during init if it
needs to open any connection to function. delaying connection setup like
this makes openStore less deterministic and graceful fallback impossible
this caused test failures where the old behaviour was required to ensure
test output stability. because of course something like that must happen
Change-Id: I946adddee026f1f4c74b4699d730b4d1ac9a5072
build-remote should not be the only thing that ever reads logs from
remote store connections. ssh-ng is better off here because it does
not transport its logs on the ssh stderr channel, but ssh *does* do
this. neither have any ssh logs shown after connection setup except
when using an ssh:// url as a remote builder, which is not markedly
helpful for debugging anything at all. let's have both kinds of ssh
store classes handle logs themselves to alleviate all of that. this
also improves the utility of remote builder checks in `nix doctor`.
Change-Id: I23947a9666bc07561eac341578ed45993f1b9839
adding logs to a daemon store can fail before the entire request has
been read, leaving non-command data in socket buffers. if the daemon
does not terminate immediately after the failure it will try to read
the remaining data as commands, which is very likely to not work and
cause unbounded memory allocation instead. this doesn't give clients
an attack vector they did not have before though, they could've just
as well sent malformed commands without a bad preceding AddBuildLog.
Change-Id: I86c22890eace19164d932cbd342ac9f52cee6531
we don't support range-based iteration yet because it's a huge hassle to
make c++ and rust iteration styles meet in any reasonable way. this is a
good start to make rust collections *actually* usable though, so here we
go. `into_iter` is not explicitly supported, but can be used regardless.
Change-Id: Iebb76c9409059bc81623e5be81ac2656b1ec5138
frames manually added with `addErrorContext` are generally a lot more
useful/informative to the average user than other frames, esp. in
the module system, which can create much better error messages than
we can.
however, before this change, frames from addErrorContext were truncated
by default if they weren't in the first 3 frames, so they were basically
useless (with `--show-trace`, you're dredging through 250 frames of
module shenanigans just to spot one singular line).
this change also removes the frames for _the call to_ `addErrorContext`,
which is just pure noise.
the way this change is done hopefully leaves a bit of space for future
similar changes to error printing, by introducing a new `TraceKind` enum
that can be used to categorize traces (i haven't done that in this CL
because that would be a pretty herculean task, given all the calls to
`BaseError::addTrace` in the codebase, and we probably want to be
careful about what categories we choose). the actual printing code could
definitely be improved tho... (e.g. by iterating twice through the trace
stack instead to first pick out the most important traces and _then_
printing less "important" traces if there's space left)
Change-Id: I52acc52f231991a9f2309d9cecae362397c6888c
this makes using the zero-copy pipes a lot more ergonomic. error
handling is unfortunately not trivial and duplicating it for all
pipe users would thus be a bad idea. we're also not oblivious to
the fact that this is a `sourceToSink`, but it's async this time
around. (at least we don't need terrible stackful coroutines..?)
Change-Id: I1ba59f27183988ad68e7f88d102935d005690f43
rpc will often need to transfer data received on push-bashed interfaces
to consumers that expect input stream sources, which are pull-based. we
want to avoid copies for performance reasons (since kj overhead as kind
of on the high side for us already), so we'll use a shared-buffer class
that behaves much like a mutex. we *don't* use mutexes because ours are
cross-thread-capable and thus require syscalls for all wakeups they do.
Change-Id: I3b14925f5d9f5e07ea2cafdf00a88a64f79e4742
Found ... by edef's harness. There was more dead code it found, which I
didn't address because it felt less obviously deletable:
- `AutoDestroyCgroup::path()` / `::delegation()`
- `TeeSource`, `LambdaSink`
- `S3BinaryCacheStore::getS3Stats()` (maybe Hydra? idk)
- `fetchers::getBoolAttr()` (technically deletable but it would be
inconsistent with the others)
Change-Id: I913d2afd61e24136fb506389fc33a1016a6a6964
sometimes we need to disambiguate based on the kind of the thing we are
filling due to how the c++ type works. in templates some constraints on
type resolution are lifted. sometimes we can make the compiler help us.
Co-authored-by: piegames <git@piegames.de>
Change-Id: Id9549764813a862b8d64959157fed258709eb6ac
This is purely aesthetically bothering me, but [] inserting stuff into
the map is such a C++ coded design decision and we don't intend to
mutate it here so we probably should not do it.
Change-Id: Idf37c51175b031fb396e8fd53dc8fe6f2039fb55
Many issues stems from `<nixpkgs>` not resolving anywhere or having the
wrong version, let's make it awfully obvious again here.
Towards #230.
Change-Id: If630e9616566e77143f028cdcfe0b7f00a1d486d
Signed-off-by: Raito Bezarius <raito@lix.systems>
without maxprocesses we can utilize core-rich systems better, and test
timeouts are reportedly to be too low to run on small systems as well.
fixes#890
Change-Id: I89386b89fcd69ef4bf77ecf0a49c0b85e1f17c3d
this fixes a large portion of tests currently marked no_daemon. most of
them only needed to set some trusted settings, which is easily done now
Change-Id: Id5a5ee94951cdc92bddd2264c738ca4f98980c8b
Some code-paths expect e.g. `nixpkgs.rev` to exist.
This change makes sure this always exists. This is done on purpose
within this let-in dance rather than in the default such that
`nix-build release.nix --arg nixpkgs ../nixpkgs` also benefits from this
fix.
Change-Id: I94f1f901823d3df433e3c44ae380bb2f67eef70e
attrpathsSuperset got renamed to preEval in nixpkgs[1] breaking Hydra's
evaluation. Updating nixpkgs to make sure this is consistent now, no
matter if flake-inputs or release.nix is being used.
[1] See commit 19a31658dc39324c9acf8d81198cd7137bdc1e92.
Change-Id: I69ab3f894534b1d89b626ddd4feda8ea502b69b3
single-user builds have a writable home directory in the build sandbox,
and the test suite does not like cargo writing anything into there. the
impact of this seems to be localized to f1 (at least on linux), but not
letting cargo write into the homedir is easy enough to do. nixpkgs also
does something much like this when compiling rustc, so we're not alone.
(it also seems that nixpkgs intended to do this for all rustc users but
broke it at some point in the past. some packages set CARGO_HOME in the
same way we do, possibly because they've had exactly the same problems)
fixes#1251
Change-Id: Ie17a0677ef09bb076ef6295bb590a0895924434e
we need to cache the current terminal size for progress bar reasons, but
we don't want to catch SIGWINCH to update the current terminal size from
a dedicated thread for repl reasons. a SIGWINCH handler function is much
easier to square with these requirements than communication with another
thread to have it change its signal mask, and since any races in handler
code affect only progress bar output and only very rarely (if ever) we'd
better chose the simplest approach. the progress bar could set a handler
of its own for this purpose, but we would much rather replace it instead
fixes#1246
Change-Id: I814d9aaf1b6fbb6a8cefc5af675a3aa372549dc8
this is a bit of a hack, but since zngur cannot handle multiple trait
implementations per type yet we will have to commit to singles types.
Change-Id: I60e7b96bbeaa9fb87cf43662d4a9a5d44116bf47
note: only two of the tests actually needed migrating, as the others
were already covered (e.g. by lang)
Change-Id: I30abb2bd728ae2b144ad76566096a216aed3b8f8
mimalloc is a compact general purpose allocator from Microsoft. It
consistently outperforms glibc's `malloc()` in allocation-heavy
workloads, such as Lix's evaluator
It's currently only linked in the main `nix` executable, as Boehm's GC
uses its own allocator. Other allocations that *do* go through glibc's
`malloc()` are still much faster, though
Benchmarked on x86_64-linux against Nixpkgs `bd07873`:
| attribute | thunks | lix@`1396012` | mimalloc | uplift |
|-------------------|---------|---------------|----------|--------|
| hello | 206444 | 0.726s | 0.620s | 1.17x |
| chromium | 1177382 | 2.273s | 2.136s | 1.06x |
| firefox-unwrapped | 1394797 | 2.521s | 2.378s | 1.06x |
| texliveFull | 3186440 | 4.992s | 4.672s | 1.07x |
| nixosTests.gnome | 7905808 | 6.115s | 5.723s | 1.07x |
Based-on: https://github.com/NixOS/nix/pull/15596
Co-authored-by: Bernardo Meurer Costa <beme@anthropic.com>
Change-Id: I3ad92eacc075efeaf3d9f7730ada7b29835536a4
Some issues are caused by users confused about their ambient search
paths, let's analyze the current configuration and make it awfully
obvious what is going on.
Towards #230.
Change-Id: I953ef3feaefc56cee68623c5aaa4d568ba1fd034
Signed-off-by: Raito Bezarius <raito@lix.systems>
We try to access to the Flake registry, if we are failing, it's hosed.
Otherwise, let's record some facts we can reuse later.
Towards #230.
Change-Id: I80dd8272d5c27f7f7ff01052e3487a0496ee29fc
Signed-off-by: Raito Bezarius <raito@lix.systems>
Based on `nix-info` and `nix --version` output.
Contributes towards #230.
Change-Id: I13bb9608b733e10037c6816456b5312e712c18ad
Signed-off-by: Raito Bezarius <raito@lix.systems>
Based on `nix-info` output and experience (LOCALE issues).
Change-Id: I350fff9592e7acc8fba0babe67c9fe1bf94a5781
Signed-off-by: Raito Bezarius <raito@lix.systems>
This is some code that is specific to systemd to fetch hostnamectl
information via the JSON flag.
This will be used for `nix doctor`.
Errors will show up similar to this:
```
❯ ./build/lix/nix/nix doctor
[INFO] Nix system type: 'x86_64-linux'
warning: could not get host information via hostnamectl: error: Expected JSON value to be of type 'string' but it is of type 'number'
```
and makes them non-fatal.
Change-Id: Idd39f29626e770b1cd3896e8eb8545416c74b041
Signed-off-by: Raito Bezarius <raito@lix.systems>
treating a file with an empty or zero hash as locked is not helpful.
these are placeholdes for "hash is not known", thus treating them as
a valid lock makes them completely useless (and confusing to users).
fixes#1233
Change-Id: If42b47281e6973fc86662b973db69e26f6346f5a
25.11 is dead, and so is mdbook 0.4
Also reenable linkcheck which was doing nothing this whole time lmao
Change-Id: I4a8b9c763b881de840d6ef1b1d85bf19941f4386
These uses of std::move are taking data out of a repeatedly-used state
object, à la Rust's std::mem::take(). But move-constructing or
move-assigning from an object does not guarantee that the moved-from
object is equivalent to a default-constructed one, or even that the
moved-from object will continue to function like a non-moved-from object
of that type (only that the object remains "valid").
Assuming there's no differing custom allocator shenanigans at play, both
libstdc++ and libc++ do leave moved-from std::map objects (and probably
others) in an empty, well-behaved, default-constructed state, at the
time of this writing. However the correct tool here is std::exchange().
Change-Id: I7cf76762a1b05a542b53fcd74be100646a6a6964
This way, other parts of the codebase can use it.
Change-Id: I58673a6a56362692980426a6c8f70f5c6aff24d4
Signed-off-by: Raito Bezarius <raito@lix.systems>
Since d0190cff6f, the `justfile` uses
`$out` from the environment as an argument to `--prefix` in its `meson`
invocation.
The `$out` variable is... A bit spicy to use since its semantics change
depending on whether `nix develop` or `nix-shell` is used.
In a `nix develop` shell, a directory *relative to PWD* (and not
relative to the Nix expression) is exported. In a `nix-shell`, the
`$out` used comes from the derivation.
The alternative to this change is to make the `justfile` use a variable
name that would not conflict with a `nix-shell` and `nix develop` shell.
Since it wants to use that value for the `outdir` variable, this would
be the name I would pick first.
I chose to instead work around the issue within the `nix-shell` since
this would be breaking the existing interface. As tenuous as it is.
Change-Id: I64cd5fd85345457e14880cb133e9ddafd55a25da
rust-analyzer binaries that don't come from nix (such as the one
shipped with vscode's rust-analyzer extension by default) need the
`RUST_SRC_PATH` env var to be the path to rustc's sources, but this
wasn't set inside the devshell. this change explicitely adds it to
the environment. (trying to add pkgs.rustLibSrc and rustcSrc to
the shell's packages unfortunately does nothing, because they are
just pure source packages, nothing more.)
(the reason this probably didn't come up for other devs is because
1. a lot of lix devs use other editors that probably just use the
binary from the path, and the devshell installs pkgs.rust-analyzer,
which is patched to always have `RUST_SRC_PATH` set, and 2. because
pkgs.vscode-extensions.rust-lang.rust-analyzer is also patched to use
pkgs.rust-analyzer by default instead of the bundled one, so installing
the extension using nix instead of from the marketplace would also hide
this problem.)
Change-Id: Ia27daf2a88756311ac5485ea0af44aae242241a9
results don't allow for streaming or pipelining. transporting errors in
exception texts is *bad*, but it's still better than not having streams
that actually work. this isn't a great situation, but with capnp that's
pretty much the best we can do. still beats the other rpc frameworks :/
Change-Id: I2abf17bc5ea5de2baeea44ff4d7b2f4db33e98dd
using result types in capnp is fraught. while it makes some sense for
interfaces that need neither streaming nor pipelining and can provide
much better error fidelity there it's also fundamentally incompatible
with those that *do* need streaming or pipelining: streams will never
be stopped early unless an exception flies, and pipelines cannot look
through result types. likely the best thing we can do is to encode an
error for transport in the capnp/kj exception description strings. :(
Change-Id: Icb7d16238fa9a7aaf92c00363f7be4076ac02a61
This is a hack, but this is the best we can do with Just unfortunately.
Maybe rewriting this file into a python script wouldn't be the worst
idea at this point …
Change-Id: I32d089f6b49165faed99fb592d068dbb8a7e48e6
This is the real deal. The new tests have surfaced one bug in the test
suite itself, and one in the RPC legacy protocol wrapper.
Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: Id75b671c10b45655d3627abd88563e2d41719446
All tests should have been clear here, but alas
Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: Id642be1f6e0fb714b844ee5348dc096866cf3906
This commit enables the parametrization for "legacy-combined" protocol.
All failing tests with that have been either fixed or disabled. Notably:
- A couple of tests (e.g. involving builders) could be made to work with
a non-local store, but this would require some refactoring to testlib in
order to make certain configuration settings generic over local and
remote operations. We've disabled those for now, in order to make
progress
- The store tests all run locally only because of their nature
- The flakes tests also all run locally only because my energy for
fixing them is limited (and it's a *lot* of test failures, with probably
little overall benefit in terms of test coverage)
Change-Id: I56fa249a64f7c17c952f688ec89e9687f2a13f12
this takes the form of a couple of hideous macros for now to get started
without worrying about semantics of rust logging and tracing crates, all
of which are in some way incompatible with the current lix expectations.
we do want to move to using those eventually, but that will have to be a
task for later. as long as only small amounts of lix code live in crates
we don't have to worry much anyway, and by the time we get around to any
Activity beimg ported we'll hopefully have a better understanding of the
requirements on either side. we'll reëvaluate the log infra around then.
Change-Id: I4aec476a486f56d0972294358cbf0b6810ab9394
This is a second attempt at https://gerrit.lix.systems/c/lix/+/5516
instead of letting meson try to be cargo we'll just have cargo at home.
this requires some contortions to link everything together due to quite
a few meson deficiencies, but at the end we get to pretend that rust is
just c++ painted orange. we'll use cxx to bring real interop back soon.
also fixes#1230
also reverts 10845bfe63
co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: Ib1fda843fa80d818705d5a65ec9054216a6a6964
This first geralizes the `nix` fixture with `pytest_generate_tests` to
iterate over all protocols (unless they are marked with `no_daemon`),
though for now the list of protocols is set to be empty. The lang tests
are all tagged with `no_daemon` because they are mostly pure and running
them multiple times would be wasteful.
Co-authored-by: rootile <lix@rootile.de>
Co-authored-by: piegames <git@piegames.de>
Change-Id: Ib407edb420ba4bf434cacf9563a71f5ae6fa8eef
This is a bit of a trip.
Commit 63b0a01d75 moved config
initialization out of static initializers.
In doing so, an oversight was made with the new
`GlobalConfig::registerGlobalConfig` calls. Previously all global
configuration would be handled well before `loadConfFile` was called.
The change placed `registerGlobalConfig` in `main` *after*
`initNix`. The `initNix` function is what calls `initLibStore`, which in
turn callse `loadConfFile`.
So, in effect, the configuration for `daemonAuthorizationSettings` and
`nixDevelopSettings` are not actually loaded.
Change-Id: I10dba2c98e032f910942ee0e0ca6bb39ef60edc2
Full warning was:
warning: Dependency of package 'lix-shell-env' uses a nested list in attribute 'nativeBuildInputs'.
This is deprecated as of Nixpkgs release 26.05, and support will be removed in a future nixpkgs release.
Change-Id: I6b6c7425e1b03d3238ea21138cb3f2cf6a6a6964
they're not doing anything, and they're slightly in the way of reworking
the rust build infra. we'll add them back once that rework is completed.
Change-Id: Idde8ba3585fd08b89b4b4298349f8f58a29a9661
blokyk on Matrix found that is not only no longer needed but now
actually breaks the very thing it sought to fix
Change-Id: I39cc0df611a6d81c763063c9fd2f0f426a6a6964
When removing the f1 repl characterization test suit, these files were
kept on accident, despite them being dead code now
Change-Id: I13b2c5a96004787c26fe7209e4cb3aeedd8ada15
This reverts commit 0d6b372c19.
Reason for revert: In the time between this getting +2 and it being merged, all repl tests and its syntax have been migrated to f2, making these changes null and to dead code
Change-Id: I00f5237bf3f2ca3a60cc1584f67fb67196805e7e
Gosh. Why was there never a README in here explaining what it does
without requiring people to read C++? There was a half-assed grammar in
a comment, but it was very half-assed, so let's replace it with a
better-written one.
I'm documenting this so that neither I nor anyone else have to read C++
while rewriting the parser in Python.
See: https://git.lix.systems/lix-project/lix/issues/1196
Change-Id: Ia34ea7b02f109eb7753833e1c03d4b5d1d3c3b75
When extracting a global pyproject.toml from the f2 one, some f2
specific things slipped through and ended up in the global one.
This commit moves those parts back where they belong
Change-Id: I95706fabec487410717fb362c95bfee900cf5b8c
if the netrcFile setting has been changed tell curl that the file is required, so
fetching fails if the file does not exist
if the caFile setting has been specified, check if the file exists and throw an
error if is it non-existent
Refs: #1106
Change-Id: Icb9330a7a715175d35b9ed894ed945f0fd4d7061
So apparently we can still use the repl if the input file fails, as long
as we have the debugger enabled.
This Commit removes the "only one output block for should_fails"
requirement to allow this behavior
Change-Id: I02568c8d7e4305ab6414953306650bbe13c6f4bc
curl does not support this configuration. we can't easily change the
config setup to mirror how curl works, but we *can* print a warning.
fixes#1223
Change-Id: Iaa1583275d9cc9e5eed5552210db183a8e0628c8
the case-sensitiveness of the "accept all options" prompt (and the fact
it silently defaults to no) was confusing a few users, so by rewording
it slightly we can avoid that case-sensitiveness and make the prompt
clearer. for consistency, i also rewrote the individual accept prompt to
make it case-insensitive.
at reviewers' requests, i also reworked the code so it re-prompts the
user on invalid answers.
Change-Id: Ie786c0f26a0aa752b22d5a5e8af1b2c7ca2846b5
lix.pc currently lists several optional libraries (such as libcpuid and
libseccomp) as required dependencies. Update the pkg-config generation
step to only list optional dependencies which are used by the Lix build.
Change-Id: Iceea7ecc9421b5b10fc08edbe95a063a4e39cea1
what the actual fuck. swallowing all errors *except* EXDEV is not okay.
renames do not do this, so moves should not do it either. luckily we do
not use moveFile anywhere except the store path registration code. this
may or may not have caused problems in the past. probably nobody knows.
Change-Id: I2b0255a5703983cbd129abc3219c11ac7171fd12
catch and rewrap std::fs errors in some meaningful way, otherwise lix
can crash with foreign exception aborts if filesystem failures happen
that aren't already being caught. renames in particular can fail with
incomprehensible errors in even pretty normal circumstances elsewise.
fixes#1219, fixes#995
Change-Id: I0b1e5696d300aac3422e43cffc17a2426803d64e
I'm not sure why this wasn't an issue before, but the generation of
jobset constituents in Hydra trips over this. The Hydra-convention is to
have the `system`-attribute last anyways, not in-between.
Since this is a single attribute, get rid of it for the sake of
simplicity.
Change-Id: I80ef1b92ed158c8712ee899534ce82130614f338
This is a Hydra jobset definition to be used to warm up a cache with Lix
versions built against several nixpkgs versions. This is only happening
after a bunch of tests have passed.
Change-Id: Id0b0e50c08581f3f377c7c793d2fd83a6007999d
If e.g. i686-linux is not in the list of systems and thus no attribute in `nixpkgsFor`
exists (e.g. on hydra.afnix.fr), don't try to instantiate it.
Change-Id: I118e2af5289ad7bd95ce7f1e6b0728afb6f0321f
...expect for the dev-shell: while it doesn't seem reasonable to me to
even build dev-shells in post-merge CI, this should still remain part of
the pre-merge checks.
Change-Id: I5cb9b4cf53484f884d694770ed94d0dc0862117e
Step II to move the code out of flake.nix. Done in a second intermediate
commit to make it easier to review.
Change-Id: I13fb386f2e9b87b1d28c366e9b7f0d2dbe03315c
That way we can use it in non-flake contexts down the chain.
We don't override the `nixpkgs` anyways since this is a test artifact
that needs an old nixpkgs input, so this isn't a concern here anyways.
Change-Id: If2aa8f7171a445e8321ef449026e206cc5a0fe3c
The upcoming change factors out a lot of code to make it reusable for a
"legacy" Hydra project that we use to build a matrix of (Lix x nixpkgs)
to have a warm cache for everyone.
The Hydra project will not use flakes to have some inputs overridable,
so the change is needed. In non-flake mode we don't have a notion of
"self", so rename it to `lixSrc` such that the next commit is purely
about moving stuff without changing any aspect.
Current stock Hydra doesn't expose `lastModified` though, so right now
that is a cache-miss. There's a pending Hydra PR fixing that[1].
[1] https://git.lix.systems/lix-project/hydra/pulls/82
Change-Id: I59d70b8b874f576c782217e7bca099eda4c7add9
The reference to the flake is used in a single place only to get the
`doc` output of Lix. Remove it so that we can also instantiate the tests
in a legacy Hydra jobset (happening further down the chain).
Change-Id: Ie3eccf42d67c81fcf991342097783e5067d512bc
Noticed on my quest to make Lix buildable against multiple nixpkgs' to
provide pre-warmed caches for the flake on AFNix's Hydra.
`pkgs.systemd` has a new patch policy[1] that only allows patches for
hotfixes and fundamental incompatibilities. As such, the patch that
allows nspawn workloads to start without a `/usr/bin` was removed with the
recommendation to create that stub yourself.
[1] https://github.com/NixOS/nixpkgs/pull/488508
Change-Id: Iaa35ae3b8971867f12daeac5f94958c20a7d9233
The option has been broken since Nix 2.4. The flag was accepted, but not
used. This change plumbs it through to computeFSClosure.
Change-Id: Id6adee8ea8a6a4f457b24b650660f319e3daa2b2
This makes it easier to build lix on a different file system from the
source repo, e.g. symlinking ./build to a directory in /tmp.
The ./outputs directory is included for completeness as another
significant source of generated artifacts.
Change-Id: I1bb4c21c700beacb4882a1be473dd31353edd177
this logs unpackTarfile calls as an `actUnknown` activity, which
unfortunately doesn't report any progress for now. for the reasoning
behind this choice, see the previous iterations of this CL (cl/5590)
as well as the attached comments (the crux of the conversion is in DMs,
unfortunately). in short:
- i could't make a new ActivityType because that would be a breaking
change to the daemon protocol, including with previous lix versions
- i can't use actFileTransfer because that would show up as a download,
which would be confusing for both users and tools
- i can't use actCopyPath /or/ actCopyPaths because that would break
NAR/substitution accounting when doing remote builds
also, ArchiveDecompressionSource, which is apparently used for
decompressing logs etc, does its own custom decompression without using
unpackTarfile, so it is not affected by this. from what i can tell,
unpackTarfile is only used for:
- fetchtarball fetcher
- git fetcher
- channel unpacking
- nix-prefetch-url
Fixes#1213
Change-Id: I68f3be520681d8889292353dd41ab19a60c5eeea
By default, when one uses `in` assertions, upon failure newlines in any
of the two strings is escaped, and everything is printed in a single
line.
This commit allows one to pass `-vv` to print out the left and righthand
side unescaped, with each line of the output being its own line,
improving readability a lot when dealing with multiline outputs.
Change-Id: Iaf67a20fce57d375e2c62a8e8a0957b14eec26ef
It's software archaeology time.
- In 2019 (Nix 2.4), Eelco added `checkOverlays`, which strictly checked
that the overlay must be a function of two arguments, one called `final`
and the other `prev`. (dc3f52a144,
gh#3573)
- In 2024, NofairKing opened an issue about checking the exact name
being a silly idea (gh#10516). Instead of trying to make the check more
sensible, the first fix attempt simply expanded the check to allow
`previous` instead of `prev` (the author's preferred name for the
attribute, supposedly; gh#10553). After some discussion, instead a PR
got merged which simply removed the `prev` check altogether, while
leaving the `final` argument check intact for some reason
(ad65a50a94a97bf1f1a1902f43542d28a2e8206b, gh#10572).
- In 2025, over at Lix and oblivious to the latest change over at
CppNix, I came across this stupid check while trying to improve the
abstractions of the AST in Nixexpr. I spent (wasted, retrospect)
considerable amounts of time removing the exact name checks while also
improving the check overall (making it check for more than two
arguments, and also generally improved error message).
(0928d4d87a,
Id4244171123dd8a228be71ce9f04d8e9f647c111)
- Fast forward to 2026, where I run into this becursed piece of code
once again, and once again during some Lixexpr cleanups. Now that I have
seen how carlessly the issue was handled at CppNix, I can't be arsed to
give a flying fuck anymore. Out the code goes. Bye.
With this commit, all casts on `Expr` subclasses outside of libexpr
itself have been removed. No more violation of abstraction boundaries.
Good riddance.
Change-Id: I939968bb01d461764cfa0f4ea7152b4fcf1acf93
pour the foundations for rpc, and let them set. this is very much
unstable and must be opted into with explicit store uris (e.g. by
setting `NIX_REMOTE=daemon?protocol=any`). the daemon sockets are
not enabled by default and must be enabled with the `rpc-sockets`
experimental feature. we will not advertise this just yet because
in the current state it one has to be *very* dedicated to the rpc
cause to deploy this, but once we have some more bits migrated we
may want to add release notes and officially as for beta testing.
Co-Authored-By: piegames <git@piegames.de>
Change-Id: I85a96ccb700b91190c1eb37154bcc6ae1c03401a
splitting it off lets us reuse it (and all the legacy protocol handling
it encapsulates) in the rpc world as well by just tunneling wire bytes.
doing this allows incremental migration of to a new rpc protocol, until
we are finally ready to call the rpc protocol ready and remove the link
to the legacy protocol altogether (and maybe merge this function back).
Change-Id: I8f2a8795c4a9d713ebbbbfd02e9eec5271a2b7a9
this adds the protocol registry. we also extend the systemd unit setup
to allow adding new protocols without much fussing, and we move deamon
instances into protocol-dependent scopes. this will make monitoring or
resource limiting of individual protocol daemon instances much easier.
Change-Id: Ic9da08eb074868b6e57385075c5502b081e8461f
we'll use this to create a registry of protocols the system understands.
also use this struct during connection setup to make that simpler later.
Change-Id: Ifa481fea7ea2efa2a1f1be4d81076a9e022d24f9
inherited classes to stack of variants
Previously the code represented the stack of unclosed arrays and
objects as a linked list of virtual objects which seemed unnecessary.
There is a closed very small (3) number of possibilities, and one of
them only exists at the top level. A vector of variants seems a lot
simpler, and as a bonus, more performant.
Change-Id: I6bc429d682d22f69ca91dc0518e6be646a6a6964
string data shares a buffer with the binary string length field. size
calculations for string read buffers always include the length field;
sufficiently large length fields can cause these calculations to wrap.
a malicious nar could use this for OOB writes in the daemon (as root).
since we use strings only as tags for archive members and for symlinks
with their OS-dependent length limits we can simply limit string size.
1 MiB should be sufficient for all symlinks, and tags are always tiny.
Change-Id: I89fb05f73c1dbeda45d91244aba4cd526a3d83e1
These parameters are now created on https://s3.afnix.fr.
Change-Id: I96b6fd913429ee46d04c412cb141edd288665ced
Signed-off-by: Raito Bezarius <raito@lix.systems>
this is not read in code paths that are called repeatedly, and the only
reader can never see it being false. we also question the wisdom of the
flag in the first place; why should opening a connection fail because a
previous connection did? conditions may have improved a lot since then!
Change-Id: Ib7c219e6239432d99a22b10024847218469f1cd4
This isn't really relevant for a setting of 10k, but now that we want to
set it to 0 for Flakes it makes a difference as to whether or not one
top-level function call is allowed or not. (It shouldn't be)
Change-Id: I6bbf99826af0289c232b9d45d172a378b81abe4b
Caught by edef's harness. I rewrote the structured build log tests in
f2 and added a test for this one.
Before: it failed the build altogether:
```
building '/tmp/jade/pytest-of-jade/pytest-77/test_invalid_fields_files0_0/nix/store/skhjs5zs082nqiakj69qz32gyywz0v1a-unusual-loggi
ng.drv'...
error: unsupported JSON type 7
```
Now the output is as expected:
```
warning: Unable to handle a JSON message from the derivation builder: error: unsupported log field type number
@nix {"action": "start", "fields": [1.5], "id": 2, "type": 1, "level": 1, "text": "abc"}
```
Change-Id: Idbd28c5a4d4cf15f27207f83b730e7946a6a6964
This is a minor adjustment to 2b22dae1ba /
I3e9d7c1c7a6599a8e68302448bbb961d051002b7
Basically having the error point to the operand instead of the operator
kind of makes sense, but it would require error spans to *truly* make
sense, and in the meantime maintaining the logic is more hassle than it
is worth.
Change-Id: Idac98dd77a0f4a6cb386cb74e0d4eb5fd2f503e5
"while evaluating a path segment" was just plain wrong
Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: I827374635df2887d17a8ee8b1755c078bf0b506b
It's useful to obtain a fully assembled command and then wrap it to
execute into something else, e.g. gdb, strace or fakeroot.
Change-Id: Ida5928fd001925549eab89edfbf0facada7ef7f1
We choose to throw an error in the invalid-input space where we *can*
produce a valid (but wrong) result to allow ourselves to, mirroring
CppNix, change the semantics to not corrupt it at all in the future,
while having a middle state where it is an error.
This is a largely-rewritten version of
https://github.com/NixOS/nix/pull/13013.
Co-authored-by: rootile <lix@rootile.de>
Fixes: https://github.com/NixOS/nix/issues/12899
Upstream-PR: https://github.com/NixOS/nix/pull/13013
Change-Id: I6a6a6964cdd1a88714952e80c660d1fa57d1f2d8
This was found by edef's harness. Unfortunately this entire module has
no coverage, so I don't really want to write a test for it immediately
as I'd have to harness it (though we have an http server in f2!).
(I think it will be much more fun and rewarding to do so once we get
coverage up and running).
Filed a bug about the missing docs for this too:
https://git.lix.systems/lix-project/lix/issues/1190
Test plan:
```nix
let pkgs = import <nixpkgs> {}; in {
foo = pkgs.fetchurl {
url = "https://jade.fyi/robots.txt";
name = "blah";
hash = "";
};
}
```
Before:
```
$ nix-prefetch-url ./foo.nix -A foo
fetching path input 'path:/nix/store/knf5m0famajw2bzbsg72b052iy1ynpck-source'
path is '/nix/store/5rcgj95as1ywr463rva6bky18088ij4x-robots.txt'
1cr11i0ih4m6kkpks9fxil71wg5k2crikah1rfjgy607qzgy8vad
```
After (note correct path name):
```
$ nix-prefetch-url ./foo.nix -A foo
fetching path input 'path:/nix/store/knf5m0famajw2bzbsg72b052iy1ynpck-source'
path is '/nix/store/hc9drd1l3vki3bbaacwhi8j1dr9907vd-blah'
1cr11i0ih4m6kkpks9fxil71wg5k2crikah1rfjgy607qzgy8vad
```
Change-Id: I6d9da1c54c6a398a0a169ff01f3f3dce6a6a6964
before this change, the innermost frame was all the way at the top,
which means that in very nested code you'd have to scroll though
dozens or hundreds of frames before even seeing where you are, because
the last lines (which are immediately visible when the command returns)
would be the nix code's entry point and other outer frames instead,
which is rarely relevant.
this CL reverses this order, so that the innermost frames are the last
ones printed, and thus are immediately visible. (note that this is
already how errors are printed by nix in other contexts anyway, the
debugger's :bt is the only thing that prints the trace in "forward"
order.)
because `DebugState.traces()` use the homegrown `libutil::Generator`
instead of an stdlib container, we have to do that reversing ourselves,
in this case by just storing every element in an `std::list` (in reverse
order) and then traversing it. this is feels kinda dirty, but i don't
know any other way, and it's not exactly performance-critical.
Change-Id: I9f23e40e57f72a251d65a335b3bba3c3f77b935d
it seems like the repl_characterization tests have not been actually
running correctly for a while, since tests from more than 2 years ago
that should have failed have been in the test suite and OK'd thus far.
the reason why some tests passed when they shouldn't have is because
the AST nodes used by the characterization parser didn't have a correct
comparison operator, so the tests would only fail when one of the output
line was of the wrong *kind*, but not if it had the wrong *value*. this
was fixed by simply adding a default comparator to the TextNode type as
well as every derived class.
i've taken the occasion to fix the tests that were failing now that we
are actually being strict about them. most of these are pretty boring
changes like indentation, but if possible i'd like to get an explicit
ACK from at least one or two people on the core team that these changes
are okay.
for reference, here is the zulip thread that lead to this CL:
https://zulip.lix.systems/#narrow/channel/8-Testing-.2F-functional2/topic/.E2.9C.94.20repl-characterization.20tests.20not.20failing/near/10276
ps: i had to turn clang-format off around the macro for text node
declarations because it re-formatted the entire macro, causing a lot
of noising and producing uglier code; if this isn't okay or if there's
a better alternative, please let me know!
Change-Id: I1d6d92eb3e9cb705ecfeccd7b9294d6cfd7fe25b
I missed some cases in https://gerrit.lix.systems/c/lix/+/5490, which is
not surprising given that I didn't have a lint for it at the time.
Some of these have JSON deserializers, which is kind of scary (in that
it was UB). Probably no impact, again, due to the lack of
-fstrict-enums, but still, yikes!
Change-Id: I895a8bfdd9972d8ade9fa6f5aa30845d6a6a6964
This started by noticing that the coverage report generated by cl/5489
showed that the error message for when allow-import-from-derivation is
disabled appeared to be not covered by any tests. Since the existing
tests for IFD were in functional, which is deprecated, I migrated those
into functional2 and added a test for erroring when IFD is disabled.
Change-Id: I565674de3dd693e023ec99d501adc4706a6a6964
as of cl/5456 prim_importNative default constructs a `Value` in order to
pass as an out parameter to a ValueInitializer function. Default
construction is deprecated, but there isn't a meaningful value to
initialize it to instead. After cl/5357 this will give a deprecation
warning (which in an asanBuild will be an error), so this commit
suppresses that warning for that `Value` construction.
Change-Id: Ie567a6e81b5753672001320e9fc816c56a6a6964
The existing function is used with good reason as explained by its
comment. This suppresses the warning it creates.
Change-Id: I58b0de3f5854266fa77b6a06b7821a636a6a6964
Motivated by needing to remove a Value default construction from
`ExprVar::maybeThunk`, so part of #744
Change-Id: Ia12b5b019e49055ebd7ffa0cc39a59c06a6a6964
Found by edef's fuzzing. We::jade don't think this is security-relevant
or likely to be mistreated by a compiler, but UB is bad.
https://eel.is/c++draft/expr.static.cast#8
> If the enumeration type does not have a fixed underlying type, the value is unchanged if the original value is within the range of the enumeration values ([dcl.enum]), and otherwise, the behavior is undefined
Notably, the range is defined as the smallest bitfield type that could hold all the values, not any real type which exists: https://eel.is/c++draft/dcl.enum#8.
This is basically a footgun, and I'm writing a clang-tidy check to forbid casting to such types. But first I needed to write clang-tidy testing infrastructure: https://gerrit.lix.systems/c/lix/+/5493.
Change-Id: Ieaaa0fe2a92fd9f24f60761e312741186a6a6964
notably the bad-store test previously tested *bash*, not lix. we also
need to extend the list of non-fatal errors, but that's probably fine
Change-Id: I983f4adc8047fe800323099de4037711869ee547
this should improve error messages when connecting to long socket paths,
and make it at all possible to detect ENOTDIR from connect helpers. this
will be needed to handle daemon sockets that live in directories we have
not previously considered like the protocol name directories we need for
rpc purposes, all of which would have resulted in hard errors previously
Change-Id: I4e16cf09fedd07c93cfdf7b0f151106984507315
The link should be to Gerrit, not Forgejo.
Issue introduced in: https://gerrit.lix.systems/c/lix/+/5005
Change-Id: Ibf93f114303f59959621a265196e6eda29b315f9
I did a pass through all files, fixing bits that have annoyed me here
and there based on vibes. Roughly:
- Replaced some out values like bools or strings with lists where
appropriate. Those tests were likely older than the value printer
- Added some parser tests where it makes sense
- Gently touched some formatting woes, while trying to not cause too
much diff noise
- Removed some dead `with (import ./lib.nix);` code
Change-Id: I8c40b2110f0b7799f68ae38ba61f049c5f1f6ee8
Related to #1162
Currently if JSON with duplicate object keys is passed into `builtins.toJSON`,
it will silently drop all but the last value, keeping only the last.
This may be surprising, but as this has been consistent reliable behavior that
users might depend on, we should test for it.
Change-Id: Icc2adefabb161530e7cbfa7330919bab6a6a6964
Without this, the primop chokes on any thunks on attributes passed in
the attrset. It even is the reason why the test contained `builtins.seq`
to work around this. Supposedly, this might have been an intentional
restriction and changing this might break things in ways I cannot forsee
due to not knowing much about Flakes, however the status quo is equally
broken:
- The error message looks like an internal error and not like some
explicitly forbidden invariant violation.
- Seemingly simple syntax literals like "-1" compile to "__sub 0 1" and
thus create a thunk which then fails, which is utterly confusing ("why
does 1 work but not -1?")
- This is a stark violation of the principle of least surprise.
- Thunking relies on maybeThunk and thunk inlining optimizations, thus
not forcing thunks turns operational details of the evaluator into
language-observable behavior. That's bad.
I am changing this now regardless of the risk of breakage, because the
bytecode evaluator will have different thunk inlining optimizations and
thus inevitably cause mismatches in behavior anyways.
Change-Id: Ifc45c4d2900e40822383670b28e4e50ab8af317a
Although the fact that some characters are illegal in derivation names
was referenced in some parts of the docs, the actual list was not
documented anywhere, and the only source of truth seems to be the
[store source code](lix/libstore/path.cc).
This adds an admonition in the `derivation` docs that specifies which
characters and strings are actually allowed (especially since some
of them can be somewhat surprising, like the string "..-foo" not
being a valid name because of `..`).
Change-Id: Ibc4b4a141e262c21306ce9c0392f92cf0abe610a
the docs say that the arguments for an overlay is info/prev/final, but
it's actually info/final/prev. this fixes the docs both in the nix.conf
manual as well as in the `nix repl` help text
Change-Id: I3de5c122e7e77cc3f4e4550e3741d6a6e906ea20
That way we can automatically push the static build (compressed as
tarball) into the AFNix S3 bucket and have it updated on each update to
main.
Change-Id: I1002727ab12c0ab6b95c8d37ae4620996607d7b7
This applies the suggested change in #1179, replacing a dead link with a
link to the content's new location. Closes#1179.
Change-Id: I543e3308516218e224e495b8cf14a7c06a6a6964
This is a small tweak to the logic added in cl/1511 to detect explicit
throws when printing stack traces. Now when deciding whether to print
"caused by explicit throw", it checks not only that the error is a
ThrowError and that we are in a throw, but also that the ThrowError was
thrown by *this* throw, and not by another while in the process of
evaluating this throw's operand.
It turns this:
```
let
set = {
inner = throw "nested throw";
};
in
throw set.inner
error:
… caused by explicit throw
at /pwd/in.nix:6:1:
5| in
6| throw set.inner
| ^
7|
… while evaluating the attribute 'inner'
at /pwd/in.nix:3:5:
2| set = {
3| inner = throw "nested throw";
| ^
4| };
… caused by explicit throw
at /pwd/in.nix:3:13:
2| set = {
3| inner = throw "nested throw";
| ^
4| };
error: nested throw
```
into this:
```
error:
… while calling the 'throw' builtin
at /pwd/in.nix:6:1:
5| in
6| throw set.inner
| ^
7|
… while evaluating the attribute 'inner'
at /pwd/in.nix:3:5:
2| set = {
3| inner = throw "nested throw";
| ^
4| };
… caused by explicit throw
at /pwd/in.nix:3:13:
2| set = {
3| inner = throw "nested throw";
| ^
4| };
error: nested throw
```
Notice the difference in the top context frame. Before it incorrectly
attributed the throw error to the both throws instead of just the one
that actually threw.
Change-Id: If3b0b8311f1ae2ff1471e260fe59d9166a6a6964
Admittedly, I do not understand what the comment means:
> As soon as Nixpkgs updates to >= 3.0.0, change to lowdown_2_0!
But this should work fine for nixpkgs versions providing lowdown 2.x or
3.x, and not cause rebuilds of the nixpkgs Lix/Nix derivations.
Change-Id: I6fb9e775335b4dbc23add76b98106970956e19fb
When index-debug-info is enabled and the store path being copied is a
regular file (not a directory), std::get_if<nar_index::Directory>
returns nullptr since the NAR root is a File variant. The loop then
immediately dereferences buildIdDir->contents on the null pointer,
causing a segfault.
Add a null check at the top of the loop to break early when the NAR
root is not a directory.
Change-Id: I3a6e792b84cc12c837ecaddf4fee889e1bcb6397
A significant part of #1136.
This is a rather large cl, because all of the primops need to be changed
together.
Change-Id: I7d92698d5344bd6186ee4fa47f5c21966a6a6964
I just copied and slightly edited the `man nix.conf(5)` docs about
`repl-overlays` to the nix repl help text (but cut out the more
complicated example and redirecting to the nix.conf docs for more
info).
Fixes#303
Change-Id: I85efaef606d8779ac66fe72cd3947d663f33fa44
This input was added in cl/2884, but was unused even then. The
`linuxPackages` within `buildPackages` is used instead.
Change-Id: I71c522ef683aa098eac0b356b22007ba6a6a6964
Calls to JSONState::add() always followed an assignment to
JSONState::value(), resulting in the new value going through
JSONState::v for no good reason. The only use of `v` outside of a
pointless stepping stone for `add` was to store the final top-level
Value before it is read out by JSONSax::result(), so it really only
makes sense for the top level JSONState to contain Value field.
Change-Id: I1758c7b770eb4b0c122e501b764dd42b6a6a6964
We would've had `build-lowdown_2_0.aarch64-linux.aarch64-linux`
otherwise. This is a bit of a problem because my way of generating a
list of constituents for Hydra's per-architecture[1] release job stops
recursing once it encounters a system.
The alternative would be to switch `hydraJobs` to `<system>.x.y` or
switching Hydra to "legacy" jobsets. For the latter I'd prefer to do the
same for Buildkite such that we don't have diverging things to build
depending on pre/post-merge CI.
[1] per-architecute because the trusted AFNix builders don't support all
the architectures we support in Lix and with Flakes there's no way
of parameterizing the list of supported systems.
Change-Id: I33f31260caf86ed5bb0f728770ca3cf1c00adf31
Closes#496
When running
nix-build ../nixpkgs --arg config.allowUnfree true -A hello-unfree
the package `hello-unfree` is now built rather than getting an
eval-error rejecting instantiation. This is because `config.allowUnfree`
is now interpreted as nested attribute-set declaration, similar to how
it's done in `nix repl`.
To prevent sudden breakage, this behavior was carefully deprecated with
Nix throwing an error if the identifier for `--arg` is not a pure
identifier, but an expression as above.
Any kind of merging is rejected. I.e. doing
nix-build ../nixpkgs --arg config '{cudaSupport = true;}' --arg config.allowUnfree true
is prohibited. That way we don't have to think about merge semantics for
cases like this (or even worse `--arg config 'rec { ... }'`). Another
nice side-effect of this is that we don't need to create an EvalState to
force the values and implement merging.
Change-Id: I8b560883a4468a3f32f915764b08f5fdd8fe71bb
While it doesn't make sense to have `foo..bar`, the attribute-path
`foo."".bar` is valid and shouldn't throw.
Change-Id: Ifcddaad6233c6ba8f17cb5c953c2101d276dfeb6
This is a little more elegant and easier to reason about than prefixing
strings with whatever type the rest of the string is.
Change-Id: I7769535303dcb9f67b79e89bef162beec990e2a0
previously, `builtins.break` didn't force its argument, resulting in
a value wrapped with `break` being opaque to most builtins if not
also wrapped with `seq`. see [lix-project/lix#1165] for more details
on what this can break.
this tiny fix just adds a call to `forceValue` inside `prim_break`,
but unfortunately this "breaks" a few existing tests because it
changes the call stack; those tests' golden outputs have been adjusted
without modifying their intended purpose.
Fixes#1165
[lix-project/lix#1165]: https://git.lix.systems/lix-project/lix/issues/1165
Change-Id: I5fe4ee3ff28b38aaf924125b8978130812e58fef
This job is used to indicate that all relevant Hydra jobs of an
architecture have built. The idea is to build some CD mechanism on
AFNix's Hydra to e.g. auto-update the nightly manual.
See https://hydra.afnix.fr/jobset/lix/demo for the current setup.
Change-Id: I41724c5884a068bbe41407ab30f8edf8e4914001
See [lix-project/lix#1156], but basically currently the `:st <n>`
debugger command doesn't allow any negative indices, and putting a plus
sign in front of the arg doesn't change anything; thus, we can exploit
that "design space" to allow users to move between different stack
frames easily, by simply prepending their arg with a +/- sign.
The actual behavior is little more nuanced when you account for errors:
as suggested by @pennae (thanks! :), when the user inputs an offset that
would result in an invalid frame index, the debugger instead clamps it
to the closest bound (i.e. 0 for negative offsets, $maxFrame for
positive ones) and just prints a warning.
Fixes#1156
[lix-project/lix#1156]: https://git.lix.systems/lix-project/lix/issues/1156
Change-Id: I02a0cdb6aaebbdb0515308880a3bf9c0d2fcd25e
The default constructor of Attr default constructs a Value, which is
itself deprecated. Therefore the default constructor of Attr must
itself be either deprecated in turn or removed. The default can't be
trivially deleted because Bindings::EMPTY depends on the default
constructor of Bindings which depends on Attr's default constructor, so
I'm settling for deprecating it for now.
Part of work towards #744
Change-Id: Ie34b08788780615c5478a0354122530b6a6a6964
This makes the removal vector a vector of Symbols instead of Attrs, and
uses a custom Compare to still be able to std::set_difference them.
std::ranges::set_difference **should** be the perfect function for this,
but because for some reason it spuriously requires
`std::indirectly_copyable<I2, Out>`, I can't use it here. This
defficiency has bee recognized before [here](https://github.com/cplusplus/papers/issues/1021),
but no one has driven the fix forward.
Part of #744
Change-Id: I6d2c016ea41e033bf38836f859541b506a6a6964
The manual for the builtins and nix.conf currently has inconsistent
indentation, which causes some of the descriptions to end up being
partially treatedas code blocks in markdown (and thus the manual).
This was simply caused by the template string for the docs having
too much indentation before the description is inserted, so this fixes
that 16-bytes mistake.
Change-Id: Ia264e3b1abb20430109029d07a2d2b0a1a726bd4
currently, the `:st <n>` command in the debugger will simply silently
fail if the argument cannot be converted to an integer, or if it falls
outside the range of valid stack indices. this isn't too big of problem,
but it can be nicer to tell the user something went wrong, rather than
not give them any output and having them guess (esp. in the second case).
this commit adds two errors, one for each case:
1. the argument is not actually an integer, or is outside INT_MIN/MAX
-> "argument '%arg' is not a valid integer"
2. the argument is an integer outside the range of stack traces
-> "stack index must be between 0 and %max_frame, but was %arg"
Change-Id: I8109feeede79a9ad3db9ee7dc95d37e7dd19741a
NixOS 25.05 which distributed lowdown < 1.4 has been EOL for a bit now,
25.11 ships lowdown 2.0.4.
Dropping support means we can tweak the lowdown options for terminal
output which have been added in 1.4.0 without having diverging behavior
in possible builds of Lix.
Change-Id: Icae97cf5e9e680766b8a6f4e85514f4c4625d1dd
lowdown 3.0.0 merged some flags into one to save on bits and did not add
any aliases for backward compatibility.
As with the changes for lowdown >= 1.4, we define a preprocessor flag to
gate the changes on and add a job to CI to ensure that lowdown < 3.0
keeps working (which is used by NixOS 25.11).
Unfortunately, we need to jump through some hoops to prevent nix and lix
from upstream Nixpkgs from being rebuilt due to a changed lowdown. Since
both implementation's packaging in Nixpkgs has their own package set /
fix point now, we can't simply inherit them from `prev` since they will
always be (re-)computed from the `final` fix point. As a consequence,
we need to expose our changed lowdown version at a non-default attribute
or break the builds of Nixpkgs derivations we test against.
Change-Id: I20a3e2fdaa05906f032ff66911c42867557fdd11
base classes
`ExprLiteral::v` needs to be initialized in ExprLiteral's constructor,
but at the time that it gets initialized subclass fields don't yet exist
so it can't reference them. Previously, `v` was default constructed and
then later assigned a proper value, but this creates a problem when
attempting to remove all default constructions of `Value` from the
codebase. This commit moves the backing fields of each of the subclasses
into new base classes, which exist just to make sure they get
initialized before the `ExprLiteral` base class.
Work towards #744
Change-Id: Ic6d24cab474460b113f2fbcc8d92ad266a6a6964
ahead of time
Rather than creating fixed size vectors of default constructed Values
before assigning to those elements, reserve the desired capacity and
then push created values onto the vector. This avoids default
constructing any Values.
Part of fixing #744
Change-Id: I36eff4275a893b181eaf3ce145b1ee446a6a6964
Progress towards #744
`alloc` default constructed a `Value` which is a problem because the
defaut constructor of `Value` is deprecated
Change-Id: I789cba20bd98728758395080a3a9cf6e6a6a6964
- launch-builder-linux.cc was missing an include for musl
and used function that are not defined in the launchers
- musl caches pids used for raise, breaking sandbox setup
- the mtls contrib plugin won't build, didn't try fixing,
static builds can't really use plugins reliably anyway.
Change-Id: I5ab1664e45ea977e5bcf05e41d825e6014e62146
This merge commit returns to the previous state prior to the release but leaves the tag in the branch history.
Release created with releng/create_release.xsh
Change-Id: Ia7a63807681389cf5ccb9d07539b880697a88be5
This avoids a large list of CLs when a user can discover all the CLs by
browsing the chain.
Change-Id: I1890eb87e8bf6f930d3d00c3f7156690bb8de068
Signed-off-by: Raito Bezarius <raito@lix.systems>
This might do more copies than are necessary. I think that's fine.
Sorry, :reload not reloading has been bothering the shit out of us.
Fixes#1115.
Change-Id: I7f4caca3dad39696ce66ce57b1a520836a6a6964
So far, we had both `fj#` and `lix#` issue references in our release
notes, due to supporting both in the md files.
This commit unifies those in order for all references to be consistent.
Change-Id: I67c75784b460bfe4ebf214c2f6ac105cc60a7bad
kj can and will throw exceptions immediately without wrapping them as
promise results, and we must take care of these as well to not crash.
fixes#1150
Change-Id: Ifa29d0f3da4a6490f3e45d4c04a8fa49eaa20f43
mdbook-linkcheck was dropped from nixpkgs and replaced with mdbook-linkcheck2,
which is not entirely a drop-in replacement, but drop-in enough for us.
Change-Id: Iab03efab89ebae70811e0de3ea7cf0a9a059a9e4
Since commit ac64c727b5, during launch of the
builder it is attempted to raise all capabilities into the ambient set.
Specifically, what "all" means here is determined by the Linux API headers Lix
was built against.
Occasionally, new capabilities are added in the Linux kernel, leading to
PR_CAP_AMBIENT_RAISE failing with EINVAL on the newly added capabilities if the
API headers are from after the change but the running kernel is from before.
Similarly to how capset already silently ignores nonexistent capabilities, we
ignore this error so the builder doesn't fail to launch unnecessarily.
Unfortunately it is very hard to test for this situation currently, since the
last time a capability was added was CAP_CHECKPOINT_RESTORE in Linux 5.9, and
all kernel versions in nixpkgs are newer than that.
Change-Id: Ibeb2f0757729b877bd3ca9f02e1aa4536a6a6964
not doing this has lix crash if stdio forwarding fails for odd reasons,
such as ssh failing to set up a connection due to transient dns errors.
cf #1148
Change-Id: I543b7444236e69d3c13b47fb6c6eb37c65b96763
This is intended to be a softer version of
`allow-import-from-derivation`, allowing for users to only flag when IFD
is encountered rather than error out
Based-on: https://github.com/NixOS/nix/pull/13279
Co-authored-by: gustavderdrache <alex.ford@determinate.systems>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Change-Id: I879a444e3a24b4e82a8cea3a5a94834c78c3924a
`builtins.storePath` allows you to use "external" knowledge of present
store paths in your store to be used in your expressions.
This is sometimes used to avoid double wrapping store paths, for example
when dealing with nixpkgs sources (`builtins.storePath pkgs.path` is a
frequent pattern).
Unfortunately, using your external knowledge of present store paths is
technically impure because Flakes is about not depending on the *STATE*
of your store, no matter what you know about it or what you do to ensure
that your knowledge is accurate.
That being said, `builtins.storePath` is already possible in the
fragment of "pure Nixlang" via a clever use of `builtins.appendContext`.
Precisely:
```
storePath = path:
let path' = builtins.unsafeDiscardStringContext path;
in
# NOTE: merging the context set in all generality is impossible
because getContext on a pure path doesn't work.
builtins.appendContext path' { ${path'} = { path = true; }; };
```
works in pure contexts.
Because of this, we deem that storePath should be allowed in pure
contexts and users do not have to come up with a `storePath` polyfill.
Fixes#402.
Change-Id: I5fff750a4feed082745431835db451f945249cfb
Signed-off-by: Raito Bezarius <raito@lix.systems>
nix3 CLI never honored that setting.
When we update a nix3 profile element with new built store paths for
which we know the deriver (the derivation path which built it), we add
the derivation paths to the profile element to prevent garbage
collection the same way the nix2 CLI logic does.
Fixes#1095.
Change-Id: Icc236d174c5ce5ffe042d1c8e1968f6cb5b50359
Signed-off-by: Raito Bezarius <raito@lix.systems>
Instead of showing untrusted configuration items one by one, collect all
of them into one batched ask with 4 options:
- apply the configuration wholesale **for now** (yes)
- apply the configuration wholesale and trust it permanently (allow
permanently)
- selectively choose subset (no)
- rejecting the configuration wholesale (no to all)
Note that we do not offer options to reject permanently a configuration,
that's what `--no-accept-flake-config` is for.
Individual prompts gain a new option to distinguish "yes for now" and
"yes forever".
Fixes#682.
Change-Id: I2492717ac277af33ea4ac9b1a11bd7d6d1bfe0db
In messages printed by the evaluator, we use showType(.) to say things
like "expected X, got Y" where X, Y can be string, string with context,
etc.
After changes in the Lix data model (pointer tagging and friends), paths
got squished inside a internal string type.
We missed updating `showType` to take this fact into account which
caused an unfortunate:
`error: expected a string but found a string with context:
/nix/store/wjf8rdpp63rhyasbzc7zfms1agf7pwyd-source`
while using a path on `builtins.getContext`
A test is added.
Change-Id: Ifb0129ead2ea1973fc17dc0133f8a694995536b1
Signed-off-by: Raito Bezarius <raito@lix.systems>
This partitions a string into two parts based on the first occurrence of
the separator.
This function handles the case when the separator never occur in the
target string by returning an optional<T> in the second argument of the
pair.
Change-Id: I9ad44375dbaa137a4d21d2408bc2ffa34fab62a7
Signed-off-by: Raito Bezarius <raito@lix.systems>
Parsing and completing enumerations on the CLI was usually done by hand.
With this enum trait, the parsing and completion is automated away.
Change-Id: Icd49729564355d4c79e2c8050ce7e8b9d8e7eda3
Signed-off-by: Raito Bezarius <raito@lix.systems>
36c4d6f592 introduced it commented. Zero
explanation why.
horrors suggested to drop it entirely because it hide the options in
categories but they are pretty common and we should make as much visible
as possible.
Change-Id: I09894b5ba0f2b2cce5869136914289fed034fce3
Signed-off-by: Raito Bezarius <raito@lix.systems>
When `max-jobs = 0`, our async semaphores have null capacity and
therefore no slot token can be acquired.
In addition, a derivation with `preferLocalBuild = true` with `max-jobs
= 0` which is not a built-in derivation is not considered suitable for
being possible to build.
Our documentation in `max-jobs` writes:
> `0` is useful when using remote builders to prevent any local builds
(except for preferLocalBuild derivation attribute which executes locally
regardless).
A valid interpretation is that `preferLocalBuild` derivations (all of
them) will always execute even if `max-jobs = 0`.
This CL achieves this promise by enabling `preferLocalBuild = true`
derivations to draw in priority capacity from a `preferLocalBuild`-only
pool and then fall back to the build pool (`maxBuildJobs`).
If there's non-zero max-jobs, we make the `preferLocalBuild`-only pool
empty.
If there's zero max-jobs, we make the `preferLocalBuild`-only pool
non-empty (capacity: 1).
Remark 1: if `max-jobs = 0`, no non-`preferLocalBuild = true`
derivations can be built without a remote builder and `preferLocalBuild
= true` derivations can be built one at a time if `extra-local-jobs` is
not overridden.
Remark 2: if `max-jobs != 0`, no `preferLocalBuild = true` derivations
can bypass capacity limits, i.e. `max-jobs` unless `extra-local-jobs` is
overridden.
Fixes#855.
Change-Id: I00a0acac7375a881aec6281ebb13a4530b26e460
Signed-off-by: Raito Bezarius <raito@lix.systems>
This is going to be used in the extra-local-job future setting.
Change-Id: Ied76f32b8fa97cb71691d76167103001aa14fa01
Signed-off-by: Raito Bezarius <raito@lix.systems>
When ref/rev are very similar looking, e.g. a revision SHA1, the logic
to reconstruct whether we had a ref or rev initially cannot work.
This regression was introduced by
I2c72d5a32e406a7ca308e271730bd0af01c5d18b which passed the information
as an attribute field called `refOrRev`, instead, we pass `ref` or `rev`
but not `refOrRev`.
We apply the same post-processing logic to distinguish rev/ref in
path-style flake references: `github:owner/repo/$refOrRev`.
Fixes#520#841.
Change-Id: I2b1985cd5f351f43fea84232c1972715e2408fb1
Signed-off-by: Raito Bezarius <raito@lix.systems>
In structured attrs derivations, __json is used to serialize the `env`
attribute into a JSON object.
If the derivation enables structured attributes AND specify its own
`__json`, one of the field will take precedence (the reader can have fun
by guessing which one).
To reduce underspecification, we disallow `__json` in structured
attributes derivations.
Fixes#380.
Change-Id: I51c3b2af1ff9449471ba81d4c72df6a7a263eef7
Signed-off-by: Raito Bezarius <raito@lix.systems>
`nix.full_path()` is the *built* path, not the *installed* path. in the
sandbox this point to a location that contains *only* `nix`, not any of
the old-cli symlinks. any test that needs them in path would thus fail.
Change-Id: Ia651db8d1151a84f8d1b52e8993f6db61b0fe104
Without the `--git-dir` git will attempt to figure out which the git directory is on its own.
This causes issues when the `GIT_DIR` envvar is set in the environment for other reasons (such as being part of a `git rebase`).
Providing this parameter overrides the environment and thus makes sure the revs are always coming from the intended repository.
Fixes: #1135
Change-Id: Ibd43cfd8e0278463c0e1ccf86779982947a6038a
Signed-off-by: benaryorg <binary@benary.org>
To stop spurious warnings, we will stop this bizarre behavior of setting
empty values to impure environment variables.
On the warning side, we verify that the presence of the environment
variable.
The corresponding test is updated to "there is a warning" to "there is
no warning".
Change-Id: I12f5c6445ef00a83d269488d7aed0b0f61aeec12
All impure environment variables are initialized as part of `initEnv()`
to an empty value if they are unset.
Whenever `NIX_SSL_CERT_FILE` is set, this cause a spurious warning.
Let's document it here as a test before we fix it.
Change-Id: Iccba8ccb487d9cc25490e940ff09c1c5a13ae097
Signed-off-by: Raito Bezarius <raito@lix.systems>
CA tests have grown quite complicated and are hard to debug and
manoeuver for new needs.
They are a prime target for F2 migration and they were extremely
pleasant to write.
Most functionality is preserved or improved with additional behaviors
asserted on warnings and the full set of reports.
Change-Id: I70cd52abbe5ff13f56744b966ab00962ad3a49d6
Signed-off-by: Raito Bezarius <raito@lix.systems>
Instead of taking in a final argument `Value &` out parameter which it
writes to, it now returns its result
Change-Id: Iab6bc3a3ac6a4b17c6d31115a766a6ea6a6a6964
we already report errors that happen during reads via exceptions
rethrown from read. reporting errors during destruction too is a
source of confusion for users since these errors are not usually
relevant and just the decompressor thread reporting short reads.
shorts reads during shutdown are not an error, they're expected.
fixes#1138
Change-Id: I7983ef0ed7722460f1d683bc8171a454b3d06e90
I could instead add the right braces or manually ensure the
XMLOpenElement destructor is run at the right time. Or I could just
write what I actually want to happen.
Fixes#1137.
Change-Id: Ie1f157de1632571fb0c21e26d8e3cae56a6a6964
This is a rebase of CL 1626 after the performance problems that blocked
that change are no longer an issue.
Change-Id: I4a17f7606c5fdc9a924933a860eb401f6a6a6964
The pseudo-constructor `Value::mkFloat` would previously be called on an
default-constructed (under-initialized) `Value` to create a properly
initialized `Value` that represents a float. This change removes mkFloat
and constructs float `Value`s directly.
Change-Id: I48ae3a836842ce5e5eda6323404ba7576a6a6964
The pseudo-constructor `Value::mkStringMove` would previously be called
on a default-constructed (under-initialized) `Value` to create a
properly initialized `Value` that represents a string, reusing an
existing GC allocated Str. This change turns that method into a proper
constructor
Change-Id: I8d7a97e2afce231a04ecf56fdd6d22d06a6a6964
it sometimes happens that one forgets to add `.run().ok()` to a command,
leading to things not actually being tested, potentially even slipping
though code reviews!
This commit adds a check at fixture teardown time, checking that all
commands have been run. Sadly, due to when this check happens, we cannot
add a test checking for this behavior
Change-Id: If3df2c20c267b9f03ae0ca099b77c57e2310f335
this includes tests that fail teardown when *skipped*, which confuses
the tap parser by reporting more tests than advertised in the header.
not sure what to do about this though, and it is better than nothing.
Change-Id: I4fc6badacbfa3b3d5f5f4a4ad76b177379bb4a30
pytest-tap didn't get it right in combination with xdist. we need to
replace the reporter *completely*, which is just incredibly weird :/
Change-Id: I2e35b9dc8ea4b24563b9b3306312778781eb674e
The pseudo-constructor `Value::mkPath` would previously be called on an
default-constructed (under-initialized) `Value` to create a properly
initialized `Value` that represents a path. This change
removes mkPath and constructs path `Value`s directly.
Change-Id: I9021de1ff59490828f9fec6866e083996a6a6964
The pseudo-constructor `Value::mkAttrs` would previously be called on an
default-constructed (under-initialized) `Value` to create a properly
initialized `Value` that represents an attribute set. This change
removes mkAttrs and constructs attr `Value`s directly.
Change-Id: I11cd801eefecd454a7a5b6229d6770d06a6a6964
This is not ca-derivations. We don't care about self-references or any of the
rewriting or stuff like that; if you want something like that, write
your user code so it figures out where it is.
The reason I want to do this is for integrating other build systems with
Lix: if you're importing something built *outside the store* with a
non-Nix build system, it makes no sense to put it in an input-addressed
path since it's not possible to come up with a derivation for it.
But you need *some* hash, so the output hashing is the most sensible option.
It is also nice because of not needing trusted user privileges to import
it, since the hash describes the exact contents of the output path.
We use this feature (implemented externally to Lix; this CL eliminates
that step) at Mercury to integrate buck2 with Nix on the output side: we
import things to the Nix store as ca paths with references.
These can then be consumed by Nix language with:
```
# Hack from https://git.lix.systems/lix-project/lix/issues/402#issuecomment-5889
path:
builtins.appendContext path {
${path} = {
path = true;
};
}
```
Test plan:
```
$ nix store add-path --references-list xx-refs-list ./README.md
/nix/store/szcwnm13d9gmxx8fly0bz82l42jgysw6-README.md
$ cat xx-refs-list
/nix/store/g9hhwjbkdrw0wnbd3axfs4icyb37nr6b-bash-interactive-5.3p3
$ nix path-info --json /nix/store/szcwnm13d9gmxx8fly0bz82l42jgysw6-README.md | jq .
[
{
"ca": "fixed:r:sha256:1pj59fy7zcrn949ry9kgv4ba17h2sy3z2is469abj912cvbl14n3",
"narHash": "sha256-w5JA12YiJLlUMkRH8YfXAp6gFtlvJp8TSTazf7xLRd4=",
"narSize": 1384,
"path": "/nix/store/szcwnm13d9gmxx8fly0bz82l42jgysw6-README.md",
"references": [
"/nix/store/g9hhwjbkdrw0wnbd3axfs4icyb37nr6b-bash-interactive-5.3p3"
],
"registrationTime": 1771266146,
"valid": true
}
]
```
Change-Id: I005a03003dfc24108e018e599dbe0b5d6a6a6964
this requires some hacks to make error reports visible, but hey. it's
better than no progress reports at all, and successful runs no longer
spew huge amounts of useless log output onto the developer's terminal
Change-Id: I9e4766b2f825a4ec451e117eb4609db65f328785
Fixes path resolution issue with meson-embedded bash script when
`build` is a symlink, or a custom build directory is passed as a
argument to `just` or `meson`.
Change-Id: I67b13c524d196241bb9a8a3b8ad2c17c2c094ed8
not all operations that involve workers need to build. for example nix
copy --from calls instantiate thir worker with the *origin* store, not
the store they copy *into*. for binary cache stores this is not likely
to ever work if cgroups are enabled and the copy is run on normal user
credentials, even though no cgroups will ever be needed to copy things
fixes#1088
Change-Id: I065e29e1a1d0f58d81823609ef0701ff82cdd1d5
Affects `github`, `gitlab` and `sourcehut` fetchers.
Previously this was an assertion, which crashed the evaluator
with SIGABRT.
Fixes: #1133
Change-Id: Ia9bb8dd29ac8b9c97bf048827f62c5076a6a6964
overridden-ness can be reset, and is usually reset such that setting
value taken from a config file are not considered as overriden. when
launching builtin builders we *do* want to send config file settings
changes to the builder, so we'll need one more getter method for it.
Change-Id: I861538a469121c77ebc1898a276439e6b756797d
diverted stores are only necessary when the logical store paths of
objects matter for the test itself, such as for derivation hashes,
substitution from golden sample nars, or actual tests of the store
diversion functionality. all other tests can use undiverted stores
to run, especially since only linux can build in diverted a store.
Change-Id: I62f0907bdef9961609af22b610195fcec54c1e57
When the destructor of PathSubstitutionGoal is run, this happens in a
sync context and can cause starvation of all ongoing IO w.r.t. to other
substitutions, including our own substitution.
While there's only a decompressor thread per stream, the other side of
the IO runs on the event loop.
In order to fix this, it is sufficient to remove the thread indirection
and inline the async code.
Fixes#1126. Great thanks to horrors' patience.
Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: I3eb37bc37d156f0f5528364e568fdaa2ced58011
Signed-off-by: Raito Bezarius <raito@lix.systems>
Looks like too much stuff is breaking on this, so let's make it only a
warning for now to ease transition
Change-Id: I52d50ceb1fe1fbe9f4e28d0aabf3537f7e4d52d1
Explicitly catch common errors (trying to escape line breaks, badly
escaping interpolations) to provide better messages and user guidance
for these cases.
Change-Id: I3dd1b2ad3bca33be393e65be5e72f4fb9544a46a
StreamFdMessageReader reads message data *lazily*. if you don't access
all segments of the message before you close the underlying stream you
may find yourselv reading from something *very* different, and in this
case that something is The Void™. this causes reads to fail, writes on
the other side to fail to match, and finally our build launch to fail.
this does not lead to happy outcomes, so we will copy the full message
into a fresh new buffer *before* we try to access any of its contents.
fixes#1118
all-analysis-by: deprekated <kate@lix.systems>
Change-Id: I105540831fde855817194e9e539acf177f54a6f4
This is the long awaited refactor of the NixSettings.
It allows one to set, unset and update any and all settings with a neat
and easy-to-use interface
closes#846
Change-Id: Id4cfb5f853cc1168b506a1f6f405076f3a7cab65
if a substituter is entirely offline and cannot be queries at all we
should not be failing if other substituters are configured. likewise
if a substituter goes offline after querying but before we try using
it we should attempt to fetch that path from some other substituter.
ideally we'd treat all substituters as a single entity instead of as
one store each, then have that single entity take care of fallbacks,
retries, error reporting, etc. that requires larger rewrites though.
fixes#1061
Change-Id: I9d8fc0544ff380bf017256e8fcc82823dc634f10
we don't use std::cerr often enough to want this, and since cerr flushes
after every operation anyway it never did anything useful to begin with.
Change-Id: Ia54be340826da5073e9a1786c463555f4f0e491c
kj exceptions get wrapped as Error instead of passed through as is.
luckily this also means that we can add context to them very easily
Change-Id: Icedab6c016f4434447dd38ba14138c102fc6149a
oops. we have to use init* here, not get*. get returns a discarding
builder for something that was not inited previously, but the linux
cause worked anyway because the first member is created by default!
Change-Id: I40f8a12a04eef2f4e3a80d1537ac9b975490a027
this allows us great flexibility in how children are launched (since the
actual launching is done by a separate executable), makes fork no longer
needed in the core codebase (outside of runProgram, anyway), and we even
get to use linux vfork to its full potential to decrease the launch cost
of sandboxes to a constant factor (previously it was O(#drvs + #deps) of
the build graph, which obviously goes to n² quickly if you are unlucky.)
Change-Id: I66e2d1b20242dc24d708666ef325fb8725bd9296
we can have a ProcessGroup for a pid that *should* be a process group
but hasn't gotten around to setting its pgid yet. in such cases we do
want to be killing the thing anyway, not shoot into the void and hope
the right thing falls over. so far this has not been a problem due to
a mixture of just not having done this and being slow enough to work.
Change-Id: I4e0e54513252d8e18256b9286b819bfa957d70dc
if we move this out of libstore and don't want to pull in all of libutil
with its myriad side effects std::filesystem is our best option, by far.
most notably we don't replace pathExists because std::filesystem::exists
behaves like `stat()` on symlinks, not like the `lstat()` as we require.
Change-Id: I1e488418dcabb33f2ebb73d8c3d1b43528aa51f1
since we're using tmpDirInSandbox as the working directory for the new
process we also rename it accordingly. buildUser likewise turns into a
different Credentials type because exposing user lock state seems odd.
Change-Id: Id4a1a6eb733f774c893f373b91c2a271a4b84185
setting the signal is not enough, we must also check that the process we
expect to be parent to actually *is* our parent, not another process (eg
init if the daemon exited). we also have to set the death signal *after*
all set[ug]id calls, otherwise it will be cleared again by such changes.
Change-Id: I4e8c9102ea407576ed85b3203c8bb9bfb56762de
it's only used this once, and it accesses a (cached) global variable for
the filter. we want to move all sandbox setup state into a single object
soon, moving filter setup will make it a little bit easier to deal with.
Change-Id: I234d92d5ca044a16644b70bd303bfb7956c120f0
when starting builders we want the inner processes to run with a blank
slate. if some signals are masked for any reason the builder processes
may attempt to send signals to each other that are never delivered; we
avoid this by unmasking all singals. since every build tree also has a
session and process group of its own we are not in danger of sending a
signal to a builder by accident from any source, so unmasking is okay.
Change-Id: I90720ed2bd44502ffb6d2bb848c05369809abeba
we don't need to mess with this rlimit for e.g. the daemon. increasing
the limit later should always be safe since we don't allocate (or map)
much before constructing the eval states that ultimately fill our heap
and could thus make stack expansion impossible after some time passes.
Change-Id: Ieafda537fbc99a6a7f83a093a981e7df947da437
this will be needed in other places in the future, and splitting it out
also clarifies the surrounding code. the keep-caps dance launchPasta is
doing doesn't have to be moved since it is only needed to allow setuid.
Change-Id: I6baaa138c2b1bca9626971ed3266c1a971a63acc
the devshell no longer worked for us due to the recent deprecations :D
let's update pre-commit too because while it doesn't fail it does warn
Change-Id: Ic2060c82e4e97bb7a96cebd29097abefabdfe733
it's ancient, it does not do what it was added for (improving
determinism), and it probably hasn't been needed for anything
in a very long time. we can probably get away with killing it
Change-Id: I66f5c841964d9ad62726cb2dae8d6fc42095399f
mostly because this will make it easier to change the data source for
them, and especially to change to data source to something immutable.
Change-Id: Ic5f1a55d02d4c2cc928c3f8b308d07167efb467b
closing them as soon as we're done ensure that the fd table doesn't
balloon in size, which in turn lets us build larger graphs. it also
makes forking processes cheaper due to less fd table copy overhead.
Change-Id: I233d1b7358ffe1b05680f845cd99de38b179f449
In Nixpkgs, there are several strings like "\d\.\d" which attempt to be
a regex but are just literally "d.d". The escaping rules are silly and
we should warn our users about that.
Co-authored-by: Commentator2.0 <lix@crystal-cavern.systems>
Change-Id: I779b0757358fc9adc34dc140e1670b83abc93b67
The indentation stripping semantics of strings are fairly bad and have a
few gotchas where they behave unintuitively. But the good news is, that
these cases are easy to catch and can be avoided.
This commit adds a warning in the parser when such strings are detected.
Unfortunately Nixpkgs uses this kind of a lot, so we won't be able to
actually enable this warning for a while to come.
Co-authored-by: Commentator2.0 <lix@crystal-cavern.systems>
Change-Id: I3b3b68c2eee4cd70959d3f4ca643cb6caf3a2217
Unfortunately, the previous code only detected superficial __overrides
and did not catch cases like `rec { __overrides.foo = 2; }`.
Change-Id: Ic38bdef1a6a3bdea91915aef44447f4ecc238259
Changed the writing style of the descriptions, expanded with more
examples and rationale, and added the new timline metadata in the
frontmatter.
Change-Id: I218389e3504fc21f4eb45a927e77a41a1a70d4f5
The code within the code-generation had tons of code-duplicates and was
overall quite meh to read, understand and expand.
This commit refactors the code-generation to make it more readable and
comprehensible, while also unifying its usage a bit more
Change-Id: I3a5df8b7d8d9b6c76e02ef47dfb151c7dab7d0ab
lix, where modularity is made up and the patterns don't matter. merging
everything to match actual behavior of the system makes using rust much
easier because we'll have only *one* library to link, not half a dozen.
Change-Id: Ie3cb1db659798511128716402497697320bd0484
starting pasta as soon as we have all namespaces it must be in available
lets it start up while we finish creating the sandbox. this may speed up
sandbox launches somewhat, but likely not enough to show up in practice.
Change-Id: Id6724cbdc48b99284cd7ba7b56c98829d74557c5
mostly as a prototype for other process launching optimizations, but
also to move linux-only bits out of runProgram2. we want to stop the
fork+exec dance we do now and migrate to posix_spawn, but linux will
always need an amount special handling for launching build children.
Change-Id: Ied55dc0de95c05b52aaca923be2e3c1d101f47e0
we'll use this to run programs from a vforked child instead of using
runProgram. ideally we would like to have runProgram be as efficient
as possible, but our mount namespace handling makes this harder than
is worth dealing with right now. the linux derivation goal is a good
place to prototype functionality like this, especially since we need
fine control over credentials and file descriptor tables of children
Change-Id: Ibc356613ae10b06ea81de9575611948f5ee353b6
this was racy previously; a builder may have spawned processes before we
added it to a cgroup. these processes would not be registered correctly.
Change-Id: If60a64a798ee806d5701590a83d1015f55d7b867
this halves sandbox launch overhead, reducing the build time for 3000
trivial runCommand derivations on our machine from 80 seconds to 44s.
as a nice side effect we also get better error message propagation in
some cases, most notably setgroups failing when run in lix sandboxes.
Change-Id: Ia7c50a844915d9b8a20475b90b2d0179fd2fff34
creating user namespaces early means we don't need pipes to signal that
a namespace has been created and set up. we'll also create the netns at
the same time because they're privilege-bound to the userns, and having
both available will let us launch pasta earlier to have it ready sooner
Change-Id: Ie66fe2ab7f50a520fb81dd90944384e168d94476
permissions and ownership will be the same since the sandbox process
will not have changed uid/gid or umask by the time it gets around to
writing these files. all of these files contain some information the
host must provide, be it contents or configuration info of some kind
Change-Id: I0b57cb0699fd29e798ccf330dff99f571f3c0692
this way we can use it anywhere without fear of it being uninitialized.
only the linux platform bit uses this anyway, so we will move it there.
Change-Id: I35e207eec91daa8aa327c4f8b36c0dc8e703a7c4
`xucred` already contain `cr_ngroups` and `cr_groups`, we just have to
use it properly.
Change-Id: Ic29b4c551f9d93e86e9ed2bef20889d27d5438f4
Signed-off-by: Raito Bezarius <raito@lix.systems>
Two things happens here:
- we consider supplementary groups that are known for authentication,
fixes#968.
- we check supplementary groups if they are our build users group and
throw an error if they are (new behavior).
Nonetheless, we did not remove the ability for `matchUser` to recurse
and check for groups that the user may have but the connection might not
have communicated as part of their groups, i.e. if a process reduces its
list of groups via a call to setgroups, it will still be authorized.
This will come in another commit.
The authorization NixOS test has been extended with a store ping test
running in systemd with DynamicUser=true *AND* a supplementary group in
allowed-users.
Co-authored-by: Tom Hubrecht <github@mail.hubrecht.ovh>
Change-Id: I25b2b8304d66a04651cea523b5585a5d15ceebe8
Signed-off-by: Raito Bezarius <raito@lix.systems>
On Linux, SO_PEERGROUPS is an option introduced in 2017, that is, in
Linux 4.13.
Thankfully, Lix targets 5.10+ kernels minimum.
It was chosen to allocate 128 gids by default and ramp up (2x) as needed
rather than allocate a full 65k of integers as it seems wasteful. I bet
the time to the 16 allocations should incur at most an additional
millisecond on a modern system, don't quote me on that though.
This is preparation to enable ability for the daemon to vet based on
supplementary groups.
Related to #968.
Suggested-by: alois31
Suggested-by: eldritch horrors
Change-Id: I26d698327db5d174bf70ca25b0afede132bd9169
Signed-off-by: Raito Bezarius <raito@lix.systems>
Rather than a bunch of booleans, use optional<T> as a way to convey that
the identifier is known or not.
This is preparatory work for SO_PEERGROUP usage.
Change-Id: I0e3ce69e4103f9df5a2161c695bfc650123c1549
Signed-off-by: Raito Bezarius <raito@lix.systems>
we can already handle this case (by forbidding any ~/... paths in option
values), and since builtin builders are now run via exec we may not have
a home dir set, or even available, for them to resolve. loading a config
file in this situation will fail with an error even if the config is ok.
fixes#1109
Change-Id: I3491b31558871faa34766462747e3c15d1eccc64
xattrs are revealing some unfortunate brittleness in real world
derivations that are getting -EINVAL errors while executing their test
code.
The reason for this is that Linux checks [1] UID delegations for xattrs
writes.
In the Lix sandbox, except if you enabled a uid-range feature, you have
exactly 3 UIDs: root, nixbld and nobody.
If your test code makes use of UIDs which have not been delegated, you
will receive an EINVAL on your operation. Test code is not resilient
with respect to the number of available UIDs in their namespace.
To avoid further issues for end users who are running into spurious
derivation build failures, we forbid xattrs again for now.
For more information about the plans, please consult or chime in [2].
Fixes#1105. Reopens#838. Fixes#1103.
[1]: https://elixir.bootlin.com/linux/v6.18.6/source/fs/posix_acl.c#L257
[2]: https://zulip.lix.systems/#narrow/channel/9-Store/topic/disablement.20of.20xattrs/with/5295
Change-Id: I864066b34cd8319d2271fac1b179cb4f950d836e
Signed-off-by: Raito Bezarius <raito@lix.systems>
this will let us configure more than one socket to connect/bind to,
which in turn lets us use posix acls on sockets for access control.
we will also need something like this for the final rpc transition.
Change-Id: I9c39f14906e9bf809055ab5c94bf687745b4f69e
now that builtin builders are regular executables we no longer need to
treat them specially during sandbox launch itself, only while we build
the command line and environment for the sandboxed process. we are not
far from being able to extract platform-dependent sandbox launch code,
ideally moving all of it into (much more replaceable) libexec helpers.
Change-Id: I9b7041314683c56cd70eec9b1b4eae6de228883f
this means that builtinFetchurl runs in a real process now, and thus we
no longer need its workarounds for running in a forked process. forking
dropped the signal handler thread and broke the curl state via sharing,
neither of which happens any more now. we can run fetchurl builtins and
their actions straight from the main thread of our builder now, and the
temporary files and settings overrides we did are now also unnecessary.
Change-Id: I738171bc120ffcd541b7ff1424fed7924c2cdc1d
this will let us migrate the fork+do_stuff combo of builtin builders we
have today to fork+exec of this new command. we use a subcommand rather
than a simple libexec helper because we would like to avoid linking all
of lix into the simple libexec helpers if possible. this is only hookup
for the builders, none of the buiders are migrated here to ease review.
Change-Id: I9358f1f3fee6ca640c81a7bd865128ae0d0e35a5
call them with their individual parameters instead. this will make it
much easier to turn them into executables we can exec in the sandbox.
Change-Id: Ib49cf44715b2a480fecadabd21216c69cd730ef5
not having the optionals set when the builder runs (e.g. because FOD
hashes were not configured correctly) could cause assertion failures
in the builder process. while this should usually not happen we *do*
want to print a useful error message here instead of a crash report.
Change-Id: I81fb1d45fcbf660d0d9a7f0c0a12c38379ecd4c6
See: cl/4840
When importing Python modules, we include `functional2` in the module
path, like this:
from functional2.testlib.fixtures.env import ManagedEnv
This means that python expects to see a file like
`functional2/testlib/fixtures/env.py`. We run `pytest` from `tests/` in
the `justfile` and have `tests/functional2/__init__.py` so `pytest` in
`meson` is able to find these imports.
However, language servers generally consider the `pyproject.toml` to be
the project root, so (e.g.) `pyright` is unable to follow any of the
`functional2` imports, leading to lots of spurious errors.
In cl/4840 I moved `tests/functional2/pyproject.toml` to
`tests/pyproject.toml`, which worked but was considered aesthetically
unappealing.
This diff is much larger but it's a more elegant solution.
Change-Id: I2983c7b87f88f59a4e3521451a9f5acd6a6a6964
like diff hooks before the build hooks also use runProgram features that
aren't present in posix_spawn. just like diff hooks we do not expect the
build hook processes to be so fast that launch overhead matters somehow.
Change-Id: If3c33dbd7d2ac20c95886e06d24feda312946c78
use a wrapper type instead. whether something should be treated as a
single process or a group is a type decision more than a runtime one
Change-Id: I966e998c19e74be64ddd72ab11af809d001f61b5
diff hooks use uid/gid switch functionality that is otherwise only
needed for linux sandbox setup and unsupported by posix_spawn. not
doing these switches inside lix core code may let us move to using
posix_spawn for most process launching in the future, and for diff
hooks the added overhead of a wrapper program really does not hurt
at all. diff hooks are expected to be expensive in terms of output
size, process launch overhead is not likely to even be noticeable.
Change-Id: Ifa4b3eedef237632db3eb88d10e6469acae01f9e
it's kind of weird to not allow diff-hooks in single user mode unless
that user is root. maybe that's why we have no tests for them? we can
at least add a test that diff hooks are run at all when we expect it.
Change-Id: I54d623b5416acda1c205cc062b0f3a33c9f4aaa7
nothing except the linux sandbox actually *needs* this for correctness.
the linux sandbox only uses it to ensure that builder uids are freed up
when the sandbox is torn down, and even *then* it only works as we need
it to with PID namespaces enabled since the parent death signals is not
inherited across fork, but when pid1 of a pid namespace dies the kernel
also kills all namespace members. in all other cases this flag does not
help us that much because it actively prevents child processes cleaning
up after themselves, possibly leaving more trash around than otherwise.
Change-Id: I9ae0a9e91833a91d1011375dda402ac6c0a7ef6f
Adds an example plugin implementing an mTLS-enabled binary cache store
(https+mtls:// scheme) using client certificates for authentication.
Darwin fix: don't link liblix* into plugins (host resolves symbols at
runtime via dynamic_lookup). Explicitly link curl so it binds to
Nix-store libcurl, not /usr/lib/libcurl. This prevents the plugin's
curl_easy_setopt calls from operating on the wrong libcurl instance.
Test portability: BSD sed -i wrapper, OpenSSL -sha256 for cert signing,
redirect test server output to log file.
Change-Id: I652b987d3ac45e31df50ff4ba1f523294438c2b6
this really doesn't have to be here, it doesn't help very much. doing it
in the parent is cheap enough to not care and sandbox setup is not async
yet *anyway*, so we would not even notice if the old way was any faster.
Change-Id: I5a3a99af0fa5928e9a42f9c6589d98ff38b8c775
this will make it easier to turn build sandbox processes into helpers.
the point they were at was effectivly unsandboxed except for a few fds
that were redirected by commonExecveingChildInit, which only made logs
of any errors that much harder to convey from the child to the parent.
Change-Id: I67006eb33e1e13311bb8d14e6a0c3d5e6baf0c13
it doesn't do anything anywhere else, and personality setup is not a
critical operation (all it does is change which arch uname returns).
Change-Id: I39d7fcc4916e6882e49191d2294f9b7ee0dbbcd0
there's no need to do it in the fork, we're not changing security
domains before opening the pty anyway. we do change who owns this
pty device, but since we change it via chown we can just *not* do
that before we have an open fd to the pty. in practice this isn't
even necessary because the daemon runs as root, but if we ever do
run the daemon as an unprivileged user we'd need this to be split
Change-Id: I35264ab2954c7ba2c9c24c927366d64acada6772
this also requires moving namespace support information into Worker and
out of function-scope static variables, otherwise we can't use async IO
for the libexec helper output. we could set the fd to blocking for just
one CL and extract the Worker changes into another that the reverts the
blocking fd usage, but that seems not warranted for the scope of these.
Change-Id: I6996fab1ae74693d50cefb6a6a9c21d61dada1d9
Those functions have been recently refactored to return a formatted
`string` or take an `std::ostream` and only do the formatting have been
renamed to match what they're actually doing.
Change-Id: I3fe32fbe8723c2d93226370b8dd297f16a6a6964
this is mostly a test and example for the libexec helper infrastructure,
but it also lets us simplify pager launching until we we can more easily
handle executable-not-found errors the launch fallbacks would cause when
using runProgram2 instead of fork. ideally we'd use `posix_spawn` later.
fixes#1104
Change-Id: Ia33cc12e8a9d60ffad6f5c055bb1b8b596810e64
the new libexec directory is not available as a setting like other
directories (e.g. binDir) are since we consider libexec helpers to
be very internal. repointing them is like repointing a .so file we
dynamically link to; it can work, but needs much more preparation.
Change-Id: I40e64be0b32276f2864c0f2eb0b998d4c8ce7c88
we don't need the full sandbox setup helpers for this: mount namespaces
do not need to be kept, loggers are not needed, and redirections can be
done by runProgram2. once the build hook is removed we will not run the
communication bits from a different process anyway, this prepares that.
Change-Id: I95d28f7c2c25e43ccd82b448d270403ce4f28852
we do not need explict redirection management now, and dieWithParent
doesn't need an override either. we'd much prefer to kill ssh if the
process in charge exits; even ssh multiplexers are not fazed by this
Change-Id: I81e28b7605df73c887878ea4716228c7ad0f5c6f
the redirections the tests did were once not possible with runProgram2,
but they have been for a while now. we should use them instead of fork.
Change-Id: Ia422c8941ed04a9403dee68bfe938d68952253da
returning the pid as a pid_t instead of as a Pid raii wrapper is only
convenient for daemon use (where it'll eventually go away). using the
released pid correctly in other places is harder without the wrapper.
Change-Id: Ib42a2f357d2f0849beabd015b321bfff31334eda
Fixes#323
Let's now all go an a little rant about spaghetti code...
The result of this code is that the missing paths are not printed
anymore. The basic issue was that the parent class of this command is
StorePathsCommand, which inherits from BuiltPathsCommand, and their
purpose is to work on path that are clearly in the store, building them
if needed (and ofc telling the user about what's missing), the sequance
of calls is:
- BuiltPathsCommand::run(ref<Store> store, Installables && installables)
- Installable::toBuiltPaths( *getEvaluator()->begin(aio()),
getEvalStore(), store, realiseMode, operateOn, installables);
where operateOn is Output by default, realiseMode is Derivation, so
the only thing that can be built are the derivations for the required
installables
- Installable::build(state, evalStore, store, mode, installables)
- Installable::build2(state, evalStore, store, mode, installables,
bMode)
And that final call has the following:
```
switch (mode) {
case Realise::Nothing:
case Realise::Derivation:
state.aio.blockOn(printMissing(store, pathsToBuild, lvlError));
```
So there were two options, hack a new spaghetti in the existing
spaghetti code, or condense all those calls that are actually useless in
our case because they mostly transform a list of installables into a map
from installables to their BuiltPath which are then iterated to retrieve
the final outputs, whereas it is possible to directly get the required
paths in a much more efficient manner and without printing unrequired
stuff through a multitude of intertwined function calls by simply
replacing one method that was previously inherited from the grandparent
class
Change-Id: I1d2baaef5a099cd98b63b5346f2613914c6cd2ac
it's no longer needed now that withPager exists. also fix a
logger-never-resumed bug that never showed up in the world.
Change-Id: I7311a50896f5291364320ec4c88506dc3bb11d4c
RunPager is weird and confusing in that it replaces what stdout *is*
depending on environmental conditions. this has not caused problems,
but it's easy to imagine situations in which it would (eg if the stl
decided to capture the stdout fd by duplicating it). using a wrapper
for this also makes clear *what* actually goes into the pager; while
the previous contract was semi-reasonable it was also very implicit,
and with the proliferation of functions we had that printed directly
to stdout it would have been easy to send wrong output to the pager.
using a wrapper also makes process management much easier because we
do not have to rely on destructors to always produce correct output.
Change-Id: Ifd3760940af1ec719fe856158c913cbb9a5bf270
this has been unused since ab3ce1cc13,
which is now almost ten years ago. surely it was just forgotten then
Change-Id: Ib93b52d4ce5f19629432c5f91ff426582c57f99b
cancelling a get() promise could leave the pool with allegedly in use
resource handles that do not actually exist. this would cause asserts
in ~Pool to fire (and crash the entire program due to bad accounting)
probably fixes#1041
Change-Id: Ic61ab9b658f02f71e8be7577b8a1115600846f86
merge all pkgconfig-found libraries into a single one and redirect all
existing lib files to the merged variant. the clean separation between
libraries is largely fictional anyway because the main config block is
in libstore and cannot reasonably be moved out, or because libutil has
assumptions built into its behavior that libmain later establishes, or
because libexpr messes with global boehmgc configuration. nearly every
kind of layering violation imaginable can be found *some*where in lix,
and not pretending that this is not the case will let us clean up some
of them. this will also allow us to merge these libraries at the meson
level which is itself necessary to make rust code easier to integrate.
Change-Id: I172f8a08d5d90ded0a0a99fd9e0fa5d804db00a7
xattrs scrubbing was gated behind Linux platforms, but Lix can safely
assume that POSIX APIs are available.
macOS implements these APIs modulo some creative ideas.
Fixes#1008.
Change-Id: I1566df4923bf24092d3fd32ccf6a96e4256e87b2
Signed-off-by: Raito Bezarius <raito@lix.systems>
Fixes#838.
xattrs were historically disabled in 2017, since then, NFS v4.2 gained
support for xattrs and almost all filesystems support xattrs. In
addition, moving the build directory to a xattrs-supporting filesystem
is always an option. Programs may exploit xattrs (including ACLs) during
their build process and testing phase, to better serve these programs,
we will remove this limitation.
Lix will use POSIX ACLs for its UNIX domain sockets in the future and be
one of these programs which will run tests making use of xattrs, while
not writing any xattrs in the derivation outputs themselves.
xattrs are still scrubbed from derivation outputs so it is not possible
to obtain an advantage by writing a security-related xattr to a well
chosen file.
Tests were added to test these scenarios on Linux. Darwin is carefully
excluded because of #1008 and #1090, solving that is welcome. In the
meantime, they are marked xfail.
Change-Id: Ia3255eeb8442e83db4f10dcb5a51cbc368a2550d
Signed-off-by: Raito Bezarius <raito@lix.systems>
one would expect the lang tests to be selectable (using `-k`) using
`eval_okay` `eval_fail` etc though this was not the case so far.
This commit renames the functions to reflect the expected names
Change-Id: I4cd340c093d774feeb4d96820d28e49c665b159f
I don't know how this was missed or no one really complained about it,
but perl-bindings was using CppNix if nix was not overlaid to our own
Lix.
If you overlay Lix manually, the resulting Lix's perl-bindings cannot
really refer to Lix itself, therefore, it's CppNix.
Change-Id: Ie5ce95564cb7fb5444acbea6124619c5121e1576
Signed-off-by: Raito Bezarius <raito@lix.systems>
during the value rewrite we accidentally broke extension of incomplete
primop application. this only shows up when binding on incomplete call
to a primop to a name, binding an incomplete call to *that* to another
name, and then finally calling the second binding with enough args for
a complete primop application. since this only shows up when calling a
primop with three or more args it took a while to surface. we have few
builtins that match this: foldl', replaceStrings, and substring. these
are not used incompletely in this manner very often, so this lingered.
fixes#1102
Change-Id: I218dffc14ae876efc86a86c7eb6c895e2405201c
these ended up referring to different directories and would break during
source reorganization (or even just including them in different places).
Change-Id: I91e8db04e3d3565efd5920982ed5bc59d2c27058
RegisterCommand was renamed in b30556f8f1
and de-static-ified in 4d9fe67430. adding
new settings was never possible with static instances since settings do
need an existing Config instance during construction; GlobalConfig must
be used to add settings. it is not possible to change existing configs.
Change-Id: I26af8abc375ba5883fce060ffefc710d6dfbca4c
Since lix-doc is no longer a separate library, the pkg-config files were broken
for static builds. Adjust them to refer to the new monocrate library instead.
Actually, since the monocrate library is static in all cases it needs to be
linked explicitly even when everything else is linked dynamically. I assume
this only doesn't break the dynamic nix-eval-jobs because it doesn't actually
use anything from it.
Change-Id: Icf29938688f4fa87425ff346a3105ef56a6a6964
Use a span representing the underlying hash, this allows comparing based
on those spans. Other minor tweaks to the header file.
Change-Id: Ie05960439e60500109083c06d347d6cdf41bfda8
When --check is ran with no output, this is seen as having *SOME*
invalid outputs, actually, ALL of them are invalid here.
Instead of reporting a confusing error, let's consider ALL of outputs
being invalid to be no output at all available and advise the user to
rerun the build without --check.
If there's *some* (>0) invalid outputs, let's print them exactly to give
a chance to the user to remediate this.
Fixes#485.
Change-Id: I00955ef9ea4f129e2c98d68c73b1e981f90278a0
Signed-off-by: Raito Bezarius <raito@lix.systems>
Yeah uh, I don't think `$shell` is ever set in the functional1
environment. Pretty sure this has been expanding to `''` THIS WHOLE
TIME.
Also `SHELL` is `/sbin/nologin` in that builder. idk why so I'm just
passing `shell` unconditionally in functional's `mkDerivation` helper.
Maybe next we should put `set -euo pipefail` into all these test
builders?
Change-Id: Ie5fb85305932c03a901ba84ffcce48f16a6a6964
macos-builder02 in particular has been having problems for a while now
that no one could reproduce. We believe we have finally found the cause:
missing shebangs, not just in the tests themselves but *also* in the
inline scripts `nix-shell` itself writes.
Fixes#1042.
I believe this will also fix#1093.
See also: https://github.com/NixOS/nix/pull/14778
Change-Id: I54d04a770b8e78484815db88a8bc88776a6a6964
When `:load`-ing a file that doesn't result in a successful outcome, the
"loaded" file is removed again from the list of loaded files.
This avoids having it be used as part of `:reload` which will probably
not result in a positive outcome.
Instead, the user is expected to retry `:load` before issuing a
`:reload`.
Fixes#50.
Change-Id: I85d025cf39f79e8e02117628bffa20c81917dc3f
Signed-off-by: Raito Bezarius <raito@lix.systems>
If `:reload` is used with no file loaded, the environment is completely
cleared.
That's a surprising behavior and one of the root cause behind #50.
Environment clear is only performed if there's files to reload.
Change-Id: Icb014f490a83aaa4d758dbf37822db16d5bb65c1
Signed-off-by: Raito Bezarius <raito@lix.systems>
Only successful loads counts towards the list of loaded flake
references.
Change-Id: I04128f90e57ef8352183d72dc6def7a09d52173a
Signed-off-by: Raito Bezarius <raito@lix.systems>
libutil config.{hh,cc} and libstore/globals.{hh,cc} contain so many
interrelated things. Hopefully we can separate them out a little bit for
clarity.
Change-Id: Ib78fc46fe305d13aada8886e2e168d446a6a6964
This makes that the default for all builds... which I'm not sure is the
right option, but it is also very cheap to do so, and adding an entire
other Lix build just for static libraries (not to be confused with
static Lix) would be prohibitively expensive.
Change-Id: Id173c77cfa62886724378ca599d21c8c6a6a6964
`build_env` is assumed to be a certain trivial environment, e.g. a
singleton search path.
An environment in general is either empty, contain one or more search
paths separated by a colon.
It seems that the intent was to set the `build_shell` first as a
prepended path in the managed environment then extend via the
`build_env` parts.
This fixes a usecase when the `BUILD_TEST_ENV` is non-trivial.
Change-Id: If5b8ab976d867a57ce0b8d29255f64695e30a8b2
Signed-off-by: Raito Bezarius <raito@lix.systems>
In an attempt to slowly split up and clean up `eval.cc`
The `Expr::eval` functions contain the most important logic when needing
to touch something on the evaluator, and having them spread across the
3kloc eval.cc file was really annoying. This should help, though more
cleanups are yet to come
Change-Id: I2c2d15a024ab1ec3fc5591f628d6768341a1c8ab
This adds a way to control the deprecated features that need to be
enable while evaluating the specific nixpkgs version
Change-Id: Ia2b9a29f1966d4d5f32239a36ba8270330007da9
This avoids rebuilding Lix each time we need to modify the lib tests
(as flake.nix stays unchanged), and adds a way to control the deprecated
features that need to be enable while evaluating the nixpkgs lib
testsuite
Change-Id: I2baa1cf90effcaf9d53337c6a7a5b24d6297178c
Certain evaluation requires to pass impurity arguments, add `flags`
to add more arguments to the instantiation.
Change-Id: I538ca2ff12c07cc66e845b42f81baa87f11b8f0f
Signed-off-by: Raito Bezarius <raito@lix.systems>
Is there a situation where `home` can be None? This hinders the ability
to use it quickly without type gating in the test code itself.
Change-Id: Idd739ddad67e10d6f7a6aad283c0e3b7cd08106a
Signed-off-by: Raito Bezarius <raito@lix.systems>
Otherwise, this result in debug traces with a \n cutting the list of
controllers.
Change-Id: I8aba6379d335c4edab9d9ddb3fb411813d8dc426
Signed-off-by: Raito Bezarius <raito@lix.systems>
The prior use of the UID is uninteresting in general for consumers who
wants to learn things about _what_ the build is, not what is the UID of
the build user running it.
Achieving this with a place where a mapping between cgroup names and
build requests are maintained is non trivial because there's no "global"
daemon that keeps track of this information and such a daemon would
require a persistent location to track this.
Rather than solving that, we will just encode the derivation hash inside
the cgroup name. Userspace can simply pull this apart and query Nix to
obtain the original corresponding derivation for more information.
This provides userspace with a way to map cgroup names to pnames for
example. Going further, if you have a map between attribute paths and
derivation hashes, you can power a database of cgroup metrics per
attribute paths by combining all of that.
In addition to this rename, we adopt a systemd-ish convention for
template units (`@`).
As a result, userspace tooling that will scrape cgroup metrics will
discover a consistent path as long as they filter out the UID part,
especially in context of Id6c458aad30eaa08c3609ac8280a7dde8e8f3cf9
change.
This problem is tracked under #1073.
Change-Id: I238d0568a3e4b1ff3057781c0639528d666b4d37
Signed-off-by: Raito Bezarius <raito@lix.systems>
nlohmann has customization points for exception throws. we can use these
instead and wrap json exceptions at the source instead of playing a game
of whack-a-mole with json errors all over the tree. since nlohmann needs
macros set to achieve this we can no longer precompile its headers *and*
must forbid including it anywhere without the proper defines or ordering
of include directives will break lix with ODR violations, if we see them
at all before we get another json-related bug report. a new lint will be
in charge of ensuring this doesn't happen. we also re-allow direct calls
to nlohmann json parsing since error handling is no longer a problem. we
will keep the wrapper for more convenient error context handling though.
fixes#1092
Change-Id: I54ecc14f5bec5e2177729b41c3703216e76cc6a3
It's 2026 and the command handler was entirely manual!
We introduce a basic command handler framework for the REPL that handles
all the previous (implicit) features reasonably well.
This also builds up the infrastructure for nicer features.
Most important feature is that now we can let external plugins adds
commands to the REPL.
There's slight REPL changes:
- errors for debugger commands are more explicit.
- short-hand have a long-hand version.
- help shows command sorted by std::map natural key sorting (based on aliases)
while keeping the special syntaxes for the REPL language at the top.
- sections were introduced for Flakes commands.
- :st was merged into one command with an optional argument marker.
Change-Id: Ibd24f994491bf5d45d4b218fd84c0944ae5050fd
Signed-off-by: Raito Bezarius <raito@lix.systems>
Status quo: We have `force$Type` and `eval$Type`, both which first
produce a value and then do a type check. The type checking logic is not
consistently implemented, with lots of code duplication.
This change does:
- Introduce new `check*` functions which unify the logic (the error
handling unfortunately still needs some duplication for now)
- Make both `force*` and `eval*` use the `check*` function for the
actual type checking
- Inline and dismantle the `eval*` functions for being of little use and
little used. This makes the `ExprOp*::eval` implementations for binary
logic operators more verbose, but IMO that's a good thing: The
implementation now needs to be a lot more explicit about the
short-circuiting semantics, something which was previously hidden behind
the short-circuiting semantics of the C++ language, in a way that could
easily be overlooked and lead to confusion, which is something that
happened to me twice in a year.
- Changes `forceAttrs` and `forceList` to include the context in case
`forceValue` fails (compared to only when the type check fails). This
was done for code consistency, because I could not find any reason why
list and attrs had different semantics here than int, float and bool. So
far the visible change is minimal (see the diff on the err.exp), however
this needs vetting for potential performance regressions.
Change-Id: I33e5c706d46850c9e1126293ee01dab85ba07587
That struct has been annoying me since I've encountered it. There is no
reason for having to track all these symbols out of band in the state.
Change-Id: I5c2c0d8174af0a51f9b456cc3651d8203a406d09
The names were prefixed with `sym_` to allow faithfully representing
names that start with __. (Previously the underscores were omitted, at
the cost of readability for people less familiar with the intricate
details of Nix. Is `toString` literally `toString` or is it
`__toString`? etc.) As a bonus, this allows to easily grep for any
symbol through the entire code base to find where it is used. I've done
exactly that and grouped and annotated the symbols based on their usage.
Change-Id: I230fecd9b0b18e0984ac8111c1e6381f45c5abf2
I was pretty confused by this when I saw it, not realizing that `build`
meant `tests/functional2/build` and not the top-level Meson `build`
directory.
Change-Id: I4865f196e5bf029419c6b488e78cbfb46a6a6964
100 causes too much diff churn w.r.t. current code style
"120 cols would be nuisance for us because our two-column layout is
exactly 118 wide (not counting the line number margin)"
"Ours is exactly 118 too"
Change-Id: Ifc7f21003e1011a17a1d799afaee775aeb927fd5
This logic is used in the various build-related REPL commands and is
factored out to make it easier to write the handlers for each
build-related REPL commands.
Change-Id: Iaa18df489db75495b12924e9a76b3fff1975eb64
Signed-off-by: Raito Bezarius <raito@lix.systems>
The completions for debug commands is purposefully left out because it
is going to go away in the next changes.
Change-Id: Ibc78b7587de3f15857bb09a3a0732334693ad550
Signed-off-by: Raito Bezarius <raito@lix.systems>
cl/4796 introduced a bug where running `just test-functional2` would
hang forever on macOS but `meson test -C build --suite installcheck
functional2` would succeed because of a mismatch between the
`$BUILD_TEST_SHELL` environment variable and the `-Dbuild-test-shell`
build option set in `mesonFlags`.
We can move these definitions to `finalAttrs.buildTestShell` and similar
and use those values in _both_ `mesonFlags` and `env` in order to make
them consistent in all cases.
See: https://gerrit.lix.systems/c/lix/+/4796/comments/50def5b4_fa5671f2
Change-Id: If300bbe46d6269ace29b44156fb4a5196a6a6964
...and add both a static bash and busybox into the sandbox for building:
$ /nix/store/fz43jxs5qfg5vldzk38y2hmrcl58qk18-busybox-1.36.1/bin/sh -c 'declare -A foo'
/nix/store/fz43jxs5qfg5vldzk38y2hmrcl58qk18-busybox-1.36.1/bin/sh: declare: not found
The reason is that busybox's `sh` doesn't know about `declare` which
means it cannot build derivations with `__structuredAttrs = true;`.
Change-Id: Ie3ca431f862fd3e59dc649582704ae739ee5834a
Nearly each tests gets a fresh store (in the test root residing under TMPDIR),
which with default settings means 8 MiB of reserved space to be deleted by the
user in case of space shortages, which of course for a test suite is just
wasted. Stop doing that to considerably reduce the litter (experimentally, from
3.7 GiB to 67 MiB per run). The old functional test suite already did the same.
Change-Id: I912cdda6e796ea37cfee0b8fe0478e976a6a6964
this is only needed on macos, and only for tests that build anything. on
linux it can actually *break* stuff due to unfortunate interactions with
store url parsing, which is required to work for any remote build tests.
Change-Id: Ic6f7e090f15e129fc365d7edc56cdbc1a5686047
apparently this file slipped through the code review (the actual test
does exist, no worries) but this empty file and init py exist, sitting
here empty. This commit drops them as they serve no purpose
see commit 1b7ad3a7ad
Change-Id: I64c4e03cd3ceedf9e430ae014127966bd8e4269d
This avoids any complications that can arise from the environment
affecting evaluation of the help pages (which don't need to be calling
out to anything external anyways)
I don't think this is actively causing problems now - though it did in
Nix! - so this is largely preventative, but I don't really see a reason
why this was ever running in impure mode in the first place either
Based-on: https://github.com/NixOS/nix/pull/14086
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
Change-Id: Ice0c760562d70c09934249edc3b767473516d372
- test_check used an old version of an error message
- test_good2 did not build an unspecified dependency
- test_attribute_selection used a fixed drv path. we
can't keep this with the current setup because the
drv `system` makes the drv hash platform-dependent
fixes#1081
Co-Authored-By: kloenk <me@kloenk.dev>
Change-Id: Ia009740f8b5432e83467ea451a5ff53d5e141b3f
the cgroups experimental feature does not work properly without this
because we do not stop subdaemons when the main daemon is shut down.
systemd needs the assigned cgroups to be empty to restart the daemon
and thus cannot cleanly restart the daemon if any connections exist.
starting a fresh unit for each connection creates a new cgroup every
time instead of sharing any delegations and thus solves the problem.
fixes#1030
Change-Id: Id6c458aad30eaa08c3609ac8280a7dde8e8f3cf9
A friend reported that this is rebuilding, which doesn't make any sense
because nixpkgs also has a lix from nightly that surely has the same
dependency and thus it surely should be built on hydra, right?
Turns out no, since they were overriding the requiredSystemFeatures to
remove big-parallel. Silly, but we can match that and get cache hits.
Change-Id: I4b1c8e850c6c38b03881354e978f41a10592ec95
This bump nixpkgs input to 25.11 and fix the two warnings that were
coming with the update:
1. `pkgs.system` => `pkgs.stdenv.hostPlatform.system`
2. `runCommandNoCC` => `runCommand`
also:
1. remove cppnix 2.3 compat tests (cppnix 2.3 is officially dead now)
2. remove lowdown 1.3 compat tests (nixpkgs no longer carries it)
Co-authored-by: Tom Hubrecht <github@mail.hubrecht.ovh>
Change-Id: I78526b5d8992a6c63ecd7f0c7c1fa6346a6a6964
It only debatably belongs in libutil since libutil doesn't know what a
progress bar is but whatever.
It was either that or libstore. Since I'm about to make it a setting.
Change-Id: Iba59b0a190087b316ff3191a0ddd9ec36a6a6964
make the global_assets folder more readable by placing asset pack files
within a dedicated folder instead of building up a mess similar to f1
Change-Id: Ia2c16f38eb6da96e1e73584bd91391ee56acb410
This will probably get the implementation of the fixtures revised when
we land the new extraction code, but we are setting it up to be generic
against that.
The operator-> thing is kind of a crime. But it also makes the code
vastly more readable so it's impossible to say if it's bad or not.
Change-Id: Ia5aca69cefaa03cd533ad19d20d856ff7e76a546
Instead of manually reading the ref file,
which gives the tag object hash when ref is a tag,
shell out to git.
The logic for finding the ref file is left for cache TTL tracking.
Fixes: #1070
Change-Id: I490b1e62f83cf602c56232c6081a52166a6a6964
Those are new warnings that come with a more up-to-date llvm
Fixes#1066
Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: I305d0d810d12a5e8d31c7d89e0cdb3a82df71556
As far as I can tell, there's no harm to let know a nix-shell invocation
about NIX_LOG_FD being stderr.
Fixes#336.
Change-Id: Ifdb3591813251d4bc481158d28dc18e1489de72c
Signed-off-by: Raito Bezarius <raito@lix.systems>
As far as I can tell, there's no harm to let stdenv know that they can
write to file descriptor 2 which is stderr inside of a nix3-develop
shell.
Contributes towards #336.
Change-Id: I51dcbcbc19a1698a0d1255f9d943d9ebd15d115d
Signed-off-by: Raito Bezarius <raito@lix.systems>
Closes#1064
The culprit here is that `genGraphString` is only invoked with the
store-paths associated with the outputs of the derivation, so when
filling `dependents`, the `graph_data.find(p)` call would return the end
of the iterator when doing this for references to other store-paths.
As a result, the code wrote information behind the graph data-structure
causing a corruption. For me, this resulted in a SIGSEGV most of the
time and in a few cases in an uncaught `map::at`-exception as reported
by Niklas.
This patch changes two aspects of the original implementation:
* When filling `dependents` in the node-set, use `map.at()` instead of
`map.find()->second`. The latter doesn't make any sense and was the
cause of corrupting memory. The `at` would've made it far easier to
spot this in the first place.
* Filter out store-paths that don't belong to a different output of the
derivation when creating `outputGraph`. This variable is used on two
places, `genGraphString` and for topological sorting.
The latter already filters out store-paths from a different drv, so
this is happening now when creating the variable in the first place
such that `genGraphString` never ends up with corrupt data in the
first place. This is the actual bugfix.
Implemented a regression-test for this case to be sure.
Change-Id: Ie02144d89c32b0a776cb1ece0601d0229315ebc3
we'll need this to modify argv for socket-activated daemons. this is our
replacement for the old savedArgv mechanism that was unscoped and fucky.
Change-Id: Ie048eb8ea99f1c9cd627a051292c836c83197068
the parent daemon does not change any settings before starting a child,
so there's nothing we may want to change that is not already set by the
config file. this also doesn't prevent changes of the config file being
applied to daemons where we do not expect it since it'll only restore a
setting to the parents' value if the child also has an override for it.
Change-Id: Ic5a9ef13458c103ec9979cb187ba8d3ce5e1e719
Lix changed to C++23 in 4ea8c9d61¹, but the perl subproject wasn't
updated with it. This commit corrects that.
[1]: 4ea8c9d643
Change-Id: I5464138d206d87cf25762720b147487d6a6a6964
We offer the three usual options: verify, repair then delete and direct
people to report us corruption so we can fix if this is induced by us.
Fixes#447.
Change-Id: I0df61769d732d227333c206f312857c0593f7bce
Signed-off-by: Raito Bezarius <raito@lix.systems>
If the derivation does not start with D, do not return a simple
"expected string 'D'" error but a full error message.
This contributes towards #447.
Change-Id: Iee05f3918e4cc43e79f244ab2fd64a52cf2bb6d2
Signed-off-by: Raito Bezarius <raito@lix.systems>
Prior to I6a6a6964d2b5ad47ae5ea9eb11af9b6373ce2141 — `sudo nix
upgrade-nix` would perform direct store access.
This ensured a certain number of desireable properties for upgrading the
Lix binary itself.
We re-introduce direct store access for upgrading Lix binaries.
Fixes#1060.
Change-Id: I523c4d3023ed5fe9eff8fde9a266c56a0de47d8c
Signed-off-by: Raito Bezarius <raito@lix.systems>
Alternative to cl/4661 discussed in #1044.
It can be assumed that `$tmpdir/build-top` can be created safely without
any risk, this way, we don't need to reuse the random directory creation
primitive.
Fixes#1044.
Change-Id: Iec52477f3047fc40959b183c607312d5a40fc8c9
Signed-off-by: Raito Bezarius <raito@lix.systems>
Otherwise, loading Perl bindings fails early with[1]
undefined symbol: Perl_pad_sv at /nix/store/h2jsb5i4yfblr2f3ac2c7zpmlmj7zjym-perl-5.40.0/lib/perl5/5.40.0/XSLoader.pm line 94
Apparently, it's expected behavior by Perl that this symbol only exists
with `DEBUGGING` being set, hence it's used by the headers. However,
`pkgs.perl` from nixpkgs is apparently not built with `-DDEBUGGING`
causing this error.
Now, `NDEBUG` is manually unset after loading the Perl
headers rather than setting `DEBUGGING` causing the error mentioned
above.
I confirmed that this not only fixes the problem described above, but
running the Hydra tests with
diff --git a/perl/lib/Nix/Store.xs b/perl/lib/Nix/Store.xs
index dfdd64d28..14788266c 100644
--- a/perl/lib/Nix/Store.xs
+++ b/perl/lib/Nix/Store.xs
@@ -27,6 +27,7 @@ using namespace nix;
static AsyncIoRoot & aio()
{
+assert(false);
static thread_local AsyncIoRoot root;
return root;
}
still results in assertion errors.
Finally, added a small install-check that importing `Nix::Store` works
fine.
[1] https://git.lix.systems/lix-project/hydra/issues/69
Change-Id: I58521777eb0f94b766a9813aa4bbd06f9052bd35
As discussed in the F2 matrix channel, not being able to put the error
message inside of the thrown exception just results in one assigning an
additional variable to immediatly throw it. The original reason for
these rules to exist are that the message isn't printed twice, though
the line above the thrown exception is printed anyways in the
stacktrace, so we still get the error message twice but now with
additional work.
Hence this commit removes the said rules
Change-Id: I0b37c0b0861334703a5772c36353d31441f19dcd
So far, our ruff config was confined to the f2 package. This meant, that
when one added additional paths to the ruff formatter, those wouldn't
get the same rules applied as f2, resulting in inconsistent styling
thoughout the project.
Due to how configs are resolved, only the "closesed" pyproject toml is
considered. This means, we need to tell f2 to extend its configuration
with the base level one. Though no change is required for other parts of
the project, as long as they don't have their own pyproject.toml
Change-Id: I145c764e7b850194020b5560e1025f4aa80411ae
Nix daemon runs in a service context, usually with systemd.
If the Nix daemon unit adds `[Service] CacheDirectory=nix-daemon`, it should
successfully use `/var/cache/nix-daemon` automatically.
Instead, it uses `/root/.cache` right now, which is really bad.
By default, we add `[Service] CacheDirectory=nix` now which means that
caches are moved into `/var/cache/nix`.
Fixes#634.
Change-Id: I854b1045bfdce8f60110aea70bda1bf6657dfd7b
Signed-off-by: Raito Bezarius <raito@lix.systems>
Recently, in 54df89f601, support for mdbook 0.5
was introduced, including some logic to handle the `sections` -> `items`
rename. However, compatibility with 0.4's `sections` was only kept on the read
path, while writing 0.5's `items` unconditionally, which ends up in the bit
bucket on 0.4, effectively disabling substitution fully and leaving the include
directives in the final documentation. Restore writing into the `sections` when
they were there so that substitution works again.
Change-Id: Idd4d7653012660f3f7fc27f81f29b82d6a6a6964
Silly me forgot to add the dependency responsible for handling
`@pytest.mark.timeout` failsafes to kill a test if it takes too long,
which results in a warning when running f2.
This commit adds the required dependency and hence removes the warning
Change-Id: I281cad05bb0cf50208f72080cb2bfd8cbfb09d0c
Running the sandbox (as unprivileged user) requires all of user, mount and PID
namespacing. Previously, only the user namespacing support was checked.
Unfortunately, newer Ubuntu kernels impose restrictions on capabilities within
unprivileged user namespaces [1]; as observed by experimentation, in particular
(re-)mounting filesystems or changing mount propagation is prohibited even
inside a mount namespace, making it impossible to use the Lix sandbox. So any
tests requiring sandboxing should check for this breakage and be skipped if
detected.
`unshare --mount` performs the same problematic operations, so that's what gets
used for the test. PID namespaces are included as well for completeness, even
though currently no instance of breakage due to them specifically is known.
This fixes [2] for the functional test suite only.
[1] https://ubuntu.com/blog/ubuntu-23-10-restricted-unprivileged-user-namespaces
[2] https://git.lix.systems/lix-project/lix/issues/545
Change-Id: If95b527bf965ccb06371bbef7b6007666a6a6964
Builds using a diverted store strictly require sandboxing. Explicit disabling
by the user is already handled properly by force-enabling the sandbox anyway
(with a warning), but sandbox-fallback was still honored, predictably leading
to hilarious breakage (like the "confusing output" in [1]) when it actually
triggered. Cause a hard failure instead.
[1] https://git.lix.systems/lix-project/lix/issues/545
Change-Id: I61ea32c3bd8bca4e5ba278312b0a49326a6a6964
Builds using a diverted store strictly require sandboxing. It therefore makes
sense to automatically enable it even if otherwise explicitly disabled by the
user. However, they should still be informed that their choice is not being
respected.
Change-Id: Ia2cbdc24a4c55d8ecc49094e35e698e16a6a6964
Disabling build sandboxing is a significant change to build isolation. If this
is done automatically due to namespacing issues (and sandbox-fallback not being
disabled), show the user a loud warning without the need to opt into a lot of
other log spam.
Change-Id: I52daccdd6934873234223c798ca3dcff6a6a6964
This contains the depth of nix shells nesting.
Fixes#826
Co-authored-by: Qyriad <qyriad@qyriad.me>
Change-Id: If584c9d02730d6c857636dafdeab0c01f4ec8e0f
If the NIX_PATH shadows internal files, this will often break things,
hence we forbid it.
Fixes#998
Change-Id: I70e5d389532ada1c9f910c60281abe565e3ce6bb
Some of the tests depended on the evaluation order chosen internally by
nix-eval-jobs, which will break when that order changes. In practice the order
is not guaranteed, and already not deterministic with multiple workers.
Change-Id: I2e85d096f4439a8075a6f4a08d31c0f16a6a6964
GCC still ICEs on coroutines, while Clang works properly. Use the latter, just
like for Lix itself.
Change-Id: I802d6df7279550a5f5355e6c1ea0533c6a6a6964
Previously the collector-side implementation of the worker interaction protocol
was a bunch of spaghetti code. Split it apart at places where it makes sense to
be easier to follow.
Change-Id: If3cc6b6fbf289dd778856b40b55316c76a6a6964
As the worker processes are now properly spawned using fork-exec, they no
longer inherit the stack size from the non-main coordinator thread, but get a
main thread with reasonable stack size on their own. For this reason the
coordinator threads can use the default stack size, and turn std::thread can be
used as the only reason for the custom wrapper was to enlarge the stack.
Change-Id: I1192474885abe9b0625ac483840b6e1a6a6a6964
Similar to how 7b37d5ea6a switched the subdaemons
from fork-only to fork-exec, do the same for the nix-eval-jobs workers. This is
the promised change that causes the signal handler thread to appear.
Change-Id: I590a5eb6bddfb546a331a01191fd788b6a6a6964
The retry added in 4c5efd4548 has never worked
properly, as releasing the pid the first time causes it to be -1 and thus
waiting for a random child on retry. While there is a good chance that this
really catches the crashed worker, this is by no means guaranteed, and it could
just as well have caught different restarting worker. Wait synchronously
instead. The worst possible outcome is a hang instead of an error message if
the worker manages to close its pipe but then fail to exit, which is not
supposed to happen in the first place.
Change-Id: I4e4c3f532ec15c3118f6f77821ab820a6a6a6964
warning: embedding a directive within macro arguments has undefined behavior
????? CLANG? why does this take -pedantic to show??
Change-Id: I6f4a9f3db1b89448f9f5d00d82b551e46a6a6964
This change brought to you by my uninstalling go but forgetting to
remove ~/.local/opt/go/bin from my PATH.
Change-Id: I937675bab11ac7e0678c896cba78204b6a6a6964
Maintains compatibility with mdbook 0.4.x.
Includes comments for what to remove once 0.5.x is the only we care
about. Some other changes technically could be changed at that point,
but currently serve to enable universal support of 0.4.x and 0.5.x
Fixes#1051.
Change-Id: Ic5b405038d180bcd357bbd9e5716879e0c26e5f5
In the same vein as be18b7dc2¹.
Before:
error: syntax error, expecting ';'
After:
error: syntax error, expecting ';' to end binding
[1]: be18b7dc25
Change-Id: Iffbcb113d2b892a50c646c9875e970376a6a6964
Lix blocks SIGPIPE, and std::cout ignores EPIPE (actually it probably stashes
an error code in some flag that no one ever reads, with the same end result).
Consequently, nix-eval-jobs would waste resources by continuing to evaluate
even when the reader interested in the results has long gone away. Instead, use
writeToStdout, which throws an error on EPIPE, leading to the process
terminating as desired.
Reported-by: Winter <winter@winter.cafe>
Change-Id: I962c09bab582a8ed27dd41c01b1519876a6a6964
Right now, the worker will not receive interrupts, as it has no signal handler
thread due to the way it forks, but this will change soon. Prepare to handle
worker interruption properly, by letting the worker exit successfully in this
case and adjusting the coordinator to handle this exit gracefully.
Change-Id: I9f79670b0d2004e7e2d8bf36cf67108e6a6a6964
For some reason that eludes me, it used to only print the last attribute
instead of the full chain.
Change-Id: I2f5f6e85e2b1ab3ab20a9c99bf06d1e45a4d09bb
`e`, `ae`, `j`, `jAttrs` … holy fuck this function was so much worse
than it needed to be. This refactoring includes:
- Descriptive variable names where possible
- More code comments indicating what even is happening
- Shuffled the control flow around to be more linear (move early return
conditions up, etc.) and have less rightwards-drift, to increase
readability
- Extracted `mergeAttrs` function for future changes and readability
Change-Id: I7253b47ce4910fdc67aa7d6edff2f3c0bb8fa55b
In one of the cases it pointed to the first instead of the second
occurence, which is a bit confusing
Change-Id: Ie508d1f84feb434708804d45aa8a9c8b1e3c5f69
How long do we have the value printer already? It's time to stop
concatenating strings like it's 2005
Change-Id: I3f5074de2439a1ad78af94de877bb141bc9f1d82
When using `builtins.derivation`, the `outputs` list can be omitted, and
in that case, it defaults to `[ "out" ]`, set this default when building
the shell derivation for `nix develop`
Fixes: #556
Change-Id: I4dc77d060f30ee499e186ec919de30bd3e5c53bb
Commit 5dc847b47b introduced usage of
std::chrono::milliseconds, which prints with the unit, leading to its
duplication. Remove this duplication again. Specifically, the explicitly
printed one is kept, due to better typography.
Change-Id: I724ea6be6f2a1349fe799602c3429f096a6a6964
We have done it!
The functional/lang framework has fully been migrated to functional2 :D
closes: #856
Change-Id: I63ad8d7dbcd9b5267ca04af68df73b1ffa3d6461
After multiple cycles of deprecation, we now use the lix repositorty as
a default, when no prefix is defined
Change-Id: I29b445fee39fb20c0ff024e3b95402f459144b62
Added builtins.warn` which takes two arguments: a message that is
displayed as a warning during evaluation which must be a string and a
value that is returned from the expression.
The next commits add new settings to control the behavior of the new
builtin: `debugger-on-warn` allows the user to start the debugger and
`abort-on-warn` aborts evaluation with an error.
Unlike upstream, I chose not to mark evaluation warnings from
`builtins.warn` as distinct from other warnings because that breaks the
commonly expected logging format `level: message`.
Co-authored-by: Qyriad <qyriad@qyriad.me>
Upstream-PR: https://github.com/NixOS/nix/pull/10592
Fixes: https://git.lix.systems/lix-project/lix/issues/579
Change-Id: I8658c88e5c27952b65e8b9f5525a572e0680cc1f
There are currently 4 different ways to run garbage collection using
lix:
- `nix-collect-garbage`
- `nix-store --gc`
- `nix store gc`
- (using the daemon directly)
As they were written all at different times, their output varies (and is
broken in some case). This unifies the display of informations in the
following ways:
- The list of paths in the results is always printed (in the nix3 cli it
is hidden unless `-v` is passed)
- The number of paths in the result set is always displayed
- The size of deleted paths is only shown when actually deleting things
(as it would have been 0B in any case)
Fixes#905
Change-Id: I40d9ec7c6d76795f6c6dd30df196d1e855bdb9db
This also allows removing a static initialization of a primop.
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Change-Id: I74da68205630f658b239e4327bf5e27bfc1b91da
This is done at the start of the command call, which allows not building
expensive things if the arguments are incorrect anyway.
Fixes#687
Change-Id: Ic924fe2115cf760684c6fdf7987279e96ab00286
the newly added f1/lang tests is required for the f1/lang framework
dismanteling to happen in a separate commit
Change-Id: Ic419c515262294c51a46d1513daa7848e4b71405
While the err file should always be empty, we prefer not have (easily
avoidable) warnings in the log, in order for actual warnings to be
spotted more easily.
Additionally this way no additional changes are required in case they
make use of some depreacted features in the future.
Change-Id: Ie078e2a851b2035d839f6b95d4188e475eb96b2d
autoargs get their own folder and are not allowed with the other
function args tests, just like they deserve their own ring of hell
It shall be noted though that this test is particularly pointless, as it
merely tests the normal argument passing (which is fine), and not the
✨magic✨ autoargs actually provide
Change-Id: Ia2dc00754a13c02e4926200141f90bc797820a15
That one is a bit confusing, as I renamed the old `in` test to `in-2`
(file identities in Git when)
Change-Id: Id0e17de414dbc1b1b616d456175704f98657a16a
Call it a bug, call it a feature, but the old testing framework
ungracefully shits itself when it has no tests to run … in other words,
we finally migrated all the parse-fail tests 🎉
Change-Id: Ibd7f1c04d9a396a20f14361af7924f0074d1ac23
The second test has also been fixed. I'd do it in two commits if
functional1 wasn't such a PITA (guess why we're doing the migration …).
Basically, the introduced syntax got changed later on, but the test
never got updated, and back then tests didn't test the output so nobody
ever noticed it now failing for a wrong reason.
Change-Id: I5b66127d507c65676b8dd4a82d1e1f2857fef145
This really looks like one of the first tests ever written, it doesn't
seem to be actively testing anything interesting
Change-Id: I442840db932203c25da2d8400ada791f8aed04ce
This merge commit returns to the previous state prior to the release but leaves the tag in the branch history.
Release created with releng/create_release.xsh
Change-Id: Ida5959473ec469dd5e5e977ebdc8aab6e694b0de
https://github.com/NixOS/nixpkgs/pull/451579 PR
enabled c-aresSupport for curl on darwin which ended up breaking
DNS resolution in FOD sandboxes:
```
nix-run> exporting https://tangled.org/@weethet.bsky.social/nix-run (rev 73d7bf6b58848fb8f42e3a69816e0847f041c689) into /nix/store/m4m951648wmipxgwrgsml9gzjwfpfhm7-nix-run-73d7bf6
nix-run> Initialized empty Git repository in /nix/store/m4m951648wmipxgwrgsml9gzjwfpfhm7-nix-run-73d7bf6/.git/
nix-run> fatal: unable to access 'https://tangled.org/@weethet.bsky.social/nix-run/': Could not resolve host: tangled.org (Could not contact DNS servers)
nix-run> fatal: unable to access 'https://tangled.org/@weethet.bsky.social/nix-run/': Could not resolve host: tangled.org (Could not contact DNS servers)
nix-run> fatal: unable to access 'https://tangled.org/@weethet.bsky.social/nix-run/': Could not resolve host: tangled.org (Could not contact DNS servers)
nix-run> Unable to checkout 73d7bf6b58848fb8f42e3a69816e0847f041c689 from https://tangled.org/@weethet.bsky.social/nix-run.
```
with these sandbox failures:
```
deny mach-lookup com.apple.SystemConfiguration.DNSConfiguration
deny file-read-metadata /private/etc/hosts
deny file-read-data /private/etc/hosts
```
We allow those so that DNS resolution work fine
Change-Id: I9102293691972feb085adf8e9b1ad915bb3a36ab
This adds the enablement code to support HTTP/3 if the user requests it.
We leave it disabled because h3 is not onpar with h2 performance.
Change-Id: I1fd3d4c97b972dcf36bccacc6c9a8290e22b31e0
Signed-off-by: Raito Bezarius <raito@lix.systems>
Channels have moved from `nixos.org/channels` to `cnannels.nixos.org`.
This udates all relevant links (excluding release notes) to use the new
canonical URLs and replaces HTTP with HTTPS.
Fixes#1031.
Change-Id: I212821c44ac5e482c8e9eaa415c7d8ee17ff8341
Signed-off-by: adam <me@adamperkowski.dev>
URL literals were deprecated in 278fddc317,
effectively stabilizing the experimental feature. Enabling it has no effect any
more. Remove it.
Change-Id: I8fbca03b1a2be6a8cddfce644043ac0a6a6a6964
We start this section with shortcomings of unsandboxed builds.
Fixes#1018.
Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: Ieb17e4340beab0c1197951813ae602de453a3fd9
Signed-off-by: Raito Bezarius <raito@lix.systems>
This is unnecessary because Darwin builds already cannot create
`set{u,g}id` files due to the minimal sandbox policy, because we can
forbid messing with the top‐level build directory directly in the
sandbox rules, and because Darwin builds can trivially avoid isolation
through temporary directories right now anyway.
This does regress the build directory isolation for builds with
`sandbox = false`, but I can’t imagine that mattering given the
above. The sandbox change prepares us for a world where we close
off shared temporary directories for `sandbox = true` builds and try
harder to achieve proper isolation on Darwin, but probably doesn’t
have a meaningful security impact one way or another for now.
With this change, we get down to 41 byte build directory paths on
Darwin, a ~2–3½ byte improvement over the old status quo. We can
also restore the 0710 permissions on Linux.
Change-Id: I6a6a6964a681c0365241fe7234831db656b76799
The minimum build directory length on Darwin with default settings
when we were still using `/tmp` was 22 bytes. Deriving build directory
names from my local store, the median and mode were 43, the mean was
around 44½, and the maximum was 127.
The switch to `/nix/var/nix/builds` over `/tmp` added a 15 byte
penalty, and the additional `/b` directory added another 2.
Now that we use opaque build directory names, the length is 48, so
we’re still at a ~3½–5 byte penalty over the previous status
quo. This change brings us down to 43, matching the previous median.
Note that these calculations do not take into account the fact that
`/tmp` is a symbolic link to `/private/tmp` on Darwin. Anything
that was canonicalizing paths would have had an additional 8 byte
penalty in the previous status quo that is not applicable here,
so we may already be ahead even without this change. If the more
opaque directory name here is undesirable, then that factor could
potentially help us squeeze by without. Alternatively, in combination
with dropping the `/b` on Darwin we could use `/nix/var/nix/bld` or
similar, but I feel that the paths in general are sufficiently opaque
that it should be okay to go with the shorter option here. Given that
some projects already had to reduce filename lengths to avoid this
limit even before the recent changes, I think it is best to try and
improve on the previous status quo.
Note that `/nix/var/nix/builds` will unfortunately not be cleaned up
on Darwin. However, we don’t clean up the directories inside it on
Darwin anyway, so hopefully that’s okay for now?
Closes: #913
Change-Id: I6a6a6964bffce7194bcddcaefb4c4a37569c7df5
We always use the default temporary directory, because
`createUniqueDir` has an interface nice enough to use directly for
the few bespoke uses.
Change-Id: I6a6a696450b7c0a0bd76655632fb14d7c5e38199
This makes the paths more nondeterministic, but more reliably unique,
and lets us remove the retry loop.
Note that this adds random entropy to the build directory visible
inside derivations on Darwin and unsandboxed Linux. It was already
non‐deterministic in the presence of concurrent builds and similar,
but now we can reliably expect it to be different every time. On the
whole I think that’s a good thing, as it is impossible to ensure
a single consistent build directory and derivation outputs should
not depend on it.
Package reproducibility isn’t great on Darwin to begin with,
though, and the reproducibility bugs this will turn up in packages
will be more urgent to fix than when the build directory was mostly
consistent. A quick survey of my local store shows that many C, C++,
and Rust binaries contain build directory references, likely due to
use of `__FILE__` and its equivalents; non‐binary offenders include:
* Install logs included in the Rust and Cargo bootstrap compilers
* Example errors in the Rust documentation referencing build paths
* Configuration information installed with CPython itself
* Python 2 metadata from resholve’s closure
* Cython metadata
* Generated headers in Facebook libraries referencing source paths
* Generated CMake files in Facebook libraries referencing source paths
I haven’t built that much in this store since the last GC, so this is
probably only a small sample of the problems across the tree. These are
all instances of <https://reproducible-builds.org/docs/build-path/>,
though, and should probably just be treated as general reproducibility
bugs outside of contexts like the Linux sandbox where we can normalize
them away entirely.
I have implemented away build directory paths for C/C++, applied some
additional fixes for non‐`__FILE__`‐related issues in binaries
from ATF and LLVM, and fixed the derivation bug causing the CPython
3 issue, and will work on upstreaming these changes. Rust is working
on the problem upstream, with some temporary workarounds we can
potentially apply in Nixpkgs for now. The rest will require some
distributed effort.
Change-Id: I6a6a69645b4915c56c0fdef904aa81684e4136c6
This does change the behaviour when the global temporary directory
does not exist, but other uses of the global temporary directory are
already broken in that circumstance, and it should be fixed centrally
if the use case is considered desirable. The logic was not present
before the recent churn around build directories – it was added now
that Lix is taking ownership of the build directory in the store –
so this should not be a meaningful regression.
Change-Id: I6a6a6964e345ea6803226c5ad759e836de7cb0ed
There is now no risk of race conditions on a system with a functioning
entropy source, and the bespoke prefixes are either redundant to the
default or unnecessary.
Change-Id: I6a6a69641211c6bb979ea48ad30aecb1a53d03f0
Relax the constraints on keeping the exact same filename format to
provide a more robust source of entropy with a simpler interface
(as previously suggested by eldritch horrors). Using 128 bits of
OS‐provided entropy ensures global uniqueness and allows us to
skip any thought of gracefully handling the case where these files
already exist.
My microbenchmark that repeatedly constructed paths like this and
printed them out showed that this takes about 1.23× the time of
the previous implementation, both taking on the order of a couple
microseconds for one iteration. Since everything that uses it is doing
things more expensive than printing to standard output, the actual
performance delta is likely to be lost in the noise. If it somehow
becomes a bottleneck, it can be optimized without sacrificing the
guarantees by reading from the system RNG only to seed a thread‐local
CSPRNG like [ChaCha8Rand], but I think that’s very unlikely.
We also tweak the recommended way of creating a temporary file inside
a directory in anticipation of later changes, and rename the `suffix`
parameter to `prefix` (it’s a prefix to the random characters and
a suffix to the root, but this way is more consistent).
[ChaCha8Rand]: https://c2sp.org/chacha8rand
Change-Id: I6a6a69648502c746d13d8c3bd2768cbbf1b90466
base32Encode now takes std::span<std::byte>, with a base32EncodeStr
convenience wrapper which takes std::string_view.
Co-authored-by: Qyriad <qyriad@qyriad.me>
Change-Id: I6a6a6964f799dc84ecbfb55c7ca03a064cff71d9
the change to shareable thunks also removed a few cases of pointer
equality checks that allowed structures containing functions to be
considered equal to other sets containing the same functions, even
if the sets themselves were pointer-equal themselves. *so* busted.
Change-Id: If87fdab658f9037ce2a654f69a9e3da6ae2f53e5
we can't change the protocol to allow daemons to do this, and we should
not try to guess what the `auto` store uri means depending on whether a
command was run by root or not due to copious side effects and not even
being able to tell whether the `auto` store uri was given explicitly or
not. while `auto` may *technically* allow this via its naming we should
resist the urge to add a hack and fix the underlying protocols instead,
especially since repairPath should be a rare, superuser only operation.
fixes#888
Change-Id: I1b53245db226199f827a89a237a2ab9907c3f766
if curl fails during a transfer and said transfer is piped through
libarchive we see very strange errors (#1009). in such cases we'll
want to prioritize reporting the source errors to allow debugging.
Change-Id: I8cc231257eca5b749471ecd38df15071170decc2
The test would fail if the identity cannot be determined
(as is the case e.g. when using auto-allocate-uids under darwin).
Instead we use the uid when the user name is not available.
Fixes: #1025
Signed-off-by: hexchen <hexchen@lilwit.ch>
Change-Id: I12f35840b78f989b5fd09bbd44c833dc6a6a6964
This helps users to debug whether their current profile symlinks are
correctly set.
Expected outputs look like this:
```
❯ sudo ./outputs/out/bin/nix doctor
[snip]
[PASS] All profiles are gcroots.
[PASS] Client protocol matches store protocol.
[INFO] You are trusted by store uri: local
[FAIL] Error: current generation cannot be discovered for profile:
'/nix/var/nix/profiles/default'
```
```
❯ ./outputs/out/bin/nix doctor
[snip]
[PASS] All profiles are gcroots.
[PASS] Client protocol matches store protocol.
[INFO] You are trusted by store uri: daemon
[PASS] You have 28 generations for profile
'/nix/var/nix/profiles/per-user/raito/profile'
The current generation number is '290'
```
Change-Id: I50c69cbeac3291d668f4c2332803411579adc944
Signed-off-by: Raito Bezarius <raito@lix.systems>
Co-authored-by: Qyriad <qyriad@qyriad.me>
Inaccessible-but-extant PATH elements can't be executed anyway, so we
can skip them as if they didn't exist, instead of erroring like before.
Change-Id: I9288c3ecb6768171e62a3088122d98421558eb03
In d2696cdd¹, libstore was made to prioritize NIX_BIN_DIR/nix (over
/proc/self/exe) if it exists. But `pathExists()` throws an exception if
lstat() returns any error other than non-existence.
Because this check happens in libstore's Settings constructor for the
static-initialized global `nix::Settings`, this case would abort Lix
well before main().
[1]: d2696cdd1e
Change-Id: I62e44c8a33b28c234900df3e4790c1badefee358
So far it was impossible to put absolute paths into files, as one
wouldn't know whre the test would be placed.
This commit adds a new Fileish variant called `EnvTemplate` which uses
the given string as a template and replaces the `@ENV_VARIABLE_NAME@`
placeholders with the according value of the environment variable.
This way one can use `@HOME@` or `@TEST_ROOT@` to build absolute paths
Change-Id: I425cb6408dceb8d7f26d136ace4ac98b1ca31ec3
Since we have async, we are unleashed and therefore we started to cause
people to run into system limits like number of open files.
We introduce an async semaphore which uses max(25 % of max open files,
max number of cores) which should lead to 256 maximum concurrent copies.
We believe that the copy operation is the only one that can cause issues
given that the daemon already runs with obscene fd limits.
Fixes#1022.
Change-Id: Iec433d10d5c5003962ea749cf8e32bafb314f0d9
Signed-off-by: Raito Bezarius <raito@lix.systems>
In case of empty messages, it is good to print the raw error code.
Additionally, we print request IDs which can help users to reconcile
what happened with the service provider.
Change-Id: I4d83c011c1b7a5514e3d1b21123df38308279044
Signed-off-by: Raito Bezarius <raito@lix.systems>
Completion status can be updated in two ways:
- progress callback
- transfer status update callback (esp. important in multipart
transfers)
It's guaranteed that one of the two will provide progress for our
promise.
This fixes#945.
Change-Id: Iac9e92df34f82fbc1facae7a411ccabf1688da52
Signed-off-by: Raito Bezarius <raito@lix.systems>
This fixes problems with S3 implementations that have mandatory payload
signing, e.g. Garage v2, AWS S3 itself, etc.
The problem manifested itself when the AWS SDK threw some error codes 99
(NETWORK_CONNECTION) with no messages and retried until it finally came
to a moment where the state machinery decided to send a
`Transfer-Encoding` header in a `Content-Encoding`/`Content-Length`-set
request with signed headers (even though payload signing is disabled),
causing the server to reject the transfer and crash the copy.
I did not debug super far what went wrong in AWS SDK, but I can confirm
this change makes transfers possible to finish with Garage v2.
Change-Id: Icc9e6a9f2afb0d760cf2d1e27816decd385a1d85
Signed-off-by: Raito Bezarius <raito@lix.systems>
Calling `nix.nix` with an empty argument list, would crash on darwin due
to a Index Out Of Range Error, as the build="auto" functionallity would
try to access the first element of the provided arguments (second
element at the place of call) resulting in a crash
Change-Id: Ia678109808b3f75a30182114baa421cedaa8759b
libarchive's xz offers single threaded xz compression which is very slow
and provides ~10-20Mbps compression speed in addition to maxing a core.
In exchange, it achieves optimal compression ratios among all our
compression methods.
Nonetheless, xz prevent the saturation of 1Gbps+ connections and slow
down significantly decompression for end users. As these connections and
faster hardware is becoming prevalent for cache servers and clients, we
offer to default to zstd.
Lix is a "compress once, decompress many times" application. To avoid
incurring a high penalty to end users very sensitive to compress ratio
(very slow Internet connections), we dampen the consequences of
switching to zstd by increasing the default zstd level to 12.
On one example, xz will compress a 4.4GB file to 632MB, zstd on 12 will
compress it to 775MB, that is a ~18 % increase over the optimal xz
compression. zstd took 18 seconds to produce this file.
Increasing to level 14 leads to a 773MB file while taking 37s.
Increasing to level 16 leads to 735MB file while taking 66s.
Finally, xz took 77s, so a 50 % reduction in time taken to compress in
exchange of an increase of 18 % of the compressed size.
This change will reduce issues encountered in #945 but is probably not
the root cause.
References:
- https://discourse.nixos.org/t/switch-cache-nixos-org-to-zstd-to-fix-slow-nixos-updates-nix-downloads/23961
Change-Id: I7beda2bf2c1fed146dcb797b8f85dc290c486ab2
Signed-off-by: Raito Bezarius <raito@lix.systems>
if a promise for an async s3 operation is cancelled is inner fulfiller
will be destroyed, but the s3 operation may still be running. once the
operation finished and attempted to fulfill the promise we'd segfault.
Change-Id: I603c75d6cc39a1aee465b3ceee06b638f7f7f596
As we saw actual users running into them throughout the debugging of
#920 and #1014.
It's best to document and point end users to this so they can take
measures for themselves.
Change-Id: I7d08407f4354055bf65fc6dd7d1624c5a9304402
Signed-off-by: Raito Bezarius <raito@lix.systems>
Fixes#1014.
Address family detection is performed by Pasta, we influence Pasta into
enabling dual stack by listing both an IPv4 & IPv6.
Change-Id: I4121e0eae7d5185c287957106adc36edf3f82a40
Signed-off-by: Raito Bezarius <raito@lix.systems>
Instead of letting inner shell perform the cleanup, we will fork/execvp
and finish the work ourselves. Once an interruption arrives, we wait
again and return the exit status code of the inner shell.
This idea was suggested by eldritch horrors.
Fixes#1020.
Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: Id38af7ac33874aa1c4b7ca9c6123e3cbd8c3cd8c
Signed-off-by: Raito Bezarius <raito@lix.systems>
This was forgotten which should have been part of 64b9247f.
Change-Id: I3cf83d8699f2c1963ce530bbea6a511075202a79
Signed-off-by: Raito Bezarius <raito@lix.systems>
Legacy commands "main function" will now return an exit code.
Fun fact: `main_nix_build` was the only one not returning integers
already.
Change-Id: Ia43a16c3c3fb9a670e8889aefc4ee9b6528a7df4
Signed-off-by: Raito Bezarius <raito@lix.systems>
This also prevent `legacy` to return nothing as it's void.
Change-Id: Ic797544a59b04b41ad9e1c46af4cad3a1cb6fdc8
Signed-off-by: Raito Bezarius <raito@lix.systems>
realistically only runProgram is ever called for interactive reasons,
and even those calls seem to be rather ill-advised in many cases. the
chance of multiple interactive processes interfering with each other,
whether for input or for output, must be very low to make this in any
way reasonable: if e.g. git calls ssh for multiple fetched inputs and
ssh requests passphrases for both we can otherwise not guarantee that
*any* input is routed correctly. misrouted output is merely annoying.
Change-Id: I794e3fdf0a3238cb9292003a89ac267f0de7a939
Eval errors will now print a simple, no frills chain of involved
derivations at the bottom. For example, trying to evaluate
`pkgs.xonsh.override { python3 = pkgs.python2; }` has the usual
Package ‘python-2.7.18.8’ in /nix/store/9v6qa656sq3xc58vkxslqy646p0ajj61-source/pkgs/development/interpreters/python/cpython/2.7/default.nix:398 is marked as insecure, refusing to evaluate.
message, but now also includes the following:
note: trace involved the following derivations:
derivation 'xonsh-0.19.9'
derivation 'python2.7-xonsh-0.19.9'
derivation 'python2.7-setuptools-44.0.0'
To give the user information about why the erroring derivation was
involved in the first place.
We would like more structured information in the future, but this should
still be a significant improvement.
Change-Id: Icf6da52abd0a043cfb63943bf0b0c160c21ee59e
the wrapper is needed by transfer streams to restart a failed transfer
if desired. curlFileTransfer itself is more of a fancy handler for the
thread we're dedicating to curl io handling. the thread will stay with
the multi handle for now because quit handling needs to stay there. we
could have CurlMulti keep only a flag, but that does not help us much.
Change-Id: I99550f0bbb635b75898ca7260f08275df86050e3
we have anonymous namespaces and no other scoping requirements, we can
move it out. this will also make it easier to move *other* things out,
such as the transport streams and all the shared multi state handling.
Change-Id: Iafa1eaeb69e096e98c4667f0531b17d5fe7c78a6
waiting for them to finish leaves us prone to deadlocking, e.g. if some
active transfers is blocked and hasn't been destroyed before our thread
was asked to exit. aborting transfers is the safer option here; even if
we cause use after frees we'll at least have a crash, not hang forever.
Change-Id: I333dcced810e8a85e2ded1f377962a0176798098
If BUILD_TEST_SHELL is not set or empty, None or the empty string respectively
should not make it into the PATH. Ensure this property.
Change-Id: I4ce9b0c06c407b465308b63b9cb64e7d6a6a6964
this is still somewhat experimental and should be considered in flux. we
will have to nail down a logger interface once we start moving the store
protocol to rpc, but until we do that we can use build hooks to test it.
Change-Id: Id20cd346c9520f45871799c31b0af040adde56ef
extracting the size-specific log handling into a new sink deduplicates
the size-limiting code and lets us clean up the derivation goal a bit:
a number of log-related fields we kept only because previous ownership
semantics were a lot more ad-hoc can be moved into the new sink class.
Change-Id: Ie770a276597c84f9d928a9624c0f0de86adfe815
we'll want to remove handleRawChildStream from DerivationGoal, but the
timeout handling is still important. the wrapper is thus useful, but a
more generic way to pass it a promise to do the handling is necessary.
Change-Id: I55bf041d51e4f8a8b8291deae053423d06724329
we never need the connection or the rpc client directly after the hook
is constructed, only the hook instance rpc object. wrapping them using
kj primitives makes this more explicit and prevents destruction errors
Change-Id: I7e0b14d7e365c2e001273e04c0e741ba4afb12f2
this makes it possible to wait for the hook to exit in one place,
process its exit status for some purpose, and later process it in
another place for a different purpose. rpc will need this to shut
down a hook cleanly after the hook has completed its assignments.
Change-Id: I5af2d5aac4b02a0a200d720d0e6f0b5df5496aaf
only LocalDerivationGoal needs these, and once the child is done
running (or is begin torn down) we can just destroy all of them.
Change-Id: Id4d9f0105aaf33d6736c435bb891d3f9af8e7894
wrapping the wait in a timeout at the call site is insufficient because
a timeout would not relock the mutex and leave the raii object invalid.
Change-Id: I0be2aaf6b3c9966070fb0f1deaab048765fd7af3
it seems like pytest is updating the message printed when skipping a set
due to no parametrization
which leads to breakage on staging-next.
this commit changes the expected message to allow both old and new skip
message
Change-Id: I34d11ad84d7075911d8229fbc89889bb880a27c9
This partially reverts commit f68233ec43.
while more modern capnp in *does* have the build system changes, it
does *not* have the patches. and those patches are rather important
to us; the clock patch because our CI machiens don't behave the way
capnp expect and the nodiscard patch because it *is* a bugsquasher.
Change-Id: Ic273777f09fd8e5e7001f815db8998dae2dd7b88
The prospective callers of this should probably be doing something
smarter or more abstracted to begin with, but this is useful as an
incremental improvement for call sites with existing `makeTempPath`
logic in the face of filename length limits.
Change-Id: I6a6a69644292f5bbf984a1df90192e06c6022b53
LocalStore::optimisePath() was made asynchronous in 38b75b736¹, but we
lost warnings for not awaiting kj::Promise in f68233ec4², since our
capnproto no longer has the [[nodiscard]] patch.
[1]: 38b75b7367
[2]: f68233ec43
Change-Id: I2363396949f40dd43eae08cc76be9ede6b488cf7
this is important for rpc error reporting: if the peer disconnects we
want to throw a useful error that can e.g. let pending builds finish,
not kill the entire process because kj errors are non-lix exceptions.
Change-Id: I41ebe10ad7fd65d886238ce52eeca1fa243b1bb2
this removes another file descriptor from the build hook. we are now
using only rpc to communicate between build hook and daemon, setting
the stage for a future world in which we don't even need build hooks
Change-Id: I706d9d3f2a420abd42fc7762c205931a45d3c8c5
interpolation of the remote store name into activity text is now done in
the build hook, so we no longer need to send it back to the daemon. this
also reduces the dependency of the daemon on how *exactly* remote builds
are done a little bit, and maybe in time we can make these fully opaque.
Change-Id: Ie51c4f2cd107350fec4ad0a8ef83621ed86db6a2
only local builds can now have a builderOutFD, remote builds only log
via json streams. these two kinds of logs have different requirements
and each is only needed by its respective build kind. splitting these
apart thus makes sense, though ideally we'd also split DerivationGoal
into a RemoteDerivationGoal to clean up the rest, but that will wait.
Change-Id: Ib577537266d1160355ab9c44b4604ebda87a7d04
treat it like ssh-ng instead, and have the build hook do the translation
of ssh stdout to the json log message steam the derivaiton goal can use.
since the ssh-ng path also handles all store urls that aren't legacy ssh
we now have a single logging system that handles every remote build kind
equally, without requiring fd passing into the build hook. this is later
required to rpc-ify the log stream emitted by build hooks to the daemon.
Change-Id: Ifb522eb8a9745029050f16b1b3b3601a6ddac748
When the maximum memory size is chosen too large (relative to the free memory
usage), so that the evaluation workers don't fit into the free memory, they
will start being swapped out. Further increase in memory usage will not be
reflected in the RSS, so they will not exit, and instead fill up all swap too
until the system runs out of memory.
The BDW-GC keeps track of its heap size, including any parts of it that may be
swapped out. For this reason it does not suffer from a similar problem. While
not all memory usage by the evaluation worker is accounted for by the GC, a
simple affine model for the overhead works reasonably well in practice. The
specific parameter values were determined using nixpkgs evaluation, and appear
to work quite well also for somewhat different workloads (like the `hydraJobs`
of Lix, which include a bunch of NixOS system configurations). So this is what
gets used.
When Lix is configured without the BDW-GC, of course this approach cannot be
used. In this case the old strategy is retained.
Change-Id: I6cb5f0a9b4ceda9dd14be165dda108cd6a6a6964
Includes link to the Buildkite SSO login, which is currently not visible
anywhere else in documentation.
Change-Id: Ic3ec6412127b2ac0df7e6efc4ec6020cc9d8b34f
These are a footgun and are not acceptable in functional2 due to thread
safety, effects on other tests, etc.
Co-authored-by: Commentator2.0 <lix@crystal-cavern.systems>
Change-Id: I8d7285061eaa9bab27edd52f3646024c8cf605e5
When providing both a `.err.exp` and a `.out.exp` for a lang test
**not** containing a toml, the test was collected twice and hence threw
a duplicate id error.
This commit adresses this issue
Change-Id: Ia781d69e0f1db2809a30192d6cd0a98861e6cc32
this restriction was added in 4af2611bd1
and extended later. it makes little sense to allow *one* activity type
specifically and drop all others on the floor, especially since broken
builders can create transfer activities with ids that would be used by
other, "real" activities. the only thing achieved here is to drop logs
written in json format in the build sandbox, which does not help much.
we may revisit this restriction later during other log-related rework.
Change-Id: I8bda494083877b71a2f958470fa52380f6ec4968
it was split up like this only because the old worker system had no
promises. since we have them now we can use them for encapsulation.
Change-Id: Idbd523eff617d4c53c14b125ab3dfce4979cdab1
currently all loggers can always accept messages and never suggest
flushing buffers. in the future this may change, and at that point
we're already fully set up for it. local loggers should never keep
asynchronous (i.e. network-backed) buffers, disk buffers are fine.
networked loggers will require buffers and periodic flushes later.
Change-Id: Ide2114f5bc17f4a1d289c92ed4f9381a1d59dacf
this keeps the call stack flat even for very deep store directories.
moving progress reporting out of the method optimizing one path also
lets us move the inevitable IO wait and associated promise overhead.
Change-Id: I5083d88b05c247c7e19a21a646bcf3f62bfa3200
create activities from loggers themselves instead of passing the logger
as a constructor argument and allow direct construction of children, no
direct logger access needed. most call sites are not changed because we
still need to handle the "no parent" case, and the logger method can do
that more cleanly than a ternary at each site that creates an activity.
we may eventually want to create a root activity, which is cleaner too.
Change-Id: I295e056228dabb08a1316eba7973874784baa113
remotes can generate the same activity ids we might generate locally,
especially if the remote is accessed over the network. in that case a
pid collision is possible, and since activity ids are sequential it's
very possible to create colliding activity ids on both sides as well.
Change-Id: Id58074a41f5f7a59171b52818d1fb5a1beb4bf40
activities are scopes, and there's no reason scopes should not be able
to move around. this also means we can *create* them elsewhere without
also having to box them in some way, making parent relationships clear
Change-Id: I5df036e12ebd8270feb4dca1f23b6bee1f08e906
Currenlty `lix` and `nixos-module` have slightly diverging version
numbering. This commit fixes the lix side of things, in accordance to
the format proposed in
https://git.lix.systems/lix-project/lix/issues/585#issuecomment-7336
which allows for better use of `builtins.compareVersions`
fixes#585
Change-Id: I2072b701f9fadf780e7aab67b62fbcd2ec5df76d
this has no performance impact in any benchmarks we've run. nul bytes
are still used as implicit truncation points in many places all over:
rejecting them in all locations that treat them as a string end point
requires large changes such as using a proper path library everywhere
Change-Id: I936158bd435f6abf009a689adfbc24496262c578
This reverts commit 42691f0d94.
Reason for revert: darwin builds time out now, apparently due to newly appearing infinite loops in the test suite
Change-Id: I81a09efd53636a287f3095ac87ed8854b6e57922
The functional2 test suite was broken in the development shell for two mostly
separate reasons leading to "no such file or directory" errors:
* The `BUILD_TEST_SHELL` (already containing the correct path) would always be
set from the Meson option `build_test_shell`, even to the empty string if
that option is not set. Skip the overwrite in this case to make the
environment variable work again.
* Sandboxed builds would fail to find their builder `/bin/sh`, since in the
development shell busybox is not found, so no sandbox shell gets configured.
Use the shell from `BUILD_TEST_SHELL` instead. (How this does not break the
old functional tests remains a mystery.)
Change-Id: I6a6a696424e8caaef3f9b68e3738bfd58ea0b056
Local store make use of SQLite as a cache for output maps, they can
diverge with actual store reality.
Causing a crash at realization time.
We can just throw an error instead.
Change-Id: I943a4bddfd2461a34933e9ec079a6638ab58e5b1
Signed-off-by: Raito Bezarius <raito@lix.systems>
Fixes fj#940.
When running `nix-shell`, the `$NIX_BUILD_TOP` environment variable is
set to `$TMPDIR` or `/tmp`.
nixpkgs stdenv uses $NIX_BUILD_TOP to create `$NIX_BUILD_TOP/env-vars`
which contains all the environment variables set by stdenv. This is used
for debugging purposes in combination with `--keep-failed` to reload the
bash environment of a derivation.
`$TMPDIR` is often unset, therefore, `/tmp/env-vars` was constantly
being created. On a multi-user system or, when you run Lix as root, you
might create a `/tmp/env-vars` with different permission bits.
As a result, `nix-shell` can cease to function because that file will
fail creation for an unprivileged user for example.
fj#940 rightfully remark that the code is not consistent between
nix3-develop and nix-shell and it should be reworked.
Change-Id: Iddf15945385d8bd497b2800b37fee5e1f97689b7
Signed-off-by: Raito Bezarius <raito@lix.systems>
This avoids mangling debug gdb scripts and breaking them resulting in no
script loading at all.
Change-Id: Ib5e7034c37a1e41d1a1c9bd8442b8b8b8758e350
Signed-off-by: Raito Bezarius <raito@lix.systems>
UnsafeValueList held pointers from Value which were not necessarily GC
allocated, causing mayhem when evaluating something with genericClosure
(texlive environments).
We get rid of storing pointers and we let comparison take places on
const references.
We keep `gc_allocator<Value>` so that GC can scan things it allocates
inside the `res` list.
Co-authored-by: eldritch horrors <pennae@lix.systems>
Reported-by: qbit
Change-Id: I4fed3a3d9e18a2ef2d751f32d81801540e196f92
Signed-off-by: Raito Bezarius <raito@lix.systems>
thunk values are shareable, and we can represent invalid/uninitialized
values with a special bit pattern that makes no sense otherwise. there
is no need to keep allocating values on the heap, instead we can treat
values like reference-counted smart pointers to heap objects, which in
turn lets us save a lot of allocations and, ultimately, gc heap space.
compared to our baseline (main of 2025-09-27) we save 15%+ memory on a
system rebuild and 17% on nix search. eval time regresses by ~3% for a
system rebuild, while nix search is 7% faster. further optimization is
probably possible (but for now this will just have to be good enough).
Change-Id: Ib6c47acdbe2fac4f76a83c2269f16f30ef66b2e1
with thunk state being shareable we no longer need to worry about value
uniqueness, only about value lifetime. this means we can liberally drop
indirections and allocations, passing references instead of pointers or
using stack memory instead of gc-managed memory for some intermediates.
Change-Id: I2d48a6fd57a376d544bd9bd2d05e5420611986d1
there's no measurable performance gain in real-world testing to abusing
our list value storage like this. we haven't tested how much storage we
actually need on the stack to offset most of the temporary storage cost
and used 128 as a "good enough" value instead, reserving 1 kiB of stack
space on 64 bit platforms in a leaf function. this should do for a bit.
Change-Id: Ie98519b1da3e6fe685da88d1c44ffb4580fb592a
Value is already tagged. Value::Acb blocks are allocated for lambdas (so
we can fit the value tag into the three bits we have available), but the
current layout is rather wasteful for this purpose. the type bits can be
stored together with parts of pointers, which in the lambda case will be
the scope the lambda captures. the expr could also be used, but Env is a
gc-allocated item and thus guaranteed to be aligned properly for tagging
Change-Id: Ia685875387c7795bc4a00d73d1ce3cfea84e7297
this is a strong prerequisite for making values themselves copyable
without duplicating evaluation side effects. with this we can treat
`Value` the way we treated `Value *` to date and drop indirections.
Change-Id: I08f30d12697614a3ae7149615f6f1da83b13f52b
preserve type information of wrapped exception when constructing a
ForeignException. this requires using `std::current_exception` for
construction in all cases as passing the exception by reference to
any function erases dynamic type information: `throw` may copy the
thrown exception from its source expression; this only works as we
need it to if static and dynamic type of the expression are equal.
fixes#1000
Change-Id: I7167f298540de7c43c2566592bdb8d7a93c34d31
same as for null: we have few of them, they're statically allocated,
and they're not the largest contributor to the value population. not
storing them in Value itself frees up resources we *will* use later.
Change-Id: I521f9f243f48f56a78f7bffdf1dc1f0bc40a5c2d
we only need the one object for it. there's no need to waste precious
bits of the value internal type enum for this simple singleton datum.
Change-Id: Ie314b5bf429015e518798d9d65ad8ab2bb84a38e
floats are used very rarely, and our float support is bad enough to
strongly discourage using them on reproducibility grounds alone. we
can thus move them to more expensive storage without hurting folks.
Change-Id: I1086f612f85e294dd3fae4a2d334e09f52bbe4a8
external values very rarely appear during eval "normal" eval, and
creating them is pretty expensive. does *anything* even use them?
Change-Id: Id50fa3f76b7e1f551d550d99996a1ed5880b2531
despite not using allocation caches this does not have a statistically
significant performance impact, with less than 1% extra memory needed.
Change-Id: Ibe51a55ba986e471f217f3724977af17880fafff
using the same nodes as tApp is possible, and thanks to multi-arg app
nodes it can even be a bit faster than the linked lists used to date.
Change-Id: Idccb7c0b54c808e62da85d1c42ee09e6e92c4f7b
these behave like the old chains of app nodes, but they can store more
than one argument per node. for tApp values themselves this is not all
that useful, but if we could share tApp and tPrimOpApp backing storage
we could avoid creating and traversing the linked lists of values that
are currently needed to represent partially applied builtin functions.
Change-Id: I5a2a02d9733e1e0be5443459e2998d62fd3b9a5b
paths already are just strings with slightly magic semantics. the type
can mirror that at no perf cost, letting us drop one internalType tag.
Change-Id: I98acaa4fe3bedd28fc0841e1b81184d8dcddacc5
memory overhead is minimal and performance impact not measurable. once
we've done something like this for all value types that don't fit in a
single machine word we can cut a word from Value, offsetting the cost.
Change-Id: I9813bacd7e851957ad3426aed8f74033179a4212
we now use a single cache set for a number of sizes from one to eight
words. this also matches small attrsets, but perf impact seems small.
Change-Id: Icf16b329b98a20fcc9fe75e6395e148f0852c798
on its own this is not very useful, but having accessors for every value
kind is a prerequisite for doing smart things with Value than the union.
the net effect for now is only to add a few parentheses across the tree.
Change-Id: I88688ac09eb08495dad1eb221034ca540f094950
If you need to disable cgroups temporarily, remember that you can do
`NIX_CONF='include /etc/nix/nix.conf\nuse-cgroups = false' nix-build ...` or
`nix-build --no-use-cgroups ...`.
## What about other service managers than systemd?
systemd has a [documentation](https://systemd.io/CGROUP_DELEGATION/) on how to
handle cgroup delegation from service management perspective.
If your service manager adheres to systemd semantics, e.g. writing an extended
attribute `user.delegate=1` on the delegated cgroup tree directory and moving
the `nix-daemon` process inside a cgroup tree to respect the inner process
rule, then, the feature will work as well.
## Why is the cgroup feature still experimental?
While the cgroup feature unlocks many use cases, its behavior and integration (e.g. user experience), especially at scale on build farms or in multi-tenant environments, are not yet fully matured. There’s also potential for deeper systemd integration (e.g. using slices and scopes) that has not been fully explored.
To avoid locking in an unstable interface, we’re keeping the experimental flag until we have validated the feature across a broader range of scenarios, including but not limited to:
synopsis:Fix develop shells for derivations with escape codes
issues:[fj#991]
cls:[4154,4155]
category:Fixes
credits:[Qyriad]
---
ASCII control characters (including `\e`, used for ANSI escape codes) in derivation variables are now correctly escaped for `nix develop` and `nix print-dev-env`, instead of erroring.
Previously, `builtins.floor` and `builtins.ceil` always cast the input into a floating point value before running the operation and casting the floating point result back into an integer.
No checks were made for precision loss in either coercing integer inputs or converting the output to an integer (and in fact in the latter case, invoked undefined behaviour).
Now, Lix checks for precision loss on integer input (to avoid a silent eval semantics change if we were to simply pass it through as-is) and on integer output.
If your code fails to evaluate after this change, use `--extra-deprecated-features floor-ceil-corrupt-integers`.
The attribute set printer, such as is seen in `nix repl` or in type errors, now prints hyperlinks on each attribute name to its definition site if it is known.
Example: all of the attributes shown here are hyperlinks to the exact definition site of the attribute in question:
1. These builds are not run via the daemon, which owns `/nix/var/nix/builds`.
2. The user lacks permissions for that path.
We considered making `build-dir` a store-level option and defaulting it to `<chroot-root>/nix/var/nix/builds` for chroot stores, but opted instead for a fallback: if the default fails, Nix now creates a safe build directory under `/tmp`.
To avoid CVE-2025-52991, the fallback uses an extra path component between `/tmp` and the build dir.
**Note**: this fallback clutters `/tmp` with build directories that are not cleaned up. To prevent this, explicitly set `build-dir` to a path managed by Lix, even for local workloads.
You're not alone. Thousands of Lix users suffer every day from excessive `builtins.toString` syndrome. It’s 2025, and we still have to cast integers to use them in strings.
To address this, Lix introduces the **`coerce-integers`** experimental feature. When enabled, interpolated integers within `"${...}"` are automatically coerced to strings. This allows writing:
To enable the feature, you need to add `coerce-integers` to your set of experimental features.
### Stabilization criteria
The `coerce-integers` feature is experimental and limited strictly to string interpolation (`"${...}"`). Before stabilization, the following must hold:
1.**Interpolation-only**
Coercion must not occur outside interpolation. Expressions like `"" + 42` must continue to fail.
2.**Expectation that no explicit cast are being observed**
Cases observing explicit coercion (e.g., via `tryEval` gadget or similar) are expected not to be load-bearing in actual production code.
### Timeline for stabilization
If the feature proves safe and is widely adopted across typical usage (e.g., actual configurations in the wild turning on the flag, non-trivial out-of-tree projects using it), the experimental flag will be removed **after six months of active use or two Lix releases**, whichever is longer.
This avoids locking the feature in experimental status indefinitely, as happened with Flakes, while allowing time for validation and ecosystem integration.
### What about coercing floats or more?
Coercion beyond integers -- such as for floats or other types -- is **not planned**, even under an experimental flag. Questions like "what is the canonical string representation of a float?" involve subtle and context-dependent trade-offs. Without a robust and principled mechanism to define and audit such behavior, introducing broader coercion risks setting unintended and hard-to-reverse precedents. The scope of `coerce-integers` is intentionally narrow and will remain so.
In terms of outlook, a proposal like https://git.lix.systems/lix-project/lix/issues/835 could pave the way for a better solution.
synopsis:"flake config warnings are now printed to stderr"
issues:[1155]
cls:[5379]
category:"Fixes"
credits:[lheckemann]
---
The settings listed in a flake-config confirmation prompt are now printed to stderr rather than stdout, which allows `nix print-dev-env` to emit valid bash again even in the presence of untrusted settings.
synopsis:"Shadowing internal files through the Nix search path is now an error"
issues:[998]
cls:[4632,5370]
category:"Breaking Changes"
credits:[thubrecht, jade, horrors]
---
As Lix uses the path `<nix/fetchurl.nix>` for bootstrapping purposes, the ability to shadow it by adding `nix=/some/path` (or `/other/path` that contains a `nix` directory) to the search path is not desirable.
Lix 2.95 deprecated this behavior with a warning, Lix 2.96 now turns it into a hard error if the `nix-path-shadow` deprecated feature isn't enabled. This deprecated feature is slated to be removed in Lix 2.98.
synopsis:Remove reliance on Bash for remote stores via SSH
issues:[fj#830, fj#805, fj#304]
cls:[3159]
category:"Fixes"
credits:[raito]
---
The pre-flight `echo started` handshake -- added years ago to catch race conditions -- has been removed.
After removal of connection sharing in Lix 2.93, it required a Bash-compatible shell and a standard `echo`, so it failed on:
* builders protected by `ForceCommand` wrappers (e.g. `nix-remote-build`),
* BusyBox / initrd images with no Bash,
* hosts using non-POSIX shells such as Nushell.
The race the probe once addressed was tied to SSH connection-sharing -- since connection-sharing code has already been removed, the probe is now pointless.
Real connection or protocol errors are now left to SSH/Nix to report directly.
This is technically a breaking change if you had scripts that relied on the literal "started" which needs to be updated to rely on other signals, e.g., exit codes.
synopsis:"repl-overlays now work in the debugger for flakes"
issues:[fj#777]
cls:[3398]
category:Fixes
credits:[jade]
---
Due to a bug, it was previously not possible to use the debugger on flakes with repl-overlays, or with pure evaluation in general:
```
$ nix repl --pure-eval
Lix 2.94.0-dev-pre20250617-87d99da
Type :? for help.
Loading 'repl-overlays'...
error: access to absolute path '/Users/jade/.config/nix/repl.nix' is forbidden in pure eval mode (use '--impure' to override)
```
This is now fixed.
The contents of the repl-overlays file itself (i.e. most typically the top level lambda in it) will be evaluated in impure mode.
It may be necessary to use `builtins.seq` to force the impure operations to happen first if one wants to do impure operations inside a repl-overlays file in pure evaluation mode.
synopsis:"`disallowedRequisites` now reports chains of disallowed requisites"
issues:[fj#334,fj#626,gh#10877]
category:Improvements
credits:[ma27,roberth]
---
When a build fails because of [`disallowedRequisites`](@docroot@/language/advanced-attributes.md#adv-attr-disallowedRequisites), the error message now includes the chain of references that led to the failure. This makes it easier to see in which derivations the chain can be broken, to resolve the problem.
Example:
```
$ nix-build -A hello
error: output '/nix/store/0b7k85gg5r28gb54px9nq7iv5986mns9-hello-2.12.2' is not allowed to refer to the following paths:
synopsis:"Lix libraries can now be linked statically"
issues:[fj#789]
cls:[3775,3778]
category:Fixes
credits:[alois31]
---
Previously the pkg-config files distributed with Lix were only suitable for dynamic linkage, causing "undefined reference to…" linker errors when trying to link statically.
Private dependency information has now been added to make static linkage work as expected without user intervention.
In addition, relevant static libraries are now prelinked to avoid strange failures due to missing static initializers.
In the Lix evaluator, **symbols** represent immutable strings, like those used
for attribute names.
In evaluator design, such strings are typically [**interned**](https://en.wikipedia.org/wiki/String_interning), stored uniquely
to save memory, and Lix inherits this approach from the original C++ codebase.
However, some builtins, like `builtins.attrNames`, must return a `Value` type
that can represent any Nix value (strings, integers, lists, etc.).
Before this change, these builtins would create lists of `Value` objects by
allocating them through the garbage collector, copying the symbol’s string
content each time.
This allocation is unnecessary if the interned symbols themselves also hold a
`Value` representation allocated outside the garbage collector, since these
live for the full duration of evaluation.
As a result, this reduces the number of allocations, leading to:
* A significant drop in maximum [resident set memory](https://en.wikipedia.org/wiki/Resident_set_size) (RSS), with some large-scale
tests showing up to 11% (about 500 MiB) savings in large colmena deployments.
* A slight decrease in CPU usage during Nix evaluations.
This change is inspired by https://github.com/NixOS/nix/pull/13258 but the approach is different.
**Note** : [`xokdvium`](https://github.com/xokdvium) is the rightful author of https://gerrit.lix.systems/c/lix/+/3300 and the credit was missed on our end during the development process. We are deeply sorry for this mistake.
Flakes try to keep their inputs and metadata "simple", to make sure no unbounded computation may happen when calling e.g. `nix flake show`.
Those checks were haphazard, a maintenance burden, and also easily circumventable.
Lix has now replaced all the old checks by a simple rule: **No function calls outside of `outputs`.**
This is easier to reason about than the previous set of inconsistent rules, and crucially now also allows syntax features that users felt like they *should* have worked in the past, like let bindings.
However, some warts still remain for now: Some syntax constructs like `-1` internally desugar to `__sub 0 1`, which is a function call and thus remains forbidden.
This will be rectified as soon as the deprecation period of the respective anti-features has been completed.
This change is **breaking** in the sense that flakes which are written with the newly allowed language features will not evaluate with an older Lix version which still uses the old, more restrictive checks.
Crucially, this also affects **all transitive dependants** of such Flakes.
> If you are building via the Lix daemon (default on Linux and macOS), it is the Lix daemon user account (that is, `root`) that should have SSH access to a user (not necessarily `root`) on the remote machine.
>
> Furthermore, `root` needs to have the public host keys for the remote system in its `.ssh/known_hosts`.
> To add them to `known_hosts` for root, do `ssh-keyscan USER@HOST | sudo tee -a ~root/.ssh/known_hosts`.
> To add them to `known_hosts` for root, do `ssh-keyscan HOST | sudo tee -a ~root/.ssh/known_hosts`.
>
> If you can’t or don’t want to configure `root` to be able to access the remote machine, you can use a private Nix store instead by passing e.g. `--store ~/my-nix` when running a Nix command from the local machine.
## Configuration
The list of remote machines can be specified on the command line or in
the Lix configuration file. The former is convenient for testing. For
example, the following command allows you to build a derivation for
- `uri`: Due to backward compatibility, the `ssh://` may be omitted for the store-uri.
- `ssh-public-host-key`: The key must be provided encoded in base64. Specifically calculated via `base64 -w0 /etc/ssh/ssh_host_ed25519_key.pub`
### Format detection
At first, the given configuration is being parsed syntactically as a toml.
If parsing fails and the given configuration contains a `"` the error is presented to the user, as those characters are necessary for TOML, but disallowed for the legacy format.
Otherwise, parsing is retried using the legacy format.
If non-syntactic errors are detected within the toml, the exception will always be shown to the user directly.
## Builder selection
The configuration(s) above specify several machines that can perform `i686-linux` builds.
However, `granite` will only do builds that have the attribute
```nix
requiredSystemFeatures = [ "benchmark" ];
```
or
```nix
requiredSystemFeatures = [ "benchmark" "kvm" ];
```
`diorite` cannot do builds that require `kvm`, but `andesite` does support
such builds. For regular builds, `diorite` will be preferred over
# [Pasta](https://passt.top/passt/about/): a network sandbox for fixed-output derivations
## Introduction
This section only applies to **Linux systems** as Pasta is a Linux-only measure.
Since [CVE-2025-46416](https://lix.systems/blog/2025-06-24-lix-cves/), the Lix project decided to adopt [Pasta](https://passt.top/passt/about/) for all fixed-output derivations, protecting against various attack vectors such as UNIX abstract domain sockets or more manipulation at the network layer from a malicious fixed-output derivation code.
Pasta acts as a translation layer between a layer-2 network interface and layer-4 sockets (TCP, UDP, ICMP/ICMPv6 echo) on the host. It requires no special privileges and can serve as a alternative to [SLiRP](https://en.wikipedia.org/wiki/Slirp) which was used [by Guix to mitigate the same problem](https://codeberg.org/guix/guix/commit/fb42611b8f27960304db5a1c0d33b8371dcde2a8).
## How to disable Pasta?
It's sufficient to pass `pasta-path = ""` in your `/etc/nix/nix.conf` or on the command line `--pasta-path ""` of a Lix invocation.
## Known issues surrounding Pasta
- Only the first DNS server in `/etc/resolv.conf` is considered: failover is not possible.
- [Reduced feature set compared to the Linux kernel](https://passt.top/passt/about/#features)
- [Performance overhead in multi-gigabits contexts and IMIX MTUs](https://passt.top/passt/about/#performance_1)
@@ -58,7 +58,7 @@ $ nix-build flake:nixpkgs -A firefox
$ nix-build flake:github:NixOS/nixpkgs/release-23.11 -A firefox
```
Finally, for legacy reasons, if a path starts with `channel:`, the rest of the argument is interpreted as the name of a *nixpkgs* channel tarball to fetch from `https://nixos.org/channels/$CHANNEL_NAME/nixexprs.tar.xz`.
Finally, for legacy reasons, if a path starts with `channel:`, the rest of the argument is interpreted as the name of a *nixpkgs* channel tarball to fetch from `https://channels.nixos.org/$CHANNEL_NAME/nixexprs.tar.xz`.
This is a **hard coded URL** pattern and is *not* related to the subscribed channels managed by the [nix-channel](./nix-channel.md) command.
> **Note**: any of the special syntaxes may always be disambiguated by prefixing the path.
Channels are a mechanism for referencing remote Nix expressions and conveniently retrieving their latest version.
The moving parts of channels are:
- The official channels listed at <https://nixos.org/channels>
- The official channels listed at <https://channels.nixos.org>
- The user-specific list of [subscribed channels](#subscribed-channels)
- The [downloaded channel contents](#channels)
- The [Nix expression search path](@docroot@/command-ref/conf-file.md#conf-nix-path), set with the [`-I` option](#opt-I) or the [`NIX_PATH` environment variable](#env-NIX_PATH)
@@ -77,9 +77,9 @@ This command has the following operations:
Subscribe to the Nixpkgs channel and run `hello` from the GNU Hello package:
@@ -177,6 +177,12 @@ Most commands in Lix accept the following command-line options:
You can override this using `--arg`, e.g., `nix-env --install --attr pkgname --arg system \"i686-freebsd\"`.
(Note that since the argument is a Nix string literal, you have to escape the quotes.)
Additionally, dots are interpreted as attribute-path separators.
I.e. `nix-instantiate '<nixpkgs>' -A hello-unfree --arg config.allowUnfree true` will result in an argument `config` with value `{ allowUnfree = true; }` being passed to `<nixpkgs>`.
Please note that merging of different arguments is rejected.
I.e. `--arg config '{ cudaSupport = true; }' --arg config.allowUnfree true` will not work whereas `--arg config.cudaSupport true --arg config.allowUnfree true` is accepted.
@@ -19,7 +19,7 @@ This description is not normative, but a feature removal may roughly happen like
1. Add a warning when the feature is being used.
2. Disable the feature by default, putting it behind a deprecated feature flag.
- If disabling the feature started out as an opt-in experimental feature, turn that experimental flag into a no-op or remove it entirely.
For example, `--extra-experimental-features=no-url-literals` becomes `--extra-deprecated-features=url-literals`.
For example, `--extra-experimental-featuresno-url-literals` becomes `--extra-deprecated-featuresurl-literals`.
3. Decide on a time frame for how long that feature will still be supported for backwards compatibility, and clearly communicate that in the error messages.
- Sometimes, automatic migration to alternatives is possible, and such should be provided if possible
- At least one NixOS release cycle should be the minimum
@@ -51,28 +51,64 @@ $ nix-shell -A native-clangStdenvPackages
### Building from the development shell
Run a clean build and test with `just clean setup buildinstall test`.
We have a [justfile](https://just.systems/) for extra convenient building.
It defaults to using `./build` as the build directory, and `$out` (`./outputs/out`) as the install directory.
For most cases, you can clean-build, install, and run the tests with:
```bash
$ just setup --wipe && just test
```
> **Note**
>
> The `--wipe` argument to `meson setup` conveniently works whether you have an existing build directory or not.
>
> However, it is *mostly*, but not *exactly* equivalent to deleting the build directory first.
> In particular, previously specified `-D` build options are **preserved** with `--wipe` (for some reason).
> For example, if you fetch and checkout a new version of Lix, and that new version *removes* a Meson build option from `./meson.options`, *and* a previous invocation in that build directory explicitly set that option, then `meson setup --wipe build` will error, complaining about the unknown option.
> For these cases, `just clean` will give you a well-and-truly-this-time-for-real clean build.
Because the integration tests require installation to work, `just test` automatically also calls `just install`, and Meson helpfully will automatically build any targets that need building when trying to install them.
You can override the build directory or install directory by setting the justfile [variables](https://just.systems/man/en/setting-variables-from-the-command-line.html) `outdir` and `builddir` on the command-line:
```bash
$ just builddir=build-before-bisect outdir=out-before-bisect setup
$ just builddir=build-before-bisect test
```
You'll have to set `builddir` for every target, but `outdir` only needs to be set for `setup`.
You can also run the unit tests and integration tests separately:
```bash
$ just setup build test-unit
$ just install test-integration
$ just setup
$ just test-unit
$ just test-integration
```
Many justfile aliases have a `-custom` variant which pass extra arguments to `meson`.
Most justfile targets forward all further arguments to the underlying Meson invocation.
For example, to work on both Lix and nix-eval-jobs you can run:
```
$ just setup-custom -Dnix-eval-jobs=enabled
$ # or
$ mesonFlags=-Dnix-eval-jobs=enabled just setup
```bash
$ just setup -Dnix-eval-jobs=enabled
```
Note that only targets which don't accept extra arguments can be used when
running multiple targets at once; `just setup build` is fine, but `just
setup-custom build` is an error. The `test` target is usually the last one to
run, so it always accepts extra arguments.
Note that only targets which *don't* accept extra arguments can have other targets following them.
`just clean setup` is equivalent to `just clean && just setup`, but `just build test` runs the `build` target with the argument `test`.
This means that if you want to, for example, build with lower parallelism, and then test, you will have to do something like this:
```bash
$ just build -j4
$ just test
```
Finally, the rewrite of the integration test suite, functional2, also has its own justfile target which allows passing extra arguments to pytest.
For example, to collect and list all functional2 tests without running them, you can pass pytest's `--collect-only` argument:
```bash
$ just test-functional2 --collect-only
```
You can also build Lix manually:
@@ -141,6 +177,21 @@ To inspect the canonical source of truth on what the state of the buildsystem co
$ meson introspect
```
#### LLD
The development shell on Linux uses LLD by default for faster link times.
This is set using `mesonFlags`, so to override it, you can simplify re-specify the linker to Meson:
```bash
$ just setup -Dc_link_args=-fuse-ld=ld -Dcpp_link_args=-fuse-ld=ld
```
While using LLD, you may find it helpful to use ThinLTO for even further improvements to link times for incremental builds:
```bash
$ just setup -Db_lto=true -Db_lto_mode=thin -Db_thinlto_cache=true
```
## Sending changes to Gerrit for review {#sending-to-gerrit}
We use Gerrit for all our code review in Lix.
@@ -149,7 +200,7 @@ Our instance is at <https://gerrit.lix.systems>.
There's much more information about how to use Gerrit in the [wiki section on Gerrit][wiki-gerrit] including how to use Jujutsu, how to use the UI and more.
The Snix project also has some Gerrit information [in their contributing docs][snix-gerrit].
The gist is that once you have your SSH key and git remote set up, you can send commits for review with:
@@ -170,6 +221,14 @@ Once you get a `Code-Review+2` vote on your change, it's rebased on `main` and C
If the change appears as "Rebase Required", you need to rebase it on `main` locally or via the Gerrit UI and wait for `Verified+1` before the Submit button is made active
The `Code-Review+2` from before will stick around through trivial rebases so no need to re-request review for a mere rebase.
## Interacting with the CI, Buildkite
We use Buildkite for our CI, usually you will not have to interact directly with it other than reviewing any errors it produces, which are linked from Gerrit.
However in certain cases a CI run will fail due to transient issues not related to your code and you will need to rerun it by hand.
You can log in to the CI via [SSO](https://buildkite.com/sso/afnix). On your job you can then hit the "Retry failed" button to rerun it, normally you will not have a repeat of the transient issue.
If the build still fails on CI issues or all builds are failing this should be reported via [Zulip on #T-infra](https://zulip.lix.systems/#narrow/channel/7-T-infra) or [Matrix on #dev](https://matrix.to/#/%23dev%3Alix.systems?via=lix.systems).
## Building Lix with `nix`
To build a release version of Lix for the current operating system and CPU architecture:
@@ -507,7 +566,7 @@ Then add the new file there, and don't forget to register it in the appropriate
The following metadata properties are supported for builtin functions:
*`name` (required): the language-facing name (as a member of the `builtins` attribute set) of the function.
*`implementation` (optional): a C++ expression specifying the implementation of the builtin.
It must be a function of signature `void(EvalState &, PosIdx, Value * *, Value &)`.
It must be a function of signature `Value(EvalState &, PosIdx, Value * *)`.
If not specified, defaults to `prim_${name}`.
*`renameInGlobalScope` (optional): whether the definition should be "hidden" in the global scope by prefixing its name with two underscores.
@@ -62,6 +62,12 @@ For `installcheck` specifically, first run `just install` before running the tes
Finer-grained filtering within a test suite is also possible using the [--gtest_filter](https://google.github.io/googletest/advanced.html#running-a-subset-of-the-tests) command-line option to a test suite executable, or the `GTEST_FILTER` environment variable.
### Inspecting failures
The test suite emits logs in `build/meson-logs/`; the full textual failure logs are in `build/meson-logs/testlog.txt`.
If you want a much nicer experience of viewing the logs in a structured manner, use `xunit-viewer --results build/meson-logs/testlog.junit.xml --server` to view them in a web browser.
### Unit test support libraries
There are headers and code which are not just used to test the library in question, but also downstream libraries.
@@ -377,7 +383,10 @@ I grepped `lix/` for `get[eE]nv\("` to find the mentions in Lix code.
Overrides compile-time configuration of various locations used by Lix. See `lix/libstore/globals.cc`.
**Expected value**: a directory
- `NIX_DAEMON_SOCKET_PATH` (optional) - Overrides the daemon socket path from `$NIX_STATE_DIR/daemon-socket/socket`.
- `LIX_DAEMON_SOCKET_DIR` (optional) - Overrides the daemon socket directory from `$NIX_STATE_DIR/daemon-socket`.
**Expected value**: a directory
- `NIX_DAEMON_SOCKET_PATH` (optional) - Overrides the daemon socket path from `$NIX_STATE_DIR/daemon-socket/socket`. Ignored if `LIX_DAEMON_SOCKET_DIR` is set.
**Expected value**: path to a socket
- `NIX_LOG_FD` (output) - An FD number for logs in `internal-json` format to be sent to.
@@ -395,7 +404,6 @@ I grepped `lix/` for `get[eE]nv\("` to find the mentions in Lix code.
**Expected value**: the path to an executable shell
- `PRINT_PATH` - Undocumented. Used by `nix-prefetch-url` as an alternative form of `--print-path`. Why???
- `_NIX_IN_TEST` - If present with any value, makes `fetchClosure` accept file URLs in addition to HTTP ones. Why is this not `_NIX_FORCE_HTTP`??
Not used anywhere else.
- `NIX_ALLOW_EVAL` - Used by eval-cache tests to block evaluation if set to `0`.
Lix in Docker is very sensitive to **functional** DNS resolution if you are running with [Pasta protections](../advanced-topics/pasta.md) which are enabled by default since Lix 2.93.0 on most distributions.
If you notice failure to download things, double check whether your **first** DNS entry in `/etc/resolv.conf` is functional.
Lix with [Pasta protections](../advanced-topics/pasta.md) does not support failing over the next entries.
In Lix, only builds done on Linux with `sandbox = true` and a functioning
`pasta-path` are isolated from the rest of the system, all other builds are
considered non-isolated to some degree.
For example, running Lix with [Pasta](@docroot@/advanced-topics/pasta.md)
disabled makes the host network visible to fixed-output derivations, reducing
isolation somewhat.
## Clean termination of non-isolated builds
Non-isolated builds may not terminate cleanly in all cases due to limitations in Lix's process management.
This occurs when a build keeps the build log file descriptor open past the end of the actual build. A common cause of this are background tasks that aren't properly terminated before the main build process exits, for example: HTTP servers run as part of a test suite.
See [issue #1018](https://git.lix.systems/lix-project/lix/issues/1018) for an example.
The only solution is to manually terminate leftover processes in your derivation, including during failure scenarios.
@@ -164,7 +164,7 @@ Note that lists are only lazy in values, and they are strict in length.
An attribute set is a collection of name-value-pairs (called *attributes*) enclosed in curly brackets (`{ }`).
An attribute name can be an identifier or a [string](#type-string).
An attribute name can be an identifier or a [double-quoted string](#type-string).
An identifier must start with a letter (`a-z`, `A-Z`) or underscore (`_`), and can otherwise contain letters (`a-z`, `A-Z`), numbers (`0-9`), underscores (`_`), apostrophes (`'`), or dashes (`-`).
FIXME(Lix): This section does not document the most common modern practices in terms of avoiding channels, pinning, declarative software installation (see flakey-profile or home-manager or NixOS), or using flakes, etc.
It is, however, likely correct at a technical level.
For more information on modern practices, see the [resources](https://lix.systems/resources) page on the Lix site.
For more information on modern practices, see the [resources](https://wiki.lix.systems/books/lix-users/page/nix-resources) page on the Lix site.
</div>
@@ -41,7 +41,7 @@ install Lix. If this is not the case for some reason, you can add it
- Remove support for daemon protocols before 2.18 [fj#510](https://git.lix.systems/lix-project/lix/issues/510) [cl/3249](https://gerrit.lix.systems/c/lix/+/3249)
Support for daemon wire protocols belonging to Nix 2.17 or older have been
removed. This impacts clients connecting to the local daemon socket or any
remote builder configured using the `ssh-ng` protocol. Builders configured
with the `ssh` protocol are still accessible from clients such as Nix 2.3.
Additionally Lix will not be able to connect to an old daemon locally, and
remote build connections to old daemons is likewise limited to `ssh` urls.
We have decided to take this step because the old protocols are very badly
tested (if at all), maintenance overhead is high, and a number of problems
with their design makes it infeasible to remain backwards compatible while
we move Lix to a more modern RPC mechanism with better versioning support.
Many thanks to [eldritch horrors](https://git.lix.systems/pennae) for this.
- Remove impure derivations and dynamic derivations [fj#815](https://git.lix.systems/lix-project/lix/issues/815) [cl/3210](https://gerrit.lix.systems/c/lix/+/3210)
The `impure-derivations` and `dynamic-derivations` experimental feature have
been removed.
New impure or dynamic derivations cannot be created from this point forward, and
any such pre-existing store derivations canot be read or built any more.
Derivation outputs created by building such a derivation are still valid
until garbage collected; existing store derivations can only be garbage
collected.
Many thanks to [eldritch horrors](https://git.lix.systems/pennae) for this.
- First argument to `--arg`/`--argstr` must be a valid Nix identifier [fj#496](https://git.lix.systems/lix-project/lix/issues/496)
The first argument to `--arg`/`--argstr` must be a valid Nix identifier, i.e.
`nix-build --arg config.allowUnfree true` is now rejected.
This is because that invocation is a false friend since it doesn't set
The idea is to change the behavior to the latter in the long-term. For that,
non-identifiers started giving a warning since 2.92 and are now rejected to give people
who depend on that a chance to notice and potentially weigh in on the discussion.
Many thanks to [ma27](https://git.lix.systems/ma27) for this.
- New cgroup delegation model [fj#537](https://git.lix.systems/lix-project/lix/issues/537) [fj#77](https://git.lix.systems/lix-project/lix/issues/77) [cl/3230](https://gerrit.lix.systems/c/lix/+/3230)
Builds using cgroups (i.e. `use-cgroups = true` and the experimental feature
`cgroups`) now always delegate a cgroup tree to the sandbox.
Compared to the original C++ Nix project, our delegation includes the
`subtree_control` file as well, which means that the sandbox can disable
certain controllers in its own cgroup tree.
This is a breaking change because this requires the Nix daemon to run with an
already delegated cgroup tree by the service manager.
## How to setup the cgroup tree with systemd?
systemd offers knobs to perform the required setup using:
```
[Service]
Delegate=yes
DelegateSubtree=supervisor
```
These directives are now included in our systemd packaging.
## What about using Nix as root without connecting to the daemon?
Builds run as `root` without connecting to the daemon relying on the cgroup
feature are now broken, i.e.
```console
# nix-build --use-cgroups --sandbox ... # will not work
If you need to disable cgroups temporarily, remember that you can do
`NIX_CONF='include /etc/nix/nix.conf\nuse-cgroups = false' nix-build ...` or
`nix-build --no-use-cgroups ...`.
## What about other service managers than systemd?
systemd has a [documentation](https://systemd.io/CGROUP_DELEGATION/) on how to
handle cgroup delegation from service management perspective.
If your service manager adheres to systemd semantics, e.g. writing an extended
attribute `user.delegate=1` on the delegated cgroup tree directory and moving
the `nix-daemon` process inside a cgroup tree to respect the inner process
rule, then, the feature will work as well.
## Why is the cgroup feature still experimental?
While the cgroup feature unlocks many use cases, its behavior and integration (e.g. user experience), especially at scale on build farms or in multi-tenant environments, are not yet fully matured. There’s also potential for deeper systemd integration (e.g. using slices and scopes) that has not been fully explored.
To avoid locking in an unstable interface, we’re keeping the experimental flag until we have validated the feature across a broader range of scenarios, including but not limited to:
* Nix as root
* Hydra-style build farms
* Forgejo CI runners
* Shared remote builders
Many thanks to [Raito Bezarius](https://git.lix.systems/raito), [eldritch horrors](https://git.lix.systems/pennae), and [lheckemann](https://git.lix.systems/lheckemann) for this.
- Enable high compress ratio zstd compression by default for binary caches uploads [fj#945](https://git.lix.systems/lix-project/lix/issues/945) [cl/4503](https://gerrit.lix.systems/c/lix/+/4503)
The default compression method for binary cache uploads has been switched from
[`xz`](https://github.com/tukaani-project/xz) to
[`zstd`](https://github.com/facebook/zstd) to address performance and usability
issues related to modern hardware and high-speed connections.
## Why?
`xz` offers compression ratios but is single-threaded in our implementation and
very slow (~10-20 Mbps in our test), preventing full utilization of 100Mbps+
connections and significantly slowing decompression for end users.
Lix is a "compress once, decompress many" application: build farms can afford
to spend more time compressing to achieve a faster download transfer for the
end user. More importantly, it matters that all end users spend the least
amount of time decompressing.
## What about compression ratios?
`zstd` cannot achieve the same peaks as `xz`, nonetheless, `zstd` compression
level has been increased to level 12 by default to balance compression ratio
and performance.
## Synthetic test case data
* **xz** (default compression level) on a 4.4GB file: ~632MB (77s)
* **zstd** (level 12) on the same file: ~775MB (18s), 18% larger but 50% faster
* **zstd** (level 14): ~773MB (37s)
* **zstd** (level 16): ~735MB (66s)
Many thanks to [eldritch horrors](https://git.lix.systems/pennae) and [Raito Bezarius](https://git.lix.systems/raito) for this.
- Repl debugger uses `--ignore-try` by default [lix#666](https://git.lix.systems/lix-project/lix/issues/666) [cl/3488](https://gerrit.lix.systems/c/lix/+/3488)
Previously, using the debugger meant that exceptions thrown in `builtins.tryEval` would trigger the debugger.
However, this caught nixpkgs initialization code, which is unhelpful in the majority of cases, so we changed the default.
We consider this a breaking change since eval results *will* change if strings
with embedded NUL bytes were used, but we also consider the old behavior to be
not intentional (seeing how inconsistent it was) but merely fallout from a old
and misguided implementation decision to be worked around, not actually fixed.
Many thanks to [eldritch horrors](https://git.lix.systems/pennae) for this.
- Fixed output derivations can be run using `pasta` network isolation [fj#285](https://git.lix.systems/lix-project/lix/issues/285) [cl/3452](https://gerrit.lix.systems/c/lix/+/3452)
Fixed output derivations traditionally run in the host network namespace.
On Linux this allows such derivations to communicate with other sandboxes
or the host using the abstract Unix domains socket namespace; this hasn't
been unproblematic in the past and has been used in two distinct exploits
to break out of the sandbox. For this reason fixed output derivations can
now run in a network namespace (provided by [`pasta`]), restricted to TCP
and UDP communication with the rest of the world. When enabled this could
be a breaking change and we classify it as such, even though we don't yet
enable or require such isolation by default. We may enforce this in later
releases of Lix once we have sufficient confidence that breakage is rare.
[`pasta`]: https://passt.top/
Many thanks to [eldritch horrors](https://git.lix.systems/pennae) and [puck](https://git.lix.systems/puck) for this.
- Function equality semantics are more consistent, but still bad [cl/4556](https://gerrit.lix.systems/c/lix/+/4556) [cl/4244](https://gerrit.lix.systems/c/lix/+/4244)
Lix has inherited a historic misfeature from CppNix in the form of pointer
equality checks built into the `==` operator. These checks were originally
meant to optimize comparison for large sets, but they have the unfortunate
side effect of producing unexpected results when sets containing functions
are compared. **Lix 2.93 and earlier** behave as shown in the repl session
```
Lix 2.93.3
Type :? for help.
nix-repl> f = x: x
Added f.
nix-repl> f == f
false
nix-repl> let s.f = f; in s.f == s.f
false
nix-repl> # however!
{ inherit f; } == { inherit f; }
true
nix-repl> [ f ] == [ f ]
true
nix-repl> # and, in another twist:
[ f ] == map f [ f ]
false
```
Nixpkgs relies on sets containing functions being comparable, so we cannot
simply deprecate this behavior. Due to changes to the object model used by
Lix ***all* comparisons above now evaluate to `true`**. This is considered
a breaking change because eval results may differ, but we also consider it
minor because the optimization is unsound (c.f. `let l = [NaN]; in l == l`
evaluates to `true` even though floating point `NaN` is incomparable). Lix
intends to remove this optimization altogether in the future, but until we
can do that we instead make it slightly less broken to allow other, *real*
optimizations. Function equality comparison remains **undefined behavior**
and should not be relied upon in Nixlang code that intends to be portable.
Many thanks to [eldritch horrors](https://git.lix.systems/pennae) for this.
- `nix eval --write-to` has been removed [fj#974](https://git.lix.systems/lix-project/lix/issues/974) [fj#227](https://git.lix.systems/lix-project/lix/issues/227) [cl/4045](https://gerrit.lix.systems/c/lix/+/4045)
`nix eval --write-to` has been removed since it was underspecified, not widely
useful, and prone to security-sensitive misbehaviors. The feature was added in
Nix 2.4 purely for internal use in the build system. According to our research
it hasn't found any use outside of some distribution packaging scripts. Please
use structured outputs formats (such as JSON) instead as they have better type
fidelity, don't conflate attributes with paths, and are useful to other tools.
Many thanks to [eldritch horrors](https://git.lix.systems/pennae) for this.
- Remove the `parse-toml-timestamps` experimental feature
The `parse-toml-timestamps` experimental feature has been removed.
This feature used in‐band signalling to mark timestamps, making it
impossible to unambiguously parse TOML documents. It also exposed
implementation‐defined behaviour in the TOML specification that
changed in the toml11 parser library.
Any interface for parsing TOML timestamps suitable for future
stabilization would necessarily involve breaking changes, and there
is no evidence this experimental feature is being relied upon in the
wild, so it has been removed.
Many thanks to [Emily](https://git.lix.systems/emilazy) for this.
* Add `http3 = true` in your Nix configuration for permanent activation.
To disable it, use `--no-http3`.
**Note**:
* `--no-http2 --http3` will still enable both HTTP/2 and HTTP/3.
* `--http2 --http3` will prioritize HTTP/3 and fall back to HTTP/2 (and then
HTTP/1.1).
These are current CLI limitations. In the future, we plan to replace `--httpX`
options with `--max-http-version [1,2,3]` for easier version selection in Lix
transfers.
Many thanks to [Raito Bezarius](https://git.lix.systems/raito) and [eldritch horrors](https://git.lix.systems/pennae) for this.
- Add hyperlinks in attr set printing [cl/3790](https://gerrit.lix.systems/c/lix/+/3790)
The attribute set printer, such as is seen in `nix repl` or in type errors, now prints hyperlinks on each attribute name to its definition site if it is known.
Example: all of the attributes shown here are hyperlinks to the exact definition site of the attribute in question:
You're not alone. Thousands of Lix users suffer every day from excessive `builtins.toString` syndrome. It’s 2025, and we still have to cast integers to use them in strings.
To address this, Lix introduces the **`coerce-integers`** experimental feature. When enabled, interpolated integers within `"${...}"` are automatically coerced to strings. This allows writing:
To enable the feature, you need to add `coerce-integers` to your set of experimental features.
### Stabilization criteria
The `coerce-integers` feature is experimental and limited strictly to string interpolation (`"${...}"`). Before stabilization, the following must hold:
1. **Interpolation-only**
Coercion must not occur outside interpolation. Expressions like `"" + 42` must continue to fail.
2. **Expectation that no explicit cast are being observed**
Cases observing explicit coercion (e.g., via `tryEval` gadget or similar) are expected not to be load-bearing in actual production code.
### Timeline for stabilization
If the feature proves safe and is widely adopted across typical usage (e.g., actual configurations in the wild turning on the flag, non-trivial out-of-tree projects using it), the experimental flag will be removed **after six months of active use or two Lix releases**, whichever is longer.
This avoids locking the feature in experimental status indefinitely, as happened with Flakes, while allowing time for validation and ecosystem integration.
### What about coercing floats or more?
Coercion beyond integers -- such as for floats or other types -- is **not planned**, even under an experimental flag. Questions like "what is the canonical string representation of a float?" involve subtle and context-dependent trade-offs. Without a robust and principled mechanism to define and audit such behavior, introducing broader coercion risks setting unintended and hard-to-reverse precedents. The scope of `coerce-integers` is intentionally narrow and will remain so.
In terms of outlook, a proposal like https://git.lix.systems/lix-project/lix/issues/835 could pave the way for a better solution.
Many thanks to [Raito Bezarius](https://git.lix.systems/raito), [delroth](https://github.com/delroth), [eldritch horrors](https://git.lix.systems/pennae), and [winter](https://git.lix.systems/winter) for this.
- nix-eval-jobs: support `--no-instantiate` flag [fj#987](https://git.lix.systems/lix-project/lix/issues/987)
`nix-eval-jobs` now supports a flag called `--no-instantiate`. With this enabled,
no write operations on the eval store are performed. That means, only evaluation is
performed, but derivations (and their gcroots) aren't created.
Many thanks to [mic92](https://github.com/mic92) and [ma27](https://git.lix.systems/ma27) for this.
## Improvements
- Assess current profile generations pointers in `nix doctor` [cl/3108](https://gerrit.lix.systems/c/lix/+/3108)
Added a new check to `nix doctor` that verifies whether the current generation of
a Nix profile can be resolved. This helps users diagnose issues with broken or
misconfigured profile symlinks.
This helps determining if you have broken symlinks or misconfigured packaging.
Many thanks to [Raito Bezarius](https://git.lix.systems/raito) for this.
- Improved susbtituter query speed
The code used to query substituters for derivations has been rewritten slightly
to take advantage of our asynchronous runtime. Such queries run for every build
that could download from substituters and processes every derivation that isn't
yet present on the local system. Previously Lix would use `http-connections` to
limit query concurrency, even for modern caches that support HTTP/2 and have no
limit on how many queries can be run concurrently on one single connection. Lix
no longer does this, resulting in approximately 60% reduction in query time for
medium-sized closures (e.g. NixOS system closures) during testing, although the
exact number depends greatly on local network latency and generally improves as
latency increases. Unlike previously setting `http-connections` to `1` or other
low values no longer brings a massive penalty in query performance if the cache
in use by the querying system supports HTTP/2 (as e.g. `cache.nixos.org` does).
Many thanks to [eldritch horrors](https://git.lix.systems/pennae) for this.
Hitting Control-C or sending `SIGINT` to Lix now prints an informational message
if it is still running after on second, the second Control-C/`SIGINT` terminates
Lix immediately without waiting for any shutdown code to finish running. Lix did
not treat the second such event differently from first in the past; this made it
impossible to easily terminate running Lix processes that got stuck in e.g. very
expensive Nixlang code that never interacted with the store. We now terminate as
soon as the user hits Control-C again without waiting any more, to much the same
effect as putting Lix into the background and killing it immediately afterwards.
This means you can now more conveniently break out of stuck Nixlang evaluations:
```
❯ nix-instantiate --eval --expr 'let f = n: if n == 0 then 0 else f (n - 1) + f (n - 1); in f 32'
^CStill shutting down. Press ^C again to abort all operations immediately.
^C
❌130 ❯
```
Many thanks to [eldritch horrors](https://git.lix.systems/pennae) for this.
- `--keep-failed` chowns the build directory to the user that request the build
Running a build with `--keep-failed` now chowns the temporary directory from the
builder user and group to the user that request the build if the build came from
a local user connected to the daemon. This makes inspecting failed derivations a
lot easier. On Linux the build directory made visible to the user will not be in
the same path as it was in the sandbox and continuing builds will usually break.
Many thanks to [eldritch horrors](https://git.lix.systems/pennae) for this.
- Better debuggability on fixed-output hash mismatches
Fixed-output derivation hash mismatch error messages will now include the path that was
produced unexpectedly, and this path will be registered as valid even if `--check`
(`nix-store`, `nix-build`) or `--rebuild` (`nix build`) was passed. This makes comparing
the expected path with the obtained path easier, and is useful for debugging when
upstreams modify previously-published releases or when changes in fixed-output
derivations' dependencies affect their output unexpectedly.
Many thanks to [lheckemann](https://git.lix.systems/lheckemann) for this.
- Add --raw flag to `nix-instantiate --eval` for unescaped output [gh#12119](https://github.com/NixOS/nix/pull/12119) [cl/2886](https://gerrit.lix.systems/c/lix/+/2886)
The `nix-instantiate --eval` command now supports a `--raw` flag. When used,
the result must be coercible to a string (as with `${...}`) and is printed
verbatim, without quotes or escaping.
Many thanks to [Martin Fischer](https://github.com/not-my-profile), [infinisil](https://github.com/infinisil), and [Raito Bezarius](https://git.lix.systems/raito) for this.
- Allow `nix store ls` to read nar listings from binary cache stores. [cl/3225](https://gerrit.lix.systems/c/lix/+/3225)
The `nix store ls` command now supports reading `.ls` nar listings from binary cache stores.
If a listing is detected for the store path being queried, the nar is no longer downloaded.
These nar listings are available in binary cache stores where the `write-nar-listing` option is
enabled, such as cache.nixos.org.
Many thanks to [Victor Fuentes](https://git.lix.systems/vlinkz) for this.
- show tree with references that lead to an output cycle [fj#551](https://git.lix.systems/lix-project/lix/issues/551)
When Lix determines a cyclic dependency between several outputs of a derivation,
it now displays which files in which outputs lead to an output cycle:
```
error: cycle detected in build of '/nix/store/gc5h2whz3rylpf34n99nswvqgkjkigmy-demo.drv' in the references of output 'bar' from output 'foo'.
Shown below are the files inside the outputs leading to the cycle:
Please note that showing the files and its contents while displaying the cycles only works
on Linux.
Many thanks to [ma27](https://git.lix.systems/ma27) for this.
- Lix now enables parallel marking in boehm-gc [fj#983](https://git.lix.systems/lix-project/lix/issues/983) [cl/3880](https://gerrit.lix.systems/c/lix/+/3880)
This brings a fairly modest performance improvement (~38% for `nixpkgs search hello`) to evaluation, especially in scenarios that necessitate larger heap sizes.
Many thanks to [Eelco Dolstra](https://github.com/edolstra) and [Seth Flynn](https://git.lix.systems/getchoo) for this.
- `disallowedRequisites` now reports chains of disallowed requisites [fj#334](https://git.lix.systems/lix-project/lix/issues/334) [fj#626](https://git.lix.systems/lix-project/lix/issues/626) [gh#10877](https://github.com/NixOS/nix/issues/10877)
When a build fails because of [`disallowedRequisites`](@docroot@/language/advanced-attributes.md#adv-attr-disallowedRequisites), the error message now includes the chain of references that led to the failure. This makes it easier to see in which derivations the chain can be broken, to resolve the problem.
Example:
```
$ nix-build -A hello
error: output '/nix/store/0b7k85gg5r28gb54px9nq7iv5986mns9-hello-2.12.2' is not allowed to refer to the following paths:
Many thanks to [ma27](https://git.lix.systems/ma27) and [Robert Hensing](https://github.com/roberth) for this.
- Stack traces now summarize involved derivations at the bottom [cl/4493](https://gerrit.lix.systems/c/lix/+/4493)
When evaluation errors and a stack trace is printed,
For example, if I add Nheko to a NixOS `environment.systemPackages` without adding `olm-3.2.16` `nixpkgs.config.permittedInsecurePackages`, then without `--show-trace`, I previously got this:
```
error:
… while calling the 'head' builtin
at /nix/store/9v6qa656sq3xc58vkxslqy646p0ajj61-source/lib/attrsets.nix:1701:13:
1700| if length values == 1 || pred here (elemAt values 1) (head values) then
1701| head values
| ^
1702| else
… while evaluating the attribute 'value'
at /nix/store/9v6qa656sq3xc58vkxslqy646p0ajj61-source/lib/modules.nix:1118:7:
1117| // {
1118| value = addErrorContext "while evaluating the option `${showOption loc}':" value;
| ^
1119| inherit (res.defsFinal') highestPrio;
(stack trace truncated; use '--show-trace' to show the full trace)
error: Package ‘olm-3.2.16’ in /nix/store/9v6qa656sq3xc58vkxslqy646p0ajj61-source/pkgs/by-name/ol/olm/package.nix:37 is marked as insecure, refusing to evaluate.
< -snip the whole explanation about olm's CVEs- >
```
This doesn't tell me anything about where `olm-3.2.16` came from.
With `--show-trace`, there's 1155 lines to sift through, but does contain lines like "while evaluating derivation 'nheko-0.12.1'".
With this change, those lines are summarized and collected at the bottom, regardless of `--show-trace`:
```
error:
… while calling the 'head' builtin
at /nix/store/9v6qa656sq3xc58vkxslqy646p0ajj61-source/lib/attrsets.nix:1701:13:
1700| if length values == 1 || pred here (elemAt values 1) (head values) then
1701| head values
| ^
1702| else
… while evaluating the attribute 'value'
at /nix/store/9v6qa656sq3xc58vkxslqy646p0ajj61-source/lib/modules.nix:1118:7:
1117| // {
1118| value = addErrorContext "while evaluating the option `${showOption loc}':" value;
| ^
1119| inherit (res.defsFinal') highestPrio;
(stack trace truncated; use '--show-trace' to show the full trace)
error: Package ‘olm-3.2.16’ in /nix/store/9v6qa656sq3xc58vkxslqy646p0ajj61-source/pkgs/by-name/ol/olm/package.nix:37 is marked as insecure, refusing to evaluate.
< -snip the whole explanation about olm's CVEs- >
note: trace involved the following derivations:
derivation 'etc'
derivation 'dbus-1'
derivation 'system-path'
derivation 'nheko-0.12.1'
derivation 'mtxclient-0.10.1'
```
Now we finally know that olm was evaluated because of Nheko, without sifting through *thousands* of lines of error message.
Many thanks to [Qyriad](https://git.lix.systems/Qyriad) for this.
- Symbols reuses once-allocated Value to reduce garbage collected allocations [cl/3308](https://gerrit.lix.systems/c/lix/+/3308) [cl/3300](https://gerrit.lix.systems/c/lix/+/3300) [cl/3314](https://gerrit.lix.systems/c/lix/+/3314) [cl/3310](https://gerrit.lix.systems/c/lix/+/3310) [cl/3312](https://gerrit.lix.systems/c/lix/+/3312) [cl/3313](https://gerrit.lix.systems/c/lix/+/3313)
In the Lix evaluator, **symbols** represent immutable strings, like those used
for attribute names.
In evaluator design, such strings are typically [**interned**](https://en.wikipedia.org/wiki/String_interning), stored uniquely
to save memory, and Lix inherits this approach from the original C++ codebase.
However, some builtins, like `builtins.attrNames`, must return a `Value` type
that can represent any Nix value (strings, integers, lists, etc.).
Before this change, these builtins would create lists of `Value` objects by
allocating them through the garbage collector, copying the symbol’s string
content each time.
This allocation is unnecessary if the interned symbols themselves also hold a
`Value` representation allocated outside the garbage collector, since these
live for the full duration of evaluation.
As a result, this reduces the number of allocations, leading to:
* A significant drop in maximum [resident set memory](https://en.wikipedia.org/wiki/Resident_set_size) (RSS), with some large-scale
tests showing up to 11% (about 500 MiB) savings in large colmena deployments.
* A slight decrease in CPU usage during Nix evaluations.
This change is inspired by https://github.com/NixOS/nix/pull/13258 but the approach is different.
**Note** : [`xokdvium`](https://github.com/xokdvium) is the rightful author of https://gerrit.lix.systems/c/lix/+/3300 and the credit was missed on our end during the development process. We are deeply sorry for this mistake.
Many thanks to [Raito Bezarius](https://git.lix.systems/raito), [eldritch horrors](https://git.lix.systems/pennae), [Tom Hubrecht](https://git.lix.systems/tom-hubrecht), [xokdvium](https://github.com/xokdvium), and [NaN-git](https://github.com/NaN-git) for this.
## Fixes
- `build-dir` no longer defaults to `temp-dir` [cl/3453](https://gerrit.lix.systems/c/lix/+/3453)
The directory in which temporary build directories are created no longer defaults
to the value of the `temp-dir` setting to avoid builders making their directories
world-accessible. This behavior has been used to escape the build sandbox and can
cause build impurities even when not used maliciously. We now default to `builds`
in `NIX_STATE_DIR` (which is `/nix/var/nix/b` in the default configuration).
Many thanks to [eldritch horrors](https://git.lix.systems/pennae) for this.
- Global certificate authorities are copied inside the builder's environment [gh#12698](https://github.com/NixOS/nix/issues/12698) [fj#885](https://git.lix.systems/lix-project/lix/issues/885) [cl/3765](https://gerrit.lix.systems/c/lix/+/3765)
Previously, CA certificates were only installed at
`/etc/ssl/certs/ca-certificates.crt` for sandboxed builds on Linux.
This setup was insufficient in light of recent changes in `nixpkgs`, which now
enforce HTTPS usage for `fetchurl`, even for fixed-output derivations, to
mitigate confidentiality risks such as `netrc` or credentials leakage.
`nixpkgs` still make use of a special package called `cacerts` which contains a
copy of the CA certificates maintained by Nixpkgs and added as a reference for
TLS-enabled fetchers.
As a result, having a consistent and trusted certificate authority in all
builder environments is becoming more essential.
On `nix-darwin`, the `NIX_SSL_CERT_FILE` environment variable is always
explicitly defined, but it is ignored by the sandbox setup.
Simultaneously, Nix evaluates and propagates impure environment variables via
`lib.proxyImpureEnvVars`, meaning that if `NIX_SSL_CERT_FILE` is set (which
influences the default value for `ssl-cert-file`), it will be forwarded
unchanged into the builder environment.
However, on Linux, Nix also *copies* the CA file into the sandbox, creating a
discrepancy between the value of `NIX_SSL_CERT_FILE` and the actual trusted
certificate path used during the build.
This divergence caused confusion and was partially addressed by attempts to
whitelist the CA path in the Darwin sandbox (see cl/2906), but that approach
involved a non-trivial path canonicalization step and is not as general as this one.
To address this properly, we now emit a warning and override
`NIX_SSL_CERT_FILE` inside the builder, explicitly pointing it to the CA file
copied into the sandbox.
This eliminates ambiguity between `NIX_SSL_CERT_FILE`
and `ssl-cert-file`, ensuring consistent trust anchors across platforms.
This warning might become a hard error as we figure out what to do regarding
`lib.proxyImpureEnvVars` in nixpkgs.
The behavior has been verified across sandboxed and unsandboxed builds on both
Linux and Darwin.
As a consequence of this change, approximately 500KB of CA certificate data is
now unconditionally copied into the build directory for fixed-output
derivations.
While this ensures consistent trust verification without having to restart the
daemon after system upgrades, it may introduce a slight overhead in build
performance. At present, no optimizations have been implemented to avoid this
copy, but if this overhead proves noticeable in your workflows, please open an
issue so we can evaluate and possibly implement different strategies to render
trust anchors visible.
Many thanks to [Raito Bezarius](https://git.lix.systems/raito) and [Emily](https://git.lix.systems/emilazy) for this.
- libstore: exponential backoff for downloads [lix#932](https://git.lix.systems/lix-project/lix/issues/932) [cl/3856](https://gerrit.lix.systems/c/lix/+/3856)
The connection timeout when downloading from e.g. a binary cache is exponentially
increased per failure. The option `connect-timeout` is now an alias to `max-connect-timeout`
which is the maximum value for a timeout. The start value is controlled
by `initial-connect-timeout` which is `5` by default.
Many thanks to [ma27](https://git.lix.systems/ma27) for this.
- Fix develop shells for derivations with escape codes [fj#991](https://git.lix.systems/lix-project/lix/issues/991) [cl/4154](https://gerrit.lix.systems/c/lix/+/4154) [cl/4155](https://gerrit.lix.systems/c/lix/+/4155)
ASCII control characters (including `\e`, used for ANSI escape codes) in derivation variables are now correctly escaped for `nix develop` and `nix print-dev-env`, instead of erroring.
Many thanks to [Qyriad](https://git.lix.systems/Qyriad) for this.
To find out why, use nix-store --query --roots and nix-store --query --referrers.
```
Many thanks to [lheckemann](https://git.lix.systems/lheckemann) for this.
- Fallback to safe temp dir when build-dir is unwritable [fj#876](https://git.lix.systems/lix-project/lix/issues/876) [cl/3501](https://gerrit.lix.systems/c/lix/+/3501)
Non-daemon builds started failing with a permission error after introducing the `build-dir` option:
1. These builds are not run via the daemon, which owns `/nix/var/nix/builds`.
2. The user lacks permissions for that path.
We considered making `build-dir` a store-level option and defaulting it to `<chroot-root>/nix/var/nix/builds` for chroot stores, but opted instead for a fallback: if the default fails, Nix now creates a safe build directory under `/tmp`.
To avoid CVE-2025-52991, the fallback uses an extra path component between `/tmp` and the build dir.
**Note**: this fallback clutters `/tmp` with build directories that are not cleaned up. To prevent this, explicitly set `build-dir` to a path managed by Lix, even for local workloads.
Many thanks to [Raito Bezarius](https://git.lix.systems/raito) and [eldritch horrors](https://git.lix.systems/pennae) for this.
- Parse overflowing JSON number literals as floating‐point [cl/3919](https://gerrit.lix.systems/c/lix/+/3919)
Previously, `builtins.fromJSON "-9223372036854775809"` would
return a floating‐point number, while `builtins.fromJSON
"9223372036854775808"` would cause an evaluation error. This was
introduced with the banning of integer overflow in Lix 2.91; previously
the latter would result in C++ undefined behaviour. These cases are
now treated consistently with JSON’s model of a single numeric type,
and JSON number literals that do not fit in a Nix‐language integer
will be parsed as floating‐point numbers.
Many thanks to [Emily](https://git.lix.systems/emilazy) for this.
- Fix handling of OSC codes in terminal output [fj#160](https://git.lix.systems/lix-project/lix/issues/160) [cl/3143](https://gerrit.lix.systems/c/lix/+/3143)
OSC codes in terminal output are now handled correctly, where OSC 8 (hyperlink) is preserved any
time color codes are allowed and all other OSC codes are stripped out. This applies not only to
output from build commands but also to rendered documentation in the REPL.
Many thanks to [lilyball](https://git.lix.systems/lilyball) for this.
- Fix nix develop for derivations that rejects dependencies with structured attrs [fj#997](https://git.lix.systems/lix-project/lix/issues/997) [cl/4182](https://gerrit.lix.systems/c/lix/+/4182)
For the sake of concision, we refer to `disallowedReferences` in what follows,
but all output checks were equally fixed:
`{dis,}allowed{References,Requisites}`.
Derivations can define *output checks* to reject unwanted dependencies, such as
interpreters like `bash` or compilers like `gcc`. This can be done in two ways:
* **Legacy style**: `disallowedReferences = [ ... ]` in the environment.
`nix-eval-jobs` doesn't clear the `NIX_PATH` from the environment anymore. This matches the behavior
of [upstream version `2.30`](https://github.com/nix-community/nix-eval-jobs/releases/tag/v2.30.0).
Many thanks to [ma27](https://git.lix.systems/ma27) and [mic92](https://github.com/mic92) for this.
- Remove reliance on Bash for remote stores via SSH [fj#830](https://git.lix.systems/lix-project/lix/issues/830) [fj#805](https://git.lix.systems/lix-project/lix/issues/805) [fj#304](https://git.lix.systems/lix-project/lix/issues/304) [cl/3159](https://gerrit.lix.systems/c/lix/+/3159)
The pre-flight `echo started` handshake -- added years ago to catch race conditions -- has been removed.
After removal of connection sharing in Lix 2.93, it required a Bash-compatible shell and a standard `echo`, so it failed on:
* builders protected by `ForceCommand` wrappers (e.g. `nix-remote-build`),
* BusyBox / initrd images with no Bash,
* hosts using non-POSIX shells such as Nushell.
The race the probe once addressed was tied to SSH connection-sharing -- since connection-sharing code has already been removed, the probe is now pointless.
Real connection or protocol errors are now left to SSH/Nix to report directly.
This is technically a breaking change if you had scripts that relied on the literal "started" which needs to be updated to rely on other signals, e.g., exit codes.
Many thanks to [Raito Bezarius](https://git.lix.systems/raito) for this.
- repl-overlays now work in the debugger for flakes [fj#777](https://git.lix.systems/lix-project/lix/issues/777) [cl/3398](https://gerrit.lix.systems/c/lix/+/3398)
Due to a bug, it was previously not possible to use the debugger on flakes with repl-overlays, or with pure evaluation in general:
```
$ nix repl --pure-eval
Lix 2.94.0-dev-pre20250617-87d99da
Type :? for help.
Loading 'repl-overlays'...
error: access to absolute path '/Users/jade/.config/nix/repl.nix' is forbidden in pure eval mode (use '--impure' to override)
```
This is now fixed.
The contents of the repl-overlays file itself (i.e. most typically the top level lambda in it) will be evaluated in impure mode.
It may be necessary to use `builtins.seq` to force the impure operations to happen first if one wants to do impure operations inside a repl-overlays file in pure evaluation mode.
Many thanks to [jade](https://git.lix.systems/jade) for this.
- `nix-shell` default shell directory is not `/tmp` anymore for `$NIX_BUILD_TOP` [fj#940](https://git.lix.systems/lix-project/lix/issues/940)
Previously, Lix `nix-shell`s could exit non-zero status when `stdenv`'s `dumpVars` phase failed to write to `$NIX_BUILD_TOP/env-vars`, despite `dumpVars` being intended as a debugging aid.
This happens when `TMPDIR` is not set and defaults therefore to `/tmp`, resulting in a `/tmp/env-vars` global file that every `nix-shell` wants to write.
We fix this issue by reusing a pre-created, unique, and writable location, as the build top directory, avoiding shell exiting from write failures silently.
Many thanks to [Raito Bezarius](https://git.lix.systems/raito) for this.
When using e.g. [Snix's nar-bridge](https://snix.dev/docs/components/overview/#nar-bridge) via
an `http`-store, Lix would create cache entries with a wrong URL to the NAR when uploading
a store-path.
This caused hard build failures for Hydra.
Lix doesn't create these entries on upload anymore. Instead, it only removes negative cache entries.
The cache entry for a narinfo is now created the first time, Lix queries the cache
for the previously uploaded store-path again.
Many thanks to [ma27](https://git.lix.systems/ma27) for this.
- Lix libraries can now be linked statically [fj#789](https://git.lix.systems/lix-project/lix/issues/789) [cl/3775](https://gerrit.lix.systems/c/lix/+/3775) [cl/3778](https://gerrit.lix.systems/c/lix/+/3778)
Previously the pkg-config files distributed with Lix were only suitable for dynamic linkage, causing "undefined reference to…" linker errors when trying to link statically.
Private dependency information has now been added to make static linkage work as expected without user intervention.
In addition, relevant static libraries are now prelinked to avoid strange failures due to missing static initializers.
Many thanks to [alois31](https://git.lix.systems/alois31) for this.
- add description to zsh completions [fj#910](https://git.lix.systems/lix-project/lix/issues/910) [cl/3632](https://gerrit.lix.systems/c/lix/+/3632)
Emit descriptions when completing args in zsh completions. This uses the descriptions we already
provided in NIX\_GET\_COMPLETIONS.
Many thanks to [matthewbauer](https://github.com/matthewbauer) for this.
## Miscellany
- Deprecation of CA derivations, dynamic derivations, and impure derivations [fj#815](https://git.lix.systems/lix-project/lix/issues/815)
Content-addressed derivations are now deprecated and slated for removal in Lix 2.94.
We're doing this because the CA derivation system has been a known cause of problems
and inconsistencies, is unmaintained, habitually makes improving the store code very
difficult (or blocks such improvements outright), and is beset by a number of design
flaws that in our opinion cannot be fixed without a full reimplementation from zero.
Dynamic derivations and impure derivations are built on the CA derivation framework,
and owing to this they too are deprecated and slated for removal in another release.
- Deprecate shadowing internal files through the Nix search path [lix#998](https://git.lix.systems/lix-project/lix/issues/998) [cl/4632](https://gerrit.lix.systems/c/lix/+/4632)
As Lix uses the path `<nix/fetchurl.nix>` for bootstrapping purposes, the ability to shadow it by adding `nix=/some/path` (or `/other/path` that contains a `nix` directory) to the search path is not desirable.
To alleviate potential issues, Lix now emits a warning when the Nix search path contains potential shadows for internal files, which will be changed to an error in a future release.
The warning can be disabled by enabling the deprecated feature `nix-path-shadow`.
Many thanks to [Tom Hubrecht](https://git.lix.systems/tom-hubrecht) for this.
- More deprecated features [cl/2092](https://gerrit.lix.systems/c/lix/+/2092) [cl/2310](https://gerrit.lix.systems/c/lix/+/2310) [cl/2311](https://gerrit.lix.systems/c/lix/+/2311) [cl/4638](https://gerrit.lix.systems/c/lix/+/4638) [cl/4652](https://gerrit.lix.systems/c/lix/+/4652) [cl/4764](https://gerrit.lix.systems/c/lix/+/4764)
This release cycle features a new batch of deprecated (anti-)features.
You can opt in into the old behavior with `--extra-deprecated-features` or any equivalent configuration option.
-`broken-string-indentation` indented strings (those starting with `''`) might produce unintended results due to how the whitespace stripping is done. Those cases will now warn the user.
-`broken-string-escape` "escaped" characters without a properly defined escape sequence evaluate to "themselves". This is in most cases unintended behaviour, both for writing regexes, and using legacy or uncommon escape sequences like `\f`. The user will now be warned, if those are present.
-`floating-without-zero` so far, one was able to declare a float using something like `.123`. This can cause confusion about accessing attributes. Floating point numbers must now always include the leading zero, i.e. `0.123`
-`rec-set-merges` Attribute sets like `{ foo = {}; foo.bar = 42;}` implicitly merge at parse time, however if one of them is marked as recursive but not the others then the recursive attribute may get lost (order-dependent). Therefore, merging attrs with mixed-`rec` is now forbidden.
-`rec-set-dynamic-attrs` Dynamic attributes have weird semantics in the presence of recursive attrsets (they evaluate *after* the rest of the set). This is now forbidden.
-`or-as-identifier``or` as an identifier has always been weird since the `or` (almost-)keyword has been introduced. We are deprecating the backcompat hacks from the early days of Nix in favor of making `or` a full and proper keyword.
-`tokens-no-whitespace` Function applications without space around the arguments like `0a`, `0.00.0` or `foo"1"2` are now forbidden. The same applies to list elements. The primary reason for this deprecation is to remove foot guns around surprising tokenization rules regarding number literals, but this will also free up some syntax for other purposes (e.g. `r""` strings) for reuse at some point in the future.
-`shadow-internal-symbols` has been expanded to also forbid shadowing `null`, `true` and `false`.
-`ancient-let` deprecation has been turned into a full parser error instead of a warning.
-`rec-set-overrides` deprecation has been turned into a full parser error instead of a warning.
Many thanks to [piegames](https://git.lix.systems/piegames), [rootile (Rutile)](https://git.lix.systems/rootile), and [eldritch horrors](https://git.lix.systems/pennae) for this.
- Move `/root/.cache/nix` to `/var/cache/nix` by default [lix#634](https://git.lix.systems/lix-project/lix/issues/634) [cl/4671](https://gerrit.lix.systems/c/lix/+/4671)
By default, Lix attempts to locate a cache directory for its operations (such
as the narinfo cache) by checking the value of `$XDG_CACHE_DIR`.
However, since the Nix daemon is a system service, using `$XDG_CACHE_DIR` is
not typical in this context.
To address this, systemd provides a better solution. Specifically, when
`CacheDirectory=` is set in the `[Service]` section of a systemd unit, it
automatically sets the `$CACHE_DIRECTORY` environment variable and systemd will
manage that cache directory for us.
Now, our systemd unit includes `CacheDirectory=nix`, which sets the
`$CACHE_DIRECTORY` and takes precedence over `$XDG_CACHE_DIR`.
If the daemon is run under user units, systemd will automatically set
`$XDG_CACHE_DIR`.
If neither of these variables is set, Lix falls back to its default behavior.
By default, Lix will try to find a cache directory for its various operations
(e.g. narinfo cache) by looking into `$XDG_CACHE_DIR`.
In summary, what was stored in `/root/.cache/nix` is now moved to
`/var/cache/nix/nix`.
Many thanks to [Raito Bezarius](https://git.lix.systems/raito) for this.
The `fetch-closure` experimental feature has been removed.
Outside of allowing the user to import closure from binary cache,
`fetchClosure` also allowed you to do the following:
* rewrite non-CA path to CA
* reject non-CA paths at fetching time
* reject CA paths at fetching time
Some people are using those mechanism to prevent users from having to build any
package and force going via the declared cache or as a way to use ancient/old
software without paying the evaluation cost of a second nixpkgs.
Both use cases are somewhat of an antipattern in Nix semantics. If the user
cannot fetch a program directly via the substituter mechanism and fall back to
local build, this is a feature *and* a misconfiguration. If the user cannot build
certain derivations because they are too expensive, the build directives should
pass `-j0` or similar.
As for the second usecase, there's a different way to do it that also allows to
have a way to reproduce the paths that are hardcoded in that file, perform
`import (fetchurl "https://my-cache/${hashparts storepath}.drv")` rather, i.e.
an IFD to a possibly well known name. The backend can generate them on the fly
or once, and possess stable names.
Finally, as for the non-CA → CA features, Lix removed ca-derivations.
fetchClosure offers ca-derivations-like features which suffers from similar
shortcomings albeit lessened. It only follows that we should deprecate
and remove these capabilities.
Many thanks to [just1602](https://git.lix.systems/just1602) for this.
## Features
-`nix store add-path` now supports references [cl/5205](https://gerrit.lix.systems/c/lix/+/5205)
Lix supports two categories of hashes in store paths: input-addressed and output-addressed.
Currently, in Nix language, there is no way to produce output-addressed paths with references, as fixed-output derivations forbid references.
However, the Nix store actually *supports* references in output-addressed paths.
This is very useful for importing build products created outside of Lix that reference dependency store paths since such build products have no associated derivation so don't make any sense to input-address.
Previously, output-addressed paths with references could only be created by writing a custom client to the rather-baroque Nix daemon protocol; now it's available in the CLI.
Using `nix store add-path --references-list-json REFS_LIST_FILE SOME_PATH` with a JSON list of string store paths, you can now create such paths with the Lix CLI.
They may be consumed from Nix language using something like `builtins.storePath` or the following which also works in pure evaluation mode:
```nix
# Hack from https://git.lix.systems/lix-project/lix/issues/402#issuecomment-5889
path:
builtins.appendContext path {
${path} = {
path = true;
};
}
```
Many thanks to [jade](https://git.lix.systems/jade) for this.
- Add `builtins.warn` for emitting warnings from Nix code [cl/2248](https://gerrit.lix.systems/c/lix/+/2248)
Lix now has a builtin function for emitting warnings.
Like `builtins.trace`, it takes two arguments: the message to emit, and the expression to return.
_Unlike_ `builtins.trace`, `builtins.warn` requires the first argument — the message — to be a string.
In the future we may extend `builtins.warn` to accept a more structured API.
To go along with this, we also have two new config settings:
- [`debugger-on-warn`](@docroot@/command-ref/conf-file.md#conf-debugger-on-warn), which, when used with `--debugger`, makes `builtins.warn` also function like [`builtins.break`](@docroot@/language/builtins.md#builtins-break).
- [`abort-on-warn`](@docroot@/command-ref/conf-file.md#conf-abort-on-warn), which aborts evaluation entirely after the warning is emitted.
Many thanks to [Emilia Bopp](https://git.lix.systems/milibopp) and [Qyriad](https://git.lix.systems/Qyriad) for this.
- `keep-env-derivations` is now supported for nix3 CLI (`nix profile`) [lix#1095](https://git.lix.systems/lix-project/lix/issues/1095) [cl/5332](https://gerrit.lix.systems/c/lix/+/5332)
The `keep-env-derivations` feature is now available for `nix profile`. This allows users to prevent the garbage collection of derivations used to install a profile, even when `keep-derivations = false` (set to `true` by default).
Previously, `nix-env` supported this feature, but `nix profile` **never** did. This caused issues when garbage collection removed the associated `.drv` files, which are required, for example, by vulnerability management tools (e.g. [vulnix](https://github.com/nix-community/vulnix)) for proper operation.
This issue has now been resolved.
Many thanks to [Raito Bezarius](https://git.lix.systems/raito) for this.
- Make `log-format` a setting [cl/4686](https://gerrit.lix.systems/c/lix/+/4686)
The [`--log-format` CLI option](@docroot@/command-ref/opt-common.md#opt-log-format) can now be set in [`nix.conf`](@docroot@/command-ref/conf-file.md#conf-log-format)!
For example, you can now persistently enable the `multiline-with-logs` log format [added in Lix 2.91](@docroot@/release-notes/rl-2.91.md) by adding the following to your `nix.conf`:
```conf
log-format = multiline-with-logs
```
Or the equivalent in a NixOS configuration:
```nix
{
nix.settings.log-format = "multiline-with-logs";
}
```
Many thanks to [Qyriad](https://git.lix.systems/Qyriad) for this.
- Allow remote builders to be configured using TOML [cl/4533](https://gerrit.lix.systems/c/lix/+/4533)
Lix now supports configuring remote builders using a TOML file instead of the old, very cursed and incomprehensible format.
This comes with not only a human-understandable file, but also with better messages and error reports on misconfiguration.
A more detailed Documentation can be found on the [distributed-builds](@docroot@/advanced-topics/distributed-builds.md) documentation page.
Many thanks to [rootile (Rutile)](https://git.lix.systems/rootile) and [Qyriad](https://git.lix.systems/Qyriad) for this.
- Emit warnings when encountering IFD with `warn-import-from-derivation` [nix#13279](https://github.com/NixOS/nix/pull/13279) [cl/3879](https://gerrit.lix.systems/c/lix/+/3879)
Instead of only being able to toggle the use of [Import from
Derivation](https://nix.dev/manual/nix/stable/language/import-from-derivation) with
`allow-import-from-derivation`, Lix is now able to warn users whenever IFD is encountered with
`warn-import-from-derivation`.
Many thanks to [Seth Flynn](https://git.lix.systems/getchoo), [gustavderdrache](https://github.com/gustavderdrache), and [Eelco Dolstra](https://github.com/edolstra) for this.
## Improvements
- Collect Flakes untrusted settings into one prompt [lix#682](https://git.lix.systems/lix-project/lix/issues/682) [cl/2921](https://gerrit.lix.systems/c/lix/+/2921)
When working with Flakes containing untrusted settings, a prompt is shown for each setting, asking whether to vet or approve it. This looks like:
```
❯ nix flake lock
warning: ignoring untrusted flake configuration setting 'allow-dirty', pass '--accept-flake-config' to trust it (may allow the flake to gain root, see the nix.conf manual page)
warning: ignoring untrusted flake configuration setting 'sandbox', pass '--accept-flake-config' to trust it (may allow the flake to gain root, see the nix.conf manual page)
The following settings require your decision:
- allow-dirty = false
- sandbox = false
Do you want to allow configuration settings to be applied?
This may allow the flake to gain root, see the nix.conf manual page (yes for now/Allow always/no/No to all)
```
In Flakes with a large number of settings to approve or reject, this process can become tedious as each option must be handled individually.
To address this, all untrusted settings are now consolidated into a single prompt: allowing for bulk acceptance permanently or not, rejection, or detailed review. For example:
### Scrutiny scenario
```console
❯ nix flake lock
warning: ignoring untrusted flake configuration setting 'allow-dirty', pass '--accept-flake-config' to trust it (may allow the flake to gain root, see the nix.conf manual page)
warning: ignoring untrusted flake configuration setting 'sandbox', pass '--accept-flake-config' to trust it (may allow the flake to gain root, see the nix.conf manual page)
The following settings require your decision:
- allow-dirty = false
- sandbox = false
Do you want to allow configuration settings to be applied?
This may allow the flake to gain root, see the nix.conf manual page (yes for now/Allow always/no/No to all) n
warning: you can set 'accept-flake-config' to 'false' to automatically reject configuration options supplied by flakes
Do you want to allow setting 'allow-dirty = false'? (yes for now/Allow always/no for now) y
Do you want to allow setting 'sandbox = false'? (yes for now/Allow always/no for now) n
```
### Reject everything scenario
```console
❯ nix flake lock
warning: ignoring untrusted flake configuration setting 'allow-dirty', pass '--accept-flake-config' to trust it (may allow the flake to gain root, see the nix.conf manual page)
warning: ignoring untrusted flake configuration setting 'sandbox', pass '--accept-flake-config' to trust it (may allow the flake to gain root, see the nix.conf manual page)
The following settings require your decision:
- allow-dirty = false
- sandbox = false
Do you want to allow configuration settings to be applied?
This may allow the flake to gain root, see the nix.conf manual page (yes for now/Allow always/no/No to all) N
Rejecting all untrusted nix.conf entries
warning: you can set 'accept-flake-config' to 'false' to automatically reject configuration options supplied by flakes
```
### Accept everything scenario
```console
❯ nix flake lock
warning: ignoring untrusted flake configuration setting 'allow-dirty', pass '--accept-flake-config' to trust it (may allow the flake to gain root, see the nix.conf manual page)
warning: ignoring untrusted flake configuration setting 'sandbox', pass '--accept-flake-config' to trust it (may allow the flake to gain root, see the nix.conf manual page)
The following settings require your decision:
- allow-dirty = false
- sandbox = false
Do you want to allow configuration settings to be applied?
This may allow the flake to gain root, see the nix.conf manual page (yes for now/Allow always/no/No to all) y
```
### Accept everything PERMANENTLY scenario
Note that accepting everything permanently will authorize these options for any
further operations.
The file containing this trust information is usually located in
`~/.local/share/nix/trusted-settings.json` and can be edited manually to revoke
this permission until Lix provides a first-class command for this manipulation.
```console
❯ nix flake lock
warning: ignoring untrusted flake configuration setting 'allow-dirty', pass '--accept-flake-config' to trust it (may allow the flake to gain root, see the nix.conf manual page)
warning: ignoring untrusted flake configuration setting 'sandbox', pass '--accept-flake-config' to trust it (may allow the flake to gain root, see the nix.conf manual page)
The following settings require your decision:
- allow-dirty = false
- sandbox = false
Do you want to allow configuration settings to be applied?
This may allow the flake to gain root, see the nix.conf manual page (yes for now/Allow always/no/No to all) A
```
Many thanks to [isabelroses](https://git.lix.systems/isabelroses), [Raito Bezarius](https://git.lix.systems/raito), and [eldritch horrors](https://git.lix.systems/pennae) for this.
- `--check` or `--rebuild` is clearer about a missing path [lix#485](https://git.lix.systems/lix-project/lix/issues/485)
Previously, when running Lix with --check or --rebuild, failures often surfaced
as an unhelpful error:
> "some outputs of '...' are not valid, so checking is not possible"
This message could mean two different things:
- The requested output paths don't exist at all, or,
- Some outputs exist but are not known to Lix
Lix cannot reliably distinguish these cases, so it treated them the same.
We've updated the error messages to clarify what Lix can determine: whether any
valid outputs (> 0) are present or whether no outputs are available.
When no valid outputs can be found, Lix will now suggest building the derivation
normally (without --check or --rebuild) before trying again.
When some valid outputs are present, Lix now reports which ones are valid,
shows the full list of known outputs, and also suggests building the derivation
normally.
In the future, Lix may automate this recovery step when it knows how to rebuild
the paths, but implementing that safely requires more extensive changes to the
codebase.
Many thanks to [Raito Bezarius](https://git.lix.systems/raito) for this.
- `nix develop` no longer ignores the env variable `SSL_CERT_FILE` [cl/5042](https://gerrit.lix.systems/c/lix/+/5042)
Running `nix develop` and `nix print-dev-env` on shells that define the environment variable `SSL_CERT_FILE` now works correctly by exporting that variable inside the built shell.
Many thanks to [Tom Hubrecht](https://git.lix.systems/tom-hubrecht) for this.
- Linux sandbox launch overhead greatly reduced [cl/5030](https://gerrit.lix.systems/c/lix/+/5030) [cl/5073](https://gerrit.lix.systems/c/lix/+/5073) [cl/5074](https://gerrit.lix.systems/c/lix/+/5074)
Sandboxed builds are now much cheaper to launch on Linux, with constant management
overhead. This will mostly be noticeable when building derivation trees containing
many small derivations like nixpkgs' `writeFile` or `runCommand` with scripts that
exit quickly. In synthetic tests we have seen build times of 3000 small runCommand
drop from 80 seconds to 14 seconds, which is the most optimistic case in practice.
Many thanks to [eldritch horrors](https://git.lix.systems/pennae) for this.
- mTLS store connections via a plugin [cl/3754](https://gerrit.lix.systems/c/lix/+/3754) [cl/3696](https://gerrit.lix.systems/c/lix/+/3696) [cl/3697](https://gerrit.lix.systems/c/lix/+/3697) [cl/3698](https://gerrit.lix.systems/c/lix/+/3698)
To support use cases requiring mutual TLS (mTLS) authentication when connecting
to remote Nix stores, e.g. private stores, we have introduced a **contributed**
mTLS plugin extending the Lix store interface.
This design follows an extensibility model which was brought up [by a proposal
This mTLS plugin serves as a concrete example of how store connection
mechanisms can be modularized through external plugins, without extending Lix
core. This idea can be generalized to integrate automatic certificate renewal
or advanced integrations with secrets engine or posture checks.
It enables custom TLS client certificates to be used for authenticating against
a remote store that enforces mTLS.
To use the plugin, configure Lix manually by setting in your `nix.conf`:
```
plugin-files = /a/path/to/libplugin_mtls_store.so
```
Currently, this must be done explicitly. In the future, Nixpkgs will provide a
mechanism to reference an up-to-date and curated set of plugins automatically.
Making plugins easily consumable outside of Nixpkgs (e.g., from external plugin
registries or binary distributions) remains an open question and will require
further design.
Contributed plugins come with significantly reduced **stability** and
**maintenance** guarantees compared to the Lix core. We encourage users who
depend on a given plugin to take on maintenance responsibilities and apply for
ownership within the Lix mono-repository. These plugins are subject to removal
at any time.
Many thanks to [Raito Bezarius](https://git.lix.systems/raito), [eldritch horrors](https://git.lix.systems/pennae), [mic92](https://github.com/mic92), [vlaci](https://github.com/vlaci), and [nkk0](https://github.com/nkk0) for this.
- Add an indication of nix-shell nesting depth [lix#826](https://git.lix.systems/lix-project/lix/issues/826) [cl/4657](https://gerrit.lix.systems/c/lix/+/4657)
When in a nix shell (either via a `nix-shell` or a `nix develop` invocation), a variable `NIX_SHELL_LEVEL` is exported to indicate the nesting depth of nix shells.
Many thanks to [Tom Hubrecht](https://git.lix.systems/tom-hubrecht) for this.
- `nix store delete` can now unlink a GC root before deleting its closure [cl/4660](https://gerrit.lix.systems/c/lix/+/4660)
Ever build something, and then you want to delete it and whatever dependencies it downloaded?
Before you had to resolve the `result` symlink and copy it, then delete it, *then* `nix store delete --delete-closure --skip-live` on the path you copied.
Now you can just pass `--unlink` and the `result` symlink itself.
Many thanks to [Qyriad](https://git.lix.systems/Qyriad) for this.
- `nix path-info` no longer lies to the user about fetching paths [lix#323](https://git.lix.systems/lix-project/lix/issues/323) [cl/4866](https://gerrit.lix.systems/c/lix/+/4866)
When running `nix path-info` with an installable that is not present in the store, Lix no longer
tells the user which paths are missing and that they will be fetched, as the documentation clearly
states that this command does not fetch missing paths.
Many thanks to [Tom Hubrecht](https://git.lix.systems/tom-hubrecht) for this.
- Derivations can now be printed in detail in `nix repl` [cl/3842](https://gerrit.lix.systems/c/lix/+/3842)
Traditionally derivations printed in the REPL would only print a formatted object
representing the path of the derivation file it refers to. This makes inspecting
the enhanced derivation attribute sets encountered from `mkDerivation` or similar
wrappers more difficult. Even the `:p`/`:print` command would not elaborate attribute sets
tagged as a derivation.
With this change you can now use `:p`/`:print` to directly inspect a derivation
by providing one as the top-level object. Derivation attribute sets will only be
printed two levels deep and internal derivation attrsets will remain in unexpanded
path form as before. `drvAttrs` will also be elided as these attributes are already
present in the top-level attribute set of the derivation. These heuristics provide
a balance between readability and functionality. When the `:p`/`:print` is omitted,
a bare derivation is printed in the path format as before.
Many thanks to [Lunaphied](https://git.lix.systems/Lunaphied) for this.
- Reject `__json` in structured attributes derivations [lix#380](https://git.lix.systems/lix-project/lix/issues/380) [cl/5286](https://gerrit.lix.systems/c/lix/+/5286)
In structured attributes derivations, `__json` is used internally to store the
JSON representation of the `env` attribute field that users can set.
Unfortunately, a user can set `__json` *and* enable structured attributes,
resulting in a broken derivation from a semantic point of view.
As no user can benefit from setting `__json` *and* enable structured attributes,
we disallow that possibility and throw an error from now on.
This is not seen as a breaking change because there's no user code that can
benefit from this behavior, hence, it's an improvement to user experience.
Many thanks to [Raito Bezarius](https://git.lix.systems/raito) for this.
- Shells support `$NIX_LOG_FD` now [lix#336](https://git.lix.systems/lix-project/lix/issues/336) [cl/4694](https://gerrit.lix.systems/c/lix/+/4694) [cl/4695](https://gerrit.lix.systems/c/lix/+/4695)
Lix's "debugging" shells (`nix3-develop` and `nix-shell`) now set the
`$NIX_LOG_FD` environment variable.
This means that [hook logging in
stdenv](https://github.com/NixOS/nixpkgs/pull/310387) appears while debugging
derivations via `nix3-develop` or `nix-shell`.
Many thanks to [Raito Bezarius](https://git.lix.systems/raito) for this.
- Supplementary groups are now supported for daemon authentication [lix#968](https://git.lix.systems/lix-project/lix/issues/968) [cl/5021](https://gerrit.lix.systems/c/lix/+/5021)
macOS, FreeBSD and Linux now support receiving supplementary groups during UNIX domain authentication to a Lix daemon.
This change is particularly beneficial for systemd units with `DynamicUser=true` that need to connect to a Lix daemon, using a `SupplementaryGroups=` allocated by systemd in the context of the process. This is desirable if you wish to harden Lix clients.
Many thanks to [Raito Bezarius](https://git.lix.systems/raito), [Tom Hubrecht](https://git.lix.systems/tom-hubrecht), [alois31](https://git.lix.systems/alois31), and [eldritch horrors](https://git.lix.systems/pennae) for this.
## Fixes
- Nix shells' `$NIX_BUILD_TOP` are shorter [lix#1044](https://git.lix.systems/lix-project/lix/issues/1044) [cl/4663](https://gerrit.lix.systems/c/lix/+/4663)
Following the changes in 2.94.0 to shorten build directory paths, aimed at [resolving UNIX domain socket length issues](https://gerrit.lix.systems/c/lix/+/4168/13) and [improving nix-shell](https://git.lix.systems/lix-project/lix/issues/940), we inadvertently introduced an excessively long path for the `$NIX_BUILD_TOP` environment variable used by Nix shells (their effective temporary `/build` directory).
To fix this, we replaced the `build-top-$HASH` directory name with simply `build-top`, reducing these paths by at least 30 characters.
We also added a test to ensure that Nix shells do not introduce more than 50 extra characters relative to their base directory (e.g., `/tmp` when `$TMPDIR` is not set).
Many thanks to [Raito Bezarius](https://git.lix.systems/raito) for this.
- Fix resolving of symlinks in flake paths [lix#106](https://git.lix.systems/lix-project/lix/issues/106) [lix#12286](https://git.lix.systems/lix-project/lix/pulls/12286) [cl/4783](https://gerrit.lix.systems/c/lix/+/4783)
Flake paths are now canonicalized to resolve symlinks. This ensures that when a flake is accessed via a symlink, paths are resolved relative to the target directory, not the symlink's location.
Many thanks to [stevalkr](https://github.com/stevalkr) and [xyenon](https://git.lix.systems/xyenon) for this.
- The REPL no longer considers failed loads for `:reload` [lix#50](https://git.lix.systems/lix-project/lix/issues/50) [cl/4864](https://gerrit.lix.systems/c/lix/+/4864) [cl/4865](https://gerrit.lix.systems/c/lix/+/4865) [cl/4700](https://gerrit.lix.systems/c/lix/+/4700) [cl/4889](https://gerrit.lix.systems/c/lix/+/4889)
The [REPL](@docroot@/command-ref/new-cli/nix3-repl.md) allows "loading" files, flakes, and expressions into the environment, with the commands `:load`/`:l`, `:load-flake`/`:lf`, and `:add`/`:a` respectively.
The results of those stay in the environment as-is even if their sources change, until the `:reload` command is used.
However `:reload` would re-perform *all* instances of `:l`/`:lf`/`:a`, meaning you would get things like this:
```nix
nix-repl> :l /tmp/texting.nix
error: getting status of '/tmp/texting.nix': No such file or directory
# oops, typo.
nix-repl> :l /tmp/testing.nix
# Do some stuff…
nix-repl> :reload
error: getting status of '/tmp/texting.nix': No such file or directory
```
This is pretty silly, but also *incredibly* annoying, as it would stop there and *not* reload the correct files anymore.
This effectively meant typoing any of the load commands would make `:reload` useless for the rest of the entire `nix repl` session!
This has been fixed, so now only *successful* loads count towards `:reload`.
Many thanks to [Raito Bezarius](https://git.lix.systems/raito) and [Qyriad](https://git.lix.systems/Qyriad) for this.
- Consistently use commit hash as rev when locking git inputs [cl/4762](https://gerrit.lix.systems/c/lix/+/4762)
Lix will now use commit hashes instead of tag object hashes in the `rev` field
when fetching git inputs by tag in `flake.lock` and `builtins.fetchTree` output.
Note that this means that Lix may change some `flake.lock` files on re-locking. Old `flake.lock` files still remain valid.
Many thanks to [goldstein](https://git.lix.systems/goldstein) for this.
## Development
- Functional lang migration [lix#856](https://git.lix.systems/lix-project/lix/issues/856) [cl/3213](https://gerrit.lix.systems/c/lix/+/3213)
We have done it! The functional/lang framework has now been fully migrated to functional2/lang.
This means: no more `just clean` and `just install` mess and whatever because one removed a test.
The lang test suite is also getting a face lift, with an improved folder structure and restructuring of many tests.
Only the first CL of the chain is provided but there's way more changes associated to this project.
Many thanks to [piegames](https://git.lix.systems/piegames) and [rootile (Rutile)](https://git.lix.systems/rootile) for this.
## Miscellany
- Warn instead of erroring when the final destination of a transfer changes in-flight [lix#1004](https://git.lix.systems/lix-project/lix/issues/1004) [cl/4641](https://gerrit.lix.systems/c/lix/+/4641)
Lix will now emit a warning during downloads where the final destination changes suddently mid-transfer instead of throwing an error.
This transfer behavior has been known to happen very rarely while fetching from some CDNs.
Many thanks to [Tom Hubrecht](https://git.lix.systems/tom-hubrecht) for this.
Linux 3.0 was released 15 years ago. The `impersonate-linux-26` setting was added
14 years ago with no mention of it being necessary to build anything, only saying
that it improves determinism—which isn't accurate since impersonating Linux 2.6.x
still allows the version string to change, and the final component of the version
does still change with each Linux release. Since this setting should be no longer
necessary in modern systems and workarounds for building old code exist (by using
e.g. `setarch --uname-2.6` to wrap builds) we are removing this setting from Lix.
Many thanks to [eldritch horrors](https://git.lix.systems/pennae) for this.
- Default to showing build logs in the new-style (nix3) CLI [cl/4674](https://gerrit.lix.systems/c/lix/+/4674)
Lix will now show logs by default, in addition to the progress bar, when invoked through the new-style "nix3" CLI (`nix build`, etc)
Many thanks to [K900](https://git.lix.systems/K900) for this.
- Lix daemons are now fully socket-activated on systemd setups [lix#1030](https://git.lix.systems/lix-project/lix/issues/1030)
When launched by systemd, Lix no longer uses a persistent daemon process and uses systemd socket
activation instead. This is necessary to support the `cgroups` and `auto-allocate-uids` features
and may improve observability of daemon behavior with common systemd-based monitoring solutions.
The old behavior with a single persistent daemon is still available, but disabled by default. It
is not possible to enable both a persistent daemon and socket activation, starting one stops the
other automatically. Existing installations should not require any changes when they're updated.
Many thanks to [eldritch horrors](https://git.lix.systems/pennae) for this.
- Plugin interfaces have changed (again) [lix#359](https://git.lix.systems/lix-project/lix/issues/359) [cl/4933](https://gerrit.lix.systems/c/lix/+/4933) [cl/4934](https://gerrit.lix.systems/c/lix/+/4934)
The `RegisterPrimOp` class used to register builtins has been removed. Plugins
must now call `PluginPrimOps::add` from their `nix_plugin_entry` with the same
parameters previously passed to `RegisterRrimOp` to register any new builtins.
The `GlobalConfig::Register` helper class has also been removed. Adding config
options to the system is now done with `GlobalConfig::registerGlobalConfig`; a
plugin can add config values by calling this function from `nix_plugin_entry`.
Many thanks to [eldritch horrors](https://git.lix.systems/pennae) for this.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
eldritch horrors