This reverts commit e356d54d7a
because this was insufficient to fix the critical correctness bugs.
Change-Id: I91c3e368ffd13ade6a3cebbbacdb42655796ea56
Using `AllowDaemon::Disallow` here broke `ssh-ng://` remote builds in
multi‐user setups where the remote builder user does not have write
access to the store, now that the automatic store selection logic
has changed. Switch to the default behaviour for this path to fix that.
This causes `ssh-ng://` builds to use the daemon by default on the
remote end, even as `root`. I think this is desirable, as the previous
change already made `ssh://` behave this way, and the pitfalls of
local stores apply to remote builds too. For instance, there were
persistent `ulimit` issues on the NixOS Hydra macOS builders that were
resolved by forcing use of the daemon, and I believe the Linux builders
also go through the daemon these days due to using non‐`root` SSH
users. I believe that the `root` vs. non‐`root` difference is just
as confusing for remote builds as it is for local ones.
`ssh-ng://root@builder?remote-store=local` can be used to revert back
to the previous default if necessary.
Closes: #884
Fixes: 9a59106c17
Change-Id: I6a6a696410f46cd3f2f5a94073ea924ad45dc99c
If `settings.buildDir` cannot be written to, because we are in a chroot
store, unprivileged or anything.
We can and should always gracefully fallback to a *secure* location
inside of /tmp, i.e. `/tmp/<a directory under 0700>/<our temporary
directory for build under 0700>/...`.
This does not reintroduce CVE-2025-52991 because we are creating a
directory in-between compared to creating only ONE level of directory.
Fixes#876.
Change-Id: Ie521202923f763225e1901ab1b9b6c6132aaf548
Signed-off-by: Raito Bezarius <raito@lix.systems>
eagerly consider outputs as not needing deletion during output
registration rather than only doing so after registration. not
waiting for registration to succeed may keep store paths alive
in the file system if registration fails for some reason; that
seem preferrable to the possibility of having another instance
of this bug. since we only leave *good* outputs around there's
not much to worry about except maybe bit of wasted disk space.
fixes#883
Change-Id: I8c22c92e39b9e203f1061278f86cde19dc4474a4
In the past, it tried direct access if it *could* [1] perform direct
access.
This solves a bunch of errors people had when they tried the cgroup
feature and their scripts did not pass NIX_REMOTE=daemon manually
(nixos-rebuild-ng, home-manager activation from a root systemd unit,
etc.)
To avoid looping infinitely while receiving daemon connections, we
forcibly change the store URI when forking for a subdaemon to do direct
access automatically, this doesn't break forward usecases where you
point a daemon to another socket because we only change NIX_REMOTE="",
NIX_REMOTE=daemon, NIX_REMOTE=auto to a local and direct access.
All these usecases would end up infinitely looping no matter what
settings are set, because we are also responsible for creating the
daemon socket.
[1]: this happened all the time if you were `root`.
Related: https://github.com/NixOS/nixpkgs/pull/415701
Change-Id: I783fc795a9c2ee25b3d9f44f453f8f94b063371f
Signed-off-by: Raito Bezarius <raito@lix.systems>
The reason this gets hit is because of the debugger in flakes. Otherwise
you never have a repl in pure mode anyway.
We evaluate the repl-overlay file in impure mode but this doesn't do
what one would initially expect.
Fixes: https://git.lix.systems/lix-project/lix/issues/777
Change-Id: I19b8ed2f5e9ce500b633b13301b42df69ab7deb3
idk how this mistake happened but it was really confusing to figure out
which one of these was right, so let's get rid of the impostor.
Change-Id: If3b6fb543e5976b1edad68fb143bfa994d1d6381
When a build fails, its scratch output paths are not cleaned up.
Until recently, this was deemed not a problem but as part of the effort
to harden the Nix builds and protect these paths against being part of a
staged attack (race conditions, etc.), we automatically cleanup after
failed builds.
Fixes CVE-2025-52992.
Change-Id: I58481b1cc83826298b9d80d37fecf81f117ccb09
Signed-off-by: Raito Bezarius <raito@lix.systems>
if a build directory is accessible to other users it is possible to
smuggle data in and out of build directories. usually this ins only
a build purity problem, but in combination with other issues it can
be used to break out of a build sandbox. to prevent this we default
to using a subdirectory of nixStateDir (which is more restrictive).
Fixes CVE-2025-52991.
Change-Id: Iacfc9b50534de158618c815f9fb99d7dae1be4d0
This allows using a userspace program, pasta, to handle comms between
the build sandbox, and the outside world; allowing for full isolation
including the network namespace, closing the "fixed-output derivation
talks to the host over an abstract domain socket" hole for good.
Fixes CVE-2025-46416.
Co-Authored-By: Puck Meerburg <puck@puckipedia.com>
Change-Id: Ifd499b7dbb3784600a6e842fede65fc031ff9f15
This is useful for certain error recovery paths (no pun intended) that
does not thread through the original path name.
Change-Id: I2d800740cb4f9912e64c923120d3f977c58ccb7e
Signed-off-by: Raito Bezarius <raito@lix.systems>
This was probably a typo introduced in
7453e2979f.
Unfortunately, AWS SDK is so well made that this typo became an assert
error in production.
AWS Outcome constructors contains
```
// Move error from other type of outcome
template<typename RT, typename ET,
enable_if_t<!std::is_convertible<RT, R>::value &&
std::is_convertible<ET, E>::value, int> = 0>
```
which means that when:
* RT → R is not possible (e.g. PutObjectOutcome → HeadObjectOutcome)
* ET → E is possible (e.g. S3Error → S3Error)
Then, we will instantiate the error-moving outcome constructor which
asserts `!o.success`… Though, the original outcome indeed succeeded.
Change-Id: I3809514ae0648e8c02b0f93fa64d91115a091cd9
Co-authored-by: Maximilian Bosch <maximilian@mbosch.me>
Signed-off-by: Raito Bezarius <raito@lix.systems>
Added an additional check that all files present within a folder must be
used/referenced. Otherwise an InvalidLangTest will be created.
This ensures that there weren't any mishaps while migrating tests
resulting in files being ignored and hence some tests not being run.
Fixes: #852
Change-Id: Ie096c5670bc20325ba72c7d6ce33c06667c66ab1
Redesigns the test.toml to use a list instead of a directory
additionally it is now possible to do toml and matrix tests on singular
files as well as on a subset of files.
Fixes: #851
Change-Id: If8635109c6274f406ad68fe35315b9125f45f67d
Currently when a lang test fails, (or any snapshot assertion for that
matter) the error message is rather bulky.
This is due to both sides being printed fully, using escaped newlines
(i.e. everything is one line)
This is awful to read and check what the actual difference is. Also
there is no indication that one can update the golden files using the
cli flag.
This commit changes the error message when comparing snapshots against
something
a list of lines is shown, where the output differed. An additional note
about how to update the files automatically was added too
Change-Id: Ibedcf48018c27f924b807fbd42362fb608d27441
When Ctrl-C is sent to the workload, even across remote builds, the
whole process possess a global flag `_isInterrupted` which is checked in
certain filesystem operations, cancelling them, e.g. writeFile will
write nothing under interruption unwinding.
In addition, if any operation throws an exception before we `rmdir` the
cgroup, we may leave it hanging while we remove the state record.
Therefore, we put the final cleanup in a block.
In practice, reading statistics could lead to failures.
Control groups cleanups are critical though and should always be
performed.
Change-Id: I48fa87317b6a9f6663559bc8fa5f8a897f37011e
Signed-off-by: Raito Bezarius <raito@lix.systems>
making stores and their users fully async requires all data streams to
be async. the most notable data streams in common usage are curl first
and remote stores second. curl is much more contained today and easier
to asyncify (with the preparatory work we've done in the past commits)
Change-Id: I2d6ff4687ee2b47e4efaa6714827b7283bed941d
We upgrade to 25.05 release, which contains the curl commit
https://github.com/curl/curl/commit/5fbd78eb2dc4afbd8884e8eed27147fc3d4318f6
done in
https://github.com/NixOS/nixpkgs/pull/396200#issuecomment-2795944006.
This fixes HTTP transfers generating arbitrary errors and possibly
failing unusually.
Users who are already depending on 25.05-small or a recent unstable
already had the fix.
Special mention to the Linux kernel who gave me the opportunity to get
on a 24 hours bisection side quest to fix the local release engineering
test.
Special thanks to everyone who had to endure me ranting.
Change-Id: I866caf65d5ea103f1fa5eccd57df8031c9eacda0
Co-authored-by: eldritch horrors <pennae@lix.systems>
Co-authored-by: helle <helle@h3l.li>
Signed-off-by: Raito Bezarius <raito@lix.systems>
Most of these are simple fixes and clarifications. One set of fixes will
come in the commit that actually upgrades nixpkgs and hence ruff as it
will otherwise cause errors here.
Change-Id: Ie857da0f6cf728478700ec2d24cf518f8c7b7815
processStderr of RemoteStore wants to be a promise and it must be used
from connection setup, so the pool factory callback must be a promise.
Change-Id: I9ac742b6048ae6dba0bfa5dcb58971386229690b
the rpc transition will require sync and async objects to share a single
io buffer (since defining serializers on async is an immense pain in the
tail, slow, and ultimately not necessary). a generic buffer class allows
us to reuse existing serializers more readily (reuse them at all, even).
Change-Id: I5ebba8449f26f2bb76016818928183c7e0123be0
We offer full cgroup delegation to our sandbox now, required for running
containers inside the sandbox.
To run systemd-nspawn or containers managers inside the sandbox, there
is a need for one extra ingredient now: control over your own cgroup
subtree inside the sandbox.
If, in addition, you need multiple UIDs, for e.g. rootless usecases, you
need to run with the `uid-range` system feature.
Therefore, when the daemon or Nix runs under the right condition, e.g.
systemd-style delegation of the cgroup subtree while placing the
nix-daemon in a supervisor sub-cgroup, we create a new sub-cgroup for
each build based on the build UID and delegate that sub-cgroup to the
builder's process.
Additionally, `uid-range` always request the `cgroups` feature now, as
`uid-range` builds would probably always benefit from having cgroups
delegated, but the converse is not true.
Inspired from https://github.com/NixOS/nix/pull/11412 with a different
design that does not use function-local statics to derive the root
cgroup.
Co-authored-by: Linus Heckemann <git@sphalerite.org>
Co-authored-by: Parker Hoyes <contact@parkerhoyes.com>
Change-Id: Ic8947c5adaf4b5bbd153386e05fad65a935274fa
Signed-off-by: Raito Bezarius <raito@lix.systems>
We drop it to re-introduce it via the concept of build context which
will control in which cgroup a certain build should be spawned.
Change-Id: I4b4705d768129a6d7c0f061dc2163ba116088b18
Signed-off-by: Raito Bezarius <raito@lix.systems>
Some source trees might not be representable inside of the NAR listing
format v1 as file paths (on Linux) are not guaranteed to be valid UTF-8.
When something like this happens on a large-scale build farm, a
mysterious "queued" but impossible to process job appears, this is
because we cannot write the NAR listing and serialization always fails.
Why did this work before? nlohmann was introduced _after_ such paths
were ingested, see: 09f00dd4d0.
What happened for such previously mis-serialized NAR listings?
```
curl -v 'https://cache.nixos.org/nz8p9hn00r6z7s57581c1hiv39pa1ia6.ls' |
brotli -d | jq .
```
This fixes the build of `sub-batch`
(https://github.com/kl/sub-batch/tree/master/tests/rename_invalid_utf8)
on ForkOS infrastructure.
Many thanks to Puck for the assistance on holding `rr` right on this one
and finding the history of these changes.
Change-Id: I2c2fbac70818e02810f9fd236c3a248187bf5fe7
Signed-off-by: Raito Bezarius <raito@lix.systems>
Currently, the typecheck for the config values is only done
half-heartedly only checking if something is either a list or non-list
item, but not checking what type the list items are
this commit fixes the typecheck and adds test for proper serialization
Change-Id: Ifd93842b19b1dd870bdb3af0c000243b4380e7aa
The error message used to only contain the last key of the merge failure
this commit changes the message to contain the full path to the merge
conflict, resolving ambiguity
Change-Id: I9848a559b1b888e50a548eef8609bf34506040de
currently, there is a small helper funciton in lang_util to check if
something is of a list type generic
to improve re-usability, this function is moved to utils and improved to
be also check for nested iterables and such
Change-Id: I92984daa4c4decf13d340a2ea5e52f724cee800e
the test is for the map that usually wraps it though because it's the
bit we're interested in replacing, and it has custom serializer code.
Change-Id: If77a236dfca738b646ed2b7a5c65515dad6b7295
the old protocols are largely untested, mostly unused, and have design
problems that make the RPC transition a lot harder, if not impossible.
in theory we could ship a transparent protocol-converting proxy that'd
isolate the daemon itself from old protocol versions, but that's a lot
of code to maintain for presumably little gain or even no gain at all.
Change-Id: I4c3f3bb34d39044f6aeb07c10caaf13b8340a220
All changes are uniform and done with the same script, so checking only
some should suffice. For that reason, any tests involving multiple files
or custom CLI flags are not included in this commit.
Change-Id: Ib2d0e08937b56e241d99771a58aad34ed3ad308a
The current `RelativeTo` design is both more complex and more confusing
than necessary. Its four variants are now reduced to only two. They are
now also represented as different classes, to better communicate the
difference in semantics and also intent.
Change-Id: Ia60fc7a2dfa0f62bdef90dde347fd8603fd3fbf9
Previously, paths not being deleted by gcDeleteSpecific would result in
(a) hardlinks not being cleaned up, and
(b) statistics not being reported correctly.
By throwing the error later, we fix both of these problems.
Change-Id: I8019f3e10d9f22e81ea87bb26b77f04ebc888a19
By default, xfail tests will always "pass" when the test fails,
disrecsarding any restrictions put on them via their parameters.
By enabling the `xfail_strict` option, xfails won't pass anymore when
the failstate is different from what is described in their parameters.
Change-Id: Ifea6e27d716d91f60210e6ba24175074fa39c304
Remote binary caches support `write-nar-listing` options where they create a `HASH.ls` file for quick indexing without having to download the nar.
This commit makes experimental `nix store ls` attempt to read these files instead of downloading the full nar.
The difference is very obvious with large packages like stellarium:
nix store ls --store "https://cache.nixos.org" /nix/store/ijpvwgs9zamqaax5dy2cd0kxgz7lr7an-stellarium-25.1 -R
Change-Id: I6a37e0788b3a91c319331a8de69c51daf3efa955
Add Documentation for usage and development within functional2
including common fixtures and where to find them
This is done to make the migration from functional easier and give devs
a reference for how one writes tests
Change-Id: I6ee73e654d245fd4ad43e495d1172e406313cb23
This creates a framework similar to the old lang.sh from functional.
Some notable changes:
- instead of having a .flags file, a test.toml can declare flags
- additionally the test.toml can also declare extra files and multiple
runners for the given input file.
- there won't be any old tests hanging around anymore which weren't
deleted properly in the installation
- all files for a single test are defined decleratively and there won't
be any residues
Tests can be placed within the functional2/lang folder
most migrations should be rather clean
Implements: #825
Change-Id: I5f9149903ec5b078008969a4ae77305417c11475
Currently, tests are marked as "passed" when golden files are updated.
With this change, the tests are marked as skipped instead.
Additionally finally introduces tests to check if the snapshot behaves
as expected
Change-Id: I438eed70e0b94d561e99cc1e0363092809da827e
Add utils for general-use functions and paths
Additionally introduces a pytest_command fixture, which creates a
testing environment for pytest within the tmp_path. This allows for
encapsulated testing of our frameworks (i.e. snapshot, lang etc)
Change-Id: Ic0a5bc4bfc0b0bfbac15bc51dd4a94fae6ee6f26
allow to pass absolute paths or similar Path entries to declaration of
files instead of just string paths relative to the requesting file
Change-Id: I616da6abbb73d1d63ead370e9ae37a401d85f42d
The first auto-GC request would not be registered as a waiter due to a logic
error. As a result, if that request was synchronous (as happens during
evaluation) it would be stuck forever waiting on a promise that will never be
fulfilled.
Register also the first request properly so that it is notified and unblocked
again when the GC has finished. Also add a test verifying that auto-GC
triggering during evaluation will not get stuck.
Fixes: https://git.lix.systems/lix-project/lix/issues/844
Change-Id: I157afdc737415261e48d6d01d46c586a2927a1ad
Raito Bezarius