This makes the paths more nondeterministic, but more reliably unique,
and lets us remove the retry loop.
Note that this adds random entropy to the build directory visible
inside derivations on Darwin and unsandboxed Linux. It was already
non‐deterministic in the presence of concurrent builds and similar,
but now we can reliably expect it to be different every time. On the
whole I think that’s a good thing, as it is impossible to ensure
a single consistent build directory and derivation outputs should
not depend on it.
Package reproducibility isn’t great on Darwin to begin with,
though, and the reproducibility bugs this will turn up in packages
will be more urgent to fix than when the build directory was mostly
consistent. A quick survey of my local store shows that many C, C++,
and Rust binaries contain build directory references, likely due to
use of `__FILE__` and its equivalents; non‐binary offenders include:
* Install logs included in the Rust and Cargo bootstrap compilers
* Example errors in the Rust documentation referencing build paths
* Configuration information installed with CPython itself
* Python 2 metadata from resholve’s closure
* Cython metadata
* Generated headers in Facebook libraries referencing source paths
* Generated CMake files in Facebook libraries referencing source paths
I haven’t built that much in this store since the last GC, so this is
probably only a small sample of the problems across the tree. These are
all instances of <https://reproducible-builds.org/docs/build-path/>,
though, and should probably just be treated as general reproducibility
bugs outside of contexts like the Linux sandbox where we can normalize
them away entirely.
I have implemented away build directory paths for C/C++, applied some
additional fixes for non‐`__FILE__`‐related issues in binaries
from ATF and LLVM, and fixed the derivation bug causing the CPython
3 issue, and will work on upstreaming these changes. Rust is working
on the problem upstream, with some temporary workarounds we can
potentially apply in Nixpkgs for now. The rest will require some
distributed effort.
Change-Id: I6a6a69645b4915c56c0fdef904aa81684e4136c6
This does change the behaviour when the global temporary directory
does not exist, but other uses of the global temporary directory are
already broken in that circumstance, and it should be fixed centrally
if the use case is considered desirable. The logic was not present
before the recent churn around build directories – it was added now
that Lix is taking ownership of the build directory in the store –
so this should not be a meaningful regression.
Change-Id: I6a6a6964e345ea6803226c5ad759e836de7cb0ed
There is now no risk of race conditions on a system with a functioning
entropy source, and the bespoke prefixes are either redundant to the
default or unnecessary.
Change-Id: I6a6a69641211c6bb979ea48ad30aecb1a53d03f0
Relax the constraints on keeping the exact same filename format to
provide a more robust source of entropy with a simpler interface
(as previously suggested by eldritch horrors). Using 128 bits of
OS‐provided entropy ensures global uniqueness and allows us to
skip any thought of gracefully handling the case where these files
already exist.
My microbenchmark that repeatedly constructed paths like this and
printed them out showed that this takes about 1.23× the time of
the previous implementation, both taking on the order of a couple
microseconds for one iteration. Since everything that uses it is doing
things more expensive than printing to standard output, the actual
performance delta is likely to be lost in the noise. If it somehow
becomes a bottleneck, it can be optimized without sacrificing the
guarantees by reading from the system RNG only to seed a thread‐local
CSPRNG like [ChaCha8Rand], but I think that’s very unlikely.
We also tweak the recommended way of creating a temporary file inside
a directory in anticipation of later changes, and rename the `suffix`
parameter to `prefix` (it’s a prefix to the random characters and
a suffix to the root, but this way is more consistent).
[ChaCha8Rand]: https://c2sp.org/chacha8rand
Change-Id: I6a6a69648502c746d13d8c3bd2768cbbf1b90466
base32Encode now takes std::span<std::byte>, with a base32EncodeStr
convenience wrapper which takes std::string_view.
Co-authored-by: Qyriad <qyriad@qyriad.me>
Change-Id: I6a6a6964f799dc84ecbfb55c7ca03a064cff71d9
The prospective callers of this should probably be doing something
smarter or more abstracted to begin with, but this is useful as an
incremental improvement for call sites with existing `makeTempPath`
logic in the face of filename length limits.
Change-Id: I6a6a69644292f5bbf984a1df90192e06c6022b53
nix develop should ignore output checks in general.
This was done only for the old way of specifying output checks, the
structured attrs way requires rewriting the JSON and removing the output
checks pieces.
We take a brutal approach of removing as many as possible including
non-recommended ways of doing it.
Fixes#997.
Change-Id: Iaf83029016c71b5171e56e15d4eadc1a60a8be98
Signed-off-by: Raito Bezarius <raito@lix.systems>
(cherry picked from commit 992c3ae981)
Fixes fj#940.
When running `nix-shell`, the `$NIX_BUILD_TOP` environment variable is
set to `$TMPDIR` or `/tmp`.
nixpkgs stdenv uses $NIX_BUILD_TOP to create `$NIX_BUILD_TOP/env-vars`
which contains all the environment variables set by stdenv. This is used
for debugging purposes in combination with `--keep-failed` to reload the
bash environment of a derivation.
`$TMPDIR` is often unset, therefore, `/tmp/env-vars` was constantly
being created. On a multi-user system or, when you run Lix as root, you
might create a `/tmp/env-vars` with different permission bits.
As a result, `nix-shell` can cease to function because that file will
fail creation for an unprivileged user for example.
fj#940 rightfully remark that the code is not consistent between
nix3-develop and nix-shell and it should be reworked.
Change-Id: Iddf15945385d8bd497b2800b37fee5e1f97689b7
Signed-off-by: Raito Bezarius <raito@lix.systems>
(cherry picked from commit feab75bde0)
Some platforms like 32-Bit PowerPC need linking against libatomic.
Try to compile and link a very simple snippet of code which uses atomics
and make libatomic required if it fails.
Change-Id: I6a6a696471e1d352fb161c537ba9023b97c2d31e
(cherry picked from commit 95448347ea)
This addresses several changes from toml11 4.0 bump in
nixpkgs [1].
1. Added more regression tests for timestamp formats.
Special attention needs to be paid to the precision
of the subsecond range for local-time. Prior versions select the closest
(upwards) multiple of 3 with a hard cap of 9 digits.
2. Normalize local datetime and offset datetime to always
use the uppercase separator `T`. This is actually the issue
surfaced in [2]. This canonicalization is basically a requirement
by (a certain reading) of rfc3339 section 5.6 [3].
3. If using toml11 >= 4.0 also keep the old behavior wrt
to the number of digits used for subsecond part of the local-time.
[1]: https://www.github.com/NixOS/nixpkgs/pull/331649
[2]: https://www.github.com/NixOS/nix/issues/11441
[3]: https://datatracker.ietf.org/doc/html/rfc3339
(cherry picked from commit dc769d72cb8ad22a0f89768682b5499a9d2b3d8b)
Upstream-PR: https://github.com/NixOS/nix/pull/13741
Change-Id: Iac4fbe5108be79be585e9670fa42dfd11f3c5e89
(cherry picked from commit 2898b9e7dc)
There's no reason to use a std::function for recursive lambdas
since there are polymorphic lambdas.
(cherry picked from commit a80a5c4dba0d944fab8f5ed57a343869ae96bf16)
Upstream-PR: https://github.com/NixOS/nix/pull/13741
Change-Id: I593bd04597e2ae000374ca1eca4d8928e986c0b5
(cherry picked from commit 5badc1bc8a)
This looks really weird after the reformat.
(cherry picked from commit df4e55ffc13c413e270af134227115a20a2341ba)
Upstream-PR: https://github.com/NixOS/nix/pull/13741
Change-Id: I8de92d58620cc4545a31d8b7d533d2f1e9f4f233
(cherry picked from commit 2ca5670ec3)
Current test suite doesn't cover the subsecond formatting at
all and toml11 is quite finicky with that. We should at the very
least test its behavior to avoid silent breakages on updates.
(cherry picked from commit 7ed0229d1abd4414144c7af396842462ce6fc1eb)
Upstream-PR: https://github.com/NixOS/nix/pull/13741
Change-Id: I6a6a696433b168072d6ad2585dce8a3c10ccbc39
(cherry picked from commit b2e48aac5c)
Without https://github.com/NixOS/nixpkgs/pull/434761 evaluation of the
`nixpkgsLibTests` will fail in CI with recent enough Lix, due to reliance on
the TOML integer saturation bug.
Reported-by: Sergei Zimmerman <sergei@zimmerman.foo>
Change-Id: I6a6a6964838009d2c525f67035f84072fdfad988
lowdown 1.4.0 changed the lowdown_opts to include a new and separate
lowdown_opts_term which allows for configuring values specific to
-Tterm (which we're using). This version should have been called 2.0.0
according to semver, hence 2.0.0 was released later without any actual
breaking changes to sort of migitate the problem.
We need to support lowdown >= 1.3 && < 1.4 since the ship has sailed for
updating lowdown in NixOS 25.05 as well as lowdown >= 1.4 or we'll be
stuck in Nixpkgs forever. Support for < 1.4 can be dropped as soon as
NixOS 25.05 is EOL, assuming this change lands before NixOS 25.11
branch-off.
We detect the changed API based on the lowdown version from pkg-config
and define LOWDOWN_SEPARATE_TERM_OPTS based on that. The ifdef is named
according to the specific API change that impacts us, so that it's
hopefully a little simpler to maintain going forward. In the new API,
all newly configurable settings use what would have been the (implicit)
default before. Changing some of these values, especially hpadding,
could be interesting in future changes.
Compared to cl/3081, this change makes sure to initialize all new fields
of lowdown_opts_term explicitly.
It seems that, while making -Tterm more configurable, lowdown's word
wrapping behavior changed slightly which broke basic_repl.test. I've
chosen to work around this by using builtins.add as an example which has
a very short documentation string, so wrapping doesn't matter.
Change-Id: Id73be4c0e43d7eb4f56e10a261b4254402698ff8
(cherry picked from commit 858de5f47a)
This merge commit returns to the previous state prior to the release but leaves the tag in the branch history.
Release created with releng/create_release.xsh
Change-Id: Ie9fa603173d7aab8a4ca8a6a0594158779701405
staging-next banned !structuredAttrs && separateDebugInfo && disallowedRequisites
due to weird output interactions. Enable structuredAttrs so we can build again.
Also, fix type confusion that makes stdenv explode (https://github.com/NixOS/nixpkgs/issues/422989).
Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: Ic0c773394ee79e10d427f27750d59892d6d1f1d1
(cherry picked from commit 378b360bf8)
libarchive *should* not break with 0710 on the tmpdir root on darwin,
just like it doesn't break on linux, but for some reason it does. the
restriction to 0710 can be weakened to 0750 with causing any trouble.
fixes#921
Change-Id: Ia9fc2f8eb9695fc19cefae9857368d5a4e58c8b9
although we only chown if the build was requested by a local daemon
user. daemonless invocations will not chown as they do not have to.
remote builds *can* chown to the remote builder user, but that does
not seem to happen (for some reason keep-failed is not propagated).
Change-Id: Ic0ead406b38b4ca0556fec42d84888efa25123bf
(cherry picked from commit ae3b8e58c3)
this makes the actual build directories used by builders invisible and
inaccessible to other processes on the system, avoiding another vector
for outside processes to interfere with builds or pass credentials the
build sandbox should not have access to into the build sandbox anyway.
fixes#919
Change-Id: Ifaa4d8e3940cfde1406e925f75c1375d2e86d81a
(cherry picked from commit 9d5a5c4dc0)
This merge commit returns to the previous state prior to the release but leaves the tag in the branch history.
Release created with releng/create_release.xsh
Change-Id: Ia72a7fd2461f07398c3eb0f49e7448300688dfe9
We’re already allowing `/tmp` anyway, so this should be harmless,
and it fixes a regression in the default configuration caused by
moving the build directories out of `temp-dir`. (For instance, that
broke the Lix `guessOrInventPath.sockets` test.)
Note that removing `/tmp` breaks quite a few builds, so although it may
be a good idea in general it would require work on the Nixpkgs side.
Fixes: 749afbbe99
Change-Id: I6a6a69645f429bc50d4cb24283feda3d3091f534
(cherry picked from commit d1db3e5fa3)
If `settings.buildDir` cannot be written to, because we are in a chroot
store, unprivileged or anything.
We can and should always gracefully fallback to a *secure* location
inside of /tmp, i.e. `/tmp/<a directory under 0700>/<our temporary
directory for build under 0700>/...`.
This does not reintroduce CVE-2025-52991 because we are creating a
directory in-between compared to creating only ONE level of directory.
Under macOS, the first level of directory has actually mode 0755 instead
of 0700 as macOS often do not possess the right primitives to chroot
inside of these directories, leading to
https://github.com/NixOS/nix/pull/11031.
Thanks to Emily for the heads-up on this type of matter.
Fixes#876.
Change-Id: Ie521202923f763225e1901ab1b9b6c6132aaf548
Signed-off-by: Raito Bezarius <raito@lix.systems>
Required to make the releng scripts work.
I know this is not optimal and we should have a proper merge commit from
releng/2.93.1 appearing here, but this is fine.
Change-Id: I11f8ccb8d2a5b124cd057d948aa80dd8be3a7ffd
Signed-off-by: Raito Bezarius <raito@lix.systems>
* Announce the deprecation of ca-derivations and various other features
as planned initially.
* Fixes papercuts in 2.93.0 (SSH connections).
* Fixes the curl download bug for non-Nixpkgs users.
* Fixes CVE-2025-46415, CVE-2025-46416, CVE-2025-52991, CVE-2025-52992,
and CVE-2025-52993.
Change-Id: I8f700396a5ac57d2a1832833f83c22645c73697d
Signed-off-by: Raito Bezarius <raito@lix.systems>
When a build fails, its scratch output paths are not cleaned up.
Until recently, this was deemed not a problem but as part of the effort
to harden the Nix builds and protect these paths against being part of a
staged attack (race conditions, etc.), we automatically cleanup after
failed builds.
Fixes CVE-2025-52992.
Change-Id: I58481b1cc83826298b9d80d37fecf81f117ccb09
Signed-off-by: Raito Bezarius <raito@lix.systems>
if a build directory is accessible to other users it is possible to
smuggle data in and out of build directories. usually this ins only
a build purity problem, but in combination with other issues it can
be used to break out of a build sandbox. to prevent this we default
to using a subdirectory of nixStateDir (which is more restrictive).
Fixes CVE-2025-52991.
Change-Id: Iacfc9b50534de158618c815f9fb99d7dae1be4d0
This allows using a userspace program, pasta, to handle comms between
the build sandbox, and the outside world; allowing for full isolation
including the network namespace, closing the "fixed-output derivation
talks to the host over an abstract domain socket" hole for good.
Fixes CVE-2025-46416.
Co-Authored-By: Puck Meerburg <puck@puckipedia.com>
Change-Id: Ifd499b7dbb3784600a6e842fede65fc031ff9f15
When calling `_deletePath` with a parent file descriptor, `openat` is
made effective by using relative paths to the directory file descriptor.
To avoid the problem, the signature is changed to resist misuse with an
assert in the prologue of the function.
Fixes CVE-2025-46415.
Change-Id: I6b3fc766bad2afe54dc27d47d1df3873e188de96
Signed-off-by: Raito Bezarius <raito@lix.systems>
This ensures that `passAsFile` data is created inside the expected
temporary build directory by `openat()` from the parent directory file
descriptor.
Fixes CVE-2025-52993.
Change-Id: Ie5273446c4a19403088d0389ae8e3f473af8879a
Signed-off-by: Raito Bezarius <raito@lix.systems>
`writeFile` lose its `sync` boolean flag to make things simpler.
A new `writeFileAndSync` function is created and all call sites are
converted to it.
Change-Id: Ib871a5283a9c047db1e4fe48a241506e4aab9192
Signed-off-by: Raito Bezarius <raito@lix.systems>
We use it immediately for the build temporary directory.
Change-Id: I180193c63a2b98721f5fb8e542c4e39c099bb947
Signed-off-by: Raito Bezarius <raito@lix.systems>
We now keep around a proper AutoCloseFD around the temporary directory
which we plan to use for openat operations and avoiding the build
directory being swapped out while we are doing something else.
Change-Id: I18d387b0f123ebf2d20c6405cd47ebadc5505f2a
Signed-off-by: Raito Bezarius <raito@lix.systems>
