This merge commit returns to the previous state prior to the release but leaves the tag in the branch history.
Release created with releng/create_release.xsh
Change-Id: I01c4b3caeaf2c180d69b406b0dbd663ed1e4392d
string data shares a buffer with the binary string length field. size
calculations for string read buffers always include the length field;
sufficiently large length fields can cause these calculations to wrap.
a malicious nar could use this for OOB writes in the daemon (as root).
since we use strings only as tags for archive members and for symlinks
with their OS-dependent length limits we can simply limit string size.
1 MiB should be sufficient for all symlinks, and tags are always tiny.
Change-Id: I89fb05f73c1dbeda45d91244aba4cd526a3d83e1
I don't understand how `ssh -l lix-releng` is supposed to work if it
doesn't say which host to target.
Change-Id: I791f3f3f49ecd5884c9e86b5d3b617fc139e031f
Signed-off-by: Raito Bezarius <raito@lix.systems>
These parameters are now created on https://s3.afnix.fr.
Change-Id: I96b6fd913429ee46d04c412cb141edd288665ced
Signed-off-by: Raito Bezarius <raito@lix.systems>
When index-debug-info is enabled and the store path being copied is a
regular file (not a directory), std::get_if<nar_index::Directory>
returns nullptr since the NAR root is a File variant. The loop then
immediately dereferences buildIdDir->contents on the null pointer,
causing a segfault.
Add a null check at the top of the loop to break early when the NAR
root is not a directory.
Change-Id: I3a6e792b84cc12c837ecaddf4fee889e1bcb6397
(cherry picked from commit 6c7ccc2588)
lowdown 3.0.0 merged some flags into one to save on bits and did not add
any aliases for backward compatibility.
As with the changes for lowdown >= 1.4, we define a preprocessor flag to
gate the changes on and add a job to CI to ensure that lowdown < 3.0
keeps working (which is used by NixOS 25.11).
The channel version this Lix branch uses doesn't have bmake with support
for all tested platforms, so we can't properly build lowdown 3.0 (which
depends on it) for tests in CI. Given this is a relatively simple
backport tested on other Lix branches, it's probably fine.
Change-Id: I20a3e2fdaa05906f032ff66911c42867557fdd11
(cherry picked from commit e839708839aa132c8694abfdf83409a619f72c5a)
This merge commit returns to the previous state prior to the release but leaves the tag in the branch history.
Release created with releng/create_release.xsh
Change-Id: I2c10392b0b80bff519501c8018ed51aa6a151ca1
When the destructor of PathSubstitutionGoal is run, this happens in a
sync context and can cause starvation of all ongoing IO w.r.t. to other
substitutions, including our own substitution.
While there's only a decompressor thread per stream, the other side of
the IO runs on the event loop.
In order to fix this, it is sufficient to remove the thread indirection
and inline the async code.
Fixes#1126. Great thanks to horrors' patience.
Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: I3eb37bc37d156f0f5528364e568fdaa2ced58011
Signed-off-by: Raito Bezarius <raito@lix.systems>
(cherry picked from commit 505d0669dc)
during the value rewrite we accidentally broke extension of incomplete
primop application. this only shows up when binding on incomplete call
to a primop to a name, binding an incomplete call to *that* to another
name, and then finally calling the second binding with enough args for
a complete primop application. since this only shows up when calling a
primop with three or more args it took a while to surface. we have few
builtins that match this: foldl', replaceStrings, and substring. these
are not used incompletely in this manner very often, so this lingered.
fixes#1102
Change-Id: I218dffc14ae876efc86a86c7eb6c895e2405201c
Closes#1064
The culprit here is that `genGraphString` is only invoked with the
store-paths associated with the outputs of the derivation, so when
filling `dependents`, the `graph_data.find(p)` call would return the end
of the iterator when doing this for references to other store-paths.
As a result, the code wrote information behind the graph data-structure
causing a corruption. For me, this resulted in a SIGSEGV most of the
time and in a few cases in an uncaught `map::at`-exception as reported
by Niklas.
This patch changes two aspects of the original implementation:
* When filling `dependents` in the node-set, use `map.at()` instead of
`map.find()->second`. The latter doesn't make any sense and was the
cause of corrupting memory. The `at` would've made it far easier to
spot this in the first place.
* Filter out store-paths that don't belong to a different output of the
derivation when creating `outputGraph`. This variable is used on two
places, `genGraphString` and for topological sorting.
The latter already filters out store-paths from a different drv, so
this is happening now when creating the variable in the first place
such that `genGraphString` never ends up with corrupt data in the
first place. This is the actual bugfix.
Implemented a regression-test for this case to be sure.
Change-Id: Ie02144d89c32b0a776cb1ece0601d0229315ebc3
(cherry picked from commit 0a5f474a25)
Prior to I6a6a6964d2b5ad47ae5ea9eb11af9b6373ce2141 — `sudo nix
upgrade-nix` would perform direct store access.
This ensured a certain number of desireable properties for upgrading the
Lix binary itself.
We re-introduce direct store access for upgrading Lix binaries.
Fixes#1060.
Change-Id: I523c4d3023ed5fe9eff8fde9a266c56a0de47d8c
Signed-off-by: Raito Bezarius <raito@lix.systems>
(cherry picked from commit d2ca1810b1)
typo of aws-cpp-sdk-transfer as aws-cpp-std-transfer prevents linking
against lix
Change-Id: Id847eab2601698696030d31fcd51288aa5f3d274
(cherry picked from commit 06f987fb0c)
This merge commit returns to the previous state prior to the release but leaves the tag in the branch history.
Release created with releng/create_release.xsh
Change-Id: Ida5959473ec469dd5e5e977ebdc8aab6e694b0de
https://github.com/NixOS/nixpkgs/pull/451579 PR
enabled c-aresSupport for curl on darwin which ended up breaking
DNS resolution in FOD sandboxes:
```
nix-run> exporting https://tangled.org/@weethet.bsky.social/nix-run (rev 73d7bf6b58848fb8f42e3a69816e0847f041c689) into /nix/store/m4m951648wmipxgwrgsml9gzjwfpfhm7-nix-run-73d7bf6
nix-run> Initialized empty Git repository in /nix/store/m4m951648wmipxgwrgsml9gzjwfpfhm7-nix-run-73d7bf6/.git/
nix-run> fatal: unable to access 'https://tangled.org/@weethet.bsky.social/nix-run/': Could not resolve host: tangled.org (Could not contact DNS servers)
nix-run> fatal: unable to access 'https://tangled.org/@weethet.bsky.social/nix-run/': Could not resolve host: tangled.org (Could not contact DNS servers)
nix-run> fatal: unable to access 'https://tangled.org/@weethet.bsky.social/nix-run/': Could not resolve host: tangled.org (Could not contact DNS servers)
nix-run> Unable to checkout 73d7bf6b58848fb8f42e3a69816e0847f041c689 from https://tangled.org/@weethet.bsky.social/nix-run.
```
with these sandbox failures:
```
deny mach-lookup com.apple.SystemConfiguration.DNSConfiguration
deny file-read-metadata /private/etc/hosts
deny file-read-data /private/etc/hosts
```
We allow those so that DNS resolution work fine
Change-Id: I9102293691972feb085adf8e9b1ad915bb3a36ab
This adds the enablement code to support HTTP/3 if the user requests it.
We leave it disabled because h3 is not onpar with h2 performance.
Change-Id: I1fd3d4c97b972dcf36bccacc6c9a8290e22b31e0
Signed-off-by: Raito Bezarius <raito@lix.systems>
Channels have moved from `nixos.org/channels` to `cnannels.nixos.org`.
This udates all relevant links (excluding release notes) to use the new
canonical URLs and replaces HTTP with HTTPS.
Fixes#1031.
Change-Id: I212821c44ac5e482c8e9eaa415c7d8ee17ff8341
Signed-off-by: adam <me@adamperkowski.dev>
URL literals were deprecated in 278fddc317,
effectively stabilizing the experimental feature. Enabling it has no effect any
more. Remove it.
Change-Id: I8fbca03b1a2be6a8cddfce644043ac0a6a6a6964
We start this section with shortcomings of unsandboxed builds.
Fixes#1018.
Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: Ieb17e4340beab0c1197951813ae602de453a3fd9
Signed-off-by: Raito Bezarius <raito@lix.systems>
This is unnecessary because Darwin builds already cannot create
`set{u,g}id` files due to the minimal sandbox policy, because we can
forbid messing with the top‐level build directory directly in the
sandbox rules, and because Darwin builds can trivially avoid isolation
through temporary directories right now anyway.
This does regress the build directory isolation for builds with
`sandbox = false`, but I can’t imagine that mattering given the
above. The sandbox change prepares us for a world where we close
off shared temporary directories for `sandbox = true` builds and try
harder to achieve proper isolation on Darwin, but probably doesn’t
have a meaningful security impact one way or another for now.
With this change, we get down to 41 byte build directory paths on
Darwin, a ~2–3½ byte improvement over the old status quo. We can
also restore the 0710 permissions on Linux.
Change-Id: I6a6a6964a681c0365241fe7234831db656b76799
The minimum build directory length on Darwin with default settings
when we were still using `/tmp` was 22 bytes. Deriving build directory
names from my local store, the median and mode were 43, the mean was
around 44½, and the maximum was 127.
The switch to `/nix/var/nix/builds` over `/tmp` added a 15 byte
penalty, and the additional `/b` directory added another 2.
Now that we use opaque build directory names, the length is 48, so
we’re still at a ~3½–5 byte penalty over the previous status
quo. This change brings us down to 43, matching the previous median.
Note that these calculations do not take into account the fact that
`/tmp` is a symbolic link to `/private/tmp` on Darwin. Anything
that was canonicalizing paths would have had an additional 8 byte
penalty in the previous status quo that is not applicable here,
so we may already be ahead even without this change. If the more
opaque directory name here is undesirable, then that factor could
potentially help us squeeze by without. Alternatively, in combination
with dropping the `/b` on Darwin we could use `/nix/var/nix/bld` or
similar, but I feel that the paths in general are sufficiently opaque
that it should be okay to go with the shorter option here. Given that
some projects already had to reduce filename lengths to avoid this
limit even before the recent changes, I think it is best to try and
improve on the previous status quo.
Note that `/nix/var/nix/builds` will unfortunately not be cleaned up
on Darwin. However, we don’t clean up the directories inside it on
Darwin anyway, so hopefully that’s okay for now?
Closes: #913
Change-Id: I6a6a6964bffce7194bcddcaefb4c4a37569c7df5
We always use the default temporary directory, because
`createUniqueDir` has an interface nice enough to use directly for
the few bespoke uses.
Change-Id: I6a6a696450b7c0a0bd76655632fb14d7c5e38199
This makes the paths more nondeterministic, but more reliably unique,
and lets us remove the retry loop.
Note that this adds random entropy to the build directory visible
inside derivations on Darwin and unsandboxed Linux. It was already
non‐deterministic in the presence of concurrent builds and similar,
but now we can reliably expect it to be different every time. On the
whole I think that’s a good thing, as it is impossible to ensure
a single consistent build directory and derivation outputs should
not depend on it.
Package reproducibility isn’t great on Darwin to begin with,
though, and the reproducibility bugs this will turn up in packages
will be more urgent to fix than when the build directory was mostly
consistent. A quick survey of my local store shows that many C, C++,
and Rust binaries contain build directory references, likely due to
use of `__FILE__` and its equivalents; non‐binary offenders include:
* Install logs included in the Rust and Cargo bootstrap compilers
* Example errors in the Rust documentation referencing build paths
* Configuration information installed with CPython itself
* Python 2 metadata from resholve’s closure
* Cython metadata
* Generated headers in Facebook libraries referencing source paths
* Generated CMake files in Facebook libraries referencing source paths
I haven’t built that much in this store since the last GC, so this is
probably only a small sample of the problems across the tree. These are
all instances of <https://reproducible-builds.org/docs/build-path/>,
though, and should probably just be treated as general reproducibility
bugs outside of contexts like the Linux sandbox where we can normalize
them away entirely.
I have implemented away build directory paths for C/C++, applied some
additional fixes for non‐`__FILE__`‐related issues in binaries
from ATF and LLVM, and fixed the derivation bug causing the CPython
3 issue, and will work on upstreaming these changes. Rust is working
on the problem upstream, with some temporary workarounds we can
potentially apply in Nixpkgs for now. The rest will require some
distributed effort.
Change-Id: I6a6a69645b4915c56c0fdef904aa81684e4136c6
This does change the behaviour when the global temporary directory
does not exist, but other uses of the global temporary directory are
already broken in that circumstance, and it should be fixed centrally
if the use case is considered desirable. The logic was not present
before the recent churn around build directories – it was added now
that Lix is taking ownership of the build directory in the store –
so this should not be a meaningful regression.
Change-Id: I6a6a6964e345ea6803226c5ad759e836de7cb0ed
There is now no risk of race conditions on a system with a functioning
entropy source, and the bespoke prefixes are either redundant to the
default or unnecessary.
Change-Id: I6a6a69641211c6bb979ea48ad30aecb1a53d03f0
Relax the constraints on keeping the exact same filename format to
provide a more robust source of entropy with a simpler interface
(as previously suggested by eldritch horrors). Using 128 bits of
OS‐provided entropy ensures global uniqueness and allows us to
skip any thought of gracefully handling the case where these files
already exist.
My microbenchmark that repeatedly constructed paths like this and
printed them out showed that this takes about 1.23× the time of
the previous implementation, both taking on the order of a couple
microseconds for one iteration. Since everything that uses it is doing
things more expensive than printing to standard output, the actual
performance delta is likely to be lost in the noise. If it somehow
becomes a bottleneck, it can be optimized without sacrificing the
guarantees by reading from the system RNG only to seed a thread‐local
CSPRNG like [ChaCha8Rand], but I think that’s very unlikely.
We also tweak the recommended way of creating a temporary file inside
a directory in anticipation of later changes, and rename the `suffix`
parameter to `prefix` (it’s a prefix to the random characters and
a suffix to the root, but this way is more consistent).
[ChaCha8Rand]: https://c2sp.org/chacha8rand
Change-Id: I6a6a69648502c746d13d8c3bd2768cbbf1b90466
base32Encode now takes std::span<std::byte>, with a base32EncodeStr
convenience wrapper which takes std::string_view.
Co-authored-by: Qyriad <qyriad@qyriad.me>
Change-Id: I6a6a6964f799dc84ecbfb55c7ca03a064cff71d9
the change to shareable thunks also removed a few cases of pointer
equality checks that allowed structures containing functions to be
considered equal to other sets containing the same functions, even
if the sets themselves were pointer-equal themselves. *so* busted.
Change-Id: If87fdab658f9037ce2a654f69a9e3da6ae2f53e5
we can't change the protocol to allow daemons to do this, and we should
not try to guess what the `auto` store uri means depending on whether a
command was run by root or not due to copious side effects and not even
being able to tell whether the `auto` store uri was given explicitly or
not. while `auto` may *technically* allow this via its naming we should
resist the urge to add a hack and fix the underlying protocols instead,
especially since repairPath should be a rare, superuser only operation.
fixes#888
Change-Id: I1b53245db226199f827a89a237a2ab9907c3f766
Raito Bezarius