This merge commit returns to the previous state prior to the release but leaves the tag in the branch history.
Release created with releng/create_release.xsh
Change-Id: I2c10392b0b80bff519501c8018ed51aa6a151ca1
When the destructor of PathSubstitutionGoal is run, this happens in a
sync context and can cause starvation of all ongoing IO w.r.t. to other
substitutions, including our own substitution.
While there's only a decompressor thread per stream, the other side of
the IO runs on the event loop.
In order to fix this, it is sufficient to remove the thread indirection
and inline the async code.
Fixes#1126. Great thanks to horrors' patience.
Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: I3eb37bc37d156f0f5528364e568fdaa2ced58011
Signed-off-by: Raito Bezarius <raito@lix.systems>
(cherry picked from commit 505d0669dc)
during the value rewrite we accidentally broke extension of incomplete
primop application. this only shows up when binding on incomplete call
to a primop to a name, binding an incomplete call to *that* to another
name, and then finally calling the second binding with enough args for
a complete primop application. since this only shows up when calling a
primop with three or more args it took a while to surface. we have few
builtins that match this: foldl', replaceStrings, and substring. these
are not used incompletely in this manner very often, so this lingered.
fixes#1102
Change-Id: I218dffc14ae876efc86a86c7eb6c895e2405201c
Closes#1064
The culprit here is that `genGraphString` is only invoked with the
store-paths associated with the outputs of the derivation, so when
filling `dependents`, the `graph_data.find(p)` call would return the end
of the iterator when doing this for references to other store-paths.
As a result, the code wrote information behind the graph data-structure
causing a corruption. For me, this resulted in a SIGSEGV most of the
time and in a few cases in an uncaught `map::at`-exception as reported
by Niklas.
This patch changes two aspects of the original implementation:
* When filling `dependents` in the node-set, use `map.at()` instead of
`map.find()->second`. The latter doesn't make any sense and was the
cause of corrupting memory. The `at` would've made it far easier to
spot this in the first place.
* Filter out store-paths that don't belong to a different output of the
derivation when creating `outputGraph`. This variable is used on two
places, `genGraphString` and for topological sorting.
The latter already filters out store-paths from a different drv, so
this is happening now when creating the variable in the first place
such that `genGraphString` never ends up with corrupt data in the
first place. This is the actual bugfix.
Implemented a regression-test for this case to be sure.
Change-Id: Ie02144d89c32b0a776cb1ece0601d0229315ebc3
(cherry picked from commit 0a5f474a25)
Prior to I6a6a6964d2b5ad47ae5ea9eb11af9b6373ce2141 — `sudo nix
upgrade-nix` would perform direct store access.
This ensured a certain number of desireable properties for upgrading the
Lix binary itself.
We re-introduce direct store access for upgrading Lix binaries.
Fixes#1060.
Change-Id: I523c4d3023ed5fe9eff8fde9a266c56a0de47d8c
Signed-off-by: Raito Bezarius <raito@lix.systems>
(cherry picked from commit d2ca1810b1)
typo of aws-cpp-sdk-transfer as aws-cpp-std-transfer prevents linking
against lix
Change-Id: Id847eab2601698696030d31fcd51288aa5f3d274
(cherry picked from commit 06f987fb0c)
This merge commit returns to the previous state prior to the release but leaves the tag in the branch history.
Release created with releng/create_release.xsh
Change-Id: Ida5959473ec469dd5e5e977ebdc8aab6e694b0de
https://github.com/NixOS/nixpkgs/pull/451579 PR
enabled c-aresSupport for curl on darwin which ended up breaking
DNS resolution in FOD sandboxes:
```
nix-run> exporting https://tangled.org/@weethet.bsky.social/nix-run (rev 73d7bf6b58848fb8f42e3a69816e0847f041c689) into /nix/store/m4m951648wmipxgwrgsml9gzjwfpfhm7-nix-run-73d7bf6
nix-run> Initialized empty Git repository in /nix/store/m4m951648wmipxgwrgsml9gzjwfpfhm7-nix-run-73d7bf6/.git/
nix-run> fatal: unable to access 'https://tangled.org/@weethet.bsky.social/nix-run/': Could not resolve host: tangled.org (Could not contact DNS servers)
nix-run> fatal: unable to access 'https://tangled.org/@weethet.bsky.social/nix-run/': Could not resolve host: tangled.org (Could not contact DNS servers)
nix-run> fatal: unable to access 'https://tangled.org/@weethet.bsky.social/nix-run/': Could not resolve host: tangled.org (Could not contact DNS servers)
nix-run> Unable to checkout 73d7bf6b58848fb8f42e3a69816e0847f041c689 from https://tangled.org/@weethet.bsky.social/nix-run.
```
with these sandbox failures:
```
deny mach-lookup com.apple.SystemConfiguration.DNSConfiguration
deny file-read-metadata /private/etc/hosts
deny file-read-data /private/etc/hosts
```
We allow those so that DNS resolution work fine
Change-Id: I9102293691972feb085adf8e9b1ad915bb3a36ab
This adds the enablement code to support HTTP/3 if the user requests it.
We leave it disabled because h3 is not onpar with h2 performance.
Change-Id: I1fd3d4c97b972dcf36bccacc6c9a8290e22b31e0
Signed-off-by: Raito Bezarius <raito@lix.systems>
Channels have moved from `nixos.org/channels` to `cnannels.nixos.org`.
This udates all relevant links (excluding release notes) to use the new
canonical URLs and replaces HTTP with HTTPS.
Fixes#1031.
Change-Id: I212821c44ac5e482c8e9eaa415c7d8ee17ff8341
Signed-off-by: adam <me@adamperkowski.dev>
URL literals were deprecated in 278fddc317,
effectively stabilizing the experimental feature. Enabling it has no effect any
more. Remove it.
Change-Id: I8fbca03b1a2be6a8cddfce644043ac0a6a6a6964
We start this section with shortcomings of unsandboxed builds.
Fixes#1018.
Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: Ieb17e4340beab0c1197951813ae602de453a3fd9
Signed-off-by: Raito Bezarius <raito@lix.systems>
This is unnecessary because Darwin builds already cannot create
`set{u,g}id` files due to the minimal sandbox policy, because we can
forbid messing with the top‐level build directory directly in the
sandbox rules, and because Darwin builds can trivially avoid isolation
through temporary directories right now anyway.
This does regress the build directory isolation for builds with
`sandbox = false`, but I can’t imagine that mattering given the
above. The sandbox change prepares us for a world where we close
off shared temporary directories for `sandbox = true` builds and try
harder to achieve proper isolation on Darwin, but probably doesn’t
have a meaningful security impact one way or another for now.
With this change, we get down to 41 byte build directory paths on
Darwin, a ~2–3½ byte improvement over the old status quo. We can
also restore the 0710 permissions on Linux.
Change-Id: I6a6a6964a681c0365241fe7234831db656b76799
The minimum build directory length on Darwin with default settings
when we were still using `/tmp` was 22 bytes. Deriving build directory
names from my local store, the median and mode were 43, the mean was
around 44½, and the maximum was 127.
The switch to `/nix/var/nix/builds` over `/tmp` added a 15 byte
penalty, and the additional `/b` directory added another 2.
Now that we use opaque build directory names, the length is 48, so
we’re still at a ~3½–5 byte penalty over the previous status
quo. This change brings us down to 43, matching the previous median.
Note that these calculations do not take into account the fact that
`/tmp` is a symbolic link to `/private/tmp` on Darwin. Anything
that was canonicalizing paths would have had an additional 8 byte
penalty in the previous status quo that is not applicable here,
so we may already be ahead even without this change. If the more
opaque directory name here is undesirable, then that factor could
potentially help us squeeze by without. Alternatively, in combination
with dropping the `/b` on Darwin we could use `/nix/var/nix/bld` or
similar, but I feel that the paths in general are sufficiently opaque
that it should be okay to go with the shorter option here. Given that
some projects already had to reduce filename lengths to avoid this
limit even before the recent changes, I think it is best to try and
improve on the previous status quo.
Note that `/nix/var/nix/builds` will unfortunately not be cleaned up
on Darwin. However, we don’t clean up the directories inside it on
Darwin anyway, so hopefully that’s okay for now?
Closes: #913
Change-Id: I6a6a6964bffce7194bcddcaefb4c4a37569c7df5
We always use the default temporary directory, because
`createUniqueDir` has an interface nice enough to use directly for
the few bespoke uses.
Change-Id: I6a6a696450b7c0a0bd76655632fb14d7c5e38199
This makes the paths more nondeterministic, but more reliably unique,
and lets us remove the retry loop.
Note that this adds random entropy to the build directory visible
inside derivations on Darwin and unsandboxed Linux. It was already
non‐deterministic in the presence of concurrent builds and similar,
but now we can reliably expect it to be different every time. On the
whole I think that’s a good thing, as it is impossible to ensure
a single consistent build directory and derivation outputs should
not depend on it.
Package reproducibility isn’t great on Darwin to begin with,
though, and the reproducibility bugs this will turn up in packages
will be more urgent to fix than when the build directory was mostly
consistent. A quick survey of my local store shows that many C, C++,
and Rust binaries contain build directory references, likely due to
use of `__FILE__` and its equivalents; non‐binary offenders include:
* Install logs included in the Rust and Cargo bootstrap compilers
* Example errors in the Rust documentation referencing build paths
* Configuration information installed with CPython itself
* Python 2 metadata from resholve’s closure
* Cython metadata
* Generated headers in Facebook libraries referencing source paths
* Generated CMake files in Facebook libraries referencing source paths
I haven’t built that much in this store since the last GC, so this is
probably only a small sample of the problems across the tree. These are
all instances of <https://reproducible-builds.org/docs/build-path/>,
though, and should probably just be treated as general reproducibility
bugs outside of contexts like the Linux sandbox where we can normalize
them away entirely.
I have implemented away build directory paths for C/C++, applied some
additional fixes for non‐`__FILE__`‐related issues in binaries
from ATF and LLVM, and fixed the derivation bug causing the CPython
3 issue, and will work on upstreaming these changes. Rust is working
on the problem upstream, with some temporary workarounds we can
potentially apply in Nixpkgs for now. The rest will require some
distributed effort.
Change-Id: I6a6a69645b4915c56c0fdef904aa81684e4136c6
This does change the behaviour when the global temporary directory
does not exist, but other uses of the global temporary directory are
already broken in that circumstance, and it should be fixed centrally
if the use case is considered desirable. The logic was not present
before the recent churn around build directories – it was added now
that Lix is taking ownership of the build directory in the store –
so this should not be a meaningful regression.
Change-Id: I6a6a6964e345ea6803226c5ad759e836de7cb0ed
There is now no risk of race conditions on a system with a functioning
entropy source, and the bespoke prefixes are either redundant to the
default or unnecessary.
Change-Id: I6a6a69641211c6bb979ea48ad30aecb1a53d03f0
Relax the constraints on keeping the exact same filename format to
provide a more robust source of entropy with a simpler interface
(as previously suggested by eldritch horrors). Using 128 bits of
OS‐provided entropy ensures global uniqueness and allows us to
skip any thought of gracefully handling the case where these files
already exist.
My microbenchmark that repeatedly constructed paths like this and
printed them out showed that this takes about 1.23× the time of
the previous implementation, both taking on the order of a couple
microseconds for one iteration. Since everything that uses it is doing
things more expensive than printing to standard output, the actual
performance delta is likely to be lost in the noise. If it somehow
becomes a bottleneck, it can be optimized without sacrificing the
guarantees by reading from the system RNG only to seed a thread‐local
CSPRNG like [ChaCha8Rand], but I think that’s very unlikely.
We also tweak the recommended way of creating a temporary file inside
a directory in anticipation of later changes, and rename the `suffix`
parameter to `prefix` (it’s a prefix to the random characters and
a suffix to the root, but this way is more consistent).
[ChaCha8Rand]: https://c2sp.org/chacha8rand
Change-Id: I6a6a69648502c746d13d8c3bd2768cbbf1b90466
base32Encode now takes std::span<std::byte>, with a base32EncodeStr
convenience wrapper which takes std::string_view.
Co-authored-by: Qyriad <qyriad@qyriad.me>
Change-Id: I6a6a6964f799dc84ecbfb55c7ca03a064cff71d9
the change to shareable thunks also removed a few cases of pointer
equality checks that allowed structures containing functions to be
considered equal to other sets containing the same functions, even
if the sets themselves were pointer-equal themselves. *so* busted.
Change-Id: If87fdab658f9037ce2a654f69a9e3da6ae2f53e5
we can't change the protocol to allow daemons to do this, and we should
not try to guess what the `auto` store uri means depending on whether a
command was run by root or not due to copious side effects and not even
being able to tell whether the `auto` store uri was given explicitly or
not. while `auto` may *technically* allow this via its naming we should
resist the urge to add a hack and fix the underlying protocols instead,
especially since repairPath should be a rare, superuser only operation.
fixes#888
Change-Id: I1b53245db226199f827a89a237a2ab9907c3f766
if curl fails during a transfer and said transfer is piped through
libarchive we see very strange errors (#1009). in such cases we'll
want to prioritize reporting the source errors to allow debugging.
Change-Id: I8cc231257eca5b749471ecd38df15071170decc2
The test would fail if the identity cannot be determined
(as is the case e.g. when using auto-allocate-uids under darwin).
Instead we use the uid when the user name is not available.
Fixes: #1025
Signed-off-by: hexchen <hexchen@lilwit.ch>
Change-Id: I12f35840b78f989b5fd09bbd44c833dc6a6a6964
This helps users to debug whether their current profile symlinks are
correctly set.
Expected outputs look like this:
```
❯ sudo ./outputs/out/bin/nix doctor
[snip]
[PASS] All profiles are gcroots.
[PASS] Client protocol matches store protocol.
[INFO] You are trusted by store uri: local
[FAIL] Error: current generation cannot be discovered for profile:
'/nix/var/nix/profiles/default'
```
```
❯ ./outputs/out/bin/nix doctor
[snip]
[PASS] All profiles are gcroots.
[PASS] Client protocol matches store protocol.
[INFO] You are trusted by store uri: daemon
[PASS] You have 28 generations for profile
'/nix/var/nix/profiles/per-user/raito/profile'
The current generation number is '290'
```
Change-Id: I50c69cbeac3291d668f4c2332803411579adc944
Signed-off-by: Raito Bezarius <raito@lix.systems>
Co-authored-by: Qyriad <qyriad@qyriad.me>
Inaccessible-but-extant PATH elements can't be executed anyway, so we
can skip them as if they didn't exist, instead of erroring like before.
Change-Id: I9288c3ecb6768171e62a3088122d98421558eb03
In d2696cdd¹, libstore was made to prioritize NIX_BIN_DIR/nix (over
/proc/self/exe) if it exists. But `pathExists()` throws an exception if
lstat() returns any error other than non-existence.
Because this check happens in libstore's Settings constructor for the
static-initialized global `nix::Settings`, this case would abort Lix
well before main().
[1]: d2696cdd1e
Change-Id: I62e44c8a33b28c234900df3e4790c1badefee358
So far it was impossible to put absolute paths into files, as one
wouldn't know whre the test would be placed.
This commit adds a new Fileish variant called `EnvTemplate` which uses
the given string as a template and replaces the `@ENV_VARIABLE_NAME@`
placeholders with the according value of the environment variable.
This way one can use `@HOME@` or `@TEST_ROOT@` to build absolute paths
Change-Id: I425cb6408dceb8d7f26d136ace4ac98b1ca31ec3
Since we have async, we are unleashed and therefore we started to cause
people to run into system limits like number of open files.
We introduce an async semaphore which uses max(25 % of max open files,
max number of cores) which should lead to 256 maximum concurrent copies.
We believe that the copy operation is the only one that can cause issues
given that the daemon already runs with obscene fd limits.
Fixes#1022.
Change-Id: Iec433d10d5c5003962ea749cf8e32bafb314f0d9
Signed-off-by: Raito Bezarius <raito@lix.systems>
In case of empty messages, it is good to print the raw error code.
Additionally, we print request IDs which can help users to reconcile
what happened with the service provider.
Change-Id: I4d83c011c1b7a5514e3d1b21123df38308279044
Signed-off-by: Raito Bezarius <raito@lix.systems>
Jade Lovelace