Closes#1064
The culprit here is that `genGraphString` is only invoked with the
store-paths associated with the outputs of the derivation, so when
filling `dependents`, the `graph_data.find(p)` call would return the end
of the iterator when doing this for references to other store-paths.
As a result, the code wrote information behind the graph data-structure
causing a corruption. For me, this resulted in a SIGSEGV most of the
time and in a few cases in an uncaught `map::at`-exception as reported
by Niklas.
This patch changes two aspects of the original implementation:
* When filling `dependents` in the node-set, use `map.at()` instead of
`map.find()->second`. The latter doesn't make any sense and was the
cause of corrupting memory. The `at` would've made it far easier to
spot this in the first place.
* Filter out store-paths that don't belong to a different output of the
derivation when creating `outputGraph`. This variable is used on two
places, `genGraphString` and for topological sorting.
The latter already filters out store-paths from a different drv, so
this is happening now when creating the variable in the first place
such that `genGraphString` never ends up with corrupt data in the
first place. This is the actual bugfix.
Implemented a regression-test for this case to be sure.
Change-Id: Ie02144d89c32b0a776cb1ece0601d0229315ebc3
(cherry picked from commit 0a5f474a25)
This is unnecessary because Darwin builds already cannot create
`set{u,g}id` files due to the minimal sandbox policy, because we can
forbid messing with the top‐level build directory directly in the
sandbox rules, and because Darwin builds can trivially avoid isolation
through temporary directories right now anyway.
This does regress the build directory isolation for builds with
`sandbox = false`, but I can’t imagine that mattering given the
above. The sandbox change prepares us for a world where we close
off shared temporary directories for `sandbox = true` builds and try
harder to achieve proper isolation on Darwin, but probably doesn’t
have a meaningful security impact one way or another for now.
With this change, we get down to 41 byte build directory paths on
Darwin, a ~2–3½ byte improvement over the old status quo. We can
also restore the 0710 permissions on Linux.
Change-Id: I6a6a6964a681c0365241fe7234831db656b76799
The test would fail if the identity cannot be determined
(as is the case e.g. when using auto-allocate-uids under darwin).
Instead we use the uid when the user name is not available.
Fixes: #1025
Signed-off-by: hexchen <hexchen@lilwit.ch>
Change-Id: I12f35840b78f989b5fd09bbd44c833dc6a6a6964
libarchive's xz offers single threaded xz compression which is very slow
and provides ~10-20Mbps compression speed in addition to maxing a core.
In exchange, it achieves optimal compression ratios among all our
compression methods.
Nonetheless, xz prevent the saturation of 1Gbps+ connections and slow
down significantly decompression for end users. As these connections and
faster hardware is becoming prevalent for cache servers and clients, we
offer to default to zstd.
Lix is a "compress once, decompress many times" application. To avoid
incurring a high penalty to end users very sensitive to compress ratio
(very slow Internet connections), we dampen the consequences of
switching to zstd by increasing the default zstd level to 12.
On one example, xz will compress a 4.4GB file to 632MB, zstd on 12 will
compress it to 775MB, that is a ~18 % increase over the optimal xz
compression. zstd took 18 seconds to produce this file.
Increasing to level 14 leads to a 773MB file while taking 37s.
Increasing to level 16 leads to 735MB file while taking 66s.
Finally, xz took 77s, so a 50 % reduction in time taken to compress in
exchange of an increase of 18 % of the compressed size.
This change will reduce issues encountered in #945 but is probably not
the root cause.
References:
- https://discourse.nixos.org/t/switch-cache-nixos-org-to-zstd-to-fix-slow-nixos-updates-nix-downloads/23961
Change-Id: I7beda2bf2c1fed146dcb797b8f85dc290c486ab2
Signed-off-by: Raito Bezarius <raito@lix.systems>
Instead of letting inner shell perform the cleanup, we will fork/execvp
and finish the work ourselves. Once an interruption arrives, we wait
again and return the exit status code of the inner shell.
This idea was suggested by eldritch horrors.
Fixes#1020.
Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: Id38af7ac33874aa1c4b7ca9c6123e3cbd8c3cd8c
Signed-off-by: Raito Bezarius <raito@lix.systems>
In the first pass I erred on the cautious side, only migrating safe
bets, to here's some trivial migrations that I missed
Change-Id: I934011919837b0aa491113afdcad603cf6b9cbbb
using sleep(1) as a synchronization mechanism does not work. use fifos instead.
fixes#690 and ci constantly falling over in the same exact fucking source line
Change-Id: I51725f8e439b6753f3212d2897dbb0620ad77a37
Previously two cryptography libraries were linked into Lix: OpenSSL used for
hashing and (in usual configurations) indirectly via curl for TLS, and Sodium
used only for handling the Ed25519 path info signatures. The latter is
functionally redundant since OpenSSL supports the same use case as well.
Reimplement the Ed25519 handling using OpenSSL and drop Sodium.
Fixes: https://git.lix.systems/lix-project/lix/issues/969
Change-Id: I6a6a696456b9d3ad7fdc2bf9b0759836a6247a38
Closes#551
This adds a special accessor that falls back to checking if a store-path
exists within a chroot if it's not a valid path. That way,
`genGraphString` can find out which files have which references before
the outputs are registered.
Change-Id: I03c9d508fa3c72e5c262194461a25d71f3f4de15
it's broken, can write arbitrary file paths when run as root, and only
supports strings and recursive sets of strings. this was only used for
manpage generation in a build system that has not woken up since 1976.
fixes#974fixes#227
Change-Id: I4f18599685a3077c15ddc02c759558f986c8c6e4
See [my comment] on the Nix PR to restore the previous behaviour
for why I believe we should remove this for the next release. The PR
should still be backported to stable releases to avoid making breaking
changes to their semantics.
[my comment]: <https://github.com/NixOS/nix/pull/13741#issuecomment-3180851635>
Fixing this across supported Lix versions is required for Nixpkgs to
update toml11, which is a blocker for the CMake 4 update.
Change-Id: I6a6a69642e6b6cb13a9fccc0778e9158b53102d5
Current test suite doesn't cover the subsecond formatting at
all and toml11 is quite finicky with that. We should at the very
least test its behavior to avoid silent breakages on updates.
(cherry picked from commit 7ed0229d1abd4414144c7af396842462ce6fc1eb)
Upstream-PR: https://github.com/NixOS/nix/pull/13741
Change-Id: I6a6a696433b168072d6ad2585dce8a3c10ccbc39
This is a collection of Lix plugins that showcase how to write one for
various usecases.
The first is a mTLS store plugin that enable mTLS cache URIs
(`https+mtls://`).
We enable meson build system support for this plugin but we are not
going to distribute it in the official packaging of Lix, we will
repackage each relevant plugin downstream in Nixpkgs.
These plugins have *NO* guarantee support, they are provided as useful
references and are possibly production-ready if your usecase is simple
enough.
Reference: https://github.com/NixOS/nix/pull/13030 (this change has
resemblances but our APIs are different, the tests harness is mostly
from CppNix).
Change-Id: Ib354271981b35dff6c134b12c4748c3eaf743fcb
Co-authored-by: Jörg Thalheim <joerg@thalheim.io>
Co-authored-by: László Vaskó <1771332+vlaci@users.noreply.github.com>
Signed-off-by: Raito Bezarius <raito@lix.systems>
This is useful to test extended features in Lix.
Change-Id: Idb2416a080329116677809b883950e6c33028a44
Signed-off-by: Raito Bezarius <raito@lix.systems>
Historically, Nix would support copying certificate authorities inside
the sandbox so you could use them.
In addition to that, the primitives consisting of leaking environment
variables via `impureEnvVars` and `extra-sandbox-paths` to render paths
external to the sandbox visible to the builder would also constitute a
mechanism to expose special inodes which should have no influence on the
output result, e.g. interception CAs.
Unfortunately, in nixpkgs, `lib.fetchers.proxyImpureEnvVars` set
`NIX_SSL_CERT_FILE` as an impure environment variable.
A confused user may set `ssl-cert-file` via `NIX_SSL_CERT_FILE` outside the
builder believing that this will set magically the right
`NIX_SSL_CERT_FILE` inside the sandbox, but this is not true.
The combination of impure environment variables and setting `caFile`
creates a weird interaction where `NIX_SSL_CERT_FILE` points to an
"outside the builder's world" inode *AND* `ssl-cert-file` creates
this very same certificate file in /etc/ssl/certs/ca-certificates.crt
without rewriting the environment variable.
This footgun is closed by making these two features mutually
incompatible with a warning and forcibly rewriting the SSL family of
environment variables even if it was set via impure environment
variables.
Users who truly meant to use `impureEnvVars` can obtain the right
behavior by setting `ssl-cert-file` to an empty string and will have to use
`extra-sandbox-paths`.
Users who meant to use `ssl-cert-file` will have everything work
automatically with a warning hinting at nixpkgs *fixing its own bug*,
i.e. passing `NIX_SSL_CERT_FILE` as an impure environment variable and
expecting the Nix interpreter to magically reconcile the diverging
values or expecting the user to actually do the work to render the path
visible available via `extra-sandbox-paths`.
Fixes#885.
Change-Id: I32f8b5ce20fe9b6a911768114c92f95fc886cc07
Signed-off-by: Raito Bezarius <raito@lix.systems>
Maximilian Bosch