Avoids incorrect behavior with large integers in `elemAt`, `substring`,
`genList`, etc, which results into crashing the Lix interpreter.
At the same time, unit tests were added for these edge cases with 2^32
as an argument of these primops.
Port of https://github.com/NixOS/nix/pull/13309.
Prior art in https://github.com/NixOS/nix/pull/7222 (forgotten by the original project…).
Change-Id: I1c43ed64f26bcb60e51869e11a74e5de2b7db53a
Co-authored-by: Raito Bezarius <raito@lix.systems>
Signed-off-by: Raito Bezarius <raito@lix.systems>
async queries easily lead to high contention on the localstore sqlite
lock. optimizing the lock wakeup scheme improves query performance by
a linear factor (with the O(waiters) wakeup replaced by O(1) wakeup).
on 100k drv closures we're now at 55s query, down from >8min in 2.93.
Change-Id: I9b96e792c4518a782c690dea92e61260f08f0bad
That way it's easier to spot whether a node is the "final" node in the
graph which is especially helpful for larger graphs.
Change-Id: I460a699f07f5455917792599f4247ebf8f430d93
Closes#334Closes#626
This is loosely based on upstream PR#10877[1], but heavily changed to
use the graph logic from `nix why-depends`.
`precise` is `false` here since the out-path of the drv being built
isn't registered yet, so the path accessor cannot scan through files
yet.
Example output (from an openssh build with `pcsclite.lib` & `glibc` in
`disallowedRequisites`):
error: output '/nix/store/hr8lmmjmd1jk6s3p5ymggyk4am7n2lmb-openssh-10.0p2' is not allowed to refer to the following paths:
/nix/store/p6r5awz3ywrz66symnrn0xb85xzmcysf-pcsclite-2.3.0-lib
/nix/store/q4wq65gl3r8fy746v9bbwgx4gzn0r2kl-glibc-2.40-66
Shown below are chains that lead to the forbidden path(s).
/nix/store/hr8lmmjmd1jk6s3p5ymggyk4am7n2lmb-openssh-10.0p2
└───/nix/store/ys91ywnwikm14xznwk3cdbprapv2m37z-libfido2-1.16.0
└───/nix/store/p6r5awz3ywrz66symnrn0xb85xzmcysf-pcsclite-2.3.0-lib
/nix/store/hr8lmmjmd1jk6s3p5ymggyk4am7n2lmb-openssh-10.0p2
├───/nix/store/q4wq65gl3r8fy746v9bbwgx4gzn0r2kl-glibc-2.40-66
├───/nix/store/6r4zqb04fq5l5l4zghq76wvcpz7dwd35-linux-pam-1.6.1
│ ├───/nix/store/q4wq65gl3r8fy746v9bbwgx4gzn0r2kl-glibc-2.40-66
[...]
[1] https://github.com/NixOS/nix/pull/10877
Co-authored-by: Robert Hensing <robert@roberthensing.nl>
Change-Id: Ib30024c0d9e45c1160bf0134f7d3ba17dbdeff47
While working on the LocalDerivationGoal code, I realized that this
attribute is only set to `false`/`true` depending on whether
`__structuredAttrs` is `true`/`false`.
Change-Id: I53868cd32cedd7e25cb6233bd93bc01111b56a07
This will be useful for other things as well such as the
disallowedRequisites error in the builder code. Additionally, print the
dependencyPath in the tree bold to spot where a change terminates.
Also implemented some unit-tests for this code.
Change-Id: I8460f3f6c5095d5bfbe390f223bc0252800dca5e
The Node struct should become an implementation detail when moving this
into libstore. A map from a node to its direct references is more
intuitive here.
Change-Id: I9fddce6b398b8bb97834e5586bee72b244885fdd
* Better name for refs
* Use std::optional<T> for distance
Suggested-by: eldritch horrors <pennae@lix.systems>
Change-Id: Ie35c3f2a7ea1a90ce3a9807025d0af9ea73e2403
Instead of logging directly, we now write into a `Strings` set that is
referenced by the caller.
While at it, added a test-case to ensure that self-reference invocations
and --all behave properly.
Change-Id: Ib183ab8e8e90436300e1c870fb3ae8f18730abbf
That way we get a line of output per test completed,
which makes it more obvious it's actually doing things.
Change-Id: Ifbbe8bdf64e7178d3c59349cf071eb5a9d0fcd32
staging-next banned !structuredAttrs && separateDebugInfo && disallowedRequisites
due to weird output interactions. Enable structuredAttrs so we can build again.
Also, fix type confusion that makes stdenv explode (https://github.com/NixOS/nixpkgs/issues/422989).
Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: Ic0c773394ee79e10d427f27750d59892d6d1f1d1
this partially reverts commit 0cc021ee15,
which for some reason is completely broken on darwin: there seems to be
no way to receive process-directed signals on a non-main thread. trying
to do it anyway will fail silently. since we only ever used kj for this
to get signal handling timeouts on darwin (which lacks sigtimedwait) to
print a nice message about retrying ^C again we can work around this by
moving the message printing into a fresh, unrelated, non-signal thread.
Change-Id: I5939c6ec62a7e1dc1b3f16067f77277533949fa0
We missed xokdvium being author on cl/3300. This is something we
absolutely want to avoid.
We credit xokdvium in the RL and add a note on this problem.
Thanks to xokdvium for reaching out in private to us so we can repair
this mistake.
Change-Id: I094d0f95b6647104621d6b228e69a4529a300304
Signed-off-by: Raito Bezarius <raito@lix.systems>
many a cleanup path has been broken by interruptions being thrown every
time checkInterrupt is called. we should only throw *once* though; more
than one Interrupted exception for the same event is not only confusing
but also breaks all cleanup paths at the first checkInterrupt call site
(e.g. #900, the cgroup cleanup saga, temp dirs not being removed, etc).
Change-Id: Ibfabf7f6af6ac2b78ad93582c254bbc48fcb3073
we must be crash-safe *anyway*, and being unable to interrupt lix if it
gets stuck somewhere that never calls checkInterrupt is really annoying
Change-Id: I7c40271c3da7e69d8735e22b7b7c4751b5306ab6
macos doesn't have sigtimedwait and we need signal wait timeouts in
order to print a "please hit ^C again" message with a bit of delay.
Change-Id: If574fb1a9de0b19975b34fc63662b089eaedc9d2
another checkInterrupt can be a makeInterruptible wrapper now. this is
also necessary to add a second daemon socket for the new rpc protocol.
Change-Id: I55055f975335a75708f1f73edb75f7bfe77a5938
it was only needed because we forked subdaemons and couldn't reuse the
main aio root. we now fork+exec, so the main aio root is always valid.
Change-Id: Ia19e20d52d65fe72721292be091f182a8a77a7cb
all uses are DoSignalSave::Save now, and introducing new DontSave uses
should be avoided as much as possible. process management is already a
mess, simplifying it somewhat will make our life easier in the future.
Change-Id: I77eecabe45bee9de18fba0dfc948403d3ce46dfe
this resolves problems with aio roots becoming invalid after fork (which
so far forced us to run the daemon loop in an aio-rootless thread), does
not require restarting the signal handler thread in the subdaemon (since
we no longer lose it), and is a step towards solving #18 (with transient
daemons doing the store manipulation started transparently when needed).
Change-Id: Iad0149cbc807e31964407c9a83d12314702c8122
posix_spawn unsets CLOEXEC for fds that are dup'd onto their existing fd
number. this is very useful when inheriting fd numbers exceeding stderr.
Change-Id: I6f14585d424ded6741fdd087f0c4d33a05936bcc
the `from`/`to` naming only made sense for unidirectional output fds,
for others (and for the dup2 api in general) it was backwards. rename
them to `dup`/`from` to make this look more like the assignment it is
Change-Id: Iee50d06f9cfcea765ace6cfbe85b192829207e5f
writing to non-blocking fds happens during remote builds due to the way
file descriptions are shared between processes. we can either poll when
writing to non-blocking fds are reset fd flags. polling is just easier.
unfortunately there is no reasonable way to test this that isn't flaky.
fixes#896
Change-Id: I1d8666df57da97199247f0770c547d0180f6ce07
This results in anything that uses nixpkgs getting stopped in the
debugger inside of nixpkgs internals, which are usually irrelevant.
Let's default to the more useful option.
Fixes: https://git.lix.systems/lix-project/lix/issues/666
Change-Id: If4b94a3d488bfb2f634ee5a2bc195e7a4b5434a5
cancelling the promise returned by makeInterruptible could free the
fulfiller before the interrupt callback handle, and no order of the
attachments made a difference. we must resort to putting fulfillers
into shared_ptrs so we can capture them in interrupt callbacks now.
(alternatively we could add another kind of interrupt callback, but
the complexity of doing that outweighs the cost of one shared_ptr.)
fixes#895
Change-Id: I008b160482fd4d81a29d7e9e452dcda858b090b9
download progress reports send a STDERR_RESULT frame. many concurrent
downloads send many STDERR_RESULT frames. each of these frames has us
run the report loop once. since many frames can happen in very little
time we may receive many frames in a single read from the socket, and
that in turn means we don't have to fcntl that socket on every round.
we must still ensure that the socket is in the correct state for each
part of the loop, and this does mean we may run two unnecessary fcntl
sequences per processStderr call. that's a small price to pay though.
Change-Id: I7af607d8c759b76aff0f6016435955e2f9456923
these are used often enough that deduplicating them is worth it. we do
lose some error fidelity, but valid fds will never cause an error here
Change-Id: I2b91b4848f546a894a2a6c2d36c32a892fb73c9f
it's only called by verifyStore, and verifyStore is only called by the
daemon and `nix-store --verify`. both pass the promise to `blockOn()`.
Change-Id: I829c0d189fa913cd8566ddd1a578c50e60fb2ddb
all of them block on a promise very soon after starting. only
queryValidPaths needs to make sure not to swallow Interrupted
exceptions to exit quickly instead of trying all paths first.
Change-Id: I4f99f5d75d7057bad109dc0131aa58e84275e362
checkInterrupt is cheap, waiting for a promise isn't. checking for
interruptions before any top-level promise is awaited lets us drop
a bunch of checkInterrupt calls elsewhere, such as in thread pools
Change-Id: Id543edf9411e53b2a5bbec77d3084a8f65aaea46
This reverts commit a0a00948df
because this was insufficient to fix the critical correctness bugs.
Change-Id: I6c7b560ebeebacbbbcc1cbf26e6ef50c38b84f7f
This reverts commit e356d54d7a
because this was insufficient to fix the critical correctness bugs.
Change-Id: I91c3e368ffd13ade6a3cebbbacdb42655796ea56
We’re already allowing `/tmp` anyway, so this should be harmless,
and it fixes a regression in the default configuration caused by
moving the build directories out of `temp-dir`. (For instance, that
broke the Lix `guessOrInventPath.sockets` test.)
Note that removing `/tmp` breaks quite a few builds, so although it may
be a good idea in general it would require work on the Nixpkgs side.
Fixes: 749afbbe99
Change-Id: I6a6a69645f429bc50d4cb24283feda3d3091f534
Using `AllowDaemon::Disallow` here broke `ssh-ng://` remote builds in
multi‐user setups where the remote builder user does not have write
access to the store, now that the automatic store selection logic
has changed. Switch to the default behaviour for this path to fix that.
This causes `ssh-ng://` builds to use the daemon by default on the
remote end, even as `root`. I think this is desirable, as the previous
change already made `ssh://` behave this way, and the pitfalls of
local stores apply to remote builds too. For instance, there were
persistent `ulimit` issues on the NixOS Hydra macOS builders that were
resolved by forcing use of the daemon, and I believe the Linux builders
also go through the daemon these days due to using non‐`root` SSH
users. I believe that the `root` vs. non‐`root` difference is just
as confusing for remote builds as it is for local ones.
`ssh-ng://root@builder?remote-store=local` can be used to revert back
to the previous default if necessary.
Closes: #884
Fixes: 9a59106c17
Change-Id: I6a6a696410f46cd3f2f5a94073ea924ad45dc99c
This allows other functions to parameterize over it themselves. An
enum class is used to avoid API misuse.
Change-Id: I6a6a6964d2b5ad47ae5ea9eb11af9b6373ce2141
Under macOS, the first level of directory has actually mode 0755 instead
of 0700 as macOS often do not possess the right primitives to chroot
inside of these directories, leading to
https://github.com/NixOS/nix/pull/11031.
Thanks to Emily for the heads-up on this type of matter.
Change-Id: I9d4e53717f61c9d573ff176f820610612804fbc3
Signed-off-by: Raito Bezarius <raito@lix.systems>
eldritch horrors