while this does require spawning a thread for every contended lock now
we don't expect performance to be impacted. only build-remote used the
synchronous method, and it only used it to serialize uploads to remote
builders. these uploads are expensive enough to dwarf the thread cost.
Change-Id: Iad0aa0cd738bc96fd06a90d655803dadffa09c47
DerivationGoal::InputStream existed only because we did not have an
error-reporting AsyncInputStream of our own yet. we do have one now
though and can thus delete old code in favor of the generic variant
Change-Id: I01c7c564554f8794bdf54603b239b7a808faeda0
it's effectively unused. one use is a write and a read immediately after
the write, the other use checks whether it's not equal to itself (..wat)
Change-Id: I5f6ce26e75a6bfa500c2e9ac3fc70e8dafc9bd74
Commits 205c59367c and
325e7e1824 introduced real glibc store paths from
current nixpkgs unstable into the source. Since nixpkgs `fetchFromGitea` (and
similar fixed-output derivations) depends on a C library, on x86_64-linux they
will fail with the forbidden reference error:
error: the fixed-output derivation '/nix/store/wnmnj3jzc82y89sfmyicr04kilg8zs2k-source.drv' must not reference store paths but 1 such references were found:
/nix/store/q4wq65gl3r8fy746v9bbwgx4gzn0r2kl-glibc-2.40-66
Falsify the store path to prevent this failure.
Change-Id: I949033567bcad070f9a0a19cefdb33a79222e421
My lix build failed today with this result:
```
lix> [----------] 3 tests from MonitorFdHup
lix> [ RUN ] MonitorFdHup.works
lix> [ OK ] MonitorFdHup.works (0 ms)
lix> [ RUN ] MonitorFdHup.works_with_pipes
lix> stderr:
lix> Using configuration: seed=6402097764877502971
lix> libc++abi: terminating due to uncaught exception of type std::__1::future_error: The state of the promise has already been set.
lix> 4/5 lix:check / libstore-unit-tests OK 1.10s
lix> 5/5 lix:check / libexpr-unit-tests OK 1.12s
lix> Summary of Failures:
lix> 3/5 lix:check / libutil-unit-tests FAIL 0.93s killed by signal 6 SIGABRT
lix> Ok: 4
lix> Fail: 1
lix> Full log written to /nix/var/nix/builds/nix-build-lix-2.94.0-dev-pre20250711-65ef28d.drv-0/source/build/meson-logs/testlog.txt
```
I had a response best described as "wtf". I think the cause of this
problem is that there's a race condition with the test in which the loop
gets gone around again a second time because it's triggered by the
terminate fd (and I guess the flags remained what they were before?
seems reasonable), and this is probably racing with the quit atomic
being first to break out of the loop.
I don't know how many hundreds of lixes I've compiled without my test
failing, but this is definitely a bug. I don't think this affects actual
usage as the only impact is repeat delivery of Ctrl-C which is harmless
and which users do regularly.
Change-Id: I60da81d4ac2e79052cd323b5171f9d8bd0aa6783
this was a mess. ssh:// remotes used the extra static fds for build
logs, ssh-ng:// remotes did not. ssh-ng remotes did not use them at
all since ssh-ng never redirected them to begin with. we now create
pipes dynamically and only for ssh:// builders, then translate logs
received over these pipes into the same format used by ssh-ng. this
requires a new activity we did not have before, but since we have a
great many activities that rarely show up already this shouldn't be
a problem for external tooling. if anything external tools can tell
what's going on much better now (at least for ssh:// remote builds)
Change-Id: I02010cee45598362a947faa3a5b04800d39daa31
Avoids incorrect behavior with large integers in `elemAt`, `substring`,
`genList`, etc, which results into crashing the Lix interpreter.
At the same time, unit tests were added for these edge cases with 2^32
as an argument of these primops.
Port of https://github.com/NixOS/nix/pull/13309.
Prior art in https://github.com/NixOS/nix/pull/7222 (forgotten by the original project…).
Change-Id: I1c43ed64f26bcb60e51869e11a74e5de2b7db53a
Co-authored-by: Raito Bezarius <raito@lix.systems>
Signed-off-by: Raito Bezarius <raito@lix.systems>
async queries easily lead to high contention on the localstore sqlite
lock. optimizing the lock wakeup scheme improves query performance by
a linear factor (with the O(waiters) wakeup replaced by O(1) wakeup).
on 100k drv closures we're now at 55s query, down from >8min in 2.93.
Change-Id: I9b96e792c4518a782c690dea92e61260f08f0bad
That way it's easier to spot whether a node is the "final" node in the
graph which is especially helpful for larger graphs.
Change-Id: I460a699f07f5455917792599f4247ebf8f430d93
Closes#334Closes#626
This is loosely based on upstream PR#10877[1], but heavily changed to
use the graph logic from `nix why-depends`.
`precise` is `false` here since the out-path of the drv being built
isn't registered yet, so the path accessor cannot scan through files
yet.
Example output (from an openssh build with `pcsclite.lib` & `glibc` in
`disallowedRequisites`):
error: output '/nix/store/hr8lmmjmd1jk6s3p5ymggyk4am7n2lmb-openssh-10.0p2' is not allowed to refer to the following paths:
/nix/store/p6r5awz3ywrz66symnrn0xb85xzmcysf-pcsclite-2.3.0-lib
/nix/store/q4wq65gl3r8fy746v9bbwgx4gzn0r2kl-glibc-2.40-66
Shown below are chains that lead to the forbidden path(s).
/nix/store/hr8lmmjmd1jk6s3p5ymggyk4am7n2lmb-openssh-10.0p2
└───/nix/store/ys91ywnwikm14xznwk3cdbprapv2m37z-libfido2-1.16.0
└───/nix/store/p6r5awz3ywrz66symnrn0xb85xzmcysf-pcsclite-2.3.0-lib
/nix/store/hr8lmmjmd1jk6s3p5ymggyk4am7n2lmb-openssh-10.0p2
├───/nix/store/q4wq65gl3r8fy746v9bbwgx4gzn0r2kl-glibc-2.40-66
├───/nix/store/6r4zqb04fq5l5l4zghq76wvcpz7dwd35-linux-pam-1.6.1
│ ├───/nix/store/q4wq65gl3r8fy746v9bbwgx4gzn0r2kl-glibc-2.40-66
[...]
[1] https://github.com/NixOS/nix/pull/10877
Co-authored-by: Robert Hensing <robert@roberthensing.nl>
Change-Id: Ib30024c0d9e45c1160bf0134f7d3ba17dbdeff47
While working on the LocalDerivationGoal code, I realized that this
attribute is only set to `false`/`true` depending on whether
`__structuredAttrs` is `true`/`false`.
Change-Id: I53868cd32cedd7e25cb6233bd93bc01111b56a07
This will be useful for other things as well such as the
disallowedRequisites error in the builder code. Additionally, print the
dependencyPath in the tree bold to spot where a change terminates.
Also implemented some unit-tests for this code.
Change-Id: I8460f3f6c5095d5bfbe390f223bc0252800dca5e
The Node struct should become an implementation detail when moving this
into libstore. A map from a node to its direct references is more
intuitive here.
Change-Id: I9fddce6b398b8bb97834e5586bee72b244885fdd
* Better name for refs
* Use std::optional<T> for distance
Suggested-by: eldritch horrors <pennae@lix.systems>
Change-Id: Ie35c3f2a7ea1a90ce3a9807025d0af9ea73e2403
Instead of logging directly, we now write into a `Strings` set that is
referenced by the caller.
While at it, added a test-case to ensure that self-reference invocations
and --all behave properly.
Change-Id: Ib183ab8e8e90436300e1c870fb3ae8f18730abbf
That way we get a line of output per test completed,
which makes it more obvious it's actually doing things.
Change-Id: Ifbbe8bdf64e7178d3c59349cf071eb5a9d0fcd32
staging-next banned !structuredAttrs && separateDebugInfo && disallowedRequisites
due to weird output interactions. Enable structuredAttrs so we can build again.
Also, fix type confusion that makes stdenv explode (https://github.com/NixOS/nixpkgs/issues/422989).
Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: Ic0c773394ee79e10d427f27750d59892d6d1f1d1
this partially reverts commit 0cc021ee15,
which for some reason is completely broken on darwin: there seems to be
no way to receive process-directed signals on a non-main thread. trying
to do it anyway will fail silently. since we only ever used kj for this
to get signal handling timeouts on darwin (which lacks sigtimedwait) to
print a nice message about retrying ^C again we can work around this by
moving the message printing into a fresh, unrelated, non-signal thread.
Change-Id: I5939c6ec62a7e1dc1b3f16067f77277533949fa0
We missed xokdvium being author on cl/3300. This is something we
absolutely want to avoid.
We credit xokdvium in the RL and add a note on this problem.
Thanks to xokdvium for reaching out in private to us so we can repair
this mistake.
Change-Id: I094d0f95b6647104621d6b228e69a4529a300304
Signed-off-by: Raito Bezarius <raito@lix.systems>
many a cleanup path has been broken by interruptions being thrown every
time checkInterrupt is called. we should only throw *once* though; more
than one Interrupted exception for the same event is not only confusing
but also breaks all cleanup paths at the first checkInterrupt call site
(e.g. #900, the cgroup cleanup saga, temp dirs not being removed, etc).
Change-Id: Ibfabf7f6af6ac2b78ad93582c254bbc48fcb3073
we must be crash-safe *anyway*, and being unable to interrupt lix if it
gets stuck somewhere that never calls checkInterrupt is really annoying
Change-Id: I7c40271c3da7e69d8735e22b7b7c4751b5306ab6
macos doesn't have sigtimedwait and we need signal wait timeouts in
order to print a "please hit ^C again" message with a bit of delay.
Change-Id: If574fb1a9de0b19975b34fc63662b089eaedc9d2
another checkInterrupt can be a makeInterruptible wrapper now. this is
also necessary to add a second daemon socket for the new rpc protocol.
Change-Id: I55055f975335a75708f1f73edb75f7bfe77a5938
it was only needed because we forked subdaemons and couldn't reuse the
main aio root. we now fork+exec, so the main aio root is always valid.
Change-Id: Ia19e20d52d65fe72721292be091f182a8a77a7cb
all uses are DoSignalSave::Save now, and introducing new DontSave uses
should be avoided as much as possible. process management is already a
mess, simplifying it somewhat will make our life easier in the future.
Change-Id: I77eecabe45bee9de18fba0dfc948403d3ce46dfe
this resolves problems with aio roots becoming invalid after fork (which
so far forced us to run the daemon loop in an aio-rootless thread), does
not require restarting the signal handler thread in the subdaemon (since
we no longer lose it), and is a step towards solving #18 (with transient
daemons doing the store manipulation started transparently when needed).
Change-Id: Iad0149cbc807e31964407c9a83d12314702c8122
posix_spawn unsets CLOEXEC for fds that are dup'd onto their existing fd
number. this is very useful when inheriting fd numbers exceeding stderr.
Change-Id: I6f14585d424ded6741fdd087f0c4d33a05936bcc
the `from`/`to` naming only made sense for unidirectional output fds,
for others (and for the dup2 api in general) it was backwards. rename
them to `dup`/`from` to make this look more like the assignment it is
Change-Id: Iee50d06f9cfcea765ace6cfbe85b192829207e5f
writing to non-blocking fds happens during remote builds due to the way
file descriptions are shared between processes. we can either poll when
writing to non-blocking fds are reset fd flags. polling is just easier.
unfortunately there is no reasonable way to test this that isn't flaky.
fixes#896
Change-Id: I1d8666df57da97199247f0770c547d0180f6ce07
This results in anything that uses nixpkgs getting stopped in the
debugger inside of nixpkgs internals, which are usually irrelevant.
Let's default to the more useful option.
Fixes: https://git.lix.systems/lix-project/lix/issues/666
Change-Id: If4b94a3d488bfb2f634ee5a2bc195e7a4b5434a5
cancelling the promise returned by makeInterruptible could free the
fulfiller before the interrupt callback handle, and no order of the
attachments made a difference. we must resort to putting fulfillers
into shared_ptrs so we can capture them in interrupt callbacks now.
(alternatively we could add another kind of interrupt callback, but
the complexity of doing that outweighs the cost of one shared_ptr.)
fixes#895
Change-Id: I008b160482fd4d81a29d7e9e452dcda858b090b9
download progress reports send a STDERR_RESULT frame. many concurrent
downloads send many STDERR_RESULT frames. each of these frames has us
run the report loop once. since many frames can happen in very little
time we may receive many frames in a single read from the socket, and
that in turn means we don't have to fcntl that socket on every round.
we must still ensure that the socket is in the correct state for each
part of the loop, and this does mean we may run two unnecessary fcntl
sequences per processStderr call. that's a small price to pay though.
Change-Id: I7af607d8c759b76aff0f6016435955e2f9456923
these are used often enough that deduplicating them is worth it. we do
lose some error fidelity, but valid fds will never cause an error here
Change-Id: I2b91b4848f546a894a2a6c2d36c32a892fb73c9f
it's only called by verifyStore, and verifyStore is only called by the
daemon and `nix-store --verify`. both pass the promise to `blockOn()`.
Change-Id: I829c0d189fa913cd8566ddd1a578c50e60fb2ddb
all of them block on a promise very soon after starting. only
queryValidPaths needs to make sure not to swallow Interrupted
exceptions to exit quickly instead of trying all paths first.
Change-Id: I4f99f5d75d7057bad109dc0131aa58e84275e362
checkInterrupt is cheap, waiting for a promise isn't. checking for
interruptions before any top-level promise is awaited lets us drop
a bunch of checkInterrupt calls elsewhere, such as in thread pools
Change-Id: Id543edf9411e53b2a5bbec77d3084a8f65aaea46
eldritch horrors