this is used by nix-env and copyPaths, which in turn is used to upload
to binary caches. for a large path set we have seen 10x a improvement.
Change-Id: Ieadd0e66180e5ceecefaf944a5bb2f0523374954
notably this also includes the symbol table because it stores real
strings that are referenced by eval values, and an upcoming change
will make it impossible to share those strings with value strings.
Change-Id: I20a3644db8aa0850efe29630e0b73d424cb2aa56
use a thread pool and allow only buffered sources for performance. we
may want to use this code path unconditionally due to gc interactions
of fibers in circumstances we should not even be able to trigger, but
if that becomes important we will have a useful implementation ready.
Change-Id: Ib4e1531fe920847d8e30a42e8df393ace549f52e
this means both the worker protocol and the serve protocol, i.e.
ssh-ng/local connections and legacy ssh connections. now we have
no blocking reads left anywhere in our client store connections.
Change-Id: I2f628d4d2e71ef0a7006918f175192f3f58eea95
shoving a nar dump directly into a framed sink created a bunch of tiny
frames, each of which requires at least two syscalls to read. this can
lead to immense performance loss when using a daemon; we have seen 15%
in benchmarks on main and even more with async code involved ... oops.
Change-Id: I8529506e3de74d92834d1f4ee228dcaf32eb756c
this lets us avoid a bunch of fcntl calls to unset and re-set O_NONBLOCK
on remote store connections. the overhead of these isn't high, but doing
it is still wasteful and a maintenance burden when we have async readers
Change-Id: I900cdca2a16202380c8b6f9b86da7d9b0f1e34ac
the former isn't even int, it's unsigned. the latter is uint64. both
should be explicit about this to avoid problems that we have already
had, such as the FramedSource wire protocol writing using 64 bits to
write frame sizes, but 32 bits to read them. large frames will cause
the reading end to crash with an unnecessary serialization exception
Change-Id: I3c15e911f649eec719d4b1c135dde1b6ba020271
they will not work well with async deserialization and are not used
consistently anyway. just like the serializing operator<< these are
protocol stability hazards: changing the type of a field influences
the wire protocol layout and type constraints, which is not amazing
Change-Id: I54b20a133048f4ca15a9fb0f4d8b94dc78f62d89
this is an equivalent of the regular kj read interface which also takes
a min/max pair. we do not need this very often though, so we'll keep it
as a separate method for now. if we do find we use it more than read we
can still rename read to readSingle and readRange to read. we will see.
Change-Id: Ib04ca146911adae7081cf4b2df097217ea5fe9f8
this should've been a filter from the start. since filter support in the
old IO model is very bad we just move it into the only use of it instead
Change-Id: Ifb9cfecf07587ae1d2d55072ddf505c86c79cc1b
we should not encourage this kind of framing. in the future we will have
to do this on async streams, which we *absolutely* should not encourage.
Change-Id: Ib89e144afb564284db64fc7367cba7fffc18fdaf
the exception no longer actually matters, only that the download stream
is destroyed before the download finishes does. exceptions during drain
calls on the returned stream will cause the stream to be destroyed, but
draining this stream is just a remnant of the old download-to-sink API.
Change-Id: Ic6de40deb2ccff09d77180148afe746f29f55d23
we don't need to report progress for every read call. that's way too
much. batching like this greatly reduces CPU usage for copies out of
or into remote buidlers due to likewise greatly reduced log traffic.
Change-Id: I3db2b2ab113fbaadefc69cfde6f977fb0c6cd5ad
PushActivity does not work with async code since we have no such thing
as promise-local storage. it will be confusing at best, and completely
wrong at worst, with the current thread-local linking state. if we can
find a way to get promise-local storage we may want to bring this back
though, explicit context passing is rather error-prone. luckily we are
not using parent links for anything important, just to keep the multi-
line activity display from filling up with stuff we're already showing
Change-Id: Ie373d713080a3db811b2d5abd681f78137735e45
checking that the remote build actually succeeded only implied-trusted
remotes or CA derivations makes *absolutely* no sense. we should check
that builds have succeeded before trying to copy them from the remote.
Change-Id: Ib2cf216c580f4c577dd9fef8849acc033ae082b9
capnp does not handle fd passing correctly in all circumstances. we hit
such cirumstances when passing large closures path lists to build-hook.
since capnp seems to ignore fds passed in non-final segments of any rpc
message we just ensure that the capability including the log fd will be
small enough to not be fragmented on the receiving side of the channel.
cf https://github.com/capnproto/capnproto/issues/2359
Change-Id: Id22309264936b3a57bcc68a0753c3bfb3c9a43d2
remote builds failures used to be signaled via exit status 1 of the
build hook, which in turn only happened because the build errors we
got from remote stores was thrown and bubbled up to main which then
logged the error and exited with code 1. with rpc we cannot do this
any more. barring a rewrite of the worker infra to allow for errors
being reported with something other than process exit codes this is
the best can do. ideally we would wrap remote builds in a new goal.
(and then remove all exit code shenanigans from DerivationGoal too)
fixes#928
Change-Id: Idc3ede3cbaca34c8c8e40247da52794f2a5013b9
drop our reimplementation of splice for non-linux in favor of using kj
pumpTo. this avoids select() for its O(maxfd) behavior, and if kj ever
uses something more efficient than read/write loops we'll benefit too.
Change-Id: Id01ba84bf8831455af2d9755bf1a3039d215bb47
libarchive *should* not break with 0710 on the tmpdir root on darwin,
just like it doesn't break on linux, but for some reason it does. the
restriction to 0710 can be weakened to 0750 with causing any trouble.
fixes#921
Change-Id: Ia9fc2f8eb9695fc19cefae9857368d5a4e58c8b9
previously we only had one build hook in waiting at most because build
hook rpc was synchronous. now that it no longer is we attempt to start
one hook per derivation, which depending on scheduling can be a *very*
large number. restrict the waiting hook count to 4 to some concurrency
without collecting a large number of hooks that may never do anything.
Change-Id: Ic0b1125cec4acd69e8a0d4639c232e71b825e01d
although we only chown if the build was requested by a local daemon
user. daemonless invocations will not chown as they do not have to.
remote builds *can* chown to the remote builder user, but that does
not seem to happen (for some reason keep-failed is not propagated).
Change-Id: Ic0ead406b38b4ca0556fec42d84888efa25123bf
this makes the actual build directories used by builders invisible and
inaccessible to other processes on the system, avoiding another vector
for outside processes to interfere with builds or pass credentials the
build sandbox should not have access to into the build sandbox anyway.
fixes#919
Change-Id: Ifaa4d8e3940cfde1406e925f75c1375d2e86d81a
this touches both libutil and libstore because with no rpc users it
doesn't make that much sense to separate the two. note that all our
strings are represented as Data (ie, blobs) because capnp Text must
be nul-terminated. while it's technically possible to use Text with
strings containing non-terminating NULs it is a bit of a hassle and
could lead to rpc users erroneously stopping at the first NUL byte.
Change-Id: I4c75e03b79a226ffa8d7cd985e3ac632a0cd7c1c
we need this to generate dependency information, and it'll be the entry
point for custom codegen once we need it. a wrapper also makes it a lot
easier to generate a whole namespace's worth of rpc definitions at once
Change-Id: Iba7a1c92a8a40bede9ed71aa3ab455477ff5e568
if the hook accepts the build request we can handle the entire request
in tryBuildHook. there is no need to punt a partially handled build to
the caller (we only did this to minimize churn during asyncification).
Change-Id: Iec3e35a8103da4fc5fbef394cc28a134ee62a198
mapping the result of an await operation before unpacking it lets us
inject rpc type conversion functions without duplicating all that is
needed for proper exception wrapping and async error traces support.
Change-Id: Ibcba1cc6d2b275757e3475881ef20f95dd4d684f
`Outcome<void, T>` and `Result<std::optional<T>>` can be interpreted as
being the same thing, but the latter is easier to use: not only do they
allow TRY_AWAIT usage for their promises, we also don't have the error/
exception confusion of outcomes (where the T above is the "error" type)
Change-Id: I92c9241481cecc97e2992445b3dced53c82a2524
while this does require spawning a thread for every contended lock now
we don't expect performance to be impacted. only build-remote used the
synchronous method, and it only used it to serialize uploads to remote
builders. these uploads are expensive enough to dwarf the thread cost.
Change-Id: Iad0aa0cd738bc96fd06a90d655803dadffa09c47
DerivationGoal::InputStream existed only because we did not have an
error-reporting AsyncInputStream of our own yet. we do have one now
though and can thus delete old code in favor of the generic variant
Change-Id: I01c7c564554f8794bdf54603b239b7a808faeda0
it's effectively unused. one use is a write and a read immediately after
the write, the other use checks whether it's not equal to itself (..wat)
Change-Id: I5f6ce26e75a6bfa500c2e9ac3fc70e8dafc9bd74
this was a mess. ssh:// remotes used the extra static fds for build
logs, ssh-ng:// remotes did not. ssh-ng remotes did not use them at
all since ssh-ng never redirected them to begin with. we now create
pipes dynamically and only for ssh:// builders, then translate logs
received over these pipes into the same format used by ssh-ng. this
requires a new activity we did not have before, but since we have a
great many activities that rarely show up already this shouldn't be
a problem for external tooling. if anything external tools can tell
what's going on much better now (at least for ssh:// remote builds)
Change-Id: I02010cee45598362a947faa3a5b04800d39daa31
async queries easily lead to high contention on the localstore sqlite
lock. optimizing the lock wakeup scheme improves query performance by
a linear factor (with the O(waiters) wakeup replaced by O(1) wakeup).
on 100k drv closures we're now at 55s query, down from >8min in 2.93.
Change-Id: I9b96e792c4518a782c690dea92e61260f08f0bad
this partially reverts commit 0cc021ee15,
which for some reason is completely broken on darwin: there seems to be
no way to receive process-directed signals on a non-main thread. trying
to do it anyway will fail silently. since we only ever used kj for this
to get signal handling timeouts on darwin (which lacks sigtimedwait) to
print a nice message about retrying ^C again we can work around this by
moving the message printing into a fresh, unrelated, non-signal thread.
Change-Id: I5939c6ec62a7e1dc1b3f16067f77277533949fa0
many a cleanup path has been broken by interruptions being thrown every
time checkInterrupt is called. we should only throw *once* though; more
than one Interrupted exception for the same event is not only confusing
but also breaks all cleanup paths at the first checkInterrupt call site
(e.g. #900, the cgroup cleanup saga, temp dirs not being removed, etc).
Change-Id: Ibfabf7f6af6ac2b78ad93582c254bbc48fcb3073
we must be crash-safe *anyway*, and being unable to interrupt lix if it
gets stuck somewhere that never calls checkInterrupt is really annoying
Change-Id: I7c40271c3da7e69d8735e22b7b7c4751b5306ab6
macos doesn't have sigtimedwait and we need signal wait timeouts in
order to print a "please hit ^C again" message with a bit of delay.
Change-Id: If574fb1a9de0b19975b34fc63662b089eaedc9d2
another checkInterrupt can be a makeInterruptible wrapper now. this is
also necessary to add a second daemon socket for the new rpc protocol.
Change-Id: I55055f975335a75708f1f73edb75f7bfe77a5938
it was only needed because we forked subdaemons and couldn't reuse the
main aio root. we now fork+exec, so the main aio root is always valid.
Change-Id: Ia19e20d52d65fe72721292be091f182a8a77a7cb
all uses are DoSignalSave::Save now, and introducing new DontSave uses
should be avoided as much as possible. process management is already a
mess, simplifying it somewhat will make our life easier in the future.
Change-Id: I77eecabe45bee9de18fba0dfc948403d3ce46dfe
this resolves problems with aio roots becoming invalid after fork (which
so far forced us to run the daemon loop in an aio-rootless thread), does
not require restarting the signal handler thread in the subdaemon (since
we no longer lose it), and is a step towards solving #18 (with transient
daemons doing the store manipulation started transparently when needed).
Change-Id: Iad0149cbc807e31964407c9a83d12314702c8122
posix_spawn unsets CLOEXEC for fds that are dup'd onto their existing fd
number. this is very useful when inheriting fd numbers exceeding stderr.
Change-Id: I6f14585d424ded6741fdd087f0c4d33a05936bcc
the `from`/`to` naming only made sense for unidirectional output fds,
for others (and for the dup2 api in general) it was backwards. rename
them to `dup`/`from` to make this look more like the assignment it is
Change-Id: Iee50d06f9cfcea765ace6cfbe85b192829207e5f
writing to non-blocking fds happens during remote builds due to the way
file descriptions are shared between processes. we can either poll when
writing to non-blocking fds are reset fd flags. polling is just easier.
unfortunately there is no reasonable way to test this that isn't flaky.
fixes#896
Change-Id: I1d8666df57da97199247f0770c547d0180f6ce07
cancelling the promise returned by makeInterruptible could free the
fulfiller before the interrupt callback handle, and no order of the
attachments made a difference. we must resort to putting fulfillers
into shared_ptrs so we can capture them in interrupt callbacks now.
(alternatively we could add another kind of interrupt callback, but
the complexity of doing that outweighs the cost of one shared_ptr.)
fixes#895
Change-Id: I008b160482fd4d81a29d7e9e452dcda858b090b9
download progress reports send a STDERR_RESULT frame. many concurrent
downloads send many STDERR_RESULT frames. each of these frames has us
run the report loop once. since many frames can happen in very little
time we may receive many frames in a single read from the socket, and
that in turn means we don't have to fcntl that socket on every round.
we must still ensure that the socket is in the correct state for each
part of the loop, and this does mean we may run two unnecessary fcntl
sequences per processStderr call. that's a small price to pay though.
Change-Id: I7af607d8c759b76aff0f6016435955e2f9456923
these are used often enough that deduplicating them is worth it. we do
lose some error fidelity, but valid fds will never cause an error here
Change-Id: I2b91b4848f546a894a2a6c2d36c32a892fb73c9f
it's only called by verifyStore, and verifyStore is only called by the
daemon and `nix-store --verify`. both pass the promise to `blockOn()`.
Change-Id: I829c0d189fa913cd8566ddd1a578c50e60fb2ddb
all of them block on a promise very soon after starting. only
queryValidPaths needs to make sure not to swallow Interrupted
exceptions to exit quickly instead of trying all paths first.
Change-Id: I4f99f5d75d7057bad109dc0131aa58e84275e362
checkInterrupt is cheap, waiting for a promise isn't. checking for
interruptions before any top-level promise is awaited lets us drop
a bunch of checkInterrupt calls elsewhere, such as in thread pools
Change-Id: Id543edf9411e53b2a5bbec77d3084a8f65aaea46
eagerly consider outputs as not needing deletion during output
registration rather than only doing so after registration. not
waiting for registration to succeed may keep store paths alive
in the file system if registration fails for some reason; that
seem preferrable to the possibility of having another instance
of this bug. since we only leave *good* outputs around there's
not much to worry about except maybe bit of wasted disk space.
fixes#883
Change-Id: I8c22c92e39b9e203f1061278f86cde19dc4474a4
the daemon must use real store paths, not virtual store paths. using
virtual paths may inadvertently delete paths in the system nix store
when a build was run on a redirected store as root, which isn't good
Change-Id: Id048b236bda0e0ab1f3be6ccba0ddc1de2a3e941
killing a cgroup via `cgroup.kill` is not synchronous, we need to give
the processes in the group some time to wake up and exit. due to a few
historical accidents in the codebase we cannot do this asycnhronously,
e.g. with a kj promise without creating yet more problems. we will, at
some point in the future, have to move cgroup management into the main
daemon rather than doing it with RAII wrappers within every subdaemon.
Change-Id: I03bf9060144b5737729f2b05c25771c674fd154c
if a build directory is accessible to other users it is possible to
smuggle data in and out of build directories. usually this ins only
a build purity problem, but in combination with other issues it can
be used to break out of a build sandbox. to prevent this we default
to using a subdirectory of nixStateDir (which is more restrictive).
Fixes CVE-2025-52991.
Change-Id: Iacfc9b50534de158618c815f9fb99d7dae1be4d0
This allows using a userspace program, pasta, to handle comms between
the build sandbox, and the outside world; allowing for full isolation
including the network namespace, closing the "fixed-output derivation
talks to the host over an abstract domain socket" hole for good.
Fixes CVE-2025-46416.
Co-Authored-By: Puck Meerburg <puck@puckipedia.com>
Change-Id: Ifd499b7dbb3784600a6e842fede65fc031ff9f15
we no longer use thread pools for querying missing derivations. this
binds queryMissing to a single thread for now, but query performance
is still greatly improved. we may want to optimize the store code in
the near future too though since queryMissing is now fully cpu bound
Change-Id: I08a9c8cc199963ef5981572ca4a32d90dbdec028
we intentionally omit writers for the new types we add for serialization
purposes since we do not plan to asyncify the legacy ssh server side. if
we ever change our mind we can extract these types into a header and add
writers as needed. due to the inevitable network overhead of the old ssh
wires we don't bother to optimize serialization too much and instead opt
to make the code more readable; the performance difference does not show
up in practice since network latency dominates the few nanoseconds spent
on extra promise allocations and awaits by a couple orders of magnitude.
Change-Id: Id3ee9a01f8bfa63fa23082fa07de5c673fd70883
protocol version 0x204 dates back to nix 2.0 in 2017. that's old enough
to not worry and drop the gratuitous assertion crash we see it instead.
Change-Id: I8cf23373d4daabccab61f1cbb670947479f0d2bc
this is a large step towards making RemoteStore a proper capnp rpc
interface, and it lets us get rid of the RemoteStore error handler
thread pool. this does mean we make six or more extra syscalls per
operation to set and clear socket non-blocking flags, but they are
pretty cheap compared to cross-thread wakeups and scheduling. once
we have real capnp rpc for store wires we can drop them again too.
Change-Id: I67dfebc8644a407cd4a8221ffcad02a938ac5abe
in the future we will want to instantiate either a sink, a source, both,
or streams, depending on how the fd is used. to do this we need to share
read buffers among sync and async readers. removing the FdSource we kept
in the connection also helps prove that we always use this buffer for io
Change-Id: Ib678e128ed6c4a07d6ce5ec1d3cde9eb3f5fc4ca
we don't need to double-buffer commands. only the subframe protocol
needs a buffered backing, and connection setup is special *anyway*.
Change-Id: I596f2bf8e297c3c5dc2befae674deafcf559d9a9
eldritch horrors