this will make it easier to turn build sandbox processes into helpers.
the point they were at was effectivly unsandboxed except for a few fds
that were redirected by commonExecveingChildInit, which only made logs
of any errors that much harder to convey from the child to the parent.
Change-Id: I67006eb33e1e13311bb8d14e6a0c3d5e6baf0c13
it doesn't do anything anywhere else, and personality setup is not a
critical operation (all it does is change which arch uname returns).
Change-Id: I39d7fcc4916e6882e49191d2294f9b7ee0dbbcd0
there's no need to do it in the fork, we're not changing security
domains before opening the pty anyway. we do change who owns this
pty device, but since we change it via chown we can just *not* do
that before we have an open fd to the pty. in practice this isn't
even necessary because the daemon runs as root, but if we ever do
run the daemon as an unprivileged user we'd need this to be split
Change-Id: I35264ab2954c7ba2c9c24c927366d64acada6772
this also requires moving namespace support information into Worker and
out of function-scope static variables, otherwise we can't use async IO
for the libexec helper output. we could set the fd to blocking for just
one CL and extract the Worker changes into another that the reverts the
blocking fd usage, but that seems not warranted for the scope of these.
Change-Id: I6996fab1ae74693d50cefb6a6a9c21d61dada1d9
this is mostly a test and example for the libexec helper infrastructure,
but it also lets us simplify pager launching until we we can more easily
handle executable-not-found errors the launch fallbacks would cause when
using runProgram2 instead of fork. ideally we'd use `posix_spawn` later.
fixes#1104
Change-Id: Ia33cc12e8a9d60ffad6f5c055bb1b8b596810e64
the new libexec directory is not available as a setting like other
directories (e.g. binDir) are since we consider libexec helpers to
be very internal. repointing them is like repointing a .so file we
dynamically link to; it can work, but needs much more preparation.
Change-Id: I40e64be0b32276f2864c0f2eb0b998d4c8ce7c88
we don't need the full sandbox setup helpers for this: mount namespaces
do not need to be kept, loggers are not needed, and redirections can be
done by runProgram2. once the build hook is removed we will not run the
communication bits from a different process anyway, this prepares that.
Change-Id: I95d28f7c2c25e43ccd82b448d270403ce4f28852
we do not need explict redirection management now, and dieWithParent
doesn't need an override either. we'd much prefer to kill ssh if the
process in charge exits; even ssh multiplexers are not fazed by this
Change-Id: I81e28b7605df73c887878ea4716228c7ad0f5c6f
the redirections the tests did were once not possible with runProgram2,
but they have been for a while now. we should use them instead of fork.
Change-Id: Ia422c8941ed04a9403dee68bfe938d68952253da
returning the pid as a pid_t instead of as a Pid raii wrapper is only
convenient for daemon use (where it'll eventually go away). using the
released pid correctly in other places is harder without the wrapper.
Change-Id: Ib42a2f357d2f0849beabd015b321bfff31334eda
it's no longer needed now that withPager exists. also fix a
logger-never-resumed bug that never showed up in the world.
Change-Id: I7311a50896f5291364320ec4c88506dc3bb11d4c
RunPager is weird and confusing in that it replaces what stdout *is*
depending on environmental conditions. this has not caused problems,
but it's easy to imagine situations in which it would (eg if the stl
decided to capture the stdout fd by duplicating it). using a wrapper
for this also makes clear *what* actually goes into the pager; while
the previous contract was semi-reasonable it was also very implicit,
and with the proliferation of functions we had that printed directly
to stdout it would have been easy to send wrong output to the pager.
using a wrapper also makes process management much easier because we
do not have to rely on destructors to always produce correct output.
Change-Id: Ifd3760940af1ec719fe856158c913cbb9a5bf270
this has been unused since ab3ce1cc13,
which is now almost ten years ago. surely it was just forgotten then
Change-Id: Ib93b52d4ce5f19629432c5f91ff426582c57f99b
cancelling a get() promise could leave the pool with allegedly in use
resource handles that do not actually exist. this would cause asserts
in ~Pool to fire (and crash the entire program due to bad accounting)
probably fixes#1041
Change-Id: Ic61ab9b658f02f71e8be7577b8a1115600846f86
merge all pkgconfig-found libraries into a single one and redirect all
existing lib files to the merged variant. the clean separation between
libraries is largely fictional anyway because the main config block is
in libstore and cannot reasonably be moved out, or because libutil has
assumptions built into its behavior that libmain later establishes, or
because libexpr messes with global boehmgc configuration. nearly every
kind of layering violation imaginable can be found *some*where in lix,
and not pretending that this is not the case will let us clean up some
of them. this will also allow us to merge these libraries at the meson
level which is itself necessary to make rust code easier to integrate.
Change-Id: I172f8a08d5d90ded0a0a99fd9e0fa5d804db00a7
during the value rewrite we accidentally broke extension of incomplete
primop application. this only shows up when binding on incomplete call
to a primop to a name, binding an incomplete call to *that* to another
name, and then finally calling the second binding with enough args for
a complete primop application. since this only shows up when calling a
primop with three or more args it took a while to surface. we have few
builtins that match this: foldl', replaceStrings, and substring. these
are not used incompletely in this manner very often, so this lingered.
fixes#1102
Change-Id: I218dffc14ae876efc86a86c7eb6c895e2405201c
these ended up referring to different directories and would break during
source reorganization (or even just including them in different places).
Change-Id: I91e8db04e3d3565efd5920982ed5bc59d2c27058
RegisterCommand was renamed in b30556f8f1
and de-static-ified in 4d9fe67430. adding
new settings was never possible with static instances since settings do
need an existing Config instance during construction; GlobalConfig must
be used to add settings. it is not possible to change existing configs.
Change-Id: I26af8abc375ba5883fce060ffefc710d6dfbca4c
nlohmann has customization points for exception throws. we can use these
instead and wrap json exceptions at the source instead of playing a game
of whack-a-mole with json errors all over the tree. since nlohmann needs
macros set to achieve this we can no longer precompile its headers *and*
must forbid including it anywhere without the proper defines or ordering
of include directives will break lix with ODR violations, if we see them
at all before we get another json-related bug report. a new lint will be
in charge of ensuring this doesn't happen. we also re-allow direct calls
to nlohmann json parsing since error handling is no longer a problem. we
will keep the wrapper for more convenient error context handling though.
fixes#1092
Change-Id: I54ecc14f5bec5e2177729b41c3703216e76cc6a3
this is only needed on macos, and only for tests that build anything. on
linux it can actually *break* stuff due to unfortunate interactions with
store url parsing, which is required to work for any remote build tests.
Change-Id: Ic6f7e090f15e129fc365d7edc56cdbc1a5686047
- test_check used an old version of an error message
- test_good2 did not build an unspecified dependency
- test_attribute_selection used a fixed drv path. we
can't keep this with the current setup because the
drv `system` makes the drv hash platform-dependent
fixes#1081
Co-Authored-By: kloenk <me@kloenk.dev>
Change-Id: Ia009740f8b5432e83467ea451a5ff53d5e141b3f
the cgroups experimental feature does not work properly without this
because we do not stop subdaemons when the main daemon is shut down.
systemd needs the assigned cgroups to be empty to restart the daemon
and thus cannot cleanly restart the daemon if any connections exist.
starting a fresh unit for each connection creates a new cgroup every
time instead of sharing any delegations and thus solves the problem.
fixes#1030
Change-Id: Id6c458aad30eaa08c3609ac8280a7dde8e8f3cf9
we'll need this to modify argv for socket-activated daemons. this is our
replacement for the old savedArgv mechanism that was unscoped and fucky.
Change-Id: Ie048eb8ea99f1c9cd627a051292c836c83197068
the parent daemon does not change any settings before starting a child,
so there's nothing we may want to change that is not already set by the
config file. this also doesn't prevent changes of the config file being
applied to daemons where we do not expect it since it'll only restore a
setting to the parents' value if the child also has an override for it.
Change-Id: Ic5a9ef13458c103ec9979cb187ba8d3ce5e1e719
the change to shareable thunks also removed a few cases of pointer
equality checks that allowed structures containing functions to be
considered equal to other sets containing the same functions, even
if the sets themselves were pointer-equal themselves. *so* busted.
Change-Id: If87fdab658f9037ce2a654f69a9e3da6ae2f53e5
we can't change the protocol to allow daemons to do this, and we should
not try to guess what the `auto` store uri means depending on whether a
command was run by root or not due to copious side effects and not even
being able to tell whether the `auto` store uri was given explicitly or
not. while `auto` may *technically* allow this via its naming we should
resist the urge to add a hack and fix the underlying protocols instead,
especially since repairPath should be a rare, superuser only operation.
fixes#888
Change-Id: I1b53245db226199f827a89a237a2ab9907c3f766
if curl fails during a transfer and said transfer is piped through
libarchive we see very strange errors (#1009). in such cases we'll
want to prioritize reporting the source errors to allow debugging.
Change-Id: I8cc231257eca5b749471ecd38df15071170decc2
if a promise for an async s3 operation is cancelled is inner fulfiller
will be destroyed, but the s3 operation may still be running. once the
operation finished and attempted to fulfill the promise we'd segfault.
Change-Id: I603c75d6cc39a1aee465b3ceee06b638f7f7f596
realistically only runProgram is ever called for interactive reasons,
and even those calls seem to be rather ill-advised in many cases. the
chance of multiple interactive processes interfering with each other,
whether for input or for output, must be very low to make this in any
way reasonable: if e.g. git calls ssh for multiple fetched inputs and
ssh requests passphrases for both we can otherwise not guarantee that
*any* input is routed correctly. misrouted output is merely annoying.
Change-Id: I794e3fdf0a3238cb9292003a89ac267f0de7a939
the wrapper is needed by transfer streams to restart a failed transfer
if desired. curlFileTransfer itself is more of a fancy handler for the
thread we're dedicating to curl io handling. the thread will stay with
the multi handle for now because quit handling needs to stay there. we
could have CurlMulti keep only a flag, but that does not help us much.
Change-Id: I99550f0bbb635b75898ca7260f08275df86050e3
we have anonymous namespaces and no other scoping requirements, we can
move it out. this will also make it easier to move *other* things out,
such as the transport streams and all the shared multi state handling.
Change-Id: Iafa1eaeb69e096e98c4667f0531b17d5fe7c78a6
waiting for them to finish leaves us prone to deadlocking, e.g. if some
active transfers is blocked and hasn't been destroyed before our thread
was asked to exit. aborting transfers is the safer option here; even if
we cause use after frees we'll at least have a crash, not hang forever.
Change-Id: I333dcced810e8a85e2ded1f377962a0176798098
this is still somewhat experimental and should be considered in flux. we
will have to nail down a logger interface once we start moving the store
protocol to rpc, but until we do that we can use build hooks to test it.
Change-Id: Id20cd346c9520f45871799c31b0af040adde56ef
extracting the size-specific log handling into a new sink deduplicates
the size-limiting code and lets us clean up the derivation goal a bit:
a number of log-related fields we kept only because previous ownership
semantics were a lot more ad-hoc can be moved into the new sink class.
Change-Id: Ie770a276597c84f9d928a9624c0f0de86adfe815
we'll want to remove handleRawChildStream from DerivationGoal, but the
timeout handling is still important. the wrapper is thus useful, but a
more generic way to pass it a promise to do the handling is necessary.
Change-Id: I55bf041d51e4f8a8b8291deae053423d06724329
we never need the connection or the rpc client directly after the hook
is constructed, only the hook instance rpc object. wrapping them using
kj primitives makes this more explicit and prevents destruction errors
Change-Id: I7e0b14d7e365c2e001273e04c0e741ba4afb12f2
this makes it possible to wait for the hook to exit in one place,
process its exit status for some purpose, and later process it in
another place for a different purpose. rpc will need this to shut
down a hook cleanly after the hook has completed its assignments.
Change-Id: I5af2d5aac4b02a0a200d720d0e6f0b5df5496aaf
only LocalDerivationGoal needs these, and once the child is done
running (or is begin torn down) we can just destroy all of them.
Change-Id: Id4d9f0105aaf33d6736c435bb891d3f9af8e7894
wrapping the wait in a timeout at the call site is insufficient because
a timeout would not relock the mutex and leave the raii object invalid.
Change-Id: I0be2aaf6b3c9966070fb0f1deaab048765fd7af3
This partially reverts commit f68233ec43.
while more modern capnp in *does* have the build system changes, it
does *not* have the patches. and those patches are rather important
to us; the clock patch because our CI machiens don't behave the way
capnp expect and the nodiscard patch because it *is* a bugsquasher.
Change-Id: Ic273777f09fd8e5e7001f815db8998dae2dd7b88
this is important for rpc error reporting: if the peer disconnects we
want to throw a useful error that can e.g. let pending builds finish,
not kill the entire process because kj errors are non-lix exceptions.
Change-Id: I41ebe10ad7fd65d886238ce52eeca1fa243b1bb2
this removes another file descriptor from the build hook. we are now
using only rpc to communicate between build hook and daemon, setting
the stage for a future world in which we don't even need build hooks
Change-Id: I706d9d3f2a420abd42fc7762c205931a45d3c8c5
interpolation of the remote store name into activity text is now done in
the build hook, so we no longer need to send it back to the daemon. this
also reduces the dependency of the daemon on how *exactly* remote builds
are done a little bit, and maybe in time we can make these fully opaque.
Change-Id: Ie51c4f2cd107350fec4ad0a8ef83621ed86db6a2
only local builds can now have a builderOutFD, remote builds only log
via json streams. these two kinds of logs have different requirements
and each is only needed by its respective build kind. splitting these
apart thus makes sense, though ideally we'd also split DerivationGoal
into a RemoteDerivationGoal to clean up the rest, but that will wait.
Change-Id: Ib577537266d1160355ab9c44b4604ebda87a7d04
treat it like ssh-ng instead, and have the build hook do the translation
of ssh stdout to the json log message steam the derivaiton goal can use.
since the ssh-ng path also handles all store urls that aren't legacy ssh
we now have a single logging system that handles every remote build kind
equally, without requiring fd passing into the build hook. this is later
required to rpc-ify the log stream emitted by build hooks to the daemon.
Change-Id: Ifb522eb8a9745029050f16b1b3b3601a6ddac748
this restriction was added in 4af2611bd1
and extended later. it makes little sense to allow *one* activity type
specifically and drop all others on the floor, especially since broken
builders can create transfer activities with ids that would be used by
other, "real" activities. the only thing achieved here is to drop logs
written in json format in the build sandbox, which does not help much.
we may revisit this restriction later during other log-related rework.
Change-Id: I8bda494083877b71a2f958470fa52380f6ec4968
it was split up like this only because the old worker system had no
promises. since we have them now we can use them for encapsulation.
Change-Id: Idbd523eff617d4c53c14b125ab3dfce4979cdab1
currently all loggers can always accept messages and never suggest
flushing buffers. in the future this may change, and at that point
we're already fully set up for it. local loggers should never keep
asynchronous (i.e. network-backed) buffers, disk buffers are fine.
networked loggers will require buffers and periodic flushes later.
Change-Id: Ide2114f5bc17f4a1d289c92ed4f9381a1d59dacf
this keeps the call stack flat even for very deep store directories.
moving progress reporting out of the method optimizing one path also
lets us move the inevitable IO wait and associated promise overhead.
Change-Id: I5083d88b05c247c7e19a21a646bcf3f62bfa3200
create activities from loggers themselves instead of passing the logger
as a constructor argument and allow direct construction of children, no
direct logger access needed. most call sites are not changed because we
still need to handle the "no parent" case, and the logger method can do
that more cleanly than a ternary at each site that creates an activity.
we may eventually want to create a root activity, which is cleaner too.
Change-Id: I295e056228dabb08a1316eba7973874784baa113
eldritch horrors