the cgroups experimental feature does not work properly without this
because we do not stop subdaemons when the main daemon is shut down.
systemd needs the assigned cgroups to be empty to restart the daemon
and thus cannot cleanly restart the daemon if any connections exist.
starting a fresh unit for each connection creates a new cgroup every
time instead of sharing any delegations and thus solves the problem.
fixes#1030
Change-Id: Id6c458aad30eaa08c3609ac8280a7dde8e8f3cf9
we'll need this to modify argv for socket-activated daemons. this is our
replacement for the old savedArgv mechanism that was unscoped and fucky.
Change-Id: Ie048eb8ea99f1c9cd627a051292c836c83197068
the parent daemon does not change any settings before starting a child,
so there's nothing we may want to change that is not already set by the
config file. this also doesn't prevent changes of the config file being
applied to daemons where we do not expect it since it'll only restore a
setting to the parents' value if the child also has an override for it.
Change-Id: Ic5a9ef13458c103ec9979cb187ba8d3ce5e1e719
the change to shareable thunks also removed a few cases of pointer
equality checks that allowed structures containing functions to be
considered equal to other sets containing the same functions, even
if the sets themselves were pointer-equal themselves. *so* busted.
Change-Id: If87fdab658f9037ce2a654f69a9e3da6ae2f53e5
we can't change the protocol to allow daemons to do this, and we should
not try to guess what the `auto` store uri means depending on whether a
command was run by root or not due to copious side effects and not even
being able to tell whether the `auto` store uri was given explicitly or
not. while `auto` may *technically* allow this via its naming we should
resist the urge to add a hack and fix the underlying protocols instead,
especially since repairPath should be a rare, superuser only operation.
fixes#888
Change-Id: I1b53245db226199f827a89a237a2ab9907c3f766
if curl fails during a transfer and said transfer is piped through
libarchive we see very strange errors (#1009). in such cases we'll
want to prioritize reporting the source errors to allow debugging.
Change-Id: I8cc231257eca5b749471ecd38df15071170decc2
if a promise for an async s3 operation is cancelled is inner fulfiller
will be destroyed, but the s3 operation may still be running. once the
operation finished and attempted to fulfill the promise we'd segfault.
Change-Id: I603c75d6cc39a1aee465b3ceee06b638f7f7f596
realistically only runProgram is ever called for interactive reasons,
and even those calls seem to be rather ill-advised in many cases. the
chance of multiple interactive processes interfering with each other,
whether for input or for output, must be very low to make this in any
way reasonable: if e.g. git calls ssh for multiple fetched inputs and
ssh requests passphrases for both we can otherwise not guarantee that
*any* input is routed correctly. misrouted output is merely annoying.
Change-Id: I794e3fdf0a3238cb9292003a89ac267f0de7a939
the wrapper is needed by transfer streams to restart a failed transfer
if desired. curlFileTransfer itself is more of a fancy handler for the
thread we're dedicating to curl io handling. the thread will stay with
the multi handle for now because quit handling needs to stay there. we
could have CurlMulti keep only a flag, but that does not help us much.
Change-Id: I99550f0bbb635b75898ca7260f08275df86050e3
we have anonymous namespaces and no other scoping requirements, we can
move it out. this will also make it easier to move *other* things out,
such as the transport streams and all the shared multi state handling.
Change-Id: Iafa1eaeb69e096e98c4667f0531b17d5fe7c78a6
waiting for them to finish leaves us prone to deadlocking, e.g. if some
active transfers is blocked and hasn't been destroyed before our thread
was asked to exit. aborting transfers is the safer option here; even if
we cause use after frees we'll at least have a crash, not hang forever.
Change-Id: I333dcced810e8a85e2ded1f377962a0176798098
this is still somewhat experimental and should be considered in flux. we
will have to nail down a logger interface once we start moving the store
protocol to rpc, but until we do that we can use build hooks to test it.
Change-Id: Id20cd346c9520f45871799c31b0af040adde56ef
extracting the size-specific log handling into a new sink deduplicates
the size-limiting code and lets us clean up the derivation goal a bit:
a number of log-related fields we kept only because previous ownership
semantics were a lot more ad-hoc can be moved into the new sink class.
Change-Id: Ie770a276597c84f9d928a9624c0f0de86adfe815
we'll want to remove handleRawChildStream from DerivationGoal, but the
timeout handling is still important. the wrapper is thus useful, but a
more generic way to pass it a promise to do the handling is necessary.
Change-Id: I55bf041d51e4f8a8b8291deae053423d06724329
we never need the connection or the rpc client directly after the hook
is constructed, only the hook instance rpc object. wrapping them using
kj primitives makes this more explicit and prevents destruction errors
Change-Id: I7e0b14d7e365c2e001273e04c0e741ba4afb12f2
this makes it possible to wait for the hook to exit in one place,
process its exit status for some purpose, and later process it in
another place for a different purpose. rpc will need this to shut
down a hook cleanly after the hook has completed its assignments.
Change-Id: I5af2d5aac4b02a0a200d720d0e6f0b5df5496aaf
only LocalDerivationGoal needs these, and once the child is done
running (or is begin torn down) we can just destroy all of them.
Change-Id: Id4d9f0105aaf33d6736c435bb891d3f9af8e7894
wrapping the wait in a timeout at the call site is insufficient because
a timeout would not relock the mutex and leave the raii object invalid.
Change-Id: I0be2aaf6b3c9966070fb0f1deaab048765fd7af3
This partially reverts commit f68233ec43.
while more modern capnp in *does* have the build system changes, it
does *not* have the patches. and those patches are rather important
to us; the clock patch because our CI machiens don't behave the way
capnp expect and the nodiscard patch because it *is* a bugsquasher.
Change-Id: Ic273777f09fd8e5e7001f815db8998dae2dd7b88
this is important for rpc error reporting: if the peer disconnects we
want to throw a useful error that can e.g. let pending builds finish,
not kill the entire process because kj errors are non-lix exceptions.
Change-Id: I41ebe10ad7fd65d886238ce52eeca1fa243b1bb2
this removes another file descriptor from the build hook. we are now
using only rpc to communicate between build hook and daemon, setting
the stage for a future world in which we don't even need build hooks
Change-Id: I706d9d3f2a420abd42fc7762c205931a45d3c8c5
interpolation of the remote store name into activity text is now done in
the build hook, so we no longer need to send it back to the daemon. this
also reduces the dependency of the daemon on how *exactly* remote builds
are done a little bit, and maybe in time we can make these fully opaque.
Change-Id: Ie51c4f2cd107350fec4ad0a8ef83621ed86db6a2
only local builds can now have a builderOutFD, remote builds only log
via json streams. these two kinds of logs have different requirements
and each is only needed by its respective build kind. splitting these
apart thus makes sense, though ideally we'd also split DerivationGoal
into a RemoteDerivationGoal to clean up the rest, but that will wait.
Change-Id: Ib577537266d1160355ab9c44b4604ebda87a7d04
treat it like ssh-ng instead, and have the build hook do the translation
of ssh stdout to the json log message steam the derivaiton goal can use.
since the ssh-ng path also handles all store urls that aren't legacy ssh
we now have a single logging system that handles every remote build kind
equally, without requiring fd passing into the build hook. this is later
required to rpc-ify the log stream emitted by build hooks to the daemon.
Change-Id: Ifb522eb8a9745029050f16b1b3b3601a6ddac748
this restriction was added in 4af2611bd1
and extended later. it makes little sense to allow *one* activity type
specifically and drop all others on the floor, especially since broken
builders can create transfer activities with ids that would be used by
other, "real" activities. the only thing achieved here is to drop logs
written in json format in the build sandbox, which does not help much.
we may revisit this restriction later during other log-related rework.
Change-Id: I8bda494083877b71a2f958470fa52380f6ec4968
it was split up like this only because the old worker system had no
promises. since we have them now we can use them for encapsulation.
Change-Id: Idbd523eff617d4c53c14b125ab3dfce4979cdab1
currently all loggers can always accept messages and never suggest
flushing buffers. in the future this may change, and at that point
we're already fully set up for it. local loggers should never keep
asynchronous (i.e. network-backed) buffers, disk buffers are fine.
networked loggers will require buffers and periodic flushes later.
Change-Id: Ide2114f5bc17f4a1d289c92ed4f9381a1d59dacf
this keeps the call stack flat even for very deep store directories.
moving progress reporting out of the method optimizing one path also
lets us move the inevitable IO wait and associated promise overhead.
Change-Id: I5083d88b05c247c7e19a21a646bcf3f62bfa3200
create activities from loggers themselves instead of passing the logger
as a constructor argument and allow direct construction of children, no
direct logger access needed. most call sites are not changed because we
still need to handle the "no parent" case, and the logger method can do
that more cleanly than a ternary at each site that creates an activity.
we may eventually want to create a root activity, which is cleaner too.
Change-Id: I295e056228dabb08a1316eba7973874784baa113
remotes can generate the same activity ids we might generate locally,
especially if the remote is accessed over the network. in that case a
pid collision is possible, and since activity ids are sequential it's
very possible to create colliding activity ids on both sides as well.
Change-Id: Id58074a41f5f7a59171b52818d1fb5a1beb4bf40
activities are scopes, and there's no reason scopes should not be able
to move around. this also means we can *create* them elsewhere without
also having to box them in some way, making parent relationships clear
Change-Id: I5df036e12ebd8270feb4dca1f23b6bee1f08e906
this has no performance impact in any benchmarks we've run. nul bytes
are still used as implicit truncation points in many places all over:
rejecting them in all locations that treat them as a string end point
requires large changes such as using a proper path library everywhere
Change-Id: I936158bd435f6abf009a689adfbc24496262c578
This reverts commit 42691f0d94.
Reason for revert: darwin builds time out now, apparently due to newly appearing infinite loops in the test suite
Change-Id: I81a09efd53636a287f3095ac87ed8854b6e57922
This avoids mangling debug gdb scripts and breaking them resulting in no
script loading at all.
Change-Id: Ib5e7034c37a1e41d1a1c9bd8442b8b8b8758e350
Signed-off-by: Raito Bezarius <raito@lix.systems>
thunk values are shareable, and we can represent invalid/uninitialized
values with a special bit pattern that makes no sense otherwise. there
is no need to keep allocating values on the heap, instead we can treat
values like reference-counted smart pointers to heap objects, which in
turn lets us save a lot of allocations and, ultimately, gc heap space.
compared to our baseline (main of 2025-09-27) we save 15%+ memory on a
system rebuild and 17% on nix search. eval time regresses by ~3% for a
system rebuild, while nix search is 7% faster. further optimization is
probably possible (but for now this will just have to be good enough).
Change-Id: Ib6c47acdbe2fac4f76a83c2269f16f30ef66b2e1
with thunk state being shareable we no longer need to worry about value
uniqueness, only about value lifetime. this means we can liberally drop
indirections and allocations, passing references instead of pointers or
using stack memory instead of gc-managed memory for some intermediates.
Change-Id: I2d48a6fd57a376d544bd9bd2d05e5420611986d1
there's no measurable performance gain in real-world testing to abusing
our list value storage like this. we haven't tested how much storage we
actually need on the stack to offset most of the temporary storage cost
and used 128 as a "good enough" value instead, reserving 1 kiB of stack
space on 64 bit platforms in a leaf function. this should do for a bit.
Change-Id: Ie98519b1da3e6fe685da88d1c44ffb4580fb592a
Value is already tagged. Value::Acb blocks are allocated for lambdas (so
we can fit the value tag into the three bits we have available), but the
current layout is rather wasteful for this purpose. the type bits can be
stored together with parts of pointers, which in the lambda case will be
the scope the lambda captures. the expr could also be used, but Env is a
gc-allocated item and thus guaranteed to be aligned properly for tagging
Change-Id: Ia685875387c7795bc4a00d73d1ce3cfea84e7297
this is a strong prerequisite for making values themselves copyable
without duplicating evaluation side effects. with this we can treat
`Value` the way we treated `Value *` to date and drop indirections.
Change-Id: I08f30d12697614a3ae7149615f6f1da83b13f52b
preserve type information of wrapped exception when constructing a
ForeignException. this requires using `std::current_exception` for
construction in all cases as passing the exception by reference to
any function erases dynamic type information: `throw` may copy the
thrown exception from its source expression; this only works as we
need it to if static and dynamic type of the expression are equal.
fixes#1000
Change-Id: I7167f298540de7c43c2566592bdb8d7a93c34d31
same as for null: we have few of them, they're statically allocated,
and they're not the largest contributor to the value population. not
storing them in Value itself frees up resources we *will* use later.
Change-Id: I521f9f243f48f56a78f7bffdf1dc1f0bc40a5c2d
we only need the one object for it. there's no need to waste precious
bits of the value internal type enum for this simple singleton datum.
Change-Id: Ie314b5bf429015e518798d9d65ad8ab2bb84a38e
floats are used very rarely, and our float support is bad enough to
strongly discourage using them on reproducibility grounds alone. we
can thus move them to more expensive storage without hurting folks.
Change-Id: I1086f612f85e294dd3fae4a2d334e09f52bbe4a8
external values very rarely appear during eval "normal" eval, and
creating them is pretty expensive. does *anything* even use them?
Change-Id: Id50fa3f76b7e1f551d550d99996a1ed5880b2531
despite not using allocation caches this does not have a statistically
significant performance impact, with less than 1% extra memory needed.
Change-Id: Ibe51a55ba986e471f217f3724977af17880fafff
using the same nodes as tApp is possible, and thanks to multi-arg app
nodes it can even be a bit faster than the linked lists used to date.
Change-Id: Idccb7c0b54c808e62da85d1c42ee09e6e92c4f7b
these behave like the old chains of app nodes, but they can store more
than one argument per node. for tApp values themselves this is not all
that useful, but if we could share tApp and tPrimOpApp backing storage
we could avoid creating and traversing the linked lists of values that
are currently needed to represent partially applied builtin functions.
Change-Id: I5a2a02d9733e1e0be5443459e2998d62fd3b9a5b
paths already are just strings with slightly magic semantics. the type
can mirror that at no perf cost, letting us drop one internalType tag.
Change-Id: I98acaa4fe3bedd28fc0841e1b81184d8dcddacc5
memory overhead is minimal and performance impact not measurable. once
we've done something like this for all value types that don't fit in a
single machine word we can cut a word from Value, offsetting the cost.
Change-Id: I9813bacd7e851957ad3426aed8f74033179a4212
we now use a single cache set for a number of sizes from one to eight
words. this also matches small attrsets, but perf impact seems small.
Change-Id: Icf16b329b98a20fcc9fe75e6395e148f0852c798
on its own this is not very useful, but having accessors for every value
kind is a prerequisite for doing smart things with Value than the union.
the net effect for now is only to add a few parentheses across the tree.
Change-Id: I88688ac09eb08495dad1eb221034ca540f094950
using sleep(1) as a synchronization mechanism does not work. use fifos instead.
fixes#690 and ci constantly falling over in the same exact fucking source line
Change-Id: I51725f8e439b6753f3212d2897dbb0620ad77a37
libarchive is not async and cannot be used async without involving green
threads, which have already proven to be very problematic. unpacking tar
archives is rare enough that spawning a new thread for each shouldn't be
too much overhead, and the additional data copy probably also won't hurt
performance too much. we may even benefit from being able to extract not
just one archive per event loop but as many archives as we can keep fed.
Change-Id: Iece82bd566ada0a2a49de54c4e69caf6d93f6720
using a sink for this has long been a bit weird anyway. originally it
was necessary due to api limitations, but it hasn't been for a while.
Change-Id: I3dfa157944618349bfd6f398ee1667fc31519d86
printTaggedWarning already colorized its messages. we can do the same
for most other log messages.
Change-Id: Idcd31bbf4f8d0d703395b0d2b7b9bc33264d969f
luckily none of these a format strings vulnerabilities because
boost::format is smart enough to throw an exception when given
fewer format string arguments than are requested by specifiers
Change-Id: I5fa78f0d1396263271f6e1dbcee9c0b2e9e18c34
always use log macros, which also have the benefit of respecting the
verbosity setting without needing virtual function calls to read it.
Change-Id: I1c605562a53e54140724d5225e040abcf49ac996
we add two variants: one that just prints a message at the warning
level (mirroring the other printer macros), and one that also adds
the colored "warning: " prefix the function added. since there are
no overriders of this function in tree it looks safe to remove it.
Change-Id: I7008fd0f31d59fbc9259472e29359c8df19ff87d
eldritch horrors