we no longer use thread pools for querying missing derivations. this
binds queryMissing to a single thread for now, but query performance
is still greatly improved. we may want to optimize the store code in
the near future too though since queryMissing is now fully cpu bound
Change-Id: I08a9c8cc199963ef5981572ca4a32d90dbdec028
we intentionally omit writers for the new types we add for serialization
purposes since we do not plan to asyncify the legacy ssh server side. if
we ever change our mind we can extract these types into a header and add
writers as needed. due to the inevitable network overhead of the old ssh
wires we don't bother to optimize serialization too much and instead opt
to make the code more readable; the performance difference does not show
up in practice since network latency dominates the few nanoseconds spent
on extra promise allocations and awaits by a couple orders of magnitude.
Change-Id: Id3ee9a01f8bfa63fa23082fa07de5c673fd70883
protocol version 0x204 dates back to nix 2.0 in 2017. that's old enough
to not worry and drop the gratuitous assertion crash we see it instead.
Change-Id: I8cf23373d4daabccab61f1cbb670947479f0d2bc
this is a large step towards making RemoteStore a proper capnp rpc
interface, and it lets us get rid of the RemoteStore error handler
thread pool. this does mean we make six or more extra syscalls per
operation to set and clear socket non-blocking flags, but they are
pretty cheap compared to cross-thread wakeups and scheduling. once
we have real capnp rpc for store wires we can drop them again too.
Change-Id: I67dfebc8644a407cd4a8221ffcad02a938ac5abe
in the future we will want to instantiate either a sink, a source, both,
or streams, depending on how the fd is used. to do this we need to share
read buffers among sync and async readers. removing the FdSource we kept
in the connection also helps prove that we always use this buffer for io
Change-Id: Ib678e128ed6c4a07d6ce5ec1d3cde9eb3f5fc4ca
we don't need to double-buffer commands. only the subframe protocol
needs a buffered backing, and connection setup is special *anyway*.
Change-Id: I596f2bf8e297c3c5dc2befae674deafcf559d9a9
this may as well be called AsyncSocketStream since that will be what we
use it for, but hopefully it will not exist for long enough to need any
other socket functions to actually justify such highly specific naming.
Change-Id: Icf2fe88cf345405218e4b1bd440267e7f132f5c7
we also extend AsyncInputStream with a drainInto variant to give async
output streams rough feature parity with sync sinks. we still will not
add serialization support to streams though, that's far too expensive.
Change-Id: I60d5ab43610c45a40ea8740470a5eafe68064aea
otherwise stores containing async objects will cause crashes during
shutdown. currently there are no such stores, but that will change.
Change-Id: I05d46ba6831c641774edfe6aa99aa7d0de457429
store objects may hold on to network connections. if those connections
are async they're bound to the lifetime of the aio runtime, which ends
long before the static object destructors we need for nix-store today.
Change-Id: I4aa5466681a82f7e5008cc0b952fcba01d5b39d7
do not rely on Source/Sink `good()` or delayed guessing about whether
an exception was thrown by the daemon or not. mark connections as bad
for all local errors happening while communication is ongoing instead,
and leave it valid only when an exception was provided by the remote.
we may drop connections a bit too eagerly now, but all cases in which
that happens were vulnerable to protocol desynchronization. there are
still a few windows for this to happen left, but those are unfixable.
Change-Id: Iefaa66c552092c436b9de77aa3f8e09f847a966e
once we make our socket fds non-blocking we won't be able to easily use
plain FdSink for serialization. performance impact of using a temporary
buffer should be low since we don't send very many messages and even in
the simple local daemon case networking overhead is already quite high.
Change-Id: I550d73142570b7d2e7b0feb1bcc57d61e9b45178
we will need this during RemoteStore wire asyncification to be able to
use the old synchronous serializers. alternatively we could define all
serializers on the async types as well, but that'd be slow and far too
much unnecessarily duplicated code (that will be deleted soon anyway).
Change-Id: I6e4f334025844b808a697ddcd8f80ddcd8c3fc9c
it was never safe. both discarded the buffer of the source object,
possibly leading to silent data corruption. FdSource discarded the
fancy EOF error string as well, possibly causing bad error reports
Change-Id: Ib5c07986471b5af03d707230cd487259201952e9
this has side-effects for FileTransfer as well since that uses S3Helper
for s3:// urls. the side effects should be entirely positive though: we
can run multiple s3 requests in parallel without explicitly running any
of them from thread pools (the aws s3 client takes care of that for us)
Change-Id: I67232e604ebb12982b63770f1661ea1d56c5087b
making stores and their users fully async requires all data streams to
be async. the most notable data streams in common usage are curl first
and remote stores second. curl is much more contained today and easier
to asyncify (with the preparatory work we've done in the past commits)
Change-Id: I2d6ff4687ee2b47e4efaa6714827b7283bed941d
we need a wrapper type for the remote exception because our Result type
does not deal well with its good type being the same as its error type.
we could have also return a `Result<Result<void>>` to fix this, but the
wrapper type clarifies via its name where the exception_ptr originates.
Change-Id: Ia6ce67b962cb8d6528b017f4cb682a55d6918939
the subframing layer is ... a bit of challenge. since the old code is
synchronous but wants to handle errors asynchronously anyway it is on
the subframing layer to *spawn a thread* that polls for errors on the
wire, while non-framed commands handle errors synchronously once they
have sent all their data. this encapsulation of the wires is far from
perfect (let alone legible), but hopefully it will be only temporary.
Change-Id: I26d8020549b767794cae121313360c488504995f
use a new helper method to send simple command data (that is, command
data that doesn't involve nested framing) to the daemon. this wraps a
large chunk of wire io, and once all wire io is wrapped thusly we can
replace the sink/source io model with new async input/output streams.
Change-Id: Ief9f520263c230a98403b8756bde917fd1cb236e
a size_t followed by as many pairs of things is exactly the format of a
vector of two-element tuples. it would also be the format of a map, but
Roots is a map of sets. rather than adding a serialization format fixed
to this map type (or some wrapper) we can deserialize the response as a
vector and convert it to the map-of-sets later as this is not run much.
Change-Id: I3950c0f7cc59661576170ace10b25a6f8af1464b
processStderr of RemoteStore wants to be a promise and it must be used
from connection setup, so the pool factory callback must be a promise.
Change-Id: I9ac742b6048ae6dba0bfa5dcb58971386229690b
async io for remote store connections needs some sync parts still for
serialization purposes, and those will have to reuse async io buffers
Change-Id: I05e066e3bf8c4318dc23306383f6a849d018ef91
the rpc transition will require sync and async objects to share a single
io buffer (since defining serializers on async is an immense pain in the
tail, slow, and ultimately not necessary). a generic buffer class allows
us to reuse existing serializers more readily (reuse them at all, even).
Change-Id: I5ebba8449f26f2bb76016818928183c7e0123be0
remote store async io will need to set O_NONBLOCK on the connection fds,
and right now the number of fds can vary between connection types: local
connections have one one fd for the sink/source pair since they use unix
sockets, but ssh connections have two because ssh uses pipes. this makes
it rather hard to manage flags correctly, and even harder to wait for io
readiness on both directions using kj. using sockets for ssh fixes this.
Change-Id: I0f563ece7627cd3fbd0f5ce21c25140469729e5a
this could've just ignored exceptions thrown by the remote. in the
current implementation there's no way such an exception could have
propagated to the client though, so there's no change in behavior.
Change-Id: Ide03bda1cb0ad7fb5f27b4ee5d16efd6c2b635ba
mostly to make moving this to async writes easier. this won't have a
performance impact because it's only a single packet, that's written
to a BufferedSink, but the connection sink only gets a single write.
Change-Id: I9a5f1afe7d3e25f5f4502ef9520ff2f2529431ba
the test is for the map that usually wraps it though because it's the
bit we're interested in replacing, and it has custom serializer code.
Change-Id: If77a236dfca738b646ed2b7a5c65515dad6b7295
the old protocols are largely untested, mostly unused, and have design
problems that make the RPC transition a lot harder, if not impossible.
in theory we could ship a transparent protocol-converting proxy that'd
isolate the daemon itself from old protocol versions, but that's a lot
of code to maintain for presumably little gain or even no gain at all.
Change-Id: I4c3f3bb34d39044f6aeb07c10caaf13b8340a220
also remove all the documentation referencing it, or rewrite the docs
to make sense in the non-floating-content-addressed world we live in.
Change-Id: I724e67839f44cc9f1cfc7d6f1c05252b62752b42
we don't need to worry about leaving around old ca data in the database:
this was always a possiblity when enabling ca derivations, and disabling
them again some time later. behavior is unchanged, but we lose dead code
Change-Id: I8c10ff7fdcee08c3badf23d64403f5ee6452e41e
we no longer need placeholders to represent all derivation output paths
as string context, and thus will not need experimental features either.
Change-Id: I9e86ce86810e976cf8397b2c2f473af11390874c
neither are actually partial now, and the the non-Static variant has a
non-Partial wrapper which merely returns the Partial result unchanged.
Change-Id: I5fa86682883c2305cc12c711ccff58537b7a278d
derivation outpaths are now statically known at all times. the one snag
here is that the wires encode even statically known paths as optionals,
forcing us to check for this any time we receive an output map. remotes
answering with nullopt paths for derivations we still support now would
be a protocol error on its own though, so we do not diagnose it deeply.
Change-Id: Ib7080b2a0c45c3506233e87c8ef6842576f61050
we don't need to touch the schema of the cache here. keeping the table
around doesn't hurt (and avoids cppnix breakage) thanks to foreign key
constraints and the ca bits of the schema being independent enough for
us to just ignore them (and not having to do any maintenance on them).
Change-Id: Ib5d8eb1cd838826d88eb65bbf8f245703a2482da
only a daemon wire operation and the perl bindings could initiate these
queries at this point. the daemon ops can throw an error instead (as if
the daemon were older) and realistically should never be queries if the
client hasn't evaluated a ca derivation on a given store, and perl code
is best off dying early. nothing known except hydra uses these bdingins
anyway, and we control our hydra so we don't need backward compat code.
Change-Id: Ia7df27aba59a4a4a692ae014f407415f3bea63f2
it's only used by the RegisterDrvOutput daemon wire operation now, and
that one we can safely stub out to throw an error when called instead.
Change-Id: If29716976392c9c7a2a05b151dfe80b2c8d9c07d
this removes the ca-derivations system feature and, perhaps most
importantly, realisation closure copy support. the latter is not
needed any more and its existence blocks some more code removal.
Change-Id: I2931b03637e25d35252ae6bd5f34f0c0168d80e9
we can't create these any more except by reading an old json-formatted
derivation that used them. since we cannot do anything with a deferred
derivation even when read we will remove json support for them as well
Change-Id: I4f9ea0b7c6469f57977784037f7710f939e40a2c
now that we have no deferred hashes (since floating ca derivations were
the only way to create them) we can safely remove this enumeration too.
Change-Id: Ic72ed90500fcee7aa5b3b5a302477fa515acf1be
only FODs can be content-addressed now, and those are always fixed.
FODs are also never sandboxed, so we do not need that field either.
Change-Id: I1be62b3ec85e08ec003cc8769723328d19777728
this mostly takes the form of removes feature checks and the associated
"ca derivations enabled" branches, but for the realisation info command
turns into a stub. we keep it around for compatibility, but from now on
it will always throws "ca derivations not implemented" errors when run.
Change-Id: I0abea5f76262013415330adcca2b498c6dca555b
this goal is only involved for output paths that aren't known at initial
build time, which in turn can only happen if they are ca paths. since we
can no longer create ca derivations during eval *or* read them from disk
we can now assume that we will never run this goal. there are still some
vestiges like output known-ness we can't remove yet, so those must stay.
Change-Id: I989e5ad4600c628bcbe8e17e1b082ce8d73a3bd9
we remove not only support for *building* a ca derivation, but also
support for *resolving* ca derivations as part of a build. we never
have to resolve derivations from here on, so this code is now dead.
Change-Id: I0346442d5fa00eb927177545ae61315f588477cc
we no longer have any experimental features depending on ca derivations,
so we can start removing them. since ca derivations are very invasive we
will need a while to remove all of the explicitly experimental code, and
even then we will not have removed *all* code related to ca derivations.
especially in the derivation goals there is a lot of code that is not as
easy to disentangle from experimental features as some would have hoped.
Change-Id: Ia456aadc6164613ded343f571318494d9310a549
only dynamic derivations could produce a non-opaque drvPath. since
dynamic derivations are no longer supported we can have drvPath be
opaque at all times, simplifying downstream code significantly and
making quite a few methods unnecessary. discardOutputPath was only
called on drvPath members anyway and thus reduces to a copy, other
operations at the very least are no longer recursive. some vestige
of dynamic derivations remains in DerivedPathMap though (for now).
Change-Id: Ifb4ad53a3c67800be5a62540068c8279d4ae0046
string context doesn't need any tests because it's never persisted or
shown to the user. getting rid of recursive string context means that
the context string parsers can be a lot simpler from here on forward.
Change-Id: I58443679ad76c0f28ea5f4eb8bfb3874f270e764
as with impure derivations it is still possible to garbage-collect
existing xp-dyn-drv derivations. we once again don't introduce any
new kinds of errors, we only change the dynamic type of exceptions
from MissingExperimentalFeature to UnimplementedError (although we
do throw FormatError when reading xp-dyn-drv derivations now, that
seems to make a little more sense than "feature not implemented").
Change-Id: Ic26e5b6c9c9e2533093e27f6cf901dc9db57c83e
only dynamic derivation produce text-hashed derivation outputs. toFile
produces text-hashed store paths, so we cannot remove text hashing now
without breaking stores, but we can disallow it in derivation outputs.
Change-Id: I95ff9882a59153a7d5fd509f5c9fd85925f30d02
with impure derivations gone we move on to dynamic derivations. this too
is not done in a single commit because dynamic derivations are invasive,
modifying semantics of all references to derivation output paths and all
derivation dependency calculations. removing dynamic derivations cleanly
is made significantly harder by the multiple did-you-mean-sum types, aka
"wrappers for std::variant", holding all derivation outpath information.
Change-Id: Ice7a7700c7b54c6a6061d4beb322b4175923d27a
