protocol version 0x204 dates back to nix 2.0 in 2017. that's old enough
to not worry and drop the gratuitous assertion crash we see it instead.
Change-Id: I8cf23373d4daabccab61f1cbb670947479f0d2bc
When Ctrl-C is sent to the workload, even across remote builds, the
whole process possess a global flag `_isInterrupted` which is checked in
certain filesystem operations, cancelling them, e.g. writeFile will
write nothing under interruption unwinding.
In addition, if any operation throws an exception before we `rmdir` the
cgroup, we may leave it hanging while we remove the state record.
Therefore, we put the final cleanup in a block.
In practice, reading statistics could lead to failures.
Control groups cleanups are critical though and should always be
performed.
Change-Id: I48fa87317b6a9f6663559bc8fa5f8a897f37011e
Signed-off-by: Raito Bezarius <raito@lix.systems>
this is a large step towards making RemoteStore a proper capnp rpc
interface, and it lets us get rid of the RemoteStore error handler
thread pool. this does mean we make six or more extra syscalls per
operation to set and clear socket non-blocking flags, but they are
pretty cheap compared to cross-thread wakeups and scheduling. once
we have real capnp rpc for store wires we can drop them again too.
Change-Id: I67dfebc8644a407cd4a8221ffcad02a938ac5abe
in the future we will want to instantiate either a sink, a source, both,
or streams, depending on how the fd is used. to do this we need to share
read buffers among sync and async readers. removing the FdSource we kept
in the connection also helps prove that we always use this buffer for io
Change-Id: Ib678e128ed6c4a07d6ce5ec1d3cde9eb3f5fc4ca
we don't need to double-buffer commands. only the subframe protocol
needs a buffered backing, and connection setup is special *anyway*.
Change-Id: I596f2bf8e297c3c5dc2befae674deafcf559d9a9
this may as well be called AsyncSocketStream since that will be what we
use it for, but hopefully it will not exist for long enough to need any
other socket functions to actually justify such highly specific naming.
Change-Id: Icf2fe88cf345405218e4b1bd440267e7f132f5c7
we also extend AsyncInputStream with a drainInto variant to give async
output streams rough feature parity with sync sinks. we still will not
add serialization support to streams though, that's far too expensive.
Change-Id: I60d5ab43610c45a40ea8740470a5eafe68064aea
otherwise stores containing async objects will cause crashes during
shutdown. currently there are no such stores, but that will change.
Change-Id: I05d46ba6831c641774edfe6aa99aa7d0de457429
store objects may hold on to network connections. if those connections
are async they're bound to the lifetime of the aio runtime, which ends
long before the static object destructors we need for nix-store today.
Change-Id: I4aa5466681a82f7e5008cc0b952fcba01d5b39d7
do not rely on Source/Sink `good()` or delayed guessing about whether
an exception was thrown by the daemon or not. mark connections as bad
for all local errors happening while communication is ongoing instead,
and leave it valid only when an exception was provided by the remote.
we may drop connections a bit too eagerly now, but all cases in which
that happens were vulnerable to protocol desynchronization. there are
still a few windows for this to happen left, but those are unfixable.
Change-Id: Iefaa66c552092c436b9de77aa3f8e09f847a966e
once we make our socket fds non-blocking we won't be able to easily use
plain FdSink for serialization. performance impact of using a temporary
buffer should be low since we don't send very many messages and even in
the simple local daemon case networking overhead is already quite high.
Change-Id: I550d73142570b7d2e7b0feb1bcc57d61e9b45178
we will need this during RemoteStore wire asyncification to be able to
use the old synchronous serializers. alternatively we could define all
serializers on the async types as well, but that'd be slow and far too
much unnecessarily duplicated code (that will be deleted soon anyway).
Change-Id: I6e4f334025844b808a697ddcd8f80ddcd8c3fc9c
it was never safe. both discarded the buffer of the source object,
possibly leading to silent data corruption. FdSource discarded the
fancy EOF error string as well, possibly causing bad error reports
Change-Id: Ib5c07986471b5af03d707230cd487259201952e9
> The resource control configuration options are configured in the
> [Slice], [Scope], [Service], [Socket], [Mount], or [Swap] sections,
> depending on the unit type.
Reported by Worm on matrix.
Change-Id: I5f942b864e40bc461e8751cdf8337b1f8c2bbce4
The previous format was a little bit messy, with inconsistent alignment of items in each line after the main error
message. The format has been cleaned up, by aligning the start of all values on the same column, and right-aligning
their labels.
Change-Id: Ic9bb3300faef00cd2e51ebb2f5e0077ade2ff949
Lowdown doesn't quite conform to CommonMark in parsing shortcut links
that are followed by a parenthesized expression, which looks like
`[link text] (unrelated text)`. CommonMark says the space there is
significant and ensures the `[link text]` is parsed as a shortcut link,
but Lowdown parses this like `[link text](unrelated text)`.
This fixes the output of `nix help`. The other case of a near-link was
in the `nix-env --install` docs, which don't get parsed by Lowdown, but
it turns out the link reference definition was missing. The generated
manpage stripped the brackets but the HTML manual page rendered the
broken link with brackets.
Change-Id: I6a6a69641fd2dbf9930bcd875ed21ea80fba909a
this has side-effects for FileTransfer as well since that uses S3Helper
for s3:// urls. the side effects should be entirely positive though: we
can run multiple s3 requests in parallel without explicitly running any
of them from thread pools (the aws s3 client takes care of that for us)
Change-Id: I67232e604ebb12982b63770f1661ea1d56c5087b
making stores and their users fully async requires all data streams to
be async. the most notable data streams in common usage are curl first
and remote stores second. curl is much more contained today and easier
to asyncify (with the preparatory work we've done in the past commits)
Change-Id: I2d6ff4687ee2b47e4efaa6714827b7283bed941d
We upgrade to 25.05 release, which contains the curl commit
https://github.com/curl/curl/commit/5fbd78eb2dc4afbd8884e8eed27147fc3d4318f6
done in
https://github.com/NixOS/nixpkgs/pull/396200#issuecomment-2795944006.
This fixes HTTP transfers generating arbitrary errors and possibly
failing unusually.
Users who are already depending on 25.05-small or a recent unstable
already had the fix.
Special mention to the Linux kernel who gave me the opportunity to get
on a 24 hours bisection side quest to fix the local release engineering
test.
Special thanks to everyone who had to endure me ranting.
Change-Id: I866caf65d5ea103f1fa5eccd57df8031c9eacda0
Co-authored-by: eldritch horrors <pennae@lix.systems>
Co-authored-by: helle <helle@h3l.li>
Signed-off-by: Raito Bezarius <raito@lix.systems>
Most of these are simple fixes and clarifications. One set of fixes will
come in the commit that actually upgrades nixpkgs and hence ruff as it
will otherwise cause errors here.
Change-Id: Ie857da0f6cf728478700ec2d24cf518f8c7b7815
we need a wrapper type for the remote exception because our Result type
does not deal well with its good type being the same as its error type.
we could have also return a `Result<Result<void>>` to fix this, but the
wrapper type clarifies via its name where the exception_ptr originates.
Change-Id: Ia6ce67b962cb8d6528b017f4cb682a55d6918939
the subframing layer is ... a bit of challenge. since the old code is
synchronous but wants to handle errors asynchronously anyway it is on
the subframing layer to *spawn a thread* that polls for errors on the
wire, while non-framed commands handle errors synchronously once they
have sent all their data. this encapsulation of the wires is far from
perfect (let alone legible), but hopefully it will be only temporary.
Change-Id: I26d8020549b767794cae121313360c488504995f
use a new helper method to send simple command data (that is, command
data that doesn't involve nested framing) to the daemon. this wraps a
large chunk of wire io, and once all wire io is wrapped thusly we can
replace the sink/source io model with new async input/output streams.
Change-Id: Ief9f520263c230a98403b8756bde917fd1cb236e
a size_t followed by as many pairs of things is exactly the format of a
vector of two-element tuples. it would also be the format of a map, but
Roots is a map of sets. rather than adding a serialization format fixed
to this map type (or some wrapper) we can deserialize the response as a
vector and convert it to the map-of-sets later as this is not run much.
Change-Id: I3950c0f7cc59661576170ace10b25a6f8af1464b
processStderr of RemoteStore wants to be a promise and it must be used
from connection setup, so the pool factory callback must be a promise.
Change-Id: I9ac742b6048ae6dba0bfa5dcb58971386229690b
async io for remote store connections needs some sync parts still for
serialization purposes, and those will have to reuse async io buffers
Change-Id: I05e066e3bf8c4318dc23306383f6a849d018ef91
the rpc transition will require sync and async objects to share a single
io buffer (since defining serializers on async is an immense pain in the
tail, slow, and ultimately not necessary). a generic buffer class allows
us to reuse existing serializers more readily (reuse them at all, even).
Change-Id: I5ebba8449f26f2bb76016818928183c7e0123be0
remote store async io will need to set O_NONBLOCK on the connection fds,
and right now the number of fds can vary between connection types: local
connections have one one fd for the sink/source pair since they use unix
sockets, but ssh connections have two because ssh uses pipes. this makes
it rather hard to manage flags correctly, and even harder to wait for io
readiness on both directions using kj. using sockets for ssh fixes this.
Change-Id: I0f563ece7627cd3fbd0f5ce21c25140469729e5a
eldritch horrors