this makes using the zero-copy pipes a lot more ergonomic. error
handling is unfortunately not trivial and duplicating it for all
pipe users would thus be a bad idea. we're also not oblivious to
the fact that this is a `sourceToSink`, but it's async this time
around. (at least we don't need terrible stackful coroutines..?)
Change-Id: I1ba59f27183988ad68e7f88d102935d005690f43
rpc will often need to transfer data received on push-bashed interfaces
to consumers that expect input stream sources, which are pull-based. we
want to avoid copies for performance reasons (since kj overhead as kind
of on the high side for us already), so we'll use a shared-buffer class
that behaves much like a mutex. we *don't* use mutexes because ours are
cross-thread-capable and thus require syscalls for all wakeups they do.
Change-Id: I3b14925f5d9f5e07ea2cafdf00a88a64f79e4742
sometimes we need to disambiguate based on the kind of the thing we are
filling due to how the c++ type works. in templates some constraints on
type resolution are lifted. sometimes we can make the compiler help us.
Co-authored-by: piegames <git@piegames.de>
Change-Id: Id9549764813a862b8d64959157fed258709eb6ac
without maxprocesses we can utilize core-rich systems better, and test
timeouts are reportedly to be too low to run on small systems as well.
fixes#890
Change-Id: I89386b89fcd69ef4bf77ecf0a49c0b85e1f17c3d
this fixes a large portion of tests currently marked no_daemon. most of
them only needed to set some trusted settings, which is easily done now
Change-Id: Id5a5ee94951cdc92bddd2264c738ca4f98980c8b
single-user builds have a writable home directory in the build sandbox,
and the test suite does not like cargo writing anything into there. the
impact of this seems to be localized to f1 (at least on linux), but not
letting cargo write into the homedir is easy enough to do. nixpkgs also
does something much like this when compiling rustc, so we're not alone.
(it also seems that nixpkgs intended to do this for all rustc users but
broke it at some point in the past. some packages set CARGO_HOME in the
same way we do, possibly because they've had exactly the same problems)
fixes#1251
Change-Id: Ie17a0677ef09bb076ef6295bb590a0895924434e
we need to cache the current terminal size for progress bar reasons, but
we don't want to catch SIGWINCH to update the current terminal size from
a dedicated thread for repl reasons. a SIGWINCH handler function is much
easier to square with these requirements than communication with another
thread to have it change its signal mask, and since any races in handler
code affect only progress bar output and only very rarely (if ever) we'd
better chose the simplest approach. the progress bar could set a handler
of its own for this purpose, but we would much rather replace it instead
fixes#1246
Change-Id: I814d9aaf1b6fbb6a8cefc5af675a3aa372549dc8
this is a bit of a hack, but since zngur cannot handle multiple trait
implementations per type yet we will have to commit to singles types.
Change-Id: I60e7b96bbeaa9fb87cf43662d4a9a5d44116bf47
treating a file with an empty or zero hash as locked is not helpful.
these are placeholdes for "hash is not known", thus treating them as
a valid lock makes them completely useless (and confusing to users).
fixes#1233
Change-Id: If42b47281e6973fc86662b973db69e26f6346f5a
results don't allow for streaming or pipelining. transporting errors in
exception texts is *bad*, but it's still better than not having streams
that actually work. this isn't a great situation, but with capnp that's
pretty much the best we can do. still beats the other rpc frameworks :/
Change-Id: I2abf17bc5ea5de2baeea44ff4d7b2f4db33e98dd
using result types in capnp is fraught. while it makes some sense for
interfaces that need neither streaming nor pipelining and can provide
much better error fidelity there it's also fundamentally incompatible
with those that *do* need streaming or pipelining: streams will never
be stopped early unless an exception flies, and pipelines cannot look
through result types. likely the best thing we can do is to encode an
error for transport in the capnp/kj exception description strings. :(
Change-Id: Icb7d16238fa9a7aaf92c00363f7be4076ac02a61
this takes the form of a couple of hideous macros for now to get started
without worrying about semantics of rust logging and tracing crates, all
of which are in some way incompatible with the current lix expectations.
we do want to move to using those eventually, but that will have to be a
task for later. as long as only small amounts of lix code live in crates
we don't have to worry much anyway, and by the time we get around to any
Activity beimg ported we'll hopefully have a better understanding of the
requirements on either side. we'll reëvaluate the log infra around then.
Change-Id: I4aec476a486f56d0972294358cbf0b6810ab9394
This first geralizes the `nix` fixture with `pytest_generate_tests` to
iterate over all protocols (unless they are marked with `no_daemon`),
though for now the list of protocols is set to be empty. The lang tests
are all tagged with `no_daemon` because they are mostly pure and running
them multiple times would be wasteful.
Co-authored-by: rootile <lix@rootile.de>
Co-authored-by: piegames <git@piegames.de>
Change-Id: Ib407edb420ba4bf434cacf9563a71f5ae6fa8eef
they're not doing anything, and they're slightly in the way of reworking
the rust build infra. we'll add them back once that rework is completed.
Change-Id: Idde8ba3585fd08b89b4b4298349f8f58a29a9661
curl does not support this configuration. we can't easily change the
config setup to mirror how curl works, but we *can* print a warning.
fixes#1223
Change-Id: Iaa1583275d9cc9e5eed5552210db183a8e0628c8
what the actual fuck. swallowing all errors *except* EXDEV is not okay.
renames do not do this, so moves should not do it either. luckily we do
not use moveFile anywhere except the store path registration code. this
may or may not have caused problems in the past. probably nobody knows.
Change-Id: I2b0255a5703983cbd129abc3219c11ac7171fd12
catch and rewrap std::fs errors in some meaningful way, otherwise lix
can crash with foreign exception aborts if filesystem failures happen
that aren't already being caught. renames in particular can fail with
incomprehensible errors in even pretty normal circumstances elsewise.
fixes#1219, fixes#995
Change-Id: I0b1e5696d300aac3422e43cffc17a2426803d64e
pour the foundations for rpc, and let them set. this is very much
unstable and must be opted into with explicit store uris (e.g. by
setting `NIX_REMOTE=daemon?protocol=any`). the daemon sockets are
not enabled by default and must be enabled with the `rpc-sockets`
experimental feature. we will not advertise this just yet because
in the current state it one has to be *very* dedicated to the rpc
cause to deploy this, but once we have some more bits migrated we
may want to add release notes and officially as for beta testing.
Co-Authored-By: piegames <git@piegames.de>
Change-Id: I85a96ccb700b91190c1eb37154bcc6ae1c03401a
splitting it off lets us reuse it (and all the legacy protocol handling
it encapsulates) in the rpc world as well by just tunneling wire bytes.
doing this allows incremental migration of to a new rpc protocol, until
we are finally ready to call the rpc protocol ready and remove the link
to the legacy protocol altogether (and maybe merge this function back).
Change-Id: I8f2a8795c4a9d713ebbbbfd02e9eec5271a2b7a9
this adds the protocol registry. we also extend the systemd unit setup
to allow adding new protocols without much fussing, and we move deamon
instances into protocol-dependent scopes. this will make monitoring or
resource limiting of individual protocol daemon instances much easier.
Change-Id: Ic9da08eb074868b6e57385075c5502b081e8461f
we'll use this to create a registry of protocols the system understands.
also use this struct during connection setup to make that simpler later.
Change-Id: Ifa481fea7ea2efa2a1f1be4d81076a9e022d24f9
string data shares a buffer with the binary string length field. size
calculations for string read buffers always include the length field;
sufficiently large length fields can cause these calculations to wrap.
a malicious nar could use this for OOB writes in the daemon (as root).
since we use strings only as tags for archive members and for symlinks
with their OS-dependent length limits we can simply limit string size.
1 MiB should be sufficient for all symlinks, and tags are always tiny.
Change-Id: I89fb05f73c1dbeda45d91244aba4cd526a3d83e1
this is not read in code paths that are called repeatedly, and the only
reader can never see it being false. we also question the wisdom of the
flag in the first place; why should opening a connection fail because a
previous connection did? conditions may have improved a lot since then!
Change-Id: Ib7c219e6239432d99a22b10024847218469f1cd4
notably the bad-store test previously tested *bash*, not lix. we also
need to extend the list of non-fatal errors, but that's probably fine
Change-Id: I983f4adc8047fe800323099de4037711869ee547
this should improve error messages when connecting to long socket paths,
and make it at all possible to detect ENOTDIR from connect helpers. this
will be needed to handle daemon sockets that live in directories we have
not previously considered like the protocol name directories we need for
rpc purposes, all of which would have resulted in hard errors previously
Change-Id: I4e16cf09fedd07c93cfdf7b0f151106984507315
- launch-builder-linux.cc was missing an include for musl
and used function that are not defined in the launchers
- musl caches pids used for raise, breaking sandbox setup
- the mtls contrib plugin won't build, didn't try fixing,
static builds can't really use plugins reliably anyway.
Change-Id: I5ab1664e45ea977e5bcf05e41d825e6014e62146
kj can and will throw exceptions immediately without wrapping them as
promise results, and we must take care of these as well to not crash.
fixes#1150
Change-Id: Ifa29d0f3da4a6490f3e45d4c04a8fa49eaa20f43
not doing this has lix crash if stdio forwarding fails for odd reasons,
such as ssh failing to set up a connection due to transient dns errors.
cf #1148
Change-Id: I543b7444236e69d3c13b47fb6c6eb37c65b96763
`nix.full_path()` is the *built* path, not the *installed* path. in the
sandbox this point to a location that contains *only* `nix`, not any of
the old-cli symlinks. any test that needs them in path would thus fail.
Change-Id: Ia651db8d1151a84f8d1b52e8993f6db61b0fe104
we already report errors that happen during reads via exceptions
rethrown from read. reporting errors during destruction too is a
source of confusion for users since these errors are not usually
relevant and just the decompressor thread reporting short reads.
shorts reads during shutdown are not an error, they're expected.
fixes#1138
Change-Id: I7983ef0ed7722460f1d683bc8171a454b3d06e90
this includes tests that fail teardown when *skipped*, which confuses
the tap parser by reporting more tests than advertised in the header.
not sure what to do about this though, and it is better than nothing.
Change-Id: I4fc6badacbfa3b3d5f5f4a4ad76b177379bb4a30
pytest-tap didn't get it right in combination with xdist. we need to
replace the reporter *completely*, which is just incredibly weird :/
Change-Id: I2e35b9dc8ea4b24563b9b3306312778781eb674e
this requires some hacks to make error reports visible, but hey. it's
better than no progress reports at all, and successful runs no longer
spew huge amounts of useless log output onto the developer's terminal
Change-Id: I9e4766b2f825a4ec451e117eb4609db65f328785
not all operations that involve workers need to build. for example nix
copy --from calls instantiate thir worker with the *origin* store, not
the store they copy *into*. for binary cache stores this is not likely
to ever work if cgroups are enabled and the copy is run on normal user
credentials, even though no cgroups will ever be needed to copy things
fixes#1088
Change-Id: I065e29e1a1d0f58d81823609ef0701ff82cdd1d5
