what the actual fuck. swallowing all errors *except* EXDEV is not okay.
renames do not do this, so moves should not do it either. luckily we do
not use moveFile anywhere except the store path registration code. this
may or may not have caused problems in the past. probably nobody knows.
Change-Id: I2b0255a5703983cbd129abc3219c11ac7171fd12
this logs unpackTarfile calls as an `actUnknown` activity, which
unfortunately doesn't report any progress for now. for the reasoning
behind this choice, see the previous iterations of this CL (cl/5590)
as well as the attached comments (the crux of the conversion is in DMs,
unfortunately). in short:
- i could't make a new ActivityType because that would be a breaking
change to the daemon protocol, including with previous lix versions
- i can't use actFileTransfer because that would show up as a download,
which would be confusing for both users and tools
- i can't use actCopyPath /or/ actCopyPaths because that would break
NAR/substitution accounting when doing remote builds
also, ArchiveDecompressionSource, which is apparently used for
decompressing logs etc, does its own custom decompression without using
unpackTarfile, so it is not affected by this. from what i can tell,
unpackTarfile is only used for:
- fetchtarball fetcher
- git fetcher
- channel unpacking
- nix-prefetch-url
Fixes#1213
Change-Id: I68f3be520681d8889292353dd41ab19a60c5eeea
string data shares a buffer with the binary string length field. size
calculations for string read buffers always include the length field;
sufficiently large length fields can cause these calculations to wrap.
a malicious nar could use this for OOB writes in the daemon (as root).
since we use strings only as tags for archive members and for symlinks
with their OS-dependent length limits we can simply limit string size.
1 MiB should be sufficient for all symlinks, and tags are always tiny.
Change-Id: I89fb05f73c1dbeda45d91244aba4cd526a3d83e1
This partitions a string into two parts based on the first occurrence of
the separator.
This function handles the case when the separator never occur in the
target string by returning an optional<T> in the second argument of the
pair.
Change-Id: I9ad44375dbaa137a4d21d2408bc2ffa34fab62a7
Signed-off-by: Raito Bezarius <raito@lix.systems>
overridden-ness can be reset, and is usually reset such that setting
value taken from a config file are not considered as overriden. when
launching builtin builders we *do* want to send config file settings
changes to the builder, so we'll need one more getter method for it.
Change-Id: I861538a469121c77ebc1898a276439e6b756797d
cancelling a get() promise could leave the pool with allegedly in use
resource handles that do not actually exist. this would cause asserts
in ~Pool to fire (and crash the entire program due to bad accounting)
probably fixes#1041
Change-Id: Ic61ab9b658f02f71e8be7577b8a1115600846f86
This will probably get the implementation of the fixtures revised when
we land the new extraction code, but we are setting it up to be generic
against that.
The operator-> thing is kind of a crime. But it also makes the code
vastly more readable so it's impossible to say if it's bad or not.
Change-Id: Ia5aca69cefaa03cd533ad19d20d856ff7e76a546
base32Encode now takes std::span<std::byte>, with a base32EncodeStr
convenience wrapper which takes std::string_view.
Co-authored-by: Qyriad <qyriad@qyriad.me>
Change-Id: I6a6a6964f799dc84ecbfb55c7ca03a064cff71d9
preserve type information of wrapped exception when constructing a
ForeignException. this requires using `std::current_exception` for
construction in all cases as passing the exception by reference to
any function erases dynamic type information: `throw` may copy the
thrown exception from its source expression; this only works as we
need it to if static and dynamic type of the expression are equal.
fixes#1000
Change-Id: I7167f298540de7c43c2566592bdb8d7a93c34d31
The variant has on the left-hand side the topologically sorted vector
and the right-hand side is a pair showing the path and its parent that
represent a cycle in the graph making the sort impossible.
The goal is to implement #551 which needs to throw an error if the
topo-sort fails. However, the error-message is supposed to contain a
graph of store-paths and the API to generate this is inherently async.
Now, catching the exception and re-throwing another one is impossible
since `co_await` is forbidden in `catch`-blocks and adding another
topoSort variant that allows an async `makeError` also seems odd. Hence,
I decided to alter the data-structure in use a bit for this use-case.
One out of two uses of the function are affected after all.
Change-Id: I70a987f470437df8beb3b1cc203ff88701d0aa1b
Closes#932
`connect-timeout` gets replaced by an exponential backoff for the
download timeout where the initial value is controlled by the setting
`initial-connect-timeout`.
Per iteration, the upper limit of the timeout is increased set to
timeout := min(max_connect_timeout, initial_connect_timeout * 2^i)
I decided to move the entire timeout / tracking of attempts into its own
class to not make the filetransfer implementation more complex. Also,
that allows us to write unit-tests for it.
Setting `--download-attempts` to `0` is forbidden now and an exception
will be thrown. For `--offline` we set it to `1`, the behavior is
equivalent to what it was before: whether the max tries were exceeded is
only checked after the first download exception got thrown, i.e. there's
still one attempt being made.
The end-result - with timeouts being caused by a wrongly set proxy -
looks like this:
$ env HTTPS_PROXY=1.1.1.1 nix store ping --store https://example.com
warning: error: unable to download 'https://example.com/nix-cache-info': Connection timed out after 5006 milliseconds (curl error code=28); retrying in 422ms ms (attempt 1/5)
warning: error: unable to download 'https://example.com/nix-cache-info': Connection timed out after 10010 milliseconds (curl error code=28); retrying in 1003ms ms (attempt 2/5)
warning: error: unable to download 'https://example.com/nix-cache-info': Connection timed out after 20020 milliseconds (curl error code=28); retrying in 2018ms ms (attempt 3/5)
warning: error: unable to download 'https://example.com/nix-cache-info': Connection timed out after 40007 milliseconds (curl error code=28); retrying in 4087ms ms (attempt 4/5)
error: unable to download 'https://example.com/nix-cache-info': Connection timed out after 80074 milliseconds (curl error code=28)
Change-Id: I9e8d08d78275bcf60080d663febc9e075243d36b
Concept: what if you could, in your fancy terminal, in the year of our
lord 2025, just click on the attrs you're looking at to go to where
they're defined. Currently we only expose this info as
builtins.unsafeGetAttrPos, which is inconvenient as it's not
discoverable to users.
By putting it in this more visible yet invisible spot, it's more likely
to be more useful to more people.
In the current state, this is not the most useful ever due to stuff like
https://github.com/neovim/neovim/discussions/35097. However, it can be
expanded by perhaps adding something like the url format setting ripgrep
has.
Change-Id: I3947f97d5c2056d59099af468d7b855486438227
this is an equivalent of the regular kj read interface which also takes
a min/max pair. we do not need this very often though, so we'll keep it
as a separate method for now. if we do find we use it more than read we
can still rename read to readSingle and readRange to read. we will see.
Change-Id: Ib04ca146911adae7081cf4b2df097217ea5fe9f8
Usually, EOFs are represented by returning 0 in the `read` APIs, at
least, this is what read(2) dictate.
As clever creature, we may sum zeroes sometimes (advanced form:
`buf->added(got)`) and forego handling the EOF condition.
To avoid the bug that lurked in remote-store.cc and caused busy looping
if the remote end disconnects suddenly, we return
`Result<Option<size_t>>` forcing the caller to perform a specific
processing for the EOF situation.
The conversion did not raise any other offending code path.
Change-Id: I185fdcb77aa82d87ab0802d66ac37c1363657a73
Signed-off-by: Raito Bezarius <raito@lix.systems>
This is useful for certain error recovery paths (no pun intended) that
does not thread through the original path name.
Change-Id: I2d800740cb4f9912e64c923120d3f977c58ccb7e
Signed-off-by: Raito Bezarius <raito@lix.systems>
processStderr of RemoteStore wants to be a promise and it must be used
from connection setup, so the pool factory callback must be a promise.
Change-Id: I9ac742b6048ae6dba0bfa5dcb58971386229690b
the rpc transition will require sync and async objects to share a single
io buffer (since defining serializers on async is an immense pain in the
tail, slow, and ultimately not necessary). a generic buffer class allows
us to reuse existing serializers more readily (reuse them at all, even).
Change-Id: I5ebba8449f26f2bb76016818928183c7e0123be0
the old protocols are largely untested, mostly unused, and have design
problems that make the RPC transition a lot harder, if not impossible.
in theory we could ship a transparent protocol-converting proxy that'd
isolate the daemon itself from old protocol versions, but that's a lot
of code to maintain for presumably little gain or even no gain at all.
Change-Id: I4c3f3bb34d39044f6aeb07c10caaf13b8340a220
This moves the original test suite for `filterANSIEscapes` into the same
file as the newer tests. There is some overlap between the old and new
tests but that doesn't hurt anything so I kept them as-is.
Change-Id: Id00000009919024a5f206ec9a7bc0022541ff612
This teaches `filterANSIEscapes()` how to find the end of an OSC
sequence. It also keeps OSC 8 (hyperlinks) when not instructed to filter
out all escapes, just as it keeps colors.
This also relaxes the parsing of CSI escapes to find the end of the
sequence for invalid sequences, and handles better escapes that don't
start CSI or OSC.
This fixes the repl output for `:doc builtins.fetchGit`.
Fixes: https://git.lix.systems/lix-project/lix/issues/160
Change-Id: Id0000000f2a6956c042c883a4545edf347fa1799
This was an absolute nightmare to diagnose. It turns out there's a
kernel bug: poll with events = POLLHUP will receive an event for NOT
POLLHUP internally in the kernel, delete their event subscription, and
then not receive events for any HUP later. lol! lmao!!
We choose to use plain old EVFILT_READ because the watched fd can be
either a socket or a pipe and it's preferable to eat some spurious
wakeups than have separate paths for those. The alternative is using
EVFILT_SOCK, a private API that's existed for years and which netty
uses for its sockets, but that doesn't work on pipes.
Fixes: https://git.lix.systems/lix-project/lix/issues/729
Change-Id: If72b5d7a39f00320a9acccdbe81121cdb1a04c45
otherwise lix may crash when e.g. nix search receives invalid regex.
we now also give better error messages for regex errors during eval.
fixes#803
Change-Id: Icc7c578ff488ba520efac5d898572ccf4486e9a8
This could previously crash lix:
Before:
$ nix eval -E '{type="derivation"; drvPath="";}'
nix: lix/libutil/file-system.cc:45: Path nix::canonPath(PathView, bool): Assertion `path != ""' failed.
Aborted (core dumped)
After:
$ nix eval -E '{type="derivation"; drvPath="";}'
error:
… while evaluating the drvPath of a derivation
at «string»:1:21:
1| {type="derivation"; drvPath="";}
| ^
error: path '' is not in the Nix store
Fixes#536
Change-Id: I406dc9e58047be8f263cf2e4bc3ed5da75a46602
There is a pull request [1] addressing these upstream--it doesn't appear
likely to be merged anytime soon though... this is a no-op til we enable
-Wdeprecated-declarations, but helps in the direction of #744
[1]: https://github.com/emil-e/rapidcheck/pull/325
Change-Id: I27e2c7d81df152de8674696f2a56d5f21c414ce3
we generally do not want to catch or throw these. catching them to print
and discard is fine, tests are largely exempt, and cases in which we can
be certain where the exception came from are also fine to *catch*. we'll
try to never *throw* (or rethrow) these if possible though because doing
so will make it impossible to construct async traces for the exceptions.
Change-Id: I3b71c32ecd16afc2246c946472f5629a1fa31f2c
Well, I was trying to figure out
https://git.lix.systems/lix-project/lix/issues/729 in which this feature
is clearly just broken on macOS, but frustratingly, it seems that it
*does* work, except for the daemon. um........ sure.
Change-Id: Iaa962f045c16fdfa82854151c90a03e5cf0eea47
Yikes!! I wonder if we have any other ones of these that just .. didn't
get added to a meson file?
Change-Id: I19480ab03cdbecf608e523d5b6c3980233f4f445
or more accurately, wrap them in a nix::Error subclass so we can display
them properly without crashing, and add some error context if available.
fixes#642fixes#753fixes#759fixes#769
Change-Id: I1aad0c0501fea83f9de3a1335eaa6adc20721616
allow opting in to serialization as integers via a trait type instead,
and add string-list serializers for the feature flag set enumerations.
fixes#738
Change-Id: I2746eb5ef1f15c01b4e681f9ba1615b6c6e64f44
we want to own this specialization fully so we can change the default
serializer behavior without also forcing downstream users of our code
to use the same behavior. it'll also let us do things we cannot do in
regular nlohmann::json, such as selectively enabling serialization of
enums as integral types, or using `to_json`/`from_json` overloads for
not-default-constructible types instead of serializer specializations
Change-Id: I91a1db362e37d654090f1824b1cd3ce783d32134
this will become a proper specialization of `nlohmann::basic_json` soon.
specialing basic_json will let us get rid of our `adl_serializer` hacks,
and it'll open the door to better enum serializing behavior without also
forcing all those who use lix as a library to set certain defines (which
may not even be possible depending on how those users use json already).
Change-Id: I5228d2b9df581a189552c993363207cfbd20f445
this doesn't do much, just wrap a few nlohmann headers in headers of our
own (and delete includes we don't need because they're transitively seen
by other includes). doing this now will make the next change much nicer.
Change-Id: I166933102ea86bb5322ebbf9ba9411f96032a53b
if a copyNAR generator was not drained to completion it would not read
the full nar data from its source. this could happen if the copier was
passed to parseAndDump wrapped as a source because copyNAR would yield
nar metadata *before* it had read it, and GeneratorSource will drain a
generator fully *only* if the source is allowed to throw EndOfFile. in
the parseAndDump case this never happened because parseAndDump expects
to be given an unterminated stream, and thus the combination left some
nar metadata in the input Source, breaking the remote store protocols.
fixes#732
Change-Id: Ia59a53375992bfcdb7bc6b37764ca779622bc8f7
eldritch horrors