warning: embedding a directive within macro arguments has undefined behavior
????? CLANG? why does this take -pedantic to show??
Change-Id: I6f4a9f3db1b89448f9f5d00d82b551e46a6a6964
the wrapper is needed by transfer streams to restart a failed transfer
if desired. curlFileTransfer itself is more of a fancy handler for the
thread we're dedicating to curl io handling. the thread will stay with
the multi handle for now because quit handling needs to stay there. we
could have CurlMulti keep only a flag, but that does not help us much.
Change-Id: I99550f0bbb635b75898ca7260f08275df86050e3
currently all loggers can always accept messages and never suggest
flushing buffers. in the future this may change, and at that point
we're already fully set up for it. local loggers should never keep
asynchronous (i.e. network-backed) buffers, disk buffers are fine.
networked loggers will require buffers and periodic flushes later.
Change-Id: Ide2114f5bc17f4a1d289c92ed4f9381a1d59dacf
If none is given, we fall back to whatever accessor we get from the
store.
To display which paths actually contain the references leading to
e.g. a cycle or triggering a disallowedRequisites error, we'd
potentially have to look into the chroot from the previously finished
build. This behavior should not be part of the local accessor by
default, but part of a "special" accessor. This change allows using such
an accessor for `genGraphString()`.
Now that we inject the accessor from the outside, we have to mock it
anyways in the tests. Hence, this also adds a testcase for the
precise=True case.
Change-Id: I58465fb944776c2b0262ba054d1f296ed2ae3406
the exception no longer actually matters, only that the download stream
is destroyed before the download finishes does. exceptions during drain
calls on the returned stream will cause the stream to be destroyed, but
draining this stream is just a remnant of the old download-to-sink API.
Change-Id: Ic6de40deb2ccff09d77180148afe746f29f55d23
Usually, EOFs are represented by returning 0 in the `read` APIs, at
least, this is what read(2) dictate.
As clever creature, we may sum zeroes sometimes (advanced form:
`buf->added(got)`) and forego handling the EOF condition.
To avoid the bug that lurked in remote-store.cc and caused busy looping
if the remote end disconnects suddenly, we return
`Result<Option<size_t>>` forcing the caller to perform a specific
processing for the EOF situation.
The conversion did not raise any other offending code path.
Change-Id: I185fdcb77aa82d87ab0802d66ac37c1363657a73
Signed-off-by: Raito Bezarius <raito@lix.systems>
Commits 205c59367c and
325e7e1824 introduced real glibc store paths from
current nixpkgs unstable into the source. Since nixpkgs `fetchFromGitea` (and
similar fixed-output derivations) depends on a C library, on x86_64-linux they
will fail with the forbidden reference error:
error: the fixed-output derivation '/nix/store/wnmnj3jzc82y89sfmyicr04kilg8zs2k-source.drv' must not reference store paths but 1 such references were found:
/nix/store/q4wq65gl3r8fy746v9bbwgx4gzn0r2kl-glibc-2.40-66
Falsify the store path to prevent this failure.
Change-Id: I949033567bcad070f9a0a19cefdb33a79222e421
That way it's easier to spot whether a node is the "final" node in the
graph which is especially helpful for larger graphs.
Change-Id: I460a699f07f5455917792599f4247ebf8f430d93
This will be useful for other things as well such as the
disallowedRequisites error in the builder code. Additionally, print the
dependencyPath in the tree bold to spot where a change terminates.
Also implemented some unit-tests for this code.
Change-Id: I8460f3f6c5095d5bfbe390f223bc0252800dca5e
making stores and their users fully async requires all data streams to
be async. the most notable data streams in common usage are curl first
and remote stores second. curl is much more contained today and easier
to asyncify (with the preparatory work we've done in the past commits)
Change-Id: I2d6ff4687ee2b47e4efaa6714827b7283bed941d
the test is for the map that usually wraps it though because it's the
bit we're interested in replacing, and it has custom serializer code.
Change-Id: If77a236dfca738b646ed2b7a5c65515dad6b7295
the old protocols are largely untested, mostly unused, and have design
problems that make the RPC transition a lot harder, if not impossible.
in theory we could ship a transparent protocol-converting proxy that'd
isolate the daemon itself from old protocol versions, but that's a lot
of code to maintain for presumably little gain or even no gain at all.
Change-Id: I4c3f3bb34d39044f6aeb07c10caaf13b8340a220
we can't create these any more except by reading an old json-formatted
derivation that used them. since we cannot do anything with a deferred
derivation even when read we will remove json support for them as well
Change-Id: I4f9ea0b7c6469f57977784037f7710f939e40a2c
we no longer have any experimental features depending on ca derivations,
so we can start removing them. since ca derivations are very invasive we
will need a while to remove all of the explicitly experimental code, and
even then we will not have removed *all* code related to ca derivations.
especially in the derivation goals there is a lot of code that is not as
easy to disentangle from experimental features as some would have hoped.
Change-Id: Ia456aadc6164613ded343f571318494d9310a549
only dynamic derivations could produce a non-opaque drvPath. since
dynamic derivations are no longer supported we can have drvPath be
opaque at all times, simplifying downstream code significantly and
making quite a few methods unnecessary. discardOutputPath was only
called on drvPath members anyway and thus reduces to a copy, other
operations at the very least are no longer recursive. some vestige
of dynamic derivations remains in DerivedPathMap though (for now).
Change-Id: Ifb4ad53a3c67800be5a62540068c8279d4ae0046
string context doesn't need any tests because it's never persisted or
shown to the user. getting rid of recursive string context means that
the context string parsers can be a lot simpler from here on forward.
Change-Id: I58443679ad76c0f28ea5f4eb8bfb3874f270e764
with impure derivations gone we move on to dynamic derivations. this too
is not done in a single commit because dynamic derivations are invasive,
modifying semantics of all references to derivation output paths and all
derivation dependency calculations. removing dynamic derivations cleanly
is made significantly harder by the multiple did-you-mean-sum types, aka
"wrappers for std::variant", holding all derivation outpath information.
Change-Id: Ice7a7700c7b54c6a6061d4beb322b4175923d27a
we don't remove the entire feature in one go to make review easier.
impure derivations are rather unintrusive on their own, at least if
we compare them to dynamic or ca derivations in general, so we will
be done with this soon. as it stands impure derivations cannot work
without ca derivations, and those we *really* want to leave behind.
Change-Id: I4f01d8d758b2c85dcd6c3078304b5ee1b52f65b0
otherwise lix may crash when e.g. nix search receives invalid regex.
we now also give better error messages for regex errors during eval.
fixes#803
Change-Id: Icc7c578ff488ba520efac5d898572ccf4486e9a8
There is a pull request [1] addressing these upstream--it doesn't appear
likely to be merged anytime soon though... this is a no-op til we enable
-Wdeprecated-declarations, but helps in the direction of #744
[1]: https://github.com/emil-e/rapidcheck/pull/325
Change-Id: I27e2c7d81df152de8674696f2a56d5f21c414ce3
This reverts commit bba678e5c5.
Reason for revert: didn't fix the bug and created new ones (fj#794)
Change-Id: I0450205d3041b6c876737151a4051081c1366f1d
or more accurately, wrap them in a nix::Error subclass so we can display
them properly without crashing, and add some error context if available.
fixes#642fixes#753fixes#759fixes#769
Change-Id: I1aad0c0501fea83f9de3a1335eaa6adc20721616
we want to own this specialization fully so we can change the default
serializer behavior without also forcing downstream users of our code
to use the same behavior. it'll also let us do things we cannot do in
regular nlohmann::json, such as selectively enabling serialization of
enums as integral types, or using `to_json`/`from_json` overloads for
not-default-constructible types instead of serializer specializations
Change-Id: I91a1db362e37d654090f1824b1cd3ce783d32134
this will become a proper specialization of `nlohmann::basic_json` soon.
specialing basic_json will let us get rid of our `adl_serializer` hacks,
and it'll open the door to better enum serializing behavior without also
forcing all those who use lix as a library to set certain defines (which
may not even be possible depending on how those users use json already).
Change-Id: I5228d2b9df581a189552c993363207cfbd20f445
this doesn't do much, just wrap a few nlohmann headers in headers of our
own (and delete includes we don't need because they're transitively seen
by other includes). doing this now will make the next change much nicer.
Change-Id: I166933102ea86bb5322ebbf9ba9411f96032a53b
also mark the sync version as NeverAsync. a blocking wait on sqlite
locks in a coroutine may never finish if it's a different coroutine
on the same executor that is holding the lock, not another process.
this propagates to the sqlite core interface, but no further. we'll
assume that caches do not block on a database for very long, and we
can't reasonably propagate never-async-ness out of stores unless we
touch everything we'd touch for the async transition, again, twice.
store code already assumes that it can block for however long it'll
feel like that moment. we keep thread pools around for this reason.
Change-Id: I62f77e1ac333cbe2e4e646dbcb1571463f2cf3fc
There's a race condition where awaitData could early-return for data
coming from a 404 response or similar and thus not rethrow the exception
that is forthcoming, and a related race during transfer setup (which
could retry a transfer *twice* per retry round).
This would then cause substitution failures like below since the exception
isn't caught in HttpBinaryCacheStore::getFile as intended, but instead
by an exception handler downstream of `drain()` which would error out
the entire operation.
Symptom:
» nix-build ./docs-service.nix -o "docs-service-result"
error: unable to download 'https://cache.nixos.org/7mr3fy8w66gi5inmf0jkkkl90lxy4jyg.narinfo': HTTP e
rror 404 ()
response body:
This is kind of a hack in how it is implemented: it assumes that you
can't intentionally be receiving a large unsuccessful response since in
such a case, `awaitData` will wait for finish() to be called to throw an
exception and will never escape until the download finishes, while
continuing to buffer the entire response into memory, which could be bad
if an error response had a large payload.
That said, nobody is sending Lix 1GiB of 404, so meh I guess, and this
is how it is seemingly intended to work. That was a design flaw of the
thing before any of the Lix team got our paws on it.
I tested this by adding _exit(0) inside the expected exception catch and
then running the offending command repeatedly to see if the symptom ever
appeared again, and it did not.
Needs cherry-pick to 2.92 and a 2.92.1 release once reviewed.
Fixes: https://git.lix.systems/lix-project/lix/issues/635
Co-Authored-By: lix@jade.fyi
Change-Id: If54f6eeaad60b5ca9d5b77d4d9232da1d295e7d1
in a daemon all calls to the logger can throw an Interrupted exception,
which so far has silently stopped the curl thread without notifying its
transfers and leaving them stuck as a result. ensuring that the loggers
can never throw Interrupted will have very unpleasant side-effects, and
throwing depending on context requires large amount of bookkeeping. for
now it is easiest to abort all transfers on Interrupted during cleanup.
the test for this is extremely sketchy because we want to hit a single,
very specifically chosen, loger call in TransferItem::finish(). the bug
was triggered by the `act.progress` further down from what we're aiming
for, but that one is much harder to select for than the debug log here.
fixes#613
Change-Id: Id72efa64dd30cbbf256d2ab2a328457a0b095c6a
the very slight speedup in config setting access is not worth the
maintenance overhead of conflating concers like this. the virtual
inheritance scheme used for configs requires too much duplication
of base class constructor arguments to be worth doing. perhaps we
should get rid of all virtual inheritance of data-membered bases?
Change-Id: I4acf5ceaedb4ed7476efe1114c2e065ec72d2c6d
Ever read gdb output and you just kinda get a headache because you have
to infer what a thread is by reading the stack trace? It's not hard, but
we could also just never have to do that again, which is also not hard.
Sample:
(gdb) info thr
Id Target Id Frame
* 1 LWP 3719283 "nix-daemon" 0x00007e558587da0f in accept ()
from target:/nix/store/c10zhkbp6jmyh0xc5kd123ga8yy2p4hk-glibc-2.39-52/lib/libc.so.6
2 LWP 3719284 "signal handler" 0x00007e55857b2bea in sigtimedwait ()
from target:/nix/store/c10zhkbp6jmyh0xc5kd123ga8yy2p4hk-glibc-2.39-52/lib/libc.so.6
The API design for this is forced by the macOS pthread_setname_np only
being able to change the current thread's name, but if we just conform
everything to that, it works everywhere.
Change-Id: I2b1d6ed41e3c94170cb0b4e73ad66f239ebd9c88
Qyriad