only dynamic derivations could produce a non-opaque drvPath. since
dynamic derivations are no longer supported we can have drvPath be
opaque at all times, simplifying downstream code significantly and
making quite a few methods unnecessary. discardOutputPath was only
called on drvPath members anyway and thus reduces to a copy, other
operations at the very least are no longer recursive. some vestige
of dynamic derivations remains in DerivedPathMap though (for now).
Change-Id: Ifb4ad53a3c67800be5a62540068c8279d4ae0046
string context doesn't need any tests because it's never persisted or
shown to the user. getting rid of recursive string context means that
the context string parsers can be a lot simpler from here on forward.
Change-Id: I58443679ad76c0f28ea5f4eb8bfb3874f270e764
with impure derivations gone we move on to dynamic derivations. this too
is not done in a single commit because dynamic derivations are invasive,
modifying semantics of all references to derivation output paths and all
derivation dependency calculations. removing dynamic derivations cleanly
is made significantly harder by the multiple did-you-mean-sum types, aka
"wrappers for std::variant", holding all derivation outpath information.
Change-Id: Ice7a7700c7b54c6a6061d4beb322b4175923d27a
we don't remove the entire feature in one go to make review easier.
impure derivations are rather unintrusive on their own, at least if
we compare them to dynamic or ca derivations in general, so we will
be done with this soon. as it stands impure derivations cannot work
without ca derivations, and those we *really* want to leave behind.
Change-Id: I4f01d8d758b2c85dcd6c3078304b5ee1b52f65b0
otherwise lix may crash when e.g. nix search receives invalid regex.
we now also give better error messages for regex errors during eval.
fixes#803
Change-Id: Icc7c578ff488ba520efac5d898572ccf4486e9a8
There is a pull request [1] addressing these upstream--it doesn't appear
likely to be merged anytime soon though... this is a no-op til we enable
-Wdeprecated-declarations, but helps in the direction of #744
[1]: https://github.com/emil-e/rapidcheck/pull/325
Change-Id: I27e2c7d81df152de8674696f2a56d5f21c414ce3
This reverts commit bba678e5c5.
Reason for revert: didn't fix the bug and created new ones (fj#794)
Change-Id: I0450205d3041b6c876737151a4051081c1366f1d
or more accurately, wrap them in a nix::Error subclass so we can display
them properly without crashing, and add some error context if available.
fixes#642fixes#753fixes#759fixes#769
Change-Id: I1aad0c0501fea83f9de3a1335eaa6adc20721616
we want to own this specialization fully so we can change the default
serializer behavior without also forcing downstream users of our code
to use the same behavior. it'll also let us do things we cannot do in
regular nlohmann::json, such as selectively enabling serialization of
enums as integral types, or using `to_json`/`from_json` overloads for
not-default-constructible types instead of serializer specializations
Change-Id: I91a1db362e37d654090f1824b1cd3ce783d32134
this will become a proper specialization of `nlohmann::basic_json` soon.
specialing basic_json will let us get rid of our `adl_serializer` hacks,
and it'll open the door to better enum serializing behavior without also
forcing all those who use lix as a library to set certain defines (which
may not even be possible depending on how those users use json already).
Change-Id: I5228d2b9df581a189552c993363207cfbd20f445
this doesn't do much, just wrap a few nlohmann headers in headers of our
own (and delete includes we don't need because they're transitively seen
by other includes). doing this now will make the next change much nicer.
Change-Id: I166933102ea86bb5322ebbf9ba9411f96032a53b
also mark the sync version as NeverAsync. a blocking wait on sqlite
locks in a coroutine may never finish if it's a different coroutine
on the same executor that is holding the lock, not another process.
this propagates to the sqlite core interface, but no further. we'll
assume that caches do not block on a database for very long, and we
can't reasonably propagate never-async-ness out of stores unless we
touch everything we'd touch for the async transition, again, twice.
store code already assumes that it can block for however long it'll
feel like that moment. we keep thread pools around for this reason.
Change-Id: I62f77e1ac333cbe2e4e646dbcb1571463f2cf3fc
There's a race condition where awaitData could early-return for data
coming from a 404 response or similar and thus not rethrow the exception
that is forthcoming, and a related race during transfer setup (which
could retry a transfer *twice* per retry round).
This would then cause substitution failures like below since the exception
isn't caught in HttpBinaryCacheStore::getFile as intended, but instead
by an exception handler downstream of `drain()` which would error out
the entire operation.
Symptom:
» nix-build ./docs-service.nix -o "docs-service-result"
error: unable to download 'https://cache.nixos.org/7mr3fy8w66gi5inmf0jkkkl90lxy4jyg.narinfo': HTTP e
rror 404 ()
response body:
This is kind of a hack in how it is implemented: it assumes that you
can't intentionally be receiving a large unsuccessful response since in
such a case, `awaitData` will wait for finish() to be called to throw an
exception and will never escape until the download finishes, while
continuing to buffer the entire response into memory, which could be bad
if an error response had a large payload.
That said, nobody is sending Lix 1GiB of 404, so meh I guess, and this
is how it is seemingly intended to work. That was a design flaw of the
thing before any of the Lix team got our paws on it.
I tested this by adding _exit(0) inside the expected exception catch and
then running the offending command repeatedly to see if the symptom ever
appeared again, and it did not.
Needs cherry-pick to 2.92 and a 2.92.1 release once reviewed.
Fixes: https://git.lix.systems/lix-project/lix/issues/635
Co-Authored-By: lix@jade.fyi
Change-Id: If54f6eeaad60b5ca9d5b77d4d9232da1d295e7d1
in a daemon all calls to the logger can throw an Interrupted exception,
which so far has silently stopped the curl thread without notifying its
transfers and leaving them stuck as a result. ensuring that the loggers
can never throw Interrupted will have very unpleasant side-effects, and
throwing depending on context requires large amount of bookkeeping. for
now it is easiest to abort all transfers on Interrupted during cleanup.
the test for this is extremely sketchy because we want to hit a single,
very specifically chosen, loger call in TransferItem::finish(). the bug
was triggered by the `act.progress` further down from what we're aiming
for, but that one is much harder to select for than the debug log here.
fixes#613
Change-Id: Id72efa64dd30cbbf256d2ab2a328457a0b095c6a
the very slight speedup in config setting access is not worth the
maintenance overhead of conflating concers like this. the virtual
inheritance scheme used for configs requires too much duplication
of base class constructor arguments to be worth doing. perhaps we
should get rid of all virtual inheritance of data-membered bases?
Change-Id: I4acf5ceaedb4ed7476efe1114c2e065ec72d2c6d
Ever read gdb output and you just kinda get a headache because you have
to infer what a thread is by reading the stack trace? It's not hard, but
we could also just never have to do that again, which is also not hard.
Sample:
(gdb) info thr
Id Target Id Frame
* 1 LWP 3719283 "nix-daemon" 0x00007e558587da0f in accept ()
from target:/nix/store/c10zhkbp6jmyh0xc5kd123ga8yy2p4hk-glibc-2.39-52/lib/libc.so.6
2 LWP 3719284 "signal handler" 0x00007e55857b2bea in sigtimedwait ()
from target:/nix/store/c10zhkbp6jmyh0xc5kd123ga8yy2p4hk-glibc-2.39-52/lib/libc.so.6
The API design for this is forced by the macOS pthread_setname_np only
being able to change the current thread's name, but if we just conform
everything to that, it works everywhere.
Change-Id: I2b1d6ed41e3c94170cb0b4e73ad66f239ebd9c88
file:// urls are no longer handled by curl itself, and this is supposed
to test the curl wrapper. use http to force the wrapper to be involved.
Change-Id: Ib13087db07b3b2f1ae44ce8e3ec7a96d935b1bab
this only ever worked for empty uploads, and there it worked only by
complete accident: curl was asked to send more data than the wrapper
would provide, which curl would not like and report as an error. the
error would cause a retry with even less data to send, until finally
failing by running into the retry limit. let's just forbid all this.
Change-Id: I229a94b3b8b33e2c6cdb8ea19edd57cd6740e6c6
This adds a new temp-dir setting for controlling the temporary directory
without having to change the TMPDIR env var. This can be used to e.g.
use a path on a case-sensitive store on macOS for temporary files
without changing the TMPDIR var used by interactive shells or commands
invoked with `nix run`.
This also stops unsetting `TMPDIR` on darwin when the env var value
starts with `/var/folders/`, preferring instead to just do the check
when reading `TMPDIR`. This way the inherited `TMPDIR` env var is
preserved for child processes (such as interactive shells).
As a side effect this changes the behavior of `nix-build -o ''` to act
like `nix-build --no-out-link` instead of failing with an error caused
by trying to create a symlink at the cwd.
Fixes: https://git.lix.systems/lix-project/lix/issues/253
Fixes: https://git.lix.systems/lix-project/lix/issues/112
Change-Id: I9ee826323f2deca62854715a77ca7a373a948a29
don't pause the entire curl thread. we have multiple consumer threads
after all, not just one, so stalling all of them is likely not great.
note that libcurl advises against using transfer pauses if compressed
encodings are allowed and automatically decoded. this should not lead
to problems in practice because our data is usually not compressed to
such a degree that curl buffering *uncompressed* data matters. should
this cause problems we can reintroduce the whole-thread pause, but we
will probably get away with this until the entire file transfer class
is made kj::Promise-using async (and *then* curl can be hardpaused if
it cannot get rid of its data, solving the problem once and for all).
Change-Id: I218e41bfa5a27c7454eafb0bdb54f2a29a7f6493
it's no longer needed. `download` can do everything `enqueueDownload`
did, and a lot more. e.g. not block the calling thread, for instance.
Change-Id: I4b36235ed707c92d117b4c33efa3db50d26f9a84
return it as a separate item in a pair instead. this will let us remove
enqueueDownload() in favor of returning metadata from download() itself
Change-Id: I74fad2ca15f920da1eefabc950c2baa2c360f2ba
it's just a uri and some headers now. those can be function arguments
with no loss of clarity. *actual* additional arguments, for example a
TLS context with additional certificates, could be added on a new and
improved FileTransfer class that carries not just a backend reference
but some real, visible context for its transfers. curl not being very
multi-threading-friendly when using multi handles will make sharing a
bit hard anyway once we drop the single global download worker thread
Change-Id: Id2112c95cbd118c6d920488f38d272d7da926460
we don't even need this outside of tests. maybe we should not do
automatic retries at this level at all and use retrying wrappers
instead? at some point we may have to do this, but not just yet.
Change-Id: If0088aa55215be81f1770c25b3bb1b5268c65cf8
Only overridden settings are sent to the daemon, and we're going to do the same
for the build hook to. It needs to be ensured that overridden settings are in
fact consistently marked as such, so that they actually get sent.
Change-Id: I7cd58d925702f86cf2c35ad121eb191ceb62a355
There was a bug report about a potential call to `memcpy` with a null
pointer which is not reproducible:
https://git.lix.systems/lix-project/lix/issues/492
This occurred in `src/libstore/filetransfer.cc` in `InnerSource::read`.
To ensure that this doesn't happen, an early return is added before
calling `memcpy` if the length of the data to be copied is 0.
This change also adds a test that ensures that when `InnerSource::read`
is called with an empty file, it throws an `EndOfFile` exception.
Change-Id: Ia18149bee9a3488576c864f28475a3a0c9eadfbb
This:
- Converts a bunch of C style casts into C++ casts.
- Removes some very silly pointer subtraction code (which is no more or
less busted on i686 than it began)
- Fixes some "technically UB" that never had to be UB in the first
place.
- Makes finally follow the noexcept status of the inner function. Maybe
in the future we should ban the function from not being noexcept, but
that is not today.
- Makes various locally-used exceptions inherit from std::exception.
Change-Id: I22e66972602604989b5e494fd940b93e0e6e9297
this is cursed. deeply and profoundly cursed. under NO CIRCUMSTANCES
must protocol serializer helpers be applied to temporaries! doing so
will inevitably cause dangling references and cause the entire thing
to crash. we need to do this even so to get rid of boost coroutines,
and likewise to encapsulate the serializers we suffer today at least
a little bit to allow a gradual migration to an actual IPC protocol.
(this isn't a problem that's unique to generators. c++ coroutines in
general cannot safely take references to arbitrary temporaries since
c++ does not have a lifetime system that can make this safe. -sigh-)
Change-Id: I2921ba451e04d86798752d140885d3c5cc08e146
without this we will not be able to get rid of makeDecompressionSink,
which in turn will be necessary to get rid of sourceToSink (since the
libarchive archive wrapper *must* be a Source due to api limitations)
Change-Id: Iccd3d333ba2cbcab49cb5a1d3125624de16bce27
even the transfer function is not all that necessary since there aren't
that many users, but we'll keep it for now. we could've kept both names
but we also kind of want to use `download` for something else very soon
Change-Id: I005e403ee59de433e139e37aa2045c26a523ccbf
We don't want to deal with these at all, let's stop doing so.
(marking this one as the fix commit since its immediate predecessors
aren't the complete fix)
Fixes: https://git.lix.systems/lix-project/lix/issues/325
Change-Id: Ieea1b0b8ac0f903d1e24e5b3e63cfe12eeec119d
eldritch horrors