I was wondering why there were only 59 succeeding jobs. We really need
to fix the buildbot at some point ;;
Change-Id: I81a22e32ac0dd0e105677f1b7bc064568009890d
Since we include kj from a header, we need to add a dependency to force
downstreams to also have it as a declared dependency or at least put it
in their include dir paths.
Given kj is what it is, this is basically unavoidable that this
dependency is visible.
Change-Id: Ia89a90f10ad8b821260db0d7fdc47aebda44e66a
the worker already wants to make a promise explicitly control-C-able,
and some other things in the future will want this as well. we do not
yet have the option to use the signal interfaces kj offers, but if we
can manage to remove all uses of the old-style notifiers we might get
there. until then we can at least wrap the old interfaces to be a bit
nicer to use, and eventually easier to replace with kj-provided code.
Change-Id: I079fdfe7720485820a9862c3615335d4e1df13e7
ifd requires building which uses async code today, fetching requires
network io which really should use async code but does not yet. eval
itself cannot be made async for performance reasons. offloading work
to other threads is possible but questionable as currently there are
no uses in which evaluation must take place from within asynchronous
operations, which are currently confined entirely to store behavior.
Change-Id: I1e3c86fca19b042d79332e9a5ef9546f90a6448b
this will let us add AsyncIoRoots to the toplevel objects and functions.
eventually we'll get rid of all offloading again byt covering the entire
space between main() and the worker entry points with promises. it won't
be a quick and easy journey; we have somewhere around 200 functions that
must be converted to async code, and some building blocks that are fully
incompatible with event loops on their own. this ranges from simple bits
like thread pools (which we can wrap in promise-fulfiller pairs), bigger
bits like the curl wrapper (which we can make async, but not easily), or
even impossible bits (like the libarchive wrapper, which can't be turned
into async code at all. not to worry though, we have *plans* for those.)
Change-Id: I95b91b0545659ece4ec1b85214df1ce2f1bc0164
evaluating promise-based things requires an event loop. we can't thread
the event loop through the code as is because that'd touch almost every
function signature. we can't declare local loops because loops must not
nest. we can't use thread-local loops because non-continuation waits on
promises are not allowed within a promise evaluation. spawning a thread
every time is the only real option besides a shared runner thread pool.
Change-Id: I763c1b2d655cee01f991a8675f3a643b6fe91363
mostly we want to hide the wait scope since using a wait scope inside a
promise is not allowed. we'll use this to push the AsyncIoContext up to
the main() function as time passes, and presumably turn a lot of things
that are regular synchronous code right now into promises as we proceed
Change-Id: Ib972568a306b540b573e946492830ed5a15c485e
Though this is a revert, it also involves fixing the code with new code
and a regression test. The problematic locking that was there before has
been removed, in any case.
This reverts commit ba728e46be.
Reason for revert: regression.
Fixes: https://git.lix.systems/lix-project/lix/issues/647
Change-Id: I326b1a10042ff69afbab38426e042e0dc2224a13
currently this header only contains TRY_AWAIT, and it's public version
LIX_TRY_AWAIT. the former is for use in our cpp files only, the latter
may be used anywhere. we don't expose TRY_AWAIT unconditionally due to
it not being namespaced according to usual rules of macro naming, i.e.
exposing it may litter user macro namespaces unnecessarily. hiding our
internal version is achieved using a new define that's not set through
pkg-config or other means. (we could also add a private header that we
simply do not install, but that gets rather messy rather very quickly)
Change-Id: I372b1a414487dd03cfbfc620b3148a6c6e56ca3f
c++ does not support result types very well. kj promises do not support
exceptions very well, but result types are fine. since we need to treat
errors like values to be compatible with kj we are forced to use result
types. creating a failure result from the current exception is our most
common error return right now, so let's have a shorthand for this. it'd
be nice to also drop the try-catch wrapper somehow, but since macros do
not work with clang-format (macro-wrapped bodies confuse the formatter,
trashing the entire file) and clang plugins tie us even harder to clang
we'll probably have to keep the wrappers (until we can rewrite in rust)
Change-Id: I463dd0a4ebedb79a5335c92251c4d12b9d64b401
This doesn't break any usages within lix, and allows avoiding copying
in some calls. The constructor already copies bits and pieces of the
data passed in.
Change-Id: Iea39906ca430195da4c509dd8a0a54d49b46ca4f
There's a race condition where awaitData could early-return for data
coming from a 404 response or similar and thus not rethrow the exception
that is forthcoming, and a related race during transfer setup (which
could retry a transfer *twice* per retry round).
This would then cause substitution failures like below since the exception
isn't caught in HttpBinaryCacheStore::getFile as intended, but instead
by an exception handler downstream of `drain()` which would error out
the entire operation.
Symptom:
» nix-build ./docs-service.nix -o "docs-service-result"
error: unable to download 'https://cache.nixos.org/7mr3fy8w66gi5inmf0jkkkl90lxy4jyg.narinfo': HTTP e
rror 404 ()
response body:
This is kind of a hack in how it is implemented: it assumes that you
can't intentionally be receiving a large unsuccessful response since in
such a case, `awaitData` will wait for finish() to be called to throw an
exception and will never escape until the download finishes, while
continuing to buffer the entire response into memory, which could be bad
if an error response had a large payload.
That said, nobody is sending Lix 1GiB of 404, so meh I guess, and this
is how it is seemingly intended to work. That was a design flaw of the
thing before any of the Lix team got our paws on it.
I tested this by adding _exit(0) inside the expected exception catch and
then running the offending command repeatedly to see if the symptom ever
appeared again, and it did not.
Needs cherry-pick to 2.92 and a 2.92.1 release once reviewed.
Fixes: https://git.lix.systems/lix-project/lix/issues/635
Co-Authored-By: lix@jade.fyi
Change-Id: If54f6eeaad60b5ca9d5b77d4d9232da1d295e7d1
It has been renamed to `emplaceExpr` to be more consistent with the fact
that it actually creates the expression pointer. Moreover, `pushExpr`
has been added which directly takes a `unique_ptr<Expr>`.
All manual calls to `exprs.emplace_back` have been removed and replaced
with `pushExpr`, and `exprs` is now a protected field. This allows for
enforcing invariants and modifying state within `pushExpr`.
Change-Id: I995f0bdc1c090cf78080c9f0b7737e2be359b3e4
The code for serialization Expr nodes back into (pseudo-)Nix has been
removed for being subtly error-prone and tedious to maintain. Instead,
`nix-instantiate --parse` now prints a JSON representation of the AST.
Usage patterns of the --parse flag I've found in the wild:
1. Check if a file is well-formed, i.e. discard output and test exit code
2. Get parser errors from a file, i.e. discard stdout and use stderr
3. Nixfmt uses --parse to test equivalence pre/post format, and that property is (should be?) preserved
None of these should break with the current change
Closes#487
Change-Id: Icdbaad17790f2ad8765fa08e02e6597ee4c7a909
the default implementation using getFile does exactly the same thing,
unless the thing being downloaded goes away immediately before we try
to resume a partial download. that's best treated as a different kind
of error though because the target did exist the first time we asked.
Change-Id: Ic0c046bdcd9d9751c6ba78167c68a38d0d2739e9
retries must happen outside of the state lock lifetime, otherwise
retries may keep transactions alive that should have been killed.
Change-Id: I9088975283aac9fc4521aa2f0bd000335740f6c7
it's dirt cheap to do on average, the complexity of sticking this under
a lock is not worth it. we may want to split this into not loading both
kinds of keys since private keys are rarely needed, but since only root
or the daemon are likely to have access to them it shouldn't be urgent.
Change-Id: I691aed100c9cc4ca32ab7e99a37b5be7d5c25e93
destruction of static objects is not sequenced before termination of
detached threads. this means that a detached thread can hold on to a
reference to a static object and access that object after main() has
returned and the static object destructor has run. ReceiveInterrupts
is one such case, but moving the shared state to the heap solves it.
Change-Id: Id597365ce9386000d171a03323169aafc542aa2c
this will make it easier to return async streams instead of sources at
some point in the future. the primary benefactors of the current state
are not greatly inconvenienced by the api change, and would need to be
changed much as they are now once async streams come around either way
Change-Id: I4db9ea8b186f358c239f7863ac8140c500986c2d
Original-Author: picnoir <picnoir@alternativebit.fr>
Inspired from
https://github.com/NixOS/nix/pull/11922/commits/ced8d311a593fcf9c3823e4e118474ac132d8e60
and adapted for Lix needs.
TL;DR: The topological sort should ensure that it is possible to delete
the path iterated upon. Nonetheless, in some cases,
`invalidatePathChecked` can still throw `PathInUse`, the exception
bubbles up and cancel the garbage collection procedure, leaving the rest
of the paths untouched. This change ensure that the error is logged for
further investigation but doesn't prevent the GC to continue when it
can.
After code review, we decided to make it a `printInfo` to inform the
user about sudden "in use" dependencies during garbage collection and
let them re-run garbage collection if they care about this.
References: https://github.com/NixOS/nix/issues/11923
References: https://git.lix.systems/lix-project/lix/issues/621
Change-Id: I5606c9afd16b5faa747b713fde2dc24016990ba3
Signed-off-by: Raito Bezarius <raito@lix.systems>
Before the change "illegal reference" was hard to interpret as it did
not mention what derivation actually hits it.
Today's `nixpkgs` example:
Before the change:
$ nix build --no-link -f. postgresql_14
...
error: derivation contains an illegal reference specifier 'man'
After the change:
$ nix build --no-link -f. postgresql_14
...
error: derivation '/nix/store/bxp6g57limvwiga61vdlyvhy7i8rp6wd-postgresql-14.15.drv' output check for 'lib' contains an illegal reference specifier 'man', expected store path or output name (one of [debug, dev, doc, lib, out])
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Maximilian Bosch <maximilian@mbosch.me>
(cherry picked from commit ec46a7e4dea8c568677d3d98588810bcd178f048)
Change-Id: I36e3e951c282123e780a920d5bef59de74de9fe0
this is misleading at best and dangerous at worst. callers expect all
exceptions thrown here to signal build startup failure, and errors in
the build itself to be signaled through the promise. since this isn't
a virtual method it will always return the same kind of promise, i.e.
we can just call handleChildOutput in the one caller of startBuilder.
Change-Id: I65d06f96ec05a3fcdf2050dbbd87c417c308ecc9
all extant derived classes implement this method, making the default
implementation not very useful. should it ever be needed again it'll
be easy enough for derived classes to return an owning StringSource.
Change-Id: I65e041e7a4e7b161f0f404f4287cea5440b5a749
references remain forbidden because std::optional does not want to
contain them, and specializing generators to use pointers where we
can't use optionals is simply too much work for a feature we don't
even need. reference wrappers and bindings still work well enough.
Change-Id: I2e6ca74719584ce16e2357c452fdd5c5a9e23d5a
This reverts commit 596b0e0a04.
cygwin dropped the patch that necessitated this hack with 3.34, which is
still the current version of sqlite in cygwin. that was in 2020, i.e. we
absolutely should not have to keep this around (especially since it will
silently corrupt the library search path of things linking to libstore!)
Change-Id: I251508b36f26c30533996e8d286aa8e5373eff31
Jade Lovelace