There's a race condition where awaitData could early-return for data
coming from a 404 response or similar and thus not rethrow the exception
that is forthcoming, and a related race during transfer setup (which
could retry a transfer *twice* per retry round).
This would then cause substitution failures like below since the exception
isn't caught in HttpBinaryCacheStore::getFile as intended, but instead
by an exception handler downstream of `drain()` which would error out
the entire operation.
Symptom:
» nix-build ./docs-service.nix -o "docs-service-result"
error: unable to download 'https://cache.nixos.org/7mr3fy8w66gi5inmf0jkkkl90lxy4jyg.narinfo': HTTP e
rror 404 ()
response body:
This is kind of a hack in how it is implemented: it assumes that you
can't intentionally be receiving a large unsuccessful response since in
such a case, `awaitData` will wait for finish() to be called to throw an
exception and will never escape until the download finishes, while
continuing to buffer the entire response into memory, which could be bad
if an error response had a large payload.
That said, nobody is sending Lix 1GiB of 404, so meh I guess, and this
is how it is seemingly intended to work. That was a design flaw of the
thing before any of the Lix team got our paws on it.
I tested this by adding _exit(0) inside the expected exception catch and
then running the offending command repeatedly to see if the symptom ever
appeared again, and it did not.
Needs cherry-pick to 2.92 and a 2.92.1 release once reviewed.
Fixes: https://git.lix.systems/lix-project/lix/issues/635
Co-Authored-By: lix@jade.fyi
Change-Id: If54f6eeaad60b5ca9d5b77d4d9232da1d295e7d1
It has been renamed to `emplaceExpr` to be more consistent with the fact
that it actually creates the expression pointer. Moreover, `pushExpr`
has been added which directly takes a `unique_ptr<Expr>`.
All manual calls to `exprs.emplace_back` have been removed and replaced
with `pushExpr`, and `exprs` is now a protected field. This allows for
enforcing invariants and modifying state within `pushExpr`.
Change-Id: I995f0bdc1c090cf78080c9f0b7737e2be359b3e4
The code for serialization Expr nodes back into (pseudo-)Nix has been
removed for being subtly error-prone and tedious to maintain. Instead,
`nix-instantiate --parse` now prints a JSON representation of the AST.
Usage patterns of the --parse flag I've found in the wild:
1. Check if a file is well-formed, i.e. discard output and test exit code
2. Get parser errors from a file, i.e. discard stdout and use stderr
3. Nixfmt uses --parse to test equivalence pre/post format, and that property is (should be?) preserved
None of these should break with the current change
Closes#487
Change-Id: Icdbaad17790f2ad8765fa08e02e6597ee4c7a909
the default implementation using getFile does exactly the same thing,
unless the thing being downloaded goes away immediately before we try
to resume a partial download. that's best treated as a different kind
of error though because the target did exist the first time we asked.
Change-Id: Ic0c046bdcd9d9751c6ba78167c68a38d0d2739e9
retries must happen outside of the state lock lifetime, otherwise
retries may keep transactions alive that should have been killed.
Change-Id: I9088975283aac9fc4521aa2f0bd000335740f6c7
it's dirt cheap to do on average, the complexity of sticking this under
a lock is not worth it. we may want to split this into not loading both
kinds of keys since private keys are rarely needed, but since only root
or the daemon are likely to have access to them it shouldn't be urgent.
Change-Id: I691aed100c9cc4ca32ab7e99a37b5be7d5c25e93
destruction of static objects is not sequenced before termination of
detached threads. this means that a detached thread can hold on to a
reference to a static object and access that object after main() has
returned and the static object destructor has run. ReceiveInterrupts
is one such case, but moving the shared state to the heap solves it.
Change-Id: Id597365ce9386000d171a03323169aafc542aa2c
this will make it easier to return async streams instead of sources at
some point in the future. the primary benefactors of the current state
are not greatly inconvenienced by the api change, and would need to be
changed much as they are now once async streams come around either way
Change-Id: I4db9ea8b186f358c239f7863ac8140c500986c2d
Original-Author: picnoir <picnoir@alternativebit.fr>
Inspired from
https://github.com/NixOS/nix/pull/11922/commits/ced8d311a593fcf9c3823e4e118474ac132d8e60
and adapted for Lix needs.
TL;DR: The topological sort should ensure that it is possible to delete
the path iterated upon. Nonetheless, in some cases,
`invalidatePathChecked` can still throw `PathInUse`, the exception
bubbles up and cancel the garbage collection procedure, leaving the rest
of the paths untouched. This change ensure that the error is logged for
further investigation but doesn't prevent the GC to continue when it
can.
After code review, we decided to make it a `printInfo` to inform the
user about sudden "in use" dependencies during garbage collection and
let them re-run garbage collection if they care about this.
References: https://github.com/NixOS/nix/issues/11923
References: https://git.lix.systems/lix-project/lix/issues/621
Change-Id: I5606c9afd16b5faa747b713fde2dc24016990ba3
Signed-off-by: Raito Bezarius <raito@lix.systems>
this is misleading at best and dangerous at worst. callers expect all
exceptions thrown here to signal build startup failure, and errors in
the build itself to be signaled through the promise. since this isn't
a virtual method it will always return the same kind of promise, i.e.
we can just call handleChildOutput in the one caller of startBuilder.
Change-Id: I65d06f96ec05a3fcdf2050dbbd87c417c308ecc9
all extant derived classes implement this method, making the default
implementation not very useful. should it ever be needed again it'll
be easy enough for derived classes to return an owning StringSource.
Change-Id: I65e041e7a4e7b161f0f404f4287cea5440b5a749
references remain forbidden because std::optional does not want to
contain them, and specializing generators to use pointers where we
can't use optionals is simply too much work for a feature we don't
even need. reference wrappers and bindings still work well enough.
Change-Id: I2e6ca74719584ce16e2357c452fdd5c5a9e23d5a
This reverts commit 596b0e0a04.
cygwin dropped the patch that necessitated this hack with 3.34, which is
still the current version of sqlite in cygwin. that was in 2020, i.e. we
absolutely should not have to keep this around (especially since it will
silently corrupt the library search path of things linking to libstore!)
Change-Id: I251508b36f26c30533996e8d286aa8e5373eff31
The fruits of a night spent figuring out the Lix release process.
Other notes:
* The release process maybe uses a horrible `builders` setting or
something to build all the architectures on the CI builders?
* Requires `nix-eval-jobs` (!!!)
Change-Id: Idf758f78326b2ea705e26c3d7f1a4638bc5980c1
This merge commit returns to the previous state prior to the release but leaves the tag in the branch history.
Release created with releng/create_release.xsh
Change-Id: I4e4650d96de82b46c35171b3a9fc1e3a6ca8e6a1
This commit remove a check for the log level in the `printVersion()`
function that was making `nix --version` behaving weirdly and
inconsistently compared to other `nix-*` commands.
The root cause is the following code in `lix/nix/main.cc` that change
the log level at runtime if nix is call interactively:
```cpp
if (isatty(STDERR_FILENO)) {
verbosity = lvlNotice;
} else {
verbosity = lvlInfo;
}
```
This should be removed, but since it has more implication it's gonna be
done in another CL.
Fix: https://git.lix.systems/lix-project/lix/issues/620
Change-Id: Id2c83c51d7ef799ee2f9b8dbdd2bfaeaf2df6188
We have obliterated the support across the build system for these
include paths anywhere, so the codemod is now redundant and can go.
Change-Id: I54082f39752c6aac6429e1c24026211adae8221a
This is a breaking change for non-migrated external clients. External
users always need to use <lix/libcmd/foo.hh> type include paths now.
This is as was always planned with the include rearrangement.
Change-Id: I269be91ff9f9cc94d5d3043cf3e0bdf8db1d8edb
This fixes a bug where flakes do not actually do purity path checks
correctly.
Tested-By: Jade Lovelace <lix@jade.fyi>
Change-Id: If7d131a8e73a5874fb15cfaa0dea3b8811ba35d2
this is more of a theoretical problem, but it does allow changing the
behavior of a flake depending on mutable machine state. it's unlikely
that this could be used to reliably do anything bad, but it does lead
to even more non-determinstic evaluation of (notionally) pure flakes.
Change-Id: I5bac7ed045046da08a36c764ab887bc9c7551542
This fixes segfaults with nix copy when there was an error processing
addMultipleToStore.
Running with ASAN/TSAN pointed at an use-after-free with threads from
the pool accessing the graph declared in processGraph after the function
was exiting and destructing the variables.
It turns out that if there is an error before pool.process() is called,
for example while we are still enqueueing tasks, then pool.process()
isn't called and threads are still left to run.
By creating the pool last we ensure that it is stopped first before
running other destructors even if an exception happens early.
fixes#618
Change-Id: I42a355f632aa0354df94c5d5d8cbe7ab5196c9a6
SourcePath only manipulates path names now. all accesses must go through
a checked path going forward to ensure we don't escape restriction lists
of pure and restricted evaluation. if a directory path is checked it can
safely be assumed that the directory itself is allowed, and its contents
will likewise be safe to access. it is tempting to assumed that contents
will also be fine, but that's only true if the content is not a symlink.
Change-Id: Icec3098d53fe9dce50997954ba958fe4f304d59b
like earlier, anything accessed during eval must be checked against the
list of path restrictions. this notably excludes `Pos::getSource` which
is run only from an unrestricted context (resolving line/column numbers
for expressions), but since positions require the parser to run and the
parser requires a checked input to produce positions this is not a leak
Change-Id: I337859e9c780590d4434885125a3ef70a11f6e93
the purpose of resolveExprPath is to produce a parser input path. parser
input paths must be validated against the path allow list so they do not
escape the restricted/pure eval sandbox. checking the input path and any
intermediate paths during resolving makes this a lot harder to do badly.
Change-Id: Ib31b5bca63fe26a5e08458a871cdc9f92f9b6a10
in pure mode it is entirely useless. in impure mode it's mostly useless
since the way in which it is used is either equivalent to not being run
at all, or is equivalent to turning the following lstat into a stat. we
add a stat method instead for all those who need final symlinks stat'd.
Change-Id: I801886d18eb34b26e62b4c05d53318c6421a69bf
