Resolves critical correctness bugs as reported in fj#883.
Following the CVE fixes.
Change-Id: Ife85fb4ca71ba920aae4b98a6dc141857e10d759
Signed-off-by: Raito Bezarius <raito@lix.systems>
We’re already allowing `/tmp` anyway, so this should be harmless,
and it fixes a regression in the default configuration caused by
moving the build directories out of `temp-dir`. (For instance, that
broke the Lix `guessOrInventPath.sockets` test.)
Note that removing `/tmp` breaks quite a few builds, so although it may
be a good idea in general it would require work on the Nixpkgs side.
Fixes: 749afbbe99
Change-Id: I6a6a69645f429bc50d4cb24283feda3d3091f534
(cherry picked from commit d1db3e5fa3)
If `settings.buildDir` cannot be written to, because we are in a chroot
store, unprivileged or anything.
We can and should always gracefully fallback to a *secure* location
inside of /tmp, i.e. `/tmp/<a directory under 0700>/<our temporary
directory for build under 0700>/...`.
This does not reintroduce CVE-2025-52991 because we are creating a
directory in-between compared to creating only ONE level of directory.
Under macOS, the first level of directory has actually mode 0755 instead
of 0700 as macOS often do not possess the right primitives to chroot
inside of these directories, leading to
https://github.com/NixOS/nix/pull/11031.
Thanks to Emily for the heads-up on this type of matter.
Fixes#876.
Change-Id: Ie521202923f763225e1901ab1b9b6c6132aaf548
Signed-off-by: Raito Bezarius <raito@lix.systems>
This is necessary to unblock releng.
This is suboptimal releng because we usually merge the branches back and
the releng script did not.
Change-Id: I12c9f6713adf344c1597e2d4adafaf2d3c62f537
Signed-off-by: Raito Bezarius <raito@lix.systems>
eagerly consider outputs as not needing deletion during output
registration rather than only doing so after registration. not
waiting for registration to succeed may keep store paths alive
in the file system if registration fails for some reason; that
seem preferrable to the possibility of having another instance
of this bug. since we only leave *good* outputs around there's
not much to worry about except maybe bit of wasted disk space.
fixes#883
Change-Id: I8c22c92e39b9e203f1061278f86cde19dc4474a4
the daemon must use real store paths, not virtual store paths. using
virtual paths may inadvertently delete paths in the system nix store
when a build was run on a redirected store as root, which isn't good
Change-Id: Id048b236bda0e0ab1f3be6ccba0ddc1de2a3e941
When a build fails, its scratch output paths are not cleaned up.
Until recently, this was deemed not a problem but as part of the effort
to harden the Nix builds and protect these paths against being part of a
staged attack (race conditions, etc.), we automatically cleanup after
failed builds.
Fixes CVE-2025-52992.
Change-Id: I58481b1cc83826298b9d80d37fecf81f117ccb09
Signed-off-by: Raito Bezarius <raito@lix.systems>
if a build directory is accessible to other users it is possible to
smuggle data in and out of build directories. usually this ins only
a build purity problem, but in combination with other issues it can
be used to break out of a build sandbox. to prevent this we default
to using a subdirectory of nixStateDir (which is more restrictive).
Fixes CVE-2025-52991.
Change-Id: Iacfc9b50534de158618c815f9fb99d7dae1be4d0
This allows using a userspace program, pasta, to handle comms between
the build sandbox, and the outside world; allowing for full isolation
including the network namespace, closing the "fixed-output derivation
talks to the host over an abstract domain socket" hole for good.
Fixes CVE-2025-46416.
Co-Authored-By: Puck Meerburg <puck@puckipedia.com>
Change-Id: Ifd499b7dbb3784600a6e842fede65fc031ff9f15
When calling `_deletePath` with a parent file descriptor, `openat` is
made effective by using relative paths to the directory file descriptor.
To avoid the problem, the signature is changed to resist misuse with an
assert in the prologue of the function.
Fixes CVE-2025-46415.
Change-Id: I6b3fc766bad2afe54dc27d47d1df3873e188de96
Signed-off-by: Raito Bezarius <raito@lix.systems>
This ensures that `passAsFile` data is created inside the expected
temporary build directory by `openat()` from the parent directory file
descriptor.
Fixes CVE-2025-52993.
Change-Id: Ie5273446c4a19403088d0389ae8e3f473af8879a
Signed-off-by: Raito Bezarius <raito@lix.systems>
`writeFile` lose its `sync` boolean flag to make things simpler.
A new `writeFileAndSync` function is created and all call sites are
converted to it.
Change-Id: Ib871a5283a9c047db1e4fe48a241506e4aab9192
Signed-off-by: Raito Bezarius <raito@lix.systems>
We use it immediately for the build temporary directory.
Change-Id: I180193c63a2b98721f5fb8e542c4e39c099bb947
Signed-off-by: Raito Bezarius <raito@lix.systems>
We now keep around a proper AutoCloseFD around the temporary directory
which we plan to use for openat operations and avoiding the build
directory being swapped out while we are doing something else.
Change-Id: I18d387b0f123ebf2d20c6405cd47ebadc5505f2a
Signed-off-by: Raito Bezarius <raito@lix.systems>
This is useful for certain error recovery paths (no pun intended) that
does not thread through the original path name.
Change-Id: I2d800740cb4f9912e64c923120d3f977c58ccb7e
Signed-off-by: Raito Bezarius <raito@lix.systems>
CHERRY-PICK: prereq to fixing the aws flags for garage shenanigans
For the user manual, we don't delete things that are missing when doing
aws s3 sync. This doesn't seem wise. If you delete a page in the manual,
it will stay public and visible.
This adds `--delete` to the `aws s3 sync` commands to fix this issue.
Closes#396
Change-Id: I6d7fb97bcdab96c0115d6c66fea0310125207df4
(cherry picked from commit 16df34b295)
Starting with commit 0dbfa7b26e access would also
be allowed to ancestors of allowed paths. This is (ironically) a significant
purity regression, since several users of the purity checks will themselves
assume that arbitrary descent is allowed. For example, `builtins.readDir` and
`builtins.path` could now refer to the filesystem root, breaking purity
entirely in the latter case by allowing to read arbitrary files. Restore the
previous behaviour of only allowing access to explicitly allowed paths.
Change-Id: Ie64180733ab735da9873255e1ccbf95ba7c9161c
(cherry picked from commit 9d99a7c2cf)
signals could cause lockFile to return without having locked the file.
the garbage collector didn't check for this, and then hilarity ensued.
Change-Id: If86d33595e8bf5510d2b032139342261dc6e07c9
(cherry picked from commit d186064c3d)
lockFile is currently interruptible by signals like SIGCHLD. which
happen a lot in daemons. now imagine that daemon enabled automatic
garbage collection. observe that the local store does not actually
check whether its lock operations have succeeded ... get the idea?
Change-Id: Ibfd7ee786c4fee3add72d4456a7e95e73e09c73e
(cherry picked from commit 79f9c39e36)
propagating the "unlock" lock type through this high-level api is
nonsense. it doesn't make sense to treat locking and unlocking as
similar operations; unlocking *must* not not interruptible by our
checkInterrupt machinery or it will just leave locks lying around
for a potentially very long time. unlock operations should not be
taking long enough to *want* them interrupted anyway. even on nfs
this makes very little sense because nfs waits *uninterruptibly*.
Change-Id: I10d605c8fe6c651bee64466eee1f8e20251d39f4
(cherry picked from commit aa87c8aa93)
a65e9e5828 did not inform `tryEval` that
(as far as it's concerned) `state.debug` moved to `state.errors.debug`
and changed types. this resulted in the REPL erroneously coming up, that
REPL having a non-debug state, and segfaulting after that REPL exited.
it's probably good that `state.debug` isn't mutated by `--ignore-try`
anymore.
Change-Id: I1918e93edacd626452aa423fc2eb825080738835
Fixes: a65e9e5828 ("libexpr: extract eval error creation into new type")
Signed-off-by: Dusk Banks <me@bb010g.com>
(cherry picked from commit 6a583136b7)
Original-Author: picnoir <picnoir@alternativebit.fr>
Inspired from
https://github.com/NixOS/nix/pull/11922/commits/ced8d311a593fcf9c3823e4e118474ac132d8e60
and adapted for Lix needs.
TL;DR: The topological sort should ensure that it is possible to delete
the path iterated upon. Nonetheless, in some cases,
`invalidatePathChecked` can still throw `PathInUse`, the exception
bubbles up and cancel the garbage collection procedure, leaving the rest
of the paths untouched. This change ensure that the error is logged for
further investigation but doesn't prevent the GC to continue when it
can.
After code review, we decided to make it a `printInfo` to inform the
user about sudden "in use" dependencies during garbage collection and
let them re-run garbage collection if they care about this.
References: https://github.com/NixOS/nix/issues/11923
References: https://git.lix.systems/lix-project/lix/issues/621
Change-Id: I5606c9afd16b5faa747b713fde2dc24016990ba3
Signed-off-by: Raito Bezarius <raito@lix.systems>
(cherry picked from commit 6a41dae49a)
There's a race condition where awaitData could early-return for data
coming from a 404 response or similar and thus not rethrow the exception
that is forthcoming, and a related race during transfer setup (which
could retry a transfer *twice* per retry round).
This would then cause substitution failures like below since the exception
isn't caught in HttpBinaryCacheStore::getFile as intended, but instead
by an exception handler downstream of `drain()` which would error out
the entire operation.
Symptom:
» nix-build ./docs-service.nix -o "docs-service-result"
error: unable to download 'https://cache.nixos.org/7mr3fy8w66gi5inmf0jkkkl90lxy4jyg.narinfo': HTTP e
rror 404 ()
response body:
This is kind of a hack in how it is implemented: it assumes that you
can't intentionally be receiving a large unsuccessful response since in
such a case, `awaitData` will wait for finish() to be called to throw an
exception and will never escape until the download finishes, while
continuing to buffer the entire response into memory, which could be bad
if an error response had a large payload.
That said, nobody is sending Lix 1GiB of 404, so meh I guess, and this
is how it is seemingly intended to work. That was a design flaw of the
thing before any of the Lix team got our paws on it.
I tested this by adding _exit(0) inside the expected exception catch and
then running the offending command repeatedly to see if the symptom ever
appeared again, and it did not.
Needs cherry-pick to 2.92 and a 2.92.1 release once reviewed.
Fixes: https://git.lix.systems/lix-project/lix/issues/635
Co-Authored-By: lix@jade.fyi
Change-Id: If54f6eeaad60b5ca9d5b77d4d9232da1d295e7d1
(cherry picked from commit de58cd6e80)
This merge commit returns to the previous state prior to the release but leaves the tag in the branch history.
Release created with releng/create_release.xsh
Change-Id: I4e4650d96de82b46c35171b3a9fc1e3a6ca8e6a1
This commit remove a check for the log level in the `printVersion()`
function that was making `nix --version` behaving weirdly and
inconsistently compared to other `nix-*` commands.
The root cause is the following code in `lix/nix/main.cc` that change
the log level at runtime if nix is call interactively:
```cpp
if (isatty(STDERR_FILENO)) {
verbosity = lvlNotice;
} else {
verbosity = lvlInfo;
}
```
This should be removed, but since it has more implication it's gonna be
done in another CL.
Fix: https://git.lix.systems/lix-project/lix/issues/620
Change-Id: Id2c83c51d7ef799ee2f9b8dbdd2bfaeaf2df6188
We have obliterated the support across the build system for these
include paths anywhere, so the codemod is now redundant and can go.
Change-Id: I54082f39752c6aac6429e1c24026211adae8221a
This is a breaking change for non-migrated external clients. External
users always need to use <lix/libcmd/foo.hh> type include paths now.
This is as was always planned with the include rearrangement.
Change-Id: I269be91ff9f9cc94d5d3043cf3e0bdf8db1d8edb
This fixes a bug where flakes do not actually do purity path checks
correctly.
Tested-By: Jade Lovelace <lix@jade.fyi>
Change-Id: If7d131a8e73a5874fb15cfaa0dea3b8811ba35d2
this is more of a theoretical problem, but it does allow changing the
behavior of a flake depending on mutable machine state. it's unlikely
that this could be used to reliably do anything bad, but it does lead
to even more non-determinstic evaluation of (notionally) pure flakes.
Change-Id: I5bac7ed045046da08a36c764ab887bc9c7551542
This fixes segfaults with nix copy when there was an error processing
addMultipleToStore.
Running with ASAN/TSAN pointed at an use-after-free with threads from
the pool accessing the graph declared in processGraph after the function
was exiting and destructing the variables.
It turns out that if there is an error before pool.process() is called,
for example while we are still enqueueing tasks, then pool.process()
isn't called and threads are still left to run.
By creating the pool last we ensure that it is stopped first before
running other destructors even if an exception happens early.
fixes#618
Change-Id: I42a355f632aa0354df94c5d5d8cbe7ab5196c9a6
SourcePath only manipulates path names now. all accesses must go through
a checked path going forward to ensure we don't escape restriction lists
of pure and restricted evaluation. if a directory path is checked it can
safely be assumed that the directory itself is allowed, and its contents
will likewise be safe to access. it is tempting to assumed that contents
will also be fine, but that's only true if the content is not a symlink.
Change-Id: Icec3098d53fe9dce50997954ba958fe4f304d59b
like earlier, anything accessed during eval must be checked against the
list of path restrictions. this notably excludes `Pos::getSource` which
is run only from an unrestricted context (resolving line/column numbers
for expressions), but since positions require the parser to run and the
parser requires a checked input to produce positions this is not a leak
Change-Id: I337859e9c780590d4434885125a3ef70a11f6e93
the purpose of resolveExprPath is to produce a parser input path. parser
input paths must be validated against the path allow list so they do not
escape the restricted/pure eval sandbox. checking the input path and any
intermediate paths during resolving makes this a lot harder to do badly.
Change-Id: Ib31b5bca63fe26a5e08458a871cdc9f92f9b6a10
in pure mode it is entirely useless. in impure mode it's mostly useless
since the way in which it is used is either equivalent to not being run
at all, or is equivalent to turning the following lstat into a stat. we
add a stat method instead for all those who need final symlinks stat'd.
Change-Id: I801886d18eb34b26e62b4c05d53318c6421a69bf
all files are physical, so this doesn't have to be optional. if it only
returns a copy of a member it's not useful either, but performance cost
Change-Id: Ib2f935ae247d96418d55bc100e04765dc586528b
after startTransfer the transfer, and thus downloadState, is gone. asan
hasn't caught this, presumably because the access is in libc somewhere.
we don't even need access to the old state; the assertion is not useful
here and clearing the previous exception is invisible to the new round.
Change-Id: I32dc1a487b96cbaabcf5061c9b1f96dbc5aaae49
Nixpkgs issues / PRs:
* https://github.com/NixOS/nixpkgs/pull/368091
* https://github.com/NixOS/nixpkgs/issues/369366
This can be triggered with the postgresql_14 derivation from nixpkgs rev
19305d94dacca226ca048b78e6de00f599c65858
(/nix/store/bxp6g57limvwiga61vdlyvhy7i8rp6wd-postgresql-14.15.drv on
x86_64-linux): for reasons unknown to me, only the `man` and `lib` outputs
are cached on cache.nixos.org:
$ nix derivation show /nix/store/bxp6g57limvwiga61vdlyvhy7i8rp6wd-postgresql-14.15.drv | jq '.[].outputs.[].path' -r | xargs nix path-info --store https://cache.nixos.org
warning: The interpretation of store paths arguments ending in `.drv` recently changed. If this command is now failing try again with '/nix/store/bxp6g57limvwiga61vdlyvhy7i8rp6wd-postgresql-14.15.drv^*'
don't know how to build these paths:
/nix/store/m9vb40xxr6gckjzpfxnqcmjqsks2gx03-postgresql-14.15
/nix/store/nm1415wa53iawar9axwxy0an6ximhayn-postgresql-14.15-dev
/nix/store/v9vrvfhiw9gk8hj9895sb15fxvxnyylj-postgresql-14.15-debug
/nix/store/zi12g1p99g2173i8093ixbqkfh9ng87b-postgresql-14.15-doc
/nix/store/3i3fpz0xss9inampf51gp3pkx24ypxpj-postgresql-14.15-man
/nix/store/db8797h2cp4rm1cnsqrf87apkkxwwdff-postgresql-14.15-lib
error: path '/nix/store/m9vb40xxr6gckjzpfxnqcmjqsks2gx03-postgresql-14.15' does not exist in the store
Also, the derivation uses the `outputChecks` feature (and thus `__structuredAttrs`)
to make sure that e.g. the `out` output doesn't reference the `man`
output:
__structuredAttrs = true;
outputs = [ "out" "dev" "doc" "lib" "man" ];
outputChecks.out.disallowedReferences = [ "dev" "doc" "man" ];
With all that in place, the following error was hit on all CppNix / Lix
versions currently supported when trying to build the derivation above:
error: derivation contains an illegal reference specifier 'man'
The following happened here:
* The `man` & `lib` outputs were substituted at some point.
* When register outputs, the reference checks are made.
* `LocalDerivationGoal::checkOutputs` gets a map of all outputs that
were built and are NOT already registered in the store. In the example
above this means `out`, `dev`, `debug` and `doc`.
* `checkOutputs` tries to resolve the `man` output and fails to do so
because it's a store-path that's already registered and thus not part
of the map passed to `checkOutputs`.
Since the map passed to `checkOutputs` is used in various other places
that appear to assume that the paths aren't registered already, I didn't
write the already registered paths into it. Instead, I created a second
map that contains all already registered outputs and pass it as third
argument to `checkOutputs`. If the other lookups fail, this map will be
now checked before the "illegal reference specifier"-error is thrown.
This fixes the problem with `postgresql_14` for me.
Also wrote a small regression test that fails locally without the patch
in place.
Change-Id: Ieacca80c001fcfbebf6f5fe97e25c49d2724c3ff
in a daemon all calls to the logger can throw an Interrupted exception,
which so far has silently stopped the curl thread without notifying its
transfers and leaving them stuck as a result. ensuring that the loggers
can never throw Interrupted will have very unpleasant side-effects, and
throwing depending on context requires large amount of bookkeeping. for
now it is easiest to abort all transfers on Interrupted during cleanup.
the test for this is extremely sketchy because we want to hit a single,
very specifically chosen, loger call in TransferItem::finish(). the bug
was triggered by the `act.progress` further down from what we're aiming
for, but that one is much harder to select for than the debug log here.
fixes#613
Change-Id: Id72efa64dd30cbbf256d2ab2a328457a0b095c6a
the flake input bump broke these due to packages being moved. also fix
renamed options while we are here, before they inevitably break later.
Change-Id: If16bb0a221e63ae7394af6b8d904c5ecd8994e6f
After gathering more community feedback, more non-trivial use cases for
overriding `__findFile` emerged. Unlike the use case of Tvix mentioned
in #599, these can't easily be worked around by overriding `nixPath`
instead.
This is the second fixup/partial revert for
81d5f0a7d9.
See also #599.
Change-Id: I7ca75e1a2b196c0da341969c61f4c168b5f657f9
Attribute names containing special characters like @ or . need to be
quoted, so we need to do our own tokenization of the command line for
completion, and quote the attribute names when we provide the completion.
Fixes: https://git.lix.systems/lix-project/lix/issues/450
Change-Id: I55a30dd272880c89445d9ded49b3f2c90cb19326
This is a useful piece of functionality to being able to eat URL
hyperlinks, for instance, which is a bug that Lix has while dealing with
terminal output today.
Change-Id: I77b2de107b2525cad7ea5dea28bfba2cc78b9e6d
This commit makes Lix include the summarized content of the value being
indexed when it is bad.
lix/lix2 » nix eval --expr '{x.y = 2;}' 'x.y.z'
error: the value being indexed in the selection path 'x.y.z' at 'x.y' should be a set but is an integer: 2
lix/lix2 » nix eval --expr '{x.y = { a = 3; };}' 'x.y.z'
error: attribute 'z' in selection path 'x.y.z' not found inside path 'x.y', whose contents are: { a = 3; }
Did you mean a?
lix/lix2 » nix eval --expr '{x.y = { a = 3; };}' 'x.y.1'
error: the expression selected by the selection path 'x.y.1' should be a list but is a set: { a = 3; }
Change-Id: I3202aba0e437e00b4c6d3ee287a2d9a7c6892dbf
I want this for being able to write reasonable expect-test style tests
for oneliners. We will still probably want something like insta for more
complicated test cases where you actually *want* the output in a
different file, but for now this will do.
cc: https://git.lix.systems/lix-project/lix/issues/595
Change-Id: I6ddc42963cc49177762cfca206fe9a9efe1ae65d
I don't know what the heck the xonsh module is doing but its obviously
crimes so it has to go. It was never intended to be running here anyway.
Fixes: https://git.lix.systems/lix-project/lix/issues/593
Change-Id: I1877698469392f85884945aaa60987c68c4e0ebc
Lix requires a non-antiquated macOS SDK, and 24.11 does not yet have a
non-antiquated one as default.
Fixes: https://git.lix.systems/lix-project/lix/issues/588
Change-Id: Iad19c06d7fefe3a736cdcb39ced185e52dcfcbb8
nixpkgs 24.11 changes how we access xonsh yet again
and updates clang.
Unfortunately, clang 18 produces significantly more
warnings on existing code that is challenging to fix.
Make sure that doesn't error when we're running
`-Werror` builds.
n.b. I had to change the "SSL certificate problem: self-signed
certificate" to the old error prior to the improved libcurl errors,
since what is presumably a difference in which TLS library is used has
cropped up between releases? Either way the curl error buffer is empty.
Seems like we aggressively cannot do anything about this.
Change-Id: If0141a46a8b445a0e7d6f86f939e8c8e03569bf5
It being overridable was an intended feature with good use cases, and
should not have been removed. However, this feature is generally in a
bad state and needs revisiting in the future.
Fixup for 81d5f0a7d9Fixes#599
Change-Id: I2d93e012caa65aa795bce3a71d8e56d7052ef9df
* changes:
libexpr: Rework error messages on ExprSelct::eval
libexpr: Track position information in attrpaths
libexpr: Assert: Don't print assertion in error message
libexpr: Split ExprAttrs and ExprLet
Calls to `show` have been removed. To counter the loss of information,
the error positions have been improved and now correctly point to the
current selector instead of the entire select expression.
Change-Id: I4771fe874af1ac15828a9863550cd4369a8f0e94
This is a pretty small change because the parser already has all
necessary information (needed for parse errors), now we want to keep it
to also provide better eval errors.
Change-Id: Ifc7a9516b9b0c8d9698f1899a6912ae91f6696ab
The `show` functionality needs to be removed because it is deeply
flawed, and given that we already print position information in the
error message (which probably wasn't always the case in the past) the
assertion printing is redundant anyways.
Change-Id: I1f5e05ab73aaa0ec92994c2211463260fd374898
ExprLet was previously inheriting from ExprAttrs for the data, while
ignoring all
set-specific operations on it. The set specific code has now been split
off so that
let doesn't inherit it anymore:
- ExprAttrs (not an Expr), containing the attributes and the related
logic
- ExprLet : Expr, ExprAttrs
- ExprSet : Expr, ExprAttrs
Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: I63f2fbcd1e790b3cffb56eec1e7565ee3cdbf964
Mostly these are bugprone-unused-local-non-trivial-variable.
Also fix instances of:
- bugprone-optional-value-conversion
- bugprone-inc-dec-in-conditions (please check this loop is correct, it
is the only non trivial code change in here)
- bugprone-unused-return-value (well, by fixing the lint config)
There are three notable changes relating to undefined vars:
- openLogFile ignoring the result. This is because openLogFile does a
whole bunch of mutation of member variables
- hiliteMatches: i am guessing this is because showing the derivation
name was unhelpful and it just got changed
- canonPath in NarAccessor: canonPath inside of a thing that is supposed
to be vfs based cannot possibly be correct, so let's delete it given
it is unused.
Fixes: https://git.lix.systems/lix-project/lix/issues/584
Change-Id: I887adc9ff28b61f726dcfed197e6796b414c2fcf
these must be tampered with before the evaluator is created, *never*
after. doing it any other way leads to interesting things like #596.
fixes#596
Change-Id: Iea253ccce44b94b1243833837a3df93c795967d9
Calling a custom type size_t is incredibly sneaky and makes all the code
around this extremely context dependent.
Change-Id: Idae684781f45fe615020d8642f12a656ab2c15ad
this finally gives us a witness type we can use to prove that a certain
call graph subtree can't be used in kj promises using only a single new
assumption: if EvalState& is never held as a reference member of a type
and instead only ever passes as an argument or held on the stack we can
be certain that anything that has access to en EvalState ref must never
be run inside a promise and, crucially, that anything that doesn't have
access to an EvalState& *can* be run inside a promise without problems.
Change-Id: I6c15ada479175ad7e6cd3e4a729a5586b3ba30d6
"persistent" here means state that outlives any particular evaluation,
i.e. everything that exist outside of a call to eval() and friends. do
note that this leave EvalState nearly empty, containing only things of
importance to evaluation *itself*, e.g. the call depth tracker. do not
worry about the self-reference ctx member. nothing to see here, go on.
Change-Id: Ie8d0af3f09d33902f81e0c36a1096dd9f334a537
for the same reason the eval caches should not keep them as references,
only worse: installables cannot fully lose their reference member since
some types of installable (flakes -sigh-) need access to caches held by
their member reference. passing that around as well at all times is not
feasible; passing in EvalState& all the time is quite bad enough as is.
Change-Id: I05d438623b5ef10fc22e7e256b89447e61a0657d
this is not necessary in any way, but it will make the following changes
smaller and easier to review. the aliases could also be added piecemeal,
but doing it here lets us lean heavily on our compilers for correctness.
(teacher notes: here the author foreshadows the shape of things to come.
not all names change, and only the names unchanged are those which will,
over time, become ever more unrecognizable. note especially nix/main.cc,
where `state` is not only cloned, but itself changes pointerness. it can
be seen as a nod to the trans community, but more realistically it is no
more than foreshadowing the future where `state` is only seen by proxy.)
Change-Id: I7732025e58df089b7f8e564fc63960cd91729d09
it's not used for anything any more. it was only passed on to the AttrDb
constructor, which itself only stashed it away to an unused member field
Change-Id: I334fde751a3754e6580f573c21ae3d04be00345a
and for the same reason DrvInfo shouldn't, only even more so. the eval
cache infrastructure wants to be a wrapper around evaluation, with its
lifetimes fully decoupled from the lifetime of the wrapped evaluation.
this can't not create problems when eval lifetime must become bounded.
Change-Id: Ib8eec649995b4decd7290c2266322f67d73b6b46
in the future eval states will be asyncio root contexts. proving that
they are used correctly is impossible if the are referenced by things
with weakly scoped lifetimes, or entirely unscoped lifetimes, as we'd
have to deal with in the case of `DrvInfos` passed as out parameters.
Change-Id: I9ca5c46a714c6c914a77c7c7500cb4939ac9aff5
it's all strings anyway. the db stores strings, the cli wants to
interact with attr paths as strings, so we will just use strings
Change-Id: Id9ea07d92343de77e8d47af8fec1e86ae225e9a1
there's no reason for these to be private. much more important things
that are now part of EvalState are not private, and having statistics
be private forces us to add otherwise unnecessary friend declarations
Change-Id: Ib37097d94a9f55c2b21969fb6c51049b1c914515
thunk creation doesn't have to be called that. thunks are more of an
implementation detail, calling it evalLazily seems a lot more clear.
Change-Id: I64bdac422eadb629002195cd6b6a89dce1b4a60f
this will let us pass the capability to create debuggable eval errors
without having to pass an entire EvalState. we could pass symbols and
debug states around just as easily, but if we add new capabilities to
our debugger we might have to change many more places than with this.
Change-Id: I2f8893012e5d98a986ef1fc888234c2dd8d5e096
if we hold the error being built behind a pointer we no longer need to
allocate the error builder itself to avoid impacting eval performance.
adding && ref qualifiers to builder functions and adding the nodiscard
attribute to the class itself also makes the nodiscard attributes kept
on the builder functions unnecessary. we do keep the noinlines though,
removing them still regresses eval performance even after this change.
Change-Id: Ibc14a66955ac32142d97fb3680b0a7e14db250cd
this is necessary to share a debug state between multiple eval states.
while doings so makes little sense at present it will be necessary for
async io support since eval states must be async io roots, which means
we must create them as needed from a shared evaluation context object.
Change-Id: Id9d4b37aae40706f65c741e3b961855582e035ab
if we define a TraceableAllocator at all times and use that in places
that want maybe-traceable allocation we can simplify things a lot. we
also unconditionally allocate cache root pointers for Value and Env-1
caches, even though we don't need them without gc (they're so cheap).
defaulting to `std::allocator` without gc recovers previous behavior.
Change-Id: I236da8c3b0669b40cdfe355ec3ec4e764d096074
that way we don't need to eval while initializing the evaluator. this
will make a lot more sense once we start pushing asyncio through here
Change-Id: I3663052438ed97d48e213b71395ad1dd5e1318bb
currently only used to hide the nix code of derivation from stack traces
and debug frames, but perhaps we'll find it useful for other things too.
Change-Id: Ie5667873d8858d25dd4113bdf454e800b59082d7
this doesn't belong here, getFlake should be handling this instead. we
do lose the debug frame for loading a root flake, but that seems fine.
there's no point in starting a debugger if the root isn't even a flake
Change-Id: I24ad32b6716baee81a1a0f8bf9ce26814d97c7aa
this was only used to enable an optimization we no longer need such
strong guarantees for: SmallVector can be used with a dynamic size.
Change-Id: I4513fb7e665827fe363ce6308448656e6c5badb7
Error::addTrace exists and is used far more often already. let's
standardize on the variant that doesn't need yet more templates.
Change-Id: If66b69ca02dbb546ce98cf385181bd13ce7ad9b5
the callback is always called immediately, which defeats the purpose of
the callback and the purpose of the template itself. there seems to not
be any performance impact of this. optimizing Value::determinePos would
be nice, but it's not used nearly often enough to matter at this point.
Change-Id: I2aec6a38103630652112f4b273653f11d2404c04
nothing needs these any more. the CLI calls evalFile, but only in places
where it'll only be called *once* per process lifetime. __import does so
too, but import doesn't care about the parse tree, only the eval result.
interestingly this improves eval performance by 6% on system eval unless
GC never occurs. no idea why this makes a GCing eval faster, but it does
Change-Id: I8289528550244e0a8b5ebc7284d8fb9aaac59e20
only generate-manpage.nix uses it any more, so we can inline it there
instead of keeping it around as a separate generated header. doing so
will also allow us to remove caching functions needed *only for this*
Change-Id: I97ee91f1dd7140ecb69dbafd8479b82fba7981b8
eval caches are not used by actual eval at all, only by the flake-shaped
wrappers around evaluation. moving caches into a subclass both clarifies
that eval caches and eval states are coupled and separates concerns that
should not have been intermixed as they were here. in the future we will
want to split up and decouple things even further. that'll have to wait.
Change-Id: I7b69510c0f8b212f05fae62e7b992d9475b4841f
installables already have a ref<EvalState>. why are we passing the same
eval state in again, by reference, everywhere? that's just unnecessary.
Change-Id: I8225ea2575146edc55d283c0b5173b804553ceec
It was never intended to be a feature to be used, and moreover it is
inconsistent: One cannot override `+`, and overriding `__lessThan` won't
affect the builtins which do comparisons.
Change-Id: Iaba54a05aa4c2eb37cdb3dc0d731fcee5a86deba
this belongs to lazy trees, which we neither have nor intend to have.
we will keep SourcePath as that may come in handy at some later date.
Change-Id: I44b8f1dd6c435d7486c393fabdcd272766b2b56b
this also fixes a debugger bug where leaving the debugger does not clean
up old debugger state completely. in such cases the fake frame withFrame
created was left behind after the corresponding caller frame was unwound
Change-Id: I45adcd116276b03b2f87076518c9eae6fe844e06
checking whether a repl hook is set does not tell whether it's running,
which in turn means we will enter a debug repl even when not debugging.
this is not very useful when we have no debug frames to inspect at all,
as commonly happens when starting a new repl with --debugger specified.
Change-Id: I7065dc4ec29743bdd53ed99c29d6592e2ceea89c
just to keep debug-related state closer together. eventually this will
also allow us to not allocate debugger state altogether unless needed.
Change-Id: Id83fea75d96d0ecbe21683cb1b57dd5b11b13535
not all of these are even used by eval itself. notably sWith wasn't used
at all, sEpsilon was only used by the eval cache one layer up, and other
attributes are used in places even further away from eval itself. we can
keep this commingling for now, but eventually we should clean it up too.
Change-Id: I5684ac614361bf008e04472130c6c02082b4c2d7
these do not rely on the GC being initialized. there's no reason for
them to not be statics, shared between all eval states in a process.
Change-Id: Ib9675f3945d3a0a7097d6c85096adcbd6f441d83
currently ref<>s are used to share eval states between whatever created
that state and the various repl instances. repls however do not own any
eval states, not even partially. they only provide interactive use of a
previously allocated state that is owned by something else, notably for
debug repls which are run from *within* an evaluation which may already
be wrapped by another repl instance. presumably this was originally set
up like this to guarantee memory safety, even though it's never needed.
Change-Id: I29a2cd1d4245c077f9270452a0873d4c47448729
there's no reason to ever *own* a repl instance. everything either wants
to run a repl and receive the result of that run, or run a repl and then
totally ignore the result. allowing non-transient repl instances doesn't
do anything useful for us, unless making ownership analysis hard counts.
this also fixes a bug in which a repl could ignore the store it was told
to use during construction, using the one returned by openStore instead.
this never showed up in lix because openStore() was always passed anyway
Change-Id: If2b1ad24ab74377340199b36af1dd629d7ce3f25
give legacy commands their program name and c++-converted argv directly
instead of passing on the (argc, argv) pair untouched. all are required
to process these things, and all of them do it in exactly the same way.
we can also remove a few old helpers only used to make this less awful.
Change-Id: I4ecd02343bca0cf85faf6fe043031d4f64c5f29c
it's a trap. it's always a trap. non-default ctors of virtual base
classes must be called with the same parameters every time or very
surprising things will happen. good luck if you virtually derive a
class more than once with different ctor arguments. that'll break.
Change-Id: I1345963e69e98f37c3cbd070bae164d89eade455
this does nothing on classes that are never used as base classes and
have only a single base class of their own. all of these classes are
also made final to prove that they are indeed in this category. once
we need to derive any of them we can erase their `final` specifiers.
Change-Id: I4bd8e50a58815227392b5a5a762fda4542d15bae
there are no uses of plain MultiCommand as a base class *except* in
NixArgs, which is the only one that does not implement run(). there
is not much of a reason not to implement a run member there though,
so let's just do that and get rid of this weird intermediate class.
Change-Id: Ie84e3acd071b43bc186a2bac87646cbfb3aff845
the very slight speedup in config setting access is not worth the
maintenance overhead of conflating concers like this. the virtual
inheritance scheme used for configs requires too much duplication
of base class constructor arguments to be worth doing. perhaps we
should get rid of all virtual inheritance of data-membered bases?
Change-Id: I4acf5ceaedb4ed7476efe1114c2e065ec72d2c6d
clangd broke because it can't look through symlinks. compile_commands
manipulation does not fix it, clangd configuration does not fix it, a
vfs overlay does not fix it, and while a combination of those can fix
it with a bind mount in place that's just too cursed to even consider
clangd bug: https://github.com/llvm/llvm-project/issues/116877
Change-Id: I8e3e8489548eb3a7aa65ac9d12a5ec8abf814aec
Ever read gdb output and you just kinda get a headache because you have
to infer what a thread is by reading the stack trace? It's not hard, but
we could also just never have to do that again, which is also not hard.
Sample:
(gdb) info thr
Id Target Id Frame
* 1 LWP 3719283 "nix-daemon" 0x00007e558587da0f in accept ()
from target:/nix/store/c10zhkbp6jmyh0xc5kd123ga8yy2p4hk-glibc-2.39-52/lib/libc.so.6
2 LWP 3719284 "signal handler" 0x00007e55857b2bea in sigtimedwait ()
from target:/nix/store/c10zhkbp6jmyh0xc5kd123ga8yy2p4hk-glibc-2.39-52/lib/libc.so.6
The API design for this is forced by the macOS pthread_setname_np only
being able to change the current thread's name, but if we just conform
everything to that, it works everywhere.
Change-Id: I2b1d6ed41e3c94170cb0b4e73ad66f239ebd9c88
give the transfer progress callback a chance to register all transfers
as aborted when an interrupt event occurs. checkInterrupt() will throw
an Interrupted exception if an interrupt signal was caught, which will
immediately break out of the curl loop. it isn't even necessary within
the curl main loop as curl guarantees to call progress callbacks about
once per second (or more often), and we already abort transfers from a
progress update if a signal was caught. this may delay shutdown a bit,
but "about one second" as should not be noticeable in most situations.
fixes#577
Change-Id: Ida4e72732562bdf6560e4f182ecdcdb663c6dda5
This allows <lix/*> includes to work from external projects not taking
Lix as a buildInput with Nix, but just using pkg-config.
Change-Id: I5b24fd0981344b013beaf406c16988dad17f0edb
These are still supported externally with pkg-config, but this commit
ensures they no longer build internally.
Change-Id: I2fa38f85e43b18f1f1dbcfdb0367de8f192aca2c
Context: we have include paths that are "types.hh" and similarly common
names. We currently have these compatibly available as
"lix/libutil/types.hh" externally but *not yet internally*. This is
because we don't have any way for the src directory to appear as
`"lix/"` from inside of Lix: the lix/ include directory is created by
the install process.
The goal of this whole thing is to make it clearer which component of
Lix that files are a part of, which should hopefully help at least a
little bit to new developers. One disadvantage of un-mixing these is
that it will cause some API changes if we ever move a file between
libraries, but that is not very common, and we don't care that much
about external API users.
This was planned for a while and is why we have a FixIncludes check to
begin with.
Personally I don't see a great benefit in rearranging our source code,
and in fact, it would probably be counterproductive:
- Moving the includes into a separate `include/` directory would just
make developers have to deal with more directories, when we can
already generate the desired layout through the build process.
- This would also decouple the .cc and .hh files which currently
conventionally have each others' definitions and declarations
respectively, right next to each other, making it easier for them to
feel decoupled and diverge.
Content: Add ../include as an include directory so that lix/ in include
paths will resolve to src/ within Lix itself, just as it does externally
today. This prepares for a further series of commits applying the actual
change to each library one-by-one by accepting both include versions at
once.
This could have been done with ../ and a symlink called lix, but we
would like to not accept libexpr/foo.hh internally for it would be
broken externally, so we need an otherwise empty directory for the
include.
Change-Id: Ideac17faadae2bcea2dffbab34eb27c582ede399
transferComplete has outlived its usefulness, and the other two phases
can be a boolean flag telling us whether we're still receiving headers
Change-Id: Ia943f95dcf0ce8ee2cc6d0f44ddf13151aef4346
things can be much simpler if the read loop knows whether the transfer
is done or not. this information is available, let's pass it back down
Change-Id: Idb99afeb06a0d7a0999c3f4a1c6ce5adeab5f054
file:// urls are no longer handled by curl itself, and this is supposed
to test the curl wrapper. use http to force the wrapper to be involved.
Change-Id: Ib13087db07b3b2f1ae44ce8e3ec7a96d935b1bab
they don't need these any more if we consider unpausing and cancelling
of transfers a cooperative affair between transfers and the curl_multi
handler. we can also remove an enable_shared_from_this base class now.
Change-Id: Iefa380df60c0cada53719cdaa500bab953892319
wrap all curl pointers in unique pointers for move correctness. the
destructor of TransferItem can go away completely since it does not
do anything special any more: memory bookkeeping is done using raii
wrappers, and its effects on the transfer state are not observable.
only TransferSource inspects the result, and even it does not check
for transfer results in its own destructor (only during transfers).
Change-Id: Ibb58484703854d5d5425e7c63cf2985d93920e97
keep the download buffer in TransferItem. keep only *one* download
buffer (not one for success returns and one for failures). we also
remove the asynchronous callbacks since they are no longer needed.
Change-Id: I9b1c1e4c59d5333602aea1c36e477c481eaaf98e
when no custom READFUNCTION is set curl will fall back to using its
READDATA as a FILE pointer and calling fread on it to retrieve data
to send to the peer. we can wrap our upload data in a FILE pointer.
doing so also allows for upload retries, but we don't do that here.
Change-Id: I0509f936229dd978e89135862dacc8a3937cd4de
Due to the cast to (unsigned int), the encoding appended broken
bytes padding. This is fixed here with a bitmask.
Fixes: https://git.lix.systems/lix-project/lix/issues/562
Change-Id: I0c93bd2b8c2f82df208d4693b7254544e3121dc3
The progress bar is not supposed to show very short activities (<10ms) in order
to prevent excessive flickering. This behaviour was inadvertedly disabled in
commit da4e46dd1f due to not actually skipping
the short activity after calculating that it should have been. Fix this
oversight.
Fixes: https://git.lix.systems/lix-project/lix/issues/561
Change-Id: Ibee256a7018cdc431662cd8ea9f6c14ef1706972
* changes:
libexpr: generate builtins from data
treewide: generate global settings from data
libutil: generate experimental and deprecated features from data
this function is called only once per object lifetime, and it's no
longer written such that it must be called by the curl thread. all
curl options set here are independent of the multi handle, there's
no good reason any more to split the constructor in two like that.
Change-Id: Ibfa9075bc1cfdbab979f995c5a942b47add9789d
this simplifies the immediate curl wrappers significantly and clarifies
control flow for retries. we can almost see the promise land from here!
Change-Id: Idc66b744631eec9c1ad5a2be388beb942a04f1f9
future changes will need to add template functions to this class, which
cannot be done for classes declared locally in unctions. otherwise this
is mostly code motion to clean up enqueueFileTransfer a little further.
Change-Id: If9a9d9eb47ceadfa75a4eebd54e2db39f2305643
it's used only once, and the transfer is used for something else
afterwards as well. returning it hides that the same transfer is
being returned unchanged, and pointlessly extends the signature.
Change-Id: Idbd0af00f9aebd3606f6b537b9b7cc9263a45cf7
we only wait for the worker thread to remove the handle from its multi
and then signal us back. after that's done the transfer can be cleaned
up independently of the multi handle and no transfer progress is made.
we also no longer remove transfer items from multi handles in the item
destructor; only the worker thread can add a transfer and as such only
the worker thread should be responsible for removing them again later.
Change-Id: I6ff57bb5e7c1e91faf8657b257e73d6a72aa928e
this only ever worked for empty uploads, and there it worked only by
complete accident: curl was asked to send more data than the wrapper
would provide, which curl would not like and report as an error. the
error would cause a retry with even less data to send, until finally
failing by running into the retry limit. let's just forbid all this.
Change-Id: I229a94b3b8b33e2c6cdb8ea19edd57cd6740e6c6
we'll need this shared code for kjified transfers that don't use
sources. it's a while out, but we can clean this up now already.
Change-Id: Ife8c160e6ab379761362d6c54aba05093deee99e
since 4ae6fb5a8f dropping a source of a
download might not properly cancel the associated curl transfer after
the transfer was paused. we have also not unpaused the transfer often
enough, only if the transfer buffer had been drained in its entirety.
Change-Id: Ic9298d9df71daa0f3d1c3fd718ed441edae9e863
Most builtins are now generated from data too, with two exceptions:
* Undocumented builtins, since supporting them would add complexity to the
generator, the harms of the current implementation mostly don't apply, and
the proper fix is to document them.
* `derivation` is somewhat magic (it is a function, but defined in the code as
a constant), so the current treatment of having it separately documented is
kept (for now, at least).
Since it is slightly easier to do and probably a good idea anyway, the builtin
function registrations generated this way are now processed directly in code
and don't go through global variables any more.
Unfortunately, a slight breaking change is introduced because the order of the
builtins' names in the symbol table changes. Hopefully, this will turn out to
not matter in practice.
Change-Id: I7b4379a93ae380b6524e41a916a21c5c6f70555e
Currently, a bunch of documentation is generated by embedding parts of it in
the nix executable, getting it out again by running it, and then postprocessing
the output. This is bad, since it creates a pointless dependency of the
documentation on the executable, and also makes documentation generation
impossible when cross-compiling.
Instead, both the code and the documentation should be generated from data, see
https://git.lix.systems/lix-project/lix/issues/292 . Here we start applying
this approach to the experimental and deprecated features, which are done in
one go since the technical implementation is very similar.
Of course, the actual benefits are not realised yet, since the offending
pattern is used in several more places. These will be fixed later.
Change-Id: I4c802052cc7e865c61119a34b8f1063c4decc9cb
Commit 4dbbd721eb intended to mark all settings
as overridden when they are. Unfortunately, due to an oversight, this marking
was accidentally performed in the implementation details of non-appendable
options. Move it to the common codepath so that it works for appendable options
too.
Fixes: https://git.lix.systems/lix-project/lix/issues/573
Change-Id: Idc3402bac48b19d832acd9b553e16e5791470c26
So I recently saw it the first time in the wild, I liked that you get
interactively asked about the nix.conf settings from the flake, but
there were a few minor things that I'd like to see changed:
* The `(y/N)` was somewhere in the middle of the line. Moved it to
the end. At first I assumed it was a bug because another thread into
my terminal while I was answering the question.
* I had to say no four times for a single flake with two options. So if
you already know you don't want any of the config for _this_ flake, I
found a `No to all` switch that ignores the rest of the nix.conf
settings a little more ergonomic than having to stop the invocation,
looking up the exact wording of `--no-accept-flake-config` and
restarting it. Hence, I added it.
* Added a note where the choices which settings to trust are persisted.
My initial assumption was that this went into `nix.conf` which is not
writable on NixOS, so I said no there as well.
Change-Id: I0a0d9c403f0662df4707697a77f08e6cd003ec6f
This adds a new temp-dir setting for controlling the temporary directory
without having to change the TMPDIR env var. This can be used to e.g.
use a path on a case-sensitive store on macOS for temporary files
without changing the TMPDIR var used by interactive shells or commands
invoked with `nix run`.
This also stops unsetting `TMPDIR` on darwin when the env var value
starts with `/var/folders/`, preferring instead to just do the check
when reading `TMPDIR`. This way the inherited `TMPDIR` env var is
preserved for child processes (such as interactive shells).
As a side effect this changes the behavior of `nix-build -o ''` to act
like `nix-build --no-out-link` instead of failing with an error caused
by trying to create a symlink at the cwd.
Fixes: https://git.lix.systems/lix-project/lix/issues/253
Fixes: https://git.lix.systems/lix-project/lix/issues/112
Change-Id: I9ee826323f2deca62854715a77ca7a373a948a29
libcurl may call callbacks during unpause. if libcurl calls these
callbacks often enough they may find that they've exhausted their
allotted buffer space, at which point they will call dataCallback
as provided in enqueueFileTransfer. download() provides callbacks
that have their own state locks and may call unpause on transfers
with *their* state locks they share with curlFileTransfer. it was
possible to cause a deadlock between these two if the curl worker
thread tried to provide some data to a callback with the transfer
state lock held and the user transfer simultaneously unpaused its
associated curl transfer, with its own state lock held. obviously
not a great situation, but avoidable by not holding any lock when
we unpause transfers and execute their callbacks, as is otherwise
the case when a transfer runs normally and is never paused at all
Change-Id: I58556d292adaf7dfb14001d3a6c5c38fa71994da
FreeBSD was left out of a few refactors over the last few months.
Add a header and register the store implementation so it's back
to working as well as it was before.
Change-Id: I6f7b2ceb557c290f2d9e0d7f207b3fea87b353ed
The devShell relied on several packages directly from `pkgs`
or used with a non-splice-aware functions.
These would be built for the host system, making them useless
in a devShell for the build system.
Make sure that all packages are for the build system when needed.
Some other minor changes also required:
* Make devShells use `clangStdenv` because GCC is currently broken
* Disable rr when making a cross stdenv
Change-Id: Iee5f8a1a0c594139a50f2261b203491bd1644866
The lix package currently fails unless it's using a clang stdenv.
However, the flake's cross build outputs (e.g. `packages.x86_64-linux.nix-armv7l-linux`)
used the default stdenv, normally gcc.
Replace this with clang to fix package build.
Also take this opportunity to remove the no longer necessary `useLLVM = true`
override on FreeBSD. Since 24.05, nixpkgs always sets `useLLVM = true`
on FreeBSD in `lib.systems.elaborate`.
Change-Id: I939302e4f6385291fa9e582d38d908c42f6db89a
This reverts commit 02c35ea9df.
Reason for revert: this code path is also used for `Input::getRev()`, i.e. flakes VCS revision validation, which, in the case of Git, are using SHA1.
As a result, this cause too much noise due to SHA1 revisions in Flakes.
Change-Id: I8064c1ebc26e4e83b627f0803a7a9ba56cfe1f37
random() is not thread-safe, it relies on global state, and calling it
from worker threads can result in multiple threads producing the same
value. It also doesn't guarantee unique values even in single-threaded
use.
Use an atomic counter for the use-case of generating temporary paths,
and switch to a thread-local RNG for the one remaining call.
This will probably fix https://github.com/NixOS/nix/issues/7273 though
I'm not willing to risk corrupting my store to find out.
Change-Id: I4c4c4c9796613573ffefd29cc8efe3d07839facc
Also tweak `pathAccessible` to ignore other relevant errors too. It was
documented as ignoring permission errors but it was only ignoring
`EPERM`, which comes from the darwin sandbox, and not ignoring `EACCESS`
which is the real permission error. I figured it also makes sense to
ignore `ELOOP`.
Fixes: https://git.lix.systems/lix-project/lix/issues/560
Change-Id: Ibb849b68d07386eb80afb52b57f7d12b3a48a202
curl handles timeouts internally when it acts as the main event loop. we
only need to wake up to add new transfers to the multi handle or restart
an existing transfer that has passed its restart wait time. periodically
waking up is not required for curl, and we don't need it any more either
Change-Id: Ic774e1d9519f807cda1a89694bc3ede75216f329
don't pause the entire curl thread. we have multiple consumer threads
after all, not just one, so stalling all of them is likely not great.
note that libcurl advises against using transfer pauses if compressed
encodings are allowed and automatically decoded. this should not lead
to problems in practice because our data is usually not compressed to
such a degree that curl buffering *uncompressed* data matters. should
this cause problems we can reintroduce the whole-thread pause, but we
will probably get away with this until the entire file transfer class
is made kj::Promise-using async (and *then* curl can be hardpaused if
it cannot get rid of its data, solving the problem once and for all).
Change-Id: I218e41bfa5a27c7454eafb0bdb54f2a29a7f6493
Sets the `X-GitHub-Api-Version` header to `2022-11-28` for calls to the
GitHub API.
This follows the later version as per
https://docs.github.com/en/rest/about-the-rest-api/api-versions?apiVersion=2022-11-28.
This affected the check on whether to use the API versus unauthenticated
calls as well, given the headers would no longer be empty if the
authentication token were missing.
The workaround used here is to use a check similar to an existing
check for the token.
In the current implementation, headers are (still) similarly sent to
non-authenticated as well as GitHub on-prem calls.
For what it's worth, manual curl calls with such a header seemed to
break nor unauthenticated calls nor ones to the github.com API.
Change-Id: I6e10839e6b99cb65eb451e923b2a64f5d3c0f578
using UTF-8 bullets in the sample avoids locale confusion where Bash
doesn't know to treat `•` as a single character.
Signed-off-by: Dusk Banks <me@bb010g.com>
Change-Id: I829019b66e93e6d33ac3a6641df07d0dd2332a5a
It's only used in a couple of tests, and only in such a way that
replacing it with a random command suffices.
I also removed a few pointless uses of the variable.
Fixes: https://git.lix.systems/lix-project/lix/issues/376
Change-Id: I90aedb61d64b02f7c9b007e72f9d614cc1b37a2e
Before:
error: derivation '/nix/store/4spy3nz1661zm15gkybsy1h5f36aliwx-python3.11-test-1.0.0.drv' may not be deterministic: output '/nix/store/ccqcp01zg18wp9iadzmzimqzdi3ll08d-python3.11-test-1.0.0-dist' differs from '/nix/store/ccqcp01zg18wp9iadzmzimqzdi3ll08d-python3.11-test-1.0.0-dist.check'
After:
error: derivation '4spy3nz1661zm15gkybsy1h5f36aliwx-python3.11-test-1.0.0.drv' may not be deterministic: outputs differ
output differs: output '/nix/store/ccqcp01zg18wp9iadzmzimqzdi3ll08d-python3.11-test-1.0.0-dist' differs from '/nix/store/ccqcp01zg18wp9iadzmzimqzdi3ll08d-python3.11-test-1.0.0-dist.check'
output differs: output '/nix/store/yl59v08356i841c560alb0zmk7q16klb-python3.11-test-1.0.0' differs from '/nix/store/yl59v08356i841c560alb0zmk7q16klb-python3.11-test-1.0.0.check'
Change-Id: Ib2871fa602bf1fa9c00e2565b3a2e1b26f908152
curl can't pause downloads of file:// urls, which is very much in the
way of making the curl wrapper fully asynchronous. we can emulate all
the things curl does, but unfortunately curl is *rather extensive* in
its support of frankly weird shit. hopefully this subset of features,
which notably does not include curl readdir support, is enough for us
Change-Id: I5f67768c4b512565655b94b0421270c7dbbd8d11
with the api cleaned up we can suddenly reunify uploads, downloads, and
existence checks through curl in the same wrapper function. uploads and
existence checks simply don't use the result source, and given that all
transfers (or at least *most* transfers to date) go through the network
the few extra allocations do not hurt us at all. even for file:// calls
the overhead won't much matter as going to disk and back *is* expensive
Change-Id: I4f9ca6681a8fc303377b4cf4c63e3363ae32c18b
it's no longer needed. `download` can do everything `enqueueDownload`
did, and a lot more. e.g. not block the calling thread, for instance.
Change-Id: I4b36235ed707c92d117b4c33efa3db50d26f9a84
this will let us return metadata from FileTransfer::download, which in
turn is necessary to remove enqueueDownload. it also opens avenues for
streaming downloads that keep download metadata instead of dropping it
Change-Id: If0fc6af5eb2aeb689fc866c345c9d7bce4d59f2d
let's use the automatic decoding functions curl provides instead of
implementing them ourselves for the dubious ability to support both
xz and bzip2 encodings as well, neither of which anything will send
Change-Id: I3edfebeb596a0e9d5c986efca9270501c996f2dd
We were calling `lstat()` twice on the link path, and a third time if it
existed. We only need to call it once.
Change-Id: Ifadc2f24681648d0ec7e4bad5c6d2dcb2e560e7b
etag changing implies with high probability that the content of the
resource changed. immutable url changing implies that the immutable
url we got previously was wrong, which is probably a server bug. if
the encoding changes our decoding will break completely, so that is
also very illegal. one notable change we still allow is etags going
away completely, mostly since this does not imply any data changes.
Change-Id: I0220ceddc3fd732cd1b3bb39b40021cc631baadc
do not retread the entire redirection path if we've seen the end of the
road. this avoids silently downloading wrong data, and notifies us when
a url we've received data from turns into a redirect when retrying. for
reasons of simplicity we don't turn of libcurl redirects on retries. if
we did that we'd have to conditionally process http status codes, which
sounds annoying and would make the header callback even more of a mess.
Change-Id: Ide0c0512ef9b2579350101246d654a2375541a39
this will let us return metadata for a transfer without having to wait
for the entire transfer to complete. more importantly for current uses
though is that we could now send retries to the effective url directly
instead of retreading the entire redirect path. this improves latency,
and in such cases where redirects change while we're downloading it'll
also improve correctness (previously we'd silently download bad data).
Change-Id: I6fcd920eb96fbdb2e960b73773c0b854e0300e99
it's always legal to call curl_multi_remove_handle on a valid pair of
multi and easy handles. removing an easy handle that is not currently
attached to a multi handle is a no-op, and removing an easy handle of
a different multi handle is something we can't reasonably trigger. if
we *did* ever manage it would result in an error we'd ignore, and the
handles in question would not be changed at all. this is just simpler
Change-Id: I85ec62ff89385981ca49d243376b9c32586bd128
return it as a separate item in a pair instead. this will let us remove
enqueueDownload() in favor of returning metadata from download() itself
Change-Id: I74fad2ca15f920da1eefabc950c2baa2c360f2ba
it's just a uri and some headers now. those can be function arguments
with no loss of clarity. *actual* additional arguments, for example a
TLS context with additional certificates, could be added on a new and
improved FileTransfer class that carries not just a backend reference
but some real, visible context for its transfers. curl not being very
multi-threading-friendly when using multi handles will make sharing a
bit hard anyway once we drop the single global download worker thread
Change-Id: Id2112c95cbd118c6d920488f38d272d7da926460
we don't even need this outside of tests. maybe we should not do
automatic retries at this level at all and use retrying wrappers
instead? at some point we may have to do this, but not just yet.
Change-Id: If0088aa55215be81f1770c25b3bb1b5268c65cf8
never set explicitly, and transfers are never instantiated with one
current activity but submitted with a *different* current activity.
Change-Id: I1a3ec57c02013565aeb9e9398ea42d0c4279095e
it's only used by HttpBinaryCacheStore, and even there used in only on
place. this one place can set the header explicitly, which it now does
Change-Id: Id89228150669e25e7f59a3d6bd939e46059ce29e
it only sets the one field anyway (and the parent activity as a side
effect that does not depend on the exact location of the constructor
call). when FileTransferRequest goes away we would need this anyway.
Change-Id: I35cf2ed3533239181449a62cf34cd282b395e5db
this is fragile, slow as fuck, breaks constantly under high concurrency,
and completely unnecessary since ssh bypasses the stdio file descriptors
*anyway*. we do still check that we see ssh messages to ensure that none
of our subprocess handling messes with ssh's /dev/tty, but that's it now
Change-Id: Ib8e31e1999f813d07a27efc63a9d3454a9e4fcdd
* changes:
treewide: make more settings conditionally available
libstore/build: only send overridden settings to the build hook
treewide: consistently mark overridden settings as such
a first little step into pushing the event loops up, up and away.
eventually we will want them to be instantiated only at the roots
of every thread (since kj binds loops to threads), but not today.
Change-Id: Ic97f1debba382a5a3f46daeaf2d6d434ee42569f
goals should be considered internal to the worker architecture due to
the tight coupling of the two, and we can finally do that. doing this
is also a prerequisite for turning Worker::run() into a real promise.
Change-Id: I7cf273d4a6fdb75b8d192fce1af07c6265ff6980
previously it was possible to fetchurl a dict server, or an ldap server,
or an imap server. this is a bit of a problem, both because rare schemes
may not be available on all systems, and because some schemes (e.g. scp)
are inherently insecure in potentially surprising ways we needn't allow.
Change-Id: I18fc567c6f58c3221b5ea8ce927f4da780057828
we already have a results type for the entire worker call, we may as
well put this bit of info in there. we do keep the assumption of the
old code that the field will only be read if some goals have failed;
fixing that is a very different mess, and not immediately necessary.
Change-Id: If3fc32649dcd88e1987cdd1758c6c5743e3b35ac
without this derivations do not show as completely processed in the
internal-json logs (or the newer multiline output). the former also
breaks external tools like nix-output-monitor which, like multiline
output, grow vertically until at least some goals are finally freed
Change-Id: I55758daf526ba29ae15fb82e0d88da8afb45bf5c
optimise-store.cc used std::regex on darwin, but forgot to include the
header. This probably compiled due to the precompiled headers file, but
it caused errors in the editor.
Change-Id: I23297c08cb66d44e4d4f303560f46e4adc7d5a43
It's not clear to me if `proc_pidinfo()` or `proc_pidfdinfo()` can
actually return negative values, the syscall wrappers convert `-1` into
zero and the semantics suggest that negative values don't make sense,
but just to be safe we'll preserve the int type until we've checked that
it's a positive value.
Fixes: https://git.lix.systems/lix-project/lix/issues/548
Change-Id: If575aec6b1e27dba63091c7a0316c7b3788747cd
some promises capture `this`. we could also allocate a shared state,
but this thing doesn't really need to ever be moved anyway. so there.
Change-Id: I50b5c44684a8ab4e984b1323de21f97ace4a864a
not doing this can freeze slots until the goal that occupied them is
freed (rather than simply complete), and then can freeze the system.
fixes#549
Change-Id: I042df04222f8ffbaa18ef4a4eae6cbd6f89b679e
Daemon client handler processes are forked off of the main nix process
and thus will not have a signal handler thread anymore. This leads to a
high likelihood of bustage, since the Worker infrastructure expects the
interrupt infrastructure to actually, you know, work, to be able to get
interrupted.
The expected behaviour after fork is either:
- Start a signal handler thread if you expect to do complicated things
that need ReceiveInterrupts.
- Call restoreProcessContext and don't handle signals specially
otherwise.
Change-Id: I73d36b5bbf96dddd21d5e1c3bd0484d715c00e8b
this was missed in 3f07c65510. if mode is
not restored, the tree will have mutability where it shouldn't.
fixes test `functional-repair`, which fails from the output path
directory being unnecessarily writable:
```
++(repair.sh:12) nix-hash /tmp/nix-shell.IIUJAq/nix-test/repair/store/j3kmp9zhc5y7gqs591l0nrscm5hw4145-dependencies-input-2
+(repair.sh:12) hash=d790f49fc89cb6f384b6dbe450790d07
+(repair.sh:15) chmod u+w /tmp/nix-shell.IIUJAq/nix-test/repair/store/j3kmp9zhc5y7gqs591l0nrscm5hw4145-dependencies-input-2
+(repair.sh:16) touch /tmp/nix-shell.IIUJAq/nix-test/repair/store/j3kmp9zhc5y7gqs591l0nrscm5hw4145-dependencies-input-2/bad
+(repair.sh:18) nix-store --verify --check-contents -v
reading the Nix store...
checking path existence...
checking link hashes...
checking store hashes...
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/20z19rjkwmwpb2ba4x29kac6xnslai38-dependencies-top.drv'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/3mmawi9lj33xz96cf6kw9989xc8v5i96-fod-input'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/5fp4r2kh1fcv6mv9dv6rywhhr1am9hhm-builder-dependencies-input-0.sh'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/9kryn4ihv6b7bjswv2rsjq4533n2w5zk-fod-input.drv'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/ap8s0fim8s3ilzj8aqwlwk7gv1crjp4j-dependencies-top'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/i2ipd9p282zkqr1zb4glqiqigv8ybsyk-dependencies.builder0.sh'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/i3xyw46h0lsx7ad6bczwi9sqjjx5f0j0-dependencies-input-0'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/j3kmp9zhc5y7gqs591l0nrscm5hw4145-dependencies-input-2'
path '/tmp/nix-shell.IIUJAq/nix-test/repair/store/j3kmp9zhc5y7gqs591l0nrscm5hw4145-dependencies-input-2' was modified! expected hash 'sha256:1rhaylnjs5lbp089lnk7qvsypqmbm5vvyvxvv7i68b4x33pncqgs', got 'sha256:0nkjrmdc6ixf935chj3zhpqph5i15p306ffdsa850qh8mncpnsmc'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/pfaxqiw8zf3bw0w8w8gazswh76d729yy-builder-dependencies-input-2.sh'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/pzzms3k5wl8d3wszv3maw29zylfkiiw0-dependencies-input-1'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/riwx84p57ckfvgli9nwhx88z6zh1c8ss-builder-fod-input.sh'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/rjsphx7xvnqh2qafdrr7fiyxqc1rljhw-dependencies-input-2.drv'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/scywzq87dvm0c5f1h16hww6mvzhcsx3f-builder-dependencies-input-1.sh'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/v65j4w033i05nyfd2h0g8h41ijmfglp7-dependencies-input-0.drv'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/ykwibh62xyp81k75vdw9c7y21kg4ibzf-dependencies-input-1.drv'
warning: not all store errors were fixed
+(repair.sh:21) nix-store --verify --check-contents --repair
reading the Nix store...
checking path existence...
checking link hashes...
checking store hashes...
path '/tmp/nix-shell.IIUJAq/nix-test/repair/store/j3kmp9zhc5y7gqs591l0nrscm5hw4145-dependencies-input-2' was modified! expected hash 'sha256:1rhaylnjs5lbp089lnk7qvsypqmbm5vvyvxvv7i68b4x33pncqgs', got 'sha256:0nkjrmdc6ixf935chj3zhpqph5i15p306ffdsa850qh8mncpnsmc'
checking path '/tmp/nix-shell.IIUJAq/nix-test/repair/store/j3kmp9zhc5y7gqs591l0nrscm5hw4145-dependencies-input-2'...
path '/tmp/nix-shell.IIUJAq/nix-test/repair/store/j3kmp9zhc5y7gqs591l0nrscm5hw4145-dependencies-input-2' is corrupted or missing!
checking path '/tmp/nix-shell.IIUJAq/nix-test/repair/store/i3xyw46h0lsx7ad6bczwi9sqjjx5f0j0-dependencies-input-0'...
repairing outputs of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/rjsphx7xvnqh2qafdrr7fiyxqc1rljhw-dependencies-input-2.drv'...
+(repair.sh:23) '[' -e /tmp/nix-shell.IIUJAq/nix-test/repair/store/j3kmp9zhc5y7gqs591l0nrscm5hw4145-dependencies-input-2/bad ']'
+(repair.sh:24) '[' -w /tmp/nix-shell.IIUJAq/nix-test/repair/store/j3kmp9zhc5y7gqs591l0nrscm5hw4145-dependencies-input-2 ']'
++(repair.sh:24) onError
++(/var/home/bb010g/Sources/Nix/lix/build/tests/functional/common/vars-and-functions.sh:244) set +x
repair.sh: test failed at:
main in repair.sh:24
```
fixes test `functional-simple`, which fails from the output path file
being unnecessarily writable:
```
+(simple.sh:11) echo 'output path is /tmp/nix-shell.IIUJAq/nix-test/simple/store/9fqn0rs99ymn1r09yip7bsifcdh3ra0y-simple'
+(simple.sh:13) '[' -w /tmp/nix-shell.IIUJAq/nix-test/simple/store/9fqn0rs99ymn1r09yip7bsifcdh3ra0y-simple ']'
++(simple.sh:13) onError
++(/var/home/bb010g/Sources/Nix/lix/build/tests/functional/common/vars-and-functions.sh:244) set +x
simple.sh: test failed at:
main in simple.sh:13
```
fixes test `functional-optimise-store`, which fails from the unnecessary
mutability making Lix avoid hard linking:
```
++(optimise-store.sh:5) nix-build - --no-out-link --auto-optimise-store
this derivation will be built:
/tmp/nix-shell.IIUJAq/nix-test/optimise-store/store/2ql4kjxhnzdard8d6n3h9hc1m3lawr2b-foo1.drv
building '/tmp/nix-shell.IIUJAq/nix-test/optimise-store/store/2ql4kjxhnzdard8d6n3h9hc1m3lawr2b-foo1.drv'...
warning: skipping suspicious writable file '/tmp/nix-shell.IIUJAq/nix-test/optimise-store/store/h7i1pp848f9a8452y0s18kgsnis77vjn-foo1/foo'
+(optimise-store.sh:5) outPath1=/tmp/nix-shell.IIUJAq/nix-test/optimise-store/store/h7i1pp848f9a8452y0s18kgsnis77vjn-foo1
++(optimise-store.sh:6) echo 'with import ./config.nix; mkDerivation { name = "foo2"; builder = builtins.toFile "builder" "mkdir $out; echo hello > $out/foo"; }'
++(optimise-store.sh:6) nix-build - --no-out-link --auto-optimise-store
this derivation will be built:
/tmp/nix-shell.IIUJAq/nix-test/optimise-store/store/wjgvfhfsp14w06im8bbp1kqzz7smdkcy-foo2.drv
building '/tmp/nix-shell.IIUJAq/nix-test/optimise-store/store/wjgvfhfsp14w06im8bbp1kqzz7smdkcy-foo2.drv'...
warning: skipping suspicious writable file '/tmp/nix-shell.IIUJAq/nix-test/optimise-store/store/gjg6gayj2f6x3h53sp5i21nbnsd7b4i3-foo2/foo'
+(optimise-store.sh:6) outPath2=/tmp/nix-shell.IIUJAq/nix-test/optimise-store/store/gjg6gayj2f6x3h53sp5i21nbnsd7b4i3-foo2
++(optimise-store.sh:8) stat --format=%i /tmp/nix-shell.IIUJAq/nix-test/optimise-store/store/h7i1pp848f9a8452y0s18kgsnis77vjn-foo1/foo
+(optimise-store.sh:8) inode1=328316
++(optimise-store.sh:9) stat --format=%i /tmp/nix-shell.IIUJAq/nix-test/optimise-store/store/gjg6gayj2f6x3h53sp5i21nbnsd7b4i3-foo2/foo
+(optimise-store.sh:9) inode2=328404
+(optimise-store.sh:10) '[' 328316 '!=' 328404 ']'
+(optimise-store.sh:11) echo 'inodes do not match'
+(optimise-store.sh:12) exit 1
```
Signed-off-by: Dusk Banks <me@bb010g.com>
Change-Id: I87eeb74e718746a587be2ac52bcc9b5b1e5529db
This is done because the one in nixpkgs has several problems and we
don't want to conflict with them fixing those problems:
- not building shared libs
- not building debuginfo (more critical for us due to
https://git.lix.systems/lix-project/lix/issues/549)
- not setting the correct cmake build type
- not setting the correct cxxflags to build the coroutine library
- not building with clang (required for the coroutine library since gcc
is known to miscompile coroutines *in kj*).
CC: https://git.lix.systems/lix-project/lix/issues/551
Fixes: https://git.lix.systems/lix-project/lix/issues/550
Change-Id: Ia5b78dc1809963fdd1a8203b127a216cb575d751
This workaround was for the stdenv not being set when callPackage'ing
package.nix for some of the stranger CI outputs.
Change-Id: I2acdd6efa721b90dd3cb04358544d25d591ff084
Some stuff wasn't working on my machine because it depended on the host
environment in some ways, fixed those.
Change-Id: Iff4931a9a26c6827978f1ee6434710f406d67a96
The old behavior results in lots of concatenations happening for no good
reason and is an artifact of the technical limitations of the old parser
(combined with some lack of care for such details).
Change-Id: I0d78d6220ca6aeaa10bc437e48e08bf7922e0bb3
This is only a minor semantical distinction, but we should be able to
properly test it, and the parser tests rely on show for that.
Change-Id: I25e868cf9544e30cdff17deb5fd50a434e0f367e
This commit should faithfully reproduce the old behavior down to the
bugs. The new code is a lot more readable, all quirks are well
documented, and it is overall much more maintainable.
Change-Id: I629585918e4f2b7d296b6b8330235cdc90b7bade
Some settings only make sense on particular platforms, or only when a certain
experimental feature is enabled. Several of those were already conditionally
available. Do the same for a bunch more instead of silently ignoring them.
Exceptionally, the use-case-hack setting is not made conditional because it is
included in the test suite.
Change-Id: I29e66ad8ee6178a7c0eff9efb55c3410fae32514
The build hook is still running locally, so it will run with the same default
settings. Hence, just as with the daemon, it is enough to send it only the
overridden settings. This will prevent warnings like
warning: Ignoring setting 'auto-allocate-uids' because experimental feature 'auto-allocate-uids' is not enabled
when the user didn't actually set those settings.
This is inspired by and an alternative to [0].
[0] https://github.com/NixOS/nix/pull/10049
Change-Id: I77ea62cd017614b16b55979dd30e75f09f860d21
Only overridden settings are sent to the daemon, and we're going to do the same
for the build hook to. It needs to be ensured that overridden settings are in
fact consistently marked as such, so that they actually get sent.
Change-Id: I7cd58d925702f86cf2c35ad121eb191ceb62a355
Before this change, expressions like:
with import <nixpkgs> {};
runCommand "foo" {} ''
echo '@nix {}' >&$NIX_LOG_FD
''
would result in Lix crashing, because accessing nonexistent fields of
a JSON object throws an exception.
Rather than handling each field individually, we just catch JSON
exceptions wholesale. Since these log messages are an unusual
circumstance, log a warning when this happens.
Fixes#544.
Change-Id: Idc2d8acf6e37046b3ec212f42e29269163dca893
By implementing the `PathSet` specialization for `PathsSetting`, we'll
be able to use `PathsSetting` for the `sandboxPaths` setting in
`src/libstore/globals.hh`.
Fixes: https://git.lix.systems/lix-project/lix/issues/498
Change-Id: I8bf7dfff98609d1774fdb36d63e57d787bcc829f
This failed due to https://github.com/NixOS/nixpkgs/pull/346945, which
makes a second lowdown-unsandboxed that works in nix builds, and the
regular lowdown has executables that fail closed when the sandbox setup
fails.
The actual failure here is only visible on nixos-unstable at the moment,
not 24.05, but this commit should fix it up for all versions.
Fixes: https://git.lix.systems/lix-project/lix/issues/547
Change-Id: I50c0ecb59518ef01a7c0181114c1b4c5a7c6b78b
This was always a terrible idea independently of whether it crashes.
Stop doing it!
This commit was verified by running nix-shell on a trivial derivation
with --debug --verbose to get the vomit-level output of the shell rc
file and then diffing it before/after this change. I have reasonable
confidence it did not regress anything, though this code is genuinely
really hard to follow (which is a second reason that I split it into two
fmt calls).
Fixes: https://git.lix.systems/lix-project/lix/issues/533
Change-Id: I8e11ddbece2b12749fda13efe0b587a71b00bfe5
This change feels kind of gross and reveals a fair bit about the
disorganization of our tests, but I think it makes parts of it a bit
better.
Change-Id: Idb8d9a00cbd75d5c156678c6b408b42b59d5e4d7
A better fix than in 104448e75d, hence a
revert + the fix.
It turns out that this commit has the side-effect that when having e.g.
`StrictHostKeyChecking=accept-new` for a remote builder, the warnings à la
Warning: Permanently added 'builder' (ED25519) to the list of known hosts.
actually end up in the derivation's log whereas hostkey verification
errors don't, but only in the stderr of the `nix-build` invocation
(which was the motivation for the patch).
This change writes the stderr from the build-hook to
* the daemon's stderr, so that the SSH errors appear in the journal
(which was the case before 104448e75d)
* the client's stderr, as a log message
* NOT to the drv log (this is handled via `handleJSONLogMessage`)
I tried to fix the issue for legacy-ssh as well, but failed and
ultimately decided to not bother.
I know that we'll sooner or later replace the entire component, however
this is the part of the patch I have working for a while, so I figured I
might still submit it for the time being.
Change-Id: I21ca1aa0d8ae281d2eacddf26e0aa825272707e5
When the git default branch is not set to master the installcheck
test suite fails. This patch adjusts the test setup scripts to
ignore the system and user git config files.
GIT_CONFIG_SYSTEM is set to /dev/null to ignore /etc/gitconfig
GIT_CONFIG_GLOBAL is not set because the global config files
are loaded from $HOME or $XDG_CONFIG_HOME which we already
reset.
git documentation: https://git-scm.com/docs/git#Documentation/git.txt-codeGITCONFIGGLOBALcode
Change-Id: Ie73bbed1db9419c9885b9d57e4edb7a4047d5cce
While debugging something else I observed that latest `main` ignores
`Control-C` on `sudo nix-build`.
After reading through the capnproto docs, it seems as if the promise
must be fulfilled to actually terminate the `promise.wait()` below.
This also applies to scenarios such as stopping the client
(`nix-build`), but the builders on the daemon-side are still running,
i.e. closes#540
Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: I9634d14df4909fc1b65d05654aad0309bcca8a0a
So we received a report that the thread pool crashed due to an
Interrupted exception.
Relevant log tail:
copying path '/nix/store/0kal2k73inviikxv9f1ciaj39lkl9a87-etc-os-release' to 'ssh://192.168.0.27'...
Lix crashed. This is a bug. We would appreciate if you report it along with what caused it at https://git.lix.systems/lix-project/lix/issues with the following information included:
error (ignored): error: interrupted by the user
Exception: nix::Interrupted: error: interrupted by the user
Relevant stack trace:
4# __cxa_rethrow in /nix/store/22nxhmsfcv2q2rpkmfvzwg2w5z1l231z-gcc-13.3.0-lib/lib/libstdc++.so.6
5# nix::ignoreExceptionExceptInterrupt(nix::Verbosity) in /nix/store/ghxr2ykqc3rrfcy8rzdys0rzx9ah5fqj-lix-2.92.0-dev-pre20241005-ed9b7f4/lib/liblixutil.so
6# nix::ThreadPool::doWork(bool) in /nix/store/ghxr2ykqc3rrfcy8rzdys0rzx9ah5fqj-lix-2.92.0-dev-pre20241005-ed9b7f4/lib/liblixutil.so
7# 0x00007FA7A00E86D3 in /nix/store/22nxhmsfcv2q2rpkmfvzwg2w5z1l231z-gcc-13.3.0-lib/lib/libstdc++.so.6
8# 0x00007FA79FE99A42 in /nix/store/3dyw8dzj9ab4m8hv5dpyx7zii8d0w6fi-glibc-2.39-52/lib/libc.so.6
9# 0x00007FA79FF1905C in /nix/store/3dyw8dzj9ab4m8hv5dpyx7zii8d0w6fi-glibc-2.39-52/lib/libc.so.6
Notably, this is *not* in the main thread, so this implies that the
thread didn't get joined properly before their destructors got called.
That, in turn, should have only possibly happened because join() threw
on a previous iteration of the loop joining threads, I think. Or if it
threw while in the ThreadPool destructor. Either way we had better stop
letting Interrupted fall out of our child threads!
If:
- Interrupted was thrown inside the action in the main thread: it would
have fallen out of doWork if state->exception was already set and got
caught by ThreadPool::process, calling shutdown() and the join loop
which would crash the process entirely.
- Interrupted was thrown inside the action on a secondary thread: it
would have been caught and put into the exception field and then
possibly rethrown to fall out of the thread (since it was previously
ignoreExceptionExceptInterrupt).
The one possible hole in this hypothesis is that there is an "error
(ignored)" line in there implying that at least one Interrupted got
eaten by an ignoreExceptionInDestructor. It's also unclear whether this
got reordered because of stderr buffering.
Fixes: https://git.lix.systems/lix-project/lix/issues/542
Change-Id: I322cf050da660af78f5cb0e08ec6e6d27d09ac76
There is absolutely no good reason these should show up in NARs besides
misconfigured systems and as long as the case hack exists, unpacking
such a NAR will cause its repacking to be wrong on systems with case
hack enabled.
This should not have any security impact on Lix to fix, but it was one
of the vectors for CVE-2024-45593:
https://github.com/NixOS/nix/security/advisories/GHSA-h4vv-h3jq-v493
Change-Id: I85b6075aacc069ee7039240b0f525804a2d8edcb
This is capped at 12 because 3.7 seconds of startup is painful enough
and 5.5 seconds with 24 was more annoying.
Change-Id: I327db40fd98deaa5330cd9cf6de99fb07b2c1cb0
This also rewrites a lot of the command handling in the fixtures
library, since we want to more precisely control which way that the nix
store is set up in the tests, rather than the previous method of
renaming /nix/store to some temp dir (which allows builds but does not
allow any /nix/store paths or stability across runs, which is a
significant issue for snapshot testing).
It uses a builder to reduce the amount of state carelessly thrown
around.
The evil NARs are inspired by CVE-2024-45593
(https://github.com/NixOS/nix/security/advisories/GHSA-h4vv-h3jq-v493).
No bugs were found in this endeavor.
Change-Id: Iee41b055fa96529c5a3c761f680ed1d0667ba5da
I am tired of bad shell scripts, let me write bad python quickly
instead. It's definitely, $100%, better.
This is not planned as an immediate replacement of the old test suite,
but we::jade would not oppose tests getting ported.
What is here is a mere starting point and there is a lot more
functionality that we need.
Fixes: https://git.lix.systems/lix-project/lix/issues/488
Change-Id: If762efce69030bb667491b263b874c36024bf7b6
I followed @pennae's advice and moved the constructor definition of
`AttrName` from the header file `nixexpr.hh` to `nixexpr.cc`.
Change-Id: I733f56c25635b366b11ba332ccec38dd7444e793
The approach that was taken here was to add default values to the type
definitions rather than specify them whenever they are missing.
Now the only remaining warning is '-Wunused-parameter' which @jade said
is usually counterproductive and that we can just disable it:
https://git.lix.systems/lix-project/lix/issues/456#issuecomment-6617
So this change adds the flags '-Wall', '-Wextra' and
'-Wno-unused-parameter', so that all warnings are enabled except for
'-Wunused-parameter'.
Change-Id: Ic223a964d67ab429e8da804c0721ba5e25d53012
There was a bug report about a potential call to `memcpy` with a null
pointer which is not reproducible:
https://git.lix.systems/lix-project/lix/issues/492
This occurred in `src/libstore/filetransfer.cc` in `InnerSource::read`.
To ensure that this doesn't happen, an early return is added before
calling `memcpy` if the length of the data to be copied is 0.
This change also adds a test that ensures that when `InnerSource::read`
is called with an empty file, it throws an `EndOfFile` exception.
Change-Id: Ia18149bee9a3488576c864f28475a3a0c9eadfbb
these two functions are now nearly trivial and much better inline into
makeGoalCommon. keeping them separate also separates information about
goal completion flows and how failure information ends up in `Worker`.
Change-Id: I6af86996e4a2346583371186595e3013c88fb082
we can use our newfound powers of Goal::work Is A Real Promise to remove
completed goals from continuation promises. apart from being much easier
to follow it's also a lot more efficient because we have the iterator to
the item we are trying to remove, skipping a linear search of the cache.
Change-Id: Ie0190d051c5f4b81304d98db478348b20c209df5
Goal::work() is a fully usable promise that does not rely on the worker
to report completion conditions. as such we no longer need the `notify`
field that enabled this interplay. we do have to clear goal caches when
destroying the worker though, otherwise goal promises may (incorrectly)
keep goals alive due to strong shared pointers created by childStarted.
Change-Id: Ie607209aafec064dbdf3464fe207d70ba9ee158a
derivation goals still hold a BuildResult member variable since parts of
these results of accumulated in different places, but the Goal class now
no longer has such a field. substitution goals don't need it at all, and
derivation goals should also be refactored to not drop their buildResult
Change-Id: Ic6d3d471cdbe790a6e09a43445e25bedec6ed446
the field is simply duplicated between the two, and now that we can
return WorkResults from Worker::run we no longer need both of them.
Change-Id: I82fc47d050b39b7bb7d1656445630d271f6c9830
this will be needed to move all interesting result fields out of Goal
proper and into WorkResult. once that is done we can treat goals as a
totally internal construct of the worker mechanism, which also allows
us to fully stop exposing unclear intermediate state to Worker users.
Change-Id: I98d7778a4b5b2590b7b070bdfc164a22a0ef7190
since we now propagate goal exceptions properly we no longer need to
check topGoals for a reason to abort early. any early abort reasons,
whether by exception or a clean top goal failure, can now be handled
by inspecting the goal result in the main loop. this greatly reduces
goal-to-goal interactions that do not happen at the main loop level.
since the underscore-free name is now available for use as variables
we'll migrate to that where we currently use `_topGoals` for locals.
Change-Id: I5727c5ea7799647c0a69ab76975b1a03a6558aa6
drop childException since it's no longer needed. also makes
waitForInput, childFinished, and childTerminated redundant.
Change-Id: I05d88ffd323c5b5c909ac21056162f69ffb0eb9f
there's no reason to have the worker set information on goals that the
goals themselves return from their entry point. doing this in the goal
`work()` function is much cleaner, and a prerequisite to removing more
implicit strong shared references to goals that are currently running.
Change-Id: Ibb3e953ab8482a6a21ce2ed659d5023a991e7923
this simplifies the worker loop, and lets us remove it entirely later.
note that ideally only one promise waiting for interrupts should exist
in the entire system. not one per event loop, one per *process*. extra
interrupt waiters make interrupt response nondeterministic and as such
aren't great for user experience. if anything wants to react to aborts
caused by explicit interruptions, or anything else, those things would
be much better served using RAII guards such as Finally (or KJ_DEFER).
Change-Id: I41d035ff40172d536e098153c7375b0972110d51
this was a triumph. i'm making a note here: huge success. it's hard to
overstate my satisfaction! i'm not even angry. i'm being so sincere ri
actually, no. we *are* angry. this was one dumbass odyssey. nobody has
asked for this. but not doing it would have locked us into old, broken
protocols forever or (possibly worse) forced us to write our own async
framework building on the old did-you-mean-continuations in Worker. if
we had done that we'd be locked into ever more, and ever more complex,
manual state management all over the place. this just could not stand.
Change-Id: I43a6de1035febff59d2eff83be9ad52af4659871
with async runtime scheduling we can no longer guarantee exact error
counts for builds that do not set keepGoing. the old behavior can be
recovered with a number of hacks that affect scheduling, but none of
those are very easy to follow now advisable. exact error counts will
like not be needed for almost all uses except tests, and *those* had
better check the actual messages rather than how many they got. more
messages can even help to avoid unnecessary rebuilds for most users.
Change-Id: I1c9aa7a401227dcaf2e19975b8cb83c5d4f85d64
In 0e6b3435a1, installation of the HTML manual
was accidentally dropped: setting install_dir on a custom_target only sets the
directory where something is going to be installed if it is installed at all,
but does not itself trigger installation. The latter has to be explicitly
requested, which is just what we do here to get the manual back.
Change-Id: Iff8b791de7e7cb4c8d747c2a9b1154b5fcc32fe0
The test suite can load the global configuration files under certain
circumstances, and, though we would really rather it didn't ever do that
at all, we should at least break the mechanism.
Fixes: https://git.lix.systems/lix-project/lix/issues/474
Change-Id: Ib27cb43dd5dfaa70ac491c395b5ba308fd7bd289
due to event loop scheduling behavior it's possible for a derivation
goal to fully finish (having seen all paths it was asked to create),
but to not notify the worker of this in time to prevent another goal
asking the recently-finished goal for more outputs. if this happened
the finished goal would ignore the request for more outputs since it
considered itself fully done, and the delayed result reporting would
cause the requesting goal to assume its request had been honored. if
the requested goal had finished *properly* the worker would recreate
it instead of asking for more outputs, and this would succeed. it is
thus safe to always recreate goals once they are done, so we now do.
Change-Id: Ifedd69ca153372c623abe9a9b49cd1523588814f
This moves the "legacy"/"nix2" commands under a new `src/legacy/`
directory, instead of being scattered around in a bunch of different
directories.
A new `liblegacy` build target is defined, and the `nix` binary is
linked against it.
Then, `RegisterLegacyCommand` is replaced with `LegacyCommand::add`
calls in functions like `registerNixCollectGarbage()`. These
registration functions are called explicitly in `src/nix/main.cc`.
See: https://git.lix.systems/lix-project/lix/issues/359
Change-Id: Id450ffc3f793374907599cfcc121863b792aac1a
This splits `ignoreException` into `ignoreExceptionExceptInterrupt`
(which ignores all exceptions except `Interrupt`, which indicates a
SIGINT/CTRL-C) and `ignoreExceptionInDestructor` (which ignores all
exceptions, so that destructors do not throw exceptions).
This prevents many cases where Nix ignores CTRL-C entirely.
See: https://github.com/NixOS/nix/issues/7245
Upstream-PR: https://github.com/NixOS/nix/pull/11618
Change-Id: Ie7d2467eedbe840d1b9fa2e88a4e88e4ab26a87b
we'll now loop to update displayed statistics, and use this loop to
limit the update rate to 50 times per second. we could have updated
much more frequently before this (once per iteration of `runImpl`),
much faster than would ever be useful in practice. aggressive stats
updates can even impede progress due to terminal or network delays.
Change-Id: Ifba755a2569f73c919b1fbb06a142c0951395d6d
Worker::run() is now entirely based on the kj event loop and promises,
so we need not handle awakeness of goals manually any more. every goal
can instead, once it has finished a partial work call, defer itself to
being called again in the next iteration of the loop. same end effect.
Change-Id: I320eee2fa60bcebaabd74d1323fa96d1402c1d15
notably we will check whether we want to do GC at all only once during
startup, and we'll only attempt GC every ten seconds rather than every
time a goal has finished a partial work call. this shouldn't cause any
problems in practice since relying on auto-gc is not deterministic and
stores in which builds can fill all remaining free space in merely ten
seconds are severely troubled even when gargage collection runs a lot.
Change-Id: I1175a56bf7f4e531f8be90157ad88750ff2ddec4
Revert submission 1946
Reason for revert: regression in building (found via bisection)
Reported by users:
> error: path '/nix/store/04ca5xwvasz6s3jg0k7njz6rzi0d225w-jq-1.7.1-dev' does not exist in the store
Reverted changes: /q/submissionid:1946
Change-Id: I6f1a4b2f7d7ef5ca430e477fc32bca62fd97036b
nothing needs to signal being still active but not actively pollable,
only that immediate polling for the next goal work phase is in order.
Change-Id: Ia43c1015e94ba4f5f6b9cb92943da608c4a01555
this was immensely inefficient on large caches, as can exist when many
derivations are buildable simultaneously. since we have smart pointers
to goals we can do cache maintenance in goal deleters instead, and use
the exact iterators instead of doing a linear search. this *does* rely
on goals being deleted to remove them from the cache, which isn't true
for toplevel goals. those would have previously been removed when done
in all cases, removing the cache entry when keep-going is set. this is
arguably incorrect since it might result in those goals being retried,
although that could only happen with dynamic derivations or the likes.
(luckily dynamic derivations not complete enough to allow this at all)
Change-Id: I8e750b868393588c33e4829333d370f2c509ce99
makeDerivationGoalCommon had the right idea, but it didn't quite go far
enough. let's do the rest and remove the remaining factory duplication.
Change-Id: I1fe32446bdfb501e81df56226fd962f85720725b
this was a debugging aid from day one that should not have any impact on
build semantics, and if it *does* have an impact on build semantics then
build semantics are seriously broken. keeping the order imposed by these
keys will be impossible once we let a real event loop schedule our jobs.
Change-Id: I5c313324e1f213ab6453d82f41ae5e59de809a5b
without circular references we do not need weak goal pointers except for
caches, which should not prevent goal destructors running. caches though
cannot create circular references even when they keep strong references.
if we removed goals from caches when their work() is fully finished, not
when their destructors are run, we could keep strong pointers in caches.
since we do not gain much from this we keep those pointers weak for now.
Change-Id: I1d4a6850ff5e264443c90eb4531da89f5e97a3a0
have DerivationGoal and its subclasses produce a wrapper promise for
their intermediate results instead, and return this wrapper promise.
Worker already handles promises that do not complete immediately, so
we do not have to duplicate this into an entire result type variant.
Change-Id: Iae8dbf63cfc742afda4d415922a29ac5a3f39348
the new event loop could very occasionally notice that a dependency of
some goal has failed, process the failure, cause the depending goal to
fail accordingly, and in the doing of the latter two steps let further
dependencies that previously have not been reported as failed do their
reporting anyway. in such cases a goal could fail with "1 dependencies
failed", but more than one dependency failure message was shown. we'll
now report the correct number of failed dependency goals in all cases.
Change-Id: I5aa95dcb2db4de4fd5fee8acbf5db833531d81a8
these can be unique rather than shared because shared_ptr has a
converting constructor. preparatory refactor for something else
and not necessary on its own, and the extra allocations we must
do for shared_ptr control blocks isn't usually relevant anyway.
Change-Id: I5391715545240c6ec8e83a031206edafdfc6462f
Since fb38459d6e, each `ref` is appended
with `refs/heads` unless it starts with `refs/` already. This regressed
two use-cases that worked fine before:
* Specifying a commit hash as `ref`: now, if `ref` looks like a commit
hash it will be directly passed to `git fetch`.
* Specifying a tag without `refs/tags` as prefix: now, the fetcher prepends
`refs/*` to a ref that doesn't start with `refs/` and doesn't look
like a commit hash. That way, both a branch and a tag specified in
`ref` can be fetched.
The order of preference in git is
* file in `refs/` (e.g. `HEAD`)
* file in `refs/tags/`
* file in `refs/heads` (i.e. a branch)
After fetching `refs/*`, ref is resolved the same way as git does.
Change-Id: Idd49b97cbdc8c6fdc8faa5a48bef3dec25e4ccc3
We have manually enabled libstdc++ assertions since cl/797. Meson 1.4.0
(the minimum version we mandate) enables this by default, so we can
remove the enabling compiler flag from the list of project arguments.
With this patch, `-D_GLIBCXX_ASSERTIONS=1` is still present in the
compile command logs when building with both gccStdenv and clangStdenv.
See: https://gerrit.lix.systems/c/lix/+/797
See: https://mesonbuild.com/Release-notes-for-1-4-0.html#ndebug-setting-now-controls-c-stdlib-assertions
Change-Id: I53483fadfe5cbd11ba35544b437d3a9ee8031631
This was already the de facto requirement, we use the method `full_path`
on a file object (introduced in Meson 1.4.0) in the functional test
suite's build.
This version of Meson is in NixOS 24.05, so there should be no
compatibility issues should this make it into a backported release of
Lix.
CC: https://git.lix.systems/lix-project/lix/issues/247
Change-Id: I5c640824807353b6eb4287e7ed09c4e89a4bdde2
Using `configure_file` to copy files has been deprecated since Meson 0.64.0.
The intended replacement is the `fs.copyfile` method.
This removes the following deprecation warning that arises when a minimum
Meson version is specified:
``
Project [...] uses feature deprecated since '0.64.0': copy arg in configure_file. Use fs.copyfile instead
``
Change-Id: I09ffc92e96311ef9ed594343a0a16d51e74b114a
In Meson, `install_subdir` is meant to be used with directories in the source
directory. When using it to install the HTML manual, we provide it with a path
under the build directory.
We should instead specify an install directory for the HTML manual as part of
the custom target that builds it.
What we do currently isn't broken, just semantically incorrect. Changing it does
get rid of the following deprecation warning, though:
``
Project [...] uses feature deprecated since '0.60.0': install_subdir with empty directory. It worked by accident and is buggy. Use install_emptydir instead.
``
Change-Id: I259583b7bdff8ecbb3b342653d70dc5f034c7fad
also gets rid of explicit strong references to dependencies of any goal,
and weak references to dependers as well. those are now only held within
promises representing goal completion and thus independent of the goal's
relation to each other. the weak references to dependers was only needed
for notifications, and that's much better handled entirely by kj itself.
Change-Id: I00d06df9090f8d6336ee4bb0c1313a7052fb016b
now that we have an event loop in the worker we can use it and its
magical execution suspending properties to replace the slot counts
we managed explicitly with semaphores and raii tokens. technically
this would not have needed an event loop base to be doable, but it
is a whole lot easier to wait for a token to be available if there
is a callback mechanism ready for use that doesn't require a whole
damn dedicated abstract method in Goal to work, and specific calls
to that dedicated method strewn all over the worker implementation
Change-Id: I1da7cf386d94e2bbf2dba9b53ff51dbce6a0cff7
with waitForAWhile turned into promised the core functionality of
waitForInput is now merely to let gc run every so often if needed
Change-Id: I68da342bbc1d67653901cf4502dabfa5bc947628
this simplifies waitForInput quite a lot, and at the same time makes
polling less thundering-herd-y. it even fixes early polling wakeups!
Change-Id: I6dfa62ce91729b8880342117d71af5ae33366414
this removes the rather janky did-you-mean-async poll loop we had so
far. sadly kj does not play well with pty file descriptors, so we do
have to add our own async input stream that does not eat pty EIO and
turns it into an exception. that's still a *lot* better than the old
code, and using a real even loop makes everything else easier later.
Change-Id: Idd7e0428c59758602cc530bcad224cd2fed4c15e
When `nix fmt` is called without an argument, Nix appends the "." argument before calling the formatter. The comment in the code is:
> Format the current flake out of the box
This also happens when formatting sub-folders.
This means that the formatter is now unable to distinguish, as an interface, whether the "." argument is coming from the flake or the user's intent to format the current folder. This decision should be up to the formatter.
Treefmt, for example, will automatically look up the project's root and format all the files. This is the desired behaviour. But because the "." argument is passed, it cannot function as expected.
Upstream-PR: https://github.com/nixos/nix/pull/11438
Change-Id: I60fb6b3ed4ec1b24f81b5f0d76c0be98470817ce
like kj::joinPromisesFailFast this allows waiting for the results of
multiple promises at once, but unlike it not all input promises must
be complete (or any of them failed) for results to become available.
Change-Id: I0e4a37e7bd90651d56b33d0bc5afbadc56cde70c
like a normal semaphore, but with awaitable acquire actions. this is
primarily intended as an intermediate concurrency limiting device in
the Worker code, but it may find other uses over time. we do not use
std::counting_semaphore as a base because the counter of that is not
inspectable as will be needed for Worker. we also do not need atomic
operations for cross-thread consistency since we don't have multiple
threads (thanks to kj event loops being confined to a single thread)
Change-Id: Ie2bcb107f3a2c0185138330f7cbba4cec6cbdd95
Without this, verifying TLS certificates would fail on macOS, as well
as any system that doesn't have a certificate file at /etc/ssl/certs/ca-certificates.crt,
which includes e.g. Fedora.
Change-Id: Iaa2e0e9db3747645b5482c82e3e0e4e8f229f5f9
This is better for privacy and to avoid leaking netrc credentials in a
MITM attack, but also the assumption that we check the hash no longer
holds in some cases (in particular for impure derivations).
Partially reverts https://github.com/NixOS/nix/commit/5db358d4d78aea7204a8f22c5bf2a309267ee038.
(cherry picked from commit c04bc17a5a0fdcb725a11ef6541f94730112e7b6)
(cherry picked from commit f2f47fa725fc87bfb536de171a2ea81f2789c9fb)
(cherry picked from commit 7b39cd631e0d3c3d238015c6f450c59bbc9cbc5b)
Upstream-PR: https://github.com/NixOS/nix/pull/11585
Change-Id: Ia973420f6098113da05a594d48394ce1fe41fbb9
These stack traces kind of suck for the reasons mentioned on the
CppTrace page here (no symbols for inline functions is a major one):
https://github.com/jeremy-rifkin/cpptrace
I would consider using CppTrace if it were packaged, but to be honest, I
think that the more reasonable option is actually to move entirely to
out-of-process crash handling and symbolization.
The reason for this is that if you want to generate anything of
substance on SIGSEGV or really any deadly signal, you are stuck in
async-signal-safe land, which is not a place to be trying to run a
symbolizer. LLVM does it anyway, probably carefully, and chromium *can*
do it on debug builds but in general uses crashpad:
https://source.chromium.org/chromium/chromium/src/+/main:base/debug/stack_trace_posix.cc;l=974;drc=82dff63dbf9db05e9274e11d9128af7b9f51ceaa;bpv=1;bpt=1
However, some stack traces are better than *no* stack traces when we get
mystery exceptions falling out the bottom of the program. I've also
promoted the path for "mystery exceptions falling out the bottom of the
program" to hard crash and generate a core dump because although there's
been some months since the last one of these, these are nonetheless
always *atrociously* diagnosed.
We can't improve the crash handling further until either we use Crashpad
(which involves more C++ deps, no thanks) or we put in the ostensibly
work in progress Rust minidump infrastructure, in which case we need to
finish full support for Rust in libutil first.
Sample report:
Lix crashed. This is a bug. We would appreciate if you report it at https://git.lix.systems/lix-project/lix/issues with the following information included:
Exception: std::runtime_error: lol
Stack trace:
0# nix::printStackTrace() in /home/jade/lix/lix3/build/src/nix/../libutil/liblixutil.so
1# 0x000073C9862331F2 in /home/jade/lix/lix3/build/src/nix/../libmain/liblixmain.so
2# 0x000073C985F2E21A in /nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libstdc++.so.6
3# 0x000073C985F2E285 in /nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libstdc++.so.6
4# nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>) in /home/jade/lix/lix3/build/src/nix/../libmain/liblixmain.so
5# 0x00005CF65B6B048B in /home/jade/lix/lix3/build/src/nix/nix
6# 0x000073C985C8810E in /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libc.so.6
7# __libc_start_main in /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libc.so.6
8# 0x00005CF65B610335 in /home/jade/lix/lix3/build/src/nix/nix
Change-Id: I1a9f6d349b617fd7145a37159b78ecb9382cb4e9
We don't support GCC anymore for building, so the overlay currently
fails to evaluate with
error: assertion '((stdenv).cc.isClang || lintInsteadOfBuild)' failed
`clangStdenv` seems like a reasonable default now.
Noticed while upgrading Lix for our Hydra fork.
Change-Id: I948a7c03b3e5648fc7c596f96e1b8053a9e7f92f
Previously, Doxygen needed to be ran from the project's source root dir
due to the relative paths in the config's `INPUT` tag. We now preprocess
the relative paths by prefixing them with the absolute path of the
project's source root dir. The HTML output remains unchanged.
Fixes: https://git.lix.systems/lix-project/lix/issues/240
Change-Id: I85f099c22bfc5fdbf26be27c2db7dcbc8155c8b2
This caused an infinite loop before since it would just keep asking the
underlying source for more data.
In practice this happened because an HTTP server served a
response to a HEAD request (for which curl will not retrieve any body or
call our write callback function) with Content-Encoding: br, leading to
decompressing nothing at all and going into an infinite loop.
This adds a test to make sure none of our compression methods do that
again, as well as just patching the HTTP client to never feed empty data
into a compression algorithm (since they absolutely have the right to
throw CompressionError on unexpectedly-short streams!).
Reported on Matrix: https://matrix.to/#/!lymvtcwDJ7ZA9Npq:lix.systems/$8BWQR_zKxCQDJ40C5NnDo4bQPId3pZ_aoDj2ANP7Itc?via=lix.systems&via=matrix.org&via=tchncs.de
Change-Id: I027566e280f0f569fdb8df40e5ecbf46c211dad1
The legitimate output of `nix path-info` may visually interfere with the
progress bar, by appending to stale progress output before the latter has been
erased. Conveniently, all expensive operations (evaluation or building) have
already been performed before, so we can simply wipe the progress bar at this
point to fix the issue.
Fixes: https://git.lix.systems/lix-project/lix/issues/343
Change-Id: Id9a807a5c882295b3e6fbf841f9c15dc96f67f6e
editline's dep on ncurses is a runtime one, so it should be in
buildInputs, not nativeBuildInputs.
CC: https://git.lix.systems/lix-project/lix/issues/527
Change-Id: I631c192a55677b0cc77faa7511986f1fa2205e91
This test suite was in desperate need of using the parameterization
available with gtest, and was a bunch of useless duplicated code. At
least now it's not duplicated code, though it still probably should be
more full of property tests.
Change-Id: Ia8ccee7ef4f02b2fa40417b79aa8c8f0626ea479
See https://git.lix.systems/lix-project/lix/issues/496.
The core idea is to be able to do e.g.
nix-instantiate -A some-nonfree-thing --arg config.allowUnfree true
which is currently not possible since `config.allowUnfree` is
interpreted as attribute name with a dot in it.
In order to change that (probably), Jade suggested to find out if there
are any folks out there relying on this behavior.
For such a use-case, it may still be possible to accept strings, i.e.
`--arg '"config.allowUnfree"'.
Change-Id: I986c73619fbd87a95b55e2f0ac03feaed3de2d2d
Apparently forgejo has a more creative interpretation of \(\) than I was
hoping in their markdown parser and thought it was maths. I have no idea
then how you put a link in parens next to another square-bracket link,
but I am not going to worry about it.
There were several more typos, which I also fixed.
Fixes: https://git.lix.systems/lix-project/lix/issues/517
Change-Id: I6b144c6881f92ca60ba72a304ce7a0bcb9c6659a
* Move the extended attribute deletion after the hardlink sanity check. We
shouldn't be removing extended attributes on random files.
* Make the entity owner-writable before attempting to remove extended
attributes, since this operation usually requires write access on the file,
and we shouldn't fail xattr deletion on a file that has been made unwritable
by the builder or a previous canonicalisation pass.
Fixes: https://git.lix.systems/lix-project/lix/issues/507
Change-Id: I7e6ccb71649185764cd5210f4a4794ee174afea6
Remove the mutable state stuff that assumes that one file is being
written a time. It's true that we don't write multiple files
interleaved, but that mutable state is evil.
Change-Id: Ia1481da48255d901e4b09a9b783e7af44fae8cff
When generating shell completions, no logging output should be visible because
it would destroy the shell prompt. Originally this was attempted to be done by
simply disabling the progress bar (ca946860ce),
since the situation is particularly bad there (the screen clearing required for
the rendering ends up erasing the shell prompt). Due to overlooking the
implementation of this hack, it was accidentally undone during a later change
(0dd1d8ca1c).
Since even with the hack correctly in place, it is still possible to mess up
the prompt by logging output (for example warnings for disabled experimental
features, or messages generated by `builtins.trace`), simply send it to the bit
bucket where it belongs. This was already done for bash and zsh
(9d840758a8), and it seems that fish was simply
missed at that time. The last trace of the no-longer-working and obsolete hack
is deleted too.
Fixes: https://git.lix.systems/lix-project/lix/issues/513
Change-Id: I59f1ebf90903034e2059298fa8d76bf970bc3315
- Rename the listener to not be called a "sink". If it were a "sink" it
would be eating bytes and conform with any of the Nix sink stuff
(maybe FileHandle should be a Sink itself! but that's a later CL's
problem). This is a parser listener.
- Move the RetrieveRegularNARSink thing into store-api.cc, which is its
only usage, and fix it to actually do what it is stated to do: crash
if its invariants are violated.
It's, of course, used to erm, unpack single-file NAR files, generated
via a horrible contraption of sources and sinks that looks like a
plumbing blueprint. Refactoring that is a future task.
- Add a description of the invariants of NARParseVisitor in preparation
of refactoring it.
Change-Id: Ifca1d74d2947204a1f66349772e54dad0743e944
This applies https://github.com/troglobit/editline/pull/70 to our build
of editline, which translates `meta-left` and `meta-right` into
`fd_word` and `bk_word`. This makes `nix repl` soooo much nicer to use!
Note: My terminal renders `meta-left` as `\e\e[C` and `meta-right` as
`\e\e[D`.
Closes https://git.lix.systems/lix-project/lix/issues/501
Change-Id: I048b10cf17231bbf4e6bf38e1d1d8572cedaa194
while gcc 12 and older miscompile our generators, gcc 13 and older
outright crash on kj coroutines. (newer gcc versions may fix this)
Change-Id: I19f12c8c147239680eb0fa5a84ef5c7de38c9263
using a proper event loop basis we no longer have to worry about most of
the intricacies of poll(), or platform-dependent replacements for it. we
may even be able to use the event loop and its promise system for all of
our scheduling in the future. we don't do any real async processing yet,
this is just preparation to separate the first such change from the huge
api design difference with the async framework we chose (kj from capnp):
kj::Promise, unlike std::future, doesn't return exceptions unmangled. it
instead wraps any non-kj exception into a kj exception, erasing all type
information and preserving mostly the what() string in the process. this
makes sense in the capnp rpc use case where unrestricted exception types
can't be transferred, and since it moves error handling styles closer to
a world we'd actually like there's no harm in doing it only here for now
Change-Id: I20f888de74d525fb2db36ca30ebba4bcfe9cc838
we're using boost::outcome rather than leaf or stl types because stl
types are not available everywhere and leaf does not provide its own
storage for error values, relying on thread-locals and the stack. if
we want to use promises we won't have a stack and would have to wrap
everything into leaf-specific allocating wrappers, so outcome it is.
Change-Id: I35111a1f9ed517e7f12a839e2162b1ba6a993f8f
When the multi-line log format is enabled, the progress bar usually occupies
multiple lines on the screen. When stopping the progress bar, only the last
line was wiped, leaving all others visible on the screen. Erase all lines
belonging to the progress bar to prevent these leftovers.
Asking the user for input is theoretically affected by a similar issue, but
this is not observed in practice since the only place where the user is asked
(whether configuration options coming from flakes should be accepted) does not
actually have multiple lines on the progress bar. However, there is no real
reason to not fix this either, so let's do it anyway.
Change-Id: Iaa5a701874fca32e6f06d85912835d86b8fa7a16
The AcceptFlakeConfig type used was missing its JSON serialisation definition,
so it was incorrectly serialised as an integer, ending up that way for example
in the nix.conf manual page. Declare a proper serialisation.
Change-Id: If8ec210f9d4dd42fe480c4e97d0a4920eb66a01e
The JSON serialisation should be declared in the header so that all translation
units can see it when needed, even though it seems that it has not been used
anywhere else so far. Unfortunately, this means we cannot use the
NLOHMANN_JSON_SERIALIZE_ENUM convenience macro, since it uses a slightly
different signature, but the code is not too bad either.
Change-Id: I6e2851b250e0b53114d2fecb8011ff1ea9379d0f
This is the repl overlay from my dotfiles, which I think provides a
reasonable and ergonomic set of variables. We can iterate on this over
time, or (perhaps?) provide a sentinel value like `repl-overlays =
<DEFAULT>` to include a "suggested default" overlay like this one.
Change-Id: I8eba3934c50fbac8367111103e66c7375b8d134e
it just makes sense to have it too, rather than just the pass/fail
information we keep so far. once we turn goals into something more
promise-shaped it'll also help detangle the current data flow mess
Change-Id: I915cf04d177cad849ea7a5833215d795326f1946
it doesn't have a purpose except cache priming, which is largely
irrelevant by default (since another code path already runs this
exact query). our store implementations do not benefit that much
from this either, and the more bursty load may indeed harm them.
Change-Id: I1cc12f8c21cede42524317736d5987f1e43fc9c9
updating statistics *immediately* when any counter changes declutters
things somewhat and makes useful status reports less dependent on the
current worker main loop. using callbacks will make it easier to move
the worker loop into kj entirely, using only promises for scheduling.
Change-Id: I695dfa83111b1ec09b1a54cff268f3c1d7743ed6
there's no reason to go through the event loop in these cases. returning
ContinueImmediately here is just a very convoluted way of jumping to the
state we've just set after unwinding one frame of the stack, which never
matters in the cases changed here because there are no live RAII guards.
Change-Id: I7c00948c22e3caf35e934c1a14ffd2d40efc5547
this is not ideal, but it's better than having this stuck in the worker
loop itself. setting ex on all failing goals is not problematic because
only toplevel goals can ever be observable, all the others are ignored.
notably only derivation goals ever set `ex`, substitution goals do not.
Change-Id: I02e2164487b2955df053fef3c8e774d557aa638a
this doesn't serve a great purpose yet except to confine construction of
goals to the stack frame of Worker::run() and its child frames. we don't
need this yet (and the goal constructors remain fully visible), but in a
future change that fully removes the current worker loop we'll need some
way of knowing which goals are top-level goals without passing the goals
themselves around. once that's possible we can remove visible goals as a
concept and rely on build result futures and a scheduler built upon them
Change-Id: Ia73cdeffcfb9ba1ce9d69b702dc0bc637a4c4ce6
whether goal errors are reported via the `ex` member or just printed to
the log depends on whether the goal is a toplevel goal or a dependency.
if goals are aware of this themselves we can move error printing out of
the worker loop, and since a running worker can only be used by running
goals it's totally sufficient to keep a `Worker::running` flag for this
Change-Id: I6b5cbe6eccee1afa5fde80653c4b968554ddd16f
(but only if it is set to relaxed. no security hole here.)
Thanks to lilyball for pointing out this omission in the docs.
Change-Id: I2408a943bfe817fe660fe1c8fefef898aaf5f7e9
Saves about 16s of CPU time. Not a lot but not nothing. Feels more like
the principle of the thing.
Change-Id: I0992d4024317c20d6985a7977d5649edfb9f46bb
Apparently the fmt contraption has some extremely popular overloads, and
the boost stuff in there gets built approximately infinite times in
every compilation unit.
Change-Id: Ideba2db7d6bf8559e4d91974bab636f5ed106198
Fixes:
- Identifiers starting with _ are prohibited
- Some driveby header dependency cleaning which wound up with doing some
extra fixups.
- Fucking C style casts, man. C++ made these 1000% worse by letting you
also do memory corruption with them with references.
- Remove casts to Expr * where ExprBlackHole is an incomplete type by
introducing an explicitly-cast eBlackHoleAddr as Expr *.
- An incredibly illegal cast of the text bytes of the StorePath hash
into a size_t directly. You can't DO THAT.
Replaced with actually parsing the hash so we get 100% of the bits
being entropy, then memcpying the start of the hash. If this shows
up in a profile we should just make the hash parser faster with a
lookup table or something sensible like that.
- This horrendous bit of UB which I thankfully slapped a deprecation
warning on, built, and it didn't trigger anywhere so it was dead
code and I just deleted it. But holy crap you *cannot* do that.
inline void mkString(const Symbol & s)
{
mkString(((const std::string &) s).c_str());
}
- Some wrong lints. Lots of wrong macro lints, one wrong
suspicious-sizeof lint triggered by the template being instantiated
with only pointers, but the calculation being correct for both
pointers and not-pointers.
- Exceptions in destructors strike again. I tried to catch the
exceptions that might actually happen rather than all the exceptions
imaginable. We can let the runtime hard-kill it on other exceptions
imo.
Change-Id: I71761620846cba64d66ee7ca231b20c061e69710
It's nice for this to be a separate function and not just inline in
`absPath`.
Prepared as part of cl/1865, though I don't think I actually ended up
using it there.
Change-Id: I24d9d4a984cee0af587010baf04b3939a1c147ec
the current test relies on derivation build order being deterministic,
which will not be a reasonable expectation for all that long any more.
Change-Id: I9be44a7725185f614a9a4c724045b8b1e6962c03
this makes WorkResult copyable, and just all around easier to deal with.
in the future we'll need this to let Goal::work() return a promise for a
WorkResult (or even just a Finished) that can be awaited by other goals.
Change-Id: Ic5a1ce04c5a0f8e683bd00a2ed2b77a2e28989c1
this should be done where we're actually trying to build something, not
in the main worker loop that shouldn't have to be aware of such details
Change-Id: I07276740c0e2e5591a8ce4828a4bfc705396527e
This caused an absolute saga which I would not like anyone else to have
to experience. Let's put in a laser targeted error message that
diagnoses this exact problem.
Fixes: https://git.lix.systems/lix-project/lix/issues/484
Change-Id: I2a79f04aeb4a1b67c10115e5e39501d958836298
This took parsing time from 1421s or so to 1060s or so. The reason is
entirely nlohmann. All of the stuff below is just Obliterated because it's
built in the PCH instead:
**** Templates that took longest to instantiate:
219051 ms: nlohmann::basic_json<>::parse<const char *> (276 times, avg 793 ms)
169675 ms: nlohmann::basic_json<>::basic_json (1127 times, avg 150 ms)
129416 ms: nlohmann::detail::parser<nlohmann::basic_json<>, nlohmann::detail::i... (276 times, avg 468 ms)
98155 ms: nlohmann::detail::parser<nlohmann::basic_json<>, nlohmann::detail::i... (276 times, avg 355 ms)
81322 ms: nlohmann::basic_json<>::json_value::json_value (1405 times, avg 57 ms)
53531 ms: nlohmann::detail::json_sax_dom_callback_parser<nlohmann::basic_json<... (276 times, avg 193 ms)
clang-only. This brings the clang build time to not far from *half* of
the gcc build time.
Also, clang does not enjoy so much to miscompile coroutines. Maybe we
should just be clang-only.
Change-Id: Id3135db0094e4560830674090e32e6da2c22fcc6
I don't know why the AWS sdk disabled it by default. It would be nice
to have test coverage of the s3 store or proxies, but neither currently
exist.
Fixes: https://git.lix.systems/lix-project/lix/issues/433
Change-Id: If1e76169a3d66dbec2e926af0d0d0eccf983b97b
This avoids C++'s standard library regexes, which aren't the same
across platforms, and have many other issues, like using stack
so much that they stack overflow when processing a lot of data.
To avoid backwards and forward compatibility issues, regexes are
processed using a function converting libstdc++ regexes into Boost
regexes, escaping characters that Boost needs to have escaped, and
rejecting features that Boost has and libstdc++ doesn't.
Related context:
- Original failed attempt to use `boost::regex` in CppNix, failed due to
boost icu dependency being large (disabling ICU is no longer necessary
because linking ICU requires using a different header file,
`boost/regex/icu.hpp`): https://github.com/NixOS/nix/pull/3826
- An attempt to use PCRE, rejected due to providing less backwards
compatibility with `std::regex` than `boost::regex`:
https://github.com/NixOS/nix/pull/7336
- Second attempt to use `boost::regex`, failed due to `}` regex failing
to compile (dealt with by writing a wrapper that parses a regular
expression and escapes `}` characters):
https://github.com/NixOS/nix/pull/7762Closes#34. Closes#476.
Change-Id: Ieb0eb9e270a93e4c7eed412ba4f9f96cb00a5fa4
nixpkgs delivered us the untimely gift of a meson 1.5 upgrade, which
*does* make our lives easier by allowing us to delete wrap generation
code, but it does so at the cost of renaming all rust crates in such a
way that the wrap logic cannot tolerate the new names on the old meson
version 😭.
It also means that support burden for this is going to be atrocious
until we either give in and vendor meson 1.5 or we make a CI target for
it. Neither seems appealing, though the latter is not super absurd for
ensuring we don't break nixpkgs unstable.
This commit causes meson 1.5 to ignore the .wrap files in subprojects/
entirely (since they have the wrong names lol) and instead use
Cargo.lock, so it now hard-depends on our workspace reshuffling
improvement.
It also deletes the hack that we were using to get the sources of Cargo
deps into meson by using a feature that went unnoticed when this code
was originally written: MESON_PACKAGE_CACHE_DIR:
https://github.com/mesonbuild/meson/blob/8a202de6ec763284cbb7160b9d43d5e7e0703f19/mesonbuild/wrap/wrap.py#L490-L502
Change-Id: I7a28f12fc2812c6ed7537b60bc3025c141a05874
This is purely to let Cargo's dependency resolver do stuff for us, we do
not actually intend to build this stuff with Cargo to begin with.
Change-Id: I4c08d55595c7c27b7096375022581e1e34308a87
There have been multiple setting types for paths that are supposed to be
canonicalised, depending on whether zero or one, one, or any number of paths is
to be specified. Naturally, they behaved in slightly different ways in the
code. Simplify things by unifying them and removing special behaviour (mainly
the "multiple paths type can coerce to boolean" thing).
Change-Id: I7c1ce95e9c8e1829a866fb37d679e167811e9705
Commit 0dd1d8ca1c included an accidental revert
of 1461e6cdda (actually slightly worse), leading
to the progress bar not being stopped properly when a legacy command was
invoked with `--log-format bar` (or similar options that show a progress bar).
Move the progress bar stopping code to its proper place again to fix this
regression.
Change-Id: I676333da096d5990b717a387924bb988c9b73fab
GCC was complaining, rightfully, about mixed-sign comparisons in there.
I removed some extra sign mixing too.
Change-Id: I949a618c7405c23d4dc3fd17440ea2d7b5c22c9d
This is required to make more meson stuff easier/possible, and honestly
it *is* now Lix sources anyhow.
Change-Id: Ia6c38fabce9aa5c53768745ee38c5cf344f5c226
lix-doc is now built with Meson, with lix-doc's dependencies built as
Meson subprojects, either fetched on demand with .wrap files, or fetched
in advance by Nix with importCargoLock. It even builds statically.
Fixes#256.
Co-authored-by: Lunaphied <lunaphied@lunaphied.me>
Co-authored-by: Jade Lovelace <lix@jade.fyi>
Change-Id: I3a4731ff13278e7117e0316bc0d7169e85f5eb0c
This check is wrong and would cause the close_range() function being called even when it's not available
Change-Id: Ide65b36830e705fe772196c37349873353622761
The <() process substitution syntax doesn't work for this one testcase
in bash for FreeBSD. The exact reason for this is unknown, possibly to
do with pipe vs file vs fifo EOF behavior. The prior behavior was this
test hanging forever, with no children of the bash process.
Change-Id: I71822a4b9dea6059b34300568256c5b7848109ac
Closes#460
I managed to trigger the issue by having the following inputs (shortened):
authentik-nix.url = "github:nix-community/authentik-nix";
authentik-nix.inputs.poetry2nix.inputs.nixpkgs.follows = "nixpkgs";
When evaluating this using
nix-eval-jobs --flake .#hydraJobs
I got the following error:
error: cannot update unlocked flake input 'authentik-nix/poetry2nix' in pure mode
The issue we have here is that `authentik-nix/poetry2nix` was written
into the `overrideMap` which caused Nix to assume it's a new input and
tried to refetch it (#460) or errored out in pure mode
(nix-eval-jobs / Hydra).
The testcase unfortunately only involves checking for the output log
and makes sure that something *is* logged on the first fetch so that
the test doesn't rot when the logging changes since I didn't
manage to trigger the error above with the reproducer from #460. In
fact, I only managed to trigger the `cannot update unlocked flake input`
error in this context with `nix-eval-jobs`.
Change-Id: Ifd00091eec9a0067ed4bb3e5765a15d027328807
this can be a proper WorkResult now. childTerminated is unfortunately a
lot more stubborn and won't be made private for quite a while yet. once
we can get rid of the Worker poll loop that *should* be possible though
Change-Id: I2218df202da5cb84e852f6a37e4c20367495b617
we'll need this once we want to pass extra information out of accepting
replies, such as fd sets or possibly even async output reader promises.
Change-Id: I5e2f18cdb80b0d2faf3067703cc18bd263329b3f
don't keep fds open we're not using. currently this does not cause any
problems, but it does increase the size of our fd table needlessly and
in the future, when we have proper async processing, having builderOut
open in the daemon once the hook has been fully started is problematic
Change-Id: I6e7fb773b280b042873103638d3e04272ca1e4fc
this is useless to do on the face of it, but it'll make it easier to
convert the entire output handling to use async io and promises soon
Change-Id: I2d1eb62c4bbf8f57bd558b9599c08710a389b1a8
only DerivationGoal can set the hook to anything at all. it always sets
buildOutFD to something that is not related to fromHook in any way, and
mixing the two would have rather dire consequences for log consistency.
Change-Id: Ida86727fd1cd5e1ecd78f07f3bde330a346658a8
Apparently it was impolite to lint with 128 jobs on our CI machine with
128 threads. Let's fix it.
Change-Id: I9ca7306294c6773c6f233690ba49d45a1da6bf7a
This is incredibly haunted, but it can happen that you change libutil,
breaking the generation of the .json files, which then does not rebuild
the files. I don't expect they are slow to build, so it does not seem so
bad to just rebuild them every time instead of extracting a list of all
the possible deps.
We want to delete this nonsense anyway and replace it with generated
code.
Change-Id: Ia576d1a3bdee48fbaefbb5ac194354428d179a84
all derivation goals need a log fd of some description. let's save this
single fd in a dedicated pointer field for all subclasses so that later
we have just the one spot to change if we turn this into async promises
Change-Id: If223adf90909247363fb823d751cae34d25d0c0b
we don't need to expose information about how busy a Worker is if the
worker can instead tell its work items whether they are in a slot. in
the future we might use this to not start items waiting for a slot if
no slots are currently available, but that requires more preparation.
Change-Id: Ibe01ac536da7e6d6f80520164117c43e772f9bd9
They are like experimental features, but opt-in instead of opt-out. They
will allow us to gracefully remove language features. See #437
Change-Id: I9ca04cc48e6926750c4d622c2b229b25cc142c42
The target_machine variable is meant for the target
of cross compilers. We are not a cross compiler, so
instead reuse our host_machine based checks.
Fixes Linux→FreeBSD cross, since Meson can't figure
out `target_machine.kernel()` in that case.
Fixes: https://git.lix.systems/lix-project/lix/issues/469
Change-Id: Ia46a64c8d507c3b08987a1de1eda171ff5e50df4
bindPath/doBind is a useful function in build that is used in several
parts of LocalDerivationGoal. Moving this function makes it easier to
split LocalDerivationGoal implementation between several files.
Change-Id: Ic5a0768479c153c1aa3ed425f12604b20bbf0f42
2024-07-27 19:40:40 +00:00
1121 changed files with 22927 additions and 16953 deletions
Other files can be included with a line like `include <path>`, where `<path>` is interpreted relative to the current configuration file.
Other files can be included with a line like `include <path>`.
A missing file is an error unless `!include` is used instead.
Paths in `include`s and option values are interpreted relative to the current configuration file.
In user configuration files, paths starting with `~/` are tilde expanded (by replacing the tilde by the value of `$HOME`).
Only user configuration files (like `$XDG_CONFIG_HOME/nix/nix.conf` or the files listed in `$NIX_USER_CONF_FILES`) can use tilde paths relative to your home directory.
Configuration listed in the `$NIX_CONFIG` environment variable may not use relative paths.
A configuration setting usually overrides any previous value.
However, for settings that take a list of items, you can prefix the name of the setting by `extra-` to *append* to the previous value.
This section describes the notion of *deprecated features*, and how it fits into the big picture of the development of Lix.
# What are deprecated features?
Deprecated features are disabled by default, with the intent to eventually remove them.
Users must explicitly enable them to keep using them, by toggling the associated [deprecated feature flags](@docroot@/command-ref/conf-file.md#conf-deprecated-features).
This allows backwards compatibility and a graceful transition away from undesired features.
# Which features can be deprecated?
Undesired features should be soft-deprecated by yielding a warning when used for a significant amount of time before the can be deprecated.
Legacy obsolete feature with little to no usage may go through this process faster.
Deprecated features should have a migration path to a preferred alternative.
# Lifecycle of a deprecated feature
This description is not normative, but a feature removal may roughly happen like this:
1. Add a warning when the feature is being used.
2. Disable the feature by default, putting it behind a deprecated feature flag.
- If disabling the feature started out as an opt-in experimental feature, turn that experimental flag into a no-op or remove it entirely.
For example, `--extra-experimental-features=no-url-literals` becomes `--extra-deprecated-features=url-literals`.
3. Decide on a time frame for how long that feature will still be supported for backwards compatibility, and clearly communicate that in the error messages.
- Sometimes, automatic migration to alternatives is possible, and such should be provided if possible
- At least one NixOS release cycle should be the minimum
4. Finally remove the feature entirely, only keeping the error message for those still using it.
In an ideal world, we'd have SemVer controls over the language and its features, cleanly allowing us to make breaking changes.
See https://wiki.lix.systems/books/lix-contributors/page/language-versioning and [RFC 137](https://github.com/nixos/rfcs/pull/137) for efforts on that.
However, we do not live in such an ideal world, and currently this goal is so far away, that "just disable it with some back-compat for a couple of years" is the most realistic solution, especially for comparatively minor changes.
@@ -399,3 +399,76 @@ The following properties are supported:
Releases have a precomputed `rl-MAJOR.MINOR.md`, and no `rl-next.md`.
Set `buildUnreleasedNotes = true;` in `flake.nix` to build the release notes on the fly.
## Adding experimental or deprecated features, global settings, or builtins
Experimental and deprecated features, global settings, and builtins are generally referenced both in the code and in the documentation.
To prevent duplication or divergence, they are defined in data files, and a script generates the necessary glue.
The data file format is similar to the release notes: it consists of a YAML metadata header, followed by the documentation in Markdown format.
### Experimental or deprecated features
Experimental and deprecated features support the following metadata properties:
*`name` (required): user-facing name of the feature, to be used in `nix.conf` options and on the command line.
This should also be the stem of the file name (with extension `md`).
*`internalName` (required): identifier used to refer to the feature inside the C++ code.
Experimental feature data files should live in `lix/libutil/experimental-features`, and deprecated features in `lix/libutil/deprecated-features`.
They must be listed in the `experimental_feature_definitions` or `deprecated_feature_definitions` lists in `lix/libutil/meson.build` respectively to be considered by the build system.
### Global settings
Global settings support the following metadata properties:
*`name` (required): user-facing name of the setting, to be used as key in `nix.conf` and in the `--option` command line argument.
*`internalName` (required): identifier used to refer to the setting inside the C++ code.
*`platforms` (optional): a list specifying the platforms on which this setting is available.
If not specified, it is available on all platforms.
Valid platform names are `darwin`, `linux`.
*`type` (optional): C++ type of the setting value.
This specifies the setting object type as `Setting<T>`; if more control is required, use `settingType` instead.
*`settingType` (required if `type` is not specified): C++ type of the setting object.
*`default` (optional): default value of the setting.
`null`, truth values, integers, strings and lists are supported as long as the correct YAML type is used, `type` is not taken into account).
Other types, machine-dependent values or non-standard representations must be handled using `defaultExpr` and `defaultText` instead.
*`defaultExpr` (required if `default` is not specified): a string containing the C++ expression representing the default value.
*`defaultText` (required if `default` is not specified): a string containing the Markdown expression representing the default value in the documentation.
Literal values are conventionally surrounded by backticks, and a system-dependent value is signaled by `*machine-specific*`.
*`aliases` (optional): a list of secondary user-facing names under which the setting is available.
Defaults to empty if not specified.
*`experimentalFeature` (optional): the user-facing name of the experimental feature which needs to be enabled to change the setting.
If not specified, no experimental feature is required.
*`deprecated` (optional): whether the setting is deprecated and shown as such in the documentation for `nix.conf`.
Defaults to false if not specified.
Settings are not collected in a single place in the source tree, so an appropriate place needs to be found for the setting to live.
Look for related setting definition files under second-level subdirectories of `lix` whose name includes `settings`.
Then add the new file there, and don't forget to register it in the appropriate `meson.build` file.
### Builtin functions
The following metadata properties are supported for builtin functions:
*`name` (required): the language-facing name (as a member of the `builtins` attribute set) of the function.
*`implementation` (optional): a C++ expression specifying the implementation of the builtin.
It must be a function of signature `void(EvalState &, PosIdx, Value * *, Value &)`.
If not specified, defaults to `prim_${name}`.
*`renameInGlobalScope` (optional): whether the definiton should be "hidden" in the global scope by prefixing its name with two underscores.
If not specified, defaults to `true`.
*`args` (required): list containing the names of the arguments, as shown in the documentation.
All arguments must be listed here since the function arity is derived as the length of this list.
*`experimental_feature` (optional): the user-facing name of the experimental feature which needs to be enabled for the bultin function to be available.
If not specified, no experimental feature is required.
New builtin function definition files must be added to `lix/libexpr/builtins` and registered in the `builtin_definitions` list in `lix/libexpr/meson.build`.
### Builtin constants
The following metadata properties are supported for builtin constants:
*`name` (required): the language-facing name (as a member of the `builtins` attribute set) of the constant.
*`type` (required): the Nix language type of the constant; the C++ type is automatically derived.
*`constructorArgs` (optional): list of strings containing C++ expressions passed as arguments to the appropriate `Value` constructor.
If the value computation is more complex, `implementation` can be used instead.
*`implementation` (required if `constructorArgs` is not specified): string containing a C++ expressing computing the value of the constant.
*`impure` (optional): whether the constant is considered impure.
Impure constants are not available when pure evaluation mode is activated.
Defaults to `false` when not specified.
New builtin constant definition files must be added to `lix/libexpr/builtin-constants` and registered in the `builtin_constant_definitions` list in `lix/libexpr/meson.build`.
@@ -13,7 +13,7 @@ The unit tests are defined using the [googletest] and [rapidcheck] frameworks.
>
> ```
> …
> ├── src
> ├── lix
> │ ├── libexpr
> │ │ ├── …
> │ │ ├── value
@@ -46,10 +46,10 @@ The unit tests are defined using the [googletest] and [rapidcheck] frameworks.
> … … … … … …
> ```
The unit tests for each Lix library (`liblixexpr`, `liblixstore`, etc..) live inside a directory `src/${library_shortname}/tests` within the directory for the library (`src/${library_shortname}`).
The unit tests for each Lix library (`liblixexpr`, `liblixstore`, etc..) live inside a directory `lix/${library_shortname}/tests` within the directory for the library (`lix/${library_shortname}`).
The data is in `tests/unit/LIBNAME/data/LIBNAME`, with one subdir per library, with the same name as where the code goes.
For example, `liblixstore` code is in `src/libstore`, and its test data is in `tests/unit/libstore/data/libstore`.
For example, `liblixstore` code is in `lix/libstore`, and its test data is in `tests/unit/libstore/data/libstore`.
The path to the unit test data directory is passed to the unit test executable with the environment variable `_NIX_TEST_UNIT_DATA`.
@@ -36,7 +36,10 @@ All users of the Lix daemon may do the following to bring things into the Nix st
- Input-addressed, so they are run in the sandbox with no network access, with the following exceptions:
- The (poorly named, since it is not *just* about chroot) property `__noChroot` is set on the derivation and `sandbox` is set to `relaxed`.
- On macOS, the derivation property `__darwinAllowLocalNetworking` allows network access to localhost from input-addressed derivations regardless of the `sandbox` setting value. This property exists with such semantics because macOS has no network namespace equivalent to isolate individual processes' localhost networking.
- On macOS, the derivation property `__darwinAllowLocalNetworking` allows network access to localhost from input-addressed derivations regardless of the `sandbox` setting value.
This property exists with such semantics because macOS has no network namespace equivalent to isolate individual processes' localhost networking.
- On macOS, the derivation property `__sandboxProfile` accepts extra sandbox profile S-expressions, allowing derivations to bypass arbitrary parts of the sandbox without altogether disabling it.
This is only permitted when `sandbox` is set to `relaxed`.
- Output-addressed, so they are run with network access but their result must match an expected hash.
Trusted users may set any setting, including `sandbox = false`, so the sandbox state can be different at runtime from what is described in `nix.conf` for builds invoked with such settings.
Following the initial mitigation of **CVE-2025-52992** in `cl/3420`, we
received reports of **unexpected deletion of in-use store paths**.
Upon investigation, we found that the patch did **not correctly cancel all
automatic deleters**, resulting in potentially critical path loss during normal
operation.
Given the severity and time-sensitive nature of the situation ([see incident
report](https://lix.systems/blog/2025-06-27-lix-critical-bug/)), we evaluated
possible options to repair the behavior safely. However, we concluded that a
rushed fix would either:
* **Overdelete**, i.e. breaking running systems, or,
* **Underdelete**, effectively **reopening CVE-2025-52992** while leaving
orphaned paths behind.
As **CVE-2025-52992 has no known exploit vector**, and correctness is critical
in the Lix project, we have **fully reverted the previous mitigations**.
Moving forward, the Lix team will rework this code path in a **long-term,
correctness-first fix** on the main branch. We will explore backporting it to
stable channels once its safety is assured.
We are deeply sorry for the stability incident and the Lix team remain
available for assisting you in recovering your systems.
Many thanks to [Raito Bezarius](https://git.lix.systems/raito) and [eldritch horrors](https://git.lix.systems/pennae) for this.
- Fallback to safe temp dir when build-dir is unwritable [fj#876](https://git.lix.systems/lix-project/lix/issues/876) [cl/3502](https://gerrit.lix.systems/c/lix/+/3502)
Non-daemon builds started failing with a permission error after introducing the `build-dir` option:
1. These builds are not run via the daemon, which owns `/nix/var/nix/builds`.
2. The user lacks permissions for that path.
We considered making `build-dir` a store-level option and defaulting it to `<chroot-root>/nix/var/nix/builds` for chroot stores, but opted instead for a fallback: if the default fails, Nix now creates a safe build directory under `/tmp`.
To avoid CVE-2025-52991, the fallback uses an extra path component between `/tmp` and the build dir.
**Note**: this fallback clutters `/tmp` with build directories that are not cleaned up. To prevent this, explicitly set `build-dir` to a path managed by Lix, even for local workloads.
Many thanks to [Raito Bezarius](https://git.lix.systems/raito) and [eldritch horrors](https://git.lix.systems/pennae) for this.
# Lix 2.91.2 (2025-06-23)
## Breaking Changes
- Fixed output derivations can be run using `pasta` network isolation [fj#285](https://git.lix.systems/lix-project/lix/issues/285) [cl/3418](https://gerrit.lix.systems/c/lix/+/3418)
Fixed output derivations traditionally run in the host network namespace.
On Linux this allows such derivations to communicate with other sandboxes
or the host using the abstract Unix domains socket namespace; this hasn't
been unproblematic in the past and has been used in two distinct exploits
to break out of the sandbox. For this reason fixed output derivations can
now run in a network namespace (provided by [`pasta`]), restricted to TCP
and UDP communication with the rest of the world. When enabled this could
be a breaking change and we classify it as such, even though we don't yet
enable or require such isolation by default. We may enforce this in later
releases of Lix once we have sufficient confidence that breakage is rare.
[`pasta`]: https://passt.top/
Many thanks to [eldritch horrors](https://git.lix.systems/pennae) and [puck](https://git.lix.systems/puck) for this.
## Fixes
- Always clean up scratch paths after derivations failed to build [cl/3420](https://gerrit.lix.systems/c/lix/+/3420)
Previously, scratch paths created during builds were not always cleaned up if
the derivation failed, potentially leaving behind unnecessary temporary files
or directories in the Nix store.
This fix ensures that such paths are consistently removed after a failed build,
improving Nix store hygiene, hardening Lix against mis-reuse of failed builds
scratch paths.
Many thanks to [Raito Bezarius](https://git.lix.systems/raito) and [eldritch horrors](https://git.lix.systems/pennae) for this.
- `build-dir` no longer defaults to `temp-dir` [cl/3419](https://gerrit.lix.systems/c/lix/+/3419)
The directory in which temporary build directories are created no longer defaults
to the value of the `temp-dir` setting to avoid builders making their directories
world-accessible. This behavior has been used to escape the build sandbox and can
cause build impurities even when not used maliciously. We now default to `builds`
in `NIX_STATE_DIR` (which is `/nix/var/nix/builds` in the default configuration).
Many thanks to [eldritch horrors](https://git.lix.systems/pennae) for this.
# Lix 2.91.1 (2024-10-18)
## Fixes
- `<nix/fetchurl.nix>` now uses TLS verification [#11585](https://github.com/NixOS/nix/pull/11585)
Previously `<nix/fetchurl.nix>` did not do TLS verification. This was because the Nix sandbox in the past did not have access to TLS certificates, and Nix checks the hash of the fetched file anyway. However, this can expose authentication data from `netrc` and URLs to man-in-the-middle attackers. In addition, Nix now in some cases (such as when using impure derivations) does *not* check the hash. Therefore we have now enabled TLS verification. This means that downloads by `<nix/fetchurl.nix>` will now fail if you're fetching from a HTTPS server that does not have a valid certificate.
`<nix/fetchurl.nix>` is also known as the builtin derivation builder `builtin:fetchurl`. It's not to be confused with the evaluation-time function `builtins.fetchurl`, which was not affected by this issue.
Many thanks to [Eelco Dolstra](https://github.com/edolstra) for this.
Following the initial mitigation of **CVE-2025-52992** in `cl/3432`, we
received reports of **unexpected deletion of in-use store paths**.
Upon investigation, we found that the patch did **not correctly cancel all
automatic deleters**, resulting in potentially critical path loss during normal
operation.
Given the severity and time-sensitive nature of the situation ([see incident
report](https://lix.systems/blog/2025-06-27-lix-critical-bug/)), we evaluated
possible options to repair the behavior safely. However, we concluded that a
rushed fix would either
* **Overdelete**, i.e. breaking running systems, or,
* **Underdelete**, effectively **reopening CVE-2025-52992** while leaving
orphaned paths behind.
As **CVE-2025-52992 has no known exploit vector**, and correctness is critical
in the Lix project, we have **fully reverted the previous mitigations**.
The affected patches (`cl/3432`) have been rolled back for the time being.
Moving forward, the Lix team will rework this code path in a **long-term,
correctness-first fix** on the main branch. We will explore backporting it to
stable channels once its safety is assured.
We are deeply sorry for the stability incident and the Lix team remain
available for assisting you in recovering your systems.
Many thanks to [Raito Bezarius](https://git.lix.systems/raito) and [eldritch horrors](https://git.lix.systems/pennae) for this.
- Fallback to safe temp dir when build-dir is unwritable [fj#876](https://git.lix.systems/lix-project/lix/issues/876) [cl/3503](https://gerrit.lix.systems/c/lix/+/3503)
Non-daemon builds started failing with a permission error after introducing the `build-dir` option:
1. These builds are not run via the daemon, which owns `/nix/var/nix/builds`.
2. The user lacks permissions for that path.
We considered making `build-dir` a store-level option and defaulting it to `<chroot-root>/nix/var/nix/builds` for chroot stores, but opted instead for a fallback: if the default fails, Nix now creates a safe build directory under `/tmp`.
To avoid CVE-2025-52991, the fallback uses an extra path component between `/tmp` and the build dir.
**Note**: this fallback clutters `/tmp` with build directories that are not cleaned up. To prevent this, explicitly set `build-dir` to a path managed by Lix, even for local workloads.
Many thanks to [Raito Bezarius](https://git.lix.systems/raito) and [eldritch horrors](https://git.lix.systems/pennae) for this.
# Lix 2.92.2 (2025-06-23)
## Breaking Changes
- Fixed output derivations can be run using `pasta` network isolation [fj#285](https://git.lix.systems/lix-project/lix/issues/285) [cl/3430](https://gerrit.lix.systems/c/lix/+/3430)
Fixed output derivations traditionally run in the host network namespace.
On Linux this allows such derivations to communicate with other sandboxes
or the host using the abstract Unix domains socket namespace; this hasn't
been unproblematic in the past and has been used in two distinct exploits
to break out of the sandbox. For this reason fixed output derivations can
now run in a network namespace (provided by [`pasta`]), restricted to TCP
and UDP communication with the rest of the world. When enabled this could
be a breaking change and we classify it as such, even though we don't yet
enable or require such isolation by default. We may enforce this in later
releases of Lix once we have sufficient confidence that breakage is rare.
[`pasta`]: https://passt.top/
Many thanks to [eldritch horrors](https://git.lix.systems/pennae) and [puck](https://git.lix.systems/puck) for this.
## Fixes
- Always clean up scratch paths after derivations failed to build [cl/3432](https://gerrit.lix.systems/c/lix/+/3432)
Previously, scratch paths created during builds were not always cleaned up if
the derivation failed, potentially leaving behind unnecessary temporary files
or directories in the Nix store.
This fix ensures that such paths are consistently removed after a failed build,
improving Nix store hygiene, hardening Lix against mis-reuse of failed builds
scratch paths.
Many thanks to [Raito Bezarius](https://git.lix.systems/raito) and [eldritch horrors](https://git.lix.systems/pennae) for this.
- `build-dir` no longer defaults to `temp-dir` [cl/3431](https://gerrit.lix.systems/c/lix/+/3431)
The directory in which temporary build directories are created no longer defaults
to the value of the `temp-dir` setting to avoid builders making their directories
world-accessible. This behavior has been used to escape the build sandbox and can
cause build impurities even when not used maliciously. We now default to `builds`
in `NIX_STATE_DIR` (which is `/nix/var/nix/builds` in the default configuration).
Many thanks to [eldritch horrors](https://git.lix.systems/pennae) for this.
- Forbid impure path accesses in pure evaluation mode again [cl/2708](https://gerrit.lix.systems/c/lix/+/2708)
Lix 2.92.0 mistakenly started allowing the access to ancestors of allowed paths in pure evaluation mode.
This made it possible to bypass the purity restrictions, for example by copying arbitrary files to the store:
```nix
builtins.path {
path = "/";
filter = …;
}
```
Restore the previous behaviour of prohibiting such impure accesses.
Many thanks to [alois31](https://git.lix.systems/alois31) for this.
- Parsing failures in flake.lock no longer crash Lix [fj#559](https://git.lix.systems/lix-project/lix/issues/559) [cl/2401](https://gerrit.lix.systems/c/lix/+/2401)
Failure to parse `flake.lock` no longer hard-crashes Lix and instead produces a nice error message.
```
error:
… while updating the lock file of flake 'git+file:///Users/jade/lix/lix2'
… while parsing the lock file at /nix/store/mm5dqh8a729yazzj82cjffxl97n5c62s-source//flake.lock
error: [json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - invalid literal;
last read: '#'
```
Many thanks to [gilice](https://git.lix.systems/gilice) for this.
When in debug mode (e.g. from using the `--debugger` flag), enabling [`ignore-try`](@docroot@/command-ref/conf-file.md#conf-ignore-try) once again properly disables debug REPLs within [`builtins.tryEval`](@docroot@/language/builtins.md#builtins-tryEval) calls. Previously, a debug REPL would be started as if `ignore-try` was disabled, but that REPL wouldn't actually be in debug mode, and upon exiting the REPL the evaluating process would segfault.
Many thanks to [Dusk Banks](https://git.lix.systems/bb010g) for this.
# Lix 2.92.0 (2025-01-18)
## Breaking Changes
- Deprecated language features [fj#437](https://git.lix.systems/lix-project/lix/issues/437) [#861](https://github.com/NixOS/nix/issues/861) [cl/1785](https://gerrit.lix.systems/c/lix/+/1785) [cl/1736](https://gerrit.lix.systems/c/lix/+/1736) [cl/1735](https://gerrit.lix.systems/c/lix/+/1735) [cl/1744](https://gerrit.lix.systems/c/lix/+/1744) [cl/2206](https://gerrit.lix.systems/c/lix/+/2206)
A system for deprecation (and then the planned removal) of undesired language features has been put into place.
It is controlled via feature flags much like experimental features, except that the deprecations are enabled default,
and can be disabled via the flags for backwards compatibility (opt-out with `--extra-deprecated-features` or the Nix configuration file).
- `url-literals`: **URL literals** have long been obsolete and discouraged of use, and now they are officially deprecated.
This means that all URLs must be properly put within quotes like all other strings.
- `rec-set-overrides`: **__overrides** is an old arcane syntax which has not been in use for more than a decade.
It is soft-deprecated with a warning only, with the plan to turn that into an error in a future release.
- `ancient-let`: **The old `let` syntax** (`let { body = …; … }`) is soft-deprecated with a warning as well. Use the regular `let … in` instead.
- `shadow-internal-symbols`: Arithmetic expressions like `5 - 3` internally expand to `__sub 5 3`, where `__sub` maps to a subtraction builtin. Shadowing such a symbols would affect the evaluation of such operations, but in a very inconsistent way, and is therefore deprecated now. **Affected symbols are:** `__sub`, `__mul`, `__div` and `__lessThan`. Note that these symbols may still be used as variable names as long as they do not shadow internal operations, so e.g. `let __sub = x: y: x + y; in __sub 3 5` remains valid code.
- **Call to action:** If you have any use cases or workflows that depend on being able to override the `__nixPath` and `__findFile` symbols, please reach out to us. We want to eventually deprecate overriding these as well, and need input on how to design a better alternative.
Many thanks to [piegames](https://git.lix.systems/piegames) and [eldritch horrors](https://git.lix.systems/pennae) for this.
- transfers no longer allow arbitrary url schemas [cl/2106](https://gerrit.lix.systems/c/lix/+/2106)
Lix no longer allows transfers using arbitrary url schemas. Only `http://`, `https://`, `ftp://`, `ftps://`, and `file://` urls are supported going forward. This affects `builtins.fetchurl`, `<nix/fetchurl.nix>`, transfers to and from binary caches, and all other uses of the internal file transfer code. Flake inputs using multi-protocol schemas (e.g. `git+ssh`) are not affected as those use external utilities to transfer data.
The `s3://` scheme is not affected at all by this change and continues to work if S3 support is built into Lix.
Many thanks to [eldritch horrors](https://git.lix.systems/pennae) for this.
- Removing the `.` default argument passed to the `nix fmt` formatter [#11438](https://github.com/NixOS/nix/pull/11438) [cl/1902](https://gerrit.lix.systems/c/lix/+/1902)
The underlying formatter no longer receives the ". " default argument when `nix fmt` is called with no arguments.
This change was necessary as the formatter wasn't able to distinguish between
a user wanting to format the current folder with `nix fmt .` or the generic
`nix fmt`.
The default behaviour is now the responsibility of the formatter itself, and
allows tools such as treefmt to format the whole tree instead of only the
current directory and below.
This may cause issues with some formatters: nixfmt, nixpkgs-fmt and alejandra currently format stdin when no arguments are passed.
Here is a small wrapper example that will restore the previous behaviour for such a formatter:
This should avoid some unnecessary trips to the repl or to the debugger by giving some information about the value being selected on that was unexpected.
Many thanks to [jade](https://git.lix.systems/jade) for this.
- Small error message improvements [cl/2185](https://gerrit.lix.systems/c/lix/+/2185) [cl/2187](https://gerrit.lix.systems/c/lix/+/2187)
When an attribute selection fails, the error message now correctly points to the attribute in the chain that failed instead of at the beginning of the entire chain.
```diff
error: attribute 'x' missing
- at /pwd/lang/eval-fail-remove.nix:4:3:
+ at /pwd/lang/eval-fail-remove.nix:4:29:
3| in
4| (removeAttrs attrs ["x"]).x
- | ^
+ | ^
5|
```
Failed asserts don't print the failed assertion expression anymore in the error message. That code was buggy and the information was redundant anyways, given that the error position already more accurately shows what exactly failed.
Many thanks to [piegames](https://git.lix.systems/piegames) for this.
- Improvements to interactive flake config [cl/2066](https://gerrit.lix.systems/c/lix/+/2066)
If `accept-flake-config` is set to `ask` and a `flake.nix` defines `nixConfig`,
Lix will ask on the CLI which of these settings should be used for the command.
Now, it's possible to answer with `N` (as opposed to `n` to only reject the setting
that is asked for) to reject _all untrusted_ entries from the flake's `nixConf`
section.
Many thanks to [ma27](https://git.lix.systems/ma27) for this.
- `nix --version` now shows details about the installation by default [fj#620](https://git.lix.systems/lix-project/lix/issues/620) [cl/2365](https://gerrit.lix.systems/c/lix/+/2365)
This happened with `nix-env --version` by default, but due to [oddities around the nix3 CLI's verbosity](https://gerrit.lix.systems/c/lix/+/1370), it used to be `nix --verbose --version`.
No longer:
```
$ nix --version
nix (Lix, like Nix) 2.92.0-dev-pre20250117-0d14c2b
System type: x86_64-linux
Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux
Features: gc, signed-caches
System configuration file: /etc/nix/nix.conf
User configuration files: /home/jade/.config/nix/nix.conf:/etc/xdg/nix/nix.conf
Store directory: /nix/store
State directory: /nix/var/nix
Data directory: /nix/store/rliimcnqkplrqdgm4z6yqclpr6c32wh6-lix-2.92.0-dev-pre20250117-0d14c2b/share
```
Many thanks to [just1602](https://git.lix.systems/just1602) for this.
The REPL (`nix repl`) now includes quotes as part of attribute names while completing with `<TAB>`,
if necessary. For example, attribute names like `"hello@example.com"` or `"hello world"` would
be suggested without quotes, resulting in invalid syntax.
Many thanks to [ian-h-chamberlain](https://git.lix.systems/ian-h-chamberlain) for this.
- Reproducibility check builds now report all differing outputs [cl/2069](https://gerrit.lix.systems/c/lix/+/2069)
`nix-build --check` allows rerunning the build of an already-built derivation to check that it produces the same output again.
If a multiple-output derivation with impure behaviour is built with `--check`, only the first output would be shown in the resulting error message (and kept for comparison):
```
error: derivation '/nix/store/4spy3nz1661zm15gkybsy1h5f36aliwx-python3.11-test-1.0.0.drv' may not be deterministic: output '/nix/store/ccqcp01zg18wp9iadzmzimqzdi3ll08d-python3.11-test
-1.0.0-dist' differs from '/nix/store/ccqcp01zg18wp9iadzmzimqzdi3ll08d-python3.11-test-1.0.0-dist.check'
```
Now, all differing outputs are kept and reported:
```
error: derivation '4spy3nz1661zm15gkybsy1h5f36aliwx-python3.11-test-1.0.0.drv' may not be deterministic: outputs differ
output differs: output '/nix/store/ccqcp01zg18wp9iadzmzimqzdi3ll08d-python3.11-test-1.0.0-dist' differs from '/nix/store/ccqcp01zg18wp9iadzmzimqzdi3ll08d-python3.11-test-1.0.0-dist.check'
output differs: output '/nix/store/yl59v08356i841c560alb0zmk7q16klb-python3.11-test-1.0.0' differs from '/nix/store/yl59v08356i841c560alb0zmk7q16klb-python3.11-test-1.0.0.check'
```
Many thanks to [lheckemann](https://git.lix.systems/lheckemann) for this.
- Some Lix crashes now produce reporting instructions and a stack trace, then abort [cl/1854](https://gerrit.lix.systems/c/lix/+/1854)
Lix, being a C++ program, can crash in a few kinds of ways.
It can obviously do a memory access violation, which will generate a core dump and thus be relatively debuggable.
But, worse, it could throw an unhandled exception, and, in the past, we would just show the message but not where it comes from, in spite of this always being a bug, since we expect all such errors to be translated to a Lix specific error.
Now the latter kind of bug should print reporting instructions, a rudimentary stack trace and (depending on system configuration) generate a core dump.
Sample output:
```
Lix crashed. This is a bug. We would appreciate if you report it along with what caused it at https://git.lix.systems/lix-project/lix/issues with the following information included:
Exception: std::runtime_error: test exception
Stack trace:
0# nix::printStackTrace() in /home/jade/lix/lix3/build/lix/nix/../libutil/liblixutil.so
1# 0x000073C9862331F2 in /home/jade/lix/lix3/build/lix/nix/../libmain/liblixmain.so
2# 0x000073C985F2E21A in /nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libstdc++.so.6
3# 0x000073C985F2E285 in /nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libstdc++.so.6
4# nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>) in /home/jade/lix/lix3/build/lix/nix/../libmain/liblixmain.so
...
```
Many thanks to [jade](https://git.lix.systems/jade) for this.
- Add a `temp-dir` setting to set the temporary directory location [#7731](https://github.com/NixOS/nix/issues/7731) [#8995](https://github.com/NixOS/nix/issues/8995) [fj#112](https://git.lix.systems/lix-project/lix/issues/112) [fj#253](https://git.lix.systems/lix-project/lix/issues/253) [cl/2103](https://gerrit.lix.systems/c/lix/+/2103)
[`temp-dir`](@docroot@/command-ref/conf-file.md#conf-temp-dir) can now be set in the Nix
configuration to change the temporary directory. This can be used to relocate all temporary files
to another filesystem without affecting the `TMPDIR` env var inherited by interactive
`nix-shell`/`nix shell` shells or `nix run` commands.
Also on macOS, the `TMPDIR` env var is no longer unset for interactive shells when pointing
to a per-session `/var/folders/` directory.
Many thanks to [lilyball](https://git.lix.systems/lilyball) for this.
## Fixes
- Build failures caused by `allowSubstitutes = false` while being the wrong system now produce a decent error [fj#484](https://git.lix.systems/lix-project/lix/issues/484) [cl/1841](https://gerrit.lix.systems/c/lix/+/1841)
Nix allows derivations to set `allowSubstitutes = false` in order to force them to be built locally without querying substituters for them.
This is useful for derivations that are very fast to build (especially if they produce large output).
However, this can shoot you in the foot if the derivation *has* to be substituted such as if the derivation is for another architecture, which is what `--always-allow-substitutes` is for.
Perhaps such derivations that are known to be impossible to build locally should ignore `allowSubstitutes` (irrespective of remote builders) in the future, but this at least reports the failure and solution directly.
```
$ nix build -f fail.nix
error: a 'unicornsandrainbows-linux' with features {} is required to build '/nix/store/...-meow.drv', but I am a 'x86_64-linux' with features {...}
Hint: the failing derivation has allowSubstitutes set to false, forcing it to be built rather than substituted.
Passing --always-allow-substitutes to force substitution may resolve this failure if the path is available in a substituter.
```
Many thanks to [jade](https://git.lix.systems/jade) for this.
- `Alt+Left` and `Alt+Right` go back/forwards by words in `nix repl` [fj#501](https://git.lix.systems/lix-project/lix/issues/501) [cl/1883](https://gerrit.lix.systems/c/lix/+/1883)
`nix repl` now recognizes `Alt+Left` and `Alt+Right` for navigating by words
when entering input in `nix repl` on more terminals/platforms.
Many thanks to [wiggles](https://git.lix.systems/rbt) for this.
- Ctrl-C stops Nix commands much more reliably and responsively [#7245](https://github.com/NixOS/nix/issues/7245) [fj#393](https://git.lix.systems/lix-project/lix/issues/393) [#11618](https://github.com/NixOS/nix/pull/11618) [cl/2016](https://gerrit.lix.systems/c/lix/+/2016)
CTRL-C will now stop Nix commands much more reliably and responsively. While
there are still some cases where a Nix command can be slow or unresponsive
following a `SIGINT` (please report these as issues!), the vast majority of
signals will now cause the Nix command to quit quickly and consistently.
Many thanks to [Robert Hensing](https://github.com/roberth) and [wiggles](https://git.lix.systems/rbt) for this.
- restore backwards-compatibility of `builtins.fetchGit` with Nix 2.3 [#5291](https://github.com/NixOS/nix/issues/5291) [#5128](https://github.com/NixOS/nix/issues/5128)
Compatibility with `builtins.fetchGit` from Nix 2.3 has been restored as follows:
* Until now, each `ref` was prefixed with `refs/heads` unless it starts with `refs/` itself.
Now, this is not done if the `ref` looks like a commit hash.
* Specifying `builtins.fetchGit { ref = "a-tag"; /* … */ }` was broken because `refs/heads` was appended.
Now, the fetcher doesn't turn a ref into `refs/heads/ref`, but into `refs/*/ref`. That way,
the value in `ref` can be either a tag or a branch.
* The ref resolution happens the same way as in git:
* If `refs/ref` exists, it's used.
* If a tag `refs/tags/ref` exists, it's used.
* If a branch `refs/heads/ref` exists, it's used.
Many thanks to [ma27](https://git.lix.systems/ma27) for this.
- Flakes/restrict-eval no longer allow reading contents of impure paths
Flakes and `--restrict-eval` now correctly restrict access to paths as intended.
In prior versions since at least 2.18, `nix eval --raw .#lol` for the following flake didn't throw an error and acted as if `--impure` was passed.
Thanks to the person who reported this for telling us about it.
This was handled as a low-severity security bug, but is not a violation of the [documented security model](../installation/multi-user.md) as untrusted Nix code should be assumed to have the privileges of the user running the evaluator.
To report a security bug, email a report to `security at lix dot systems`.
```nix
{
inputs = {};
outputs = {...}: {
lol = builtins.readFile "${/etc/passwd}";
};
}
```
Many thanks to [eldritch horrors](https://git.lix.systems/pennae) for this.
- HTTP proxy environment variables are now respected for S3 binary cache stores [fj#433](https://git.lix.systems/lix-project/lix/issues/433) [cl/1788](https://gerrit.lix.systems/c/lix/+/1788)
Due to "legacy reasons" (according to the AWS C++ SDK docs), the AWS SDK ignores system proxy configuration by default.
We turned it back on.
Many thanks to [jade](https://git.lix.systems/jade) for this.
- Fix potential store corruption with auto-optimise-store [#7273](https://github.com/NixOS/nix/issues/7273) [cl/2100](https://gerrit.lix.systems/c/lix/+/2100)
Optimising store paths (and other operations involving temporary files) no longer use `random(3)`
to generate filenames. On darwin systems this was observed to potentially cause store corruption
when using [`auto-optimise-store`](@docroot@/command-ref/conf-file.md#conf-auto-optimise-store),
though this corruption was possible on any system whose `random(3)` does not have locking around
the global state.
Many thanks to [lilyball](https://git.lix.systems/lilyball) for this.
- Change `nix-build -o ""` to behave like `--no-out-link` [cl/2103](https://gerrit.lix.systems/c/lix/+/2103)
[`nix-build`](@docroot@/command-ref/nix-build.md) now treats <code>[--out-link](@docroot@/command-ref/nix-build.md#opt-out-link) ''</code>
the same as [`--no-out-link`](@docroot@/command-ref/nix-build.md#opt-no-out-link). This matches
[`nix build`](@docroot@/command-ref/new-cli/nix3-build.md) behavior. Previously when building the default output it
would have resulted in throwing an error saying the current working directory already exists, and when building any
other output it would have resulted in a symlink starting with a hyphen such as `-doc`, which is a footgun for
terminal commands.
Many thanks to [lilyball](https://git.lix.systems/lilyball) for this.
[`ssl-cert-file`](@docroot@/command-ref/conf-file.md#conf-ssl-cert-file) now checks its default
value for a broken symlink before using it. This fixes a problem on macOS where uninstalling
nix-darwin may leave behind a broken symlink at `/etc/ssl/certs/ca-certificates.crt` that was
stopping Lix from using the cert at `/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt`.
Many thanks to [lilyball](https://git.lix.systems/lilyball) for this.
- `<nix/fetchurl.nix>` now uses TLS verification [#11585](https://github.com/NixOS/nix/pull/11585)
Previously `<nix/fetchurl.nix>` did not do TLS verification. This was because the Nix sandbox in the past did not have access to TLS certificates, and Nix checks the hash of the fetched file anyway. However, this can expose authentication data from `netrc` and URLs to man-in-the-middle attackers. In addition, Nix now in some cases (such as when using impure derivations) does *not* check the hash. Therefore we have now enabled TLS verification. This means that downloads by `<nix/fetchurl.nix>` will now fail if you're fetching from a HTTPS server that does not have a valid certificate.
`<nix/fetchurl.nix>` is also known as the builtin derivation builder `builtin:fetchurl`. It's not to be confused with the evaluation-time function `builtins.fetchurl`, which was not affected by this issue.
Many thanks to [Eelco Dolstra](https://github.com/edolstra) for this.
## Packaging
- readline support removed [cl/1885](https://gerrit.lix.systems/c/lix/+/1885)
Support for building Lix with [`readline`][readline] instead of
[`editline`][editline] has been removed. `readline` support hasn't worked for a
long time (attempting to use it would lead to build errors) and would make Lix
subject to the GPL if it did work. In the future, we're hoping to replace
`editline` with [`rustyline`][rustyline] for improved ergonomics in the `nix
Many thanks to [wiggles](https://git.lix.systems/rbt) for this.
## Development
- Includes are now qualified with library name everywhere [cl/2178](https://gerrit.lix.systems/c/lix/+/2178) [cl/2362](https://gerrit.lix.systems/c/lix/+/2362)
The Lix includes have all been rearranged to be of the form `"lix/libexpr/foo.hh"` instead of `"foo.hh"`.
This was already supported externally for a migration period, but it is now being applied to all the internal usages within Lix itself.
The goal of this change is to both clarify where a file is from and to avoid polluting global include paths with things like `config.h` that might conflict with other projects.
Lix 2.92 removes support for the old `"foo.hh"` include form either internally or externally (that is, via pkg-config for things linking to Lix).
For other details, see the release notes of Lix 2.90.0, under "Rename all the libraries" in Breaking Changes.
To fix an external project with sources in `src` which has a separate build directory (such that headers are in `../src` relative to where the compiler is running), use a checkout of Lix 2.91 to run the following:
Many thanks to [jade](https://git.lix.systems/jade) for this.
- The beginnings of a new pytest-based functional test suite [cl/2036](https://gerrit.lix.systems/c/lix/+/2036) [cl/2037](https://gerrit.lix.systems/c/lix/+/2037)
The existing integration/functional test suite is based on a large volume of shell scripts.
This often makes it somewhat challenging to debug at the best of times.
The goal of the pytest test suite is to make tests have more obvious dependencies on files and to make tests more concise and easier to write, as well as making new testing methods like snapshot testing easy.
Many thanks to [jade](https://git.lix.systems/jade) for this.
- Dependency on monolithic coreutils removed [cl/2108](https://gerrit.lix.systems/c/lix/+/2108)
Previously, the build erroneously depended on a `coreutils` binary, which requires `coreutils` to be built with a specific configuration. This was only used in one test and was not required to be a single binary. This dependency is removed now.
Many thanks to [Vigress](https://git.lix.systems/vigress8) for this.
- All Lix threads are named [cl/2210](https://gerrit.lix.systems/c/lix/+/2210)
Lix now sets thread names on all of its secondary threads, which will make debugger usage slightly nicer and easier.
```
(gdb) info thr
Id Target Id Frame
* 1 LWP 3719283 "nix-daemon" 0x00007e558587da0f in accept ()
from target:/nix/store/c10zhkbp6jmyh0xc5kd123ga8yy2p4hk-glibc-2.39-52/lib/libc.so.6
2 LWP 3719284 "signal handler" 0x00007e55857b2bea in sigtimedwait ()
from target:/nix/store/c10zhkbp6jmyh0xc5kd123ga8yy2p4hk-glibc-2.39-52/lib/libc.so.6
```
Many thanks to [jade](https://git.lix.systems/jade) for this.
- Set `X-GitHub-Api-Version` header [fj#255](https://git.lix.systems/lix-project/lix/issues/255) [cl/1925](https://gerrit.lix.systems/c/lix/+/1925)
Sets the `X-GitHub-Api-Version` header to `2022-11-28` for calls to the
'''+(f'''This function is only available if the [{builtin.experimental_feature}](@docroot@/contributing/experimental-features.md#xp-feature-{builtin.experimental_feature}) experimental feature is enabled.
f' - [`{feature.name}`](@docroot@/contributing/{"deprecated"ifargs.deprecatedelse"experimental"}-features.md#{"dp"ifargs.deprecatedelse"xp"}-feature-{feature.name})\n')
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Raito Bezarius