guiwf/lix
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

merge into: guiwf/lix:sb/jade/coverage
Branches Tags
guiwf/lix:main guiwf/lix:adaptive-routing guiwf/lix:release-2.95 guiwf/lix:ci-config guiwf/lix:release-2.93 guiwf/lix:release-2.94 guiwf/lix:sb/jade/coverage guiwf/lix:sb/raitobezarius/cgroups-observability guiwf/lix:sb/jade/wip-tars guiwf/lix:sb/raito/flakes-out-of-core guiwf/lix:sb/raito/eval-explorations guiwf/lix:sb/pennae/ci-config guiwf/lix:release-2.91 guiwf/lix:release-2.92 guiwf/lix:sb/raito/darwin-kqueue-disablement guiwf/lix:sb/raito/hydra-merge guiwf/lix:sb/raito/phantom-referrers-gc guiwf/lix:repl-overlay-example guiwf/lix:reuse-compliance guiwf/lix:release-2.90 guiwf/lix:sb/pennae/io-rewrite guiwf/lix:git-series/activity-cleanup guiwf/lix:sb/tom-hubrecht/primops.cc-v2 guiwf/lix:sb/bacchanalia/cache-strlen%topic=substr-perf guiwf/lix:sb/qyriad/beta-1 guiwf/lix:sb/qyriad/2.90-beta.0 guiwf/lix:sb/rbt/test-branch guiwf/lix:sb/rbt/justfile guiwf/lix:sb/rbt/repl-overlays guiwf/lix:sb/qyriad/maint-meson guiwf/lix:sb/qyriad/maint/meson guiwf/lix:sb/puck/meson-c-api guiwf/lix:sb/jade/burn-it-all guiwf/lix:dependabot/github_actions/zeebe-io/backport-action-2.4.1 guiwf/lix:dependabot/github_actions/cachix/install-nix-action-25 guiwf/lix:dependabot/github_actions/actions/labeler-5 guiwf/lix:dependabot/github_actions/cachix/cachix-action-14
guiwf/lix:2.95.3 guiwf/lix:2.93.4 guiwf/lix:2.94.2 guiwf/lix:2.95.2 guiwf/lix:2.95.1 guiwf/lix:2.94.1 guiwf/lix:2.95.0 guiwf/lix:2.94.0 guiwf/lix:2.93.3 guiwf/lix:2.92.3 guiwf/lix:2.93.2 guiwf/lix:2.91.3 guiwf/lix:2.92.2 guiwf/lix:2.91.2 guiwf/lix:2.93.1 guiwf/lix:2.93.0 guiwf/lix:2.92.0 guiwf/lix:2.91.1 guiwf/lix:2.91.0 guiwf/lix:2.90.0 guiwf/lix:2.90.0-rc1 guiwf/lix:2.90-beta.1 guiwf/lix:2.90-beta.0 guiwf/lix:1.11.1 guiwf/lix:1.11 guiwf/lix:1.10 guiwf/lix:1.9 guiwf/lix:1.8 guiwf/lix:1.7 guiwf/lix:1.6.1 guiwf/lix:1.6 guiwf/lix:1.5.3 guiwf/lix:1.5.2 guiwf/lix:1.5.1 guiwf/lix:1.5 guiwf/lix:1.4 guiwf/lix:1.3 guiwf/lix:1.2 guiwf/lix:1.1 guiwf/lix:1.0
...
pull from: guiwf/lix:main
Branches Tags
guiwf/lix:adaptive-routing guiwf/lix:main guiwf/lix:release-2.95 guiwf/lix:ci-config guiwf/lix:release-2.93 guiwf/lix:release-2.94 guiwf/lix:sb/jade/coverage guiwf/lix:sb/raitobezarius/cgroups-observability guiwf/lix:sb/jade/wip-tars guiwf/lix:sb/raito/flakes-out-of-core guiwf/lix:sb/raito/eval-explorations guiwf/lix:sb/pennae/ci-config guiwf/lix:release-2.91 guiwf/lix:release-2.92 guiwf/lix:sb/raito/darwin-kqueue-disablement guiwf/lix:sb/raito/hydra-merge guiwf/lix:sb/raito/phantom-referrers-gc guiwf/lix:repl-overlay-example guiwf/lix:reuse-compliance guiwf/lix:release-2.90 guiwf/lix:sb/pennae/io-rewrite guiwf/lix:git-series/activity-cleanup guiwf/lix:sb/tom-hubrecht/primops.cc-v2 guiwf/lix:sb/bacchanalia/cache-strlen%topic=substr-perf guiwf/lix:sb/qyriad/beta-1 guiwf/lix:sb/qyriad/2.90-beta.0 guiwf/lix:sb/rbt/test-branch guiwf/lix:sb/rbt/justfile guiwf/lix:sb/rbt/repl-overlays guiwf/lix:sb/qyriad/maint-meson guiwf/lix:sb/qyriad/maint/meson guiwf/lix:sb/puck/meson-c-api guiwf/lix:sb/jade/burn-it-all guiwf/lix:dependabot/github_actions/zeebe-io/backport-action-2.4.1 guiwf/lix:dependabot/github_actions/cachix/install-nix-action-25 guiwf/lix:dependabot/github_actions/actions/labeler-5 guiwf/lix:dependabot/github_actions/cachix/cachix-action-14
guiwf/lix:2.95.3 guiwf/lix:2.93.4 guiwf/lix:2.94.2 guiwf/lix:2.95.2 guiwf/lix:2.95.1 guiwf/lix:2.94.1 guiwf/lix:2.95.0 guiwf/lix:2.94.0 guiwf/lix:2.93.3 guiwf/lix:2.92.3 guiwf/lix:2.93.2 guiwf/lix:2.91.3 guiwf/lix:2.92.2 guiwf/lix:2.91.2 guiwf/lix:2.93.1 guiwf/lix:2.93.0 guiwf/lix:2.92.0 guiwf/lix:2.91.1 guiwf/lix:2.91.0 guiwf/lix:2.90.0 guiwf/lix:2.90.0-rc1 guiwf/lix:2.90-beta.1 guiwf/lix:2.90-beta.0 guiwf/lix:1.11.1 guiwf/lix:1.11 guiwf/lix:1.10 guiwf/lix:1.9 guiwf/lix:1.8 guiwf/lix:1.7 guiwf/lix:1.6.1 guiwf/lix:1.6 guiwf/lix:1.5.3 guiwf/lix:1.5.2 guiwf/lix:1.5.1 guiwf/lix:1.5 guiwf/lix:1.4 guiwf/lix:1.3 guiwf/lix:1.2 guiwf/lix:1.1 guiwf/lix:1.0
204 Commits
sb/jade/coverage ... main
Author SHA1 Message Date
eldritch horrors 1e48ac50ca meson: fix clang-tidy 
meson fixed the bugs that prevented using the sanitizer args, but using
such a fixed meson will provide broken flags to the subprojects. *sigh*

Change-Id: I0ab54351b19a3879e9a97daed1621dc5f730d376
 2026-07-18 00:22:19 +02:00
rootile 3dd53ecdef tests: migrate fetchers.sh 
Change-Id: I3c2856dad2e06993717bcba437c5a0b46e74821f
 2026-07-17 21:06:08 +00:00
eldritch horrors 289fab6a68 libstore: remove ssh{,-ng}:// connection pooling 
it's used very rarely, on ssh it's pretty much broken, and on ssh-ng
it's standing squarely in the way of the rpc transition. we have not
found significant use of this feature in public configs (with all of
*two* repositories using this store parameter), so why even keep it?

Change-Id: Ifcafdf794d815001a1a55a771e5823010e5b174b
 2026-07-17 19:46:00 +00:00
eldritch horrors f5b9f105e2 libstore: drop Store::connect 
there's no reason a store should not open a connection during init if it
needs to open any connection to function. delaying connection setup like
this makes openStore less deterministic and graceful fallback impossible

this caused test failures where the old behaviour was required to ensure
test output stability. because of course something like that must happen

Change-Id: I946adddee026f1f4c74b4699d730b4d1ac9a5072
 2026-07-17 18:00:14 +00:00
eldritch horrors 57fb4f534d libstore: handle ssh{,-ng}:// logs in their store class 
build-remote should not be the only thing that ever reads logs from
remote store connections. ssh-ng is better off here because it does
not transport its logs on the ssh stderr channel, but ssh *does* do
this. neither have any ssh logs shown after connection setup except
when using an ssh:// url as a remote builder, which is not markedly
helpful for debugging anything at all. let's have both kinds of ssh
store classes handle logs themselves to alleviate all of that. this
also improves the utility of remote builder checks in `nix doctor`.

Change-Id: I23947a9666bc07561eac341578ed45993f1b9839
 2026-07-17 18:00:14 +00:00
eldritch horrors 553ae61ffd libstore: fix AddBuildLog denial of service 
adding logs to a daemon store can fail before the entire request has
been read, leaving non-command data in socket buffers. if the daemon
does not terminate immediately after the failure it will try to read
the remaining data as commands, which is very likely to not work and
cause unbounded memory allocation instead. this doesn't give clients
an attack vector they did not have before though, they could've just
as well sent malformed commands without a bad preceding AddBuildLog.

Change-Id: I86c22890eace19164d932cbd342ac9f52cee6531
 2026-07-17 16:48:28 +00:00
rootile 33d0984e27 justfile/clean: also run cargo clean 
Change-Id: I069a106ced8f164f1aaff48a9a9211668cbe012b
 2026-07-17 15:42:05 +00:00
eldritch horrors 93ca179023 f2: fix nix fixture store setup 
log dir and state dir are not taken from the environment if root is set.

Change-Id: I3752dcf7ac81b291e61beafbb3fed9601e3cfcd1
 2026-07-17 13:33:19 +00:00
rootileandeldritch horrors ece46d54c7 lix-rs: link cpp also when building tests 
Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: I656a2f1b6999395c421f9a5b301b324c77f8ac52
 2026-07-17 10:27:39 +00:00
eldritch horrorsandRutile 4c4d8b8ccd lix-rs: enable iterators 
we don't support range-based iteration yet because it's a huge hassle to
make c++ and rust iteration styles meet in any reasonable way. this is a
good start to make rust collections *actually* usable though, so here we
go. `into_iter` is not explicitly supported, but can be used regardless.

Change-Id: Iebb76c9409059bc81623e5be81ac2656b1ec5138
 2026-07-17 09:56:35 +00:00
blokyk f91bdc9367 libutil: always print addErrorContext frames 
frames manually added with `addErrorContext` are generally a lot more
useful/informative to the average user than other frames, esp. in
the module system, which can create much better error messages than
we can.

however, before this change, frames from addErrorContext were truncated
by default if they weren't in the first 3 frames, so they were basically
useless (with `--show-trace`, you're dredging through 250 frames of
module shenanigans just to spot one singular line).

this change also removes the frames for _the call to_ `addErrorContext`,
which is just pure noise.

the way this change is done hopefully leaves a bit of space for future
similar changes to error printing, by introducing a new `TraceKind` enum
that can be used to categorize traces (i haven't done that in this CL
because that would be a pretty herculean task, given all the calls to
`BaseError::addTrace` in the codebase, and we probably want to be
careful about what categories we choose). the actual printing code could
definitely be improved tho... (e.g. by iterating twice through the trace
stack instead to first pick out the most important traces and _then_
printing less "important" traces if there's space left)

Change-Id: I52acc52f231991a9f2309d9cecae362397c6888c
 2026-07-16 15:20:59 +00:00
rootile 25d47498bd cli/doctor: fix crash when having relative paths in PATH 
fixes #1244

Change-Id: Ifefab62dcedaea4bccd1546bf63e2081fc040ce1
 2026-07-16 13:04:58 +00:00
rootile 99c6ad45b5 cli/doctor: check machines toml and remote stores 
fixes #1076

Change-Id: I37e904f3b83b48618a09cc119067133775cf01b6
 2026-07-16 13:04:10 +00:00
eldritch horrorsandpiegames 0e180a479f libutil/rpc-types: Add map type 
Co-authored-by: piegames <git@piegames.de>
Change-Id: I4215efb0e9839925d34ad45d47879148dbdfc4c2
 2026-07-16 12:09:24 +00:00
eldritch horrorsandpiegames 7307475218 libutil/rpc-types: Add Option type 
Co-authored-by: piegames <git@piegames.de>
Change-Id: I75d5db5f567a3a86870ed2aceb36840ddcbb53db
 2026-07-16 12:09:24 +00:00
eldritch horrors 66090810f1 libutil: add async input-consumer-to-output-stream adapter 
this makes using the zero-copy pipes a lot more ergonomic. error
handling is unfortunately not trivial and duplicating it for all
pipe users would thus be a bad idea. we're also not oblivious to
the fact that this is a `sourceToSink`, but it's async this time
around. (at least we don't need terrible stackful coroutines..?)

Change-Id: I1ba59f27183988ad68e7f88d102935d005690f43
 2026-07-16 12:09:24 +00:00
eldritch horrors 9bd9947f41 libutil: add a zero-copy async pipe 
rpc will often need to transfer data received on push-bashed interfaces
to consumers that expect input stream sources, which are pull-based. we
want to avoid copies for performance reasons (since kj overhead as kind
of on the high side for us already), so we'll use a shared-buffer class
that behaves much like a mutex. we *don't* use mutexes because ours are
cross-thread-capable and thus require syscalls for all wakeups they do.

Change-Id: I3b14925f5d9f5e07ea2cafdf00a88a64f79e4742
 2026-07-16 12:09:24 +00:00
rootile 25f53b755c tests: migrate config.sh 
Change-Id: Ia37a84e0036dd52af112a359817c709a15aa8c90
 2026-07-15 10:53:10 +00:00
Jade Lovelace f51148a75b tidy: delete a bunch of dead code 
Found ... by edef's harness. There was more dead code it found, which I
didn't address because it felt less obviously deletable:

- `AutoDestroyCgroup::path()` / `::delegation()`
- `TeeSource`, `LambdaSink`
- `S3BinaryCacheStore::getS3Stats()` (maybe Hydra? idk)
- `fetchers::getBoolAttr()` (technically deletable but it would be
  inconsistent with the others)

Change-Id: I913d2afd61e24136fb506389fc33a1016a6a6964
 2026-07-14 20:17:11 +00:00
eldritch horrors e6ddca402b libutil: make blanket rpc converters more useful 
having from() work on primitive types lets us specialize less elsewhere.

Change-Id: Ia6eb72ab03c264c6ba3149cb8fe21fa5b3ab4b4d
 2026-07-14 17:47:07 +00:00
piegamesandeldritch horrors c739701077 rpc.hh: Document footgun in LIX_TRY_AWAIT_RPC 
*piegames is tired of C++*

Change-Id: I9d6dd935afaadf809d5a779fbcc64f48b4af1cba
 2026-07-14 17:47:07 +00:00
eldritch horrors 734cfd1d48 libutil/rpc: add more filler macros for rpc conversions 
sometimes we need to disambiguate based on the kind of the thing we are
filling due to how the c++ type works. in templates some constraints on
type resolution are lifted. sometimes we can make the compiler help us.

Co-authored-by: piegames <git@piegames.de>

Change-Id: Id9549764813a862b8d64959157fed258709eb6ac
 2026-07-14 17:47:07 +00:00
Jade Lovelaceandrootile fca58aa044 main: don't insert null function pointers into the legacy commands map 
This is purely aesthetically bothering me, but [] inserting stuff into
the map is such a C++ coded design decision and we don't intend to
mutate it here so we probably should not do it.

Change-Id: Idf37c51175b031fb396e8fd53dc8fe6f2039fb55
 2026-07-13 19:51:09 +02:00
Raito Bezariusandrootile 782d967342 nix/doctor: check nixpkgs provenance 
Many issues stems from `<nixpkgs>` not resolving anywhere or having the
wrong version, let's make it awfully obvious again here.

Towards #230.

Change-Id: If630e9616566e77143f028cdcfe0b7f00a1d486d
Signed-off-by: Raito Bezarius <raito@lix.systems>
 2026-07-13 16:47:18 +02:00
rootile 08f18285ef f2/nix: add fake nixpkgs fixture 
Change-Id: I76fe321bd93d9f79b4b1d98e529c50581d23d3f7
 2026-07-13 16:46:01 +02:00
eldritch horrors c6d22874d6 f2: increase timeouts and max worker count 
without maxprocesses we can utilize core-rich systems better, and test
timeouts are reportedly to be too low to run on small systems as well.

fixes #890

Change-Id: I89386b89fcd69ef4bf77ecf0a49c0b85e1f17c3d
 2026-07-12 20:31:05 +02:00
eldritch horrors 1657d0eb47 f2: add mark for nix settings defaults and use it 
this fixes a large portion of tests currently marked no_daemon. most of
them only needed to set some trusted settings, which is easily done now

Change-Id: Id5a5ee94951cdc92bddd2264c738ca4f98980c8b
 2026-07-12 20:28:05 +02:00
Maximilian Bosch 527ecba696 release.nix: mock rev/lastModified attrs when nixpkgs is a path 
Some code-paths expect e.g. `nixpkgs.rev` to exist.
This change makes sure this always exists. This is done on purpose
within this let-in dance rather than in the default such that
`nix-build release.nix --arg nixpkgs ../nixpkgs` also benefits from this
fix.

Change-Id: I94f1f901823d3df433e3c44ae380bb2f67eef70e
 2026-07-11 21:18:59 +02:00
Maximilian Bosch 72f8000b8f flake: update nixpkgs, fix eval 
attrpathsSuperset got renamed to preEval in nixpkgs[1] breaking Hydra's
evaluation. Updating nixpkgs to make sure this is consistent now, no
matter if flake-inputs or release.nix is being used.

[1] See commit 19a31658dc39324c9acf8d81198cd7137bdc1e92.

Change-Id: I69ab3f894534b1d89b626ddd4feda8ea502b69b3
 2026-07-11 21:18:59 +02:00
eldritch horrors 8294860ccf packaging: set CARGO_HOME 
single-user builds have a writable home directory in the build sandbox,
and the test suite does not like cargo writing anything into there. the
impact of this seems to be localized to f1 (at least on linux), but not
letting cargo write into the homedir is easy enough to do. nixpkgs also
does something much like this when compiling rustc, so we're not alone.
(it also seems that nixpkgs intended to do this for all rustc users but
broke it at some point in the past. some packages set CARGO_HOME in the
same way we do, possibly because they've had exactly the same problems)

fixes #1251

Change-Id: Ie17a0677ef09bb076ef6295bb590a0895924434e
 2026-07-11 12:11:49 +00:00
rootile 3b286bbd62 fetchers/git: cleanup code 
Change-Id: I643ea90f53b4e57980bed45b33837950f5c1e02e
 2026-07-10 18:06:26 +00:00
rootile 28fc206387 rust: add rootcause as error library 
Change-Id: I65460dda21331d79f0dfadfe06a7d6738c9ddefe
 2026-07-10 19:47:06 +02:00
eldritch horrors 64c486e22c libutil/libmain: never mask SIGWINCH 
we need to cache the current terminal size for progress bar reasons, but
we don't want to catch SIGWINCH to update the current terminal size from
a dedicated thread for repl reasons. a SIGWINCH handler function is much
easier to square with these requirements than communication with another
thread to have it change its signal mask, and since any races in handler
code affect only progress bar output and only very rarely (if ever) we'd
better chose the simplest approach. the progress bar could set a handler
of its own for this purpose, but we would much rather replace it instead

fixes #1246

Change-Id: I814d9aaf1b6fbb6a8cefc5af675a3aa372549dc8
 2026-07-10 16:28:11 +00:00
Qyriad 1a2853bf7b package: restore original curl after cl/5842 
Change-Id: I09b5cdd1a91ef471233cfcaf1c7ec1d16a6a6964
 2026-07-09 15:09:26 +00:00
QyriadandSergei Zimmerman 81084c7fc9 libstore: Fix libcurl thread wakeup with curl >= 8.21 
Since https://github.com/curl/curl/commit/2a2104f3cff44bb28bb570a093be52bbeeed8f23
libcurl sometimes just swallows events in curl_multi_perform, we entirely miss them,
and curl_multi_poll blocks forever.

Fixes #1234.

Based on https://github.com/NixOS/nix/pull/16090.

Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
Change-Id: I453dce192de0ea9e92bfb0114020dc9d6a6a6964
 2026-07-09 15:09:26 +00:00
rootileandeldritch horrors 41bad096e3 rust: migrate libutil/git.cc 
Change-Id: Ic3932fdf3afd3dc09019623516597f1c648f7fc4
 2026-07-08 20:03:10 +00:00
eldritch horrors dd8f1843cc lix-rs: allow exposing ord/eq operators from rust 
this is a bit of a hack, but since zngur cannot handle multiple trait
implementations per type yet we will have to commit to singles types.

Change-Id: I60e7b96bbeaa9fb87cf43662d4a9a5d44116bf47
 2026-07-08 18:04:40 +00:00
Linus Heckemann 9302845601 package: use buildPackages for licxxbridge build 
This fixes a regression in cross builds introduced in https://gerrit.lix.systems/c/lix/+/5704.

Change-Id: I3031d7c4daa5554cc16f654d28311f486a6a6964
 2026-07-07 10:59:53 +02:00
rootile 523df65fba tests: migrate misc.sh 
note: only two of the tests actually needed migrating, as the others
were already covered (e.g. by lang)

Change-Id: I30abb2bd728ae2b144ad76566096a216aed3b8f8
 2026-07-06 12:39:47 +00:00
rootile de67d1424d bench: fix outdated nixpkgs reference 
Change-Id: I293f4e4c399c192b0fec30aa8554b64641eee0bf
 2026-07-05 13:21:25 +00:00
isabelMic92Yannik Sander
 		405ff2eb03 nix-eval-jobs: --apply 
Based off of https://github.com/NixOS/nix-eval-jobs/pull/332 and
9f292a402a6edf620b872bedaaad5f0a45c23826

closes https://git.lix.systems/lix-project/lix/issues/1214

Co-authored-by: Mic92 <joerg@thalheim.io>
Co-authored-by: Yannik Sander <yannik@floxdev.com>
Change-Id: Ifba7081b384d18fbdc47437de261d30b65a7930a
 2026-07-04 14:46:08 +00:00
Seth FlynnandBernardo Meurer Costa c1370bc81c build: use mimalloc 
mimalloc is a compact general purpose allocator from Microsoft. It
consistently outperforms glibc's `malloc()` in allocation-heavy
workloads, such as Lix's evaluator

It's currently only linked in the main `nix` executable, as Boehm's GC
uses its own allocator. Other allocations that *do* go through glibc's
`malloc()` are still much faster, though

Benchmarked on x86_64-linux against Nixpkgs `bd07873`:

| attribute         | thunks  | lix@`1396012` | mimalloc | uplift |
|-------------------|---------|---------------|----------|--------|
| hello             | 206444  | 0.726s        | 0.620s   | 1.17x  |
| chromium          | 1177382 | 2.273s        | 2.136s   | 1.06x  |
| firefox-unwrapped | 1394797 | 2.521s        | 2.378s   | 1.06x  |
| texliveFull       | 3186440 | 4.992s        | 4.672s   | 1.07x  |
| nixosTests.gnome  | 7905808 | 6.115s        | 5.723s   | 1.07x  |

Based-on: https://github.com/NixOS/nix/pull/15596
Co-authored-by: Bernardo Meurer Costa <beme@anthropic.com>
Change-Id: I3ad92eacc075efeaf3d9f7730ada7b29835536a4
 2026-07-04 10:39:30 +00:00
Raito Bezariusandrootile 12f10fca7e nix/doctor: check ambient search paths 
Some issues are caused by users confused about their ambient search
paths, let's analyze the current configuration and make it awfully
obvious what is going on.

Towards #230.

Change-Id: I953ef3feaefc56cee68623c5aaa4d568ba1fd034
Signed-off-by: Raito Bezarius <raito@lix.systems>
 2026-07-03 20:12:07 +02:00
Raito Bezariusandrootile b62b37296c nix/doctor: check flake registry 
We try to access to the Flake registry, if we are failing, it's hosed.
Otherwise, let's record some facts we can reuse later.

Towards #230.

Change-Id: I80dd8272d5c27f7f7ff01052e3487a0496ee29fc
Signed-off-by: Raito Bezarius <raito@lix.systems>
 2026-07-03 20:02:35 +02:00
Raito Bezariusandrootile abeefa0e4f nix/doctor: print nix related information 
Based on `nix-info` and `nix --version` output.

Contributes towards #230.

Change-Id: I13bb9608b733e10037c6816456b5312e712c18ad
Signed-off-by: Raito Bezarius <raito@lix.systems>
 2026-07-03 19:44:16 +02:00
Raito Bezariusandrootile 4126e671cc nix/doctor: print general system information 
Based on `nix-info` output and experience (LOCALE issues).

Change-Id: I350fff9592e7acc8fba0babe67c9fe1bf94a5781
Signed-off-by: Raito Bezarius <raito@lix.systems>
 2026-07-03 19:44:16 +02:00
Raito Bezariusandrootile ad199c4723 libutil/systemd/hostnamectl: init 
This is some code that is specific to systemd to fetch hostnamectl
information via the JSON flag.

This will be used for `nix doctor`.

Errors will show up similar to this:
```
❯ ./build/lix/nix/nix doctor
[INFO] Nix system type: 'x86_64-linux'
warning: could not get host information via hostnamectl: error: Expected JSON value to be of type 'string' but it is of type 'number'
```

and makes them non-fatal.

Change-Id: Idd39f29626e770b1cd3896e8eb8545416c74b041
Signed-off-by: Raito Bezarius <raito@lix.systems>
 2026-07-03 17:45:26 +02:00
eldritch horrors e9a01a5be6 libfetchers: don't treat empty/zero hashes as valid locks 
treating a file with an empty or zero hash as locked is not helpful.
these are placeholdes for "hash is not known", thus treating them as
a valid lock makes them completely useless (and confusing to users).

fixes #1233

Change-Id: If42b47281e6973fc86662b973db69e26f6346f5a
 2026-07-03 11:06:57 +00:00
K900 48f7747deb treewide: drop support for mdbook < 0.5 
25.11 is dead, and so is mdbook 0.4

Also reenable linkcheck which was doing nothing this whole time lmao

Change-Id: I4a8b9c763b881de840d6ef1b1d85bf19941f4386
 2026-07-02 18:06:50 +03:00
Qyriad 2c45e056e9 libstore/curl: std::move is not mem::take; fix correctness 
These uses of std::move are taking data out of a repeatedly-used state
object, à la Rust's std::mem::take(). But move-constructing or
move-assigning from an object does not guarantee that the moved-from
object is equivalent to a default-constructed one, or even that the
moved-from object will continue to function like a non-moved-from object
of that type (only that the object remains "valid").

Assuming there's no differing custom allocator shenanigans at play, both
libstdc++ and libc++ do leave moved-from std::map objects (and probably
others) in an empty, well-behaved, default-constructed state, at the
time of this writing. However the correct tool here is std::exchange().


Change-Id: I7cf76762a1b05a542b53fcd74be100646a6a6964
 2026-07-01 23:03:27 +02:00
Qyriad d4e445f251 libstore/filetransfer: reörganize includes 
Change-Id: I74310683b6e721f3ad0a7b2e2e2aa28c6a6a6964
 2026-07-01 23:03:27 +02:00
Qyriad 43c5aa2076 libstore: factor out curlFileTransfer 
As part of efforts to make debugging our Curl usage easier


Change-Id: Ib1f8567daa8b3d3d559a164bc85a8b906a6a6964
 2026-07-01 23:03:27 +02:00
Qyriad dbf1ef3d42 libstore: factor out CurlMulti 
As part of efforts to make debugging our Curl usage easier


Change-Id: I182bb3e968f1dfcd192ecd61103cba376a6a6964
 2026-07-01 23:03:27 +02:00
Qyriad 325364fc26 libstore: factor out TransferItem 
As part of efforts to make debugging our Curl usage easier


Change-Id: Idb8c32b33b6bfd0eebb9c9fded7b251d6a6a6964
 2026-07-01 23:03:27 +02:00
Qyriad cde71e45a1 libstore: fully inline FileTransferError 
To avoid template instantiation problems during the incomming
extractions


Change-Id: Ia0179da00bdfa45034559be88a2b923c6a6a6964
 2026-07-01 23:03:27 +02:00
Qyriad 1705515e79 libstore/filetransfer: reörganize includes 
Change-Id: If1a9c9d45e50af1d06d07a8510ab003e6a6a6964
 2026-07-01 23:03:27 +02:00
Raito Bezariusandrootile d26a73bfe6 libmain/shared: factor out nix features 
This way, other parts of the codebase can use it.

Change-Id: I58673a6a56362692980426a6c8f70f5c6aff24d4
Signed-off-by: Raito Bezarius <raito@lix.systems>
 2026-07-01 17:46:00 +02:00
Samuel Dionne-Riel cf1e565a0a devshell: Fix usage with nix-shell 
Since d0190cff6f, the `justfile` uses
`$out` from the environment as an argument to `--prefix` in its `meson`
invocation.

The `$out` variable is...  A bit spicy to use since its semantics change
depending on whether `nix develop` or `nix-shell` is used.

In a `nix develop` shell, a directory *relative to PWD* (and not
relative to the Nix expression) is exported. In a `nix-shell`, the
`$out` used comes from the derivation.

The alternative to this change is to make the `justfile` use a variable
name that would not conflict with a `nix-shell` and `nix develop` shell.
Since it wants to use that value for the `outdir` variable, this would
be the name I would pick first.

I chose to instead work around the issue within the `nix-shell` since
this would be breaking the existing interface. As tenuous as it is.

Change-Id: I64cd5fd85345457e14880cb133e9ddafd55a25da
 2026-07-01 11:02:38 +00:00
blokyk a92e4b5c42 devshell: fix usage with non-nix rust-analyzer 
rust-analyzer binaries that don't come from nix (such as the one
shipped with vscode's rust-analyzer extension by default) need the
`RUST_SRC_PATH` env var to be the path to rustc's sources, but this
wasn't set inside the devshell. this change explicitely adds it to
the environment. (trying to add pkgs.rustLibSrc and rustcSrc to
the shell's packages unfortunately does nothing, because they are
just pure source packages, nothing more.)

(the reason this probably didn't come up for other devs is because
1. a lot of lix devs use other editors that probably just use the
binary from the path, and the devshell installs pkgs.rust-analyzer,
which is patched to always have `RUST_SRC_PATH` set, and 2. because
pkgs.vscode-extensions.rust-lang.rust-analyzer is also patched to use
pkgs.rust-analyzer by default instead of the bundled one, so installing
the extension using nix instead of from the marketplace would also hide
this problem.)

Change-Id: Ia27daf2a88756311ac5485ea0af44aae242241a9
 2026-07-01 11:55:10 +02:00
eldritch horrors 418918b599 libstore: derive legacy-tunneled rpc id from package version 
this way each git revision trivially gets its own rpc protocol version.

Change-Id: I066c58ebff697e534479db5d02271dac1ea71709
 2026-06-30 19:58:17 +00:00
eldritch horrors 6e19a940b1 rpc: transport errors in encoded exception descriptions, not results 
results don't allow for streaming or pipelining. transporting errors in
exception texts is *bad*, but it's still better than not having streams
that actually work. this isn't a great situation, but with capnp that's
pretty much the best we can do. still beats the other rpc frameworks :/

Change-Id: I2abf17bc5ea5de2baeea44ff4d7b2f4db33e98dd
 2026-06-30 19:56:57 +00:00
eldritch horrors 57c00a20c5 libutil: make exception wrapping accessible outside of TRY_AWAIT 
this will be useful for rpc cases, and it reduces macro code expasions.

Change-Id: I47c837fb5efab8eb408fcf9617055ea1562fc422
 2026-06-30 19:55:45 +00:00
eldritch horrors d0deb1a150 libutil: add error encoding that survives capnp exception transport 
using result types in capnp is fraught. while it makes some sense for
interfaces that need neither streaming nor pipelining and can provide
much better error fidelity there it's also fundamentally incompatible
with those that *do* need streaming or pipelining: streams will never
be stopped early unless an exception flies, and pipelines cannot look
through result types. likely the best thing we can do is to encode an
error for transport in the capnp/kj exception description strings. :(

Change-Id: Icb7d16238fa9a7aaf92c00363f7be4076ac02a61
 2026-06-30 19:55:45 +00:00
Qyriad 22e2ecdc43 build: update integer overflow sanitizer handling for fixed meson bugs 
Change-Id: I27f33e66e107d4af13abe62257be8c696a6a6964
 2026-06-30 18:56:51 +00:00
piegames 5434ddf4cd justfile: Don't require nested escapes on forwarded commands anymore 
This is a hack, but this is the best we can do with Just unfortunately.
Maybe rewriting this file into a python script wouldn't be the worst
idea at this point …

Change-Id: I32d089f6b49165faed99fb592d068dbb8a7e48e6
 2026-06-30 10:58:58 +00:00
piegamesandeldritch horrors 7ef274600d testlib: default nix fixture to also test all daemon protocols (3/3) 
This is the real deal. The new tests have surfaced one bug in the test
suite itself, and one in the RPC legacy protocol wrapper.

Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: Id75b671c10b45655d3627abd88563e2d41719446
 2026-06-28 19:09:49 +00:00
K900 12c1132f07 package: revert the breaking curl change on 8.21+ 
Change-Id: I374b37c5d428c6c7db71341d783b6a1cddf45b19
 2026-06-28 17:19:51 +00:00
piegamesandeldritch horrors d5044f0016 testlib: default nix fixture to also test all daemon protocols (2.5/3) 
All tests should have been clear here, but alas

Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: Id642be1f6e0fb714b844ee5348dc096866cf3906
 2026-06-28 16:52:12 +00:00
eldritch horrors 2605d3eab1 libcmd|rs: use rustyline instead of editline for the repl 
Change-Id: Ic414de3cc2435d9615305f05a85aa9c923e82a69
 2026-06-28 16:37:46 +00:00
piegamesandeldritch horrors 567348a90c testlib: default nix fixture to also test all daemon protocols (2/3) 
This commit enables the parametrization for "legacy-combined" protocol.
All failing tests with that have been either fixed or disabled. Notably:

- A couple of tests (e.g. involving builders) could be made to work with
a non-local store, but this would require some refactoring to testlib in
order to make certain configuration settings generic over local and
remote operations. We've disabled those for now, in order to make
progress
- The store tests all run locally only because of their nature
- The flakes tests also all run locally only because my energy for
fixing them is limited (and it's a *lot* of test failures, with probably
little overall benefit in terms of test coverage)

Change-Id: I56fa249a64f7c17c952f688ec89e9687f2a13f12
 2026-06-28 18:17:57 +02:00
eldritch horrors 1c2b5d4b5a libutil: expose loggers to rust 
this takes the form of a couple of hideous macros for now to get started
without worrying about semantics of rust logging and tracing crates, all
of which are in some way incompatible with the current lix expectations.
we do want to move to using those eventually, but that will have to be a
task for later. as long as only small amounts of lix code live in crates
we don't have to worry much anyway, and by the time we get around to any
Activity beimg ported we'll hopefully have a better understanding of the
requirements on either side. we'll reëvaluate the log infra around then.

Change-Id: I4aec476a486f56d0972294358cbf0b6810ab9394
 2026-06-28 13:02:07 +00:00
eldritch horrors c5bcb9a312 libutil: disable direct access to logger verbosity 
that way we can sync it with rust for nearly free.

Change-Id: Ib3f1e0335884e077a0293f4b13c5919297748522
 2026-06-28 13:01:45 +00:00
eldritch horrors 91663cca2d libutil|rs: bring back c++ interop with zngur 
Change-Id: Ia9e9881f464d5b25110b56fb9d72ed30463eaeee
 2026-06-28 12:47:18 +00:00
Qyriad d9ecb5301e build: cargo entrypoint II 
This is a second attempt at https://gerrit.lix.systems/c/lix/+/5516

instead of letting meson try to be cargo we'll just have cargo at home.
this requires some contortions to link everything together due to quite
a few meson deficiencies, but at the end we get to pretend that rust is
just c++ painted orange. we'll use cxx to bring real interop back soon.

also fixes #1230
also reverts 10845bfe63

co-authored-by: eldritch horrors <pennae@lix.systems>

Change-Id: Ib1fda843fa80d818705d5a65ec9054216a6a6964
 2026-06-28 12:46:20 +00:00
piegamesandeldritch horrors 9e846fbaea testlib: Small feature additions 
Change-Id: I8940e66a1fb89cf8024b1e94e328b28a008a1452
 2026-06-28 13:45:48 +02:00
eldritch horrorsrootilepiegames
 		84fb93a009 testlib: default nix fixture to also test all daemon protocols (1/3) 
This first geralizes the `nix` fixture with `pytest_generate_tests` to
iterate over all protocols (unless they are marked with `no_daemon`),
though for now the list of protocols is set to be empty. The lang tests
are all tagged with `no_daemon` because they are mostly pure and running
them multiple times would be wasteful.

Co-authored-by: rootile <lix@rootile.de>
Co-authored-by: piegames <git@piegames.de>
Change-Id: Ib407edb420ba4bf434cacf9563a71f5ae6fa8eef
 2026-06-28 13:45:46 +02:00
Samuel Dionne-Riel 7868f52c2a tests: Add regression test for daemonAuthorizationSettings 
Regression test for extra-trusted-users being ignored
  https://git.lix.systems/lix-project/lix/issues/1183

Change-Id: I764a6add0215b535fc4abd21b41d1c23b21fc06b
 2026-06-27 18:37:27 +00:00
Samuel Dionne-Riel fc6772a7dd main: Run registerGlobalConfig before loading config 
This is a bit of a trip.

Commit 63b0a01d75 moved config
initialization out of static initializers.

In doing so, an oversight was made with the new
`GlobalConfig::registerGlobalConfig` calls. Previously all global
configuration would be handled well before `loadConfFile` was called.

The change placed `registerGlobalConfig` in `main` *after*
`initNix`. The `initNix` function is what calls `initLibStore`, which in
turn callse `loadConfFile`.

So, in effect, the configuration for `daemonAuthorizationSettings` and
`nixDevelopSettings` are not actually loaded.

Change-Id: I10dba2c98e032f910942ee0e0ca6bb39ef60edc2
 2026-06-27 18:37:27 +00:00
rootileandeldritch horrors fce63f6dbf parser: improve error message for ill-defined escape sequences 
fixes #1169

Change-Id: Ic93e0cb5442b717964de3ae9eb22f4cd48e4ad0a
 2026-06-25 21:03:09 +00:00
Qyriad de68de1d11 tests/test_build_fod: fix overly strict pattern 
Change-Id: Idd3146f41d527a7263ee61e7a5ce0e836a6a6964
 2026-06-25 09:54:50 +00:00
Qyriad 071b810f95 package: override Curl to remove backported patch that breaks our tests 
Cherry-picked-from: https://github.com/nixos-cuda/nixpkgs/commit/3f596088797d2e3a62e20cae2d2936e8548a690e
Co-authored-by: Atemu <git@atemu.net>

Change-Id: I5d060872d4d0ae283d73b4268af612326a6a6964
 2026-06-24 13:45:53 +00:00
Qyriad 63fb295ad8 build: fix C++ dylibs on Musl by working around Nixpkgs brokenness 
Change-Id: Iad30ec38eef8205e2d73553caddbbbc96a6a6964
 2026-06-22 21:18:56 +02:00
Qyriad 2ea4b5d9e1 fix Nixpkgs 26.05 warning about nested lists in nativeBuildInputs 
Full warning was:

warning: Dependency of package 'lix-shell-env' uses a nested list in attribute 'nativeBuildInputs'.
This is deprecated as of Nixpkgs release 26.05, and support will be removed in a future nixpkgs release.


Change-Id: I6b6c7425e1b03d3238ea21138cb3f2cf6a6a6964
 2026-06-22 21:18:56 +02:00
rootile 8902b78442 tests: migrate derivation-json.sh 
Change-Id: I8a648231677bde28611dc9511e3a03d87608e7b4
 2026-06-22 18:16:17 +00:00
Justin !andQyriad 30ca2e38c4 nixpkgs: bump 25.11 -> 26.05 
Change-Id: I7747d38d38f3be69404f5dcc8147e5366a6a6964
 2026-06-22 17:58:20 +02:00
Qyriad 3f7bc8ffb8 HACK: patch out broken Perl version compatability check on macOS 
Change-Id: I85cfc3863299d06f36138753e8f1f0206a6a6964
 2026-06-22 17:58:20 +02:00
eldritch horrors 20453629ab build: remove libutil rust components 
they're not doing anything, and they're slightly in the way of reworking
the rust build infra. we'll add them back once that rework is completed.

Change-Id: Idde8ba3585fd08b89b4b4298349f8f58a29a9661
 2026-06-21 17:16:17 +02:00
eldritch horrors 0cddd2d38a build: always link plugins with unresolved symbols allowed 
we're quite surprised that this hasn't caused problems sooner.

Change-Id: I33b779c1563e15db715c7fd4d3cb6516cf34391d
 2026-06-21 17:16:17 +02:00
Qyriadandeldritch horrors 10845bfe63 build: move all remaining C++ files to a library (except main.cc, yet) 
Change-Id: I2a69701a67faf6f9ad33e30bbe7aa4386a6a6964
 2026-06-21 17:16:17 +02:00
Qyriadandeldritch horrors 05e9ca3e26 build: remove file prefix mapping 
blokyk on Matrix found that is not only no longer needed but now
actually breaks the very thing it sought to fix


Change-Id: I39cc0df611a6d81c763063c9fd2f0f426a6a6964
 2026-06-21 17:16:17 +02:00
ash ff920956a9 libstore/build: fix URL guesswork for failing structuredAttrs FODs 
Fixes #1175

Change-Id: I6a074f489620f6de0615858f4d0d872c6a6a6964
 2026-06-20 14:31:17 +01:00
rootile cf0ce785b7 tests: remove forgotten cli-literate-parser 
When removing the f1 repl characterization test suit, these files were
kept on accident, despite them being dead code now

Change-Id: I13b2c5a96004787c26fe7209e4cb3aeedd8ada15
 2026-06-17 12:03:36 +02:00
RutileandLix Systems Gerrit d748b4e509 Revert "repl-characterization: document the syntax" 
This reverts commit 0d6b372c19.

Reason for revert: In the time between this getting +2 and it being merged, all repl tests and its syntax have been migrated to f2, making these changes null and to dead code

Change-Id: I00f5237bf3f2ca3a60cc1584f67fb67196805e7e
 2026-06-17 10:00:54 +00:00
Jade Lovelace 0d6b372c19 repl-characterization: document the syntax 
Gosh. Why was there never a README in here explaining what it does
without requiring people to read C++? There was a half-assed grammar in
a comment, but it was very half-assed, so let's replace it with a
better-written one.

I'm documenting this so that neither I nor anyone else have to read C++
while rewriting the parser in Python.

See: https://git.lix.systems/lix-project/lix/issues/1196

Change-Id: Ia34ea7b02f109eb7753833e1c03d4b5d1d3c3b75
 2026-06-17 01:01:13 +00:00
rootile 7bf0edcc0d build-remote/logging: use machine names instead of uris 
Change-Id: Iafe7b3e780d3a69dc4bb2d745ff40db6df7111e8
 2026-06-16 17:58:23 +02:00
rootile 4b0371bff3 libstore/machines: add name attribute 
Change-Id: Ie0f246dee219ffb3cade7e8b9a9596eb2a06f2dd
 2026-06-15 22:09:15 +02:00
rootile c019a776e9 python-infra: move f2 specific stuff into f2 
When extracting a global pyproject.toml from the f2 one, some f2
specific things slipped through and ended up in the global one.
This commit moves those parts back where they belong

Change-Id: I95706fabec487410717fb362c95bfee900cf5b8c
 2026-06-14 20:56:18 +02:00
rootile 93c872ab25 tests: migrate binary-cache-build-remote.sh 
Change-Id: Ie159da4f00cf05627a777bae80c1368485dd4fcc
 2026-06-14 20:33:11 +02:00
rootile 2ea51a6c71 tests: migrate structured-attrs.sh 
Change-Id: I6d2b82e3aa236d01b9a345c148b7fef3703a446d
 2026-06-14 20:19:58 +02:00
rootile fe79d62b73 tests: migrate check-reqs.sh 
Change-Id: Icc22331f6f7e1be0919a74155cd93514c5d70a65
 2026-06-14 20:19:58 +02:00
rootile 9614f12908 tests: migrate export.sh 
Change-Id: I9416b71b0a15a7703aa9d7c2dea866a1f556e657
 2026-06-14 20:19:58 +02:00
rootile bc8a95457f tests: migrate export-graph.sh 
Change-Id: I0de67ba6b1f710434f504814dd9b5eb7d486e65a
 2026-06-14 19:02:11 +02:00
rootile ba3a026f40 tests: migrate eval-store.sh 
Change-Id: Id4a721871218160b441e848f6f252c06187320bf
 2026-06-14 18:46:39 +02:00
Astreaprtcl 12baca7bea store/fetchers: fail if ca or netrc file has been specified but doesn't exist 
if the netrcFile setting has been changed tell curl that the file is required, so
fetching fails if the file does not exist

if the caFile setting has been specified, check if the file exists and throw an
error if is it non-existent

Refs: #1106
Change-Id: Icb9330a7a715175d35b9ed894ed945f0fd4d7061
 2026-06-13 23:38:45 +02:00
rootile abfe9d0c3e tests(functional): remove old repl characterization test suite 
Change-Id: I2c13e0332298600cafa97b06113ca56f66668324
 2026-06-11 16:30:56 +02:00
rootile 64b434b87b tests(f2/repl): migrate stacktrace tests 
Change-Id: Ic6795f22d68aea113fa2037cbd0a731ac5e698f8
 2026-06-11 16:30:56 +02:00
rootile da9f0c494e tests(f2/repl): migrate failing pure eval test 
Change-Id: Ic8db38346c2420b2d3c5ab51fe71ef0b88f826b3
 2026-06-11 16:27:49 +02:00
rootile a50b85b7d6 tests(f2/repl): migrate overlay tests 
Change-Id: Ibd2b123f919abd84ecb9333bd952cf0bdb813992
 2026-06-11 16:26:34 +02:00
rootile cb4703019b refactor(f2/repl): move overlays to overlay folder 
Change-Id: I16e42eae00dc7a1f06300a81d69c637b9fa342df
 2026-06-11 16:19:20 +02:00
rootile ab8cdcc9b8 tests(f2/repl): migrate basic tests 
Change-Id: I22241734a7249d376c86b3a226c5f1a82c03fd33
 2026-06-11 16:19:20 +02:00
rootile f7ce20d879 tests(f2/repl): migrate debugger tests 
Change-Id: I2cc1d45db2e5e0b8cea6798c70e13ede57fea6ee
 2026-06-11 16:19:20 +02:00
rootile ab149a6cd8 tests(f2/repl): migrate builtins_warn.test 
Change-Id: Iaed1fb8651840b855ec7bdf5087681dc5eb9d54c
 2026-06-09 19:40:07 +02:00
rootile 99175779c4 tests(f/repl): remove duplicate repl tests 
Change-Id: I07635a7f62bf56b4ee9526b5a34f51049554e875
 2026-06-09 19:39:52 +02:00
rootile a4634bca67 feat(f2/repl): allow should_fail to have repl input 
So apparently we can still use the repl if the input file fails, as long
as we have the debugger enabled.
This Commit removes the "only one output block for should_fails"
requirement to allow this behavior

Change-Id: I02568c8d7e4305ab6414953306650bbe13c6f4bc
 2026-06-09 19:16:46 +02:00
rootile c6fdf7d843 feat(f2/repl): add block for catching startup messages 
Change-Id: I8465ceb70ea698a00e1cdc7cef0c560382bb73a3
 2026-06-09 19:16:46 +02:00
rootile 5d91fcba5f fix(f2/repl): accept tests should not replace delimiter on updates 
Change-Id: I468bf84af1b4a5dab34e186c55830e11ba9b9474
 2026-06-09 19:16:46 +02:00
rootile 28a8c5c42a formatting: disable formatting of repl test files 
Change-Id: I1b5fedbf3f54ada2a8e105f1f318956154169760
 2026-06-09 19:16:46 +02:00
rootile 868ec73e16 docs(f2): move lang and repl tests to their own files, closer to where the docs are actually needed 
Change-Id: I2d003144ef97f12069a27f9a4be4219ec14d6056
 2026-06-09 15:21:06 +02:00
rootile 1a1234cf68 f2: lay foundations for repl testing 
Change-Id: I0f5588adbf4ea11e2846b2a3655308d1dfb711b0
 2026-06-09 14:06:04 +02:00
eldritch horrors daa2bc82bc libstore: warn if http3 && !http2 
curl does not support this configuration. we can't easily change the
config setup to mirror how curl works, but we *can* print a warning.

fixes #1223

Change-Id: Iaa1583275d9cc9e5eed5552210db183a8e0628c8
 2026-06-08 12:47:19 +00:00
blokyk 0cad504c8a libexpr/flake-config: improve untrusted flake config prompt ux 
the case-sensitiveness of the "accept all options" prompt (and the fact
it silently defaults to no) was confusing a few users, so by rewording
it slightly we can avoid that case-sensitiveness and make the prompt
clearer. for consistency, i also rewrote the individual accept prompt to
make it case-insensitive.

at reviewers' requests, i also reworked the code so it re-prompts the
user on invalid answers.

Change-Id: Ie786c0f26a0aa752b22d5a5e8af1b2c7ca2846b5
 2026-06-07 21:23:39 +00:00
Alex James 1396012a03 meson: conditionally define optional dependencies in pkg-config 
lix.pc currently lists several optional libraries (such as libcpuid and
libseccomp) as required dependencies. Update the pkg-config generation
step to only list optional dependencies which are used by the Lix build.

Change-Id: Iceea7ecc9421b5b10fc08edbe95a063a4e39cea1
 2026-06-06 15:49:20 +00:00
Linus Heckemannandeldritch horrors 267a8e7c9b nix-eval-jobs: fix version 
Change-Id: Ie237ddb2b1a0829b0954b05794506afd6a6a6964
 2026-06-06 15:20:47 +00:00
toonn 2698fe8611 manual: Update Attribute Set section to exclude indented strings and URIs 
Indented strings and URIs are string literals but they are not allowed
as names for attrsets, `let` or `inherit`. The following snippets
illustrate that they lead to syntax errors.

==> examples/ind_str-attr.nix <==
{
  ''indented string'' = "error: syntax error, expecting '}'";
}

==> examples/ind_str-inherit.nix <==
{ inherit ({ "non-indented string" = "error: syntax error, expecting ';'"; })
    ''non-indented string'';
}

==> examples/ind_str-let.nix <==
let
  ''indented string'' = "error: syntax error, expecting 'in'";
in
  true

==> examples/uri-attr.nix <==
{
  http://example.org/foo.tar.bz2 = "error: syntax error, expecting '}'";
}

==> examples/uri-inherit.nix <==
{ inherit ({ "http://example.org/foo.tar.bz2" = "error: syntax error, expecting ';'"; })
    http://example.org/foo.tar.bz2;
}

==> examples/uri-let.nix <==
let
  http://example.org/foo.tar.bz2 = "error: syntax error, expecting 'in'";
in
  true

Change-Id: I6774af93baf9f533877642a563775667c518ea88
 2026-06-06 14:52:54 +00:00
eldritch horrors d5cf115a3c libstore: fix file transfer cancellation deadlock 
file transfer cancellation should not get lost, or bad things happen.

fixes #1218, probably

Change-Id: I1be9d523236655f1248883d59b2b7ef5db7873a2
 2026-06-06 14:19:53 +00:00
eldritch horrors 4f0214b4bc tidy: add a lint for std::fs exceptions 
Change-Id: Iba3c6ed858a7c4304c8c8825aaf5df3ec60b91df
 2026-06-05 18:43:08 +00:00
eldritch horrors 6cd8fcd9d8 libutil: handle all rename errors in moveFile 
what the actual fuck. swallowing all errors *except* EXDEV is not okay.
renames do not do this, so moves should not do it either. luckily we do
not use moveFile anywhere except the store path registration code. this
may or may not have caused problems in the past. probably nobody knows.

Change-Id: I2b0255a5703983cbd129abc3219c11ac7171fd12
 2026-06-05 18:26:21 +00:00
eldritch horrors e108432cb7 libutil: catch std::filesystem errors 
catch and rewrap std::fs errors in some meaningful way, otherwise lix
can crash with foreign exception aborts if filesystem failures happen
that aren't already being caught. renames in particular can fail with
incomprehensible errors in even pretty normal circumstances elsewise.

fixes #1219, fixes #995

Change-Id: I0b1e5696d300aac3422e43cffc17a2426803d64e
 2026-06-05 17:17:18 +00:00
Maximilian Bosch c13ca9adcf nix-support/build/outputs: don't expose rl-next.user 
I'm not sure why this wasn't an issue before, but the generation of
jobset constituents in Hydra trips over this. The Hydra-convention is to
have the `system`-attribute last anyways, not in-between.

Since this is a single attribute, get rid of it for the sake of
simplicity.

Change-Id: I80ef1b92ed158c8712ee899534ce82130614f338
 2026-06-04 16:26:04 +02:00
Maximilian Bosch 82d23e8f1b release.nix: init 
This is a Hydra jobset definition to be used to warm up a cache with Lix
versions built against several nixpkgs versions. This is only happening
after a bunch of tests have passed.

Change-Id: Id0b0e50c08581f3f377c7c793d2fd83a6007999d
 2026-06-04 16:26:04 +02:00
Maximilian Bosch 1c70995b25 flake: remove release job 
This is Hydra-specific and will be re-added in the commit adding a
release.nix.

Change-Id: I13900821b02fe268356932f612734187188832b8
 2026-06-04 15:41:13 +02:00
Maximilian Bosch 4931a5f10c nix-supports/outputs: only build packages on platforms that are available 
If e.g. i686-linux is not in the list of systems and thus no attribute in `nixpkgsFor`
exists (e.g. on hydra.afnix.fr), don't try to instantiate it.

Change-Id: I118e2af5289ad7bd95ce7f1e6b0728afb6f0321f
 2026-06-04 15:40:48 +02:00
Maximilian Bosch 9b2a19530c flake: split off common CI artifacts 
...expect for the dev-shell: while it doesn't seem reasonable to me to
even build dev-shells in post-merge CI, this should still remain part of
the pre-merge checks.

Change-Id: I5cb9b4cf53484f884d694770ed94d0dc0862117e
 2026-06-04 15:38:58 +02:00
Maximilian Bosch 9fe22f4e5e flake: split off tests 
Change-Id: Ic7957eb2928bc66e9213051d9de7447d96e55fe6
 2026-06-04 15:30:46 +02:00
Maximilian Bosch 52449f3f8e flake: split off packages attribute 
Change-Id: Ic1e0ab81cb034f6468edfc03b7a716bbf157146f
 2026-06-04 15:30:45 +02:00
Maximilian Bosch eb4997592d nix-support/build/inputs: init from flake bindings 
Change-Id: I84b462d8fa18f656d4a19d2def6d69ecfb90c156
 2026-06-04 15:30:45 +02:00
Maximilian Bosch 8d581f42d9 flake: turn variables into a scope 
Step II to move the code out of flake.nix. Done in a second intermediate
commit to make it easier to review.

Change-Id: I13fb386f2e9b87b1d28c366e9b7f0d2dbe03315c
 2026-06-04 15:30:45 +02:00
Maximilian Bosch 79d6056d26 flake: import nix_2_18 as path 
That way we can use it in non-flake contexts down the chain.

We don't override the `nixpkgs` anyways since this is a test artifact
that needs an old nixpkgs input, so this isn't a concern here anyways.

Change-Id: If2aa8f7171a445e8321ef449026e206cc5a0fe3c
 2026-06-04 15:30:45 +02:00
Maximilian Bosch 13bcd07385 flake: don't use self in let block 
The upcoming change factors out a lot of code to make it reusable for a
"legacy" Hydra project that we use to build a matrix of (Lix x nixpkgs)
to have a warm cache for everyone.

The Hydra project will not use flakes to have some inputs overridable,
so the change is needed. In non-flake mode we don't have a notion of
"self", so rename it to `lixSrc` such that the next commit is purely
about moving stuff without changing any aspect.

Current stock Hydra doesn't expose `lastModified` though, so right now
that is a cache-miss. There's a pending Hydra PR fixing that[1].

[1] https://git.lix.systems/lix-project/hydra/pulls/82

Change-Id: I59d70b8b874f576c782217e7bca099eda4c7add9
 2026-06-04 15:30:20 +02:00
Maximilian Bosch 1b9570162e tests/nixos: don't pass in self 
The reference to the flake is used in a single place only to get the
`doc` output of Lix. Remove it so that we can also instantiate the tests
in a legacy Hydra jobset (happening further down the chain).

Change-Id: Ie3eccf42d67c81fcf991342097783e5067d512bc
 2026-06-04 15:30:08 +02:00
Maximilian Bosch 003946f062 tests.container: fix for 26.05 
Noticed on my quest to make Lix buildable against multiple nixpkgs' to
provide pre-warmed caches for the flake on AFNix's Hydra.

`pkgs.systemd` has a new patch policy[1] that only allows patches for
hotfixes and fundamental incompatibilities. As such, the patch that
allows nspawn workloads to start without a `/usr/bin` was removed with the
recommendation to create that stub yourself.

[1] https://github.com/NixOS/nixpkgs/pull/488508

Change-Id: Iaa35ae3b8971867f12daeac5f94958c20a7d9233
 2026-05-31 12:27:17 +02:00
rootile c64fbcbdb2 f2/command: allow piping stderr to stdout 
Change-Id: I280d4888e9dc11cce93bacfddd9b4383a11525e1
 2026-05-27 11:49:49 +00:00
Erik Jensen eef57410d7 Fix nix-copy-closure --include-outputs 
The option has been broken since Nix 2.4. The flag was accepted, but not
used. This change plumbs it through to computeFSClosure.

Change-Id: Id6adee8ea8a6a4f457b24b650660f319e3daa2b2
 2026-05-26 19:40:07 +00:00
Linus Heckemann 8edb857248 libfetchers: lock inputs before fetching them 
Fixes #1122

Change-Id: I9243e692779a795066417a15c6315dd56a6a6964
 2026-05-26 15:14:13 +00:00
Qyriad b113be5f13 flake: provide devshells for static cross systems too 
Change-Id: I27ec97eaac88236a0958953a81d09ee16a6a6964
 2026-05-23 21:14:54 +02:00
Qyriad 720306b693 package: deduplicate python environment packages 
Change-Id: I2653d2db71a40d763d4cd3b5c4ef05306a6a6964
 2026-05-23 21:14:54 +02:00
Qyriad f6cfd6b3ea package: deduplicate pyxattr override 
Change-Id: I4c8f34cef4d09f2ccb2df23e0679addd6a6a6964
 2026-05-23 21:14:53 +02:00
Qyriad 839ae964fb justfile: clean build and outputs contents, not directories 
This makes it easier to build lix on a different file system from the
source repo, e.g. symlinking ./build to a directory in /tmp.

The ./outputs directory is included for completeness as another
significant source of generated artifacts.

Change-Id: I1bb4c21c700beacb4882a1be473dd31353edd177
 2026-05-23 21:14:53 +02:00
blokyk e5ca862ea9 libfetchers/tarball,git: log when unpacking tarballs 
this logs unpackTarfile calls as an `actUnknown` activity, which
unfortunately doesn't report any progress for now. for the reasoning
behind this choice, see the previous iterations of this CL (cl/5590)
as well as the attached comments (the crux of the conversion is in DMs,
unfortunately). in short:
- i could't make a new ActivityType because that would be a breaking
  change to the daemon protocol, including with previous lix versions
- i can't use actFileTransfer because that would show up as a download,
  which would be confusing for both users and tools
- i can't use actCopyPath /or/ actCopyPaths because that would break
  NAR/substitution accounting when doing remote builds

also, ArchiveDecompressionSource, which is apparently used for
decompressing logs etc, does its own custom decompression without using
unpackTarfile, so it is not affected by this. from what i can tell,
unpackTarfile is only used for:
- fetchtarball fetcher
- git fetcher
- channel unpacking
- nix-prefetch-url

Fixes #1213

Change-Id: I68f3be520681d8889292353dd41ab19a60c5eeea
 2026-05-23 05:21:56 +00:00
Linus Heckemann 1f100371d7 doc/hacking: switch buildkite sso url to afnix 
Change-Id: I7f7e65fe2dc52b40d4771e59198dd66a6a6a6964
 2026-05-18 18:13:47 +02:00
K900 b118785864 nix-eval-jobs: fix nested list in inputs 
Change-Id: I08b02ecb85512288b24a02a500348eaf78f030be
 2026-05-18 11:55:30 +00:00
rootile 1c81bd926d tests/f2: improve error message for failed in assertions 
By default, when one uses `in` assertions, upon failure newlines in any
of the two strings is escaped, and everything is printed in a single
line.

This commit allows one to pass `-vv` to print out the left and righthand
side unescaped, with each line of the output being its own line,
improving readability a lot when dealing with multiline outputs.

Change-Id: Iaf67a20fce57d375e2c62a8e8a0957b14eec26ef
 2026-05-18 10:47:00 +00:00
rootile b34f4d7b17 OWNERS: Do not inherit parent owners for f2/testlib 
see [lix-weekly of
2026-04-22](https://wiki.lix.systems/link/153#bkmrk-code-ownership-of-f2)

Change-Id: I3e3df0d864a5614bb66a2d756df42c63305a7c8f
 2026-05-13 22:32:30 +02:00
rootile 1971824580 OWNERS: remove Helle from test codeowners. 
See [lix-weekly of
2026-04-22](https://wiki.lix.systems/link/153#bkmrk-code-ownership-of-f2)

Change-Id: Ic4345dd70e2cf5f7dd4509b32543c1f61d76cf07
 2026-05-13 22:32:30 +02:00
Qyriad 7831c98a4d package: remove no-op deprecated 'pie' hardeningDisable 
PIE was enabled by default in GCC Nixpkgs 25.11¹, and the hardening flag
was subsequently removed² and has no effect. This commit thus fixes a
warning about it.

[1]: https://github.com/NixOS/nixpkgs/pull/439314
[2]: https://github.com/NixOS/nixpkgs/pull/442510


Change-Id: I27850f947b1888c01045af9bbd5e737f6a6a6964
 2026-05-07 11:45:08 +02:00
Qyriad cc271148b0 treewide: fix sys/poll.h -> poll.h 
poll() and pollfd are defined in poll.h per POSIX¹, and musl complains²
about it.

[1]: https://man.archlinux.org/man/poll.h.0p
[2]: https://github.com/kraj/musl/blob/8cb84492b0245d70b2cd0edd523e2b55c7ad67a9/include/sys/poll.h#L1


Change-Id: I4fff447394903cbf55e330f07ed82b906a6a6964
 2026-05-06 20:51:25 +02:00
skye 2b409ad322 meson.build: Warn on deprecated declarations 
Can finally do this after fixing all Value based deprecations.
Resolves #744

Change-Id: Ic050b42a5af140c6dc74dd1545e3f1d76a6a6964
 2026-05-06 12:08:03 -04:00
piegames 7cdda9c5fa flakes: checkOverlay: Don't check the second argument 
It's software archaeology time.

- In 2019 (Nix 2.4), Eelco added `checkOverlays`, which strictly checked
that the overlay must be a function of two arguments, one called `final`
and the other `prev`. (dc3f52a144,
gh#3573)
- In 2024, NofairKing opened an issue about checking the exact name
being a silly idea (gh#10516). Instead of trying to make the check more
sensible, the first fix attempt simply expanded the check to allow
`previous` instead of `prev` (the author's preferred name for the
attribute, supposedly; gh#10553). After some discussion, instead a PR
got merged which simply removed the `prev` check altogether, while
leaving the `final` argument check intact for some reason
(ad65a50a94a97bf1f1a1902f43542d28a2e8206b, gh#10572).
- In 2025, over at Lix and oblivious to the latest change over at
CppNix, I came across this stupid check while trying to improve the
abstractions of the AST in Nixexpr. I spent (wasted, retrospect)
considerable amounts of time removing the exact name checks while also
improving the check overall (making it check for more than two
arguments, and also generally improved error message).
(0928d4d87a,
Id4244171123dd8a228be71ce9f04d8e9f647c111)
- Fast forward to 2026, where I run into this becursed piece of code
once again, and once again during some Lixexpr cleanups. Now that I have
seen how carlessly the issue was handled at CppNix, I can't be arsed to
give a flying fuck anymore. Out the code goes. Bye.

With this commit, all casts on `Expr` subclasses outside of libexpr
itself have been removed. No more violation of abstraction boundaries.
Good riddance.

Change-Id: I939968bb01d461764cfa0f4ea7152b4fcf1acf93
 2026-05-06 17:22:19 +02:00
piegames 481fc30ae7 libexpr/flakes: Replace the AST checks with maxCallDepth = 0 
Change-Id: I7130cc941b4b7432df76a3995d298edfb28e6a01
 2026-05-06 17:22:19 +02:00
eldritch horrorsandpiegames d0a4b55a0e treewide: pour the rpc foundations 
pour the foundations for rpc, and let them set. this is very much
unstable and must be opted into with explicit store uris (e.g. by
setting `NIX_REMOTE=daemon?protocol=any`). the daemon sockets are
not enabled by default and must be enabled with the `rpc-sockets`
experimental feature. we will not advertise this just yet because
in the current state it one has to be *very* dedicated to the rpc
cause to deploy this, but once we have some more bits migrated we
may want to add release notes and officially as for beta testing.

Co-Authored-By: piegames <git@piegames.de>
Change-Id: I85a96ccb700b91190c1eb37154bcc6ae1c03401a
 2026-05-06 10:55:08 +00:00
eldritch horrors b2c95d41c9 libstore: prepare meson for more rpc definitions 
Change-Id: I4c495d574acdaa976bdd4ab51bf543aae63903af
 2026-05-05 12:03:34 +00:00
eldritch horrors 010d062926 libstore: move legacy request handler loop to new function 
splitting it off lets us reuse it (and all the legacy protocol handling
it encapsulates) in the rpc world as well by just tunneling wire bytes.
doing this allows incremental migration of to a new rpc protocol, until
we are finally ready to call the rpc protocol ready and remove the link
to the legacy protocol altogether (and maybe merge this function back).

Change-Id: I8f2a8795c4a9d713ebbbbfd02e9eec5271a2b7a9
 2026-05-05 12:03:26 +00:00
eldritch horrors af4d064be7 libstore: processConnection -> processLegacyConnection 
Change-Id: I608085c845cce05f4186f83b9ef0d84e8e4a776e
 2026-05-05 12:02:47 +00:00
eldritch horrors 7edb421e1b treewide: generalize socket handling 
this adds the protocol registry. we also extend the systemd unit setup
to allow adding new protocols without much fussing, and we move deamon
instances into protocol-dependent scopes. this will make monitoring or
resource limiting of individual protocol daemon instances much easier.

Change-Id: Ic9da08eb074868b6e57385075c5502b081e8461f
 2026-05-05 12:02:16 +00:00
eldritch horrors de3a2451eb libstore: move daemon protocol struct out of Settings 
we'll use this to create a registry of protocols the system understands.
also use this struct during connection setup to make that simpler later.

Change-Id: Ifa481fea7ea2efa2a1f1be4d81076a9e022d24f9
 2026-05-05 12:02:07 +00:00
eldritch horrors 4da9a53b66 f2: add test for daemon socket connection 
tests both order of sockets tried and that all sockets work as expected.

Change-Id: I5aeb506e159564a0b14b6fbbe5df1e5ccdb437d5
 2026-05-05 12:01:58 +00:00
skye 8ab7547a74 libexpr/json-to-value: Refactor json-to-value.cc from linked list of 
inherited classes to stack of variants

Previously the code represented the stack of unclosed arrays and
objects as a linked list of virtual objects which seemed unnecessary.
There is a closed very small (3) number of possibilities, and one of
them only exists at the top level. A vector of variants seems a lot
simpler, and as a bonus, more performant.

Change-Id: I6bc429d682d22f69ca91dc0518e6be646a6a6964
 2026-05-04 15:42:55 -04:00
eldritch horrorsandRaito Bezarius 2f7d7444f1 libutil: fix nar parser buffer overflow 
string data shares a buffer with the binary string length field. size
calculations for string read buffers always include the length field;
sufficiently large length fields can cause these calculations to wrap.
a malicious nar could use this for OOB writes in the daemon (as root).

since we use strings only as tags for archive members and for symlinks
with their OS-dependent length limits we can simply limit string size.
1 MiB should be sufficient for all symlinks, and tags are always tiny.

Change-Id: I89fb05f73c1dbeda45d91244aba4cd526a3d83e1
 2026-05-04 19:00:19 +02:00
Raito Bezarius c595477002 releng/keys: update the way to receive the ephemeral key 
Change-Id: I791f3f3f49ecd5884c9e86b5d3b617fc139e031f
Signed-off-by: Raito Bezarius <raito@lix.systems>
 2026-05-04 18:57:20 +02:00
Raito Bezarius c71d2ebace releng/environment: update staging parameters 
These parameters are now created on https://s3.afnix.fr.

Change-Id: I96b6fd913429ee46d04c412cb141edd288665ced
Signed-off-by: Raito Bezarius <raito@lix.systems>
 2026-05-04 18:57:20 +02:00
eldritch horrors 65346101a2 libstore: rename openConnectionWrapper to something more meaningful 
Change-Id: I46c57476ef73c6a8b851ab34ba67872c8f6b826b
 2026-05-04 09:44:10 +00:00
eldritch horrors 0d2c48a797 libstore: remove RemoteStore::failed 
this is not read in code paths that are called repeatedly, and the only
reader can never see it being false. we also question the wisdom of the
flag in the first place; why should opening a connection fail because a
previous connection did? conditions may have improved a lot since then!

Change-Id: Ib7c219e6239432d99a22b10024847218469f1cd4
 2026-05-04 09:44:10 +00:00
piegames 078bbe6171 libexpr/eval: Fix off by one in maxCallDepth calculation 
This isn't really relevant for a setting of 10k, but now that we want to
set it to 0 for Flakes it makes a difference as to whether or not one
top-level function call is allowed or not. (It shouldn't be)

Change-Id: I6bbf99826af0289c232b9d45d172a378b81abe4b
 2026-05-03 13:05:46 +02:00
Lunaphied cd573beb0a docs/hacking: fix to respect changes in justfile 
We no longer have setup-custom, just use setup

Change-Id: I2a151d04dc3d90a98b79859b05c4e6706a6a6964
 2026-05-01 11:12:54 +00:00
Jade Lovelace d94bb49cf4 fix: structured build logs: bad field types no longer fail builds 
Caught by edef's harness. I rewrote the structured build log tests in
f2 and added a test for this one.

Before: it failed the build altogether:
```
building '/tmp/jade/pytest-of-jade/pytest-77/test_invalid_fields_files0_0/nix/store/skhjs5zs082nqiakj69qz32gyywz0v1a-unusual-loggi
ng.drv'...
error: unsupported JSON type 7
```

Now the output is as expected:

```
warning: Unable to handle a JSON message from the derivation builder: error: unsupported log field type number
@nix {"action": "start", "fields": [1.5], "id": 2, "type": 1, "level": 1, "text": "abc"}
```

Change-Id: Idbd28c5a4d4cf15f27207f83b730e7946a6a6964
 2026-04-30 17:28:55 +00:00
piegames 076d1aa4cd eval: switch up ExprOpConcatLists error position again 
This is a minor adjustment to 2b22dae1ba /
I3e9d7c1c7a6599a8e68302448bbb961d051002b7

Basically having the error point to the operand instead of the operator
kind of makes sense, but it would require error spans to *truly* make
sense, and in the meantime maintaining the logic is more hassle than it
is worth.

Change-Id: Idac98dd77a0f4a6cb386cb74e0d4eb5fd2f503e5
 2026-04-30 17:14:43 +02:00
piegamesandeldritch horrors 8782c7533f eval: Improve ExprConcatStrings error messages and positions 
"while evaluating a path segment" was just plain wrong

Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: I827374635df2887d17a8ee8b1755c078bf0b506b
 2026-04-30 17:14:43 +02:00
rootile 7017def185 f2/testlib: support wrapping commands 
It's useful to obtain a fully assembled command and then wrap it to
execute into something else, e.g. gdb, strace or fakeroot.

Change-Id: Ida5928fd001925549eab89edfbf0facada7ef7f1
 2026-04-30 16:59:28 +02:00
Jade Lovelaceandrootile e70ae36b3f libexpr/builtins: correctly handle integer edge cases in floor/ceil 
We choose to throw an error in the invalid-input space where we *can*
produce a valid (but wrong) result to allow ourselves to, mirroring
CppNix, change the semantics to not corrupt it at all in the future,
while having a middle state where it is an error.

This is a largely-rewritten version of
https://github.com/NixOS/nix/pull/13013.

Co-authored-by: rootile <lix@rootile.de>

Fixes: https://github.com/NixOS/nix/issues/12899
Upstream-PR: https://github.com/NixOS/nix/pull/13013
Change-Id: I6a6a6964cdd1a88714952e80c660d1fa57d1f2d8
 2026-04-30 16:10:47 +02:00
Jade Lovelace 89f5974125 fix: inverted null check in nix-prefetch-url 
This was found by edef's harness. Unfortunately this entire module has
no coverage, so I don't really want to write a test for it immediately
as I'd have to harness it (though we have an http server in f2!).
(I think it will be much more fun and rewarding to do so once we get
coverage up and running).

Filed a bug about the missing docs for this too:
https://git.lix.systems/lix-project/lix/issues/1190

Test plan:

```nix
let pkgs = import <nixpkgs> {}; in {
  foo = pkgs.fetchurl {
    url = "https://jade.fyi/robots.txt";
    name = "blah";
    hash = "";
  };
}
```

Before:

```
$ nix-prefetch-url ./foo.nix -A foo
fetching path input 'path:/nix/store/knf5m0famajw2bzbsg72b052iy1ynpck-source'
path is '/nix/store/5rcgj95as1ywr463rva6bky18088ij4x-robots.txt'
1cr11i0ih4m6kkpks9fxil71wg5k2crikah1rfjgy607qzgy8vad
```

After (note correct path name):

```
$ nix-prefetch-url ./foo.nix -A foo
fetching path input 'path:/nix/store/knf5m0famajw2bzbsg72b052iy1ynpck-source'
path is '/nix/store/hc9drd1l3vki3bbaacwhi8j1dr9907vd-blah'
1cr11i0ih4m6kkpks9fxil71wg5k2crikah1rfjgy607qzgy8vad
```

Change-Id: I6d9da1c54c6a398a0a169ff01f3f3dce6a6a6964
 2026-04-30 13:19:23 +00:00
eldritch horrors faaa14a303 tests/libutil: disable guessOrInventPath sockets test 
not very useful, fails in docker and on macos. let's disable it for now.

fixes #1113

Change-Id: If83ea11ff7120eb7fb30601096374d167107573c
 2026-04-28 10:52:57 +00:00
blokyk f8b13b254b libcmd/repl: print backtraces from outer to innermost frame 
before this change, the innermost frame was all the way at the top,
which means that in very nested code you'd have to scroll though
dozens or hundreds of frames before even seeing where you are, because
the last lines (which are immediately visible when the command returns)
would be the nix code's entry point and other outer frames instead,
which is rarely relevant.

this CL reverses this order, so that the innermost frames are the last
ones printed, and thus are immediately visible. (note that this is
already how errors are printed by nix in other contexts anyway, the
debugger's :bt is the only thing that prints the trace in "forward"
order.)

because `DebugState.traces()` use the homegrown `libutil::Generator`
instead of an stdlib container, we have to do that reversing ourselves,
in this case by just storing every element in an `std::list` (in reverse
order) and then traversing it. this is feels kinda dirty, but i don't
know any other way, and it's not exactly performance-critical.

Change-Id: I9f23e40e57f72a251d65a335b3bba3c3f77b935d
 2026-04-28 12:09:13 +02:00
blokyk 68834b7841 tests/f1/repl-characterization: match golden text correctly 
it seems like the repl_characterization tests have not been actually
running correctly for a while, since tests from more than 2 years ago
that should have failed have been in the test suite and OK'd thus far.

the reason why some tests passed when they shouldn't have is because
the AST nodes used by the characterization parser didn't have a correct
comparison operator, so the tests would only fail when one of the output
line was of the wrong *kind*, but not if it had the wrong *value*. this
was fixed by simply adding a default comparator to the TextNode type as
well as every derived class.

i've taken the occasion to fix the tests that were failing now that we
are actually being strict about them. most of these are pretty boring
changes like indentation, but if possible i'd like to get an explicit
ACK from at least one or two people on the core team that these changes
are okay.

for reference, here is the zulip thread that lead to this CL:
  https://zulip.lix.systems/#narrow/channel/8-Testing-.2F-functional2/topic/.E2.9C.94.20repl-characterization.20tests.20not.20failing/near/10276

ps: i had to turn clang-format off around the macro for text node
declarations because it re-formatted the entire macro, causing a lot
of noising and producing uglier code; if this isn't okay or if there's
a better alternative, please let me know!

Change-Id: I1d6d92eb3e9cb705ecfeccd7b9294d6cfd7fe25b
 2026-04-27 23:40:56 +02:00
Yureka 0c63036c7d releng: Adapt for AFNix S3 
Change-Id: I29dbd62dcc70595ba3f2ac2a466a5c26a28aea99
 2026-04-27 11:39:58 +00:00
Qyriad de212be82b upgrade-nix: use --store local, so we don't kill our own access to the daemon 
Tested in a Debian container.

Fixes #1189
Fixes https://git.lix.systems/lix-project/lix-installer/issues/79

Change-Id: Id7de28bf81eb122dd60755622b7816716a6a6964
 2026-04-24 15:56:19 +00:00
Jade Lovelace bb8bc31e58 fix: lint warnings from our clang-tidy enum cast check 
I missed some cases in https://gerrit.lix.systems/c/lix/+/5490, which is
not surprising given that I didn't have a lint for it at the time.

Some of these have JSON deserializers, which is kind of scary (in that
it was UB). Probably no impact, again, due to the lack of
-fstrict-enums, but still, yikes!

Change-Id: I895a8bfdd9972d8ade9fa6f5aa30845d6a6a6964
 2026-04-23 16:09:33 +00:00
skye be34bc0481 testing: migrate and expand import-derivation.sh and warn-import-from-derivation.sh 
This started by noticing that the coverage report generated by cl/5489
showed that the error message for when allow-import-from-derivation is
disabled appeared to be not covered by any tests. Since the existing
tests for IFD were in functional, which is deprecated, I migrated those
into functional2 and added a test for erroring when IFD is disabled.

Change-Id: I565674de3dd693e023ec99d501adc4706a6a6964
 2026-04-22 10:26:15 -04:00
skye eeac529d5e libexpr/primops: Suppress deprecation warning in prim_importNative 
as of cl/5456 prim_importNative default constructs a `Value` in order to
pass as an out parameter to a ValueInitializer function. Default
construction is deprecated, but there isn't a meaningful value to
initialize it to instead. After cl/5357 this will give a deprecation
warning (which in an asanBuild will be an error), so this commit
suppresses that warning for that `Value` construction.

Change-Id: Ie567a6e81b5753672001320e9fc816c56a6a6964
 2026-04-22 08:39:55 -04:00
skye 47c9f73d36 doc/manual: Update signature of primop fn in hacking.md 
This should have been part of cl/5456, but I didn't notice it at the time.

Change-Id: I3afe75c5862ecd3ce2f616a2ae30aff96a6a6964
 2026-04-22 08:39:49 -04:00
skye 84347f94da libexec/kill-user: suppress syscall deprecation warning on darwin 
The existing function is used with good reason as explained by its
comment. This suppresses the warning it creates.

Change-Id: I58b0de3f5854266fa77b6a06b7821a636a6a6964
 2026-04-22 08:37:17 -04:00
skye cda9e01053 nix-eval-jobs: Remove Value default construction from releaseExprTopLevelValue 
Change-Id: I7461323771a50729c66d9e3160c4768f6a6a6964
 2026-04-22 08:37:17 -04:00
skye c40afdea73 libexpr/eval-expr: extract makeThunk function that constructs and increments counters 
Motivated by needing to remove a Value default construction from
`ExprVar::maybeThunk`, so part of #744

Change-Id: Ia12b5b019e49055ebd7ffa0cc39a59c06a6a6964
 2026-04-22 08:37:17 -04:00
vczf dda519baff gitignore: ignore top-level build and outputs directories 
Change-Id: Id51439c15c0216e7353ba48ae1940d3906473d4f
 2026-04-22 11:47:09 +00:00
Jade Lovelace af99ede1fd fix: UB casts to unspecified width enums in proto 
Found by edef's fuzzing. We::jade don't think this is security-relevant
or likely to be mistreated by a compiler, but UB is bad.

https://eel.is/c++draft/expr.static.cast#8

> If the enumeration type does not have a fixed underlying type, the value is unchanged if the original value is within the range of the enumeration values ([dcl.enum]), and otherwise, the behavior is undefined

Notably, the range is defined as the smallest bitfield type that could hold all the values, not any real type which exists: https://eel.is/c++draft/dcl.enum#8.

This is basically a footgun, and I'm writing a clang-tidy check to forbid casting to such types. But first I needed to write clang-tidy testing infrastructure: https://gerrit.lix.systems/c/lix/+/5493.

Change-Id: Ieaaa0fe2a92fd9f24f60761e312741186a6a6964
 2026-04-22 10:52:04 +00:00
eldritch horrors 0a09782cc8 testing: migrate store-ping.sh 
notably the bad-store test previously tested *bash*, not lix. we also
need to extend the list of non-fatal errors, but that's probably fine

Change-Id: I983f4adc8047fe800323099de4037711869ee547
 2026-04-20 16:08:22 +00:00
Felix Uhl e3ee75633e Fix broken user override of NIX_SSL_CERT_FILE in fish shell 
Change-Id: If15fac01104cc413102e417805c2e92aa9007eba
 2026-04-20 13:02:17 +00:00
eldritch horrors d505a477a4 libstore: asyncify RemoteStore::openConnection 
Change-Id: Icc7e705314b9a099687c37859700959139840631
 2026-04-20 09:28:24 +00:00
eldritch horrors f006b41a3c libutil: transfer bind/connect errors from libexec helper 
this should improve error messages when connecting to long socket paths,
and make it at all possible to detect ENOTDIR from connect helpers. this
will be needed to handle daemon sockets that live in directories we have
not previously considered like the protocol name directories we need for
rpc purposes, all of which would have resulted in hard errors previously

Change-Id: I4e16cf09fedd07c93cfdf7b0f151106984507315
 2026-04-20 08:19:17 +00:00
Michael Daniels a5205265c0 manual: correct broken link to Gerrit wiki 
Change-Id: Ida19e872da316c7d3170f0ef2bc60712c31b2ae1
 2026-04-20 03:30:28 +00:00
Michael Daniels 23b2797321 lix/code-generation: correct link to changes for deprecated features 
The link should be to Gerrit, not Forgejo.

Issue introduced in: https://gerrit.lix.systems/c/lix/+/5005

Change-Id: Ibf93f114303f59959621a265196e6eda29b315f9
 2026-04-19 22:47:10 +00:00
skye 0e971cddee libexpr: Delete EvalState::mkSingleDerivedPathString dead function with Value& out param 
Part of #1136, I believe this cleans up the final instance of `Value&`
used as an out parameter.

Change-Id: I40153b1f11d5ca9af06cd427f22896d06a6a6964
 2026-04-19 12:45:56 -04:00
piegames 5afda956a7 tests/functional2/lang: Some refactorings 
I did a pass through all files, fixing bits that have annoyed me here
and there based on vibes. Roughly:

- Replaced some out values like bools or strings with lists where
appropriate. Those tests were likely older than the value printer
- Added some parser tests where it makes sense
- Gently touched some formatting woes, while trying to not cause too
much diff noise
- Removed some dead `with (import ./lib.nix);` code

Change-Id: I8c40b2110f0b7799f68ae38ba61f049c5f1f6ee8
 2026-04-19 16:07:54 +02:00
rootile ae4b3d585d f2/nix: provide a high-level wrapper for calling builtins 
Change-Id: I357bd7630b96e55d000fbff1032efcf8c98dfb7b
 2026-04-19 10:10:37 +00:00
rootile 62b728e766 f2/nix: provide a function to serialise python objects to nix code 
Change-Id: I863029263edbc701a18b18da4b58fcb35f575614
 2026-04-19 10:10:22 +00:00
501 changed files with 14703 additions and 7041 deletions
Expand all files Collapse all files
.cargo/config.toml
+14
View File
@@ -0,0 +1,14 @@
[target.'cfg(true)']
rustflags = [
# rustc will pass `-nodefaultlibs` without this, but we need the C++ standard library.
'-Cdefault-linker-libraries=yes',
]
[target.'cfg(target_env = "musl")']
rustflags = [
'-Cdefault-linker-libraries=yes',
# musl, at least in Nixpkgs, is not compiled with -fPIE.
# XXX: nevermind? as of Nixpkgs 26.05??
# Oh gods do we need to gate this??
#'-Crelocation-model=static',
]
.gitignore
+2 -1
View File
@@ -1,4 +1,5 @@
outputs/
/build
/outputs
# GNU Global
GPATH
Cargo.lock
Generated
+878 -4
View File
@@ -2,18 +2,320 @@
# It is not intended for manual editing.
version = 4
[[package]]
name = "aho-corasick"
version = "1.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ddd31a130427c27518df266943a5308ed92d4b226cc639f5a8f1002816174301"
dependencies = [
"memchr",
]
[[package]]
name = "allocator-api2"
version = "0.2.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "683d7910e743518b0e34f1186f92494becacb047c7b6bf616c96772180fef923"
[[package]]
name = "anstream"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "824a212faf96e9acacdbd09febd34438f8f711fb84e09a8916013cd7815ca28d"
dependencies = [
"anstyle",
"anstyle-parse",
"anstyle-query",
"anstyle-wincon",
"colorchoice",
"is_terminal_polyfill",
"utf8parse",
]
[[package]]
name = "anstyle"
version = "1.0.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "940b3a0ca603d1eade50a4846a2afffd5ef57a9feac2c0e2ec2e14f9ead76000"
[[package]]
name = "anstyle-parse"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "52ce7f38b242319f7cabaa6813055467063ecdc9d355bbb4ce0c68908cd8130e"
dependencies = [
"utf8parse",
]
[[package]]
name = "anstyle-query"
version = "1.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc"
dependencies = [
"windows-sys",
]
[[package]]
name = "anstyle-wincon"
version = "3.0.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d"
dependencies = [
"anstyle",
"once_cell_polyfill",
"windows-sys",
]
[[package]]
name = "ar_archive_writer"
version = "0.5.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4087686b4b0a3427190bae57a1d9a478dbb2d40c5dc1bd6e2b6d797913bdd348"
dependencies = [
"object",
]
[[package]]
name = "ariadne"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "72fe02fc62033df9ba41cba57ee19acf5e742511a140c7dbc3a873e19a19a1bd"
dependencies = [
"unicode-width 0.1.14",
"yansi",
]
[[package]]
name = "askama"
version = "0.12.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b79091df18a97caea757e28cd2d5fda49c6cd4bd01ddffd7ff01ace0c0ad2c28"
dependencies = [
"askama_derive",
"askama_escape",
"humansize",
"num-traits",
"percent-encoding",
]
[[package]]
name = "askama_derive"
version = "0.12.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "19fe8d6cb13c4714962c072ea496f3392015f0989b1a2847bb4b2d9effd71d83"
dependencies = [
"askama_parser",
"basic-toml",
"mime",
"mime_guess",
"proc-macro2",
"quote",
"serde",
"syn",
]
[[package]]
name = "askama_escape"
version = "0.10.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "619743e34b5ba4e9703bba34deac3427c72507c7159f5fd030aea8cac0cfe341"
[[package]]
name = "askama_parser"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "acb1161c6b64d1c3d83108213c2a2533a342ac225aabd0bda218278c2ddb00c0"
dependencies = [
"nom",
]
[[package]]
name = "autocfg"
version = "1.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53"
[[package]]
name = "basic-toml"
version = "0.1.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ba62675e8242a4c4e806d12f11d136e626e6c8361d6b829310732241652a178a"
dependencies = [
"serde",
]
[[package]]
name = "bitflags"
version = "2.13.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b4388bee8683e3d04af747c73422af53102d2bd24d9eadb6cbc100baef4b43f8"
[[package]]
name = "block-buffer"
version = "0.10.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71"
dependencies = [
"generic-array",
]
[[package]]
name = "cc"
version = "1.2.65"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e228eec9be7c17ccb640b59b36a5cd805ea2a564a4c5e162c2f659fea30d3b96"
dependencies = [
"find-msvc-tools",
"shlex",
]
[[package]]
name = "cfg-if"
version = "1.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801"
[[package]]
name = "cfg_aliases"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724"
[[package]]
name = "chumsky"
version = "1.0.0-alpha.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0e82d74e6c83060ec269fe9e0d408d6de4a1645d525f9a0bbbb841ba4efd91ac"
dependencies = [
"hashbrown 0.15.5",
"regex-automata 0.3.9",
"serde",
"stacker",
"unicode-ident",
"unicode-segmentation",
]
[[package]]
name = "clap"
version = "4.6.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1ddb117e43bbf7dacf0a4190fef4d345b9bad68dfc649cb349e7d17d28428e51"
dependencies = [
"clap_builder",
"clap_derive",
]
[[package]]
name = "clap_builder"
version = "4.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "714a53001bf66416adb0e2ef5ac857140e7dc3a0c48fb28b2f10762fc4b5069f"
dependencies = [
"anstream",
"anstyle",
"clap_lex",
"strsim",
]
[[package]]
name = "clap_derive"
version = "4.6.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f2ce8604710f6733aa641a2b3731eaa1e8b3d9973d5e3565da11800813f997a9"
dependencies = [
"heck",
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "clap_lex"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9"
[[package]]
name = "clipboard-win"
version = "5.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bde03770d3df201d4fb868f2c9c59e66a3e4e2bd06692a0fe701e7103c7e84d4"
dependencies = [
"error-code",
]
[[package]]
name = "colorchoice"
version = "1.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570"
[[package]]
name = "countme"
version = "3.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7704b5fdd17b18ae31c4c1da5a2e0305a2bf17b5249300a9ee9ed7b72114c636"
[[package]]
name = "cpufeatures"
version = "0.2.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280"
dependencies = [
"libc",
]
[[package]]
name = "crypto-common"
version = "0.1.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a"
dependencies = [
"generic-array",
"typenum",
]
[[package]]
name = "digest"
version = "0.10.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292"
dependencies = [
"block-buffer",
"crypto-common",
]
[[package]]
name = "dissimilar"
version = "1.0.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "59f8e79d1fbf76bdfbde321e902714bf6c49df88a7dda6fc682fc2979226962d"
[[package]]
name = "either"
version = "1.16.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "91622ff5e7162018101f2fea40d6ebf4a78bbe5a49736a2020649edf9693679e"
[[package]]
name = "endian-type"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "869b0adbda23651a9c5c0c3d270aac9fcb52e8622a8f2b17e57802d7791962f2"
[[package]]
name = "equivalent"
version = "1.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f"
[[package]]
name = "error-code"
version = "3.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dea2df4cf52843e0452895c455a1a2cfbb842a1e7329671acf418fdc53ed4c59"
[[package]]
name = "expect-test"
version = "1.5.0"
@@ -24,12 +326,139 @@ dependencies = [
"once_cell",
]
[[package]]
name = "find-msvc-tools"
version = "0.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582"
[[package]]
name = "foldhash"
version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2"
[[package]]
name = "generic-array"
version = "0.14.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a"
dependencies = [
"typenum",
"version_check",
]
[[package]]
name = "hashbrown"
version = "0.14.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1"
[[package]]
name = "hashbrown"
version = "0.15.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1"
dependencies = [
"allocator-api2",
"equivalent",
"foldhash",
]
[[package]]
name = "hashbrown"
version = "0.17.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a"
[[package]]
name = "heck"
version = "0.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea"
[[package]]
name = "hex"
version = "0.4.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70"
[[package]]
name = "home"
version = "0.5.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cc627f471c528ff0c4a49e1d5e60450c8f6461dd6d10ba9dcd3a61d3dff7728d"
dependencies = [
"windows-sys",
]
[[package]]
name = "humansize"
version = "2.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6cb51c9a029ddc91b07a787f1d86b53ccfa49b0e86688c946ebe8d3555685dd7"
dependencies = [
"libm",
]
[[package]]
name = "indexmap"
version = "2.14.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9"
dependencies = [
"equivalent",
"hashbrown 0.17.1",
]
[[package]]
name = "is_terminal_polyfill"
version = "1.70.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a6cb138bb79a146c1bd460005623e142ef0181e3d0219cb493e02f7d08a35695"
[[package]]
name = "itertools"
version = "0.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b1c173a5686ce8bfa551b3563d0c2170bf24ca44da99c7ca4bfdab5418c3fe57"
dependencies = [
"either",
]
[[package]]
name = "libc"
version = "0.2.186"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66"
[[package]]
name = "libm"
version = "0.2.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981"
[[package]]
name = "licxxbridge"
version = "0.0.0"
dependencies = [
"clap",
"zngur",
]
[[package]]
name = "lix"
version = "0.0.0"
dependencies = [
"lix-doc",
"pkg-config",
"regex",
"rootcause",
"rustyline",
"rustyline-derive",
"zngur",
]
[[package]]
name = "lix-doc"
version = "0.0.1"
@@ -40,8 +469,87 @@ dependencies = [
]
[[package]]
name = "lixutil-rs"
version = "0.0.0"
name = "log"
version = "0.4.33"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad"
[[package]]
name = "memchr"
version = "2.8.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4"
[[package]]
name = "mime"
version = "0.3.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a"
[[package]]
name = "mime_guess"
version = "2.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f7c44f8e672c00fe5308fa235f821cb4198414e1c77935c1ab6948d3fd78550e"
dependencies = [
"mime",
"unicase",
]
[[package]]
name = "minimal-lexical"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "68354c5c6bd36d73ff3feceb05efa59b6acb7626617f4962be322a825e61f79a"
[[package]]
name = "nibble_vec"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "77a5d83df9f36fe23f0c3648c6bbb8b0298bb5f1939c8f2704431371f4b84d43"
dependencies = [
"smallvec",
]
[[package]]
name = "nix"
version = "0.31.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cf20d2fde8ff38632c426f1165ed7436270b44f199fc55284c38276f9db47c3d"
dependencies = [
"bitflags",
"cfg-if",
"cfg_aliases",
"libc",
]
[[package]]
name = "nom"
version = "7.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d273983c5a657a70a3e8f2a01329822f3b8c8172b73826411a55751e404a0a4a"
dependencies = [
"memchr",
"minimal-lexical",
]
[[package]]
name = "num-traits"
version = "0.2.19"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841"
dependencies = [
"autocfg",
]
[[package]]
name = "object"
version = "0.37.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ff76201f031d8863c38aa7f905eca4f53abbfa15f609db4277d44cd8938f33fe"
dependencies = [
"memchr",
]
[[package]]
name = "once_cell"
@@ -49,6 +557,108 @@ version = "1.19.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3fdb12b2476b595f9358c5161aa467c2438859caa136dec86c26fdd2efe17b92"
[[package]]
name = "once_cell_polyfill"
version = "1.70.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe"
[[package]]
name = "percent-encoding"
version = "2.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220"
[[package]]
name = "pkg-config"
version = "0.3.33"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e"
[[package]]
name = "proc-macro2"
version = "1.0.106"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934"
dependencies = [
"unicode-ident",
]
[[package]]
name = "psm"
version = "0.1.31"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "645dbe486e346d9b5de3ef16ede18c26e6c70ad97418f4874b8b1889d6e761ea"
dependencies = [
"ar_archive_writer",
"cc",
]
[[package]]
name = "quote"
version = "1.0.45"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924"
dependencies = [
"proc-macro2",
]
[[package]]
name = "radix_trie"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3b4431027dcd37fc2a73ef740b5f233aa805897935b8bce0195e41bbf9a3289a"
dependencies = [
"endian-type",
"nibble_vec",
]
[[package]]
name = "regex"
version = "1.12.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f1292b7759ae1cb9ec195452d1390a074f0cd8541ab7a5a8c31cd6db45d4a6ba"
dependencies = [
"aho-corasick",
"memchr",
"regex-automata 0.4.14",
"regex-syntax 0.8.11",
]
[[package]]
name = "regex-automata"
version = "0.3.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "59b23e92ee4318893fa3fe3e6fb365258efbfe6ac6ab30f090cdcbb7aa37efa9"
dependencies = [
"aho-corasick",
"memchr",
"regex-syntax 0.7.5",
]
[[package]]
name = "regex-automata"
version = "0.4.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6e1dd4122fc1595e8162618945476892eefca7b88c52820e74af6262213cae8f"
dependencies = [
"aho-corasick",
"memchr",
"regex-syntax 0.8.11",
]
[[package]]
name = "regex-syntax"
version = "0.7.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dbb5fb1acd8a1a18b3dd5be62d25485eb770e05afb408a9627d14d451bae12da"
[[package]]
name = "regex-syntax"
version = "0.8.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4"
[[package]]
name = "rnix"
version = "0.12.0"
@@ -58,6 +668,28 @@ dependencies = [
"rowan",
]
[[package]]
name = "rootcause"
version = "0.13.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b660d9968fae12f4e691f2b2be5d9a3a6de875300c682e8d2cb89a618dd60875"
dependencies = [
"hashbrown 0.17.1",
"indexmap",
"rootcause-internals",
"rustc-hash 2.1.3",
"triomphe",
]
[[package]]
name = "rootcause-internals"
version = "0.13.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0184f6fcff3b58b7c963aee6e3cc915c04331aa6eef974f79d7d44d21e246c24"
dependencies = [
"triomphe",
]
[[package]]
name = "rowan"
version = "0.15.16"
@@ -65,8 +697,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0a542b0253fa46e632d27a1dc5cf7b930de4df8659dc6e720b647fc72147ae3d"
dependencies = [
"countme",
"hashbrown",
"rustc-hash",
"hashbrown 0.14.5",
"rustc-hash 1.1.0",
"text-size",
]
@@ -76,8 +708,250 @@ version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "08d43f7aa6b08d49f382cde6a7982047c3426db949b1424bc4b7ec9ae12c6ce2"
[[package]]
name = "rustc-hash"
version = "2.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6b1e7f9a428571be2dc5bc0505c13fb6bf936822b894ec87abf8a08a4e51742d"
[[package]]
name = "rustyline"
version = "18.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "53f6a737db68eb1a8ccff86b584b2fc13eca6a7bb6f78ebc7c529547e3ab9684"
dependencies = [
"bitflags",
"cfg-if",
"clipboard-win",
"home",
"libc",
"log",
"memchr",
"nix",
"radix_trie",
"unicode-segmentation",
"unicode-width 0.2.2",
"utf8parse",
"windows-sys",
]
[[package]]
name = "rustyline-derive"
version = "0.12.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "64e5587417a3c4e16a4415e8d7d07f80998ed835ade621d19dfbe9fbe3205b0f"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "serde"
version = "1.0.228"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e"
dependencies = [
"serde_core",
"serde_derive",
]
[[package]]
name = "serde_core"
version = "1.0.228"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad"
dependencies = [
"serde_derive",
]
[[package]]
name = "serde_derive"
version = "1.0.228"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "sha2"
version = "0.10.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283"
dependencies = [
"cfg-if",
"cpufeatures",
"digest",
]
[[package]]
name = "shlex"
version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba"
[[package]]
name = "smallvec"
version = "1.15.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8ed6a63f02c8539c91a8685a86f4099661ba3da017932f6ebbea6de3f0fa7c90"
[[package]]
name = "stacker"
version = "0.1.24"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "640c8cdd92b6b12f5bcb1803ca3bbf5ab96e5e6b6b96b9ab77dabe9e880b3190"
dependencies = [
"cc",
"cfg-if",
"libc",
"psm",
"windows-sys",
]
[[package]]
name = "strsim"
version = "0.11.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f"
[[package]]
name = "syn"
version = "2.0.118"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1b9ae57f904213ebb649ce6895b8a66c66f0203b9319718f69a5612a065b1422"
dependencies = [
"proc-macro2",
"quote",
"unicode-ident",
]
[[package]]
name = "text-size"
version = "1.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f18aa187839b2bdb1ad2fa35ead8c4c2976b64e4363c386d45ac0f7ee85c9233"
[[package]]
name = "triomphe"
version = "0.1.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b40688ea6389c8171614b25491f71d4a27946e0c7ce2da1c6de27e25abf1a0ae"
[[package]]
name = "typenum"
version = "1.20.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20"
[[package]]
name = "unicase"
version = "2.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dbc4bc3a9f746d862c45cb89d705aa10f187bb96c76001afab07a0d35ce60142"
[[package]]
name = "unicode-ident"
version = "1.0.24"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75"
[[package]]
name = "unicode-segmentation"
version = "1.13.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c6f5d3c3b1bf09027a88a6bc961fc00497d651009560b5463668dc81b0fa87a8"
[[package]]
name = "unicode-width"
version = "0.1.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7dd6e30e90baa6f72411720665d41d89b9a3d039dc45b8faea1ddd07f617f6af"
[[package]]
name = "unicode-width"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b4ac048d71ede7ee76d585517add45da530660ef4390e49b098733c6e897f254"
[[package]]
name = "utf8parse"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821"
[[package]]
name = "version_check"
version = "0.9.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a"
[[package]]
name = "windows-link"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
[[package]]
name = "windows-sys"
version = "0.61.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc"
dependencies = [
"windows-link",
]
[[package]]
name = "yansi"
version = "0.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "09041cd90cf85f7f8b2df60c646f853b7f535ce68f85244eb6731cf89fa498ec"
[[package]]
name = "zngur"
version = "0.10.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7fc912d12934b4d04aabc52c14db6fc88ae8aa5a47902f27e8759c2de254723f"
dependencies = [
"zngur-generator",
]
[[package]]
name = "zngur-def"
version = "0.10.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8f27e49a62db537cea43a6c122ded7eda99c4ac0ca1d7cfd74f2bf6a6c88712d"
dependencies = [
"indexmap",
"itertools",
]
[[package]]
name = "zngur-generator"
version = "0.10.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5ffad8c6994c477023aba7613bdf17c470003bbd191846d3096e84ffecf86d4c"
dependencies = [
"askama",
"hex",
"indexmap",
"itertools",
"sha2",
"zngur-def",
"zngur-parser",
]
[[package]]
name = "zngur-parser"
version = "0.10.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8d66c1b85ca6eab9576df5de31758cb1f9046b4ce47b25f45dabcb0fc7ef8789"
dependencies = [
"ariadne",
"chumsky",
"itertools",
"zngur-def",
]
Cargo.toml
+23 -1
View File
@@ -1,6 +1,28 @@
[workspace]
resolver = "2"
members = ["lix/lix-doc", "lix/libutil"]
members = [
"lix/lix-doc",
"lix/lix-rs",
"tools/licxxbridge",
]
[workspace.package]
edition = "2021"
[workspace.dependencies]
clap = "4"
regex = "1.12.4"
rootcause = "0.13.0"
rustyline = "18"
rustyline-derive = "0.12"
syn = "2.0"
zngur = "0.10"
pkg-config = "0.3.33"
[profile.dev]
opt-level = 1
[profile.release]
debug = "full"
debug-assertions = true
overflow-checks = true
bench/bench.py
+2 -1
View File
@@ -10,6 +10,7 @@ import platform
import shlex
import textwrap
import dataclasses
from pathlib import Path
flake_args = ["--extra-experimental-features", "nix-command flakes"]
cases = {
@@ -18,7 +19,7 @@ cases = {
*flake_args,
"search",
"--no-eval-cache",
"github:nixos/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870",
f"path:{Path('./bench/nixpkgs/').readlink()}",
"hello",
],
"rebuild": lambda build: [
contrib/plugins/meson.build
+7 -13
View File
@@ -1,19 +1,13 @@
# Darwin: don't link liblix* into plugins (host process provides them at runtime).
# Explicitly link curl so it binds to Nix-store libcurl, not /usr/lib/libcurl.
if is_darwin
plugin_deps = [
liblix.partial_dependency(includes : true, compile_args : true),
curl,
]
else
plugin_deps = [liblix, curl]
endif
plugin_mtls_store = shared_module(
'plugin_mtls_store',
'plugin_mtls_store.cc',
dependencies : plugin_deps,
# don't link liblix* into plugins (host process provides them at runtime).
# Explicitly link curl so it binds to Nix-store libcurl, not /usr/lib/libcurl.
dependencies : [
liblix.partial_dependency(includes : true, compile_args : true),
curl,
],
install : false,
build_by_default : true,
link_args : is_darwin ? shared_module_link_args : strict_shared_module_link_args,
link_args : plugin_link_args,
)
doc/manual/anchors.jq
+1 -2
View File
@@ -24,8 +24,7 @@ def map_contents_recursively(transformer):
def process_command:
.[0] as $context |
.[1] as $body |
# XXX FUTURE: drop sections once mdBook is at 0.5.0 or above in nixpkgs
$body | (.items? // .sections) |= map(map_contents_recursively(if $context.renderer == "html" then transform_anchors_html else transform_anchors_strip end))
$body | .items |= map(map_contents_recursively(if $context.renderer == "html" then transform_anchors_html else transform_anchors_strip end))
;
process_command
doc/manual/book.toml
+3 -4
View File
@@ -31,12 +31,11 @@ command = "jq --from-file anchors.jq"
[output.markdown]
# XXX FUTURE: may be reenabled once mdBook 0.5.0 or above and matching mdbook-linkchecker are in nixpkgs
#[output.linkcheck]
[output.linkcheck2]
# no Internet during the build (in the sandbox)
#follow-web-links = false
follow-web-links = false
# mdbook-linkcheck does not understand [foo]{#bar} style links, resulting in
# excessive "Potential incomplete link" warnings. No other kind of warning was
# produced at the time of writing.
#warning-policy = "ignore"
warning-policy = "ignore"
doc/manual/change-authors.yml
+22
View File
@@ -48,6 +48,11 @@ artemist:
display_name: Artemis Tosini
forgejo: artemist
astreaprtcl:
display_name: Astreaprtcl
forgejo: astreaprtcl
github: astreaprtcl
bb010g:
display_name: Dusk Banks
forgejo: bb010g
@@ -77,6 +82,9 @@ detroyejr:
display_name: Jonathan De Troye
github: detroyejr
edef:
github: edef1c
edolstra:
display_name: Eelco Dolstra
github: edolstra
@@ -142,6 +150,9 @@ kasimeka:
forgejo: janw4ld
github: kasimeka
keysmashes:
github: keysmashes
kfears:
display_name: KFears
forgejo: kfearsoff
@@ -245,6 +256,11 @@ raito:
forgejo: raito
github: RaitoBezarius
rkjnsn:
display_name: Erik Jensen
forgejo: rkjnsn
github: rkjnsn
roberth:
display_name: Robert Hensing
github: roberth
@@ -253,6 +269,9 @@ rootile:
display_name: rootile (Rutile)
forgejo: rootile
sandydoo:
github: sandydoo
seppel3210:
github: Seppel3210
@@ -317,5 +336,8 @@ yorickvp:
yshui:
github: yshui
ysndr:
github: ysndr
zimbatm:
github: zimbatm
doc/manual/rl-next/check-netrc-or-ca-file.md
+9
View File
@@ -0,0 +1,9 @@
---
synopsis: "check for missing ca-file or netrc-file if one is specified"
cls: [5646]
category: "Improvements"
credits: [astreaprtcl]
issues: [fj#1106]
---
If the settings `ssl-cert-file` or `netrc-file` have been set by the user, check if those files actually exist and fail if they are missing.
doc/manual/rl-next/fetch-empty-hash.md
+10
View File
@@ -0,0 +1,10 @@
---
synopsis: "don't treat tarball fetches with empty or zero hash as locked"
cls: []
category: "Fixes"
credits: [horrors]
issues: [fj#1233]
---
Lix no longer treats tarball fetches with empty or zero hashes as locked.
All such fetches are now also affected by `tarball-ttl` as a consequence.
doc/manual/rl-next/floor-ceil-corrupt-integers.md
+13
View File
@@ -0,0 +1,13 @@
---
synopsis: builtins.floor/builtins.ceil handle out-of-range inputs correctly
issues: [nix#12899]
cls: [3923]
prs: [nix#13013]
category: "Breaking Changes"
credits: [jade, nan-git, rootile]
---
Previously, `builtins.floor` and `builtins.ceil` always cast the input into a floating point value before running the operation and casting the floating point result back into an integer.
No checks were made for precision loss in either coercing integer inputs or converting the output to an integer (and in fact in the latter case, invoked undefined behaviour).
Now, Lix checks for precision loss on integer input (to avoid a silent eval semantics change if we were to simply pass it through as-is) and on integer output.
If your code fails to evaluate after this change, use `--extra-deprecated-features floor-ceil-corrupt-integers`.
doc/manual/rl-next/libfetchers-lock-cache-input.md
+21
View File
@@ -0,0 +1,21 @@
---
synopsis: "Use a lock when fetching inputs"
issues: [1122]
cls: [5438]
category: "Fixes"
credits: [lheckemann]
---
Up to now, attempting to fetch the same git input from multiple processes
concurrently when the input is not yet cached presented multiple issues:
- If the input was not already present, it would unnecessarily be fetched
multiple times;
- Access to the fetcher cache database was contentious, and could lead to
evaluation or flake locking failing unnecessary because the fetcher cache
was locked.
We now acquire a lock on a path based on a hash of the input specification
before accessing the fetcher db, reducing contention significantly, and
preventing more than one process from fetching the same path at the same time.
doc/manual/rl-next/mimalloc.md
+16
View File
@@ -0,0 +1,16 @@
---
synopsis: "Use mimalloc for faster evaluation"
cls: [5645]
category: Features
credits: [getchoo, lovesegfault]
---
Lix now links with [mimalloc](https://github.com/microsoft/mimalloc),
replacing the system's default `malloc()` for all non-GC allocations.
This yields a **5–12% wall-clock improvement** on evaluation workloads,
ranging from `nix-instantiate hello` to `nix-env -qa` and full NixOS
configurations.
The allocator can be disabled at build time with `-Dmimalloc=disabled`,
or by passing the `useMimalloc = false` override to the `lix` package.
doc/manual/rl-next/nej-apply-flag.md
+12
View File
@@ -0,0 +1,12 @@
---
synopsis: "nix-eval-jobs support `--apply` flag"
cls: [5748]
category: "Features"
credits: [isabelroses,mic92,ysndr]
issues: [fj#1214]
---
`nix-eval-jobs` now supports the `--apply` flag. With this you can apply the
provided function to the each derivation, the result of this function will then
be serialized as a JSON value and stored inside `"extraValue"` key of the json
line output.
doc/manual/rl-next/nix-copy-closure-include-outputs.md
+10
View File
@@ -0,0 +1,10 @@
---
synopsis: "Fix `nix-copy-closure --include-outputs`"
issues: [gh#5105]
cls: [5588]
category: "Fixes"
credits: [rkjnsn]
---
The `--include-outputs` flag for `nix-copy-closure` now works as intended.
Previously, the option was accepted but silently ignored.
doc/manual/rl-next/nix-doctor.md
+14
View File
@@ -0,0 +1,14 @@
---
synopsis: "Improve nix doctor"
cls: [5316, 5317, 5318, 5319, 5320, 5768, 5829]
category: Features
credits: [rootile, raito]
---
The `nix doctor` diagnosics interface now provides a lot more useful information including, but not limited to:
- General system information (OS, Hardware etc)
- Nix Information like Sandbox, Version, Store, State and other directories
- Flake registry
- Search path Information
- Nixpkgs provenance
- Remote builder configuration (including remote connection)
- fix crash when having relative Paths in PATH
doc/manual/rl-next/no-max-connections.md
+11
View File
@@ -0,0 +1,11 @@
---
synopsis: "Remove `max-connections` store parameters for `ssh://` and `ssh-ng://` stores"
cls: []
category: Miscellany
credits: [horrors]
---
The `max-connections` parameter was undocumented, untested, and (in the case of `ssh`) even ignored
entirely for remote builds. During a survey of public nixos configurations we have found *two* uses
of `max-connections` for `ssh-ng`, and none at all for `ssh`. Since it is so rarely used but brings
significant internal complexity that hinders improvements we have decided to remove these features.
doc/manual/rl-next/repl-reverse-bt.md
+17
View File
@@ -0,0 +1,17 @@
---
synopsis: "Print REPL backtraces in more convenient order"
issues: []
cls: [5491]
category: "Improvements"
credits: [blokyk]
---
When using the debugger, stack traces printed with the `:bt` command were
previously printed in reverse order compared to most other situations where they
appeared: the current stack frame would be printed at the very top, with the
most outer frame at the bottom, meaning that you'd have to scroll up to get a
sense of where you are.
With this change, the stack frames are printed such that the most relevant ones
are immediatly visible at the bottom, just like other traces in lix (e.g.
ones caused by errors).
doc/manual/rl-next/rustyline.md
+12
View File
@@ -0,0 +1,12 @@
---
synopsis: "REPL now uses rustyline"
cls: [5703]
category: "Improvements"
credits: [horrors]
issues: []
---
The REPL now uses [rustyline](https://github.com/kkawakam/rustyline) for input processing instead
of editline. This comes with some improvements to REPL behavior: wrapping lines no longer confuse
the line editor, unicode is fully supported, pasting multiline expressions is noew possible, even
undo commands are now available! We plan to improve the REPL further using these newfound powers.
doc/manual/rl-next/structuredattrs-url-guesswork.md
+11
View File
@@ -0,0 +1,11 @@
---
synopsis: "Hash mismatch diagnostics now work with `structuredAttrs`"
issues: [fj#1175]
cls: [5441]
category: Fixes
credits: [keysmashes]
---
Nixpkgs fetchers like `fetchurl` now use `structuredAttrs`, which broke the
hash mismatch diagnostics added in Lix 2.91. This has been fixed and the likely
URL is now shown again.
doc/manual/rl-next/trivial-flakes.md
+18
View File
@@ -0,0 +1,18 @@
---
synopsis: "Changes to `flake.nix` validation"
cls: [5523]
category: "Breaking Changes"
credits: [piegames, Qyriad, horrors]
issues: [gh#4945]
---
Flakes try to keep their inputs and metadata "simple", to make sure no unbounded computation may happen when calling e.g. `nix flake show`.
Those checks were haphazard, a maintenance burden, and also easily circumventable.
Lix has now replaced all the old checks by a simple rule: **No function calls outside of `outputs`.**
This is easier to reason about than the previous set of inconsistent rules, and crucially now also allows syntax features that users felt like they *should* have worked in the past, like let bindings.
However, some warts still remain for now: Some syntax constructs like `-1` internally desugar to `__sub 0 1`, which is a function call and thus remains forbidden.
This will be rectified as soon as the deprecation period of the respective anti-features has been completed.
This change is **breaking** in the sense that flakes which are written with the newly allowed language features will not evaluate with an older Lix version which still uses the old, more restrictive checks.
Crucially, this also affects **all transitive dependants** of such Flakes.
doc/manual/rl-next/unsigned-overflow-vuln.md
+28
View File
@@ -0,0 +1,28 @@
---
synopsis: "Fix unsigned overflow leading to out-of-band write in the NAR parser"
cls: [5554]
category: "Fixes"
credits: [horrors, raito, edef, sandydoo]
issues: []
---
The NAR parser contained an unsigned integer overflow that could be used by an
attacker to write arbitrary data to an unknown memory location and possibly
achieve code execution. A successful attack on the system-wide Lix daemon
could lead to privilege escalation to root. Any process that involves NAR
serialization could trigger this issue, including (but not limited to)
- local user interaction, whether the users are trusted or untrusted
- malicious substituters sending malformed NARs
- remote builders sending malformed build results
- remote daemons sending malformed inputs when requesting remote builds
Successful attacks using this bug require ASLR weakening of some sort, whether
by architecture constraints (e.g. on 32 bit systems, where little randomization
is possible) or system configuration (e.g. low ASLR entropy when loading
libraries), and millions of attempts. Local attacks can be mounted in less than
an hour. Remote builds typically require a fresh SSH connection for each build
and are thus less susceptible. Only one attempt can be made by substituters for
every build using substituters, they are thus not a likely vector for attacks.
At the time of writing, MITRE has not assigned this a CVE yet.
doc/manual/rl-next/user-traces.md
+23
View File
@@ -0,0 +1,23 @@
---
synopsis: "Always print frames from `addErrorContext` in error traces"
cls: [5847]
category: "Improvements"
credits: [blokyk]
issues: []
---
The [`builtins.addErrorContext`](@docroot@/language/builtins.md#builtins-addErrorContext)
function allows an author to add artificial stack frames with custom messages to
help end-users understand the context of an error and the path the code took to
get there, without having to read and understand the original source code. A
particularly notable user of this is the Nixpkgs module system, which adds
custom frames detailing what option it's evaluating or which definition it's
looking at.
However, previously, these frames would end up treated just as any other,
meaning they would most often not be visible without `--show-trace`; yet, using
`--show-trace`, they would be drowned out in the noise of the hundreds of other
frames, rendering them just as unusable.
With this change, these frames are now unconditionally shown, even without
`--show-trace`, which makes basic error traces much more informative.
doc/manual/src/contributing/hacking.md
+6 -6
View File
@@ -51,7 +51,7 @@ $ nix-shell -A native-clangStdenvPackages
### Building from the development shell
We have a [justfile](just.systems) for extra convenient building.
We have a [justfile](https://just.systems/) for extra convenient building.
It defaults to using `./build` as the build directory, and `$out` (`./outputs/out`) as the install directory.
For most cases, you can clean-build, install, and run the tests with:
@@ -183,13 +183,13 @@ The development shell on Linux uses LLD by default for faster link times.
This is set using `mesonFlags`, so to override it, you can simplify re-specify the linker to Meson:
```bash
$ just setup-custom -Dc_link_args=-fuse-ld=ld -Dcpp_link_args=-fuse-ld=ld
$ just setup -Dc_link_args=-fuse-ld=ld -Dcpp_link_args=-fuse-ld=ld
```
While using LLD, you may find it helpful to use ThinLTO for even further improvements to link times for incremental builds:
```bash
$ just setup-custom -Db_lto=true -Db_lto_mode=thin -Db_thinlto_cache=true
$ just setup -Db_lto=true -Db_lto_mode=thin -Db_thinlto_cache=true
```
## Sending changes to Gerrit for review {#sending-to-gerrit}
@@ -200,7 +200,7 @@ Our instance is at <https://gerrit.lix.systems>.
There's much more information about how to use Gerrit in the [wiki section on Gerrit][wiki-gerrit] including how to use Jujutsu, how to use the UI and more.
The Snix project also has some Gerrit information [in their contributing docs][snix-gerrit].
[wiki-gerrit]: https://wiki.lix.systems/books/lix-contributors/chapter/gerrit
[wiki-gerrit]: https://wiki.lix.systems/books/contributing/chapter/intro-to-gerrit
[snix-gerrit]: https://snix.dev/docs/guides/contributing/
The gist is that once you have your SSH key and git remote set up, you can send commits for review with:
@@ -226,7 +226,7 @@ The `Code-Review+2` from before will stick around through trivial rebases so no
We use Buildkite for our CI, usually you will not have to interact directly with it other than reviewing any errors it produces, which are linked from Gerrit.
However in certain cases a CI run will fail due to transient issues not related to your code and you will need to rerun it by hand.
You can log in to the CI via [SSO](https://buildkite.com/sso/lix-project). On your job you can then hit the "Retry failed" button to rerun it, normally you will not have a repeat of the transient issue.
You can log in to the CI via [SSO](https://buildkite.com/sso/afnix). On your job you can then hit the "Retry failed" button to rerun it, normally you will not have a repeat of the transient issue.
If the build still fails on CI issues or all builds are failing this should be reported via [Zulip on #T-infra](https://zulip.lix.systems/#narrow/channel/7-T-infra) or [Matrix on #dev](https://matrix.to/#/%23dev%3Alix.systems?via=lix.systems).
## Building Lix with `nix`
@@ -566,7 +566,7 @@ Then add the new file there, and don't forget to register it in the appropriate
The following metadata properties are supported for builtin functions:
* `name` (required): the language-facing name (as a member of the `builtins` attribute set) of the function.
* `implementation` (optional): a C++ expression specifying the implementation of the builtin.
It must be a function of signature `void(EvalState &, PosIdx, Value * *, Value &)`.
It must be a function of signature `Value(EvalState &, PosIdx, Value * *)`.
If not specified, defaults to `prim_${name}`.
* `renameInGlobalScope` (optional): whether the definition should be "hidden" in the global scope by prefixing its name with two underscores.
If not specified, defaults to `true`.
doc/manual/src/installation/prerequisites-source.md
-4
View File
@@ -50,10 +50,6 @@ The most current alternative to this section is to read `package.nix` and see wh
- The `boost` library of version 1.66.0 or higher. It can be obtained
from the official web site <https://www.boost.org/>.
- The `editline` library of version 1.14.0 or higher. It can be
obtained from the its repository
<https://github.com/troglobit/editline>.
- Recent versions of Bison and Flex to build the parser. (This is
because Nix needs GLR support in Bison and reentrancy support in
Flex.) For Bison, you need version 2.6, which can be obtained from
doc/manual/src/language/values.md
+1 -1
View File
@@ -164,7 +164,7 @@ Note that lists are only lazy in values, and they are strict in length.
An attribute set is a collection of name-value-pairs (called *attributes*) enclosed in curly brackets (`{ }`).
An attribute name can be an identifier or a [string](#type-string).
An attribute name can be an identifier or a [double-quoted string](#type-string).
An identifier must start with a letter (`a-z`, `A-Z`) or underscore (`_`), and can otherwise contain letters (`a-z`, `A-Z`), numbers (`0-9`), underscores (`_`), apostrophes (`'`), or dashes (`-`).
> *name* = *identifier* | *string* \
doc/manual/substitute.py
-5
View File
@@ -70,11 +70,6 @@ def do_include(content: str, relative_md_path: Path, source_root: Path, search_p
def recursive_replace(data, book_root, search_path):
match data:
# XXX FUTURE: drop sections once mdBook is at 0.5.0 or above in nixpkgs
case {'sections': sections}:
return data | dict(
sections = [recursive_replace(section, book_root, search_path) for section in sections],
)
case {'items': items}:
return data | dict(
items = [recursive_replace(item, book_root, search_path) for item in items],
flake.lock
Generated
+4 -4
View File
@@ -106,16 +106,16 @@
},
"nixpkgs_2": {
"locked": {
"lastModified": 1773082486,
"narHash": "sha256-TKUDrM0nKUo5s/b8jhjXa2prcu5KU5Cck3HBTRLDjfo=",
"lastModified": 1783770249,
"narHash": "sha256-K8pGvFito5dp9T0+clr60q+bJPGEskK75aAJ39w7HBM=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "7f8b8875bdb38a70c7b5ceb9ba6a6a8d69859e16",
"rev": "62463162b3ce92919f19ada41a70a0d943a08da8",
"type": "github"
},
"original": {
"owner": "NixOS",
"ref": "nixos-25.11-small",
"ref": "nixos-26.05-small",
"repo": "nixpkgs",
"type": "github"
}
flake.nix
+56 -435
View File
@@ -2,7 +2,7 @@
description = "Lix: A modern, delicious implementation of the Nix package manager";
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.11-small";
nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05-small";
nixpkgs-regression.url = "github:NixOS/nixpkgs/215d4d0fd80ca5163643b03a33fde804a29cc1e2";
# Required because Nix 2.18 is not in Nixpkgs ≥ 25.05 anymore.
@@ -42,6 +42,7 @@
let
inherit (nixpkgs) lib;
lixSrc = self;
# This notice gets echoed as a dev shell hook, and can be turned off with
# `touch .nocontribmsg`
@@ -76,206 +77,35 @@
(Run `touch .nocontribmsg` to hide this message.)
'';
versionJson = builtins.fromJSON (builtins.readFile ./version.json);
officialRelease = versionJson.official_release;
scope = import ./nix-support/build/inputs.nix {
inherit
lib
nixpkgs
nix_2_18
nix2container
lixSrc
nixpkgs-regression
;
};
# Set to true to build the release notes for the next release.
buildUnreleasedNotes = true;
inherit (scope)
crossSystems
darwinSystems
forAllStdenvs
forAllSystems
forAvailableSystems
linux64BitSystems
nixpkgsFor
overlayFor
systems
versionSuffix
;
versionSuffix =
if officialRelease then
""
else
"pre${
builtins.substring 0 8 (self.lastModifiedDate or self.lastModified or "19700101")
}-dev_${self.shortRev or "dirty"}";
linux32BitSystems = [ "i686-linux" ];
linux64BitSystems = [
"x86_64-linux"
"aarch64-linux"
];
linuxSystems = linux32BitSystems ++ linux64BitSystems;
darwinSystems = [
"x86_64-darwin"
"aarch64-darwin"
];
nonDarwinSystems = linuxSystems;
systems = linuxSystems ++ darwinSystems;
# If you add something here, please update the list in doc/manual/src/contributing/hacking.md.
# Thanks~
crossSystems = [
"armv6l-linux"
"armv7l-linux"
"riscv64-linux"
"aarch64-linux"
"x86_64-freebsd"
# FIXME: broken dev shell due to python
# "x86_64-netbsd"
];
stdenvs = [
# see assertion in package.nix why these two are disabled
# "stdenv"
# "gccStdenv"
"clangStdenv"
"libcxxStdenv"
"ccacheStdenv"
];
forAllSystems = lib.genAttrs systems;
# Same as forAllSystems, but removes nulls, in case something is broken
# on that system.
forAvailableSystems =
f: lib.filterAttrs (name: value: value != null && value != { }) (forAllSystems f);
forAllCrossSystems = lib.genAttrs crossSystems;
forAllStdenvs =
f:
lib.listToAttrs (
map (stdenvName: {
name = "${stdenvName}Packages";
value = f stdenvName;
}) stdenvs
)
// {
# TODO delete this and reënable gcc stdenvs once gcc compiles kj coros correctly
stdenvPackages = f "clangStdenv";
};
# Memoize nixpkgs for different platforms for efficiency.
nixpkgsFor = forAllSystems (
system:
let
make-pkgs =
crossSystem: stdenv:
import nixpkgs {
localSystem = {
inherit system;
};
crossSystem = if crossSystem == null then null else { system = crossSystem; };
overlays = [ (overlayFor (p: p.${stdenv})) ];
};
stdenvs = forAllStdenvs (make-pkgs null);
native = stdenvs.stdenvPackages;
in
{
inherit stdenvs native;
static = native.pkgsStatic;
cross = forAllCrossSystems (crossSystem: make-pkgs crossSystem "clangStdenv");
}
);
overlayFor =
getStdenv: final: prev:
let
currentStdenv = getStdenv final;
in
{
nixStable = prev.nix;
nixVersions = prev.nixVersions // {
# Nix 2.18 has been removed from Nixpkgs ≥ 25.05, so we need to reintroduce it ourselves for our tests.
nix_2_18 =
nix_2_18.outputs.packages.${currentStdenv.hostPlatform.system}.default.overrideAttrs
(_: {
pname = "nix";
});
};
check-headers = final.buildPackages.callPackage ./maintainers/check-headers.nix { };
check-syscalls = final.buildPackages.callPackage ./maintainers/check-syscalls.nix { };
default-busybox-sandbox-shell = final.busybox.override {
useMusl = true;
enableStatic = true;
enableMinimal = true;
extraConfig = ''
CONFIG_FEATURE_FANCY_ECHO y
CONFIG_FEATURE_SH_MATH y
CONFIG_FEATURE_SH_MATH_64 y
CONFIG_ASH y
CONFIG_ASH_OPTIMIZE_FOR_SIZE y
CONFIG_ASH_ALIAS y
CONFIG_ASH_BASH_COMPAT y
CONFIG_ASH_CMDCMD y
CONFIG_ASH_ECHO y
CONFIG_ASH_GETOPTS y
CONFIG_ASH_INTERNAL_GLOB y
CONFIG_ASH_JOB_CONTROL y
CONFIG_ASH_PRINTF y
CONFIG_ASH_TEST y
'';
};
nix = final.callPackage ./package.nix {
inherit versionSuffix officialRelease;
stdenv = currentStdenv;
busybox-sandbox-shell = final.busybox-sandbox-shell or final.default-busybox-sandbox-shell;
# See below
lowdown = final.lowdown_3_0;
lowdown-unsandboxed = final.lowdown_3_0.override { enableDarwinSandbox = false; };
};
lix-clang-tidy = final.callPackage ./subprojects/lix-clang-tidy { };
nix-eval-jobs = final.callPackage ./subprojects/nix-eval-jobs {
stdenv = currentStdenv;
srcDir = ./subprojects/nix-eval-jobs;
};
# HACK: We need nix-prefetch-git for fetchCargoVendor for Rust stuff,
# so it can't use Lix, or we infrec:
# lix -> Rust stuff -> fetchCargoVendor -> nix-prefetch-git -> nix (lix)
# This will eventually become a problem upstream, but until then,
# apply some duct tape and pray.
nix-prefetch-git =
if (lib.functionArgs prev.nix-prefetch-git.override) ? "nix" then
prev.nix-prefetch-git.override { nix = prev.nix; }
else
prev.nix-prefetch-git;
# Export the patched version of boehmgc that Lix uses into the overlay
# for consumers of this flake.
boehmgc-nix = final.nix.passthru.boehmgc-nix;
# And same thing for our build-release-notes package.
build-release-notes = final.nix.passthru.build-release-notes;
# As soon as Nixpkgs updates to >= 3.0.0, change to lowdown_2_0!
# We don't change the default version in order to not change the hash
# of Nix/Lix from upstream Nixpkgs.
lowdown_3_0 =
if (lib.versions.major prev.lowdown.version == "3") then
prev.lowdown
else
prev.lowdown.overrideAttrs (
finalAttrs: _prevAttrs: {
version = "3.0.0";
src = final.fetchurl {
url = "https://kristaps.bsd.lv/lowdown/snapshots/lowdown-${finalAttrs.version}.tar.gz";
sha512 = "94e97234d598382c3c3dc27f9bfdb3a3a2fcf7dbb6a8df3c85ee09f27f792449034a41d49d9cfd3d8450d2de01b8562c20c3d120e65c81af4d7d6c9454119e93";
};
}
);
capnproto = prev.capnproto.overrideAttrs (old: {
patches =
old.patches or [ ]
++ [
# backport of https://github.com/capnproto/capnproto/pull/1810
./misc/capnproto-promise-nodiscard.patch
]
++ lib.optionals (lib.versionOlder old.version "1.2.0") [
# backport of https://github.com/capnproto/capnproto/pull/2296
./misc/capnproto-monotonic-clocks-are-a-lie.patch
];
});
};
inherit (scope.callPackage ./nix-support/build/outputs.nix { })
packages
ciArtifacts
tests
;
in
{
# for repl debugging
@@ -285,202 +115,13 @@
# 'nix.perl-bindings' packages.
overlays.default = overlayFor (p: p.clangStdenv);
hydraJobs = {
# Aggregate job that is finished in Hydra _after_ all constituent jobs (here: grouped by system)
# succeed.
# This is used to run CD scripts once all builds are finished on Hydra.
release = forAllSystems (
system:
let
pkgs = nixpkgsFor.${system}.native;
in
pkgs.runCommand "release"
{
_hydraAggregate = true;
constituents = lib.filter (x: x != null) (
lib.mapAttrsToListRecursiveCond
(_: val: !(lib.isDerivation val || builtins.any (system': val ? ${system'}) systems))
(
path: drv:
if drv ? ${system} then
lib.concatStringsSep "." (path ++ [ system ])
else if drv.system or null == system then
lib.concatStringsSep "." path
else
null
)
(
removeAttrs self.hydraJobs [
"devShell"
"release"
"rl-next"
]
)
);
}
''
touch $out
''
);
# Binary package for various platforms.
build = forAllSystems (system: self.packages.${system}.nix);
# Ensure support for lowdown < 3.0 doesn't regress for NixOS 25.11
build-lowdown_2_0 = lib.genAttrs [ "aarch64-linux" ] (
system:
assert lib.versionOlder nixpkgsFor.${system}.native.lowdown.version "3.0.0";
self.packages.${system}.nix.override {
lowdown = nixpkgsFor.${system}.native.lowdown;
lowdown-unsandboxed = nixpkgsFor.${system}.native.lowdown-unsandboxed;
}
);
buildStatic = lib.genAttrs linux64BitSystems (system: self.packages.${system}.nix-static);
hydraJobs = ciArtifacts // {
devShell = forAllSystems (system: {
default = self.devShells.${system}.default;
clang = self.devShells.${system}.native-clangStdenvPackages;
});
rl-next = forAllSystems (
system:
let
rl-next-check =
name: dir:
let
pkgs = nixpkgsFor.${system}.native;
in
pkgs.buildPackages.runCommand "test-${name}-release-notes" { } ''
LANG=C.UTF-8 ${lib.getExe pkgs.build-release-notes} --change-authors ${./doc/manual/change-authors.yml} ${dir} >$out
'';
in
{
user = rl-next-check "rl-next" ./doc/manual/rl-next;
}
);
# Completion tests for the Nix REPL.
repl-completion = forAllSystems (
system: nixpkgsFor.${system}.native.callPackage ./tests/repl-completion.nix { }
);
# Perl bindings for various platforms.
perlBindings = forAllSystems (system: nixpkgsFor.${system}.native.nix.passthru.perl-bindings);
# nix-eval-jobs can be built against this Lix.
nix-eval-jobs = forAllSystems (system: nixpkgsFor.${system}.native.nix-eval-jobs);
# Binary tarball for various platforms, containing a Nix store
# with the closure of 'nix' package.
binaryTarball = forAllSystems (system: nixpkgsFor.${system}.native.nix.passthru.binaryTarball);
# docker image with Lix inside
dockerImage = lib.genAttrs linux64BitSystems (system: self.packages.${system}.dockerImage);
# API docs for Nix's unstable internal C++ interfaces.
internal-api-docs =
let
nixpkgs = nixpkgsFor.x86_64-linux.native;
inherit (nixpkgs) pkgs;
nix = pkgs.callPackage ./package.nix {
inherit versionSuffix officialRelease buildUnreleasedNotes;
inherit (pkgs) build-release-notes;
# Required since we don't support gcc stdenv
stdenv = pkgs.clangStdenv;
internalApiDocs = true;
busybox-sandbox-shell = pkgs.busybox-sandbox-shell;
};
in
nix.overrideAttrs (prev: {
# This Hydra job is just for the internal API docs.
# We don't need the build artifacts here.
dontBuild = true;
doCheck = false;
doInstallCheck = false;
});
# System tests.
tests =
import ./tests/nixos {
inherit
self
lib
nixpkgs
nixpkgsFor
;
}
// {
# the n-e-j test suite is unusably slow in darwin ci. disbled until anywho fixes this.
nix-eval-jobs = (lib.genAttrs nonDarwinSystems) (
system: self.packages.${system}.nix-eval-jobs.tests.nix-eval-jobs
);
# This is x86_64-linux only, just because we have significantly
# cheaper x86_64-linux compute in CI.
# It is clangStdenv because clang's sanitizers are nicer.
asanBuild = self.packages.x86_64-linux.nix-clangStdenv.override {
# Improve caching of non-code changes by not changing the
# derivation name every single time, since this will never be seen
# by users anyway.
versionSuffix = "";
sanitize = [
"address"
"undefined"
];
# it is very hard to make *every* CI build use this option such
# that we don't wind up building Lix twice, so we do it here where
# we are already doing so.
werror = true;
};
# Although this might be nicer to do with pre-commit, that would
# require adding 12MB of nodejs to the dev shell, whereas building it
# in CI with Nix avoids that at a cost of slower feedback on rarely
# touched files.
jsSyntaxCheck =
let
nixpkgs = nixpkgsFor.x86_64-linux.native;
inherit (nixpkgs) pkgs;
docSources = lib.fileset.toSource {
root = ./doc;
fileset = lib.fileset.fileFilter (f: f.hasExt "js") ./doc;
};
in
pkgs.runCommand "js-syntax-check" { } ''
find ${docSources} -type f -print -exec ${pkgs.nodejs-slim}/bin/node --check '{}' ';'
touch $out
'';
# clang-tidy run against the Lix codebase using the Lix clang-tidy plugin
clang-tidy = forAllSystems (
system:
let
pkgs = nixpkgsFor.${system}.native;
in
pkgs.callPackage ./package.nix {
# Required since we don't support gcc stdenv
stdenv = pkgs.clangStdenv;
versionSuffix = "";
lintInsteadOfBuild = true;
}
);
# Make sure that nix-env still produces the exact same result
# on a particular version of Nixpkgs.
evalNixpkgs = nixpkgsFor.x86_64-linux.native.callPackage ./tests/nixpkgs/eval.nix {
inherit nixpkgs-regression;
};
nixpkgsLibTests = forAllSystems (
system:
nixpkgsFor.${system}.native.callPackage ./tests/nixpkgs/lib.nix {
inherit nixpkgs system;
inherit (self.packages.${system}) nix;
}
);
};
inherit tests;
pre-commit = forAvailableSystems (
system:
@@ -540,45 +181,7 @@
}
);
packages = forAllSystems (
system:
rec {
inherit (nixpkgsFor.${system}.native) nix;
default = nix;
inherit (nixpkgsFor.${system}.native) lix-clang-tidy nix-eval-jobs;
}
// (
lib.optionalAttrs (builtins.elem system linux64BitSystems) {
# python doesn't work in static builds as of 2025-06-27
nix-static = nixpkgsFor.${system}.static.nix.overrideAttrs (_: {
doCheck = false;
});
dockerImage =
let
pkgs = nixpkgsFor.${system}.native;
nix2container' = import nix2container { inherit pkgs; };
in
import ./docker.nix {
inherit pkgs;
nix2container = nix2container'.nix2container;
tag = pkgs.nix.version;
};
}
// builtins.listToAttrs (
map (crossSystem: {
name = "nix-${crossSystem}";
value = nixpkgsFor.${system}.cross.${crossSystem}.nix;
}) crossSystems
)
// builtins.listToAttrs (
map (stdenvName: {
name = "nix-${stdenvName}";
value = nixpkgsFor.${system}.stdenvs."${stdenvName}Packages".nix;
}) stdenvs
)
)
);
inherit packages;
devShells =
let
@@ -611,12 +214,30 @@
in
(makeShells "native" nixpkgsFor.${system}.native)
// (makeShells "static" nixpkgsFor.${system}.static)
// (forAllCrossSystems (
crossSystem:
let
pkgs = nixpkgsFor.${system}.cross.${crossSystem};
in
makeShell pkgs pkgs.clangStdenv
// (lib.listToAttrs (
# Provide e.g., both '.#native-aarch64-linux` and `.#static-aarch64-linux`,
# for each cross-system.
# "native" feels like a misnomer here since it's literally cross compiling,
# but at least it's consistent with the native/static dichotomy we've set up.
lib.concatMap (
crossSystem:
let
pkgs = nixpkgsFor.${system}.cross.${crossSystem};
inherit (pkgs) pkgsStatic;
native = makeShell pkgs pkgs.clangStdenv;
static = makeShell pkgsStatic pkgsStatic.clangStdenv;
in
[
{
name = "native-${crossSystem}";
value = native;
}
{
name = "static-${crossSystem}";
value = static;
}
]
) crossSystems
))
// {
default = self.devShells.${system}.native-clangStdenvPackages;
justfile
+26 -8
View File
@@ -3,6 +3,14 @@
# Take a look at ./doc/manual/src/contributing/hacking.md for a detailed
# explanation on how to use this file!
# Pin the shell to bash (anything sufficiently POSIX-y would do)
# HACK: We use https://github.com/casey/just#positional-arguments
# and `"@$"` to forward arguments to the inner commands.
# The reason we require this is that `{{ OPTIONS }}` does not escape any values,
# and thus requires one additional level of escaping when running `just` commands with e.g. spaces in them.
# just provides no good solution to this problem, so we have to rely on its forwarding of arguments and shell semantics.
set shell := ["bash", "-uc"]
outdir := x"${out:-$PWD/outputs/out}"
builddir := "build"
@@ -10,28 +18,33 @@ builddir := "build"
list:
just --list
# Clean build artifacts
# Clean build artifacts and outputs.
clean:
rm -rf {{ builddir }}
rm -rf {{ quote(builddir) }}/* {{ quote(builddir) }}/.* {{ quote(outdir) }}/* {{ quote(outdir) }}/.*
cargo clean
# Prepare meson for building.
[positional-arguments]
setup *OPTIONS:
meson setup {{ builddir }} --reconfigure --prefix="{{outdir}}" $mesonFlags {{ OPTIONS }}
meson setup {{ builddir }} --reconfigure --prefix="{{outdir}}" $mesonFlags "$@"
# Build lix with extra options
[positional-arguments]
build *OPTIONS:
meson compile -C {{ builddir }} {{ OPTIONS }}
meson compile -C {{ builddir }} "$@"
alias compile := build
# `meson install` will automatically build anything that needs to be built to install it.
[doc("Install Lix for local development")]
[positional-arguments]
install *OPTIONS:
meson install --quiet -C {{ builddir }} {{ OPTIONS }}
meson install --quiet -C {{ builddir }} "$@"
# Run all tests tests (installs first).
[positional-arguments]
test *OPTIONS: (install)
meson test -C {{ builddir }} --print-errorlogs --max-lines 10000 {{ OPTIONS }}
meson test -C {{ builddir }} --print-errorlogs --max-lines 10000 "$@"
# Run unit tests only
test-unit *OPTIONS: (test "--suite" "check")
@@ -40,10 +53,15 @@ test-unit *OPTIONS: (test "--suite" "check")
test-integration *OPTIONS: (test "--suite" "installcheck" OPTIONS)
# Run functional2 tests using pytest directly, allowing for additional arguments to be passed to pytest e.g. for more granular test selection
[positional-arguments]
test-functional2 *OPTIONS:
cd tests/functional2 && python -m pytest -v {{ OPTIONS }}
cd tests/functional2 && python -m pytest -v "$@"
alias clang-tidy := lint
# special target for cargo because meson cannot be convinced to not mangle cargo test output,
# and getting properly colored test output any other way also doesn't look all that possible.
[positional-arguments]
test-rs *OPTIONS:
meson test -C {{ builddir }} --interactive lix-rs-tests "$@"
# Lint with `clang-tidy`
lint:
lix/code-generation/build_extra_features.py
+1 -1
View File
@@ -52,7 +52,7 @@ class ExtraFeature:
{
"\n ".join(
[
f"- {event.date}, {event.release}: {event.message} [{", ".join([f'[CL {cl}](https://git.lix.systems/c/lix/+/{cl})' for cl in event.cls])}]"
f"- {event.date}, {event.release}: {event.message} [{", ".join([f'[CL {cl}](https://gerrit.lix.systems/c/lix/+/{cl})' for cl in event.cls])}]"
for event in self.timeline
]
)
lix/legacy/build-remote.cc
+32 -112
View File
@@ -5,6 +5,7 @@
#include "lix/libutil/file-descriptor.hh"
#include "lix/libutil/logging-rpc.hh"
#include "lix/libutil/logging.hh"
#include "lix/libutil/result.hh"
#include "lix/libutil/rpc.hh"
#include "lix/libutil/types-rpc.hh" // IWYU pragma: keep
#include "lix/libutil/types.hh"
@@ -50,7 +51,7 @@ struct Instance final : rpc::build_remote::HookInstance::Server
kj::Promise<void> init(InitContext context) override;
kj::Promise<void> buildImpl(BuildContext context);
kj::Promise<Result<void>> buildImpl(BuildContext context);
kj::Promise<void> build(BuildContext context) override;
};
}
@@ -96,7 +97,7 @@ static std::tuple<bool, Machine *, AutoCloseFD> selectBestMachine(
uint64_t bestLoad = 0;
for (auto & m : machines) {
debug("considering building on remote machine '%s'", m.storeUri);
debug("considering building on remote machine '%s'", m.name);
if (m.enabled && m.systemSupported(neededSystem) && m.allSupported(requiredFeatures)
&& m.mandatoryMet(requiredFeatures))
@@ -184,67 +185,6 @@ struct BuilderConnection
AutoCloseFD slotLock;
std::shared_ptr<Store> sshStore;
std::string storeUri;
Pipe logPipe;
// start the thread that reads ssh stderr and turns it into log items.
// this future *must* outlive sshStore, otherwise it will never finish
kj::Promise<Result<void>> startLogThread(std::string buildDescription, std::string drvPath)
try {
if (!logPipe.readSide) {
co_return result::success();
}
logPipe.writeSide.close();
// NOTE this is very similar to handleBuilderOutput in DerivationGoal, but unlike
// the derivation goal we do not need to handle EIO from a pty here. we also have
// no timeouts or limits to keep track of, which makes deduplication less useful.
auto act = logger->startActivity(
lvlInfo, actBuild, buildDescription, Logger::Fields{drvPath, storeUri, 1, 1}
);
std::map<ActivityId, Activity> activities;
auto reader = AIO().lowLevelProvider.wrapInputFd(logPipe.readSide.get());
LogLineSplitter splitter;
auto flushLine = [&](const std::string & line) {
if (const auto state =
handleJSONLogMessage(line, act, activities, "the derivation builder"))
{
return *state;
} else {
return act.result(resBuildLogLine, line);
}
};
auto buf = kj::heapArray<char>(4096);
while (true) {
const auto got = co_await reader->tryRead(buf.begin(), 1, buf.size());
if (got == 0) {
break;
}
std::string_view data{buf.begin(), got};
while (!data.empty()) {
if (auto line = splitter.feed(data)) {
if (flushLine(*line) == Logger::BufferState::NeedsFlush) {
TRY_AWAIT(act.getLogger().flush());
}
}
}
}
if (auto line = splitter.finish(); !line.empty()) {
(void) flushLine(line);
TRY_AWAIT(act.getLogger().flush());
}
co_return result::success();
} catch (...) {
co_return result::current_exception();
}
};
struct AcceptedBuild final : rpc::build_remote::HookInstance::AcceptedBuild::Server
@@ -261,7 +201,7 @@ struct AcceptedBuild final : rpc::build_remote::HookInstance::AcceptedBuild::Ser
{
}
kj::Promise<void> runImpl(RunContext context);
kj::Promise<Result<void>> runImpl(RunContext context);
kj::Promise<void> run(RunContext context) override;
};
@@ -326,26 +266,15 @@ try {
lock.reset();
std::shared_ptr<Store> sshStore;
Pipe logPipe;
try {
auto act = logger->startActivity(
lvlTalkative, actUnknown, fmt("connecting to '%s'", bestMachine->storeUri)
);
auto act =
logger->startActivity(lvlTalkative, actUnknown, fmt("connecting to '%s'", bestMachine->name));
std::tie(sshStore, logPipe) = TRY_AWAIT(bestMachine->openStore());
TRY_AWAIT(sshStore->connect());
co_return BuilderConnection{
std::move(bestSlotLock), sshStore, bestMachine->storeUri, std::move(logPipe)
};
sshStore = TRY_AWAIT(bestMachine->openStore());
co_return BuilderConnection{std::move(bestSlotLock), sshStore, bestMachine->storeUri};
} catch (std::exception & e) { // NOLINT(lix-foreign-exceptions)
std::string msg = logPipe.readSide ? chomp(drainFD(logPipe.readSide.get(), false)) : "";
printError(
"cannot build on '%s': %s%s",
bestMachine->storeUri,
e.what(),
msg.empty() ? "" : ": " + msg
);
printError("cannot build on '%s': %s", bestMachine->name, e.what());
bestMachine->enabled = false;
}
}
@@ -366,7 +295,7 @@ static int main_build_remote(AsyncIoRoot & aio, std::string programName, Strings
if (argv.size() != 1)
throw UsageError("called without required arguments");
verbosity = (Verbosity) std::stoll(argv.front());
setVerbosity((Verbosity) std::stoll(argv.front()));
auto conn = aio.kj.lowLevelProvider->wrapUnixSocketFd(1);
capnp::TwoPartyServer srv(kj::heap<Instance>());
@@ -395,16 +324,14 @@ kj::Promise<void> Instance::init(InitContext context)
initPlugins();
initialized = true;
context.getResults().initResult().setGood();
} catch (...) {
RPC_FILL(context.getResults(), initResult, std::current_exception());
rpc::rethrow_as_rpc_error();
}
return kj::READY_NOW;
}
kj::Promise<void> Instance::buildImpl(BuildContext context)
kj::Promise<Result<void>> Instance::buildImpl(BuildContext context)
try {
if (!initialized) {
throw Error("build hook not fully initialized");
@@ -431,8 +358,8 @@ try {
debug("got %d remote builders", machines.size());
if (machines.empty()) {
context.getResults().initResult().initGood().setDeclinePermanently();
co_return;
context.getResults().initResult().setDeclinePermanently();
co_return result::success();
}
auto amWilling = context.getParams().getAmWilling();
@@ -448,21 +375,23 @@ try {
if (auto immediateResponse = std::get_if<BuildRejected>(&result)) {
switch (*immediateResponse) {
case BuildRejected::Temporarily:
context.getResults().initResult().initGood().setPostpone();
co_return;
context.getResults().initResult().setPostpone();
co_return result::success();
case BuildRejected::Permanently:
context.getResults().initResult().initGood().setDecline();
co_return;
context.getResults().initResult().setDecline();
co_return result::success();
}
}
auto builder = std::get_if<BuilderConnection>(&result);
assert(builder);
auto ac = context.getResults().initResult().initGood().initAccept();
auto ac = context.getResults().initResult().initAccept();
ac.setMachine(kj::heap<AcceptedBuild>(store, drvPath, std::move(*builder)));
co_return result::success();
} catch (...) {
RPC_FILL(context.getResults(), initResult, std::current_exception());
co_return result::current_exception();
}
kj::Promise<void> Instance::build(BuildContext context)
@@ -471,11 +400,12 @@ try {
throw Error("build hooks can only accept a single job");
}
used = true; // lock out other rpc calls during processing
co_await buildImpl(context);
auto result = co_await buildImpl(context);
TRY_AWAIT(logger->flush());
used = context.getResults().getResult().getGood().isAccept();
used = result.has_value() && context.getResults().getResult().isAccept();
result.value();
} catch (...) {
RPC_FILL(context.getResults(), getResult, std::current_exception());
rpc::rethrow_as_rpc_error();
}
kj::Promise<void> AcceptedBuild::run(RunContext context)
@@ -493,23 +423,17 @@ kj::Promise<void> AcceptedBuild::run(RunContext context)
throw Error("build hooks builds are single-use items");
}
used = true;
co_await runImpl(context);
auto result = co_await runImpl(context);
TRY_AWAIT(logger->flush());
result.value();
} catch (...) {
RPC_FILL(context.getResults(), getResult, std::current_exception());
rpc::rethrow_as_rpc_error();
}
}
kj::Promise<void> AcceptedBuild::runImpl(RunContext context)
kj::Promise<Result<void>> AcceptedBuild::runImpl(RunContext context)
{
try {
auto logHandler = builder.startLogThread(
fmt("%s on '%s'",
rpc::to<std::string_view>(context.getParams().getDescription()),
builder.storeUri),
store->printStorePath(drvPath)
);
auto & sshStore = builder.sshStore;
auto & storeUri = builder.storeUri;
@@ -619,13 +543,9 @@ kj::Promise<void> AcceptedBuild::runImpl(RunContext context)
);
}
// drop store connection, let log handler process any remaining input
builder.sshStore = nullptr;
TRY_AWAIT(logHandler);
context.getResults().initResult().setGood();
co_return result::success();
} catch (...) {
RPC_FILL(context.getResults(), initResult, std::current_exception());
co_return result::current_exception();
}
}
lix/legacy/builtin-builder.cc
+1 -1
View File
@@ -35,7 +35,7 @@ static int main_builtin_builder(AsyncIoRoot & aio, std::string programName, Stri
};
if (auto val = string2Int<int>(getArg("verbosity"))) {
verbosity = verbosityFromIntClamped(*val);
setVerbosity(verbosityFromIntClamped(*val));
} else {
throw Error("expected a verbosity argument");
}
lix/legacy/nix-copy-closure.cc
+1 -1
View File
@@ -55,7 +55,7 @@ static int main_nix_copy_closure(AsyncIoRoot & aio, std::string programName, Str
for (auto & path : storePaths)
storePaths2.insert(from->followLinksToStorePath(path));
aio.blockOn(copyClosure(*from, *to, storePaths2, NoRepair, NoCheckSigs, useSubstitutes));
aio.blockOn(copyClosure(*from, *to, storePaths2, NoRepair, NoCheckSigs, useSubstitutes, includeOutputs));
return 0;
}
lix/legacy/nix-store.cc
+1 -1
View File
@@ -948,7 +948,7 @@ opServe(std::shared_ptr<Store> store, AsyncIoRoot & aio, Strings opFlags, String
auto getBuildSettings = [&]() {
// FIXME: changing options here doesn't work if we're
// building through the daemon.
verbosity = lvlError;
setVerbosity(lvlError);
settings.keepLog.override(false);
settings.useSubstitutes.override(false);
settings.maxSilentTime.override(readNum<unsigned>(in));
lix/libcmd/repl-interacter.cc
+60 -215
View File
@@ -1,251 +1,101 @@
#include "lix/libutil/c-calls.hh"
#include "libutil/fmt.hh"
#include "libutil/terminal.hh"
#include "lix/libutil/error.hh"
#include "lix/libutil/file-system.hh"
#include "lix/libutil/logging.hh"
#include <csignal>
#include <cstdio>
#include "lix/lix-rs/main.gen.hh"
#include "lix/lix-rs/utils.hh"
#include <iostream>
#include <memory>
#include <string>
#include <string_view>
#include <cerrno>
// editline < 1.15.2 don't wrap their API for C++ usage
// (added in https://github.com/troglobit/editline/commit/91398ceb3427b730995357e9d120539fb9bb7461).
// This results in linker errors due to to name-mangling of editline C symbols.
// For compatibility with these versions, we wrap the API here
// (wrapping multiple times on newer versions is no problem).
extern "C" {
#include <editline.h>
}
#include "lix/libutil/finally.hh"
#include "lix/libutil/strings.hh"
#include "lix/libcmd/repl-interacter.hh"
namespace rust {
Vec<String> Impl<lix::repl::CxxCompleter, lix::repl::ReplCompleter>::complete(
Ref<lix::repl::CxxCompleter> self, Ref<Str> input
)
try {
auto s = to_std_string(input);
auto result = std::vec::Vec<std::string::String>::new_();
for (auto & possible : self.cpp().completePrefix(s)) {
result.push(rust::to_string(possible));
}
return result;
} catch (...) {
// the completer should have logged anything interesting.
return std::vec::Vec<std::string::String>::new_();
}
}
namespace nix {
namespace {
// Used to communicate to NixRepl::getLine whether a signal occurred in ::readline.
volatile sig_atomic_t g_signal_received = 0;
ReadlineLikeInteracter::ReadlineLikeInteracter(std::string historyFile) : historyFile(historyFile) {}
void sigintHandler(int signo)
void ReadlineLikeInteracter::init(detail::ReplCompleterMixin * repl)
{
g_signal_received = signo;
}
static detail::ReplCompleterMixin * curRepl; // ugly
/**
* @return a null-terminated list of completions as expected by `el_print_columns`
*/
char ** copyCompletions(const StringSet& possible)
{
int ac = 0;
char ** vp = nullptr;
auto check = [&](auto * p) {
if (!p) {
if (vp) {
while (--ac >= 0)
free(vp[ac]);
free(static_cast<void *>(vp));
}
throw Error("allocation failure");
}
return p;
};
vp = check(static_cast<char **>(malloc(possible.size() * sizeof(char *))));
for (auto & p : possible)
vp[ac++] = check(strdup(p.c_str()));
return vp;
}
// Instead of using the readline-provided prefix, do our own tokenization
// to avoid the default behavior of treating dots/quotes as word boundaries.
// See the definition of SEPS for what it treats as a boundary:
// https://github.com/troglobit/editline/blob/caf4b3c0ce3b0785791198b11de6f3134e9f05d8/src/editline.c
std::string getLastTokenBeforeCursor()
{
std::string_view line{rl_line_buffer, static_cast<size_t>(rl_point)};
auto tokens = tokenizeString<std::vector<std::string>>(
line,
// Same as editline's SEPS, except for double and single quotes:
"#$&()*:;<=>?[\\]^`{|}~\n\t "
);
if (tokens.empty()) {
return "";
}
return tokens.back();
}
// Sometimes inserting text or listing possible completions has a side effect
// of hiding the text after the cursor (even though it remains in the buffer).
// This helper just refreshes the display while keeping the cursor in place.
//
// Inserting text also sometimes moves the whole buffer down one line, usually
// if the cursor is inside a quoted attr name. I'm not sure why (vs unquoted)
// but it still seems to work pretty well and is just a visual artifact.
el_status_t redisplay()
{
int cursorPos = rl_point;
rl_refresh_line(0, 0);
rl_point = cursorPos;
return (rl_point == rl_end) ? CSstay : CSmove;
}
};
static el_status_t doCompletion() {
auto s = getLastTokenBeforeCursor();
auto possible = curRepl->completePrefix(s);
if (possible.empty()) {
return el_ring_bell();
}
if (possible.size() == 1) {
const auto completion = *possible.cbegin();
if (completion.size() > s.size()) {
rl_insert_text(requireCString(completion.substr(s.size())));
return redisplay();
}
return el_ring_bell();
}
auto checkAllHaveSameAt = [&](size_t pos) {
auto & first = *possible.begin();
for (auto & p : possible) {
if (p.size() <= pos || p[pos] != first[pos]) {
return false;
}
}
return true;
};
size_t start = s.size();
size_t len = 0;
while (checkAllHaveSameAt(start + len)) {
++len;
}
if (len > 0) {
auto commonPrefix = possible.begin()->substr(start, len);
rl_insert_text(requireCString(commonPrefix));
el_ring_bell();
return redisplay();
}
char** columns = copyCompletions(possible);
el_print_columns(possible.size(), columns);
return redisplay();
}
ReadlineLikeInteracter::Guard ReadlineLikeInteracter::init(detail::ReplCompleterMixin * repl)
{
// Allow nix-repl specific settings in .inputrc
rl_readline_name = "nix-repl";
try {
createDirs(dirOf(historyFile));
} catch (SysError & e) {
logWarning(e.info());
}
el_hist_size = 1000;
read_history(requireCString(historyFile));
auto oldRepl = curRepl;
curRepl = repl;
Guard restoreRepl([oldRepl] { curRepl = oldRepl; });
// editline does its own escaping of completions, so we rebind tab
// to our own completion function to skip that and do nix escaping
// instead of shell escaping.
el_bind_key(CTL('I'), doCompletion);
return restoreRepl;
auto rl = repl::Rustyline::new_(rust::to_string(historyFile).as_str(), *repl);
match_result(
std::move(rl),
[&](repl::Rustyline ok) { this->rl = std::make_unique<repl::Rustyline>(std::move(ok)); },
[&](rust::Box<rust::Dyn<rust::std::error::Error>> err) {
throw Error("%s", Uncolored(to_std_string(err.to_string())));
}
);
}
static constexpr const char * promptForType(ReplPromptType promptType)
static rust::Ref<rust::Str> promptForType(ReplPromptType promptType)
{
switch (promptType) {
case ReplPromptType::ReplPrompt:
return "nix-repl> ";
return "nix-repl> "_rs;
case ReplPromptType::ContinuationPrompt:
return " ";
return " "_rs;
}
assert(false);
}
bool ReadlineLikeInteracter::getLine(std::string & input, ReplPromptType promptType)
{
struct sigaction act, old;
sigset_t savedSignalMask, set;
auto s = rl->ask(promptForType(promptType));
auto setupSignals = [&]() {
act.sa_handler = sigintHandler;
sigfillset(&act.sa_mask);
act.sa_flags = 0;
if (sigaction(SIGINT, &act, &old))
throw SysError("installing handler for SIGINT");
// rustyline temporarily sets a SIGWINCH handler
KJ_DEFER(invalidateWindowSize());
sigemptyset(&set);
sigaddset(&set, SIGINT);
if (sigprocmask(SIG_UNBLOCK, &set, &savedSignalMask))
throw SysError("unblocking SIGINT");
};
auto restoreSignals = [&]() {
if (sigprocmask(SIG_SETMASK, &savedSignalMask, nullptr))
throw SysError("restoring signals");
return match_result(
std::move(s),
[&](rust::String ok) {
input += to_std_string(ok);
input += '\n';
return true;
},
[&](rust::rustyline::error::ReadlineError err) {
if (err.matches_Interrupted()) {
input.clear();
return true;
}
if (sigaction(SIGINT, &old, 0))
throw SysError("restoring handler for SIGINT");
};
if (err.matches_Eof()) {
return false;
}
setupSignals();
char * s = readline(promptForType(promptType)); // NOLINT(lix-unsafe-c-calls)
Finally doFree([&]() { free(s); });
restoreSignals();
if (g_signal_received) {
g_signal_received = 0;
input.clear();
return true;
}
if (!s)
return false;
this->writeHistory();
input += s;
input += '\n';
return true;
throw Error("%s", Uncolored(to_std_string(err.into().to_string())));
}
);
}
void ReadlineLikeInteracter::writeHistory()
{
int ret = write_history(requireCString(historyFile));
int writeHistErr = errno;
if (ret == 0) {
return;
if (rl) {
rl->write_history();
}
// If the open fails, editline returns EOF. If the close fails, editline
// forwards the return value of fclose(), which is EOF on error.
// readline however, returns the errno.
// So if we didn't get exactly EOF, then consider the return value the error
// code; otherwise use the errno we saved above.
// https://github.com/troglobit/editline/issues/66
if (ret != EOF) {
writeHistErr = ret;
}
// In any of these cases, we should explicitly ignore the error, but log
// them so the user isn't confused why their history is getting eaten.
std::string_view const errMsg(std::strerror(writeHistErr));
printTaggedWarning("ignoring error writing repl history to %s: %s", this->historyFile, errMsg);
}
ReadlineLikeInteracter::~ReadlineLikeInteracter()
@@ -253,11 +103,6 @@ ReadlineLikeInteracter::~ReadlineLikeInteracter()
this->writeHistory();
}
AutomationInteracter::Guard AutomationInteracter::init(detail::ReplCompleterMixin *)
{
return Guard([] {});
}
// ASCII ENQ character
constexpr const char * automationPrompt = "\x05";
lix/libcmd/repl-interacter.hh
+11 -11
View File
@@ -1,11 +1,14 @@
#pragma once
/// @file
#include "lix/libutil/finally.hh"
#include "lix/libutil/types.hh"
#include <functional>
#include <memory>
#include <string>
namespace rust::lix::repl {
struct Rustyline;
}
namespace nix {
namespace detail {
@@ -24,9 +27,7 @@ enum class ReplPromptType {
class ReplInteracter
{
public:
using Guard = Finally<std::function<void()>>;
virtual Guard init(detail::ReplCompleterMixin * repl) = 0;
virtual void init(detail::ReplCompleterMixin * repl) {}
/** Returns a boolean of whether the interacter got EOF */
virtual bool getLine(std::string & input, ReplPromptType promptType) = 0;
virtual ~ReplInteracter(){};
@@ -35,12 +36,12 @@ public:
class ReadlineLikeInteracter final : public ReplInteracter
{
std::string historyFile;
std::unique_ptr<rust::lix::repl::Rustyline> rl;
public:
ReadlineLikeInteracter(std::string historyFile)
: historyFile(historyFile)
{
}
virtual Guard init(detail::ReplCompleterMixin * repl) override;
ReadlineLikeInteracter(std::string historyFile);
virtual void init(detail::ReplCompleterMixin * repl) override;
virtual bool getLine(std::string & input, ReplPromptType promptType) override;
/** Writes the current history to the history file.
*
@@ -54,7 +55,6 @@ class AutomationInteracter final : public ReplInteracter
{
public:
AutomationInteracter() = default;
virtual Guard init(detail::ReplCompleterMixin * repl) override;
virtual bool getLine(std::string & input, ReplPromptType promptType) override;
virtual ~AutomationInteracter() override = default;
};
lix/libcmd/repl.cc
+22 -19
View File
@@ -1,6 +1,5 @@
#include <algorithm>
#include <cstdio>
#include <editline.h>
#include <iostream>
#include <cstdlib>
#include <cstring>
@@ -380,7 +379,7 @@ ReplExitStatus NixRepl::mainLoop()
std::list<ReplLoadable> loadables = std::exchange(loaded, {});
loadFiles(loadables);
auto _guard = interacter->init(static_cast<detail::ReplCompleterMixin *>(this));
interacter->init(static_cast<detail::ReplCompleterMixin *>(this));
/* Stop the progress bar because it interferes with the display of
the repl. */
@@ -441,15 +440,10 @@ StringSet NixRepl::completePrefix(const std::string &prefix)
{
StringSet completions;
// We should only complete colon commands if there's a colon at the beginning,
// but editline (for... whatever reason) doesn't *give* us the colon in the
// completion callback. If the user types :rel<TAB>, `prefix` will only be `rel`.
// Luckily, editline provides a global variable for its current buffer, so we can
// check for the presence of a colon there.
if (rl_line_buffer != nullptr && rl_line_buffer[0] == ':') {
if (prefix.starts_with(':')) {
for (auto const & [colonCmd, cmd] : registeredCommands) {
if ((!cmd->attributes.debugModeOnly || inDebugger()) && colonCmd.starts_with(prefix)) {
completions.insert(std::string(colonCmd));
if ((!cmd->attributes.debugModeOnly || inDebugger()) && colonCmd.starts_with(prefix.substr(1))) {
completions.insert(":" + colonCmd);
}
}
@@ -699,17 +693,26 @@ void NixRepl::initDebugBuiltinCommands()
addCommand(
"backtrace",
[](NixRepl & repl, const std::string & _arg) {
auto traces = repl.evaluator.debug->traces();
for (const auto & [idx, i] : enumerate(traces)) {
std::cout << "\n" << ANSI_BLUE << idx << ANSI_NORMAL << ": ";
showDebugTrace(std::cout, repl.evaluator.positions, *i);
auto tracesGenerator = repl.evaluator.debug->traces();
// since we want to print the stack trace in reverse order,
// we have to first traverse all frames and accumulate them
// in a list (in which we store each new trace at the /beginning/)
std::list<std::pair<size_t, const DebugTrace *>> reversedTraces;
for (const auto trace : tracesGenerator) {
// because the original traces are indexed from 0 upto N,
// this gives us their original index
auto idx = reversedTraces.size();
reversedTraces.push_front({idx, trace});
}
for (const auto & [traceIdx, trace] : reversedTraces) {
std::cout << "\n" << ANSI_BLUE << traceIdx << ANSI_NORMAL << ": ";
showDebugTrace(std::cout, repl.evaluator.positions, *trace);
}
return ProcessLineResult::PromptAgain;
},
{.aliases = {"bt"},
.debugModeOnly = true,
.help = "Show trace stack",
.section = "Debug mode"}
{.aliases = {"bt"}, .debugModeOnly = true, .help = "Show trace stack", .section = "Debug mode"}
);
addCommand(
@@ -817,7 +820,7 @@ void NixRepl::initDebugBuiltinCommands()
{.aliases = {"st"},
.debugModeOnly = true,
.help = "Show current trace. If an integer is provided, this switches to that stack "
"beforehand. If the integer has an explicit + or - sign, it is treated as"
"beforehand. If the integer has an explicit + or - sign, it is treated as "
"relative to the current stack index.",
.section = "Debug mode",
.positionalArgsSpecifiers = {{.placeholderText = "integer index", .optional = true}}}
lix/libexec/kill-user.cc
+3
View File
@@ -36,9 +36,12 @@ int helperMain(const char * name, std::span<char *> args) noexcept
other things, determines if kill(-1, signo) affects the
calling process. In the OSX libc, it's set to true,
which means "follow POSIX", which we don't want here */
#pragma clang diagnostic push
#pragma clang diagnostic ignored "-Wdeprecated-declarations"
if (syscall(SYS_kill, -1, SIGKILL, false) == 0) {
break;
}
#pragma clang diagnostic pop
#else
if (kill(-1, SIGKILL) == 0) {
break;
lix/libexec/launch-builder-linux.cc
+6 -6
View File
@@ -59,7 +59,7 @@ static void setPersonality(std::string_view system)
bool pathExists(const fs::path & path)
{
return fs::exists(fs::symlink_status(path));
return fs::exists(fs::symlink_status(path)); // NOLINT(lix-foreign-exceptions): caught by main
}
void bindPath(const fs::path & source, const fs::path & target, bool optional = false)
@@ -72,7 +72,7 @@ void bindPath(const fs::path & source, const fs::path & target, bool optional =
}
};
auto st = fs::symlink_status(source);
auto st = fs::symlink_status(source); // NOLINT(lix-foreign-exceptions): caught by main
if (st.type() == fs::file_type::not_found) {
if (optional) {
return;
@@ -82,14 +82,14 @@ void bindPath(const fs::path & source, const fs::path & target, bool optional =
}
if (st.type() == fs::file_type::directory) {
fs::create_directories(target);
fs::create_directories(target); // NOLINT(lix-foreign-exceptions): caught by main
bindMount();
} else if (st.type() == fs::file_type::symlink) {
// Symlinks can (apparently) not be bind-mounted, so just copy it
fs::create_directories(target.parent_path());
fs::copy_symlink(source, target);
fs::create_directories(target.parent_path()); // NOLINT(lix-foreign-exceptions): caught by main
fs::copy_symlink(source, target); // NOLINT(lix-foreign-exceptions): caught by main
} else {
fs::create_directories(target.parent_path());
fs::create_directories(target.parent_path()); // NOLINT(lix-foreign-exceptions): caught by main
if (kj::AutoCloseFd file{open(target.c_str(), O_RDWR | O_CREAT, 0644)}; file == nullptr) {
throw SysError("could not create %s", target);
}
lix/libexec/unix-bind-connect.cc
+20 -4
View File
@@ -2,7 +2,17 @@
#include <sys/socket.h>
#include <sys/un.h>
LIBEXEC_HELPER(4)
LIBEXEC_HELPER(5)
static int resultFd = -1;
static void sendResult(int result)
{
// NOTE posix says pipe writes smaller than PIPE_BUF must be atomic, so this either
// succeeds or fails (pipe bufs of four bytes make no sense at all for our systems)
int error = result;
DIE_UNLESS_SYS("writing result", ::write(resultFd, &error, sizeof(error)));
}
int helperMain(const char *, std::span<char *> args) noexcept
{
@@ -10,8 +20,12 @@ int helperMain(const char *, std::span<char *> args) noexcept
std::string_view method = args[1];
const auto dir = args[2];
const auto name = args[3];
resultFd = argToInt<int>("result-fd", args[4]);
DIE_UNLESS_SYS("chdir", chdir(dir));
if (chdir(dir)) {
sendResult(errno);
return 0;
}
struct sockaddr_un addr;
addr.sun_family = AF_UNIX;
@@ -23,9 +37,11 @@ int helperMain(const char *, std::span<char *> args) noexcept
}
if (method == "bind") {
DIE_UNLESS_SYS("bind", bind(socket, reinterpret_cast<struct sockaddr *>(&addr), sizeof(addr)));
auto result = bind(socket, reinterpret_cast<struct sockaddr *>(&addr), sizeof(addr)) ? errno : 0;
sendResult(result);
} else if (method == "connect") {
DIE_UNLESS_SYS("connect", connect(socket, reinterpret_cast<struct sockaddr *>(&addr), sizeof(addr)));
auto result = connect(socket, reinterpret_cast<struct sockaddr *>(&addr), sizeof(addr)) ? errno : 0;
sendResult(result);
} else {
die(std::format("invalid method %s", method));
}
lix/libexpr/builtins/addErrorContext.md
+2 -58
View File
@@ -23,70 +23,14 @@ countDown 2
Then, evaluating the file will give the following stack trace:
```console
$ nix-instantiate --show-trace err.nix
$ nix-instantiate err.nix
error:
… from call site
at /home/plop/git.lix.systems/lix-project/lix/err.nix:9:1:
8| in
9| countDown 2
| ^
10|
… while calling 'countDown'
at /home/plop/git.lix.systems/lix-project/lix/err.nix:3:5:
2| countDown =
3| n:
| ^
4| if n == 0 then
… while calling the 'addErrorContext' builtin
at /home/plop/git.lix.systems/lix-project/lix/err.nix:7:7:
6| else
7| builtins.addErrorContext "while counting down; n = ${toString n}" ("x" + countDown (n - 1));
| ^
8| in
… while counting down; n = 2
… from call site
at /home/plop/git.lix.systems/lix-project/lix/err.nix:7:80:
6| else
7| builtins.addErrorContext "while counting down; n = ${toString n}" ("x" + countDown (n - 1));
| ^
8| in
… while calling 'countDown'
at /home/plop/git.lix.systems/lix-project/lix/err.nix:3:5:
2| countDown =
3| n:
| ^
4| if n == 0 then
… while calling the 'addErrorContext' builtin
at /home/plop/git.lix.systems/lix-project/lix/err.nix:7:7:
6| else
7| builtins.addErrorContext "while counting down; n = ${toString n}" ("x" + countDown (n - 1));
| ^
8| in
… while counting down; n = 1
… from call site
at /home/plop/git.lix.systems/lix-project/lix/err.nix:7:80:
6| else
7| builtins.addErrorContext "while counting down; n = ${toString n}" ("x" + countDown (n - 1));
| ^
8| in
… while calling 'countDown'
at /home/plop/git.lix.systems/lix-project/lix/err.nix:3:5:
2| countDown =
3| n:
| ^
4| if n == 0 then
… caused by explicit throw
at /home/plop/git.lix.systems/lix-project/lix/err.nix:5:7:
at err.nix:5:7:
4| if n == 0 then
5| throw "kaboom"
| ^
lix/libexpr/builtins/ceil.md
+6 -5
View File
@@ -1,9 +1,10 @@
---
name: ceil
args: [double]
args: [number]
---
Converts an IEEE-754 double-precision floating-point number (*double*) to
the next higher integer.
Returns an integer value containing the smallest integer greater than or equal to *number* (which is either a floating-point or integer value).
If the datatype is neither an integer nor a "float", an evaluation error will be
thrown.
If the result is out of range for the integer type, such as NaN, infinity, or a number with magnitude out of range, Lix throws an evaluation error.
Lix currently throws an evaluation error for some *integer* inputs between 2\*\*52 and 2\*\*63 - 1 as those previously experienced floating-point precision loss due to a Nix bug (https://github.com/NixOS/nix/issues/12899).
In a future release, such integers will be passed through.
lix/libexpr/builtins/floor.md
+6 -5
View File
@@ -1,9 +1,10 @@
---
name: floor
args: [double]
args: [number]
---
Converts an IEEE-754 double-precision floating-point number (*double*) to
the next lower integer.
Returns an integer value containing the largest integer less than or equal to *number* (which is either a float or integer).
If the datatype is neither an integer nor a "float", an evaluation error will be
thrown.
If the result is out of range for the integer type, such as NaN, infinity, or a number with magnitude out of range, Lix throws an evaluation error.
Lix currently throws an evaluation error for some *integer* inputs between 2\*\*52 and 2\*\*63 - 1 as those previously experienced floating-point precision loss due to a Nix bug (https://github.com/NixOS/nix/issues/12899).
In a future release, such integers will be passed through.
lix/libexpr/eval-cache.hh
+1 -1
View File
@@ -53,7 +53,7 @@ public:
ref<AttrCursor> getRoot();
};
enum AttrType {
enum AttrType : uint8_t {
Placeholder = 0,
FullAttrs = 1,
String = 2,
lix/libexpr/eval-expr.cc
+29 -24
View File
@@ -1,4 +1,5 @@
#include "eval.hh"
#include "libexpr/nixexpr.hh"
#include "primops.hh"
#include "gc-small-vector.hh"
@@ -10,14 +11,19 @@
namespace nix {
Value Expr::makeThunk(Evaluator & ctx, Env & env)
{
ctx.stats.nrThunks++;
return {NewValueAs::thunk, ctx.mem, env, *this};
}
/* Create a thunk for the delayed computation of the given expression
in the given environment. But if the expression is a variable,
then look it up right away. This significantly reduces the number
of thunks allocated. */
Value Expr::maybeThunk(EvalState & state, Env & env)
{
state.ctx.stats.nrThunks++;
return {NewValueAs::thunk, state.ctx.mem, env, *this};
return makeThunk(state.ctx, env);
}
Value ExprVar::maybeThunk(EvalState & state, Env & env)
@@ -99,18 +105,10 @@ Value ExprSet::eval(EvalState & state, Env & env)
in the original environment. */
Displacement displ = 0;
for (auto & i : attrs) {
Value vAttr;
if (hasOverrides && i.second.kind != ExprAttrs::AttrDef::Kind::Inherited) {
vAttr = {
NewValueAs::thunk,
state.ctx.mem,
*i.second.chooseByKind(&env2, &env, inheritEnv),
*i.second.e
};
state.ctx.stats.nrThunks++;
} else {
vAttr = i.second.e->maybeThunk(state, *i.second.chooseByKind(&env2, &env, inheritEnv));
}
Env & thunkEnv = *i.second.chooseByKind(&env2, &env, inheritEnv);
Value vAttr = hasOverrides && i.second.kind != ExprAttrs::AttrDef::Kind::Inherited
? i.second.e->makeThunk(state.ctx, thunkEnv)
: i.second.e->maybeThunk(state, thunkEnv);
env2.values[displ++] = vAttr;
v.attrs()->push_back(Attr(i.first, vAttr, i.second.pos));
}
@@ -358,12 +356,14 @@ Value ExprOpConcatLists::eval(EvalState & state, Env & env)
{
state.ctx.stats.nrListConcats++;
/* We don't call into `concatLists` as that loses the position information of the expressions. */
/* We could simply call into `state.concatLists`, but that would add a redundant trace to our errors,
* and to fix that we would need to make the error on it and `forceList` optional, and *sigh*
*/
Value v1 = e1->eval(state, env);
state.checkList(v1, env, *e1);
state.checkList(v1, env, *this); // Pass in `this` instead of `e1` to make the error point to the `++`
Value v2 = e2->eval(state, env);
state.checkList(v2, env, *e2);
state.checkList(v2, env, *this); // Pass in `this` instead of `e2` to make the error point to the `++`
size_t l1 = v1.listSize(), l2 = v2.listSize(), len = l1 + l2;
@@ -436,7 +436,7 @@ Value ExprConcatStrings::eval(EvalState & state, Env & env)
} else {
state.ctx.errors
.make<EvalError>("integer overflow in adding %1% + %2%", n, vTmp.integer())
.atPos(i_pos)
.atPos(isInterpolation ? i_pos : pos)
.debugThrow();
}
} else if (vTmp.type() == nFloat) {
@@ -446,7 +446,7 @@ Value ExprConcatStrings::eval(EvalState & state, Env & env)
nf += vTmp.fpoint();
} else {
state.ctx.errors.make<EvalError>("cannot add %1% to an integer", showType(vTmp))
.atPos(i_pos)
.atPos(isInterpolation ? i_pos : pos)
.withFrame(env, *this)
.debugThrow();
}
@@ -457,7 +457,7 @@ Value ExprConcatStrings::eval(EvalState & state, Env & env)
nf += vTmp.fpoint();
} else {
state.ctx.errors.make<EvalError>("cannot add %1% to a float", showType(vTmp))
.atPos(i_pos)
.atPos(isInterpolation ? i_pos : pos)
.withFrame(env, *this)
.debugThrow();
}
@@ -473,13 +473,18 @@ Value ExprConcatStrings::eval(EvalState & state, Env & env)
: StringCoercionMode::Strict;
/* skip canonization of first path, which would only be not
canonized in the first place if it's coming from a ./${foo} type
path */
canonized in the first place if it's coming from a ./${foo} type
path */
auto part = state.coerceToString(
i_pos,
isInterpolation ? i_pos : pos,
vTmp,
context,
"while evaluating a path segment",
(isInterpolation && firstType == nPath)
? "while evaluating a path interpolation"
: (isInterpolation ? "while evaluating a string interpolation"
// TODO: to the person who eventually cleans up all of this mess,
// please turn this into "cannot concatenate $type to a $type" instead.
: "while concatenating"),
coercionMode,
firstType == nString,
!first
lix/libexpr/eval-inline.hh
-1
View File
@@ -6,7 +6,6 @@
#include "lix/libexpr/eval-error.hh"
#include "lix/libexpr/gc-alloc.hh"
#include "value.hh"
#include <cstdint>
namespace nix {
lix/libexpr/eval.cc
+14 -19
View File
@@ -847,8 +847,7 @@ Value::List * EvalMemory::newList(size_t size)
Value Evaluator::evalLazily(Expr & e)
{
stats.nrThunks++;
return {NewValueAs::thunk, mem, builtins.env, e};
return e.makeThunk(*this, builtins.env);
}
Value EvalState::mkPos(PosIdx p)
@@ -908,19 +907,6 @@ std::string EvalState::mkSingleDerivedPathStringRaw(
}
void EvalState::mkSingleDerivedPathString(
const SingleDerivedPath & p,
Value & v)
{
v = {
NewValueAs::string,
mkSingleDerivedPathStringRaw(p),
NixStringContext{
std::visit([](auto && v) -> NixStringContextElem { return v; }, p),
}
};
}
struct CachedEvalFile
{
Value result;
@@ -1132,8 +1118,9 @@ Env & AttrsPattern::match(
Value EvalState::callFunction(Value & fun, std::span<Value> args, const PosIdx pos)
{
if (callDepth > evalSettings.maxCallDepth)
if (callDepth >= evalSettings.maxCallDepth) {
ctx.errors.make<EvalError>("stack overflow; max-call-depth exceeded").atPos(pos).debugThrow();
}
MaintainCount _level(callDepth);
auto trace = evalSettings.traceFunctionCalls
@@ -1208,12 +1195,18 @@ Value EvalState::callFunction(Value & fun, std::span<Value> args, const PosIdx p
// was being evaluated and an explicit thrown error.
if (fn->name == "throw" && !e.hasTrace()) {
e.addTrace(ctx.positions[pos], "caused by explicit %s", "throw");
} else {
}
// otherwise, print a trace of this builtin, as long as it isn't
// a 'addErrorContext' call (which would just create noise)
else if (fn->name != "addErrorContext")
{
e.addTrace(ctx.positions[pos], "while calling the '%s' builtin", fn->name);
}
throw;
} catch (Error & e) {
e.addTrace(ctx.positions[pos], "while calling the '%1%' builtin", fn->name);
if (fn->name != "addErrorContext") {
e.addTrace(ctx.positions[pos], "while calling the '%1%' builtin", fn->name);
}
throw;
}
@@ -1259,7 +1252,9 @@ Value EvalState::callFunction(Value & fun, std::span<Value> args, const PosIdx p
// so the debugger allows to inspect the wrong parameters passed to the builtin.
vCur = fn->fun(*this, vArgs.data());
} catch (Error & e) {
e.addTrace(ctx.positions[pos], "while calling the '%1%' builtin", fn->name);
if (fn->name != "addErrorContext") {
e.addTrace(ctx.positions[pos], "while calling the '%1%' builtin", fn->name);
}
throw;
}
}
lix/libexpr/eval.hh
-9
View File
@@ -817,15 +817,6 @@ public:
*/
Value mkOutputString(const SingleDerivedPath::Built & b, const StorePath & staticOutputPath);
/**
* Create a string representing a `SingleDerivedPath`.
*
* A combination of `mkStorePathString` and `mkOutputString`.
*/
void mkSingleDerivedPathString(
const SingleDerivedPath & p,
Value & v);
Value updateAttrs(const Value & v1, const Value & v2);
Value concatLists(std::span<Value> lists, const PosIdx pos, std::string_view errorCtx);
lix/libexpr/flake/config.cc
+119 -63
View File
@@ -4,6 +4,7 @@
#include "lix/libutil/json.hh"
#include "lix/libutil/users.hh"
#include "lix/libfetchers/fetch-settings.hh"
#include <cctype>
namespace nix::flake {
@@ -30,6 +31,64 @@ static void writeTrustedList(const TrustedList & trustedList)
writeFile(path, JSON(trustedList).dump());
}
static bool
askForEachSetting(TrustedList & trustedList, std::map<std::string, std::string> & untrustedSettings)
{
// clang-format off
constexpr auto prompt =
"[" ANSI_BOLD "y" ANSI_NORMAL "]es for now/"
"[" ANSI_BOLD "N" ANSI_NORMAL "]o for now/"
"[" ANSI_BOLD "a" ANSI_NORMAL "]lways allow";
// clang-format on
auto didTrustedListChange = false;
int acceptedCount = 0;
for (const auto & [name, valueS] : untrustedSettings) {
auto reply =
logger
->ask(
fmt("Do you want to allow setting '" ANSI_MAGENTA "%s = %s" ANSI_NORMAL "'? (%s) ",
name,
valueS,
prompt)
)
.value_or("n");
reply = toLower(reply);
static const std::string yes = "yes for now";
static const std::string no = "no for now";
static const std::string always = "always allow";
while (true) {
if (no.starts_with(reply)) {
break;
}
if (yes.starts_with(reply) || always.starts_with(reply)) {
if (reply[0] == 'a') {
trustedList[name][valueS] = true;
didTrustedListChange = true;
}
globalConfig.set(name, valueS);
acceptedCount++;
break;
}
// if the reply wasn't a prefix of any answer, ask the user again
reply = logger->ask(fmt("Couldn't understand reply.\n%s: ", prompt)).value_or("n");
}
}
if (didTrustedListChange) {
writeTrustedList(trustedList);
}
// return false if *none* of the settings were accepted
return acceptedCount > 0;
}
static bool batchAskForSetting(
bool & negativeTrustOverride,
TrustedList & trustedList,
@@ -43,77 +102,74 @@ static bool batchAskForSetting(
printWarning("%s", warning);
auto reply = logger
->ask(
fmt("Do you want to allow configuration settings to be applied?\nThis may allow the "
"flake to gain root, see the nix.conf manual page (" ANSI_BOLD "y" ANSI_NORMAL
"es for now/" ANSI_BOLD "A" ANSI_NORMAL "llow always/" ANSI_BOLD "n" ANSI_NORMAL
"o/" ANSI_BOLD "N" ANSI_NORMAL "o to all) ")
)
.value_or('n');
// clang-format off
constexpr auto globalPrompt =
"[" ANSI_BOLD "y" ANSI_NORMAL "]es for now/"
"[" ANSI_BOLD "n" ANSI_NORMAL "]o for now/"
"[" ANSI_BOLD "a" ANSI_NORMAL "]lways allow/"
"[" ANSI_BOLD "I" ANSI_NORMAL "]ndividually review";
// clang-format on
if (reply == 'N') {
printWarning("Rejecting all untrusted nix.conf entries");
printTaggedWarning(
"you can set '%s' to '%b' to automatically reject configuration options supplied by "
"flakes",
"accept-flake-config",
false
);
negativeTrustOverride = true;
return false;
}
auto reply =
logger
->ask(
fmt("Do you want to allow these configuration settings to be applied?\n" ANSI_BOLD
"This may allow the flake to gain root" ANSI_NORMAL ", see the nix.conf manual page.\n"
"(%s) ",
globalPrompt)
)
.value_or("I"); // if the answer is empty (or there is no interactive prompt),
// just default to reviewing each individually
reply = toLower(reply);
static const std::string yes = "yes for now";
static const std::string no = "no for now";
static const std::string always = "always allow";
static const std::string review = "individually review";
// when interactive, loops and reprompts until the reply is one of y/n/a/i (or any prefix of the answers)
while (true) {
if (no.starts_with(reply)) {
printWarning("Rejecting all untrusted nix.conf entries");
printTaggedWarning(
"you can set '%s' to '%b' to automatically reject configuration options supplied by "
"flakes",
"accept-flake-config",
false
);
negativeTrustOverride = true;
return false;
}
if (yes.starts_with(reply) || always.starts_with(reply)) {
auto alwaysAllow = reply[0] == 'a';
for (const auto & [name, valueS] : untrustedSettings) {
if (alwaysAllow) {
trustedList[name][valueS] = true;
}
globalConfig.set(name, valueS);
}
if (reply == 'y' || reply == 'A') {
auto alwaysAllow = reply == 'A';
for (const auto & [name, valueS] : untrustedSettings) {
if (alwaysAllow) {
trustedList[name][valueS] = true;
}
globalConfig.set(name, valueS);
}
if (alwaysAllow) {
writeTrustedList(trustedList);
}
return true;
} else {
printTaggedWarning(
"you can set '%s' to '%b' to automatically reject configuration options supplied "
"by flakes",
"accept-flake-config",
false
);
}
auto didTrustedListChange = false;
for (const auto & [name, valueS] : untrustedSettings) {
auto individualReply = logger
->ask(
fmt("Do you want to allow setting '%s = %s'? (" ANSI_BOLD
"y" ANSI_NORMAL "es for now/" ANSI_BOLD "A" ANSI_NORMAL
"llow always/" ANSI_BOLD "n" ANSI_NORMAL "o for now) ",
name,
valueS)
)
.value_or('n');
if (individualReply == 'y' || individualReply == 'A') {
if (individualReply == 'A') {
trustedList[name][valueS] = true;
didTrustedListChange = true;
printTaggedWarning(
"adding these configuration settings to the trusted list at %s, "
"edit it if you want to remove them in the future",
trustedListPath()
);
writeTrustedList(trustedList);
}
globalConfig.set(name, valueS);
return true;
}
}
if (didTrustedListChange) {
writeTrustedList(trustedList);
}
if (review.starts_with(reply)) {
return askForEachSetting(trustedList, untrustedSettings);
}
return false;
// if the reply wasn't a prefix of any, ask the user again
reply = logger->ask(fmt("Couldn't understand reply.\n%s: ", globalPrompt)).value_or("n");
}
}
void ConfigFile::apply()
lix/libexpr/flake/flake.cc
+23 -6
View File
@@ -78,10 +78,21 @@ try {
co_return result::current_exception();
}
/** Force a value that cannot contain any function calls */
static void forceTrivialValue(EvalState & state, Value & value, const PosIdx pos)
{
if (value.isThunk() && value.isTrivial())
state.forceValue(value, pos);
/* /piegames sighs at the settings API */
auto prevOverridden = evalSettings.maxCallDepth.overridden;
auto prevMaxCallDepth = evalSettings.maxCallDepth;
evalSettings.maxCallDepth.override(0);
/* /piegames sighs at C++ */
KJ_DEFER({
evalSettings.maxCallDepth.override(prevMaxCallDepth);
evalSettings.maxCallDepth.overridden = prevOverridden;
});
state.forceValue(value, pos);
}
@@ -327,11 +338,17 @@ static Flake getFlake(
Expr & flakeExpr = state.ctx.parseExprFromFile(resolvedFlakeFile);
// Enforce that 'flake.nix' is a direct attrset, not a computation.
if (!flakeExpr.try_cast<ExprAttrs>()) {
state.ctx.errors.make<EvalError>("file '%s' must be an attribute set", resolvedFlakeFile).debugThrow();
}
// We do this by disallowing any function calls (maxCallDepth 0) and checking that the resulting value is
// an attrset.
// The logic is already implemented in `forceTrivialValue`, but that takes a value instead of an Expr, so
// we wrap the expression in a thunk to be able to call it.
Value vInfo = flakeExpr.maybeThunk(state, state.ctx.builtins.env);
forceTrivialValue(state, vInfo, flakeExpr.getPos());
Value vInfo = state.eval(flakeExpr);
if (vInfo.type() != nAttrs) {
state.ctx.errors.make<EvalError>("file '%s' must be an attribute set", resolvedFlakeFile)
.debugThrow();
}
if (auto description = vInfo.attrs()->get(state.ctx.symbols.sym_description)) {
expectType(state, nString, description->value, description->pos);
lix/libexpr/flake/flakeref.cc
-20
View File
@@ -67,16 +67,6 @@ FlakeRef parseFlakeRef(
return flakeRef;
}
std::optional<FlakeRef> maybeParseFlakeRef(
const std::string & url, const std::optional<Path> & baseDir)
{
try {
return parseFlakeRef(url, baseDir);
} catch (Error &) {
return {};
}
}
std::pair<FlakeRef, std::string> parseFlakeRefWithFragment(
const std::string & url,
const std::optional<Path> & baseDir,
@@ -234,16 +224,6 @@ std::pair<FlakeRef, std::string> parseFlakeRefWithFragment(
}
}
std::optional<std::pair<FlakeRef, std::string>> maybeParseFlakeRefWithFragment(
const std::string & url, const std::optional<Path> & baseDir)
{
try {
return parseFlakeRefWithFragment(url, baseDir);
} catch (Error & e) {
return {};
}
}
FlakeRef FlakeRef::fromAttrs(const fetchers::Attrs & attrs)
{
auto attrs2(attrs);
lix/libexpr/flake/flakeref.hh
-6
View File
@@ -70,18 +70,12 @@ FlakeRef parseFlakeRef(
bool allowMissing = false,
bool isFlake = true);
std::optional<FlakeRef> maybeParseFlake(
const std::string & url, const std::optional<Path> & baseDir = {});
std::pair<FlakeRef, std::string> parseFlakeRefWithFragment(
const std::string & url,
const std::optional<Path> & baseDir = {},
bool allowMissing = false,
bool isFlake = true);
std::optional<std::pair<FlakeRef, std::string>> maybeParseFlakeRefWithFragment(
const std::string & url, const std::optional<Path> & baseDir = {});
std::tuple<FlakeRef, std::string, ExtendedOutputsSpec> parseFlakeRefWithFragmentAndExtendedOutputsSpec(
const std::string & url,
const std::optional<Path> & baseDir = {},
lix/libexpr/json-to-value.cc
+80 -92
View File
@@ -1,125 +1,75 @@
#include "lix/libexpr/json-to-value.hh"
#include "gc-alloc.hh"
#include "libutil/types.hh"
#include "lix/libexpr/value.hh"
#include "lix/libexpr/eval.hh"
#include "lix/libutil/json.hh"
#include <limits>
#include <variant>
namespace nix {
// for more information, refer to
// https://github.com/nlohmann/json/blob/master/include/nlohmann/detail/input/json_sax.hpp
class JSONSax : nlohmann::json_sax<JSON> {
class JSONState {
protected:
std::unique_ptr<JSONState> parent;
JSONState() = default;
public:
virtual std::unique_ptr<JSONState> resolve(EvalState &) = 0;
explicit JSONState(std::unique_ptr<JSONState> && p) : parent(std::move(p)) {}
JSONState(JSONState & p) = delete;
virtual Value & finalValue()
{
assert(false && "tried to read a final value from a non-toplevel json parser state");
}
virtual ~JSONState() {}
virtual void addValue(Value v) = 0;
};
class TopLevelJSONState : public JSONState
struct WIPObject
{
RootValue v;
public:
std::unique_ptr<JSONState> resolve(EvalState &) override
{
assert(false && "tried to close toplevel json parser state");
}
TopLevelJSONState() = default;
TopLevelJSONState(TopLevelJSONState & p) = delete;
Value & finalValue() override
{
assert(v && "tried to read nonexistent final value from json parser");
return *v;
}
void addValue(Value v) override
{
assert(!this->v && "duplicate value in toplevel JSON scope");
this->v = allocRootValue(v);
}
GcMap<Symbol, Value> entries;
Symbol nextKey;
};
using WIPArray = GcVector<Value>;
class JSONObjectState : public JSONState {
using JSONState::JSONState;
GcMap<Symbol, Value> attrs;
Symbol _key;
std::unique_ptr<JSONState> resolve(EvalState & state) override
{
auto attrs2 = state.ctx.buildBindings(attrs.size());
for (auto & i : attrs)
attrs2.insert(i.first, i.second);
parent->addValue({NewValueAs::attrs, attrs2.alreadySorted()});
return std::move(parent);
}
void addValue(Value v) override
{
attrs.insert_or_assign(_key, v);
}
public:
void key(string_t & name, EvalState & state)
{
_key = state.ctx.symbols.create(name);
}
};
// A stack of all of the currently opened and not yet closed JSON objects and arrays at the current moment
// of parsing
std::vector<std::variant<WIPObject, WIPArray>> stack;
class JSONListState : public JSONState {
GcVector<Value> values;
std::unique_ptr<JSONState> resolve(EvalState & state) override
{
auto list = state.ctx.mem.newList(values.size());
parent->addValue({NewValueAs::list, list});
for (size_t n = 0; n < values.size(); ++n) {
list->elems[n] = values[n];
}
return std::move(parent);
std::optional<Value> final_value;
void addValue(Value v)
{
if (stack.size() == 0) {
assert(!final_value);
final_value = v;
return;
}
void addValue(Value v) override
{
values.push_back(v);
}
public:
JSONListState(std::unique_ptr<JSONState> && p, std::size_t reserve) : JSONState(std::move(p))
{
values.reserve(reserve);
}
};
std::visit(
overloaded{
[&](WIPObject & currentObject) {
currentObject.entries.insert_or_assign(currentObject.nextKey, v);
currentObject.nextKey = {}; // clear the current symbol
},
[&](WIPArray & arr) { arr.push_back(v); }
},
stack.back()
);
}
EvalState & state;
std::unique_ptr<JSONState> rs;
public:
JSONSax(EvalState & state) : state(state), rs(new TopLevelJSONState()) {};
JSONSax(EvalState & state) : state(state) {};
Value result()
{
return rs->finalValue();
return final_value.value();
}
bool null() override
{
rs->addValue(Value::VNULL);
addValue(Value::VNULL);
return true;
}
bool boolean(bool val) override
{
rs->addValue({NewValueAs::boolean, val});
addValue({NewValueAs::boolean, val});
return true;
}
bool number_integer(number_integer_t val) override
{
rs->addValue({NewValueAs::integer, val});
addValue({NewValueAs::integer, val});
return true;
}
@@ -131,19 +81,19 @@ public:
return number_float(static_cast<number_float_t>(val_), "");
}
NixInt::Inner val = val_;
rs->addValue({NewValueAs::integer, val});
addValue({NewValueAs::integer, val});
return true;
}
bool number_float(number_float_t val, const string_t & s) override
{
rs->addValue({NewValueAs::floating, val});
addValue({NewValueAs::floating, val});
return true;
}
bool string(string_t & val) override
{
rs->addValue({NewValueAs::string, val});
addValue({NewValueAs::string, val});
return true;
}
@@ -158,28 +108,66 @@ public:
bool start_object(std::size_t len) override
{
rs = std::make_unique<JSONObjectState>(std::move(rs));
stack.emplace_back(std::in_place_type<WIPObject>);
return true;
}
bool key(string_t & name) override
{
dynamic_cast<JSONObjectState*>(rs.get())->key(name, state);
auto & back = stack.back();
std::visit(
overloaded{
[&](WIPObject & frame) {
assert(!frame.nextKey);
frame.nextKey = state.ctx.symbols.create(name);
},
[](const WIPArray &) { assert(false && "tried adding a json key to an array value??"); }
},
back
);
return true;
}
bool end_object() override {
rs = rs->resolve(state);
auto back = std::move(stack.back());
stack.pop_back();
std::visit(
overloaded{
[&](const WIPObject & frame) {
auto attrs2 = state.ctx.buildBindings(frame.entries.size());
for (auto & i : frame.entries) {
attrs2.insert(i.first, i.second);
}
addValue({NewValueAs::attrs, attrs2.alreadySorted()});
},
[](const WIPArray &) { assert(false && "tried to close a JSON object while in an array"); }
},
back
);
return true;
}
bool end_array() override {
return end_object();
auto back = std::move(stack.back());
stack.pop_back();
std::visit(
overloaded{
[](const WIPObject &) { assert(false && "tried to close a JSON array while in an object"); },
[&](const WIPArray & values) {
auto list = state.ctx.mem.newList(values.size());
std::ranges::copy(values, list->elems);
addValue({NewValueAs::list, list});
}
},
back
);
return true;
}
bool start_array(size_t len) override {
rs = std::make_unique<JSONListState>(std::move(rs),
len != std::numeric_limits<size_t>::max() ? len : 128);
auto v = WIPArray{};
v.reserve(len != std::numeric_limits<size_t>::max() ? len : 128);
stack.push_back(std::move(v));
return true;
}
lix/libexpr/nixexpr.cc
-18
View File
@@ -99,13 +99,6 @@ JSON ExprVar::toJSON(const SymbolTable & symbols) const
return {{"_type", "ExprVar"}, {"value", stringToJSON(symbols[name])}};
}
JSON ExprInheritFrom::toJSON(SymbolTable const & symbols) const
{
return {
{"_type", "ExprInheritFrom"}
};
}
JSON ExprSelect::toJSON(const SymbolTable & symbols) const
{
JSON out = {
@@ -747,17 +740,6 @@ void ExprLambda::setName(Symbol name)
body->setName(name);
}
std::string ExprLambda::showNamePos(const EvalState & state) const
{
std::string id(name
? concatStrings("'", state.ctx.symbols[name], "'")
: "anonymous function");
return fmt("%1% at %2%", id, state.ctx.positions[pos]);
}
/* Position table. */
Pos PosTable::operator[](PosIdx p) const
lix/libexpr/nixexpr.hh
+12 -3
View File
@@ -126,9 +126,10 @@ public:
std::unique_ptr<Expr> parsed, Evaluator & es, const std::shared_ptr<const StaticEnv> & env
);
virtual JSON toJSON(const SymbolTable & symbols) const;
virtual JSON toJSON(const SymbolTable & symbols) const = 0;
virtual void accept(ExprVisitor & ev, std::unique_ptr<Expr> & ptr) = 0;
virtual Value eval(EvalState & state, Env & env);
Value makeThunk(Evaluator & ctx, Env & env);
virtual Value maybeThunk(EvalState & state, Env & env);
/* Lambdas have a name associated with them, when they are declared in a binding:
* `identity = x: x` will print the resulting value as `«lambda identity @ «string»:1:14»`.
@@ -302,7 +303,11 @@ struct ExprInheritFrom : Expr
{
}
JSON toJSON(const SymbolTable & symbols) const override;
JSON toJSON(const SymbolTable & symbols) const override
{
abort();
}
Value eval(EvalState & state, Env & env) override;
void accept(ExprVisitor & ev, std::unique_ptr<Expr> & ptr) override { ev.visit(*this, ptr); }
};
@@ -517,7 +522,6 @@ struct ExprLambda : Expr
{
}
void setName(Symbol name) override;
std::string showNamePos(const EvalState & state) const;
/** Returns the name of the lambda,
* or "anonymous lambda" if it doesn't have one.
@@ -656,6 +660,11 @@ struct ExprPos : Expr
/* only used to mark thunks as black holes. */
struct ExprBlackHole : Expr
{
JSON toJSON(const SymbolTable & symbols) const override
{
abort();
}
Value eval(EvalState & state, Env & env) override;
void accept(ExprVisitor & ev, std::unique_ptr<Expr> & ptr) override { ev.visit(*this, ptr); }
};
lix/libexpr/parser/state.hh
+12 -2
View File
@@ -133,14 +133,24 @@ inline void State::badEscapeFound(const PosIdx pos, char found, bool isIndented)
{
auto escape = std::string(isIndented ? "''\\" : "\\");
auto interpolEscape = std::string(isIndented ? "''${" : "\\${");
auto backslashChar = std::string("\\") + found;
HintFmt dedicatedInsert = isIndented
? HintFmt("you can simply write it as %s in the string", backslashChar)
: HintFmt("you need to escape the %s itself: %s", escape, escape + escape + found);
HintFmt msg = HintFmt(
"%s is an ill-defined escape. You can drop the %s and simply write %s instead. "
"%s is an ill-defined escape sequence. In Nix, it simply means %s, therefore the %s is redundant and "
"should be removed. If the intent of the string was to mean %s instead (e.g. in a regex), %s. "
"Use %s to silence this warning.",
escape + found,
escape,
found,
escape,
backslashChar,
Uncolored(dedicatedInsert.str()),
"--extra-deprecated-features broken-string-escape"
);
/* Special case some common escapes to provide better messages */
if (found == '$' || found == '{') {
/* Someone possibly tried to escape an interpolation but used the wrong sequence.
lix/libexpr/primops.cc
+79 -6
View File
@@ -1,3 +1,4 @@
#include "libutil/error-trace.hh"
#include "lix/libutil/archive.hh"
#include "lix/libstore/derivations.hh"
#include "lix/libexpr/eval.hh"
@@ -36,6 +37,7 @@
#include <dlfcn.h>
#include <cmath>
#include <cfenv>
namespace nix {
@@ -300,8 +302,16 @@ Value prim_importNative(EvalState & state, Value ** args)
state.ctx.errors.make<EvalError>("symbol '%1%' from '%2%' resolved to NULL when a function pointer was expected", sym, path).debugThrow();
}
// Default construction of `Value` is deprecated, and uses of `Value&` out parameters
// have mostly been removed in favor of returning a `Value` instead. However in this
// particular case, this signature (ValueInitializer defined above) is externally visible,
// changing it would be an API breaking change, so for this one instance we just suppress
// the warning instead.
#pragma clang diagnostic push
#pragma clang diagnostic ignored "-Wdeprecated-declarations"
Value v;
(func)(state, v);
#pragma clang diagnostic pop
/* We don't dlclose because v may be a primop referencing a function in the shared object file */
return v;
@@ -690,22 +700,85 @@ static Value prim_addErrorContext(EvalState & state, Value ** args)
auto message = state.coerceToString(noPos, *args[0], context,
"while evaluating the error message passed to builtins.addErrorContext",
StringCoercionMode::Strict, false).toOwned();
e.addTrace(nullptr, HintFmt(message));
e.addTrace(nullptr, HintFmt(message), TraceKind::UserTrace);
throw;
}
}
static std::optional<NixInt> floatToIntChecked(NixFloat f)
{
// Required to detect overflows when converting floats to integers
#pragma STDC FENV_ACCESS ON
std::feclearexcept(FE_ALL_EXCEPT);
auto converted = llrint(f);
if (std::fetestexcept(FE_ALL_EXCEPT)) {
return std::nullopt;
}
return NixInt{converted};
}
/*
Note [floor/ceil corrupt integers]:
Integers above 2**54 that aren't a power of two get corrupted when passed
through floor/ceil.
Corrupting integers would become impossible if we just passed through integer
inputs, since the corruption actually happens when casting from int to float,
*not* from float to int, which is the one we actually safely cast.
Floats generate an error as intended if they are out of range.
In a future release, we'd like to pass through all integers, but it would be
an eval semantics change, so it's safer to error first before relaxing the
semantics again.
*/
using FloorCeilFunc = auto (*)(NixFloat) -> NixFloat;
static Value
floorCeil(std::string_view const which, FloorCeilFunc f, EvalState & state, NixFloat value, Value * arg0)
{
bool isInt = arg0->type() == nInt;
NixFloat result = f(value);
if (auto checked = floatToIntChecked(result); checked.has_value()) {
// See Note [floor/ceil corrupt integers].
if (isInt && *checked != arg0->integer()
&& !featureSettings.isEnabled(DeprecatedFeature::FloorCeilCorruptIntegers))
{
state.ctx.errors
.make<EvalError>(
"%s was corrupting your integer (was %d, became %d) in previous versions due to a "
"historical Nix bug (https://github.com/NixOS/nix/issues/12899).\n"
"This may be changed in the future to pass through integers as-is, which will change the "
"semantics of this code.\n"
"To suppress this error, use %s",
which,
arg0->integer(),
*checked,
"--extra-deprecated-features floor-ceil-corrupt-integers"
)
.debugThrow();
}
return {NewValueAs::integer, NixInt::Inner(*checked)};
} else {
state.ctx.errors.make<EvalError>("%s result %f is out of range for Nix integer (i64)", which, result)
.debugThrow();
}
}
static Value prim_ceil(EvalState & state, Value ** args)
{
auto value = state.forceFloat(*args[0], noPos,
"while evaluating the first argument passed to builtins.ceil");
return {NewValueAs::integer, NixInt::Inner(ceil(value))};
auto value = state.forceFloat(*args[0], noPos, "while evaluating the argument passed to builtins.ceil");
return floorCeil("builtins.ceil", ceil, state, value, args[0]);
}
static Value prim_floor(EvalState & state, Value ** args)
{
auto value = state.forceFloat(*args[0], noPos, "while evaluating the first argument passed to builtins.floor");
return {NewValueAs::integer, NixInt::Inner(floor(value))};
auto value = state.forceFloat(*args[0], noPos, "while evaluating the argument passed to builtins.floor");
return floorCeil("builtins.floor", floor, state, value, args[0]);
}
/* Try evaluating the argument. Success => {success=true; value=something;},
lix/libexpr/primops/fetchTree.cc
+8 -2
View File
@@ -308,10 +308,16 @@ static Value fetch(
// https://github.com/NixOS/nix/issues/4313
auto storePath = unpack
? state.aio
.blockOn(fetchers::downloadTarball(state.ctx.store, *url, name, (bool) expectedHash))
.blockOn(
fetchers::downloadTarball(
state.ctx.store, *url, name, expectedHash != Hash(HashType::SHA256)
)
)
.tree.storePath
: state.aio
.blockOn(fetchers::downloadFile(state.ctx.store, *url, name, (bool) expectedHash))
.blockOn(
fetchers::downloadFile(state.ctx.store, *url, name, expectedHash != Hash(HashType::SHA256))
)
.storePath;
if (expectedHash) {
lix/libexpr/value.cc
-9
View File
@@ -53,15 +53,6 @@ void Value::print(EvalState & state, std::ostream & str, PrintOptions options)
printValue(state, str, *this, options);
}
bool Value::isTrivial() const
{
return internalType() != tApp
&& (internalType() != tThunk
|| (thunk().expr->try_cast<ExprSet>()
&& static_cast<ExprSet *>(thunk().expr)->dynamicAttrs.empty())
|| thunk().expr->try_cast<ExprLambda>() || thunk().expr->try_cast<ExprList>());
}
Value::Value(string_t, Str * s, const NixStringContext & context)
: Value(NewValueAs::string, s, copyContext(context))
{
lix/libexpr/value.hh
+3 -7
View File
@@ -284,6 +284,7 @@ private:
InternalType internalType() const
{
// NOLINTNEXTLINE(lix-cast-to-non-fixed-enum): TAG_MASK ensures it's in range
return InternalType(raw & TAG_MASK);
}
@@ -723,6 +724,8 @@ public:
Type type() const
{
// TAG_MASK == 7, max enumerator is 5, so all possible values are in range
// NOLINTNEXTLINE(lix-cast-to-non-fixed-enum)
return Type(raw & TAG_MASK);
}
};
@@ -775,13 +778,6 @@ public:
size_t listSize() const;
/**
* Check whether forcing this value requires a trivial amount of
* computation. In particular, function applications are
* non-trivial.
*/
bool isTrivial() const;
auto listItems() const
{
struct ListIterable
lix/libfetchers/cache.cc
+24
View File
@@ -1,4 +1,6 @@
#include "lix/libfetchers/cache.hh"
#include "libstore/pathlocks.hh"
#include "libutil/hash.hh"
#include "lix/libstore/sqlite.hh"
#include "lix/libutil/async.hh"
#include "lix/libutil/sync.hh"
@@ -90,6 +92,28 @@ struct CacheImpl : Cache
co_return result::current_exception();
}
kj::Promise<Result<std::variant<std::pair<Attrs, StorePath>, PathLock>>> lookupOrLock(ref<Store> store, const Attrs & inAttrs) override try {
// In order to avoid fetching the same input multiple times
// concurrently, we first acquire a lock based on the input attributes.
auto hashResult = hashString(HashType::SHA256, attrsToJSON(inAttrs).dump());
auto lockPath = getCacheDir() + "/nix/fetcher-lock-" + hashResult.to_string(HashFormat::Base32, false);
auto pathLock = TRY_AWAIT(lockPathAsync(lockPath));
// Once this lock is acquired, we check if the input is already present
// in the cache (which requires locking the cache db)
auto lookedUp = TRY_AWAIT(lookup(store, inAttrs));
// We return either the cache hit or the lock; this allows fetchers to
// keep a lock on fetching the input in question if it's not already in
// the cache, and otherwise frees the cache db up again.
std::variant<std::pair<Attrs, StorePath>, PathLock> result(std::move(pathLock));
if (lookedUp) {
// Reassigning the variant frees the lock
result = *lookedUp;
}
co_return result;
} catch (...) {
co_return result::current_exception();
}
kj::Promise<Result<std::optional<LookupResult>>> lookupExpired(
ref<Store> store,
const Attrs & inAttrs) override
lix/libfetchers/cache.hh
+5
View File
@@ -1,6 +1,7 @@
#pragma once
///@file
#include "libstore/pathlocks.hh"
#include "lix/libfetchers/fetchers.hh"
#include "lix/libstore/path.hh"
@@ -21,6 +22,10 @@ struct Cache
ref<Store> store,
const Attrs & inAttrs) = 0;
virtual kj::Promise<Result<std::variant<std::pair<Attrs, StorePath>, PathLock>>> lookupOrLock(
ref<Store> store,
const Attrs & inAttrs) = 0;
struct LookupResult
{
bool expired = false;
lix/libfetchers/git.cc
+43 -16
View File
@@ -1,3 +1,4 @@
#include "libfetchers/attrs.hh"
#include "lix/libutil/archive.hh"
#include "lix/libutil/async-io.hh"
#include "lix/libutil/async.hh"
@@ -15,9 +16,10 @@
#include "lix/libutil/url-parts.hh"
#include "lix/libstore/pathlocks.hh"
#include "lix/libutil/users.hh"
#include "lix/libutil/git.hh"
#include "lix/libutil/logging.hh"
#include "lix/libutil/finally.hh"
#include "lix/lix-rs/main.gen.hh"
#include "lix/lix-rs/utils.hh"
#include "lix/libfetchers/fetch-settings.hh"
@@ -80,17 +82,17 @@ try {
));
std::string_view line = output;
line = line.substr(0, line.find("\n"));
if (const auto parseResult = git::parseLsRemoteLine(line)) {
switch (parseResult->kind) {
case git::LsRemoteRefLine::Kind::Symbolic:
debug("resolved HEAD ref '%s' for repo '%s'", parseResult->target, path);
break;
case git::LsRemoteRefLine::Kind::Object:
debug("resolved HEAD rev '%s' for repo '%s'", parseResult->target, path);
break;
auto line_rs = rust::to_string(line.substr(0, line.find("\n")));
if (const auto parseResult =
to_std(rust::lix::fetchers::git::LsRemoteRefLine::try_from(line_rs.as_str()).ok()))
{
auto target = to_std_string(parseResult->target.as_str());
if (parseResult->kind.matches_Object()) {
debug("resolved HEAD rev '%s' for repo '%s'", target, path);
} else {
debug("resolved HEAD ref '%s' for repo '%s'", target, path);
}
co_return parseResult->target;
co_return to_std_string(parseResult->target.as_str());
}
co_return std::nullopt;
} catch (ExecError &) {
@@ -605,9 +607,18 @@ struct GitInputScheme : InputScheme
return {std::move(storePath), input};
};
// If the input isn't present in the fetcher cache, we get a lock for
// fetching the given input. We want to hold this lock until we're done
// fetching, so we keep it in this scope.
std::optional<PathLock> fetchLock;
if (input.getRev()) {
if (auto res = TRY_AWAIT(getCache()->lookup(store, getLockedAttrs())))
auto resOrLock = TRY_AWAIT(getCache()->lookupOrLock(store, getLockedAttrs()));
if (auto res = std::get_if<std::pair<Attrs, StorePath>>(&resOrLock))
co_return makeResult(res->first, std::move(res->second));
auto lock = std::get_if<PathLock>(&resOrLock);
assert(lock);
fetchLock = std::move(*lock);
}
auto [isLocal, actualUrl_] = getActualUrl(input);
@@ -676,7 +687,10 @@ struct GitInputScheme : InputScheme
}
}
if (auto res = TRY_AWAIT(getCache()->lookup(store, unlockedAttrs))) {
// If the input isn't in the cache, we get a lock for fetching it.
// Make sure to keep this until we're done fetching!
auto cached = TRY_AWAIT(getCache()->lookupOrLock(store, unlockedAttrs));
if (auto res = std::get_if<std::pair<Attrs, StorePath>>(&cached)) {
auto rev2 = Hash::parseAny(getStrAttr(res->first, "rev"), HashType::SHA1);
if (!input.getRev() || input.getRev() == rev2) {
input.attrs.insert_or_assign("rev", rev2.gitRev());
@@ -858,8 +872,21 @@ struct GitInputScheme : InputScheme
/* Now that we know the ref, check again whether we have it in
the store. */
if (auto res = TRY_AWAIT(getCache()->lookup(store, getLockedAttrs())))
co_return makeResult(res->first, std::move(res->second));
// If we already have a lock for fetching this path, we can't use lookupOrLock because it would deadlock.
if (fetchLock) {
if (auto res = TRY_AWAIT(getCache()->lookup(store, getLockedAttrs()))) {
co_return makeResult(res->first, std::move(res->second));
}
} else {
// If we don't have the lock, we need to acquire it by using lookupOrLock.
auto resultOrLock = TRY_AWAIT(getCache()->lookupOrLock(store, getLockedAttrs()));
if (auto res = std::get_if<std::pair<Attrs, StorePath>>(&resultOrLock)) {
co_return makeResult(res->first, std::move(res->second));
}
auto lock = std::get_if<PathLock>(&resultOrLock);
assert(lock);
fetchLock = std::move(*lock);
}
Path tmpDir = createTempDir();
AutoDelete delTmpDir(tmpDir, true);
@@ -974,7 +1001,7 @@ struct GitInputScheme : InputScheme
});
Finally const _wait([&] { proc.waitAndCheck(); });
TRY_AWAIT(unpackTarfile(*proc.getStdout(), tmpDir));
TRY_AWAIT(unpackTarfile(_input.toURLString(), *proc.getStdout(), tmpDir));
}
auto storePath = TRY_AWAIT(
lix/libfetchers/github.cc
+12 -7
View File
@@ -7,10 +7,11 @@
#include "lix/libutil/result.hh"
#include "lix/libutil/types.hh"
#include "lix/libutil/url-parts.hh"
#include "lix/libutil/git.hh"
#include "lix/libutil/json.hh"
#include "lix/libfetchers/fetchers.hh"
#include "lix/libfetchers/fetch-settings.hh"
#include "lix/lix-rs/main.gen.hh"
#include "lix/lix-rs/utils.hh"
#include <optional>
#include <fstream>
@@ -486,15 +487,16 @@ struct SourceHutInputScheme : GitArchiveInputScheme
std::string line;
getline(is, line);
auto remoteLine = git::parseLsRemoteLine(line);
auto remoteLine = to_std(
rust::lix::fetchers::git::LsRemoteRefLine::try_from(rust::to_string(line).as_str()).ok()
);
if (!remoteLine) {
throw BadURL("in '%d', couldn't resolve HEAD ref '%d'", input.to_string(), ref);
}
refUri = remoteLine->target;
refUri = to_std_string(remoteLine->target.as_str());
} else {
refUri = fmt("refs/(heads|tags)/%s", ref);
}
std::regex refRegex = regex::parse(refUri);
auto file = store->toRealPath(
TRY_AWAIT(downloadFile(store, fmt("%s/info/refs", base_url), "source", false, headers))
@@ -505,9 +507,12 @@ struct SourceHutInputScheme : GitArchiveInputScheme
std::string line;
std::optional<std::string> id;
while(!id && getline(is, line)) {
auto parsedLine = git::parseLsRemoteLine(line);
if (parsedLine && parsedLine->reference && std::regex_match(*parsedLine->reference, refRegex))
id = parsedLine->target;
auto parsedLine = to_std(
rust::lix::fetchers::git::LsRemoteRefLine::try_from(rust::to_string(line).as_str()).ok()
);
if (parsedLine && (*parsedLine).matches_ref_uri(rust::to_string(refUri).as_str())) {
id = to_std_string(parsedLine->target.as_str());
}
}
if(!id)
lix/libfetchers/tarball.cc
+1 -1
View File
@@ -195,7 +195,7 @@ try {
} else {
Path tmpDir = createTempDir();
AutoDelete autoDelete(tmpDir, true);
unpackTarfile(store->toRealPath(res.storePath), tmpDir);
unpackTarfile(url, store->toRealPath(res.storePath), tmpDir);
auto members = readDirectory(tmpDir);
if (members.size() != 1)
throw nix::Error("tarball '%s' contains an unexpected number of top-level files", url);
lix/libmain/common-args.cc
+3 -3
View File
@@ -16,21 +16,21 @@ MixCommonArgs::MixCommonArgs(const std::string & programName)
.shortName = 'v',
.description = "Increase the logging verbosity level.",
.category = loggingCategory,
.handler = {[]() { verbosity = verbosityFromIntClamped(int(verbosity) + 1); }},
.handler = {[]() { setVerbosity(verbosityFromIntClamped(int(getVerbosity()) + 1)); }},
});
addFlag({
.longName = "quiet",
.description = "Decrease the logging verbosity level.",
.category = loggingCategory,
.handler = {[]() { verbosity = verbosityFromIntClamped(int(verbosity) - 1); }},
.handler = {[]() { setVerbosity(verbosityFromIntClamped(int(getVerbosity()) - 1)); }},
});
addFlag({
.longName = "debug",
.description = "Set the logging verbosity level to 'debug'.",
.category = loggingCategory,
.handler = {[]() { verbosity = lvlDebug; }},
.handler = {[]() { setVerbosity(lvlDebug); }},
});
addFlag({
lix/libmain/progress-bar.cc
+9 -6
View File
@@ -106,7 +106,7 @@ bool ProgressBar::isVerbose()
Logger::BufferState ProgressBar::log(Verbosity lvl, std::string_view s)
{
if (lvl > verbosity) {
if (lvl > getVerbosity()) {
return BufferState::HasSpace;
}
auto state(state_.lock());
@@ -142,8 +142,9 @@ Logger::BufferState ProgressBar::startActivityImpl(
{
auto state(state_.lock());
if (lvl <= verbosity && !s.empty() && type != actBuildWaiting)
if (lvl <= getVerbosity() && !s.empty() && type != actBuildWaiting) {
(void) log(*state, lvl, s + "...");
}
state->activities.emplace_back(ActInfo {
.s = s,
@@ -197,8 +198,10 @@ Logger::BufferState ProgressBar::startActivityImpl(
if ((type == actFileTransfer && hasAncestor(*state, actCopyPath, parent))
|| (type == actFileTransfer && hasAncestor(*state, actQueryPathInfo, parent))
|| (type == actCopyPath && hasAncestor(*state, actSubstitute, parent)))
|| (type == actCopyPath && hasAncestor(*state, actSubstitute, parent)) || (s == "daemon connection"))
{
i->visible = false;
}
update(*state);
return BufferState::HasSpace;
@@ -560,7 +563,7 @@ void ProgressBar::writeToStdout(std::string_view s)
restoreProgressDisplay(*state);
}
std::optional<char> ProgressBar::ask(std::string_view msg)
std::optional<std::string> ProgressBar::ask(std::string_view msg)
{
auto state(state_.lock());
if (state->paused > 0 || !isatty(STDIN_FILENO)) return {};
@@ -569,9 +572,9 @@ std::optional<char> ProgressBar::ask(std::string_view msg)
std::cerr << msg;
auto s = trim(readLine(STDIN_FILENO));
writeLogsToStderr("\e[?2026h"); // begin synchronized update
if (s.size() != 1) return {};
restoreProgressDisplay(*state);
return s[0];
// only return the string if it's not empty
return s.size() != 0 ? s : std::optional<std::string>{};
}
void ProgressBar::setPrintBuildLogs(bool printBuildLogs)
lix/libmain/progress-bar.hh
+1 -1
View File
@@ -103,7 +103,7 @@ struct ProgressBar : public Logger
void writeToStdout(std::string_view s) override;
std::optional<char> ask(std::string_view msg) override;
std::optional<std::string> ask(std::string_view msg) override;
void setPrintBuildLogs(bool printBuildLogs) override;
lix/libmain/shared.cc
+2 -12
View File
@@ -21,6 +21,7 @@
#include "lix/libutil/terminal.hh"
#include "lix/libutil/strings.hh"
#include "lix/libutil/exit.hh"
#include "lix/libutil/compile-time-features.hh"
#include <algorithm>
#include <exception>
@@ -175,12 +176,6 @@ void initNix()
}
#if __APPLE__
/* HACK: on darwin, we need can’t use sigprocmask with SIGWINCH.
* Instead, add a dummy sigaction handler, and signalHandlerThread
* can handle the rest. */
act.sa_handler = sigHandler;
if (sigaction(SIGWINCH, &act, 0)) throw SysError("handling SIGWINCH");
/* Disable SA_RESTART for interrupts, so that system calls on this thread
* error with EINTR like they do on Linux.
* Most signals on BSD systems default to SA_RESTART on, but Nix
@@ -302,14 +297,9 @@ bool LegacyArgs::processArgs(const Strings & args, bool finish)
void printVersion(const std::string & programName)
{
std::cout << fmt("%1% (Lix, like Nix) %2%", programName, nixVersion) << std::endl;
Strings cfg;
#if HAVE_BOEHMGC
cfg.push_back("gc");
#endif
cfg.push_back("signed-caches");
std::cout << "System type: " << settings.thisSystem << "\n";
std::cout << "Additional system types: " << concatStringsSep(", ", settings.extraPlatforms.get()) << "\n";
std::cout << "Features: " << concatStringsSep(", ", cfg) << "\n";
std::cout << "Features: " << concatStringsSep(", ", getNixFeatures()) << "\n";
std::cout << "System configuration file: " << settings.nixConfDir + "/nix.conf" << "\n";
std::cout << "User configuration files: " <<
concatStringsSep(":", settings.nixUserConfFiles)
lix/libstore/build-result.hh
+1 -1
View File
@@ -21,7 +21,7 @@ struct BuildResult
* Therefore, don't remove status codes, and only add new status
* codes at the end of the list.
*/
enum Status {
enum Status : uint8_t {
Built = 0,
Substituted,
AlreadyValid,
lix/libstore/build/derivation-goal.cc
+5 -6
View File
@@ -1094,8 +1094,8 @@ try {
RPC_FILL(buildReq, setNeededSystem, drv->platform);
RPC_FILL(buildReq, initDrvPath, drvPath, worker.store);
RPC_FILL(buildReq, initRequiredFeatures, parsedDrv->getRequiredSystemFeatures());
auto buildRespPromise = buildReq.send();
auto buildResp = TRY_AWAIT_RPC(buildRespPromise);
auto buildRespV = TRY_AWAIT_RPC(buildReq.send());
auto buildResp = buildRespV.getResult();
debug("hook reply is '%1%'", buildResp.toString().flatten().cStr());
@@ -1136,10 +1136,9 @@ try {
missingOutputs.insert(outputName);
}
RPC_FILL(runReq, initWantedOutputs, missingOutputs);
RPC_FILL(runReq, setDescription, buildDescription());
}
auto runPromise = runReq.send();
auto runPromise = LIX_WRAP_RPC_PROMISE_V1(runReq.send());
// build via hook is now properly running. wait for it to finish
actLock.reset();
@@ -1150,8 +1149,8 @@ try {
wrapChildHandler(runPromise.then([&](auto result) -> kj::Promise<Result<WorkResult>> {
try {
std::shared_ptr<Error> remoteError;
if (result.getResult().isBad()) {
remoteError = std::make_shared<Error>(from(result.getResult().getBad()));
if (result.has_error()) {
remoteError = std::make_shared<Error>(detail::wrap_exception_as_lix(result.error()));
logErrorInfo(remoteError->info().level, remoteError->info());
}
// close the rpc connection to have the hook exit
lix/libstore/build/derivation-goal.hh
-4
View File
@@ -360,10 +360,6 @@ public:
{
return false;
}
JobCategory jobCategory() const override {
return JobCategory::Build;
};
};
MakeError(NotDeterministic, BuildError);
lix/libstore/build/goal.hh
+1 -24
View File
@@ -27,23 +27,6 @@ typedef std::shared_ptr<Goal> GoalPtr;
*/
typedef std::set<GoalPtr> Goals;
/**
* Used as a hint to the worker on how to schedule a particular goal. For example,
* builds are typically CPU- and memory-bound, while substitutions are I/O bound.
* Using this information, the worker might decide to schedule more or fewer goals
* of each category in parallel.
*/
enum struct JobCategory {
/**
* A build of a derivation; it will use CPU and disk resources.
*/
Build,
/**
* A substitution an arbitrary store object; it will use network resources.
*/
Substitution,
};
struct Goal
{
typedef enum {ecSuccess, ecFailed, ecNoSubstituters, ecIncompleteClosure} ExitCode;
@@ -139,13 +122,7 @@ public:
return name;
}
virtual void cleanup() { }
/**
* @brief Hint for the scheduler, which concurrency limit applies.
* @see JobCategory
*/
virtual JobCategory jobCategory() const = 0;
virtual void cleanup() {}
};
}
lix/libstore/build/hook-instance.capnp
+4 -5
View File
@@ -8,14 +8,13 @@ using Types = import "/lix/libutil/types.capnp";
using Log = import "/lix/libutil/logging.capnp";
using StoreTypes = import "/lix/libstore/types.capnp";
interface HookInstance {
interface HookInstance $Types.throws(Types.v1Errors) {
interface AcceptedBuild {
run @0 (
logger :Log.LogStream,
inputs :List(StoreTypes.StorePath), # actual a set
wantedOutputs :List(Data), # actually StringSet
description :Text, # root activity description for this build
) -> (result :Types.ResultV);
);
}
struct BuildResponse {
@@ -32,11 +31,11 @@ interface HookInstance {
init @0 (
logger :Log.LogStream,
settings :Types.Settings,
) -> (result :Types.ResultV);
);
build @1 (
amWilling :Bool,
neededSystem :Data,
drvPath :StoreTypes.StorePath,
requiredFeatures :List(Data),
) -> (result :Types.Result(BuildResponse));
) -> (result :BuildResponse);
}
lix/libstore/build/hook-instance.cc
+1 -1
View File
@@ -75,7 +75,7 @@ try {
auto buildHook = canonPath(buildHookArgs.front());
buildHookArgs.emplace(std::next(buildHookArgs.begin()), baseNameOf(buildHook));
buildHookArgs.push_back(std::to_string(verbosity));
buildHookArgs.push_back(std::to_string(getVerbosity()));
/* Create the communication pipes. */
auto [selfRPC, hookRPC] = SocketPair::stream();
lix/libstore/build/local-derivation-goal.cc
+19 -2
View File
@@ -897,7 +897,7 @@ try {
if (drv->isBuiltin()) {
args.push_back("builtin-builder");
args.push_back(std::to_string(verbosity));
args.push_back(std::to_string(getVerbosity()));
std::map<std::string, AbstractConfig::SettingInfo> changedSettings;
globalConfig.getChangedSettings(changedSettings);
@@ -984,7 +984,7 @@ try {
auto groups = buildUser->getSupplementaryGIDs();
creds.setSupplementaryGroups({groups.data(), groups.size()});
}
request.setDebug(verbosity >= lvlDebug);
request.setDebug(getVerbosity() >= lvlDebug);
fillBuilderConfig(request);
@@ -1711,6 +1711,23 @@ try {
anyHashMismatchSeen = true;
// XXX: shameless layering violation hack that makes the hash mismatch error at least not utterly worthless
auto guessedUrl = getOr(drv->env, "urls", getOr(drv->env, "url", "(unknown)"));
if (guessedUrl == "(unknown)") {
if (auto structuredAttrs = parsedDrv->getStructuredAttrs()) {
auto json = *structuredAttrs;
if (json.contains("urls") && json["urls"].is_array() && !json["urls"].empty()) {
guessedUrl = "";
for (auto it = json["urls"].begin(); it != json["urls"].end(); ++it) {
if (!it->is_string()) continue;
if (guessedUrl != "") {
guessedUrl += " or ";
}
guessedUrl += it->get<std::string>();
}
} else if (json.contains("url") && json["url"].is_string()) {
guessedUrl = json["url"].get<std::string>();
}
}
}
delayedException = std::make_exception_ptr(BuildError(
"hash mismatch in fixed-output derivation '%s':\n likely URL: %s\n "
"specified: %s\n got: %s\n expected path: %s\n got path: %s",
lix/libstore/build/substitution-goal.hh
-4
View File
@@ -90,10 +90,6 @@ public:
kj::Promise<Result<WorkResult>> referencesValid() noexcept;
kj::Promise<Result<WorkResult>> tryToRun() noexcept;
kj::Promise<Result<WorkResult>> finished() noexcept;
JobCategory jobCategory() const override {
return JobCategory::Substitution;
};
};
}
lix/libstore/builtins/unpack-channel.cc
+1 -1
View File
@@ -7,7 +7,7 @@ void builtinUnpackChannel(const Path & out, const std::string & channelName, con
{
createDirs(out);
unpackTarfile(src, out);
unpackTarfile(fmt("channel:%s", channelName), src, out);
auto entries = readDirectory(out);
if (entries.size() != 1)
lix/libstore/curlfiletransfer.cc
+470
View File
@@ -0,0 +1,470 @@
#include <cstddef>
#include <new>
#include <optional>
#include <string>
#include <tuple>
#include <utility>
#include <curl/urlapi.h>
#include <kj/async.h>
#include <kj/common.h>
#include <kj/encoding.h>
#include <sys/stat.h>
#include <unistd.h>
#if ENABLE_S3
#include <aws/core/client/ClientConfiguration.h>
#endif
#include "lix/libstore/curlfiletransfer.hh"
#include "lix/libstore/curlmulti.hh"
#include "lix/libstore/filetransfer.hh"
#include "lix/libstore/s3.hh"
#include "lix/libstore/store-api.hh"
#include "lix/libutil/async-io.hh"
#include "lix/libutil/async.hh"
#include "lix/libutil/box_ptr.hh"
#include "lix/libutil/c-calls.hh"
#include "lix/libutil/error.hh"
#include "lix/libutil/logging.hh"
#include "lix/libutil/result.hh"
#include "lix/libutil/tracepoint.hh"
#include "lix/libutil/types.hh"
#if ENABLE_DTRACE
#include "trace-probes.gen.hh"
#endif
namespace nix {
curlFileTransfer::curlFileTransfer(unsigned int baseRetryTimeMs)
: multi(std::make_shared<CurlMulti>(baseRetryTimeMs))
{
}
curlFileTransfer::~curlFileTransfer()
{
multi->stopWorkerThread();
}
#if ENABLE_S3
std::tuple<std::string, std::string, StoreConfig::Params> curlFileTransfer::parseS3Uri(std::string uri)
{
auto [path, params] = splitUriAndParams(uri);
auto slash = path.find('/', 5); // 5 is the length of "s3://" prefix
if (slash == std::string::npos)
throw nix::Error("bad S3 URI '%s'", path);
std::string bucketName(path, 5, slash - 5);
std::string key(path, slash + 1);
return {bucketName, key, params};
}
#endif
kj::Promise<Result<void>> curlFileTransfer::upload(
const std::string & uri,
std::string data,
FileTransferOptions options,
const Activity * context
)
try {
TRY_AWAIT(enqueueFileTransfer(uri, std::move(options), std::move(data), false, context));
co_return result::success();
} catch (...) {
co_return result::current_exception();
}
kj::Promise<Result<std::optional<std::pair<FileTransferResult, box_ptr<AsyncInputStream>>>>>
curlFileTransfer::tryEagerTransfers(
const std::string & uri,
const FileTransferOptions & options,
const std::optional<std::string> & data,
bool noBody
)
try {
// curl transfers using file:// urls cannot be paused, and are a bit unruly
// in other ways too. since their metadata is trivial and we already have a
// backend for simple file system reads we can use that instead. we'll pass
// uploads to files to curl even so, those will fail in enqueueItem anyway.
// on all other decoding failures we also let curl fail for us a bit later.
//
// note that we use kj to decode the url, not curl. curl uses only the path
// component of the url to determine the file name, but it does note expose
// the decoding method it uses for this. for file:// transfers curl forbids
// only \0 characters in the urldecoded path, not all control characters as
// it does in the public curl_url_get(CURLUPART_PATH, CURLU_URLDECODE) api.
//
// also note: everything weird you see here is for compatibility with curl.
// we can't even fix it because nix-channel relies on this. even reading of
// directories being allowed and returning something (though hopefully it's
// enough to return anything instead of a directory listing like curl does)
if (uri.starts_with("file://") && !data.has_value()) {
if (!uri.starts_with("file:///")) {
throw FileTransferError(NotFound, std::nullopt, "file not found");
}
auto url = curl_url();
if (!url) {
throw std::bad_alloc();
}
KJ_DEFER(curl_url_cleanup(url));
curl_url_set(url, CURLUPART_URL, requireCString(uri), 0);
char * path = nullptr;
curl_url_get(url, CURLUPART_PATH, &path, 0);
auto decoded = kj::decodeUriComponent(kj::arrayPtr(path, path + strlen(path)));
if (!decoded.hadErrors && decoded.findFirst(0) == nullptr) {
Path fsPath(decoded.cStr(), decoded.size());
FileTransferResult metadata{.effectiveUri = std::string("file://") + path};
struct stat st;
AutoCloseFD fd(sys::open(fsPath, O_RDONLY));
if (!fd || fstat(fd.get(), &st) != 0) {
throw FileTransferError(
NotFound, std::nullopt, "%s: file not found (%s)", fsPath, strerror(errno)
);
}
if (S_ISDIR(st.st_mode)) {
co_return std::pair{
std::move(metadata), make_box_ptr<AsyncStringInputStream>("")
};
}
struct OwningFdStream : AsyncInputStream
{
AutoCloseFD fd;
OwningFdStream(AutoCloseFD fd) : fd(std::move(fd)) {}
kj::Promise<Result<std::optional<size_t>>>
read(void * buffer, size_t size) override
{
// NOTE the synchronous implementation used to have a buffer for
// file data, but we cannot be bothered to treat this edge case.
if (const auto got = ::read(fd.get(), buffer, size); got >= 0) {
if (got == 0) {
return {result::success(std::nullopt)};
} else {
return {result::success(got)};
}
} else {
return {result::failure(std::make_exception_ptr(SysError("reading file")
))};
}
}
};
co_return std::pair{
std::move(metadata), make_box_ptr<OwningFdStream>(std::move(fd))
};
}
}
/* Ugly hack to support s3:// URIs. */
if (uri.starts_with("s3://")) {
// FIXME: do this on a worker thread
#if ENABLE_S3
auto [bucketName, key, params] = parseS3Uri(uri);
std::string profile = getOr(params, "profile", "");
std::string region = getOr(params, "region", Aws::Region::US_EAST_1);
std::string scheme = getOr(params, "scheme", "");
std::string endpoint = getOr(params, "endpoint", "");
S3Helper s3Helper(profile, region, scheme, endpoint);
// FIXME: implement ETag
auto s3Res = TRY_AWAIT(s3Helper.getObject(bucketName, key));
FileTransferResult res;
if (!s3Res.data)
throw FileTransferError(NotFound, "S3 object '%s' does not exist", uri);
struct OwningStringStream : private std::string, AsyncStringInputStream
{
OwningStringStream(std::string data)
: std::string(std::move(data))
, AsyncStringInputStream(*this)
{
}
};
co_return std::pair{res, make_box_ptr<OwningStringStream>(std::move(*s3Res.data))};
#else
throw nix::Error(
"cannot download '%s' because Lix is not built with S3 support", uri
);
#endif
}
co_return std::nullopt;
} catch (...) {
co_return result::current_exception();
}
kj::Promise<Result<std::pair<FileTransferResult, box_ptr<AsyncInputStream>>>>
curlFileTransfer::enqueueFileTransfer(
const std::string & uri,
FileTransferOptions && options,
std::optional<std::string> data,
bool noBody,
const Activity * context
)
try {
if (auto eager = TRY_AWAIT(tryEagerTransfers(uri, options, data, noBody))) {
co_return std::move(*eager);
}
auto source = make_box_ptr<TransferStream>(
*this, uri, std::move(options), std::move(data), noBody, context
);
TRY_AWAIT(source->init());
TRY_AWAIT(source->awaitData());
co_return {source->metadata, std::move(source)};
} catch (...) {
co_return result::current_exception();
}
curlFileTransfer::TransferStream::TransferStream(
curlFileTransfer & parent,
const std::string & uri,
FileTransferOptions && options,
std::optional<std::string> data,
bool noBody,
const Activity * context
)
: parent(parent.multi)
, uri(uri)
, options(options)
, data(std::move(data))
, noBody(noBody)
, parentAct(context)
, backoff(backoffTimeouts(
fileTransferSettings.tries,
std::chrono::seconds(fileTransferSettings.maxConnectTimeout.get()),
std::chrono::seconds(fileTransferSettings.initialConnectTimeout.get()),
std::chrono::milliseconds(this->parent->baseRetryTimeMs)
))
{
}
curlFileTransfer::TransferStream::~TransferStream()
{
// wake up the download thread if it's still going and have it abort
try {
if (transfer) {
parent->cancel(transfer);
}
} catch (...) {
ignoreExceptionInDestructor();
}
}
kj::Promise<Result<void>> curlFileTransfer::TransferStream::init()
try {
metadata = TRY_AWAIT(withRetries(
[&]() {
return startTransfer(
uri, std::chrono::seconds(fileTransferSettings.initialConnectTimeout.get())
);
},
[&](const std::chrono::milliseconds & timeout) {
return startTransfer(uri, timeout);
}
));
co_return result::success();
} catch (...) {
co_return result::current_exception();
}
kj::Promise<Result<FileTransferResult>> curlFileTransfer::TransferStream::startTransfer(
const std::string & uri,
const std::chrono::milliseconds & timeout,
curl_off_t offset
)
try {
auto uploadData = data ? std::optional(std::string_view(*data)) : std::nullopt;
auto pfp = kj::newPromiseAndCrossThreadFulfiller<Result<FileTransferResult>>();
transfer = std::make_shared<TransferItem>(
uri,
std::move(options),
parentAct,
uploadData,
noBody,
offset,
std::move(pfp.fulfiller),
timeout
);
parent->enqueueItem(transfer);
co_return TRY_AWAIT(pfp.promise);
} catch (...) {
co_return result::current_exception();
}
kj::Promise<Result<void>> curlFileTransfer::TransferStream::prepareRetry(
const std::string & context,
const std::chrono::milliseconds & waitTime,
unsigned int attempt
)
try {
if (totalReceived) {
printTaggedWarning(
"%s; retrying from offset %d in %d ms (attempt %d/%d)",
Uncolored(context),
totalReceived,
waitTime.count(),
Uncolored(attempt),
Uncolored(tries)
);
} else {
printTaggedWarning(
"%s; retrying in %d ms (attempt %d/%d)",
Uncolored(context),
waitTime.count(),
Uncolored(attempt),
Uncolored(tries)
);
}
co_await AIO().provider.getTimer().afterDelay(waitTime.count() * kj::MILLISECONDS);
co_return result::success();
} catch (...) {
co_return result::current_exception();
}
kj::Promise<Result<void>> curlFileTransfer::TransferStream::restartTransfer(const std::chrono::milliseconds & timeout)
try {
auto onChange =
[&](std::string_view what, std::string_view from, std::string_view to, bool throw_
) -> void {
if (!from.empty() && from != to) {
FileTransferError e = FileTransferError(
Misc,
{},
"uri %s changed %s from %s to %s during transfer",
uri,
what,
from,
to
);
if (throw_) {
throw e;
}
logWarning(e.info());
}
};
// use the effective URI of the previous transfer for retries. this avoids
// some silent corruption if a redirect changes between starting and retry
const auto & uri = metadata.effectiveUri.empty() ? this->uri : metadata.effectiveUri;
auto newMeta = TRY_AWAIT(startTransfer(uri, timeout, totalReceived));
onChange("final destination", metadata.effectiveUri, newMeta.effectiveUri, false);
onChange("ETag", metadata.etag, newMeta.etag, true);
onChange(
"immutable url",
metadata.immutableUrl.value_or(""),
newMeta.immutableUrl.value_or(""),
true
);
co_return result::success();
} catch (...) {
co_return result::current_exception();
}
kj::Promise<Result<bool>> curlFileTransfer::TransferStream::waitForData()
try {
/* Grab data if available, otherwise wait for the download
thread to wake us up. */
std::optional<kj::Promise<void>> signal;
while (buffered.empty()) {
if (signal) {
co_await *signal;
signal.reset();
}
auto state(transfer->downloadState.lock());
if (!state->data.empty()) {
chunk = std::exchange(state->data, {});
buffered = chunk;
totalReceived += chunk.size();
parent->unpause(transfer);
} else if (state->exc) {
std::rethrow_exception(state->exc);
} else if (state->done) {
co_return false;
} else {
parent->unpause(transfer);
signal = state->wait();
}
}
co_return true;
} catch (...) {
co_return result::current_exception();
}
kj::Promise<Result<bool>> curlFileTransfer::TransferStream::restartAndWaitForData(const std::chrono::milliseconds & timeout)
try {
TRY_AWAIT(restartTransfer(timeout));
co_return TRY_AWAIT(waitForData());
} catch (...) {
co_return result::current_exception();
}
kj::Promise<Result<bool>> curlFileTransfer::TransferStream::awaitData()
try {
co_return TRY_AWAIT(withRetries(
[&] { return waitForData(); },
[&](const std::chrono::milliseconds & timeout) {
return restartAndWaitForData(timeout);
}
));
} catch (...) {
co_return result::current_exception();
}
kj::Promise<Result<std::optional<size_t>>> curlFileTransfer::TransferStream::read(void * buffer, size_t len)
try {
TRACE(LIX_STORE_FILETRANSFER_READ(uri.c_str(), len));
size_t total = 0;
auto data = static_cast<char *>(buffer);
while (total < len && TRY_AWAIT(awaitData())) {
const auto available = std::min(len - total, buffered.size());
memcpy(data + total, buffered.data(), available);
buffered.remove_prefix(available);
total += available;
}
if (total == 0) {
co_return std::nullopt;
} else {
co_return total;
}
} catch (...) {
co_return result::current_exception();
}
kj::Promise<Result<bool>>
curlFileTransfer::exists(const std::string & uri, FileTransferOptions options, const Activity * context)
try {
try {
TRY_AWAIT(enqueueFileTransfer(uri, std::move(options), std::nullopt, true, context));
co_return true;
} catch (FileTransferError & e) {
/* S3 buckets return 403 if a file doesn't exist and the
bucket is unlistable, so treat 403 as 404. */
if (e.error == FileTransfer::NotFound || e.error == FileTransfer::Forbidden)
co_return false;
throw;
}
} catch (...) {
co_return result::current_exception();
}
kj::Promise<Result<std::pair<FileTransferResult, box_ptr<AsyncInputStream>>>> curlFileTransfer::download(
const std::string & uri, FileTransferOptions options, const Activity * context
)
{
return enqueueFileTransfer(uri, std::move(options), std::nullopt, false, context);
}
}
lix/libstore/curlfiletransfer.hh
+170
View File
@@ -0,0 +1,170 @@
#pragma once
///@file
#include "lix/libstore/filetransfer.hh"
#include "lix/libstore/store-api.hh"
#include "lix/libstore/transferitem.hh"
#include "lix/libutil/async-io.hh"
#include "lix/libutil/backoff.hh"
#include "lix/libutil/box_ptr.hh"
#include "lix/libutil/result.hh"
#include <memory>
#include <optional>
#include <string>
#include <string_view>
#include <tuple>
#include <utility>
#include <kj/async.h>
namespace nix {
struct CurlMulti;
struct curlFileTransfer : public FileTransfer
{
// Types.
public:
template<typename T>
using Async = kj::Promise<Result<T>>;
struct TransferStream : AsyncInputStream
{
std::shared_ptr<CurlMulti> parent;
std::string uri;
FileTransferOptions options;
std::optional<std::string> data;
bool noBody;
const Activity * parentAct;
std::shared_ptr<TransferItem> transfer;
FileTransferResult metadata;
std::string chunk;
std::string_view buffered;
const size_t tries = fileTransferSettings.tries;
curl_off_t totalReceived = 0;
Generator<BackoffTiming> backoff;
TransferStream(
curlFileTransfer & parent,
const std::string & uri,
FileTransferOptions && options,
std::optional<std::string> data,
bool noBody,
const Activity * context
);
~TransferStream();
kj::Promise<Result<void>> init();
inline auto withRetries(auto && initial, auto && retry) -> decltype(initial())
try {
std::optional<std::string> retryContext;
BackoffTiming timings;
while (true) {
try {
if (retryContext) {
TRY_AWAIT(prepareRetry(*retryContext, timings.waitTime, timings.attempt));
co_return TRY_AWAIT(retry(timings.downloadTimeout));
} else {
co_return TRY_AWAIT(initial());
}
} catch (FileTransferError & e) {
auto next = backoff.next();
// If this is a transient error, then maybe retry after a while. after any
// bytes have been received we require range support to proceed, otherwise
// we'd need to start from scratch and discard everything we already have.
if (e.error != Transient || data.has_value() || !next.has_value()
|| (totalReceived > 0 && !transfer->acceptsRanges()))
{
throw;
}
retryContext = e.what();
timings = *next;
}
}
} catch (...) {
co_return result::current_exception();
}
kj::Promise<Result<FileTransferResult>> startTransfer(
const std::string & uri,
const std::chrono::milliseconds & timeout,
curl_off_t offset = 0
);
kj::Promise<Result<void>> prepareRetry(
const std::string & context,
const std::chrono::milliseconds & waitTime,
unsigned int attempt
);
kj::Promise<Result<void>> restartTransfer(const std::chrono::milliseconds & timeout);
kj::Promise<Result<bool>> waitForData();
kj::Promise<Result<bool>> restartAndWaitForData(const std::chrono::milliseconds & timeout);
kj::Promise<Result<bool>> awaitData();
kj::Promise<Result<std::optional<size_t>>> read(void * buffer, size_t len) override;
};
// Fields.
public:
std::shared_ptr<CurlMulti> multi;
// Specials.
public:
curlFileTransfer(unsigned int baseRetryTimeMs);
~curlFileTransfer();
#if ENABLE_S3
using S3Uri = std::tuple<std::string, std::string, StoreConfig::Params>;
static auto parseS3Uri(std::string url) -> S3Uri;
#endif
// Overrides.
public:
auto upload(
std::string const & uri,
std::string data,
FileTransferOptions options,
Activity const * context
) -> Async<void> override;
auto exists(
std::string const & uri,
FileTransferOptions options,
Activity const * context
) -> Async<bool> override;
auto download(
std::string const & uri,
FileTransferOptions options,
Activity const * context = nullptr
) -> Async<std::pair<FileTransferResult, box_ptr<AsyncInputStream>>> override;
// Actual API.
public:
auto tryEagerTransfers(
std::string const & url,
FileTransferOptions const & options,
std::optional<std::string> const & data,
bool noBody
) -> Async<std::optional<std::pair<FileTransferResult, box_ptr<AsyncInputStream>>>>;
auto enqueueFileTransfer(
std::string const & uri,
FileTransferOptions && options,
std::optional<std::string> data,
bool noBody,
Activity const * context
) -> Async<std::pair<FileTransferResult, box_ptr<AsyncInputStream>>>;
};
}
lix/libstore/curlmulti.cc
+259
View File
@@ -0,0 +1,259 @@
#include "lix/libstore/curlmulti.hh"
#include "lix/libstore/filetransfer.hh"
#include "lix/libutil/error.hh"
#include "lix/libutil/logging.hh"
#include "lix/libutil/namespaces.hh"
#include "lix/libutil/signals.hh"
#include "lix/libutil/thread-name.hh"
#include <algorithm>
#include <climits>
#include <cstdint>
#include <exception>
#include <future>
#include <map>
#include <memory>
#include <mutex>
#include <thread>
#include <vector>
#include <curl/curl.h>
#include <curl/multi.h>
#include <kj/common.h>
namespace nix {
void CurlMulti::State::quit()
{
quitting = true;
// We will not be processing any more requests.
for (auto & item : incoming) {
item->finish(CURLE_ABORTED_BY_CALLBACK);
}
incoming.clear();
unpause.clear();
// make a note that we're dying and acknowledge all pending cancel
// requests by individual transfers. not doing this can cause bugs
// like #1218 in which the process deadlocks waiting for transfers
// to cancel with no download thread to make this happen; this was
// likely caused by a transfer requesting a cancellation *exactly*
// before a signal was received, causing the curl thread to die in
// a hurry without processing cancellations. the transfer is stuck
// from that point on, and since this happened in a destructor the
// entire process locked up solid. curl exceptions could have also
// caused this; we set the `quit` flag just in case to avoid this.
for (auto & [item, promise] : cancel) {
promise.set_value();
}
}
CurlMulti::CurlMulti(unsigned int baseRetryTimeMs)
: curlm(curl_multi_init())
, baseRetryTimeMs(baseRetryTimeMs)
{
if (curlm == nullptr) {
throw FileTransferError(FileTransfer::Misc, {}, "could not allocate curl handle");
}
static std::once_flag globalInit;
std::call_once(globalInit, curl_global_init, CURL_GLOBAL_ALL);
curl_multi_setopt(curlm.get(), CURLMOPT_PIPELINING, CURLPIPE_MULTIPLEX);
curl_multi_setopt(curlm.get(), CURLMOPT_MAX_TOTAL_CONNECTIONS,
fileTransferSettings.httpConnections.get());
workerThread = std::thread([&]() {
setCurrentThreadName("curlFileTransfer worker");
workerThreadEntry();
});
}
CurlMulti::~CurlMulti()
{
try {
stopWorkerThread();
} catch (nix::Error & e) {
// This can only fail if a socket to our own process cannot be
// written to, so it is always a bug in the program if it fails.
//
// Joining the thread would probably only cause a deadlock if this
// happened, so just die on purpose.
printError("failed to join curl file transfer worker thread: %1%", e.what());
std::terminate();
}
workerThread.join();
}
void CurlMulti::unpause(const std::shared_ptr<TransferItem> & transfer)
{
auto lock = state_.lock();
lock->unpause.push_back(transfer);
wakeup(*lock);
}
void CurlMulti::cancel(const std::shared_ptr<TransferItem> & transfer)
{
std::promise<void> promise;
auto wait = promise.get_future();
{
auto lock = state_.lock();
if (lock->quitting) {
return;
}
lock->cancel[transfer] = std::move(promise);
wakeup(*lock);
}
wait.get();
}
void CurlMulti::wakeup(State & locked)
{
locked.workAvailable = true;
if (auto mc = curl_multi_wakeup(curlm.get()))
throw nix::Error("unexpected error from curl_multi_wakeup(): %s", curl_multi_strerror(mc));
}
void CurlMulti::stopWorkerThread()
{
/* Signal the worker thread to exit. */
auto state(state_.lock());
state->quit();
wakeup(*state);
}
void CurlMulti::workerThreadMain()
{
/* Cause this thread to be notified on SIGINT. */
auto callback = createInterruptCallback([&]() {
stopWorkerThread();
});
unshareFilesystem();
std::map<CURL *, std::shared_ptr<TransferItem>> items;
// clear all current transfers in case of an early exit, as can happen
// via Interrupted if the interruption occured right before a log call
KJ_DEFER({
for (auto & [_, item] : items) {
item->finish(CURLE_ABORTED_BY_CALLBACK);
}
auto lock = state_.lock();
lock->quit();
});
bool quit = false;
// NOTE: we will need to use CURLMOPT_TIMERFUNCTION to integrate this
// loop with kj. until then curl will handle its timeouts internally.
int64_t timeoutMs = INT64_MAX;
while (!quit) {
/* Let curl do its thing. */
int running;
CURLMcode mc = curl_multi_perform(curlm.get(), &running);
if (mc != CURLM_OK)
throw nix::Error("unexpected error from curl_multi_perform(): %s", curl_multi_strerror(mc));
/* Set the promises of any finished requests. */
CURLMsg * msg;
int left;
while ((msg = curl_multi_info_read(curlm.get(), &left))) {
if (msg->msg == CURLMSG_DONE) {
auto i = items.find(msg->easy_handle);
assert(i != items.end());
i->second->finish(msg->data.result);
curl_multi_remove_handle(curlm.get(), i->second->req.get());
items.erase(i);
}
}
{
auto cancel = [&] { return std::exchange(state_.lock()->cancel, {}); }();
for (auto & [item, promise] : cancel) {
curl_multi_remove_handle(curlm.get(), item->req.get());
items.erase(item->req.get());
promise.set_value();
}
}
/* Add new curl requests from the incoming requests queue,
except for requests that are embargoed (waiting for a
retry timeout to expire). */
std::vector<std::shared_ptr<TransferItem>> incoming;
std::vector<std::shared_ptr<TransferItem>> unpause;
bool haveWork = false;
timeoutMs = 10'000;
{
auto state(state_.lock());
incoming = std::exchange(state->incoming, {});
unpause = std::exchange(state->unpause, {});
haveWork = std::exchange(state->workAvailable, false);
quit = state->quitting;
}
for (auto & item : incoming) {
debug("starting %s of %s", item->verb(), item->uri);
curl_multi_add_handle(curlm.get(), item->req.get());
items[item->req.get()] = item;
}
// exit immediately and abort all running transfers. waiting for transfers to finish
// before exiting this loop may hang the shutdown procedure forever, e.g. if blocked
// transfers would be destroyed (thus aborted) after the curl thread for any reason.
if (quit) {
break;
}
for (auto & item : unpause) {
curl_easy_pause(item->req.get(), CURLPAUSE_CONT);
}
if (haveWork) {
timeoutMs = 0;
}
/* Wait for activity, including wakeup events. */
mc = curl_multi_poll(curlm.get(), nullptr, 0, timeoutMs, nullptr);
if (mc != CURLM_OK)
throw nix::Error("unexpected error from curl_multi_poll(): %s", curl_multi_strerror(mc));
}
debug("download thread shutting down");
}
void CurlMulti::workerThreadEntry()
{
try {
workerThreadMain();
} catch (nix::Interrupted & e) {
} catch (std::exception & e) { // NOLINT(lix-foreign-exceptions)
printError("unexpected error in download thread: %s", e.what());
} catch (...) {
printError("unexpected error in download thread");
}
state_.lock()->quit();
}
void CurlMulti::enqueueItem(std::shared_ptr<TransferItem> item)
{
if (item->uploadData
&& !item->uri.starts_with("http://")
&& !item->uri.starts_with("https://"))
throw nix::Error("uploading to '%s' is not supported", item->uri);
{
auto state(state_.lock());
if (state->quitting)
throw nix::Error("cannot enqueue download request because the download thread is shutting down");
state->incoming.push_back(item);
wakeup(*state);
}
}
} // namespace nix
lix/libstore/curlmulti.hh
+61
View File
@@ -0,0 +1,61 @@
#pragma once
///@file
#include "lix/libstore/transferitem.hh"
#include "lix/libutil/sync.hh"
#include <future>
#include <map>
#include <memory>
#include <thread>
#include <vector>
#include <curl/multi.h>
namespace nix {
struct CurlMulti
{
// Types.
public:
struct State
{
bool quitting = false;
bool workAvailable = false;
std::vector<std::shared_ptr<TransferItem>> incoming;
std::vector<std::shared_ptr<TransferItem>> unpause;
std::map<std::shared_ptr<TransferItem>, std::promise<void>> cancel;
void quit();
};
// Fields.
public:
std::unique_ptr<CURLM, decltype([](auto * m) { curl_multi_cleanup(m); })> curlm;
const unsigned int baseRetryTimeMs;
Sync<State> state_;
std::thread workerThread;
// Specials.
public:
CurlMulti(unsigned int baseRetryTimeMs);
~CurlMulti();
// Actual API.
public:
void unpause(std::shared_ptr<TransferItem> const & transfer);
void cancel(std::shared_ptr<TransferItem> const & transfer);
void wakeup(State & locked);
void stopWorkerThread();
void workerThreadMain();
void workerThreadEntry();
void enqueueItem(std::shared_ptr<TransferItem> item);
};
} // namespace nix
lix/libstore/daemon.capnp
+54
View File
@@ -0,0 +1,54 @@
@0xd8aa4d286ba6797b;
# IMPORTANT NOTICE
#
# these definitions are EXPERIMENTAL and come with NO stability guarantees
using Cxx = import "/capnp/c++.capnp";
$Cxx.namespace("nix::rpc::daemon");
$Cxx.allowCancellation;
using T = import "/lix/libutil/types.capnp";
using Log = import "/lix/libutil/logging.capnp";
struct ProtocolDescription {
id @0 :Text;
description @1 :Text;
}
interface Bootstrap {
supported @0 () -> (protocols :List(ProtocolDescription));
request @1 (
clientInfo :Text,
protocol :Text,
) -> (result :Protocol);
}
interface Protocol {
# TODO maybe add information or something
}
interface LegacyBoot extends(Protocol) $T.throws(T.v1Errors) {
enum Trust {
unknown @0;
untrusted @1;
trusted @2;
}
init @0 (
logger :Log.LogStream,
replyStream :LegacyStream,
) -> (result :InitResult);
struct InitResult {
requestStream @0 :LegacyStream;
trust @1 :Trust;
version @2 :Text;
}
}
interface LegacyStream $T.throws(T.v1Errors) {
feed @0 (raw :Data) -> stream;
# must be called before a new op is started, otherwise errors may get lost
sync @1 ();
}
lix/libstore/daemon.cc
+353 -51
View File
@@ -1,5 +1,7 @@
#include "lix/libstore/daemon.hh"
#include "filetransfer.hh"
#include "libutil/async.hh"
#include "libutil/logging-rpc.hh"
#include "lix/libutil/async-io.hh"
#include "lix/libutil/monitor-fd.hh"
#include "lix/libstore/worker-protocol.hh"
@@ -17,12 +19,21 @@
#include "lix/libutil/serialise.hh"
#include "lix/libutil/strings.hh"
#include "lix/libutil/args.hh"
#include "lix/libstore/daemon.capnp.h"
#include "lix/libutil/rpc.hh"
#include "lix/libutil/types-rpc.hh"
#include <boost/core/demangle.hpp>
#include <capnp/rpc-twoparty.h>
#include <cstdint>
#include <ctime>
#include <kj/encoding.h>
#include <kj/exception.h>
#include <kj/memory.h>
#include <sstream>
const std::string nix::rpc::daemon::UNSTABLE_LEGACY_TUNNELED = "lix/legacy/" PACKAGE_VERSION;
namespace nix::daemon {
Sink & operator << (Sink & sink, const Logger::Fields & fields)
@@ -86,7 +97,7 @@ struct TunnelLogger : public Logger
BufferState log(Verbosity lvl, std::string_view s) override
{
if (lvl > verbosity) {
if (lvl > getVerbosity()) {
return BufferState::HasSpace;
}
@@ -97,7 +108,7 @@ struct TunnelLogger : public Logger
BufferState logEI(const ErrorInfo & ei) override
{
if (ei.level > verbosity) {
if (ei.level > getVerbosity()) {
return BufferState::HasSpace;
}
@@ -186,7 +197,7 @@ struct ClientSettings
settings.keepFailed.override(keepFailed);
settings.keepGoing.override(keepGoing);
settings.tryFallback.override(tryFallback);
nix::verbosity = verbosity;
setVerbosity(verbosity);
settings.maxBuildJobs.override(maxBuildJobs);
settings.maxSilentTime.override(maxSilentTime);
settings.verboseBuild = verboseBuild;
@@ -834,11 +845,12 @@ static void performOp(AsyncIoRoot & aio, TunnelLogger * logger, ref<Store> store
case WorkerProto::Op::AddBuildLog: {
StorePath path{readString(from)};
logger->startWork();
if (!trusted)
throw Error("you are not privileged to add logs");
auto & logStore = require<LogStore>(*store);
{
FramedSource source(from);
if (!trusted) {
throw Error("you are not privileged to add logs");
}
StringSink sink;
source.drainInto(sink);
aio.blockOn(logStore.addBuildLog(path, sink.s));
@@ -857,12 +869,66 @@ static void performOp(AsyncIoRoot & aio, TunnelLogger * logger, ref<Store> store
}
}
void processConnection(
static void processLegacyRequests(
AsyncIoRoot & aio,
Logger * prevLogger,
TunnelLogger * tunnelLogger,
ref<Store> store,
FdSource & from,
FdSink & to,
TrustedFlag trusted)
TrustedFlag trusted,
WorkerProto::Version clientVersion
)
{
unsigned int opCount = 0;
Finally finally([&]() { printMsgUsing(prevLogger, lvlDebug, "%d operations", opCount); });
while (true) {
WorkerProto::Op op;
try {
op = (enum WorkerProto::Op) readNum<unsigned>(from);
} catch (Interrupted & e) {
break;
} catch (EndOfFile & e) {
break;
}
printMsgUsing(prevLogger, lvlDebug, "received daemon op %d", op);
opCount++;
debug("performing daemon worker op: %d", op);
try {
KJ_DEFER(aio.blockOn(logger->flush()));
performOp(aio, tunnelLogger, store, trusted, clientVersion, from, to, op);
} catch (Error & e) {
/* If we're not in a state where we can send replies, then
something went wrong processing the input of the
client. This can happen especially if I/O errors occur
during addTextToStore() / importPath(). If that
happens, just send the error message and exit. */
bool errorAllowed = tunnelLogger->state_.lock()->canSendStderr;
tunnelLogger->stopWork(&e);
if (!errorAllowed) {
throw;
}
} catch (std::bad_alloc & e) {
auto ex = Error("Lix daemon out of memory");
tunnelLogger->stopWork(&ex);
throw;
}
to.flush();
assert(!tunnelLogger->state_.lock()->canSendStderr);
}
}
void processLegacyConnection(
AsyncIoRoot & aio, ref<Store> store, FdSource & from, FdSink & to, TrustedFlag trusted
)
{
auto monitor = std::make_unique<MonitorFdHup>(from.fd);
@@ -880,10 +946,6 @@ void processConnection(
auto prevLogger = nix::logger;
logger = tunnelLogger;
unsigned int opCount = 0;
Finally finally([&]() { printMsgUsing(prevLogger, lvlDebug, "%d operations", opCount); });
// FIXME: what is *supposed* to be in this even?
if (readNum<unsigned>(from)) {
// Obsolete CPU affinity.
@@ -906,49 +968,10 @@ void processConnection(
tunnelLogger->startWork();
try {
tunnelLogger->stopWork();
to.flush();
/* Process client requests. */
while (true) {
WorkerProto::Op op;
try {
op = (enum WorkerProto::Op) readNum<unsigned>(from);
} catch (Interrupted & e) {
break;
} catch (EndOfFile & e) {
break;
}
printMsgUsing(prevLogger, lvlDebug, "received daemon op %d", op);
opCount++;
debug("performing daemon worker op: %d", op);
try {
performOp(aio, tunnelLogger, store, trusted, clientVersion, from, to, op);
} catch (Error & e) {
/* If we're not in a state where we can send replies, then
something went wrong processing the input of the
client. This can happen especially if I/O errors occur
during addTextToStore() / importPath(). If that
happens, just send the error message and exit. */
bool errorAllowed = tunnelLogger->state_.lock()->canSendStderr;
tunnelLogger->stopWork(&e);
if (!errorAllowed) throw;
} catch (std::bad_alloc & e) {
auto ex = Error("Lix daemon out of memory");
tunnelLogger->stopWork(&ex);
throw;
}
to.flush();
assert(!tunnelLogger->state_.lock()->canSendStderr);
};
processLegacyRequests(aio, prevLogger, tunnelLogger, store, from, to, trusted, clientVersion);
} catch (Error & e) {
tunnelLogger->stopWork(&e);
to.flush();
@@ -967,4 +990,283 @@ void processConnection(
}
}
namespace {
using namespace rpc::daemon;
// Shared state for all legacy protocol implementation structs
struct LegacyState
{
ref<Store> store;
TrustedFlag trusted;
LegacyState(ref<Store> store, TrustedFlag trusted) : store(store), trusted(trusted) {}
};
struct RequestStreamImpl final : LegacyStream::Server
{
ref<LegacyState> state;
std::exception_ptr error;
AsyncFdIoStream workerSock;
kj::Promise<void> responseForwarder;
RequestStreamImpl(
ref<LegacyState> state,
kj::Promise<std::exception_ptr> error,
LegacyStream::Client callbacks,
AutoCloseFD workerFd
)
: state(state)
, workerSock(std::move(workerFd))
, responseForwarder(forwardResponse(callbacks).exclusiveJoin(
error.then([&](std::exception_ptr e) -> kj::Promise<void> {
onError(e);
return kj::NEVER_DONE;
})
))
{
}
void onError(std::exception_ptr e)
{
if (!error) {
error = e;
}
}
kj::Promise<void> forwardResponse(LegacyStream::Client callbacks)
try {
std::array<char, 8192> buf;
while (true) {
if (auto got = TRY_AWAIT(workerSock.read(buf.data(), buf.size())); !got) {
break;
} else {
auto req = callbacks.feedRequest();
req.initRaw(*got);
std::copy(buf.begin(), buf.begin() + *got, req.getRaw().begin());
TRY_AWAIT_RPC(req.send());
}
}
TRY_AWAIT_RPC(callbacks.syncRequest().send());
} catch (...) {
onError(std::current_exception());
}
kj::Promise<void> feed(FeedContext context) override
try {
if (error) {
std::rethrow_exception(error);
}
auto bytes = context.getParams().getRaw();
TRY_AWAIT(workerSock.writeFull(bytes.begin(), bytes.size()));
} catch (...) {
onError(std::current_exception());
rpc::rethrow_as_rpc_error();
}
kj::Promise<void> sync(SyncContext context) override
try {
TRY_AWAIT(logger->flush());
if (error) {
std::rethrow_exception(error);
}
} catch (...) {
rpc::rethrow_as_rpc_error();
}
};
struct LegacyBootImpl final : LegacyBoot::Server
{
ref<LegacyState> state;
bool used = false;
LegacyBootImpl(TrustedFlag trusted, ref<Store> store) : state(make_ref<LegacyState>(store, trusted)) {}
kj::Promise<void> init(InitContext context) override
try {
if (used) {
throw Error("connection already initialized");
}
auto prevLogger = logger;
logger = rpc::log::makeRpcLoggerClient(context.getParams().getLogger());
auto args = context.getParams();
auto result = context.initResults().initResult();
// We and the underlying store both need to trust the client for it to be trusted.
if (!state->trusted) {
result.setTrust(LegacyBoot::Trust::UNTRUSTED);
} else if (auto trust = TRY_AWAIT(state->store->isTrustedClient()); trust) {
result.setTrust(*trust ? LegacyBoot::Trust::TRUSTED : LegacyBoot::Trust::UNTRUSTED);
} else {
result.setTrust(LegacyBoot::Trust::UNKNOWN);
}
result.setVersion(PACKAGE_VERSION);
auto [rpcSock, workerSock] = SocketPair::stream();
auto pfp = kj::newPromiseAndCrossThreadFulfiller<std::exception_ptr>();
struct Request
{
AutoCloseFD fd;
kj::Own<kj::CrossThreadPromiseFulfiller<std::exception_ptr>> signal;
Request(AutoCloseFD fd, kj::Own<kj::CrossThreadPromiseFulfiller<std::exception_ptr>> fulfiller)
: fd(std::move(fd))
, signal(std::move(fulfiller))
{
}
};
auto req = make_ref<Request>(std::move(rpcSock), std::move(pfp.fulfiller));
auto legacyThread = std::async(std::launch::async, [prevLogger, state{state}, req] {
AsyncIoRoot aio;
FdSource from(req->fd.get());
FdSink to(req->fd.get());
TunnelLogger logger(to, PROTOCOL_VERSION);
try {
processLegacyRequests(
aio, prevLogger, &logger, state->store, from, to, state->trusted, PROTOCOL_VERSION
);
} catch (Error & e) {
req->signal->fulfill(std::current_exception());
} catch (std::bad_alloc & e) {
req->signal->fulfill(std::make_exception_ptr(Error("Lix daemon out of memory")));
} catch (std::exception & e) { // NOLINT(lix-foreign-exceptions)
// TODO print stack trace to daemon log, maybe crash?
// boost stacktrace has from_current_exception (at a cost) with not-great symbolization,
// cpptrace has a *much* better symbolizer (at unknown cost)
req->signal->fulfill(
std::make_exception_ptr(Error(
"Unexpected exception on the Lix daemon; this is a bug in Lix.\n"
"We would appreciate a report of the circumstances it happened in at "
"https://git.lix.systems/lix-project/lix.\n%s: %s",
Uncolored(boost::core::demangle(typeid(e).name())),
e.what()
))
);
} catch (...) {
// TODO print stack trace to daemon log, maybe crash?
req->signal->fulfill(
std::make_exception_ptr(Error(
"Unexpected exception on the Lix daemon; this is a bug in Lix.\n"
"We would appreciate a report of the circumstances it happened in at "
"https://git.lix.systems/lix-project/lix.\n"
))
);
}
});
result.setRequestStream(
kj::heap<RequestStreamImpl>(
state, std::move(pfp.promise), args.getReplyStream(), std::move(workerSock)
)
.attach(std::move(legacyThread))
);
used = true;
} catch (...) {
rpc::rethrow_as_rpc_error();
}
};
struct BootstrapImpl final : Bootstrap::Server
{
struct ProtocolEntry
{
std::string description;
std::function<rpc::daemon::Protocol::Client(TrustedFlag, ref<Store>)> factory;
};
TrustedFlag trusted;
ref<Store> store;
std::map<kj::StringPtr, ProtocolEntry> protocols;
bool used = false;
BootstrapImpl(TrustedFlag trusted, ref<Store> store) : trusted(trusted), store(store)
{
if (store->isThreadSafe()) {
protocols.emplace(
rpc::daemon::UNSTABLE_LEGACY_TUNNELED,
ProtocolEntry{"tunneled legacy wire protocol", [](TrustedFlag trusted, ref<Store> store) {
return kj::heap<LegacyBootImpl>(trusted, store);
}}
);
}
}
kj::Promise<void> supported(SupportedContext context) override
{
if (!experimentalFeatureSettings.isEnabled(Xp::RpcSockets)) {
kj::throwFatalException(
kj::Exception(
kj::Exception::Type::UNIMPLEMENTED, "main", 0, kj::str("rpc sockets not enabled")
)
);
}
auto result = context.initResults();
auto protocols = result.initProtocols(this->protocols.size());
for (auto [i, proto] : enumerate(this->protocols)) {
protocols[i].setId(proto.first);
protocols[i].setDescription(proto.second.description);
}
return kj::READY_NOW;
}
kj::Promise<void> request(RequestContext context) override
{
auto id = rpc::to<std::string>(context.getParams().getProtocol());
if (used) {
kj::throwFatalException(
kj::Exception(
kj::Exception::Type::FAILED, "main", 0, kj::str("connection already initialized")
)
);
} else if (const auto & protocol = get(protocols, id)) {
used = true;
context.initResults().setResult(protocol->factory(trusted, store));
} else {
kj::throwFatalException(
kj::Exception(
kj::Exception::Type::UNIMPLEMENTED, "main", 0, kj::str("unsupported protocol", id)
)
);
}
return kj::READY_NOW;
}
};
}
kj::Promise<Result<void>>
processConnection(ref<Store> store, kj::AsyncIoStream & connection, TrustedFlag trusted)
try {
// TODO trace encoders can do neat error info things, use them. we could stuff some serialized
// error struct into the remote trace field instead of using result types and get pipelineable
// calls out of it. needs more investigation to say if it's worth the possible reporting skew.
capnp::TwoPartyServer server{kj::heap<BootstrapImpl>(trusted, store)};
// NOTE we can't easily disconnect a peer without shutting shutting down the socket connection
// independently of capnp since capnp does not offer such functionality. shutting down sockets
// in this manner is very disruptive and pretty unreliable, so we will have to find some other
// way to disconnect clients. or we just don't do it because the DoS risk is not large anyway.
//
// since we have control over the promise we can have the following await finish early to stop
// processing events, and if we close the socket after that we've dropped the connection. this
// does not guarantee that responses have been sent though, so we can only do this on requests
// received *after* a fatal error response has been *sent*, inflicting per-operation overhead.
{
auto prevLogger = logger;
co_await server.accept(connection).exclusiveJoin(connection.whenWriteDisconnected());
// NOTE: we do not flush the logger here because the connection is already closed! we only
// delete the non-local logger (if one was set) to ensure all rpc references were dropped.
if (prevLogger != logger) {
std::swap(prevLogger, logger);
delete prevLogger;
}
}
co_return result::success();
} catch (...) {
co_return result::current_exception();
}
}
lix/libstore/daemon.hh
+12 -6
View File
@@ -4,14 +4,20 @@
#include "lix/libutil/async.hh"
#include "lix/libutil/serialise.hh"
#include "lix/libstore/store-api.hh"
#include <kj/async-io.h>
namespace nix::rpc::daemon {
// legacy boot protocol. EXPLICITLY UNSTABLE, this id will change frequently and without notice.
// every change to the experimental tunneling protocol may also change this protocol identifier.
extern const std::string UNSTABLE_LEGACY_TUNNELED;
}
namespace nix::daemon {
void processConnection(
AsyncIoRoot & aio,
ref<Store> store,
FdSource & from,
FdSink & to,
TrustedFlag trusted);
void processLegacyConnection(
AsyncIoRoot & aio, ref<Store> store, FdSource & from, FdSink & to, TrustedFlag trusted
);
kj::Promise<Result<void>>
processConnection(ref<Store> store, kj::AsyncIoStream & connection, TrustedFlag trusted);
}
lix/libstore/derivations.cc
-6
View File
@@ -744,12 +744,6 @@ WireFormatGenerator serializeDerivation(const Store & store, const BasicDerivati
}
}
void writeDerivation(Sink & out, const Store & store, const BasicDerivation & drv)
{
out << serializeDerivation(store, drv);
}
std::string hashPlaceholder(const OutputNameView outputName)
{
// FIXME: memoize?
lix/libstore/derivations.hh
-1
View File
@@ -359,7 +359,6 @@ struct Sink;
Source & readDerivation(Source & in, const Store & store, BasicDerivation & drv, std::string_view name);
WireFormatGenerator serializeDerivation(const Store & store, const BasicDerivation & drv);
void writeDerivation(Sink & out, const Store & store, const BasicDerivation & drv);
/**
* This creates an opaque and almost certainly unique string
lix/libstore/file-transfer-settings/http2.md
+1
View File
@@ -5,3 +5,4 @@ type: bool
default: true
---
Whether to enable HTTP/2 support.
HTTP/2 support cannot be disabled if HTTP/3 support is enabled, the `http2` setting will be ignored in this case.
lix/libstore/file-transfer-settings/http3.md
+1
View File
@@ -5,3 +5,4 @@ type: bool
default: false
---
Whether to enable HTTP/3 support.
Enabling HTTP/3 support forcibly enables HTTP/2 as well; the `http2` will be ignored in this case.
lix/libstore/filetransfer.cc
+10 -1271
View File
File diff suppressed because it is too large Load Diff
lix/libstore/filetransfer.hh
+15 -4
View File
@@ -3,17 +3,17 @@
#include "lix/libutil/async-io.hh"
#include "lix/libutil/box_ptr.hh"
#include "lix/libutil/ref.hh"
#include "lix/libutil/config.hh"
#include "lix/libutil/logging.hh"
#include "lix/libutil/ref.hh"
#include "lix/libutil/result.hh"
#include "lix/libutil/serialise.hh"
#include "lix/libutil/strings.hh"
#include "lix/libutil/types.hh"
#include "lix/libutil/config.hh"
#include <kj/async.h>
#include <curl/curl.h>
#include <string>
#include <future>
namespace nix {
@@ -118,7 +118,18 @@ public:
std::optional<std::string> response;
template<typename... Args>
FileTransferError(FileTransfer::Error error, std::optional<std::string> response, const Args & ... args);
FileTransferError(FileTransfer::Error error, std::optional<std::string> response, const Args & ... args)
: Error(args...), error(error), response(response)
{
const auto hf = HintFmt(args...);
// FIXME: Due to https://github.com/NixOS/nix/issues/3841 we don't know how
// to print different messages for different verbosity levels. For now
// we add some heuristics for detecting when we want to show the response.
if (response && (response->size() < 1024 || response->find("<html>") != std::string::npos))
err.msg = HintFmt("%1%\n\nresponse body:\n\n%2%", Uncolored(hf.str()), chomp(*response));
else
err.msg = hf;
}
};
}
lix/libstore/gc-store.hh
+2 -2
View File
@@ -39,13 +39,13 @@ struct GCOptions
* Any that could not be deleted are returned via the
* `kept` field of GCResults.
*/
typedef enum {
enum GCAction : uint8_t {
gcReturnLive,
gcReturnDead,
gcDeleteDead,
gcDeleteSpecific,
gcTryDeleteSpecific,
} GCAction;
};
GCAction action{gcDeleteDead};
lix/libstore/globals.cc
+47 -5
View File
@@ -67,15 +67,16 @@ Settings::Settings()
, nixManDir(canonPath(NIX_MAN_DIR))
{
if (auto socketDirFromEnv = getEnvNonEmpty("LIX_DAEMON_SOCKET_DIR")) {
nixDaemonSockets_ = {{canonPath(*socketDirFromEnv + LEGACY_SOCKET_COMBINED)}};
nixDaemonSockets_ = daemon::supportedProtocols(*socketDirFromEnv);
socketsPath = *socketDirFromEnv;
} else if (auto socketPathFromEnv = getEnvNonEmpty("NIX_DAEMON_SOCKET_PATH")) {
nixDaemonSockets_ = {{canonPath(*socketPathFromEnv)}};
nixDaemonSockets_ = {{canonPath(*socketPathFromEnv), daemon::Protocol::LEGACY_COMBINED}};
socketsPath = *socketPathFromEnv;
} else {
auto baseDir = nixStateDir + DEFAULT_SOCKET_DIR;
socketsPath = baseDir;
// this should always match the list of sockets created by daemonLoop and the socket units
nixDaemonSockets_ = {
{canonPath(baseDir + LEGACY_SOCKET_COMBINED)},
};
nixDaemonSockets_ = daemon::supportedProtocols(baseDir);
}
buildUsersGroup.setDefault(getuid() == 0 ? "nixbld" : "");
@@ -530,4 +531,45 @@ void initLibStore()
initLibStoreDone = true;
}
namespace daemon {
static inline constexpr auto LEGACY_COMBINED_STR = "legacy-combined";
static inline constexpr auto LEGACY_STR = "legacy";
static inline constexpr auto LIX_XP_1_STR = "lix-xp-1";
std::list<daemon::Protocol> supportedProtocols(std::optional<PathView> prefix)
{
const Path base = prefix ? *prefix + "/" : "";
return {
{base + "lix-xp-1/socket", Protocol::RPC_V1},
{base + "socket", Protocol::LEGACY},
};
}
std::string_view daemon::Protocol::id() const
{
switch (type) {
case LEGACY_COMBINED:
return LEGACY_COMBINED_STR;
case LEGACY:
return LEGACY_STR;
case RPC_V1:
return LIX_XP_1_STR;
}
}
daemon::Protocol getProtocol(std::string_view protocol, std::optional<PathView> prefix)
{
const Path base = prefix ? *prefix + "/" : "";
if (protocol == LEGACY_COMBINED_STR) {
return {prefix ? Path(*prefix) : "", Protocol::LEGACY_COMBINED};
} else if (protocol == LEGACY_STR) {
return {base + "socket", Protocol::LEGACY_COMBINED};
} else if (protocol == LIX_XP_1_STR) {
return {base + "lix-xp-1/socket", Protocol::RPC_V1};
} else {
throw Error("unsupported daemon protocol %s", protocol);
}
}
}
}
lix/libstore/globals.hh
+38 -9
View File
@@ -10,6 +10,37 @@
namespace nix {
namespace daemon {
struct Protocol
{
/// path of socket. may be relative or absolute, depending on context.
Path path;
enum Type : int {
LEGACY_COMBINED,
LEGACY,
RPC_V1,
} type;
/// external identifier of the protocol (eg for `protocol` store parameters)
std::string_view id() const;
};
/**
* returns the list of protocols we can serve, most preferred first. set the `prefix` to
* retrieve socket paths beneath the prefix, otherwise the search suffices are returned.
*/
std::list<Protocol> supportedProtocols(std::optional<PathView> prefix = {});
/**
* get the protocol description with id `protocol` at `prefix`. if `prefix` is `nullopt`
* the returned protocol will contain a relative path. throws an exception if `protocol`
* is not supported or unknown. understands all protocols in `supportedProtocols` and is
* able to resolve `legacy-combined` even though `supportedProtocols` would never return
* it; this is mainly useful for client-side socket resolution using default store uris.
*/
Protocol getProtocol(std::string_view protocol, std::optional<PathView> prefix = {});
}
typedef enum { smEnabled, smRelaxed, smDisabled } SandboxMode;
void to_json(JSON & j, const SandboxMode & e);
@@ -63,14 +94,7 @@ const uint32_t maxIdsPerBuild =
class Settings : public Config
{
public:
struct DaemonSocketPath
{
Path path;
};
private:
std::list<DaemonSocketPath> nixDaemonSockets_;
std::list<daemon::Protocol> nixDaemonSockets_;
unsigned int getDefaultCores();
@@ -105,6 +129,11 @@ public:
*/
Path nixStateDir;
/**
* The directory where sockets are stored.
*/
Path socketsPath;
/**
* The directory where system configuration files are stored.
*/
@@ -128,7 +157,7 @@ public:
/**
* Socket paths a client should connect to, in order of decreasing preference.
*/
const std::list<DaemonSocketPath> & nixDaemonSockets() const
const std::list<daemon::Protocol> & nixDaemonSockets() const
{
return nixDaemonSockets_;
}
lix/libstore/legacy-ssh-store.cc
+139 -79
View File
@@ -1,4 +1,7 @@
#include "lix/libstore/legacy-ssh-store.hh"
#include "libutil/error.hh"
#include "libutil/logging.hh"
#include "libutil/sync.hh"
#include "lix/libutil/archive.hh"
#include "lix/libutil/async-io.hh"
#include "lix/libutil/async.hh"
@@ -20,6 +23,8 @@
#include "path-info.hh"
#include "path.hh"
#include <cstdint>
#include <exception>
#include <kj/async.h>
#include <optional>
namespace nix {
@@ -82,9 +87,6 @@ struct LegacySSHStoreConfig : CommonSSHStoreConfig
const Setting<Path> remoteProgram{this, "nix-store", "remote-program",
"Path to the `nix-store` executable on the remote machine."};
const Setting<int> maxConnections{this, 1, "max-connections",
"Maximum number of concurrent SSH connections."};
const std::string name() override { return "SSH Store"; }
std::string doc() override
@@ -95,25 +97,25 @@ struct LegacySSHStoreConfig : CommonSSHStoreConfig
}
};
struct LegacySSHStoreConfigWithLog : LegacySSHStoreConfig
{
using LegacySSHStoreConfig::LegacySSHStoreConfig;
// Hack for getting remote build log output.
// Intentionally not in `LegacySSHStoreConfig` so that it doesn't appear in
// the documentation
const Setting<int> logFD{this, -1, "log-fd", "file descriptor to which SSH's stderr is connected"};
};
struct LegacySSHStore final : public Store
{
LegacySSHStoreConfigWithLog config_;
LegacySSHStoreConfig config_;
LegacySSHStoreConfigWithLog & config() override { return config_; }
const LegacySSHStoreConfigWithLog & config() const override { return config_; }
LegacySSHStoreConfig & config() override
{
return config_;
}
const LegacySSHStoreConfig & config() const override
{
return config_;
}
struct Connection
{
// make sure this is destroyed last so the sshConn that feeds it dies first
kj::Promise<void> logHandlerPromise{nullptr};
std::list<Activity> act;
ref<IoBuffer> fromBuf{make_ref<IoBuffer>()};
std::unique_ptr<SSH::Connection> sshConn;
ServeProto::Version remoteVersion;
@@ -136,6 +138,54 @@ struct LegacySSHStore final : public Store
};
}
kj::Promise<void> logHandler(std::string storeUri)
try {
// NOTE this is very similar to handleBuilderOutput in DerivationGoal, but unlike
// the derivation goal we do not need to handle EIO from a pty here. we also have
// no timeouts or limits to keep track of, which makes deduplication less useful.
std::map<ActivityId, Activity> activities;
auto stderrPipe = std::move(sshConn->stderrPipe);
auto reader = AIO().lowLevelProvider.wrapInputFd(stderrPipe.get());
LogLineSplitter splitter;
auto flushLine = [&](const std::string & line) {
if (const auto state = handleJSONLogMessage(line, act.back(), activities, storeUri)) {
return *state;
} else {
return act.back().result(resBuildLogLine, line);
}
};
auto buf = kj::heapArray<char>(4096);
while (true) {
const auto got = co_await reader->tryRead(buf.begin(), 1, buf.size());
if (got == 0) {
break;
}
std::string_view data{buf.begin(), got};
while (!data.empty()) {
if (auto line = splitter.feed(data)) {
if (flushLine(*line) == Logger::BufferState::NeedsFlush) {
TRY_AWAIT(act.back().getLogger().flush());
}
}
}
}
if (auto line = splitter.finish(); !line.empty()) {
(void) flushLine(line);
TRY_AWAIT(act.back().getLogger().flush());
}
} catch (std::exception & e) { // NOLINT(lix-foreign-exceptions)
logException("remote store error", e);
} catch (...) {
std::terminate();
}
template<typename Arg>
kj::Promise<Result<void>>
sendArg(AsyncOutputStream & stream, StringSink & buffer, Arg && arg)
@@ -204,72 +254,70 @@ struct LegacySSHStore final : public Store
std::string host;
ref<Pool<Connection>> connections;
Sync<Connection, AsyncMutex> connection;
SSH ssh;
static std::set<std::string> uriSchemes() { return {"ssh"}; }
LegacySSHStore(
const std::string & scheme, const std::string & host, LegacySSHStoreConfigWithLog config
)
LegacySSHStore(const std::string & scheme, const std::string & host, LegacySSHStoreConfig config)
: Store(config)
, config_(std::move(config))
, host(host)
, connections(make_ref<Pool<Connection>>(
std::max(1, (int) config_.maxConnections),
[this]() { return openConnection(); },
[](const ref<Connection> & r) { return r->good; }
))
, ssh(
host,
config_.port,
config_.sshKey,
config_.sshPublicHostKey,
config_.compress,
config_.logFD)
, ssh(host, config_.port, config_.sshKey, config_.sshPublicHostKey, config_.compress)
{
}
kj::Promise<Result<ref<Connection>>> openConnection()
kj::Promise<Result<decltype(connection)::Lock>> getConnection()
try {
auto conn = make_ref<Connection>();
conn->sshConn = ssh.startCommand(
fmt("%s --serve --write", config_.remoteProgram)
+ (config_.remoteStore.get() == ""
? ""
: " --store " + shellEscape(config_.remoteStore.get()))
);
FdSink to(conn->sshConn->socket.get());
FdSource from(conn->sshConn->socket.get(), conn->fromBuf);
conn->store = this;
try {
to << SERVE_MAGIC_1 << SERVE_PROTOCOL_VERSION;
to.flush();
uint64_t magic = readNum<uint64_t>(from);
if (magic != SERVE_MAGIC_2)
throw Error("'nix-store --serve' protocol mismatch from '%s'", host);
conn->remoteVersion = readNum<unsigned>(from);
if (GET_PROTOCOL_MAJOR(conn->remoteVersion) != 0x200)
throw Error("unsupported 'nix-store --serve' protocol version on '%s'", host);
/* No longer support protocols this old*/
if (GET_PROTOCOL_MINOR(conn->remoteVersion) < 4) {
throw Error(
"remote '%s' is too old (protocol version %x)", host, conn->remoteVersion
);
}
} catch (EndOfFile & e) {
throw Error("cannot connect to '%1%'", host);
auto conn = co_await connection.lock();
if (conn->good && conn->sshConn) {
co_return conn;
}
return {conn};
*conn = {};
conn->sshConn = ssh.startCommand(
fmt("%s --serve --write", config_.remoteProgram)
+ (config_.remoteStore.get() == "" ? "" : " --store " + shellEscape(config_.remoteStore.get()))
);
try {
FdSink to(conn->sshConn->socket.get());
FdSource from(conn->sshConn->socket.get(), conn->fromBuf);
conn->store = this;
try {
to << SERVE_MAGIC_1 << SERVE_PROTOCOL_VERSION;
to.flush();
uint64_t magic = readNum<uint64_t>(from);
if (magic != SERVE_MAGIC_2) {
throw Error("'nix-store --serve' protocol mismatch from '%s'", host);
}
conn->remoteVersion = readNum<unsigned>(from);
if (GET_PROTOCOL_MAJOR(conn->remoteVersion) != 0x200) {
throw Error("unsupported 'nix-store --serve' protocol version on '%s'", host);
}
/* No longer support protocols this old*/
if (GET_PROTOCOL_MINOR(conn->remoteVersion) < 4) {
throw Error("remote '%s' is too old (protocol version %x)", host, conn->remoteVersion);
}
} catch (EndOfFile & e) {
throw Error("cannot connect to '%1%'", host);
}
conn->act.emplace_back(logger->startActivity(lvlDebug, actUnknown, "remote store " + getUri()));
conn->logHandlerPromise = conn->logHandler(host);
co_return conn;
} catch (Error & e) {
std::string msg = chomp(drainFD(conn->sshConn->stderrPipe.get(), false));
throw Error("cannot connect to %s: %s (%s)", getUri(), e.msg(), msg);
}
} catch (...) {
return {result::current_exception()};
};
co_return result::current_exception();
}
std::string getUri() override
{
@@ -279,7 +327,7 @@ struct LegacySSHStore final : public Store
kj::Promise<Result<std::shared_ptr<const ValidPathInfo>>>
queryPathInfoUncached(const StorePath & path, const Activity * context) override
try {
auto conn(TRY_AWAIT(connections->get()));
auto conn(TRY_AWAIT(getConnection()));
debug("querying remote host '%s' for info on '%s'", host, printStorePath(path));
@@ -307,7 +355,7 @@ struct LegacySSHStore final : public Store
try {
debug("adding path '%s' to remote host '%s'", printStorePath(info.path), host);
auto conn(TRY_AWAIT(connections->get()));
auto conn(TRY_AWAIT(getConnection()));
unsigned result;
if (GET_PROTOCOL_MINOR(conn->remoteVersion) >= 5) {
@@ -351,16 +399,16 @@ struct LegacySSHStore final : public Store
kj::Promise<Result<box_ptr<AsyncInputStream>>>
narFromPath(const StorePath & path, const Activity * context) override
try {
auto conn(TRY_AWAIT(connections->get()));
auto conn(TRY_AWAIT(getConnection()));
struct NarStream : AsyncInputStream
{
Pool<Connection>::Handle conn;
Sync<Connection, AsyncMutex>::Lock conn;
AsyncFdIoStream stream{AsyncFdIoStream::shared_fd{}, conn->sshConn->socket.get()};
AsyncBufferedInputStream buffered{stream, conn->fromBuf};
box_ptr<AsyncInputStream> copier{copyNAR(buffered)};
NarStream(Pool<Connection>::Handle conn) : conn(std::move(conn)) {}
NarStream(Sync<Connection, AsyncMutex>::Lock conn) : conn(std::move(conn)) {}
kj::Promise<Result<std::optional<size_t>>> read(void * buffer, size_t size) override
{
@@ -424,7 +472,19 @@ public:
const StorePath & drvPath, const BasicDerivation & drv, BuildMode buildMode
) override
try {
auto conn(TRY_AWAIT(connections->get()));
auto conn(TRY_AWAIT(getConnection()));
// this is a duplicate of DerivationGoal::buildDescription because ugh.
// getting that information into here where needed is nigh *impossible*
auto description = fmt(buildMode == bmRepair ? "repairing outputs of '%s'"
: buildMode == bmCheck ? "checking outputs of '%s'"
: "building '%s'",
printStorePath(drvPath))
+ "on " + getUri();
conn->act.emplace_back(logger->startActivity(
lvlInfo, actBuild, description, Logger::Fields{printStorePath(drvPath), getUri(), 1, 1}
));
KJ_DEFER(conn->act.pop_back());
co_return TRY_AWAIT(conn->sendCommand<BuildResult>(
ServeProto::Command::BuildDerivation,
@@ -445,7 +505,7 @@ public:
if (evalStore && evalStore.get() != this)
throw Error("building on an SSH store is incompatible with '--eval-store'");
auto conn(TRY_AWAIT(connections->get()));
auto conn(TRY_AWAIT(getConnection()));
Strings ss;
for (auto & p : drvPaths) {
@@ -504,7 +564,7 @@ public:
co_return result::success();
}
auto conn(TRY_AWAIT(connections->get()));
auto conn(TRY_AWAIT(getConnection()));
out.merge(TRY_AWAIT(conn->sendCommand<StorePathSet>(
ServeProto::Command::QueryClosure, includeOutputs, ServeProto::write(*conn, paths)
@@ -518,7 +578,7 @@ public:
kj::Promise<Result<StorePathSet>> queryValidPaths(const StorePathSet & paths,
SubstituteFlag maybeSubstitute = NoSubstitute) override
try {
auto conn(TRY_AWAIT(connections->get()));
auto conn(TRY_AWAIT(getConnection()));
co_return TRY_AWAIT(conn->sendCommand<StorePathSet>(
ServeProto::Command::QueryValidPaths,
@@ -530,9 +590,9 @@ public:
co_return result::current_exception();
}
kj::Promise<Result<void>> connect() override
kj::Promise<Result<void>> init() override
try {
auto conn(TRY_AWAIT(connections->get()));
auto conn(TRY_AWAIT(getConnection()));
co_return result::success();
} catch (...) {
co_return result::current_exception();
@@ -540,7 +600,7 @@ public:
kj::Promise<Result<unsigned int>> getProtocol() override
try {
auto conn(TRY_AWAIT(connections->get()));
auto conn(TRY_AWAIT(getConnection()));
co_return conn->remoteVersion;
} catch (...) {
co_return result::current_exception();
lix/libstore/machines.cc
+7 -12
View File
@@ -34,19 +34,11 @@ bool Machine::mandatoryMet(const std::set<std::string> & features) const
});
}
kj::Promise<Result<std::pair<ref<Store>, Pipe>>> Machine::openStore() const
kj::Promise<Result<ref<Store>>> Machine::openStore() const
try {
Pipe pipe;
StoreConfig::Params storeParams;
if (storeUri.starts_with("ssh://")) {
// Remote builds become flakey, when having more than one ssh connection
storeParams["max-connections"] = "1";
}
if (storeUri.starts_with("ssh://") || storeUri.starts_with("ssh-ng://")) {
pipe.create();
storeParams["log-fd"] = std::to_string(pipe.writeSide.get());
if (sshKey != "")
storeParams["ssh-key"] = sshKey;
if (sshPublicHostKey != "")
@@ -65,7 +57,7 @@ try {
append(mandatoryFeatures);
}
co_return {TRY_AWAIT(nix::openStore(storeUri, storeParams)), std::move(pipe)};
co_return TRY_AWAIT(nix::openStore(storeUri, storeParams));
} catch (...) {
co_return result::current_exception();
}
@@ -167,6 +159,7 @@ static Machine parseBuilderLine(const std::string & line)
}
return {
storeUri,
storeUri,
systemTypes,
sshKey,
@@ -262,7 +255,8 @@ static toml::result<float, std::string> getSpeedFactor(const toml::value & data)
return toml::success(1.0f);
}
static toml::result<Machine, std::vector<std::string>> parseMachine(const toml::value & data)
static toml::result<Machine, std::vector<std::string>>
parseMachine(const std::string name, const toml::value & data)
{
std::vector<std::string> errs;
@@ -339,6 +333,7 @@ static toml::result<Machine, std::vector<std::string>> parseMachine(const toml::
return toml::failure(errs);
}
return toml::success<Machine>({
name,
storeUri.unwrap(),
std::set(systemTypes.unwrap().begin(), systemTypes.unwrap().end()),
sshKey.unwrap(),
@@ -404,7 +399,7 @@ static toml::result<Machines, std::vector<std::string>> parseToml(const toml::va
}
for (const auto & [name, machine] : data.at(array_name).as_table()) {
auto const res = parseMachine(machine);
auto const res = parseMachine(name, machine);
if (res.is_err()) {
auto err = res.as_err();
parserErrors.push_back(fmt("for machine %s:", name));
lix/libstore/machines.hh
+2 -1
View File
@@ -14,6 +14,7 @@ class Store;
struct Machine {
const std::string name;
const std::string storeUri;
const std::set<std::string> systemTypes;
const std::string sshKey;
@@ -41,7 +42,7 @@ struct Machine {
*/
bool mandatoryMet(const std::set<std::string> & features) const;
kj::Promise<Result<std::pair<ref<Store>, Pipe>>> openStore() const;
kj::Promise<Result<ref<Store>>> openStore() const;
};
typedef std::vector<Machine> Machines;
lix/libstore/meson.build
+31 -20
View File
@@ -11,28 +11,33 @@ endforeach
libstore_rpc = []
libstore_rpc += custom_target(
command : [
capnpc_wrapper,
'--language=c++',
'--src-prefix=@CURRENT_SOURCE_DIR@',
'--outdir=@OUTDIR@',
'--depfile=@DEPFILE@',
'-I@SOURCE_ROOT@',
'@INPUT@',
],
input : files(
# keep-sorted start
'types.capnp',
# keep-sorted end
),
output : [
'@PLAINNAME@.h',
'@PLAINNAME@.c++',
],
depfile : '@PLAINNAME@.d',
libstore_rpc_files = files(
# keep-sorted start
'daemon.capnp',
'types.capnp',
# keep-sorted end
)
foreach rpc_file : libstore_rpc_files
libstore_rpc += custom_target(
command : [
capnpc_wrapper,
'--language=c++',
'--src-prefix=@CURRENT_SOURCE_DIR@',
'--outdir=@OUTDIR@',
'--depfile=@DEPFILE@',
'-I@SOURCE_ROOT@',
'@INPUT@',
],
input : rpc_file,
output : [
'@PLAINNAME@.h',
'@PLAINNAME@.c++',
],
depfile : '@PLAINNAME@.d',
)
endforeach
# rpc definitions only here to get relative paths of generated files right
subdir('build')
@@ -189,6 +194,8 @@ liblix_sources += files(
'common-protocol.cc',
'content-address.cc',
'crypto.cc',
'curlfiletransfer.cc',
'curlmulti.cc',
'daemon.cc',
'derivations.cc',
'derived-path.cc',
@@ -232,6 +239,7 @@ liblix_sources += files(
'ssh.cc',
'store-api.cc',
'temporary-dir.cc',
'transferitem.cc',
'uds-remote-store.cc',
'worker-protocol.cc',
# keep-sorted end
@@ -255,6 +263,8 @@ libstore_headers = files(
'common-protocol.hh',
'content-address.hh',
'crypto.hh',
'curlfiletransfer.hh',
'curlmulti.hh',
'daemon.hh',
'derivations.hh',
'derived-path.hh',
@@ -302,6 +312,7 @@ libstore_headers = files(
'store-api.hh',
'store-cast.hh',
'temporary-dir.hh',
'transferitem.hh',
'types-rpc.hh',
'uds-remote-store.hh',
'worker-protocol-impl.hh',

Some files were not shown because too many files have changed in this diff Show More