Test coverage for native code WIP

Part of: https://git.lix.systems/lix-project/lix/issues/1186

doc/manual/change-authors.yml

@@ -57,6 +57,10 @@ blitz:
   display_name: Julian Stecklina
   github: blitz
 
+blokyk:
+  display_name: blokyk
+  github: blokyk
+
 cole-h:
   display_name: Cole Helbling
   github: cole-h
@@ -252,6 +256,10 @@ rootile:
 seppel3210:
   github: Seppel3210
 
+sterni:
+  forgejo: sterni
+  github: sternenseemann
+
 stevalkr:
   github: stevalkr
 

doc/manual/rl-next/arg-behavior.md

@@ -0,0 +1,10 @@
+---
+synopsis: "allow setting nested attributes via `--arg`/`--argstr`"
+cls: [5338]
+category: "Features"
+credits: [ma27]
+issues: [fj#496]
+---
+
+Passing `--arg config.allowUnfree true` to e.g. `nix-build` now results in `config` with value
+`{ allowUnfree = true; }` passed to the expression.

doc/manual/rl-next/empty-attr-paths.md

@@ -0,0 +1,10 @@
+---
+synopsis: "libexpr: allow empty attr-names in parseAttrPath if they are quoted"
+cls: [5375]
+category: "Miscellany"
+credits: [ma27]
+---
+
+Empty strings are now allowed in attribute paths as consumed by e.g. `nix-build`.
+I.e. `nix-build -A 'foo."".bar'` works now.
+The quotes are necessary, i.e. `nix-build -A foo..bar` will throw an error.

doc/manual/rl-next/libexpr-fix-break.md

@@ -0,0 +1,14 @@
+---
+synopsis: "builtins.break doesn't break expression anymore"
+issues: [1165]
+cls: [5422]
+category: "Fixes"
+credits: [blokyk]
+---
+
+Wrapping an expression in `builtins.break` used to break some builtins like
+`map` and the `is*` functions, which could modify the execution path of code
+inadvertently, made debugging nix harder than it already is, and in some cases
+even crashed the interpreter. Now, using `break` should be completely
+transparent to whatever function receives it as an input, preventing the
+above-mentioned issues.

doc/manual/rl-next/libexpr-fix-stdout.md

@@ -0,0 +1,9 @@
+---
+synopsis: "flake config warnings are now printed to stderr"
+issues: [1155]
+cls: [5379]
+category: "Fixes"
+credits: [lheckemann]
+---
+
+The settings listed in a flake-config confirmation prompt are now printed to stderr rather than stdout, which allows `nix print-dev-env` to emit valid bash again even in the presence of untrusted settings.

doc/manual/rl-next/minimum-lowdown-1.4.md

@@ -0,0 +1,10 @@
+---
+synopsis: "Lix now requires lowdown 1.4.0 or later"
+issues: []
+cls: [5374]
+category: Packaging
+credits: [sterni]
+---
+
+Support for linking against `lowdown < 1.4.0` has been removed from Lix since
+all supported Nixpkgs channels distribute lowdown 2.0.4 or later.

doc/manual/rl-next/nix-path-shadow-error.md

@@ -0,0 +1,11 @@
+---
+synopsis: "Shadowing internal files through the Nix search path is now an error"
+issues: [998]
+cls: [4632, 5370]
+category: "Breaking Changes"
+credits: [thubrecht, jade, horrors]
+---
+
+As Lix uses the path `<nix/fetchurl.nix>` for bootstrapping purposes, the ability to shadow it by adding `nix=/some/path` (or `/other/path` that contains a `nix` directory) to the search path is not desirable.
+
+Lix 2.95 deprecated this behavior with a warning, Lix 2.96 now turns it into a hard error if the `nix-path-shadow` deprecated feature isn't enabled. This deprecated feature is slated to be removed in Lix 2.98.

doc/manual/rl-next/repl-relative-st.md

@@ -0,0 +1,18 @@
+---
+synopsis: "Allow moving between stack frames relative to current debugger frame"
+issues: [1156]
+cls: [5411]
+category: "Improvements"
+credits: [blokyk]
+---
+
+Debugging functional programs often involve switching between a bunch of stack
+frames to get the full context of what's happening and who's calling who.
+Before this change, going up or down the stack in the nix debugger with `:st`
+meant remembering the absolute index of each stack frame, instead of their
+positions relative to one another; this got tiring *fast*.
+
+Now, you can prepend `:st`'s argument with a + or - sign to indicate you want to
+move relative to the current stack frame. For example, typing `:st +3` when you
+were on frame `10` will go frame `13`; vice-versa, typing `:st -4` on frame `6`
+will go to frame `2`.

doc/manual/rl-next/repl-st-errors.md

@@ -0,0 +1,14 @@
+---
+synopsis: "invalid arguments to :st now print an error"
+cls: [5386]
+category: "Improvements"
+credits: [blokyk]
+---
+
+When using the debugger, the `:st` command used to traverse the call stack would
+silently fail and put the debugger in an invalid state if the argument given to
+it wasn't a valid stack frame index.
+
+This change adds an error message warning the user if the given index wasn't a
+valid frame (telling them the range of valid indices), as well as if it wasn't
+even a valid integer to begin with.

doc/manual/src/command-ref/opt-common.md

@@ -177,6 +177,12 @@ Most commands in Lix accept the following command-line options:
   You can override this using `--arg`, e.g., `nix-env --install --attr pkgname --arg system \"i686-freebsd\"`.
   (Note that since the argument is a Nix string literal, you have to escape the quotes.)
 
+  Additionally, dots are interpreted as attribute-path separators.
+  I.e. `nix-instantiate '<nixpkgs>' -A hello-unfree --arg config.allowUnfree true` will result in an argument `config` with value `{ allowUnfree = true; }` being passed to `<nixpkgs>`.
+
+  Please note that merging of different arguments is rejected.
+  I.e. `--arg config '{ cudaSupport = true; }' --arg config.allowUnfree true` will not work whereas `--arg config.cudaSupport true --arg config.allowUnfree true` is accepted.
+
 - <span id="opt-argstr">[`--argstr`](#opt-argstr)</span> *name* *value*
 
   This option is like `--arg`, only the value is not a Nix expression but a string.

doc/manual/src/contributing/hacking.md

@@ -51,28 +51,64 @@ $ nix-shell -A native-clangStdenvPackages
 
 ### Building from the development shell
 
-Run a clean build and test with `just clean setup build install test`.
+We have a [justfile](just.systems) for extra convenient building.
+It defaults to using `./build` as the build directory, and `$out` (`./outputs/out`) as the install directory.
+For most cases, you can clean-build, install, and run the tests with:
+
+```bash
+$ just setup --wipe && just test
+```
+
+> **Note**
+>
+> The `--wipe` argument to `meson setup` conveniently works whether you have an existing build directory or not.
+>
+> However, it is *mostly*, but not *exactly* equivalent to deleting the build directory first.
+> In particular, previously specified `-D` build options are **preserved** with `--wipe` (for some reason).
+> For example, if you fetch and checkout a new version of Lix, and that new version *removes* a Meson build option from `./meson.options`, *and* a previous invocation in that build directory explicitly set that option, then `meson setup --wipe build` will error, complaining about the unknown option.
+> For these cases, `just clean` will give you a well-and-truly-this-time-for-real clean build.
+
+Because the integration tests require installation to work, `just test` automatically also calls `just install`, and Meson helpfully will automatically build any targets that need building when trying to install them.
+
+You can override the build directory or install directory by setting the justfile [variables](https://just.systems/man/en/setting-variables-from-the-command-line.html) `outdir` and `builddir` on the command-line:
+
+```bash
+$ just builddir=build-before-bisect outdir=out-before-bisect setup
+$ just builddir=build-before-bisect test
+```
+
+You'll have to set `builddir` for every target, but `outdir` only needs to be set for `setup`.
 
 You can also run the unit tests and integration tests separately:
 
 ```bash
-$ just setup build test-unit
-$ just install test-integration
+$ just setup
+$ just test-unit
+$ just test-integration
 ```
 
-Many justfile aliases have a `-custom` variant which pass extra arguments to `meson`.
+Most justfile targets forward all further arguments to the underlying Meson invocation.
 For example, to work on both Lix and nix-eval-jobs you can run:
 
-```
-$ just setup-custom -Dnix-eval-jobs=enabled
-$ # or
-$ mesonFlags=-Dnix-eval-jobs=enabled just setup
+```bash
+$ just setup -Dnix-eval-jobs=enabled
 ```
 
-Note that only targets which don't accept extra arguments can be used when
-running multiple targets at once; `just setup build` is fine, but `just
-setup-custom build` is an error. The `test` target is usually the last one to
-run, so it always accepts extra arguments.
+Note that only targets which *don't* accept extra arguments can have other targets following them.
+`just clean setup` is equivalent to `just clean && just setup`, but `just build test` runs the `build` target with the argument `test`.
+This means that if you want to, for example, build with lower parallelism, and then test, you will have to do something like this:
+
+```bash
+$ just build -j4
+$ just test
+```
+
+Finally, the rewrite of the integration test suite, functional2, also has its own justfile target which allows passing extra arguments to pytest.
+For example, to collect and list all functional2 tests without running them, you can pass pytest's `--collect-only` argument:
+
+```bash
+$ just test-functional2 --collect-only
+```
 
 You can also build Lix manually:
 
@@ -408,6 +444,18 @@ You can build it yourself:
 # xdg-open ./result/coverage/index.html
 ```
 
+Or, in a dev shell, set `-Dcoverage=true` when running `meson setup`.
+Coverage data goes into `build/profraw` when you run executables in the dev shell.
+Then, run `ninja -C build coverage-report` to produce an HTML report of coverage in `build/coverage/index.html` alongside a LLVM `.lcov` file.
+
+> [!NOTE]
+> We use the [llvm source-based coverage], which has better precision than using clang with gcov, which is debuginfo based (but likely worse performance, which is fine).
+>
+> It should be noted that Meson [allegedly has coverage support][meson-coverage], but it only supports gcov-style coverage, so we don't use it.
+
+[llvm source-based coverage]: https://clang.llvm.org/docs/SourceBasedCodeCoverage.html
+[meson-coverage]: https://mesonbuild.com/Unit-tests.html#coverage
+
 Metrics about the change in line/function coverage over time will be available in the future (FIXME(lix-hydra)).
 
 ## Add a release note {#release-notes}

doc/manual/src/language/derivations.md

@@ -17,6 +17,12 @@ the attributes of which specify the inputs of the build.
     string. This is used as a symbolic name for the package by
     `nix-env`, and it is appended to the output paths of the derivation.
 
+    > **Note**
+    >
+    > Names can only contain alphanumerical characters (0-9, a-z, A-Z)
+    > as well as `+`, `-`, `.`, `_`, `?` and `=`. Names must be neither
+    > `.` nor `..`, and must not start with `.-` or `..-`.
+
   - There must be an attribute named [`builder`]{#attr-builder} that identifies the
     program that is executed to perform the build. It can be either a
     derivation or a source (a local file reference, e.g.,

doc/manual/src/quick-start.md

@@ -5,7 +5,7 @@
 FIXME(Lix): This chapter is quite outdated with respect to recommended practices in 2024 and needs updating.
 The commands in here will work, however, and the installation section is up to date.
 
-For more updated guidance, see the links on <https://lix.systems/resources/>
+For more updated guidance, see the links on <https://wiki.lix.systems/books/lix-users/page/nix-resources>
 
 </div>
 

flake.nix

@@ -185,9 +185,6 @@
                 });
           };
 
-          # Forward from the previous stage as we don’t want it to pick the lowdown override
-          nixUnstable = prev.nixUnstable;
-
           check-headers = final.buildPackages.callPackage ./maintainers/check-headers.nix { };
           check-syscalls = final.buildPackages.callPackage ./maintainers/check-syscalls.nix { };
 
@@ -219,6 +216,9 @@
             inherit versionSuffix officialRelease;
             stdenv = currentStdenv;
             busybox-sandbox-shell = final.busybox-sandbox-shell or final.default-busybox-sandbox-shell;
+            # See below
+            lowdown = final.lowdown_3_0;
+            lowdown-unsandboxed = final.lowdown_3_0.override { enableDarwinSandbox = false; };
           };
 
           lix-clang-tidy = final.callPackage ./subprojects/lix-clang-tidy { };
@@ -245,9 +245,23 @@
           # And same thing for our build-release-notes package.
           build-release-notes = final.nix.passthru.build-release-notes;
 
-          lowdown =
-            assert lib.versionAtLeast prev.lowdown.version "2.0.0";
-            prev.lowdown;
+          # As soon as Nixpkgs updates to >= 3.0.0, change to lowdown_2_0!
+          # We don't change the default version in order to not change the hash
+          # of Nix/Lix from upstream Nixpkgs.
+          lowdown_3_0 =
+            if (lib.versions.major prev.lowdown.version == "3") then
+              prev.lowdown
+            else
+              prev.lowdown.overrideAttrs (
+                finalAttrs: _prevAttrs: {
+                  version = "3.0.0";
+
+                  src = final.fetchurl {
+                    url = "https://kristaps.bsd.lv/lowdown/snapshots/lowdown-${finalAttrs.version}.tar.gz";
+                    sha512 = "94e97234d598382c3c3dc27f9bfdb3a3a2fcf7dbb6a8df3c85ee09f27f792449034a41d49d9cfd3d8450d2de01b8562c20c3d120e65c81af4d7d6c9454119e93";
+                  };
+                }
+              );
 
           capnproto = prev.capnproto.overrideAttrs (old: {
             patches =
@@ -272,25 +286,58 @@
       overlays.default = overlayFor (p: p.clangStdenv);
 
       hydraJobs = {
+        # Aggregate job that is finished in Hydra _after_ all constituent jobs (here: grouped by system)
+        # succeed.
+        # This is used to run CD scripts once all builds are finished on Hydra.
+        release = forAllSystems (
+          system:
+          let
+            pkgs = nixpkgsFor.${system}.native;
+          in
+          pkgs.runCommand "release"
+            {
+              _hydraAggregate = true;
+              constituents = lib.filter (x: x != null) (
+                lib.mapAttrsToListRecursiveCond
+                  (_: val: !(lib.isDerivation val || builtins.any (system': val ? ${system'}) systems))
+                  (
+                    path: drv:
+                    if drv ? ${system} then
+                      lib.concatStringsSep "." (path ++ [ system ])
+                    else if drv.system or null == system then
+                      lib.concatStringsSep "." path
+                    else
+                      null
+                  )
+                  (
+                    removeAttrs self.hydraJobs [
+                      "devShell"
+                      "release"
+                      "rl-next"
+                    ]
+                  )
+              );
+            }
+            ''
+              touch $out
+            ''
+        );
+
         # Binary package for various platforms.
         build = forAllSystems (system: self.packages.${system}.nix);
 
-        # Building Lix twice in CI is expensive, but we can catch a lot of static
-        # build regressions by at least making sure it evals and configures.
-        configure-static = lib.genAttrs linux64BitSystems (
+        # Ensure support for lowdown < 3.0 doesn't regress for NixOS 25.11
+        build-lowdown_2_0 = lib.genAttrs [ "aarch64-linux" ] (
           system:
-          self.packages.${system}.nix-static.overrideAttrs {
-            dontBuild = true;
-            installPhase = ''
-              runHook preInstall
-
-              echo "configure-static complete. exiting with success"
-              mkdir -p "$out"
-              exit 0
-            '';
+          assert lib.versionOlder nixpkgsFor.${system}.native.lowdown.version "3.0.0";
+          self.packages.${system}.nix.override {
+            lowdown = nixpkgsFor.${system}.native.lowdown;
+            lowdown-unsandboxed = nixpkgsFor.${system}.native.lowdown-unsandboxed;
           }
         );
 
+        buildStatic = lib.genAttrs linux64BitSystems (system: self.packages.${system}.nix-static);
+
         devShell = forAllSystems (system: {
           default = self.devShells.${system}.default;
           clang = self.devShells.${system}.native-clangStdenvPackages;

justfile

@@ -1,4 +1,10 @@
 # https://just.systems/man/en/
+#
+# Take a look at ./doc/manual/src/contributing/hacking.md for a detailed
+# explanation on how to use this file!
+
+outdir := x"${out:-$PWD/outputs/out}"
+builddir := "build"
 
 # List all available targets
 list:
@@ -6,40 +12,32 @@ list:
 
 # Clean build artifacts
 clean:
-    rm -rf build
+    rm -rf {{ builddir }}
 
-# Prepare meson for building with extra options
-setup-custom *OPTIONS:
-    meson setup build --prefix="$PWD/outputs/out" $mesonFlags {{ OPTIONS }}
-
-# Prepare meson for building
-setup: (setup-custom)
+# Prepare meson for building.
+setup *OPTIONS:
+    meson setup {{ builddir }} --reconfigure --prefix="{{outdir}}" $mesonFlags {{ OPTIONS }}
 
 # Build lix with extra options
-build-custom *OPTIONS:
-    meson compile -C build {{ OPTIONS }}
-
-# Build lix
-build: (build-custom)
+build *OPTIONS:
+    meson compile -C {{ builddir }} {{ OPTIONS }}
 
 alias compile := build
 
-# Install lix for local development with extra options
-install-custom *OPTIONS: (build-custom OPTIONS)
-    meson install -C build
+# `meson install` will automatically build anything that needs to be built to install it.
+[doc("Install Lix for local development")]
+install *OPTIONS:
+    meson install --quiet -C {{ builddir }} {{ OPTIONS }}
 
-# Install lix for local development
-install: (install-custom)
-
-# Run tests (usually requires `install`) with extra options
-test *OPTIONS:
-    meson test -C build --print-errorlogs --max-lines 10000 {{ OPTIONS }}
+# Run all tests tests (installs first).
+test *OPTIONS: (install)
+    meson test -C {{ builddir }} --print-errorlogs --max-lines 10000 {{ OPTIONS }}
 
 # Run unit tests only
 test-unit *OPTIONS: (test "--suite" "check")
 
 # Run integration tests only
-test-integration *OPTIONS: install (test "--suite" "installcheck")
+test-integration *OPTIONS: (test "--suite" "installcheck" OPTIONS)
 
 # Run functional2 tests using pytest directly, allowing for additional arguments to be passed to pytest e.g. for more granular test selection
 test-functional2 *OPTIONS:

lix/code-generation/build_builtins.py

@@ -52,7 +52,7 @@ class Builtin:
             </dt>
             <dd>
 
-            {indent(self.documentation, "    " * 3)}
+{indent(self.documentation, "    " * 3)}
 
             {
             f"This function is only available if the [{self.experimental_feature}](@docroot@/contributing/experimental-features.md#xp-feature-{self.experimental_feature}) experimental feature is enabled."

lix/code-generation/build_settings.py

@@ -90,7 +90,7 @@ class Setting:
         aliases = [f"`{item}`" for item in self.aliases]
         description = dedent(f"""
 
-            {indent(indentation, self.documentation)}
+{indent(indentation, self.documentation)}
 
             {indent(indentation, PLATFORM_WARNING.format(platforms=str(platforms)[1:-1])) if self.platforms else ""}
             {indent(indentation, XP_WARNING.format(feature=self.experimental_feature, name=self.name)) if self.experimental_feature is not None else ""}

lix/legacy/nix-build.cc

@@ -213,7 +213,7 @@ static int main_nix_build(AsyncIoRoot & aio, std::string programName, Strings ar
     auto autoArgsWithInNixShell = autoArgs;
     if (runEnv) {
         auto newArgs = evaluator->buildBindings(autoArgsWithInNixShell->size() + 1);
-        newArgs.alloc("inNixShell") = {NewValueAs::boolean, true};
+        newArgs.insert("inNixShell", {NewValueAs::boolean, true});
         for (auto & i : *autoArgs) newArgs.insert(i);
         autoArgsWithInNixShell = newArgs.finish();
     }

lix/legacy/nix-env.cc

@@ -154,8 +154,7 @@ static void getAllExprs(Evaluator & state,
             Value vArg = {NewValueAs::string, path2.canonical().abs()};
             if (seen.size() == maxAttrs)
                 throw Error("too many Nix expressions in directory '%1%'", path);
-            attrs.alloc(attrName
-            ) = {NewValueAs::app, state.mem, state.builtins.get("import"), vArg};
+            attrs.insert(attrName, {NewValueAs::app, state.mem, state.builtins.get("import"), vArg});
         }
         else if (st.type == InputAccessor::tDirectory)
             /* `path2' is a directory (with no default.nix in it);
@@ -180,7 +179,7 @@ static Value loadSourceExpr(EvalState & state, const SourcePath & path_)
        directory). */
     else if (st.type == InputAccessor::tDirectory) {
         auto attrs = state.ctx.buildBindings(maxAttrs);
-        attrs.alloc("_combineChannels") = Value::EMPTY_LIST;
+        attrs.insert("_combineChannels", Value::EMPTY_LIST);
         StringSet seen;
         getAllExprs(state.ctx, path, seen, attrs);
         return {NewValueAs::attrs, attrs};

lix/legacy/nix-instantiate.cc

@@ -42,11 +42,7 @@ void processExpr(EvalState & state, const Strings & attrPaths,
 
         NixStringContext context;
         if (evalOnly) {
-            Value vRes;
-            if (autoArgs.empty())
-                vRes = v;
-            else
-                vRes = state.autoCallFunction(autoArgs, v, noPos);
+            Value vRes = autoArgs.empty() ? v : state.autoCallFunction(autoArgs, v, noPos);
             if (output == okRaw)
                 std::cout << *state.coerceToString(noPos, vRes, context, "while generating the nix-instantiate output", StringCoercionMode::Strict);
                 // We intentionally don't output a newline here. The default PS1 for Bash in NixOS starts with a newline

lix/legacy/user-env.cc

@@ -46,30 +46,31 @@ bool createUserEnv(EvalState & state, DrvInfos & elems,
 
         auto attrs = state.ctx.buildBindings(7 + outputs.size());
 
-        attrs.alloc(state.ctx.symbols.sym_type) = {NewValueAs::string, "derivation"};
-        attrs.alloc(state.ctx.symbols.sym_name) = {NewValueAs::string, i.queryName(state)};
+        attrs.insert(state.ctx.symbols.sym_type, {NewValueAs::string, "derivation"});
+        attrs.insert(state.ctx.symbols.sym_name, {NewValueAs::string, i.queryName(state)});
         auto system = i.querySystem(state);
         if (!system.empty())
-            attrs.alloc(state.ctx.symbols.sym_system) = {NewValueAs::string, system};
-        attrs.alloc(state.ctx.symbols.sym_outPath) = {
-            NewValueAs::string, state.ctx.store->printStorePath(i.queryOutPath(state))
-        };
+            attrs.insert(state.ctx.symbols.sym_system, {NewValueAs::string, system});
+        attrs.insert(
+            state.ctx.symbols.sym_outPath,
+            {NewValueAs::string, state.ctx.store->printStorePath(i.queryOutPath(state))}
+        );
         if (drvPath)
-            attrs.alloc(state.ctx.symbols.sym_drvPath) = {
-                NewValueAs::string, state.ctx.store->printStorePath(*drvPath)
-            };
+            attrs.insert(
+                state.ctx.symbols.sym_drvPath, {NewValueAs::string, state.ctx.store->printStorePath(*drvPath)}
+            );
 
         // Copy each output meant for installation.
-        auto & vOutputs = attrs.alloc(state.ctx.symbols.sym_outputs);
         auto outputsList = state.ctx.mem.newList(outputs.size());
-        vOutputs = {NewValueAs::list, outputsList};
+        attrs.insert(state.ctx.symbols.sym_outputs, {NewValueAs::list, outputsList});
         for (const auto & [m, j] : enumerate(outputs)) {
             outputsList->elems[m] = {NewValueAs::string, j.first};
             auto outputAttrs = state.ctx.buildBindings(2);
-            outputAttrs.alloc(state.ctx.symbols.sym_outPath) = {
-                NewValueAs::string, state.ctx.store->printStorePath(*j.second)
-            };
-            attrs.alloc(j.first) = {NewValueAs::attrs, outputAttrs};
+            outputAttrs.insert(
+                state.ctx.symbols.sym_outPath,
+                {NewValueAs::string, state.ctx.store->printStorePath(*j.second)}
+            );
+            attrs.insert(j.first, {NewValueAs::attrs, outputAttrs});
 
             /* This is only necessary when installing store paths, e.g.,
                `nix-env -i /nix/store/abcd...-foo'. */
@@ -87,7 +88,7 @@ bool createUserEnv(EvalState & state, DrvInfos & elems,
             meta.insert(state.ctx.symbols.create(j), *v);
         }
 
-        attrs.alloc(state.ctx.symbols.sym_meta) = {NewValueAs::attrs, meta};
+        attrs.insert(state.ctx.symbols.sym_meta, {NewValueAs::attrs, meta});
 
         manifest->elems[n++] = {NewValueAs::attrs, attrs};
 
@@ -111,7 +112,7 @@ bool createUserEnv(EvalState & state, DrvInfos & elems,
     /* Construct a Nix expression that calls the user environment
        builder with the manifest as argument. */
     auto attrs = state.ctx.buildBindings(3);
-    state.ctx.paths.mkStorePathString(manifestFile, attrs.alloc("manifest"));
+    attrs.insert("manifest", state.ctx.paths.mkStorePathString(manifestFile));
     attrs.insert(state.ctx.symbols.create("derivations"), vManifest);
     Value args = {NewValueAs::attrs, attrs};
 

lix/libcmd/common-eval-args.cc

@@ -1,3 +1,6 @@
+#include "libexpr/value.hh"
+#include "libutil/strings.hh"
+#include "lix/libexpr/attr-path.hh"
 #include "lix/libexpr/eval-settings.hh"
 #include "lix/libcmd/common-eval-args.hh"
 #include "lix/libmain/shared.hh"
@@ -10,31 +13,10 @@
 #include "lix/libcmd/command.hh"
 #include "lix/libutil/async.hh"
 #include "lix/libutil/error.hh"
-#include "lix/libutil/regex.hh"
-
-#include <regex>
+#include <deque>
 
 namespace nix {
 
-static std::regex const identifierRegex = regex::parse("^[A-Za-z_][A-Za-z0-9_'-]*$");
-static void checkValidNixIdentifier(const std::string & name)
-{
-    std::smatch match;
-    if (!std::regex_match(name, match, identifierRegex)) {
-        throw UsageError(
-            "This invocation specifies a value for argument '%s' "
-            "which isn't a valid Nix identifier. "
-            "The project is dropping support for this so that it's possible to make e.g. "
-            "'%s' evaluating to '%s' in the future. "
-            "If you depend on this behavior, please reach out in "
-            "<https://git.lix.systems/lix-project/lix/issues/496> so we can discuss your use-case.",
-            name,
-            "--arg config.allowUnfree true",
-            "{ config.allowUnfree = true; }"
-        );
-    }
-}
-
 MixEvalArgs::MixEvalArgs()
 {
     addFlag(
@@ -42,10 +24,7 @@ MixEvalArgs::MixEvalArgs()
          .description = "Pass the value *expr* as the argument *name* to Nix functions.",
          .category = category,
          .labels = {"name", "expr"},
-         .handler = {[&](std::string name, std::string expr) {
-             checkValidNixIdentifier(name);
-             autoArgs[name] = 'E' + expr;
-         }}}
+         .handler = {[&](std::string name, std::string expr) { autoArgs[name] = ExprArgument(expr); }}}
     );
 
     addFlag({
@@ -53,10 +32,7 @@ MixEvalArgs::MixEvalArgs()
         .description = "Pass the string *string* as the argument *name* to Nix functions.",
         .category = category,
         .labels = {"name", "string"},
-        .handler = {[&](std::string name, std::string s) {
-            checkValidNixIdentifier(name);
-            autoArgs[name] = 'S' + s;
-        }},
+        .handler = {[&](std::string name, std::string s) { autoArgs[name] = StringArgument(s); }},
     });
 
     addFlag({
@@ -179,18 +155,90 @@ MixEvalArgs::MixEvalArgs()
     });
 }
 
+struct AutoArgsContainer
+{
+    std::map<Symbol, std::variant<Value, AutoArgsContainer>> data;
+
+    Bindings * toBindings(Evaluator & state)
+    {
+        auto bb = state.buildBindings(data.size());
+
+        for (auto & [sym, v] : data) {
+            bb.insert(
+                sym,
+                std::visit(
+                    overloaded{
+                        [&](Value & v) { return v; },
+                        [&](AutoArgsContainer & aac) -> Value {
+                            return {NewValueAs::attrs, aac.toBindings(state)};
+                        }
+                    },
+                    v
+                )
+            );
+        }
+
+        return bb.finish();
+    }
+};
+
+static void addAutoArgRecursive(
+    AutoArgsContainer & container,
+    Evaluator & state,
+    std::vector<std::string> && path,
+    Value & val,
+    const std::string_view pathStr
+)
+{
+    auto * data = &container.data;
+    auto size = path.size();
+    for (auto [i, pathCmp] : enumerate(path)) {
+        auto next = state.symbols.create(pathCmp);
+        auto entry = data->find(next);
+
+        if (entry == data->end()) {
+            if (i == size - 1) {
+                (*data)[next] = val;
+            } else {
+                (*data)[next] = AutoArgsContainer{};
+                data = &std::get<AutoArgsContainer>((*data)[next]).data;
+            }
+        } else {
+            std::visit(
+                overloaded{
+                    [&](Value & v) {
+                        throw Error(
+                            "Cannot set %s via --arg/--argstr when it's the path-extension of another "
+                            "auto-argument!",
+                            pathStr
+                        );
+                    },
+                    [&](AutoArgsContainer & v) { data = &v.data; }
+                },
+                entry->second
+            );
+        }
+    }
+}
+
 Bindings * MixEvalArgs::getAutoArgs(Evaluator & state)
 {
-    auto res = state.buildBindings(autoArgs.size());
-    for (auto & i : autoArgs) {
-        Value v;
-        if (i.second[0] == 'E')
-            v = state.evalLazily(state.parseExprFromString(i.second.substr(1), CanonPath::fromCwd()));
-        else
-            v = {NewValueAs::string, ((std::string_view) i.second).substr(1)};
-        res.insert(state.symbols.create(i.first), v);
+    AutoArgsContainer aac;
+    for (auto & [name, value] : autoArgs) {
+        Value v = std::visit(
+            overloaded{
+                [&](StringArgument & str) -> Value { return {NewValueAs::string, (std::string_view) str.value}; },
+                [&](ExprArgument & e) -> Value {
+                    return state.evalLazily(state.parseExprFromString(e.expr, CanonPath::fromCwd()));
+                }
+            },
+            value
+        );
+
+        addAutoArgRecursive(aac, state, parseAttrPath(name, false), v, name);
     }
-    return res.finish();
+
+    return aac.toBindings(state);
 }
 
 kj::Promise<Result<EvalPaths::PathResult<SourcePath, ThrownError>>>

lix/libcmd/common-eval-args.hh

@@ -14,6 +14,15 @@ class EvalState;
 class Bindings;
 struct SourcePath;
 
+struct StringArgument
+{
+    std::string value;
+};
+struct ExprArgument
+{
+    std::string expr;
+};
+
 struct MixEvalArgs : virtual Args, virtual MixRepair
 {
     static constexpr auto category = "Common evaluation options";
@@ -27,7 +36,7 @@ struct MixEvalArgs : virtual Args, virtual MixRepair
     std::optional<std::string> evalStoreUrl;
 
 private:
-    std::map<std::string, std::string> autoArgs;
+    std::map<std::string, std::variant<StringArgument, ExprArgument>> autoArgs;
 };
 
 /** @brief Resolve an argument that is generally a file, but could be something that

lix/libcmd/installables.cc

@@ -446,18 +446,18 @@ Installables SourceExprCommand::parseInstallables(
             throw UsageError("'--file' and '--expr' are exclusive");
 
         auto evaluator = getEvaluator();
-        Value vFile;
 
-        if (file == "-") {
-            auto & e = evaluator->parseStdin();
-            vFile = state.eval(e);
-        }
-        else if (file)
-            vFile = state.evalFile(state.aio.blockOn(lookupFileArg(*evaluator, *file)).unwrap());
-        else {
-            auto & e = evaluator->parseExprFromString(*expr, CanonPath::fromCwd());
-            vFile = state.eval(e);
-        }
+        Value vFile = [&](NeverAsync = {}) {
+            if (file == "-") {
+                auto & e = evaluator->parseStdin();
+                return state.eval(e);
+            } else if (file) {
+                return state.evalFile(state.aio.blockOn(lookupFileArg(*evaluator, *file)).unwrap());
+            } else {
+                auto & e = evaluator->parseExprFromString(*expr, CanonPath::fromCwd());
+                return state.eval(e);
+            }
+        }();
 
         for (auto & s : ss) {
             auto [prefix, extendedOutputsSpec] = ExtendedOutputsSpec::parse(s);

lix/libcmd/markdown.cc

@@ -55,7 +55,6 @@ std::string renderMarkdownToTerminal(std::string_view markdown, StandardOutputSt
 
     struct lowdown_opts opts{
         .type = LOWDOWN_TERM,
-#ifdef LOWDOWN_SEPARATE_TERM_OPTS
         .term =
             {
                 .cols = lowdown_cols,
@@ -65,16 +64,13 @@ std::string renderMarkdownToTerminal(std::string_view markdown, StandardOutputSt
                 .vmargin = 0,
                 .centre = 0,
             },
-        // maxdepth needs to be part of the ifdefs to match declaration order
         .maxdepth = 20,
-#else
-        .maxdepth = 20,
-        .cols = lowdown_cols,
-        .hmargin = 0,
-        .vmargin = 0,
-#endif /* LOWDOWN_SEPARATE_TERM_OPTS */
         .feat = LOWDOWN_COMMONMARK | LOWDOWN_FENCED | LOWDOWN_DEFLIST | LOWDOWN_TABLES,
+#ifdef LOWDOWN_CONSOLIDATED_OFLAGS
+        .oflags = LOWDOWN_NOLINK,
+#else
         .oflags = LOWDOWN_TERM_NOLINK,
+#endif /* LOWDOWN_CONSOLIDATED_OFLAGS */
     };
     if (!shouldANSI(fileno)) {
         opts.oflags |= LOWDOWN_TERM_NOANSI;

lix/libcmd/repl.cc

@@ -1,10 +1,13 @@
+#include <algorithm>
 #include <cstdio>
 #include <editline.h>
 #include <iostream>
 #include <cstdlib>
 #include <cstring>
+#include <optional>
 #include <string_view>
 
+#include "libutil/logging.hh"
 #include "lix/libexpr/value.hh"
 #include "lix/libutil/box_ptr.hh"
 #include "lix/libcmd/repl-interacter.hh"
@@ -711,29 +714,111 @@ void NixRepl::initDebugBuiltinCommands()
 
     addCommand(
         "show-trace",
+        // this command has a bit of nuance to its function and error states.
+        // it can either:
+        //      1. be called without any argument
+        //          -> just display the current stack frame (still have to walk up the stack :/)
+        //      2. be called with an absolute index
+        //          -> try to go to that frame
+        //          -> if it doesn't exist, print an "arg out of range" error
+        //      3. be called with a relative index
+        //          -> if the final offset is in-bounds, go to that frame
+        //          -> otherwise: clamp the index, i.e. go to 0/$max instead of out-of-bounds
+        //
+        // because the collection of frames is lazy and isn't a random-access list,
+        // we need to iterate the whole stack for most of these if we want to have
+        // good error messages; this is the biggest reason why this function is so
+        // long/complex compared to its role
+        //
         [](NixRepl & repl, const std::string & arg) {
-            try {
-                repl.debugTraceIndex = stoi(arg);
-            } catch (...) {
+            auto setTrace = [&](size_t traceIdx, const DebugTrace * trace) {
+                repl.debugTraceIndex = traceIdx;
+                std::cout << "\n" << ANSI_BLUE << traceIdx << ANSI_NORMAL << ": ";
+                showDebugTrace(std::cout, repl.evaluator.positions, *trace);
+                std::cout << std::endl;
+                printEnvBindings(repl.state, trace->expr, trace->env);
+                repl.loadDebugTraceEnv(*trace);
+            };
+
+            // tries to find a trace at a given index.
+            // - if it is found, it returns the requested trace, along with its
+            //   index, which will be *the same* as requested
+            // - otherwise, it returns the last (=outermost) trace, along with
+            //   its index, which will be *different* than the one requested
+            auto tryFindTrace = [&](size_t traceIdx) -> std::pair<size_t, const DebugTrace *> {
+                size_t lastIndex = 0;
+                const DebugTrace * lastTrace;
+                auto traces = repl.evaluator.debug->traces();
+                for (const auto & [idx, i] : enumerate(traces)) {
+                    lastTrace = i;
+                    lastIndex = idx;
+                    if (idx == traceIdx) {
+                        return std::pair(idx, i);
+                    }
+                }
+                return std::pair(lastIndex, lastTrace);
+            };
+
+            bool isRelativeIdx = false;
+            int requestedTraceIdx;
+            if (arg.length() == 0) {
+                // if there's no argument, just re-print the current frame
+                requestedTraceIdx = repl.debugTraceIndex;
+            } else {
+                std::optional<int> maybeIdx = string2Int<int>(arg);
+                if (!maybeIdx) {
+                    throw Error("argument '%s' is not a valid integer", arg);
+                }
+
+                isRelativeIdx = arg.starts_with('+') || arg.starts_with('-');
+                requestedTraceIdx =
+                    isRelativeIdx ? maybeIdx.value() + repl.debugTraceIndex : maybeIdx.value();
             }
 
-            auto traces = repl.evaluator.debug->traces();
-            for (const auto & [idx, i] : enumerate(traces)) {
-                if (idx == repl.debugTraceIndex) {
-                    std::cout << "\n" << ANSI_BLUE << idx << ANSI_NORMAL << ": ";
-                    showDebugTrace(std::cout, repl.evaluator.positions, *i);
-                    std::cout << std::endl;
-                    printEnvBindings(repl.state, i->expr, i->env);
-                    repl.loadDebugTraceEnv(*i);
-                    break;
-                }
+            auto [actualTraceIdx, trace] = tryFindTrace((size_t) requestedTraceIdx);
+
+            // if we *did* find the frame we wanted originally, all is well
+            // in the world and we can just load it and exit
+            if (actualTraceIdx == (size_t) requestedTraceIdx) {
+                setTrace(actualTraceIdx, trace);
+                return ProcessLineResult::PromptAgain;
+            }
+
+            // if we couldn't immediately find the requested trace on the "happy path", then either:
+
+            // a) it was an absolute index but didn't exist
+            //      -> print a specific error showing the exact valid range
+            if (!isRelativeIdx) {
+                throw Error(
+                    "stack index must be between %ld and %ld (inclusive), but was %ld",
+                    0,
+                    actualTraceIdx, // tryFindTrace sets *idx to the final (max) frame index if it fails
+                    requestedTraceIdx
+                );
+            }
+
+            // b) it was a relative index
+            //      -> clamp the index to the bounds and print a warning
+            if (requestedTraceIdx < 0) {
+                // just load frame 0 but print a warning about the bounds
+                std::tie(actualTraceIdx, trace) = tryFindTrace(0);
+                setTrace(actualTraceIdx, trace);
+                printTaggedWarning("stopped at stack frame %ld, cannot go any deeper", 0);
+                return ProcessLineResult::PromptAgain;
+            } else {
+                // (if we're here, then requestedTraceIdx > $max, since tryFindTrace failed)
+                // load the max frame (that `tryFindFrame` kindly already got for us),
+                // but print a warning that we can't go any further
+                setTrace(actualTraceIdx, trace);
+                printTaggedWarning("stopped at stack frame %ld, cannot go any higher", actualTraceIdx);
+                return ProcessLineResult::PromptAgain;
             }
-            return ProcessLineResult::PromptAgain;
         },
         {.aliases = {"st"},
          .debugModeOnly = true,
          .help = "Show current trace. If an integer is provided, this switches to that stack "
-                 "beforehand.",
+                 "beforehand. If the integer has an explicit + or - sign, it is treated as"
+                 "relative to the current stack index.",
          .section = "Debug mode",
          .positionalArgsSpecifiers = {{.placeholderText = "integer index", .optional = true}}}
     );
@@ -867,8 +952,7 @@ void NixRepl::initBuiltinCommands()
         [](NixRepl & repl, const std::string & arg) {
             Value v = repl.evalString(arg);
             Value f = repl.evalString(
-                R""("drv: (import <nixpkgs> {}).runCommand "shell") ""
-                R""({ buildInputs = [ drv ]; } "")""
+                R"(drv: (import <nixpkgs> {}).runCommand "shell" { buildInputs = [ drv ]; } "")"
             );
             Value result = repl.state.callFunction(f, v, PosIdx());
 
@@ -1318,12 +1402,10 @@ void NixRepl::loadFlake(const std::string & flakeRefS)
         .kind = ReplLoadKind::Flake,
     };
 
-    Value v;
-
     try {
         loaded.remove(loadable);
         loaded.push_back(loadable);
-        v = flake::callFlake(
+        Value v = flake::callFlake(
             state,
             flake::lockFlake(
                 state,
@@ -1451,7 +1533,6 @@ Value NixRepl::getReplOverlaysEvalFunction()
     }
 
     auto evalReplInitFilesPath = CanonPath::root + "repl-overlays.nix";
-    *replOverlaysEvalFunction = Value{};
     auto code =
         #include "repl-overlays.nix.gen.hh"
         ;
@@ -1461,16 +1542,15 @@ Value NixRepl::getReplOverlaysEvalFunction()
         evaluator.builtins.staticEnv
     );
 
-    **replOverlaysEvalFunction = state.eval(expr);
+    *replOverlaysEvalFunction = state.eval(expr);
 
     return **replOverlaysEvalFunction;
 }
 
 Value NixRepl::replOverlays()
 {
-    Value replInits;
     auto replInitStorage = evaluator.mem.newList(evalSettings.replOverlays.get().size());
-    replInits = {NewValueAs::list, replInitStorage};
+    Value replInits = {NewValueAs::list, replInitStorage};
 
     size_t i = 0;
     for (auto path : evalSettings.replOverlays.get()) {

lix/libexec/launch-builder-linux.cc

@@ -3,6 +3,7 @@
 #include "lix/libutil/rpc.hh"
 #include <cassert>
 #include <csignal>
+#include <fcntl.h>
 #include <filesystem>
 #include <format>
 #include <kj/io.h>
@@ -224,7 +225,11 @@ bool prepareChildSetup(build::Request::Reader request)
                     };
             const fs::path dst = chrootRootDir / target.relative_path();
             fs::create_directories(dst.parent_path());
-            writeFile(dst, std::string_view((const char *) sh, sizeof(sh)));
+            kj::AutoCloseFd fd(open(dst.c_str(), O_RDWR | O_CREAT, 0755));
+            if (fd == nullptr) {
+                throw SysError("cannot create sandbox shell");
+            }
+            writeFull(fd.get(), std::string_view((const char *) sh, sizeof(sh)));
             fs::permissions(dst, fs::perms(0555));
         } else
 #endif

lix/libexpr/attr-path.cc

@@ -7,8 +7,7 @@
 
 namespace nix {
 
-
-std::vector<std::string> parseAttrPath(std::string_view const s)
+std::vector<std::string> parseAttrPath(std::string_view const s, bool allowRhsTrailingDot)
 {
     std::vector<std::string> res;
     std::string cur;
@@ -16,6 +15,22 @@ std::vector<std::string> parseAttrPath(std::string_view const s)
     auto i = s.begin();
     while (i != s.end()) {
         if (*i == '.') {
+            if (!haveData) {
+                if (res.empty()) {
+                    throw ParseError(
+                        "Leading dot in attribute selection path '%1%' is not allowed! If the attribute name "
+                        "is an empty string, use '\"\".foo.bar'",
+                        s
+                    );
+                } else {
+                    throw ParseError(
+                        "consecutive dots not allowed in selection path '%1%', use 'foo.\"\".bar' to denote "
+                        "an "
+                        "empty attribute name",
+                        s
+                    );
+                }
+            }
             res.push_back(cur);
             haveData = false;
             cur.clear();
@@ -36,7 +51,11 @@ std::vector<std::string> parseAttrPath(std::string_view const s)
         }
         ++i;
     }
-    if (haveData) res.push_back(cur);
+    if (haveData) {
+        res.push_back(cur);
+    } else if (!allowRhsTrailingDot) {
+        throw ParseError("Trailing dot on the right-hand side of path expr '%1%' is not allowed!", s);
+    };
     return res;
 }
 
@@ -90,9 +109,6 @@ findAlongAttrPath(EvalState & state, const std::string & attrPath, Bindings & au
            according to what is specified in the attrPath. */
 
         if (!attrIndex) {
-            if (attr.empty())
-                throw Error("empty attribute name in selection path '%1%'", attrPath);
-
             if (v.type() != nAttrs) {
                 auto pathPart =
                     std::vector<std::string>(tokens.begin(), tokens.begin() + attrPathIdx);
@@ -163,13 +179,14 @@ findAlongAttrPath(EvalState & state, const std::string & attrPath, Bindings & au
 
 std::pair<SourcePath, uint32_t> findPackageFilename(EvalState & state, Value & v, std::string what)
 {
-    Value v2;
-    try {
-        auto dummyArgs = state.ctx.mem.allocBindings(0);
-        v2 = findAlongAttrPath(state, "meta.position", *dummyArgs, v).first;
-    } catch (Error &) {
-        throw NoPositionInfo("package '%s' has no source location information", what);
-    }
+    Value v2 = [&]() {
+        try {
+            auto dummyArgs = state.ctx.mem.allocBindings(0);
+            return findAlongAttrPath(state, "meta.position", *dummyArgs, v).first;
+        } catch (Error &) {
+            throw NoPositionInfo("package '%s' has no source location information", what);
+        }
+    }();
 
     // FIXME: is it possible to extract the Pos object instead of doing this
     //        toString + parsing?

lix/libexpr/attr-path.hh

@@ -25,7 +25,7 @@ std::pair<SourcePath, uint32_t> findPackageFilename(EvalState & state, Value & v
  * Such an attr path is a dot-separated sequence of attribute names, which are possibly quoted.
  * No escaping is performed; attribute names containing double quotes are unrepresentable.
  */
-std::vector<std::string> parseAttrPath(std::string_view const s);
+std::vector<std::string> parseAttrPath(std::string_view const s, bool allowRhsTrailingDot = true);
 
 /**
  * Converts an attr path from a list of strings into a string once more.

lix/libexpr/attr-set.cc

@@ -23,20 +23,11 @@ Bindings * EvalMemory::allocBindings(size_t capacity)
     return new (allocBytes(sizeof(Bindings) + sizeof(Attr) * capacity)) Bindings();
 }
 
-
-Value & BindingsBuilder::alloc(Symbol name, PosIdx pos)
+void BindingsBuilder::insert(std::string_view name, Value value, PosIdx pos)
 {
-    bindings->push_back(Attr(name, {}, pos));
-    return (bindings->end() - 1)->value;
+    return insert(symbols.create(name), value, pos);
 }
 
-
-Value & BindingsBuilder::alloc(std::string_view name, PosIdx pos)
-{
-    return alloc(symbols.create(name), pos);
-}
-
-
 void Bindings::sort()
 {
     if (size_) std::sort(begin(), end());

lix/libexpr/attr-set.hh

@@ -25,7 +25,8 @@ struct Attr
     PosIdx pos;
     mutable Value value;
     Attr(Symbol name, Value value, PosIdx pos = noPos) : name(name), pos(pos), value(value) {}
-    Attr() { };
+    [[deprecated]]
+    Attr() {};
     bool operator < (const Attr & a) const
     {
         return name < a.name;
@@ -72,8 +73,9 @@ public:
 
     const Attr * get(Symbol name)
     {
-        Attr key(name, {});
-        iterator i = std::lower_bound(begin(), end(), key);
+        iterator i = std::lower_bound(begin(), end(), name, [](const Attr & value, const Symbol & compare) {
+            return value.name < compare;
+        });
         if (i != end() && i->name == name) return &*i;
         return nullptr;
     }
@@ -140,6 +142,8 @@ public:
         insert(Attr(name, value, pos));
     }
 
+    void insert(std::string_view name, Value value, PosIdx pos = noPos);
+
     void insert(const Attr & attr)
     {
         push_back(attr);
@@ -151,10 +155,6 @@ public:
         bindings->push_back(attr);
     }
 
-    Value & alloc(Symbol name, PosIdx pos = noPos);
-
-    Value & alloc(std::string_view name, PosIdx pos = noPos);
-
     [[nodiscard("must use created bindings")]]
     Bindings * finish()
     {

lix/libexpr/builtins/path.md

@@ -10,7 +10,8 @@ present in *args*. All are optional except `path`:
 
   - name\
     The name of the path when added to the store. This can used to
-    reference paths that have nix-illegal characters in their names,
+    reference paths that have
+    [nix-illegal characters in their names](./derivations.md),
     like `@`.
 
   - filter\

lix/libexpr/builtins/scopedImport.md

@@ -1,6 +1,6 @@
 ---
 name: scopedImport
-implementation: "[](EvalState & state, Value ** args, Value & v) { import(state, *args[1], args[0], v); }"
+implementation: "[](EvalState & state, Value ** args) -> Value { return import(state, *args[1], args[0]); }"
 args: [scope, path]
 renameInGlobalScope: false
 ---

lix/libexpr/eval-expr.cc

@@ -162,18 +162,18 @@ Value ExprSet::eval(EvalState & state, Env & env)
          * http://github.com/NixOS/nix/issues/7012. Any accesses to the output attrset will thus infrec.
          */
         Value vBackup = v;
-        Value nameVal;
+        Symbol nameSym;
         {
             KJ_DEFER(v = vBackup);
             v = Value{NewValueAs::blackhole};
-            nameVal = i.nameExpr->eval(state, *dynamicEnv);
+            Value nameVal = i.nameExpr->eval(state, *dynamicEnv);
             state.forceValue(nameVal, i.pos);
             if (nameVal.type() == nNull) {
                 continue;
             }
             state.forceStringNoCtx(nameVal, i.pos, "while evaluating the name of a dynamic attribute");
+            nameSym = state.ctx.symbols.create(nameVal.str());
         }
-        auto nameSym = state.ctx.symbols.create(nameVal.str());
         auto j = v.attrs()->get(nameSym);
         if (j) {
             state.ctx.errors
@@ -187,7 +187,29 @@ Value ExprSet::eval(EvalState & state, Env & env)
                 .debugThrow();
         }
 
-        i.valueExpr->setName(nameSym);
+        // clang-format off
+        /* This line is so wrong that it is best kept in here with the documentation why it is wrong,
+         * lest some naive soul may add it once again some year in the future.
+         * See the following witness as to why it is wrong:
+         *
+         * nix-repl> fun = (name: { ${name} = x: x; }) # This function creates a dynamic attribute with a variable name
+         * Added fun.
+         * nix-repl> revSeq = x: y: builtins.seq x (builtins.seq y x) # evaluate x, then y in sequence, then return x
+         * Added revSeq.
+         * nix-repl> fun "foo" # The code seemingly works
+         * { foo = «lambda foo @ «string»:1:26»; }
+         * nix-repl> fun "bar" #
+         * { bar = «lambda bar @ «string»:1:26»; }
+         * nix-repl> revSeq (fun "foo") (fun "bar") # Until it doesn't
+         * { foo = «lambda bar @ «string»:1:26»; }
+         *
+         * What happened? Expressions are AST bound, therefore all lambdas share the same Expr and thus *the same name*.
+         * Using `setName` here updates the name of *all* lambdas from that expression, not just of the value at hand.
+         * And this is why all expressions must be treated as immutable after parsing.
+         */
+        /* i.valueExpr->setName(nameSym); */
+        // clang-format on
+
         /* Keep sorted order so find can catch duplicates */
         v.attrs()->push_back(Attr(nameSym, i.valueExpr->maybeThunk(state, *dynamicEnv), i.pos));
         v.attrs()->sort(); // FIXME: inefficient
@@ -329,45 +351,7 @@ Value ExprOpUpdate::eval(EvalState & state, Env & env)
     Value v2 = e2->eval(state, env);
     state.checkAttrs(v2, env, *e2);
 
-    state.ctx.stats.nrOpUpdates++;
-
-    if (v1.attrs()->size() == 0) {
-        return v2;
-    }
-    if (v2.attrs()->size() == 0) {
-        return v1;
-    }
-
-    auto attrs = state.ctx.buildBindings(v1.attrs()->size() + v2.attrs()->size());
-
-    /* Merge the sets, preferring values from the second set.  Make
-       sure to keep the resulting vector in sorted order. */
-    Bindings::iterator i = v1.attrs()->begin();
-    Bindings::iterator j = v2.attrs()->begin();
-
-    while (i != v1.attrs()->end() && j != v2.attrs()->end()) {
-        if (i->name == j->name) {
-            attrs.insert(*j);
-            ++i;
-            ++j;
-        } else if (i->name < j->name) {
-            attrs.insert(*i++);
-        } else {
-            attrs.insert(*j++);
-        }
-    }
-
-    while (i != v1.attrs()->end()) {
-        attrs.insert(*i++);
-    }
-    while (j != v2.attrs()->end()) {
-        attrs.insert(*j++);
-    }
-
-    Value v = {NewValueAs::attrs, attrs.alreadySorted()};
-
-    state.ctx.stats.nrOpUpdateValuesCopied += v.attrs()->size();
-    return v;
+    return state.updateAttrs(v1, v2);
 }
 
 Value ExprOpConcatLists::eval(EvalState & state, Env & env)
@@ -429,12 +413,12 @@ Value ExprConcatStrings::eval(EvalState & state, Env & env)
     };
 
     // List of returned strings. References to these Values must NOT be persisted.
-    SmallTemporaryValueVector<conservativeStackReservation> values(es.size());
-    Value * vTmpP = values.data();
+    SmallTemporaryValueVector<conservativeStackReservation> values;
+    values.reserve(es.size());
 
     for (auto & [i_pos, i] : es) {
-        Value & vTmp = *vTmpP++;
-        vTmp = i->eval(state, env);
+        values.push_back(i->eval(state, env));
+        Value & vTmp = values.back();
 
         /* If the first element is a path, then the result will also
            be a path, we don't copy anything (yet - that's done later,
@@ -527,9 +511,7 @@ Value ExprConcatStrings::eval(EvalState & state, Env & env)
 
 Value ExprPos::eval(EvalState & state, Env & env)
 {
-    Value v;
-    state.mkPos(v, pos);
-    return v;
+    return state.mkPos(pos);
 }
 
 Value ExprBlackHole::eval(EvalState & state, Env & env)
@@ -619,19 +601,20 @@ Value ExprSelect::eval(EvalState & state, Env & env)
     // Position for the current selector in this select chain.
     PosIdx posCurrentSyntax;
 
-    Value baseSelectee;
-    try {
-        // Evaluate the original thing we're selecting on.
-        baseSelectee = e->eval(state, env);
-    } catch (Error & e) {
-        // clang-format off
-        e.addTrace(state.ctx.positions[getPos()], HintFmt(
-            "while evaluating an expression to select '%s' on it",
-            showAttrPath(state.ctx.symbols, attrPath)
-        ));
-        // clang-format on
-        throw;
-    }
+    Value baseSelectee = [&]() {
+        try {
+            // Evaluate the original thing we're selecting on.
+            return e->eval(state, env);
+        } catch (Error & e) {
+            // clang-format off
+            e.addTrace(state.ctx.positions[getPos()], HintFmt(
+                "while evaluating an expression to select '%s' on it",
+                showAttrPath(state.ctx.symbols, attrPath)
+            ));
+            // clang-format on
+            throw;
+        }
+    }();
 
     try {
         // With the original selectee evaluated, we'll walk the selection path starting
@@ -717,9 +700,10 @@ Value ExprCall::eval(EvalState & state, Env & env)
     // 5: under 10
     // This excluded attrset lambdas (`{...}:`). Contributions of mixed lambdas appears insignificant at ~150
     // total.
-    SmallValueVector<4> vArgs(args.size());
+    SmallValueVector<4> vArgs;
+    vArgs.reserve(args.size());
     for (size_t i = 0; i < args.size(); ++i) {
-        vArgs[i] = args[i]->maybeThunk(state, env);
+        vArgs.push_back(args[i]->maybeThunk(state, env));
     }
 
     return state.callFunction(vFun, vArgs, pos);

lix/libexpr/eval.cc

@@ -297,30 +297,29 @@ EvalPaths::EvalPaths(
         }
     }
 
-#if LIX_MAJOR >= 3 || (LIX_MAJOR == 2 && LIX_MINOR >= 96)
+#if LIX_MAJOR >= 3 || (LIX_MAJOR == 2 && LIX_MINOR >= 98)
 #warning \
-    "The feature nix-path-shadow was deprecated in 2.95 with a warning, which needs to be turned into an error in 2.96"
+    "The feature nix-path-shadow was deprecated in 2.95 with a warning, error in 2.96, and we should consider removing the bypass in 2.98"
 #endif
+
     if (!featureSettings.isEnabled(DeprecatedFeature::NixPathShadow)) {
         for (auto & [prefix, path] : searchPath_.elements) {
             // Match on the 'nix' prefix
             if (prefix.s == "nix") {
-                logWarning(
-                    {.msg = HintFmt(
-                         "The prefix '%s' is reserved for internal use by Lix in the Nix search "
-                         "path, its usage is deprecated and will be forbidden in the future.\n"
-                         "Use %s to silence this warning.\n"
-                         "This is due to adding '%s=%s' in the Nix search path, either through the "
-                         "environment variable '%s' or by passing the flag %s to the nix "
-                         "invocation.",
-                         "nix",
-                         "--extra-deprecated-features nix-path-shadow",
-                         prefix.s,
-                         path.s,
-                         "NIX_PATH",
-                         "-I"
-                     )}
-                );
+                throw EvalError(HintFmt(
+                    "The prefix '%s' is reserved for internal use by Lix in the Nix search "
+                    "path, its usage is deprecated and will be forbidden in the future.\n"
+                    "Use %s to silence this error.\n"
+                    "This is due to adding '%s=%s' in the Nix search path, either through the "
+                    "environment variable '%s' or by passing the flag %s to the nix "
+                    "invocation.",
+                    "nix",
+                    "--extra-deprecated-features nix-path-shadow",
+                    prefix.s,
+                    path.s,
+                    "NIX_PATH",
+                    "-I"
+                ));
             } else
                 // Match prefixless paths that contain a `nix` directory
                 if (auto res =
@@ -333,21 +332,19 @@ EvalPaths::EvalPaths(
                                 });
                         }))
                 {
-                    logWarning(
-                        {.msg = HintFmt(
-                             "Shadowing '%s' by configuring the nix-path is deprecated and "
-                             "will be forbidden in the future.\n"
-                             "Use %s to silence this warning.\n"
-                             "This is due to adding '%s' to the nix-path without a prefix, "
-                             "either by passing the flag '-I %s' to the nix invocation or by "
-                             "adding this path to the environment variable '%s'.",
-                             "<nix/...>",
-                             "--extra-deprecated-features nix-path-shadow",
-                             path.s,
-                             path.s,
-                             "NIX_PATH"
-                         )}
-                    );
+                    throw EvalError(HintFmt(
+                        "Shadowing '%s' by configuring the nix-path is deprecated and "
+                        "will be forbidden in the future.\n"
+                        "Use %s to silence this error.\n"
+                        "This is due to adding '%s' to the nix-path without a prefix, "
+                        "either by passing the flag '-I %s' to the nix invocation or by "
+                        "adding this path to the environment variable '%s'.",
+                        "<nix/...>",
+                        "--extra-deprecated-features nix-path-shadow",
+                        path.s,
+                        path.s,
+                        "NIX_PATH"
+                    ));
                 }
         }
     }
@@ -437,11 +434,11 @@ void EvalPaths::allowPath(const StorePath & storePath)
         allowPath(store->toRealPath(storePath));
 }
 
-void EvalPaths::allowAndSetStorePathString(const StorePath & storePath, Value & v)
+Value EvalPaths::allowAndSetStorePathString(const StorePath & storePath)
 {
     allowPath(storePath);
 
-    mkStorePathString(storePath, v);
+    return mkStorePathString(storePath);
 }
 
 CheckedSourcePath EvalPaths::checkSourcePath(const SourcePath & path_)
@@ -736,13 +733,13 @@ void mapStaticEnvBindings(const SymbolTable & st, const StaticEnv & se, const En
             // add 'with' bindings.
             Bindings::iterator j = env.values[0].attrs()->begin();
             while (j != env.values[0].attrs()->end()) {
-                vm[std::string(st[j->name])] = j->value;
+                vm.insert_or_assign(std::string(st[j->name]), j->value);
                 ++j;
             }
         } else {
             // iterate through staticenv bindings and add them.
             for (auto & i : se.vars)
-                vm[std::string(st[i.first])] = env.values[i.second];
+                vm.insert_or_assign(std::string(st[i.first]), env.values[i.second]);
         }
     }
 }
@@ -854,22 +851,23 @@ Value Evaluator::evalLazily(Expr & e)
     return {NewValueAs::thunk, mem, builtins.env, e};
 }
 
-void EvalState::mkPos(Value & v, PosIdx p)
+Value EvalState::mkPos(PosIdx p)
 {
     auto origin = ctx.positions.originOf(p);
     if (auto path = std::get_if<CheckedSourcePath>(&origin)) {
         auto attrs = ctx.buildBindings(3);
-        attrs.alloc(ctx.symbols.sym_file) = {NewValueAs::string, path->to_string()};
-        makePositionThunks(*this, p, attrs.alloc(ctx.symbols.sym_line), attrs.alloc(ctx.symbols.sym_column));
-        v = {NewValueAs::attrs, attrs};
+        attrs.insert(ctx.symbols.sym_file, {NewValueAs::string, path->to_string()});
+        auto [line, col] = makePositionThunks(*this, p);
+        attrs.insert(ctx.symbols.sym_line, line);
+        attrs.insert(ctx.symbols.sym_column, col);
+        return {NewValueAs::attrs, attrs};
     } else
-        v.mkNull();
+        return Value::VNULL;
 }
 
-
-void EvalPaths::mkStorePathString(const StorePath & p, Value & v)
+Value EvalPaths::mkStorePathString(const StorePath & p)
 {
-    v = {
+    return {
         NewValueAs::string,
         store->printStorePath(p),
         NixStringContext{
@@ -885,13 +883,9 @@ std::string EvalState::mkOutputStringRaw(
     return ctx.store->printStorePath(staticOutputPath);
 }
 
-
-void EvalState::mkOutputString(
-    Value & value,
-    const SingleDerivedPath::Built & b,
-    const StorePath & staticOutputPath)
+Value EvalState::mkOutputString(const SingleDerivedPath::Built & b, const StorePath & staticOutputPath)
 {
-    value = {NewValueAs::string, mkOutputStringRaw(staticOutputPath), NixStringContext{b}};
+    return {NewValueAs::string, mkOutputStringRaw(staticOutputPath), NixStringContext{b}};
 }
 
 
@@ -1208,11 +1202,11 @@ Value EvalState::callFunction(Value & fun, std::span<Value> args, const PosIdx p
                     for (unsigned i = 0; i < argsLeft; i++) {
                         pargs[i] = &args[i];
                     }
-                    fn->fun(*this, pargs.data(), vCur);
+                    vCur = fn->fun(*this, pargs.data());
                 } catch (ThrownError & e) {
                     // Distinguish between an error that simply happened while "throw"
                     // was being evaluated and an explicit thrown error.
-                    if (fn->name == "throw") {
+                    if (fn->name == "throw" && !e.hasTrace()) {
                         e.addTrace(ctx.positions[pos], "caused by explicit %s", "throw");
                     } else {
                         e.addTrace(ctx.positions[pos], "while calling the '%s' builtin", fn->name);
@@ -1263,7 +1257,7 @@ Value EvalState::callFunction(Value & fun, std::span<Value> args, const PosIdx p
                     // 1. Unify this and above code. Heavily redundant.
                     // 2. Create a fake env (arg1, arg2, etc.) and a fake expr (arg1: arg2: etc: builtins.name arg1 arg2 etc)
                     //    so the debugger allows to inspect the wrong parameters passed to the builtin.
-                    fn->fun(*this, vArgs.data(), vCur);
+                    vCur = fn->fun(*this, vArgs.data());
                 } catch (Error & e) {
                     e.addTrace(ctx.positions[pos], "while calling the '%1%' builtin", fn->name);
                     throw;
@@ -1364,9 +1358,49 @@ https://docs.lix.systems/manual/lix/stable/language/constructs.html#functions)",
     return callFunction(fun, vAttrs, pos);
 }
 
-void EvalState::concatLists(
-    Value & v, std::span<Value> lists, const PosIdx pos, std::string_view errorCtx
-)
+Value EvalState::updateAttrs(const Value & v1, const Value & v2)
+{
+    ctx.stats.nrOpUpdates++;
+
+    if (v1.attrs()->size() == 0) {
+        return v2;
+    }
+    if (v2.attrs()->size() == 0) {
+        return v1;
+    }
+
+    auto attrs = ctx.buildBindings(v1.attrs()->size() + v2.attrs()->size());
+
+    /* Merge the sets, preferring values from the second set.  Make
+       sure to keep the resulting vector in sorted order. */
+    Bindings::iterator i = v1.attrs()->begin();
+    Bindings::iterator j = v2.attrs()->begin();
+
+    while (i != v1.attrs()->end() && j != v2.attrs()->end()) {
+        if (i->name == j->name) {
+            attrs.insert(*j);
+            ++i;
+            ++j;
+        } else if (i->name < j->name) {
+            attrs.insert(*i++);
+        } else {
+            attrs.insert(*j++);
+        }
+    }
+
+    while (i != v1.attrs()->end()) {
+        attrs.insert(*i++);
+    }
+    while (j != v2.attrs()->end()) {
+        attrs.insert(*j++);
+    }
+
+    Value v = {NewValueAs::attrs, attrs.alreadySorted()};
+    ctx.stats.nrOpUpdateValuesCopied += v.attrs()->size();
+    return v;
+}
+
+Value EvalState::concatLists(std::span<Value> lists, const PosIdx pos, std::string_view errorCtx)
 {
     ctx.stats.nrListConcats++;
 
@@ -1382,12 +1416,10 @@ void EvalState::concatLists(
     }
 
     if (nonEmpty && len == nonEmpty->listSize()) {
-        v = *nonEmpty;
-        return;
+        return *nonEmpty;
     }
 
     auto list = ctx.mem.newList(len);
-    v = {NewValueAs::list, list};
     auto out = list->elems;
     for (size_t n = 0, pos = 0; n < lists.size(); ++n) {
         auto l = lists[n].listSize();
@@ -1396,6 +1428,7 @@ void EvalState::concatLists(
         }
         pos += l;
     }
+    return {NewValueAs::list, list};
 }
 
 // always force this to be separate, otherwise forceValue may inline it and take

lix/libexpr/eval.hh

@@ -370,7 +370,7 @@ public:
     /**
      * Allow access to a store path and return it as a string.
      */
-    void allowAndSetStorePathString(const StorePath & storePath, Value & v);
+    Value allowAndSetStorePathString(const StorePath & storePath);
 
     /**
      * Check whether access to a path is allowed and throw an error if
@@ -449,7 +449,7 @@ public:
      * The string is the printed store path with a context containing a
      * single `NixStringContextElem::Opaque` element of that store path.
      */
-    void mkStorePathString(const StorePath & storePath, Value & v);
+    Value mkStorePathString(const StorePath & storePath);
 };
 
 struct EvalStatistics
@@ -800,7 +800,7 @@ public:
      */
     Value autoCallFunction(Bindings & args, Value & fun, PosIdx pos);
 
-    void mkPos(Value & v, PosIdx pos);
+    Value mkPos(PosIdx pos);
 
     /**
      * Create a string representing a `SingleDerivedPath::Built`.
@@ -809,18 +809,13 @@ public:
      * single `NixStringContextElem::Built` element of the drv path and
      * output name.
      *
-     * @param value Value we are settings
-     *
      * @param b the drv whose output we are making a string for, and the
      * output
      *
      * @param staticOutputPath Output path for that string.
      * Will be printed to form string.
      */
-    void mkOutputString(
-        Value & value,
-        const SingleDerivedPath::Built & b,
-        const StorePath & staticOutputPath);
+    Value mkOutputString(const SingleDerivedPath::Built & b, const StorePath & staticOutputPath);
 
     /**
      * Create a string representing a `SingleDerivedPath`.
@@ -831,8 +826,8 @@ public:
         const SingleDerivedPath & p,
         Value & v);
 
-    void
-    concatLists(Value & v, std::span<Value> lists, const PosIdx pos, std::string_view errorCtx);
+    Value updateAttrs(const Value & v1, const Value & v2);
+    Value concatLists(std::span<Value> lists, const PosIdx pos, std::string_view errorCtx);
 
 private:
 

lix/libexpr/extra-primops.hh

@@ -8,25 +8,24 @@ namespace nix {
 class EvalState;
 struct Value;
 
-void prim_addDrvOutputDependencies(EvalState & state, Value * * args, Value & v);
-void prim_fetchTree(EvalState & state, Value * * args, Value & v);
-void prim_fetchGit(EvalState & state, Value * * args, Value & v);
-void prim_fetchMercurial(EvalState & state, Value ** args, Value & v);
-void prim_fetchTarball(EvalState & state, Value * * args, Value & v);
-void prim_fetchurl(EvalState & state, Value * * args, Value & v);
-void prim_fromTOML(EvalState & state, Value * * args, Value & v);
-void prim_appendContext(EvalState & state, Value ** args, Value & v);
-void prim_getContext(EvalState & state, Value * * args, Value & v);
-void prim_hasContext(EvalState & state, Value * * args, Value & v);
-void prim_unsafeDiscardOutputDependency(EvalState & state, Value * * args, Value & v);
-void prim_unsafeDiscardStringContext(EvalState & state, Value ** args, Value & v);
+Value prim_addDrvOutputDependencies(EvalState & state, Value ** args);
+Value prim_fetchTree(EvalState & state, Value ** args);
+Value prim_fetchGit(EvalState & state, Value ** args);
+Value prim_fetchMercurial(EvalState & state, Value ** args);
+Value prim_fetchTarball(EvalState & state, Value ** args);
+Value prim_fetchurl(EvalState & state, Value ** args);
+Value prim_fromTOML(EvalState & state, Value ** args);
+Value prim_appendContext(EvalState & state, Value ** args);
+Value prim_getContext(EvalState & state, Value ** args);
+Value prim_hasContext(EvalState & state, Value ** args);
+Value prim_unsafeDiscardOutputDependency(EvalState & state, Value ** args);
+Value prim_unsafeDiscardStringContext(EvalState & state, Value ** args);
 
 namespace flake {
 
-void prim_flakeRefToString(EvalState & state, Value * * args, Value & v);
-void prim_getFlake(EvalState & state, Value * * args, Value & v);
-void prim_parseFlakeRef(EvalState & state, Value * * args, Value & v);
-
+Value prim_flakeRefToString(EvalState & state, Value ** args);
+Value prim_getFlake(EvalState & state, Value ** args);
+Value prim_parseFlakeRef(EvalState & state, Value ** args);
 }
 
 }

lix/libexpr/flake/config.cc

@@ -35,12 +35,14 @@ static bool batchAskForSetting(
     TrustedList & trustedList,
     std::map<std::string, std::string> & untrustedSettings)
 {
-    printWarning("The following settings require your decision:");
+    std::string warning("The following settings require your decision:");
     for (const auto & [name, valueS] : untrustedSettings) {
         // FIXME: filter ANSI escapes, newlines, \r, etc.
-        logger->cout("- %s = %s", name, valueS);
+        warning += fmt("\n- %s = %s", name, valueS);
     }
 
+    printWarning("%s", warning);
+
     auto reply = logger
                      ->ask(
                          fmt("Do you want to allow configuration settings to be applied?\nThis may allow the "

lix/libexpr/flake/flake.cc

@@ -942,10 +942,7 @@ LockedFlake lockFlake(
 
 Value callFlake(EvalState & state, const LockedFlake & lockedFlake)
 {
-    Value vLocks;
-    Value vRootSubdir;
-
-    vLocks = {NewValueAs::string, lockedFlake.lockFile.to_string()};
+    Value vLocks = {NewValueAs::string, lockedFlake.lockFile.to_string()};
 
     Value vRootSrc = emitTreeAttrs(
         state.ctx,
@@ -955,14 +952,13 @@ Value callFlake(EvalState & state, const LockedFlake & lockedFlake)
         lockedFlake.flake.forceDirty
     );
 
-    vRootSubdir = {NewValueAs::string, lockedFlake.flake.lockedRef.subdir};
+    Value vRootSubdir = {NewValueAs::string, lockedFlake.flake.lockedRef.subdir};
 
     if (!state.ctx.caches.vCallFlake) {
-        state.ctx.caches.vCallFlake = allocRootValue({});
-        *state.ctx.caches.vCallFlake = state.eval(state.ctx.parseExprFromString(
+        state.ctx.caches.vCallFlake = allocRootValue(state.eval(state.ctx.parseExprFromString(
 #include "call-flake.nix.gen.hh"
             , CanonPath::root
-        ));
+        )));
     }
 
     Value vTmp1 = state.callFunction(*state.ctx.caches.vCallFlake, vLocks, noPos);
@@ -970,14 +966,14 @@ Value callFlake(EvalState & state, const LockedFlake & lockedFlake)
     return state.callFunction(vTmp2, vRootSubdir, noPos);
 }
 
-void prim_getFlake(EvalState & state, Value * * args, Value & v)
+Value prim_getFlake(EvalState & state, Value ** args)
 {
     std::string flakeRefS(state.forceStringNoCtx(*args[0], noPos, "while evaluating the argument passed to builtins.getFlake"));
     auto flakeRef = parseFlakeRef(flakeRefS, {}, true);
     if (evalSettings.pureEval && !flakeRef.input.isLocked())
         throw Error("cannot call 'getFlake' on unlocked flake reference '%s' (use --impure to override)", flakeRefS);
 
-    v = callFlake(
+    return callFlake(
         state,
         lockFlake(
             state,
@@ -992,10 +988,7 @@ void prim_getFlake(EvalState & state, Value * * args, Value & v)
     );
 }
 
-void prim_parseFlakeRef(
-    EvalState & state,
-    Value * * args,
-    Value & v)
+Value prim_parseFlakeRef(EvalState & state, Value ** args)
 {
     std::string flakeRefS(state.forceStringNoCtx(*args[0], noPos,
         "while evaluating the argument passed to builtins.parseFlakeRef"));
@@ -1003,28 +996,26 @@ void prim_parseFlakeRef(
     auto binds = state.ctx.buildBindings(attrs.size());
     for (const auto & [key, value] : attrs) {
         auto s = state.ctx.symbols.create(key);
-        auto & vv = binds.alloc(s);
-        std::visit(
+        Value vv = std::visit(
             overloaded{
-                [&vv](const std::string & value) { vv = {NewValueAs::string, value}; },
-                [&vv](const uint64_t & value) { vv = {NewValueAs::integer, NixInt::Inner(value)}; },
-                [&vv](const Explicit<bool> & value) { vv = {NewValueAs::boolean, value.t}; }
+                [](const std::string & value) -> Value { return {NewValueAs::string, value}; },
+                [](const uint64_t & value) -> Value { return {NewValueAs::integer, NixInt::Inner(value)}; },
+                [](const Explicit<bool> & value) -> Value { return {NewValueAs::boolean, value.t}; }
             },
             value
         );
+        binds.insert(s, vv);
     }
-    v = {NewValueAs::attrs, binds};
+    return {NewValueAs::attrs, binds};
 }
 
-void prim_flakeRefToString(
-    EvalState & state,
-    Value * * args,
-    Value & v)
+Value prim_flakeRefToString(EvalState & state, Value ** args)
 {
     state.forceAttrs(*args[0], noPos,
         "while evaluating the argument passed to builtins.flakeRefToString");
     fetchers::Attrs attrs;
     for (const auto & attr : *args[0]->attrs()) {
+        state.forceValue(attr.value, noPos);
         auto t = attr.value.type();
         if (t == nInt) {
             auto intValue = attr.value.integer().value;
@@ -1051,7 +1042,7 @@ void prim_flakeRefToString(
         }
     }
     auto flakeRef = FlakeRef::fromAttrs(attrs);
-    v = {NewValueAs::string, flakeRef.to_string()};
+    return {NewValueAs::string, flakeRef.to_string()};
 }
 
 }

lix/libexpr/json-to-value.cc

@@ -14,24 +14,39 @@ class JSONSax : nlohmann::json_sax<JSON> {
     class JSONState {
     protected:
         std::unique_ptr<JSONState> parent;
+        JSONState() = default;
+    public:
+        virtual std::unique_ptr<JSONState> resolve(EvalState &) = 0;
+        explicit JSONState(std::unique_ptr<JSONState> && p) : parent(std::move(p)) {}
+        JSONState(JSONState & p) = delete;
+        virtual Value & finalValue()
+        {
+            assert(false && "tried to read a final value from a non-toplevel json parser state");
+        }
+        virtual ~JSONState() {}
+        virtual void addValue(Value v) = 0;
+    };
+
+    class TopLevelJSONState : public JSONState
+    {
         RootValue v;
     public:
-        virtual std::unique_ptr<JSONState> resolve(EvalState &)
+        std::unique_ptr<JSONState> resolve(EvalState &) override
         {
             assert(false && "tried to close toplevel json parser state");
         }
-        explicit JSONState(std::unique_ptr<JSONState> && p) : parent(std::move(p)) {}
-        JSONState() = default;
-        JSONState(JSONState & p) = delete;
-        Value & value()
+        TopLevelJSONState() = default;
+        TopLevelJSONState(TopLevelJSONState & p) = delete;
+        Value & finalValue() override
         {
-            if (!v) {
-                v = allocRootValue({});
-            }
+            assert(v && "tried to read nonexistent final value from json parser");
             return *v;
         }
-        virtual ~JSONState() {}
-        virtual void add() {}
+        void addValue(Value v) override
+        {
+            assert(!this->v && "duplicate value in toplevel JSON scope");
+            this->v = allocRootValue(v);
+        }
     };
 
     class JSONObjectState : public JSONState {
@@ -43,13 +58,12 @@ class JSONSax : nlohmann::json_sax<JSON> {
             auto attrs2 = state.ctx.buildBindings(attrs.size());
             for (auto & i : attrs)
                 attrs2.insert(i.first, i.second);
-            parent->value() = {NewValueAs::attrs, attrs2.alreadySorted()};
+            parent->addValue({NewValueAs::attrs, attrs2.alreadySorted()});
             return std::move(parent);
         }
-        void add() override
+        void addValue(Value v) override
         {
-            attrs.insert_or_assign(_key, value());
-            v = nullptr;
+            attrs.insert_or_assign(_key, v);
         }
     public:
         void key(string_t & name, EvalState & state)
@@ -63,16 +77,15 @@ class JSONSax : nlohmann::json_sax<JSON> {
         std::unique_ptr<JSONState> resolve(EvalState & state) override
         {
             auto list = state.ctx.mem.newList(values.size());
-            parent->value() = {NewValueAs::list, list};
+            parent->addValue({NewValueAs::list, list});
             for (size_t n = 0; n < values.size(); ++n) {
                 list->elems[n] = values[n];
             }
             return std::move(parent);
         }
-        void add() override
+        void addValue(Value v) override
         {
-            values.push_back(*v);
-            v = nullptr;
+            values.push_back(v);
         }
     public:
         JSONListState(std::unique_ptr<JSONState> && p, std::size_t reserve) : JSONState(std::move(p))
@@ -85,31 +98,28 @@ class JSONSax : nlohmann::json_sax<JSON> {
     std::unique_ptr<JSONState> rs;
 
 public:
-    JSONSax(EvalState & state) : state(state), rs(new JSONState()) {};
+    JSONSax(EvalState & state) : state(state), rs(new TopLevelJSONState()) {};
 
     Value result()
     {
-        return rs->value();
+        return rs->finalValue();
     }
 
     bool null() override
     {
-        rs->value().mkNull();
-        rs->add();
+        rs->addValue(Value::VNULL);
         return true;
     }
 
     bool boolean(bool val) override
     {
-        rs->value() = {NewValueAs::boolean, val};
-        rs->add();
+        rs->addValue({NewValueAs::boolean, val});
         return true;
     }
 
     bool number_integer(number_integer_t val) override
     {
-        rs->value() = {NewValueAs::integer, val};
-        rs->add();
+        rs->addValue({NewValueAs::integer, val});
         return true;
     }
 
@@ -121,22 +131,19 @@ public:
             return number_float(static_cast<number_float_t>(val_), "");
         }
         NixInt::Inner val = val_;
-        rs->value() = {NewValueAs::integer, val};
-        rs->add();
+        rs->addValue({NewValueAs::integer, val});
         return true;
     }
 
     bool number_float(number_float_t val, const string_t & s) override
     {
-        rs->value() = {NewValueAs::floating, val};
-        rs->add();
+        rs->addValue({NewValueAs::floating, val});
         return true;
     }
 
     bool string(string_t & val) override
     {
-        rs->value() = {NewValueAs::string, val};
-        rs->add();
+        rs->addValue({NewValueAs::string, val});
         return true;
     }
 
@@ -163,7 +170,6 @@ public:
 
     bool end_object() override {
         rs = rs->resolve(state);
-        rs->add();
         return true;
     }
 
@@ -184,13 +190,13 @@ public:
     }
 };
 
-void parseJSON(EvalState & state, const std::string_view & s_, Value & v)
+Value parseJSON(EvalState & state, const std::string_view & s_)
 {
     JSONSax parser(state);
     bool res = JSON::sax_parse(s_, &parser);
     if (!res)
         throw JSONParseError("Invalid JSON Value");
-    v = parser.result();
+    return parser.result();
 }
 
 }

lix/libexpr/json-to-value.hh

@@ -12,6 +12,5 @@ struct Value;
 
 MakeError(JSONParseError, Error);
 
-void parseJSON(EvalState & state, const std::string_view & s, Value & v);
-
+Value parseJSON(EvalState & state, const std::string_view & s);
 }

lix/libexpr/nixexpr.hh

@@ -130,6 +130,10 @@ public:
     virtual void accept(ExprVisitor & ev, std::unique_ptr<Expr> & ptr) = 0;
     virtual Value eval(EvalState & state, Env & env);
     virtual Value maybeThunk(EvalState & state, Env & env);
+    /* Lambdas have a name associated with them, when they are declared in a binding:
+     * `identity = x: x` will print the resulting value as `«lambda identity @ «string»:1:14»`.
+     * This is set in the parser. After parsing, all expressions are immutable.
+     */
     virtual void setName(Symbol name);
     PosIdx getPos() const { return pos; }
 
@@ -175,7 +179,7 @@ struct ExprLiteral : Expr
 {
 protected:
     Value v;
-    ExprLiteral(const PosIdx pos) : Expr(pos) {};
+    ExprLiteral(const PosIdx pos, Value v) : Expr(pos), v(v) {};
 public:
     Value maybeThunk(EvalState & state, Env & env) override;
     JSON toJSON(const SymbolTable & symbols) const override;
@@ -183,34 +187,46 @@ public:
     void accept(ExprVisitor & ev, std::unique_ptr<Expr> & ptr) override { ev.visit(*this, ptr); }
 };
 
-struct ExprInt : ExprLiteral
+struct ExprInt : private std::tuple<Value::Int>, ExprLiteral
 {
-    Value::Int i;
-    ExprInt(const PosIdx pos, NixInt n) : ExprLiteral(pos), i{{Value::Acb::tInt}, n}
+    ExprInt(const PosIdx pos, NixInt n)
+        : tuple({{Value::Acb::tInt}, n})
+        , ExprLiteral(
+              pos,
+              Value::isTaggableInteger(n) ? Value{NewValueAs::integer, n} : Value(std::get<Value::Int>(*this))
+          )
     {
-        v = Value::isTaggableInteger(n) ? Value{NewValueAs::integer, n} : Value(i);
     }
     ExprInt(const PosIdx pos, NixInt::Inner n) : ExprInt(pos, NixInt(n)) {}
 };
 
-struct ExprFloat : ExprLiteral
+struct ExprFloat : private std::tuple<Value::Float>, ExprLiteral
 {
-    Value::Float f;
     ExprFloat(const PosIdx pos, NewValueAs::floating_t, double f)
-        : ExprLiteral(pos)
-        , f{{Value::Acb::tFloat}, f}
+        : tuple({{Value::Acb::tFloat}, f})
+        , ExprLiteral(pos, Value(std::get<Value::Float>(*this)))
     {
-        v = Value(this->f);
     }
 };
 
-struct ExprString : ExprLiteral
+struct ExprStringBase
 {
     std::unique_ptr<Value::Str, Value::Str::Deleter> contents;
-    Value::String strcb{.content = contents.get(), .context = nullptr};
-    ExprString(const PosIdx pos, std::string s) : ExprLiteral(pos), contents(Value::Str::copy(s))
+    Value::String strcb;
+protected:
+    ExprStringBase(std::string_view s, const char ** context = nullptr)
+        : contents(Value::Str::copy(s))
+        , strcb{.content = contents.get(), .context = context}
+    {
+    }
+};
+
+struct ExprString : private ExprStringBase, ExprLiteral
+{
+    ExprString(const PosIdx pos, std::string s)
+        : ExprStringBase(s)
+        , ExprLiteral(pos, Value{NewValueAs::string, &strcb})
     {
-        v = {NewValueAs::string, &strcb};
     }
 
     std::string_view str() const
@@ -219,11 +235,11 @@ struct ExprString : ExprLiteral
     }
 };
 
-struct ExprPath : ExprLiteral
+struct ExprPath : private ExprStringBase, ExprLiteral
 {
-    std::unique_ptr<Value::Str, Value::Str::Deleter> contents;
-    Value::String strcb{.content = contents.get(), .context = Value::String::path};
-    ExprPath(const PosIdx pos, std::string s) : ExprLiteral(pos), contents(Value::Str::copy(s))
+    ExprPath(const PosIdx pos, std::string s)
+        : ExprStringBase(s, Value::String::path)
+        , ExprLiteral(pos, Value{NewValueAs::path, &strcb})
     {
         v = Value{NewValueAs::path, &strcb};
     }

lix/libexpr/primops.hh

@@ -48,13 +48,12 @@ public:
 /**
  * Load a ValueInitializer from a DSO and return whatever it initializes
  */
-void prim_importNative(EvalState & state, Value * * args, Value & v);
+Value prim_importNative(EvalState & state, Value ** args);
 
 /**
  * Execute a program and parse its output
  */
-void prim_exec(EvalState & state, Value * * args, Value & v);
-
-void makePositionThunks(EvalState & state, const PosIdx pos, Value & line, Value & column);
+Value prim_exec(EvalState & state, Value ** args);
 
+std::tuple<Value, Value> makePositionThunks(EvalState & state, const PosIdx pos);
 }

lix/libexpr/primops/context.cc

@@ -7,22 +7,21 @@
 
 namespace nix {
 
-void prim_unsafeDiscardStringContext(EvalState & state, Value ** args, Value & v)
+Value prim_unsafeDiscardStringContext(EvalState & state, Value ** args)
 {
     NixStringContext context;
     auto s = state.coerceToString(noPos, *args[0], context, "while evaluating the argument passed to builtins.unsafeDiscardStringContext");
-    v = {NewValueAs::string, *s};
+    return {NewValueAs::string, *s};
 }
 
-void prim_hasContext(EvalState & state, Value * * args, Value & v)
+Value prim_hasContext(EvalState & state, Value ** args)
 {
     NixStringContext context;
     state.forceString(*args[0], context, noPos, "while evaluating the argument passed to builtins.hasContext");
-    v = {NewValueAs::boolean, !context.empty()};
+    return {NewValueAs::boolean, !context.empty()};
 }
 
-
-void prim_unsafeDiscardOutputDependency(EvalState & state, Value * * args, Value & v)
+Value prim_unsafeDiscardOutputDependency(EvalState & state, Value ** args)
 {
     NixStringContext context;
     auto s = state.coerceToString(noPos, *args[0], context, "while evaluating the argument passed to builtins.unsafeDiscardOutputDependency");
@@ -39,11 +38,10 @@ void prim_unsafeDiscardOutputDependency(EvalState & state, Value * * args, Value
         }
     }
 
-    v = {NewValueAs::string, *s, context2};
+    return {NewValueAs::string, *s, context2};
 }
 
-
-void prim_addDrvOutputDependencies(EvalState & state, Value * * args, Value & v)
+Value prim_addDrvOutputDependencies(EvalState & state, Value ** args)
 {
     NixStringContext context;
     auto s = state.coerceToString(noPos, *args[0], context, "while evaluating the argument passed to builtins.addDrvOutputDependencies");
@@ -82,7 +80,7 @@ void prim_addDrvOutputDependencies(EvalState & state, Value * * args, Value & v)
         }, context.begin()->raw) }),
     };
 
-    v = {NewValueAs::string, *s, context2};
+    return {NewValueAs::string, *s, context2};
 }
 
 
@@ -105,7 +103,7 @@ void prim_addDrvOutputDependencies(EvalState & state, Value * * args, Value & v)
    Note that for a given path any combination of the above attributes
    may be present.
 */
-void prim_getContext(EvalState & state, Value * * args, Value & v)
+Value prim_getContext(EvalState & state, Value ** args)
 {
     struct ContextInfo {
         bool path = false;
@@ -136,20 +134,19 @@ void prim_getContext(EvalState & state, Value * * args, Value & v)
     for (const auto & info : contextInfos) {
         auto infoAttrs = state.ctx.buildBindings(3);
         if (info.second.path)
-            infoAttrs.alloc(state.ctx.symbols.sym_path) = {NewValueAs::boolean, true};
+            infoAttrs.insert(state.ctx.symbols.sym_path, {NewValueAs::boolean, true});
         if (info.second.allOutputs)
-            infoAttrs.alloc(sAllOutputs) = {NewValueAs::boolean, true};
+            infoAttrs.insert(sAllOutputs, {NewValueAs::boolean, true});
         if (!info.second.outputs.empty()) {
-            auto & outputsVal = infoAttrs.alloc(state.ctx.symbols.sym_outputs);
             auto content = state.ctx.mem.newList(info.second.outputs.size());
-            outputsVal = {NewValueAs::list, content};
+            infoAttrs.insert(state.ctx.symbols.sym_outputs, {NewValueAs::list, content});
             for (const auto & [i, output] : enumerate(info.second.outputs))
                 content->elems[i] = {NewValueAs::string, output};
         }
-        attrs.alloc(state.ctx.store->printStorePath(info.first)) = {NewValueAs::attrs, infoAttrs};
+        attrs.insert(state.ctx.store->printStorePath(info.first), {NewValueAs::attrs, infoAttrs});
     }
 
-    v = {NewValueAs::attrs, attrs};
+    return {NewValueAs::attrs, attrs};
 }
 
 
@@ -158,7 +155,7 @@ void prim_getContext(EvalState & state, Value * * args, Value & v)
    See the commentary above unsafeGetContext for details of the
    context representation.
 */
-void prim_appendContext(EvalState & state, Value ** args, Value & v)
+Value prim_appendContext(EvalState & state, Value ** args)
 {
     NixStringContext context;
     auto orig = state.forceString(*args[0], context, noPos, "while evaluating the first argument passed to builtins.appendContext");
@@ -232,6 +229,6 @@ void prim_appendContext(EvalState & state, Value ** args, Value & v)
         }
     }
 
-    v = {NewValueAs::string, orig, context};
+    return {NewValueAs::string, orig, context};
 }
 }

lix/libexpr/primops/fetchMercurial.cc

@@ -5,7 +5,7 @@
 
 namespace nix {
 
-void prim_fetchMercurial(EvalState & state, Value ** args, Value & v)
+Value prim_fetchMercurial(EvalState & state, Value ** args)
 {
     std::string url;
     std::optional<Hash> rev;
@@ -88,18 +88,18 @@ void prim_fetchMercurial(EvalState & state, Value ** args, Value & v)
     auto [tree, input2] = state.aio.blockOn(input.fetch(state.ctx.store));
 
     auto attrs2 = state.ctx.buildBindings(8);
-    state.ctx.paths.mkStorePathString(tree.storePath, attrs2.alloc(state.ctx.symbols.sym_outPath));
+    attrs2.insert(state.ctx.symbols.sym_outPath, state.ctx.paths.mkStorePathString(tree.storePath));
     if (input2.getRef())
-        attrs2.alloc("branch") = {NewValueAs::string, *input2.getRef()};
+        attrs2.insert("branch", {NewValueAs::string, *input2.getRef()});
     // Backward compatibility: set 'rev' to
     // 0000000000000000000000000000000000000000 for a dirty tree.
     auto rev2 = input2.getRev().value_or(Hash(HashType::SHA1));
-    attrs2.alloc("rev") = {NewValueAs::string, rev2.gitRev()};
-    attrs2.alloc("shortRev") = {NewValueAs::string, rev2.gitRev().substr(0, 12)};
+    attrs2.insert("rev", {NewValueAs::string, rev2.gitRev()});
+    attrs2.insert("shortRev", {NewValueAs::string, rev2.gitRev().substr(0, 12)});
     if (auto revCount = input2.getRevCount())
-        attrs2.alloc("revCount") = {NewValueAs::integer, NixInt::Inner(*revCount)};
-    v = {NewValueAs::attrs, attrs2};
+        attrs2.insert("revCount", {NewValueAs::integer, NixInt::Inner(*revCount)});
 
     state.ctx.paths.allowPath(tree.storePath);
+    return {NewValueAs::attrs, attrs2};
 }
 }

lix/libexpr/primops/fetchTree.cc

@@ -26,49 +26,51 @@ Value emitTreeAttrs(
 
     auto attrs = state.buildBindings(10);
 
-    state.paths.mkStorePathString(tree.storePath, attrs.alloc(state.symbols.sym_outPath));
+    attrs.insert(state.symbols.sym_outPath, state.paths.mkStorePathString(tree.storePath));
 
     // FIXME: support arbitrary input attributes.
 
     auto narHash = input.getNarHash();
     assert(narHash);
-    attrs.alloc("narHash") = {NewValueAs::string, narHash->to_string()};
+    attrs.insert("narHash", {NewValueAs::string, narHash->to_string()});
 
     if (input.getType() == "git")
-        attrs.alloc("submodules") = {
-            NewValueAs::boolean, fetchers::maybeGetBoolAttr(input.attrs, "submodules").value_or(false)
-        };
+        attrs.insert(
+            "submodules",
+            {NewValueAs::boolean, fetchers::maybeGetBoolAttr(input.attrs, "submodules").value_or(false)}
+        );
 
     if (!forceDirty) {
 
         if (auto rev = input.getRev()) {
-            attrs.alloc("rev") = {NewValueAs::string, rev->gitRev()};
-            attrs.alloc("shortRev") = {NewValueAs::string, rev->gitShortRev()};
+            attrs.insert("rev", {NewValueAs::string, rev->gitRev()});
+            attrs.insert("shortRev", {NewValueAs::string, rev->gitShortRev()});
         } else if (emptyRevFallback) {
             // Backwards compat for `builtins.fetchGit`: dirty repos return an empty sha1 as rev
             auto emptyHash = Hash(HashType::SHA1);
-            attrs.alloc("rev") = {NewValueAs::string, emptyHash.gitRev()};
-            attrs.alloc("shortRev") = {NewValueAs::string, emptyHash.gitShortRev()};
+            attrs.insert("rev", {NewValueAs::string, emptyHash.gitRev()});
+            attrs.insert("shortRev", {NewValueAs::string, emptyHash.gitShortRev()});
         }
 
         if (auto revCount = input.getRevCount())
-            attrs.alloc("revCount") = {NewValueAs::integer, NixInt::Inner(*revCount)};
+            attrs.insert("revCount", {NewValueAs::integer, NixInt::Inner(*revCount)});
         else if (emptyRevFallback)
-            attrs.alloc("revCount") = {NewValueAs::integer, 0};
+            attrs.insert("revCount", {NewValueAs::integer, 0});
     }
 
     if (auto dirtyRev = fetchers::maybeGetStrAttr(input.attrs, "dirtyRev")) {
-        attrs.alloc("dirtyRev") = {NewValueAs::string, *dirtyRev};
-        attrs.alloc("dirtyShortRev") = {
-            NewValueAs::string, *fetchers::maybeGetStrAttr(input.attrs, "dirtyShortRev")
-        };
+        attrs.insert("dirtyRev", {NewValueAs::string, *dirtyRev});
+        attrs.insert(
+            "dirtyShortRev", {NewValueAs::string, *fetchers::maybeGetStrAttr(input.attrs, "dirtyShortRev")}
+        );
     }
 
     if (auto lastModified = input.getLastModified()) {
-        attrs.alloc("lastModified") = {NewValueAs::integer, *lastModified};
-        attrs.alloc("lastModifiedDate") = {
-            NewValueAs::string, fmt("%s", std::put_time(std::gmtime(&*lastModified), "%Y%m%d%H%M%S"))
-        };
+        attrs.insert("lastModified", {NewValueAs::integer, *lastModified});
+        attrs.insert(
+            "lastModifiedDate",
+            {NewValueAs::string, fmt("%s", std::put_time(std::gmtime(&*lastModified), "%Y%m%d%H%M%S"))}
+        );
     }
 
     return {NewValueAs::attrs, attrs};
@@ -106,14 +108,14 @@ struct FetchTreeParams {
     bool allowNameArgument = false;
 };
 
-static void fetchTree(
+static Value fetchTree(
     EvalState & state,
     const PosIdx pos,
-    Value * * args,
-    Value & v,
+    Value ** args,
     std::optional<std::string> type,
     const FetchTreeParams & params = FetchTreeParams{}
-) {
+)
+{
     fetchers::Input input;
     NixStringContext context;
 
@@ -223,16 +225,17 @@ static void fetchTree(
 
     state.ctx.paths.allowPath(tree.storePath);
 
-    v = emitTreeAttrs(state.ctx, tree, input2, params.emptyRevFallback, false);
+    return emitTreeAttrs(state.ctx, tree, input2, params.emptyRevFallback, false);
 }
 
-void prim_fetchTree(EvalState & state, Value * * args, Value & v)
+Value prim_fetchTree(EvalState & state, Value ** args)
 {
-    fetchTree(state, noPos, args, v, std::nullopt, FetchTreeParams { .allowNameArgument = false });
+    return fetchTree(state, noPos, args, std::nullopt, FetchTreeParams{.allowNameArgument = false});
 }
 
-static void fetch(EvalState & state, const PosIdx pos, Value * * args, Value & v,
-    const std::string & who, bool unpack, std::string name)
+static Value fetch(
+    EvalState & state, const PosIdx pos, Value ** args, const std::string & who, bool unpack, std::string name
+)
 {
     std::optional<std::string> url;
     std::optional<Hash> expectedHash;
@@ -297,8 +300,7 @@ static void fetch(EvalState & state, const PosIdx pos, Value * * args, Value & v
             });
 
         if (state.aio.blockOn(state.ctx.store->isValidPath(expectedPath))) {
-            state.ctx.paths.allowAndSetStorePathString(expectedPath, v);
-            return;
+            return state.ctx.paths.allowAndSetStorePathString(expectedPath);
         }
     }
 
@@ -329,22 +331,24 @@ static void fetch(EvalState & state, const PosIdx pos, Value * * args, Value & v
         }
     }
 
-    state.ctx.paths.allowAndSetStorePathString(storePath, v);
+    return state.ctx.paths.allowAndSetStorePathString(storePath);
 }
 
-void prim_fetchurl(EvalState & state, Value * * args, Value & v)
+Value prim_fetchurl(EvalState & state, Value ** args)
 {
-    fetch(state, noPos, args, v, "fetchurl", false, "");
+    return fetch(state, noPos, args, "fetchurl", false, "");
 }
 
-void prim_fetchTarball(EvalState & state, Value * * args, Value & v)
+Value prim_fetchTarball(EvalState & state, Value ** args)
 {
-    fetch(state, noPos, args, v, "fetchTarball", true, "source");
+    return fetch(state, noPos, args, "fetchTarball", true, "source");
 }
 
-void prim_fetchGit(EvalState & state, Value * * args, Value & v)
+Value prim_fetchGit(EvalState & state, Value ** args)
 {
-    fetchTree(state, noPos, args, v, "git", FetchTreeParams { .emptyRevFallback = true, .allowNameArgument = true });
+    return fetchTree(
+        state, noPos, args, "git", FetchTreeParams{.emptyRevFallback = true, .allowNameArgument = true}
+    );
 }
 
 }

lix/libexpr/primops/fromTOML.cc

@@ -7,7 +7,7 @@
 
 namespace nix {
 
-void prim_fromTOML(EvalState & state, Value ** args, Value & val)
+Value prim_fromTOML(EvalState & state, Value ** args)
 {
     auto toml = state.forceStringNoCtx(
         *args[0], noPos, "while evaluating the argument passed to builtins.fromTOML"
@@ -15,62 +15,53 @@ void prim_fromTOML(EvalState & state, Value ** args, Value & val)
 
     std::istringstream tomlStream(std::string{toml});
 
-    auto visit = [&](this const auto & self, Value & v, toml::value t) -> void {
+    auto visit = [&](this const auto & self, toml::value t) -> Value {
         switch (t.type()) {
         case toml::value_t::table: {
             auto table = toml::get<toml::table>(t);
             auto attrs = state.ctx.buildBindings(table.size());
 
             for (auto & elem : table) {
-                self(attrs.alloc(elem.first), elem.second);
+                attrs.insert(elem.first, self(elem.second));
             }
 
-            v = {NewValueAs::attrs, attrs};
-        } break;
+            return {NewValueAs::attrs, attrs};
+        }
         case toml::value_t::array: {
             auto array = toml::get<std::vector<toml::value>>(t);
 
             size_t size = array.size();
             auto list = state.ctx.mem.newList(size);
-            v = {NewValueAs::list, list};
             for (size_t i = 0; i < size; ++i) {
-                self(list->elems[i], array[i]);
+                list->elems[i] = self(array[i]);
             }
-        } break;
+            return {NewValueAs::list, list};
+        }
         case toml::value_t::boolean:
-            v = {NewValueAs::boolean, toml::get<bool>(t)};
-            break;
+            return {NewValueAs::boolean, toml::get<bool>(t)};
         case toml::value_t::integer:
-            v = {NewValueAs::integer, toml::get<int64_t>(t)};
-            break;
+            return {NewValueAs::integer, toml::get<int64_t>(t)};
         case toml::value_t::floating:
-            v = {NewValueAs::floating, toml::get<NixFloat>(t)};
-            break;
+            return {NewValueAs::floating, toml::get<NixFloat>(t)};
         case toml::value_t::string:
-            v = {NewValueAs::string, toml::get<std::string>(t)};
-            break;
+            return {NewValueAs::string, toml::get<std::string>(t)};
         case toml::value_t::local_datetime:
         case toml::value_t::offset_datetime:
         case toml::value_t::local_date:
         case toml::value_t::local_time:
             // NOLINTNEXTLINE(lix-foreign-exceptions)
             throw std::runtime_error("Dates and times are not supported");
-            break;
         case toml::value_t::empty:
-            v.mkNull();
-            break;
+            return Value::VNULL;
         }
     };
 
     try {
-        visit(
-            val,
+        return visit(
             toml::parse(
                 tomlStream,
                 "fromTOML", /* the "filename" */
-                toml::spec::v(
-                    1, 0, 0
-                ) // Be explicit that we are parsing TOML 1.0.0 without extensions
+                toml::spec::v(1, 0, 0) // Be explicit that we are parsing TOML 1.0.0 without extensions
             )
         );
     } catch (std::exception & e) { // NOLINT(lix-foreign-exceptions) // TODO: toml::syntax_error

lix/libexpr/settings/repl-overlays.md

@@ -17,10 +17,10 @@ Each file is called with three arguments:
    except that it's available in
    [`pure-eval`](@docroot@/command-ref/conf-file.html#conf-pure-eval)
    mode).
-2. The top-level bindings produced by the previous `repl-overlays`
-   value (or the default top-level bindings).
-3. The final top-level bindings produced by calling all
+2. The final top-level bindings produced by calling all
    `repl-overlays`.
+3. The top-level bindings produced by the previous `repl-overlays`
+   value (or the default top-level bindings).
 
 For example, the following file would alias `pkgs` to
 `legacyPackages.${info.currentSystem}` (if that attribute is defined):

lix/libexpr/value.cc

@@ -12,8 +12,10 @@ namespace nix
 
 static const Value::List emptyListData{.size = 0};
 Value Value::EMPTY_LIST{Value::list_t{}, &emptyListData};
+Value Value::EMPTY_SET{attrs_t{}, &Bindings::EMPTY};
 
 const Value::Null Value::NULL_ACB = {{Value::Acb::tNull}};
+Value Value::VNULL{null_t{}};
 
 static_assert(alignof(Value::String) >= Value::TAG_ALIGN);
 static_assert(alignof(Bindings) >= Value::TAG_ALIGN);

lix/libexpr/value.hh

@@ -32,7 +32,7 @@ struct Value;
 /**
  * Function that implements a primop.
  */
-using PrimOpImpl = void(EvalState & state, Value ** args, Value & v);
+using PrimOpImpl = Value(EvalState & state, Value ** args);
 
 /**
  * Info about a primitive operation, and its implementation
@@ -356,6 +356,8 @@ public:
      * Empty list constant.
      */
     static Value EMPTY_LIST;
+    static Value EMPTY_SET;
+    static Value VNULL;
 
     struct String;
     struct Acb;
@@ -764,16 +766,6 @@ public:
      */
     inline ValueType type(bool invalidIsThunk = false) const;
 
-    inline void mkNull()
-    {
-        *this = {NewValueAs::null};
-    }
-
-    inline void mkExternal(ExternalValueBase * e)
-    {
-        *this = {NewValueAs::external, *e};
-    }
-
     bool isList() const
     {
         return internalType() == tList;

lix/libstore/binary-cache-store.cc

@@ -221,6 +221,7 @@ try {
 
         auto * buildIdDir = std::get_if<nar_index::Directory>(&narIndex);
         for (auto subdir : { "lib", "debug", ".build-id" }) {
+            if (!buildIdDir) break;
             // get returns nullptr subdir does not exist, and std::get_if propagates it.
             buildIdDir = std::get_if<nar_index::Directory>(get(buildIdDir->contents, subdir));
         }

lix/libstore/platform/linux.cc

@@ -1348,7 +1348,10 @@ Pid LinuxLocalDerivationGoal::startChild(AutoCloseFD setupFD, AutoCloseFD logPTY
             (wantUserNS ? CLONE_NEWUSER : 0) | (wantNetNS ? CLONE_NEWNET : 0) | CLONE_VM | CLONE_FILES,
             []() -> int {
                 for (;;) {
-                    raise(SIGSTOP);
+                    // NOTE: musl apparently caches the pid of the process, which fucks with raise().
+                    // we must explicitly use getpid() to bypass this cache instead of using raise; a
+                    // raise(SIGSTOP) would stop the *daemon* process, and this breaks sandbox setup.
+                    kill(getpid(), SIGSTOP);
                 }
             }
         )};

lix/libutil/chunked-vector.hh

@@ -37,7 +37,7 @@ private:
     }
 
 public:
-    ChunkedVector(uint32_t reserve)
+    ChunkedVector(uint32_t reserve = 1)
     {
         chunks.reserve(reserve);
         addChunk();

lix/libutil/linear-map.hh

@@ -39,13 +39,13 @@ private:
     }
 
 public:
-    using typename base::const_iterator, typename base::value_type;
+    using typename base::const_iterator, typename base::const_reverse_iterator, typename base::value_type;
     LinearMap() = default;
     LinearMap(size_t expectedSize)
     {
         reserve(expectedSize);
     }
-    using base::size, base::reserve, base::clear, base::cbegin, base::cend;
+    using base::size, base::reserve, base::clear, base::cbegin, base::cend, base::crbegin, base::crend;
 
     /* Insert an element at the correct position, shifting later elements back by
      * one place. Returns `true` if a previous element with that key was
@@ -148,5 +148,13 @@ public:
     {
         return cend();
     }
+    const_reverse_iterator rbegin() const
+    {
+        return crbegin();
+    }
+    const_reverse_iterator rend() const
+    {
+        return crend();
+    }
 };
 } // namespace nix

lix/lix-doc/meson.build

@@ -20,8 +20,6 @@ lix_doc = static_library(
     rnix,
   ],
   rust_args : [
-    # Empty when default_library=static
-    rust_dynamic_args,
     lix_doc_rust_args,
   ],
 )

lix/nix/repl.md

@@ -83,4 +83,69 @@ On startup, it loads the Nix expressions named *files* and adds them
 into the lexical scope. You can load addition files using the `:l
 <filename>` command, or reload all files using `:r`.
 
+# Adding default variables in REPL sessions
+
+It is possible to automatically load variables from a list of files
+into each new REPL session using the
+[`repl-overlays`](@docroot@/command-ref/conf-file.html#conf-repl-overlays)
+configuration option.
+
+Each file should contain a Nix function taking three taking and
+returning an
+[attribute set](@docroot@/language/values.html#attribute-set).
+These three arguments are:
+
+1. An [attribute set](@docroot@/language/values.html#attribute-set)
+    containing at least a `currentSystem` attribute (this is identical
+    to
+    [`builtins.currentSystem`](@docroot@/language/builtin-constants.md#builtins-currentSystem),
+    except that it's available in
+    [`pure-eval`](@docroot@/command-ref/conf-file.html#conf-pure-eval)
+    mode).
+2. The final top-level bindings produced by calling all
+    `repl-overlays`.
+3. The top-level bindings produced by the previous `repl-overlays`
+    value (or the default top-level bindings).
+
+## Examples
+
+* Aliasing `legacyPackages.${currentSystem}` to `pkgs`
+
+  A file named `/home/alice/my-overlays.nix` containing the following code
+  would, if `legacyPackages` exists, add a variable named `pkgs` into
+  the global REPL scope, which returns the value of
+  `legacyPackages.${currentSystem}`.
+
+  ```nix
+  info: final: prev:
+  if prev ? legacyPackages
+    && prev.legacyPackages ? ${info.currentSystem}
+  then
+  {
+    pkgs = prev.legacyPackages.${info.currentSystem};
+  }
+  else
+  { }
+  ```
+
+  This file can be loaded automatically for every REPL session by
+  adding it to the value of
+  [`repl-overlays`](@docroot@/command-ref/conf-file.html#conf-repl-overlays)
+  inside `nix.conf`. For a single session, it is possible to add it
+  using `--option repl-overlays /home/alice/my-overlay.nix`:
+
+  ```console
+  # nix repl --option repl-overlays /home/alice/my-overlay.nix nixpkgs
+
+  nix-repl> pkgs == legacyPackages.${builtins.currentSystem}
+  true
+
+  nix-repl> pkgs.hello
+  «derivation /nix/store/qdzln99hynf92vrz8sz91hlf1dmb1vdy-hello-2.12.2.drv»
+  ```
+
+See the `nix.conf`
+[`repl-overlays`](@docroot@/command-ref/conf-file.html#conf-repl-overlays)
+documentation for more information.
+
 )""

meson.build

@@ -376,11 +376,11 @@ curl = dependency('libcurl', 'curl', required : true, include_type : 'system')
 
 editline = dependency('libeditline', 'editline', version : '>=1.14', required : true, include_type : 'system')
 
-lowdown = dependency('lowdown', version : '>=0.9.0', required : true, include_type : 'system')
+lowdown = dependency('lowdown', version : '>=1.4.0', required : true, include_type : 'system')
 
-# TODO(sterni): drop the corresponding #ifdef after NixOS 25.05 is EOL which still distributes lowdown < 1.4.0
-if lowdown.version().version_compare('>= 1.4.0')
-  add_project_arguments('-DLOWDOWN_SEPARATE_TERM_OPTS', language: 'cpp')
+# TODO(sterni): drop the corresponding #ifdef after NixOS 25.11 is EOL which still distributes lowdown < 3.0.0
+if lowdown.version().version_compare('>= 3.0.0')
+  add_project_arguments('-DLOWDOWN_CONSOLIDATED_OFLAGS', language: 'cpp')
 endif
 
 # HACK(Qyriad): rapidcheck's pkg-config doesn't include the libs lol
@@ -688,6 +688,11 @@ if cxx.get_id() in ['clang', 'gcc']
     language : 'cpp',
   )
 endif
+add_project_arguments(
+  # likewise for rust
+  '--remap-path-prefix=../lix=lix',
+  language : 'rust',
+)
 
 if is_darwin
   fs.copyfile(
@@ -713,10 +718,24 @@ capnpc_wrapper = custom_target(
   output : 'capnpc_wrapper',
 )
 
+coverage_test_env = {}
+coverage = get_option('coverage')
+if coverage
+  if cxx.get_id() != 'clang'
+    error('-Dcoverage=true is llvm-only')
+  endif
+  # not necessarily just for our own tests, so we don't gate on tests being
+  # enabled
+  subdir('tests/coverage/args')
+endif
+
+
 subdir('lix')
 subdir('scripts')
 subdir('misc')
 
+coverage_objects = [nix, liblix_all]
+
 if enable_docs
   subdir('doc/manual')
 endif
@@ -734,3 +753,9 @@ endif
 subdir('meson/clang-tidy')
 
 subproject('nix-eval-jobs', required : enable_nix_eval_jobs)
+
+if coverage
+  # targets defined by coverage
+  subdir('tests/coverage')
+endif
+

meson.options

@@ -108,3 +108,7 @@ option('disable-fibers', type : 'boolean', value : false,
 option('builtin-dep-closure', type : 'array',
   description : 'dependency closure used for builtin builder sandboxes. the install paths are included automatically.',
 )
+
+option('coverage', type : 'boolean',
+  description : 'Use LLVM\'s source-based coverage while building and testing Lix'
+)

package.nix

@@ -32,7 +32,6 @@
   libcpuid,
   libseccomp,
   libsystemtap,
-  linuxPackages,
   lix-clang-tidy ? null,
   llvmPackages,
   lsof,
@@ -366,7 +365,10 @@ stdenv.mkDerivation (finalAttrs: {
       "-Dc_link_args=-fuse-ld=lld"
       "-Dcpp_link_args=-fuse-ld=lld"
     ]
-    ++ lib.optional hostPlatform.isStatic "-Denable-embedded-sandbox-shell=true"
+    ++ lib.optionals hostPlatform.isStatic [
+      "-Denable-embedded-sandbox-shell=true"
+      "-Denable-contrib-plugins=false"
+    ]
     ++ lib.optional ciBuildAndDeleteBothLibraries "-Ddefault_library=both"
     # musl doesn't support fibers, and we can't detect this with meson alone.
     ++ lib.optional hostPlatform.isMusl "-Ddisable-fibers=true"

tests/coverage/args/meson.build

@@ -0,0 +1,36 @@
+# Early initialization of LLVM source-based coverage.
+#
+# Needs to run before defining any targets.
+
+add_project_arguments(
+  '-fprofile-instr-generate',
+  '-fcoverage-mapping',
+  # TODO: -mllvm -runtime-counter-relocation may fix problems with tests with
+  # nix run/fmt/etc that execvp, bypassing atexit. need to confirm that's real.
+  language : 'cpp',
+)
+# N.B. This is a link argument because it needs to link the LLVM profiling runtime, I believe.
+add_project_link_arguments(
+  '-fprofile-instr-generate',
+  language: 'cpp',
+)
+
+add_project_arguments(
+  '-Cinstrument-coverage',
+  language : 'rust',
+)
+
+coverage_profraw_dir = meson.project_build_root() / 'profraw'
+run_command('mkdir', '-p', coverage_profraw_dir, check : true)
+coverage_test_env = {
+  # TODO: may need %c, but that may only work if you set runtime counter relocation, need to find that out
+  #
+  # See: https://clang.llvm.org/docs/SourceBasedCodeCoverage.html#running-the-instrumented-program
+  # %20m -> use 20 raw profiles and merge at runtime, so that our large number of
+  # invocations of lix in the test suite don't create unreasonable numbers of
+  # files. This, I think, also limits the test concurrency.
+  'LLVM_PROFILE_FILE': coverage_profraw_dir / '%20m.profraw'
+}
+
+llvm_profdata = find_program('llvm-profdata', required : true)
+llvm_cov = find_program('llvm-cov', required : true)

tests/coverage/coverage-report.py

@@ -0,0 +1,135 @@
+"""
+Generates a report of code coverage using llvm's line-based coverage tool.
+
+This merges together/indexes all the profraw files: https://clang.llvm.org/docs/SourceBasedCodeCoverage.html#creating-coverage-reports
+
+Then it generates reports.
+TODO(review): should this maybe be two separate things? idk!
+"""
+
+import sys
+from pathlib import Path
+from dataclasses import dataclass
+import glob
+import tempfile
+import subprocess
+import shlex
+import logging
+
+log = logging.getLogger(__name__)
+log.setLevel(logging.INFO)
+fmt = logging.Formatter(
+    "{asctime} {levelname} {name}: {message}", datefmt="%b %d %H:%M:%S", style="{"
+)
+hand = logging.StreamHandler()
+hand.setFormatter(fmt)
+log.addHandler(hand)
+
+
+def run(args, *rest, check: bool = True, **kwargs):
+    # stringify all path values
+    args = [str(a) for a in args]
+
+    logging.info("Run: %s", shlex.join(args))
+    return subprocess.run(args, *rest, check=check, **kwargs)
+
+
+@dataclass
+class LLVMCovArgs:
+    llvm_cov: Path
+    objects: list[str]
+    source_root: Path
+    profdata: Path
+
+    def to_opts(self) -> list[str]:
+        args = [self.objects[0]]
+        for obj in self.objects:
+            args.extend(["-object", obj])
+
+        args.extend([f"-compilation-dir={self.source_root}", f"-instr-profile={self.profdata}"])
+
+        return args
+
+    def show_html(self, out_dir: Path, *args):
+        run(
+            [
+                self.llvm_cov,
+                "show",
+                *self.to_opts(),
+                "-format=html",
+                f"-output-dir={out_dir}",
+                # by default, it doesn't show coverage of particular
+                # instantiations of template functions, but let's turn it on for
+                # fun!
+                "-show-instantiation-summary",
+                *args,
+            ]
+        )
+
+    def export_lcov(self, out_file: Path, *args):
+        with out_file.open("w") as h:
+            run([self.llvm_cov, "export", *self.to_opts(), "-format=lcov"], stdout=h)
+
+
+def main() -> int:
+    import argparse
+
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument(
+        "--profraw-dir", type=Path, help="Directory of the .profraw files to ingest"
+    )
+    parser.add_argument(
+        "--out-dir", type=Path, help="Output directory for reports and intermediates (indexes)"
+    )
+    parser.add_argument(
+        "--source-root", type=Path, help="Source root, given to llvm-cov as -compilation-dir"
+    )
+    parser.add_argument("--llvm-cov", type=Path, help="llvm-cov executable")
+    parser.add_argument("--llvm-profdata", type=Path, help="llvm-profdata executable")
+    parser.add_argument("objects", nargs="+")
+    args = parser.parse_args()
+
+    profraw_dir: Path = args.profraw_dir
+    out_dir: Path = args.out_dir
+    source_root: Path = args.source_root
+    llvm_profdata: Path = args.llvm_profdata
+    llvm_cov: Path = args.llvm_cov
+    objects: list[str] = args.objects
+
+    profraw_dir.mkdir(parents=True, exist_ok=True)
+    out_dir.mkdir(parents=True, exist_ok=True)
+
+    all_profraws = sorted(glob.glob(str(profraw_dir / "**/*.profraw"), recursive=True))
+
+    with tempfile.TemporaryDirectory() as tmp:
+        tmp = Path(tmp)
+
+        inputs_file = tmp / "inputs"
+        inputs_file.write_text("\n".join(all_profraws) + "\n")
+
+        profdata = out_dir / "merged.profdata"
+
+        run(
+            [
+                llvm_profdata,
+                "merge",
+                # ostensibly faster or smaller according to LLVM docs
+                "-sparse",
+                # TODO: '--failure-mode=warn' ?
+                "-o",
+                profdata,
+                f"--input-files={inputs_file}",
+            ]
+        )
+
+        llvm_cov_args = LLVMCovArgs(
+            llvm_cov=llvm_cov, objects=objects, source_root=source_root, profdata=profdata
+        )
+        llvm_cov_args.show_html(out_dir=out_dir)
+        llvm_cov_args.export_lcov(out_file=out_dir / "coverage.lcov")
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

tests/coverage/meson.build

@@ -0,0 +1,12 @@
+# LLVM source-based coverage for Lix
+run_target('coverage-report',
+  command : [
+    python, meson.project_source_root() / 'tests/coverage/coverage-report.py',
+    '--profraw-dir', coverage_profraw_dir,
+    '--out-dir', meson.project_build_root() / 'meson-logs/coverage',
+    '--source-root', meson.project_source_root(),
+    '--llvm-profdata', llvm_profdata.full_path(),
+    '--llvm-cov', llvm_cov.full_path(),
+    coverage_objects,
+  ],
+)

tests/functional/meson.build

@@ -57,7 +57,6 @@ functional_tests_scripts = [
   'fetchurl.sh',
   'fetchPath.sh',
   'fetchTree-file.sh',
-  'simple.sh',
   'referrers.sh',
   'substitute-with-invalid-ca.sh',
   'signing.sh',
@@ -155,7 +154,7 @@ foreach script : functional_tests_scripts
     suite : 'installcheck',
     env : {
       'MESON_BUILD_ROOT': meson.project_build_root(),
-    },
+    } + coverage_test_env,
     # some tests take 15+ seconds even on an otherwise idle machine, on a loaded machine
     # this can easily drive them to failure. give them more time, 5min rather than 30sec
     timeout : 300,

tests/functional/plugins/plugintest.cc

@@ -25,13 +25,13 @@ static void maybeRequireMeowForDlopen() {
     meow();
 }
 
-static void prim_anotherNull (EvalState & state, Value ** args, Value & v)
+static Value prim_anotherNull(EvalState & state, Value ** args)
 {
     assert(entryCalled);
     if (mySettings.settingSet)
-        v.mkNull();
+        return Value::VNULL;
     else
-        v = {NewValueAs::boolean, false};
+        return {NewValueAs::boolean, false};
 }
 
 extern "C" void nix_plugin_entry()

tests/functional/repl_characterization/data/debug_frames.test

@@ -40,7 +40,7 @@
     :bt, :backtrace                       Show trace stack
     :c, :continue                         Go until end of program, exception or builtins.break
     :s, :step                             Go one step
-    :st, :show-trace [integer index]      Show current trace. If an integer is provided, this switches to that stack beforehand.
+    :st, :show-trace [integer index]      Show current trace. If an integer is provided, this switches to that stack beforehand. If the integer has an explicit + or - sign, it is treated as relative to the current stack index.
 
       Flakes commands
 

tests/functional/repl_characterization/data/regression_9917.test

@@ -14,26 +14,56 @@ This test ensures that continues don't skip opportunities to enter the debugger.
   0: error: breakpoint reached
   $TEST_DATA/regression_9917.nix:3:5
 
-       2|   a = builtins.trace "before inner break" (
-       3|     builtins.break { msg = "hello"; }
-        |     ^
-       4|   );
+      2|   a = builtins.trace "before inner break" (
+      3|     builtins.break { msg = "hello"; }
+      |     ^
+      4|   );
 
   1: while calling a function
   $TEST_DATA/regression_9917.nix:3:5
 
-       2|   a = builtins.trace "before inner break" (
-       3|     builtins.break { msg = "hello"; }
-        |     ^
-       4|   );
+      2|   a = builtins.trace "before inner break" (
+      3|     builtins.break { msg = "hello"; }
+      |     ^
+      4|   );
 
   2: while calling a function
   $TEST_DATA/regression_9917.nix:2:7
 
-       1| let
-       2|   a = builtins.trace "before inner break" (
-        |       ^
-       3|     builtins.break { msg = "hello"; }
+      1| let
+      2|   a = builtins.trace "before inner break" (
+      |       ^
+      3|     builtins.break { msg = "hello"; }
+
+  3: while calling a function
+  $TEST_DATA/regression_9917.nix:6:5
+
+      5|   b = builtins.trace "before outer break" (
+      6|     builtins.break a
+      |     ^
+      7|   );
+
+  4: while calling a function
+  $TEST_DATA/regression_9917.nix:5:7
+
+      4|   );
+      5|   b = builtins.trace "before outer break" (
+      |       ^
+      6|     builtins.break a
+
+  5: while evaluating a 'let' expression
+  $TEST_DATA/regression_9917.nix:1:1
+
+      1| let
+      | ^
+      2|   a = builtins.trace "before inner break" (
+
+  6: while evaluating the file '$TEST_DATA/regression_9917.nix':
+  $TEST_DATA/regression_9917.nix:1:1
+
+      1| let
+      | ^
+      2|   a = builtins.trace "before inner break" (
 
   nix-repl> :c
 

tests/functional/repl_characterization/data/stack_vars.test

@@ -87,18 +87,22 @@ If we :st past the frame in the backtrace with the meow in it, the meow should n
 
   nix-repl> :quit
   error:
-         … while calling the 'trace' builtin
-           at $TEST_DATA/stack_vars.nix:2:7:
-              1| let
-              2|   a = builtins.trace "before inner break" (
-               |       ^
-              3|     let meow' = 3; in builtins.break { msg = "hello"; }
+         … while evaluating the file '$TEST_DATA/stack_vars.nix':
 
-         … while calling the 'break' builtin
-           at $TEST_DATA/stack_vars.nix:3:23:
-              2|   a = builtins.trace "before inner break" (
-              3|     let meow' = 3; in builtins.break { msg = "hello"; }
-               |                       ^
+         … while evaluating b
+           at $TEST_DATA/stack_vars.nix:9:3:
+              8| in
+              9|   b
+               |   ^
+             10|
+
+         … while calling the 'trace' builtin
+           at $TEST_DATA/stack_vars.nix:5:7:
               4|   );
+              5|   b = builtins.trace "before outer break" (
+               |       ^
+              6|     let meow = 2; in builtins.break a
+
+         (stack trace truncated; use '--show-trace' to show the full trace)
 
          error: breakpoint reached

tests/functional/repl_characterization/data/stacktrace_invalid_arg.test

@@ -0,0 +1,73 @@
+@args --debugger
+  nix-repl> throw "(forever?????????)"
+  error: (forever?????????)
+
+argument-less :st works fine
+  nix-repl> :st
+
+  0: error: (forever?????????)
+  «string»:1:1
+
+       1| throw "(forever?????????)"
+        | ^
+
+  Env level 0
+  static:
+
+  Env level 1
+  abort baseNameOf break builtins derivation derivationStrict dirOf false fetchGit fetchMercurial fetchTarball fetchTree fromTOML import isNull map null placeholder removeAttrs scopedImport throw toString true
+
+a non-numeric strings produces an error
+  nix-repl> :st chat
+  error: argument 'chat' is not a valid integer
+
+  nix-repl> :st bedroom community
+  error: argument 'bedroom community' is not a valid integer
+
+...even when they start with a digit
+  nix-repl> :st 6up
+  error: argument '6up' is not a valid integer
+
+...or when they're floats
+  nix-repl> :st 4.50
+  error: argument '4.50' is not a valid integer
+
+an integer outside the range produces an error
+  nix-repl> :st 23571113171923
+  error: argument '23571113171923' is not a valid integer
+
+argument-less :st is still at the same index after errors
+  nix-repl> :st 1
+
+  1: while calling a function
+  «string»:1:1
+
+       1| throw "(forever?????????)"
+        | ^
+
+  Env level 0
+  static:
+
+  Env level 1
+  abort baseNameOf break builtins derivation derivationStrict dirOf false fetchGit fetchMercurial fetchTarball fetchTree fromTOML import isNull map null placeholder removeAttrs scopedImport throw toString true
+
+  nix-repl> :st foo
+  error: argument 'foo' is not a valid integer
+
+  nix-repl> :st
+
+  1: while calling a function
+  «string»:1:1
+
+       1| throw "(forever?????????)"
+        | ^
+
+  Env level 0
+  static:
+
+  Env level 1
+  abort baseNameOf break builtins derivation derivationStrict dirOf false fetchGit fetchMercurial fetchTarball fetchTree fromTOML import isNull map null placeholder removeAttrs scopedImport throw toString true
+
+quit
+  nix-repl> :quit
+  error: (forever?????????)

tests/functional/repl_characterization/data/stacktrace_oob.test

@@ -0,0 +1,110 @@
+@args --debugger
+  nix-repl> let f = _: throw "x_x"; x = f 5; in x
+  error: x_x
+
+frames from 0 up to 4 work fine
+  nix-repl> :st 0
+
+  0: error: x_x
+  «string»:1:12
+
+       1| let f = _: throw "x_x"; x = f 5; in x
+        |            ^
+
+  Env level 0
+  static: _
+
+  Env level 1
+  static: f x
+
+  Env level 2
+  static:
+
+  Env level 3
+  abort baseNameOf break builtins derivation derivationStrict dirOf false fetchGit fetchMercurial fetchTarball fetchTree fromTOML import isNull map null placeholder removeAttrs scopedImport throw toString true
+
+  nix-repl> :st 4
+
+  4: while evaluating a 'let' expression
+  «string»:1:1
+
+       1| let f = _: throw "x_x"; x = f 5; in x
+        | ^
+
+  Env level 0
+  static: f x
+
+  Env level 1
+  static:
+
+  Env level 2
+  abort baseNameOf break builtins derivation derivationStrict dirOf false fetchGit fetchMercurial fetchTarball fetchTree fromTOML import isNull map null placeholder removeAttrs scopedImport throw toString true
+
+absolute frames out of bounds print an error
+  nix-repl> :st 5
+  error: stack index must be between 0 and 4, but was 5
+
+argument-less :st is still at the same after absolute oob
+  nix-repl> :st
+
+  4: while evaluating a 'let' expression
+  «string»:1:1
+
+       1| let f = _: throw "x_x"; x = f 5; in x
+        | ^
+
+  Env level 0
+  static: f x
+
+  Env level 1
+  static:
+
+  Env level 2
+  abort baseNameOf break builtins derivation derivationStrict dirOf false fetchGit fetchMercurial fetchTarball fetchTree fromTOML import isNull map null placeholder removeAttrs scopedImport throw toString true
+
+positive relative frames oob clamp to upper bound and print a warning
+  nix-repl> :st +5
+
+  4: while evaluating a 'let' expression
+  «string»:1:1
+
+       1| let f = _: throw "x_x"; x = f 5; in x
+        | ^
+
+  Env level 0
+  static: f x
+
+  Env level 1
+  static:
+
+  Env level 2
+  abort baseNameOf break builtins derivation derivationStrict dirOf false fetchGit fetchMercurial fetchTarball fetchTree fromTOML import isNull map null placeholder removeAttrs scopedImport throw toString true
+
+  warning: stopped at stack frame 4, cannot go any higher
+
+negative relative frames oob clamp to lower bound and print a warning
+  nix-repl> :st -5
+
+  0: error: x_x
+  «string»:1:12
+
+       1| let f = _: throw "x_x"; x = f 5; in x
+        |            ^
+
+  Env level 0
+  static: _
+
+  Env level 1
+  static: f x
+
+  Env level 2
+  static:
+
+  Env level 3
+  abort baseNameOf break builtins derivation derivationStrict dirOf false fetchGit fetchMercurial fetchTarball fetchTree fromTOML import isNull map null placeholder removeAttrs scopedImport throw toString true
+
+  warning: stopped at stack frame 0, cannot go any deeper
+
+quit
+  nix-repl> :quit
+  error: x_x

tests/functional/repl_characterization/data/stacktrace_relative.test

@@ -0,0 +1,64 @@
+@args --debugger
+  nix-repl> let f = _: throw "x_x"; x = f 5; in x
+  error: x_x
+
+absolute indices still work:
+  nix-repl> :st 1
+
+  1: while calling a function
+  «string»:1:12
+
+      1| let f = _: throw "x_x"; x = f 5; in x
+        |            ^
+
+  Env level 0
+  static: _
+
+  Env level 1
+  static: f x
+
+  Env level 2
+  static:
+
+  Env level 3
+  abort baseNameOf break builtins derivation derivationStrict dirOf false fetchGit fetchMercurial fetchTarball fetchTree fromTOML import isNull map null placeholder removeAttrs scopedImport throw toString true
+
+index with + goes up the stack relative to current (1 in this case):
+  nix-repl> :st +3
+
+  4: while evaluating a 'let' expression
+  «string»:1:1
+
+      1| let f = _: throw "x_x"; x = f 5; in x
+        | ^
+
+  Env level 0
+  static: f x
+
+  Env level 1
+  static:
+
+  Env level 2
+  abort baseNameOf break builtins derivation derivationStrict dirOf false fetchGit fetchMercurial fetchTarball fetchTree fromTOML import isNull map null placeholder removeAttrs scopedImport throw toString true
+
+index with - goes down and is also relative to current (4):
+  nix-repl> :st -1
+
+  3: while calling a function
+  «string»:1:29
+
+      1| let f = _: throw "x_x"; x = f 5; in x
+        |                             ^
+
+  Env level 0
+  static: f x
+
+  Env level 1
+  static:
+
+  Env level 2
+  abort baseNameOf break builtins derivation derivationStrict dirOf false fetchGit fetchMercurial fetchTarball fetchTree fromTOML import isNull map null placeholder removeAttrs scopedImport throw toString true
+
+quit
+  nix-repl> :quit
+  error: x_x

tests/functional/repl_characterization/meson.build

@@ -36,6 +36,6 @@ test(
   env : {
     '_NIX_TEST_UNIT_DATA': meson.current_build_dir() / 'data',
     'MESON_BUILD_ROOT': meson.project_build_root(),
-  },
+  } + coverage_test_env,
   suite : 'installcheck',
 )

tests/functional/repl_characterization/repl_characterization.cc

@@ -197,5 +197,8 @@ REPL_TEST(idempotent);
 REPL_TEST(debug_frames);
 REPL_TEST(debug_ignore_try);
 REPL_TEST(debug_ignore_try_defaults);
+REPL_TEST(stacktrace_invalid_arg);
+REPL_TEST(stacktrace_oob);
+REPL_TEST(stacktrace_relative);
 
 }; // namespace nix

tests/functional/simple.sh

@@ -1,33 +0,0 @@
-source common.sh
-
-drvPath=$(nix-instantiate simple.nix)
-
-test "$(nix-store -q --binding system "$drvPath")" = "$system"
-
-echo "derivation is $drvPath"
-
-outPath=$(nix-store -rvv "$drvPath")
-
-echo "output path is $outPath"
-
-(! [ -w $outPath ])
-
-text=$(cat "$outPath"/hello)
-if test "$text" != "Hello World!"; then exit 1; fi
-
-# Directed delete: $outPath is not reachable from a root, so it should
-# be deleteable.
-nix-store --delete $outPath
-(! [ -e $outPath/hello ])
-
-outPath="$(NIX_REMOTE=local?store=/foo\&real=$TEST_ROOT/real-store nix-instantiate --readonly-mode hash-check.nix)"
-if test "$outPath" != "/foo/lfy1s6ca46rm5r6w4gg9hc0axiakjcnm-dependencies.drv"; then
-    echo "hashDerivationModulo appears broken, got $outPath"
-    exit 1
-fi
-
-outPath="$(NIX_REMOTE=local?store=/foo\&real=$TEST_ROOT/real-store nix-instantiate --readonly-mode big-derivation-attr.nix)"
-if test "$outPath" != "/foo/xxiwa5zlaajv6xdjynf9yym9g319d6mn-big-derivation-attr.drv"; then
-    echo "big-derivation-attr.nix hash appears broken, got $outPath. Memory corruption in large drv attr?"
-    exit 1
-fi

tests/functional2/daemon/test_trust.py

@@ -1,7 +1,7 @@
 import pytest
 import json
 
-from testlib.fixtures.nix import Nix
+from testlib.fixtures.nix import Nix, NixDaemon
 
 
 @pytest.mark.parametrize(
@@ -13,9 +13,9 @@ from testlib.fixtures.nix import Nix
         (["*"], ["--force-untrusted"], False),
     ],
 )
-def test_trust(nix: Nix, trusted: list[str], flags: list[str], expected: bool):
+def test_trust(nix: Nix, daemon: NixDaemon, trusted: list[str], flags: list[str], expected: bool):
     nix.settings.add_xp_feature("nix-command", "daemon-trust-override")
 
-    with nix.daemon(flags, settings={"trusted-users": trusted}) as inner:
+    with daemon(nix, flags, settings={"trusted-users": trusted}) as inner:
         trusted = json.loads(inner.nix(["store", "ping", "--json"]).run().ok().stdout)
         assert trusted["trusted"] == expected

tests/functional2/eval/test_attr_paths.py

@@ -23,12 +23,15 @@ ERR_CASES: list[ShouldError] = [
         "1",
         """error: the expression selected by the selection path '1' should be a list but is a set: { }""",
     ),
-    ShouldError("{}", ".", """error: empty attribute name in selection path '.'"""),
     ShouldError(
-        '{ x."" = 2; }', 'x.""', """error: empty attribute name in selection path 'x.""'"""
+        "{}",
+        ".",
+        """error: Leading dot in attribute selection path '.' is not allowed! If the attribute name is an empty string, use '\"\".foo.bar'""",
     ),
     ShouldError(
-        '{ x."".y = 2; }', 'x."".y', """error: empty attribute name in selection path 'x."".y'"""
+        "{}",
+        "bla..blub",
+        """error: consecutive dots not allowed in selection path 'bla..blub', use 'foo."".bar' to denote an empty attribute name""",
     ),
     ShouldError(
         "[]", "1", """error: list index 1 in selection path '1' is out of range for list [ ]"""

tests/functional2/eval/test_autoargs_cli.py

@@ -0,0 +1,110 @@
+from typing import Any
+
+import pytest
+
+from testlib.fixtures.nix import Nix
+
+
+def do_evaluate(nix: Nix, args: list[str], expect_success: bool = True) -> dict[str, Any] | str:
+    res = (
+        nix.nix_instantiate(
+            [
+                "--eval",
+                "--json",
+                "-E",
+                "{ arg1, arg2 ? null }: { inherit arg1 arg2; }",
+                "--strict",
+                *args,
+            ]
+        )
+        .run()
+        .expect(0 if expect_success else 1)
+    )
+
+    if expect_success:
+        return res.json()
+    return res.stderr_s
+
+
+def test_trivial(nix: Nix):
+    res = do_evaluate(nix, args=["--arg", "arg1", "[ 1 2 3 ]", "--arg", "arg2", "1"])
+
+    assert res["arg1"] == [1, 2, 3]
+    assert res["arg2"] == 1
+
+
+def test_recursive(nix: Nix):
+    res = do_evaluate(
+        nix,
+        args=["--arg", "arg1.foo.bar", "1", "--arg", "arg1.foo.baz", "2", "--arg", "arg1.bar", "3"],
+    )
+
+    assert res["arg1"] == {"foo": {"bar": 1, "baz": 2}, "bar": 3}
+    assert res["arg2"] is None
+
+
+@pytest.mark.parametrize(
+    ("attribute_path", "expected"),
+    [("arg1", 2), ("arg1.foo", {"foo": 2}), ("arg1.foo.bar", {"foo": {"bar": 2}})],
+)
+def test_override(nix: Nix, attribute_path: str, expected: Any):
+    res = do_evaluate(nix, args=["--arg", attribute_path, "1", "--arg", attribute_path, "2"])
+
+    assert res["arg1"] == expected
+
+
+def test_quoting(nix: Nix):
+    res = do_evaluate(
+        nix,
+        args=[
+            "--arg",
+            "arg1.foo.bar",
+            "1",
+            "--arg",
+            'arg1."foo bar baz".baz',
+            "2",
+            "--arg",
+            "arg1.bar",
+            "2",
+        ],
+    )
+
+    assert res["arg1"] == {"foo": {"bar": 1}, "bar": 2, "foo bar baz": {"baz": 2}}
+    assert res["arg2"] is None
+
+
+def test_quoting_error(nix: Nix):
+    res = do_evaluate(nix, ["--arg", 'arg1."foo bar.baz', "1"], expect_success=False)
+    assert "error: missing closing quote in selection path 'arg1.\"foo bar.baz'" in res
+
+
+def test_trailing_dot(nix: Nix):
+    # This is what `parseAttrPath` from `libutil` does and is consistent with the selection path
+    # passed to e.g. `nix-build -A`.
+    res = do_evaluate(nix, args=["--arg", "arg1.bar.", "[ 1 2 3 ]"], expect_success=False)
+
+    assert (
+        "error: Trailing dot on the right-hand side of path expr 'arg1.bar.' is not allowed!" in res
+    )
+
+
+@pytest.mark.parametrize(
+    "args",
+    [
+        ["--arg", "arg1.foo", "1", "--arg", "arg1.foo.bar", "2"],
+        ["--arg", "arg1.foo.bar", "2", "--arg", "arg1.foo", "1"],
+    ],
+)
+def test_conflict(nix: Nix, args: list[str]):
+    res = do_evaluate(nix, args, expect_success=False)
+
+    assert (
+        "error: Cannot set arg1.foo.bar via --arg/--argstr when it's the path-extension of another auto-argument!"
+        in res
+    )
+
+
+@pytest.mark.parametrize("selection", ["foo..bar", "foo.bar.."])
+def test_no_empty_items(nix: Nix, selection: str):
+    res = do_evaluate(nix, ["--arg", selection, "1"], expect_success=False)
+    assert f"error: consecutive dots not allowed in selection path '{selection}'" in res

tests/functional2/eval/test_debugger.py

@@ -27,3 +27,59 @@ def test_debugger_output(nix: Nix):
     assert "error: oh snap" in res.stderr_plain
     assert re.findall(r"with: .*a", res.stdout_plain)
     assert re.findall(r"static: .*x", res.stdout_plain)
+
+
+def test_transparent_break(nix: Nix):
+    """
+    Make sure that adding a call to builtins.break doesn't
+    change the result of an expression
+    """
+
+    expr = dedent("""
+        let
+          inherit (builtins)
+            attrNames
+            break
+            elem
+            functionArgs
+            head
+            isAttrs
+            isPath
+            isFunction
+            map
+            mapAttrs
+            removeAttrs
+            toJSON
+            typeOf;
+        in
+        builtins.all (b: b) [
+            ((attrNames { a = 5; }) == (attrNames (break { a = 5; })))
+            ((elem 5 [1 5]) == (elem 5 (break [1 5])))
+            ((elem (2+3) [1 (2+3)]) == (elem (2+3) (break [1 (2+3)])))
+            ((functionArgs ({ a }: 5)) == (functionArgs (break ({ a }: 5))))
+            ((head [1 2]) == (head (break [1 2])))
+            ((isAttrs { a = 5; }) == (isAttrs (break { a = 5; })))
+            ((isPath ./.) == (isPath (break ./.)))
+            ((isPath ./${".meow"}) == (isPath (break ./${".meow"})))
+            ((isFunction (x: x)) == (isFunction (break (x: x))))
+            ((map (x: x) [1 5]) == (map (x: x) (break [1 5])))
+            ((mapAttrs (n: v: v) { a = 5; }) == (mapAttrs (n: v: v) (break { a = 5; })))
+            ((removeAttrs { a = 5; b = 6; } ["a"]) == (removeAttrs (break { a = 5; b = 6; }) ["a"]))
+            ((removeAttrs { ab = 5; } [("a"+"b")]) == (removeAttrs { ab = 5; } [(break ("a"+"b"))]))
+            ((toJSON { a = 5; }) == (toJSON (break { a = 5; })))
+            ((toJSON { a = [(1+2)]; }) == (toJSON { a = break [(1+2)]; }))
+            ((typeOf { a = 5; }) == (typeOf (break { a = 5; })))
+            ((typeOf (1+2)) == (typeOf (break (1+2))))
+        ]
+    """)
+
+    res_no_dbg = nix.nix(["eval", "--expr", expr], flake=True).run().expect(0)
+    assert "true" in res_no_dbg.stdout_plain
+
+    res_with_dbg = (
+        nix.nix(["eval", "--debugger", "--expr", expr], flake=True)
+        .with_stdin(b":c\n" * 50)
+        .run()
+        .expect(0)
+    )
+    assert "true" in res_with_dbg.stdout_plain

tests/functional2/lang/builtins.flakeRefToString/eval-fail-negative-integer.err.exp

@@ -1,14 +1,9 @@
 error:
-       … while calling the 'seq' builtin
-         at /pwd/in.nix:1:16:
-            1| let n = -1; in builtins.seq n (builtins.flakeRefToString {
-             |                ^
-            2|   type  = "github";
-
        … while calling the 'flakeRefToString' builtin
-         at /pwd/in.nix:1:32:
-            1| let n = -1; in builtins.seq n (builtins.flakeRefToString {
-             |                                ^
-            2|   type  = "github";
+         at /pwd/in.nix:4:1:
+            3| in
+            4| builtins.flakeRefToString {
+             | ^
+            5|   type  = "github";
 
        error: negative value given for flake ref attr repo: -1

tests/functional2/lang/builtins.flakeRefToString/in-negative-integer.nix

@@ -1,7 +1,10 @@
-let n = -1; in builtins.seq n (builtins.flakeRefToString {
+let
+  n = -1;
+in
+builtins.flakeRefToString {
   type  = "github";
   owner = "NixOS";
   repo  = n;
   ref   = "23.05";
   dir   = "lib";
-})
+}

tests/functional2/lang/builtins.fromJSON/eval-okay-duplicate-keys.out.exp

@@ -0,0 +1 @@
+{ key = "three"; }

tests/functional2/lang/builtins.fromJSON/in-duplicate-keys.nix

@@ -0,0 +1,8 @@
+# Duplicate JSON keys should always drop all but the latest value
+builtins.fromJSON ''
+  {
+    "key": "one",
+    "key": "two",
+    "key": "three"
+  }
+''

tests/functional2/lang/search-path/eval-okay-prefixed.err.exp

@@ -1,3 +1,3 @@
-warning: The prefix 'nix' is reserved for internal use by Lix in the Nix search path, its usage is deprecated and will be forbidden in the future.
-         Use --extra-deprecated-features nix-path-shadow to silence this warning.
-         This is due to adding 'nix=nix-shadow' in the Nix search path, either through the environment variable 'NIX_PATH' or by passing the flag -I to the nix invocation.
+error: The prefix 'nix' is reserved for internal use by Lix in the Nix search path, its usage is deprecated and will be forbidden in the future.
+       Use --extra-deprecated-features nix-path-shadow to silence this error.
+       This is due to adding 'nix=nix-shadow' in the Nix search path, either through the environment variable 'NIX_PATH' or by passing the flag -I to the nix invocation.

tests/functional2/lang/search-path/eval-okay-prefixless.err.exp

@@ -1,3 +1,3 @@
-warning: Shadowing '<nix/...>' by configuring the nix-path is deprecated and will be forbidden in the future.
-         Use --extra-deprecated-features nix-path-shadow to silence this warning.
-         This is due to adding 'nix-shadow' to the nix-path without a prefix, either by passing the flag '-I nix-shadow' to the nix invocation or by adding this path to the environment variable 'NIX_PATH'.
+error: Shadowing '<nix/...>' by configuring the nix-path is deprecated and will be forbidden in the future.
+       Use --extra-deprecated-features nix-path-shadow to silence this error.
+       This is due to adding 'nix-shadow' to the nix-path without a prefix, either by passing the flag '-I nix-shadow' to the nix invocation or by adding this path to the environment variable 'NIX_PATH'.

tests/functional2/lang/search-path/test_search_path.py

@@ -1,7 +1,7 @@
 from collections.abc import Callable
 from pathlib import Path
 
-from lang.test_lang import test_eval_okay as nix_eval
+from lang.test_lang import test_eval_fail as nix_eval_fail, test_eval_okay as nix_eval_okay
 from testlib.fixtures.file_helper import AssetSymlink, CopyFile, CopyTree, with_files
 from testlib.fixtures.nix import Nix
 from testlib.fixtures.snapshot import Snapshot
@@ -21,7 +21,7 @@ from testlib.fixtures.snapshot import Snapshot
 )
 def test_search_path(files: Path, nix: Nix, snapshot: Callable[[str], Snapshot]):
     nix.env.set_env("NIX_PATH", "dir3:dir4")
-    nix_eval(
+    nix_eval_okay(
         files,
         nix,
         [
@@ -43,12 +43,26 @@ def test_search_path(files: Path, nix: Nix, snapshot: Callable[[str], Snapshot])
         "nix-shadow": CopyTree("nix-shadow"),
         "in.nix": CopyFile("in-fetchurl.nix"),
         "out.exp": AssetSymlink("eval-okay-prefixed.out.exp"),
+    }
+)
+def test_prefixed_search_path_deprecated(
+    files: Path, nix: Nix, snapshot: Callable[[str], Snapshot]
+):
+    nix.env.set_env("NIX_PATH", "nix=nix-shadow")
+    nix.settings.add_dp_feature("nix-path-shadow")
+    nix_eval_okay(files, nix, [], snapshot)
+
+
+@with_files(
+    {
+        "nix-shadow": CopyTree("nix-shadow"),
+        "in.nix": CopyFile("in-fetchurl.nix"),
         "err.exp": AssetSymlink("eval-okay-prefixed.err.exp"),
     }
 )
 def test_prefixed_search_path(files: Path, nix: Nix, snapshot: Callable[[str], Snapshot]):
     nix.env.set_env("NIX_PATH", "nix=nix-shadow")
-    nix_eval(files, nix, [], snapshot)
+    nix_eval_fail(files, nix, [], snapshot)
 
 
 @with_files(
@@ -56,11 +70,24 @@ def test_prefixed_search_path(files: Path, nix: Nix, snapshot: Callable[[str], S
         "nix-shadow": CopyTree("nix-shadow"),
         "in.nix": CopyFile("in-fetchurl.nix"),
         "out.exp": AssetSymlink("eval-okay-prefixless.out.exp"),
+    }
+)
+def test_prefixless_search_path_deprecated(
+    files: Path, nix: Nix, snapshot: Callable[[str], Snapshot]
+):
+    nix.settings.add_dp_feature("nix-path-shadow")
+    nix_eval_okay(files, nix, ["-I", "nix-shadow"], snapshot)
+
+
+@with_files(
+    {
+        "nix-shadow": CopyTree("nix-shadow"),
+        "in.nix": CopyFile("in-fetchurl.nix"),
         "err.exp": AssetSymlink("eval-okay-prefixless.err.exp"),
     }
 )
 def test_prefixless_search_path(files: Path, nix: Nix, snapshot: Callable[[str], Snapshot]):
-    nix_eval(files, nix, ["-I", "nix-shadow"], snapshot)
+    nix_eval_fail(files, nix, ["-I", "nix-shadow"], snapshot)
 
 
 @with_files(
@@ -71,4 +98,4 @@ def test_prefixless_search_path(files: Path, nix: Nix, snapshot: Callable[[str],
     }
 )
 def test_empty_search_path(files: Path, nix: Nix, snapshot: Callable[[str], Snapshot]):
-    nix_eval(files, nix, [], snapshot)
+    nix_eval_okay(files, nix, [], snapshot)

tests/functional2/lang/select-err/eval-fail-2.err.exp

@@ -0,0 +1,23 @@
+error:
+       … while calling the 'throw' builtin
+         at /pwd/in.nix:6:1:
+            5| in
+            6| throw set.inner
+             | ^
+            7|
+
+       … while evaluating the attribute 'inner'
+         at /pwd/in.nix:3:5:
+            2|   set = {
+            3|     inner = throw "nested throw";
+             |     ^
+            4|   };
+
+       … caused by explicit throw
+         at /pwd/in.nix:3:13:
+            2|   set = {
+            3|     inner = throw "nested throw";
+             |             ^
+            4|   };
+
+       error: nested throw

tests/functional2/lang/select-err/in-2.nix

@@ -0,0 +1,6 @@
+let
+  set = {
+    inner = throw "nested throw";
+  };
+in
+throw set.inner

tests/functional2/meson.build

@@ -23,6 +23,10 @@ if build_test_env != ''
 endif
 functional2_env.set('system', host_system)
 
+foreach name, val : coverage_test_env
+  functional2_env.set(name, val)
+endforeach
+
 test(
   'functional2',
   bash,

tests/functional2/store/test_optimise_store.py

@@ -1,7 +1,7 @@
 from pathlib import Path
 
 from testlib.fixtures.file_helper import with_files
-from testlib.fixtures.nix import Nix
+from testlib.fixtures.nix import Nix, NixDaemon
 from testlib.utils import get_global_asset
 
 
@@ -40,7 +40,7 @@ class TestOptimizeStore:
     def test_optimise_store(self, nix: Nix):
         self._test_optimise_store(nix)
 
-    def test_optimise_store_daemon(self, nix: Nix):
+    def test_optimise_store_daemon(self, nix: Nix, daemon: NixDaemon):
         nix.settings.auto_optimise_store = True
-        with nix.daemon([], {"trusted-users": "*"}) as inner:
+        with daemon(nix, [], {"trusted-users": "*"}) as inner:
             self._test_optimise_store(inner)

tests/functional2/store/test_simple.py

@@ -0,0 +1,72 @@
+from pathlib import Path
+
+import pytest
+
+from testlib.fixtures.file_helper import with_files, CopyFile, File
+from testlib.fixtures.nix import Nix
+from testlib.utils import get_global_asset_pack
+from testlib.environ import environ
+
+_files = get_global_asset_pack("simple-drv") | {
+    "hash-check.nix": CopyFile("assets/test_simple/hash-check.nix"),
+    "big-derivation-attr.nix": CopyFile("assets/test_simple/big-derivation-attr.nix"),
+    "dummy": File("Hello World\n"),
+}
+
+
+@pytest.fixture
+def drv(nix: Nix) -> str:
+    res = nix.nix_instantiate(["simple.nix"]).run().ok()
+    return res.stdout_plain
+
+
+@with_files(_files)
+def test_store_system(nix: Nix, drv: str):
+    res = nix.nix_store(["-q", "--binding", "system", drv]).run().ok()
+    assert res.stdout_plain == environ.get("system")
+
+
+@with_files(_files)
+def test_out_path(nix: Nix, drv: str):
+    res = nix.nix_store(["-rvv", drv]).run().ok()
+    out_path = Path(res.stdout_plain)
+
+    assert out_path.exists()
+    text_path = out_path / "hello"
+    assert text_path.read_text() == "Hello World!\n"
+
+    # Directed delete: $outPath is not reachable from a root, so it should
+    # be deleteable.
+    nix.nix_store(["--delete", str(out_path)]).run().ok()
+    assert not text_path.exists()
+
+    res = (
+        nix.nix(
+            [
+                "eval",
+                "--store",
+                f"local?store=/foo&real={nix.env.dirs.real_store_dir}",
+                "--read-only",
+                "-f",
+                "hash-check.nix",
+            ],
+            flake=True,
+        )
+        .run()
+        .ok()
+    )
+    assert (
+        res.stdout_plain == "«derivation /foo/lfy1s6ca46rm5r6w4gg9hc0axiakjcnm-dependencies.drv»"
+    ), "hashDerivationModulo appears broken"
+
+    nix.env.set_env("NIX_REMOTE", f"local?store=/foo&real={nix.env.dirs.real_store_dir}")
+    nix.settings.store = None
+    res = nix.nix_instantiate(["--readonly-mode", "hash-check.nix"]).run().ok()
+    assert res.stdout_plain == "/foo/lfy1s6ca46rm5r6w4gg9hc0axiakjcnm-dependencies.drv", (
+        "hashDerivationModulo appears broken"
+    )
+
+    res = nix.nix_instantiate(["--readonly-mode", "big-derivation-attr.nix"]).run().ok()
+    assert res.stdout_plain == "/foo/xxiwa5zlaajv6xdjynf9yym9g319d6mn-big-derivation-attr.drv", (
+        "big-derivation-attr.nix hash appears broken. Memory corruption in large drv attr?"
+    )

tests/functional2/testlib/fixtures/env.py

@@ -221,8 +221,12 @@ class ManagedEnv:
         if platform.system() == "Darwin":
             # Darwin / Apple behaves differently and requires _NIX_TEST_NO_SANDBOX to be set for whatever reason
             self._env |= {"_NIX_TEST_NO_SANDBOX": "1"}
-            # copy global path to maintain features usually provided by busybox
-            [self.path.append(p) for p in global_path.split(":") if Path(p).exists()]
+            # Copy coreutils from the global path to maintain availability of commands that are not part of
+            # XCode Developer Tools and provided by busybox on Linux, which does not build on Darwin
+            for p in global_path.split(":"):
+                if Path(p).exists() and "coreutils" in p:
+                    self.path.append(p)
+                    break
 
     def set_env(self, name: str, value: str):
         if name in self.dirs.get_env_keys():

tests/functional2/testlib/fixtures/nix.py

@@ -4,7 +4,7 @@ import dataclasses
 import sys
 from functools import partialmethod
 from pathlib import Path
-from typing import Any
+from typing import Any, Literal, get_args
 from collections.abc import Callable, Generator
 import shutil
 import subprocess
@@ -20,9 +20,9 @@ from testlib.utils import is_value_of_type
 type _NixSettingValue = str | int | list[str] | bool | None
 
 
-def _serialise(value: Any) -> str:
+def _serialise_config(value: _NixSettingValue) -> str:
     if is_value_of_type(value, list[str]):
-        return " ".join(_serialise(e) for e in value)
+        return " ".join(_serialise_config(e) for e in value)
     if is_value_of_type(value, bool):
         return "true" if value else "false"
     if is_value_of_type(value, str | int):
@@ -105,7 +105,7 @@ class NixSettings:
 
         self["extra-sandbox-paths"] += env.path.to_sandbox_paths()
 
-        def field_may(name: str, value: Any, serializer: Callable[[Any], str] = _serialise):
+        def field_may(name: str, value: Any, serializer: Callable[[Any], str] = _serialise_config):
             nonlocal config
             if value is not None:
                 config += f"{name} = {serializer(value)}\n"
@@ -158,7 +158,6 @@ class Nix:
     def nix_cmd(self, argv: list[str], flake: bool = False, cwd: Path | None = None) -> Command:
         """
         Constructs a NixCommand with the appropriate settings.
-        :param build: if the executed command wants to build stuff. This is required due to darwin shenanigans. "auto" will try to autodetect, override using `True` or `False`. Has no effect on linux.
         """
         # Create a copy of settings to not have a writing side effect
         settings = self.settings.clone()
@@ -173,57 +172,6 @@ class Nix:
     ) -> Command:
         return self.nix_cmd([nix_exe, *cmd], flake=flake, cwd=cwd)
 
-    @contextlib.contextmanager
-    def daemon(
-        self,
-        args: list[str] | None = None,
-        settings: dict[str, _NixSettingValue] | None = None,
-        **kwargs,
-    ) -> "Nix":
-        daemon = copy.deepcopy(self)
-        daemon.logger = self.logger.getChild("daemon")
-        daemon.settings["allowed-users"] = ["*"]
-        daemon.settings["trusted-users"] = []
-        daemon.settings.store = f"local?root={self.env.dirs.test_root}"
-        daemon.settings.update(settings)
-
-        sockets_dir = Path(daemon.env.dirs.nix_state_dir) / "daemon-socket"
-        sockets = [sockets_dir / "socket"]
-        for p in sockets:
-            p.unlink(missing_ok=True)
-
-        proc = daemon.nix(args or [], nix_exe="nix-daemon", **kwargs).start()
-
-        def log_daemon_result(result: CommandResult | None, level: int):
-            if result:
-                daemon.logger.log(level, "daemon exited with code %i", result.rc)
-                daemon.logger.log(level, "stdout: %s", result.stdout_s)
-                daemon.logger.log(level, "stderr: %s", result.stderr_s)
-            else:
-                daemon.logger.error("daemon exited unexpectedly")
-
-        # wait for daemon to come up. this may take a while under load.
-        # we only test the *last* socket in the list because that's the
-        # last one the daemon creates, once it's there the daemon is up
-        while not sockets[-1].exists():
-            if status := proc.wait(0.01):
-                log_daemon_result(status, logging.ERROR)
-                raise RuntimeError("daemon exited during startup")
-
-        inner = copy.deepcopy(self)
-        inner.settings.store = f"unix://{sockets[-1]}"  # missing multi socket support
-
-        try:
-            timeout, level = 1, logging.ERROR
-            yield inner
-            # 5 seconds should be enough to wait for a *graceful* exit.
-            timeout, level = 5, logging.DEBUG
-        finally:
-            result = proc.terminate(timeout)
-            if not result:
-                result = proc.kill()
-            log_daemon_result(result, level)
-
     # Mark each of these as correct as they are not ClassVars, but we also don't want to turn off RUF045
     nix_build = partialmethod(nix, nix_exe="nix-build")  # noqa: RUF045
     nix_shell = partialmethod(nix, nix_exe="nix-shell")  # noqa: RUF045
@@ -342,6 +290,75 @@ def nix(tmp_path: Path, env: ManagedEnv, logger: logging.Logger) -> Generator[Ni
     cmd.run().ok()
 
 
+type NixDaemon = Callable[..., contextlib.AbstractAsyncContextManager[Nix]]
+
+type NixDaemonProtocol = Literal["legacy-combined"]
+
+daemon_protocols: list[NixDaemonProtocol] = get_args(NixDaemonProtocol.__value__)
+
+
+# paramterize every daemon tests to run using all supported nix protocols
+@pytest.fixture(params=daemon_protocols)
+def daemon(request: pytest.FixtureRequest) -> NixDaemon:
+    default_protocol = request.param
+
+    @contextlib.contextmanager
+    def wrapper(
+        nix: Nix,
+        args: list[str] | None = None,
+        settings: dict[str, _NixSettingValue] | None = None,
+        protocol: NixDaemonProtocol | None = None,
+        **kwargs,
+    ) -> contextlib.AbstractAsyncContextManager[Nix]:
+        protocol = protocol or default_protocol
+
+        daemon = copy.deepcopy(nix)
+        daemon.logger = nix.logger.getChild("daemon")
+        daemon.settings["allowed-users"] = ["*"]
+        daemon.settings["trusted-users"] = []
+        daemon.settings.store = f"local?root={nix.env.dirs.test_root}"
+        daemon.settings.update(settings)
+
+        sockets_dir = Path(daemon.env.dirs.nix_state_dir) / "daemon-socket"
+        sockets = [sockets_dir / "socket"]
+        for p in sockets:
+            p.unlink(missing_ok=True)
+
+        proc = daemon.nix(args or [], nix_exe="nix-daemon", **kwargs).start()
+
+        def log_daemon_result(result: CommandResult | None, level: int):
+            if result:
+                daemon.logger.log(level, "daemon exited with code %i", result.rc)
+                daemon.logger.log(level, "stdout: %s", result.stdout_s)
+                daemon.logger.log(level, "stderr: %s", result.stderr_s)
+            else:
+                daemon.logger.error("daemon exited unexpectedly")
+
+        # wait for daemon to come up. this may take a while under load.
+        # we only test the *last* socket in the list because that's the
+        # last one the daemon creates, once it's there the daemon is up
+        while not sockets[-1].exists():
+            if status := proc.wait(0.01):
+                log_daemon_result(status, logging.ERROR)
+                raise RuntimeError("daemon exited during startup")
+
+        inner = copy.deepcopy(nix)
+        inner.settings.store = f"unix://{sockets[-1]}"  # missing multi socket support
+
+        try:
+            timeout, level = 1, logging.ERROR
+            yield inner
+            # 5 seconds should be enough to wait for a *graceful* exit.
+            timeout, level = 5, logging.DEBUG
+        finally:
+            result = proc.terminate(timeout)
+            if not result:
+                result = proc.kill()
+            log_daemon_result(result, level)
+
+    return wrapper
+
+
 @pytest.fixture
 def enable_diverted_store(nix: Nix):
     """

tests/functional2/testlib/test_utils.py

@@ -30,6 +30,14 @@ def test_list_type_nested_single_invalid():
     assert not is_value_of_type([[1], [2, 3], ["a"]], list[list[int]])
 
 
+def test_none_type():
+    assert is_value_of_type(None, None)
+
+
+def test_none_type_union():
+    assert is_value_of_type(None, int | None)
+
+
 def test_weird_type_valid():
     assert is_value_of_type(42, Literal[42])
 

tests/functional2/testlib/utils.py

@@ -116,6 +116,8 @@ def is_value_of_type(value: Any, expected_type: type[Any] | UnionType) -> bool:
         return True
     match origin:
         case None:
+            if expected_type is None:
+                return origin is None
             return isinstance(value, expected_type)
         case types.UnionType:
             return any(is_value_of_type(value, t) for t in get_args(expected_type))

tests/unit/libexpr/attr-path.cc

@@ -38,12 +38,56 @@ std::pair<Value, PosIdx> AttrPathEval::testFindAlongAttrPath(std::string expr, s
 
 // n.b. I do not know why we throw for empty attrs but they are apparently
 // disallowed.
-TEST_F(AttrPathEval, emptyAttrsThrows)
+TEST_F(AttrPathEval, emptyAttrsThrowsWithoutQuotes)
 {
     std::string expr = "{a.\"\".b = 2;}";
     ASSERT_NO_THROW(testFindAlongAttrPath(expr, "a"));
-    ASSERT_THROW(testFindAlongAttrPath(expr, "a.\"\".b"), Error);
-    ASSERT_THROW(testFindAlongAttrPath(expr, "a.\"\""), Error);
+    ASSERT_NO_THROW(testFindAlongAttrPath(expr, "a.\"\".b"));
+    ASSERT_THROW(testFindAlongAttrPath(expr, "a..b"), Error);
+    ASSERT_NO_THROW(testFindAlongAttrPath(expr, "a.\"\""));
 }
 
+TEST(attr_path_eval, quotes)
+{
+    auto p1 = parseAttrPath("foo.\"foo bar\".baz");
+    ASSERT_EQ(3, p1.size());
+    ASSERT_EQ("foo", p1[0]);
+    ASSERT_EQ("foo bar", p1[1]);
+    ASSERT_EQ("baz", p1[2]);
+
+    auto p2 = parseAttrPath("foo.\"foo bar\"");
+    ASSERT_EQ(2, p2.size());
+    ASSERT_EQ("foo", p2[0]);
+    ASSERT_EQ("foo bar", p2[1]);
+
+    auto p3 = parseAttrPath("\"foo bar\"");
+    ASSERT_EQ(1, p3.size());
+    ASSERT_EQ("foo bar", p3[0]);
+}
+
+TEST(attr_path_eval, quotes_empty)
+{
+    auto p1 = parseAttrPath("foo.\"\".bar");
+    ASSERT_EQ(3, p1.size());
+    ASSERT_EQ("foo", p1[0]);
+    ASSERT_EQ("", p1[1]);
+    ASSERT_EQ("bar", p1[2]);
+
+    auto p2 = parseAttrPath("foo.\"\"");
+    ASSERT_EQ(2, p2.size());
+    ASSERT_EQ("foo", p2[0]);
+    ASSERT_EQ("", p2[1]);
+
+    auto p3 = parseAttrPath("\"\"");
+    ASSERT_EQ(1, p3.size());
+    ASSERT_EQ("", p3[0]);
+}
+
+TEST(attr_path_eval, quotes_syntax)
+{
+    ASSERT_THROW(parseAttrPath("foo.\"bar"), ParseError);
+
+    // escaped quotes (\") are not supported
+    ASSERT_THROW(parseAttrPath("foo.\"bar\\\"\""), ParseError);
+}
 }

tests/unit/libexpr/derived-path.cc

@@ -27,8 +27,7 @@ RC_GTEST_FIXTURE_PROP(
     prop_opaque_path_round_trip,
     (const SingleDerivedPath::Opaque & o))
 {
-    Value v;
-    evaluator.paths.mkStorePathString(o.path, v);
+    Value v = evaluator.paths.mkStorePathString(o.path);
     auto d = state.coerceToSingleDerivedPath(noPos, v, "");
     RC_ASSERT(SingleDerivedPath { o } == d);
 }
@@ -41,8 +40,7 @@ RC_GTEST_FIXTURE_PROP(
     prop_derived_path_built_out_path_round_trip,
     (const SingleDerivedPath::Built & b, const StorePath & outPath))
 {
-    Value v;
-    state.mkOutputString(v, b, outPath);
+    Value v = state.mkOutputString(b, outPath);
     auto [d, _] = state.coerceToSingleDerivedPathUnchecked(noPos, v, "");
     RC_ASSERT(SingleDerivedPath { b } == d);
 }

tests/unit/libexpr/json.cc

@@ -15,8 +15,7 @@ namespace nix {
     };
 
     TEST_F(JSONValueTest, null) {
-        Value v;
-        v.mkNull();
+        Value v = Value::VNULL;
         ASSERT_EQ(getJSONValue(v), "null");
     }
 

tests/unit/libexpr/value/print-type.cc

@@ -152,8 +152,7 @@ TEST_F(TypeValuePrintingTests, vExternal)
         }
     } myExternal;
 
-    Value vExternal;
-    vExternal.mkExternal(&myExternal);
+    Value vExternal = {NewValueAs::external, myExternal};
 
     test(vExternal, "an external value from MyExternal");
 }