release: merge release 2.93.4 back to mainline

This merge commit returns to the previous state prior to the release but leaves the tag in the branch history.
Release created with releng/create_release.xsh

Change-Id: I34efc0973f445d42e2afaacf2b7fb65c5566bec9

.editorconfig

@@ -33,7 +33,3 @@ max_line_length = 0
 [meson.build]
 indent_style = space
 indent_size = 2
-
-[*.json]
-indent_style = space
-indent_size = 4

.gitignore

@@ -39,5 +39,3 @@ buildtime.bin
 
 # Python compiled files from the code generators and test suite
 *.pyc
-
-**/.idea

doc/manual/change-authors.yml

@@ -66,13 +66,13 @@ delan:
   forgejo: delan
   github: delan
 
-delroth:
-  github: delroth
-
 detroyejr:
   display_name: Jonathan De Troye
   github: detroyejr
 
+edef:
+  github: edef1c
+
 edolstra:
   display_name: Eelco Dolstra
   github: edolstra
@@ -200,6 +200,9 @@ roberth:
   display_name: Robert Hensing
   github: roberth
 
+sandydoo:
+  github: sandydoo
+
 seppel3210:
   github: Seppel3210
 

doc/manual/rl-next/deprecated-derivation-types.md

@@ -1,16 +0,0 @@
----
-synopsis: Deprecation of CA derivations, dynamic derivations, and impure derivations
-issues: [fj#815]
-cls: []
-significance: significant
-category: Miscellany
-credits: []
----
-
-Content-addressed derivations are now deprecated and slated for removal in Lix 2.94.
-We're doing this because the CA derivation system has been a known cause of problems
-and inconsistencies, is unmaintained, habitually makes improving the store code very
-difficult (or blocks such improvements outright), and is beset by a number of design
-flaws that in our opinion cannot be fixed without a full reimplementation from zero.
-Dynamic derivations and impure derivations are built on the CA derivation framework,
-and owing to this they too are deprecated and slated for removal in another release.

doc/manual/rl-next/integer-coercion.md

@@ -1,47 +0,0 @@
----
-synopsis: Experimental integer coercion in interpolated strings
-issues: []
-cls: [3198]
-category: "Features"
-credits: [raito, delroth, horrors, winter]
----
-
-Ever tried interpolating a port number in Lix and ended up with something like this?
-
-```nix
-"http://${config.network.host}:${builtins.toString config.network.port}/"
-```
-
-You're not alone. Thousands of Lix users suffer every day from excessive `builtins.toString` syndrome. It’s 2025, and we still have to cast integers to use them in strings.
-
-To address this, Lix introduces the **`coerce-integers`** experimental feature. When enabled, interpolated integers within `"${...}"` are automatically coerced to strings. This allows writing:
-
-```nix
-"http://${config.network.host}:${config.network.port}/"
-```
-
-without additional conversion.
-
-To enable the feature, you need to add `coerce-integers` to your set of experimental features.
-
-### Stabilization criteria
-
-The `coerce-integers` feature is experimental and limited strictly to string interpolation (`"${...}"`). Before stabilization, the following must hold:
-
-1. **Interpolation-only**
-   Coercion must not occur outside interpolation. Expressions like `"" + 42` must continue to fail.
-
-2. **Expectation that no explicit cast are being observed**
-   Cases observing explicit coercion (e.g., via `tryEval` gadget or similar) are expected not to be load-bearing in actual production code.
-
-### Timeline for stabilization
-
-If the feature proves safe and is widely adopted across typical usage (e.g., actual configurations in the wild turning on the flag, non-trivial out-of-tree projects using it), the experimental flag will be removed **after six months of active use or two Lix releases**, whichever is longer.
-
-This avoids locking the feature in experimental status indefinitely, as happened with Flakes, while allowing time for validation and ecosystem integration.
-
-### What about coercing floats or more?
-
-Coercion beyond integers -- such as for floats or other types -- is **not planned**, even under an experimental flag. Questions like "what is the canonical string representation of a float?" involve subtle and context-dependent trade-offs. Without a robust and principled mechanism to define and audit such behavior, introducing broader coercion risks setting unintended and hard-to-reverse precedents. The scope of `coerce-integers` is intentionally narrow and will remain so.
-
-In terms of outlook, a proposal like https://git.lix.systems/lix-project/lix/issues/835 could pave the way for a better solution.

doc/manual/rl-next/log-output-terminal-codes.md

@@ -1,11 +0,0 @@
----
-synopsis: Fix handling of OSC codes in terminal output
-issues: [fj#160]
-cls: [3143]
-category: Fixes
-credits: [lilyball]
----
-
-OSC codes in terminal output are now handled correctly, where OSC 8 (hyperlink) is preserved any
-time color codes are allowed and all other OSC codes are stripped out. This applies not only to
-output from build commands but also to rendered documentation in the REPL.

doc/manual/rl-next/mismatch-check-path.md

@@ -1,14 +0,0 @@
----
-synopsis: Better debuggability on fixed-output hash mismatches
-issues: []
-cls: []
-category: Improvements
-credits: [lheckemann]
----
-
-Fixed-output derivation hash mismatch error messages will now include the path that was
-produced unexpectedly, and this path will be registered as valid even if `--check`
-(`nix-store`, `nix-build`) or `--rebuild` (`nix build`) was passed. This makes comparing
-the expected path with the obtained path easier, and is useful for debugging when
-upstreams modify previously-published releases or when changes in fixed-output
-derivations' dependencies affect their output unexpectedly.

doc/manual/rl-next/reliance-on-bash-for-ssh.md

@@ -1,21 +0,0 @@
----
-synopsis: Remove reliance on Bash for remote stores via SSH
-issues: [fj#830, fj#805, fj#304]
-cls: [3159]
-category: "Fixes"
-credits: [raito]
----
-
-The pre-flight `echo started` handshake -- added years ago to catch race conditions -- has been removed.
-
-After removal of connection sharing in Lix 2.93, it required a Bash-compatible shell and a standard `echo`, so it failed on:
-
-  * builders protected by `ForceCommand` wrappers (e.g. `nix-remote-build`),
-  * BusyBox / initrd images with no Bash,
-  * hosts using non-POSIX shells such as Nushell.
-
-The race the probe once addressed was tied to SSH connection-sharing -- since connection-sharing code has already been removed, the probe is now pointless.
-
-Real connection or protocol errors are now left to SSH/Nix to report directly.
-
-This is technically a breaking change if you had scripts that relied on the literal "started" which needs to be updated to rely on other signals, e.g., exit codes.

doc/manual/rl-next/remove-xp-derivations.md

@@ -1,17 +0,0 @@
----
-synopsis: Remove impure derivations and dynamic derivations
-issues: [fj#815]
-cls: [3210]
-significance: significant
-category: "Breaking Changes"
-credits: [horrors]
----
-
-The `impure-derivations` and `dynamic-derivations` experimental feature have
-been removed.
-
-New impure or dynamic derivations cannot be created from this point forward, and
-any such pre-existing store derivations canot be read or built any more.
-Derivation outputs created by building such a derivation are still valid
-until garbage collected; existing store derivations can only be garbage
-collected.

doc/manual/src/SUMMARY.md

@@ -20,6 +20,7 @@
   - [Basic Package Management](package-management/basic-package-mgmt.md)
   - [Profiles](package-management/profiles.md)
   - [Garbage Collection](package-management/garbage-collection.md)
+    - [Garbage Collector Roots](package-management/garbage-collector-roots.md)
   - [Sharing Packages Between Machines](package-management/sharing-packages.md)
     - [Serving a Nix store via HTTP](package-management/binary-cache-substituter.md)
     - [Copying Closures via SSH](package-management/copy-closure.md)

doc/manual/src/command-ref/nix-store/realise.md

@@ -15,6 +15,7 @@ Each of *paths* is processed as follows:
   1. If it is not [valid], substitute the store derivation file itself.
   2. Realise its [output paths]:
     - Try to fetch from [substituters] the [store objects] associated with the output paths in the store derivation's [closure].
+      - With [content-addressed derivations] (experimental): Determine the output paths to realise by querying content-addressed realisation entries in the [Nix database].
     - For any store paths that cannot be substituted, produce the required store objects. This involves first realising all outputs of the derivation's dependencies and then running the derivation's [`builder`](@docroot@/language/derivations.md#attr-builder) executable. <!-- TODO: Link to build process page #8888 -->
 - Otherwise, and if the path is not already valid: Try to fetch the associated [store objects] in the path's [closure] from [substituters].
 
@@ -27,6 +28,7 @@ If no substitutes are available and no store derivation is given, realisation fa
 [store objects]: @docroot@/glossary.md#gloss-store-object
 [closure]: @docroot@/glossary.md#gloss-closure
 [substituters]: @docroot@/command-ref/conf-file.md#conf-substituters
+[content-addressed derivations]: @docroot@/contributing/experimental-features.md#xp-feature-ca-derivations
 [Nix database]: @docroot@/glossary.md#gloss-nix-database
 
 The resulting paths are printed on standard output.

doc/manual/src/contributing/hacking.md

@@ -39,7 +39,7 @@ $ nix-shell -A native-clangStdenvPackages
 
 ### Building from the development shell
 
-Run a clean build and test with `just clean setup build install test`.
+Run a clean build and test with `just clean build install test`.
 
 You can also run the unit tests and integration tests separately:
 

doc/manual/src/contributing/testing.md

@@ -449,6 +449,9 @@ I grepped `lix/` for `get[eE]nv\("` to find the mentions in Lix code.
 - `NIX_CLIENT_PACKAGE` - Runs the test suite against an alternate Nix client with the current daemon.
 
   **Expected value**: something like `/nix/store/...-nix-2.18.2`
+- `NIX_TESTS_CA_BY_DEFAULT` - Pass `__contentAddressed`, `outputHashMode` and `outputHashAlgo` to builds of some input-addressed derivations in the test suite.
+
+  **Expected value**: 1
 - `TEST_DATA` - Not an environment variable! This is used in repl characterization tests to refer to `tests/functional/repl_characterization/data`.
   More specifically, that path is replaced with the string `$TEST_DATA` in output for reproducibility.
 - `TEST_HOME` (output) - Set to the temporary directory that is set as `$HOME` inside the tests, underneath `$TEST_ROOT`.

doc/manual/src/glossary.md

@@ -41,6 +41,12 @@
 
   [realise]: #gloss-realise
 
+- [content-addressed derivation]{#gloss-content-addressed-derivation}
+
+  A derivation which has the
+  [`__contentAddressed`](./language/advanced-attributes.md#adv-attr-__contentAddressed)
+  attribute set to `true`.
+
 - [fixed-output derivation]{#gloss-fixed-output-derivation}
 
   A derivation which includes the
@@ -108,13 +114,14 @@
 - [input-addressed store object]{#gloss-input-addressed-store-object}
 
   A store object produced by building a
+  non-[content-addressed](#gloss-content-addressed-derivation),
   non-[fixed-output](#gloss-fixed-output-derivation)
   derivation.
 
 - [output-addressed store object]{#gloss-output-addressed-store-object}
 
   A [store object] whose [store path] is determined by its contents.
-  This includes derivations and the outputs of [fixed-output derivations](#gloss-fixed-output-derivation).
+  This includes derivations, the outputs of [content-addressed derivations](#gloss-content-addressed-derivation), and the outputs of [fixed-output derivations](#gloss-fixed-output-derivation).
 
 - [substitute]{#gloss-substitute}
 

doc/manual/src/language/advanced-attributes.md

@@ -209,8 +209,15 @@ Derivations can declare some infrequently used optional attributes.
 
   - [`__contentAddressed`]{#adv-attr-__contentAddressed}
     > **Warning**
-    > This attribute is part of a removed [experimental feature](@docroot@/contributing/experimental-features.md).
-    > Setting this flag *will* cause eval errors.
+    > This attribute is part of an [experimental feature](@docroot@/contributing/experimental-features.md).
+    >
+    > To use this attribute, you must enable the
+    > [`ca-derivations`](@docroot@/contributing/experimental-features.md#xp-feature-ca-derivations) experimental feature.
+    > For example, in [nix.conf](../command-ref/conf-file.md) you could add:
+    >
+    > ```
+    > extra-experimental-features = ca-derivations
+    > ```
 
     If this attribute is set to `true`, then the derivation
     outputs will be stored in a content-addressed location rather than the

doc/manual/src/package-management/garbage-collection.md

@@ -71,62 +71,3 @@ $ nix-collect-garbage -d
 ```
 
 is a quick and easy way to clean up your system.
-
-## Garbage Collector Roots
-
-### Explicit roots
-
-All store paths to which there are symlinks in the directory
-`prefix/nix/var/nix/gcroots` will be used as roots by the garbage
-collector. For instance, the following command makes the path
-`/nix/store/d718ef...-foo` a root of the collector:
-
-```console
-$ ln -s /nix/store/d718ef...-foo /nix/var/nix/gcroots/bar
-```
-
-That is, after this command, the garbage collector will not remove
-`/nix/store/d718ef...-foo` or any of its dependencies.
-
-Subdirectories of `prefix/nix/var/nix/gcroots` are also searched for
-symlinks.
-
-Symlinks may also point to paths outside the nix store. If the
-destination of the symlink is itself a symlink to a store path, it
-is also considered a root. This style of GC root is called an
-"indirect root", and is created by tools like `nix-build` to avoid
-garbage-collecting paths that are being used on-the-fly rather than
-installed in profiles.
-
-
-### In-use roots
-
-Lix will also perform a best-effort detection of paths that are in use
-by running processes when scanning for garbage collection roots, to
-avoid removing paths that are still needed by running processes.
-
-Exact details vary between platforms, but the following will generally
-be taken into account:
-
-- Executables in the store that are currently running;
-- Other files in the store that are mapped into a process's address space (e.g. shared libraries);
-- Files in the store to which processes have open handles;
-- Store paths found in processes' environment variables.
-
-Note that this detection is susceptible to missing paths that may still be in use for multiple reasons:
-
-- Time-of-check-to-time-of-use (TOCTTOU): new processes may appear
-  after Lix has enumerated the currently running processes, and will
-  not be taken into account;
-- Access privileges: if the garbage collection is not running as the
-  root user (this is typically the case for single-user
-  installations), it will not be able to scan processes belonging to
-  other users;
-- Other types of references: store paths may be stored in parts of the
-  filesystem (e.g. databases) or process memory (e.g. environment
-  variables changed since the start of the process) that Lix does not
-  scan.
-
-For this reason, it is recommended to create explicit roots whenever
-using store paths that aren't obtained from some existing explicit GC
-root.

doc/manual/src/package-management/garbage-collector-roots.md

@@ -0,0 +1,18 @@
+# Garbage Collector Roots
+
+The roots of the garbage collector are all store paths to which there
+are symlinks in the directory `prefix/nix/var/nix/gcroots`. For
+instance, the following command makes the path
+`/nix/store/d718ef...-foo` a root of the collector:
+
+```console
+$ ln -s /nix/store/d718ef...-foo /nix/var/nix/gcroots/bar
+```
+
+That is, after this command, the garbage collector will not remove
+`/nix/store/d718ef...-foo` or any of its dependencies.
+
+Subdirectories of `prefix/nix/var/nix/gcroots` are also searched for
+symlinks. Symlinks to non-store paths are followed and searched for
+roots, but symlinks to non-store paths *inside* the paths reached in
+that way are not followed to prevent infinite recursion.

doc/manual/src/protocols/derivation-aterm.md

@@ -2,8 +2,18 @@
 
 For historical reasons, [derivations](@docroot@/glossary.md#gloss-store-derivation) are stored on-disk in [ATerm](https://homepages.cwi.nl/~daybuild/daily-books/technology/aterm-guide/aterm-guide.html) format.
 
-Derivations are serialised in the following format:
+Derivations are serialised in one of the following formats:
 
-```
-Derive(...)
-```
+- ```
+  Derive(...)
+  ```
+
+  For all stable derivations.
+
+- ```
+  DrvWithVersion(<version-string>, ...)
+  ```
+
+  The only `version-string`s that are in use today are for [experimental features](@docroot@/contributing/experimental-features.md):
+
+  - `"xp-dyn-drv"` for the [`dynamic-derivations`](@docroot@/contributing/experimental-features.md#xp-feature-dynamic-derivations) experimental feature.

doc/manual/src/release-notes/rl-2.93.md

@@ -1,4 +1,242 @@
 # Lix 2.93 "Bici Bici" (2025-05-09)
+# Lix 2.93.4 (2026-05-04)
+## Fixes
+
+- `build-dir` no longer defaults to `temp-dir` [cl/3453](https://gerrit.lix.systems/c/lix/+/3453)
+
+  The directory in which temporary build directories are created no longer defaults
+  to the value of the `temp-dir` setting to avoid builders making their directories
+  world-accessible. This behavior has been used to escape the build sandbox and can
+  cause build impurities even when not used maliciously. We now default to `builds`
+  in `NIX_STATE_DIR` (which is `/nix/var/nix/b` in the default configuration).
+
+  Many thanks to [eldritch horrors](https://git.lix.systems/pennae) for this.
+
+- Fix develop shells for derivations with escape codes [fj#991](https://git.lix.systems/lix-project/lix/issues/991) [cl/4154](https://gerrit.lix.systems/c/lix/+/4154) [cl/4155](https://gerrit.lix.systems/c/lix/+/4155)
+
+  ASCII control characters (including `\e`, used for ANSI escape codes) in derivation variables are now correctly escaped for `nix develop` and `nix print-dev-env`, instead of erroring.
+
+  Many thanks to [Qyriad](https://git.lix.systems/Qyriad) for this.
+
+- Fix nix develop for derivations that rejects dependencies with structured attrs [fj#997](https://git.lix.systems/lix-project/lix/issues/997) [cl/4182](https://gerrit.lix.systems/c/lix/+/4182) [cl/4214](https://gerrit.lix.systems/c/lix/+/4214)
+
+  For the sake of concision, we refer to `disallowedReferences` in what follows,
+  but all output checks were equally fixed:
+  `{dis,}allowed{References,Requisites}`.
+
+  Derivations can define *output checks* to reject unwanted dependencies, such as
+  interpreters like `bash` or compilers like `gcc`. This can be done in two ways:
+
+  * **Legacy style**: `disallowedReferences = [ ... ]` in the environment.
+  * **Structured attrs**: `outputChecks.<output>.disallowedReferences = [ ... ]`,
+    typically used in `__json`.
+
+  Only the structured form supports derivations with multiple outputs.
+
+  `nix develop` internally rewrites derivations to create development shells. It
+  relied on the legacy `disallowedReferences`, and failed to honor the structured
+  variant. This led to broken shells in cases where `bashInteractive` was
+  explicitly disallowed using structured output checks, e.g. `nix develop
+  nixpkgs#systemd` after the "bash-less NixOS" changes.
+
+  This fix teaches `nix develop` to respect structured output checks, restoring
+  support for such derivations.
+
+  Many thanks to [Raito Bezarius](https://git.lix.systems/raito) for this.
+
+- `nix-shell` default shell directory is not `/tmp` anymore for `$NIX_BUILD_TOP` [fj#940](https://git.lix.systems/lix-project/lix/issues/940)
+
+  Previously, Lix `nix-shell`s could exit non-zero status when `stdenv`'s `dumpVars` phase failed to write to `$NIX_BUILD_TOP/env-vars`, despite `dumpVars` being intended as a debugging aid.
+
+  This happens when `TMPDIR` is not set and defaults therefore to `/tmp`, resulting in a `/tmp/env-vars` global file that every `nix-shell` wants to write.
+
+  We fix this issue by reusing a pre-created, unique, and writable location, as the build top directory, avoiding shell exiting from write failures silently.
+
+  Many thanks to [Raito Bezarius](https://git.lix.systems/raito) for this.
+
+- Fix unsigned overflow leading to out-of-band write in the NAR parser [cl/5537](https://gerrit.lix.systems/c/lix/+/5537)
+
+  The NAR parser contained an unsigned integer overflow that could be used by an
+  attacker to write arbitrary data to an unknown memory location and possibly
+  achieve code execution. A successful attack on the system-wide Lix daemon
+  could lead to privilege escalation to root. Any process that involves NAR
+  serialization could trigger this issue, including (but not limited to)
+
+    - local user interaction, whether the users are trusted or untrusted
+    - malicious substituters sending malformed NARs
+    - remote builders sending malformed build results
+    - remote daemons sending malformed inputs when requesting remote builds
+
+  Successful attacks using this bug require ASLR weakening of some sort, whether
+  by architecture constraints (e.g. on 32 bit systems, where little randomization
+  is possible) or system configuration (e.g. low ASLR entropy when loading
+  libraries), and millions of attempts. Local attacks can be mounted in less than
+  an hour. Remote builds typically require a fresh SSH connection for each build
+  and are thus less susceptible. Only one attempt can be made by substituters for
+  every build using substituters, they are thus not a likely vector for attacks.
+
+  At the time of writing, MITRE has not assigned this a CVE yet.
+
+  Many thanks to [eldritch horrors](https://git.lix.systems/pennae), [Raito Bezarius](https://git.lix.systems/raito), [edef](https://github.com/edef1c), and [sandydoo](https://github.com/sandydoo) for this.
+
+
+
+
+# Lix 2.93.3 (2025-07-22)
+## Improvements
+
+- `--keep-failed` chowns the build directory to the user that request the build [cl/3678](https://gerrit.lix.systems/c/lix/+/3678)
+
+  Running a build with `--keep-failed` now chowns the temporary directory from the
+  builder user and group to the user that request the build if the build came from
+  a local user connected to the daemon. This makes inspecting failed derivations a
+  lot easier. On Linux the build directory made visible to the user will not be in
+  the same path as it was in the sandbox and continuing builds will usually break.
+
+  Many thanks to [eldritch horrors](https://git.lix.systems/pennae) for this.
+
+
+
+
+# Lix 2.93.2 (2025-06-30)
+## Fixes
+
+- Revert CVE-2025-52992 failed mitigation [fj#883](https://git.lix.systems/lix-project/lix/issues/883) [fj#887](https://git.lix.systems/lix-project/lix/issues/887) [cl/3444](https://gerrit.lix.systems/c/lix/+/3444) [cl/3528](https://gerrit.lix.systems/c/lix/+/3528)
+
+  Following the initial mitigation of **CVE-2025-52992** in `cl/3444`, we
+  received reports of **unexpected deletion of in-use store paths**.
+
+  Upon investigation, we found that the patch did **not correctly cancel all
+  automatic deleters**, resulting in potentially critical path loss during normal
+  operation.
+
+  Given the severity and time-sensitive nature of the situation ([see incident
+  report](https://lix.systems/blog/2025-06-27-lix-critical-bug/)), we evaluated
+  possible options to repair the behavior safely. However, we concluded that a
+  rushed fix would either
+
+  * **Overdelete**, i.e. breaking running systems, or,
+  * **Underdelete**, effectively **reopening CVE-2025-52992** while leaving
+    orphaned paths behind.
+
+  As **CVE-2025-52992 has no known exploit vector**, and correctness is critical
+  in the Lix project, we have **fully reverted the previous mitigations**.
+
+  The affected patches (`cl/3444`) have been rolled back for the time being.
+
+  Moving forward, the Lix team will rework this code path in a **long-term,
+  correctness-first fix** on the main branch. We will explore backporting it to
+  stable channels once its safety is assured.
+
+  We are deeply sorry for the stability incident and the Lix team remain
+  available for assisting you in recovering your systems.
+
+  Many thanks to [Raito Bezarius](https://git.lix.systems/raito) and [eldritch horrors](https://git.lix.systems/pennae) for this.
+
+- Fallback to safe temp dir when build-dir is unwritable [fj#876](https://git.lix.systems/lix-project/lix/issues/876) [cl/3501](https://gerrit.lix.systems/c/lix/+/3501)
+
+  Non-daemon builds started failing with a permission error after introducing the `build-dir` option:
+
+  ```
+  $ nix build --store ~/scratch nixpkgs#hello --rebuild
+  error: creating directory '/nix/var/nix/builds/nix-build-hello-2.12.2.drv-0': Permission denied
+  ```
+
+  This happens because:
+
+  1. These builds are not run via the daemon, which owns `/nix/var/nix/builds`.
+  2. The user lacks permissions for that path.
+
+  We considered making `build-dir` a store-level option and defaulting it to `<chroot-root>/nix/var/nix/builds` for chroot stores, but opted instead for a fallback: if the default fails, Nix now creates a safe build directory under `/tmp`.
+
+  To avoid CVE-2025-52991, the fallback uses an extra path component between `/tmp` and the build dir.
+
+  **Note**: this fallback clutters `/tmp` with build directories that are not cleaned up. To prevent this, explicitly set `build-dir` to a path managed by Lix, even for local workloads.
+
+  Many thanks to [Raito Bezarius](https://git.lix.systems/raito) and [eldritch horrors](https://git.lix.systems/pennae) for this.
+
+
+
+
+# Lix 2.93.1 (2025-06-23)
+## Breaking Changes
+
+- Fixed output derivations can be run using `pasta` network isolation [fj#285](https://git.lix.systems/lix-project/lix/issues/285) [cl/3442](https://gerrit.lix.systems/c/lix/+/3442)
+
+  Fixed output derivations traditionally run in the host network namespace.
+  On Linux this allows such derivations to communicate with other sandboxes
+  or the host using the abstract Unix domains socket namespace; this hasn't
+  been unproblematic in the past and has been used in two distinct exploits
+  to break out of the sandbox. For this reason fixed output derivations can
+  now run in a network namespace (provided by [`pasta`]), restricted to TCP
+  and UDP communication with the rest of the world. When enabled this could
+  be a breaking change and we classify it as such, even though we don't yet
+  enable or require such isolation by default. We may enforce this in later
+  releases of Lix once we have sufficient confidence that breakage is rare.
+
+  [`pasta`]: https://passt.top/
+
+  Many thanks to [eldritch horrors](https://git.lix.systems/pennae) and [puck](https://git.lix.systems/puck) for this.
+
+
+## Fixes
+
+- Always clean up scratch paths after derivations failed to build [cl/3444](https://gerrit.lix.systems/c/lix/+/3444)
+
+  Previously, scratch paths created during builds were not always cleaned up if
+  the derivation failed, potentially leaving behind unnecessary temporary files
+  or directories in the Nix store.
+
+  This fix ensures that such paths are consistently removed after a failed build,
+  improving Nix store hygiene, hardening Lix against mis-reuse of failed builds
+  scratch paths.
+
+  Many thanks to [Raito Bezarius](https://git.lix.systems/raito) and [eldritch horrors](https://git.lix.systems/pennae) for this.
+
+- `build-dir` no longer defaults to `temp-dir` [cl/3443](https://gerrit.lix.systems/c/lix/+/3443)
+
+  The directory in which temporary build directories are created no longer defaults
+  to the value of the `temp-dir` setting to avoid builders making their directories
+  world-accessible. This behavior has been used to escape the build sandbox and can
+  cause build impurities even when not used maliciously. We now default to `builds`
+  in `NIX_STATE_DIR` (which is `/nix/var/nix/builds` in the default configuration).
+
+  Many thanks to [eldritch horrors](https://git.lix.systems/pennae) for this.
+
+- Remove reliance on Bash for remote stores via SSH [fj#830](https://git.lix.systems/lix-project/lix/issues/830) [fj#805](https://git.lix.systems/lix-project/lix/issues/805) [fj#304](https://git.lix.systems/lix-project/lix/issues/304) [cl/3159](https://gerrit.lix.systems/c/lix/+/3159)
+
+  The pre-flight `echo started` handshake -- added years ago to catch race conditions -- has been removed.
+
+  After removal of connection sharing in Lix 2.93, it required a Bash-compatible shell and a standard `echo`, so it failed on:
+
+    * builders protected by `ForceCommand` wrappers (e.g. `nix-remote-build`),
+    * BusyBox / initrd images with no Bash,
+    * hosts using non-POSIX shells such as Nushell.
+
+  The race the probe once addressed was tied to SSH connection-sharing -- since connection-sharing code has already been removed, the probe is now pointless.
+
+  Real connection or protocol errors are now left to SSH/Nix to report directly.
+
+  This is technically a breaking change if you had scripts that relied on the literal "started" which needs to be updated to rely on other signals, e.g., exit codes.
+
+  Many thanks to [Raito Bezarius](https://git.lix.systems/raito) for this.
+
+
+## Miscellany
+
+- Deprecation of CA derivations, dynamic derivations, and impure derivations [fj#815](https://git.lix.systems/lix-project/lix/issues/815)
+
+  Content-addressed derivations are now deprecated and slated for removal in Lix 2.94.
+  We're doing this because the CA derivation system has been a known cause of problems
+  and inconsistencies, is unmaintained, habitually makes improving the store code very
+  difficult (or blocks such improvements outright), and is beset by a number of design
+  flaws that in our opinion cannot be fixed without a full reimplementation from zero.
+  Dynamic derivations and impure derivations are built on the CA derivation framework,
+  and owing to this they too are deprecated and slated for removal in another release.
+
+
+
+
 
 
 # Lix 2.93.0 (2025-05-09)

docker.nix

@@ -62,38 +62,37 @@ let
     ++ autoLayered
     ++ extraPkgs;
 
-  users =
-    {
+  users = {
 
-      root = {
-        uid = 0;
-        shell = "${pkgs.bashInteractive}/bin/bash";
-        home = "/root";
-        gid = 0;
-        groups = [ "root" ];
-        description = "System administrator";
-      };
+    root = {
+      uid = 0;
+      shell = "${pkgs.bashInteractive}/bin/bash";
+      home = "/root";
+      gid = 0;
+      groups = [ "root" ];
+      description = "System administrator";
+    };
 
-      nobody = {
-        uid = 65534;
-        shell = "${pkgs.shadow}/bin/nologin";
-        home = "/var/empty";
-        gid = 65534;
-        groups = [ "nobody" ];
-        description = "Unprivileged account (don't use!)";
+    nobody = {
+      uid = 65534;
+      shell = "${pkgs.shadow}/bin/nologin";
+      home = "/var/empty";
+      gid = 65534;
+      groups = [ "nobody" ];
+      description = "Unprivileged account (don't use!)";
+    };
+  }
+  // lib.listToAttrs (
+    map (n: {
+      name = "nixbld${toString n}";
+      value = {
+        uid = 30000 + n;
+        gid = 30000;
+        groups = [ "nixbld" ];
+        description = "Nix build user ${toString n}";
       };
-    }
-    // lib.listToAttrs (
-      map (n: {
-        name = "nixbld${toString n}";
-        value = {
-          uid = 30000 + n;
-          gid = 30000;
-          groups = [ "nixbld" ];
-          description = "Nix build user ${toString n}";
-        };
-      }) (lib.lists.range 1 32)
-    );
+    }) (lib.lists.range 1 32)
+  );
 
   groups = {
     root.gid = 0;
@@ -193,13 +192,11 @@ let
             in
             ''
               {
-                ${
-                  lib.concatStringsSep "\n" (
-                    builtins.map (output: ''
-                      ${output} = { outPath = "${lib.getOutput output drv}"; };
-                    '') outputs
-                  )
-                }
+                ${lib.concatStringsSep "\n" (
+                  builtins.map (output: ''
+                    ${output} = { outPath = "${lib.getOutput output drv}"; };
+                  '') outputs
+                )}
                 outputs = [ ${lib.concatStringsSep " " (builtins.map (x: "\"${x}\"") outputs)} ];
                 name = "${drv.name}";
                 outPath = "${drv}";
@@ -361,8 +358,10 @@ let
         "org.opencontainers.image.source" = "https://git.lix.systems/lix-project/lix";
         "org.opencontainers.image.vendor" = "Lix project";
         "org.opencontainers.image.version" = pkgs.nix.version;
-        "org.opencontainers.image.description" = "Minimal Lix container image, with some batteries included.";
-      } // lib.optionalAttrs (lixRevision != null) { "org.opencontainers.image.revision" = lixRevision; };
+        "org.opencontainers.image.description" =
+          "Minimal Lix container image, with some batteries included.";
+      }
+      // lib.optionalAttrs (lixRevision != null) { "org.opencontainers.image.revision" = lixRevision; };
     };
 
     meta = {

flake.lock

@@ -16,6 +16,22 @@
         "type": "github"
       }
     },
+    "lowdown-src": {
+      "flake": false,
+      "locked": {
+        "lastModified": 1633514407,
+        "narHash": "sha256-Dw32tiMjdK9t3ETl5fzGrutQTzh2rufgZV4A/BbxuD4=",
+        "owner": "kristapsdz",
+        "repo": "lowdown",
+        "rev": "d2c2b44ff6c27b936ec27358a2653caaef8f73b8",
+        "type": "github"
+      },
+      "original": {
+        "owner": "kristapsdz",
+        "repo": "lowdown",
+        "type": "github"
+      }
+    },
     "nix2container": {
       "flake": false,
       "locked": {
@@ -32,18 +48,44 @@
         "type": "github"
       }
     },
-    "nixpkgs": {
+    "nix_2_18": {
+      "inputs": {
+        "flake-compat": [
+          "flake-compat"
+        ],
+        "lowdown-src": "lowdown-src",
+        "nixpkgs": "nixpkgs",
+        "nixpkgs-regression": [
+          "nixpkgs-regression"
+        ]
+      },
       "locked": {
-        "lastModified": 1733348545,
-        "narHash": "sha256-b4JrUmqT0vFNx42aEN9LTWOHomkTKL/ayLopflVf81U=",
+        "lastModified": 1730375271,
+        "narHash": "sha256-RrOFlDGmRXcVRV2p2HqHGqvzGNyWoD0Dado/BNlJ1SI=",
         "owner": "NixOS",
-        "repo": "nixpkgs",
-        "rev": "9ecb50d2fae8680be74c08bb0a995c5383747f89",
+        "repo": "nix",
+        "rev": "0f665ff6779454f2117dcc32e44380cda7f45523",
         "type": "github"
       },
       "original": {
         "owner": "NixOS",
-        "ref": "nixos-24.11-small",
+        "ref": "2.18.9",
+        "repo": "nix",
+        "type": "github"
+      }
+    },
+    "nixpkgs": {
+      "locked": {
+        "lastModified": 1705033721,
+        "narHash": "sha256-K5eJHmL1/kev6WuqyqqbS1cdNnSidIZ3jeqJ7GbrYnQ=",
+        "owner": "NixOS",
+        "repo": "nixpkgs",
+        "rev": "a1982c92d8980a0114372973cbdfe0a307f1bdea",
+        "type": "github"
+      },
+      "original": {
+        "owner": "NixOS",
+        "ref": "nixos-23.05-small",
         "repo": "nixpkgs",
         "type": "github"
       }
@@ -64,6 +106,22 @@
         "type": "github"
       }
     },
+    "nixpkgs_2": {
+      "locked": {
+        "lastModified": 1757198069,
+        "narHash": "sha256-m3VUcOD4rTs8J7S+3dOjWMrAjw6RcITC3XYQ98zhEFs=",
+        "owner": "NixOS",
+        "repo": "nixpkgs",
+        "rev": "0747026fc57ecb9c28901c7f7a2b5dc40e8af43c",
+        "type": "github"
+      },
+      "original": {
+        "owner": "NixOS",
+        "ref": "nixos-25.05-small",
+        "repo": "nixpkgs",
+        "type": "github"
+      }
+    },
     "pre-commit-hooks": {
       "flake": false,
       "locked": {
@@ -84,7 +142,8 @@
       "inputs": {
         "flake-compat": "flake-compat",
         "nix2container": "nix2container",
-        "nixpkgs": "nixpkgs",
+        "nix_2_18": "nix_2_18",
+        "nixpkgs": "nixpkgs_2",
         "nixpkgs-regression": "nixpkgs-regression",
         "pre-commit-hooks": "pre-commit-hooks"
       }

flake.nix

@@ -2,8 +2,19 @@
   description = "Lix: A modern, delicious implementation of the Nix package manager";
 
   inputs = {
-    nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.11-small";
+    nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.05-small";
     nixpkgs-regression.url = "github:NixOS/nixpkgs/215d4d0fd80ca5163643b03a33fde804a29cc1e2";
+
+    # Required because Nix 2.18 is not in Nixpkgs ≥ 25.05 anymore.
+    nix_2_18 = {
+      url = "github:NixOS/nix/2.18.9";
+      # NOTE(Raito): this is not possible because patches on libseccomp does not apply anymore on this Nix.
+      # Let's keep the latest known nixpkgs useable with Nix 2.18 for our tests.
+      # inputs.nixpkgs.follows = "nixpkgs";
+      inputs.nixpkgs-regression.follows = "nixpkgs-regression";
+      inputs.flake-compat.follows = "flake-compat";
+    };
+
     pre-commit-hooks = {
       url = "github:cachix/git-hooks.nix";
       flake = false;
@@ -25,6 +36,7 @@
       nixpkgs-regression,
       pre-commit-hooks,
       nix2container,
+      nix_2_18,
       flake-compat,
     }:
 
@@ -163,6 +175,16 @@
         {
           nixStable = prev.nix;
 
+          nixVersions = prev.nixVersions // {
+            nix_2_3 = prev.nixVersions.nix_2_3.overrideAttrs (old: {
+              meta = old.meta // {
+                knownVulnerabilities = [ ];
+              };
+            });
+            # Nix 2.18 has been removed from Nixpkgs ≥ 25.05, so we need to reintroduce it ourselves for our tests.
+            nix_2_18 = nix_2_18.outputs.packages.${currentStdenv.hostPlatform.system}.default;
+          };
+
           # Forward from the previous stage as we don’t want it to pick the lowdown override
           nixUnstable = prev.nixUnstable;
 
@@ -197,6 +219,9 @@
             inherit versionSuffix officialRelease;
             stdenv = currentStdenv;
             busybox-sandbox-shell = final.busybox-sandbox-shell or final.default-busybox-sandbox-shell;
+            # See below
+            lowdown = final.lowdown_3_0;
+            lowdown-unsandboxed = final.lowdown_3_0.override { enableDarwinSandbox = false; };
           };
 
           lix-clang-tidy = final.callPackage ./subprojects/lix-clang-tidy { };
@@ -221,6 +246,30 @@
           boehmgc-nix = final.nix.passthru.boehmgc-nix;
           # And same thing for our build-release-notes package.
           build-release-notes = final.nix.passthru.build-release-notes;
+
+          # As soon as Nixpkgs updates to >= 3.0.0, change to lowdown_2_0!
+          # We don't change the default version in order to not change the hash
+          # of Nix/Lix from upstream Nixpkgs.
+          lowdown_3_0 =
+            assert lib.versionOlder prev.lowdown.version "3.0.0";
+            prev.lowdown.overrideAttrs (
+              finalAttrs: prevAttrs: {
+                version = "3.0.1";
+                src = final.fetchurl {
+                  url = "https://kristaps.bsd.lv/lowdown/snapshots/lowdown-${finalAttrs.version}.tar.gz";
+                  sha512 = "fe68e1b7ff23f3992398356d7aa9a330dfd7b72e22bea9a91eeef74182b209ecea0c9f3e2b2216e1a07b2358da2b746238ec9cbbdeebdd3551cef14dd2d79f46";
+                };
+
+                # no longer compiles with GNU make
+                nativeBuildInputs = prevAttrs.nativeBuildInputs ++ [ final.bmake ];
+                # dylib fixups on darwin are no longer necessary
+                postInstall = "";
+                # doesn't work on darwin due to disallowed nested sandboxes
+                doInstallCheck = prevAttrs.doInstallCheck && !(final.stdenv.hostPlatform.isDarwin);
+                doCheck = prevAttrs.doCheck && !(final.stdenv.hostPlatform.isDarwin);
+              }
+            );
+
         };
     in
     {
@@ -235,6 +284,15 @@
         # Binary package for various platforms.
         build = forAllSystems (system: self.packages.${system}.nix);
 
+        # Ensure support for lowdown < 3.0 doesn't regress for NixOS 25.11
+        build-lowdown_2_0.aarch64-linux = lib.genAttrs [ "aarch64-linux" ] (
+          system:
+          self.packages.${system}.nix.override {
+            lowdown = nixpkgsFor.${system}.native.lowdown;
+            lowdown-unsandboxed = nixpkgsFor.${system}.native.lowdown-unsandboxed;
+          }
+        );
+
         devShell = forAllSystems (system: {
           default = self.devShells.${system}.default;
           clang = self.devShells.${system}.native-clangStdenvPackages;
@@ -381,13 +439,16 @@
               in
               pkgs.symlinkJoin {
                 name = "nixpkgs-lib-tests";
-                paths =
-                  [ testWithNix ]
-                  # NOTE: nixpkgs 24.11 is being ... *creative*, and requires this dance to override
-                  # the evaluator used for the test. it will break again in the future, don't worry.
-                  ++ lib.optionals pkgs.stdenv.isLinux [
-                    (pkgs.callPackage "${nixpkgs}/ci/eval" { nixVersions.nix_2_24 = nix; }).attrpathsSuperset
-                  ];
+                paths = [
+                  testWithNix
+                ]
+                # NOTE: nixpkgs 25.05 is being ... *creative*, and requires this dance to override
+                # the evaluator used for the test. it will break again in the future, don't worry.
+                ++ lib.optionals pkgs.stdenv.isLinux [
+                  ((pkgs.callPackage "${nixpkgs}/ci/eval" { inherit nix; }).attrpathsSuperset {
+                    evalSystem = system;
+                  })
+                ];
               }
             );
           };

lix/legacy/build-remote.cc

@@ -324,7 +324,7 @@ connected:
             //
             // 2. Changing the `inputSrcs` set changes the associated
             //    output ids, which break CA derivations
-            if (!drv.inputDrvs.empty())
+            if (!drv.inputDrvs.map.empty())
                 drv.inputSrcs = store->parseStorePathSet(inputs);
             optResult = aio.blockOn(sshStore->buildDerivation(*drvPath, (const BasicDerivation &) drv));
             auto & result = *optResult;
@@ -336,7 +336,7 @@ connected:
             ));
             auto res = aio.blockOn(sshStore->buildPathsWithResults({
                 DerivedPath::Built {
-                    .drvPath = makeConstantStorePath(*drvPath),
+                    .drvPath = makeConstantStorePathRef(*drvPath),
                     .outputs = OutputsSpec::All {},
                 }
             }));
@@ -346,11 +346,31 @@ connected:
         }
 
 
+        auto outputHashes = aio.blockOn(staticOutputHashes(*store, drv));
+        std::set<Realisation> missingRealisations;
         StorePathSet missingPaths;
-        auto outputPaths = drv.outputsAndPaths(*store);
-        for (auto & [outputName, outputPath] : outputPaths) {
-            if (!aio.blockOn(store->isValidPath(outputPath.second)))
-                missingPaths.insert(outputPath.second);
+        if (experimentalFeatureSettings.isEnabled(Xp::CaDerivations) && !drv.type().hasKnownOutputPaths()) {
+            for (auto & outputName : wantedOutputs) {
+                auto thisOutputHash = outputHashes.at(outputName);
+                auto thisOutputId = DrvOutput{ thisOutputHash, outputName };
+                if (!aio.blockOn(store->queryRealisation(thisOutputId))) {
+                    debug("missing output %s", outputName);
+                    assert(optResult);
+                    auto & result = *optResult;
+                    auto i = result.builtOutputs.find(outputName);
+                    assert(i != result.builtOutputs.end());
+                    auto & newRealisation = i->second;
+                    missingRealisations.insert(newRealisation);
+                    missingPaths.insert(newRealisation.outPath);
+                }
+            }
+        } else {
+            auto outputPaths = drv.outputsAndOptPaths(*store);
+            for (auto & [outputName, hopefullyOutputPath] : outputPaths) {
+                assert(hopefullyOutputPath.second);
+                if (!aio.blockOn(store->isValidPath(*hopefullyOutputPath.second)))
+                    missingPaths.insert(*hopefullyOutputPath.second);
+            }
         }
 
         if (!missingPaths.empty()) {
@@ -362,6 +382,13 @@ connected:
                 copyPaths(*sshStore, *store, missingPaths, NoRepair, NoCheckSigs, NoSubstitute)
             );
         }
+        // XXX: Should be done as part of `copyPaths`
+        for (auto & realisation : missingRealisations) {
+            // Should hold, because if the feature isn't enabled the set
+            // of missing realisations should be empty
+            experimentalFeatureSettings.require(Xp::CaDerivations);
+            aio.blockOn(store->registerDrvOutput(realisation));
+        }
 
         return 0;
     }

lix/legacy/nix-build.cc

@@ -187,7 +187,8 @@ static void main_nix_build(AsyncIoRoot & aio, std::string programName, Strings a
     if (packages && fromArgs)
         throw UsageError("'-p' and '-E' are mutually exclusive");
 
-    AutoDelete tmpDir(createTempDir("", myName));
+    AutoDelete tmpDir(createTempDir(myName));
+    AutoDelete buildTopTmpDir(createTempSubdir(tmpDir, "build-top"));
     if (outLink.empty())
         outLink = (Path) tmpDir + "/result";
 
@@ -355,7 +356,7 @@ static void main_nix_build(AsyncIoRoot & aio, std::string programName, Strings a
 
                 auto bashDrv = drv->requireDrvPath(*state);
                 pathsToBuild.push_back(DerivedPath::Built {
-                    .drvPath = makeConstantStorePath(bashDrv),
+                    .drvPath = makeConstantStorePathRef(bashDrv),
                     .outputs = OutputsSpec::Names {"out"},
                 });
                 pathsToCopy.insert(bashDrv);
@@ -368,16 +369,22 @@ static void main_nix_build(AsyncIoRoot & aio, std::string programName, Strings a
             }
         }
 
-        auto accumDerivedPath = [&](SingleDerivedPath::Opaque inputDrv, const StringSet & inputNode) {
-            if (!inputNode.empty())
+        std::function<void(ref<SingleDerivedPath>, const DerivedPathMap<StringSet>::ChildNode &)> accumDerivedPath;
+
+        accumDerivedPath = [&](ref<SingleDerivedPath> inputDrv, const DerivedPathMap<StringSet>::ChildNode & inputNode) {
+            if (!inputNode.value.empty())
                 pathsToBuild.push_back(DerivedPath::Built {
                     .drvPath = inputDrv,
-                    .outputs = OutputsSpec::Names { inputNode },
+                    .outputs = OutputsSpec::Names { inputNode.value },
                 });
+            for (const auto & [outputName, childNode] : inputNode.childMap)
+                accumDerivedPath(
+                    make_ref<SingleDerivedPath>(SingleDerivedPath::Built { inputDrv, outputName }),
+                    childNode);
         };
 
         // Build or fetch all dependencies of the derivation.
-        for (const auto & [inputDrv0, inputNode] : drv.inputDrvs) {
+        for (const auto & [inputDrv0, inputNode] : drv.inputDrvs.map) {
             // To get around lambda capturing restrictions in the
             // standard.
             const auto & inputDrv = inputDrv0;
@@ -386,7 +393,7 @@ static void main_nix_build(AsyncIoRoot & aio, std::string programName, Strings a
                         return !std::regex_search(store->printStorePath(inputDrv), regex::parse(exclude));
                     }))
             {
-                accumDerivedPath(makeConstantStorePath(inputDrv), inputNode);
+                accumDerivedPath(makeConstantStorePathRef(inputDrv), inputNode);
                 pathsToCopy.insert(inputDrv);
             }
         }
@@ -401,8 +408,14 @@ static void main_nix_build(AsyncIoRoot & aio, std::string programName, Strings a
 
         if (shellDrv) {
             auto shellDrvOutputs =
-                aio.blockOn(store->queryDerivationOutputMap(shellDrv.value(), &*evalStore));
-            shell = store->printStorePath(shellDrvOutputs.at("out")) + "/bin/bash";
+                aio.blockOn(store->queryPartialDerivationOutputMap(shellDrv.value(), &*evalStore));
+            shell = store->printStorePath(shellDrvOutputs.at("out").value()) + "/bin/bash";
+        }
+
+        if (experimentalFeatureSettings.isEnabled(Xp::CaDerivations)) {
+            auto resolvedDrv = aio.blockOn(drv.tryResolve(*store));
+            assert(resolvedDrv && "Successfully resolved the derivation");
+            drv = *resolvedDrv;
         }
 
         // Set the environment.
@@ -419,7 +432,8 @@ static void main_nix_build(AsyncIoRoot & aio, std::string programName, Strings a
         }
 
         // Don't use defaultTempDir() here! We want to preserve the user's TMPDIR for the shell
-        env["NIX_BUILD_TOP"] = env["TMPDIR"] = env["TEMPDIR"] = env["TMP"] = env["TEMP"] = getEnvNonEmpty("TMPDIR").value_or("/tmp");
+        env["NIX_BUILD_TOP"] = env["TMPDIR"] = env["TEMPDIR"] = env["TMP"] = env["TEMP"] =
+            getEnvNonEmpty("TMPDIR").value_or(buildTopTmpDir);
         env["NIX_STORE"] = store->config().storeDir;
         env["NIX_BUILD_CORES"] = std::to_string(settings.buildCores);
 
@@ -443,16 +457,20 @@ static void main_nix_build(AsyncIoRoot & aio, std::string programName, Strings a
         if (env.count("__json")) {
             StorePathSet inputs;
 
-            auto accumInputClosure = [&](const StorePath & inputDrv, const StringSet & inputNode) {
+            std::function<void(const StorePath &, const DerivedPathMap<StringSet>::ChildNode &)> accumInputClosure;
+
+            accumInputClosure = [&](const StorePath & inputDrv, const DerivedPathMap<StringSet>::ChildNode & inputNode) {
                 auto outputs =
-                    aio.blockOn(store->queryDerivationOutputMap(inputDrv, &*evalStore));
-                for (auto & i : inputNode) {
+                    aio.blockOn(store->queryPartialDerivationOutputMap(inputDrv, &*evalStore));
+                for (auto & i : inputNode.value) {
                     auto o = outputs.at(i);
-                    aio.blockOn(store->computeFSClosure(o, inputs));
+                    aio.blockOn(store->computeFSClosure(*o, inputs));
                 }
+                for (const auto & [outputName, childNode] : inputNode.childMap)
+                    accumInputClosure(*outputs.at(outputName), childNode);
             };
 
-            for (const auto & [inputDrv, inputNode] : drv.inputDrvs)
+            for (const auto & [inputDrv, inputNode] : drv.inputDrvs.map)
                 accumInputClosure(inputDrv, inputNode);
 
             ParsedDerivation parsedDrv(drvInfo.requireDrvPath(*state), drv);
@@ -565,7 +583,7 @@ static void main_nix_build(AsyncIoRoot & aio, std::string programName, Strings a
                 throw Error("derivation '%s' lacks an 'outputName' attribute", store->printStorePath(drvPath));
 
             pathsToBuild.push_back(DerivedPath::Built{
-                .drvPath = makeConstantStorePath(drvPath),
+                .drvPath = makeConstantStorePathRef(drvPath),
                 .outputs = OutputsSpec::Names{outputName},
             });
             pathsToBuildOrdered.push_back({drvPath, {outputName}});
@@ -591,9 +609,11 @@ static void main_nix_build(AsyncIoRoot & aio, std::string programName, Strings a
                 drvPrefix += fmt("-%d", counter + 1);
 
             auto builtOutputs =
-                aio.blockOn(store->queryDerivationOutputMap(drvPath, &*evalStore));
+                aio.blockOn(store->queryPartialDerivationOutputMap(drvPath, &*evalStore));
 
-            auto outputPath = builtOutputs.at(outputName);
+            auto maybeOutputPath = builtOutputs.at(outputName);
+            assert(maybeOutputPath);
+            auto outputPath = *maybeOutputPath;
 
             if (auto store2 = store.try_cast_shared<LocalFSStore>()) {
                 std::string symlink = drvPrefix;

lix/legacy/nix-env.cc

@@ -495,7 +495,7 @@ static void printMissing(EvalState & state, DrvInfos & elems)
     for (auto & i : elems)
         if (auto drvPath = i.queryDrvPath(state))
             targets.emplace_back(DerivedPath::Built{
-                .drvPath = makeConstantStorePath(*drvPath),
+                .drvPath = makeConstantStorePathRef(*drvPath),
                 .outputs = OutputsSpec::All { },
             });
         else
@@ -792,7 +792,7 @@ static void opSet(Globals & globals, Strings opFlags, Strings opArgs)
     std::vector<DerivedPath> paths {
         drvPath
         ? (DerivedPath) (DerivedPath::Built {
-            .drvPath = makeConstantStorePath(*drvPath),
+            .drvPath = makeConstantStorePathRef(*drvPath),
             .outputs = OutputsSpec::All { },
         })
         : (DerivedPath) (DerivedPath::Opaque {

lix/legacy/nix-store.cc

@@ -254,8 +254,10 @@ try {
         StorePathSet outputs;
         if (forceRealise)
             co_return TRY_AWAIT(store->queryDerivationOutputs(storePath));
-        for (auto & i : drv.outputsAndPaths(*store)) {
-            outputs.insert(i.second.second);
+        for (auto & i : drv.outputsAndOptPaths(*store)) {
+            if (!i.second.second)
+                throw UsageError("Cannot use output path of floating content-addressed derivation until we know what it is (e.g. by building it)");
+            outputs.insert(*i.second.second);
         }
         co_return outputs;
     }

lix/libcmd/built-path.cc

@@ -13,9 +13,9 @@ namespace nix {
     bool MY_TYPE ::operator COMPARATOR (const MY_TYPE & other) const \
     { \
         const MY_TYPE* me = this; \
-        auto fields1 = std::tie(me->drvPath, me->FIELD); \
+        auto fields1 = std::tie(*me->drvPath, me->FIELD); \
         me = &other; \
-        auto fields2 = std::tie(me->drvPath, me->FIELD); \
+        auto fields2 = std::tie(*me->drvPath, me->FIELD); \
         return fields1 COMPARATOR fields2; \
     }
 #define CMP(CHILD_TYPE, MY_TYPE, FIELD) \
@@ -23,6 +23,10 @@ namespace nix {
     CMP_ONE(CHILD_TYPE, MY_TYPE, FIELD, !=) \
     CMP_ONE(CHILD_TYPE, MY_TYPE, FIELD, <)
 
+#define FIELD_TYPE std::pair<std::string, StorePath>
+CMP(SingleBuiltPath, SingleBuiltPathBuilt, output)
+#undef FIELD_TYPE
+
 #define FIELD_TYPE std::map<std::string, StorePath>
 CMP(SingleBuiltPath, BuiltPathBuilt, outputs)
 #undef FIELD_TYPE
@@ -30,6 +34,16 @@ CMP(SingleBuiltPath, BuiltPathBuilt, outputs)
 #undef CMP
 #undef CMP_ONE
 
+StorePath SingleBuiltPath::outPath() const
+{
+    return std::visit(
+        overloaded{
+            [](const SingleBuiltPath::Opaque & p) { return p.path; },
+            [](const SingleBuiltPath::Built & b) { return b.output.second; },
+        }, raw()
+    );
+}
+
 StorePathSet BuiltPath::outPaths() const
 {
     return std::visit(
@@ -45,10 +59,32 @@ StorePathSet BuiltPath::outPaths() const
     );
 }
 
+SingleDerivedPath::Built SingleBuiltPath::Built::discardOutputPath() const
+{
+    return SingleDerivedPath::Built {
+        .drvPath = make_ref<SingleDerivedPath>(drvPath->discardOutputPath()),
+        .output = output.first,
+    };
+}
+
+SingleDerivedPath SingleBuiltPath::discardOutputPath() const
+{
+    return std::visit(
+        overloaded{
+            [](const SingleBuiltPath::Opaque & p) -> SingleDerivedPath {
+                return p;
+            },
+            [](const SingleBuiltPath::Built & b) -> SingleDerivedPath {
+                return b.discardOutputPath();
+            },
+        }, raw()
+    );
+}
+
 kj::Promise<Result<JSON>> BuiltPath::Built::toJSON(const Store & store) const
 try {
     JSON res;
-    res["drvPath"] = TRY_AWAIT(drvPath.toJSON(store));
+    res["drvPath"] = TRY_AWAIT(drvPath->toJSON(store));
     for (const auto & [outputName, outputPath] : outputs) {
         res["outputs"][outputName] = store.printStorePath(outputPath);
     }
@@ -57,6 +93,36 @@ try {
     co_return result::current_exception();
 }
 
+kj::Promise<Result<JSON>> SingleBuiltPath::Built::toJSON(const Store & store) const
+try {
+    JSON res;
+    res["drvPath"] = TRY_AWAIT(drvPath->toJSON(store));
+    auto & [outputName, outputPath] = output;
+    res["output"] = outputName;
+    res["outputPath"] = store.printStorePath(outputPath);
+    co_return res;
+} catch (...) {
+    co_return result::current_exception();
+}
+
+kj::Promise<Result<JSON>> SingleBuiltPath::toJSON(const Store & store) const
+try {
+    co_return TRY_AWAIT(std::visit([&](const auto & buildable) {
+        return buildable.toJSON(store);
+    }, raw()));
+} catch (...) {
+    co_return result::current_exception();
+}
+
+
+kj::Promise<Result<JSON>> BuiltPath::toJSON(const Store & store) const
+try {
+    co_return TRY_AWAIT(std::visit([&](const auto & buildable) {
+        return buildable.toJSON(store);
+    }, raw()));
+} catch (...) {
+    co_return result::current_exception();
+}
 
 kj::Promise<Result<RealisedPath::Set>> BuiltPath::toRealisedPaths(Store & store) const
 try {
@@ -75,10 +141,24 @@ try {
         [&](const BuiltPath::Built & p) -> kj::Promise<Result<void>> {
             try {
                 auto drvHashes = TRY_AWAIT(
-                    staticOutputHashes(store, TRY_AWAIT(store.readDerivation(p.drvPath.path)))
+                    staticOutputHashes(store, TRY_AWAIT(store.readDerivation(p.drvPath->outPath())))
                 );
                 for (auto& [outputName, outputPath] : p.outputs) {
-                    res.insert(outputPath);
+                    if (experimentalFeatureSettings.isEnabled(
+                                Xp::CaDerivations)) {
+                        auto drvOutput = get(drvHashes, outputName);
+                        if (!drvOutput)
+                            throw Error(
+                                "the derivation '%s' has unrealised output '%s' (derived-path.cc/toRealisedPaths)",
+                                store.printStorePath(p.drvPath->outPath()), outputName);
+                        auto thisRealisation = TRY_AWAIT(store.queryRealisation(
+                            DrvOutput{*drvOutput, outputName}));
+                        assert(thisRealisation);  // We’ve built it, so we must
+                                                  // have the realisation
+                        res.insert(*thisRealisation);
+                    } else {
+                        res.insert(outputPath);
+                    }
                 }
                 co_return result::success();
             } catch (...) {

lix/libcmd/built-path.hh

@@ -7,15 +7,63 @@
 
 namespace nix {
 
+struct SingleBuiltPath;
+
+struct SingleBuiltPathBuilt {
+    ref<SingleBuiltPath> drvPath;
+    std::pair<std::string, StorePath> output;
+
+    SingleDerivedPathBuilt discardOutputPath() const;
+
+    std::string to_string(const Store & store) const;
+    static SingleBuiltPathBuilt parse(const Store & store, std::string_view, std::string_view);
+    kj::Promise<Result<JSON>> toJSON(const Store & store) const;
+
+    DECLARE_CMP(SingleBuiltPathBuilt);
+};
+
+namespace built_path::detail {
+using SingleBuiltPathRaw = std::variant<
+    DerivedPathOpaque,
+    SingleBuiltPathBuilt
+>;
+}
+
+struct SingleBuiltPath : built_path::detail::SingleBuiltPathRaw {
+    using Raw = built_path::detail::SingleBuiltPathRaw;
+    using Raw::Raw;
+
+    using Opaque = DerivedPathOpaque;
+    using Built = SingleBuiltPathBuilt;
+
+    inline const Raw & raw() const {
+        return static_cast<const Raw &>(*this);
+    }
+
+    StorePath outPath() const;
+
+    SingleDerivedPath discardOutputPath() const;
+
+    static SingleBuiltPath parse(const Store & store, std::string_view);
+    kj::Promise<Result<JSON>> toJSON(const Store & store) const;
+};
+
+static inline ref<SingleBuiltPath> staticDrv(StorePath drvPath)
+{
+    return make_ref<SingleBuiltPath>(SingleBuiltPath::Opaque { drvPath });
+}
+
 /**
  * A built derived path with hints in the form of optional concrete output paths.
  *
  * See 'BuiltPath' for more an explanation.
  */
 struct BuiltPathBuilt {
-    DerivedPathOpaque drvPath;
+    ref<SingleBuiltPath> drvPath;
     std::map<std::string, StorePath> outputs;
 
+    std::string to_string(const Store & store) const;
+    static BuiltPathBuilt parse(const Store & store, std::string_view, std::string_view);
     kj::Promise<Result<JSON>> toJSON(const Store & store) const;
 
     DECLARE_CMP(BuiltPathBuilt);
@@ -45,6 +93,8 @@ struct BuiltPath : built_path::detail::BuiltPathRaw {
 
     StorePathSet outPaths() const;
     kj::Promise<Result<RealisedPath::Set>> toRealisedPaths(Store & store) const;
+
+    kj::Promise<Result<JSON>> toJSON(const Store & store) const;
 };
 
 typedef std::vector<BuiltPath> BuiltPaths;

lix/libcmd/installable-attr-path.cc

@@ -76,7 +76,7 @@ DerivedPathsWithInfo InstallableAttrPath::toDerivedPaths(EvalState & state)
     for (auto & [drvPath, outputs] : byDrvPath)
         res.push_back({
             .path = DerivedPath::Built {
-                .drvPath = makeConstantStorePath(drvPath),
+                .drvPath = makeConstantStorePathRef(drvPath),
                 .outputs = outputs,
             },
             .info = make_ref<ExtraPathInfoValue>(ExtraPathInfoValue::Value {

lix/libcmd/installable-derived-path.cc

@@ -35,7 +35,7 @@ InstallableDerivedPath InstallableDerivedPath::parse(
             // Remove this prior to stabilizing the new CLI.
             if (storePath.isDerivation()) {
                 auto oldDerivedPath = DerivedPath::Built {
-                    .drvPath = makeConstantStorePath(storePath),
+                    .drvPath = makeConstantStorePathRef(storePath),
                     .outputs = OutputsSpec::All { },
                 };
                 warn(
@@ -48,7 +48,8 @@ InstallableDerivedPath InstallableDerivedPath::parse(
         },
         // If the user did use ^, we just do exactly what is written.
         [&](const ExtendedOutputsSpec::Explicit & outputSpec) -> DerivedPath {
-            auto drv = DerivedPathOpaque::parse(*store, prefix);
+            auto drv = make_ref<SingleDerivedPath>(SingleDerivedPath::parse(*store, prefix));
+            drvRequireExperiment(*drv);
             return DerivedPath::Built {
                 .drvPath = std::move(drv),
                 .outputs = outputSpec,

lix/libcmd/installable-flake.cc

@@ -98,7 +98,7 @@ DerivedPathsWithInfo InstallableFlake::toDerivedPaths(EvalState & state)
 
     return {{
         .path = DerivedPath::Built {
-            .drvPath = makeConstantStorePath(std::move(drvPath)),
+            .drvPath = makeConstantStorePathRef(std::move(drvPath)),
             .outputs = std::visit(overloaded {
                 [&](const ExtendedOutputsSpec::Default & d) -> OutputsSpec {
                     std::set<std::string> outputsToInstall;

lix/libcmd/installables.cc

@@ -524,6 +524,36 @@ ref<Installable> SourceExprCommand::parseInstallable(
     return installables.front();
 }
 
+static kj::Promise<Result<SingleBuiltPath>> getBuiltPath(ref<Store> evalStore, ref<Store> store, const SingleDerivedPath & b)
+try {
+    auto handlers = overloaded{
+        [&](const SingleDerivedPath::Opaque & bo) -> kj::Promise<Result<SingleBuiltPath>> {
+            return {SingleBuiltPath::Opaque { bo.path }};
+        },
+        // NOLINTNEXTLINE(cppcoreguidelines-avoid-capturing-lambda-coroutines)
+        [&](const SingleDerivedPath::Built & bfd) -> kj::Promise<Result<SingleBuiltPath>> {
+            try {
+                auto drvPath = TRY_AWAIT(getBuiltPath(evalStore, store, *bfd.drvPath));
+                // Resolving this instead of `bfd` will yield the same result, but avoid duplicative work.
+                SingleDerivedPath::Built truncatedBfd {
+                    .drvPath = makeConstantStorePathRef(drvPath.outPath()),
+                    .output = bfd.output,
+                };
+                auto outputPath = TRY_AWAIT(resolveDerivedPath(*store, truncatedBfd, &*evalStore));
+                co_return SingleBuiltPath::Built {
+                    .drvPath = make_ref<SingleBuiltPath>(std::move(drvPath)),
+                    .output = { bfd.output, outputPath },
+                };
+            } catch (...) {
+                co_return result::current_exception();
+            }
+        },
+    };
+    co_return TRY_AWAIT(std::visit(handlers, b.raw()));
+} catch (...) {
+    co_return result::current_exception();
+}
+
 std::vector<BuiltPathWithResult> Installable::build(
     EvalState & state,
     ref<Store> evalStore,
@@ -612,7 +642,7 @@ std::vector<std::pair<ref<Installable>, BuiltPathWithResult>> Installable::build
                             state.aio.blockOn(resolveDerivedPath(*store, bfd, &*evalStore));
                         res.push_back({aux.installable, {
                             .path = BuiltPath::Built {
-                                .drvPath = bfd.drvPath,
+                                .drvPath = make_ref<SingleBuiltPath>(state.aio.blockOn(getBuiltPath(evalStore, store, *bfd.drvPath))),
                                 .outputs = outputs,
                              },
                             .info = aux.info}});
@@ -644,7 +674,7 @@ std::vector<std::pair<ref<Installable>, BuiltPathWithResult>> Installable::build
                             outputs.emplace(outputName, realisation.outPath);
                         res.push_back({aux.installable, {
                             .path = BuiltPath::Built {
-                                .drvPath = bfd.drvPath,
+                                .drvPath = make_ref<SingleBuiltPath>(state.aio.blockOn(getBuiltPath(evalStore, store, *bfd.drvPath))),
                                 .outputs = outputs,
                             },
                             .info = aux.info,
@@ -759,7 +789,7 @@ StorePathSet Installable::toDerivations(
                         : throw Error("argument '%s' did not evaluate to a derivation", i->what()));
                 },
                 [&](const DerivedPath::Built & bfd) {
-                    drvPaths.insert(bfd.drvPath.path);
+                    drvPaths.insert(state.aio.blockOn(resolveDerivedPath(*store, *bfd.drvPath)));
                 },
             }, b.path.raw());
 

lix/libcmd/markdown.cc

@@ -8,20 +8,39 @@
 
 namespace nix {
 
-std::string renderMarkdownToTerminal(std::string_view markdown, StandardOutputStream fileno)
+std::string renderMarkdownToTerminal(std::string_view markdown)
 {
     int windowWidth = getWindowSize().second;
+    size_t lowdown_cols = std::max(windowWidth - 5, 60);
 
-    struct lowdown_opts opts {
+    struct lowdown_opts opts{
         .type = LOWDOWN_TERM,
+#ifdef LOWDOWN_SEPARATE_TERM_OPTS
+        .term =
+            {
+                .cols = lowdown_cols,
+                .width = 0,
+                .hmargin = 0,
+                .hpadding = 4,
+                .vmargin = 0,
+                .centre = 0,
+            },
+        // maxdepth needs to be part of the ifdefs to match declaration order
         .maxdepth = 20,
-        .cols = (size_t) std::max(windowWidth - 5, 60),
+#else
+        .maxdepth = 20,
+        .cols = lowdown_cols,
         .hmargin = 0,
         .vmargin = 0,
+#endif /* LOWDOWN_SEPARATE_TERM_OPTS */
         .feat = LOWDOWN_COMMONMARK | LOWDOWN_FENCED | LOWDOWN_DEFLIST | LOWDOWN_TABLES,
+#ifdef LOWDOWN_CONSOLIDATED_OFLAGS
+        .oflags = LOWDOWN_NOLINK,
+#else
         .oflags = LOWDOWN_TERM_NOLINK,
+#endif /* LOWDOWN_CONSOLIDATED_OFLAGS */
     };
-    if (!shouldANSI(fileno)) {
+    if (!shouldANSI()) {
         opts.oflags |= LOWDOWN_TERM_NOANSI;
     }
 

lix/libcmd/markdown.hh

@@ -1,11 +1,10 @@
 #pragma once
 ///@file
 
-#include "lix/libutil/terminal.hh"
 #include "lix/libutil/types.hh"
 
 namespace nix {
 
-std::string renderMarkdownToTerminal(std::string_view markdown, StandardOutputStream fileno = StandardOutputStream::Stdout);
+std::string renderMarkdownToTerminal(std::string_view markdown);
 
 }

lix/libcmd/repl.cc

@@ -254,7 +254,7 @@ void runNix(Path program, const Strings & args)
         .program = settings.nixBinDir+ "/" + program,
         .args = args,
         .environment = subprocessEnv,
-    }).wait();
+    }).waitAndCheck();
 
     return;
 }
@@ -672,7 +672,7 @@ ProcessLineResult NixRepl::processLine(std::string line)
 
         // runProgram redirects stdout to a StringSink,
         // using runProgram2 to allow editors to display their UI
-        runProgram2(RunOptions { .program = editor, .searchPath = true, .args = args }).wait();
+        runProgram2(RunOptions { .program = editor, .searchPath = true, .args = args }).waitAndCheck();
 
         // Reload right after exiting the editor if path is not in store
         // Store is immutable, so there could be no changes, so there's no need to reload
@@ -760,7 +760,7 @@ ProcessLineResult NixRepl::processLine(std::string line)
 
             state.aio.blockOn(evaluator.store->buildPaths({
                 DerivedPath::Built {
-                    .drvPath = makeConstantStorePath(drvPath),
+                    .drvPath = makeConstantStorePathRef(drvPath),
                     .outputs = OutputsSpec::All { },
                 },
             }));
@@ -1056,24 +1056,19 @@ template<typename T, typename NameFn, typename ValueFn>
 void NixRepl::addToScope(T && things, NameFn nameFn, ValueFn valueFn)
 {
     size_t added = 0;
+    for (auto && thing : things) {
+        if (displ + 1 >= envSize)
+            throw Error("environment full; cannot add more variables");
 
-    staticEnv->vars.unsafe_insert_bulk([&] (auto & map) {
-        auto oldSize = map.size();
-        for (auto && thing : things) {
-            if (displ + 1 >= envSize)
-                throw Error("environment full; cannot add more variables");
-
-            const auto name = nameFn(thing);
-            map.emplace_back(name, displ);
-            env->values[displ++] = valueFn(thing);
-            varNames.emplace(evaluator.symbols[name]);
-            added++;
-        }
-        // safety: we sort the range that we inserted so that we don't have to push that
-        // invariant up to the caller
-        std::sort(map.begin() + oldSize, map.end());
-    });
+        const auto name = nameFn(thing);
+        staticEnv->vars.emplace_back(name, displ);
+        env->values[displ++] = valueFn(thing);
+        varNames.emplace(evaluator.symbols[name]);
+        added++;
+    }
 
+    staticEnv->sort();
+    staticEnv->deduplicate();
     if (added > 0) {
         notice("Added %1% variables.", added);
     }
@@ -1098,11 +1093,14 @@ void NixRepl::addVarToScope(const Symbol name, Value & v)
 {
     if (displ >= envSize)
         throw Error("environment full; cannot add more variables");
-    if (staticEnv->vars.insert_or_assign(name, displ).second) {
+    if (auto oldVar = staticEnv->find(name); oldVar != staticEnv->vars.end()) {
+        staticEnv->vars.erase(oldVar);
         notice("Updated %s.", evaluator.symbols[name]);
     } else {
         notice("Added %s.", evaluator.symbols[name]);
     }
+    staticEnv->vars.emplace_back(name, displ);
+    staticEnv->sort();
     env->values[displ++] = &v;
     varNames.emplace(evaluator.symbols[name]);
 }

lix/libexpr/builtins/fetchGit.md

@@ -48,69 +48,6 @@ attribute with the following attributes (all except `url` optional):
   With this argument being true, it's possible to load a `rev` from *any* `ref`
   (by default only `rev`s from the specified `ref` are supported).
 
-- `narHash`
-
-  If given, the source is first looked-up in the Nix store and the [substituters](@docroot@/command-ref/conf-file.md#conf-substituters), and only fetched if not available.
-
-The return value is an attrset containing the following keys:
-
-- `lastModified` (`integer`)
-
-  Unix timestamp of the last update.
-  This corresponds to the timestamp of the "committer" timestamp embedded in the fetched commit.
-
-- `lastModifiedDate` (`string`)
-
-  Textual representation of the `lastModified` timestamp in UTC (the timezone embedded in the git commit is discarded).
-
-- `outPath` (`string`)
-
-  Resulting store path of the fetch process.
-
-- `narHash` (`string`)
-
-  SRI representation of the hash of the `outPath`.
-
-- `rev` (`string`)
-
-  The full-length revision fetched from the remote.
-  For further information see the `rev` input parameter.
-  This will usually be the output of `git rev-parse <rev>` (or `ref` when no `rev` is provided as an input parameter).
-
-- `revCount` (`integer`)
-
-  Number of revisions in the history of the revision fetched.
-  For a repository with a single commit (the root) this number equals 1.
-  Fetches of shallow repositories report a value of 0.
-
-- `shortRev` (`string`)
-
-  A short representation of the `rev`.
-  This string is a *truncated* version of the `rev`.
-  It is of fixed length and therefore not guaranteed to be unique (unlike the output of `git rev-parse --short`).
-  Future versions of Lix may change the length of this string only as part of a breaking change.
-  For maximum reproducibility and interoperability it is recommended to not rely on this value and to truncate the returned `rev` to an appropriate value instead.
-
-- `submodules` (`boolean`)
-
-  Indicates whether submodules have been fetched.
-  If this value is set to `true`, any submodules are already checked out in the resulting `outPath`.
-
-A full example of the output:
-
-```nix
-{
-  lastModified = 1746827286;
-  lastModifiedDate = "20250509214806";
-  narHash = "sha256-qCRBy8Bbh5XhPalPkhonxNgfsbw3lP0UIXBLSrhxAvI=";
-  outPath = "/nix/store/2qdnzhzccspwm70mni7jkvrfkpwcb3jn-source";
-  rev = "dcb0a97000d50b2868ed4f8d9fd465c5a5b8eb3a";
-  revCount = 17845;
-  shortRev = "dcb0a97";
-  submodules = false;
-}
-```
-
 Here are some examples of how to use `fetchGit`.
 
   - To fetch a private repository over SSH:

lix/libexpr/builtins/outputOf.md

@@ -0,0 +1,23 @@
+---
+name: outputOf
+args: [derivation-reference, output-name]
+experimentalFeature: dynamic-derivations
+---
+Return the output path of a derivation, literally or using a placeholder if needed.
+
+If the derivation has a statically-known output path (i.e. the derivation output is input-addressed, or fixed content-addressed), the output path will just be returned.
+But if the derivation is content-addressed or if the derivation is itself not-statically produced (i.e. is the output of another derivation), a placeholder will be returned instead.
+
+*`derivation reference`* must be a string that may contain a regular store path to a derivation, or may be a placeholder reference. If the derivation is produced by a derivation, you must explicitly select `drv.outPath`.
+This primop can be chained arbitrarily deeply.
+For instance,
+
+```nix
+builtins.outputOf
+  (builtins.outputOf myDrv "out)
+  "out"
+```
+
+will return a placeholder for the output of the output of `myDrv`.
+
+This primop corresponds to the `^` sigil for derivable paths, e.g. as part of installable syntax on the command line.

lix/libexpr/eval-cache.cc

@@ -580,7 +580,7 @@ string_t AttrCursor::getStringWithContext(EvalState & state)
                             return d.drvPath;
                         },
                         [&](const NixStringContextElem::Built & b) -> const StorePath & {
-                            return b.drvPath.path;
+                            return b.drvPath->getBaseStorePath();
                         },
                         [&](const NixStringContextElem::Opaque & o) -> const StorePath & {
                             return o.path;

lix/libexpr/eval.cc

@@ -14,6 +14,7 @@
 #include "lix/libutil/types.hh"
 #include "lix/libstore/store-api.hh"
 #include "lix/libstore/derivations.hh"
+#include "lix/libstore/downstream-placeholder.hh"
 #include "lix/libexpr/gc-alloc.hh"
 #include "lix/libstore/filetransfer.hh"
 #include "lix/libexpr/function-trace.hh"
@@ -548,7 +549,7 @@ void EvalBuiltins::addConstant(const std::string & name, Value * v, Constant inf
             assert(info.type == gotType);
 
         /* Install value the base environment. */
-        staticEnv->vars.insert_or_assign(symbols.create(name), baseEnvDispl);
+        staticEnv->vars.emplace_back(symbols.create(name), baseEnvDispl);
         env.values[baseEnvDispl++] = v;
         env.values[0]->attrs->push_back(Attr(symbols.create(name2), v));
     }
@@ -583,7 +584,7 @@ Value * EvalBuiltins::addPrimOp(PrimOp && primOp)
 
     Value * v = mem.allocValue();
     v->mkPrimOp(new PrimOp(primOp));
-    staticEnv->vars.insert_or_assign(auto(envName), baseEnvDispl);
+    staticEnv->vars.emplace_back(envName, baseEnvDispl);
     env.values[baseEnvDispl++] = v;
     env.values[0]->attrs->push_back(Attr(symbols.create(primOp.name), v));
     return v;
@@ -887,18 +888,29 @@ void EvalPaths::mkStorePathString(const StorePath & p, Value & v)
 
 
 std::string EvalState::mkOutputStringRaw(
-    const StorePath & staticOutputPath)
+    const SingleDerivedPath::Built & b,
+    std::optional<StorePath> optStaticOutputPath,
+    const ExperimentalFeatureSettings & xpSettings)
 {
-    return ctx.store->printStorePath(staticOutputPath);
+    /* In practice, this is testing for the case of CA derivations, or
+       dynamic derivations. */
+    return optStaticOutputPath
+        ? ctx.store->printStorePath(std::move(*optStaticOutputPath))
+        /* Downstream we would substitute this for an actual path once
+           we build the floating CA derivation */
+        : DownstreamPlaceholder::fromSingleDerivedPathBuilt(b, xpSettings).render();
 }
 
 
 void EvalState::mkOutputString(
     Value & value,
     const SingleDerivedPath::Built & b,
-    const StorePath & staticOutputPath)
+    std::optional<StorePath> optStaticOutputPath,
+    const ExperimentalFeatureSettings & xpSettings)
 {
-    value.mkString(mkOutputStringRaw(staticOutputPath), NixStringContext { b });
+    value.mkString(
+        mkOutputStringRaw(b, optStaticOutputPath, xpSettings),
+        NixStringContext { b });
 }
 
 
@@ -910,12 +922,19 @@ std::string EvalState::mkSingleDerivedPathStringRaw(
             return ctx.store->printStorePath(o.path);
         },
         [&](const SingleDerivedPath::Built & b) {
-            auto drv = aio.blockOn(ctx.store->readDerivation(b.drvPath.path));
-            auto i = drv.outputs.find(b.output);
-            if (i == drv.outputs.end())
-                throw Error("derivation '%s' does not have output '%s'", b.drvPath.to_string(*ctx.store), b.output);
-            auto staticOutputPath = i->second.path(*ctx.store, drv.name, b.output);
-            return mkOutputStringRaw(staticOutputPath);
+            auto optStaticOutputPath = std::visit(overloaded {
+                [&](const SingleDerivedPath::Opaque & o) {
+                    auto drv = aio.blockOn(ctx.store->readDerivation(o.path));
+                    auto i = drv.outputs.find(b.output);
+                    if (i == drv.outputs.end())
+                        throw Error("derivation '%s' does not have output '%s'", b.drvPath->to_string(*ctx.store), b.output);
+                    return i->second.path(*ctx.store, drv.name, b.output);
+                },
+                [&](const SingleDerivedPath::Built & o) -> std::optional<StorePath> {
+                    return std::nullopt;
+                },
+            }, b.drvPath->raw());
+            return mkOutputStringRaw(b, optStaticOutputPath);
         }
     }, p.raw());
 }
@@ -1954,7 +1973,7 @@ void ExprConcatStrings::eval(EvalState & state, Env & env, Value & v)
     NixInt n{0};
     NixFloat nf = 0;
 
-    bool first = !isInterpolation;
+    bool first = !forceString;
     ValueType firstType = nString;
 
     const auto str = [&] {
@@ -2017,17 +2036,12 @@ void ExprConcatStrings::eval(EvalState & state, Env & env, Value & v)
                 state.ctx.errors.make<EvalError>("cannot add %1% to a float", showType(vTmp)).atPos(i_pos).withFrame(env, *this).debugThrow();
         } else {
             if (s.empty()) s.reserve(es.size());
-
-            /* If we are coercing inside of an interpolation, we may allow slightly more comfort by coercing things like integers. */
-            auto coercionMode = isInterpolation && featureSettings.isEnabled(Xp::CoerceIntegers)
-                ? StringCoercionMode::Interpolation : StringCoercionMode::Strict;
-
             /* skip canonization of first path, which would only be not
             canonized in the first place if it's coming from a ./${foo} type
             path */
             auto part = state.coerceToString(i_pos, vTmp, context,
                                              "while evaluating a path segment",
-                                             coercionMode, firstType == nString, !first);
+                                             false, firstType == nString, !first);
             sSize += part->size();
             s.emplace_back(std::move(part));
         }
@@ -2260,7 +2274,7 @@ bool EvalState::isDerivation(Value & v)
 
 
 std::optional<std::string> EvalState::tryAttrsToString(const PosIdx pos, Value & v,
-    NixStringContext & context, StringCoercionMode mode, bool copyToStore)
+    NixStringContext & context, bool coerceMore, bool copyToStore)
 {
     auto i = v.attrs->find(ctx.s.toString);
     if (i != v.attrs->end()) {
@@ -2269,7 +2283,7 @@ std::optional<std::string> EvalState::tryAttrsToString(const PosIdx pos, Value &
             callFunction(*i->value, v, v1, i->pos);
             return coerceToString(pos, v1, context,
                     "while evaluating the result of the `__toString` attribute",
-                    mode, copyToStore).toOwned();
+                    coerceMore, copyToStore).toOwned();
         } catch (EvalError & e) {
             e.addTrace(ctx.positions[pos], "while converting a set to string");
             throw;
@@ -2284,7 +2298,7 @@ BackedStringView EvalState::coerceToString(
     Value & v,
     NixStringContext & context,
     std::string_view errorCtx,
-    StringCoercionMode mode,
+    bool coerceMore,
     bool copyToStore,
     bool canonicalizePath)
 {
@@ -2308,7 +2322,7 @@ BackedStringView EvalState::coerceToString(
     }
 
     if (v.type() == nAttrs) {
-        auto maybeString = tryAttrsToString(pos, v, context, mode, copyToStore);
+        auto maybeString = tryAttrsToString(pos, v, context, coerceMore, copyToStore);
         if (maybeString)
             return std::move(*maybeString);
         auto i = v.attrs->find(ctx.s.outPath);
@@ -2322,29 +2336,24 @@ BackedStringView EvalState::coerceToString(
                 .debugThrow();
         }
         return coerceToString(pos, *i->value, context, errorCtx,
-                              mode, copyToStore, canonicalizePath);
+                              coerceMore, copyToStore, canonicalizePath);
     }
 
     if (v.type() == nExternal) {
         try {
-            return v.external->coerceToString(*this, pos, context, mode, copyToStore);
+            return v.external->coerceToString(*this, pos, context, coerceMore, copyToStore);
         } catch (Error & e) {
             e.addTrace(nullptr, errorCtx);
             throw;
         }
     }
 
-    /* Raito: Any addition to this mode is subject to extra scrutiny
-     * until we have better formatting tools. */
-    if (mode >= StringCoercionMode::Interpolation) {
-        if (v.type() == nInt) return std::to_string(v.integer.value);
-    }
-
-    if (mode >= StringCoercionMode::ToString) {
+    if (coerceMore) {
         /* Note that `false' is represented as an empty string for
            shell scripting convenience, just like `null'. */
         if (v.type() == nBool && v.boolean) return "1";
         if (v.type() == nBool && !v.boolean) return "";
+        if (v.type() == nInt) return std::to_string(v.integer.value);
         if (v.type() == nFloat) return std::to_string(v.fpoint);
         if (v.type() == nNull) return "";
 
@@ -2354,7 +2363,7 @@ BackedStringView EvalState::coerceToString(
                 try {
                     result += *coerceToString(pos, *v2, context,
                             "while evaluating one element of the list",
-                            mode, copyToStore, canonicalizePath);
+                            coerceMore, copyToStore, canonicalizePath);
                 } catch (Error & e) {
                     e.addTrace(ctx.positions[pos], errorCtx);
                     throw;
@@ -2411,7 +2420,7 @@ try {
 
 SourcePath EvalState::coerceToPath(const PosIdx pos, Value & v, NixStringContext & context, std::string_view errorCtx)
 {
-    auto path = coerceToString(pos, v, context, errorCtx, StringCoercionMode::Strict, false, true).toOwned();
+    auto path = coerceToString(pos, v, context, errorCtx, false, false, true).toOwned();
     if (path == "" || path[0] != '/')
         ctx.errors.make<EvalError>("string '%1%' doesn't represent an absolute path", path).withTrace(pos, errorCtx).debugThrow();
     return CanonPath(path);
@@ -2420,7 +2429,7 @@ SourcePath EvalState::coerceToPath(const PosIdx pos, Value & v, NixStringContext
 
 StorePath EvalState::coerceToStorePath(const PosIdx pos, Value & v, NixStringContext & context, std::string_view errorCtx)
 {
-    auto path = coerceToString(pos, v, context, errorCtx, StringCoercionMode::Strict, false, true).toOwned();
+    auto path = coerceToString(pos, v, context, errorCtx, false, false, true).toOwned();
     if (auto storePath = ctx.store->maybeParseStorePath(path))
         return *storePath;
     ctx.errors.make<EvalError>("path '%1%' is not in the Nix store", path).withTrace(pos, errorCtx).debugThrow();
@@ -2475,7 +2484,7 @@ SingleDerivedPath EvalState::coerceToSingleDerivedPath(const PosIdx pos, Value &
             [&](const SingleDerivedPath::Built & b) {
                 ctx.errors.make<EvalError>(
                     "string '%s' has context with the output '%s' from derivation '%s', but the string is not the right placeholder for this derivation output. It should be '%s'",
-                    s, b.output, b.drvPath.to_string(*ctx.store), sExpected)
+                    s, b.output, b.drvPath->to_string(*ctx.store), sExpected)
                     .withTrace(pos, errorCtx).debugThrow(always_progresses);
             }
         }, derivedPath.raw());
@@ -2882,7 +2891,7 @@ try {
 }
 
 
-std::string ExternalValueBase::coerceToString(EvalState & state, const PosIdx & pos, NixStringContext & context, StringCoercionMode mode, bool copyToStore) const
+std::string ExternalValueBase::coerceToString(EvalState & state, const PosIdx & pos, NixStringContext & context, bool copyMore, bool copyToStore) const
 {
     state.ctx.errors.make<TypeError>(
         "cannot coerce %1% to a string: %2%", showType(), *this

lix/libexpr/eval.hh

@@ -767,18 +767,19 @@ public:
     bool isDerivation(Value & v);
 
     std::optional<std::string> tryAttrsToString(const PosIdx pos, Value & v,
-        NixStringContext & context, StringCoercionMode mode = StringCoercionMode::Strict, bool copyToStore = true);
+        NixStringContext & context, bool coerceMore = false, bool copyToStore = true);
 
     /**
      * String coercion.
      *
      * Converts strings, paths and derivations to a
-     * string. If `copyToStore` is set,
+     * string.  If `coerceMore` is set, also converts nulls, integers,
+     * booleans and lists to a string.  If `copyToStore` is set,
      * referenced paths are copied to the Nix store as a side effect.
      */
     BackedStringView coerceToString(const PosIdx pos, Value & v, NixStringContext & context,
         std::string_view errorCtx,
-        StringCoercionMode mode = StringCoercionMode::Strict, bool copyToStore = true,
+        bool coerceMore = false, bool copyToStore = true,
         bool canonicalizePath = true);
 
     /**
@@ -803,7 +804,8 @@ public:
     /**
      * Coerce to `SingleDerivedPath`.
      *
-     * Must be a string which is either a literal store path.
+     * Must be a string which is either a literal store path or a
+     * "placeholder (see `DownstreamPlaceholder`).
      *
      * Even more importantly, the string context must be exactly one
      * element, which is either a `NixStringContextElem::Opaque` or
@@ -868,13 +870,19 @@ public:
      * @param b the drv whose output we are making a string for, and the
      * output
      *
-     * @param staticOutputPath Output path for that string.
-     * Will be printed to form string.
+     * @param optStaticOutputPath Optional output path for that string.
+     * Must be passed if and only if output store object is
+     * input-addressed or fixed output. Will be printed to form string
+     * if passed, otherwise a placeholder will be used (see
+     * `DownstreamPlaceholder`).
+     *
+     * @param xpSettings Stop-gap to avoid globals during unit tests.
      */
     void mkOutputString(
         Value & value,
         const SingleDerivedPath::Built & b,
-        const StorePath & staticOutputPath);
+        std::optional<StorePath> optStaticOutputPath,
+        const ExperimentalFeatureSettings & xpSettings = experimentalFeatureSettings);
 
     /**
      * Create a string representing a `SingleDerivedPath`.
@@ -894,7 +902,9 @@ private:
      * string Value, which would also have a string context.
      */
     std::string mkOutputStringRaw(
-        const StorePath & staticOutputPath);
+        const SingleDerivedPath::Built & b,
+        std::optional<StorePath> optStaticOutputPath,
+        const ExperimentalFeatureSettings & xpSettings = experimentalFeatureSettings);
 
     /**
      * Like `mkSingleDerivedPathStringRaw` but just creates a raw string

lix/libexpr/fetchurl.nix

@@ -34,7 +34,6 @@
   executable ? false,
   unpack ? false,
   name ? baseNameOf (toString url),
-  # still translates to __impure to trigger derivationStrict error checks.
   impure ? false,
 }:
 

lix/libexpr/flake/flake.cc

@@ -298,7 +298,7 @@ static Flake getFlake(
                 NixStringContext emptyContext = {};
                 flake.config.settings.emplace(
                     state.ctx.symbols[setting.name],
-                    state.coerceToString(setting.pos, *setting.value, emptyContext, "", StringCoercionMode::Strict, true, true) .toOwned());
+                    state.coerceToString(setting.pos, *setting.value, emptyContext, "", false, true, true) .toOwned());
             }
             else if (setting.value->type() == nInt)
                 flake.config.settings.emplace(

lix/libexpr/meson.build

@@ -115,6 +115,7 @@ builtin_definitions = files(
   'builtins/mapAttrs.md',
   'builtins/match.md',
   'builtins/mul.md',
+  'builtins/outputOf.md',
   'builtins/parseDrvName.md',
   'builtins/parseFlakeRef.md',
   'builtins/partition.md',

lix/libexpr/nixexpr.cc

@@ -273,7 +273,7 @@ JSON ExprConcatStrings::toJSON(const SymbolTable & symbols) const
         parts.push_back(part->toJSON(symbols));
     return {
         {"_type", "ExprConcatStrings"},
-        {"isInterpolation", isInterpolation},
+        {"forceString", forceString},
         {"es", parts}
     };
 }
@@ -465,7 +465,7 @@ void VarBinder::visit(ExprVar & e, std::unique_ptr<Expr> & ptr)
         if (curEnv->isWith) {
             if (withLevel == -1) withLevel = level;
         } else {
-            auto i = curEnv->vars.find(e.name);
+            auto i = curEnv->find(e.name);
             if (i != curEnv->vars.end()) {
                 if (e.needsRoot && !curEnv->isRoot) {
                     throw ParseError({
@@ -523,12 +523,9 @@ std::shared_ptr<const StaticEnv> ExprAttrs::buildRecursiveEnv(const std::shared_
 {
     auto newEnv = std::make_shared<StaticEnv>(nullptr, env.get(), attrs.size());
 
-    // safety: the attrs is already sorted
-    newEnv->vars.unsafe_insert_bulk([&] (auto & map) {
-        Displacement displ = 0;
-        for (auto & i : attrs)
-            map.emplace_back(i.first, i.second.displ = displ++);
-    });
+    Displacement displ = 0;
+    for (auto & i : attrs)
+        newEnv->vars.emplace_back(i.first, i.second.displ = displ++);
     return newEnv;
 }
 
@@ -665,7 +662,7 @@ void VarBinder::visit(ExprPos & e, std::unique_ptr<Expr> & ptr)
 std::shared_ptr<const StaticEnv> SimplePattern::buildEnv(const StaticEnv * up)
 {
     auto newEnv = std::make_shared<StaticEnv>(nullptr, up, 1);
-    newEnv->vars.insert_or_assign(name, 0);
+    newEnv->vars.emplace_back(name, 0);
     return newEnv;
 }
 
@@ -680,14 +677,12 @@ std::shared_ptr<const StaticEnv> AttrsPattern::buildEnv(const StaticEnv * up)
 
     Displacement displ = 0;
 
-    if (name) newEnv->vars.insert_or_assign(name, displ++);
+    if (name) newEnv->vars.emplace_back(name, displ++);
 
-    // safety: The formals are already sorted
-    newEnv->vars.unsafe_insert_bulk([&] (auto & map) {
-        for (auto & i : formals)
-            map.emplace_back(i.name, displ++);
-    });
+    for (auto & i : formals)
+        newEnv->vars.emplace_back(i.name, displ++);
 
+    newEnv->sort();
     return newEnv;
 }
 

lix/libexpr/nixexpr.hh

@@ -11,7 +11,6 @@
 #include "lix/libexpr/eval-error.hh"
 #include "lix/libexpr/pos-idx.hh"
 #include "lix/libutil/strings.hh"
-#include "lix/libutil/linear-map.hh"
 
 namespace nix {
 
@@ -313,7 +312,7 @@ struct ExprAttrs
         AttrDef() { };
 
         template<typename T>
-        const T & chooseByKind(const T & plain, const T & inherited, const T & inheritedFrom) const
+        T chooseByKind(const T & plain, const T & inherited, const T & inheritedFrom) const
         {
             switch (kind) {
             case Kind::Plain:
@@ -573,10 +572,10 @@ MakeBinOp(ExprOpConcatLists, "++")
 
 struct ExprConcatStrings : Expr
 {
-    bool isInterpolation;
+    bool forceString;
     std::vector<std::pair<PosIdx, std::unique_ptr<Expr>>> es;
-    ExprConcatStrings(const PosIdx & pos, bool isInterpolation, std::vector<std::pair<PosIdx, std::unique_ptr<Expr>>> es)
-        : Expr(pos), isInterpolation(isInterpolation), es(std::move(es)) { };
+    ExprConcatStrings(const PosIdx & pos, bool forceString, std::vector<std::pair<PosIdx, std::unique_ptr<Expr>>> es)
+        : Expr(pos), forceString(forceString), es(std::move(es)) { };
     JSON toJSON(const SymbolTable & symbols) const override;
     void eval(EvalState & state, Env & env, Value & v) override;
     void accept(ExprVisitor & ev, std::unique_ptr<Expr> & ptr) override { ev.visit(*this, ptr); }
@@ -608,7 +607,9 @@ struct StaticEnv
     ExprWith * isWith;
     const StaticEnv * up;
 
-    LinearMap<Symbol, Displacement> vars;
+    // Note: these must be in sorted order.
+    typedef std::vector<std::pair<Symbol, Displacement>> Vars;
+    Vars vars;
 
     /* See ExprVar::needsRoot */
     bool isRoot = false;
@@ -616,6 +617,31 @@ struct StaticEnv
     StaticEnv(ExprWith * isWith, const StaticEnv * up, size_t expectedSize = 0) : isWith(isWith), up(up) {
         vars.reserve(expectedSize);
     };
+
+    void sort()
+    {
+        std::stable_sort(vars.begin(), vars.end(),
+            [](const Vars::value_type & a, const Vars::value_type & b) { return a.first < b.first; });
+    }
+
+    void deduplicate()
+    {
+        auto it = vars.begin(), jt = it, end = vars.end();
+        while (jt != end) {
+            *it = *jt++;
+            while (jt != end && it->first == jt->first) *it = *jt++;
+            it++;
+        }
+        vars.erase(it, end);
+    }
+
+    Vars::const_iterator find(Symbol name) const
+    {
+        Vars::value_type key(name, 0);
+        auto i = std::lower_bound(vars.begin(), vars.end(), key);
+        if (i != vars.end() && i->first == name) return i;
+        return vars.end();
+    }
 };
 
 

lix/libexpr/parser/grammar.hh

@@ -686,10 +686,12 @@ struct nothing : p::nothing<Rule> {
     static_assert(!std::is_base_of_v<semantic, Rule>);
 };
 
-
-
 template<typename Self, typename OpCtx, typename AttrPathT, typename ExprT>
-struct operator_semantics {
+struct operator_semantics
+{
+private:
+    operator_semantics() = default;
+    friend Self;
 public:
     struct has_attr : grammar::v1::op::has_attr {
         AttrPathT path;
@@ -775,5 +777,4 @@ public:
         return popExpr();
     }
 };
-
 }

lix/libexpr/primops.cc

@@ -1,5 +1,6 @@
 #include "lix/libutil/archive.hh"
 #include "lix/libstore/derivations.hh"
+#include "lix/libstore/downstream-placeholder.hh"
 #include "lix/libexpr/eval.hh"
 #include "lix/libexpr/eval-settings.hh"
 #include "lix/libexpr/extra-primops.hh"
@@ -57,7 +58,7 @@ StringMap EvalState::realiseContext(const NixStringContext & context)
                     .drvPath = b.drvPath,
                     .outputs = OutputsSpec::Names { b.output },
                 });
-                return ensureValid(b.drvPath.path);
+                return ensureValid(b.drvPath->getBaseStorePath());
             },
             [&](const NixStringContextElem::Opaque & o) {
                 auto ctxS = ctx.store->printStorePath(o.path);
@@ -92,6 +93,18 @@ StringMap EvalState::realiseContext(const NixStringContext & context)
         auto outputs = aio.blockOn(resolveDerivedPath(*ctx.buildStore, drv, &*ctx.store));
         for (auto & [outputName, outputPath] : outputs) {
             outputsToCopyAndAllow.insert(outputPath);
+
+            /* Get all the output paths corresponding to the placeholders we had */
+            if (experimentalFeatureSettings.isEnabled(Xp::CaDerivations)) {
+                res.insert_or_assign(
+                    DownstreamPlaceholder::fromSingleDerivedPathBuilt(
+                        SingleDerivedPath::Built {
+                            .drvPath = drv.drvPath,
+                            .output = outputName,
+                        }).render(),
+                    ctx.buildStore->printStorePath(outputPath)
+                );
+            }
         }
     }
 
@@ -153,7 +166,7 @@ static void mkOutputString(
     state.mkOutputString(
         attrs.alloc(o.first),
         SingleDerivedPath::Built {
-            .drvPath = makeConstantStorePath(drvPath),
+            .drvPath = makeConstantStorePathRef(drvPath),
             .output = o.first,
         },
         o.second.path(*state.ctx.store, Derivation::nameFromPath(drvPath), o.first));
@@ -229,14 +242,14 @@ static void import(EvalState & state, Value & vPath, Value * vScope, Value & v)
                 nullptr, state.ctx.builtins.staticEnv.get(), vScope->attrs->size()
             );
 
-            staticEnv->vars.unsafe_insert_bulk([&] (auto & map) {
-                unsigned int displ = 0;
-                for (auto & attr : *vScope->attrs) {
-                    // safety: args[0]->attrs is already sorted.
-                    map.emplace_back(attr.name, displ);
-                    env->values[displ++] = attr.value;
-                }
-            });
+            unsigned int displ = 0;
+            for (auto & attr : *vScope->attrs) {
+                staticEnv->vars.emplace_back(attr.name, displ);
+                env->values[displ++] = attr.value;
+            }
+
+            // No need to call staticEnv.sort(), because
+            // args[0]->attrs is already sorted.
 
             debug("evaluating file '%1%'", path);
             Expr & e = state.ctx.parseExprFromFile(state.ctx.paths.resolveExprPath(path), staticEnv);
@@ -302,13 +315,13 @@ void prim_exec(EvalState & state, Value * * args, Value & v)
     NixStringContext context;
     auto program = state.coerceToString(noPos, *elems[0], context,
             "while evaluating the first element of the argument passed to builtins.exec",
-            StringCoercionMode::Strict, false).toOwned();
+            false, false).toOwned();
     Strings commandArgs;
     for (unsigned int i = 1; i < args[0]->listSize(); ++i) {
         commandArgs.push_back(
                 state.coerceToString(noPos, *elems[i], context,
                         "while evaluating an element of the argument passed to builtins.exec",
-                        StringCoercionMode::Strict, false).toOwned());
+                        false, false).toOwned());
     }
     try {
         auto _ = state.realiseContext(context); // FIXME: Handle CA derivations
@@ -597,7 +610,7 @@ static void prim_addErrorContext(EvalState & state, Value * * args, Value & v)
         NixStringContext context;
         auto message = state.coerceToString(noPos, *args[0], context,
                 "while evaluating the error message passed to builtins.addErrorContext",
-                StringCoercionMode::Strict, false).toOwned();
+                false, false).toOwned();
         e.addTrace(nullptr, HintFmt(message));
         throw;
     }
@@ -797,6 +810,8 @@ drvName, Bindings * attrs, Value & v)
 
     NixStringContext context;
 
+    bool contentAddressed = false;
+    bool isImpure = false;
     std::optional<std::string> outputHash;
     std::string outputHashAlgo;
     std::optional<ContentAddressMethod> ingestionMethod;
@@ -812,7 +827,10 @@ drvName, Bindings * attrs, Value & v)
         auto handleHashMode = [&](const std::string_view s, NeverAsync = {}) {
             if (s == "recursive") ingestionMethod = FileIngestionMethod::Recursive;
             else if (s == "flat") ingestionMethod = FileIngestionMethod::Flat;
-            else
+            else if (s == "text") {
+                experimentalFeatureSettings.require(Xp::DynamicDerivations);
+                ingestionMethod = TextIngestionMethod {};
+            } else
                 state.ctx.errors.make<EvalError>(
                     "invalid value '%s' for 'outputHashMode' attribute", s
                 ).debugThrow();
@@ -850,13 +868,13 @@ drvName, Bindings * attrs, Value & v)
             }
 
             if (i->name == state.ctx.s.contentAddressed && state.forceBool(*i->value, noPos, context_below)) {
-                state.ctx.errors.make<EvalError>("ca derivations are not supported in Lix")
-                    .debugThrow();
+                contentAddressed = true;
+                experimentalFeatureSettings.require(Xp::CaDerivations);
             }
 
             else if (i->name == state.ctx.s.impure && state.forceBool(*i->value, noPos, context_below)) {
-                state.ctx.errors.make<EvalError>("impure derivations are not supported in Lix")
-                    .debugThrow();
+                isImpure = true;
+                experimentalFeatureSettings.require(Xp::ImpureDerivations);
             }
 
             /* The `args' attribute is special: it supplies the
@@ -866,7 +884,7 @@ drvName, Bindings * attrs, Value & v)
                 for (auto elem : i->value->listItems()) {
                     auto s = state.coerceToString(noPos, *elem, context,
                             "while evaluating an element of the argument list",
-                            StringCoercionMode::ToString).toOwned();
+                            true).toOwned();
                     drv.args.push_back(s);
                 }
             }
@@ -915,7 +933,7 @@ drvName, Bindings * attrs, Value & v)
 
 
                 } else {
-                    auto s = state.coerceToString(noPos, *i->value, context, context_below, StringCoercionMode::ToString).toOwned();
+                    auto s = state.coerceToString(noPos, *i->value, context, context_below, true).toOwned();
                     drv.env.emplace(key, s);
                     if (i->name == state.ctx.s.builder) drv.builder = std::move(s);
                     else if (i->name == state.ctx.s.system) drv.platform = std::move(s);
@@ -957,13 +975,13 @@ drvName, Bindings * attrs, Value & v)
                 for (auto & j : refs) {
                     drv.inputSrcs.insert(j);
                     if (j.isDerivation()) {
-                        drv.inputDrvs[j] =
+                        drv.inputDrvs.map[j].value =
                             state.aio.blockOn(state.ctx.store->readDerivation(j)).outputNames();
                     }
                 }
             },
             [&](const NixStringContextElem::Built & b) {
-                drv.inputDrvs[b.drvPath.path].insert(b.output);
+                drv.inputDrvs.ensureSlot(*b.drvPath).value.insert(b.output);
             },
             [&](const NixStringContextElem::Opaque & o) {
                 drv.inputSrcs.insert(o.path);
@@ -981,10 +999,15 @@ drvName, Bindings * attrs, Value & v)
             .debugThrow();
 
     /* Check whether the derivation name is valid. */
-    if (isDerivation(drvName)) {
-        state.ctx.errors
-            .make<EvalError>("derivation names are not allowed to end in '%s'", drvExtension)
-            .debugThrow();
+    if (isDerivation(drvName) &&
+        !(ingestionMethod == ContentAddressMethod { TextIngestionMethod { } } &&
+          outputs.size() == 1 &&
+          *(outputs.begin()) == "out"))
+    {
+        state.ctx.errors.make<EvalError>(
+            "derivation names are allowed to end in '%s' only if they produce a single derivation file",
+            drvExtension
+        ).debugThrow();
     }
 
     if (outputHash) {
@@ -1012,6 +1035,31 @@ drvName, Bindings * attrs, Value & v)
         drv.outputs.insert_or_assign("out", std::move(dof));
     }
 
+    else if (contentAddressed || isImpure) {
+        if (contentAddressed && isImpure)
+            state.ctx.errors.make<EvalError>("derivation cannot be both content-addressed and impure")
+                .debugThrow();
+
+        auto ht = parseHashTypeOpt(outputHashAlgo).value_or(HashType::SHA256);
+        auto method = ingestionMethod.value_or(FileIngestionMethod::Recursive);
+
+        for (auto & i : outputs) {
+            drv.env[i] = hashPlaceholder(i);
+            if (isImpure)
+                drv.outputs.insert_or_assign(i,
+                    DerivationOutput::Impure {
+                        .method = method,
+                        .hashType = ht,
+                    });
+            else
+                drv.outputs.insert_or_assign(i,
+                    DerivationOutput::CAFloating {
+                        .method = method,
+                        .hashType = ht,
+                    });
+        }
+    }
+
     else {
         /* Compute a hash over the "masked" store derivation, which is
            the final one except that in the list of outputs, the
@@ -1022,25 +1070,34 @@ drvName, Bindings * attrs, Value & v)
         for (auto & i : outputs) {
             drv.env[i] = "";
             drv.outputs.insert_or_assign(i,
-                DerivationOutput::InputAddressed { .path = StorePath::dummy });
+                DerivationOutput::Deferred { });
         }
 
         auto hashModulo =
             state.aio.blockOn(hashDerivationModulo(*state.ctx.store, Derivation(drv), true));
-        for (auto & i : outputs) {
-            auto h = get(hashModulo.hashes, i);
-            if (!h)
-                state.ctx.errors.make<AssertionError>(
-                    "derivation produced no hash for output '%s'",
-                    i
-                ).debugThrow();
-            auto outPath = state.ctx.store->makeOutputPath(i, *h, drvName);
-            drv.env[i] = state.ctx.store->printStorePath(outPath);
-            drv.outputs.insert_or_assign(
-                i,
-                DerivationOutput::InputAddressed {
-                    .path = std::move(outPath),
-                });
+        switch (hashModulo.kind) {
+        case DrvHash::Kind::Regular:
+            for (auto & i : outputs) {
+                auto h = get(hashModulo.hashes, i);
+                if (!h)
+                    state.ctx.errors.make<AssertionError>(
+                        "derivation produced no hash for output '%s'",
+                        i
+                    ).debugThrow();
+                auto outPath = state.ctx.store->makeOutputPath(i, *h, drvName);
+                drv.env[i] = state.ctx.store->printStorePath(outPath);
+                drv.outputs.insert_or_assign(
+                    i,
+                    DerivationOutput::InputAddressed {
+                        .path = std::move(outPath),
+                    });
+            }
+            break;
+            ;
+        case DrvHash::Kind::Deferred:
+            for (auto & i : outputs) {
+                drv.outputs.insert_or_assign(i, DerivationOutput::Deferred {});
+            }
         }
     }
 
@@ -1175,7 +1232,7 @@ static void prim_baseNameOf(EvalState & state, Value * * args, Value & v)
     NixStringContext context;
     v.mkString(baseNameOf(*state.coerceToString(noPos, *args[0], context,
             "while evaluating the first argument passed to builtins.baseNameOf",
-            StringCoercionMode::Strict, false)), context);
+            false, false)), context);
 }
 
 /* Return the directory of the given path, i.e., everything before the
@@ -1191,7 +1248,7 @@ static void prim_dirOf(EvalState & state, Value * * args, Value & v)
         NixStringContext context;
         auto path = state.coerceToString(noPos, *args[0], context,
             "while evaluating the first argument passed to 'builtins.dirOf'",
-            StringCoercionMode::Strict, false);
+            false, false);
         auto dir = dirOf(*path);
         v.mkString(dir, context);
     }
@@ -1252,7 +1309,7 @@ static void prim_findFile(EvalState & state, Value * * args, Value & v)
         NixStringContext context;
         auto path = state.coerceToString(noPos, *i->value, context,
                 "while evaluating the `path` attribute of an element of the list passed to builtins.findFile",
-                StringCoercionMode::Strict, false).toOwned();
+                false, false).toOwned();
 
         try {
             auto rewrites = state.realiseContext(context);
@@ -1345,6 +1402,21 @@ static void prim_readDir(EvalState & state, Value * * args, Value & v)
     v.mkAttrs(attrs);
 }
 
+/* Extend single element string context with another output. */
+static void prim_outputOf(EvalState & state, Value * * args, Value & v)
+{
+    SingleDerivedPath drvPath = state.coerceToSingleDerivedPath(noPos, *args[0], "while evaluating the first argument to builtins.outputOf");
+
+    OutputNameView outputName = state.forceStringNoCtx(*args[1], noPos, "while evaluating the second argument to builtins.outputOf");
+
+    state.mkSingleDerivedPathString(
+        SingleDerivedPath::Built {
+            .drvPath = make_ref<SingleDerivedPath>(drvPath),
+            .output = std::string { outputName },
+        },
+        v);
+}
+
 /*************************************************************
  * Creating files
  *************************************************************/
@@ -2416,7 +2488,7 @@ static void prim_toString(EvalState & state, Value * * args, Value & v)
     NixStringContext context;
     auto s = state.coerceToString(noPos, *args[0], context,
             "while evaluating the first argument passed to builtins.toString",
-            StringCoercionMode::ToString, false);
+            true, false);
     v.mkString(*s, context);
 }
 
@@ -2798,6 +2870,7 @@ void EvalBuiltins::createBaseEnv(const SearchPath & searchPath, const Path & sto
        because attribute lookups expect it to be sorted. */
     env.values[0]->attrs->sort();
 
+    staticEnv->sort();
     staticEnv->isRoot = true;
 }
 

lix/libexpr/primops/context.cc

@@ -127,7 +127,9 @@ void prim_getContext(EvalState & state, Value * * args, Value & v)
                 contextInfos[std::move(d.drvPath)].allOutputs = true;
             },
             [&](NixStringContextElem::Built && b) {
-                auto drvPath = b.drvPath.path;
+                // FIXME should eventually show string context as is, no
+                // resolving here.
+                auto drvPath = state.aio.blockOn(resolveDerivedPath(*state.ctx.store, *b.drvPath));
                 contextInfos[std::move(drvPath)].outputs.emplace_back(std::move(b.output));
             },
             [&](NixStringContextElem::Opaque && o) {
@@ -217,7 +219,7 @@ static void prim_appendContext(EvalState & state, Value * * args, Value & v)
             for (auto elem : iter->value->listItems()) {
                 auto outputName = state.forceStringNoCtx(*elem, iter->pos, "while evaluating an output name within a string context");
                 context.emplace(NixStringContextElem::Built {
-                    .drvPath = makeConstantStorePath(namePath),
+                    .drvPath = makeConstantStorePathRef(namePath),
                     .output = std::string { outputName },
                 });
             }

lix/libexpr/primops/fetchMercurial.cc

@@ -22,7 +22,7 @@ static void prim_fetchMercurial(EvalState & state, Value * * args, Value & v)
             if (n == "url")
                 url = state.coerceToString(attr.pos, *attr.value, context,
                         "while evaluating the `url` attribute passed to builtins.fetchMercurial",
-                        StringCoercionMode::Strict, false).toOwned();
+                        false, false).toOwned();
             else if (n == "rev") {
                 // Ugly: unlike fetchGit, here the "rev" attribute can
                 // be both a revision or a branch/tag name.
@@ -44,7 +44,7 @@ static void prim_fetchMercurial(EvalState & state, Value * * args, Value & v)
     } else
         url = state.coerceToString(noPos, *args[0], context,
                 "while evaluating the first argument passed to builtins.fetchMercurial",
-                StringCoercionMode::Strict, false).toOwned();
+                false, false).toOwned();
 
     // FIXME: git externals probably can be used to bypass the URI
     // whitelist. Ah well.

lix/libexpr/primops/fetchTree.cc

@@ -139,7 +139,7 @@ static void fetchTree(
             if (attr.name == state.ctx.s.type) continue;
             state.forceValue(*attr.value, attr.pos);
             if (attr.value->type() == nPath || attr.value->type() == nString) {
-                auto s = state.coerceToString(attr.pos, *attr.value, context, "", StringCoercionMode::Strict, false).toOwned();
+                auto s = state.coerceToString(attr.pos, *attr.value, context, "", false, false).toOwned();
                 attrs.emplace(state.ctx.symbols[attr.name],
                     state.ctx.symbols[attr.name] == "url"
                     ? type == "git"
@@ -169,18 +169,11 @@ static void fetchTree(
                     "attribute 'name' isn’t supported in call to 'fetchTree'"
                 ).atPos(pos).debugThrow();
 
-        // HACK: When using `fetchGit`, locking with only the hash should happen
-        //       as we don't care about flake shenanigans about `lastModified`
-        if (type == "git" && attrs.contains("narHash")) {
-            using namespace std::literals::string_literals;
-            attrs["type"] = "\0git-locked"s;
-        }
-
         input = fetchers::Input::fromAttrs(std::move(attrs));
     } else {
         auto url = state.coerceToString(pos, *args[0], context,
                 "while evaluating the first argument passed to the fetcher",
-                StringCoercionMode::Strict, false).toOwned();
+                false, false).toOwned();
 
         if (type == "git") {
             fetchers::Attrs attrs;

lix/libexpr/primops/fromTOML.cc

@@ -6,82 +6,164 @@
 
 namespace nix {
 
-void prim_fromTOML(EvalState & state, Value * * args, Value & val)
+#if HAVE_TOML11_4
+
+/**
+ * This is what toml11 < 4.0 did when choosing the subsecond precision.
+ * TOML 1.0.0 spec doesn't define how sub-millisecond ranges should be handled and calls it
+ * implementation defined behavior. For a lack of a better choice we stick with what older versions
+ * of toml11 did [1].
+ *
+ * [1]:
+ * https://github.com/ToruNiina/toml11/blob/dcfe39a783a94e8d52c885e5883a6fbb21529019/toml/datetime.hpp#L282
+ */
+static size_t normalizeSubsecondPrecision(toml::local_time lt)
 {
-    auto toml = state.forceStringNoCtx(*args[0], noPos, "while evaluating the argument passed to builtins.fromTOML");
+    auto millis = lt.millisecond;
+    auto micros = lt.microsecond;
+    auto nanos = lt.nanosecond;
+    if (millis != 0 || micros != 0 || nanos != 0) {
+        if (micros != 0 || nanos != 0) {
+            if (nanos != 0) {
+                return 9;
+            }
+            return 6;
+        }
+        return 3;
+    }
+    return 0;
+}
+
+/**
+ * Normalize date/time formats to serialize to the same strings as versions prior to toml11 4.0.
+ *
+ * Several things to consider:
+ *
+ * 1. Sub-millisecond range is represented the same way as in toml11 versions prior to 4.0.
+ * Precision is rounded towards the next multiple of 3 or capped at 9 digits.
+ * 2. Seconds must be specified. This may become optional in (yet unreleased) TOML 1.1.0, but 1.0.0
+ * defined local time in terms of RFC3339 [1].
+ * 3. date-time separator (`t`, `T` or space ` `) is canonicalized to an upper T. This is compliant
+ * with RFC3339 [1] 5.6: > Applications that generate this format SHOULD use upper case letters.
+ *
+ * [1]: https://datatracker.ietf.org/doc/html/rfc3339#section-5.6
+ */
+static void normalizeDatetimeFormat(toml::value & t)
+{
+    if (t.is_local_datetime()) {
+        auto & ldt = t.as_local_datetime();
+        t.as_local_datetime_fmt() = {
+            .delimiter = toml::datetime_delimiter_kind::upper_T,
+            // https://datatracker.ietf.org/doc/html/rfc3339#section-5.6
+            .has_seconds = true, // Mandated by TOML 1.0.0
+            .subsecond_precision = normalizeSubsecondPrecision(ldt.time),
+        };
+        return;
+    }
+
+    if (t.is_offset_datetime()) {
+        auto & odt = t.as_offset_datetime();
+        t.as_offset_datetime_fmt() = {
+            .delimiter = toml::datetime_delimiter_kind::upper_T,
+            // https://datatracker.ietf.org/doc/html/rfc3339#section-5.6
+            .has_seconds = true, // Mandated by TOML 1.0.0
+            .subsecond_precision = normalizeSubsecondPrecision(odt.time),
+        };
+        return;
+    }
+
+    if (t.is_local_time()) {
+        auto & lt = t.as_local_time();
+        t.as_local_time_fmt() = {
+            .has_seconds = true, // Mandated by TOML 1.0.0
+            .subsecond_precision = normalizeSubsecondPrecision(lt),
+        };
+        return;
+    }
+}
+
+#endif
+
+void prim_fromTOML(EvalState & state, Value ** args, Value & val)
+{
+    auto toml = state.forceStringNoCtx(
+        *args[0], noPos, "while evaluating the argument passed to builtins.fromTOML"
+    );
 
     std::istringstream tomlStream(std::string{toml});
 
-    std::function<void(Value &, toml::value)> visit;
+    auto visit = [&](this const auto & self, Value & v, toml::value t) -> void {
+        switch (t.type()) {
+        case toml::value_t::table: {
+            auto table = toml::get<toml::table>(t);
+            auto attrs = state.ctx.buildBindings(table.size());
 
-    visit = [&](Value & v, toml::value t) {
+            for (auto & elem : table) {
+                self(attrs.alloc(elem.first), elem.second);
+            }
 
-        switch(t.type())
-        {
-            case toml::value_t::table:
-                {
-                    auto table = toml::get<toml::table>(t);
-
-                    size_t size = 0;
-                    for (auto & i : table) { (void) i; size++; }
-
-                    auto attrs = state.ctx.buildBindings(size);
-
-                    for(auto & elem : table)
-                        visit(attrs.alloc(elem.first), elem.second);
-
-                    v.mkAttrs(attrs);
-                }
-                break;;
-            case toml::value_t::array:
-                {
-                    auto array = toml::get<std::vector<toml::value>>(t);
-
-                    size_t size = array.size();
-                    v = state.ctx.mem.newList(size);
-                    for (size_t i = 0; i < size; ++i)
-                        visit(*(v.listElems()[i] = state.ctx.mem.allocValue()), array[i]);
-                }
-                break;;
-            case toml::value_t::boolean:
-                v.mkBool(toml::get<bool>(t));
-                break;;
-            case toml::value_t::integer:
-                v.mkInt(toml::get<int64_t>(t));
-                break;;
-            case toml::value_t::floating:
-                v.mkFloat(toml::get<NixFloat>(t));
-                break;;
-            case toml::value_t::string:
-                v.mkString(toml::get<std::string>(t));
-                break;;
-            case toml::value_t::local_datetime:
-            case toml::value_t::offset_datetime:
-            case toml::value_t::local_date:
-            case toml::value_t::local_time:
-                {
-                    if (experimentalFeatureSettings.isEnabled(Xp::ParseTomlTimestamps)) {
-                        auto attrs = state.ctx.buildBindings(2);
-                        attrs.alloc("_type").mkString("timestamp");
-                        std::ostringstream s;
-                        s << t;
-                        attrs.alloc("value").mkString(s.str());
-                        v.mkAttrs(attrs);
-                    } else {
-                        // NOLINTNEXTLINE(lix-foreign-exceptions)
-                        throw std::runtime_error("Dates and times are not supported");
-                    }
-                }
-                break;;
-            case toml::value_t::empty:
-                v.mkNull();
-                break;;
+            v.mkAttrs(attrs);
+        } break;
+        case toml::value_t::array: {
+            auto array = toml::get<std::vector<toml::value>>(t);
 
+            size_t size = array.size();
+            v = state.ctx.mem.newList(size);
+            for (size_t i = 0; i < size; ++i) {
+                self(*(v.listElems()[i] = state.ctx.mem.allocValue()), array[i]);
+            }
+        } break;
+        case toml::value_t::boolean:
+            v.mkBool(toml::get<bool>(t));
+            break;
+        case toml::value_t::integer:
+            v.mkInt(toml::get<int64_t>(t));
+            break;
+        case toml::value_t::floating:
+            v.mkFloat(toml::get<NixFloat>(t));
+            break;
+        case toml::value_t::string:
+            v.mkString(toml::get<std::string>(t));
+            break;
+        case toml::value_t::local_datetime:
+        case toml::value_t::offset_datetime:
+        case toml::value_t::local_date:
+        case toml::value_t::local_time: {
+            if (experimentalFeatureSettings.isEnabled(Xp::ParseTomlTimestamps)) {
+#if HAVE_TOML11_4
+                normalizeDatetimeFormat(t);
+#endif
+                auto attrs = state.ctx.buildBindings(2);
+                attrs.alloc("_type").mkString("timestamp");
+                std::ostringstream s;
+                s << t;
+                attrs.alloc("value").mkString(s.str());
+                v.mkAttrs(attrs);
+            } else {
+                // NOLINTNEXTLINE(lix-foreign-exceptions)
+                throw std::runtime_error("Dates and times are not supported");
+            }
+        } break;
+        case toml::value_t::empty:
+            v.mkNull();
+            break;
         }
     };
 
     try {
-        visit(val, toml::parse(tomlStream, "fromTOML" /* the "filename" */));
+        visit(
+            val,
+            toml::parse(
+                tomlStream,
+                "fromTOML" /* the "filename" */
+#if HAVE_TOML11_4
+                ,
+                toml::spec::v(
+                    1, 0, 0
+                ) // Be explicit that we are parsing TOML 1.0.0 without extensions
+#endif
+            )
+        );
     } catch (std::exception & e) { // NOLINT(lix-foreign-exceptions) // TODO: toml::syntax_error
         state.ctx.errors.make<EvalError>("while parsing TOML: %s", e.what()).debugThrow();
     }

lix/libexpr/value-to-json.cc

@@ -46,7 +46,7 @@ JSON printValueAsJSON(EvalState & state, bool strict,
             break;
 
         case nAttrs: {
-            auto maybeString = state.tryAttrsToString(pos, v, context, StringCoercionMode::Strict, false);
+            auto maybeString = state.tryAttrsToString(pos, v, context, false, false);
             if (maybeString) {
                 out = *maybeString;
                 break;

lix/libexpr/value.hh

@@ -10,7 +10,6 @@
 #include "lix/libexpr/gc-alloc.hh"
 #include "lix/libexpr/symbol-table.hh"
 #include "lix/libexpr/value/context.hh"
-#include "lix/libutil/logging.hh"
 #include "lix/libutil/source-path.hh"
 #include "lix/libexpr/print-options.hh"
 #include "lix/libutil/checked-arithmetic.hh"
@@ -60,26 +59,6 @@ typedef enum {
     nExternal
 } ValueType;
 
-
-/**
- * Modes of string coercion.
- *
- * Determines how permissive the coercion functions are when converting
- * values to strings.
- *
- * - Strict: Only allow coercion of values that are already strings,
- *   paths, or derivations.
- * - Interpolation: Additionally allow coercion of unambiguously printable values in a string, for
- *   now: integers. This mode is meant as a stopgap measure until we get better formatting tools.
- * - ToString: Additionally allow coercion of integers, booleans, null,
- *   and lists to strings.
- */
-enum class StringCoercionMode {
-    Strict,
-    Interpolation,
-    ToString,
-};
-
 class Bindings;
 struct Env;
 struct Expr;
@@ -127,7 +106,7 @@ class ExternalValueBase
      * Coerce the value to a string. Defaults to uncoercable, i.e. throws an
      * error.
      */
-    virtual std::string coerceToString(EvalState & state, const PosIdx & pos, NixStringContext & context, StringCoercionMode mode, bool copyToStore) const;
+    virtual std::string coerceToString(EvalState & state, const PosIdx & pos, NixStringContext & context, bool copyMore, bool copyToStore) const;
 
     /**
      * Compare to another value of the same type. Defaults to uncomparable,

lix/libexpr/value/context.cc

@@ -2,10 +2,33 @@
 
 namespace nix {
 
-NixStringContextElem NixStringContextElem::parse(std::string_view s0)
+NixStringContextElem NixStringContextElem::parse(
+    std::string_view s0,
+    const ExperimentalFeatureSettings & xpSettings)
 {
     std::string_view s = s0;
 
+    std::function<SingleDerivedPath()> parseRest;
+    parseRest = [&]() -> SingleDerivedPath {
+        // Case on whether there is a '!'
+        size_t index = s.find("!");
+        if (index == std::string_view::npos) {
+            return SingleDerivedPath::Opaque {
+                .path = StorePath { s },
+            };
+        } else {
+            std::string output { s.substr(0, index) };
+            // Advance string to parse after the '!'
+            s = s.substr(index + 1);
+            auto drv = make_ref<SingleDerivedPath>(parseRest());
+            drvRequireExperiment(*drv, xpSettings);
+            return SingleDerivedPath::Built {
+                .drvPath = std::move(drv),
+                .output = std::move(output),
+            };
+        }
+    };
+
     if (s.size() == 0) {
         throw BadNixStringContextElem(s0,
             "String context element should never be an empty string");
@@ -17,20 +40,14 @@ NixStringContextElem NixStringContextElem::parse(std::string_view s0)
         s = s.substr(1);
 
         // Find *second* '!'
-        size_t index = s.find("!");
-        if (index == std::string_view::npos) {
+        if (s.find("!") == std::string_view::npos) {
             throw BadNixStringContextElem(s0,
                 "String content element beginning with '!' should have a second '!'");
         }
 
-        std::string output { s.substr(0, index) };
-        // Advance string to parse after the '!'
-        s = s.substr(index + 1);
-        auto drv = SingleDerivedPath::Opaque{StorePath{s}};
-        return SingleDerivedPath::Built{
-            .drvPath = std::move(drv),
-            .output = std::move(output),
-        };
+        return std::visit(
+            [&](auto x) -> NixStringContextElem { return std::move(x); },
+            parseRest());
     }
     case '=': {
         return NixStringContextElem::DrvDeep {
@@ -43,9 +60,9 @@ NixStringContextElem NixStringContextElem::parse(std::string_view s0)
             throw BadNixStringContextElem(s0,
                 "String content element not beginning with '!' should not have a second '!'");
         }
-        return SingleDerivedPath::Opaque{
-            .path = StorePath{s},
-        };
+        return std::visit(
+            [&](auto x) -> NixStringContextElem { return std::move(x); },
+            parseRest());
     }
     }
 }
@@ -54,15 +71,27 @@ std::string NixStringContextElem::to_string() const
 {
     std::string res;
 
+    std::function<void(const SingleDerivedPath &)> toStringRest;
+    toStringRest = [&](auto & p) {
+        std::visit(overloaded {
+            [&](const SingleDerivedPath::Opaque & o) {
+                res += o.path.to_string();
+            },
+            [&](const SingleDerivedPath::Built & o) {
+                res += o.output;
+                res += '!';
+                toStringRest(*o.drvPath);
+            },
+        }, p.raw());
+    };
+
     std::visit(overloaded {
         [&](const NixStringContextElem::Built & b) {
             res += '!';
-            res += b.output;
-            res += '!';
-            res += b.drvPath.path.to_string();
+            toStringRest(b);
         },
         [&](const NixStringContextElem::Opaque & o) {
-            res += o.path.to_string();
+            toStringRest(o);
         },
         [&](const NixStringContextElem::DrvDeep & d) {
             res += '=';

lix/libexpr/value/context.hh

@@ -69,8 +69,12 @@ struct NixStringContextElem {
      * - ‘<path>’
      * - ‘=<path>’
      * - ‘!<name>!<path>’
+     *
+     * @param xpSettings Stop-gap to avoid globals during unit tests.
      */
-    static NixStringContextElem parse(std::string_view s);
+    static NixStringContextElem parse(
+        std::string_view s,
+        const ExperimentalFeatureSettings & xpSettings = experimentalFeatureSettings);
     std::string to_string() const;
 };
 

lix/libfetchers/builtin-fetchers.hh

@@ -10,7 +10,6 @@ std::unique_ptr<InputScheme> makePathInputScheme();
 std::unique_ptr<InputScheme> makeFileInputScheme();
 std::unique_ptr<InputScheme> makeTarballInputScheme();
 std::unique_ptr<InputScheme> makeGitInputScheme();
-std::unique_ptr<InputScheme> makeGitLockedInputScheme();
 std::unique_ptr<InputScheme> makeMercurialInputScheme();
 std::unique_ptr<InputScheme> makeGitHubInputScheme();
 std::unique_ptr<InputScheme> makeGitLabInputScheme();

lix/libfetchers/fetchers.cc

@@ -22,7 +22,6 @@ void initLibFetchers()
     registerInputScheme(makeTarballInputScheme());
     registerInputScheme(makeFileInputScheme());
     registerInputScheme(makeGitInputScheme());
-    registerInputScheme(makeGitLockedInputScheme());
     registerInputScheme(makeMercurialInputScheme());
     registerInputScheme(makeGitHubInputScheme());
     registerInputScheme(makeGitLabInputScheme());
@@ -305,24 +304,6 @@ std::optional<time_t> Input::getLastModified() const
     return maybeGetIntAttr(attrs, "lastModified");
 }
 
-std::optional<Input> InputScheme::inputFromAttrs(const Attrs & attrs) const
-{
-    if (maybeGetStrAttr(attrs, "type") != schemeType()) return {};
-
-    Attrs finalAttrs = preprocessAttrs(attrs);
-
-    for (auto & [name, value] : finalAttrs)
-        // All attrs need to accept a `type` and `narHash` key, the rest is scheme-specific
-        if (name != "type" && name != "narHash" && !allowedAttrs().contains(name))
-            throw UnsupportedAttributeError("unsupported input attribute '%s' for the '%s' scheme", name, schemeType());
-
-    Input input;
-    input.attrs = finalAttrs;
-    return input;
-}
-
-
-
 ParsedURL InputScheme::toURL(const Input & input) const
 {
     throw Error("don't know how to convert input '%s' to a URL", attrsToJSON(input.attrs));

lix/libfetchers/fetchers.hh

@@ -2,7 +2,6 @@
 ///@file
 
 #include "lix/libstore/content-address.hh"
-#include "lix/libutil/error.hh"
 #include "lix/libutil/result.hh"
 #include "lix/libutil/types.hh"
 #include "lix/libutil/hash.hh"
@@ -20,8 +19,6 @@ namespace nix { class Store; }
 
 namespace nix::fetchers {
 
-MakeError(UnsupportedAttributeError, Error);
-
 struct Tree
 {
     Path actualPath;
@@ -128,6 +125,7 @@ public:
     std::optional<time_t> getLastModified() const;
 };
 
+
 /**
  * The InputScheme represents a type of fetcher.  Each fetcher
  * registers with nix at startup time.  When processing an input for a
@@ -144,12 +142,7 @@ struct InputScheme
 
     virtual std::optional<Input> inputFromURL(const ParsedURL & url, bool requireTree) const = 0;
 
-    virtual Attrs preprocessAttrs(const Attrs & attrs) const = 0;
-
-    // The scheme type, which is used to match attributes to a specific scheme
-    virtual std::string schemeType() const = 0;
-
-    virtual std::optional<Input> inputFromAttrs(const Attrs & attrs) const;
+    virtual std::optional<Input> inputFromAttrs(const Attrs & attrs) const = 0;
 
     virtual ParsedURL toURL(const Input & input) const;
 
@@ -181,9 +174,6 @@ struct InputScheme
     virtual bool isLockedByRev() const { return true; }
 
 protected:
-    // The set of allowed attributes for this specific fetcher
-    virtual const std::set<std::string> & allowedAttrs() const = 0;
-
     void emplaceURLQueryIntoAttrs(
         const ParsedURL & parsedURL,
         Attrs & attrs,

lix/libfetchers/git.cc

@@ -22,6 +22,7 @@
 #include <string.h>
 #include <sys/time.h>
 #include <sys/wait.h>
+#include <unistd.h>
 
 using namespace std::string_literals;
 
@@ -168,7 +169,7 @@ WorkdirInfo getWorkdirInfo(const Input & input, const Path & workdir)
         .program = "git",
         .args = { "-C", workdir, "--git-dir", gitDir, "rev-parse", "--verify", "--no-revs", "HEAD^{commit}" },
         .environment = env,
-        .mergeStderrToStdout = true
+        .redirections = {{.from = STDERR_FILENO, .to = STDOUT_FILENO}},
     });
     auto exitCode = WEXITSTATUS(result.first);
     auto errorMessage = result.second;
@@ -288,28 +289,8 @@ static std::optional<Path> resolveRefToCachePath(
     return std::nullopt;
 }
 
-static const std::set<std::string> allowedGitAttrs = {
-    "allRefs",
-    "dirtyRev",
-    "dirtyShortRev",
-    "lastModified",
-    "name",
-    "ref",
-    "rev",
-    "revCount",
-    "shallow",
-    "submodules",
-    "url",
-};
-
 struct GitInputScheme : InputScheme
 {
-    std::string schemeType() const override { return "git"; }
-
-    const std::set<std::string> & allowedAttrs() const override {
-        return allowedGitAttrs;
-    }
-
     std::optional<Input> inputFromURL(const ParsedURL & url, bool requireTree) const override
     {
         if (url.scheme != "git" &&
@@ -336,7 +317,14 @@ struct GitInputScheme : InputScheme
         return inputFromAttrs(attrs);
     }
 
-    Attrs preprocessAttrs(const Attrs & attrs) const override {
+    std::optional<Input> inputFromAttrs(const Attrs & attrs) const override
+    {
+        if (maybeGetStrAttr(attrs, "type") != "git") return {};
+
+        for (auto & [name, value] : attrs)
+            if (name != "type" && name != "url" && name != "ref" && name != "rev" && name != "shallow" && name != "submodules" && name != "lastModified" && name != "revCount" && name != "narHash" && name != "allRefs" && name != "name" && name != "dirtyRev" && name != "dirtyShortRev")
+                throw Error("unsupported Git input attribute '%s'", name);
+
         parseURL(getStrAttr(attrs, "url"));
         maybeGetBoolAttr(attrs, "shallow");
         maybeGetBoolAttr(attrs, "submodules");
@@ -347,7 +335,9 @@ struct GitInputScheme : InputScheme
                 throw BadURL("invalid Git branch/tag name '%s'", *ref);
         }
 
-        return attrs;
+        Input input;
+        input.attrs = attrs;
+        return input;
     }
 
     ParsedURL toURL(const Input & input) const override
@@ -712,7 +702,7 @@ struct GitInputScheme : InputScheme
         auto result = runProgram(RunOptions {
             .program = "git",
             .args = { "-C", repoDir, "--git-dir", gitDir, "cat-file", "commit", input.getRev()->gitRev() },
-            .mergeStderrToStdout = true
+            .redirections = {{.from = STDERR_FILENO, .to = STDOUT_FILENO}},
         });
         if (WEXITSTATUS(result.first) == 128
             && result.second.find("bad file") != std::string::npos)
@@ -777,7 +767,7 @@ struct GitInputScheme : InputScheme
                 .args = { "-C", repoDir, "--git-dir", gitDir, "archive", input.getRev()->gitRev() },
                 .captureStdout = true,
             });
-            Finally const _wait([&] { proc.wait(); });
+            Finally const _wait([&] { proc.waitAndCheck(); });
 
             unpackTarfile(*proc.getStdout(), tmpDir);
         }
@@ -823,22 +813,4 @@ std::unique_ptr<InputScheme> makeGitInputScheme()
     return std::make_unique<GitInputScheme>();
 }
 
-struct GitLockedInputScheme : GitInputScheme {
-
-    std::string schemeType() const override {
-        using namespace std::literals::string_literals;
-        return "\0git-locked"s;
-    }
-
-    bool hasAllInfo(const Input & input) const override {
-        return true;
-    }
-
-};
-
-std::unique_ptr<InputScheme> makeGitLockedInputScheme()
-{
-    return std::make_unique<GitLockedInputScheme>();
-}
-
 }

lix/libfetchers/github.cc

@@ -1,5 +1,7 @@
 #include "lix/libfetchers/attrs.hh"
 #include "lix/libstore/filetransfer.hh"
+#include "lix/libfetchers/cache.hh"
+#include "lix/libstore/globals.hh"
 #include "lix/libfetchers/builtin-fetchers.hh"
 #include "lix/libstore/store-api.hh"
 #include "lix/libutil/async.hh"
@@ -22,30 +24,19 @@ struct DownloadUrl
     Headers headers;
 };
 
-static const std::set<std::string> allowedGitArchiveAttrs = {
-    "host",
-    "lastModified",
-    "owner",
-    "ref",
-    "repo",
-    "rev",
-};
-
 // A github, gitlab, or sourcehut host
 const static std::string hostRegexS = "[a-zA-Z0-9.-]*"; // FIXME: check
 std::regex hostRegex = regex::parse(hostRegexS, std::regex::ECMAScript);
 
 struct GitArchiveInputScheme : InputScheme
 {
-    const std::set<std::string> & allowedAttrs() const override {
-        return allowedGitArchiveAttrs;
-    }
+    virtual std::string type() const = 0;
 
     virtual std::optional<std::pair<std::string, std::string>> accessHeaderFromToken(const std::string & token) const = 0;
 
     std::optional<Input> inputFromURL(const ParsedURL & url, bool requireTree) const override
     {
-        if (url.scheme != schemeType()) return {};
+        if (url.scheme != type()) return {};
 
         auto path = tokenizeString<std::vector<std::string>>(url.path, "/");
 
@@ -72,7 +63,7 @@ struct GitArchiveInputScheme : InputScheme
             throw BadURL("URL '%s' is invalid", url.url);
 
         Attrs attrs;
-        attrs.emplace("type", schemeType());
+        attrs.emplace("type", type());
         attrs.emplace("owner", path[0]);
         attrs.emplace("repo", path[1]);
 
@@ -102,18 +93,20 @@ struct GitArchiveInputScheme : InputScheme
         return inputFromAttrs(attrs);
     }
 
-    Attrs preprocessAttrs(const Attrs & attrs) const override
+    std::optional<Input> inputFromAttrs(const Attrs & attrs) const override
     {
         // Attributes can contain refOrRev and it needs to be figured out
         // which one it is (see inputFromURL for when that may happen).
         // The correct one (ref or rev) will be written into finalAttrs and
         // it needs to be mutable for that.
         Attrs finalAttrs(attrs);
+        auto type_ = maybeGetStrAttr(finalAttrs, "type");
+        if (type_ != type()) return {};
 
         auto owner = getStrAttr(finalAttrs, "owner");
         auto repo = getStrAttr(finalAttrs, "repo");
 
-        auto url = fmt("%s:%s/%s", schemeType(), owner, repo);
+        auto url = fmt("%s:%s/%s", *type_, owner, repo);
         if (auto host = maybeGetStrAttr(finalAttrs, "host")) {
             if (!std::regex_match(*host, hostRegex)) {
                 throw BadURL("URL '%s' contains an invalid instance host", url);
@@ -139,7 +132,15 @@ struct GitArchiveInputScheme : InputScheme
             }
         }
 
-        return finalAttrs;
+        for (auto & [name, value] : finalAttrs) {
+            if (name != "type" && name != "owner" && name != "repo" && name != "ref" && name != "rev" && name != "narHash" && name != "lastModified" && name != "host") {
+                throw Error("unsupported input attribute '%s'", name);
+            }
+        }
+
+        Input input;
+        input.attrs = finalAttrs;
+        return input;
     }
 
     ParsedURL toURL(const Input & input) const override
@@ -153,7 +154,7 @@ struct GitArchiveInputScheme : InputScheme
         if (ref) path += "/" + *ref;
         if (rev) path += "/" + rev->to_string(Base::Base16, false);
         return ParsedURL {
-            .scheme = schemeType(),
+            .scheme = type(),
             .path = path,
         };
     }
@@ -238,7 +239,7 @@ struct GitArchiveInputScheme : InputScheme
 
 struct GitHubInputScheme : GitArchiveInputScheme
 {
-    std::string schemeType() const override { return "github"; }
+    std::string type() const override { return "github"; }
 
     std::optional<std::pair<std::string, std::string>> accessHeaderFromToken(const std::string & token) const override
     {
@@ -328,7 +329,7 @@ struct GitHubInputScheme : GitArchiveInputScheme
 
 struct GitLabInputScheme : GitArchiveInputScheme
 {
-    std::string schemeType() const override { return "gitlab"; }
+    std::string type() const override { return "gitlab"; }
 
     std::optional<std::pair<std::string, std::string>> accessHeaderFromToken(const std::string & token) const override
     {
@@ -404,7 +405,7 @@ struct GitLabInputScheme : GitArchiveInputScheme
 
 struct SourceHutInputScheme : GitArchiveInputScheme
 {
-    std::string schemeType() const override { return "sourcehut"; }
+    std::string type() const override { return "sourcehut"; }
 
     std::optional<std::pair<std::string, std::string>> accessHeaderFromToken(const std::string & token) const override
     {

lix/libfetchers/indirect.cc

@@ -8,20 +8,8 @@ namespace nix::fetchers {
 
 std::regex flakeRegex = regex::parse("[a-zA-Z][a-zA-Z0-9_-]*", std::regex::ECMAScript);
 
-static const std::set<std::string> allowedIndirectAttrs = {
-    "id",
-    "ref",
-    "rev",
-};
-
 struct IndirectInputScheme : InputScheme
 {
-    std::string schemeType() const override { return "indirect"; }
-
-    const std::set<std::string> & allowedAttrs() const override {
-        return allowedIndirectAttrs;
-    }
-
     std::optional<Input> inputFromURL(const ParsedURL & url, bool requireTree) const override
     {
         if (url.scheme != "flake") return {};
@@ -59,7 +47,14 @@ struct IndirectInputScheme : InputScheme
         return inputFromAttrs(attrs);
     }
 
-    Attrs preprocessAttrs(const Attrs & attrs) const override {
+    std::optional<Input> inputFromAttrs(const Attrs & attrs) const override
+    {
+        if (maybeGetStrAttr(attrs, "type") != "indirect") return {};
+
+        for (auto & [name, value] : attrs)
+            if (name != "type" && name != "id" && name != "ref" && name != "rev" && name != "narHash")
+                throw Error("unsupported indirect input attribute '%s'", name);
+
         auto id = getStrAttr(attrs, "id");
         if (!std::regex_match(id, flakeRegex))
             throw BadURL("'%s' is not a valid flake ID", id);
@@ -76,17 +71,9 @@ struct IndirectInputScheme : InputScheme
             }
         }
 
-        return attrs;
-    }
-
-    std::optional<Input> inputFromAttrs(const Attrs & attrs) const override
-    {
-        std::optional<Input> input = InputScheme::inputFromAttrs(attrs);
-
-        if (input) {
-            input->direct = false;
-        }
-
+        Input input;
+        input.direct = false;
+        input.attrs = attrs;
         return input;
     }
 

lix/libfetchers/mercurial.cc

@@ -44,22 +44,8 @@ static std::string runHg(const Strings & args)
     return res.second;
 }
 
-static const std::set<std::string> allowedMercurialAttrs = {
-    "name",
-    "ref",
-    "rev",
-    "revCount",
-    "url",
-};
-
 struct MercurialInputScheme : InputScheme
 {
-    std::string schemeType() const override { return "hg"; }
-
-    const std::set<std::string> & allowedAttrs() const override {
-        return allowedMercurialAttrs;
-    }
-
     std::optional<Input> inputFromURL(const ParsedURL & url, bool requireTree) const override
     {
         if (url.scheme != "hg+http" &&
@@ -81,8 +67,14 @@ struct MercurialInputScheme : InputScheme
         return inputFromAttrs(attrs);
     }
 
-    Attrs preprocessAttrs(const Attrs & attrs) const override
+    std::optional<Input> inputFromAttrs(const Attrs & attrs) const override
     {
+        if (maybeGetStrAttr(attrs, "type") != "hg") return {};
+
+        for (auto & [name, value] : attrs)
+            if (name != "type" && name != "url" && name != "ref" && name != "rev" && name != "revCount" && name != "narHash" && name != "name")
+                throw Error("unsupported Mercurial input attribute '%s'", name);
+
         parseURL(getStrAttr(attrs, "url"));
 
         if (auto ref = maybeGetStrAttr(attrs, "ref")) {
@@ -90,7 +82,9 @@ struct MercurialInputScheme : InputScheme
                 throw BadURL("invalid Mercurial branch/tag name '%s'", *ref);
         }
 
-        return attrs;
+        Input input;
+        input.attrs = attrs;
+        return input;
     }
 
     ParsedURL toURL(const Input & input) const override

lix/libfetchers/path.cc

@@ -6,25 +6,8 @@
 
 namespace nix::fetchers {
 
-/* Allow the user to pass in "fake" tree info
-   attributes. This is useful for making a pinned tree
-   work the same as the repository from which is exported
-   (e.g. path:/nix/store/...-source?lastModified=1585388205&rev=b0c285...). */
-static const std::set<std::string> allowedPathAttrs = {
-    "lastModified",
-    "path",
-    "rev",
-    "revCount",
-};
-
 struct PathInputScheme : InputScheme
 {
-    std::string schemeType() const override { return "path"; }
-
-    const std::set<std::string> & allowedAttrs() const override {
-        return allowedPathAttrs;
-    }
-
     std::optional<Input> inputFromURL(const ParsedURL & url, bool requireTree) const override
     {
         if (url.scheme != "path") return {};
@@ -51,11 +34,26 @@ struct PathInputScheme : InputScheme
         return input;
     }
 
-    Attrs preprocessAttrs(const Attrs & attrs) const override
+    std::optional<Input> inputFromAttrs(const Attrs & attrs) const override
     {
+        if (maybeGetStrAttr(attrs, "type") != "path") return {};
+
         getStrAttr(attrs, "path");
 
-        return attrs;
+        for (auto & [name, value] : attrs)
+            /* Allow the user to pass in "fake" tree info
+               attributes. This is useful for making a pinned tree
+               work the same as the repository from which is exported
+               (e.g. path:/nix/store/...-source?lastModified=1585388205&rev=b0c285...). */
+            if (name == "type" || name == "rev" || name == "revCount" || name == "lastModified" || name == "narHash" || name == "path")
+                // checked in Input::fromAttrs
+                ;
+            else
+                throw Error("unsupported path input attribute '%s'", name);
+
+        Input input;
+        input.attrs = attrs;
+        return input;
     }
 
     bool isLockedByRev() const override { return false; }

lix/libfetchers/tarball.cc

@@ -230,25 +230,12 @@ try {
     co_return result::current_exception();
 }
 
-// FIXME: some of these only apply to TarballInputScheme.
-static const std::set<std::string> allowedCurlAttrs = {
-    "lastModified",
-    "name",
-    "rev",
-    "revCount",
-    "unpack",
-    "url",
-};
-
 // An input scheme corresponding to a curl-downloadable resource.
 struct CurlInputScheme : InputScheme
 {
+    virtual const std::string inputType() const = 0;
     const std::set<std::string> transportUrlSchemes = {"file", "http", "https"};
 
-    const std::set<std::string> & allowedAttrs() const override {
-        return allowedCurlAttrs;
-    }
-
     bool hasTarballExtension(std::string_view path) const
     {
         return path.ends_with(".zip") || path.ends_with(".tar")
@@ -267,7 +254,7 @@ struct CurlInputScheme : InputScheme
         auto url = _url;
 
         Attrs attrs;
-        attrs.emplace("type", schemeType());
+        attrs.emplace("type", inputType());
 
         url.scheme = parseUrlScheme(url.scheme).transport;
 
@@ -277,16 +264,24 @@ struct CurlInputScheme : InputScheme
         return inputFromAttrs(attrs);
     }
 
-    Attrs preprocessAttrs(const Attrs & attrs) const override
+    std::optional<Input> inputFromAttrs(const Attrs & attrs) const override
     {
+        auto type = maybeGetStrAttr(attrs, "type");
+        if (type != inputType()) return {};
+
+        // FIXME: some of these only apply to TarballInputScheme.
+        std::set<std::string> allowedNames = {"type", "url", "narHash", "name", "unpack", "rev", "revCount", "lastModified"};
         for (auto & [name, value] : attrs)
-            if (name != "type" && name != "narHash" && !allowedAttrs().contains(name))
-                throw UnsupportedAttributeError("unsupported tarball input attribute '%s'. If you wanted to fetch a tarball with a query parameter, please use '{ type = \"tarball\"; url = \"...\"; }'", name);
+            if (!allowedNames.count(name))
+                throw Error("unsupported %s input attribute '%s'. If you wanted to fetch a tarball with a query parameter, please use '{ type = \"tarball\"; url = \"...\"; }'", *type, name);
 
-        return attrs;
+        Input input;
+        input.attrs = attrs;
+
+        //input.locked = (bool) maybeGetStrAttr(input.attrs, "hash");
+        return input;
     }
 
-
     ParsedURL toURL(const Input & input) const override
     {
         auto url = parseURL(getStrAttr(input.attrs, "url"));
@@ -308,14 +303,14 @@ struct CurlInputScheme : InputScheme
 
 struct FileInputScheme : CurlInputScheme
 {
-    std::string schemeType() const override { return "file"; }
+    const std::string inputType() const override { return "file"; }
 
     bool isValidURL(const ParsedURL & url, bool requireTree) const override
     {
         auto parsedUrlScheme = parseUrlScheme(url.scheme);
         return transportUrlSchemes.count(std::string(parsedUrlScheme.transport))
             && (parsedUrlScheme.application
-                ? parsedUrlScheme.application.value() == schemeType()
+                ? parsedUrlScheme.application.value() == inputType()
                 : (!requireTree && !hasTarballExtension(url.path)));
     }
 
@@ -333,7 +328,7 @@ struct FileInputScheme : CurlInputScheme
 
 struct TarballInputScheme : CurlInputScheme
 {
-    std::string schemeType() const override { return "tarball"; }
+    const std::string inputType() const override { return "tarball"; }
 
     bool isValidURL(const ParsedURL & url, bool requireTree) const override
     {
@@ -341,7 +336,7 @@ struct TarballInputScheme : CurlInputScheme
 
         return transportUrlSchemes.count(std::string(parsedUrlScheme.transport))
             && (parsedUrlScheme.application
-                ? parsedUrlScheme.application.value() == schemeType()
+                ? parsedUrlScheme.application.value() == inputType()
                 : (requireTree || hasTarballExtension(url.path)));
     }
 

lix/libstore/binary-cache-store.cc

@@ -190,6 +190,7 @@ try {
 
         auto * buildIdDir = std::get_if<nar_index::Directory>(&narIndex);
         for (auto subdir : { "lib", "debug", ".build-id" }) {
+            if (!buildIdDir) break;
             // get returns nullptr subdir does not exist, and std::get_if propagates it.
             buildIdDir = std::get_if<nar_index::Directory>(get(buildIdDir->contents, subdir));
         }
@@ -505,6 +506,32 @@ try {
     co_return result::current_exception();
 }
 
+kj::Promise<Result<std::shared_ptr<const Realisation>>>
+BinaryCacheStore::queryRealisationUncached(const DrvOutput & id)
+try {
+    auto outputInfoFilePath = realisationsPrefix + "/" + id.to_string() + ".doi";
+
+    auto data = getFileContents(outputInfoFilePath);
+    if (!data) co_return result::success(nullptr);
+
+    auto realisation = Realisation::fromJSON(
+        json::parse(*data), outputInfoFilePath);
+    co_return std::make_shared<const Realisation>(realisation);
+} catch (...) {
+    co_return result::current_exception();
+}
+
+kj::Promise<Result<void>> BinaryCacheStore::registerDrvOutput(const Realisation& info)
+try {
+    if (diskCache)
+        diskCache->upsertRealisation(getUri(), info);
+    auto filePath = realisationsPrefix + "/" + info.id.to_string() + ".doi";
+    upsertFile(filePath, info.toJSON().dump(), "application/json");
+    co_return result::success();
+} catch (...) {
+    co_return result::current_exception();
+}
+
 ref<FSAccessor> BinaryCacheStore::getFSAccessor()
 {
     return make_ref<RemoteFSAccessor>(ref<Store>(*this), config().localNarCache);

lix/libstore/binary-cache-store.hh

@@ -146,6 +146,11 @@ public:
         const StorePathSet & references,
         RepairFlag repair) override;
 
+    kj::Promise<Result<void>> registerDrvOutput(const Realisation & info) override;
+
+    kj::Promise<Result<std::shared_ptr<const Realisation>>>
+    queryRealisationUncached(const DrvOutput &) override;
+
     kj::Promise<Result<box_ptr<Source>>> narFromPath(const StorePath & path) override;
 
     ref<FSAccessor> getFSAccessor() override;

lix/libstore/build/derivation-goal.cc

@@ -10,6 +10,7 @@
 #include "lix/libstore/common-protocol-impl.hh" // IWYU pragma: keep
 #include "lix/libstore/local-store.hh" // TODO remove, along with remaining downcasts
 #include "lix/libstore/build/substitution-goal.hh"
+#include "lix/libstore/build/drv-output-substitution-goal.hh"
 #include "lix/libutil/result.hh"
 #include "lix/libutil/strings.hh"
 
@@ -73,7 +74,7 @@ DerivationGoal::DerivationGoal(const StorePath & drvPath,
 {
     name = fmt(
         "building of '%s' from .drv file",
-        DerivedPath::Built { makeConstantStorePath(drvPath), wantedOutputs }.to_string(worker.store));
+        DerivedPath::Built { makeConstantStorePathRef(drvPath), wantedOutputs }.to_string(worker.store));
     trace("created");
 
     mcExpectedBuilds = worker.expectedBuilds.addTemporarily(1);
@@ -92,7 +93,7 @@ DerivationGoal::DerivationGoal(DrvHasRoot, const StorePath & drvPath, const Basi
 
     name = fmt(
         "building of '%s' from in-memory derivation",
-        DerivedPath::Built { makeConstantStorePath(drvPath), drv.outputNames() }.to_string(worker.store));
+        DerivedPath::Built { makeConstantStorePathRef(drvPath), drv.outputNames() }.to_string(worker.store));
     trace("created");
 
     mcExpectedBuilds = worker.expectedBuilds.addTemporarily(1);
@@ -198,6 +199,8 @@ try {
     /* Get the derivation. It is probably in the eval store, but it might be inthe main store:
 
          - Resolved derivation are resolved against main store realisations, and so must be stored there.
+
+         - Dynamic derivations are built, and so are found in the main store.
      */
     for (auto * drvStore : { &worker.evalStore, &worker.store }) {
         if (TRY_AWAIT(drvStore->isValidPath(drvPath))) {
@@ -219,8 +222,34 @@ try {
 
     parsedDrv = std::make_unique<ParsedDerivation>(drvPath, *drv);
 
-    for (auto & i : drv->outputsAndPaths(worker.store))
-        TRY_AWAIT(worker.store.addTempRoot(i.second.second));
+    if (!drv->type().hasKnownOutputPaths())
+        experimentalFeatureSettings.require(Xp::CaDerivations);
+
+    if (!drv->type().isPure()) {
+        experimentalFeatureSettings.require(Xp::ImpureDerivations);
+
+        for (auto & [outputName, output] : drv->outputs) {
+            auto randomPath = StorePath::random(outputPathName(drv->name, outputName));
+            assert(!TRY_AWAIT(worker.store.isValidPath(randomPath)));
+            initialOutputs.insert({
+                outputName,
+                InitialOutput {
+                    .wanted = true,
+                    .outputHash = impureOutputHash,
+                    .known = InitialOutputStatus {
+                        .path = randomPath,
+                        .status = PathStatus::Absent
+                    }
+                }
+            });
+        }
+
+        co_return co_await gaveUpOnSubstitution();
+    }
+
+    for (auto & i : drv->outputsAndOptPaths(worker.store))
+        if (i.second.second)
+            TRY_AWAIT(worker.store.addTempRoot(*i.second.second));
 
     auto outputHashes = TRY_AWAIT(staticOutputHashes(worker.evalStore, *drv));
     for (auto & [outputName, outputHash] : outputHashes)
@@ -248,13 +277,14 @@ try {
         if (parsedDrv->substitutesAllowed()) {
             for (auto & [outputName, status] : initialOutputs) {
                 if (!status.wanted) continue;
-                if (!status.known) {
-                    // TODO remove somehow
-                    throw Error(
-                        "congrats, you hit vestigial CA code. sigh.\n"
-                        "please report a bug at https://git.lix.systems/lix-project/lix/issues"
+                if (!status.known)
+                    dependencies.add(
+                        worker.goalFactory().makeDrvOutputSubstitutionGoal(
+                            DrvOutput{status.outputHash, outputName},
+                            buildMode == bmRepair ? Repair : NoRepair
+                        )
                     );
-                } else {
+                else {
                     auto * cap = getDerivationCA(*drv);
                     dependencies.add(worker.goalFactory().makePathSubstitutionGoal(
                         status.known->path,
@@ -279,6 +309,8 @@ kj::Promise<Result<Goal::WorkResult>> DerivationGoal::outputsSubstitutionTried()
 try {
     trace("all outputs substituted (maybe)");
 
+    assert(drv->type().isPure());
+
     if (nrFailed > 0 && nrFailed > nrNoSubstituters + nrIncompleteClosure && !settings.tryFallback)
     {
         co_return done(
@@ -365,18 +397,34 @@ try {
     /* The inputs must be built before we can build this goal. */
     inputDrvOutputs.clear();
     if (useDerivation) {
-        auto addWaiteeDerivedPath = [&](DerivedPathOpaque inputDrv, const StringSet & inputNode) {
-            if (!inputNode.empty())
+        std::function<void(ref<SingleDerivedPath>, const DerivedPathMap<StringSet>::ChildNode &)> addWaiteeDerivedPath;
+
+        addWaiteeDerivedPath = [&](ref<SingleDerivedPath> inputDrv, const DerivedPathMap<StringSet>::ChildNode & inputNode) {
+            if (!inputNode.value.empty())
                 dependencies.add(worker.goalFactory().makeGoal(
                     DerivedPath::Built {
                         .drvPath = inputDrv,
-                        .outputs = inputNode,
+                        .outputs = inputNode.value,
                     },
                     buildMode == bmRepair ? bmRepair : bmNormal));
+            for (const auto & [outputName, childNode] : inputNode.childMap)
+                addWaiteeDerivedPath(
+                    make_ref<SingleDerivedPath>(SingleDerivedPath::Built { inputDrv, outputName }),
+                    childNode);
         };
 
-        for (const auto & [inputDrvPath, inputNode] : dynamic_cast<Derivation *>(drv.get())->inputDrvs) {
-            addWaiteeDerivedPath(makeConstantStorePath(inputDrvPath), inputNode);
+        for (const auto & [inputDrvPath, inputNode] : dynamic_cast<Derivation *>(drv.get())->inputDrvs.map) {
+            /* Ensure that pure, non-fixed-output derivations don't
+               depend on impure derivations. */
+            if (experimentalFeatureSettings.isEnabled(Xp::ImpureDerivations) && drv->type().isPure() && !drv->type().isFixed()) {
+                auto inputDrv = TRY_AWAIT(worker.evalStore.readDerivation(inputDrvPath));
+                if (!inputDrv.type().isPure())
+                    throw Error("pure derivation '%s' depends on impure derivation '%s'",
+                        worker.store.printStorePath(drvPath),
+                        worker.store.printStorePath(inputDrvPath));
+            }
+
+            addWaiteeDerivedPath(makeConstantStorePathRef(inputDrvPath), inputNode);
         }
     }
 
@@ -412,6 +460,8 @@ try {
 
 kj::Promise<Result<Goal::WorkResult>> DerivationGoal::repairClosure() noexcept
 try {
+    assert(drv->type().isPure());
+
     /* If we're repairing, we now know that our own outputs are valid.
        Now check whether the other paths in the outputs closure are
        good.  If not, then start derivation goals for the derivations
@@ -438,9 +488,10 @@ try {
     for (auto & i : inputClosure)
         if (i.isDerivation()) {
             auto depOutputs =
-                TRY_AWAIT(worker.store.queryDerivationOutputMap(i, &worker.evalStore));
+                TRY_AWAIT(worker.store.queryPartialDerivationOutputMap(i, &worker.evalStore));
             for (auto & j : depOutputs)
-                outputsToDrv.insert_or_assign(j.second, i);
+                if (j.second)
+                    outputsToDrv.insert_or_assign(*j.second, i);
         }
 
     /* Check each path (slow!). */
@@ -456,7 +507,7 @@ try {
         else
             dependencies.add(worker.goalFactory().makeGoal(
                 DerivedPath::Built {
-                    .drvPath = makeConstantStorePath(drvPath2->second),
+                    .drvPath = makeConstantStorePathRef(drvPath2->second),
                     .outputs = OutputsSpec::All { },
                 },
                 bmRepair));
@@ -521,8 +572,68 @@ try {
     if (useDerivation) {
         auto & fullDrv = *dynamic_cast<Derivation *>(drv.get());
 
+        auto drvType = fullDrv.type();
+        bool resolveDrv = std::visit(overloaded {
+            [&](const DerivationType::InputAddressed & ia) {
+                /* must resolve if deferred. */
+                return ia.deferred;
+            },
+            [&](const DerivationType::ContentAddressed & ca) {
+                return !fullDrv.inputDrvs.map.empty() && (
+                    ca.fixed
+                    /* Can optionally resolve if fixed, which is good
+                       for avoiding unnecessary rebuilds. */
+                    ? experimentalFeatureSettings.isEnabled(Xp::CaDerivations)
+                    /* Must resolve if floating and there are any inputs
+                       drvs. */
+                    : true);
+            },
+            [&](const DerivationType::Impure &) {
+                return true;
+            }
+        }, drvType.raw);
+
+        if (resolveDrv && !fullDrv.inputDrvs.map.empty()) {
+            experimentalFeatureSettings.require(Xp::CaDerivations);
+
+            /* We are be able to resolve this derivation based on the
+               now-known results of dependencies. If so, we become a
+               stub goal aliasing that resolved derivation goal. */
+            std::optional attempt = TRY_AWAIT(fullDrv.tryResolve(worker.store, inputDrvOutputs));
+            if (!attempt) {
+              /* TODO (impure derivations-induced tech debt) (see below):
+                 The above attempt should have found it, but because we manage
+                 inputDrvOutputs statefully, sometimes it gets out of sync with
+                 the real source of truth (store). So we query the store
+                 directly if there's a problem. */
+              attempt = TRY_AWAIT(fullDrv.tryResolve(worker.store, &worker.evalStore));
+            }
+            assert(attempt);
+            Derivation drvResolved { std::move(*attempt) };
+
+            auto pathResolved = TRY_AWAIT(writeDerivation(worker.store, drvResolved));
+
+            auto msg = fmt("resolved derivation: '%s' -> '%s'",
+                worker.store.printStorePath(drvPath),
+                worker.store.printStorePath(pathResolved));
+            act = std::make_unique<Activity>(*logger, lvlInfo, actBuildWaiting, msg,
+                Logger::Fields {
+                       worker.store.printStorePath(drvPath),
+                       worker.store.printStorePath(pathResolved),
+                   });
+
+            auto dependency = worker.goalFactory().makeDerivationGoal(
+                pathResolved, wantedOutputs, buildMode);
+            resolvedDrvGoal = dependency.first;
+
+            TRY_AWAIT(waitForGoals(std::move(dependency)));
+            co_return co_await resolvedFinished();
+        }
+
+        std::function<kj::Promise<Result<void>>(const StorePath &, const DerivedPathMap<StringSet>::ChildNode &)> accumInputPaths;
+
         // NOLINTNEXTLINE(cppcoreguidelines-avoid-capturing-lambda-coroutines)
-        auto accumInputPaths = [&](const StorePath & depDrvPath, const StringSet & inputNode) -> kj::Promise<Result<void>> {
+        accumInputPaths = [&](const StorePath & depDrvPath, const DerivedPathMap<StringSet>::ChildNode & inputNode) -> kj::Promise<Result<void>> {
             try {
                 /* Add the relevant output closures of the input derivation
                    `i' as input paths.  Only add the closures of output paths
@@ -565,19 +676,21 @@ try {
                     }
                 };
 
-                for (auto & outputName : inputNode) {
+                for (auto & outputName : inputNode.value) {
                     TRY_AWAIT(
                         worker.store.computeFSClosure(TRY_AWAIT(getOutput(outputName)), inputPaths)
                     );
                 }
 
+                for (auto & [outputName, childNode] : inputNode.childMap)
+                    TRY_AWAIT(accumInputPaths(TRY_AWAIT(getOutput(outputName)), childNode));
                 co_return result::success();
             } catch (...) {
                 co_return result::current_exception();
             }
         };
 
-        for (auto & [depDrvPath, depNode] : fullDrv.inputDrvs)
+        for (auto & [depDrvPath, depNode] : fullDrv.inputDrvs.map)
             TRY_AWAIT(accumInputPaths(depDrvPath, depNode));
     }
 
@@ -622,6 +735,8 @@ retry:
        other goal can start a build, and if not, the main loop will sleep a few
        seconds and then retry this goal. */
     PathSet lockFiles;
+    /* FIXME: Should lock something like the drv itself so we don't build same
+       CA drv concurrently */
     if (dynamic_cast<LocalStore *>(&worker.store)) {
         /* If we aren't a local store, we might need to use the local store as
            a build remote, but that would cause a deadlock. */
@@ -630,8 +745,13 @@ retry:
         /* FIXME: find some way to lock for scheduling for the other stores so
            a forking daemon with --store still won't farm out redundant builds.
            */
-        for (auto & i : drv->outputsAndPaths(worker.store)) {
-            lockFiles.insert(worker.store.Store::toRealPath(i.second.second));
+        for (auto & i : drv->outputsAndOptPaths(worker.store)) {
+            if (i.second.second)
+                lockFiles.insert(worker.store.Store::toRealPath(*i.second.second));
+            else
+                lockFiles.insert(
+                    worker.store.Store::toRealPath(drvPath) + "." + i.first
+                );
         }
     }
 
@@ -765,11 +885,7 @@ void replaceValidPath(const Path & storePath, const Path & tmpPath)
        we're repairing (say) Glibc, we end up with a broken system. */
     Path oldPath;
     if (pathExists(storePath)) {
-        do {
-            oldPath = makeTempPath(storePath, ".old");
-            // store paths are often directories so we can't just unlink() it
-            // let's make sure the path doesn't exist before we try to use it
-        } while (pathExists(oldPath));
+        oldPath = makeTempSiblingPath(storePath);
         movePath(storePath, oldPath);
     }
 
@@ -891,11 +1007,11 @@ void runPostBuildHook(
         .program = settings.postBuildHook,
         .environment = hookEnvironment,
         .captureStdout = true,
-        .mergeStderrToStdout = true,
+        .redirections = {{.from = STDERR_FILENO, .to = STDOUT_FILENO}},
     });
     Finally const _wait([&] {
         try {
-            proc.wait();
+            proc.waitAndCheck();
         } catch (nix::Error & e) {
             e.addTrace(nullptr,
                 "while running the post-build-hook %s for derivation %s",
@@ -1029,6 +1145,88 @@ try {
     co_return result::current_exception();
 }
 
+kj::Promise<Result<Goal::WorkResult>> DerivationGoal::resolvedFinished() noexcept
+try {
+    trace("resolved derivation finished");
+
+    assert(resolvedDrvGoal);
+    auto resolvedDrv = *resolvedDrvGoal->drv;
+    auto & resolvedResult = resolvedDrvGoal->buildResult;
+
+    SingleDrvOutputs builtOutputs;
+
+    if (resolvedResult.success()) {
+        auto resolvedHashes = TRY_AWAIT(staticOutputHashes(worker.store, resolvedDrv));
+
+        StorePathSet outputPaths;
+
+        for (auto & outputName : resolvedDrv.outputNames()) {
+            auto initialOutput = get(initialOutputs, outputName);
+            auto resolvedHash = get(resolvedHashes, outputName);
+            if ((!initialOutput) || (!resolvedHash))
+                throw Error(
+                    "derivation '%s' doesn't have expected output '%s' (derivation-goal.cc/resolvedFinished,resolve)",
+                    worker.store.printStorePath(drvPath), outputName);
+
+            // NOLINTNEXTLINE(cppcoreguidelines-avoid-capturing-lambda-coroutines)
+            auto realisation = TRY_AWAIT([&]() -> kj::Promise<Result<Realisation>> {
+                try {
+                    auto take1 = get(resolvedResult.builtOutputs, outputName);
+                    if (take1) co_return *take1;
+
+                    /* The above `get` should work. But sateful tracking of
+                       outputs in resolvedResult, this can get out of sync with the
+                       store, which is our actual source of truth. For now we just
+                       check the store directly if it fails. */
+                    auto take2 = TRY_AWAIT(
+                        worker.evalStore.queryRealisation(DrvOutput{*resolvedHash, outputName})
+                    );
+                    if (take2) co_return *take2;
+
+                    throw Error(
+                        "derivation '%s' doesn't have expected output '%s' (derivation-goal.cc/resolvedFinished,realisation)",
+                        worker.store.printStorePath(resolvedDrvGoal->drvPath), outputName);
+                } catch (...) {
+                    co_return result::current_exception();
+                }
+            }());
+
+            if (drv->type().isPure()) {
+                auto newRealisation = realisation;
+                newRealisation.id = DrvOutput { initialOutput->outputHash, outputName };
+                newRealisation.signatures.clear();
+                if (!drv->type().isFixed()) {
+                    auto & drvStore = TRY_AWAIT(worker.evalStore.isValidPath(drvPath))
+                        ? worker.evalStore
+                        : worker.store;
+                    newRealisation.dependentRealisations = TRY_AWAIT(
+                        drvOutputReferences(worker.store, *drv, realisation.outPath, &drvStore)
+                    );
+                }
+                signRealisation(newRealisation);
+                TRY_AWAIT(worker.store.registerDrvOutput(newRealisation));
+            }
+            outputPaths.insert(realisation.outPath);
+            builtOutputs.emplace(outputName, realisation);
+        }
+
+        runPostBuildHook(
+            worker.store,
+            *logger,
+            drvPath,
+            outputPaths
+        );
+    }
+
+    auto status = resolvedResult.status;
+    if (status == BuildResult::AlreadyValid)
+        status = BuildResult::ResolvesToAlreadyValid;
+
+    co_return done(status, std::move(builtOutputs));
+} catch (...) {
+    co_return result::current_exception();
+}
+
 HookReply DerivationGoal::tryBuildHook()
 {
     if (!worker.hook.available || !useDerivation) return HookReply::Decline{};
@@ -1416,12 +1614,43 @@ void DerivationGoal::flushLine()
 }
 
 
+kj::Promise<Result<std::map<std::string, std::optional<StorePath>>>> DerivationGoal::queryPartialDerivationOutputMap()
+try {
+    assert(drv->type().isPure());
+    if (!useDerivation || drv->type().hasKnownOutputPaths()) {
+        std::map<std::string, std::optional<StorePath>> res;
+        for (auto & [name, output] : drv->outputs)
+            res.insert_or_assign(name, output.path(worker.store, drv->name, name));
+        co_return res;
+    } else {
+        for (auto * drvStore : {&worker.evalStore, &worker.store}) {
+            if (TRY_AWAIT(drvStore->isValidPath(drvPath))) {
+                co_return TRY_AWAIT(worker.store.queryPartialDerivationOutputMap(drvPath, drvStore)
+                );
+            }
+        }
+        assert(false);
+    }
+} catch (...) {
+    co_return result::current_exception();
+}
+
 kj::Promise<Result<OutputPathMap>> DerivationGoal::queryDerivationOutputMap()
 try {
-    OutputPathMap res;
-    for (auto & [name, output] : drv->outputsAndPaths(worker.store))
-        res.insert_or_assign(name, output.second);
-    co_return res;
+    assert(drv->type().isPure());
+    if (!useDerivation || drv->type().hasKnownOutputPaths()) {
+        OutputPathMap res;
+        for (auto & [name, output] : drv->outputsAndOptPaths(worker.store))
+            res.insert_or_assign(name, *output.second);
+        co_return res;
+    } else {
+        for (auto * drvStore : {&worker.evalStore, &worker.store}) {
+            if (TRY_AWAIT(drvStore->isValidPath(drvPath))) {
+                co_return TRY_AWAIT(worker.store.queryDerivationOutputMap(drvPath, drvStore));
+            }
+        }
+        assert(false);
+    }
 } catch (...) {
     co_return result::current_exception();
 }
@@ -1429,6 +1658,8 @@ try {
 
 kj::Promise<Result<std::pair<bool, SingleDrvOutputs>>> DerivationGoal::checkPathValidity()
 try {
+    if (!drv->type().isPure()) co_return { false, SingleDrvOutputs{} };
+
     bool checkHash = buildMode == bmRepair;
     auto wantedOutputsLeft = std::visit(overloaded {
         [&](const OutputsSpec::All &) {
@@ -1440,7 +1671,7 @@ try {
     }, wantedOutputs.raw);
     SingleDrvOutputs validOutputs;
 
-    for (auto & i : TRY_AWAIT(queryDerivationOutputMap())) {
+    for (auto & i : TRY_AWAIT(queryPartialDerivationOutputMap())) {
         auto initialOutput = get(initialOutputs, i.first);
         if (!initialOutput)
             // this is an invalid output, gets catched with (!wantedOutputsLeft.empty())
@@ -1449,16 +1680,37 @@ try {
         info.wanted = wantedOutputs.contains(i.first);
         if (info.wanted)
             wantedOutputsLeft.erase(i.first);
-        auto & outputPath = i.second;
-        info.known = {
-            .path = outputPath,
-            .status = !TRY_AWAIT(worker.store.isValidPath(outputPath))
-                ? PathStatus::Absent
-                : !checkHash || TRY_AWAIT(worker.pathContentsGood(outputPath))
-                ? PathStatus::Valid
-                : PathStatus::Corrupt,
-        };
+        if (i.second) {
+            auto outputPath = *i.second;
+            info.known = {
+                .path = outputPath,
+                .status = !TRY_AWAIT(worker.store.isValidPath(outputPath))
+                    ? PathStatus::Absent
+                    : !checkHash || TRY_AWAIT(worker.pathContentsGood(outputPath))
+                    ? PathStatus::Valid
+                    : PathStatus::Corrupt,
+            };
+        }
         auto drvOutput = DrvOutput{info.outputHash, i.first};
+        if (experimentalFeatureSettings.isEnabled(Xp::CaDerivations)) {
+            if (auto real = TRY_AWAIT(worker.store.queryRealisation(drvOutput))) {
+                info.known = {
+                    .path = real->outPath,
+                    .status = PathStatus::Valid,
+                };
+            } else if (info.known && info.known->isValid()) {
+                // We know the output because it's a static output of the
+                // derivation, and the output path is valid, but we don't have
+                // its realisation stored (probably because it has been built
+                // without the `ca-derivations` experimental flag).
+                TRY_AWAIT(worker.store.registerDrvOutput(
+                    Realisation {
+                        drvOutput,
+                        info.known->path,
+                    }
+                ));
+            }
+        }
         if (info.known && info.known->isValid())
             validOutputs.emplace(i.first, Realisation { drvOutput, info.known->path });
     }
@@ -1555,13 +1807,13 @@ void DerivationGoal::waiteeDone(GoalPtr waitee)
 
     auto & fullDrv = *dynamic_cast<Derivation *>(drv.get());
 
-    auto * nodeP = get(fullDrv.inputDrvs, dg->drvPath);
+    auto * nodeP = fullDrv.inputDrvs.findSlot(DerivedPath::Opaque { .path = dg->drvPath });
     if (!nodeP) return;
-    auto & outputs = *nodeP;
+    auto & outputs = nodeP->value;
 
     for (auto & outputName : outputs) {
         auto buildResult = dg->buildResult.restrictTo(DerivedPath::Built {
-            .drvPath = makeConstantStorePath(dg->drvPath),
+            .drvPath = makeConstantStorePathRef(dg->drvPath),
             .outputs = OutputsSpec::Names { outputName },
         });
         if (buildResult.success()) {

lix/libstore/build/derivation-goal.hh

@@ -87,6 +87,11 @@ struct DerivationGoal : public Goal
     /** The path of the derivation. */
     StorePath drvPath;
 
+    /**
+     * The goal for the corresponding resolved derivation
+     */
+    std::shared_ptr<DerivationGoal> resolvedDrvGoal;
+
     /**
      * The specific outputs that we need to build.
      */
@@ -274,6 +279,8 @@ struct DerivationGoal : public Goal
     virtual kj::Promise<Result<WorkResult>> tryLocalBuild() noexcept;
     kj::Promise<Result<WorkResult>> buildDone() noexcept;
 
+    kj::Promise<Result<WorkResult>> resolvedFinished() noexcept;
+
     /**
      * Is the build hook willing to perform the build?
      */
@@ -292,6 +299,11 @@ struct DerivationGoal : public Goal
      */
     Path openLogFile();
 
+    /**
+     * Sign the newly built realisation if the store allows it
+     */
+    virtual void signRealisation(Realisation&) {}
+
     /**
      * Close the log file.
      */
@@ -326,10 +338,11 @@ protected:
 
 public:
     /**
-     * Wrappers around the corresponding Store method that first consults the
+     * Wrappers around the corresponding Store methods that first consult the
      * derivation.  This is currently needed because when there is no drv file
      * there also is no DB entry.
      */
+    kj::Promise<Result<std::map<std::string, std::optional<StorePath>>>> queryPartialDerivationOutputMap();
     kj::Promise<Result<OutputPathMap>> queryDerivationOutputMap();
 
     /**
@@ -367,6 +380,8 @@ public:
         return false;
     }
 
+    StorePathSet exportReferences(const StorePathSet & storePaths);
+
     JobCategory jobCategory() const override {
         return JobCategory::Build;
     };

lix/libstore/build/drv-output-substitution-goal.cc

@@ -0,0 +1,164 @@
+#include "lix/libstore/build/drv-output-substitution-goal.hh"
+#include "lix/libstore/build-result.hh"
+#include "lix/libutil/async.hh"
+#include "lix/libutil/finally.hh"
+#include "lix/libstore/build/worker.hh"
+#include "lix/libstore/build/substitution-goal.hh"
+#include "lix/libutil/signals.hh"
+#include <kj/array.h>
+#include <kj/async.h>
+#include <kj/vector.h>
+
+namespace nix {
+
+DrvOutputSubstitutionGoal::DrvOutputSubstitutionGoal(
+    const DrvOutput & id,
+    Worker & worker,
+    bool isDependency,
+    RepairFlag repair,
+    std::optional<ContentAddress> ca)
+    : Goal(worker, isDependency)
+    , id(id)
+{
+    name = fmt("substitution of '%s'", id.to_string());
+    trace("created");
+}
+
+
+kj::Promise<Result<Goal::WorkResult>> DrvOutputSubstitutionGoal::workImpl() noexcept
+try {
+    trace("init");
+
+    /* If the derivation already exists, we’re done */
+    if (TRY_AWAIT(worker.store.queryRealisation(id))) {
+        co_return WorkResult{ecSuccess};
+    }
+
+    subs = settings.useSubstitutes ? TRY_AWAIT(getDefaultSubstituters()) : std::list<ref<Store>>();
+    co_return co_await tryNext();
+} catch (...) {
+    co_return result::current_exception();
+}
+
+kj::Promise<Result<Goal::WorkResult>> DrvOutputSubstitutionGoal::tryNext() noexcept
+try {
+    trace("trying next substituter");
+
+    if (!slotToken.valid()) {
+        slotToken = co_await worker.substitutions.acquire();
+    }
+
+    maintainRunningSubstitutions = worker.runningSubstitutions.addTemporarily(1);
+
+    if (subs.size() == 0) {
+        /* None left.  Terminate this goal and let someone else deal
+           with it. */
+        debug("derivation output '%s' is required, but there is no substituter that can provide it", id.to_string());
+
+        if (substituterFailed) {
+            worker.failedSubstitutions++;
+        }
+
+        /* Hack: don't indicate failure if there were no substituters.
+           In that case the calling derivation should just do a
+           build. */
+        co_return WorkResult{substituterFailed ? ecFailed : ecNoSubstituters};
+    }
+
+    sub = subs.front();
+    subs.pop_front();
+
+    /* The async call to a curl download below can outlive `this` (if
+       some other error occurs), so it must not touch `this`. So put
+       the shared state in a separate refcounted object. */
+    downloadState = std::make_shared<DownloadState>();
+    auto pipe = kj::newPromiseAndCrossThreadFulfiller<void>();
+    downloadState->outPipe = kj::mv(pipe.fulfiller);
+
+    downloadState->result =
+        std::async(std::launch::async, [downloadState{downloadState}, id{id}, sub{sub}] {
+            Finally updateStats([&]() { downloadState->outPipe->fulfill(); });
+            ReceiveInterrupts receiveInterrupts;
+            AsyncIoRoot aio;
+            return aio.blockOn(sub->queryRealisation(id));
+        });
+
+    co_await pipe.promise;
+    co_return co_await realisationFetched();
+} catch (...) {
+    co_return result::current_exception();
+}
+
+kj::Promise<Result<Goal::WorkResult>> DrvOutputSubstitutionGoal::realisationFetched() noexcept
+try {
+    maintainRunningSubstitutions.reset();
+    slotToken = {};
+
+    try {
+        outputInfo = downloadState->result.get();
+    } catch (std::exception & e) { // NOLINT(lix-foreign-exceptions)
+        printError(e.what());
+        substituterFailed = true;
+    }
+
+    if (!outputInfo) {
+        co_return co_await tryNext();
+    }
+
+    kj::Vector<std::pair<GoalPtr, kj::Promise<Result<WorkResult>>>> dependencies;
+    for (const auto & [depId, depPath] : outputInfo->dependentRealisations) {
+        if (depId != id) {
+            if (auto localOutputInfo = TRY_AWAIT(worker.store.queryRealisation(depId));
+                localOutputInfo && localOutputInfo->outPath != depPath) {
+                warn(
+                    "substituter '%s' has an incompatible realisation for '%s', ignoring.\n"
+                    "Local:  %s\n"
+                    "Remote: %s",
+                    sub->getUri(),
+                    depId.to_string(),
+                    worker.store.printStorePath(localOutputInfo->outPath),
+                    worker.store.printStorePath(depPath)
+                );
+                co_return co_await tryNext();
+            }
+            dependencies.add(worker.goalFactory().makeDrvOutputSubstitutionGoal(depId));
+        }
+    }
+
+    dependencies.add(worker.goalFactory().makePathSubstitutionGoal(outputInfo->outPath));
+
+    if (!dependencies.empty()) {
+        TRY_AWAIT(waitForGoals(dependencies.releaseAsArray()));
+    }
+    co_return co_await outPathValid();
+} catch (...) {
+    co_return result::current_exception();
+}
+
+kj::Promise<Result<Goal::WorkResult>> DrvOutputSubstitutionGoal::outPathValid() noexcept
+try {
+    assert(outputInfo);
+    trace("output path substituted");
+
+    if (nrFailed > 0) {
+        debug("The output path of the derivation output '%s' could not be substituted", id.to_string());
+        co_return WorkResult{
+            nrNoSubstituters > 0 || nrIncompleteClosure > 0 ? ecIncompleteClosure : ecFailed,
+        };
+    }
+
+    TRY_AWAIT(worker.store.registerDrvOutput(*outputInfo));
+    co_return TRY_AWAIT(finished());
+} catch (...) {
+    co_return result::current_exception();
+}
+
+kj::Promise<Result<Goal::WorkResult>> DrvOutputSubstitutionGoal::finished() noexcept
+try {
+    trace("finished");
+    return {WorkResult{ecSuccess}};
+} catch (...) {
+    return {result::current_exception()};
+}
+
+}

lix/libstore/build/drv-output-substitution-goal.hh

@@ -0,0 +1,80 @@
+#pragma once
+///@file
+
+#include "lix/libutil/notifying-counter.hh"
+#include "lix/libstore/store-api.hh"
+#include "lix/libstore/build/goal.hh"
+#include "lix/libstore/realisation.hh"
+#include <future>
+
+namespace nix {
+
+class Worker;
+
+/**
+ * Substitution of a derivation output.
+ * This is done in three steps:
+ * 1. Fetch the output info from a substituter
+ * 2. Substitute the corresponding output path
+ * 3. Register the output info
+ */
+class DrvOutputSubstitutionGoal : public Goal {
+
+    /**
+     * The drv output we're trying to substitute
+     */
+    DrvOutput id;
+
+    /**
+     * The realisation corresponding to the given output id.
+     * Will be filled once we can get it.
+     */
+    std::shared_ptr<const Realisation> outputInfo;
+
+    /**
+     * The remaining substituters.
+     */
+    std::list<ref<Store>> subs;
+
+    /**
+     * The current substituter.
+     */
+    std::shared_ptr<Store> sub;
+
+    NotifyingCounter<uint64_t>::Bump maintainRunningSubstitutions;
+
+    struct DownloadState
+    {
+        kj::Own<kj::CrossThreadPromiseFulfiller<void>> outPipe;
+        std::future<std::shared_ptr<const Realisation>> result;
+    };
+
+    std::shared_ptr<DownloadState> downloadState;
+
+    /**
+     * Whether a substituter failed.
+     */
+    bool substituterFailed = false;
+
+public:
+    DrvOutputSubstitutionGoal(
+        const DrvOutput & id,
+        Worker & worker,
+        bool isDependency,
+        RepairFlag repair = NoRepair,
+        std::optional<ContentAddress> ca = std::nullopt
+    );
+
+    kj::Promise<Result<WorkResult>> tryNext() noexcept;
+    kj::Promise<Result<WorkResult>> realisationFetched() noexcept;
+    kj::Promise<Result<WorkResult>> outPathValid() noexcept;
+    kj::Promise<Result<WorkResult>> finished() noexcept;
+
+    kj::Promise<Result<WorkResult>> workImpl() noexcept override;
+
+    JobCategory jobCategory() const override {
+        return JobCategory::Substitution;
+    };
+};
+
+}

lix/libstore/build/entry-points.cc

@@ -93,7 +93,7 @@ try {
         }));
         auto & result = results.goals.begin()->second;
         co_return result.result.restrictTo(DerivedPath::Built {
-            .drvPath = makeConstantStorePath(drvPath),
+            .drvPath = makeConstantStorePathRef(drvPath),
             .outputs = OutputsSpec::All {},
         });
     } catch (Error & e) {
@@ -151,7 +151,7 @@ try {
                 Worker::Targets goals;
                 goals.emplace_back(gf.makeGoal(
                     DerivedPath::Built{
-                        .drvPath = makeConstantStorePath(*info->deriver),
+                        .drvPath = makeConstantStorePathRef(*info->deriver),
                         // FIXME: Should just build the specific output we need.
                         .outputs = OutputsSpec::All{},
                     },

lix/libstore/build/local-derivation-goal.cc

@@ -13,6 +13,8 @@
 #include "lix/libutil/archive.hh"
 #include "lix/libstore/daemon.hh"
 #include "lix/libutil/regex.hh"
+#include "lix/libutil/file-descriptor.hh"
+#include "lix/libutil/file-system.hh"
 #include "lix/libutil/result.hh"
 #include "lix/libutil/topo-sort.hh"
 #include "lix/libutil/json.hh"
@@ -25,13 +27,16 @@
 #include "lix/libutil/mount.hh"
 #include "lix/libutil/strings.hh"
 #include "lix/libutil/thread-name.hh"
+#include "platform/linux.hh"
 
 #include <cstddef>
+#include <dirent.h>
 #include <exception>
 #include <regex>
 #include <queue>
 
 #include <stdexcept>
+#include <sys/stat.h>
 #include <sys/un.h>
 #include <fcntl.h>
 #include <termios.h>
@@ -110,7 +115,11 @@ LocalDerivationGoal::~LocalDerivationGoal() noexcept(false)
     /* Careful: we should never ever throw an exception from a
        destructor. */
     try { killChild(); } catch (...) { ignoreExceptionInDestructor(); }
-    try { deleteTmpDir(false, true); } catch (...) { ignoreExceptionInDestructor(); }
+    try {
+        finalizeTmpDir(false, true);
+    } catch (...) {
+        ignoreExceptionInDestructor();
+    }
 }
 
 
@@ -360,13 +369,14 @@ bool LocalDerivationGoal::cleanupDecideWhetherDiskFull()
         if (statvfs(localStore.config().realStoreDir.get().c_str(), &st) == 0 &&
             (uint64_t) st.f_bavail * st.f_bsize < required)
             diskFull = true;
-        if (statvfs(tmpDir.c_str(), &st) == 0 &&
-            (uint64_t) st.f_bavail * st.f_bsize < required)
+        if (statvfs(tmpDirRoot.c_str(), &st) == 0 && (uint64_t) st.f_bavail * st.f_bsize < required)
+        {
             diskFull = true;
+        }
     }
 #endif
 
-    deleteTmpDir(false);
+    finalizeTmpDir(false);
 
     /* Move paths out of the chroot for easier debugging of
        build failures. */
@@ -385,7 +395,7 @@ bool LocalDerivationGoal::cleanupDecideWhetherDiskFull()
 
 void LocalDerivationGoal::cleanupPostOutputsRegisteredModeCheck()
 {
-    deleteTmpDir(true);
+    finalizeTmpDir(true);
 }
 
 
@@ -480,17 +490,77 @@ try {
         });
     }
 
-    /* Create a temporary directory where the build will take
-       place. */
-    tmpDir = createTempDir(
-        settings.buildDir.get().value_or(""),
-        "nix-build-" + std::string(drvPath.name()),
-        false,
-        false,
-        0700
-    );
+    try {
+        auto buildDir = worker.buildDirOverride.value_or(settings.buildDir.get());
 
-    chownToBuilder(tmpDir);
+        createDirs(buildDir);
+
+        /* Create a temporary directory where the build will take
+           place. */
+        tmpDirRoot = createTempSubdir(buildDir, std::nullopt, 0700);
+    } catch (SysError & e) {
+        /*
+         * Fallback to the global tmpdir and create a safe space there
+         * only if it's a permission error.
+         */
+        if (e.errNo != EACCES) {
+            throw;
+        }
+
+        auto nixBuildsTmp = createTempDir(fmt("nix-builds-%s", geteuid()), 0700);
+        warn(
+            "Failed to use the system-wide build directory '%s', falling back to a temporary "
+            "directory inside '%s'",
+            settings.buildDir.get(),
+            nixBuildsTmp
+        );
+        tmpDirRoot = createTempSubdir(nixBuildsTmp, std::nullopt, 0700);
+        worker.buildDirOverride = nixBuildsTmp;
+    }
+    /* The TOCTOU between the previous mkdir call and this open call is unavoidable due to
+     * POSIX semantics.*/
+    tmpDirRootFd = AutoCloseFD{open(tmpDirRoot.c_str(), O_RDONLY | O_NOFOLLOW | O_DIRECTORY)};
+    if (!tmpDirRootFd) {
+        throw SysError("failed to open the build temporary directory descriptor '%1%'", tmpDirRoot);
+    }
+
+#if __APPLE__
+    // The Darwin sandbox ensures that builds cannot change the
+    // permissions of their own build directory. Unsandboxed builds
+    // disable this, but have no isolation by design anyway. The
+    // minimal sandbox (applied even when `sandbox = false`, though not
+    // when `_NIX_TEST_NO_SANDBOX` is set) prevents the creation of
+    // `set{u,g}id` files regardless.
+    tmpDir = tmpDirRoot;
+    tmpDirFd = std::move(tmpDirRootFd);
+#else
+    // place the actual build directory in a subdirectory of tmpDirRoot. if
+    // we do not do this a build can `chown 777` its build directory and so
+    // make it accessible to everyone in the system, breaking isolation. we
+    // also need the intermediate level to be inaccessible to others. build
+    // processes must be able to at least traverse to the directory though,
+    // without being able to chmod. this means either mode 0750 or 0710. we
+    // use 0710 just to be extra safe; if we ever add more directories they
+    // will not be enumerable to other processes in the builder user group.
+    tmpDir = tmpDirRoot + "/b";
+    if (mkdirat(tmpDirRootFd.get(), "b", 0700)) {
+        throw SysError("failed to create the build temporary directory '%1%'", tmpDir);
+    }
+    tmpDirFd = AutoCloseFD{openat(tmpDirRootFd.get(), "b", O_RDONLY | O_NOFOLLOW | O_DIRECTORY)};
+    if (!tmpDirFd)
+        throw SysError("failed to open the build temporary directory descriptor '%1%'", tmpDir);
+
+    if (buildUser) {
+        if (fchown(tmpDirRootFd.get(), -1, buildUser->getGID()) == -1) {
+            throw SysError("cannot change ownership of '%1%'", tmpDirRoot);
+        }
+        if (fchmod(tmpDirRootFd.get(), 0710) == -1) {
+            throw SysError("cannot change mode of '%1%'", tmpDirRoot);
+        }
+    }
+#endif
+
+    chownToBuilder(tmpDirFd);
 
     for (auto & [outputName, status] : initialOutputs) {
         /* Set scratch path we'll actually use during the build.
@@ -858,8 +928,13 @@ void LocalDerivationGoal::initTmpDir() {
                 auto hash = hashString(HashType::SHA256, i.first);
                 std::string fn = ".attr-" + hash.to_string(Base::Base32, false);
                 Path p = tmpDir + "/" + fn;
-                writeFile(p, rewriteStrings(i.second, inputRewrites));
-                chownToBuilder(p);
+                /* TODO(jade): we should have BorrowedFD instead of OwnedFD. */
+                AutoCloseFD passAsFileFd{openat(tmpDirFd.get(), fn.c_str(), O_WRONLY | O_TRUNC | O_CREAT | O_CLOEXEC | O_EXCL | O_NOFOLLOW, 0666)};
+                if (!passAsFileFd) {
+                    throw SysError("opening `passAsFile` file in the sandbox '%1%'", p);
+                }
+                writeFile(passAsFileFd, rewriteStrings(i.second, inputRewrites));
+                chownToBuilder(passAsFileFd);
                 env[i.first + "Path"] = tmpDirInSandbox + "/" + fn;
             }
         }
@@ -975,6 +1050,13 @@ void LocalDerivationGoal::chownToBuilder(const Path & path)
         throw SysError("cannot change ownership of '%1%'", path);
 }
 
+void LocalDerivationGoal::chownToBuilder(const AutoCloseFD & fd)
+{
+    if (!buildUser) return;
+    if (fchown(fd.get(), buildUser->getUID(), buildUser->getGID()) == -1)
+        throw SysError("cannot change ownership of file '%1%'", fd.guessOrInventPath());
+}
+
 
 void LocalDerivationGoal::runChild()
 {
@@ -1105,7 +1187,7 @@ void LocalDerivationGoal::runChild()
                 /* N.B. it is realistic that these paths might not exist. It
                    happens when testing Nix building fixed-output derivations
                    within a pure derivation. */
-                for (auto & path : { "/etc/resolv.conf", "/etc/services", "/etc/hosts" })
+                for (auto & path : { "/etc/services", "/etc/hosts" })
                     if (pathExists(path)) {
                         // Copy the actual file, not the symlink, because we don't know where
                         // the symlink is pointing, and we don't want to chase down the entire
@@ -1126,6 +1208,11 @@ void LocalDerivationGoal::runChild()
                         copyFile(path, chrootRootDir + path, { .followSymlinks = true });
                     }
 
+                if (pathExists("/etc/resolv.conf")) {
+                    const auto resolvConf = rewriteResolvConf(readFile("/etc/resolv.conf"));
+                    writeFile(chrootRootDir + "/etc/resolv.conf", resolvConf);
+                }
+
                 if (settings.caFile != "" && pathExists(settings.caFile)) {
                     // For the same reasons as above, copy the CA certificates file too.
                     // It should be even less likely to change during the build than resolv.conf.
@@ -1253,6 +1340,36 @@ void LocalDerivationGoal::runChild()
             if (setuid(sandboxUid()) == -1)
                 throw SysError("setuid failed");
 
+            if (runPasta) {
+                // wait for the pasta interface to appear. pasta can't signal us when
+                // it's done setting up the namespace, so we have to wait for a while
+                AutoCloseFD fd(socket(PF_INET, SOCK_DGRAM, IPPROTO_IP));
+                if (!fd) throw SysError("cannot open IP socket");
+
+                struct ifreq ifr;
+                strcpy(ifr.ifr_name, LinuxLocalDerivationGoal::PASTA_NS_IFNAME);
+                // wait two minutes for the interface to appear. if it does not do so
+                // we are either grossly overloaded, or pasta startup failed somehow.
+                static constexpr int SINGLE_WAIT_US = 1000;
+                static constexpr int TOTAL_WAIT_US = 120'000'000;
+                for (unsigned tries = 0; ; tries++) {
+                    if (tries > TOTAL_WAIT_US / SINGLE_WAIT_US) {
+                        throw Error(
+                            "sandbox network setup timed out, please check daemon logs for "
+                            "possible error output."
+                        );
+                    } else if (ioctl(fd.get(), SIOCGIFFLAGS, &ifr) == 0) {
+                        if ((ifr.ifr_ifru.ifru_flags & IFF_UP) != 0) {
+                            break;
+                        }
+                    } else if (errno == ENODEV) {
+                        usleep(SINGLE_WAIT_US);
+                    } else {
+                        throw SysError("cannot get loopback interface flags");
+                    }
+                }
+            }
+
             setUser = false;
         }
 #endif
@@ -1424,6 +1541,8 @@ void LocalDerivationGoal::runChild()
 
         if (getEnv("_NIX_TEST_NO_SANDBOX") != "1") {
             Strings sandboxArgs;
+            sandboxArgs.push_back("_NIX_BUILD_TOP");
+            sandboxArgs.push_back(tmpDir);
             sandboxArgs.push_back("_GLOBAL_TMP_DIR");
             sandboxArgs.push_back(globalTmpDir);
             if (allowLocalNetworking) {
@@ -1739,14 +1858,14 @@ try {
             return res;
         };
 
-        auto newInfoFromCA = [&](ContentAddressMethod method, HashType hashType) -> ValidPathInfo {
+        auto newInfoFromCA = [&](const DerivationOutput::CAFloating outputHash) -> ValidPathInfo {
             auto st = get(outputStats, outputName);
             if (!st)
                 throw BuildError(
                     "output path %1% without valid stats info",
                     actualPath);
-            if (method == ContentAddressMethod { FileIngestionMethod::Flat } ||
-                method == ContentAddressMethod { TextIngestionMethod {} })
+            if (outputHash.method == ContentAddressMethod { FileIngestionMethod::Flat } ||
+                outputHash.method == ContentAddressMethod { TextIngestionMethod {} })
             {
                 /* The output path should be a regular file without execute permission. */
                 if (!S_ISREG(st->st_mode) || (st->st_mode & S_IXUSR) != 0)
@@ -1771,11 +1890,11 @@ try {
                     }
                     assert(false);
                 },
-            }, method.raw);
-            auto got = computeHashModulo(hashType, oldHashPart, input).first;
+            }, outputHash.method.raw);
+            auto got = computeHashModulo(outputHash.hashType, oldHashPart, input).first;
 
             auto optCA = ContentAddressWithReferences::fromPartsOpt(
-                method,
+                outputHash.method,
                 std::move(got),
                 rewriteRefs());
             if (!optCA) {
@@ -1840,7 +1959,10 @@ try {
                 movePath(actualPath, tmpOutput);
                 copyFile(tmpOutput, actualPath, { .deleteAfter = true });
 
-                auto newInfo0 = newInfoFromCA(dof.ca.method, wanted.type);
+                auto newInfo0 = newInfoFromCA(DerivationOutput::CAFloating {
+                    .method = dof.ca.method,
+                    .hashType = wanted.type,
+                });
 
                 /* Check wanted hash */
                 assert(newInfo0.ca);
@@ -1852,13 +1974,11 @@ try {
                     // XXX: shameless layering violation hack that makes the hash mismatch error at least not utterly worthless
                     auto guessedUrl = getOr(drv->env, "urls", getOr(drv->env, "url", "(unknown)"));
                     delayedException = std::make_exception_ptr(
-                        BuildError("hash mismatch in fixed-output derivation '%s':\n likely URL: %s\n  specified: %s\n     got:    %s\n expected path: %s\n   got path: %s",
+                        BuildError("hash mismatch in fixed-output derivation '%s':\n likely URL: %s\n  specified: %s\n     got:    %s",
                             worker.store.printStorePath(drvPath),
                             guessedUrl,
                             wanted.to_string(Base::SRI, true),
-                            got.to_string(Base::SRI, true),
-                            worker.store.printStorePath(dof.path(worker.store, drv->name, outputName)),
-                            worker.store.printStorePath(newInfo0.path)));
+                            got.to_string(Base::SRI, true)));
                 }
                 if (!newInfo0.references.empty()) {
                     std::string references;
@@ -1878,6 +1998,24 @@ try {
 
                 return newInfo0;
             },
+
+            [&](const DerivationOutput::CAFloating & dof) {
+                return newInfoFromCA(dof);
+            },
+
+            [&](const DerivationOutput::Deferred &) -> ValidPathInfo {
+                // No derivation should reach that point without having been
+                // rewritten first
+                assert(false);
+            },
+
+            [&](const DerivationOutput::Impure & doi) {
+                return newInfoFromCA(DerivationOutput::CAFloating {
+                    .method = doi.method,
+                    .hashType = doi.hashType,
+                });
+            },
+
         }, output->raw);
 
         /* FIXME: set proper permissions in restorePath() so
@@ -1893,8 +2031,10 @@ try {
            floating CA derivations and hash-mismatching fixed-output
            derivations. */
         std::optional<PathLock> dynamicOutputLock;
-        auto fixedPath = output->path(worker.store, drv->name, outputName);
-        if (worker.store.printStorePath(fixedPath) != finalDestPath) {
+        auto optFixedPath = output->path(worker.store, drv->name, outputName);
+        if (!optFixedPath ||
+            worker.store.printStorePath(*optFixedPath) != finalDestPath)
+        {
             assert(newInfo.ca);
             dynamicOutputLock = TRY_AWAIT(lockPathAsync(worker.store.toRealPath(finalDestPath)));
         }
@@ -1905,10 +2045,9 @@ try {
                 /* Path already exists, need to replace it */
                 replaceValidPath(worker.store.toRealPath(finalDestPath), actualPath);
                 actualPath = worker.store.toRealPath(finalDestPath);
-            } else if (buildMode == bmCheck && TRY_AWAIT(worker.store.isValidPath(newInfo.path))) {
-                /* Path already exists, and we want to compare, so we
-                   don't replace the previously existing output with
-                   the new one. */
+            } else if (buildMode == bmCheck) {
+                /* Path already exists, and we want to compare, so we leave out
+                   new path in place. */
             } else if (TRY_AWAIT(worker.store.isValidPath(newInfo.path))) {
                 /* Path already exists because CA path produced by something
                    else. No moving needed. */
@@ -1923,12 +2062,8 @@ try {
 
         auto & localStore = getLocalStore();
 
-        // Check determinism and run the diff hook for input-addressed
-        // paths if we're in check mode.
-        // TODO: implement this for content-addressed paths too.
-        if (buildMode == bmCheck && !newInfo.ca) {
+        if (buildMode == bmCheck) {
 
-            // We can only do this if we have a previous output path to compare.
             if (!TRY_AWAIT(worker.store.isValidPath(newInfo.path))) continue;
             ValidPathInfo oldInfo(*TRY_AWAIT(worker.store.queryPathInfo(newInfo.path)));
             if (newInfo.narHash != oldInfo.narHash) {
@@ -1999,11 +2134,12 @@ try {
             }
             throw NotDeterministic(msg.str());
         }
-        /* In case of fixed-output derivations with hash mismatches,
-           we don't want to rethrow the exception until later so that
-           the unexpected path is still registered as valid. */
-        if (!delayedException)
-            co_return TRY_AWAIT(assertPathValidity());
+        /* In case of fixed-output derivations, if there are
+           mismatches on `--check` an error must be thrown as this is
+           also a source for non-determinism. */
+        if (delayedException)
+            std::rethrow_exception(delayedException);
+        co_return TRY_AWAIT(assertPathValidity());
     }
 
     /* Apply output checks. */
@@ -2044,6 +2180,12 @@ try {
             },
             .outPath = newInfo.path
         };
+        if (experimentalFeatureSettings.isEnabled(Xp::CaDerivations)
+            && drv->type().isPure())
+        {
+            signRealisation(thisRealisation);
+            TRY_AWAIT(worker.store.registerDrvOutput(thisRealisation));
+        }
         builtOutputs.emplace(outputName, thisRealisation);
     }
 
@@ -2052,6 +2194,11 @@ try {
     co_return result::current_exception();
 }
 
+void LocalDerivationGoal::signRealisation(Realisation & realisation)
+{
+    getLocalStore().signRealisation(realisation);
+}
+
 
 kj::Promise<Result<void>> LocalDerivationGoal::checkOutputs(const std::map<std::string, ValidPathInfo> & newlyBuiltOutputs, const std::map<std::string, StorePath> & alreadyRegisteredOutputs)
 try {
@@ -2269,21 +2416,78 @@ try {
     co_return result::current_exception();
 }
 
-
-void LocalDerivationGoal::deleteTmpDir(bool force, bool duringDestruction)
+// make `entry` in `parentFd` visible to the given user and group, preserving
+// inode modes as much as possible. if the builder sets the mode of any inode
+// to not be readable by the owner we keep this; not doing so could interfere
+// with error analysis. if the builder used multiple uids or gids we will not
+// keep them around and instead collapse them all onto the uid/gid given here
+// to not leave around inodes owned by unassigned uids/gids in the system. we
+// also clear setuid/setgid/sticky bits just to be safe even though a builder
+// should not be able to set them to begin, otherwise we may leave setuid/gid
+// executables in the tree even with user/group set to -1/-1. there have been
+// enough bugs of this kind in the past to warrant some extra attention here.
+static void makeVisible(int parentFd, const char * entry, uid_t user, gid_t group)
 {
-    if (tmpDir != "") {
+    struct stat st;
+    if (fstatat(parentFd, entry, &st, AT_SYMLINK_NOFOLLOW)) {
+        throw SysError("fstat(%s)", guessOrInventPathFromFD(parentFd));
+    }
+    if (S_ISDIR(st.st_mode)) {
+        int dirfd = openat(parentFd, entry, O_RDONLY | O_DIRECTORY | O_NOFOLLOW);
+        if (dirfd < 0) {
+            throw SysError("openat(%s/%s)", guessOrInventPathFromFD(parentFd), entry);
+        }
+        AutoCloseDir dir(fdopendir(dirfd));
+        if (!dir) {
+            close(dirfd);
+            throw SysError("fdopendir(%s/%s)", guessOrInventPathFromFD(parentFd), entry);
+        }
+
+        struct dirent * dirent;
+        while (errno = 0, dirent = readdir(dir.get())) {
+            if (strcmp(dirent->d_name, ".") == 0 || strcmp(dirent->d_name, "..") == 0) {
+                continue;
+            }
+            makeVisible(dirfd, dirent->d_name, user, group);
+        }
+    }
+
+    // ignore permissions errors for symlinks. linux can't chmod them.
+    // clear special permission bits while we're here, just to be safe
+    if (fchmodat(parentFd, entry, st.st_mode & 0777, AT_SYMLINK_NOFOLLOW) && !S_ISLNK(st.st_mode)) {
+        throw SysError("fchmod(%s)", guessOrInventPathFromFD(parentFd));
+    }
+    if (user != uid_t(-1) && group != gid_t(-1)
+        && fchownat(parentFd, entry, user, group, AT_SYMLINK_NOFOLLOW))
+    {
+        throw SysError("fchown(%s)", guessOrInventPathFromFD(parentFd));
+    }
+}
+
+void LocalDerivationGoal::finalizeTmpDir(bool force, bool duringDestruction)
+{
+    if (tmpDirRoot != "") {
         /* Don't keep temporary directories for builtins because they
            might have privileged stuff (like a copy of netrc). */
         if (settings.keepFailed && !force && !drv->isBuiltin()) {
-            printError("note: keeping build directory '%s'", tmpDir);
-            chmod(tmpDir.c_str(), 0755);
+            printError("note: keeping build directory '%s'", tmpDirRoot);
+            try {
+                // always make visible, but don't always chown. if we run as
+                // root we may not want to chown things to root:root so much
+                auto creds = worker.store.associatedCredentials();
+                makeVisible(
+                    tmpDirFd.get(), ".", creds ? creds->user : -1, creds ? creds->group : -1
+                );
+            } catch (SysError & e) {
+                printError("error making '%s' accessible: %s", tmpDir, e.what());
+            }
+            chmod(tmpDirRoot.c_str(), 0755);
         }
         else if (duringDestruction)
-            deletePathUninterruptible(tmpDir);
+            deletePathUninterruptible(tmpDirRoot);
         else
-            deletePath(tmpDir);
-        tmpDir = "";
+            deletePath(tmpDirRoot);
+        tmpDirRoot = "";
     }
 }
 

lix/libstore/build/local-derivation-goal.hh

@@ -29,7 +29,12 @@ struct LocalDerivationGoal : public DerivationGoal
     /**
      * The temporary directory.
      */
-    Path tmpDir;
+    Path tmpDirRoot, tmpDir;
+
+    /**
+     * The temporary directory file descriptor
+     */
+    AutoCloseFD tmpDirRootFd, tmpDirFd;
 
     /**
      * The path of the temporary directory in the sandbox.
@@ -127,6 +132,11 @@ struct LocalDerivationGoal : public DerivationGoal
 
     const static Path homeDir;
 
+    /**
+     * The daemon worker threads.
+     */
+    std::vector<std::thread> daemonWorkerThreads;
+
     /**
      * Create a LocalDerivationGoal without an on-disk .drv file,
      * possibly a platform-specific subclass
@@ -186,10 +196,18 @@ struct LocalDerivationGoal : public DerivationGoal
     kj::Promise<Result<void>> writeStructuredAttrs();
 
     /**
-     * Make a file owned by the builder.
+     * Make a file owned by the builder addressed by its path.
+     *
+     * SAFETY: this function is prone to TOCTOU as it receives a path and not a descriptor.
+     * It's only safe to call in a child of a directory only visible to the owner.
      */
     void chownToBuilder(const Path & path);
 
+    /**
+     * Make a file owned by the builder addressed by its file descriptor.
+     */
+    void chownToBuilder(const AutoCloseFD & fd);
+
     int getChildStatus() override;
 
     /**
@@ -203,6 +221,8 @@ struct LocalDerivationGoal : public DerivationGoal
      */
     kj::Promise<Result<SingleDrvOutputs>> registerOutputs() override;
 
+    void signRealisation(Realisation &) override;
+
     /**
      * Check that an output meets the requirements specified by the
      * 'outputChecks' attribute (or the legacy
@@ -226,9 +246,12 @@ struct LocalDerivationGoal : public DerivationGoal
     void cleanupPostOutputsRegisteredModeNonCheck() override;
 
     /**
-     * Delete the temporary directory, if we have one.
+     * Delete the temporary directory or make it visible to the user requesting
+     * this build, if a temporary directory was created at all. Temporary files
+     * of derivations using builtin builders are deleted even for `keep-failed`
+     * builds as otherwise we may expose secrets (e.g. from the system .netrc).
      */
-    void deleteTmpDir(bool force, bool duringDestruction = false);
+    void finalizeTmpDir(bool force, bool duringDestruction = false);
 
     /**
      * Forcibly kill the child process, if any.
@@ -262,6 +285,12 @@ struct LocalDerivationGoal : public DerivationGoal
 protected:
     using DerivationGoal::DerivationGoal;
 
+    /**
+     * Whether to run pasta for network-endowed derivations. Running pasta
+     * currently requires actively waiting for its net-ns setup to finish.
+     */
+    bool runPasta = false;
+
     /**
      * Setup dependencies outside the sandbox.
      * Called in the parent nix process.
@@ -271,6 +300,15 @@ protected:
         throw Error("sandboxing builds is not supported on this platform");
     };
 
+    /**
+     * Rewrite resolv.conf for use in the sandbox. Used in the linux platform
+     * to replace nameservers * when using pasta for fixed output derivations.
+     */
+    virtual std::string rewriteResolvConf(std::string fromHost)
+    {
+        return fromHost;
+    }
+
     /**
      * Create a new process that runs `openSlave` and `runChild`
      * On some platforms this process is created with sandboxing flags.

lix/libstore/build/sandbox-defaults.sb

@@ -26,12 +26,28 @@ R""(
 ; Allow getpwuid.
 (allow mach-lookup (global-name "com.apple.system.opendirectoryd.libinfo"))
 
-; Access to /tmp.
+; Disallow messing with the top‐level build directory.
+(deny file-write-owner file-write-flags file-write-xattr file-write-mount
+      file-write-unmount
+      (literal (param "_NIX_BUILD_TOP")))
+; Nixpkgs does `chmod -R` on `$NIX_BUILD_TOP/$sourceRoot` by default,
+; which results in it trying to set the mode of `$NIX_BUILD_TOP` when
+; derivations set `sourceRoot = ".";`. Thankfully, the GNU `chmod(1)`
+; treats `ENOTSUP` as a non‐fatal, non‐reported error in this case, and
+; continues to descend into the directory tree.
+;
+; See: <https://gitweb.git.savannah.gnu.org/gitweb/?p=coreutils.git;a=blob;f=src/chmod.c;hb=refs/tags/v9.7#l312>
+(deny file-write-mode (with errno ENOTSUP)
+      (literal (param "_NIX_BUILD_TOP")))
+
+; Access to /tmp and the build directory.
 ; The network-outbound/network-inbound ones are for unix domain sockets, which
 ; we allow access to in TMPDIR (but if we allow them more broadly, you could in
 ; theory escape the sandbox)
 (allow file* process-exec network-outbound network-inbound
-       (literal "/tmp") (subpath TMPDIR))
+       (literal "/tmp")
+       (subpath TMPDIR)
+       (subpath (param "_NIX_BUILD_TOP")))
 
 ; Some packages like to read the system version.
 (allow file-read*

lix/libstore/build/worker.cc

@@ -5,6 +5,7 @@
 #include "lix/libstore/build/worker.hh"
 #include "lix/libutil/finally.hh"
 #include "lix/libstore/build/substitution-goal.hh"
+#include "lix/libstore/build/drv-output-substitution-goal.hh"
 #include "lix/libstore/build/local-derivation-goal.hh"
 #include "lix/libutil/signals.hh"
 #include "lix/libstore/build/hook-instance.hh" // IWYU pragma: keep
@@ -49,6 +50,7 @@ Worker::~Worker()
     children.clear();
 
     derivationGoals.clear();
+    drvOutputSubstitutionGoals.clear();
     substitutionGoals.clear();
 
     assert(expectedSubstitutions == 0);
@@ -181,11 +183,28 @@ Worker::makePathSubstitutionGoal(
 }
 
 
+std::pair<std::shared_ptr<DrvOutputSubstitutionGoal>, kj::Promise<Result<Goal::WorkResult>>>
+Worker::makeDrvOutputSubstitutionGoal(
+    const DrvOutput & id, RepairFlag repair, std::optional<ContentAddress> ca
+)
+{
+    return makeGoalCommon(
+        drvOutputSubstitutionGoals,
+        id,
+        [&] { return std::make_unique<DrvOutputSubstitutionGoal>(id, *this, running, repair, ca); },
+        [&](auto &) { return true; }
+    );
+}
+
+
 std::pair<GoalPtr, kj::Promise<Result<Goal::WorkResult>>> Worker::makeGoal(const DerivedPath & req, BuildMode buildMode)
 {
     return std::visit(overloaded {
         [&](const DerivedPath::Built & bfd) -> std::pair<GoalPtr, kj::Promise<Result<Goal::WorkResult>>> {
-            return makeDerivationGoal(bfd.drvPath.path, bfd.outputs, buildMode);
+            if (auto bop = std::get_if<DerivedPath::Opaque>(&*bfd.drvPath))
+                return makeDerivationGoal(bop->path, bfd.outputs, buildMode);
+            else
+                throw UnimplementedError("Building dynamic derivations in one shot is not yet implemented.");
         },
         [&](const DerivedPath::Opaque & bo) -> std::pair<GoalPtr, kj::Promise<Result<Goal::WorkResult>>> {
             return makePathSubstitutionGoal(bo.path, buildMode == bmRepair ? Repair : NoRepair);

lix/libstore/build/worker.hh

@@ -21,6 +21,7 @@ namespace nix {
 /* Forward definition. */
 struct DerivationGoal;
 struct PathSubstitutionGoal;
+class DrvOutputSubstitutionGoal;
 class LocalStore;
 
 typedef std::chrono::time_point<std::chrono::steady_clock> steady_time_point;
@@ -53,6 +54,12 @@ public:
         RepairFlag repair = NoRepair,
         std::optional<ContentAddress> ca = std::nullopt
     ) = 0;
+    virtual std::pair<std::shared_ptr<DrvOutputSubstitutionGoal>, kj::Promise<Result<Goal::WorkResult>>>
+    makeDrvOutputSubstitutionGoal(
+        const DrvOutput & id,
+        RepairFlag repair = NoRepair,
+        std::optional<ContentAddress> ca = std::nullopt
+    ) = 0;
 
     /**
      * Make a goal corresponding to the `DerivedPath`.
@@ -69,6 +76,7 @@ class WorkerBase : protected GoalFactory
 {
     friend struct DerivationGoal;
     friend struct PathSubstitutionGoal;
+    friend class DrvOutputSubstitutionGoal;
 
 protected:
     GoalFactory & goalFactory() { return *this; }
@@ -131,6 +139,7 @@ private:
      */
     std::map<StorePath, CachedGoal<DerivationGoal>> derivationGoals;
     std::map<StorePath, CachedGoal<PathSubstitutionGoal>> substitutionGoals;
+    std::map<DrvOutput, CachedGoal<DrvOutputSubstitutionGoal>> drvOutputSubstitutionGoals;
 
     /**
      * Cache for pathContentsGood().
@@ -186,6 +195,7 @@ public:
     Store & store;
     Store & evalStore;
     AsyncSemaphore substitutions, localBuilds;
+    std::optional<Path> buildDirOverride;
 
 private:
     kj::TaskSet children;
@@ -256,6 +266,12 @@ private:
         RepairFlag repair = NoRepair,
         std::optional<ContentAddress> ca = std::nullopt
     ) override;
+    std::pair<std::shared_ptr<DrvOutputSubstitutionGoal>, kj::Promise<Result<Goal::WorkResult>>>
+    makeDrvOutputSubstitutionGoal(
+        const DrvOutput & id,
+        RepairFlag repair = NoRepair,
+        std::optional<ContentAddress> ca = std::nullopt
+    ) override;
 
     /**
      * Make a goal corresponding to the `DerivedPath`.

lix/libstore/ca-specific-schema.sql

@@ -0,0 +1,41 @@
+-- Extension of the sql schema for content-addressed derivations.
+-- Won't be loaded unless the experimental feature `ca-derivations`
+-- is enabled
+
+create table if not exists Realisations (
+    id integer primary key autoincrement not null,
+    drvPath text not null,
+    outputName text not null, -- symbolic output id, usually "out"
+    outputPath integer not null,
+    signatures text, -- space-separated list
+    foreign key (outputPath) references ValidPaths(id) on delete cascade
+);
+
+create index if not exists IndexRealisations on Realisations(drvPath, outputName);
+
+-- We can end-up in a weird edge-case where a path depends on itself because
+-- it’s an output of a CA derivation, that happens to be the same as one of its
+-- dependencies.
+-- In that case we have a dependency loop (path -> realisation1 -> realisation2
+-- -> path) that we need to break by removing the dependencies between the
+-- realisations
+create trigger if not exists DeleteSelfRefsViaRealisations before delete on ValidPaths
+  begin
+    delete from RealisationsRefs where realisationReference in (
+      select id from Realisations where outputPath = old.id
+    );
+  end;
+
+create table if not exists RealisationsRefs (
+    referrer integer not null,
+    realisationReference integer,
+    foreign key (referrer) references Realisations(id) on delete cascade,
+    foreign key (realisationReference) references Realisations(id) on delete restrict
+);
+-- used by deletion trigger
+create index if not exists IndexRealisationsRefsRealisationReference on RealisationsRefs(realisationReference);
+
+-- used by QueryRealisationReferences
+create index if not exists IndexRealisationsRefs on RealisationsRefs(referrer);
+-- used by cascade deletion when ValidPaths is deleted
+create index if not exists IndexRealisationsRefsOnOutputPath on Realisations(outputPath);

lix/libstore/crypto.cc

@@ -45,8 +45,15 @@ std::string SecretKey::signDetached(std::string_view data) const
 {
     unsigned char sig[crypto_sign_BYTES];
     unsigned long long sigLen;
-    crypto_sign_detached(sig, &sigLen, charptr_cast<const unsigned char *>(data.data()), data.size(),
-        charptr_cast<const unsigned char *>(key.data()));
+    crypto_sign_detached(
+        sig,
+        &sigLen,
+        // the following is not a string function so no null termination issues are possible here.
+        // NOLINTNEXTLINE(bugprone-suspicious-stringview-data-usage)
+        charptr_cast<const unsigned char *>(data.data()),
+        data.size(),
+        charptr_cast<const unsigned char *>(key.data())
+    );
     return name + ":" + base64Encode(std::string(reinterpret_cast<char *>(sig), sigLen));
 }
 

lix/libstore/daemon.cc

@@ -387,7 +387,7 @@ static void performOp(AsyncIoRoot & aio, TunnelLogger * logger, ref<Store> store
     case WorkerProto::Op::QueryDerivationOutputMap: {
         auto path = store->parseStorePath(readString(from));
         logger->startWork();
-        auto outputs = aio.blockOn(store->queryDerivationOutputMap(path));
+        auto outputs = aio.blockOn(store->queryPartialDerivationOutputMap(path));
         logger->stopWork();
         to << WorkerProto::write(*store, wconn, outputs);
         break;
@@ -946,9 +946,36 @@ static void performOp(AsyncIoRoot & aio, TunnelLogger * logger, ref<Store> store
         break;
     }
 
-    case WorkerProto::Op::RegisterDrvOutput:
+    case WorkerProto::Op::RegisterDrvOutput: {
+        logger->startWork();
+        if (GET_PROTOCOL_MINOR(clientVersion) < 31) {
+            auto outputId = DrvOutput::parse(readString(from));
+            auto outputPath = StorePath(readString(from));
+            aio.blockOn(store->registerDrvOutput(Realisation{
+                .id = outputId, .outPath = outputPath}));
+        } else {
+            auto realisation = WorkerProto::Serialise<Realisation>::read(*store, rconn);
+            aio.blockOn(store->registerDrvOutput(realisation));
+        }
+        logger->stopWork();
+        break;
+    }
+
     case WorkerProto::Op::QueryRealisation: {
-        throw UnimplementedError("ca derivations are not supported");
+        logger->startWork();
+        auto outputId = DrvOutput::parse(readString(from));
+        auto info = aio.blockOn(store->queryRealisation(outputId));
+        logger->stopWork();
+        if (GET_PROTOCOL_MINOR(clientVersion) < 31) {
+            std::set<StorePath> outPaths;
+            if (info) outPaths.insert(info->outPath);
+            to << WorkerProto::write(*store, wconn, outPaths);
+        } else {
+            std::set<Realisation> realisations;
+            if (info) realisations.insert(*info);
+            to << WorkerProto::write(*store, wconn, realisations);
+        }
+        break;
     }
 
     case WorkerProto::Op::AddBuildLog: {

lix/libstore/derivations.cc

@@ -1,4 +1,5 @@
 #include "lix/libstore/derivations.hh"
+#include "lix/libstore/downstream-placeholder.hh"
 #include "lix/libstore/store-api.hh"
 #include "lix/libstore/globals.hh"
 #include "lix/libutil/json.hh"
@@ -13,14 +14,25 @@
 
 namespace nix {
 
-StorePath DerivationOutput::path(const Store & store, std::string_view drvName, OutputNameView outputName) const
+std::optional<StorePath> DerivationOutput::path(const Store & store, std::string_view drvName, OutputNameView outputName) const
 {
     return std::visit(overloaded {
-        [](const DerivationOutput::InputAddressed & doi) -> StorePath {
-            return doi.path;
+        [](const DerivationOutput::InputAddressed & doi) -> std::optional<StorePath> {
+            return { doi.path };
         },
-        [&](const DerivationOutput::CAFixed & dof) -> StorePath {
-            return dof.path(store, drvName, outputName);
+        [&](const DerivationOutput::CAFixed & dof) -> std::optional<StorePath> {
+            return {
+                dof.path(store, drvName, outputName)
+            };
+        },
+        [](const DerivationOutput::CAFloating & dof) -> std::optional<StorePath> {
+            return std::nullopt;
+        },
+        [](const DerivationOutput::Deferred &) -> std::optional<StorePath> {
+            return std::nullopt;
+        },
+        [](const DerivationOutput::Impure &) -> std::optional<StorePath> {
+            return std::nullopt;
         },
     }, raw);
 }
@@ -47,6 +59,9 @@ bool DerivationType::isCA() const
         [](const ContentAddressed & ca) {
             return true;
         },
+        [](const Impure &) {
+            return true;
+        },
     }, raw);
 }
 
@@ -57,11 +72,30 @@ bool DerivationType::isFixed() const
             return false;
         },
         [](const ContentAddressed & ca) {
-            return true;
+            return ca.fixed;
+        },
+        [](const Impure &) {
+            return false;
         },
     }, raw);
 }
 
+bool DerivationType::hasKnownOutputPaths() const
+{
+    return std::visit(overloaded {
+        [](const InputAddressed & ia) {
+            return !ia.deferred;
+        },
+        [](const ContentAddressed & ca) {
+            return ca.fixed;
+        },
+        [](const Impure &) {
+            return false;
+        },
+    }, raw);
+}
+
+
 bool DerivationType::isSandboxed() const
 {
     return std::visit(overloaded {
@@ -69,6 +103,25 @@ bool DerivationType::isSandboxed() const
             return true;
         },
         [](const ContentAddressed & ca) {
+            return ca.sandboxed;
+        },
+        [](const Impure &) {
+            return false;
+        },
+    }, raw);
+}
+
+
+bool DerivationType::isPure() const
+{
+    return std::visit(overloaded {
+        [](const InputAddressed & ia) {
+            return true;
+        },
+        [](const ContentAddressed & ca) {
+            return true;
+        },
+        [](const Impure &) {
             return false;
         },
     }, raw);
@@ -85,7 +138,7 @@ kj::Promise<Result<StorePath>> writeDerivation(Store & store,
     const Derivation & drv, RepairFlag repair, bool readOnly)
 try {
     auto references = drv.inputSrcs;
-    for (auto & i : drv.inputDrvs)
+    for (auto & i : drv.inputDrvs.map)
         references.insert(i.first);
     /* Note that the outputs of a derivation are *not* references
        (that can be missing (of course) and should not necessarily be
@@ -220,10 +273,16 @@ static DerivationOutput parseDerivationOutput(
     if (hashAlgo != "") {
         ContentAddressMethod method = ContentAddressMethod::parsePrefix(hashAlgo);
         if (method == TextIngestionMethod {})
-            throw UnimplementedError("dynamic derivations are not supported");
+            xpSettings.require(Xp::DynamicDerivations);
         const auto hashType = parseHashType(hashAlgo);
         if (hashS == "impure") {
-            throw UnimplementedError("impure derivations are not supported");
+            xpSettings.require(Xp::ImpureDerivations);
+            if (pathS != "")
+                throw FormatError("impure derivation output should not specify output path");
+            return DerivationOutput::Impure {
+                .method = std::move(method),
+                .hashType = std::move(hashType),
+            };
         } else if (hashS != "") {
             validatePath(pathS);
             auto hash = Hash::parseNonSRIUnprefixed(hashS, hashType);
@@ -234,11 +293,17 @@ static DerivationOutput parseDerivationOutput(
                 },
             };
         } else {
-            throw UnimplementedError("ca derivations are not supported");
+            xpSettings.require(Xp::CaDerivations);
+            if (pathS != "")
+                throw FormatError("content-addressed derivation output should not specify output path");
+            return DerivationOutput::CAFloating {
+                .method = std::move(method),
+                .hashType = std::move(hashType),
+            };
         }
     } else {
         if (pathS == "") {
-            throw UnimplementedError("deferred input-addressed derivations are not supported");
+            return DerivationOutput::Deferred { };
         }
         validatePath(pathS);
         return DerivationOutput::InputAddressed {
@@ -259,11 +324,67 @@ static DerivationOutput parseDerivationOutput(
     return parseDerivationOutput(store, *pathS, *hashAlgo, *hash, xpSettings);
 }
 
-static StringSet parseDerivedPathMapNode(
+/**
+ * All ATerm Derivation format versions currently known.
+ *
+ * Unknown versions are rejected at the parsing stage.
+ */
+enum struct DerivationATermVersion {
+    /**
+     * Older unversioned form
+     */
+    Traditional,
+
+    /**
+     * Newer versioned form; only this version so far.
+     */
+    DynamicDerivations,
+};
+
+static DerivedPathMap<StringSet>::ChildNode parseDerivedPathMapNode(
     const Store & store,
-    StringViewStream & str)
+    StringViewStream & str,
+    DerivationATermVersion version)
 {
-    return parseStrings(str, false);
+    DerivedPathMap<StringSet>::ChildNode node;
+
+    auto parseNonDynamic = [&]() {
+        node.value = parseStrings(str, false);
+    };
+
+    // Older derivation should never use new form, but newer
+    // derivaiton can use old form.
+    switch (version) {
+    case DerivationATermVersion::Traditional:
+        parseNonDynamic();
+        break;
+    case DerivationATermVersion::DynamicDerivations:
+        switch (str.peek()) {
+        case '[':
+            parseNonDynamic();
+            break;
+        case '(':
+            expect(str, "(");
+            node.value = parseStrings(str, false);
+            expect(str, ",[");
+            while (!endOfList(str)) {
+                expect(str, "(");
+                auto outputName = parseString(str).toOwned();
+                expect(str, ",");
+                node.childMap.insert_or_assign(outputName, parseDerivedPathMapNode(store, str, version));
+                expect(str, ")");
+            }
+            expect(str, ")");
+            break;
+        default:
+            throw FormatError("invalid inputDrvs entry in derivation");
+        }
+        break;
+    default:
+        // invalid format, not a parse error but internal error
+        assert(false);
+    }
+    return node;
 }
 
 
@@ -276,14 +397,24 @@ Derivation parseDerivation(
 
     StringViewStream str{s};
     expect(str, "D");
+    DerivationATermVersion version;
     switch (str.peek()) {
     case 'e':
         expect(str, "erive(");
+        version = DerivationATermVersion::Traditional;
         break;
     case 'r': {
         expect(str, "rvWithVersion(");
         auto versionS = parseString(str);
-        throw FormatError("Unknown derivation ATerm format version '%s'", *versionS);
+        if (*versionS == "xp-dyn-drv") {
+            // Only verison we have so far
+            version = DerivationATermVersion::DynamicDerivations;
+            xpSettings.require(Xp::DynamicDerivations);
+        } else {
+            throw FormatError("Unknown derivation ATerm format version '%s'", *versionS);
+        }
+        expect(str, ",");
+        break;
     }
     default:
         throw Error("derivation does not start with 'Derive' or 'DrvWithVersion'");
@@ -303,7 +434,7 @@ Derivation parseDerivation(
         expect(str, "(");
         auto drvPath = parsePath(str);
         expect(str, ",");
-        drv.inputDrvs.insert_or_assign(store.parseStorePath(*drvPath), parseDerivedPathMapNode(store, str));
+        drv.inputDrvs.map.insert_or_assign(store.parseStorePath(*drvPath), parseDerivedPathMapNode(store, str, version));
         expect(str, ")");
     }
 
@@ -391,20 +522,64 @@ static void printUnquotedStrings(std::string & res, ForwardIterator i, ForwardIt
 }
 
 
-static void unparseDerivedPathMapNode(const Store & store, std::string & s, const StringSet & node)
+static void unparseDerivedPathMapNode(const Store & store, std::string & s, const DerivedPathMap<StringSet>::ChildNode & node)
 {
     s += ',';
-    printUnquotedStrings(s, node.begin(), node.end());
+    if (node.childMap.empty()) {
+        printUnquotedStrings(s, node.value.begin(), node.value.end());
+    } else {
+        s += "(";
+        printUnquotedStrings(s, node.value.begin(), node.value.end());
+        s += ",[";
+        bool first = true;
+        for (auto & [outputName, childNode] : node.childMap) {
+            if (first) first = false; else s += ',';
+            s += '('; printUnquotedString(s, outputName);
+            unparseDerivedPathMapNode(store, s, childNode);
+            s += ')';
+        }
+        s += "])";
+    }
+}
+
+
+/**
+ * Does the derivation have a dependency on the output of a dynamic
+ * derivation?
+ *
+ * In other words, does it on the output of derivation that is itself an
+ * ouput of a derivation? This corresponds to a dependency that is an
+ * inductive derived path with more than one layer of
+ * `DerivedPath::Built`.
+ */
+static bool hasDynamicDrvDep(const Derivation & drv)
+{
+    return
+        std::find_if(
+            drv.inputDrvs.map.begin(),
+            drv.inputDrvs.map.end(),
+            [](auto & kv) { return !kv.second.childMap.empty(); })
+        != drv.inputDrvs.map.end();
 }
 
 
 std::string Derivation::unparse(const Store & store, bool maskOutputs,
-    std::map<std::string, StringSet> * actualInputs) const
+    DerivedPathMap<StringSet>::ChildNode::Map * actualInputs) const
 {
     std::string s;
     s.reserve(65536);
 
-    s += "Derive(";
+    /* Use older unversioned form if possible, for wider compat. Use
+       newer form only if we need it, which we do for
+       `Xp::DynamicDerivations`. */
+    if (hasDynamicDrvDep(*this)) {
+        s += "DrvWithVersion(";
+        // Only version we have so far
+        printUnquotedString(s, "xp-dyn-drv");
+        s += ",";
+    } else {
+        s += "Derive(";
+    }
 
     bool first = true;
     s += "[";
@@ -422,6 +597,22 @@ std::string Derivation::unparse(const Store & store, bool maskOutputs,
                 s += ','; printUnquotedString(s, dof.ca.printMethodAlgo());
                 s += ','; printUnquotedString(s, dof.ca.hash.to_string(Base::Base16, false));
             },
+            [&](const DerivationOutput::CAFloating & dof) {
+                s += ','; printUnquotedString(s, "");
+                s += ','; printUnquotedString(s, dof.method.renderPrefix() + printHashType(dof.hashType));
+                s += ','; printUnquotedString(s, "");
+            },
+            [&](const DerivationOutput::Deferred &) {
+                s += ','; printUnquotedString(s, "");
+                s += ','; printUnquotedString(s, "");
+                s += ','; printUnquotedString(s, "");
+            },
+            [&](const DerivationOutput::Impure & doi) {
+                // FIXME
+                s += ','; printUnquotedString(s, "");
+                s += ','; printUnquotedString(s, doi.method.renderPrefix() + printHashType(doi.hashType));
+                s += ','; printUnquotedString(s, "impure");
+            }
         }, i.second.raw);
         s += ')';
     }
@@ -436,7 +627,7 @@ std::string Derivation::unparse(const Store & store, bool maskOutputs,
             s += ')';
         }
     } else {
-        for (auto & [drvPath, childMap] : inputDrvs) {
+        for (auto & [drvPath, childMap] : inputDrvs.map) {
             if (first) first = false; else s += ',';
             s += '('; printUnquotedString(s, store.printStorePath(drvPath));
             unparseDerivedPathMapNode(store, s, childMap);
@@ -488,7 +679,11 @@ DerivationType BasicDerivation::type() const
 {
     std::set<std::string_view>
         inputAddressedOutputs,
-        fixedCAOutputs;
+        fixedCAOutputs,
+        floatingCAOutputs,
+        deferredIAOutputs,
+        impureOutputs;
+    std::optional<HashType> floatingHashType;
 
     for (auto & i : outputs) {
         std::visit(overloaded {
@@ -498,28 +693,83 @@ DerivationType BasicDerivation::type() const
             [&](const DerivationOutput::CAFixed &) {
                 fixedCAOutputs.insert(i.first);
             },
+            [&](const DerivationOutput::CAFloating & dof) {
+                floatingCAOutputs.insert(i.first);
+                if (!floatingHashType) {
+                    floatingHashType = dof.hashType;
+                } else {
+                    if (*floatingHashType != dof.hashType)
+                        throw Error("all floating outputs must use the same hash type");
+                }
+            },
+            [&](const DerivationOutput::Deferred &) {
+                deferredIAOutputs.insert(i.first);
+            },
+            [&](const DerivationOutput::Impure &) {
+                impureOutputs.insert(i.first);
+            },
         }, i.second.raw);
     }
 
     if (inputAddressedOutputs.empty()
-        && fixedCAOutputs.empty())
+        && fixedCAOutputs.empty()
+        && floatingCAOutputs.empty()
+        && deferredIAOutputs.empty()
+        && impureOutputs.empty())
         throw Error("must have at least one output");
 
     if (!inputAddressedOutputs.empty()
-        && fixedCAOutputs.empty())
-        return DerivationType::InputAddressed {};
+        && fixedCAOutputs.empty()
+        && floatingCAOutputs.empty()
+        && deferredIAOutputs.empty()
+        && impureOutputs.empty())
+        return DerivationType::InputAddressed {
+            .deferred = false,
+        };
 
     if (inputAddressedOutputs.empty()
-        && !fixedCAOutputs.empty())
+        && !fixedCAOutputs.empty()
+        && floatingCAOutputs.empty()
+        && deferredIAOutputs.empty()
+        && impureOutputs.empty())
     {
         if (fixedCAOutputs.size() > 1)
             // FIXME: Experimental feature?
             throw Error("only one fixed output is allowed for now");
         if (*fixedCAOutputs.begin() != "out")
             throw Error("single fixed output must be named \"out\"");
-        return DerivationType::ContentAddressed {};
+        return DerivationType::ContentAddressed {
+            .sandboxed = false,
+            .fixed = true,
+        };
     }
 
+    if (inputAddressedOutputs.empty()
+        && fixedCAOutputs.empty()
+        && !floatingCAOutputs.empty()
+        && deferredIAOutputs.empty()
+        && impureOutputs.empty())
+        return DerivationType::ContentAddressed {
+            .sandboxed = true,
+            .fixed = false,
+        };
+
+    if (inputAddressedOutputs.empty()
+        && fixedCAOutputs.empty()
+        && floatingCAOutputs.empty()
+        && !deferredIAOutputs.empty()
+        && impureOutputs.empty())
+        return DerivationType::InputAddressed {
+            .deferred = true,
+        };
+
+    if (inputAddressedOutputs.empty()
+        && fixedCAOutputs.empty()
+        && floatingCAOutputs.empty()
+        && deferredIAOutputs.empty()
+        && !impureOutputs.empty())
+        return DerivationType::Impure { };
+
     throw Error("can't mix derivation output types");
 }
 
@@ -587,17 +837,46 @@ try {
         }
         co_return DrvHash {
             .hashes = outputHashes,
+            .kind = DrvHash::Kind::Regular,
         };
     }
 
-    std::map<std::string, StringSet> inputs2;
-    for (auto & [drvPath, node] : drv.inputDrvs) {
+    if (!type.isPure()) {
+        std::map<std::string, Hash> outputHashes;
+        for (const auto & [outputName, _] : drv.outputs)
+            outputHashes.insert_or_assign(outputName, impureOutputHash);
+        co_return DrvHash {
+            .hashes = outputHashes,
+            .kind = DrvHash::Kind::Deferred,
+        };
+    }
+
+    auto kind = std::visit(overloaded {
+        [](const DerivationType::InputAddressed & ia) {
+            /* This might be a "pesimistically" deferred output, so we don't
+               "taint" the kind yet. */
+            return DrvHash::Kind::Regular;
+        },
+        [](const DerivationType::ContentAddressed & ca) {
+            return ca.fixed
+                ? DrvHash::Kind::Regular
+                : DrvHash::Kind::Deferred;
+        },
+        [](const DerivationType::Impure &) -> DrvHash::Kind {
+            assert(false);
+        }
+    }, drv.type().raw);
+
+    DerivedPathMap<StringSet>::ChildNode::Map inputs2;
+    for (auto & [drvPath, node] : drv.inputDrvs.map) {
         const auto & res = TRY_AWAIT(pathDerivationModulo(store, drvPath));
-        for (auto & outputName : node) {
+        if (res.kind == DrvHash::Kind::Deferred)
+            kind = DrvHash::Kind::Deferred;
+        for (auto & outputName : node.value) {
             const auto h = get(res.hashes, outputName);
             if (!h)
                 throw Error("no hash for output '%s' of derivation '%s'", outputName, drv.name);
-            inputs2[h->to_string(Base::Base16, false)].insert(outputName);
+            inputs2[h->to_string(Base::Base16, false)].value.insert(outputName);
         }
     }
 
@@ -610,6 +889,7 @@ try {
 
     co_return DrvHash {
         .hashes = outputHashes,
+        .kind = kind,
     };
 } catch (...) {
     co_return result::current_exception();
@@ -642,16 +922,16 @@ StringSet BasicDerivation::outputNames() const
     return names;
 }
 
-DerivationOutputsAndPaths BasicDerivation::outputsAndPaths(const Store & store) const
+DerivationOutputsAndOptPaths BasicDerivation::outputsAndOptPaths(const Store & store) const
 {
-    DerivationOutputsAndPaths outsAndPaths;
+    DerivationOutputsAndOptPaths outsAndOptPaths;
     for (auto & [outputName, output] : outputs)
-        outsAndPaths.insert(std::make_pair(
+        outsAndOptPaths.insert(std::make_pair(
             outputName,
             std::make_pair(output, output.path(store, name, outputName))
             )
         );
-    return outsAndPaths;
+    return outsAndOptPaths;
 }
 
 std::string_view BasicDerivation::nameFromPath(const StorePath & drvPath)
@@ -708,6 +988,21 @@ void writeDerivation(Sink & out, const Store & store, const BasicDerivation & dr
                     << dof.ca.printMethodAlgo()
                     << dof.ca.hash.to_string(Base::Base16, false);
             },
+            [&](const DerivationOutput::CAFloating & dof) {
+                out << ""
+                    << (dof.method.renderPrefix() + printHashType(dof.hashType))
+                    << "";
+            },
+            [&](const DerivationOutput::Deferred &) {
+                out << ""
+                    << ""
+                    << "";
+            },
+            [&](const DerivationOutput::Impure & doi) {
+                out << ""
+                    << (doi.method.renderPrefix() + printHashType(doi.hashType))
+                    << "impure";
+            },
         }, i.second.raw);
     }
     out << CommonProto::write(store,
@@ -729,6 +1024,151 @@ std::string hashPlaceholder(const OutputNameView outputName)
 
 
 
+static kj::Promise<Result<void>>
+rewriteDerivation(Store & store, BasicDerivation & drv, const StringMap & rewrites)
+try {
+    debug("Rewriting the derivation");
+
+    for (auto & rewrite : rewrites) {
+        debug("rewriting %s as %s", rewrite.first, rewrite.second);
+    }
+
+    drv.builder = rewriteStrings(drv.builder, rewrites);
+    for (auto & arg : drv.args) {
+        arg = rewriteStrings(arg, rewrites);
+    }
+
+    StringPairs newEnv;
+    for (auto & envVar : drv.env) {
+        auto envName = rewriteStrings(envVar.first, rewrites);
+        auto envValue = rewriteStrings(envVar.second, rewrites);
+        newEnv.emplace(envName, envValue);
+    }
+    drv.env = newEnv;
+
+    auto hashModulo = TRY_AWAIT(hashDerivationModulo(store, Derivation(drv), true));
+    for (auto & [outputName, output] : drv.outputs) {
+        if (std::holds_alternative<DerivationOutput::Deferred>(output.raw)) {
+            auto h = get(hashModulo.hashes, outputName);
+            if (!h)
+                throw Error("derivation '%s' output '%s' has no hash (derivations.cc/rewriteDerivation)",
+                    drv.name, outputName);
+            auto outPath = store.makeOutputPath(outputName, *h, drv.name);
+            drv.env[outputName] = store.printStorePath(outPath);
+            output = DerivationOutput::InputAddressed {
+                .path = std::move(outPath),
+            };
+        }
+    }
+
+    co_return result::success();
+} catch (...) {
+    co_return result::current_exception();
+}
+
+kj::Promise<Result<std::optional<BasicDerivation>>>
+Derivation::tryResolve(Store & store, Store * evalStore) const
+try {
+    std::map<std::pair<StorePath, std::string>, StorePath> inputDrvOutputs;
+
+    std::function<
+        kj::Promise<Result<void>>(const StorePath &, const DerivedPathMap<StringSet>::ChildNode &)>
+        accum;
+    // NOLINTNEXTLINE(cppcoreguidelines-avoid-capturing-lambda-coroutines)
+    accum = [&](auto & inputDrv, auto & node) -> kj::Promise<Result<void>> {
+        try {
+            for (auto & [outputName, outputPath] :
+                 TRY_AWAIT(store.queryPartialDerivationOutputMap(inputDrv, evalStore)))
+            {
+                if (outputPath) {
+                    inputDrvOutputs.insert_or_assign({inputDrv, outputName}, *outputPath);
+                    if (auto p = get(node.childMap, outputName))
+                        TRY_AWAIT(accum(*outputPath, *p));
+                }
+            }
+            co_return result::success();
+        } catch (...) {
+            co_return result::current_exception();
+        }
+    };
+
+    for (auto & [inputDrv, node] : inputDrvs.map)
+        TRY_AWAIT(accum(inputDrv, node));
+
+    co_return TRY_AWAIT(tryResolve(store, inputDrvOutputs));
+} catch (...) {
+    co_return result::current_exception();
+}
+
+static bool tryResolveInput(
+    Store & store, StorePathSet & inputSrcs, StringMap & inputRewrites,
+    const DownstreamPlaceholder * placeholderOpt,
+    const StorePath & inputDrv, const DerivedPathMap<StringSet>::ChildNode & inputNode,
+    const std::map<std::pair<StorePath, std::string>, StorePath> & inputDrvOutputs)
+{
+    auto getOutput = [&](const std::string & outputName) {
+        auto * actualPathOpt = get(inputDrvOutputs, { inputDrv, outputName });
+        if (!actualPathOpt)
+            warn("output %s of input %s missing, aborting the resolving",
+                outputName,
+                store.printStorePath(inputDrv)
+            );
+        return actualPathOpt;
+    };
+
+    auto getPlaceholder = [&](const std::string & outputName) {
+        return placeholderOpt
+            ? DownstreamPlaceholder::unknownDerivation(*placeholderOpt, outputName)
+            : DownstreamPlaceholder::unknownCaOutput(inputDrv, outputName);
+    };
+
+    for (auto & outputName : inputNode.value) {
+        auto actualPathOpt = getOutput(outputName);
+        if (!actualPathOpt) return false;
+        auto actualPath = *actualPathOpt;
+        if (experimentalFeatureSettings.isEnabled(Xp::CaDerivations)) {
+            inputRewrites.emplace(
+                getPlaceholder(outputName).render(),
+                store.printStorePath(actualPath));
+        }
+        inputSrcs.insert(std::move(actualPath));
+    }
+
+    for (auto & [outputName, childNode] : inputNode.childMap) {
+        auto actualPathOpt = getOutput(outputName);
+        if (!actualPathOpt) return false;
+        auto actualPath = *actualPathOpt;
+        auto nextPlaceholder = getPlaceholder(outputName);
+        if (!tryResolveInput(store, inputSrcs, inputRewrites,
+            &nextPlaceholder, actualPath, childNode,
+            inputDrvOutputs))
+            return false;
+    }
+    return true;
+}
+
+kj::Promise<Result<std::optional<BasicDerivation>>> Derivation::tryResolve(
+    Store & store,
+    const std::map<std::pair<StorePath, std::string>, StorePath> & inputDrvOutputs) const
+try {
+    BasicDerivation resolved { *this };
+
+    // Input paths that we'll want to rewrite in the derivation
+    StringMap inputRewrites;
+
+    for (auto & [inputDrv, inputNode] : inputDrvs.map)
+        if (!tryResolveInput(store, resolved.inputSrcs, inputRewrites,
+            nullptr, inputDrv, inputNode, inputDrvOutputs))
+            co_return std::nullopt;
+
+    TRY_AWAIT(rewriteDerivation(store, resolved, inputRewrites));
+
+    co_return resolved;
+} catch (...) {
+    co_return result::current_exception();
+}
+
+
 kj::Promise<Result<void>>
 Derivation::checkInvariants(Store & store, const StorePath & drvPath) const
 try {
@@ -787,6 +1227,18 @@ try {
                     return {result::current_exception()};
                 }
             },
+            [](const DerivationOutput::CAFloating &) -> kj::Promise<Result<void>> {
+                /* Nothing to check */
+                return {result::success()};
+            },
+            [](const DerivationOutput::Deferred &) -> kj::Promise<Result<void>> {
+                /* Nothing to check */
+                return {result::success()};
+            },
+            [](const DerivationOutput::Impure &) -> kj::Promise<Result<void>> {
+                /* Nothing to check */
+                return {result::success()};
+            },
         }, i.second.raw));
     }
     co_return result::success();
@@ -795,6 +1247,8 @@ try {
 }
 
 
+const Hash impureOutputHash = hashString(HashType::SHA256, "impure");
+
 JSON DerivationOutput::toJSON(
     const Store & store, std::string_view drvName, OutputNameView outputName) const
 {
@@ -809,6 +1263,14 @@ JSON DerivationOutput::toJSON(
             res["hash"] = dof.ca.hash.to_string(Base::Base16, false);
             // FIXME print refs?
         },
+        [&](const DerivationOutput::CAFloating & dof) {
+            res["hashAlgo"] = dof.method.renderPrefix() + printHashType(dof.hashType);
+        },
+        [&](const DerivationOutput::Deferred &) {},
+        [&](const DerivationOutput::Impure & doi) {
+            res["hashAlgo"] = doi.method.renderPrefix() + printHashType(doi.hashType);
+            res["impure"] = true;
+        },
     }, raw);
     return res;
 }
@@ -832,7 +1294,7 @@ DerivationOutput DerivationOutput::fromJSON(
         std::string_view s = hashAlgo;
         ContentAddressMethod method = ContentAddressMethod::parsePrefix(s);
         if (method == TextIngestionMethod {})
-            throw UnimplementedError("dynamic derivations are not supported");
+            xpSettings.require(Xp::DynamicDerivations);
         auto hashType = parseHashType(s);
         return { std::move(method), std::move(hashType) };
     };
@@ -857,15 +1319,25 @@ DerivationOutput DerivationOutput::fromJSON(
     }
 
     else if (keys == (std::set<std::string_view> { "hashAlgo" })) {
-        throw UnimplementedError("ca derivations are not supported");
+        xpSettings.require(Xp::CaDerivations);
+        auto [method, hashType] = methodAlgo();
+        return DerivationOutput::CAFloating {
+            .method = std::move(method),
+            .hashType = std::move(hashType),
+        };
     }
 
     else if (keys == (std::set<std::string_view> { })) {
-        throw UnimplementedError("deferred input-addressed derivations are not supported");
+        return DerivationOutput::Deferred {};
     }
 
     else if (keys == (std::set<std::string_view> { "hashAlgo", "impure" })) {
-        throw UnimplementedError("impure derivations are not supported");
+        xpSettings.require(Xp::ImpureDerivations);
+        auto [method, hashType] = methodAlgo();
+        return DerivationOutput::Impure {
+            .method = std::move(method),
+            .hashType = hashType,
+        };
     }
 
     else {
@@ -896,16 +1368,22 @@ JSON Derivation::toJSON(const Store & store) const
     }
 
     {
-        auto doInput = [&](const auto & inputNode) {
+        std::function<JSON(const DerivedPathMap<StringSet>::ChildNode &)> doInput;
+        doInput = [&](const auto & inputNode) {
             auto value = JSON::object();
-            value["outputs"] = inputNode;
-            value["dynamicOutputs"] = JSON::object(); // for compatibility with cppnix
+            value["outputs"] = inputNode.value;
+            {
+                auto next = JSON::object();
+                for (auto & [outputId, childNode] : inputNode.childMap)
+                    next[outputId] = doInput(childNode);
+                value["dynamicOutputs"] = std::move(next);
+            }
             return value;
         };
         {
             auto& inputDrvsObj = res["inputDrvs"];
             inputDrvsObj = JSON::object();
-            for (auto & [inputDrv, inputNode] : inputDrvs) {
+            for (auto & [inputDrv, inputNode] : inputDrvs.map) {
                 inputDrvsObj[store.printStorePath(inputDrv)] = doInput(inputNode);
             }
         }
@@ -955,18 +1433,20 @@ Derivation Derivation::fromJSON(
     }
 
     try {
-        auto doInput = [&](const auto & json) {
-            StringSet node;
-            node = static_cast<const StringSet &>(
+        std::function<DerivedPathMap<StringSet>::ChildNode(const JSON &)> doInput;
+        doInput = [&](const auto & json) {
+            DerivedPathMap<StringSet>::ChildNode node;
+            node.value = static_cast<const StringSet &>(
                 ensureType(valueAt(json, "outputs"), value_t::array));
-            if (!ensureType(valueAt(json, "dynamicOutputs"), value_t::object).empty()) {
-                throw UnimplementedError("dynamic derivations are not supported");
+            for (auto & [outputId, childNode] : ensureType(valueAt(json, "dynamicOutputs"), value_t::object).items()) {
+                xpSettings.require(Xp::DynamicDerivations);
+                node.childMap[outputId] = doInput(childNode);
             }
             return node;
         };
         auto & inputDrvsObj = ensureType(valueAt(json, "inputDrvs"), value_t::object);
         for (auto & [inputDrvPath, inputOutputs] : inputDrvsObj.items())
-            res.inputDrvs[store.parseStorePath(inputDrvPath)] =
+            res.inputDrvs.map[store.parseStorePath(inputDrvPath)] =
                 doInput(inputOutputs);
     } catch (Error & e) {
         e.addTrace({}, "while reading key 'inputDrvs'");

lix/libstore/derivations.hh

@@ -2,16 +2,15 @@
 ///@file
 
 #include "lix/libstore/path.hh"
-#include "lix/libutil/config.hh"
 #include "lix/libutil/result.hh"
 #include "lix/libutil/types.hh"
 #include "lix/libutil/hash.hh"
 #include "lix/libstore/content-address.hh"
 #include "lix/libutil/repair-flag.hh"
+#include "lix/libstore/derived-path-map.hh"
 #include "lix/libutil/sync.hh"
 #include "lix/libutil/comparator.hh"
 #include "lix/libutil/variant-wrapper.hh"
-#include "outputs-spec.hh"
 
 #include <kj/async.h>
 #include <map>
@@ -63,9 +62,59 @@ struct DerivationOutput
         GENERATE_CMP(CAFixed, me->ca);
     };
 
+    /**
+     * Floating-output derivations, whose output paths are content
+     * addressed, but not fixed, and so are dynamically calculated from
+     * whatever the output ends up being.
+     * */
+    struct CAFloating
+    {
+        /**
+         * How the file system objects will be serialized for hashing
+         */
+        ContentAddressMethod method;
+
+        /**
+         * How the serialization will be hashed
+         */
+        HashType hashType;
+
+        GENERATE_CMP(CAFloating, me->method, me->hashType);
+    };
+
+    /**
+     * Input-addressed output which depends on a (CA) derivation whose hash
+     * isn't known yet.
+     */
+    struct Deferred {
+        GENERATE_CMP(Deferred);
+    };
+
+    /**
+     * Impure output which is moved to a content-addressed location (like
+     * CAFloating) but isn't registered as a realization.
+     */
+    struct Impure
+    {
+        /**
+         * How the file system objects will be serialized for hashing
+         */
+        ContentAddressMethod method;
+
+        /**
+         * How the serialization will be hashed
+         */
+        HashType hashType;
+
+        GENERATE_CMP(Impure, me->method, me->hashType);
+    };
+
     typedef std::variant<
         InputAddressed,
-        CAFixed
+        CAFixed,
+        CAFloating,
+        Deferred,
+        Impure
     > Raw;
 
     Raw raw;
@@ -83,9 +132,9 @@ struct DerivationOutput
      * \note when you use this function you should make sure that you're
      * passing the right derivation name. When in doubt, you should use
      * the safer interface provided by
-     * BasicDerivation::outputsAndPaths
+     * BasicDerivation::outputsAndOptPaths
      */
-    StorePath path(const Store & store, std::string_view drvName, OutputNameView outputName) const;
+    std::optional<StorePath> path(const Store & store, std::string_view drvName, OutputNameView outputName) const;
 
     JSON toJSON(
         const Store & store,
@@ -106,12 +155,12 @@ typedef std::map<std::string, DerivationOutput> DerivationOutputs;
 
 /**
  * These are analogues to the previous DerivationOutputs data type,
- * but they also contains, for each output, the store
+ * but they also contains, for each output, the (optional) store
  * path in which it would be written. To calculate values of these
  * types, see the corresponding functions in BasicDerivation.
  */
-typedef std::map<std::string, std::pair<DerivationOutput, StorePath>>
-  DerivationOutputsAndPaths;
+typedef std::map<std::string, std::pair<DerivationOutput, std::optional<StorePath>>>
+  DerivationOutputsAndOptPaths;
 
 /**
  * For inputs that are sub-derivations, we specify exactly which
@@ -124,19 +173,54 @@ struct DerivationType {
      * Input-addressed derivation types
      */
     struct InputAddressed {
-        GENERATE_CMP(InputAddressed);
+        /**
+         * True iff the derivation type can't be determined statically,
+         * for instance because it (transitively) depends on a content-addressed
+         * derivation.
+        */
+        bool deferred;
+
+        GENERATE_CMP(InputAddressed, me->deferred);
     };
 
     /**
      * Content-addressed derivation types
      */
     struct ContentAddressed {
-        GENERATE_CMP(ContentAddressed);
+        /**
+         * Whether the derivation should be built safely inside a sandbox.
+         */
+        bool sandboxed;
+        /**
+         * Whether the derivation's outputs' content-addresses are "fixed"
+         * or "floating".
+         *
+         *  - Fixed: content-addresses are written down as part of the
+         *    derivation itself. If the outputs don't end up matching the
+         *    build fails.
+         *
+         *  - Floating: content-addresses are not written down, we do not
+         *    know them until we perform the build.
+         */
+        bool fixed;
+
+        GENERATE_CMP(ContentAddressed, me->sandboxed, me->fixed);
+    };
+
+    /**
+     * Impure derivation type
+     *
+     * This is similar at build-time to the content addressed, not sandboxed, not fixed
+     * type, but has some restrictions on its usage.
+     */
+    struct Impure {
+        GENERATE_CMP(Impure);
     };
 
     typedef std::variant<
         InputAddressed,
-        ContentAddressed
+        ContentAddressed,
+        Impure
     > Raw;
 
     Raw raw;
@@ -169,6 +253,21 @@ struct DerivationType {
      * controlled separately. Always true for non-CA derivations.
      */
     bool isSandboxed() const;
+
+    /**
+     * Whether the derivation is expected to produce the same result
+     * every time, and therefore it only needs to be built once. This is
+     * only false for derivations that have the attribute '__impure =
+     * true'.
+     */
+    bool isPure() const;
+
+    /**
+     * Does the derivation knows its own output paths?
+     * Only true when there's no floating-ca derivation involved in the
+     * closure, or if fixed output.
+     */
+    bool hasKnownOutputPaths() const;
 };
 
 struct BasicDerivation
@@ -207,7 +306,7 @@ struct BasicDerivation
      * augmented with knowledge of the Store paths they would be written
      * into.
      */
-    DerivationOutputsAndPaths outputsAndPaths(const Store & store) const;
+    DerivationOutputsAndOptPaths outputsAndOptPaths(const Store & store) const;
 
     static std::string_view nameFromPath(const StorePath & storePath);
 
@@ -226,13 +325,34 @@ struct Derivation : BasicDerivation
     /**
      * inputs that are sub-derivations
      */
-    std::map<StorePath, std::set<OutputName>> inputDrvs;
+    DerivedPathMap<std::set<OutputName>> inputDrvs;
 
     /**
      * Print a derivation.
      */
     std::string unparse(const Store & store, bool maskOutputs,
-        std::map<std::string, StringSet> * actualInputs = nullptr) const;
+        DerivedPathMap<StringSet>::ChildNode::Map * actualInputs = nullptr) const;
+
+    /**
+     * Return the underlying basic derivation but with these changes:
+     *
+     * 1. Input drvs are emptied, but the outputs of them that were used
+     *    are added directly to input sources.
+     *
+     * 2. Input placeholders are replaced with realized input store
+     *    paths.
+     */
+    kj::Promise<Result<std::optional<BasicDerivation>>>
+    tryResolve(Store & store, Store * evalStore = nullptr) const;
+
+    /**
+     * Like the above, but instead of querying the Nix database for
+     * realisations, uses a given mapping from input derivation paths +
+     * output names to actual output store paths.
+     */
+    kj::Promise<Result<std::optional<BasicDerivation>>> tryResolve(
+        Store & store,
+        const std::map<std::pair<StorePath, std::string>, StorePath> & inputDrvOutputs) const;
 
     /**
      * Check that the derivation is valid and does not present any
@@ -300,7 +420,7 @@ std::string outputPathName(std::string_view drvName, OutputNameView outputName);
  * The hashes modulo of a derivation.
  *
  * Each output is given a hash, although in practice only the content-addressed
- * derivations (i.e. fixed-output) will have a different hash for each
+ * derivations (fixed-output or not) will have a different hash for each
  * output.
  */
 struct DrvHash {
@@ -308,8 +428,29 @@ struct DrvHash {
      * Map from output names to hashes
      */
     std::map<std::string, Hash> hashes;
+
+    enum struct Kind : bool {
+        /**
+         * Statically determined derivations.
+         * This hash will be directly used to compute the output paths
+         */
+        Regular,
+
+        /**
+         * Floating-output derivations (and their reverse dependencies).
+         */
+        Deferred,
+    };
+
+    /**
+     * The kind of derivation this is, simplified for just "derivation hash
+     * modulo" purposes.
+     */
+    Kind kind;
 };
 
+void operator |= (DrvHash::Kind & self, const DrvHash::Kind & other) noexcept;
+
 /**
  * Returns hashes with the details of fixed-output subderivations
  * expunged.
@@ -370,4 +511,6 @@ void writeDerivation(Sink & out, const Store & store, const BasicDerivation & dr
  */
 std::string hashPlaceholder(const OutputNameView outputName);
 
+extern const Hash impureOutputHash;
+
 }

lix/libstore/derived-path-map.cc

@@ -0,0 +1,69 @@
+#include "lix/libstore/derived-path-map.hh"
+
+namespace nix {
+
+template<typename V>
+typename DerivedPathMap<V>::ChildNode & DerivedPathMap<V>::ensureSlot(const SingleDerivedPath & k)
+{
+    std::function<ChildNode &(const SingleDerivedPath & )> initIter;
+    initIter = [&](const auto & k) -> auto & {
+        return std::visit(overloaded {
+            [&](const SingleDerivedPath::Opaque & bo) -> auto & {
+                // will not overwrite if already there
+                return map[bo.path];
+            },
+            [&](const SingleDerivedPath::Built & bfd) -> auto & {
+                auto & n = initIter(*bfd.drvPath);
+                return n.childMap[bfd.output];
+            },
+        }, k.raw());
+    };
+    return initIter(k);
+}
+
+template<typename V>
+typename DerivedPathMap<V>::ChildNode * DerivedPathMap<V>::findSlot(const SingleDerivedPath & k)
+{
+    std::function<ChildNode *(const SingleDerivedPath & )> initIter;
+    initIter = [&](const auto & k) {
+        return std::visit(overloaded {
+            [&](const SingleDerivedPath::Opaque & bo) {
+                auto it = map.find(bo.path);
+                return it != map.end()
+                    ? &it->second
+                    : nullptr;
+            },
+            [&](const SingleDerivedPath::Built & bfd) {
+                auto * n = initIter(*bfd.drvPath);
+                if (!n) return (ChildNode *)nullptr;
+
+                auto it = n->childMap.find(bfd.output);
+                return it != n->childMap.end()
+                    ? &it->second
+                    : nullptr;
+            },
+        }, k.raw());
+    };
+    return initIter(k);
+}
+
+}
+
+// instantiations
+
+namespace nix {
+
+GENERATE_CMP_EXT(
+    template<>,
+    DerivedPathMap<std::set<std::string>>::ChildNode,
+    me->value,
+    me->childMap);
+
+GENERATE_CMP_EXT(
+    template<>,
+    DerivedPathMap<std::set<std::string>>,
+    me->map);
+
+template struct DerivedPathMap<std::set<std::string>>;
+
+};

lix/libstore/derived-path-map.hh

@@ -0,0 +1,96 @@
+#pragma once
+///@file
+
+#include "lix/libutil/types.hh"
+#include "lix/libstore/derived-path.hh"
+
+namespace nix {
+
+/**
+ * A simple Trie, of sorts. Conceptually a map of `SingleDerivedPath` to
+ * values.
+ *
+ * Concretely, an n-ary tree, as described below. A
+ * `SingleDerivedPath::Opaque` maps to the value of an immediate child
+ * of the root node. A `SingleDerivedPath::Built` maps to a deeper child
+ * node: the `SingleDerivedPath::Built::drvPath` is first mapped to a a
+ * child node (inductively), and then the
+ * `SingleDerivedPath::Built::output` is used to look up that child's
+ * child via its map. In this manner, every `SingleDerivedPath` is
+ * mapped to a child node.
+ *
+ * @param V A type to instantiate for each output. It should probably
+ * should be an "optional" type so not every interior node has to have a
+ * value. `* const Something` or `std::optional<Something>` would be
+ * good choices for "optional" types.
+ */
+template<typename V>
+struct DerivedPathMap {
+    /**
+     * A child node (non-root node).
+     */
+    struct ChildNode {
+        /**
+         * Value of this child node.
+         *
+         * @see DerivedPathMap for what `V` should be.
+         */
+        V value;
+
+        /**
+         * The map type for the root node.
+         */
+        using Map = std::map<OutputName, ChildNode>;
+
+        /**
+         * The map of the root node.
+         */
+        Map childMap;
+
+        DECLARE_CMP(ChildNode);
+    };
+
+    /**
+     * The map type for the root node.
+     */
+    using Map = std::map<StorePath, ChildNode>;
+
+    /**
+     * The map of root node.
+     */
+    Map map;
+
+    DECLARE_CMP(DerivedPathMap);
+
+    /**
+     * Find the node for `k`, creating it if needed.
+     *
+     * The node is referred to as a "slot" on the assumption that `V` is
+     * some sort of optional type, so the given key can be set or unset
+     * by changing this node.
+     */
+    ChildNode & ensureSlot(const SingleDerivedPath & k);
+
+    /**
+     * Like `ensureSlot` but does not create the slot if it doesn't exist.
+     *
+     * Read the entire description of `ensureSlot` to understand an
+     * important caveat here that "have slot" does *not* imply "key is
+     * set in map". To ensure a key is set one would need to get the
+     * child node (with `findSlot` or `ensureSlot`) *and* check the
+     * `ChildNode::value`.
+     */
+    ChildNode * findSlot(const SingleDerivedPath & k);
+};
+
+
+DECLARE_CMP_EXT(
+    template<>,
+    DerivedPathMap<std::set<std::string>>::,
+    DerivedPathMap<std::set<std::string>>);
+DECLARE_CMP_EXT(
+    template<>,
+    DerivedPathMap<std::set<std::string>>::ChildNode::,
+    DerivedPathMap<std::set<std::string>>::ChildNode);
+
+}

lix/libstore/derived-path.cc

@@ -10,9 +10,9 @@ namespace nix {
     bool MY_TYPE ::operator COMPARATOR (const MY_TYPE & other) const \
     { \
         const MY_TYPE* me = this; \
-        auto fields1 = std::tie(me->drvPath, me->FIELD); \
+        auto fields1 = std::tie(*me->drvPath, me->FIELD); \
         me = &other; \
-        auto fields2 = std::tie(me->drvPath, me->FIELD); \
+        auto fields2 = std::tie(*me->drvPath, me->FIELD); \
         return fields1 COMPARATOR fields2; \
     }
 #define CMP(CHILD_TYPE, MY_TYPE, FIELD) \
@@ -34,41 +34,105 @@ try {
     return {result::current_exception()};
 }
 
+kj::Promise<Result<JSON>> SingleDerivedPath::Built::toJSON(Store & store) const
+try {
+    JSON res;
+    res["drvPath"] = TRY_AWAIT(drvPath->toJSON(store));
+    // Fallback for the input-addressed derivation case: We expect to always be
+    // able to print the output paths, so let’s do it
+    // FIXME try-resolve on drvPath
+    const auto outputMap = TRY_AWAIT(
+        store.queryPartialDerivationOutputMap(TRY_AWAIT(resolveDerivedPath(store, *drvPath)))
+    );
+    res["output"] = output;
+    auto outputPathIter = outputMap.find(output);
+    if (outputPathIter == outputMap.end())
+        res["outputPath"] = nullptr;
+    else if (std::optional p = outputPathIter->second)
+        res["outputPath"] = store.printStorePath(*p);
+    else
+        res["outputPath"] = nullptr;
+    co_return res;
+} catch (...) {
+    co_return result::current_exception();
+}
+
 kj::Promise<Result<JSON>> DerivedPath::Built::toJSON(Store & store) const
 try {
     JSON res;
-    res["drvPath"] = TRY_AWAIT(drvPath.toJSON(store));
+    res["drvPath"] = TRY_AWAIT(drvPath->toJSON(store));
     // Fallback for the input-addressed derivation case: We expect to always be
     // able to print the output paths, so let’s do it
-    const auto outputMap = TRY_AWAIT(store.queryDerivationOutputMap(drvPath.path));
+    // FIXME try-resolve on drvPath
+    const auto outputMap = TRY_AWAIT(
+        store.queryPartialDerivationOutputMap(TRY_AWAIT(resolveDerivedPath(store, *drvPath)))
+    );
     for (const auto & [output, outputPathOpt] : outputMap) {
         if (!outputs.contains(output)) continue;
-        res["outputs"][output] = store.printStorePath(outputPathOpt);
+        if (outputPathOpt)
+            res["outputs"][output] = store.printStorePath(*outputPathOpt);
+        else
+            res["outputs"][output] = nullptr;
     }
     co_return res;
 } catch (...) {
     co_return result::current_exception();
 }
 
+kj::Promise<Result<JSON>> SingleDerivedPath::toJSON(Store & store) const
+try {
+    co_return TRY_AWAIT(std::visit([&](const auto & buildable) {
+        return buildable.toJSON(store);
+    }, raw()));
+} catch (...) {
+    co_return result::current_exception();
+}
+
+kj::Promise<Result<JSON>> DerivedPath::toJSON(Store & store) const
+try {
+    co_return TRY_AWAIT(std::visit([&](const auto & buildable) {
+        return buildable.toJSON(store);
+    }, raw()));
+} catch (...) {
+    co_return result::current_exception();
+}
+
 std::string DerivedPath::Opaque::to_string(const Store & store) const
 {
     return store.printStorePath(path);
 }
 
+std::string SingleDerivedPath::Built::to_string(const Store & store) const
+{
+    return drvPath->to_string(store) + "^" + output;
+}
+
+std::string SingleDerivedPath::Built::to_string_legacy(const Store & store) const
+{
+    return drvPath->to_string(store) + "!" + output;
+}
+
 std::string DerivedPath::Built::to_string(const Store & store) const
 {
-    return drvPath.to_string(store)
+    return drvPath->to_string(store)
         + '^'
         + outputs.to_string();
 }
 
 std::string DerivedPath::Built::to_string_legacy(const Store & store) const
 {
-    return drvPath.to_string(store)
+    return drvPath->to_string_legacy(store)
         + "!"
         + outputs.to_string();
 }
 
+std::string SingleDerivedPath::to_string(const Store & store) const
+{
+    return std::visit(
+        [&](const auto & req) { return req.to_string(store); },
+        raw());
+}
+
 std::string DerivedPath::to_string(const Store & store) const
 {
     return std::visit(
@@ -76,6 +140,14 @@ std::string DerivedPath::to_string(const Store & store) const
         raw());
 }
 
+std::string SingleDerivedPath::to_string_legacy(const Store & store) const
+{
+    return std::visit(overloaded {
+        [&](const SingleDerivedPath::Built & req) { return req.to_string_legacy(store); },
+        [&](const SingleDerivedPath::Opaque & req) { return req.to_string(store); },
+    }, this->raw());
+}
+
 std::string DerivedPath::to_string_legacy(const Store & store) const
 {
     return std::visit(overloaded {
@@ -90,29 +162,63 @@ DerivedPath::Opaque DerivedPath::Opaque::parse(const Store & store, std::string_
     return {store.parseStorePath(s)};
 }
 
-DerivedPath::Built DerivedPath::Built::parse(
-    const Store & store, DerivedPathOpaque drv,
-    OutputNameView outputsS)
+void drvRequireExperiment(
+    const SingleDerivedPath & drv,
+    const ExperimentalFeatureSettings & xpSettings)
 {
+    std::visit(overloaded {
+        [&](const SingleDerivedPath::Opaque &) {
+            // plain drv path; no experimental features required.
+        },
+        [&](const SingleDerivedPath::Built &) {
+            xpSettings.require(Xp::DynamicDerivations);
+        },
+    }, drv.raw());
+}
+
+SingleDerivedPath::Built SingleDerivedPath::Built::parse(
+    const Store & store, ref<SingleDerivedPath> drv,
+    OutputNameView output,
+    const ExperimentalFeatureSettings & xpSettings)
+{
+    drvRequireExperiment(*drv, xpSettings);
     return {
-        .drvPath = std::move(drv),
+        .drvPath = drv,
+        .output = std::string { output },
+    };
+}
+
+DerivedPath::Built DerivedPath::Built::parse(
+    const Store & store, ref<SingleDerivedPath> drv,
+    OutputNameView outputsS,
+    const ExperimentalFeatureSettings & xpSettings)
+{
+    drvRequireExperiment(*drv, xpSettings);
+    return {
+        .drvPath = drv,
         .outputs = OutputsSpec::parse(outputsS),
     };
 }
 
 template <typename DerivedPathT>
 static DerivedPathT parseDerivedPath(
-    const Store & store, std::string_view s, std::string_view separator)
+    const Store & store, std::string_view s, std::string_view separator,
+    const ExperimentalFeatureSettings & xpSettings)
 {
     size_t n = s.rfind(separator);
     if (n == s.npos) {
         return DerivedPathT::Opaque::parse(store, s);
     } else {
         auto path = DerivedPathT::Built::parse(store,
-            DerivedPathT::Opaque::parse(store, s.substr(0, n)),
-            s.substr(n + 1));
+            make_ref<SingleDerivedPath>(parseDerivedPath<SingleDerivedPath>(
+                store,
+                s.substr(0, n),
+                separator,
+                xpSettings)),
+            s.substr(n + 1),
+            xpSettings);
 
-        const auto& basePath = path.drvPath.path;
+        const auto& basePath = path.getBaseStorePath();
         if (!basePath.isDerivation()) {
             throw InvalidPath("cannot use output selection ('%s') on non-derivation store path '%s'",
                               separator, basePath.to_string());
@@ -122,9 +228,36 @@ static DerivedPathT parseDerivedPath(
     }
 }
 
-DerivedPath DerivedPath::parseLegacy(const Store & store, std::string_view s)
+SingleDerivedPath SingleDerivedPath::parse(
+    const Store & store,
+    std::string_view s,
+    const ExperimentalFeatureSettings & xpSettings)
 {
-    return parseDerivedPath<DerivedPath>(store, s, "!");
+    return parseDerivedPath<SingleDerivedPath>(store, s, "^", xpSettings);
+}
+
+SingleDerivedPath SingleDerivedPath::parseLegacy(
+    const Store & store,
+    std::string_view s,
+    const ExperimentalFeatureSettings & xpSettings)
+{
+    return parseDerivedPath<SingleDerivedPath>(store, s, "!", xpSettings);
+}
+
+DerivedPath DerivedPath::parse(
+    const Store & store,
+    std::string_view s,
+    const ExperimentalFeatureSettings & xpSettings)
+{
+    return parseDerivedPath<DerivedPath>(store, s, "^", xpSettings);
+}
+
+DerivedPath DerivedPath::parseLegacy(
+    const Store & store,
+    std::string_view s,
+    const ExperimentalFeatureSettings & xpSettings)
+{
+    return parseDerivedPath<DerivedPath>(store, s, "!", xpSettings);
 }
 
 DerivedPath DerivedPath::fromSingle(const SingleDerivedPath & req)
@@ -142,19 +275,34 @@ DerivedPath DerivedPath::fromSingle(const SingleDerivedPath & req)
     }, req.raw());
 }
 
+const StorePath & SingleDerivedPath::Built::getBaseStorePath() const
+{
+	return drvPath->getBaseStorePath();
+}
+
+const StorePath & DerivedPath::Built::getBaseStorePath() const
+{
+	return drvPath->getBaseStorePath();
+}
+
 template<typename DP>
 static inline const StorePath & getBaseStorePath_(const DP & derivedPath)
 {
     return std::visit(overloaded {
-        [&](const typename DP::Built & bfd) -> const StorePath & {
-            return bfd.drvPath.path;
+        [&](const typename DP::Built & bfd) -> auto & {
+            return bfd.drvPath->getBaseStorePath();
         },
-        [&](const typename DP::Opaque & bo) -> const StorePath & {
+        [&](const typename DP::Opaque & bo) -> auto & {
             return bo.path;
         },
     }, derivedPath.raw());
 }
 
+const StorePath & SingleDerivedPath::getBaseStorePath() const
+{
+	return getBaseStorePath_(*this);
+}
+
 const StorePath & DerivedPath::getBaseStorePath() const
 {
 	return getBaseStorePath_(*this);

lix/libstore/derived-path.hh

@@ -43,9 +43,40 @@ struct SingleDerivedPath;
  * path of the given output name.
  */
 struct SingleDerivedPathBuilt {
-    DerivedPathOpaque drvPath;
+    ref<SingleDerivedPath> drvPath;
     OutputName output;
 
+    /**
+     * Get the store path this is ultimately derived from (by realising
+     * and projecting outputs).
+     *
+     * Note that this is *not* a property of the store object being
+     * referred to, but just of this path --- how we happened to be
+     * referring to that store object. In other words, this means this
+     * function breaks "referential transparency". It should therefore
+     * be used only with great care.
+     */
+    const StorePath & getBaseStorePath() const;
+
+    /**
+     * Uses `^` as the separator
+     */
+    std::string to_string(const Store & store) const;
+    /**
+     * Uses `!` as the separator
+     */
+    std::string to_string_legacy(const Store & store) const;
+    /**
+     * The caller splits on the separator, so it works for both variants.
+     *
+     * @param xpSettings Stop-gap to avoid globals during unit tests.
+     */
+    static SingleDerivedPathBuilt parse(
+        const Store & store, ref<SingleDerivedPath> drvPath,
+        OutputNameView outputs,
+        const ExperimentalFeatureSettings & xpSettings = experimentalFeatureSettings);
+    kj::Promise<Result<JSON>> toJSON(Store & store) const;
+
     DECLARE_CMP(SingleDerivedPathBuilt);
 };
 
@@ -77,11 +108,51 @@ struct SingleDerivedPath : derived_path::detail::SingleDerivedPathRaw {
     inline const Raw & raw() const {
         return static_cast<const Raw &>(*this);
     }
+
+    /**
+     * Get the store path this is ultimately derived from (by realising
+     * and projecting outputs).
+     *
+     * Note that this is *not* a property of the store object being
+     * referred to, but just of this path --- how we happened to be
+     * referring to that store object. In other words, this means this
+     * function breaks "referential transparency". It should therefore
+     * be used only with great care.
+     */
+    const StorePath & getBaseStorePath() const;
+
+    /**
+     * Uses `^` as the separator
+     */
+    std::string to_string(const Store & store) const;
+    /**
+     * Uses `!` as the separator
+     */
+    std::string to_string_legacy(const Store & store) const;
+    /**
+     * Uses `^` as the separator
+     *
+     * @param xpSettings Stop-gap to avoid globals during unit tests.
+     */
+    static SingleDerivedPath parse(
+        const Store & store,
+        std::string_view,
+        const ExperimentalFeatureSettings & xpSettings = experimentalFeatureSettings);
+    /**
+     * Uses `!` as the separator
+     *
+     * @param xpSettings Stop-gap to avoid globals during unit tests.
+     */
+    static SingleDerivedPath parseLegacy(
+        const Store & store,
+        std::string_view,
+        const ExperimentalFeatureSettings & xpSettings = experimentalFeatureSettings);
+    kj::Promise<Result<JSON>> toJSON(Store & store) const;
 };
 
-static inline DerivedPathOpaque makeConstantStorePath(StorePath drvPath)
+static inline ref<SingleDerivedPath> makeConstantStorePathRef(StorePath drvPath)
 {
-    return SingleDerivedPath::Opaque { std::move(drvPath) };
+    return make_ref<SingleDerivedPath>(SingleDerivedPath::Opaque { drvPath });
 }
 
 /**
@@ -97,9 +168,21 @@ static inline DerivedPathOpaque makeConstantStorePath(StorePath drvPath)
  * output name.
  */
 struct DerivedPathBuilt {
-    DerivedPathOpaque drvPath;
+    ref<SingleDerivedPath> drvPath;
     OutputsSpec outputs;
 
+    /**
+     * Get the store path this is ultimately derived from (by realising
+     * and projecting outputs).
+     *
+     * Note that this is *not* a property of the store object being
+     * referred to, but just of this path --- how we happened to be
+     * referring to that store object. In other words, this means this
+     * function breaks "referential transparency". It should therefore
+     * be used only with great care.
+     */
+    const StorePath & getBaseStorePath() const;
+
     /**
      * Uses `^` as the separator
      */
@@ -110,8 +193,13 @@ struct DerivedPathBuilt {
     std::string to_string_legacy(const Store & store) const;
     /**
      * The caller splits on the separator, so it works for both variants.
+     *
+     * @param xpSettings Stop-gap to avoid globals during unit tests.
      */
-    static DerivedPathBuilt parse(const Store & store, DerivedPathOpaque, std::string_view);
+    static DerivedPathBuilt parse(
+        const Store & store, ref<SingleDerivedPath>,
+        std::string_view,
+        const ExperimentalFeatureSettings & xpSettings = experimentalFeatureSettings);
     kj::Promise<Result<JSON>> toJSON(Store & store) const;
 
     DECLARE_CMP(DerivedPathBuilt);
@@ -166,16 +254,44 @@ struct DerivedPath : derived_path::detail::DerivedPathRaw {
      */
     std::string to_string_legacy(const Store & store) const;
     /**
-     * Uses `!` as the separator
+     * Uses `^` as the separator
+     *
+     * @param xpSettings Stop-gap to avoid globals during unit tests.
      */
-    static DerivedPath parseLegacy(const Store & store, std::string_view);
+    static DerivedPath parse(
+        const Store & store,
+        std::string_view,
+        const ExperimentalFeatureSettings & xpSettings = experimentalFeatureSettings);
+    /**
+     * Uses `!` as the separator
+     *
+     * @param xpSettings Stop-gap to avoid globals during unit tests.
+     */
+    static DerivedPath parseLegacy(
+        const Store & store,
+        std::string_view,
+        const ExperimentalFeatureSettings & xpSettings = experimentalFeatureSettings);
 
     /**
      * Convert a `SingleDerivedPath` to a `DerivedPath`.
      */
     static DerivedPath fromSingle(const SingleDerivedPath &);
+
+    kj::Promise<Result<JSON>> toJSON(Store & store) const;
 };
 
 typedef std::vector<DerivedPath> DerivedPaths;
 
+/**
+ * Used by various parser functions to require experimental features as
+ * needed.
+ *
+ * Somewhat unfortunate this cannot just be an implementation detail for
+ * this module.
+ *
+ * @param xpSettings Stop-gap to avoid globals during unit tests.
+ */
+void drvRequireExperiment(
+    const SingleDerivedPath & drv,
+    const ExperimentalFeatureSettings & xpSettings = experimentalFeatureSettings);
 }

lix/libstore/downstream-placeholder.cc

@@ -0,0 +1,58 @@
+#include "lix/libstore/downstream-placeholder.hh"
+#include "lix/libstore/derivations.hh"
+
+namespace nix {
+
+std::string DownstreamPlaceholder::render() const
+{
+    return "/" + hash.to_string(Base::Base32, false);
+}
+
+
+DownstreamPlaceholder DownstreamPlaceholder::unknownCaOutput(
+    const StorePath & drvPath,
+    OutputNameView outputName,
+    const ExperimentalFeatureSettings & xpSettings)
+{
+    xpSettings.require(Xp::CaDerivations);
+    auto drvNameWithExtension = drvPath.name();
+    auto drvName = drvNameWithExtension.substr(0, drvNameWithExtension.size() - 4);
+    auto clearText = "nix-upstream-output:" + std::string { drvPath.hashPart() } + ":" + outputPathName(drvName, outputName);
+    return DownstreamPlaceholder {
+        hashString(HashType::SHA256, clearText)
+    };
+}
+
+DownstreamPlaceholder DownstreamPlaceholder::unknownDerivation(
+    const DownstreamPlaceholder & placeholder,
+    OutputNameView outputName,
+    const ExperimentalFeatureSettings & xpSettings)
+{
+    xpSettings.require(Xp::DynamicDerivations);
+    auto compressed = compressHash(placeholder.hash, 20);
+    auto clearText = "nix-computed-output:"
+        + compressed.to_string(Base::Base32, false)
+        + ":" + std::string { outputName };
+    return DownstreamPlaceholder {
+        hashString(HashType::SHA256, clearText)
+    };
+}
+
+DownstreamPlaceholder DownstreamPlaceholder::fromSingleDerivedPathBuilt(
+    const SingleDerivedPath::Built & b,
+    const ExperimentalFeatureSettings & xpSettings)
+{
+    return std::visit(overloaded {
+        [&](const SingleDerivedPath::Opaque & o) {
+            return DownstreamPlaceholder::unknownCaOutput(o.path, b.output, xpSettings);
+        },
+        [&](const SingleDerivedPath::Built & b2) {
+            return DownstreamPlaceholder::unknownDerivation(
+                DownstreamPlaceholder::fromSingleDerivedPathBuilt(b2, xpSettings),
+                b.output,
+                xpSettings);
+        },
+    }, b.drvPath->raw());
+}
+
+}

lix/libstore/downstream-placeholder.hh

@@ -0,0 +1,91 @@
+#pragma once
+///@file
+
+#include "lix/libutil/hash.hh"
+#include "lix/libstore/path.hh"
+#include "lix/libstore/derived-path.hh"
+
+namespace nix {
+
+/**
+ * Downstream Placeholders are opaque and almost certainly unique values
+ * used to allow derivations to refer to store objects which are yet to
+ * be built and for we do not yet have store paths for.
+ *
+ * They correspond to `DerivedPaths` that are not `DerivedPath::Opaque`,
+ * except for the cases involving input addressing or fixed outputs
+ * where we do know a store path for the derivation output in advance.
+ *
+ * Unlike `DerivationPath`, however, `DownstreamPlaceholder` is
+ * purposefully opaque and obfuscated. This is so they are hard to
+ * create by accident, and so substituting them (once we know what the
+ * path to store object is) is unlikely to capture other stuff it
+ * shouldn't.
+ *
+ * We use them with `Derivation`: the `render()` method is called to
+ * render an opaque string which can be used in the derivation, and the
+ * resolving logic can substitute those strings for store paths when
+ * resolving `Derivation.inputDrvs` to `BasicDerivation.inputSrcs`.
+ */
+class DownstreamPlaceholder
+{
+    /**
+     * `DownstreamPlaceholder` is just a newtype of `Hash`.
+     * This its only field.
+     */
+    Hash hash;
+
+    /**
+     * Newtype constructor
+     */
+    DownstreamPlaceholder(Hash hash) : hash(hash) { }
+
+public:
+    /**
+     * This creates an opaque and almost certainly unique string
+     * deterministically from the placeholder.
+     */
+    std::string render() const;
+
+    /**
+     * Create a placeholder for an unknown output of a content-addressed
+     * derivation.
+     *
+     * The derivation itself is known (we have a store path for it), but
+     * the output doesn't yet have a known store path.
+     *
+     * @param xpSettings Stop-gap to avoid globals during unit tests.
+     */
+    static DownstreamPlaceholder unknownCaOutput(
+        const StorePath & drvPath,
+        OutputNameView outputName,
+        const ExperimentalFeatureSettings & xpSettings = experimentalFeatureSettings);
+
+    /**
+     * Create a placehold for the output of an unknown derivation.
+     *
+     * The derivation is not yet known because it is a dynamic
+     * derivaiton --- it is itself an output of another derivation ---
+     * and we just have (another) placeholder for it.
+     *
+     * @param xpSettings Stop-gap to avoid globals during unit tests.
+     */
+    static DownstreamPlaceholder unknownDerivation(
+        const DownstreamPlaceholder & drvPlaceholder,
+        OutputNameView outputName,
+        const ExperimentalFeatureSettings & xpSettings = experimentalFeatureSettings);
+
+    /**
+     * Convenience constructor that handles both cases (unknown
+     * content-addressed output and unknown derivation), delegating as
+     * needed to `unknownCaOutput` and `unknownDerivation`.
+     *
+     * Recursively builds up a placeholder from a
+     * `SingleDerivedPath::Built.drvPath` chain.
+     */
+    static DownstreamPlaceholder fromSingleDerivedPathBuilt(
+        const SingleDerivedPath::Built & built,
+        const ExperimentalFeatureSettings & xpSettings = experimentalFeatureSettings);
+};
+
+}

lix/libstore/dummy-store.cc

@@ -74,6 +74,10 @@ struct DummyStore final : public Store
     kj::Promise<Result<box_ptr<Source>>> narFromPath(const StorePath & path) override
     try { unsupported("narFromPath"); } catch (...) { return {result::current_exception()}; }
 
+    kj::Promise<Result<std::shared_ptr<const Realisation>>>
+    queryRealisationUncached(const DrvOutput &) override
+    { co_return result::success(nullptr); }
+
     virtual ref<FSAccessor> getFSAccessor() override
     { unsupported("getFSAccessor"); }
 };

lix/libstore/gc.cc

@@ -96,7 +96,7 @@ void LocalStore::createTempRootsFile()
 
     /* Create the temporary roots file for this process. */
     while (true) {
-        auto tmp = makeTempPath(fnTempRoots, ".tmp");
+        auto tmp = makeTempPath(fnTempRoots);
         AutoCloseFD fd{open(tmp.c_str(), O_RDWR | O_CREAT | O_EXCL | O_CLOEXEC, 0600)};
         if (!fd && errno != EEXIST) {
             throw SysError("opening lock file '%1%'", tmp);
@@ -774,12 +774,13 @@ try {
                     /* If keep-derivations is set and this is a
                        derivation, then visit the derivation outputs. */
                     if (gcKeepDerivations && path->isDerivation()) {
-                        for (auto & [name, outPath] :
-                             TRY_AWAIT(queryDerivationOutputMap(*path)))
+                        for (auto & [name, maybeOutPath] :
+                             TRY_AWAIT(queryPartialDerivationOutputMap(*path)))
                         {
-                            if (TRY_AWAIT(isValidPath(outPath)) &&
-                                TRY_AWAIT(queryPathInfo(outPath))->deriver == *path)
-                                enqueue(outPath);
+                            if (maybeOutPath &&
+                                TRY_AWAIT(isValidPath(*maybeOutPath)) &&
+                                TRY_AWAIT(queryPathInfo(*maybeOutPath))->deriver == *path)
+                                enqueue(*maybeOutPath);
                         }
                     }
 
@@ -965,9 +966,9 @@ try {
 
     {
         auto state(_gcState.lock());
+        state->gcWaiters.push_back(std::move(pfp.fulfiller));
 
         if (state->gcRunning) {
-            state->gcWaiters.push_back(std::move(pfp.fulfiller));
             debug("waiting for auto-GC to finish");
             goto sync;
         }

lix/libstore/globals.cc

@@ -87,6 +87,9 @@ Settings::Settings()
 #if defined(__linux__) && defined(SANDBOX_SHELL)
     sandboxPaths.setDefault(tokenizeString<StringSet>("/bin/sh=" SANDBOX_SHELL));
 #endif
+#if defined(__linux__) && defined(PASTA_PATH)
+    pastaPath.setDefault(PASTA_PATH);
+#endif
 
     /* chroot-like behavior from Apple's sandbox */
 #if __APPLE__
@@ -243,7 +246,11 @@ StringSet Settings::getDefaultExtraPlatforms()
     // x86_64 in aarch64 environments or vice versa since they can
     // always exec with their own binary preferences.
     if (std::string{SYSTEM} == "aarch64-darwin" &&
-        runProgram(RunOptions {.program = "arch", .args = {"-arch", "x86_64", "/usr/bin/true"}, .mergeStderrToStdout = true}).first == 0)
+        runProgram(RunOptions {
+            .program = "arch",
+            .args = {"-arch", "x86_64", "/usr/bin/true"},
+            .redirections = {{.from = STDERR_FILENO, .to = STDOUT_FILENO}}
+        }).first == 0)
         extraPlatforms.insert("x86_64-darwin");
 #endif
 

lix/libstore/legacy-ssh-store.cc

@@ -463,6 +463,11 @@ public:
     {
         return {result::success(std::nullopt)};
     }
+
+    kj::Promise<Result<std::shared_ptr<const Realisation>>>
+    queryRealisationUncached(const DrvOutput &) override
+    // TODO: Implement
+    try { unsupported("queryRealisation"); } catch (...) { co_return result::current_exception(); }
 };
 
 void registerLegacySSHStore() {

lix/libstore/local-binary-cache-store.cc

@@ -63,11 +63,10 @@ protected:
         const std::string & mimeType) override
     {
         auto path2 = binaryCacheDir + "/" + path;
-        static std::atomic<int> counter{0};
-        Path tmp = fmt("%s.tmp.%d.%d", path2, getpid(), ++counter);
+        Path tmp = makeTempPath(path2);
         AutoDelete del(tmp, false);
         StreamToSourceAdapter source(istream);
-        writeFile(tmp, source);
+        writeFileExcl(tmp, source);
         renameFile(tmp, path2);
         del.cancel();
     }

lix/libstore/local-store.cc

@@ -68,10 +68,16 @@ struct LocalStore::DBState::Stmts {
     SQLiteStmt QueryReferrers;
     SQLiteStmt InvalidatePath;
     SQLiteStmt AddDerivationOutput;
+    SQLiteStmt RegisterRealisedOutput;
+    SQLiteStmt UpdateRealisedOutput;
     SQLiteStmt QueryValidDerivers;
     SQLiteStmt QueryDerivationOutputs;
+    SQLiteStmt QueryRealisedOutput;
+    SQLiteStmt QueryAllRealisedOutputs;
     SQLiteStmt QueryPathFromHashPart;
     SQLiteStmt QueryValidPaths;
+    SQLiteStmt QueryRealisationReferences;
+    SQLiteStmt AddRealisationReference;
 };
 
 int getSchema(Path schemaPath)
@@ -87,6 +93,91 @@ int getSchema(Path schemaPath)
     return curSchema;
 }
 
+static void migrateCASchema(SQLite& db, Path schemaPath, AutoCloseFD& lockFd, NeverAsync = {})
+{
+    const int nixCASchemaVersion = 4;
+    int curCASchema = getSchema(schemaPath);
+    if (curCASchema != nixCASchemaVersion) {
+        if (curCASchema > nixCASchemaVersion) {
+            throw Error("current Nix store ca-schema is version %1%, but I only support %2%",
+                 curCASchema, nixCASchemaVersion);
+        }
+
+        if (!tryLockFile(lockFd.get(), ltWrite)) {
+            printInfo("waiting for exclusive access to the Nix store for ca drvs...");
+            unlockFile(lockFd.get()); // We have acquired a shared lock; release it to prevent deadlocks
+            lockFile(lockFd.get(), ltWrite);
+        }
+
+        if (curCASchema == 0) {
+            static const char schema[] =
+              #include "ca-specific-schema.sql.gen.hh"
+                ;
+            db.exec(schema);
+            curCASchema = nixCASchemaVersion;
+        }
+
+        if (curCASchema < 2) {
+            SQLiteTxn txn = db.beginTransaction();
+            // Ugly little sql dance to add a new `id` column and make it the primary key
+            db.exec(R"(
+                create table Realisations2 (
+                    id integer primary key autoincrement not null,
+                    drvPath text not null,
+                    outputName text not null, -- symbolic output id, usually "out"
+                    outputPath integer not null,
+                    signatures text, -- space-separated list
+                    foreign key (outputPath) references ValidPaths(id) on delete cascade
+                );
+                insert into Realisations2 (drvPath, outputName, outputPath, signatures)
+                    select drvPath, outputName, outputPath, signatures from Realisations;
+                drop table Realisations;
+                alter table Realisations2 rename to Realisations;
+            )");
+            db.exec(R"(
+                create index if not exists IndexRealisations on Realisations(drvPath, outputName);
+
+                create table if not exists RealisationsRefs (
+                    referrer integer not null,
+                    realisationReference integer,
+                    foreign key (referrer) references Realisations(id) on delete cascade,
+                    foreign key (realisationReference) references Realisations(id) on delete restrict
+                );
+            )");
+            txn.commit();
+        }
+
+        if (curCASchema < 3) {
+            SQLiteTxn txn = db.beginTransaction();
+            // Apply new indices added in this schema update.
+            db.exec(R"(
+                -- used by QueryRealisationReferences
+                create index if not exists IndexRealisationsRefs on RealisationsRefs(referrer);
+                -- used by cascade deletion when ValidPaths is deleted
+                create index if not exists IndexRealisationsRefsOnOutputPath on Realisations(outputPath);
+            )");
+            txn.commit();
+        }
+        if (curCASchema < 4) {
+            SQLiteTxn txn = db.beginTransaction();
+            db.exec(R"(
+                create trigger if not exists DeleteSelfRefsViaRealisations before delete on ValidPaths
+                begin
+                    delete from RealisationsRefs where realisationReference in (
+                    select id from Realisations where outputPath = old.id
+                    );
+                end;
+                -- used by deletion trigger
+                create index if not exists IndexRealisationsRefsRealisationReference on RealisationsRefs(realisationReference);
+            )");
+            txn.commit();
+        }
+
+        writeFileAndSync(schemaPath, fmt("%d", nixCASchemaVersion), 0666);
+        lockFile(lockFd.get(), ltRead);
+    }
+}
+
 // NOTE this constructor uses NeverAsync functions, but they are limited to schema migrations.
 // since these migrations run reasonably quickly *and* approximately never we are not going to
 // bother asyncifying this constructor (especially since it'll propagate to all other stores).
@@ -224,7 +315,7 @@ void LocalStore::initDB(DBState & state)
     else if (curSchema == 0) { /* new store */
         curSchema = nixSchemaVersion;
         openDB(state, true);
-        writeFile(schemaPath, fmt("%1%", nixSchemaVersion), 0666, true);
+        writeFileAndSync(schemaPath, fmt("%1%", nixSchemaVersion), 0666);
     }
 
     else if (curSchema < nixSchemaVersion) {
@@ -277,13 +368,21 @@ void LocalStore::initDB(DBState & state)
             txn.commit();
         }
 
-        writeFile(schemaPath, fmt("%1%", nixSchemaVersion), 0666, true);
+        writeFileAndSync(schemaPath, fmt("%1%", nixSchemaVersion), 0666);
 
         lockFile(globalLock.get(), ltRead, always_progresses);
     }
 
     else openDB(state, false);
 
+    if (experimentalFeatureSettings.isEnabled(Xp::CaDerivations)) {
+        if (!config_.readOnly) {
+            migrateCASchema(state.db, dbDir + "/ca-schema", globalLock, always_progresses);
+        } else {
+            throw Error("need to migrate to content-addressed schema, but this cannot be done in read-only mode");
+        }
+    }
+
     prepareStatements(state);
 }
 
@@ -315,6 +414,50 @@ void LocalStore::prepareStatements(DBState & state)
     state.stmts->QueryPathFromHashPart = state.db.create(
         "select path from ValidPaths where path >= ? limit 1;");
     state.stmts->QueryValidPaths = state.db.create("select path from ValidPaths");
+    if (experimentalFeatureSettings.isEnabled(Xp::CaDerivations)) {
+        state.stmts->RegisterRealisedOutput = state.db.create(
+            R"(
+                insert into Realisations (drvPath, outputName, outputPath, signatures)
+                values (?, ?, (select id from ValidPaths where path = ?), ?)
+                ;
+            )");
+        state.stmts->UpdateRealisedOutput = state.db.create(
+            R"(
+                update Realisations
+                    set signatures = ?
+                where
+                    drvPath = ? and
+                    outputName = ?
+                ;
+            )");
+        state.stmts->QueryRealisedOutput = state.db.create(
+            R"(
+                select Realisations.id, Output.path, Realisations.signatures from Realisations
+                    inner join ValidPaths as Output on Output.id = Realisations.outputPath
+                    where drvPath = ? and outputName = ?
+                    ;
+            )");
+        state.stmts->QueryAllRealisedOutputs = state.db.create(
+            R"(
+                select outputName, Output.path from Realisations
+                    inner join ValidPaths as Output on Output.id = Realisations.outputPath
+                    where drvPath = ?
+                    ;
+            )");
+        state.stmts->QueryRealisationReferences = state.db.create(
+            R"(
+                select drvPath, outputName from Realisations
+                    join RealisationsRefs on realisationReference = Realisations.id
+                    where referrer = ?;
+            )");
+        state.stmts->AddRealisationReference = state.db.create(
+            R"(
+                insert or replace into RealisationsRefs (referrer, realisationReference)
+                values (
+                    (select id from Realisations where drvPath = ? and outputName = ?),
+                    (select id from Realisations where drvPath = ? and outputName = ?));
+            )");
+    }
 }
 
 
@@ -623,6 +766,82 @@ void canonicalisePathMetaData(const Path & path,
 }
 
 
+kj::Promise<Result<void>>
+LocalStore::registerDrvOutput(const Realisation & info, CheckSigsFlag checkSigs)
+try {
+    experimentalFeatureSettings.require(Xp::CaDerivations);
+    if (checkSigs == NoCheckSigs || !realisationIsUntrusted(info))
+        TRY_AWAIT(registerDrvOutput(info));
+    else
+        throw Error("cannot register realisation '%s' because it lacks a signature by a trusted key", info.outPath.to_string());
+    co_return result::success();
+} catch (...) {
+    co_return result::current_exception();
+}
+
+kj::Promise<Result<void>> LocalStore::registerDrvOutput(const Realisation & info)
+try {
+    experimentalFeatureSettings.require(Xp::CaDerivations);
+    // NOLINTNEXTLINE(cppcoreguidelines-avoid-capturing-lambda-coroutines)
+    TRY_AWAIT(retrySQLite([&]() -> kj::Promise<Result<void>> {
+        try {
+            auto state = co_await _dbState.lock();
+            if (auto oldR = queryRealisation_(*state, info.id)) {
+                if (info.isCompatibleWith(*oldR)) {
+                    auto combinedSignatures = oldR->signatures;
+                    combinedSignatures.insert(info.signatures.begin(),
+                        info.signatures.end());
+                    state->stmts->UpdateRealisedOutput.use()
+                        (concatStringsSep(" ", combinedSignatures))
+                        (info.id.strHash())
+                        (info.id.outputName)
+                        .exec();
+                } else {
+                    throw Error("Trying to register a realisation of '%s', but we already "
+                                "have another one locally.\n"
+                                "Local:  %s\n"
+                                "Remote: %s",
+                        info.id.to_string(),
+                        printStorePath(oldR->outPath),
+                        printStorePath(info.outPath)
+                    );
+                }
+            } else {
+                state->stmts->RegisterRealisedOutput.use()
+                    (info.id.strHash())
+                    (info.id.outputName)
+                    (printStorePath(info.outPath))
+                    (concatStringsSep(" ", info.signatures))
+                    .exec();
+            }
+            for (auto & [outputId, depPath] : info.dependentRealisations) {
+                auto localRealisation = queryRealisationCore_(*state, outputId);
+                if (!localRealisation)
+                    throw Error("unable to register the derivation '%s' as it "
+                                "depends on the non existent '%s'",
+                        info.id.to_string(), outputId.to_string());
+                if (localRealisation->second.outPath != depPath)
+                    throw Error("unable to register the derivation '%s' as it "
+                                "depends on a realisation of '%s' that doesn’t"
+                                "match what we have locally",
+                        info.id.to_string(), outputId.to_string());
+                state->stmts->AddRealisationReference.use()
+                    (info.id.strHash())
+                    (info.id.outputName)
+                    (outputId.strHash())
+                    (outputId.outputName)
+                    .exec();
+            }
+            co_return result::success();
+        } catch (...) {
+            co_return result::current_exception();
+        }
+    }));
+    co_return result::success();
+} catch (...) {
+    co_return result::current_exception();
+}
+
 void LocalStore::cacheDrvOutputMapping(
     DBState & state,
     const uint64_t deriver,
@@ -670,8 +889,11 @@ try {
            registration above is undone. */
         if (checkOutputs) TRY_AWAIT(drv.checkInvariants(*this, info.path));
 
-        for (auto & i : drv.outputsAndPaths(*this)) {
-            cacheDrvOutputMapping(state, id, i.first, i.second.second);
+        for (auto & i : drv.outputsAndOptPaths(*this)) {
+            /* Floating CA derivations have indeterminate output paths until
+               they are built, so don't register anything in that case */
+            if (i.second.second)
+                cacheDrvOutputMapping(state, id, i.first, *i.second.second);
         }
     }
 
@@ -883,15 +1105,15 @@ try {
 }
 
 
-kj::Promise<Result<std::map<std::string, StorePath>>>
-LocalStore::queryStaticDerivationOutputMap(const StorePath & path)
+kj::Promise<Result<std::map<std::string, std::optional<StorePath>>>>
+LocalStore::queryStaticPartialDerivationOutputMap(const StorePath & path)
 try {
     co_return TRY_AWAIT(
         // NOLINTNEXTLINE(cppcoreguidelines-avoid-capturing-lambda-coroutines)
-        retrySQLite([&]() -> kj::Promise<Result<std::map<std::string, StorePath>>> {
+        retrySQLite([&]() -> kj::Promise<Result<std::map<std::string, std::optional<StorePath>>>> {
             try {
                 auto state = co_await _dbState.lock();
-                std::map<std::string, StorePath> outputs;
+                std::map<std::string, std::optional<StorePath>> outputs;
                 uint64_t drvId;
                 drvId = queryValidPathId(*state, path);
                 auto use(state->stmts->QueryDerivationOutputs.use()(drvId));
@@ -1082,6 +1304,11 @@ bool LocalStore::pathInfoIsUntrusted(const ValidPathInfo & info)
     return config_.requireSigs && !info.checkSignatures(*this, getPublicKeys());
 }
 
+bool LocalStore::realisationIsUntrusted(const Realisation & realisation)
+{
+    return config_.requireSigs && !realisation.checkSignatures(getPublicKeys());
+}
+
 kj::Promise<Result<void>> LocalStore::addToStore(
     const ValidPathInfo & info,
     AsyncInputStream & source,
@@ -1402,7 +1629,7 @@ std::pair<Path, AutoCloseFD> LocalStore::createTempDirInStore()
         /* There is a slight possibility that `tmpDir' gets deleted by
            the GC between createTempDir() and when we acquire a lock on it.
            We'll repeat until 'tmpDir' exists and we've locked it. */
-        tmpDirFn = createTempDir(config_.realStoreDir, "tmp");
+        tmpDirFn = createTempSubdir(config_.realStoreDir, "tmp");
         tmpDirFd = AutoCloseFD{open(tmpDirFn.c_str(), O_RDONLY | O_DIRECTORY)};
         if (tmpDirFd.get() < 0) {
             continue;
@@ -1668,6 +1895,18 @@ try {
 }
 
 
+void LocalStore::signRealisation(Realisation & realisation)
+{
+    // FIXME: keep secret keys in memory.
+
+    auto secretKeyFiles = settings.secretKeyFiles;
+
+    for (auto & secretKeyFile : secretKeyFiles.get()) {
+        SecretKey secretKey(readFile(secretKeyFile));
+        realisation.sign(secretKey);
+    }
+}
+
 void LocalStore::signPathInfo(ValidPathInfo & info)
 {
     // FIXME: keep secret keys in memory.
@@ -1681,6 +1920,81 @@ void LocalStore::signPathInfo(ValidPathInfo & info)
 }
 
 
+std::optional<std::pair<int64_t, Realisation>> LocalStore::queryRealisationCore_(
+        LocalStore::DBState & state,
+        const DrvOutput & id)
+{
+    auto useQueryRealisedOutput(
+            state.stmts->QueryRealisedOutput.use()
+                (id.strHash())
+                (id.outputName));
+    if (!useQueryRealisedOutput.next())
+        return std::nullopt;
+    auto realisationDbId = useQueryRealisedOutput.getInt(0);
+    auto outputPath = parseStorePath(useQueryRealisedOutput.getStr(1));
+    auto signatures =
+        tokenizeString<StringSet>(useQueryRealisedOutput.getStr(2));
+
+    return {{
+        realisationDbId,
+        Realisation{
+            .id = id,
+            .outPath = outputPath,
+            .signatures = signatures,
+        }
+    }};
+}
+
+std::optional<const Realisation> LocalStore::queryRealisation_(
+            LocalStore::DBState & state,
+            const DrvOutput & id)
+{
+    auto maybeCore = queryRealisationCore_(state, id);
+    if (!maybeCore)
+        return std::nullopt;
+    auto [realisationDbId, res] = *maybeCore;
+
+    std::map<DrvOutput, StorePath> dependentRealisations;
+    auto useRealisationRefs(
+        state.stmts->QueryRealisationReferences.use()
+            (realisationDbId));
+    while (useRealisationRefs.next()) {
+        auto depId = DrvOutput {
+            Hash::parseAnyPrefixed(useRealisationRefs.getStr(0)),
+            useRealisationRefs.getStr(1),
+        };
+        auto dependentRealisation = queryRealisationCore_(state, depId);
+        assert(dependentRealisation); // Enforced by the db schema
+        auto outputPath = dependentRealisation->second.outPath;
+        dependentRealisations.insert({depId, outputPath});
+    }
+
+    res.dependentRealisations = dependentRealisations;
+
+    return { res };
+}
+
+kj::Promise<Result<std::shared_ptr<const Realisation>>>
+LocalStore::queryRealisationUncached(const DrvOutput & id)
+try {
+    auto maybeRealisation =
+        // NOLINTNEXTLINE(cppcoreguidelines-avoid-capturing-lambda-coroutines)
+        TRY_AWAIT(retrySQLite([&]() -> kj::Promise<Result<std::optional<const Realisation>>> {
+            try {
+                auto state = co_await _dbState.lock();
+                co_return queryRealisation_(*state, id);
+            } catch (...) {
+                co_return result::current_exception();
+            }
+        }));
+    if (maybeRealisation)
+        co_return std::make_shared<const Realisation>(maybeRealisation.value());
+    else
+        co_return result::success(nullptr);
+} catch (...) {
+    co_return result::current_exception();
+}
+
 ContentAddress LocalStore::hashCAPath(
     const ContentAddressMethod & method, const HashType & hashType,
     const StorePath & path)
@@ -1729,9 +2043,9 @@ try {
 
     createDirs(dirOf(logPath));
 
-    auto tmpFile = fmt("%s.tmp.%d", logPath, getpid());
+    auto tmpFile = makeTempSiblingPath(logPath);
 
-    writeFile(tmpFile, compress("bzip2", log));
+    writeFileExcl(tmpFile, compress("bzip2", log));
 
     renameFile(tmpFile, logPath);
     co_return result::success();

lix/libstore/local-store.hh

@@ -135,6 +135,8 @@ private:
 
     Sync<GCState> _gcState;
 
+    std::optional<AssociatedCredentials> association;
+
 public:
 
     const Path dbDir;
@@ -148,6 +150,16 @@ public:
     LocalStoreConfig & config() override { return config_; }
     const LocalStoreConfig & config() const override { return config_; }
 
+    std::optional<AssociatedCredentials> associatedCredentials() const override
+    {
+        return association;
+    }
+
+    void associateWithCredentials(uid_t user, gid_t group)
+    {
+        association = {user, group};
+    }
+
 private:
 
     const PublicKeys & getPublicKeys();
@@ -201,8 +213,8 @@ public:
 
     kj::Promise<Result<StorePathSet>> queryValidDerivers(const StorePath & path) override;
 
-    kj::Promise<Result<std::map<std::string, StorePath>>>
-    queryStaticDerivationOutputMap(const StorePath & path) override;
+    kj::Promise<Result<std::map<std::string, std::optional<StorePath>>>>
+    queryStaticPartialDerivationOutputMap(const StorePath & path) override;
 
     kj::Promise<Result<std::optional<StorePath>>>
     queryPathFromHashPart(const std::string & hashPart) override;
@@ -210,6 +222,7 @@ public:
     kj::Promise<Result<StorePathSet>> querySubstitutablePaths(const StorePathSet & paths) override;
 
     bool pathInfoIsUntrusted(const ValidPathInfo &) override;
+    bool realisationIsUntrusted(const Realisation & ) override;
 
     kj::Promise<Result<void>> addToStore(const ValidPathInfo & info, AsyncInputStream & source,
         RepairFlag repair, CheckSigsFlag checkSigs) override;
@@ -318,12 +331,20 @@ public:
      * Register the store path 'output' as the output named 'outputName' of
      * derivation 'deriver'.
      */
+    kj::Promise<Result<void>> registerDrvOutput(const Realisation & info) override;
+    kj::Promise<Result<void>>
+    registerDrvOutput(const Realisation & info, CheckSigsFlag checkSigs) override;
     void cacheDrvOutputMapping(
         DBState & state,
         const uint64_t deriver,
         const std::string & outputName,
         const StorePath & output);
 
+    std::optional<const Realisation> queryRealisation_(DBState & state, const DrvOutput & id);
+    std::optional<std::pair<int64_t, Realisation>> queryRealisationCore_(DBState & state, const DrvOutput & id);
+    kj::Promise<Result<std::shared_ptr<const Realisation>>>
+    queryRealisationUncached(const DrvOutput&) override;
+
     kj::Promise<Result<std::optional<std::string>>> getVersion() override;
 
 private:
@@ -391,6 +412,7 @@ private:
      * specified by the ‘secret-key-files’ option.
      */
     void signPathInfo(ValidPathInfo & info);
+    void signRealisation(Realisation &);
 
     // XXX: Make a generic `Store` method
     ContentAddress hashCAPath(

lix/libstore/meson.build

@@ -1,5 +1,5 @@
 libstore_generated_headers = []
-foreach header : [ 'schema.sql' ]
+foreach header : [ 'schema.sql', 'ca-specific-schema.sql' ]
   libstore_generated_headers += custom_target(
     command : [ 'bash', '-c', 'echo \'R"__NIX_STR(\' | cat - @INPUT@ && echo \')__NIX_STR"\'' ],
     input : header,
@@ -82,6 +82,7 @@ libstore_setting_definitions = files(
   'settings/narinfo-cache-negative-ttl.md',
   'settings/narinfo-cache-positive-ttl.md',
   'settings/netrc-file.md',
+  'settings/pasta-path.md',
   'settings/plugin-files.md',
   'settings/post-build-hook.md',
   'settings/pre-build-hook.md',
@@ -140,6 +141,7 @@ libstore_sources = files(
   'build-result.cc',
   'build/child.cc',
   'build/derivation-goal.cc',
+  'build/drv-output-substitution-goal.cc',
   'build/entry-points.cc',
   'build/goal.cc',
   'build/hook-instance.cc',
@@ -155,7 +157,9 @@ libstore_sources = files(
   'crypto.cc',
   'daemon.cc',
   'derivations.cc',
+  'derived-path-map.cc',
   'derived-path.cc',
+  'downstream-placeholder.cc',
   'dummy-store.cc',
   'export-import.cc',
   'filetransfer.cc',
@@ -207,6 +211,7 @@ libstore_headers = files(
   'build-result.hh',
   'build/child.hh',
   'build/derivation-goal.hh',
+  'build/drv-output-substitution-goal.hh',
   'build/goal.hh',
   'build/hook-instance.hh',
   'build/local-derivation-goal.hh',
@@ -221,7 +226,9 @@ libstore_headers = files(
   'crypto.hh',
   'daemon.hh',
   'derivations.hh',
+  'derived-path-map.hh',
   'derived-path.hh',
+  'downstream-placeholder.hh',
   'dummy-store.hh',
   'filetransfer.hh',
   'fs-accessor.hh',
@@ -326,6 +333,12 @@ elif busybox.found()
   }
 endif
 
+if pasta.found()
+  cpp_str_defines += {
+    'PASTA_PATH': pasta.full_path(),
+  }
+endif
+
 cpp_args = []
 
 foreach name, value : cpp_str_defines

lix/libstore/misc.cc

@@ -37,9 +37,9 @@ try {
                         res.insert(i);
 
                 if (includeDerivers && path.isDerivation())
-                    for (auto& [_, outPath] : TRY_AWAIT(queryDerivationOutputMap(path)))
-                        if (TRY_AWAIT(isValidPath(outPath)))
-                            res.insert(outPath);
+                    for (auto& [_, maybeOutPath] : TRY_AWAIT(queryPartialDerivationOutputMap(path)))
+                        if (maybeOutPath && TRY_AWAIT(isValidPath(*maybeOutPath)))
+                            res.insert(*maybeOutPath);
                 co_return res;
             } catch (...) {
                 co_return result::current_exception();
@@ -56,9 +56,9 @@ try {
                         res.insert(ref);
 
                 if (includeOutputs && path.isDerivation())
-                    for (auto& [_, outPath] : TRY_AWAIT(queryDerivationOutputMap(path)))
-                        if (TRY_AWAIT(isValidPath(outPath)))
-                            res.insert(outPath);
+                    for (auto& [_, maybeOutPath] : TRY_AWAIT(queryPartialDerivationOutputMap(path)))
+                        if (maybeOutPath && TRY_AWAIT(isValidPath(*maybeOutPath)))
+                            res.insert(*maybeOutPath);
 
                 if (includeDerivers && info->deriver && TRY_AWAIT(isValidPath(*info->deriver)))
                     res.insert(*info->deriver);
@@ -150,13 +150,17 @@ struct QueryMissingContext
 
     kj::Promise<Result<void>> queryMissing(const std::vector<DerivedPath> & targets);
 
-    void enqueueDerivedPaths(DerivedPathOpaque inputDrv, const StringSet & inputNode)
+    void enqueueDerivedPaths(ref<SingleDerivedPath> inputDrv, const DerivedPathMap<StringSet>::ChildNode & inputNode)
     {
-        if (!inputNode.empty()) {
-            pool.enqueueWithAio([this, path{DerivedPath::Built{std::move(inputDrv), inputNode}}](
+        if (!inputNode.value.empty()) {
+            pool.enqueueWithAio([this, path{DerivedPath::Built{inputDrv, inputNode.value}}](
                                     AsyncIoRoot & aio
                                 ) { doPath(aio, path); });
         }
+        for (const auto & [outputName, childNode] : inputNode.childMap)
+            enqueueDerivedPaths(
+                make_ref<SingleDerivedPath>(SingleDerivedPath::Built { inputDrv, outputName }),
+                childNode);
     }
 
     void mustBuildDrv(const StorePath & drvPath, const Derivation & drv)
@@ -166,8 +170,8 @@ struct QueryMissingContext
             state->willBuild.insert(drvPath);
         }
 
-        for (const auto & [inputDrv, inputNode] : drv.inputDrvs) {
-            enqueueDerivedPaths(makeConstantStorePath(inputDrv), inputNode);
+        for (const auto & [inputDrv, inputNode] : drv.inputDrvs.map) {
+            enqueueDerivedPaths(makeConstantStorePathRef(inputDrv), inputNode);
         }
     }
 
@@ -230,7 +234,13 @@ struct QueryMissingContext
 
     void doPathBuilt(AsyncIoRoot & aio, const DerivedPath::Built & bfd)
     {
-        auto & drvPath = bfd.drvPath.path;
+        auto drvPathP = std::get_if<DerivedPath::Opaque>(&*bfd.drvPath);
+        if (!drvPathP) {
+            // TODO make work in this case.
+            warn("Ignoring dynamic derivation %s while querying missing paths; not yet implemented", bfd.drvPath->to_string(store));
+            return;
+        }
+        auto & drvPath = drvPathP->path;
 
         if (!aio.blockOn(store.isValidPath(drvPath))) {
             // FIXME: we could try to substitute the derivation.
@@ -240,18 +250,55 @@ struct QueryMissingContext
         }
 
         StorePathSet invalid;
-        for (auto & [outputName, path] :
-             aio.blockOn(store.queryDerivationOutputMap(drvPath)))
+        /* true for regular derivations, and CA derivations for which we
+            have a trust mapping for all wanted outputs. */
+        auto knownOutputPaths = true;
+        for (auto & [outputName, pathOpt] :
+             aio.blockOn(store.queryPartialDerivationOutputMap(drvPath)))
         {
-            if (bfd.outputs.contains(outputName) && !aio.blockOn(store.isValidPath(path)))
-                invalid.insert(path);
+            if (!pathOpt) {
+                knownOutputPaths = false;
+                break;
+            }
+            if (bfd.outputs.contains(outputName) && !aio.blockOn(store.isValidPath(*pathOpt)))
+                invalid.insert(*pathOpt);
         }
-        if (invalid.empty()) return;
+        if (knownOutputPaths && invalid.empty()) return;
 
         auto drv = make_ref<Derivation>(aio.blockOn(store.derivationFromPath(drvPath)));
         ParsedDerivation parsedDrv(StorePath(drvPath), *drv);
 
-        if (settings.useSubstitutes && parsedDrv.substitutesAllowed()) {
+        if (!knownOutputPaths && settings.useSubstitutes && parsedDrv.substitutesAllowed()) {
+            experimentalFeatureSettings.require(Xp::CaDerivations);
+
+            // If there are unknown output paths, attempt to find if the
+            // paths are known to substituters through a realisation.
+            auto outputHashes = aio.blockOn(staticOutputHashes(store, *drv));
+            knownOutputPaths = true;
+
+            for (auto [outputName, hash] : outputHashes) {
+                if (!bfd.outputs.contains(outputName))
+                    continue;
+
+                bool found = false;
+                for (auto &sub : aio.blockOn(getDefaultSubstituters())) {
+                    auto realisation = aio.blockOn(sub->queryRealisation({hash, outputName}));
+                    if (!realisation)
+                        continue;
+                    found = true;
+                    if (!aio.blockOn(store.isValidPath(realisation->outPath)))
+                        invalid.insert(realisation->outPath);
+                    break;
+                }
+                if (!found) {
+                    // Some paths did not have a realisation, this must be built.
+                    knownOutputPaths = false;
+                    break;
+                }
+            }
+        }
+
+        if (knownOutputPaths && settings.useSubstitutes && parsedDrv.substitutesAllowed()) {
             auto drvState = make_ref<Sync<DrvState>>(DrvState(invalid.size()));
             for (auto & output : invalid) {
                 pool.enqueueWithAio([=, this](AsyncIoRoot & aio) {
@@ -349,32 +396,189 @@ try {
     co_return result::current_exception();
 }
 
+static kj::Promise<Result<std::map<DrvOutput, StorePath>>> drvOutputReferences(
+    const std::set<Realisation> & inputRealisations,
+    const StorePathSet & pathReferences)
+try {
+    std::map<DrvOutput, StorePath> res;
+
+    for (const auto & input : inputRealisations) {
+        if (pathReferences.count(input.outPath)) {
+            res.insert({input.id, input.outPath});
+        }
+    }
+
+    co_return res;
+} catch (...) {
+    co_return result::current_exception();
+}
+
+kj::Promise<Result<std::map<DrvOutput, StorePath>>> drvOutputReferences(
+    Store & store,
+    const Derivation & drv,
+    const StorePath & outputPath,
+    Store * evalStore_)
+try {
+    auto & evalStore = evalStore_ ? *evalStore_ : store;
+
+    std::set<Realisation> inputRealisations;
+
+    std::function<
+        kj::Promise<Result<void>>(const StorePath &, const DerivedPathMap<StringSet>::ChildNode &)>
+        accumRealisations;
+
+    // NOLINTNEXTLINE(cppcoreguidelines-avoid-capturing-lambda-coroutines)
+    accumRealisations = [&](const StorePath & inputDrv,
+                            const DerivedPathMap<StringSet>::ChildNode & inputNode
+                        ) -> kj::Promise<Result<void>> {
+        try {
+            if (!inputNode.value.empty()) {
+                auto outputHashes = TRY_AWAIT(
+                    staticOutputHashes(evalStore, TRY_AWAIT(evalStore.readDerivation(inputDrv)))
+                );
+                for (const auto & outputName : inputNode.value) {
+                    auto outputHash = get(outputHashes, outputName);
+                    if (!outputHash)
+                        throw Error(
+                            "output '%s' of derivation '%s' isn't realised", outputName,
+                            store.printStorePath(inputDrv));
+                    auto thisRealisation = TRY_AWAIT(store.queryRealisation(
+                        DrvOutput{*outputHash, outputName}));
+                    if (!thisRealisation)
+                        throw Error(
+                            "output '%s' of derivation '%s' isn’t built", outputName,
+                            store.printStorePath(inputDrv));
+                    inputRealisations.insert(*thisRealisation);
+                }
+            }
+            if (!inputNode.value.empty()) {
+                auto d = makeConstantStorePathRef(inputDrv);
+                for (const auto & [outputName, childNode] : inputNode.childMap) {
+                    SingleDerivedPath next = SingleDerivedPath::Built { d, outputName };
+                    TRY_AWAIT(accumRealisations(
+                        // TODO deep resolutions for dynamic derivations, issue #8947, would go here.
+                        TRY_AWAIT(resolveDerivedPath(store, next, evalStore_)),
+                        childNode));
+                }
+            }
+            co_return result::success();
+        } catch (...) {
+            co_return result::current_exception();
+        }
+    };
+
+    for (const auto & [inputDrv, inputNode] : drv.inputDrvs.map)
+        TRY_AWAIT(accumRealisations(inputDrv, inputNode));
+
+    auto info = TRY_AWAIT(store.queryPathInfo(outputPath));
+
+    co_return TRY_AWAIT(drvOutputReferences(
+        TRY_AWAIT(Realisation::closure(store, inputRealisations)), info->references
+    ));
+} catch (...) {
+    co_return result::current_exception();
+}
+
 kj::Promise<Result<OutputPathMap>>
 resolveDerivedPath(Store & store, const DerivedPath::Built & bfd, Store * evalStore_)
 try {
-    auto drvPath = bfd.drvPath.path;
+    auto drvPath = TRY_AWAIT(resolveDerivedPath(store, *bfd.drvPath, evalStore_));
 
-    auto outputs_ = TRY_AWAIT(store.queryDerivationOutputMap(drvPath, evalStore_));
+    auto outputsOpt_ = TRY_AWAIT(store.queryPartialDerivationOutputMap(drvPath, evalStore_));
 
-    co_return std::visit(overloaded {
+    auto outputsOpt = std::visit(overloaded {
         [&](const OutputsSpec::All &) {
             // Keep all outputs
-            return std::move(outputs_);
+            return std::move(outputsOpt_);
         },
         [&](const OutputsSpec::Names & names) {
             // Get just those mentioned by name
-            std::map<std::string, StorePath> outputsOpt;
+            std::map<std::string, std::optional<StorePath>> outputsOpt;
             for (auto & output : names) {
-                auto * pOutputPathOpt = get(outputs_, output);
+                auto * pOutputPathOpt = get(outputsOpt_, output);
                 if (!pOutputPathOpt)
                     throw Error(
                         "the derivation '%s' doesn't have an output named '%s'",
-                        bfd.drvPath.to_string(store), output);
+                        bfd.drvPath->to_string(store), output);
                 outputsOpt.insert_or_assign(output, std::move(*pOutputPathOpt));
             }
             return outputsOpt;
         },
     }, bfd.outputs.raw);
+
+    OutputPathMap outputs;
+    for (auto & [outputName, outputPathOpt] : outputsOpt) {
+        if (!outputPathOpt)
+            throw MissingRealisation(bfd.drvPath->to_string(store), outputName);
+        auto & outputPath = *outputPathOpt;
+        outputs.insert_or_assign(outputName, outputPath);
+    }
+    co_return outputs;
+} catch (...) {
+    co_return result::current_exception();
+}
+
+
+kj ::Promise<Result<StorePath>>
+resolveDerivedPath(Store & store, const SingleDerivedPath & req, Store * evalStore_)
+try {
+    auto & evalStore = evalStore_ ? *evalStore_ : store;
+
+    auto handlers = overloaded {
+        [&](const SingleDerivedPath::Opaque & bo) -> kj::Promise<Result<StorePath>> {
+            return {bo.path};
+        },
+        // NOLINTNEXTLINE(cppcoreguidelines-avoid-capturing-lambda-coroutines)
+        [&](const SingleDerivedPath::Built & bfd) -> kj::Promise<Result<StorePath>> {
+            try {
+                auto drvPath = TRY_AWAIT(resolveDerivedPath(store, *bfd.drvPath, evalStore_));
+                auto outputPaths =
+                    TRY_AWAIT(evalStore.queryPartialDerivationOutputMap(drvPath, evalStore_));
+                if (outputPaths.count(bfd.output) == 0)
+                    throw Error("derivation '%s' does not have an output named '%s'",
+                        store.printStorePath(drvPath), bfd.output);
+                auto & optPath = outputPaths.at(bfd.output);
+                if (!optPath)
+                    throw MissingRealisation(bfd.drvPath->to_string(store), bfd.output);
+                co_return *optPath;
+            } catch (...) {
+                co_return result::current_exception();
+            }
+        },
+    };
+    co_return TRY_AWAIT(std::visit(handlers, req.raw()));
+} catch (...) {
+    co_return result::current_exception();
+}
+
+
+kj::Promise<Result<OutputPathMap>>
+resolveDerivedPath(Store & store, const DerivedPath::Built & bfd)
+try {
+    auto drvPath = TRY_AWAIT(resolveDerivedPath(store, *bfd.drvPath));
+    auto outputMap = TRY_AWAIT(store.queryDerivationOutputMap(drvPath));
+    auto outputsLeft = std::visit(overloaded {
+        [&](const OutputsSpec::All &) {
+            return StringSet {};
+        },
+        [&](const OutputsSpec::Names & names) {
+            return static_cast<StringSet>(names);
+        },
+    }, bfd.outputs.raw);
+    for (auto iter = outputMap.begin(); iter != outputMap.end();) {
+        auto & outputName = iter->first;
+        if (bfd.outputs.contains(outputName)) {
+            outputsLeft.erase(outputName);
+            ++iter;
+        } else {
+            iter = outputMap.erase(iter);
+        }
+    }
+    if (!outputsLeft.empty())
+        throw Error("derivation '%s' does not have an outputs %s",
+            store.printStorePath(drvPath),
+            concatStringsSep(", ", quoteStrings(std::get<OutputsSpec::Names>(bfd.outputs.raw))));
+    co_return outputMap;
 } catch (...) {
     co_return result::current_exception();
 }

lix/libstore/nar-info-disk-cache.cc

@@ -43,6 +43,15 @@ create table if not exists NARs (
     foreign key (cache) references BinaryCaches(id) on delete cascade
 );
 
+create table if not exists Realisations (
+    cache integer not null,
+    outputId text not null,
+    content blob, -- Json serialisation of the realisation, or null if the realisation is absent
+    timestamp        integer not null,
+    primary key (cache, outputId),
+    foreign key (cache) references BinaryCaches(id) on delete cascade
+);
+
 create table if not exists LastPurge (
     dummy            text primary key,
     value            integer
@@ -72,7 +81,8 @@ public:
     {
         SQLite db;
         SQLiteStmt insertCache, queryCache, insertNAR, insertMissingNAR,
-            queryNAR, purgeCache;
+            queryNAR, insertRealisation, insertMissingRealisation,
+            queryRealisation, purgeCache;
         std::map<std::string, Cache> caches;
     };
 
@@ -106,6 +116,26 @@ public:
         state->queryNAR = state->db.create(
             "select present, namePart, url, compression, fileHash, fileSize, narHash, narSize, refs, deriver, sigs, ca from NARs where cache = ? and hashPart = ? and ((present = 0 and timestamp > ?) or (present = 1 and timestamp > ?))");
 
+        state->insertRealisation = state->db.create(
+            R"(
+                insert or replace into Realisations(cache, outputId, content, timestamp)
+                    values (?, ?, ?, ?)
+            )");
+
+        state->insertMissingRealisation = state->db.create(
+            R"(
+                insert or replace into Realisations(cache, outputId, timestamp)
+                    values (?, ?, ?)
+            )");
+
+        state->queryRealisation = state->db.create(
+            R"(
+                select content from Realisations
+                    where cache = ? and outputId = ?  and
+                        ((content is null and timestamp > ?) or
+                         (content is not null and timestamp > ?))
+            )");
+
         /* Periodically purge expired entries from the database. */
         retrySQLite([&]() {
             auto now = time(0);
@@ -252,6 +282,37 @@ public:
         }, always_progresses);
     }
 
+    std::pair<Outcome, std::shared_ptr<Realisation>> lookupRealisation(
+        const std::string & uri, const DrvOutput & id) override
+    {
+        return retrySQLite([&]() -> std::pair<Outcome, std::shared_ptr<Realisation>> {
+            auto state(_state.lock());
+
+            auto & cache(getCache(*state, uri));
+
+            auto now = time(0);
+
+            auto queryRealisation(state->queryRealisation.use()
+                (cache.id)
+                (id.to_string())
+                (now - settings.ttlNegativeNarInfoCache)
+                (now - settings.ttlPositiveNarInfoCache));
+
+            if (!queryRealisation.next())
+                return {oUnknown, 0};
+
+            if (queryRealisation.isNull(0))
+                return {oInvalid, 0};
+
+            auto realisation =
+                std::make_shared<Realisation>(Realisation::fromJSON(
+                    json::parse(queryRealisation.getStr(0), "a nar cache entry"),
+                    "Local disk cache"));
+
+            return {oValid, realisation};
+        }, always_progresses);
+    }
+
     void upsertNarInfo(
         const std::string & uri, const std::string & hashPart,
         std::shared_ptr<const ValidPathInfo> info) override
@@ -291,6 +352,39 @@ public:
             }
         }, always_progresses);
     }
+
+    void upsertRealisation(
+        const std::string & uri,
+        const Realisation & realisation) override
+    {
+        retrySQLite([&]() {
+            auto state(_state.lock());
+
+            auto & cache(getCache(*state, uri));
+
+            state->insertRealisation.use()
+                (cache.id)
+                (realisation.id.to_string())
+                (realisation.toJSON().dump())
+                (time(0)).exec();
+        }, always_progresses);
+
+    }
+
+    virtual void upsertAbsentRealisation(
+        const std::string & uri,
+        const DrvOutput & id) override
+    {
+        retrySQLite([&]() {
+            auto state(_state.lock());
+
+            auto & cache(getCache(*state, uri));
+            state->insertMissingRealisation.use()
+                (cache.id)
+                (id.to_string())
+                (time(0)).exec();
+        }, always_progresses);
+    }
 };
 
 ref<NarInfoDiskCache> getNarInfoDiskCache()

lix/libstore/nar-info-disk-cache.hh

@@ -32,6 +32,15 @@ public:
     virtual void upsertNarInfo(
         const std::string & uri, const std::string & hashPart,
         std::shared_ptr<const ValidPathInfo> info) = 0;
+
+    virtual void upsertRealisation(
+        const std::string & uri,
+        const Realisation & realisation) = 0;
+    virtual void upsertAbsentRealisation(
+        const std::string & uri,
+        const DrvOutput & id) = 0;
+    virtual std::pair<Outcome, std::shared_ptr<Realisation>> lookupRealisation(
+        const std::string & uri, const DrvOutput & id) = 0;
 };
 
 /**

lix/libstore/optimise-store.cc

@@ -217,8 +217,7 @@ void LocalStore::optimisePath_(Activity * act, OptimiseStats & stats,
        its timestamp back to 0. */
     MakeReadOnly makeReadOnly(mustToggle ? dirOfPath : "");
 
-    Path tempLink = makeTempPath(config().realStoreDir, "/.tmp-link");
-    unlink(tempLink.c_str()); // just in case; ignore errors
+    Path tempLink = makeTempPath(config().realStoreDir + "/");
 
     if (link(linkPath.c_str(), tempLink.c_str()) == -1) {
         if (errno == EMLINK) {

lix/libstore/parsed-derivations.cc

@@ -96,6 +96,8 @@ StringSet ParsedDerivation::getRequiredSystemFeatures() const
     StringSet res;
     for (auto & i : getStringsAttr("requiredSystemFeatures").value_or(Strings()))
         res.insert(i);
+    if (!drv.type().hasKnownOutputPaths())
+        res.insert("ca-derivations");
     return res;
 }