libexpr: remove fetchClosure exp feature

fixes #1010

Change-Id: I149cd01bf08655c91dc2d3817954937c6a6a6964
This commit is contained in:
Justin !
2025-11-20 08:45:29 +00:00
parent 2c73f3c492
commit 2a308bca00
12 changed files with 38 additions and 431 deletions
@@ -0,0 +1,37 @@
---
synopsis: Remove `fetch-closure` experimental feature
issues: [fj#1010]
cls: [4595]
category: "Breaking Changes"
credits: [just1602]
---
The `fetch-closure` experimental feature has been removed.
Outside of allowing the user to import closure from binary cache,
`fetchClosure` also allow you to do the following:
* rewrite non-CA path to CA
* reject non-CA paths at fetching time
* reject CA paths at fetching time
Some people are using those mechanism to prevent users from having to build any
package and force going via the declared cache or as a way to use ancient/old
software without paying the evaluation cost of a second nixpkgs.
Both use cases are somewhat of an antipattern in Nix semantics. If the user
cannot fetch a program directly via the substituter mechanism and fall back to
local build, this is a feature AND a misconfiguration. If the user cannot build
certain derivations because they are too expensive, the build directives should
pass `-j0` or similar.
As for the second usecase, there's a different way to do it that also allows to
have a way to reproduce the paths that are hardcoded in that file, perform
`import (fetchurl "https://my-cache/${hashparts storepath}.drv")` rather, i.e.
an IFD to a possibly well known name. The backend can generate them on the fly
or once, and possess stable names.
Finally, as for the non-CA → CA features, Lix removed ca-derivations.
fetchClosure offers ca-derivations-like features which suffers from similar
shortcomings albeit lessened. It only follows that we should rather deprecate
and remove these capabilities.
-1
View File
@@ -401,7 +401,6 @@ I grepped `lix/` for `get[eE]nv\("` to find the mentions in Lix code.
**Expected value**: the path to an executable shell
- `PRINT_PATH` - Undocumented. Used by `nix-prefetch-url` as an alternative form of `--print-path`. Why???
- `_NIX_IN_TEST` - If present with any value, makes `fetchClosure` accept file URLs in addition to HTTP ones. Why is this not `_NIX_FORCE_HTTP`??
Not used anywhere else.
- `NIX_ALLOW_EVAL` - Used by eval-cache tests to block evaluation if set to `0`.
-72
View File
@@ -1,72 +0,0 @@
---
name: fetchClosure
args: [args]
experimentalFeature: fetch-closure
---
Fetch a store path [closure](@docroot@/glossary.md#gloss-closure) from a binary cache, and return the store path as a string with context.
This function can be invoked in three ways, that we will discuss in order of preference.
**Fetch a content-addressed store path**
Example:
```nix
builtins.fetchClosure {
fromStore = "https://cache.nixos.org";
fromPath = /nix/store/ldbhlwhh39wha58rm61bkiiwm6j7211j-git-2.33.1;
}
```
This is the simplest invocation, and it does not require the user of the expression to configure [`trusted-public-keys`](@docroot@/command-ref/conf-file.md#conf-trusted-public-keys) to ensure their authenticity.
If your store path is [input addressed](@docroot@/glossary.md#gloss-input-addressed-store-object) instead of content addressed, consider the other two invocations.
**Fetch any store path and rewrite it to a fully content-addressed store path**
Example:
```nix
builtins.fetchClosure {
fromStore = "https://cache.nixos.org";
fromPath = /nix/store/r2jd6ygnmirm2g803mksqqjm4y39yi6i-git-2.33.1;
toPath = /nix/store/ldbhlwhh39wha58rm61bkiiwm6j7211j-git-2.33.1;
}
```
This example fetches `/nix/store/r2jd...` from the specified binary cache,
and rewrites it into the content-addressed store path
`/nix/store/ldbh...`.
Like the previous example, no extra configuration or privileges are required.
To find out the correct value for `toPath` given a `fromPath`,
use [`nix store make-content-addressed`](@docroot@/command-ref/new-cli/nix3-store-make-content-addressed.md):
```console
# nix store make-content-addressed --from https://cache.nixos.org /nix/store/r2jd6ygnmirm2g803mksqqjm4y39yi6i-git-2.33.1
rewrote '/nix/store/r2jd6ygnmirm2g803mksqqjm4y39yi6i-git-2.33.1' to '/nix/store/ldbhlwhh39wha58rm61bkiiwm6j7211j-git-2.33.1'
```
Alternatively, set `toPath = ""` and find the correct `toPath` in the error message.
**Fetch an input-addressed store path as is**
Example:
```nix
builtins.fetchClosure {
fromStore = "https://cache.nixos.org";
fromPath = /nix/store/r2jd6ygnmirm2g803mksqqjm4y39yi6i-git-2.33.1;
inputAddressed = true;
}
```
It is possible to fetch an [input-addressed store path](@docroot@/glossary.md#gloss-input-addressed-store-object) and return it as is.
However, this is the least preferred way of invoking `fetchClosure`, because it requires that the input-addressed paths are trusted by the Lix configuration.
**`builtins.storePath`**
`fetchClosure` is similar to [`builtins.storePath`](#builtins-storePath) in that it allows you to use a previously built store path in a Nix expression.
However, `fetchClosure` is more reproducible because it specifies a binary cache from which the path can be fetched.
Also, using content-addressed store paths does not require users to configure [`trusted-public-keys`](@docroot@/command-ref/conf-file.md#conf-trusted-public-keys) to ensure their authenticity.
-2
View File
@@ -13,5 +13,3 @@ in a new path (e.g. `/nix/store/ld01dnzc…-source-source`).
Not available in [pure evaluation mode](@docroot@/command-ref/conf-file.md#conf-pure-eval).
Lix may change this, tracking issue: <https://git.lix.systems/lix-project/lix/issues/402>
See also [`builtins.fetchClosure`](#builtins-fetchClosure).
-1
View File
@@ -9,7 +9,6 @@ class EvalState;
struct Value;
void prim_addDrvOutputDependencies(EvalState & state, Value * * args, Value & v);
void prim_fetchClosure(EvalState & state, Value * * args, Value & v);
void prim_fetchTree(EvalState & state, Value * * args, Value & v);
void prim_fetchGit(EvalState & state, Value * * args, Value & v);
void prim_fetchTarball(EvalState & state, Value * * args, Value & v);
-2
View File
@@ -71,7 +71,6 @@ builtin_definitions = files(
'builtins/div.md',
'builtins/elem.md',
'builtins/elemAt.md',
'builtins/fetchClosure.md',
'builtins/fetchGit.md',
'builtins/fetchTarball.md',
'builtins/fetchTree.md',
@@ -219,7 +218,6 @@ libexpr_sources = files(
'parser/parser.cc',
'primops.cc',
'primops/context.cc',
'primops/fetchClosure.cc',
'primops/fetchMercurial.cc',
'primops/fetchTree.cc',
'primops/fromTOML.cc',
-201
View File
@@ -1,201 +0,0 @@
#include "lix/libexpr/eval.hh"
#include "lix/libexpr/extra-primops.hh"
#include "lix/libstore/store-api.hh"
#include "lix/libstore/make-content-addressed.hh"
#include "lix/libutil/async.hh"
#include "lix/libutil/url.hh"
namespace nix {
/**
* Handler for the content addressed case.
*
* @param state Evaluator state and store to write to.
* @param fromStore Store containing the path to rewrite.
* @param fromPath Source path to be rewritten.
* @param toPathMaybe Path to write the rewritten path to. If empty, the error shows the actual path.
* @param v Return `Value`
*/
static void runFetchClosureWithRewrite(EvalState & state, Store & fromStore, const StorePath & fromPath, const std::optional<StorePath> & toPathMaybe, Value &v) {
// establish toPath or throw
if (!toPathMaybe || !state.aio.blockOn(state.ctx.store->isValidPath(*toPathMaybe))) {
auto rewrittenPath =
state.aio.blockOn(makeContentAddressed(fromStore, *state.ctx.store, fromPath));
if (toPathMaybe && *toPathMaybe != rewrittenPath)
throw Error({
.msg = HintFmt("rewriting '%s' to content-addressed form yielded '%s', while '%s' was expected",
state.ctx.store->printStorePath(fromPath),
state.ctx.store->printStorePath(rewrittenPath),
state.ctx.store->printStorePath(*toPathMaybe)),
});
if (!toPathMaybe)
throw Error({
.msg = HintFmt(
"rewriting '%s' to content-addressed form yielded '%s'\n"
"Use this value for the 'toPath' attribute passed to 'fetchClosure'",
state.ctx.store->printStorePath(fromPath),
state.ctx.store->printStorePath(rewrittenPath)),
});
}
auto toPath = *toPathMaybe;
// check and return
auto resultInfo = state.aio.blockOn(state.ctx.store->queryPathInfo(toPath));
if (!resultInfo->isContentAddressed(*state.ctx.store)) {
// We don't perform the rewriting when outPath already exists, as an optimisation.
// However, we can quickly detect a mistake if the toPath is input addressed.
throw Error({
.msg = HintFmt(
"The 'toPath' value '%s' is input-addressed, so it can't possibly be the result of rewriting to a content-addressed path.\n\n"
"Set 'toPath' to an empty string to make Lix report the correct content-addressed path.",
state.ctx.store->printStorePath(toPath)),
});
}
state.ctx.paths.mkStorePathString(toPath, v);
}
/**
* Fetch the closure and make sure it's content addressed.
*/
static void runFetchClosureWithContentAddressedPath(EvalState & state, Store & fromStore, const StorePath & fromPath, Value & v) {
if (!state.aio.blockOn(state.ctx.store->isValidPath(fromPath)))
state.aio.blockOn(copyClosure(fromStore, *state.ctx.store, RealisedPath::Set { fromPath }));
auto info = state.aio.blockOn(state.ctx.store->queryPathInfo(fromPath));
if (!info->isContentAddressed(*state.ctx.store)) {
throw Error({
.msg = HintFmt(
"The 'fromPath' value '%s' is input-addressed, but 'inputAddressed' is set to 'false' (default).\n\n"
"If you do intend to fetch an input-addressed store path, add\n\n"
" inputAddressed = true;\n\n"
"to the 'fetchClosure' arguments.\n\n"
"Note that to ensure authenticity input-addressed store paths, users must configure a trusted binary cache public key on their systems. This is not needed for content-addressed paths.",
state.ctx.store->printStorePath(fromPath)),
});
}
state.ctx.paths.mkStorePathString(fromPath, v);
}
/**
* Fetch the closure and make sure it's input addressed.
*/
static void runFetchClosureWithInputAddressedPath(EvalState & state, Store & fromStore, const StorePath & fromPath, Value & v) {
if (!state.aio.blockOn(state.ctx.store->isValidPath(fromPath)))
state.aio.blockOn(copyClosure(fromStore, *state.ctx.store, RealisedPath::Set { fromPath }));
auto info = state.aio.blockOn(state.ctx.store->queryPathInfo(fromPath));
if (info->isContentAddressed(*state.ctx.store)) {
throw Error({
.msg = HintFmt(
"The store object referred to by 'fromPath' at '%s' is not input-addressed, but 'inputAddressed' is set to 'true'.\n\n"
"Remove the 'inputAddressed' attribute (it defaults to 'false') to expect 'fromPath' to be content-addressed",
state.ctx.store->printStorePath(fromPath)),
});
}
state.ctx.paths.mkStorePathString(fromPath, v);
}
typedef std::optional<StorePath> StorePathOrGap;
void prim_fetchClosure(EvalState & state, Value * * args, Value & v)
{
state.forceAttrs(*args[0], noPos, "while evaluating the argument passed to builtins.fetchClosure");
std::optional<std::string> fromStoreUrl;
std::optional<StorePath> fromPath;
std::optional<StorePathOrGap> toPath;
std::optional<bool> inputAddressedMaybe;
for (auto & attr : *args[0]->attrs()) {
const auto & attrName = state.ctx.symbols[attr.name];
auto attrHint = [&]() -> std::string {
return "while evaluating the '" + attrName + "' attribute passed to builtins.fetchClosure";
};
if (attrName == "fromPath") {
NixStringContext context;
fromPath = state.coerceToStorePath(attr.pos, attr.value, context, attrHint());
}
else if (attrName == "toPath") {
state.forceValue(attr.value, attr.pos);
bool isEmptyString = attr.value.type() == nString && attr.value.str().empty();
if (isEmptyString) {
toPath = StorePathOrGap {};
}
else {
NixStringContext context;
toPath = state.coerceToStorePath(attr.pos, attr.value, context, attrHint());
}
}
else if (attrName == "fromStore")
fromStoreUrl = state.forceStringNoCtx(attr.value, attr.pos, attrHint());
else if (attrName == "inputAddressed")
inputAddressedMaybe = state.forceBool(attr.value, attr.pos, attrHint());
else
throw Error({
.msg = HintFmt("attribute '%s' isn't supported in call to 'fetchClosure'", attrName),
});
}
if (!fromPath)
throw Error({
.msg = HintFmt("attribute '%s' is missing in call to 'fetchClosure'", "fromPath"),
});
bool inputAddressed = inputAddressedMaybe.value_or(false);
if (inputAddressed) {
if (toPath)
throw Error({
.msg = HintFmt("attribute '%s' is set to true, but '%s' is also set. Please remove one of them",
"inputAddressed",
"toPath"),
});
}
if (!fromStoreUrl)
throw Error({
.msg = HintFmt("attribute '%s' is missing in call to 'fetchClosure'", "fromStore"),
});
auto parsedURL = parseURL(*fromStoreUrl);
if (parsedURL.scheme != "http" &&
parsedURL.scheme != "https" &&
!(getEnv("_NIX_IN_TEST").has_value() && parsedURL.scheme == "file"))
throw Error({
.msg = HintFmt("'fetchClosure' only supports http:// and https:// stores"),
});
if (!parsedURL.query.empty())
throw Error({
.msg = HintFmt("'fetchClosure' does not support URL query parameters (in '%s')", *fromStoreUrl),
});
auto fromStore = state.aio.blockOn(openStore(parsedURL.to_string()));
if (toPath)
runFetchClosureWithRewrite(state, *fromStore, *fromPath, *toPath, v);
else if (inputAddressed)
runFetchClosureWithInputAddressedPath(state, *fromStore, *fromPath, v);
else
runFetchClosureWithContentAddressedPath(state, *fromStore, *fromPath, v);
}
}
@@ -1,5 +0,0 @@
---
name: fetch-closure
internalName: FetchClosure
---
Enable the use of the [`fetchClosure`](@docroot@/language/builtins.md#builtins-fetchClosure) built-in function in the Nix language.
-1
View File
@@ -158,7 +158,6 @@ experimental_feature_definitions = files(
'experimental-features/cgroups.md',
'experimental-features/coerce-integers.md',
'experimental-features/daemon-trust-override.md',
'experimental-features/fetch-closure.md',
'experimental-features/flake-self-attrs.md',
'experimental-features/flakes.md',
'experimental-features/lix-custom-sub-commands.md',
+1 -2
View File
@@ -521,8 +521,7 @@ int mainWrapped(AsyncIoRoot & aio, int argc, char ** argv)
if (argc == 2 && std::string(argv[1]) == "__dump-language") {
experimentalFeatureSettings.experimentalFeatures.override(ExperimentalFeatures{}
| Xp::Flakes
| Xp::FetchClosure);
| Xp::Flakes);
evalSettings.pureEval.override(false);
Evaluator state(aio, {}, aio.blockOn(openStore("dummy://")));
auto res = JSON::object();
-143
View File
@@ -1,143 +0,0 @@
source common.sh
enableFeatures "fetch-closure"
clearStore
clearCacheCache
# Initialize binary cache.
nonCaPath=$(nix build --json --file ./dependencies.nix --no-link | jq -r .[].outputs.out)
caPath=$(nix store make-content-addressed --json $nonCaPath | jq -r '.rewrites | map(.) | .[]')
nix copy --to file://$cacheDir $nonCaPath
# Test basic fetchClosure rewriting from non-CA to CA.
clearStore
[ ! -e $nonCaPath ]
[ ! -e $caPath ]
[[ $(nix eval -v --raw --expr "
builtins.fetchClosure {
fromStore = \"file://$cacheDir\";
fromPath = $nonCaPath;
toPath = $caPath;
}
") = $caPath ]]
[ ! -e $nonCaPath ]
[ -e $caPath ]
clearStore
# The daemon will reject input addressed paths unless configured to trust the
# cache key or the user. This behavior should be covered by another test, so we
# skip this part when using the daemon.
if [[ "$NIX_REMOTE" != "daemon" ]]; then
# If we want to return a non-CA path, we have to be explicit about it.
expectStderr 1 nix eval --raw --no-require-sigs --expr "
builtins.fetchClosure {
fromStore = \"file://$cacheDir\";
fromPath = $nonCaPath;
}
" | grepQuiet -E "The .fromPath. value .* is input-addressed, but .inputAddressed. is set to .false."
# TODO: Should the closure be rejected, despite single user mode?
# [ ! -e $nonCaPath ]
[ ! -e $caPath ]
# We can use non-CA paths when we ask explicitly.
[[ $(nix eval --raw --no-require-sigs --expr "
builtins.fetchClosure {
fromStore = \"file://$cacheDir\";
fromPath = $nonCaPath;
inputAddressed = true;
}
") = $nonCaPath ]]
[ -e $nonCaPath ]
[ ! -e $caPath ]
fi
[ ! -e $caPath ]
# 'toPath' set to empty string should fail but print the expected path.
expectStderr 1 nix eval -v --json --expr "
builtins.fetchClosure {
fromStore = \"file://$cacheDir\";
fromPath = $nonCaPath;
toPath = \"\";
}
" | grep "error: rewriting.*$nonCaPath.*yielded.*$caPath"
# If fromPath is CA, then toPath isn't needed.
nix copy --to file://$cacheDir $caPath
clearStore
[ ! -e $caPath ]
[[ $(nix eval -v --raw --expr "
builtins.fetchClosure {
fromStore = \"file://$cacheDir\";
fromPath = $caPath;
}
") = $caPath ]]
[ -e $caPath ]
# Check that URL query parameters aren't allowed.
clearStore
narCache=$TEST_ROOT/nar-cache
rm -rf $narCache
(! nix eval -v --raw --expr "
builtins.fetchClosure {
fromStore = \"file://$cacheDir?local-nar-cache=$narCache\";
fromPath = $caPath;
}
")
(! [ -e $narCache ])
# If toPath is specified but wrong, we check it (only) when the path is missing.
clearStore
badPath=$(echo $caPath | sed -e 's!/store/................................-!/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-!')
[ ! -e $badPath ]
expectStderr 1 nix eval -v --raw --expr "
builtins.fetchClosure {
fromStore = \"file://$cacheDir\";
fromPath = $nonCaPath;
toPath = $badPath;
}
" | grep "error: rewriting.*$nonCaPath.*yielded.*$caPath.*while.*$badPath.*was expected"
[ ! -e $badPath ]
# We only check it when missing, as a performance optimization similar to what we do for fixed output derivations. So if it's already there, we don't check it.
# It would be nice for this to fail, but checking it would be too(?) slow.
[ -e $caPath ]
[[ $(nix eval -v --raw --expr "
builtins.fetchClosure {
fromStore = \"file://$cacheDir\";
fromPath = $badPath;
toPath = $caPath;
}
") = $caPath ]]
# However, if the output address is unexpected, we can report it
expectStderr 1 nix eval -v --raw --expr "
builtins.fetchClosure {
fromStore = \"file://$cacheDir\";
fromPath = $caPath;
inputAddressed = true;
}
" | grepQuiet 'error.*The store object referred to by.*fromPath.* at .* is not input-addressed, but .*inputAddressed.* is set to .*true.*'
-1
View File
@@ -151,7 +151,6 @@ functional_tests_scripts = [
'nix-profile.sh',
'suggestions.sh',
'store-ping.sh',
'fetchClosure.sh',
'completions.sh',
'flakes/show.sh',
'path-from-hash-part.sh',