diff --git a/doc/manual/rl-next/remove-xp-fetch-closure-.md b/doc/manual/rl-next/remove-xp-fetch-closure-.md new file mode 100644 index 000000000..3f42614c2 --- /dev/null +++ b/doc/manual/rl-next/remove-xp-fetch-closure-.md @@ -0,0 +1,37 @@ +--- +synopsis: Remove `fetch-closure` experimental feature +issues: [fj#1010] +cls: [4595] +category: "Breaking Changes" +credits: [just1602] +--- + +The `fetch-closure` experimental feature has been removed. + +Outside of allowing the user to import closure from binary cache, +`fetchClosure` also allow you to do the following: + +* rewrite non-CA path to CA +* reject non-CA paths at fetching time +* reject CA paths at fetching time + +Some people are using those mechanism to prevent users from having to build any +package and force going via the declared cache or as a way to use ancient/old +software without paying the evaluation cost of a second nixpkgs. + +Both use cases are somewhat of an antipattern in Nix semantics. If the user +cannot fetch a program directly via the substituter mechanism and fall back to +local build, this is a feature AND a misconfiguration. If the user cannot build +certain derivations because they are too expensive, the build directives should +pass `-j0` or similar. + +As for the second usecase, there's a different way to do it that also allows to +have a way to reproduce the paths that are hardcoded in that file, perform +`import (fetchurl "https://my-cache/${hashparts storepath}.drv")` rather, i.e. +an IFD to a possibly well known name. The backend can generate them on the fly +or once, and possess stable names. + +Finally, as for the non-CA → CA features, Lix removed ca-derivations. +fetchClosure offers ca-derivations-like features which suffers from similar +shortcomings albeit lessened. It only follows that we should rather deprecate +and remove these capabilities. diff --git a/doc/manual/src/contributing/testing.md b/doc/manual/src/contributing/testing.md index 0e2969781..2bc518467 100644 --- a/doc/manual/src/contributing/testing.md +++ b/doc/manual/src/contributing/testing.md @@ -401,7 +401,6 @@ I grepped `lix/` for `get[eE]nv\("` to find the mentions in Lix code. **Expected value**: the path to an executable shell - `PRINT_PATH` - Undocumented. Used by `nix-prefetch-url` as an alternative form of `--print-path`. Why??? -- `_NIX_IN_TEST` - If present with any value, makes `fetchClosure` accept file URLs in addition to HTTP ones. Why is this not `_NIX_FORCE_HTTP`?? Not used anywhere else. - `NIX_ALLOW_EVAL` - Used by eval-cache tests to block evaluation if set to `0`. diff --git a/lix/libexpr/builtins/fetchClosure.md b/lix/libexpr/builtins/fetchClosure.md deleted file mode 100644 index 6fc1ee6fe..000000000 --- a/lix/libexpr/builtins/fetchClosure.md +++ /dev/null @@ -1,72 +0,0 @@ ---- -name: fetchClosure -args: [args] -experimentalFeature: fetch-closure ---- -Fetch a store path [closure](@docroot@/glossary.md#gloss-closure) from a binary cache, and return the store path as a string with context. - -This function can be invoked in three ways, that we will discuss in order of preference. - -**Fetch a content-addressed store path** - -Example: - -```nix -builtins.fetchClosure { - fromStore = "https://cache.nixos.org"; - fromPath = /nix/store/ldbhlwhh39wha58rm61bkiiwm6j7211j-git-2.33.1; -} -``` - -This is the simplest invocation, and it does not require the user of the expression to configure [`trusted-public-keys`](@docroot@/command-ref/conf-file.md#conf-trusted-public-keys) to ensure their authenticity. - -If your store path is [input addressed](@docroot@/glossary.md#gloss-input-addressed-store-object) instead of content addressed, consider the other two invocations. - -**Fetch any store path and rewrite it to a fully content-addressed store path** - -Example: - -```nix -builtins.fetchClosure { - fromStore = "https://cache.nixos.org"; - fromPath = /nix/store/r2jd6ygnmirm2g803mksqqjm4y39yi6i-git-2.33.1; - toPath = /nix/store/ldbhlwhh39wha58rm61bkiiwm6j7211j-git-2.33.1; -} -``` - -This example fetches `/nix/store/r2jd...` from the specified binary cache, -and rewrites it into the content-addressed store path -`/nix/store/ldbh...`. - -Like the previous example, no extra configuration or privileges are required. - -To find out the correct value for `toPath` given a `fromPath`, -use [`nix store make-content-addressed`](@docroot@/command-ref/new-cli/nix3-store-make-content-addressed.md): - -```console -# nix store make-content-addressed --from https://cache.nixos.org /nix/store/r2jd6ygnmirm2g803mksqqjm4y39yi6i-git-2.33.1 -rewrote '/nix/store/r2jd6ygnmirm2g803mksqqjm4y39yi6i-git-2.33.1' to '/nix/store/ldbhlwhh39wha58rm61bkiiwm6j7211j-git-2.33.1' -``` - -Alternatively, set `toPath = ""` and find the correct `toPath` in the error message. - -**Fetch an input-addressed store path as is** - -Example: - -```nix -builtins.fetchClosure { - fromStore = "https://cache.nixos.org"; - fromPath = /nix/store/r2jd6ygnmirm2g803mksqqjm4y39yi6i-git-2.33.1; - inputAddressed = true; -} -``` - -It is possible to fetch an [input-addressed store path](@docroot@/glossary.md#gloss-input-addressed-store-object) and return it as is. -However, this is the least preferred way of invoking `fetchClosure`, because it requires that the input-addressed paths are trusted by the Lix configuration. - -**`builtins.storePath`** - -`fetchClosure` is similar to [`builtins.storePath`](#builtins-storePath) in that it allows you to use a previously built store path in a Nix expression. -However, `fetchClosure` is more reproducible because it specifies a binary cache from which the path can be fetched. -Also, using content-addressed store paths does not require users to configure [`trusted-public-keys`](@docroot@/command-ref/conf-file.md#conf-trusted-public-keys) to ensure their authenticity. diff --git a/lix/libexpr/builtins/storePath.md b/lix/libexpr/builtins/storePath.md index 0ef025cf3..82363ed56 100644 --- a/lix/libexpr/builtins/storePath.md +++ b/lix/libexpr/builtins/storePath.md @@ -13,5 +13,3 @@ in a new path (e.g. `/nix/store/ld01dnzc…-source-source`). Not available in [pure evaluation mode](@docroot@/command-ref/conf-file.md#conf-pure-eval). Lix may change this, tracking issue: - -See also [`builtins.fetchClosure`](#builtins-fetchClosure). diff --git a/lix/libexpr/extra-primops.hh b/lix/libexpr/extra-primops.hh index 745acbc94..8743bb6e6 100644 --- a/lix/libexpr/extra-primops.hh +++ b/lix/libexpr/extra-primops.hh @@ -9,7 +9,6 @@ class EvalState; struct Value; void prim_addDrvOutputDependencies(EvalState & state, Value * * args, Value & v); -void prim_fetchClosure(EvalState & state, Value * * args, Value & v); void prim_fetchTree(EvalState & state, Value * * args, Value & v); void prim_fetchGit(EvalState & state, Value * * args, Value & v); void prim_fetchTarball(EvalState & state, Value * * args, Value & v); diff --git a/lix/libexpr/meson.build b/lix/libexpr/meson.build index 199b12de4..ea1762825 100644 --- a/lix/libexpr/meson.build +++ b/lix/libexpr/meson.build @@ -71,7 +71,6 @@ builtin_definitions = files( 'builtins/div.md', 'builtins/elem.md', 'builtins/elemAt.md', - 'builtins/fetchClosure.md', 'builtins/fetchGit.md', 'builtins/fetchTarball.md', 'builtins/fetchTree.md', @@ -219,7 +218,6 @@ libexpr_sources = files( 'parser/parser.cc', 'primops.cc', 'primops/context.cc', - 'primops/fetchClosure.cc', 'primops/fetchMercurial.cc', 'primops/fetchTree.cc', 'primops/fromTOML.cc', diff --git a/lix/libexpr/primops/fetchClosure.cc b/lix/libexpr/primops/fetchClosure.cc deleted file mode 100644 index 36d6094f4..000000000 --- a/lix/libexpr/primops/fetchClosure.cc +++ /dev/null @@ -1,201 +0,0 @@ -#include "lix/libexpr/eval.hh" -#include "lix/libexpr/extra-primops.hh" -#include "lix/libstore/store-api.hh" -#include "lix/libstore/make-content-addressed.hh" -#include "lix/libutil/async.hh" -#include "lix/libutil/url.hh" - -namespace nix { - -/** - * Handler for the content addressed case. - * - * @param state Evaluator state and store to write to. - * @param fromStore Store containing the path to rewrite. - * @param fromPath Source path to be rewritten. - * @param toPathMaybe Path to write the rewritten path to. If empty, the error shows the actual path. - * @param v Return `Value` - */ -static void runFetchClosureWithRewrite(EvalState & state, Store & fromStore, const StorePath & fromPath, const std::optional & toPathMaybe, Value &v) { - - // establish toPath or throw - - if (!toPathMaybe || !state.aio.blockOn(state.ctx.store->isValidPath(*toPathMaybe))) { - auto rewrittenPath = - state.aio.blockOn(makeContentAddressed(fromStore, *state.ctx.store, fromPath)); - if (toPathMaybe && *toPathMaybe != rewrittenPath) - throw Error({ - .msg = HintFmt("rewriting '%s' to content-addressed form yielded '%s', while '%s' was expected", - state.ctx.store->printStorePath(fromPath), - state.ctx.store->printStorePath(rewrittenPath), - state.ctx.store->printStorePath(*toPathMaybe)), - }); - if (!toPathMaybe) - throw Error({ - .msg = HintFmt( - "rewriting '%s' to content-addressed form yielded '%s'\n" - "Use this value for the 'toPath' attribute passed to 'fetchClosure'", - state.ctx.store->printStorePath(fromPath), - state.ctx.store->printStorePath(rewrittenPath)), - }); - } - - auto toPath = *toPathMaybe; - - // check and return - - auto resultInfo = state.aio.blockOn(state.ctx.store->queryPathInfo(toPath)); - - if (!resultInfo->isContentAddressed(*state.ctx.store)) { - // We don't perform the rewriting when outPath already exists, as an optimisation. - // However, we can quickly detect a mistake if the toPath is input addressed. - throw Error({ - .msg = HintFmt( - "The 'toPath' value '%s' is input-addressed, so it can't possibly be the result of rewriting to a content-addressed path.\n\n" - "Set 'toPath' to an empty string to make Lix report the correct content-addressed path.", - state.ctx.store->printStorePath(toPath)), - }); - } - - state.ctx.paths.mkStorePathString(toPath, v); -} - -/** - * Fetch the closure and make sure it's content addressed. - */ -static void runFetchClosureWithContentAddressedPath(EvalState & state, Store & fromStore, const StorePath & fromPath, Value & v) { - - if (!state.aio.blockOn(state.ctx.store->isValidPath(fromPath))) - state.aio.blockOn(copyClosure(fromStore, *state.ctx.store, RealisedPath::Set { fromPath })); - - auto info = state.aio.blockOn(state.ctx.store->queryPathInfo(fromPath)); - - if (!info->isContentAddressed(*state.ctx.store)) { - throw Error({ - .msg = HintFmt( - "The 'fromPath' value '%s' is input-addressed, but 'inputAddressed' is set to 'false' (default).\n\n" - "If you do intend to fetch an input-addressed store path, add\n\n" - " inputAddressed = true;\n\n" - "to the 'fetchClosure' arguments.\n\n" - "Note that to ensure authenticity input-addressed store paths, users must configure a trusted binary cache public key on their systems. This is not needed for content-addressed paths.", - state.ctx.store->printStorePath(fromPath)), - }); - } - - state.ctx.paths.mkStorePathString(fromPath, v); -} - -/** - * Fetch the closure and make sure it's input addressed. - */ -static void runFetchClosureWithInputAddressedPath(EvalState & state, Store & fromStore, const StorePath & fromPath, Value & v) { - - if (!state.aio.blockOn(state.ctx.store->isValidPath(fromPath))) - state.aio.blockOn(copyClosure(fromStore, *state.ctx.store, RealisedPath::Set { fromPath })); - - auto info = state.aio.blockOn(state.ctx.store->queryPathInfo(fromPath)); - - if (info->isContentAddressed(*state.ctx.store)) { - throw Error({ - .msg = HintFmt( - "The store object referred to by 'fromPath' at '%s' is not input-addressed, but 'inputAddressed' is set to 'true'.\n\n" - "Remove the 'inputAddressed' attribute (it defaults to 'false') to expect 'fromPath' to be content-addressed", - state.ctx.store->printStorePath(fromPath)), - }); - } - - state.ctx.paths.mkStorePathString(fromPath, v); -} - -typedef std::optional StorePathOrGap; - -void prim_fetchClosure(EvalState & state, Value * * args, Value & v) -{ - state.forceAttrs(*args[0], noPos, "while evaluating the argument passed to builtins.fetchClosure"); - - std::optional fromStoreUrl; - std::optional fromPath; - std::optional toPath; - std::optional inputAddressedMaybe; - - for (auto & attr : *args[0]->attrs()) { - const auto & attrName = state.ctx.symbols[attr.name]; - auto attrHint = [&]() -> std::string { - return "while evaluating the '" + attrName + "' attribute passed to builtins.fetchClosure"; - }; - - if (attrName == "fromPath") { - NixStringContext context; - fromPath = state.coerceToStorePath(attr.pos, attr.value, context, attrHint()); - } - - else if (attrName == "toPath") { - state.forceValue(attr.value, attr.pos); - bool isEmptyString = attr.value.type() == nString && attr.value.str().empty(); - if (isEmptyString) { - toPath = StorePathOrGap {}; - } - else { - NixStringContext context; - toPath = state.coerceToStorePath(attr.pos, attr.value, context, attrHint()); - } - } - - else if (attrName == "fromStore") - fromStoreUrl = state.forceStringNoCtx(attr.value, attr.pos, attrHint()); - - else if (attrName == "inputAddressed") - inputAddressedMaybe = state.forceBool(attr.value, attr.pos, attrHint()); - - else - throw Error({ - .msg = HintFmt("attribute '%s' isn't supported in call to 'fetchClosure'", attrName), - }); - } - - if (!fromPath) - throw Error({ - .msg = HintFmt("attribute '%s' is missing in call to 'fetchClosure'", "fromPath"), - }); - - bool inputAddressed = inputAddressedMaybe.value_or(false); - - if (inputAddressed) { - if (toPath) - throw Error({ - .msg = HintFmt("attribute '%s' is set to true, but '%s' is also set. Please remove one of them", - "inputAddressed", - "toPath"), - }); - } - - if (!fromStoreUrl) - throw Error({ - .msg = HintFmt("attribute '%s' is missing in call to 'fetchClosure'", "fromStore"), - }); - - auto parsedURL = parseURL(*fromStoreUrl); - - if (parsedURL.scheme != "http" && - parsedURL.scheme != "https" && - !(getEnv("_NIX_IN_TEST").has_value() && parsedURL.scheme == "file")) - throw Error({ - .msg = HintFmt("'fetchClosure' only supports http:// and https:// stores"), - }); - - if (!parsedURL.query.empty()) - throw Error({ - .msg = HintFmt("'fetchClosure' does not support URL query parameters (in '%s')", *fromStoreUrl), - }); - - auto fromStore = state.aio.blockOn(openStore(parsedURL.to_string())); - - if (toPath) - runFetchClosureWithRewrite(state, *fromStore, *fromPath, *toPath, v); - else if (inputAddressed) - runFetchClosureWithInputAddressedPath(state, *fromStore, *fromPath, v); - else - runFetchClosureWithContentAddressedPath(state, *fromStore, *fromPath, v); -} - -} diff --git a/lix/libutil/experimental-features/fetch-closure.md b/lix/libutil/experimental-features/fetch-closure.md deleted file mode 100644 index 45692f030..000000000 --- a/lix/libutil/experimental-features/fetch-closure.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -name: fetch-closure -internalName: FetchClosure ---- -Enable the use of the [`fetchClosure`](@docroot@/language/builtins.md#builtins-fetchClosure) built-in function in the Nix language. diff --git a/lix/libutil/meson.build b/lix/libutil/meson.build index c1f4d82d5..ec7861154 100644 --- a/lix/libutil/meson.build +++ b/lix/libutil/meson.build @@ -158,7 +158,6 @@ experimental_feature_definitions = files( 'experimental-features/cgroups.md', 'experimental-features/coerce-integers.md', 'experimental-features/daemon-trust-override.md', - 'experimental-features/fetch-closure.md', 'experimental-features/flake-self-attrs.md', 'experimental-features/flakes.md', 'experimental-features/lix-custom-sub-commands.md', diff --git a/lix/nix/main.cc b/lix/nix/main.cc index d9bc1b06e..469c2ee88 100644 --- a/lix/nix/main.cc +++ b/lix/nix/main.cc @@ -521,8 +521,7 @@ int mainWrapped(AsyncIoRoot & aio, int argc, char ** argv) if (argc == 2 && std::string(argv[1]) == "__dump-language") { experimentalFeatureSettings.experimentalFeatures.override(ExperimentalFeatures{} - | Xp::Flakes - | Xp::FetchClosure); + | Xp::Flakes); evalSettings.pureEval.override(false); Evaluator state(aio, {}, aio.blockOn(openStore("dummy://"))); auto res = JSON::object(); diff --git a/tests/functional/fetchClosure.sh b/tests/functional/fetchClosure.sh deleted file mode 100644 index 934d28506..000000000 --- a/tests/functional/fetchClosure.sh +++ /dev/null @@ -1,143 +0,0 @@ -source common.sh - -enableFeatures "fetch-closure" - -clearStore -clearCacheCache - -# Initialize binary cache. -nonCaPath=$(nix build --json --file ./dependencies.nix --no-link | jq -r .[].outputs.out) -caPath=$(nix store make-content-addressed --json $nonCaPath | jq -r '.rewrites | map(.) | .[]') -nix copy --to file://$cacheDir $nonCaPath - -# Test basic fetchClosure rewriting from non-CA to CA. -clearStore - -[ ! -e $nonCaPath ] -[ ! -e $caPath ] - -[[ $(nix eval -v --raw --expr " - builtins.fetchClosure { - fromStore = \"file://$cacheDir\"; - fromPath = $nonCaPath; - toPath = $caPath; - } -") = $caPath ]] - -[ ! -e $nonCaPath ] -[ -e $caPath ] - -clearStore - -# The daemon will reject input addressed paths unless configured to trust the -# cache key or the user. This behavior should be covered by another test, so we -# skip this part when using the daemon. -if [[ "$NIX_REMOTE" != "daemon" ]]; then - - # If we want to return a non-CA path, we have to be explicit about it. - expectStderr 1 nix eval --raw --no-require-sigs --expr " - builtins.fetchClosure { - fromStore = \"file://$cacheDir\"; - fromPath = $nonCaPath; - } - " | grepQuiet -E "The .fromPath. value .* is input-addressed, but .inputAddressed. is set to .false." - - # TODO: Should the closure be rejected, despite single user mode? - # [ ! -e $nonCaPath ] - - [ ! -e $caPath ] - - # We can use non-CA paths when we ask explicitly. - [[ $(nix eval --raw --no-require-sigs --expr " - builtins.fetchClosure { - fromStore = \"file://$cacheDir\"; - fromPath = $nonCaPath; - inputAddressed = true; - } - ") = $nonCaPath ]] - - [ -e $nonCaPath ] - [ ! -e $caPath ] - - -fi - -[ ! -e $caPath ] - -# 'toPath' set to empty string should fail but print the expected path. -expectStderr 1 nix eval -v --json --expr " - builtins.fetchClosure { - fromStore = \"file://$cacheDir\"; - fromPath = $nonCaPath; - toPath = \"\"; - } -" | grep "error: rewriting.*$nonCaPath.*yielded.*$caPath" - -# If fromPath is CA, then toPath isn't needed. -nix copy --to file://$cacheDir $caPath - -clearStore - -[ ! -e $caPath ] - -[[ $(nix eval -v --raw --expr " - builtins.fetchClosure { - fromStore = \"file://$cacheDir\"; - fromPath = $caPath; - } -") = $caPath ]] - -[ -e $caPath ] - -# Check that URL query parameters aren't allowed. -clearStore -narCache=$TEST_ROOT/nar-cache -rm -rf $narCache -(! nix eval -v --raw --expr " - builtins.fetchClosure { - fromStore = \"file://$cacheDir?local-nar-cache=$narCache\"; - fromPath = $caPath; - } -") -(! [ -e $narCache ]) - -# If toPath is specified but wrong, we check it (only) when the path is missing. -clearStore - -badPath=$(echo $caPath | sed -e 's!/store/................................-!/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-!') - -[ ! -e $badPath ] - -expectStderr 1 nix eval -v --raw --expr " - builtins.fetchClosure { - fromStore = \"file://$cacheDir\"; - fromPath = $nonCaPath; - toPath = $badPath; - } -" | grep "error: rewriting.*$nonCaPath.*yielded.*$caPath.*while.*$badPath.*was expected" - -[ ! -e $badPath ] - -# We only check it when missing, as a performance optimization similar to what we do for fixed output derivations. So if it's already there, we don't check it. -# It would be nice for this to fail, but checking it would be too(?) slow. -[ -e $caPath ] - -[[ $(nix eval -v --raw --expr " - builtins.fetchClosure { - fromStore = \"file://$cacheDir\"; - fromPath = $badPath; - toPath = $caPath; - } -") = $caPath ]] - - -# However, if the output address is unexpected, we can report it - - -expectStderr 1 nix eval -v --raw --expr " - builtins.fetchClosure { - fromStore = \"file://$cacheDir\"; - fromPath = $caPath; - inputAddressed = true; - } -" | grepQuiet 'error.*The store object referred to by.*fromPath.* at .* is not input-addressed, but .*inputAddressed.* is set to .*true.*' diff --git a/tests/functional/meson.build b/tests/functional/meson.build index c514bd9e5..4e7f79bea 100644 --- a/tests/functional/meson.build +++ b/tests/functional/meson.build @@ -151,7 +151,6 @@ functional_tests_scripts = [ 'nix-profile.sh', 'suggestions.sh', 'store-ping.sh', - 'fetchClosure.sh', 'completions.sh', 'flakes/show.sh', 'path-from-hash-part.sh',