Some code-paths expect e.g. `nixpkgs.rev` to exist.
This change makes sure this always exists. This is done on purpose
within this let-in dance rather than in the default such that
`nix-build release.nix --arg nixpkgs ../nixpkgs` also benefits from this
fix.
Change-Id: I94f1f901823d3df433e3c44ae380bb2f67eef70e
attrpathsSuperset got renamed to preEval in nixpkgs[1] breaking Hydra's
evaluation. Updating nixpkgs to make sure this is consistent now, no
matter if flake-inputs or release.nix is being used.
[1] See commit 19a31658dc39324c9acf8d81198cd7137bdc1e92.
Change-Id: I69ab3f894534b1d89b626ddd4feda8ea502b69b3
I'm not sure why this wasn't an issue before, but the generation of
jobset constituents in Hydra trips over this. The Hydra-convention is to
have the `system`-attribute last anyways, not in-between.
Since this is a single attribute, get rid of it for the sake of
simplicity.
Change-Id: I80ef1b92ed158c8712ee899534ce82130614f338
This is a Hydra jobset definition to be used to warm up a cache with Lix
versions built against several nixpkgs versions. This is only happening
after a bunch of tests have passed.
Change-Id: Id0b0e50c08581f3f377c7c793d2fd83a6007999d
If e.g. i686-linux is not in the list of systems and thus no attribute in `nixpkgsFor`
exists (e.g. on hydra.afnix.fr), don't try to instantiate it.
Change-Id: I118e2af5289ad7bd95ce7f1e6b0728afb6f0321f
...expect for the dev-shell: while it doesn't seem reasonable to me to
even build dev-shells in post-merge CI, this should still remain part of
the pre-merge checks.
Change-Id: I5cb9b4cf53484f884d694770ed94d0dc0862117e
Step II to move the code out of flake.nix. Done in a second intermediate
commit to make it easier to review.
Change-Id: I13fb386f2e9b87b1d28c366e9b7f0d2dbe03315c
That way we can use it in non-flake contexts down the chain.
We don't override the `nixpkgs` anyways since this is a test artifact
that needs an old nixpkgs input, so this isn't a concern here anyways.
Change-Id: If2aa8f7171a445e8321ef449026e206cc5a0fe3c
The upcoming change factors out a lot of code to make it reusable for a
"legacy" Hydra project that we use to build a matrix of (Lix x nixpkgs)
to have a warm cache for everyone.
The Hydra project will not use flakes to have some inputs overridable,
so the change is needed. In non-flake mode we don't have a notion of
"self", so rename it to `lixSrc` such that the next commit is purely
about moving stuff without changing any aspect.
Current stock Hydra doesn't expose `lastModified` though, so right now
that is a cache-miss. There's a pending Hydra PR fixing that[1].
[1] https://git.lix.systems/lix-project/hydra/pulls/82
Change-Id: I59d70b8b874f576c782217e7bca099eda4c7add9
The reference to the flake is used in a single place only to get the
`doc` output of Lix. Remove it so that we can also instantiate the tests
in a legacy Hydra jobset (happening further down the chain).
Change-Id: Ie3eccf42d67c81fcf991342097783e5067d512bc
Noticed on my quest to make Lix buildable against multiple nixpkgs' to
provide pre-warmed caches for the flake on AFNix's Hydra.
`pkgs.systemd` has a new patch policy[1] that only allows patches for
hotfixes and fundamental incompatibilities. As such, the patch that
allows nspawn workloads to start without a `/usr/bin` was removed with the
recommendation to create that stub yourself.
[1] https://github.com/NixOS/nixpkgs/pull/488508
Change-Id: Iaa35ae3b8971867f12daeac5f94958c20a7d9233
That way we can automatically push the static build (compressed as
tarball) into the AFNix S3 bucket and have it updated on each update to
main.
Change-Id: I1002727ab12c0ab6b95c8d37ae4620996607d7b7
We would've had `build-lowdown_2_0.aarch64-linux.aarch64-linux`
otherwise. This is a bit of a problem because my way of generating a
list of constituents for Hydra's per-architecture[1] release job stops
recursing once it encounters a system.
The alternative would be to switch `hydraJobs` to `<system>.x.y` or
switching Hydra to "legacy" jobsets. For the latter I'd prefer to do the
same for Buildkite such that we don't have diverging things to build
depending on pre/post-merge CI.
[1] per-architecute because the trusted AFNix builders don't support all
the architectures we support in Lix and with Flakes there's no way
of parameterizing the list of supported systems.
Change-Id: I33f31260caf86ed5bb0f728770ca3cf1c00adf31
Closes#496
When running
nix-build ../nixpkgs --arg config.allowUnfree true -A hello-unfree
the package `hello-unfree` is now built rather than getting an
eval-error rejecting instantiation. This is because `config.allowUnfree`
is now interpreted as nested attribute-set declaration, similar to how
it's done in `nix repl`.
To prevent sudden breakage, this behavior was carefully deprecated with
Nix throwing an error if the identifier for `--arg` is not a pure
identifier, but an expression as above.
Any kind of merging is rejected. I.e. doing
nix-build ../nixpkgs --arg config '{cudaSupport = true;}' --arg config.allowUnfree true
is prohibited. That way we don't have to think about merge semantics for
cases like this (or even worse `--arg config 'rec { ... }'`). Another
nice side-effect of this is that we don't need to create an EvalState to
force the values and implement merging.
Change-Id: I8b560883a4468a3f32f915764b08f5fdd8fe71bb
While it doesn't make sense to have `foo..bar`, the attribute-path
`foo."".bar` is valid and shouldn't throw.
Change-Id: Ifcddaad6233c6ba8f17cb5c953c2101d276dfeb6
This is a little more elegant and easier to reason about than prefixing
strings with whatever type the rest of the string is.
Change-Id: I7769535303dcb9f67b79e89bef162beec990e2a0
This job is used to indicate that all relevant Hydra jobs of an
architecture have built. The idea is to build some CD mechanism on
AFNix's Hydra to e.g. auto-update the nightly manual.
See https://hydra.afnix.fr/jobset/lix/demo for the current setup.
Change-Id: I41724c5884a068bbe41407ab30f8edf8e4914001
...and add both a static bash and busybox into the sandbox for building:
$ /nix/store/fz43jxs5qfg5vldzk38y2hmrcl58qk18-busybox-1.36.1/bin/sh -c 'declare -A foo'
/nix/store/fz43jxs5qfg5vldzk38y2hmrcl58qk18-busybox-1.36.1/bin/sh: declare: not found
The reason is that busybox's `sh` doesn't know about `declare` which
means it cannot build derivations with `__structuredAttrs = true;`.
Change-Id: Ie3ca431f862fd3e59dc649582704ae739ee5834a
Closes#1064
The culprit here is that `genGraphString` is only invoked with the
store-paths associated with the outputs of the derivation, so when
filling `dependents`, the `graph_data.find(p)` call would return the end
of the iterator when doing this for references to other store-paths.
As a result, the code wrote information behind the graph data-structure
causing a corruption. For me, this resulted in a SIGSEGV most of the
time and in a few cases in an uncaught `map::at`-exception as reported
by Niklas.
This patch changes two aspects of the original implementation:
* When filling `dependents` in the node-set, use `map.at()` instead of
`map.find()->second`. The latter doesn't make any sense and was the
cause of corrupting memory. The `at` would've made it far easier to
spot this in the first place.
* Filter out store-paths that don't belong to a different output of the
derivation when creating `outputGraph`. This variable is used on two
places, `genGraphString` and for topological sorting.
The latter already filters out store-paths from a different drv, so
this is happening now when creating the variable in the first place
such that `genGraphString` never ends up with corrupt data in the
first place. This is the actual bugfix.
Implemented a regression-test for this case to be sure.
Change-Id: Ie02144d89c32b0a776cb1ece0601d0229315ebc3
Otherwise, loading Perl bindings fails early with[1]
undefined symbol: Perl_pad_sv at /nix/store/h2jsb5i4yfblr2f3ac2c7zpmlmj7zjym-perl-5.40.0/lib/perl5/5.40.0/XSLoader.pm line 94
Apparently, it's expected behavior by Perl that this symbol only exists
with `DEBUGGING` being set, hence it's used by the headers. However,
`pkgs.perl` from nixpkgs is apparently not built with `-DDEBUGGING`
causing this error.
Now, `NDEBUG` is manually unset after loading the Perl
headers rather than setting `DEBUGGING` causing the error mentioned
above.
I confirmed that this not only fixes the problem described above, but
running the Hydra tests with
diff --git a/perl/lib/Nix/Store.xs b/perl/lib/Nix/Store.xs
index dfdd64d28..14788266c 100644
--- a/perl/lib/Nix/Store.xs
+++ b/perl/lib/Nix/Store.xs
@@ -27,6 +27,7 @@ using namespace nix;
static AsyncIoRoot & aio()
{
+assert(false);
static thread_local AsyncIoRoot root;
return root;
}
still results in assertion errors.
Finally, added a small install-check that importing `Nix::Store` works
fine.
[1] https://git.lix.systems/lix-project/hydra/issues/69
Change-Id: I58521777eb0f94b766a9813aa4bbd06f9052bd35
Closes#987
The patch adds a flag `--no-instantiate` which only performs evaluation
without instantiating any derivations. Hence, GC root creation is also
skipped. To achieve that, Lix is also put in read-only mode and all
operations that require reading a derivation (e.g. constituents or
listing input derivations) are disabled fallback values are set.
This is a port of an upstream PR[1]. Given the divergence of the
codebases (different restructurings on both ends, no more CA derivations)
I decided to redo large portions from scratch instead of
cherry-picking the patches. Hence, the authorship.
Additionally the clean up of casts down to a local store are removed or
guarded behind an if, as done in the upstream PR.
[1] https://github.com/nix-community/nix-eval-jobs/pull/379
Co-authored-by: Jörg Thalheim <joerg@thalheim.io>
Change-Id: Ib84f44e7799bc5577fd2ee98912458f16ebeab81
Closes#551
This adds a special accessor that falls back to checking if a store-path
exists within a chroot if it's not a valid path. That way,
`genGraphString` can find out which files have which references before
the outputs are registered.
Change-Id: I03c9d508fa3c72e5c262194461a25d71f3f4de15
That way it's possible to inherit from LocalStoreAccessor to implement
special behavior such as an accessor that falls back to the chroot
directory if it can't find a store-path (which is what we'll do in the
next commit).
Change-Id: If689eb3f410e81e629f1d13cc2b48594fecb1001
That way we do now have linear complexity to determine output
references per output within each step of the topological sort.
Instead, this is done before and the topo-sort only filters the output
map for other derivation outputs.
Following up on this, we can re-use `outputGraph` to generate a tree
with references to display which files cause an output reference cycle
if needed.
Change-Id: Ibdd46e7b2e895bfeeebc173046d1297b41998181
If none is given, we fall back to whatever accessor we get from the
store.
To display which paths actually contain the references leading to
e.g. a cycle or triggering a disallowedRequisites error, we'd
potentially have to look into the chroot from the previously finished
build. This behavior should not be part of the local accessor by
default, but part of a "special" accessor. This change allows using such
an accessor for `genGraphString()`.
Now that we inject the accessor from the outside, we have to mock it
anyways in the tests. Hence, this also adds a testcase for the
precise=True case.
Change-Id: I58465fb944776c2b0262ba054d1f296ed2ae3406
The variant has on the left-hand side the topologically sorted vector
and the right-hand side is a pair showing the path and its parent that
represent a cycle in the graph making the sort impossible.
The goal is to implement #551 which needs to throw an error if the
topo-sort fails. However, the error-message is supposed to contain a
graph of store-paths and the API to generate this is inherently async.
Now, catching the exception and re-throwing another one is impossible
since `co_await` is forbidden in `catch`-blocks and adding another
topoSort variant that allows an async `makeError` also seems odd. Hence,
I decided to alter the data-structure in use a bit for this use-case.
One out of two uses of the function are affected after all.
Change-Id: I70a987f470437df8beb3b1cc203ff88701d0aa1b
Closes#932
`connect-timeout` gets replaced by an exponential backoff for the
download timeout where the initial value is controlled by the setting
`initial-connect-timeout`.
Per iteration, the upper limit of the timeout is increased set to
timeout := min(max_connect_timeout, initial_connect_timeout * 2^i)
I decided to move the entire timeout / tracking of attempts into its own
class to not make the filetransfer implementation more complex. Also,
that allows us to write unit-tests for it.
Setting `--download-attempts` to `0` is forbidden now and an exception
will be thrown. For `--offline` we set it to `1`, the behavior is
equivalent to what it was before: whether the max tries were exceeded is
only checked after the first download exception got thrown, i.e. there's
still one attempt being made.
The end-result - with timeouts being caused by a wrongly set proxy -
looks like this:
$ env HTTPS_PROXY=1.1.1.1 nix store ping --store https://example.com
warning: error: unable to download 'https://example.com/nix-cache-info': Connection timed out after 5006 milliseconds (curl error code=28); retrying in 422ms ms (attempt 1/5)
warning: error: unable to download 'https://example.com/nix-cache-info': Connection timed out after 10010 milliseconds (curl error code=28); retrying in 1003ms ms (attempt 2/5)
warning: error: unable to download 'https://example.com/nix-cache-info': Connection timed out after 20020 milliseconds (curl error code=28); retrying in 2018ms ms (attempt 3/5)
warning: error: unable to download 'https://example.com/nix-cache-info': Connection timed out after 40007 milliseconds (curl error code=28); retrying in 4087ms ms (attempt 4/5)
error: unable to download 'https://example.com/nix-cache-info': Connection timed out after 80074 milliseconds (curl error code=28)
Change-Id: I9e8d08d78275bcf60080d663febc9e075243d36b
Snix's nar-bridge[1] stores NARs under a different URL, i.e.
`nar/snix-castore/<hash>.nar` rather than `nar/<filehash>.nar`. Right
now, when copying into such a store via the http binary-cache, we'd end
up with wrong cache entries that point to the wrong NAR URL.
On Hydra, this is a fatal error, i.e. builds that depend on previously
built paths (that were written to the cache before by the queue runner)
would be aborted because of that.
This patch removes the caching since we'd have to re-fetch the narinfo
to do taht and this can also happen the next time, the narinfo is
queried. Also, removes the negative cache entry indicating that the
store-path doesn't exist in the store.
We don't have any coverage for http-stores so far, so I wrote a small
testcase for the "default" case and the nar-bridge case in functional2
since it has a very nice fixture for an HTTP server ready. I'm aware
that there's a CL for a nicer cache server[2], but given I'm down a
pretty deep rabbit hole by playing around with Snix, I decided to not
finish the CL and write something small for the tests in here. This can
be replaced by the fixtures from that CL later on as well.
[1] https://snix.dev/docs/components/overview/#nar-bridge
[2] https://gerrit.lix.systems/c/lix/+/2431/1
Change-Id: I4fcdf47a6bf9c3c8fbeb235eeca7a48914a4d693
That way it's easier to spot whether a node is the "final" node in the
graph which is especially helpful for larger graphs.
Change-Id: I460a699f07f5455917792599f4247ebf8f430d93
Closes#334Closes#626
This is loosely based on upstream PR#10877[1], but heavily changed to
use the graph logic from `nix why-depends`.
`precise` is `false` here since the out-path of the drv being built
isn't registered yet, so the path accessor cannot scan through files
yet.
Example output (from an openssh build with `pcsclite.lib` & `glibc` in
`disallowedRequisites`):
error: output '/nix/store/hr8lmmjmd1jk6s3p5ymggyk4am7n2lmb-openssh-10.0p2' is not allowed to refer to the following paths:
/nix/store/p6r5awz3ywrz66symnrn0xb85xzmcysf-pcsclite-2.3.0-lib
/nix/store/q4wq65gl3r8fy746v9bbwgx4gzn0r2kl-glibc-2.40-66
Shown below are chains that lead to the forbidden path(s).
/nix/store/hr8lmmjmd1jk6s3p5ymggyk4am7n2lmb-openssh-10.0p2
└───/nix/store/ys91ywnwikm14xznwk3cdbprapv2m37z-libfido2-1.16.0
└───/nix/store/p6r5awz3ywrz66symnrn0xb85xzmcysf-pcsclite-2.3.0-lib
/nix/store/hr8lmmjmd1jk6s3p5ymggyk4am7n2lmb-openssh-10.0p2
├───/nix/store/q4wq65gl3r8fy746v9bbwgx4gzn0r2kl-glibc-2.40-66
├───/nix/store/6r4zqb04fq5l5l4zghq76wvcpz7dwd35-linux-pam-1.6.1
│ ├───/nix/store/q4wq65gl3r8fy746v9bbwgx4gzn0r2kl-glibc-2.40-66
[...]
[1] https://github.com/NixOS/nix/pull/10877
Co-authored-by: Robert Hensing <robert@roberthensing.nl>
Change-Id: Ib30024c0d9e45c1160bf0134f7d3ba17dbdeff47
While working on the LocalDerivationGoal code, I realized that this
attribute is only set to `false`/`true` depending on whether
`__structuredAttrs` is `true`/`false`.
Change-Id: I53868cd32cedd7e25cb6233bd93bc01111b56a07
This will be useful for other things as well such as the
disallowedRequisites error in the builder code. Additionally, print the
dependencyPath in the tree bold to spot where a change terminates.
Also implemented some unit-tests for this code.
Change-Id: I8460f3f6c5095d5bfbe390f223bc0252800dca5e
The Node struct should become an implementation detail when moving this
into libstore. A map from a node to its direct references is more
intuitive here.
Change-Id: I9fddce6b398b8bb97834e5586bee72b244885fdd
* Better name for refs
* Use std::optional<T> for distance
Suggested-by: eldritch horrors <pennae@lix.systems>
Change-Id: Ie35c3f2a7ea1a90ce3a9807025d0af9ea73e2403
Instead of logging directly, we now write into a `Strings` set that is
referenced by the caller.
While at it, added a test-case to ensure that self-reference invocations
and --all behave properly.
Change-Id: Ib183ab8e8e90436300e1c870fb3ae8f18730abbf
Step two for #496.
The idea is to allow `nix-build --arg config.allowUnfree true` do the
right thing in the future. However, that's a breaking change since
people might be relying on the ability to set `"config.allowUnfree"` as
attribute-name when auto-calling a Nix-expression.
As a first step, a warning got introduced in 2.92, the next step is now
to reject this usage in 2.94 and await feedback if any so that we can do
the change in a future Lix release.
Change-Id: I6e38fafe26e234204f5bba2a3a4c1da10f80e5f2
This introduces three new things:
* `handleException` which prints out exception details and its stack
trace.
* `handleExceptionWithAsyncTrace` which does the same, but also prints
the async trace if any.
* `LIX_BLOCK_ON` which is awaits a promise and adds an exception trace
if an exception got thrown, similar to `LIX_TRY_AWAIT`. However, this
is not supposed to be used in async functions, but on callsites of
`aio.blockOn()` which is especially useful for Hydra[1].
For `LIX_BLOCK_ON` I had to introduce another function because there's
apparently no way to implement all of it in a macro: on macros with
compound statements the return value must be a trivial expression at the
bottom, i.e. no `try`/`catch`. Now, returning the value from the
`try`-block requires the variable to be defined up-front, but for that
we'd need to know the type-name. Hence the construction with a
template-function being invoked by a macro that injects the current
source-location.
[1] https://git.lix.systems/lix-project/hydra/pulls/52
Change-Id: I56cc92c94f7e8f0be5d4dc5a7d8cb21a92e776ef
It's possible to put a path into the store in pure mode by pretending
it's locked like this:
$ echo 'lalala' > testfile
$ nix eval --expr '(builtins.fetchTree { path = "/home/ma27/testfile"; rev = "0000000000000000000000000000000000000000"; type = "path"; })'
{ lastModified = 1723656303; lastModifiedDate = "20240814172503"; narHash = "sha256-hOMY06A0ohaaCLwnhpZIMoAqi/8kG2vk30NRiqi0dfc="; outPath = "/nix/store/lhfz259iipmv9ky995rml8018jvriynh-source"; rev = "0000000000000000000000000000000000000000"; shortRev = "0000000"; }
$ cat /nix/store/lhfz259iipmv9ky995rml8018jvriynh-source
lalala
There was a fix in CppNix[1], but Puck noted that it's breaking
backwards-compatibility because
> [...] a github fetch with a fully specified rev is no longer considered "locked"
> this is for "purity" reasons, but it breaks any existing flake.nix files
I tried a way smaller correctness fix here:
* Each scheme can denote whether a `rev` is enough to consider itself
locked.
* If a `rev` is given and the scheme is OK with just a `rev` to be
locked, the input is marked as locked.
For `path` this is not the case anymore, i.e. it requires a NAR hash to
be locked down.
[1] https://github.com/nixos/nix/commit/071dd2b3a4e6c0b2106f1b6f14ec26e153d97446
Change-Id: Ibbbf4733c82bcfa1c24dfe099a896d8aaecd81cc
Right now, Lix waits for quite a while five times[1] if a substituter is
down. If the substituter is not reachable for that long, it's probably
down or so slow that it doesn't make sense to download from it. Also,
most people would abort earlier assuming that Lix just behaves weird in
that case.
Reducing the timeout to 5s to make the behavior a little better.
If people live in areas with poor connection where a higher timeout
would be appropriate, I'd argue that they should increase the timeout
rather than requiring everbody else to set it to a non-zero value.
Additionally, communicate how often we're re-trying.
[1] Because of `download-attempts` from `nix.conf(5)`
Change-Id: I8500dce0c8230247dd492e43cc7af4d3b58c4710
Closes#703
The problem boils down to:
* A testcase with a segfaulting worker due to a stack overflow from the
evaluation.
* When the controller encounters that the pipes are down, the worker's
state is checked and a potential infrec is reported if it stopped with
SIGSEGV.
* However, in a few cases, the worker didn't exit yet, but only had closed
pipes (which is how the error is detected). In that case, a different
error message is printed breaking the test.
This bug handling only affects error cases, so this race doesn't happen
on normal shutdown. To mitigate the problem a bit, the controller will
wait for a second and re-check the state of the worker process then.
While this should make the error reporting for end-users a little more
reliable, it's still no guarantee for a non-racy test. Hence, the test
asserts against both cases now and it passes if either the infrec error
or the "worker still running, but pipes are closed" error is returned.
Change-Id: Ifdc7a05bc86b6aecd5d03118d3e3ffc9affe1c5e
Last week I was hunting a production issue at work related to CppNix and
the only hint I had was this warning. While the cause was clearly a
CppNix issue since it was related to the libgit2-backed tarball cache,
the fact that this exception was hidden, made it way harder to discover
this. It would've saved me a lot of time[1] if the error was just
printed out (and thus would've ended up in the logs).
This is what this patch results in:
$ nix-instantiate -I https://notexisting.com --find-file . --option connect-timeout 1
warning: error: unable to download 'https://notexisting.com': Resolving timed out after 1001 milliseconds (curl error code=28); retrying in 265 ms
warning: error: unable to download 'https://notexisting.com/': Connection timed out after 1002 milliseconds (curl error code=28); retrying in 523 ms
warning: error: unable to download 'https://notexisting.com/': Connection timed out after 1001 milliseconds (curl error code=28); retrying in 1324 ms
warning: error: unable to download 'https://notexisting.com/': Connection timed out after 1000 milliseconds (curl error code=28); retrying in 2670 ms
warning:
… while downloading https://notexisting.com to satisfy NIX_PATH lookup (ignoring)
warning: unable to download 'https://notexisting.com/': Connection timed out after 1001 milliseconds (curl error code=28)
/path/to/nix/path/root
The `unable to download` part was hidden before. Granted, this is not the most
interesting showcase given the retries before, but I'd still argue that
this is useful for cases where you don't have a connection timeout (and
this was just the most trivial way to test for me).
[1] Unlucky as I was, I had to mount an older backup and nixos-enter
into the mount to reproduce this.
Change-Id: If2894369fc0b159a307b448e1ce7d9b54a327df6
Hydra used to support aggregate jobs that only succeeded when their
constituents succeed. This is still used by e.g. nixpkgs[1].
Prior art:
* https://git.lix.systems/lix-project/nix-eval-jobs/pulls/17: got ported
into the CppNix implementation[2]
* https://github.com/nix-community/nix-eval-jobs/pull/349: implements
glob expressions for constituents - something we needed at work. This
also restructures the code a bit which is what I re-used here. The
globbing is not part of this patch.
Essentially, the following things happen here (assuming `--constituents`
is set):
* Derivations with `_hydraAggregate = true;` are considered aggregates.
These are not written to stdout when received by a worker, but stored
until the end.
* Constituents can be drv paths or strings (that must be the `attr` of
another job). In that case, the derivation of the aggregate job is
rewritten so that it depends on the drv of the constituent job.
* At the very end the aggregate jobs are also written to stdout.
Additionally, this fixes one bug, the old `hydra-eval-jobs`
implementation had (and we actually hit at work):
Given the leaf jobs `packages.foo` & `packages.bar`, an aggregate job
`aggregate0` with
_hydraAggregate = true;
constituents = [ "packages.bar" "packages.foo" ];
and an aggregate job `aggregate1` with
constituents = [ "aggregate0" ];
then it may happen depending on the order of evaluation that `aggregate1`
depends on the old derivation of `aggregate0` (i.e. the one without
rewritten constituents) and doesn't depend on `packages.foo` and
`packages.bar` because it was rewritten before `aggregate0` was
rewritten.
This is done in here correctly, but topologically sorting the aggregate
jobs before rewriting those.
[1] https://github.com/NixOS/nixpkgs/blob/bba6b37c9d0898867a7d9c38a1b5b77efcfb07b9/nixos/release-combined.nix#L69
[2] https://github.com/nix-community/nix-eval-jobs/pull/340
Change-Id: I5baad5e57336b4985ef8595e903814de83eb01c1
* Point `meta.homepage` to the correct location
* Remove `meta.maintainers` field: the folks listed in there maintain
the upstream package. I decided to remove it since we don't use
`meta.maintainers` in this repository to indicate maintainership of a
component.
Change-Id: Iec7b0eb463bc8f41907909e87ec9280c19dc86f5
There's an `aio.blockOn` missing to resolve the promise. Also added a
test to verify that the GC roots are created as expected.
Change-Id: I88816cc1161372674ded15374b13323f41846453
This exposes the `pytest` suite as a Nix build. Previously, this didn't
exist and upstream just runs this in GHA. As a result, the test was
actually broken since the assumptions about the directory structure have
changed.
To not require network access, the nixpkgs used by the Lix flake is
injected into the test. The evaluation will be done in a chroot store
since the `openStore` call would otherwise need write access to
/nix/store.
Change-Id: Ib4ff5f0319ce3aabb3b7243a076ea169e27e3533
Nixpkgs issues / PRs:
* https://github.com/NixOS/nixpkgs/pull/368091
* https://github.com/NixOS/nixpkgs/issues/369366
This can be triggered with the postgresql_14 derivation from nixpkgs rev
19305d94dacca226ca048b78e6de00f599c65858
(/nix/store/bxp6g57limvwiga61vdlyvhy7i8rp6wd-postgresql-14.15.drv on
x86_64-linux): for reasons unknown to me, only the `man` and `lib` outputs
are cached on cache.nixos.org:
$ nix derivation show /nix/store/bxp6g57limvwiga61vdlyvhy7i8rp6wd-postgresql-14.15.drv | jq '.[].outputs.[].path' -r | xargs nix path-info --store https://cache.nixos.org
warning: The interpretation of store paths arguments ending in `.drv` recently changed. If this command is now failing try again with '/nix/store/bxp6g57limvwiga61vdlyvhy7i8rp6wd-postgresql-14.15.drv^*'
don't know how to build these paths:
/nix/store/m9vb40xxr6gckjzpfxnqcmjqsks2gx03-postgresql-14.15
/nix/store/nm1415wa53iawar9axwxy0an6ximhayn-postgresql-14.15-dev
/nix/store/v9vrvfhiw9gk8hj9895sb15fxvxnyylj-postgresql-14.15-debug
/nix/store/zi12g1p99g2173i8093ixbqkfh9ng87b-postgresql-14.15-doc
/nix/store/3i3fpz0xss9inampf51gp3pkx24ypxpj-postgresql-14.15-man
/nix/store/db8797h2cp4rm1cnsqrf87apkkxwwdff-postgresql-14.15-lib
error: path '/nix/store/m9vb40xxr6gckjzpfxnqcmjqsks2gx03-postgresql-14.15' does not exist in the store
Also, the derivation uses the `outputChecks` feature (and thus `__structuredAttrs`)
to make sure that e.g. the `out` output doesn't reference the `man`
output:
__structuredAttrs = true;
outputs = [ "out" "dev" "doc" "lib" "man" ];
outputChecks.out.disallowedReferences = [ "dev" "doc" "man" ];
With all that in place, the following error was hit on all CppNix / Lix
versions currently supported when trying to build the derivation above:
error: derivation contains an illegal reference specifier 'man'
The following happened here:
* The `man` & `lib` outputs were substituted at some point.
* When register outputs, the reference checks are made.
* `LocalDerivationGoal::checkOutputs` gets a map of all outputs that
were built and are NOT already registered in the store. In the example
above this means `out`, `dev`, `debug` and `doc`.
* `checkOutputs` tries to resolve the `man` output and fails to do so
because it's a store-path that's already registered and thus not part
of the map passed to `checkOutputs`.
Since the map passed to `checkOutputs` is used in various other places
that appear to assume that the paths aren't registered already, I didn't
write the already registered paths into it. Instead, I created a second
map that contains all already registered outputs and pass it as third
argument to `checkOutputs`. If the other lookups fail, this map will be
now checked before the "illegal reference specifier"-error is thrown.
This fixes the problem with `postgresql_14` for me.
Also wrote a small regression test that fails locally without the patch
in place.
Change-Id: Ieacca80c001fcfbebf6f5fe97e25c49d2724c3ff
So I recently saw it the first time in the wild, I liked that you get
interactively asked about the nix.conf settings from the flake, but
there were a few minor things that I'd like to see changed:
* The `(y/N)` was somewhere in the middle of the line. Moved it to
the end. At first I assumed it was a bug because another thread into
my terminal while I was answering the question.
* I had to say no four times for a single flake with two options. So if
you already know you don't want any of the config for _this_ flake, I
found a `No to all` switch that ignores the rest of the nix.conf
settings a little more ergonomic than having to stop the invocation,
looking up the exact wording of `--no-accept-flake-config` and
restarting it. Hence, I added it.
* Added a note where the choices which settings to trust are persisted.
My initial assumption was that this went into `nix.conf` which is not
writable on NixOS, so I said no there as well.
Change-Id: I0a0d9c403f0662df4707697a77f08e6cd003ec6f
A better fix than in 104448e75d, hence a
revert + the fix.
It turns out that this commit has the side-effect that when having e.g.
`StrictHostKeyChecking=accept-new` for a remote builder, the warnings à la
Warning: Permanently added 'builder' (ED25519) to the list of known hosts.
actually end up in the derivation's log whereas hostkey verification
errors don't, but only in the stderr of the `nix-build` invocation
(which was the motivation for the patch).
This change writes the stderr from the build-hook to
* the daemon's stderr, so that the SSH errors appear in the journal
(which was the case before 104448e75d)
* the client's stderr, as a log message
* NOT to the drv log (this is handled via `handleJSONLogMessage`)
I tried to fix the issue for legacy-ssh as well, but failed and
ultimately decided to not bother.
I know that we'll sooner or later replace the entire component, however
this is the part of the patch I have working for a while, so I figured I
might still submit it for the time being.
Change-Id: I21ca1aa0d8ae281d2eacddf26e0aa825272707e5
While debugging something else I observed that latest `main` ignores
`Control-C` on `sudo nix-build`.
After reading through the capnproto docs, it seems as if the promise
must be fulfilled to actually terminate the `promise.wait()` below.
This also applies to scenarios such as stopping the client
(`nix-build`), but the builders on the daemon-side are still running,
i.e. closes#540
Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: I9634d14df4909fc1b65d05654aad0309bcca8a0a
Since fb38459d6e, each `ref` is appended
with `refs/heads` unless it starts with `refs/` already. This regressed
two use-cases that worked fine before:
* Specifying a commit hash as `ref`: now, if `ref` looks like a commit
hash it will be directly passed to `git fetch`.
* Specifying a tag without `refs/tags` as prefix: now, the fetcher prepends
`refs/*` to a ref that doesn't start with `refs/` and doesn't look
like a commit hash. That way, both a branch and a tag specified in
`ref` can be fetched.
The order of preference in git is
* file in `refs/` (e.g. `HEAD`)
* file in `refs/tags/`
* file in `refs/heads` (i.e. a branch)
After fetching `refs/*`, ref is resolved the same way as git does.
Change-Id: Idd49b97cbdc8c6fdc8faa5a48bef3dec25e4ccc3
We don't support GCC anymore for building, so the overlay currently
fails to evaluate with
error: assertion '((stdenv).cc.isClang || lintInsteadOfBuild)' failed
`clangStdenv` seems like a reasonable default now.
Noticed while upgrading Lix for our Hydra fork.
Change-Id: I948a7c03b3e5648fc7c596f96e1b8053a9e7f92f
See https://git.lix.systems/lix-project/lix/issues/496.
The core idea is to be able to do e.g.
nix-instantiate -A some-nonfree-thing --arg config.allowUnfree true
which is currently not possible since `config.allowUnfree` is
interpreted as attribute name with a dot in it.
In order to change that (probably), Jade suggested to find out if there
are any folks out there relying on this behavior.
For such a use-case, it may still be possible to accept strings, i.e.
`--arg '"config.allowUnfree"'.
Change-Id: I986c73619fbd87a95b55e2f0ac03feaed3de2d2d
Closes#460
I managed to trigger the issue by having the following inputs (shortened):
authentik-nix.url = "github:nix-community/authentik-nix";
authentik-nix.inputs.poetry2nix.inputs.nixpkgs.follows = "nixpkgs";
When evaluating this using
nix-eval-jobs --flake .#hydraJobs
I got the following error:
error: cannot update unlocked flake input 'authentik-nix/poetry2nix' in pure mode
The issue we have here is that `authentik-nix/poetry2nix` was written
into the `overrideMap` which caused Nix to assume it's a new input and
tried to refetch it (#460) or errored out in pure mode
(nix-eval-jobs / Hydra).
The testcase unfortunately only involves checking for the output log
and makes sure that something *is* logged on the first fetch so that
the test doesn't rot when the logging changes since I didn't
manage to trigger the error above with the reproducer from #460. In
fact, I only managed to trigger the `cannot update unlocked flake input`
error in this context with `nix-eval-jobs`.
Change-Id: Ifd00091eec9a0067ed4bb3e5765a15d027328807
The original attempt at this introduced a regression; this commit
reverts the revert and fixes the regression.
This reverts commit 3e151d4d77.
Fix to the regression:
flakeref: fix handling of `?dir=` param for flakes in subdirs
As reported in #419[1], accessing a flake in a subdir of a Git
repository fails with the previous commit[2] applied with the error
error: unsupported Git input attribute 'dir'
The problem is that the `dir`-param is inserted into the parsed URL if a
flake is fetched from the subdir of a Git repository. However, for the
fetching part this isn't even needed. The fix is to just pass `subdir`
as second argument to `FlakeRef` (which needs a `basedir` that can be
empty) and leave the parsedURL as-is.
Added a regression test to make sure we don't run into this again.
[1] https://git.lix.systems/lix-project/lix/issues/419
[2] e22172aaf6b6a366cecd3c025590e68fa2b91bcc,
originally 3e151d4d77
Change-Id: I2c72d5a32e406a7ca308e271730bd0af01c5d18b
This comes quite often when the available job slots on all remote
builders are exhausted and this is pretty spammy.
This isn't really an issue, but expected behavior.
A better way to display this is a nom-like approach where all scheduled
builds are shown in a tree and pending builds are being marked as such
IMHO.
Change-Id: I6bc14e6054f84e3eb0768127b490e263d8cdcf89
The original idea was to fix lix#174, but for a user friendly solution,
I figured that we'd need more consistency:
* Invalid query params will cause an error, just like invalid
attributes. This has the following two consequences:
* The `?dir=`-param from flakes will be removed before the URL to be
fetched is passed to libfetchers.
* The tarball fetcher doesn't allow URLs with custom query params
anymore. I think this was questionable anyways given that an
arbitrary set of query params was silently removed from the URL you
wanted to fetch. The correct way is to use an attribute-set
with a key `url` that contains the tarball URL to fetch.
* Same for the git & mercurial fetchers: in that case it doesn't even
matter though: both fetchers added unused query params to the URL
that's passed from the input scheme to the fetcher (`url2` in the code).
It turns out that this was never used since the query parameters were
erased again in `getActualUrl`.
* Validation happens for both attributes and URLs. Previously, a lot of
fetchers validated e.g. refs/revs only when specified in a URL and
the validity of attribute names only in `inputFromAttrs`.
Now, all the validation is done in `inputFromAttrs` and `inputFromURL`
constructs attributes that will be passed to `inputFromAttrs`.
* Accept all attributes as URL query parameters. That also includes
lesser used ones such as `narHash`.
And "output" attributes like `lastModified`: these could be declared
already when declaring inputs as attribute rather than URL. Now the
behavior is at least consistent.
Personally, I think we should differentiate in the future between
"fetched input" (basically the attr-set that ends up in the lock-file)
and "unfetched input" earlier: both inputFrom{Attrs,URL} entrypoints
are probably OK for unfetched inputs, but for locked/fetched inputs
a custom entrypoint should be used. Then, the current entrypoints
wouldn't have to allow these attributes anymore.
Change-Id: I1be1992249f7af8287cfc37891ab505ddaa2e8cd
Basically I'd expect the same behavior as with `nix-build`, i.e.
with `--keep-going` the hash-mismatch error of each failing
fixed-output derivation is shown.
The approach is derived from `Store::buildPaths` (`entry-point.cc`):
instead of throwing the first build-result, check if there are any build
errors and if so, display all of them and throw after that.
Unfortunately, the BuildResult struct doesn't have an `ErrorInfo`
(there's a FIXME for that at least), so I have to construct my own here.
This is a rather cheap bugfix and I decided against touching too many
parts of libstore for that (also I don't know if that's in line with the
ongoing refactoring work).
Closes https://git.lix.systems/lix-project/lix/issues/302
Change-Id: I378ab984fa271e6808c6897c45e0f070eb4c6fac
This includes the update to libseccomp 2.5.5[1], so we don't need to
override it on our own.
[1] https://nixpk.gs/pr-tracker.html?pr=306070
Change-Id: I1fa9c7fcc23e501d75f774745107c6bb086ced70
