sadly this is a visitor-only interface; async generators are not yet a
thing and preliminary benchmarks say that overhead would be too large.
Change-Id: I0460d18eba94441cf3101d46bfffdb69cdda81d1
we could have constructed a nar accessor from an index just as well, but
we don't need any of the advanced accessor features like retrieving file
contents or full path-string based access to the archive. using an index
directly is simpler and faster, although in practice we shouldn't notice
Change-Id: Ic521536cd89e88cbe5aba4f26bf40f0c40d09341
this lives with the NarAccessor (for now) because only the nar accessor
consumes this format in-tree, and produces the same format for a second
type of data source (an FSAccessor instead of a precomputed nar index).
Change-Id: I54cddcf59a4e0501b5cc296c606f063feee5fb3b
we'll use this in NarAccessor to provide actually safe indexing of
archives. NarAccessor currently is not fully correct: it relies on
the parser not buffering anything to produce correct file offsets,
but only the nar implementation itself can reasonably expect that.
Change-Id: I64f300b86d8844b876a3ace723546ea7d7b4628b
this wasn't fully correct to begin with; if restorePath threw an
exception we would not have discarded the nar, but if deletePath
threw an exception we would have. discarding the nar in this way
also lets us not swallow nar parser exceptions when adding a nar
for a valid path, potentially notifying clients of corruption in
their own data sources. ideally we would never read the nar when
we're not interested anyway, but that needs a new wire protocol.
Change-Id: Ie5e10a9bee05b67ec2adb9c4f7c9319a4e644e31
the web of sinks and sources is really convoluted. all we want to do is
to always write the full path dump into narHashSink, also write the nar
(during recursive import) or only file data (during flat import) into a
dedicate non-sha256 hash sink, and maybe check that the path was a file
(when doing flat imports). the graphviz diagram did not improve things.
Change-Id: I2d036358a78963222da3db885cf6971e51369344
this fully decouples the possibly-never-async bits of dumping from the
generation of dump bitstream. having the two separate will allow us to
change store import methods to use async streams, not our sync sources
Change-Id: I9dbd5e30ad3ee380c244b4a3760c11e37db3895f
filtering of paths may call into nixlang code, and in turn may call into
the async runtime. dumps and dump preparations should thus be considered
never-async, but we can't mark them as such yet because async methods do
call them without issue at time of writing. we cannot mark them as such;
only a single function in libexpr may cause such problems. fetchers also
use dumpPath with filters, but those filters never call into async code.
Change-Id: Ica0bd853419e08cd5b0f7926820e3c5a8d0f4688
only a single caller uses flat fetching at all. it still makes sense to
not inline that single caller for now, mostly due to activity reporting
Change-Id: I1a6420868443c3a684deafc7a4f567d4d4b1bd53
Last week I was hunting a production issue at work related to CppNix and
the only hint I had was this warning. While the cause was clearly a
CppNix issue since it was related to the libgit2-backed tarball cache,
the fact that this exception was hidden, made it way harder to discover
this. It would've saved me a lot of time[1] if the error was just
printed out (and thus would've ended up in the logs).
This is what this patch results in:
$ nix-instantiate -I https://notexisting.com --find-file . --option connect-timeout 1
warning: error: unable to download 'https://notexisting.com': Resolving timed out after 1001 milliseconds (curl error code=28); retrying in 265 ms
warning: error: unable to download 'https://notexisting.com/': Connection timed out after 1002 milliseconds (curl error code=28); retrying in 523 ms
warning: error: unable to download 'https://notexisting.com/': Connection timed out after 1001 milliseconds (curl error code=28); retrying in 1324 ms
warning: error: unable to download 'https://notexisting.com/': Connection timed out after 1000 milliseconds (curl error code=28); retrying in 2670 ms
warning:
… while downloading https://notexisting.com to satisfy NIX_PATH lookup (ignoring)
warning: unable to download 'https://notexisting.com/': Connection timed out after 1001 milliseconds (curl error code=28)
/path/to/nix/path/root
The `unable to download` part was hidden before. Granted, this is not the most
interesting showcase given the retries before, but I'd still argue that
this is useful for cases where you don't have a connection timeout (and
this was just the most trivial way to test for me).
[1] Unlucky as I was, I had to mount an older backup and nixos-enter
into the mount to reproduce this.
Change-Id: If2894369fc0b159a307b448e1ce7d9b54a327df6
the two variants compute their store path hashes very differently, and
the flat version ignores the filter passed in by the user entirely. in
a future change we want to move filters out of store methods entirely,
splitting them now will make that task a lot easier when we are ready.
Change-Id: I5f23a26ca08e81923f33adf687056d0d464c2cb8
it's only used once at all, only with SHA256, and the hash it returned
was always ignored. probably a relic of a bygone age, but today is not
the time to clean this up. not before addToStore is rewritten, anyway.
Change-Id: I20e0b5a9bc85ccd05db79bf44f3e972fbd174902
Starting with commit 0dbfa7b26e access would also
be allowed to ancestors of allowed paths. This is (ironically) a significant
purity regression, since several users of the purity checks will themselves
assume that arbitrary descent is allowed. For example, `builtins.readDir` and
`builtins.path` could now refer to the filesystem root, breaking purity
entirely in the latter case by allowing to read arbitrary files. Restore the
previous behaviour of only allowing access to explicitly allowed paths.
Change-Id: Ie64180733ab735da9873255e1ccbf95ba7c9161c
`writeLogsToStderr` uses a static mutex in order to prevent log output
from being interleaved. On macOS, it was possible for a logger in a
non-main thread to call this function after the static mutex was
destructed, leading to strange errors. Deliberately leaking the mutex
prevents the destructor from being called, fixing the issue.
Closes#702
Co-Authored-By: Jade Lovelace <jadel@mercury.com>
Co-Authored-By: eldritch horrors <pennae@lix.systems>
Change-Id: I14c80134cc493972752fad56b7f15fad8e4d5a5b
Hydra used to support aggregate jobs that only succeeded when their
constituents succeed. This is still used by e.g. nixpkgs[1].
Prior art:
* https://git.lix.systems/lix-project/nix-eval-jobs/pulls/17: got ported
into the CppNix implementation[2]
* https://github.com/nix-community/nix-eval-jobs/pull/349: implements
glob expressions for constituents - something we needed at work. This
also restructures the code a bit which is what I re-used here. The
globbing is not part of this patch.
Essentially, the following things happen here (assuming `--constituents`
is set):
* Derivations with `_hydraAggregate = true;` are considered aggregates.
These are not written to stdout when received by a worker, but stored
until the end.
* Constituents can be drv paths or strings (that must be the `attr` of
another job). In that case, the derivation of the aggregate job is
rewritten so that it depends on the drv of the constituent job.
* At the very end the aggregate jobs are also written to stdout.
Additionally, this fixes one bug, the old `hydra-eval-jobs`
implementation had (and we actually hit at work):
Given the leaf jobs `packages.foo` & `packages.bar`, an aggregate job
`aggregate0` with
_hydraAggregate = true;
constituents = [ "packages.bar" "packages.foo" ];
and an aggregate job `aggregate1` with
constituents = [ "aggregate0" ];
then it may happen depending on the order of evaluation that `aggregate1`
depends on the old derivation of `aggregate0` (i.e. the one without
rewritten constituents) and doesn't depend on `packages.foo` and
`packages.bar` because it was rewritten before `aggregate0` was
rewritten.
This is done in here correctly, but topologically sorting the aggregate
jobs before rewriting those.
[1] https://github.com/NixOS/nixpkgs/blob/bba6b37c9d0898867a7d9c38a1b5b77efcfb07b9/nixos/release-combined.nix#L69
[2] https://github.com/nix-community/nix-eval-jobs/pull/340
Change-Id: I5baad5e57336b4985ef8595e903814de83eb01c1
I've always been annoyed that it just silently succeeded without any
feedback, but now with the upcoming improvements on defining variables
this is more necessary than ever.
Change-Id: I565897fa2f97cf6f567d4449dcc8d4ad4eb73fce
- `pos` is not used anymore and can be removed
- `path` and `value` are more correctly placed in their own sub state,
to mirror `InheritState`
Change-Id: Ie7c8a3aea3fe4af0f9c5c15ab5897a59d1b3f7b0
I absolutely want to go back to allowing them, I am strongly of the
opinion that nothing should be special about NUL in a string, but this
will have to wait until at least the GC rewrite, so for now the least I
can do is to disarm this gun.
Change-Id: Id14b6037bc8b274c6c60ad970b1c74d436fb62a7
They are broken beyond repair, and barring a language version revision
this is our only option besides doing nothing about it.
Change-Id: I25fa4f032ca9b5ca67356946bcd889289583592e
macOS uses opendirectory for users and groups, which supports nested
groups and groups with synthesized membership. This means that asking
for a group's users isn't sufficient to test for group membership.
With this change, groups like `@localaccounts` or `@_developer` will
work in `trusted-users` and `allowed-users`.
Fixes https://github.com/NixOS/nix/issues/5885
Change-Id: I3b0783ce7cec303de5aba32c8e5ac0f976112c72
now we can feed the nar parser from something that isn't a Source, such
as a kj async stream. we will not be able to wrap the nar members quite
as nicely in that case since async generators are not a thing yet. once
we need that we can come up with some async generator mechanism though.
Change-Id: If9f5d7865c18e4293aa037b9acd29718e080b3e4
with copyNAR no longer using it it doesn't have to be public any more,
and since parseDump does not need to pass through its input data we'll
also turn restore() into a plain function instead of a wire generator.
Change-Id: Idd3a0270089ff47a7dc8ca96eb03b002fa5ea37e
eldritch horrors