When testing specific internal functionallity while needing things from
the testlib, so far, the tests for the testlib have always been copied
too.
To reduce the amount of additional program required when later making
the env of the pytest_command declarative, and to not test the same
tests a multitude of times (and potentially reaching infinite recursion)
those tests will no longer be copied
Change-Id: I36ec3824a21ed30f9b8ff19948031d1edbf6c76c
this means both the worker protocol and the serve protocol, i.e.
ssh-ng/local connections and legacy ssh connections. now we have
no blocking reads left anywhere in our client store connections.
Change-Id: I2f628d4d2e71ef0a7006918f175192f3f58eea95
`mesonCheckPhase` of course runs `preCheck` and `postCheck` hooks,
not `preInstallCheck`, so this was not doing the right thing. Really
sorry for breaking CI :(
Fixes: e6da29ad6b
Change-Id: I6a6a69642a242e2c8bfe10ac61d3f6756d666548
Snix's nar-bridge[1] stores NARs under a different URL, i.e.
`nar/snix-castore/<hash>.nar` rather than `nar/<filehash>.nar`. Right
now, when copying into such a store via the http binary-cache, we'd end
up with wrong cache entries that point to the wrong NAR URL.
On Hydra, this is a fatal error, i.e. builds that depend on previously
built paths (that were written to the cache before by the queue runner)
would be aborted because of that.
This patch removes the caching since we'd have to re-fetch the narinfo
to do taht and this can also happen the next time, the narinfo is
queried. Also, removes the negative cache entry indicating that the
store-path doesn't exist in the store.
We don't have any coverage for http-stores so far, so I wrote a small
testcase for the "default" case and the nar-bridge case in functional2
since it has a very nice fixture for an HTTP server ready. I'm aware
that there's a CL for a nicer cache server[2], but given I'm down a
pretty deep rabbit hole by playing around with Snix, I decided to not
finish the CL and write something small for the tests in here. This can
be replaced by the fixtures from that CL later on as well.
[1] https://snix.dev/docs/components/overview/#nar-bridge
[2] https://gerrit.lix.systems/c/lix/+/2431/1
Change-Id: I4fcdf47a6bf9c3c8fbeb235eeca7a48914a4d693
shoving a nar dump directly into a framed sink created a bunch of tiny
frames, each of which requires at least two syscalls to read. this can
lead to immense performance loss when using a daemon; we have seen 15%
in benchmarks on main and even more with async code involved ... oops.
Change-Id: I8529506e3de74d92834d1f4ee228dcaf32eb756c
Notably, this adds the Requires.private required for proper static linkage of
Lix libraries. Some minor missing or duplicated dependencies are also fixed
along the way (although some optional dependencies are omitted due to their
size).
Fixes: https://git.lix.systems/lix-project/lix/issues/789
Change-Id: I6a6a696413d538124d9ac75c68f100cc3089284f
A bunch of dependencies were superfluous, while others were missing (in
particular on internal sub-libraries) and things just happened to work because
they leaked in or were explicitly required in other places. Make efforts to fix
them all.
Change-Id: I6a6a69643e35ac4b0b66a72f4e42a2ba5ed52488
Some of the Lix libraries always need to be linked in full due to their
reliance on static initializers. This was achieved internally using link_whole,
but they are still easy to abuse by external users who manually need to
remember passing linker flags such as `--whole-archive` (GNU) or `-force_load`
(Apple), and the obvious way to shove it in pkg-config breaks Meson due to
potentially including a library's flags multiple times, and then deduplicating
only the file names leaving a stale `-force_load` around causing trouble.
Instead we now "prelink" the static libraries, by merging them into one object
file. Since the static linker will always link entire object files, this will
have the same effect as whole-archive linking (except the library won't be
included if it's completely unused, which should not cause trouble since it's
unused after all, and dynamic libraries behave the same way). Unfortunately
Meson's native prelink functionality cannot be used due to missing (non-Apple)
Clang support [1], so write our own one. While not particularly portable, it
should work with Clang which is the only officially supported compiler, as well
as GCC.
[1] https://github.com/mesonbuild/meson/pull/14846
Change-Id: I6a6a6964a82241ce3b0b11fe8397fd451b8027f2
Meson only requires the `modules` option for Boost libraries that should be
linked against [1]. However, we use only header-only portions of the Boost
container library; in fact we only do not run into the disallowedReferences
because the linker drops the unused library. Remove the misleading option.
[1] https://mesonbuild.com/Dependencies.html#boost
Change-Id: I6a6a69648b806bf6bcf784391263f5ee9cd63a0b
This gets us `--print-errorlogs` and `--timeout-multiplier=0` for free,
and also fixes the concatenation logic with `__structuredAttrs`.
Change-Id: I6a6a69643ea7224cb24508505c554143f83ae243
Otherwise we'll get system names like `x86-linux`, which is nonesense.
Also add fixups for 32-bit ARM, and MIPS.
Change-Id: I6bd773bec82dcc54b40bbc018641cd614b272a28
this lets us avoid a bunch of fcntl calls to unset and re-set O_NONBLOCK
on remote store connections. the overhead of these isn't high, but doing
it is still wasteful and a maintenance burden when we have async readers
Change-Id: I900cdca2a16202380c8b6f9b86da7d9b0f1e34ac
the former isn't even int, it's unsigned. the latter is uint64. both
should be explicit about this to avoid problems that we have already
had, such as the FramedSource wire protocol writing using 64 bits to
write frame sizes, but 32 bits to read them. large frames will cause
the reading end to crash with an unnecessary serialization exception
Change-Id: I3c15e911f649eec719d4b1c135dde1b6ba020271
they will not work well with async deserialization and are not used
consistently anyway. just like the serializing operator<< these are
protocol stability hazards: changing the type of a field influences
the wire protocol layout and type constraints, which is not amazing
Change-Id: I54b20a133048f4ca15a9fb0f4d8b94dc78f62d89
this is an equivalent of the regular kj read interface which also takes
a min/max pair. we do not need this very often though, so we'll keep it
as a separate method for now. if we do find we use it more than read we
can still rename read to readSingle and readRange to read. we will see.
Change-Id: Ib04ca146911adae7081cf4b2df097217ea5fe9f8
this should've been a filter from the start. since filter support in the
old IO model is very bad we just move it into the only use of it instead
Change-Id: Ifb9cfecf07587ae1d2d55072ddf505c86c79cc1b
we should not encourage this kind of framing. in the future we will have
to do this on async streams, which we *absolutely* should not encourage.
Change-Id: Ib89e144afb564284db64fc7367cba7fffc18fdaf
the exception no longer actually matters, only that the download stream
is destroyed before the download finishes does. exceptions during drain
calls on the returned stream will cause the stream to be destroyed, but
draining this stream is just a remnant of the old download-to-sink API.
Change-Id: Ic6de40deb2ccff09d77180148afe746f29f55d23
Nixpkgs expects the `builtin.currentSystem` for POWER CPUs to be:
`powerpc[64][le]-linux`
But using `host_machine.cpu_family()` for the CPU part of the system
string on POWER produces this instead:
`ppc[64]-linux`
So evaluating Nixpkgs errors out on:
`error: Unknown CPU type: ppc64`
To fix this, change
`ppc` -> `powerpc`
`ppc64` -> `powerpc64`
and append `le` if `host_machine.endian() == 'little'`.
I can't actually test this on hardware rn due to hitting a kernel bug
on the host system when linking big things[1], but the approach here
is similar to how it was fixed in cppnix[2][3], so it *should* be fine.
[1] https://git.adelielinux.org/adelie/packages/-/issues/1315
[2] https://github.com/NixOS/nix/pull/13514
[3] https://github.com/NixOS/nix/pull/13520
Change-Id: Ib82839cdaf2198bf18b89e82caaa1217f88e11ed
This is a collection of Lix plugins that showcase how to write one for
various usecases.
The first is a mTLS store plugin that enable mTLS cache URIs
(`https+mtls://`).
We enable meson build system support for this plugin but we are not
going to distribute it in the official packaging of Lix, we will
repackage each relevant plugin downstream in Nixpkgs.
These plugins have *NO* guarantee support, they are provided as useful
references and are possibly production-ready if your usecase is simple
enough.
Reference: https://github.com/NixOS/nix/pull/13030 (this change has
resemblances but our APIs are different, the tests harness is mostly
from CppNix).
Change-Id: Ib354271981b35dff6c134b12c4748c3eaf743fcb
Co-authored-by: Jörg Thalheim <joerg@thalheim.io>
Co-authored-by: László Vaskó <1771332+vlaci@users.noreply.github.com>
Signed-off-by: Raito Bezarius <raito@lix.systems>
This is useful to test extended features in Lix.
Change-Id: Idb2416a080329116677809b883950e6c33028a44
Signed-off-by: Raito Bezarius <raito@lix.systems>
we don't need to report progress for every read call. that's way too
much. batching like this greatly reduces CPU usage for copies out of
or into remote buidlers due to likewise greatly reduced log traffic.
Change-Id: I3db2b2ab113fbaadefc69cfde6f977fb0c6cd5ad
Historically, Nix would support copying certificate authorities inside
the sandbox so you could use them.
In addition to that, the primitives consisting of leaking environment
variables via `impureEnvVars` and `extra-sandbox-paths` to render paths
external to the sandbox visible to the builder would also constitute a
mechanism to expose special inodes which should have no influence on the
output result, e.g. interception CAs.
Unfortunately, in nixpkgs, `lib.fetchers.proxyImpureEnvVars` set
`NIX_SSL_CERT_FILE` as an impure environment variable.
A confused user may set `ssl-cert-file` via `NIX_SSL_CERT_FILE` outside the
builder believing that this will set magically the right
`NIX_SSL_CERT_FILE` inside the sandbox, but this is not true.
The combination of impure environment variables and setting `caFile`
creates a weird interaction where `NIX_SSL_CERT_FILE` points to an
"outside the builder's world" inode *AND* `ssl-cert-file` creates
this very same certificate file in /etc/ssl/certs/ca-certificates.crt
without rewriting the environment variable.
This footgun is closed by making these two features mutually
incompatible with a warning and forcibly rewriting the SSL family of
environment variables even if it was set via impure environment
variables.
Users who truly meant to use `impureEnvVars` can obtain the right
behavior by setting `ssl-cert-file` to an empty string and will have to use
`extra-sandbox-paths`.
Users who meant to use `ssl-cert-file` will have everything work
automatically with a warning hinting at nixpkgs *fixing its own bug*,
i.e. passing `NIX_SSL_CERT_FILE` as an impure environment variable and
expecting the Nix interpreter to magically reconcile the diverging
values or expecting the user to actually do the work to render the path
visible available via `extra-sandbox-paths`.
Fixes#885.
Change-Id: I32f8b5ce20fe9b6a911768114c92f95fc886cc07
Signed-off-by: Raito Bezarius <raito@lix.systems>
Sometimes, `bindPath` will detect the source is a symlink and we are not
using the new mount API which support symlinks (kernel ≥ 5.12 IIRC?).
In those instances, we copy the inode to the target.
But some callers may want to follow the symlink in such circumstances,
we add a new default argument to the previous value and let caller
decide for themselves.
Change-Id: I8505b613fc614ce539eb89258fbbb7eaecebe23b
Signed-off-by: Raito Bezarius <raito@lix.systems>
`pathContentsGood` is used to assess the validity of a path as part of
derivation goals *in repair mode*.
When repair is used with a diverted store, i.e. a store where
fsPath(toRealPath(s)) != fsPath(s) for s a store path, this result in
utterly broken behavior because it will attempt to assess the goodness
of the *logical* store locations, most of the time: /nix/store/...
So, if you are repairing your system using a live NixOS ISO. Your ISO
contains a `/nix/store` (assumed to be good) and you repair your system
which is rooted at /mnt and contains its own /nix/store, that is, a Nix
store at /mnt/nix/store.
Performing the following operation `nix-store --verify --repair --store
/mnt` will assess the contents goodness of the ISO's Nix store.
To avoid this, we assess the path existence of the *physical path*, aka
the result of `store.toRealPath` applied to a *logical* store path
string representation and we verify the hash of the *physical path*.
The error messages are not taken care of in this CL as those are purely
cosmetic and helps the user understand what is going on.
Fixes#892.
Change-Id: Ib9e0153cb5683edcf37f1963ebf065ceba5e5dfb
Signed-off-by: Raito Bezarius <raito@lix.systems>
Instead of showing logical Nix store paths, we show the actual physical
location so that the user can stat by copy-pasting these paths.
The bad thing about this change is that certain Nix porcelain only
accept their logical counterparts.
Change-Id: Id0eb45d1bf08a23508dfc2bb694c88155654f585
Signed-off-by: Raito Bezarius <raito@lix.systems>
When a user runs a repair or check sequence, they might be confused of
seeing references to a logical /nix/store path rather than the actual
physical store location.
Change-Id: I042031a6159cdd1054e7e200a220bf6c321d5fb8
Signed-off-by: Raito Bezarius <raito@lix.systems>
jade: I rewrote this PR to be consistent between nix-instantiate and nix
eval. It turns out that nix eval *doesn't* copy to store with `--json`,
whereas nix-instantiate does. Wat.
Closes: https://github.com/lix-project/lix/pull/17
Co-authored-by: tyberiusprime <tyberiusprime@noreply.git.lix.systems>
Change-Id: Id22deec1cee0fed3bd5689567869b70bab26bae5
Originally, libboost_context and dependent dynamic libraries have been copied
into the nix package to remove boost from the closure and consequently reduce
the closure size. Since commit ef0de7c79f we do
not depend on Boost coroutines any more, so these libraries are not needed at
all any more and (somewhat ironically) only increase the package size. Remove
them.
Change-Id: I6a6a6964dc3e0b29dfad8b2b232b428ba3cc653c
Meson conveniently does lets you pass feature objects to `required :`
arguments[1], which is handy
[1]: https://mesonbuild.com/Build-options.html#features
Change-Id: I54194b235a9b3dc207f3f78e0a8c50f957e1fd1f
