JSON has only one numeric type, with vague semantics. [RFC 8259] says:
> This specification allows implementations to set limits on the range
> and precision of numbers accepted. Since software that implements
> IEEE 754 binary64 (double precision) numbers [IEEE754] is generally
> available and widely used, good interoperability can be achieved by
> implementations that expect no more precision or range than these
> provide, in the sense that implementations will approximate JSON
> numbers within the expected precision. A JSON number such as 1E400
> or 3.141592653589793238462643383279 may indicate potential
> interoperability problems, since it suggests that the software that
> created it expects receiving software to have greater capabilities
> for numeric magnitude and precision than is widely available.
>
> Note that when such software is used, numbers that are integers and
> are in the range [-(2**53)+1, (2**53)-1] are interoperable in the
> sense that implementations will agree exactly on their numeric
> values.
[RFC 8259]: <https://www.rfc-editor.org/rfc/rfc8259.html#section-6>
Floating‐point numbers are annoying to deal with in Nix, so it
optimistically parses integer‐looking literals as Nix‐language
integers where possible. Nixpkgs relies on this behaviour, as it backs
its `lib.toInt` family of functions with `builtins.fromJSON` in lieu
of a real integer‐parsing built‐in, and treats floating‐point
outputs as an error. Therefore, dealing with integer‐looking
JSON number literals that are outside the interoperable range is
unavoidable.
However, this raises the question of how literals that look
like integers, but exceed the range of a Nix‐language integer,
should be handled. The JSON library we use attempts to represent
integer‐looking literals as a unsigned or signed C++ integer type
before falling back to floating‐point numbers.
This means that we were parsing literals below −2⁶³ as
floating‐point numbers, while rejecting ones above (2⁶³ + 1)
with an error. This was done to avoid the C++ undefined behaviour in
the previous code path, but is hard to justify. This change causes
them to both be parsed as floating‐point numbers.
The alternative would be to reject the signed case too. However,
I believe that is less consistent with how JSON handles its single
numeric type, less interoperable with JSON documents found in the wild,
and that it is preferable to avoid the function being needlessly
partial in this case. It does mean that round‐tripping is less
lossless than before, but extreme floating‐point values already
caused these kinds of issues, and from JSON’s point of view that’s
exactly what these literals represent. Only numeric values outside the
RFC’s suggested interoperable range will have any round‐tripping
issues, and we continue to guarantee the behaviour on all values that
can be represented as Nix‐language integers.
Change-Id: I6a6a696412383e8a2cc160397716cb7f7bc7a2d4
So far, the environment used by `command` was completely leaky and the
one used by `nix` was very leaky despite it trying to be a "hermetic"
environment.
This commit moves the hermaticity to `command` and changes its
implementation to be not leak anything.
To achieve this, the following changes were also nessecary:
- the `files` and `snapshot` fixture now use the folder `test-home`
within the tmp_path directory by default, as the `HOME` environment
variable is set to there. (extraction not possible due to dependencies
of command etc also using this directory)
Fixes: #847, #848
Change-Id: I55f86ee0e1615e73fcf442ee2f28f3b89893bbb4
Current test suite doesn't cover the subsecond formatting at
all and toml11 is quite finicky with that. We should at the very
least test its behavior to avoid silent breakages on updates.
(cherry picked from commit 7ed0229d1abd4414144c7af396842462ce6fc1eb)
Upstream-PR: https://github.com/NixOS/nix/pull/13741
Change-Id: I6a6a696433b168072d6ad2585dce8a3c10ccbc39
use a thread pool and allow only buffered sources for performance. we
may want to use this code path unconditionally due to gc interactions
of fibers in circumstances we should not even be able to trigger, but
if that becomes important we will have a useful implementation ready.
Change-Id: Ib4e1531fe920847d8e30a42e8df393ace549f52e
We need capnproto-lix to be provided by callPackage, otherwise it won't
be spliced and the same (built for the cross host platform) for both
buildInputs (correct) and nativeBuildInputs (incorrect). We thus move it
into the overlay.
A similar problem exists for the lowdown build. We thus use callPackage
to override it as well. This is horrible (especially because it means we
need to pass the enableDarwinSandbox package argument through
ourselves), but at least it builds...
Fixes: https://git.lix.systems/lix-project/lix/issues/939
Change-Id: I802152072d852903401ef701f526195aa99475f2
When testing specific internal functionallity while needing things from
the testlib, so far, the tests for the testlib have always been copied
too.
To reduce the amount of additional program required when later making
the env of the pytest_command declarative, and to not test the same
tests a multitude of times (and potentially reaching infinite recursion)
those tests will no longer be copied
Change-Id: I36ec3824a21ed30f9b8ff19948031d1edbf6c76c
this means both the worker protocol and the serve protocol, i.e.
ssh-ng/local connections and legacy ssh connections. now we have
no blocking reads left anywhere in our client store connections.
Change-Id: I2f628d4d2e71ef0a7006918f175192f3f58eea95
`mesonCheckPhase` of course runs `preCheck` and `postCheck` hooks,
not `preInstallCheck`, so this was not doing the right thing. Really
sorry for breaking CI :(
Fixes: e6da29ad6b
Change-Id: I6a6a69642a242e2c8bfe10ac61d3f6756d666548
Snix's nar-bridge[1] stores NARs under a different URL, i.e.
`nar/snix-castore/<hash>.nar` rather than `nar/<filehash>.nar`. Right
now, when copying into such a store via the http binary-cache, we'd end
up with wrong cache entries that point to the wrong NAR URL.
On Hydra, this is a fatal error, i.e. builds that depend on previously
built paths (that were written to the cache before by the queue runner)
would be aborted because of that.
This patch removes the caching since we'd have to re-fetch the narinfo
to do taht and this can also happen the next time, the narinfo is
queried. Also, removes the negative cache entry indicating that the
store-path doesn't exist in the store.
We don't have any coverage for http-stores so far, so I wrote a small
testcase for the "default" case and the nar-bridge case in functional2
since it has a very nice fixture for an HTTP server ready. I'm aware
that there's a CL for a nicer cache server[2], but given I'm down a
pretty deep rabbit hole by playing around with Snix, I decided to not
finish the CL and write something small for the tests in here. This can
be replaced by the fixtures from that CL later on as well.
[1] https://snix.dev/docs/components/overview/#nar-bridge
[2] https://gerrit.lix.systems/c/lix/+/2431/1
Change-Id: I4fcdf47a6bf9c3c8fbeb235eeca7a48914a4d693
shoving a nar dump directly into a framed sink created a bunch of tiny
frames, each of which requires at least two syscalls to read. this can
lead to immense performance loss when using a daemon; we have seen 15%
in benchmarks on main and even more with async code involved ... oops.
Change-Id: I8529506e3de74d92834d1f4ee228dcaf32eb756c
Notably, this adds the Requires.private required for proper static linkage of
Lix libraries. Some minor missing or duplicated dependencies are also fixed
along the way (although some optional dependencies are omitted due to their
size).
Fixes: https://git.lix.systems/lix-project/lix/issues/789
Change-Id: I6a6a696413d538124d9ac75c68f100cc3089284f
A bunch of dependencies were superfluous, while others were missing (in
particular on internal sub-libraries) and things just happened to work because
they leaked in or were explicitly required in other places. Make efforts to fix
them all.
Change-Id: I6a6a69643e35ac4b0b66a72f4e42a2ba5ed52488
Some of the Lix libraries always need to be linked in full due to their
reliance on static initializers. This was achieved internally using link_whole,
but they are still easy to abuse by external users who manually need to
remember passing linker flags such as `--whole-archive` (GNU) or `-force_load`
(Apple), and the obvious way to shove it in pkg-config breaks Meson due to
potentially including a library's flags multiple times, and then deduplicating
only the file names leaving a stale `-force_load` around causing trouble.
Instead we now "prelink" the static libraries, by merging them into one object
file. Since the static linker will always link entire object files, this will
have the same effect as whole-archive linking (except the library won't be
included if it's completely unused, which should not cause trouble since it's
unused after all, and dynamic libraries behave the same way). Unfortunately
Meson's native prelink functionality cannot be used due to missing (non-Apple)
Clang support [1], so write our own one. While not particularly portable, it
should work with Clang which is the only officially supported compiler, as well
as GCC.
[1] https://github.com/mesonbuild/meson/pull/14846
Change-Id: I6a6a6964a82241ce3b0b11fe8397fd451b8027f2
Meson only requires the `modules` option for Boost libraries that should be
linked against [1]. However, we use only header-only portions of the Boost
container library; in fact we only do not run into the disallowedReferences
because the linker drops the unused library. Remove the misleading option.
[1] https://mesonbuild.com/Dependencies.html#boost
Change-Id: I6a6a69648b806bf6bcf784391263f5ee9cd63a0b
This gets us `--print-errorlogs` and `--timeout-multiplier=0` for free,
and also fixes the concatenation logic with `__structuredAttrs`.
Change-Id: I6a6a69643ea7224cb24508505c554143f83ae243
Otherwise we'll get system names like `x86-linux`, which is nonesense.
Also add fixups for 32-bit ARM, and MIPS.
Change-Id: I6bd773bec82dcc54b40bbc018641cd614b272a28
this lets us avoid a bunch of fcntl calls to unset and re-set O_NONBLOCK
on remote store connections. the overhead of these isn't high, but doing
it is still wasteful and a maintenance burden when we have async readers
Change-Id: I900cdca2a16202380c8b6f9b86da7d9b0f1e34ac
the former isn't even int, it's unsigned. the latter is uint64. both
should be explicit about this to avoid problems that we have already
had, such as the FramedSource wire protocol writing using 64 bits to
write frame sizes, but 32 bits to read them. large frames will cause
the reading end to crash with an unnecessary serialization exception
Change-Id: I3c15e911f649eec719d4b1c135dde1b6ba020271
they will not work well with async deserialization and are not used
consistently anyway. just like the serializing operator<< these are
protocol stability hazards: changing the type of a field influences
the wire protocol layout and type constraints, which is not amazing
Change-Id: I54b20a133048f4ca15a9fb0f4d8b94dc78f62d89
this is an equivalent of the regular kj read interface which also takes
a min/max pair. we do not need this very often though, so we'll keep it
as a separate method for now. if we do find we use it more than read we
can still rename read to readSingle and readRange to read. we will see.
Change-Id: Ib04ca146911adae7081cf4b2df097217ea5fe9f8
this should've been a filter from the start. since filter support in the
old IO model is very bad we just move it into the only use of it instead
Change-Id: Ifb9cfecf07587ae1d2d55072ddf505c86c79cc1b
we should not encourage this kind of framing. in the future we will have
to do this on async streams, which we *absolutely* should not encourage.
Change-Id: Ib89e144afb564284db64fc7367cba7fffc18fdaf
the exception no longer actually matters, only that the download stream
is destroyed before the download finishes does. exceptions during drain
calls on the returned stream will cause the stream to be destroyed, but
draining this stream is just a remnant of the old download-to-sink API.
Change-Id: Ic6de40deb2ccff09d77180148afe746f29f55d23
Nixpkgs expects the `builtin.currentSystem` for POWER CPUs to be:
`powerpc[64][le]-linux`
But using `host_machine.cpu_family()` for the CPU part of the system
string on POWER produces this instead:
`ppc[64]-linux`
So evaluating Nixpkgs errors out on:
`error: Unknown CPU type: ppc64`
To fix this, change
`ppc` -> `powerpc`
`ppc64` -> `powerpc64`
and append `le` if `host_machine.endian() == 'little'`.
I can't actually test this on hardware rn due to hitting a kernel bug
on the host system when linking big things[1], but the approach here
is similar to how it was fixed in cppnix[2][3], so it *should* be fine.
[1] https://git.adelielinux.org/adelie/packages/-/issues/1315
[2] https://github.com/NixOS/nix/pull/13514
[3] https://github.com/NixOS/nix/pull/13520
Change-Id: Ib82839cdaf2198bf18b89e82caaa1217f88e11ed
This is a collection of Lix plugins that showcase how to write one for
various usecases.
The first is a mTLS store plugin that enable mTLS cache URIs
(`https+mtls://`).
We enable meson build system support for this plugin but we are not
going to distribute it in the official packaging of Lix, we will
repackage each relevant plugin downstream in Nixpkgs.
These plugins have *NO* guarantee support, they are provided as useful
references and are possibly production-ready if your usecase is simple
enough.
Reference: https://github.com/NixOS/nix/pull/13030 (this change has
resemblances but our APIs are different, the tests harness is mostly
from CppNix).
Change-Id: Ib354271981b35dff6c134b12c4748c3eaf743fcb
Co-authored-by: Jörg Thalheim <joerg@thalheim.io>
Co-authored-by: László Vaskó <1771332+vlaci@users.noreply.github.com>
Signed-off-by: Raito Bezarius <raito@lix.systems>
This is useful to test extended features in Lix.
Change-Id: Idb2416a080329116677809b883950e6c33028a44
Signed-off-by: Raito Bezarius <raito@lix.systems>
we don't need to report progress for every read call. that's way too
much. batching like this greatly reduces CPU usage for copies out of
or into remote buidlers due to likewise greatly reduced log traffic.
Change-Id: I3db2b2ab113fbaadefc69cfde6f977fb0c6cd5ad
Emily