There is now no risk of race conditions on a system with a functioning
entropy source, and the bespoke prefixes are either redundant to the
default or unnecessary.
Change-Id: I6a6a69648a3b8060333e97269ea8b72499614559
Relax the constraints on keeping the exact same filename format to
provide a more robust source of entropy with a simpler interface
(as previously suggested by eldritch horrors). Using 128 bits of
OS‐provided entropy ensures global uniqueness and allows us to
skip any thought of gracefully handling the case where these files
already exist.
My microbenchmark that repeatedly constructed paths like this and
printed them out showed that this takes about 1.23× the time of
the previous implementation, both taking on the order of a couple
microseconds for one iteration. Since everything that uses it is doing
things more expensive than printing to standard output, the actual
performance delta is likely to be lost in the noise. If it somehow
becomes a bottleneck, it can be optimized without sacrificing the
guarantees by reading from the system RNG only to seed a thread‐local
CSPRNG like [ChaCha8Rand], but I think that’s very unlikely.
We also tweak the recommended way of creating a temporary file inside
a directory in anticipation of later changes, and rename the `suffix`
parameter to `prefix` (it’s a prefix to the random characters and
a suffix to the root, but this way is more consistent).
[ChaCha8Rand]: https://c2sp.org/chacha8rand
Change-Id: I5bd7badf1392243f485935c4a016c1f833cb16d3
The prospective callers of this should probably be doing something
smarter or more abstracted to begin with, but this is useful as an
incremental improvement for call sites with existing `makeTempPath`
logic in the face of filename length limits.
Change-Id: I6a6a6964374f47abbf0ec10aa8d945c4e50a43af
notably this also includes the symbol table because it stores real
strings that are referenced by eval values, and an upcoming change
will make it impossible to share those strings with value strings.
Change-Id: I20a3644db8aa0850efe29630e0b73d424cb2aa56
See [my comment] on the Nix PR to restore the previous behaviour
for why I believe we should remove this for the next release. The PR
should still be backported to stable releases to avoid making breaking
changes to their semantics.
[my comment]: <https://github.com/NixOS/nix/pull/13741#issuecomment-3180851635>
Fixing this across supported Lix versions is required for Nixpkgs to
update toml11, which is a blocker for the CMake 4 update.
Change-Id: I6a6a69642e6b6cb13a9fccc0778e9158b53102d5
This version changes the handling of TOML timestamps, and throws an
error on out‐of‐range integer literals rather than the previous
saturating behaviour, as required by [the TOML v1.0.0 specification]:
> Arbitrary 64-bit signed integers (from −2^63 to 2^63−1) should be
> accepted and handled losslessly. If an integer cannot be represented
> losslessly, an error must be thrown.
[the TOML v1.0.0 specification]: <https://toml.io/en/v1.0.0#integer>
The only known use of this is a questionable Nixpkgs test that I have
proposed [a fix] for.
[a fix]: <https://github.com/NixOS/nixpkgs/pull/433710>
Bumping this ahead of Nixpkgs ensures we can test these cases on
HEAD in advance. I presume that the next Lix major version will be
released after 25.05 goes out of support, so it should be fine to
drop support for the old version of toml11.
The co‐authors of this commit are the contributors to the vendored
package definition from Nixpkgs.
Co-authored-by: Anderson Torres <torres.anderson.85@protonmail.com>
Co-authored-by: Artturin <Artturin@artturin.com>
Co-authored-by: Silvan Mosberger <silvan.mosberger@moduscreate.com>
Change-Id: I6a6a69644a188b6e09eee5c9cf91ddd3c81d24ee
This behaviour is bad and will be fixed, but adding a test for the
present state documents the change.
Change-Id: I6a6a6964b88dc929269ee136804857b3852dfafe
This addresses several changes from toml11 4.0 bump in
nixpkgs [1].
1. Added more regression tests for timestamp formats.
Special attention needs to be paid to the precision
of the subsecond range for local-time. Prior versions select the closest
(upwards) multiple of 3 with a hard cap of 9 digits.
2. Normalize local datetime and offset datetime to always
use the uppercase separator `T`. This is actually the issue
surfaced in [2]. This canonicalization is basically a requirement
by (a certain reading) of rfc3339 section 5.6 [3].
3. If using toml11 >= 4.0 also keep the old behavior wrt
to the number of digits used for subsecond part of the local-time.
[1]: https://www.github.com/NixOS/nixpkgs/pull/331649
[2]: https://www.github.com/NixOS/nix/issues/11441
[3]: https://datatracker.ietf.org/doc/html/rfc3339
(cherry picked from commit dc769d72cb8ad22a0f89768682b5499a9d2b3d8b)
Upstream-PR: https://github.com/NixOS/nix/pull/13741
Change-Id: Iac4fbe5108be79be585e9670fa42dfd11f3c5e89
There's no reason to use a std::function for recursive lambdas
since there are polymorphic lambdas.
(cherry picked from commit a80a5c4dba0d944fab8f5ed57a343869ae96bf16)
Upstream-PR: https://github.com/NixOS/nix/pull/13741
Change-Id: I593bd04597e2ae000374ca1eca4d8928e986c0b5
This looks really weird after the reformat.
(cherry picked from commit df4e55ffc13c413e270af134227115a20a2341ba)
Upstream-PR: https://github.com/NixOS/nix/pull/13741
Change-Id: I8de92d58620cc4545a31d8b7d533d2f1e9f4f233
JSON has only one numeric type, with vague semantics. [RFC 8259] says:
> This specification allows implementations to set limits on the range
> and precision of numbers accepted. Since software that implements
> IEEE 754 binary64 (double precision) numbers [IEEE754] is generally
> available and widely used, good interoperability can be achieved by
> implementations that expect no more precision or range than these
> provide, in the sense that implementations will approximate JSON
> numbers within the expected precision. A JSON number such as 1E400
> or 3.141592653589793238462643383279 may indicate potential
> interoperability problems, since it suggests that the software that
> created it expects receiving software to have greater capabilities
> for numeric magnitude and precision than is widely available.
>
> Note that when such software is used, numbers that are integers and
> are in the range [-(2**53)+1, (2**53)-1] are interoperable in the
> sense that implementations will agree exactly on their numeric
> values.
[RFC 8259]: <https://www.rfc-editor.org/rfc/rfc8259.html#section-6>
Floating‐point numbers are annoying to deal with in Nix, so it
optimistically parses integer‐looking literals as Nix‐language
integers where possible. Nixpkgs relies on this behaviour, as it backs
its `lib.toInt` family of functions with `builtins.fromJSON` in lieu
of a real integer‐parsing built‐in, and treats floating‐point
outputs as an error. Therefore, dealing with integer‐looking
JSON number literals that are outside the interoperable range is
unavoidable.
However, this raises the question of how literals that look
like integers, but exceed the range of a Nix‐language integer,
should be handled. The JSON library we use attempts to represent
integer‐looking literals as a unsigned or signed C++ integer type
before falling back to floating‐point numbers.
This means that we were parsing literals below −2⁶³ as
floating‐point numbers, while rejecting ones above (2⁶³ + 1)
with an error. This was done to avoid the C++ undefined behaviour in
the previous code path, but is hard to justify. This change causes
them to both be parsed as floating‐point numbers.
The alternative would be to reject the signed case too. However,
I believe that is less consistent with how JSON handles its single
numeric type, less interoperable with JSON documents found in the wild,
and that it is preferable to avoid the function being needlessly
partial in this case. It does mean that round‐tripping is less
lossless than before, but extreme floating‐point values already
caused these kinds of issues, and from JSON’s point of view that’s
exactly what these literals represent. Only numeric values outside the
RFC’s suggested interoperable range will have any round‐tripping
issues, and we continue to guarantee the behaviour on all values that
can be represented as Nix‐language integers.
Change-Id: I6a6a696412383e8a2cc160397716cb7f7bc7a2d4
So far, the environment used by `command` was completely leaky and the
one used by `nix` was very leaky despite it trying to be a "hermetic"
environment.
This commit moves the hermaticity to `command` and changes its
implementation to be not leak anything.
To achieve this, the following changes were also nessecary:
- the `files` and `snapshot` fixture now use the folder `test-home`
within the tmp_path directory by default, as the `HOME` environment
variable is set to there. (extraction not possible due to dependencies
of command etc also using this directory)
Fixes: #847, #848
Change-Id: I55f86ee0e1615e73fcf442ee2f28f3b89893bbb4
Current test suite doesn't cover the subsecond formatting at
all and toml11 is quite finicky with that. We should at the very
least test its behavior to avoid silent breakages on updates.
(cherry picked from commit 7ed0229d1abd4414144c7af396842462ce6fc1eb)
Upstream-PR: https://github.com/NixOS/nix/pull/13741
Change-Id: I6a6a696433b168072d6ad2585dce8a3c10ccbc39
use a thread pool and allow only buffered sources for performance. we
may want to use this code path unconditionally due to gc interactions
of fibers in circumstances we should not even be able to trigger, but
if that becomes important we will have a useful implementation ready.
Change-Id: Ib4e1531fe920847d8e30a42e8df393ace549f52e
We need capnproto-lix to be provided by callPackage, otherwise it won't
be spliced and the same (built for the cross host platform) for both
buildInputs (correct) and nativeBuildInputs (incorrect). We thus move it
into the overlay.
A similar problem exists for the lowdown build. We thus use callPackage
to override it as well. This is horrible (especially because it means we
need to pass the enableDarwinSandbox package argument through
ourselves), but at least it builds...
Fixes: https://git.lix.systems/lix-project/lix/issues/939
Change-Id: I802152072d852903401ef701f526195aa99475f2
When testing specific internal functionallity while needing things from
the testlib, so far, the tests for the testlib have always been copied
too.
To reduce the amount of additional program required when later making
the env of the pytest_command declarative, and to not test the same
tests a multitude of times (and potentially reaching infinite recursion)
those tests will no longer be copied
Change-Id: I36ec3824a21ed30f9b8ff19948031d1edbf6c76c
this means both the worker protocol and the serve protocol, i.e.
ssh-ng/local connections and legacy ssh connections. now we have
no blocking reads left anywhere in our client store connections.
Change-Id: I2f628d4d2e71ef0a7006918f175192f3f58eea95
`mesonCheckPhase` of course runs `preCheck` and `postCheck` hooks,
not `preInstallCheck`, so this was not doing the right thing. Really
sorry for breaking CI :(
Fixes: e6da29ad6b
Change-Id: I6a6a69642a242e2c8bfe10ac61d3f6756d666548
Snix's nar-bridge[1] stores NARs under a different URL, i.e.
`nar/snix-castore/<hash>.nar` rather than `nar/<filehash>.nar`. Right
now, when copying into such a store via the http binary-cache, we'd end
up with wrong cache entries that point to the wrong NAR URL.
On Hydra, this is a fatal error, i.e. builds that depend on previously
built paths (that were written to the cache before by the queue runner)
would be aborted because of that.
This patch removes the caching since we'd have to re-fetch the narinfo
to do taht and this can also happen the next time, the narinfo is
queried. Also, removes the negative cache entry indicating that the
store-path doesn't exist in the store.
We don't have any coverage for http-stores so far, so I wrote a small
testcase for the "default" case and the nar-bridge case in functional2
since it has a very nice fixture for an HTTP server ready. I'm aware
that there's a CL for a nicer cache server[2], but given I'm down a
pretty deep rabbit hole by playing around with Snix, I decided to not
finish the CL and write something small for the tests in here. This can
be replaced by the fixtures from that CL later on as well.
[1] https://snix.dev/docs/components/overview/#nar-bridge
[2] https://gerrit.lix.systems/c/lix/+/2431/1
Change-Id: I4fcdf47a6bf9c3c8fbeb235eeca7a48914a4d693
shoving a nar dump directly into a framed sink created a bunch of tiny
frames, each of which requires at least two syscalls to read. this can
lead to immense performance loss when using a daemon; we have seen 15%
in benchmarks on main and even more with async code involved ... oops.
Change-Id: I8529506e3de74d92834d1f4ee228dcaf32eb756c
Notably, this adds the Requires.private required for proper static linkage of
Lix libraries. Some minor missing or duplicated dependencies are also fixed
along the way (although some optional dependencies are omitted due to their
size).
Fixes: https://git.lix.systems/lix-project/lix/issues/789
Change-Id: I6a6a696413d538124d9ac75c68f100cc3089284f
A bunch of dependencies were superfluous, while others were missing (in
particular on internal sub-libraries) and things just happened to work because
they leaked in or were explicitly required in other places. Make efforts to fix
them all.
Change-Id: I6a6a69643e35ac4b0b66a72f4e42a2ba5ed52488
Some of the Lix libraries always need to be linked in full due to their
reliance on static initializers. This was achieved internally using link_whole,
but they are still easy to abuse by external users who manually need to
remember passing linker flags such as `--whole-archive` (GNU) or `-force_load`
(Apple), and the obvious way to shove it in pkg-config breaks Meson due to
potentially including a library's flags multiple times, and then deduplicating
only the file names leaving a stale `-force_load` around causing trouble.
Instead we now "prelink" the static libraries, by merging them into one object
file. Since the static linker will always link entire object files, this will
have the same effect as whole-archive linking (except the library won't be
included if it's completely unused, which should not cause trouble since it's
unused after all, and dynamic libraries behave the same way). Unfortunately
Meson's native prelink functionality cannot be used due to missing (non-Apple)
Clang support [1], so write our own one. While not particularly portable, it
should work with Clang which is the only officially supported compiler, as well
as GCC.
[1] https://github.com/mesonbuild/meson/pull/14846
Change-Id: I6a6a6964a82241ce3b0b11fe8397fd451b8027f2
Meson only requires the `modules` option for Boost libraries that should be
linked against [1]. However, we use only header-only portions of the Boost
container library; in fact we only do not run into the disallowedReferences
because the linker drops the unused library. Remove the misleading option.
[1] https://mesonbuild.com/Dependencies.html#boost
Change-Id: I6a6a69648b806bf6bcf784391263f5ee9cd63a0b
This gets us `--print-errorlogs` and `--timeout-multiplier=0` for free,
and also fixes the concatenation logic with `__structuredAttrs`.
Change-Id: I6a6a69643ea7224cb24508505c554143f83ae243
Otherwise we'll get system names like `x86-linux`, which is nonesense.
Also add fixups for 32-bit ARM, and MIPS.
Change-Id: I6bd773bec82dcc54b40bbc018641cd614b272a28
this lets us avoid a bunch of fcntl calls to unset and re-set O_NONBLOCK
on remote store connections. the overhead of these isn't high, but doing
it is still wasteful and a maintenance burden when we have async readers
Change-Id: I900cdca2a16202380c8b6f9b86da7d9b0f1e34ac
Emily