Commit Graph
7 Commits
Author SHA1 Message Date
eldritch horrors d90e4a6581 libutil: reject malformed nars harder with better errors
fixes #993

Change-Id: I56aff32498ecd32e65f1576661d5a15808a4439f
2025-09-12 23:22:11 +02:00
eldritch horrors 935901cd46 nix/ls: fall back to downloading the nar if the .ls is bad
fixes #959

Change-Id: I1715f6fcc938dac7287d9cf4a85e93c9d66768a0
2025-08-15 18:05:06 +02:00
Raito BezariusandMaximilian Bosch 3c614a136b libstore/binary-cache: fix catching JSON exceptions
We were catching ForeignExceptions believing it came from the TRY_AWAIT
handler, but this was misguided.

`j.dump()` is evaluated in synchronous context, outside of the `try {
... } catch (...)` block from `TRY_AWAIT`.

Therefore, we need to use `JSON::Exception` directly.

The previous test case did not catch it because:

(1) https://git.lix.systems/lix-project/lix/issues/865 hid the fact that
`--arg` was wrong.
(2) we did not grep for the warning because… we were not even copying
the strange store path to the binary cache.
(3) checking for the NAR happened after the NAR directory was emptied
for test reasons and this was not even caught neither.

Anyway, the test case was completely busted and has now been tested
without this commit and after this commit and we can confirm that prior
to this commit, the test will fail with an exception trace.

Co-authored-by: Maximilian Bosch <maximilian@mbosch.me>
Change-Id: I8df5befd06c4a449072b987f82a67bc4437e7e49
Signed-off-by: Raito Bezarius <raito@lix.systems>
2025-07-25 21:04:16 +02:00
Raito Bezarius 21dbd7745d libstore/binary-cache-store: skip NAR listings if it's not possible to serialize it
Some source trees might not be representable inside of the NAR listing
format v1 as file paths (on Linux) are not guaranteed to be valid UTF-8.

When something like this happens on a large-scale build farm, a
mysterious "queued" but impossible to process job appears, this is
because we cannot write the NAR listing and serialization always fails.

Why did this work before? nlohmann was introduced _after_ such paths
were ingested, see: 09f00dd4d0.

What happened for such previously mis-serialized NAR listings?

```
curl -v 'https://cache.nixos.org/nz8p9hn00r6z7s57581c1hiv39pa1ia6.ls' |
brotli -d | jq .
```

This fixes the build of `sub-batch`
(https://github.com/kl/sub-batch/tree/master/tests/rename_invalid_utf8)
on ForkOS infrastructure.

Many thanks to Puck for the assistance on holding `rr` right on this one
and finding the history of these changes.

Change-Id: I2c2fbac70818e02810f9fd236c3a248187bf5fe7
Signed-off-by: Raito Bezarius <raito@lix.systems>
2025-06-10 15:22:22 +00:00
Victor Fuentes ab1e58e948 feat(nix store ls): support reading nar listings from binary cache
Remote binary caches support `write-nar-listing` options where they create a `HASH.ls` file for quick indexing without having to download the nar.
This commit makes experimental `nix store ls` attempt to read these files instead of downloading the full nar.

The difference is very obvious with large packages like stellarium:

nix store ls --store "https://cache.nixos.org" /nix/store/ijpvwgs9zamqaax5dy2cd0kxgz7lr7an-stellarium-25.1 -R

Change-Id: I6a37e0788b3a91c319331a8de69c51daf3efa955
2025-06-01 22:47:11 -07:00
eldritch horrors 2633ca3f88 Merge pull request #9288 from edolstra/fix-nar-access-test
Fix nar-access test on macOS

(cherry picked from commit e9a857e4189241d23111e18e284ab7ac9d56b798)
Change-Id: I2fd2ec3e6ccb88350bdcd2ba15ccfd77275399c9
2024-03-04 05:27:18 +01:00
John Ericson 30dcc19d1f Put functional tests in tests/functional
I think it is bad for these reasons when `tests/` contains a mix of
functional and integration tests

 - Concepts is harder to understand, the documentation makes a good
   unit vs functional vs integration distinction, but when the
   integration tests are just two subdirs within `tests/` this is not
   clear.

 - Source filtering in the `flake.nix` is more complex. We need to
   filter out some of the dirs from `tests/`, rather than simply pick
   the dirs we want and take all of them. This is a good sign the
   structure of what we are trying to do is not matching the structure
   of the files.

With this change we have a clean:
```shell-session
$ git show 'HEAD:tests'
tree HEAD:tests

functional/
installer/
nixos/
```

(cherry picked from commit 68c81c737571794f7246db53fb4774e94fcf4b7e)
2023-12-01 12:06:43 -05:00