Snix's nar-bridge[1] stores NARs under a different URL, i.e.
`nar/snix-castore/<hash>.nar` rather than `nar/<filehash>.nar`. Right
now, when copying into such a store via the http binary-cache, we'd end
up with wrong cache entries that point to the wrong NAR URL.
On Hydra, this is a fatal error, i.e. builds that depend on previously
built paths (that were written to the cache before by the queue runner)
would be aborted because of that.
This patch removes the caching since we'd have to re-fetch the narinfo
to do taht and this can also happen the next time, the narinfo is
queried. Also, removes the negative cache entry indicating that the
store-path doesn't exist in the store.
We don't have any coverage for http-stores so far, so I wrote a small
testcase for the "default" case and the nar-bridge case in functional2
since it has a very nice fixture for an HTTP server ready. I'm aware
that there's a CL for a nicer cache server[2], but given I'm down a
pretty deep rabbit hole by playing around with Snix, I decided to not
finish the CL and write something small for the tests in here. This can
be replaced by the fixtures from that CL later on as well.
[1] https://snix.dev/docs/components/overview/#nar-bridge
[2] https://gerrit.lix.systems/c/lix/+/2431/1
Change-Id: I4fcdf47a6bf9c3c8fbeb235eeca7a48914a4d693
We were catching ForeignExceptions believing it came from the TRY_AWAIT
handler, but this was misguided.
`j.dump()` is evaluated in synchronous context, outside of the `try {
... } catch (...)` block from `TRY_AWAIT`.
Therefore, we need to use `JSON::Exception` directly.
The previous test case did not catch it because:
(1) https://git.lix.systems/lix-project/lix/issues/865 hid the fact that
`--arg` was wrong.
(2) we did not grep for the warning because… we were not even copying
the strange store path to the binary cache.
(3) checking for the NAR happened after the NAR directory was emptied
for test reasons and this was not even caught neither.
Anyway, the test case was completely busted and has now been tested
without this commit and after this commit and we can confirm that prior
to this commit, the test will fail with an exception trace.
Co-authored-by: Maximilian Bosch <maximilian@mbosch.me>
Change-Id: I8df5befd06c4a449072b987f82a67bc4437e7e49
Signed-off-by: Raito Bezarius <raito@lix.systems>
PushActivity does not work with async code since we have no such thing
as promise-local storage. it will be confusing at best, and completely
wrong at worst, with the current thread-local linking state. if we can
find a way to get promise-local storage we may want to bring this back
though, explicit context passing is rather error-prone. luckily we are
not using parent links for anything important, just to keep the multi-
line activity display from filling up with stuff we're already showing
Change-Id: Ie373d713080a3db811b2d5abd681f78137735e45
Usually, EOFs are represented by returning 0 in the `read` APIs, at
least, this is what read(2) dictate.
As clever creature, we may sum zeroes sometimes (advanced form:
`buf->added(got)`) and forego handling the EOF condition.
To avoid the bug that lurked in remote-store.cc and caused busy looping
if the remote end disconnects suddenly, we return
`Result<Option<size_t>>` forcing the caller to perform a specific
processing for the EOF situation.
The conversion did not raise any other offending code path.
Change-Id: I185fdcb77aa82d87ab0802d66ac37c1363657a73
Signed-off-by: Raito Bezarius <raito@lix.systems>
all of them block on a promise very soon after starting. only
queryValidPaths needs to make sure not to swallow Interrupted
exceptions to exit quickly instead of trying all paths first.
Change-Id: I4f99f5d75d7057bad109dc0131aa58e84275e362
Some source trees might not be representable inside of the NAR listing
format v1 as file paths (on Linux) are not guaranteed to be valid UTF-8.
When something like this happens on a large-scale build farm, a
mysterious "queued" but impossible to process job appears, this is
because we cannot write the NAR listing and serialization always fails.
Why did this work before? nlohmann was introduced _after_ such paths
were ingested, see: 09f00dd4d0.
What happened for such previously mis-serialized NAR listings?
```
curl -v 'https://cache.nixos.org/nz8p9hn00r6z7s57581c1hiv39pa1ia6.ls' |
brotli -d | jq .
```
This fixes the build of `sub-batch`
(https://github.com/kl/sub-batch/tree/master/tests/rename_invalid_utf8)
on ForkOS infrastructure.
Many thanks to Puck for the assistance on holding `rr` right on this one
and finding the history of these changes.
Change-Id: I2c2fbac70818e02810f9fd236c3a248187bf5fe7
Signed-off-by: Raito Bezarius <raito@lix.systems>
only a daemon wire operation and the perl bindings could initiate these
queries at this point. the daemon ops can throw an error instead (as if
the daemon were older) and realistically should never be queries if the
client hasn't evaluated a ca derivation on a given store, and perl code
is best off dying early. nothing known except hydra uses these bdingins
anyway, and we control our hydra so we don't need backward compat code.
Change-Id: Ia7df27aba59a4a4a692ae014f407415f3bea63f2
it's only used by the RegisterDrvOutput daemon wire operation now, and
that one we can safely stub out to throw an error when called instead.
Change-Id: If29716976392c9c7a2a05b151dfe80b2c8d9c07d
otherwise lix may crash when e.g. nix search receives invalid regex.
we now also give better error messages for regex errors during eval.
fixes#803
Change-Id: Icc7c578ff488ba520efac5d898572ccf4486e9a8
448c7d50e1 was a bit over-eager and didn't
make the `ref(enable_shared_from_this &)` constructor explicit. this has
confused hydra maintainers, and is just generally bad practice since any
reference is allowed to implicitly convert, even if those references are
not associated with an active control block. we can't avoid this problem
entirely, but we can make what happens more explicit by ... well, making
the involved constructor explicit. enable_shared_from_this is statically
unsafe in principle and we really have to get rid of this nonsense soon.
Change-Id: I8b48ef4353e6301b61af3569083f42bc7379b0a4
don't throw if a downcast fails, have that particular casting method
return an optional instead and check the result at the call site. we
also rename dynamic_pointer_cast to try_cast_shared for consistency.
Change-Id: I65ee0fb498b1b2b8b418935fce4358a654f984e5
we don't need to unsafely cast a shared_from_this pointer to create a
ref (the pointer must be non-null to not invoke nasal demons anyway).
Change-Id: I133fe5f07b2cff8ec2c925f0528f4c5288827261
or more accurately, wrap them in a nix::Error subclass so we can display
them properly without crashing, and add some error context if available.
fixes#642fixes#753fixes#759fixes#769
Change-Id: I1aad0c0501fea83f9de3a1335eaa6adc20721616
this will become a proper specialization of `nlohmann::basic_json` soon.
specialing basic_json will let us get rid of our `adl_serializer` hacks,
and it'll open the door to better enum serializing behavior without also
forcing all those who use lix as a library to set certain defines (which
may not even be possible depending on how those users use json already).
Change-Id: I5228d2b9df581a189552c993363207cfbd20f445
this doesn't do much, just wrap a few nlohmann headers in headers of our
own (and delete includes we don't need because they're transitively seen
by other includes). doing this now will make the next change much nicer.
Change-Id: I166933102ea86bb5322ebbf9ba9411f96032a53b
In cl/1549, this function was changed from calling `getFile(path, sink)`
to returning `getFile(path)`. As a result, the `StringSink` is no longer
used. However, the sink was still being allocated.
Change-Id: I26c8556f20bc4f922912d8a921e8f47aa2bf975e
we could have constructed a nar accessor from an index just as well, but
we don't need any of the advanced accessor features like retrieving file
contents or full path-string based access to the archive. using an index
directly is simpler and faster, although in practice we shouldn't notice
Change-Id: Ic521536cd89e88cbe5aba4f26bf40f0c40d09341
this fully decouples the possibly-never-async bits of dumping from the
generation of dump bitstream. having the two separate will allow us to
change store import methods to use async streams, not our sync sources
Change-Id: I9dbd5e30ad3ee380c244b4a3760c11e37db3895f
this will make it easier to return async streams instead of sources at
some point in the future. the primary benefactors of the current state
are not greatly inconvenienced by the api change, and would need to be
changed much as they are now once async streams come around either way
Change-Id: I4db9ea8b186f358c239f7863ac8140c500986c2d
all extant derived classes implement this method, making the default
implementation not very useful. should it ever be needed again it'll
be easy enough for derived classes to return an owning StringSource.
Change-Id: I65e041e7a4e7b161f0f404f4287cea5440b5a749
the very slight speedup in config setting access is not worth the
maintenance overhead of conflating concers like this. the virtual
inheritance scheme used for configs requires too much duplication
of base class constructor arguments to be worth doing. perhaps we
should get rid of all virtual inheritance of data-membered bases?
Change-Id: I4acf5ceaedb4ed7476efe1114c2e065ec72d2c6d
clangd broke because it can't look through symlinks. compile_commands
manipulation does not fix it, clangd configuration does not fix it, a
vfs overlay does not fix it, and while a combination of those can fix
it with a bind mount in place that's just too cursed to even consider
clangd bug: https://github.com/llvm/llvm-project/issues/116877
Change-Id: I8e3e8489548eb3a7aa65ac9d12a5ec8abf814aec
Maximilian Bosch