The issue requiring these environment variables to be set for Meson to find
Boost [1] has been fixed [2] for quite some time now. Drop them since they are
unnecessary in all supported nixpkgs releases (in fact they have already been
removed in the lix package there).
[1] https://github.com/NixOS/nixpkgs/issues/86131
[2] https://github.com/NixOS/nixpkgs/pull/315998
Change-Id: I6a6a69640a30e917cd7a983b9d846d023b393dcd
In certain scenarios, a path may exist but is a broken symlink. For
instance, this happens frequently if you are rescuing an existing NixOS
system via `nixos-enter` or a manual `chroot` invocation because the
rescued system `/etc` may have broken links or the system prior to the
root pivot may interfere.
Nonetheless, these dangling symlinks are not always fatal for the builds
so we can just warn and skip their setup.
The warnings will provide a better diagnostics for system administrators
also.
Fixes#893.
Change-Id: Ifa12be3a43f23c973d7b466e8b73bd776abf3e7b
Signed-off-by: Raito Bezarius <raito@lix.systems>
We were catching ForeignExceptions believing it came from the TRY_AWAIT
handler, but this was misguided.
`j.dump()` is evaluated in synchronous context, outside of the `try {
... } catch (...)` block from `TRY_AWAIT`.
Therefore, we need to use `JSON::Exception` directly.
The previous test case did not catch it because:
(1) https://git.lix.systems/lix-project/lix/issues/865 hid the fact that
`--arg` was wrong.
(2) we did not grep for the warning because… we were not even copying
the strange store path to the binary cache.
(3) checking for the NAR happened after the NAR directory was emptied
for test reasons and this was not even caught neither.
Anyway, the test case was completely busted and has now been tested
without this commit and after this commit and we can confirm that prior
to this commit, the test will fail with an exception trace.
Co-authored-by: Maximilian Bosch <maximilian@mbosch.me>
Change-Id: I8df5befd06c4a449072b987f82a67bc4437e7e49
Signed-off-by: Raito Bezarius <raito@lix.systems>
PushActivity does not work with async code since we have no such thing
as promise-local storage. it will be confusing at best, and completely
wrong at worst, with the current thread-local linking state. if we can
find a way to get promise-local storage we may want to bring this back
though, explicit context passing is rather error-prone. luckily we are
not using parent links for anything important, just to keep the multi-
line activity display from filling up with stuff we're already showing
Change-Id: Ie373d713080a3db811b2d5abd681f78137735e45
checking that the remote build actually succeeded only implied-trusted
remotes or CA derivations makes *absolutely* no sense. we should check
that builds have succeeded before trying to copy them from the remote.
Change-Id: Ib2cf216c580f4c577dd9fef8849acc033ae082b9
Including the `.cc` is possible but is suspicious.
Change-Id: Ie18fef1e30e517edff4ab96f4a9c339e2b3145b5
Signed-off-by: Raito Bezarius <raito@lix.systems>
postInstall needs to know the name of the (versioned) .so file since it
needs to be renamed for Darwin platforms. Unfortunately, the .so version
is not properly overrideable, so we need to use string replacement.
Change-Id: Idf9671f84fac955a52d82a20ec0f381d05fdc762
lowdown 1.4.0 changed the lowdown_opts to include a new and separate
lowdown_opts_term which allows for configuring values specific to
-Tterm (which we're using). This version should have been called 2.0.0
according to semver, hence 2.0.0 was released later without any actual
breaking changes to sort of migitate the problem.
We need to support lowdown >= 1.3 && < 1.4 since the ship has sailed for
updating lowdown in NixOS 25.05 as well as lowdown >= 1.4 or we'll be
stuck in Nixpkgs forever. Support for < 1.4 can be dropped as soon as
NixOS 25.05 is EOL, assuming this change lands before NixOS 25.11
branch-off.
We detect the changed API based on the lowdown version from pkg-config
and define LOWDOWN_SEPARATE_TERM_OPTS based on that. The ifdef is named
according to the specific API change that impacts us, so that it's
hopefully a little simpler to maintain going forward. In the new API,
all newly configurable settings use what would have been the (implicit)
default before. Changing some of these values, especially hpadding,
could be interesting in future changes.
Compared to cl/3081, this change makes sure to initialize all new fields
of lowdown_opts_term explicitly.
It seems that, while making -Tterm more configurable, lowdown's word
wrapping behavior changed slightly which broke basic_repl.test. I've
chosen to work around this by using builtins.add as an example which has
a very short documentation string, so wrapping doesn't matter.
Change-Id: Id73be4c0e43d7eb4f56e10a261b4254402698ff8
capnp does not handle fd passing correctly in all circumstances. we hit
such cirumstances when passing large closures path lists to build-hook.
since capnp seems to ignore fds passed in non-final segments of any rpc
message we just ensure that the capability including the log fd will be
small enough to not be fragmented on the receiving side of the channel.
cf https://github.com/capnproto/capnproto/issues/2359
Change-Id: Id22309264936b3a57bcc68a0753c3bfb3c9a43d2
Some users may have arbitrary needs to connect to their store URIs, e.g.
mTLS authentication, Kerberos authentication, custom renewal using any
RPC mechanism of their preference and so on.
To avoid encoding all these patterns in Lix itself, we push the
configuration to the plugin boundaries and offer a hook for end users to
inherit from `HttpBinaryCacheStore` and provide new store schemes like
`https+mtls://my.very.secure.cache?tls-certificate=...&tls-key=...` or
`https+krb5://my.kerberos.enabled.cache`.
Co-authored-by: George Shammas <george@shamm.as>
Co-authored-by: eldritch horrors <pennae@lix.systems>
Signed-off-by: Raito Bezarius <raito@lix.systems>
Change-Id: I79f322b1a74632500fc79d53f5c920f9e43fd0c4
Usually, EOFs are represented by returning 0 in the `read` APIs, at
least, this is what read(2) dictate.
As clever creature, we may sum zeroes sometimes (advanced form:
`buf->added(got)`) and forego handling the EOF condition.
To avoid the bug that lurked in remote-store.cc and caused busy looping
if the remote end disconnects suddenly, we return
`Result<Option<size_t>>` forcing the caller to perform a specific
processing for the EOF situation.
The conversion did not raise any other offending code path.
Change-Id: I185fdcb77aa82d87ab0802d66ac37c1363657a73
Signed-off-by: Raito Bezarius <raito@lix.systems>
Also adds an assert that store path hash part length is what we expect
because it's alarmingly easy to forget to truncate a hash before
throwing it into there. It's kind of messy code, someone could improve
it more later.
Change-Id: I5296ea3d5b854323d092f0256defb598dd5b87e8
remote builds failures used to be signaled via exit status 1 of the
build hook, which in turn only happened because the build errors we
got from remote stores was thrown and bubbled up to main which then
logged the error and exited with code 1. with rpc we cannot do this
any more. barring a rewrite of the worker infra to allow for errors
being reported with something other than process exit codes this is
the best can do. ideally we would wrap remote builds in a new goal.
(and then remove all exit code shenanigans from DerivationGoal too)
fixes#928
Change-Id: Idc3ede3cbaca34c8c8e40247da52794f2a5013b9
drop our reimplementation of splice for non-linux in favor of using kj
pumpTo. this avoids select() for its O(maxfd) behavior, and if kj ever
uses something more efficient than read/write loops we'll benefit too.
Change-Id: Id01ba84bf8831455af2d9755bf1a3039d215bb47
libarchive *should* not break with 0710 on the tmpdir root on darwin,
just like it doesn't break on linux, but for some reason it does. the
restriction to 0710 can be weakened to 0750 with causing any trouble.
fixes#921
Change-Id: Ia9fc2f8eb9695fc19cefae9857368d5a4e58c8b9
previously we only had one build hook in waiting at most because build
hook rpc was synchronous. now that it no longer is we attempt to start
one hook per derivation, which depending on scheduling can be a *very*
large number. restrict the waiting hook count to 4 to some concurrency
without collecting a large number of hooks that may never do anything.
Change-Id: Ic0b1125cec4acd69e8a0d4639c232e71b825e01d
although we only chown if the build was requested by a local daemon
user. daemonless invocations will not chown as they do not have to.
remote builds *can* chown to the remote builder user, but that does
not seem to happen (for some reason keep-failed is not propagated).
Change-Id: Ic0ead406b38b4ca0556fec42d84888efa25123bf
this makes the actual build directories used by builders invisible and
inaccessible to other processes on the system, avoiding another vector
for outside processes to interfere with builds or pass credentials the
build sandbox should not have access to into the build sandbox anyway.
fixes#919
Change-Id: Ifaa4d8e3940cfde1406e925f75c1375d2e86d81a
this touches both libutil and libstore because with no rpc users it
doesn't make that much sense to separate the two. note that all our
strings are represented as Data (ie, blobs) because capnp Text must
be nul-terminated. while it's technically possible to use Text with
strings containing non-terminating NULs it is a bit of a hassle and
could lead to rpc users erroneously stopping at the first NUL byte.
Change-Id: I4c75e03b79a226ffa8d7cd985e3ac632a0cd7c1c
we need this to generate dependency information, and it'll be the entry
point for custom codegen once we need it. a wrapper also makes it a lot
easier to generate a whole namespace's worth of rpc definitions at once
Change-Id: Iba7a1c92a8a40bede9ed71aa3ab455477ff5e568
if the hook accepts the build request we can handle the entire request
in tryBuildHook. there is no need to punt a partially handled build to
the caller (we only did this to minimize churn during asyncification).
Change-Id: Iec3e35a8103da4fc5fbef394cc28a134ee62a198
mapping the result of an await operation before unpacking it lets us
inject rpc type conversion functions without duplicating all that is
needed for proper exception wrapping and async error traces support.
Change-Id: Ibcba1cc6d2b275757e3475881ef20f95dd4d684f
Goals:
- Distribute reviews to people who can do the reviews
- Not prevent anything from getting done
- Allow giving away more commit access
Anti-goals:
- Silo people into particular areas
- Discourage contributing to any area
This was drafted by glancing at git logs. It is not likely to be very
accurate; the goal here is that we figure out a way to distribute
reviews to the right people.
Change-Id: I8be44bf7fdeca23da8099124eec7bc3a30e34627
`Outcome<void, T>` and `Result<std::optional<T>>` can be interpreted as
being the same thing, but the latter is easier to use: not only do they
allow TRY_AWAIT usage for their promises, we also don't have the error/
exception confusion of outcomes (where the T above is the "error" type)
Change-Id: I92c9241481cecc97e2992445b3dced53c82a2524
while this does require spawning a thread for every contended lock now
we don't expect performance to be impacted. only build-remote used the
synchronous method, and it only used it to serialize uploads to remote
builders. these uploads are expensive enough to dwarf the thread cost.
Change-Id: Iad0aa0cd738bc96fd06a90d655803dadffa09c47
DerivationGoal::InputStream existed only because we did not have an
error-reporting AsyncInputStream of our own yet. we do have one now
though and can thus delete old code in favor of the generic variant
Change-Id: I01c7c564554f8794bdf54603b239b7a808faeda0
it's effectively unused. one use is a write and a read immediately after
the write, the other use checks whether it's not equal to itself (..wat)
Change-Id: I5f6ce26e75a6bfa500c2e9ac3fc70e8dafc9bd74
Commits 205c59367c and
325e7e1824 introduced real glibc store paths from
current nixpkgs unstable into the source. Since nixpkgs `fetchFromGitea` (and
similar fixed-output derivations) depends on a C library, on x86_64-linux they
will fail with the forbidden reference error:
error: the fixed-output derivation '/nix/store/wnmnj3jzc82y89sfmyicr04kilg8zs2k-source.drv' must not reference store paths but 1 such references were found:
/nix/store/q4wq65gl3r8fy746v9bbwgx4gzn0r2kl-glibc-2.40-66
Falsify the store path to prevent this failure.
Change-Id: I949033567bcad070f9a0a19cefdb33a79222e421
My lix build failed today with this result:
```
lix> [----------] 3 tests from MonitorFdHup
lix> [ RUN ] MonitorFdHup.works
lix> [ OK ] MonitorFdHup.works (0 ms)
lix> [ RUN ] MonitorFdHup.works_with_pipes
lix> stderr:
lix> Using configuration: seed=6402097764877502971
lix> libc++abi: terminating due to uncaught exception of type std::__1::future_error: The state of the promise has already been set.
lix> 4/5 lix:check / libstore-unit-tests OK 1.10s
lix> 5/5 lix:check / libexpr-unit-tests OK 1.12s
lix> Summary of Failures:
lix> 3/5 lix:check / libutil-unit-tests FAIL 0.93s killed by signal 6 SIGABRT
lix> Ok: 4
lix> Fail: 1
lix> Full log written to /nix/var/nix/builds/nix-build-lix-2.94.0-dev-pre20250711-65ef28d.drv-0/source/build/meson-logs/testlog.txt
```
I had a response best described as "wtf". I think the cause of this
problem is that there's a race condition with the test in which the loop
gets gone around again a second time because it's triggered by the
terminate fd (and I guess the flags remained what they were before?
seems reasonable), and this is probably racing with the quit atomic
being first to break out of the loop.
I don't know how many hundreds of lixes I've compiled without my test
failing, but this is definitely a bug. I don't think this affects actual
usage as the only impact is repeat delivery of Ctrl-C which is harmless
and which users do regularly.
Change-Id: I60da81d4ac2e79052cd323b5171f9d8bd0aa6783
this was a mess. ssh:// remotes used the extra static fds for build
logs, ssh-ng:// remotes did not. ssh-ng remotes did not use them at
all since ssh-ng never redirected them to begin with. we now create
pipes dynamically and only for ssh:// builders, then translate logs
received over these pipes into the same format used by ssh-ng. this
requires a new activity we did not have before, but since we have a
great many activities that rarely show up already this shouldn't be
a problem for external tooling. if anything external tools can tell
what's going on much better now (at least for ssh:// remote builds)
Change-Id: I02010cee45598362a947faa3a5b04800d39daa31
Avoids incorrect behavior with large integers in `elemAt`, `substring`,
`genList`, etc, which results into crashing the Lix interpreter.
At the same time, unit tests were added for these edge cases with 2^32
as an argument of these primops.
Port of https://github.com/NixOS/nix/pull/13309.
Prior art in https://github.com/NixOS/nix/pull/7222 (forgotten by the original project…).
Change-Id: I1c43ed64f26bcb60e51869e11a74e5de2b7db53a
Co-authored-by: Raito Bezarius <raito@lix.systems>
Signed-off-by: Raito Bezarius <raito@lix.systems>
Alois Wohlschlager