Lix blocks SIGPIPE, and std::cout ignores EPIPE (actually it probably stashes
an error code in some flag that no one ever reads, with the same end result).
Consequently, nix-eval-jobs would waste resources by continuing to evaluate
even when the reader interested in the results has long gone away. Instead, use
writeToStdout, which throws an error on EPIPE, leading to the process
terminating as desired.
Reported-by: Winter <winter@winter.cafe>
Change-Id: I962c09bab582a8ed27dd41c01b1519876a6a6964
Right now, the worker will not receive interrupts, as it has no signal handler
thread due to the way it forks, but this will change soon. Prepare to handle
worker interruption properly, by letting the worker exit successfully in this
case and adjusting the coordinator to handle this exit gracefully.
Change-Id: I9f79670b0d2004e7e2d8bf36cf67108e6a6a6964
This also prevent `legacy` to return nothing as it's void.
Change-Id: Ic797544a59b04b41ad9e1c46af4cad3a1cb6fdc8
Signed-off-by: Raito Bezarius <raito@lix.systems>
When the maximum memory size is chosen too large (relative to the free memory
usage), so that the evaluation workers don't fit into the free memory, they
will start being swapped out. Further increase in memory usage will not be
reflected in the RSS, so they will not exit, and instead fill up all swap too
until the system runs out of memory.
The BDW-GC keeps track of its heap size, including any parts of it that may be
swapped out. For this reason it does not suffer from a similar problem. While
not all memory usage by the evaluation worker is accounted for by the GC, a
simple affine model for the overhead works reasonably well in practice. The
specific parameter values were determined using nixpkgs evaluation, and appear
to work quite well also for somewhat different workloads (like the `hydraJobs`
of Lix, which include a bunch of NixOS system configurations). So this is what
gets used.
When Lix is configured without the BDW-GC, of course this approach cannot be
used. In this case the old strategy is retained.
Change-Id: I6cb5f0a9b4ceda9dd14be165dda108cd6a6a6964
thunk values are shareable, and we can represent invalid/uninitialized
values with a special bit pattern that makes no sense otherwise. there
is no need to keep allocating values on the heap, instead we can treat
values like reference-counted smart pointers to heap objects, which in
turn lets us save a lot of allocations and, ultimately, gc heap space.
compared to our baseline (main of 2025-09-27) we save 15%+ memory on a
system rebuild and 17% on nix search. eval time regresses by ~3% for a
system rebuild, while nix search is 7% faster. further optimization is
probably possible (but for now this will just have to be good enough).
Change-Id: Ib6c47acdbe2fac4f76a83c2269f16f30ef66b2e1
with thunk state being shareable we no longer need to worry about value
uniqueness, only about value lifetime. this means we can liberally drop
indirections and allocations, passing references instead of pointers or
using stack memory instead of gc-managed memory for some intermediates.
Change-Id: I2d48a6fd57a376d544bd9bd2d05e5420611986d1
on its own this is not very useful, but having accessors for every value
kind is a prerequisite for doing smart things with Value than the union.
the net effect for now is only to add a few parentheses across the tree.
Change-Id: I88688ac09eb08495dad1eb221034ca540f094950
Closes#987
The patch adds a flag `--no-instantiate` which only performs evaluation
without instantiating any derivations. Hence, GC root creation is also
skipped. To achieve that, Lix is also put in read-only mode and all
operations that require reading a derivation (e.g. constituents or
listing input derivations) are disabled fallback values are set.
This is a port of an upstream PR[1]. Given the divergence of the
codebases (different restructurings on both ends, no more CA derivations)
I decided to redo large portions from scratch instead of
cherry-picking the patches. Hence, the authorship.
Additionally the clean up of casts down to a local store are removed or
guarded behind an if, as done in the upstream PR.
[1] https://github.com/nix-community/nix-eval-jobs/pull/379
Co-authored-by: Jörg Thalheim <joerg@thalheim.io>
Change-Id: Ib84f44e7799bc5577fd2ee98912458f16ebeab81
Some platforms like 32-Bit PowerPC need linking against libatomic.
Try to compile and link a very simple snippet of code which uses atomics
and make libatomic required if it fails.
Because we're using `dependency('atomic')`, the required meson versions
gets bumped to 1.7.0. See https://mesonbuild.com/Dependencies.html#atomic-stdatomic
Change-Id: I6a6a696471e1d352fb161c537ba9023b97c2d31e
Without https://github.com/NixOS/nixpkgs/pull/434761 evaluation of the
`nixpkgsLibTests` will fail in CI with recent enough Lix, due to reliance on
the TOML integer saturation bug.
Reported-by: Sergei Zimmerman <sergei@zimmerman.foo>
Change-Id: I6a6a6964838009d2c525f67035f84072fdfad988
printTaggedWarning already colorized its messages. we can do the same
for most other log messages.
Change-Id: Idcd31bbf4f8d0d703395b0d2b7b9bc33264d969f
luckily none of these a format strings vulnerabilities because
boost::format is smart enough to throw an exception when given
fewer format string arguments than are requested by specifiers
Change-Id: I5fa78f0d1396263271f6e1dbcee9c0b2e9e18c34
always use log macros, which also have the benefit of respecting the
verbosity setting without needing virtual function calls to read it.
Change-Id: I1c605562a53e54140724d5225e040abcf49ac996
notably this also includes the symbol table because it stores real
strings that are referenced by eval values, and an upcoming change
will make it impossible to share those strings with value strings.
Change-Id: I20a3644db8aa0850efe29630e0b73d424cb2aa56
We need capnproto-lix to be provided by callPackage, otherwise it won't
be spliced and the same (built for the cross host platform) for both
buildInputs (correct) and nativeBuildInputs (incorrect). We thus move it
into the overlay.
A similar problem exists for the lowdown build. We thus use callPackage
to override it as well. This is horrible (especially because it means we
need to pass the enableDarwinSandbox package argument through
ourselves), but at least it builds...
Fixes: https://git.lix.systems/lix-project/lix/issues/939
Change-Id: I802152072d852903401ef701f526195aa99475f2
Goals:
- Distribute reviews to people who can do the reviews
- Not prevent anything from getting done
- Allow giving away more commit access
Anti-goals:
- Silo people into particular areas
- Discourage contributing to any area
This was drafted by glancing at git logs. It is not likely to be very
accurate; the goal here is that we figure out a way to distribute
reviews to the right people.
Change-Id: I8be44bf7fdeca23da8099124eec7bc3a30e34627
That way we get a line of output per test completed,
which makes it more obvious it's actually doing things.
Change-Id: Ifbbe8bdf64e7178d3c59349cf071eb5a9d0fcd32
This fixes Meson's "Project does not target a minimum version but uses
feature introduced in '1.1': meson.options file" warning.
Silly Meson.
I also added a note in the top-level meson.build to indicate
`meson_version` is specified in more than one place.
Change-Id: I2c04278bb46a562a1c96cd2e5e4d9ce59ce8e125
now that we have no deferred hashes (since floating ca derivations were
the only way to create them) we can safely remove this enumeration too.
Change-Id: Ic72ed90500fcee7aa5b3b5a302477fa515acf1be
this mostly takes the form of removes feature checks and the associated
"ca derivations enabled" branches, but for the realisation info command
turns into a stub. we keep it around for compatibility, but from now on
it will always throws "ca derivations not implemented" errors when run.
Change-Id: I0abea5f76262013415330adcca2b498c6dca555b
Inspired by cl/3191 and
https://git.lix.systems/delroth/lix/commit/ae0247cbb4fc739ab013dc87d02e5f3191cf25ab.
`coerceToString` takes now an enumeration that lives in `value.hh`, this
enumeration is meant to represent increasing subsets of behaviors, e.g.
any level above Strict should do what the previous levels do and extra
behavior until `ToString`, which transforms many Nix values into an
arbitrary string representation, e.g. `null` to `""`.
Change-Id: Ief7a4756e8c0660e197623efebeaf07710746ec7
Signed-off-by: Raito Bezarius <raito@lix.systems>
Co-authored-by: Pierre Bourdon <delroth@gmail.com>
add a position parameter to the autocaller instead, and pass it much
more accurate position information where we have it easily available
Change-Id: If2f1c3006ca3f2b413492842905d079a8b752542
otherwise lix may crash when e.g. nix search receives invalid regex.
we now also give better error messages for regex errors during eval.
fixes#803
Change-Id: Icc7c578ff488ba520efac5d898572ccf4486e9a8
we generally do not want to catch or throw these. catching them to print
and discard is fine, tests are largely exempt, and cases in which we can
be certain where the exception came from are also fine to *catch*. we'll
try to never *throw* (or rethrow) these if possible though because doing
so will make it impossible to construct async traces for the exceptions.
Change-Id: I3b71c32ecd16afc2246c946472f5629a1fa31f2c
don't throw if a downcast fails, have that particular casting method
return an optional instead and check the result at the call site. we
also rename dynamic_pointer_cast to try_cast_shared for consistency.
Change-Id: I65ee0fb498b1b2b8b418935fce4358a654f984e5
or more accurately, wrap them in a nix::Error subclass so we can display
them properly without crashing, and add some error context if available.
fixes#642fixes#753fixes#759fixes#769
Change-Id: I1aad0c0501fea83f9de3a1335eaa6adc20721616
technically it doesn't *have* to be NeverAsync, but not marking it as
such unconditionally requires templating DebugState over asyncness of
its callback (which then requires templating EvalState, which, *NO*.)
Change-Id: I4980d45b541c2e40328beac139b18c6c1ba0957c
this will become a proper specialization of `nlohmann::basic_json` soon.
specialing basic_json will let us get rid of our `adl_serializer` hacks,
and it'll open the door to better enum serializing behavior without also
forcing all those who use lix as a library to set certain defines (which
may not even be possible depending on how those users use json already).
Change-Id: I5228d2b9df581a189552c993363207cfbd20f445
this doesn't do much, just wrap a few nlohmann headers in headers of our
own (and delete includes we don't need because they're transitively seen
by other includes). doing this now will make the next change much nicer.
Change-Id: I166933102ea86bb5322ebbf9ba9411f96032a53b
this would've caught the missing aio await. also disable some rather
annoying deprecation and missing initializer warnings in dev builds.
Change-Id: Ibbb9dfc67eada0e7843081b669b7c8726a02ec29
Closes#703
The problem boils down to:
* A testcase with a segfaulting worker due to a stack overflow from the
evaluation.
* When the controller encounters that the pipes are down, the worker's
state is checked and a potential infrec is reported if it stopped with
SIGSEGV.
* However, in a few cases, the worker didn't exit yet, but only had closed
pipes (which is how the error is detected). In that case, a different
error message is printed breaking the test.
This bug handling only affects error cases, so this race doesn't happen
on normal shutdown. To mitigate the problem a bit, the controller will
wait for a second and re-check the state of the worker process then.
While this should make the error reporting for end-users a little more
reliable, it's still no guarantee for a non-racy test. Hence, the test
asserts against both cases now and it passes if either the infrec error
or the "worker still running, but pipes are closed" error is returned.
Change-Id: Ifdc7a05bc86b6aecd5d03118d3e3ffc9affe1c5e
Hydra used to support aggregate jobs that only succeeded when their
constituents succeed. This is still used by e.g. nixpkgs[1].
Prior art:
* https://git.lix.systems/lix-project/nix-eval-jobs/pulls/17: got ported
into the CppNix implementation[2]
* https://github.com/nix-community/nix-eval-jobs/pull/349: implements
glob expressions for constituents - something we needed at work. This
also restructures the code a bit which is what I re-used here. The
globbing is not part of this patch.
Essentially, the following things happen here (assuming `--constituents`
is set):
* Derivations with `_hydraAggregate = true;` are considered aggregates.
These are not written to stdout when received by a worker, but stored
until the end.
* Constituents can be drv paths or strings (that must be the `attr` of
another job). In that case, the derivation of the aggregate job is
rewritten so that it depends on the drv of the constituent job.
* At the very end the aggregate jobs are also written to stdout.
Additionally, this fixes one bug, the old `hydra-eval-jobs`
implementation had (and we actually hit at work):
Given the leaf jobs `packages.foo` & `packages.bar`, an aggregate job
`aggregate0` with
_hydraAggregate = true;
constituents = [ "packages.bar" "packages.foo" ];
and an aggregate job `aggregate1` with
constituents = [ "aggregate0" ];
then it may happen depending on the order of evaluation that `aggregate1`
depends on the old derivation of `aggregate0` (i.e. the one without
rewritten constituents) and doesn't depend on `packages.foo` and
`packages.bar` because it was rewritten before `aggregate0` was
rewritten.
This is done in here correctly, but topologically sorting the aggregate
jobs before rewriting those.
[1] https://github.com/NixOS/nixpkgs/blob/bba6b37c9d0898867a7d9c38a1b5b77efcfb07b9/nixos/release-combined.nix#L69
[2] https://github.com/nix-community/nix-eval-jobs/pull/340
Change-Id: I5baad5e57336b4985ef8595e903814de83eb01c1
Alois Wohlschlager