Instead of taking in a final argument `Value &` out parameter which it
writes to, it now returns its result
Change-Id: Iab6bc3a3ac6a4b17c6d31115a766a6ea6a6a6964
we don't need to mess with this rlimit for e.g. the daemon. increasing
the limit later should always be safe since we don't allocate (or map)
much before constructing the eval states that ultimately fill our heap
and could thus make stack expansion impossible after some time passes.
Change-Id: Ieafda537fbc99a6a7f83a093a981e7df947da437
merge all pkgconfig-found libraries into a single one and redirect all
existing lib files to the merged variant. the clean separation between
libraries is largely fictional anyway because the main config block is
in libstore and cannot reasonably be moved out, or because libutil has
assumptions built into its behavior that libmain later establishes, or
because libexpr messes with global boehmgc configuration. nearly every
kind of layering violation imaginable can be found *some*where in lix,
and not pretending that this is not the case will let us clean up some
of them. this will also allow us to merge these libraries at the meson
level which is itself necessary to make rust code easier to integrate.
Change-Id: I172f8a08d5d90ded0a0a99fd9e0fa5d804db00a7
nlohmann has customization points for exception throws. we can use these
instead and wrap json exceptions at the source instead of playing a game
of whack-a-mole with json errors all over the tree. since nlohmann needs
macros set to achieve this we can no longer precompile its headers *and*
must forbid including it anywhere without the proper defines or ordering
of include directives will break lix with ODR violations, if we see them
at all before we get another json-related bug report. a new lint will be
in charge of ensuring this doesn't happen. we also re-allow direct calls
to nlohmann json parsing since error handling is no longer a problem. we
will keep the wrapper for more convenient error context handling though.
fixes#1092
Change-Id: I54ecc14f5bec5e2177729b41c3703216e76cc6a3
That struct has been annoying me since I've encountered it. There is no
reason for having to track all these symbols out of band in the state.
Change-Id: I5c2c0d8174af0a51f9b456cc3651d8203a406d09
Some of the tests depended on the evaluation order chosen internally by
nix-eval-jobs, which will break when that order changes. In practice the order
is not guaranteed, and already not deterministic with multiple workers.
Change-Id: I2e85d096f4439a8075a6f4a08d31c0f16a6a6964
GCC still ICEs on coroutines, while Clang works properly. Use the latter, just
like for Lix itself.
Change-Id: I802d6df7279550a5f5355e6c1ea0533c6a6a6964
Previously the collector-side implementation of the worker interaction protocol
was a bunch of spaghetti code. Split it apart at places where it makes sense to
be easier to follow.
Change-Id: If3cc6b6fbf289dd778856b40b55316c76a6a6964
As the worker processes are now properly spawned using fork-exec, they no
longer inherit the stack size from the non-main coordinator thread, but get a
main thread with reasonable stack size on their own. For this reason the
coordinator threads can use the default stack size, and turn std::thread can be
used as the only reason for the custom wrapper was to enlarge the stack.
Change-Id: I1192474885abe9b0625ac483840b6e1a6a6a6964
Similar to how 7b37d5ea6a switched the subdaemons
from fork-only to fork-exec, do the same for the nix-eval-jobs workers. This is
the promised change that causes the signal handler thread to appear.
Change-Id: I590a5eb6bddfb546a331a01191fd788b6a6a6964
The retry added in 4c5efd4548 has never worked
properly, as releasing the pid the first time causes it to be -1 and thus
waiting for a random child on retry. While there is a good chance that this
really catches the crashed worker, this is by no means guaranteed, and it could
just as well have caught different restarting worker. Wait synchronously
instead. The worst possible outcome is a hang instead of an error message if
the worker manages to close its pipe but then fail to exit, which is not
supposed to happen in the first place.
Change-Id: I4e4c3f532ec15c3118f6f77821ab820a6a6a6964
Lix blocks SIGPIPE, and std::cout ignores EPIPE (actually it probably stashes
an error code in some flag that no one ever reads, with the same end result).
Consequently, nix-eval-jobs would waste resources by continuing to evaluate
even when the reader interested in the results has long gone away. Instead, use
writeToStdout, which throws an error on EPIPE, leading to the process
terminating as desired.
Reported-by: Winter <winter@winter.cafe>
Change-Id: I962c09bab582a8ed27dd41c01b1519876a6a6964
Right now, the worker will not receive interrupts, as it has no signal handler
thread due to the way it forks, but this will change soon. Prepare to handle
worker interruption properly, by letting the worker exit successfully in this
case and adjusting the coordinator to handle this exit gracefully.
Change-Id: I9f79670b0d2004e7e2d8bf36cf67108e6a6a6964
This also prevent `legacy` to return nothing as it's void.
Change-Id: Ic797544a59b04b41ad9e1c46af4cad3a1cb6fdc8
Signed-off-by: Raito Bezarius <raito@lix.systems>
When the maximum memory size is chosen too large (relative to the free memory
usage), so that the evaluation workers don't fit into the free memory, they
will start being swapped out. Further increase in memory usage will not be
reflected in the RSS, so they will not exit, and instead fill up all swap too
until the system runs out of memory.
The BDW-GC keeps track of its heap size, including any parts of it that may be
swapped out. For this reason it does not suffer from a similar problem. While
not all memory usage by the evaluation worker is accounted for by the GC, a
simple affine model for the overhead works reasonably well in practice. The
specific parameter values were determined using nixpkgs evaluation, and appear
to work quite well also for somewhat different workloads (like the `hydraJobs`
of Lix, which include a bunch of NixOS system configurations). So this is what
gets used.
When Lix is configured without the BDW-GC, of course this approach cannot be
used. In this case the old strategy is retained.
Change-Id: I6cb5f0a9b4ceda9dd14be165dda108cd6a6a6964
thunk values are shareable, and we can represent invalid/uninitialized
values with a special bit pattern that makes no sense otherwise. there
is no need to keep allocating values on the heap, instead we can treat
values like reference-counted smart pointers to heap objects, which in
turn lets us save a lot of allocations and, ultimately, gc heap space.
compared to our baseline (main of 2025-09-27) we save 15%+ memory on a
system rebuild and 17% on nix search. eval time regresses by ~3% for a
system rebuild, while nix search is 7% faster. further optimization is
probably possible (but for now this will just have to be good enough).
Change-Id: Ib6c47acdbe2fac4f76a83c2269f16f30ef66b2e1
with thunk state being shareable we no longer need to worry about value
uniqueness, only about value lifetime. this means we can liberally drop
indirections and allocations, passing references instead of pointers or
using stack memory instead of gc-managed memory for some intermediates.
Change-Id: I2d48a6fd57a376d544bd9bd2d05e5420611986d1
on its own this is not very useful, but having accessors for every value
kind is a prerequisite for doing smart things with Value than the union.
the net effect for now is only to add a few parentheses across the tree.
Change-Id: I88688ac09eb08495dad1eb221034ca540f094950
Closes#987
The patch adds a flag `--no-instantiate` which only performs evaluation
without instantiating any derivations. Hence, GC root creation is also
skipped. To achieve that, Lix is also put in read-only mode and all
operations that require reading a derivation (e.g. constituents or
listing input derivations) are disabled fallback values are set.
This is a port of an upstream PR[1]. Given the divergence of the
codebases (different restructurings on both ends, no more CA derivations)
I decided to redo large portions from scratch instead of
cherry-picking the patches. Hence, the authorship.
Additionally the clean up of casts down to a local store are removed or
guarded behind an if, as done in the upstream PR.
[1] https://github.com/nix-community/nix-eval-jobs/pull/379
Co-authored-by: Jörg Thalheim <joerg@thalheim.io>
Change-Id: Ib84f44e7799bc5577fd2ee98912458f16ebeab81
Some platforms like 32-Bit PowerPC need linking against libatomic.
Try to compile and link a very simple snippet of code which uses atomics
and make libatomic required if it fails.
Because we're using `dependency('atomic')`, the required meson versions
gets bumped to 1.7.0. See https://mesonbuild.com/Dependencies.html#atomic-stdatomic
Change-Id: I6a6a696471e1d352fb161c537ba9023b97c2d31e
Without https://github.com/NixOS/nixpkgs/pull/434761 evaluation of the
`nixpkgsLibTests` will fail in CI with recent enough Lix, due to reliance on
the TOML integer saturation bug.
Reported-by: Sergei Zimmerman <sergei@zimmerman.foo>
Change-Id: I6a6a6964838009d2c525f67035f84072fdfad988
printTaggedWarning already colorized its messages. we can do the same
for most other log messages.
Change-Id: Idcd31bbf4f8d0d703395b0d2b7b9bc33264d969f
luckily none of these a format strings vulnerabilities because
boost::format is smart enough to throw an exception when given
fewer format string arguments than are requested by specifiers
Change-Id: I5fa78f0d1396263271f6e1dbcee9c0b2e9e18c34
always use log macros, which also have the benefit of respecting the
verbosity setting without needing virtual function calls to read it.
Change-Id: I1c605562a53e54140724d5225e040abcf49ac996
notably this also includes the symbol table because it stores real
strings that are referenced by eval values, and an upcoming change
will make it impossible to share those strings with value strings.
Change-Id: I20a3644db8aa0850efe29630e0b73d424cb2aa56
We need capnproto-lix to be provided by callPackage, otherwise it won't
be spliced and the same (built for the cross host platform) for both
buildInputs (correct) and nativeBuildInputs (incorrect). We thus move it
into the overlay.
A similar problem exists for the lowdown build. We thus use callPackage
to override it as well. This is horrible (especially because it means we
need to pass the enableDarwinSandbox package argument through
ourselves), but at least it builds...
Fixes: https://git.lix.systems/lix-project/lix/issues/939
Change-Id: I802152072d852903401ef701f526195aa99475f2
Goals:
- Distribute reviews to people who can do the reviews
- Not prevent anything from getting done
- Allow giving away more commit access
Anti-goals:
- Silo people into particular areas
- Discourage contributing to any area
This was drafted by glancing at git logs. It is not likely to be very
accurate; the goal here is that we figure out a way to distribute
reviews to the right people.
Change-Id: I8be44bf7fdeca23da8099124eec7bc3a30e34627
That way we get a line of output per test completed,
which makes it more obvious it's actually doing things.
Change-Id: Ifbbe8bdf64e7178d3c59349cf071eb5a9d0fcd32
This fixes Meson's "Project does not target a minimum version but uses
feature introduced in '1.1': meson.options file" warning.
Silly Meson.
I also added a note in the top-level meson.build to indicate
`meson_version` is specified in more than one place.
Change-Id: I2c04278bb46a562a1c96cd2e5e4d9ce59ce8e125
now that we have no deferred hashes (since floating ca derivations were
the only way to create them) we can safely remove this enumeration too.
Change-Id: Ic72ed90500fcee7aa5b3b5a302477fa515acf1be
this mostly takes the form of removes feature checks and the associated
"ca derivations enabled" branches, but for the realisation info command
turns into a stub. we keep it around for compatibility, but from now on
it will always throws "ca derivations not implemented" errors when run.
Change-Id: I0abea5f76262013415330adcca2b498c6dca555b
Inspired by cl/3191 and
https://git.lix.systems/delroth/lix/commit/ae0247cbb4fc739ab013dc87d02e5f3191cf25ab.
`coerceToString` takes now an enumeration that lives in `value.hh`, this
enumeration is meant to represent increasing subsets of behaviors, e.g.
any level above Strict should do what the previous levels do and extra
behavior until `ToString`, which transforms many Nix values into an
arbitrary string representation, e.g. `null` to `""`.
Change-Id: Ief7a4756e8c0660e197623efebeaf07710746ec7
Signed-off-by: Raito Bezarius <raito@lix.systems>
Co-authored-by: Pierre Bourdon <delroth@gmail.com>
