Commit Graph
17506 Commits
Author SHA1 Message Date
Maximilian BoschandGerrit Code Review 596ffc290e Merge changes I88816cc1,Ib4ff5f03 into main
* changes:
  subprojects/nix-eval-jobs: fix gcroots
  subprojects/nix-eval-jobs: build tests in Hydra jobs
2025-02-26 18:51:08 +00:00
eldritch horrors 78f8ffefbf libstore: *always* delete tmpdirs during derivation goal destruction
deletePath checks for interruption, and when interrupted simply exits
without doing anything of note. this may be useful in some instances,
but in the case of a user interrupting a build running on a daemon it
leaves temporary directories sitting around until they are deleted by
the user or by some automated mechanism. we simply shouldn't do this.

note the this means that large build directories now cause the daemon
(and single-user processes) to not be interruptible during cleanup of
cancelled builds. we plan to address this by getting rid of this very
haphazard mechanism once asyncification is completed (hopefully soon)

fixes #678 (again)

Change-Id: Ifff3769620bc9ef7684bc84354aed6c7f932bd87
2025-02-26 17:24:59 +00:00
eldritch horrorsandGerrit Code Review ee49ed56c3 Merge "crash handling: remove catch-alls in main, crash hard" into main 2025-02-26 14:54:34 +00:00
eldritch horrors 11cde67e21 libstore: asyncify Store::addMultipleToStore source set
this will let us asyncify narFromPath, in some distant future.

Change-Id: I722718e434ee105d74aa14688829cca6c29ac442
2025-02-26 00:15:05 +00:00
eldritch horrors 492659714d libstore: account path copy progress with new source type
generators are nice, but they can't be made async. narFromPath will
eventually have to be made async. we just can't win with this stuff

Change-Id: Ia24727893be0001491e232ea27a09cc921b1c935
2025-02-26 00:15:05 +00:00
eldritch horrors f60a492884 libstore: pass source constructors to Store::addMultipleToStore
this makes explicit that activities are only valid while a path is being copied.

Change-Id: I8e26d898bf9e97ffefddfb41763626d5f996281a
2025-02-26 00:15:05 +00:00
eldritch horrors eeaf5ff909 libstore: add an async withFramedSink
Change-Id: I565606ee6e1a9cbd08bde930a1ddb07bf6ea56ef
2025-02-26 00:15:05 +00:00
eldritch horrors b6b7a087f4 libstore: de-generatorize RemoteStore::addMultipleToStore
with the only reason to put this into a source gone it's no longer needed.

Change-Id: Ic25ba3aa06f5b15c4ca5b80b0198f8b4ad5a6f91
2025-02-26 00:15:05 +00:00
eldritch horrors ea6f2214e6 libstore: remove Store::addMultipleToStore(Source&, ...)
only RemoteStore and the daemon loop ever call this method, and the
remote store call site isn't even particularly well done. we do not
have to create a bitstream, check whether our remote understands it
at all, only to parse it again if the remote doesn't understand it.
we can instead just use data we already have and avoid many copies.

Change-Id: Ib20740df16555269cbb19c7b12937280a00fc1f0
2025-02-26 00:15:05 +00:00
eldritch horrors bb0b1278a0 libstore: asyncify RemoteStore::addCAToStore
Change-Id: I9a491041e49d10d54b6c78d5a48871d32c430c52
2025-02-26 00:15:05 +00:00
Teo Camarasu ffe2dd40f4 doc: improve entry for nix-store --query --valid-derivers
Let's use similar terminology and formatting to the other entries,
and make it a bit clearer.

The information here is based on the changelog entry from when this
feature was added.

Change-Id: Ia859c710588180ca827c665a70650eb4db5a1c45
2025-02-25 16:46:51 +00:00
eldritch horrors 42a8fb9656 libstore: asyncify Realisation::closure
Change-Id: I8ed52455bea0e166098de9c77d55a2f843530d64
2025-02-25 02:09:22 +00:00
eldritch horrors c566a69e79 libstore: asyncify Store::query{,Partial}DerivationOutputMap
Change-Id: I1383f7986d454963409096501df47ddd4aa33602
2025-02-25 02:09:22 +00:00
Jade Lovelace 5687881a1f crash handling: remove catch-alls in main, crash hard
This is because C++ exceptions are very tricky, and most notably,
because a catch is run *after* things have been cleaned up, and we don't
*want* them cleaned up if we want usable stack traces.

    $ nix nar
    Lix crashed. This is a bug. We would appreciate if you report it along with what caused it at https://git.lix.systems/lix-project/lix/issues with the following information included:

    Exception: std::runtime_error: lol
    Stack trace:
     0# nix::getStackTrace[abi:cxx11]() in /home/jade/lix/lix4/outputs/out/lib/liblixutil.so
     1# 0x00007D58FC909292 in /home/jade/lix/lix4/outputs/out/lib/liblixmain.so
     2# 0x00007D58FC49220A in /nix/store/ybjcla5bhj8g1y84998pn4a2drfxybkv-gcc-13.3.0-lib/lib/libstdc++.so.6
     3# 0x00007D58FC492275 in /nix/store/ybjcla5bhj8g1y84998pn4a2drfxybkv-gcc-13.3.0-lib/lib/libstdc++.so.6
     4# 0x00007D58FC4924C7 in /nix/store/ybjcla5bhj8g1y84998pn4a2drfxybkv-gcc-13.3.0-lib/lib/libstdc++.so.6
     5# 0x00005B9799B45036 in nix
     6# 0x00005B9799B3697F in nix
     7# 0x00005B9799B3924D in nix
     8# nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>) in /home/jade/lix/lix4/outputs/out/lib/liblixmain.so
     9# 0x00005B9799B38D2B in nix
    10# 0x00007D58FC1E227E in /nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/libc.so.6
    11# __libc_start_main in /nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/libc.so.6
    12# 0x00005B9799A7CF05 in nix

    zsh: IOT instruction (core dumped)  nix nar

Sadly our symbolizer sucks. But that's, well, improvable. The core dump
is at least much more useful now.

Obviously with async things being full of catch-alls, we have the same
problem. I think we might want to change those in another commit to only
catch nix::BaseError subclasses and let the rest fall into a noexcept
function so it will crash hard.

The daemon also could use this treatment, but I would like to be able to
send the client the crash message in phase 1 unwind. No idea how to
achieve that; all the ways I have seen are heinous.

Fixes: https://git.lix.systems/lix-project/lix/issues/698

Change-Id: I0b61f2e7a7003b92c39b1a0e90756979e5e59cb4
2025-02-24 17:26:49 -08:00
Raito BezariusandGerrit Code Review 446af4c6fe Merge "flake: Add support for x86_64-freebsd." into main 2025-02-24 23:12:12 +00:00
Raito Bezarius 55cbabd316 feat(libstore/build): always create a new cgroup namespace
There's no reason to make it dependent upon `Xp::Cgroups`, this
increases the purity of the builders' views.

Change-Id: I3d934428177e6bb78cd700d1cce1ded8a88e9da0
Signed-off-by: Raito Bezarius <raito@lix.systems>
2025-02-25 00:00:48 +01:00
Artemis TosiniandRaito Bezarius f42d6f0b52 flake: Add support for x86_64-freebsd.
Now that we're using nixpkgs 24.11 it's possible to build cross
for FreeBSD from linux, both in a derivation and a devShell.

Change-Id: I06fa257023397a97cdc72af51170e8523e9f1bfb
2025-02-24 23:54:04 +01:00
piegamesandGerrit Code Review 945e674bc8 Merge "libexpr: Refactor formals handling" into main 2025-02-24 18:28:52 +00:00
eldritch horrors ea6ba6a460 libstore: asyncify Store::queryDerivationOutputs
Change-Id: Ie9f67ff0b9193978021d58afb2bf2758288215f7
2025-02-24 15:09:02 +00:00
eldritch horrors acd7916af4 libstore: asyncify LogStore::addBuildLog
Change-Id: Idfbeecc9ad8ecba53eea25cbbee65bf85a550e79
2025-02-24 15:09:02 +00:00
eldritch horrors 63a96b1d31 libstore: asyncify LogStore::getBuildLogExact
Change-Id: Ie8492c2a0dbdbaeebc0d1608ad461f3e7a7f5b52
2025-02-24 15:09:02 +00:00
eldritch horrors 719f58b930 libstore: asyncify Store::topoSortPaths
Change-Id: Iaad42a6f021434a5f160c88ca7c8afe6ff280806
2025-02-24 15:09:02 +00:00
eldritch horrors e6fee559f7 libmain: asyncify printMissing
Change-Id: Iac87ecbbab4c65ddd6e3a32bdb64107ed7f9b3c0
2025-02-24 15:09:02 +00:00
eldritch horrors 4042d9c167 libstore: asyncify copyStorePath
Change-Id: I846bc5c95b54313594b2714981bb7434dbca96b7
2025-02-24 15:09:02 +00:00
eldritch horrors 381eca9947 libstore: asyncify Store::makeValidityRegistration
Change-Id: Ic1c31240d73db6783842664fbfa9607401a3b4a7
2025-02-24 15:09:02 +00:00
eldritch horrors 3cdf9fed2f libstore: asyncify Store::init
Change-Id: I38d59b40ab806996018f0ed17896e3ca9dc303be
2025-02-24 15:09:02 +00:00
eldritch horrors c6f5a6a196 perl: fix computeFSClosure
we must await the promise. also add a Werror argument to not break this again.

Change-Id: Ia02fee1720c0280853bc299d3d20301370d93e56
2025-02-24 15:09:02 +00:00
piegames 2e5239d2e5 libexpr: Refactor formals handling
This is kind of a painful change, because a lot of code grew around the
bad abstraction, but the goal here is to abstract formals in a way that
allows adding other means of pattern matching / argument destructuring
in the future.

Change-Id: If4e681a4be3d1f42ceea81a8e07297f0d08acc80
2025-02-24 13:48:29 +01:00
piegames 0928d4d87a flakes: Make checkOverlay less strict
I'm sorry but checking the name of the function arguments to be "final"
and "prev" is just absurd. That's like worst of
type-checking-meets-static-analysis. And the error message of the second
`throw` was not even good, as it mixed all three failure cases together.

Change-Id: Id4244171123dd8a228be71ce9f04d8e9f647c111
2025-02-24 12:54:48 +01:00
Raito BezariusandGerrit Code Review 43b0902390 Merge "feat: add support for external lix- prefixed commands in the CLI" into main 2025-02-24 10:51:27 +00:00
eldritch horrors b6dadd5cf0 libstore: asyncify Store::queryValidPaths
Change-Id: I3a3303b288321cd812f4df87c80a676f997e57b9
2025-02-23 17:18:48 +00:00
eldritch horrors f09f7c88a5 libutil: allow async threadpool waiting
Change-Id: I133e7390ebe05120e0de18d58a1c743c6c2ba973
2025-02-23 17:18:48 +00:00
eldritch horrors a4e7cfdb57 libutil: make PushActivity movable
while not copy-assignable it was still copy-constructible, which can
lead to some very unfortunate interactions with closure captures. we
will also need to move current-activity handles in upcoming changes.

Change-Id: I63cede83b9790820c6bd8784fb5d47e5246b7c4a
2025-02-23 17:18:48 +00:00
eldritch horrors f9a05605c1 libstore: asyncify resolveDerivedPath
Change-Id: I4606cdc9c935a4b40f68f12d742bc1e40d35b310
2025-02-23 17:18:48 +00:00
eldritch horrors 91387bbb13 libstore: asyncify Store::computeFSClosure
Change-Id: I79d3e9c60886a9eb58051a28cdda25dadcead9f1
2025-02-23 17:18:48 +00:00
eldritch horrors f06040f5d8 libutil: add computeClosureAsync
Change-Id: I5d9a11a106e9e0d9e49d08d1d04f579ba55255a9
2025-02-23 17:18:48 +00:00
eldritch horrors cc3a9897e3 libstore: asyncify printClosureDiff
Change-Id: I4982d34e565f9e59931c4e675c2d142fcacce81f
2025-02-23 17:18:48 +00:00
eldritch horrors bc0e0786f0 libstore: asyncify Store::getClosureSize
Change-Id: I625c92f94e2a77b64569d385d486434ae41df2da
2025-02-23 17:18:48 +00:00
eldritch horrors bd8272faec libstore: asyncify RealisedPath::closure
Change-Id: Ibc81c4cd504665410542274d2ffe97c72c347ac2
2025-02-23 17:18:48 +00:00
eldritch horrors a514e22166 libstore: asyncify a bunch of toJSON methods
why these are allowed to do IO that can end up blocking is beyond us

Change-Id: I9718900b016b3c0755d5bb668a5c270546cc4b17
2025-02-23 17:18:48 +00:00
eldritch horrors 990b3b0cfc libstore: asyncify Derivation::tryResolve
Change-Id: Id00e1d2a2316ab1daafcd308ee5e48c36e9c3a43
2025-02-23 17:18:48 +00:00
eldritch horrors 645a90b3e2 libstore: asyncify GcStore::findRoots
also includes LocalStore methods findRoots, findRootsNoTemp, findRuntimeRoots

Change-Id: I721cfabaf9a7bf44ac8af513a53cd4cd01109e53
2025-02-23 17:18:48 +00:00
eldritch horrors b33214a643 libstore: asyncify GcStore::collectGarbage
Change-Id: I0406444b518f66c22eb043fcbc7de90122974c42
2025-02-23 17:18:48 +00:00
Raito Bezarius 5f50741ce9 feat: add support for external lix- prefixed commands in the CLI
This feature allows calling external binaries starting with `lix-` as
subcommands directly within the `lix` CLI, similar to how Git handles
external commands.

For example, a binary named `lix-example` can now be invoked using `lix
example`.

This change opens up the possibility for broader community-driven
extensions of the Lix CLI, offering flexibility that Flakes has somewhat
restricted by centralizing certain features around its own model of
dependency management.

By enabling users to introduce custom subcommands, we encourage
experimentation and diverse workflows, addressing the needs of those who
seek more modular, feature-rich approaches to package management and
system configuration beyond what Flakes currently offers [1] [2] [3].

This adds `lix` as a new binary to be able to use this feature.

[1]: https://github.com/nmattia/niv
[2]: https://github.com/andir/npins
[3]: https://github.com/nikstur/lon

Change-Id: Ic6344424a6a46fc9fb30432f00e21c3509659f8a
Signed-off-by: Raito Bezarius <raito@lix.systems>
2025-02-23 15:24:28 +01:00
eldritch horrors aaab224bea libstore: kill sandboxes before removing paths
otherwise we may end up trying to delete a temporary directory that is
still actively being written to if a derivation is killed due to other
derivations failing or the build is aborted due to a user interruption

fixes #678

Change-Id: I4678723f11bdd261b27af8784d040a787690b107
2025-02-23 13:41:37 +00:00
Raito BezariusandGerrit Code Review c64a9db6bb Merge "doc/authors: add lilyinstarlight" into main 2025-02-23 09:47:26 +00:00
jadeandGerrit Code Review 33f13385f6 Merge changes Iba04f70f,I6b5bd980 into main
* changes:
  rl-next: fix mistake in issue number
  crash handling: log to syslog also
2025-02-23 07:56:32 +00:00
jadeandGerrit Code Review 595778e2d2 Merge "releng: use aws s3 sync --delete when uploading manual" into main 2025-02-23 02:37:05 +00:00
Jade Lovelace 392ba80bbd rl-next: fix mistake in issue number
Should be fj#foo for a lix issue.

Change-Id: Iba04f70fd8258e5c54a2c5b92a41472f8fb68376
2025-02-22 18:33:51 -08:00
Jade Lovelace 5468d340c3 crash handling: log to syslog also
This is mostly to deal with fatal errors in NixOS tests, but it's useful
for all cases; assertion failures landing in syslog would be nice too,
but that's not as easy to do.

CC: https://git.lix.systems/lix-project/lix/issues/693

Change-Id: I6b5bd9800db3f5affe33f2946ea218e4a78f0c07
2025-02-22 18:33:51 -08:00