Commit Graph
17471 Commits
Author SHA1 Message Date
eldritch horrors a4e7cfdb57 libutil: make PushActivity movable
while not copy-assignable it was still copy-constructible, which can
lead to some very unfortunate interactions with closure captures. we
will also need to move current-activity handles in upcoming changes.

Change-Id: I63cede83b9790820c6bd8784fb5d47e5246b7c4a
2025-02-23 17:18:48 +00:00
eldritch horrors f9a05605c1 libstore: asyncify resolveDerivedPath
Change-Id: I4606cdc9c935a4b40f68f12d742bc1e40d35b310
2025-02-23 17:18:48 +00:00
eldritch horrors 91387bbb13 libstore: asyncify Store::computeFSClosure
Change-Id: I79d3e9c60886a9eb58051a28cdda25dadcead9f1
2025-02-23 17:18:48 +00:00
eldritch horrors f06040f5d8 libutil: add computeClosureAsync
Change-Id: I5d9a11a106e9e0d9e49d08d1d04f579ba55255a9
2025-02-23 17:18:48 +00:00
eldritch horrors cc3a9897e3 libstore: asyncify printClosureDiff
Change-Id: I4982d34e565f9e59931c4e675c2d142fcacce81f
2025-02-23 17:18:48 +00:00
eldritch horrors bc0e0786f0 libstore: asyncify Store::getClosureSize
Change-Id: I625c92f94e2a77b64569d385d486434ae41df2da
2025-02-23 17:18:48 +00:00
eldritch horrors bd8272faec libstore: asyncify RealisedPath::closure
Change-Id: Ibc81c4cd504665410542274d2ffe97c72c347ac2
2025-02-23 17:18:48 +00:00
eldritch horrors a514e22166 libstore: asyncify a bunch of toJSON methods
why these are allowed to do IO that can end up blocking is beyond us

Change-Id: I9718900b016b3c0755d5bb668a5c270546cc4b17
2025-02-23 17:18:48 +00:00
eldritch horrors 990b3b0cfc libstore: asyncify Derivation::tryResolve
Change-Id: Id00e1d2a2316ab1daafcd308ee5e48c36e9c3a43
2025-02-23 17:18:48 +00:00
eldritch horrors 645a90b3e2 libstore: asyncify GcStore::findRoots
also includes LocalStore methods findRoots, findRootsNoTemp, findRuntimeRoots

Change-Id: I721cfabaf9a7bf44ac8af513a53cd4cd01109e53
2025-02-23 17:18:48 +00:00
eldritch horrors b33214a643 libstore: asyncify GcStore::collectGarbage
Change-Id: I0406444b518f66c22eb043fcbc7de90122974c42
2025-02-23 17:18:48 +00:00
eldritch horrors aaab224bea libstore: kill sandboxes before removing paths
otherwise we may end up trying to delete a temporary directory that is
still actively being written to if a derivation is killed due to other
derivations failing or the build is aborted due to a user interruption

fixes #678

Change-Id: I4678723f11bdd261b27af8784d040a787690b107
2025-02-23 13:41:37 +00:00
Raito BezariusandGerrit Code Review c64a9db6bb Merge "doc/authors: add lilyinstarlight" into main 2025-02-23 09:47:26 +00:00
jadeandGerrit Code Review 33f13385f6 Merge changes Iba04f70f,I6b5bd980 into main
* changes:
  rl-next: fix mistake in issue number
  crash handling: log to syslog also
2025-02-23 07:56:32 +00:00
jadeandGerrit Code Review 595778e2d2 Merge "releng: use aws s3 sync --delete when uploading manual" into main 2025-02-23 02:37:05 +00:00
Jade Lovelace 392ba80bbd rl-next: fix mistake in issue number
Should be fj#foo for a lix issue.

Change-Id: Iba04f70fd8258e5c54a2c5b92a41472f8fb68376
2025-02-22 18:33:51 -08:00
Jade Lovelace 5468d340c3 crash handling: log to syslog also
This is mostly to deal with fatal errors in NixOS tests, but it's useful
for all cases; assertion failures landing in syslog would be nice too,
but that's not as easy to do.

CC: https://git.lix.systems/lix-project/lix/issues/693

Change-Id: I6b5bd9800db3f5affe33f2946ea218e4a78f0c07
2025-02-22 18:33:51 -08:00
eldritch horrors dd2241ea82 libcmd: asyncify getBuiltPath
Change-Id: I301ab541c0367100f8a68041d482d4cb326fc028
2025-02-22 23:24:26 +00:00
eldritch horrors 8fc0e0d24b libstore: asyncify Store::exportPath{,s}
Change-Id: I92915a3cb27bda73e7b9f3b32dc1feea3f47abda
2025-02-22 23:24:26 +00:00
eldritch horrors a2e523aa8b libstore: asyncify BinaryCacheStore::addToStoreCommon
Change-Id: I46710b11547515615f719e74501c1c8b4640ecc3
2025-02-22 23:24:26 +00:00
Sebastian Widua 82c7e76c9c libstore: allow specifying port for ssh[-ng] stores
This also adds documentation for it in distributed-builds.md as that's
possibly the most common use-case for remote ssh stores

Support for `std::optional<uint16_t>` in Setting is also added since
setting a port is optional.

The line `#include "lix/libutil/strings.hh"` fixes that templates instanciations
in lix/libutil/config-impl.hh were using string utils without including
the header (why are they even there btw)

Change-Id: Id806c117c48cdf158d9d1cb1e639b0df31d9bf11
2025-02-22 20:37:45 +01:00
eldritch horrors 044b4c4500 libutil: split retrySQLite into sync and async
also mark the sync version as NeverAsync. a blocking wait on sqlite
locks in a coroutine may never finish if it's a different coroutine
on the same executor that is holding the lock, not another process.

this propagates to the sqlite core interface, but no further. we'll
assume that caches do not block on a database for very long, and we
can't reasonably propagate never-async-ness out of stores unless we
touch everything we'd touch for the async transition, again, twice.
store code already assumes that it can block for however long it'll
feel like that moment. we keep thread pools around for this reason.

Change-Id: I62f77e1ac333cbe2e4e646dbcb1571463f2cf3fc
2025-02-22 17:59:54 +00:00
eldritch horrors 184922de19 libstore: drop support for db schema version 7
the upgrade to version 7 was done in 2013. it's about time.

Change-Id: I4520cfd1a13b693259ae71710d04f42a1303eb5d
2025-02-22 16:38:12 +00:00
jadeandGerrit Code Review 148f4eefe9 Merge changes Ia8ac6af0,I9f478750,I7d594835 into main
* changes:
  deps: backport capnproto promise nodiscard PR
  libstore: await promises instead of discarding
  libutil: fix some thread pool bugs
2025-02-22 01:47:10 +00:00
Rebecca Turner 16df34b295 releng: use aws s3 sync --delete when uploading manual
For the user manual, we don't delete things that are missing when doing
aws s3 sync. This doesn't seem wise. If you delete a page in the manual,
it will stay public and visible.

This adds `--delete` to the `aws s3 sync` commands to fix this issue.

Closes #396

Change-Id: I6d7fb97bcdab96c0115d6c66fea0310125207df4
2025-02-21 11:45:22 -08:00
jadeandGerrit Code Review e7aa55ca34 Merge "nix-eval-jobs: passthru nix" into main 2025-02-21 02:35:53 +00:00
Lunaphied 61e1a06aeb hotfix for releng scripts, fixes #440
Change-Id: I33b29fbf3920a2d4fb53b58db477dff5bf7a1af1
2025-02-21 01:02:04 +00:00
eldritch horrors 0eb1164a8d deps: backport capnproto promise nodiscard PR
this does not change the API exposed by the library, so it'll be safe
for users to substitute stock capnp. we do want it for our own builds
though, at least until we can switch to capnp 2 or fully go for rust.

Change-Id: Ia8ac6af0165f6f61ee02345179dc0a93a0889a06
2025-02-21 00:35:11 +01:00
eldritch horrors 3644b519a9 libstore: await promises instead of discarding
oops.

Change-Id: I9f47875085500fa392f74ee51293f9a95c455997
2025-02-21 00:35:11 +01:00
eldritch horrors e47bff547a libutil: fix some thread pool bugs
4138fc7622 mistakenly removed an early
exit from non-main worker threads. this led to exceptions not ending
`process()` calls in a timely manner and instead draining the entire
work queue first, which for e.g. Interrupted errors would cause many
duplicated reports per worker thread instead of only one per thread.

it also did not properly rethrow a work item exception in all cases,
e.g. when all work had completed by the time `process()` was called.

due to a mistake in the thread starting check it was possible that a
system would require n² work items to start n worker threads if some
work items process quickly enough while other items block for a bit.

Change-Id: I7d59483580cda1c19980f9296074216a0fbf2c4e
2025-02-21 00:35:11 +01:00
Linus Heckemann 6eda9a95a4 nix-eval-jobs: passthru nix
Change-Id: Ied6d4abb8a67dcbe221271e6be1a14b71f9543d1
2025-02-20 22:36:53 +01:00
Lunaphied e335a26d5c make nix store sign use a thread pool like nix store copy-sigs does
Fixes #399.

Change-Id: I0571036362eaf52a2efad99ba07788d7e3fd7f35
2025-02-20 04:28:28 +00:00
Lily BallardandGerrit Code Review df336747e6 Merge "libstore: use BEGIN IMMEDIATE SQLite transactions" into main 2025-02-19 04:51:39 +00:00
jadeandGerrit Code Review e4650caa69 Merge changes I75b828ae,I4baa5721,I68798300 into main
* changes:
  fix: pre-build-hook did not run when useChroot == false
  libstore/build: say what was failing when pre/post build hooks fail
  daemon: complain much louder about unknown std::exception instances falling out
2025-02-18 17:02:09 +00:00
Teo CamarasuandGerrit Code Review 8908fbdf37 Merge "Avoid lix daemon killing unrelated processes when using sandboxes under Linux" into main 2025-02-18 16:47:02 +00:00
Teo Camarasu ab57463df4 Avoid lix daemon killing unrelated processes when using sandboxes under Linux
The lix daemon wants to avoid orphan processes outliving a lix build.
In order to do that it kills all processes under the build user's UID
after and before a build.

When using sandboxes under Linux, this is unecessary, as builds are run
inside a PID namespace, which guarantees that processes cannot outlive
the "init" process.

Partially fixes https://git.lix.systems/lix-project/lix/issues/667

Change-Id: Idb2cdaad30169b75d730e8a18b360330516faf8b
2025-02-18 15:59:59 +00:00
Lily Ballard 98ccb179f9 libstore: use BEGIN IMMEDIATE SQLite transactions
Upgrading a read transaction to a write transaction will immediately
fail if another connection/process started a write transaction in the
meantime. For transactions that we expect to perform a write after a
read, starting with `BEGIN IMMEDIATE` gives us a write transaction
immediately instead of beginning with a read transaction and upgrading
it.

Change-Id: I82b8a3843382940a4fa4ffe45f30b365b690ff86
2025-02-17 19:07:59 -08:00
Lily BallardandGerrit Code Review 0069c59eda Merge "libstore: use monotonic clock for SQLITE_BUSY warnings" into main 2025-02-18 02:44:28 +00:00
Jade Lovelace c9b2e8b1d9 fix: pre-build-hook did not run when useChroot == false
It turns out there is actually some history here.
Eight years ago, in 7f5b750b40, the
pre-build-hook condition got changed to check that useChroot is true
(which we will put down as "eelco making a mistake" because it makes no
sense) and that this->drv was a Derivation (as opposed to a
BasicDerivation as would be the case if the derivation was not present
on disk). The intent of this change was that pre-build-hook would not
run when the derivation doesn't exist, so that some hydras would not
explode.

However, this broke later when both cases became Derivation such that
AFAICT it will always run it if useChroot is true, which is absolutely
not the intended behaviour, but it is a reasonable interpretation as
well, just inconvenient for certain pre-build-hook usages.

So, I think the safest and most compatible behaviour is to run the
pre-build-hook but not guarantee that the derivation actually exists on
disk, since it already didn't run in many cases. Maybe the more correct
way is to actually write out the derivation if we are running a
pre-build-hook, but post-build-hook has exactly the same situation and
it's expected, so idk.

Fixes: https://git.lix.systems/lix-project/lix/issues/674
Change-Id: I75b828ae2a07ab373f44083507eb243459bd081d
2025-02-17 16:05:06 -08:00
Jade Lovelace 24a6759940 libstore/build: say what was failing when pre/post build hooks fail
Fixes: https://git.lix.systems/lix-project/lix/issues/638
Change-Id: I4baa5721e79b425c37b9e7057ba4989003bced82
2025-02-17 15:39:53 -08:00
Jade Lovelace 8a5d1c45d2 daemon: complain much louder about unknown std::exception instances falling out
This made debugging https://git.lix.systems/lix-project/lix/issues/681 a
pain in the neck. This is partially a fix, in a certain sense, for
https://git.lix.systems/lix-project/lix/issues/379, but that one also
addresses expected exceptions from the daemon.

Another instance of this error generation site being shit recently:
https://git.lix.systems/lix-project/lix/issues/638

I don't know how we should improve that particular site but we
definitely should complain about uncaught std exceptions with type ids.

Change-Id: I68798300448ee9ebae65c6469ba69e3f933b2895
2025-02-17 14:42:35 -08:00
Charles HallandGerrit Code Review c68457a906 Merge "document the behavior of negative substring len" into main 2025-02-17 21:48:39 +00:00
Charles HallandGerrit Code Review 8238e9dc1a Merge "use sri hash format in fetch{url,Tarball} errors" into main 2025-02-17 21:48:10 +00:00
Charles Hall c8cf70b601 use sri hash format in fetch{url,Tarball} errors
Specifically, for errors where the provided and expected hashes differ.

Also, I've updated `fetchTarball`'s docs to explain the allowed values
for the `sha256` attribute.

Change-Id: Ic9aa777dc0a3c1ccdb4c98048745067f529bd986
2025-02-17 12:04:16 -08:00
Linus HeckemannandGerrit Code Review 02c12314e4 Merge "libstore: more precise origin info for strange structured logs" into main 2025-02-17 16:28:19 +00:00
Lily Ballard ed03c3985d libstore: use monotonic clock for SQLITE_BUSY warnings
Using time_t meant a warning could be printed arbitrarily soon depending
on when the second rolled over relative to when the SQLite transaction
was attempted.

Change-Id: I93401aea65f90e37cd450a293c4dd4d64fba7628
2025-02-16 22:48:45 -08:00
Charles Hall 93a4f0e82d document the behavior of negative substring len
Change-Id: I0777580a3aa374400586b76afdc74da5b5f6645a
2025-02-16 15:18:19 -08:00
eldritch horrors d745b28500 libstore: make LocalStore locking async-compatible
the two instances that used synchronous locking from promises weren't in
danger of deadlocks: both only serialized access to the sqlite database.

Change-Id: I2041a7a31f39cffde2080d0bb2402e2ead8d953f
2025-02-16 22:47:06 +00:00
eldritch horrors 5b98b75f74 libstore: extend retrySQLite for promise blocks
Change-Id: I91c8aad9e49ba3d470c5d6270b4e23da2501a0b8
2025-02-16 22:20:33 +00:00
eldritch horrors 415075884c libcmd: asyncify BuiltPath::toRealisedPaths
Change-Id: Ie819879b00db57c90927a7fbc833923718050856
2025-02-16 15:00:10 +00:00