Commit Graph
17439 Commits
Author SHA1 Message Date
Lunaphied e335a26d5c make nix store sign use a thread pool like nix store copy-sigs does
Fixes #399.

Change-Id: I0571036362eaf52a2efad99ba07788d7e3fd7f35
2025-02-20 04:28:28 +00:00
Lily BallardandGerrit Code Review df336747e6 Merge "libstore: use BEGIN IMMEDIATE SQLite transactions" into main 2025-02-19 04:51:39 +00:00
jadeandGerrit Code Review e4650caa69 Merge changes I75b828ae,I4baa5721,I68798300 into main
* changes:
  fix: pre-build-hook did not run when useChroot == false
  libstore/build: say what was failing when pre/post build hooks fail
  daemon: complain much louder about unknown std::exception instances falling out
2025-02-18 17:02:09 +00:00
Teo CamarasuandGerrit Code Review 8908fbdf37 Merge "Avoid lix daemon killing unrelated processes when using sandboxes under Linux" into main 2025-02-18 16:47:02 +00:00
Teo Camarasu ab57463df4 Avoid lix daemon killing unrelated processes when using sandboxes under Linux
The lix daemon wants to avoid orphan processes outliving a lix build.
In order to do that it kills all processes under the build user's UID
after and before a build.

When using sandboxes under Linux, this is unecessary, as builds are run
inside a PID namespace, which guarantees that processes cannot outlive
the "init" process.

Partially fixes https://git.lix.systems/lix-project/lix/issues/667

Change-Id: Idb2cdaad30169b75d730e8a18b360330516faf8b
2025-02-18 15:59:59 +00:00
Lily Ballard 98ccb179f9 libstore: use BEGIN IMMEDIATE SQLite transactions
Upgrading a read transaction to a write transaction will immediately
fail if another connection/process started a write transaction in the
meantime. For transactions that we expect to perform a write after a
read, starting with `BEGIN IMMEDIATE` gives us a write transaction
immediately instead of beginning with a read transaction and upgrading
it.

Change-Id: I82b8a3843382940a4fa4ffe45f30b365b690ff86
2025-02-17 19:07:59 -08:00
Lily BallardandGerrit Code Review 0069c59eda Merge "libstore: use monotonic clock for SQLITE_BUSY warnings" into main 2025-02-18 02:44:28 +00:00
Jade Lovelace c9b2e8b1d9 fix: pre-build-hook did not run when useChroot == false
It turns out there is actually some history here.
Eight years ago, in 7f5b750b40, the
pre-build-hook condition got changed to check that useChroot is true
(which we will put down as "eelco making a mistake" because it makes no
sense) and that this->drv was a Derivation (as opposed to a
BasicDerivation as would be the case if the derivation was not present
on disk). The intent of this change was that pre-build-hook would not
run when the derivation doesn't exist, so that some hydras would not
explode.

However, this broke later when both cases became Derivation such that
AFAICT it will always run it if useChroot is true, which is absolutely
not the intended behaviour, but it is a reasonable interpretation as
well, just inconvenient for certain pre-build-hook usages.

So, I think the safest and most compatible behaviour is to run the
pre-build-hook but not guarantee that the derivation actually exists on
disk, since it already didn't run in many cases. Maybe the more correct
way is to actually write out the derivation if we are running a
pre-build-hook, but post-build-hook has exactly the same situation and
it's expected, so idk.

Fixes: https://git.lix.systems/lix-project/lix/issues/674
Change-Id: I75b828ae2a07ab373f44083507eb243459bd081d
2025-02-17 16:05:06 -08:00
Jade Lovelace 24a6759940 libstore/build: say what was failing when pre/post build hooks fail
Fixes: https://git.lix.systems/lix-project/lix/issues/638
Change-Id: I4baa5721e79b425c37b9e7057ba4989003bced82
2025-02-17 15:39:53 -08:00
Jade Lovelace 8a5d1c45d2 daemon: complain much louder about unknown std::exception instances falling out
This made debugging https://git.lix.systems/lix-project/lix/issues/681 a
pain in the neck. This is partially a fix, in a certain sense, for
https://git.lix.systems/lix-project/lix/issues/379, but that one also
addresses expected exceptions from the daemon.

Another instance of this error generation site being shit recently:
https://git.lix.systems/lix-project/lix/issues/638

I don't know how we should improve that particular site but we
definitely should complain about uncaught std exceptions with type ids.

Change-Id: I68798300448ee9ebae65c6469ba69e3f933b2895
2025-02-17 14:42:35 -08:00
Charles HallandGerrit Code Review c68457a906 Merge "document the behavior of negative substring len" into main 2025-02-17 21:48:39 +00:00
Charles HallandGerrit Code Review 8238e9dc1a Merge "use sri hash format in fetch{url,Tarball} errors" into main 2025-02-17 21:48:10 +00:00
Charles Hall c8cf70b601 use sri hash format in fetch{url,Tarball} errors
Specifically, for errors where the provided and expected hashes differ.

Also, I've updated `fetchTarball`'s docs to explain the allowed values
for the `sha256` attribute.

Change-Id: Ic9aa777dc0a3c1ccdb4c98048745067f529bd986
2025-02-17 12:04:16 -08:00
Linus HeckemannandGerrit Code Review 02c12314e4 Merge "libstore: more precise origin info for strange structured logs" into main 2025-02-17 16:28:19 +00:00
Lily Ballard ed03c3985d libstore: use monotonic clock for SQLITE_BUSY warnings
Using time_t meant a warning could be printed arbitrarily soon depending
on when the second rolled over relative to when the SQLite transaction
was attempted.

Change-Id: I93401aea65f90e37cd450a293c4dd4d64fba7628
2025-02-16 22:48:45 -08:00
Charles Hall 93a4f0e82d document the behavior of negative substring len
Change-Id: I0777580a3aa374400586b76afdc74da5b5f6645a
2025-02-16 15:18:19 -08:00
eldritch horrors d745b28500 libstore: make LocalStore locking async-compatible
the two instances that used synchronous locking from promises weren't in
danger of deadlocks: both only serialized access to the sqlite database.

Change-Id: I2041a7a31f39cffde2080d0bb2402e2ead8d953f
2025-02-16 22:47:06 +00:00
eldritch horrors 5b98b75f74 libstore: extend retrySQLite for promise blocks
Change-Id: I91c8aad9e49ba3d470c5d6270b4e23da2501a0b8
2025-02-16 22:20:33 +00:00
eldritch horrors 415075884c libcmd: asyncify BuiltPath::toRealisedPaths
Change-Id: Ie819879b00db57c90927a7fbc833923718050856
2025-02-16 15:00:10 +00:00
eldritch horrors 993cc99f0d libstore: asyncify drvOutputReferences
Change-Id: I68bc77eadb1ddf3d0e257e773f8b9b29289ea2bc
2025-02-16 15:00:10 +00:00
eldritch horrors 3b9020d37c libstore/libfetchers: do more async lazy init
while these two are currently safe using synchronous mutexes we can't
easily guarantee this forever. since we have async mutexes too now we
may as well use them; the overhead they incur is small enough that we
can just ignore it here. this can be revisited later, when necessary.

Change-Id: Ia4cb2361d5638aa48e3b982a05e75ae9c4f254cd
2025-02-16 15:00:10 +00:00
eldritch horrors 7cb28186f2 libutil: add promise-compatible Sync specialization
we can't use Sync with stl mutex types since yielding from one coroutine
to another while holding a lock is a deadlock hazard. if we don't expose
blocking lock operations from Sync though and emulate blocking with more
promises we can reuse most of the Sync code instead of rewriting it all.
hiding them completely would make it impossible to migrate current users
to promise-based code though, so we use our new NeverAsync lint instead.

Change-Id: I7eebf18e29473963040e7448da27c52012f1f134
2025-02-16 15:00:10 +00:00
eldritch horrors 7f95f9a85c libstore: mark blocking file locks as never async
Change-Id: Icf4633b5cd540cb777004c3dce6aa9d79a7f4063
2025-02-16 14:59:32 +00:00
eldritch horrors e12deec9fb clang-tidy: add lint to forbid blocking code
Change-Id: I3df6f67f363b0a5ac014bf817e0b35fde398827f
2025-02-16 14:16:15 +00:00
Robert HensingandLinus Heckemann 66da3fd63f libstore: more precise origin info for strange structured logs
Robert made these improvements on the PR that cherry-picked
e55cd3beea /
https://gerrit.lix.systems/c/lix/+/2057
from Lix into CppNix:
https://github.com/NixOS/nix/pull/11921

Push log source description out of libutil and report build hook @nix warning correctly

(cherry picked from commit 03d4bfd852dce9a050f984e887c887a43581796c)

test: Move unusual-logging to run only in logging test case

(cherry picked from commit 1421420e862434321c46511a3152016e443dd479)

Remove redundant warning: prefix from structured build log warning

(cherry picked from commit f3c722cab24f7a0de8c3573d25e91749f4f16234)
Change-Id: I7da99046f2a41b3c58e62351119bc89bcc25a703
2025-02-16 14:15:36 +01:00
eldritch horrors 1077bc626e libstore: use lockPathAsync in more places
these were found by an upcoming lint.

Change-Id: I4647baa2c0971f3007d348e8ced6f75e45037a20
2025-02-14 15:28:50 +00:00
eldritch horrors 65ee7f5da4 libstore: use non-blocking locks in createTempDirInStore
gc tryLocks first, then deletes. we can use the same order since there's
no sequence of operations that would let us lock our new directory *and*
have the gc delete it (except outright bugs). the worst possible outcome
is that we create a few directories that'll be deleted by a later gc run

Change-Id: I6d3524f4990f8deee10804e6fbb0d0a26f3d57fe
2025-02-14 15:28:50 +00:00
eldritch horrors 81275c9ad7 libstore: asyncify Store::addTempRoot
Change-Id: Ie3ef2a637d54b22f0bf9c8aba14d2f7546c6cd9a
2025-02-14 15:28:50 +00:00
eldritch horrors 84dd5864f3 libstore: simplify LocalStore::createTempRootsFile
we now create a temporary file, lock it, move it into place, and return.
if any of these steps (except creation) fails the GC must have found and
deleted our new file before we got the chance to use it. we don't need a
lock around file creation anymore, we especially don't need a mutex that
covers an flock call, and we don't need the GC to clobber what it finds.
(clobbering *is* kept for compatibility with older nix implementations.)

if we ever leave trash around it'll be cleaned up by some future gc run.

Change-Id: I078ae55fb7a9dfcdae28d6917d138842188ed528
2025-02-14 15:28:50 +00:00
Maximilian Bosch 406f4fed35 doc: show how to work on n-e-j
Change-Id: I51a65597ef712c283f3347351ae9aed2bc329f5e
2025-02-13 12:25:48 +01:00
eldritch horrors a987d92bd0 libstore: asyncify LocalStore::autoGC
sadly we need both a synchronous and an asynchronous promise for this
since destructors cannot be async. we also cannot use forked promises
since multiple threads may be waiting for an auto-gc to complete, but
forked promises are bound to an event loop (and thus a single thread)

Change-Id: I7b4fdbd229c4a1e01bbf80858e93347024f2e333
2025-02-12 19:22:20 +00:00
eldritch horrors 4fec8b3214 libstore: asyncify Worker::makeBasicDerivationGoal
Change-Id: I04af5dc0c70084daa18de61aa0f84c8e18981e26
2025-02-12 19:22:20 +00:00
eldritch horrors d7d0acdbb4 libstore: create drv temp roots in Worker
constructors can't be async, and root creation must be made async.
creating roots outside of goal classes and passing a witness type
is a lot easier than changing the constructor structure of today.

Change-Id: Ic84a92f3db2e1a9a9164047456f227048c6871bd
2025-02-12 01:32:01 +00:00
eldritch horrors 3bca42eb00 libstore: asyncify Store::optimiseStore
Change-Id: I443fe4bddb0bec519ee36d55060e082ae66802cc
2025-02-10 12:54:19 +00:00
eldritch horrors 6a27c18479 libstore: asyncify Store::addTextToStore
Change-Id: I4181bef0a8c7f0001c359646f39c9d452b44f23b
2025-02-10 12:54:19 +00:00
eldritch horrors f18c39f46d libstore: asyncify writeDerivation
Change-Id: Ia4e25dbc84e522b27c5257564febbd1ccf4df8a3
2025-02-10 12:54:19 +00:00
eldritch horrors e19a1a25e8 libstore: asyncify Store::getBuildDerivationPath
Change-Id: Ie9c41eab4bfa548aa19f6587d60d9c0250ef2f54
2025-02-10 12:54:19 +00:00
eldritch horrors 3b1ade4b32 libstore: asyncify LogStore::getBuildLog
Change-Id: I98aed93f8fa4d7f6e99f6848a4ee00fc77d2e787
2025-02-10 12:54:19 +00:00
eldritch horrors fd6e995139 libstore:: asyncify Store::addToStoreFromDump
Change-Id: I6577a157d66765e08530633868d714f70e46891d
2025-02-10 12:54:19 +00:00
eldritch horrors af43fdec70 libstore: asyncify Store::addToStore
Change-Id: I0da60f3dc74525395bbf75fe6db56fce572cca7c
2025-02-10 12:54:19 +00:00
eldritch horrors 6b86f0bf49 libutil: make aio roots usable in processGraph
addMultipleToStore calls addToStore through processGraph, so for now we
need an aio root in processGraph callbacks. eventually we'll drop them,
but that can only be done once we know that the single-threaded runtime
will not get deadlocked. the threads guarantee progress, for time being

Change-Id: I62869a72ef8085c508609a8d4274330e004ab685
2025-02-10 12:54:06 +00:00
Jade Lovelace 7d4912bcf9 filetransfer: report errbuf for failing http codes
This was annoying in the case of
https://git.lix.systems/lix-project/lix/issues/662 and it's just an
oversight.

Change-Id: I5af8494dff4621ae2525c28a2c318304545ed665
2025-02-09 13:41:05 -08:00
eldritch horrors 132d11c2d8 libstore: asyncify Store::addToStoreSlow
Change-Id: Id5a3a29adf7b726b466d4b0b7461345af9a03cc9
2025-02-08 17:26:19 +00:00
eldritch horrors ed2d1080be libfetchers: asyncify fetchToStore
Change-Id: I0153cefbe974137ab3d4fd9218c8a77137976f8c
2025-02-08 17:26:19 +00:00
eldritch horrors 5a0c979afa libexpr: asyncify EvalPaths::copyPathToStore
Change-Id: I5ffa734a1b389049f8fd30c8554e073369c8523a
2025-02-08 17:26:19 +00:00
eldritch horrors 2c52a20dd2 libstore: asyncify Store::addMultipleToStore
Change-Id: I6a552fecfef8c32979afdec8caa9cdeaf875b511
2025-02-08 17:26:19 +00:00
eldritch horrors 65edbcc37a libcmd: asyncify ProfileManifest::build
Change-Id: Ib1d7ce47c81259d305f3e993618516e4c165a084
2025-02-08 17:26:19 +00:00
eldritch horrors e85242d6da libstore: asyncify Store::importPaths
Change-Id: Ic330b9213209fd7d3cccf90af7f3d6b41aede9b5
2025-02-08 17:26:19 +00:00
eldritch horrors 70026b1696 libfetchers: asyncify Cache::lookup{,Expired}
Change-Id: I0c2b364ebf7051b3118366b4e8c2b52860d5ce09
2025-02-08 17:26:19 +00:00
eldritch horrors 8bfd2d4bb1 libfetchers: asyncify downloadFile, downloadTarball
Change-Id: I6f6e5b61667efeda8c575ae1f3bd87ee75fb9f9e
2025-02-08 17:26:19 +00:00