Commit Graph
17417 Commits
Author SHA1 Message Date
eldritch horrors 7cb28186f2 libutil: add promise-compatible Sync specialization
we can't use Sync with stl mutex types since yielding from one coroutine
to another while holding a lock is a deadlock hazard. if we don't expose
blocking lock operations from Sync though and emulate blocking with more
promises we can reuse most of the Sync code instead of rewriting it all.
hiding them completely would make it impossible to migrate current users
to promise-based code though, so we use our new NeverAsync lint instead.

Change-Id: I7eebf18e29473963040e7448da27c52012f1f134
2025-02-16 15:00:10 +00:00
eldritch horrors 7f95f9a85c libstore: mark blocking file locks as never async
Change-Id: Icf4633b5cd540cb777004c3dce6aa9d79a7f4063
2025-02-16 14:59:32 +00:00
eldritch horrors e12deec9fb clang-tidy: add lint to forbid blocking code
Change-Id: I3df6f67f363b0a5ac014bf817e0b35fde398827f
2025-02-16 14:16:15 +00:00
eldritch horrors 1077bc626e libstore: use lockPathAsync in more places
these were found by an upcoming lint.

Change-Id: I4647baa2c0971f3007d348e8ced6f75e45037a20
2025-02-14 15:28:50 +00:00
eldritch horrors 65ee7f5da4 libstore: use non-blocking locks in createTempDirInStore
gc tryLocks first, then deletes. we can use the same order since there's
no sequence of operations that would let us lock our new directory *and*
have the gc delete it (except outright bugs). the worst possible outcome
is that we create a few directories that'll be deleted by a later gc run

Change-Id: I6d3524f4990f8deee10804e6fbb0d0a26f3d57fe
2025-02-14 15:28:50 +00:00
eldritch horrors 81275c9ad7 libstore: asyncify Store::addTempRoot
Change-Id: Ie3ef2a637d54b22f0bf9c8aba14d2f7546c6cd9a
2025-02-14 15:28:50 +00:00
eldritch horrors 84dd5864f3 libstore: simplify LocalStore::createTempRootsFile
we now create a temporary file, lock it, move it into place, and return.
if any of these steps (except creation) fails the GC must have found and
deleted our new file before we got the chance to use it. we don't need a
lock around file creation anymore, we especially don't need a mutex that
covers an flock call, and we don't need the GC to clobber what it finds.
(clobbering *is* kept for compatibility with older nix implementations.)

if we ever leave trash around it'll be cleaned up by some future gc run.

Change-Id: I078ae55fb7a9dfcdae28d6917d138842188ed528
2025-02-14 15:28:50 +00:00
Maximilian Bosch 406f4fed35 doc: show how to work on n-e-j
Change-Id: I51a65597ef712c283f3347351ae9aed2bc329f5e
2025-02-13 12:25:48 +01:00
eldritch horrors a987d92bd0 libstore: asyncify LocalStore::autoGC
sadly we need both a synchronous and an asynchronous promise for this
since destructors cannot be async. we also cannot use forked promises
since multiple threads may be waiting for an auto-gc to complete, but
forked promises are bound to an event loop (and thus a single thread)

Change-Id: I7b4fdbd229c4a1e01bbf80858e93347024f2e333
2025-02-12 19:22:20 +00:00
eldritch horrors 4fec8b3214 libstore: asyncify Worker::makeBasicDerivationGoal
Change-Id: I04af5dc0c70084daa18de61aa0f84c8e18981e26
2025-02-12 19:22:20 +00:00
eldritch horrors d7d0acdbb4 libstore: create drv temp roots in Worker
constructors can't be async, and root creation must be made async.
creating roots outside of goal classes and passing a witness type
is a lot easier than changing the constructor structure of today.

Change-Id: Ic84a92f3db2e1a9a9164047456f227048c6871bd
2025-02-12 01:32:01 +00:00
eldritch horrors 3bca42eb00 libstore: asyncify Store::optimiseStore
Change-Id: I443fe4bddb0bec519ee36d55060e082ae66802cc
2025-02-10 12:54:19 +00:00
eldritch horrors 6a27c18479 libstore: asyncify Store::addTextToStore
Change-Id: I4181bef0a8c7f0001c359646f39c9d452b44f23b
2025-02-10 12:54:19 +00:00
eldritch horrors f18c39f46d libstore: asyncify writeDerivation
Change-Id: Ia4e25dbc84e522b27c5257564febbd1ccf4df8a3
2025-02-10 12:54:19 +00:00
eldritch horrors e19a1a25e8 libstore: asyncify Store::getBuildDerivationPath
Change-Id: Ie9c41eab4bfa548aa19f6587d60d9c0250ef2f54
2025-02-10 12:54:19 +00:00
eldritch horrors 3b1ade4b32 libstore: asyncify LogStore::getBuildLog
Change-Id: I98aed93f8fa4d7f6e99f6848a4ee00fc77d2e787
2025-02-10 12:54:19 +00:00
eldritch horrors fd6e995139 libstore:: asyncify Store::addToStoreFromDump
Change-Id: I6577a157d66765e08530633868d714f70e46891d
2025-02-10 12:54:19 +00:00
eldritch horrors af43fdec70 libstore: asyncify Store::addToStore
Change-Id: I0da60f3dc74525395bbf75fe6db56fce572cca7c
2025-02-10 12:54:19 +00:00
eldritch horrors 6b86f0bf49 libutil: make aio roots usable in processGraph
addMultipleToStore calls addToStore through processGraph, so for now we
need an aio root in processGraph callbacks. eventually we'll drop them,
but that can only be done once we know that the single-threaded runtime
will not get deadlocked. the threads guarantee progress, for time being

Change-Id: I62869a72ef8085c508609a8d4274330e004ab685
2025-02-10 12:54:06 +00:00
Jade Lovelace 7d4912bcf9 filetransfer: report errbuf for failing http codes
This was annoying in the case of
https://git.lix.systems/lix-project/lix/issues/662 and it's just an
oversight.

Change-Id: I5af8494dff4621ae2525c28a2c318304545ed665
2025-02-09 13:41:05 -08:00
eldritch horrors 132d11c2d8 libstore: asyncify Store::addToStoreSlow
Change-Id: Id5a3a29adf7b726b466d4b0b7461345af9a03cc9
2025-02-08 17:26:19 +00:00
eldritch horrors ed2d1080be libfetchers: asyncify fetchToStore
Change-Id: I0153cefbe974137ab3d4fd9218c8a77137976f8c
2025-02-08 17:26:19 +00:00
eldritch horrors 5a0c979afa libexpr: asyncify EvalPaths::copyPathToStore
Change-Id: I5ffa734a1b389049f8fd30c8554e073369c8523a
2025-02-08 17:26:19 +00:00
eldritch horrors 2c52a20dd2 libstore: asyncify Store::addMultipleToStore
Change-Id: I6a552fecfef8c32979afdec8caa9cdeaf875b511
2025-02-08 17:26:19 +00:00
eldritch horrors 65edbcc37a libcmd: asyncify ProfileManifest::build
Change-Id: Ib1d7ce47c81259d305f3e993618516e4c165a084
2025-02-08 17:26:19 +00:00
eldritch horrors e85242d6da libstore: asyncify Store::importPaths
Change-Id: Ic330b9213209fd7d3cccf90af7f3d6b41aede9b5
2025-02-08 17:26:19 +00:00
eldritch horrors 70026b1696 libfetchers: asyncify Cache::lookup{,Expired}
Change-Id: I0c2b364ebf7051b3118366b4e8c2b52860d5ce09
2025-02-08 17:26:19 +00:00
eldritch horrors 8bfd2d4bb1 libfetchers: asyncify downloadFile, downloadTarball
Change-Id: I6f6e5b61667efeda8c575ae1f3bd87ee75fb9f9e
2025-02-08 17:26:19 +00:00
eldritch horrors 60f391ef3d libstore: asyncify LocalFSStore::addPermRoot
Change-Id: I0e19470e7fcc3d563b26c078c1dc831dbc5cf422
2025-02-08 17:26:19 +00:00
eldritch horrors 614a63b49a libstore: asyncify createGeneration
Change-Id: Iae47a0f732b2f0fe54525917c091169f4be2214e
2025-02-08 17:26:19 +00:00
eldritch horrors 1ce46c318c libfetchers: asyncify getRegistries, lookupInRegistries
Change-Id: I5a668ad05440e2d7d574e8a470c269b670282c6f
2025-02-08 17:26:19 +00:00
eldritch horrors 07691dd851 libexpr: asyncify FlakeRef::resolve
Change-Id: I31a1080652f03bdddd63b3115fa0686aa340dc34
2025-02-08 17:26:19 +00:00
eldritch horrors 17b3992e78 libcmd: add an aio root to completeFlakeRef
the other completers already inherit one through EvalState. all of them
eventually need it since completing a flake ref may require network io.

Change-Id: Ie664a129a68457a6f0908d226057d7d7fb610ae4
2025-02-08 17:01:31 +00:00
eldritch horrors 0bd93b5208 libfetchers: asyncify GitArchiveInputScheme::getRevFromRef
Change-Id: Id409ed6c13e71eed3628a8ed10f6fd03060c79b0
2025-02-08 17:01:31 +00:00
eldritch horrors 6e95540610 libstore: asyncify Store::pathInfoToJSON
Change-Id: I2e04c6804444b3ca53486ce6acb0c159e7814938
2025-02-08 17:01:31 +00:00
eldritch horrors 6ffa9f4298 libfetchers: asyncify InputScheme::fetch
Change-Id: I8d4d8d1d4137635ade3e4b05b5d4cd3ecb3390c9
2025-02-08 17:01:31 +00:00
eldritch horrors b60314f32e libstore: asyncify makeContentAddressed
Change-Id: I3cdad058a9f015983db5306271c56a9996aaef28
2025-02-08 13:50:33 +00:00
eldritch horrors 5fd2d24e63 cli: asyncify print{DotGraph,GraphML}
Change-Id: I0fc13a861acee55c53c4ae5f6ba77b4100fd8735
2025-02-08 13:50:33 +00:00
eldritch horrors 9b0cd7a7c9 libstore: asyncify copyPaths
Change-Id: Ie7cd94d8760abd9c7593f73fd281cbba91a5be77
2025-02-08 13:50:33 +00:00
eldritch horrors fb9e3f6cb5 libstore: asyncify copyClosure
Change-Id: I18cc324d80b502ac87dda3fba2166d4e25f53664
2025-02-08 13:50:33 +00:00
eldritch horrors ecfda8abe2 libstore: asyncify openStore
Change-Id: Ia152bfd2014851590328e732f434dff6695304bd
2025-02-08 12:45:15 +00:00
eldritch horrors af507e587f libstore: asyncify getDefaultSubstituters
Change-Id: I078c5575f4941ab0d06ef9bf1e549b789e695afc
2025-02-08 12:45:15 +00:00
eldritch horrors bf1c5cb27d libstore: asyncify Store::querySubstitutablePathInfos
this also calls getDefaultSubstituters.

Change-Id: I9c80d8b8311a712a3e73964445c42d10a5dab7db
2025-02-08 12:45:15 +00:00
eldritch horrors c2f71d5b30 libstore: asyncify Store::querySubstitutablePaths
the LocalStore implementation calls `getDefaultSubstituters`, which
calls `openStore`, which calls `Store::init`, which does network io
in binary caches, which will eventually be properly asyncified curl

Change-Id: If6a7a0294e2c4ba8e35722583be6ccde21ed6585
2025-02-08 12:45:15 +00:00
eldritch horrors 6e12f8922c libstore: add async path locking
just single path locks for now since we don't need more.

Change-Id: I0f1be156e762fbe7355dc1381295a7b276bad720
2025-02-08 12:45:15 +00:00
eldritch horrors ec44c8bd6c libstore: rework path locking infra entirely
we now allow for single locked paths rather than only sets, the lock
holder classes are finally resource-safe, and not locking files from
within constructors means we can (in theory!) make path locks async.

Change-Id: I1e1807299d370c07b15c332d5b2ff77b64456e7d
2025-02-08 13:39:49 +01:00
eldritch horrors 72326c4044 libstore: always delete PathLocks lockfiles
they are not being created with O_EXCL and their existence is not
checked for using the PathLocks api, so we will boldy assume that
*nothing* checks for lock file existence. deleting lockfiles when
we're done with them in all cases makes the code cleaner, and any
failing builds no longer litter their store with stale lockfiles.

Change-Id: Iffb588d29b00e6aca32fcb6776980455238ac2bd
2025-02-08 13:14:48 +01:00
eldritch horrors 80189b3c26 libstore: add async FdLock constructor
the local store doesn't have to block the entire thread waiting for a
lock with such support. we'll still block *some* thread, but there is
currently no good way around that without e.g. a lock manager daemon.

Change-Id: I037fdc749e1b56cf9e0f1ca7ac70274442fb2b8c
2025-02-08 08:54:16 +00:00
eldritch horrors 8f5f33c9d8 libstore: bind FdLock to AutoCloseFD and make resource-safe
Change-Id: I2451c15bd0e7d3a5a537bbb65502fd4a18ae0c22
2025-02-08 08:33:26 +00:00
Justin !andGerrit Code Review 00a1afe022 Merge "libutil: make runProgram2 print debug infos" into main 2025-02-07 15:10:53 +00:00