this is a connection pool of size 1. it's also extremely in the way of
making store methods async since all methods are necessarily entangled
with all other methods via this blocking connection pool. we also will
not be able to convert *all* public methods to async code at once (not
without losing what little remains of our sanity, anyway). we'll treat
the connection pool as unbounded for a while, until the conversion has
been completed. if it turns out to be too expensive to keep db handles
in a pool like this we can tune it back down to small, or even size 1.
Change-Id: I0f3adb66df6f45a84e480c090d6627a368917db5
processGraph is not always safe to use in async code, but whether that's
the case depends on whether the caller is holding locks or not. (-sigh-)
Change-Id: Ieecea9c6d3c6abae222332bc6e3869bfd97631dd
this could be a method of AsyncCollect, but putting there would require
another overload that synthesizes the empty key types we use here. even
then a separate function is slightly nicer because it spares us writing
things like `asyncCollect(someIterableOfPromises).collect()` or similar
Change-Id: Ia92abc105016414a2171e237f2a0bc6233d3ccbf
more precisely, split it into store preparation (which remains in the
constructor proper), db initialization, and prepared statements init.
this will let us more easily turn the dbState into a connection pool.
Change-Id: I894acd4553bc83a2d86229922b757c573e394cac
this is one of the cases retrySQLite is supposed to handle. with busy
timeouts set at the connection level we may instead wait for up to an
hour to acquire locks before actually falling back to the retrySQLite
loop, blocking an entire thread in the process. this isn't painful in
the current system that uses dedicated threads for everything, but an
async executor actually suffers. since we use sqlite in WAL mode, and
WAL mode still allows reads to happen concurrently with writes, we'll
want to not block an executor on sqlite write locks. (note that while
we do have a mutex around db access that mutex is only held inside of
retrySQLite callbacks, so it will be released where we'd block today)
do note that we don't disable the timeout completely since that could
drastically reduce write performance. multiple daemon processes using
the same database file will naturally conflict, but each write is not
usually that expensive. going to sleep immediately instead of waiting
a bit can greatly impact throughput. waiting for 50ms should be fine.
Change-Id: I2d8e36ed7a0183bbc213490ebde2c79fc6e2f4b2
this is like a condition variable associated with the mutex. it's all
we'll need for the async transition, so we don't add a separate type.
Change-Id: Id9aefadd3b50e5f14f7b4d6bad41721f5249ce59
sadly this is a visitor-only interface; async generators are not yet a
thing and preliminary benchmarks say that overhead would be too large.
Change-Id: I0460d18eba94441cf3101d46bfffdb69cdda81d1
we could have constructed a nar accessor from an index just as well, but
we don't need any of the advanced accessor features like retrieving file
contents or full path-string based access to the archive. using an index
directly is simpler and faster, although in practice we shouldn't notice
Change-Id: Ic521536cd89e88cbe5aba4f26bf40f0c40d09341
this lives with the NarAccessor (for now) because only the nar accessor
consumes this format in-tree, and produces the same format for a second
type of data source (an FSAccessor instead of a precomputed nar index).
Change-Id: I54cddcf59a4e0501b5cc296c606f063feee5fb3b
we'll use this in NarAccessor to provide actually safe indexing of
archives. NarAccessor currently is not fully correct: it relies on
the parser not buffering anything to produce correct file offsets,
but only the nar implementation itself can reasonably expect that.
Change-Id: I64f300b86d8844b876a3ace723546ea7d7b4628b
this wasn't fully correct to begin with; if restorePath threw an
exception we would not have discarded the nar, but if deletePath
threw an exception we would have. discarding the nar in this way
also lets us not swallow nar parser exceptions when adding a nar
for a valid path, potentially notifying clients of corruption in
their own data sources. ideally we would never read the nar when
we're not interested anyway, but that needs a new wire protocol.
Change-Id: Ie5e10a9bee05b67ec2adb9c4f7c9319a4e644e31
the web of sinks and sources is really convoluted. all we want to do is
to always write the full path dump into narHashSink, also write the nar
(during recursive import) or only file data (during flat import) into a
dedicate non-sha256 hash sink, and maybe check that the path was a file
(when doing flat imports). the graphviz diagram did not improve things.
Change-Id: I2d036358a78963222da3db885cf6971e51369344
this fully decouples the possibly-never-async bits of dumping from the
generation of dump bitstream. having the two separate will allow us to
change store import methods to use async streams, not our sync sources
Change-Id: I9dbd5e30ad3ee380c244b4a3760c11e37db3895f
filtering of paths may call into nixlang code, and in turn may call into
the async runtime. dumps and dump preparations should thus be considered
never-async, but we can't mark them as such yet because async methods do
call them without issue at time of writing. we cannot mark them as such;
only a single function in libexpr may cause such problems. fetchers also
use dumpPath with filters, but those filters never call into async code.
Change-Id: Ica0bd853419e08cd5b0f7926820e3c5a8d0f4688
only a single caller uses flat fetching at all. it still makes sense to
not inline that single caller for now, mostly due to activity reporting
Change-Id: I1a6420868443c3a684deafc7a4f567d4d4b1bd53
eldritch horrors