Commit Graph
100 Commits
Author SHA1 Message Date
eldritch horrors 4857feb910 libutil: add Bytes type
not used anywhere yet, but we'll use this a lot soon for generators that
return file contents, wire protocol fragments, or indeed any byte stream

Change-Id: I01a46f9bf9d75aaf4a5d7662773b99f498862a28
2024-07-03 11:46:53 +00:00
eldritch horrors 73ddc4540f libutil: generator type with on-yield value mapping
this will be the basis of non-boost coroutines in lix. anything that is
a boost coroutine *should* be representable with a Generator coroutine,
and many things that are not currently boost coroutines but behave much
like one (such as, notably, serializers) should be as well. this allows
us to greatly simplify many things that look like iteration but aren't.

Change-Id: I2cebcefa0148b631fb30df4c8cfa92167a407e34
2024-07-03 11:46:53 +00:00
eldritch horrors e7517419a6 libmain: better fix for #424, #425
not printing activities at all when no progress information is available
hides *all* progress information from e.g. flake show. this is not ideal
and needs to be fixed, but the fix *still* has problems with flake show:
in multiline mode we will overwrite all useful flake show output as soon
as the progress bar is redrawn. flake show output is also mangled in any
number of other situations (like -v being set), so we should probably be
not too worried about it and fix progress reporting properly another day

Change-Id: I6d39d670e261bbae00560b6a8e15dec8e16b35c4
2024-07-02 17:16:30 +02:00
eldritch horrors 010ff57ebb enable -Werror=unused-result
we do not have any of these warnings appearing at the moment, but it
seems like a good idea to enable [[nodiscard]] checking anyway. once
we start introducing more functions with must-use conditions we will
need such checking, and the rust stdlib has proven them very useful.

Change-Id: Ibb6b042ae1ec5f527f8dc2809a7816a4c1548ae2
2024-06-30 21:30:43 +00:00
eldritch horrors 3dd7d023f4 libmain: don't print empty lines
this most notably affects `nix eval`: if there is no progress bar to be
shown and no activities going on we should not print anything at all. a
progress bar with no activities would print a bunch of terminal escapes
*and a space*, which is not helpful in simple cases like nix eval -E 1.
notably this does *not* affect nix eval called on non-terminal outputs,
but it is slightly confusing nevertheless (and not difficult to avoid).

fixes https://git.lix.systems/lix-project/lix/issues/424

Change-Id: Iee793c79ba5a485d6606e0d292ed2eae6dfb7216
2024-06-26 17:44:04 +00:00
eldritch horrors e6cd67591b libexpr: rewrite the parser with pegtl instead of flex/bison
this gives about 20% performance improvements on pure parsing. obviously
it will be less on full eval, but depending on how much parsing is to be
done (e.g. including hackage-packages.nix or not) it's more like 4%-10%.

this has been tested (with thousands of core hours of fuzzing) to ensure
that the ASTs produced by the new parser are exactly the same as the old
one would have produced. error messages will change (sometimes by a lot)
and are not yet perfect, but we would rather leave this as is for later.

test results for running only the parser (excluding the variable binding
code) in a tight loop with inputs and parameters as given are promising:

  - 40% faster on lix's package.nix at 10000 iterations
  - 1.3% faster on nixpkgs all-packages.nix at 1000 iterations
  - equivalent on all of nixpkgs concatenated at 100 iterations
    (excluding invalid files, each file surrounded with parens)

more realistic benchmarks are somewhere in between the extremes, parsing
once again getting the largest uplift. other realistic workloads improve
by a few percentage points as well, notably system builds are 4% faster.

Benchmarks summary (from ./bench/summarize.jq bench/bench-*.json)
old/bin/nix --extra-experimental-features 'nix-command flakes' eval -f bench/nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  mean:     0.408s ± 0.025s
            user: 0.355s | system: 0.033s
  median:   0.389s
  range:    0.388s ... 0.442s
  relative: 1

new/bin/nix --extra-experimental-features 'nix-command flakes' eval -f bench/nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  mean:     0.332s ± 0.024s
            user: 0.279s | system: 0.033s
  median:   0.314s
  range:    0.313s ... 0.361s
  relative: 0.814

---

old/bin/nix --extra-experimental-features 'nix-command flakes' eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  mean:     6.133s ± 0.022s
            user: 5.395s | system: 0.437s
  median:   6.128s
  range:    6.099s ... 6.183s
  relative: 1

new/bin/nix --extra-experimental-features 'nix-command flakes' eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  mean:     5.925s ± 0.025s
            user: 5.176s | system: 0.456s
  median:   5.934s
  range:    5.861s ... 5.943s
  relative: 0.966

---

GC_INITIAL_HEAP_SIZE=10g old/bin/nix eval --extra-experimental-features 'nix-command flakes' --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  mean:     4.503s ± 0.027s
            user: 3.731s | system: 0.547s
  median:   4.499s
  range:    4.478s ... 4.541s
  relative: 1

GC_INITIAL_HEAP_SIZE=10g new/bin/nix eval --extra-experimental-features 'nix-command flakes' --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  mean:     4.285s ± 0.031s
            user: 3.504s | system: 0.571s
  median:   4.281s
  range:    4.221s ... 4.328s
  relative: 0.951

---

old/bin/nix --extra-experimental-features 'nix-command flakes' search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 hello
  mean:     16.475s ± 0.07s
            user: 14.088s | system: 1.572s
  median:   16.495s
  range:    16.351s ... 16.536s
  relative: 1

new/bin/nix --extra-experimental-features 'nix-command flakes' search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 hello
  mean:     15.973s ± 0.013s
            user: 13.558s | system: 1.615s
  median:   15.973s
  range:    15.946s ... 15.99s
  relative: 0.97

---

Change-Id: Ie66ec2d045dec964632c6541e25f8f0797319ee2
2024-06-25 12:24:58 +00:00
eldritch horrors d477b34d1d libutil: remove runProgram2 stdin functionality
this was only used in one place, and that place has been rewritten to
use a temporary file instead. keeping this around is not very helpful
at this time, and in any case we'd be better off rewriting subprocess
handling in rust where we not only have a much safer library for such
things but also async frameworks necessary for this easily available.

Change-Id: I6f8641b756857c84ae2602cdf41f74ee7a1fda02
2024-06-23 17:29:40 +00:00
eldritch horrors 2bbdaf0b19 libfetchers: write git commit message to tempfile
we want to remove runProgram's ability to provide stdin to a process
because the concurrency issues of handling both stdin and stdout are
much more pronounced once runProgram returns not is collected output
but a source. this is possible in the current c++ framework, however
it isn't necessary in almost all cases (as demonstrated by only this
single user existing) and in much better handled with a proper async
concurrency model that lets the caller handle both at the same time.

Change-Id: I29da1e1ad898d45e2e33a7320b246d5003e7712b
2024-06-23 17:29:40 +00:00
eldritch horrors 2fe9157808 flakes: add --commit-lock-file message test
we had no test to ensure that we generated a commit message at all?

Change-Id: Ic9aa8fde92b83e1ea6f61cd2a21867aa73d4e885
2024-06-23 17:29:40 +00:00
eldritch horrors ce6cb14995 libutil: return Pid from startProcess, not pid_t
Change-Id: Icc8a15090c77f54ea7d9220aadedcd4a19922814
2024-06-23 11:52:49 +00:00
eldritch horrors 3d155fc509 libutil: give Pid proper resource semantics
copy-constructing or assigning from pid_t can easily lead to duplicate
Pid instances for the same process if a pid_t was used carelessly, and
Pid itself was copy-constructible. both could cause surprising results
such as killing processes twice (which could become very problemantic,
but luckily modern systems don't reuse PIDs all that quickly), or more
than one piece of the code believing it owns a process when neither do

Change-Id: Ifea7445f84200b34c1a1d0acc2cdffe0f01e20c6
2024-06-23 11:52:49 +00:00
eldritch horrors b43a2e84c4 libutil: make Pid -> pid_t operations explicit
Change-Id: I3137cc140590001fe7ba542844e735944a0a9255
2024-06-23 11:52:49 +00:00
eldritch horrors a9949f4760 libutil: add some serialize.hh serializer tests
Change-Id: I0116265a18bc44bba16c07bf419af70d5195f07d
2024-06-23 11:52:49 +00:00
eldritch horrors 39a1e248c9 libutil: remove sinkToSource eof callback
this is only used in one place, and only to set a nicer error message on
EndOfFile. the only caller that actually *catches* this exception should
provide an error message in that catch block rather than forcing support
for setting error message so deep into the stack. copyStorePath is never
called outside of PathSubstitutionGoal anyway, which catches everything.

Change-Id: Ifbae8706d781c388737706faf4c8a8b7917ca278
2024-06-23 11:52:49 +00:00
eldritch horrors c55dcc6c13 filetransfer: return a Source from download()
without this we will not be able to get rid of makeDecompressionSink,
which in turn will be necessary to get rid of sourceToSink (since the
libarchive archive wrapper *must* be a Source due to api limitations)

Change-Id: Iccd3d333ba2cbcab49cb5a1d3125624de16bce27
2024-06-19 10:50:12 +00:00
eldritch horrors 11f4a5bc7e libutil: return a source from readFile
don't consume a sink, return a source instead. the only reason to not do
this is a very slight reduction in dynamic allocations, but since we are
going to *at least* do disk io that will not be a lot of overhead anyway

Change-Id: Iae2f879ec64c3c3ac1d5310eeb6a85e696d4614a
2024-06-19 10:50:12 +00:00
eldritch horrors 67f778670c libutil: add makeDecompressionSource
Change-Id: Iac7f24d79e24417436b9b5cbefd6af051aeea0a6
2024-06-19 10:50:12 +00:00
eldritch horrors 3425e90d76 libstore: BinaryCacheStore::getFile{ -> Contents}
if we want have getFile return a source instead of consuming a sink
we'll have to disambiguate this overload another way, eg like this.

Change-Id: Ia26de2020c309a37e7ccc3775c1ad1f32e0a778b
2024-06-19 10:50:12 +00:00
eldritch horrors a960576f58 libutil: BrotliDecompression{Sink -> Source}
Change-Id: I9579dd08f7bd0f927bde9d3128515b0cee15f320
2024-06-19 00:54:06 +00:00
eldritch horrors 0b9a72524a filetransfer: {up,down}load -> transfer
even the transfer function is not all that necessary since there aren't
that many users, but we'll keep it for now. we could've kept both names
but we also kind of want to use `download` for something else very soon

Change-Id: I005e403ee59de433e139e37aa2045c26a523ccbf
2024-06-18 23:58:25 +00:00
eldritch horrors bcb774688f libexpr: add expr memory management
with the prepatory work done this mostly means turning plain pointers
into unique_ptrs, with all the associated churn that necessitates. we
might want to change some of these to box_ptrs at some point as well,
but that would be a semantic change that isn't fully appropriate yet.

Change-Id: I0c238c118617420650432f4ed45569baa3e3f413
2024-06-17 19:46:44 +00:00
eldritch horrors ad5366c2ad libexpr: pass Exprs as references, not pointers
almost all places where Exprs are passed as pointers expect the pointers
to be non-null. pass them as references to encode this constraint in the
type system as well (and also communicate that Exprs must not be freed).

Change-Id: Ia98f166fec3c23151f906e13acb4a0954a5980a2
2024-06-17 19:46:44 +00:00
eldritch horrors b8f49a8eaf libexpr: store ExprConcatStrings elements as direct vector
storing a pointer only adds an unnecessary indirection at runtime.

Change-Id: If06dd05effdf1ccb0df0873580f50c775608925d
2024-06-17 19:46:44 +00:00
eldritch horrors dad8bc679e libexpr: don't immediately throw parser errors
now that destructors are hooked up we want to give the C skeleton every
real chance to actually run them. since bison does not call destructors
on values that have been passed to semantic actions even when an action
causes an abort we will also have to delete some things manually still.

Change-Id: Ia22bdaa9e969b74e17a6c496e35e6c2d86b7d750
2024-06-17 19:46:44 +00:00
eldritch horrors 9592a9fd57 libexpr: hook up bison destructors for state objects
this doesn't help much yet since the state objects themselves also leak
all memory they are given, but it is a first necessary step to properly
managing parser memory. notably we have to clear $$ when returning from
the parser since even the start symbol is subject to automatic deletion
by the bison-generated parser before returning control to the call site

Change-Id: I80245b0c747308e80923e7f18ce4e1a4898f93b0
2024-06-17 19:46:44 +00:00
eldritch horrors dd4a2c1759 libstore: fix http abuses no longer working
while refactoring the curl wrapper we inadvertently broken the immutable
flake protocol, because the immutable flake protocol accumulates headers
across the entire redirect chain instead of using only the headers given
in the final response of the chain. this is a problem because Some Known
Providers Of Flake Infrastructure set rel=immutable link headers only in
the penultimate entry of the redirect chain, and curl does not regard it
as worth returning to us via its response header enumeration mechanisms.

fixes https://git.lix.systems/lix-project/lix/issues/358

Change-Id: I645c3932b465cde848bd6a3565925a1e3cbcdda0
2024-05-29 22:35:29 +00:00
eldritch horrors ed6b3165ea worker: process timeouts first, and exclusively
check goals for timeouts first, and their activity fds only if no
timeout has occurred. checking for timeouts *after* activity sets
us up for assertion failures by running multiple build completion
notifiers, the first of which will kill/reap the the goal process
and consuming the Pid instance. when the second notifier attempts
to do the same it will core dump with an assertion failure in Pid
and take down not only the single goal, but the entire daemon and
all goals it was building. luckily this is rare in practice since
it requires a build to both finish and time out at the same time.

writing a test for this is not feasible due to how much it relies
on scheduling to actually trigger the underlying bug, but on idle
machines it can usually be triggered by running multiple sleeping
builds with timeout set to the sleep duration and `--keep-going`:

    nix-build --timeout 10 --builders '' --keep-going -E '
      with import <nixpkgs> {};
      builtins.genList
        (i: runCommand "foo-${toString i}" {} "sleep 10")
        100
    '

Change-Id: I394d36b2e5ffb909cf8a19977d569bbdb71cb67b
2024-05-26 20:57:09 +00:00
eldritch horrorsandGerrit Code Review 829521b91a Merge "doc: sort change-authors.yml" into main 2024-05-23 02:35:53 +00:00
eldritch horrorsandGerrit Code Review f281727e70 Merge "libstore: remove unused copyPath function" into main 2024-05-23 02:31:50 +00:00
eldritch horrors 9a75150d19 doc: sort change-authors.yml
this should make it easier to spot future instances of entries being
duplicated by accident. also add a pre-commit check to remain sorted

Change-Id: I500caf862e93480b38c9d51144273bb2dcab1af0
2024-05-23 02:09:05 +00:00
eldritch horrors 47523944c5 libstore: remove unused copyPath function
Change-Id: Ibfb0848d8b337bd38947a745c240838865cea401
2024-05-22 18:26:01 +02:00
eldritch horrors 774c56094f libstore: fix old RemoteStore::addToStore serializer
having the serializer write into `*conn` is not legal because we are
in a sinkToSource that will be drained by the remote we're connected
to. writing into `*conn` directly can break the framing protocol. it
is unlikely this code was ever run: to protocol it caters to is from
2016(!) and thoroughly untested in-tree, and since it's been present
since nix 2.17 and the 1.18 protocol broken here is nix 2.0 we might
safely assume that daemons older than nix 2.1 are no longer used now

see also #325 (though that wants <2.3 gone, this is sadly only <2.1)

Change-Id: I9d674c18f6d802f61c5d85dfd9608587b73e70a5
2024-05-19 11:57:55 +00:00
eldritch horrors ceccac835c libutil: remove callback.hh
it's no longer used. it really shouldn't have existed this long since it
was just a mashup of both std::promise and std::packaged_task in a shape
that makes composition unnecessarily difficult. all but a single case of
Callback pattern calls were fully synchronous anyway, and even this sole
outlier was by far not important enough to justify the extra complexity.

Change-Id: I208aec4572bf2501cdbd0f331f27d505fca3a62f
2024-05-10 02:21:11 +02:00
eldritch horrors b66451ae7f libstore: de-callback-ify FileTransfer
also add a few more tests for exception propagation behavior. using
packaged_tasks and futures (which only allow a single call to a few
of their methods) introduces error paths that weren't there before.

Change-Id: I42ca5236f156fefec17df972f6e9be45989cf805
2024-05-10 02:21:11 +02:00
eldritch horrors 28a98d152c libstore: de-callback-ify Store::queryRealisation
Change-Id: I8d74745c519518f163f51dfaa39063836f17599e
2024-05-09 23:18:05 +02:00
eldritch horrors 17965bf11c libstore: un-callback-ify Store::queryRealisationUncached
Change-Id: I4a328f46eaac3bb8b19ddc091306de83348be9cf
2024-05-09 23:18:05 +02:00
eldritch horrors 2f4a1dd6e0 libstore: de-callback-ify Store::queryPathInfoUncached
Change-Id: I23a156aaff5328f67ca16ccd85c0ea1711b21e35
2024-05-09 23:18:05 +02:00
eldritch horrors c77bd88259 libstore: de-callback-ify BinaryCacheStore::getFile
Change-Id: I36b3eb9f645aa04058151e7b2353e15e6f29057b
2024-05-09 23:18:05 +02:00
eldritch horrors 1a002d1a11 libstore: de-callback-ify CA realisation substitution
this is the *only* real user of file transfer download completion
callbacks, and a pretty spurious user at that (seeing how nothing
here is even turned on by default and indeed a dependency of path
substitution which *isn't* async, and concurrency-limited). it'll
be a real pain to keep this around, and realistically it would be
a lot better to overhaul substitution in general to be *actually*
async. that requires a proper async framework footing though, and
we don't have anything of the sort, but it's also blocking *that*

Change-Id: I1bf671f217c654a67377087607bf608728cbfc83
2024-05-09 23:18:05 +02:00
eldritch horrors 964ac8b0e8 libutil: de-callback-ify computeClosure
only two users of this function exist. only one used it in a way that
even bears resemblance to asynchronicity, and even that one didn't do
it right. fully async and parallel computation would have only worked
if any getEdgesAsync never calls the continuation it receives itself,
only from more derived callbacks running on other threads. calling it
directly would cause the decoupling promise to be awaited immediately
*on the original thread*, completely negating all nice async effects.

Change-Id: I0aa640950cf327533a32dee410105efdabb448df
2024-05-07 14:35:20 +00:00
eldritch horrors 230860dbb8 libstore: limit CA realisation info substitution concurrency
this seems to be an oversight, considering that regular substitutions
are concurrency-limited. while not particularly necessary at present,
once we've removed the `Callback` based interfaces it will be needed.

Change-Id: Ide2d08169fcc24752cbd07a1d33fb8482f7034f5
2024-05-07 14:35:20 +00:00
eldritch horrors 29f93e1e0d libutil: throw EndOfFile at sourceToSink end
... how has this never broken anything before

Change-Id: If3789c02028e8f929481514f63d76b0b46bfc182
2024-05-07 14:35:20 +00:00
eldritch horrors f75d0752ce filetransfer: correctly abort empty transfers
returning 0 from the callback for errors signals successful transfer if
the source returned no data even though the exception we've just caught
clearly disagrees. while this is not all that important (since the only
viable cause of such errors will be dataCallback, and the sole instance
of it being used already takes care of exceptions) we can just do this.

Change-Id: I2bb150eff447121d82e8e3aa4e00057c40523ac6
2024-05-06 21:18:23 +02:00
eldritch horrors 121edecf65 filetransfer: extract decompressor creation
this will be necessary if we want download() to return a source instead
of consuming a sink, which will in turn be needed to remove coroutines.

Change-Id: I34ec241e9bbc5d32fbcd243b244e29c3757533aa
2024-05-06 20:19:03 +02:00
eldritch horrors 6b08138929 filetransfer: abort transfer on receiver exception
not doing this will cause transfers that had their readers disappear to
linger. with lingering transfers the curl thread can't shut down, which
will cause nix itself to not shut down until the transfer finishes some
other way (most likely network timeouts). also add a new test for this.

Change-Id: Id2401b3ac85731c824db05918d4079125be25b57
2024-05-05 18:09:31 +00:00
eldritch horrors 6a5f100b8b doc: fix build littering doc/
mdbook has the unfortunate habit of creating stub files for chapters it
can't find on disk. turn off this helpful feature as it masks errors in
the summary file, and fix a recently introduced instance of this error.

Change-Id: I10d86aac0489c9c494bd5c8a50047415f4d4b18d
2024-05-05 16:13:27 +00:00
eldritch horrors d55b158e24 libutil: make rewriteStrings sound
this is used in CA rewriting, replacement of placeholders in
derivations, generating scripts for devShells, and some more
places. in all of these transitive replacements are unsound,
and overlapping replacements would be as well. there even is
a test that transitive replacements do not happen (in the CA
RewriteSink suite), but none for overlapping replacements. a
minimally surprising binary rewriter surely would not do any
of these replacements, the only reason we have not seen this
break yet is probably that rewriteStrings is only called for
store paths and things that look like store paths (and those
should never overlap nor admit such transitive replacements)

Change-Id: I6fc29f939d5061d9f56c752624a823ece8437c07
2024-05-03 00:50:31 +00:00
eldritch horrors a1ad4e52a6 filetransfer: don't decompress in curl wrapper itself
only decompress the response once all data has been received (in the
fully buffered case), or at least outside of the curl wrapper itself
(in the receive-to-sink case). unfortunately this means we will have
to duplicate decompression logic for these two cases for time being,
but once the curl wrapper has been rewritten to return a real future
or Source we can deduplicate this logic again. the curl wrapper will
have to turn into a proper Source first and use decompression source
logic which also does not currently exist—only decompression *sinks*

Change-Id: I66bc692f07d9b9e69fe10689ee73a2de8d65e35c
2024-04-26 15:26:37 +00:00
eldritch horrors fb0996aaa8 filetransfer: remove dataCallback from interface
this is highly questionable. single-arg download calls will misbehave
with it set, and two-arg download calls will just overwrite it. being
an implementation detail this should not have been in the API at all.

Change-Id: I613772951ee03d8302366085f06a53601d13f132
2024-04-26 15:26:37 +00:00
eldritch horrors dfe3baea12 filetransfer: make two-arg download abstract
this lets each implementation of FileTransfer (of which currently only
the one exists at all) implement appropriate handling for its internal
behaviours that are not otherwise exposed. in curl this lets us switch
the buffer-full handling method from "block the entire curl thread" to
"pause just the one transfer", move the non-libcurl body decompression
out of the actual curl wrapper (which will let us eventually morph the
curl wrapper intto an actual source of Sources), and some other things

Change-Id: Id6d3593cde6b4915aab3e90a43b175c103cc3f18
2024-04-26 15:26:37 +00:00
eldritch horrors 5420b3afd6 filetransfer: drop errorSink
just accumulate error data into result.data as we would for successful
transfers without a dataCallback. errorSink and data would contain the
same data in error cases anyway, so splitting them is not very useful.

Change-Id: I00e449866454389ac6a564ab411c903fd357dabf
2024-04-25 01:33:22 +02:00
eldritch horrors 5e69f8aa3d filetransfer: restore http status line reporting
this was broken in 75b62e5260.

Change-Id: If8583e802afbcde822623036bf41a9708fbc7c8d
2024-04-25 01:33:08 +02:00
eldritch horrors 38442e3123 filetransfer: remove decompress request parameter
this is never read.

Change-Id: I4c46f140519843a21e452958900e81edd2f78be2
2024-04-25 01:33:08 +02:00
eldritch horrors 86bfede948 libstore: use curl functions for reading headers
don't reimplement header parsing. this was only really needed due to the
ancient github bug we no longer care about, everything else we have done
in custom code can also be done using curl itself. doing this also fixes
possible sources of header smuggling (because the header function didn't
unfold headers and we'd trim them before parsing, which would've made us
read contents of one header as a fully formed header in itself). this is
a slight behavior change because we now honor only the first instance of
a given header where previous behavior was to honor either the last or a
combination of all of them (accept-ranges was logical-or'd by accident).

Change-Id: I93cb93ddb91ab98c8991f846014926f6ef039fdb
2024-04-23 01:04:56 +00:00
eldritch horrors 257d7ffa7b libstore: remove github etag workaround
this was a workaround for a *github* bug that happend *in 2015*.
not only is github no longer buggy, it shouldn't have been nix's
responsibility to work around these bugs like this to begin with

while we're at it we'll also remove another workaround—again for
github specifically and again for etag handling—from 2021 that's
also not needed any more. future workarounds for serverside bugs
should probably come with an expiration date that mutates into a
build warning after a while, otherwise this *will* happen again.

Change-Id: I74f739ae3e36d40350f78bebcb5869aa8cc9adcd
2024-04-23 01:04:56 +00:00
eldritch horrors ff9a4fc336 libstore: use curl_multi_{poll,wakeup}
the previous solution to the wakeup problem (adding a pipe and passing
it as an additional fd to curl_multi_wait) worked, but there have been
builtin alternatives for this since 2020. not only do these save code,
they're also a lot more likely to work natively on windows when needed

Change-Id: Iab751b900997110a8d15de45ea3ab0c42f7e5973
2024-04-22 21:37:20 +00:00
eldritch horrors e5903aab65 libstore: remove ancient libcurl feature checks
the oldest version checked for here is 7.47, which was released in
2016. it's probably safe to say that we do not need these any more

Change-Id: I003411f6b2ce6d56f7ca337390df3ea86bd59a99
2024-04-22 19:45:22 +00:00
eldritch horrors a326344253 tests: unhaunt the flakes nixos tests
these should really wait for networks to come up, otherwise they can fail.

fixes #235

Change-Id: I08989e8bdb0de280df74660ac43983de5c34fa9d
2024-04-18 20:09:19 +00:00
eldritch horrorsandGerrit Code Review e4a8c01bdf Merge changes Iedf46484,I76b51eac,I6a084827,I60193f9f into main
* changes:
  meson: fix log-dir
  manual: build docs with dummy envs
  libcmd: install generated headers as well
  docs: redo content generation for mdbook and manual
2024-04-11 14:33:16 +00:00
eldritch horrors cd79b8d65a meson: fix log-dir
the make build system sets this with an extra /nix segment.

Change-Id: Iedf464843196faeae5b59698837faca3a4f23586
2024-04-11 13:36:04 +00:00
eldritch horrors adab839c98 manual: build docs with dummy envs
this was previously used because the macOS docs build would otherwise
pull files out of the host nix store. or something. not sure about it

Change-Id: I76b51eac1ebc5de5f00e2e4be086dd8db3eeb8e6
2024-04-11 13:36:04 +00:00
eldritch horrors f42678802c libcmd: install generated headers as well
these seem to have been forgotten.

Change-Id: I6a084827d087f8098c19b62f2060a874d87202a1
2024-04-11 13:36:04 +00:00
eldritch horrors 725f5cd358 docs: redo content generation for mdbook and manual
manpages can be rendered using the markdown output of mdbook, the rest
of the manual can generated out of the main doc/manual source tree. we
still use lowdown to actually render manpages instead of eg mdbook-man
because lowdown does generate reasonably good manpages (though that is
also somewhat debatable, but they're a lot better than mdbook-man).

doing this not only lets us drastically simplify the lowdown pipeline,
but also remove all custom {{#include}} handling since now mdbook does
all of it, even for the manpage builds. even the lowdown wrapper isn't
entirely necessary because lowdown can take all wrapper arguments with
command line flags rather than bits of input file content.

This also implements running mdbook in Meson, in order to generate the
manpages. The mdbook outputs are also installed in the usual location.

Co-authored-by: Qyriad <qyriad@qyriad.me>

Change-Id: I60193f9fd0f15d48872f071af35855cda2a0f40b
2024-04-11 13:32:06 +00:00
eldritch horrors a0875f6adf libstore: fix glossary link in documentation
this should be a link, not an anchor. it should also point to the
`gloss-store` element, not the `#gloss-store` element.

Change-Id: I1f2803093179549637e10f917ad73399a419131b
2024-04-11 02:34:45 +02:00
eldritch horrors e6aab36d60 docs: put the manual build into a known tmpdir
this isn't strictly necessary, but it'll make it a lot easier to put the
generated files used by the autoconf build system in this directory too.
doing this now already will make the meson transition a lot easier later

Change-Id: I5fb39eade2ff88b6093c9ee436c9e8db793e9448
2024-04-09 02:09:36 +00:00
eldritch horrors c1ae35d34a docs: don't compute SUMMARY.md during build
this would make meson build compatibility unnecessarily hard and
the cli does not change often enough to justify this complexity.

Change-Id: I17b1870cdf8538feeaa01a9945db97af2175a642
2024-04-09 02:09:36 +00:00
eldritch horrors d9bc197ff4 docs: don't compute rl-next.md during build
not sure why this was done the way it was considering that includes are
a feature the doc toolchain had previously. let's just always have some
kind of entry for the upcoming release in the dev manual builds even if
that means having a completely empty release notes chapter.

the release notes generation script isn't entirely functional right now
due to pre-commit hooks, but it's good enough for time being. we need a
better release process for notes anyway.

Change-Id: Ifda6912cf5233db013f72a30247a62d6f22b1565

Change-Id: I9eb347ec4aabc5be2b816ff0fd3e4be45f93b934
2024-04-09 02:09:36 +00:00
eldritch horrors 07ef9f9f5d docs: skip do custom include processing for mdbook
mdbook already does include processing of its own, and the custom
processing code has always admitted as much. we don't need it for
the mdbook build at this point if we run our preprocessors in the
right order, and maybe we can even have mdbook to return complete
pages to us that we only have to pass to lowdown without any more
preprocessing of our own.

Change-Id: Icd978acbc3b1e215fee8f062c53ab2cb2a222ab1
2024-04-09 02:09:36 +00:00
eldritch horrors 14088e1ec5 docs: remove many link-to-self
for some reason these three were anchors, not links, but had they been
links they wouldn't've worked because they're not defined anywhere but
here. in the print version of the manual they're duplicated many times
over (creating id collisions), so we should better remove them anyway.

Change-Id: I8988a7c32c812dee0f0b6d4953faa7cd1255228d
2024-04-09 02:09:36 +00:00
eldritch horrors 1b5b9de04e flake: move release note checks to hydraJobs
having them in checks only does not run them in CI, which can cause
broken release notes entries to pass.

fixes #228

Change-Id: If0ba7b1be0b6525fc884a27e941cbc84b5a160f9
2024-04-06 15:51:52 +00:00
eldritch horrorsandGerrit Code Review e9e1b6963c Merge changes I1fa30114,I3ca208b6,Ide4c6e00,I74c46b9f,I05fa6a9d, ... into main
* changes:
  Revert "libutil: drop Pool resources on exceptional free"
  Revert "libutil: remove Pool::Handle::bad"
  Revert "libstore: remove one Resource::good flag"
  Revert "libstore: using throwing finally in withFramedSink"
  Revert "libutil: allow graceful dropping of Pool::Handle"
  Revert "libutil: drop Fs{Source,Sink}::good"
  libutil: guard Finally against invalid exception throws
2024-04-05 23:17:18 +00:00
eldritch horrors 38dc6f5b69 Revert "libutil: drop Pool resources on exceptional free"
This reverts commit de2884b82b376d10de5c400d8e73bc7d98f195d2.

Change-Id: I1fa301149d7c2ed3d266a40c15b2d010e12e44e6
2024-04-05 20:13:02 +00:00
eldritch horrors 52f741c23a Revert "libutil: remove Pool::Handle::bad"
This reverts commit 792844fb861ea7367ac2316c78fec055363f2f9e.

Change-Id: I3ca208b62edfd5cd1199478f75cd2edf19a364f6
2024-04-05 20:13:02 +00:00
eldritch horrors 0b8a17cab6 Revert "libstore: remove one Resource::good flag"
This reverts commit 87249eb579bf57f4f09e9fca100588a4d6b90b4c.

Change-Id: Ide4c6e00c4155216a17e46671ff47151d7bb85b4
2024-04-05 20:13:02 +00:00
eldritch horrors ad30972867 Revert "libstore: using throwing finally in withFramedSink"
This reverts commit 491caad6f62c21ffbcdebe662e63ec0f72e6f3a2.

this is not actually legal for nix! throwing exceptions in destructors
is fine, but the way nix is set up we'll end up throwing the exception
we received from the remote *twice* in some cases, and such cases will
cause an immediate terminate without active exception.

Change-Id: I74c46b9f26fd791086e4193ec60eb1deb9a5bb2a
2024-04-05 20:13:02 +00:00
eldritch horrors c77b6e1fdd Revert "libutil: allow graceful dropping of Pool::Handle"
This reverts commit 8075541d82d05347321d35b9934ccee5f82142f4.

Change-Id: I05fa6a9de1308a4827a6557cf2807eb47ca64da6
2024-04-05 20:13:02 +00:00
eldritch horrors 821ad98beb Revert "libutil: drop Fs{Source,Sink}::good"
This reverts commit 1340807e30dba4b3972c31f02861bbaeaeb60e61.

Change-Id: I34d2a80eb3c3e9d79cb02b92cd1189da32d18cb6
2024-04-05 20:13:02 +00:00
eldritch horrors 6c777476c9 libutil: guard Finally against invalid exception throws
throwing exceptions is fine, but throwing exceptions during exception
handling is hard enough to do correctly that we should just forbid it
entirely out of an overabundance of caution. in cases where terminate
is the correct answer the users of Finally must call it manually now.

Change-Id: Ia51a2cb4a0638500550bfabc89cf01a6d8098983
2024-04-05 20:13:02 +00:00
eldritch horrors d2aca15f31 doc: remove the -- unescaping hack
lowdown has an option to disable this behavior. use it.

Change-Id: I06d4201b7f528021ddb8e94ac6baff385402e080
2024-04-01 20:15:28 +00:00
eldritch horrors ed2bb919e7 meson: add switch to disable docs
the make build system can do this too.

Change-Id: I8c07d159cab54a8749c50dc33615f60bd251a86d
2024-04-01 20:15:28 +00:00
eldritch horrors 45623f077f libutil: drop Fs{Source,Sink}::good
setting this only on exceptions caused by actual fd access is not
sufficient to diagnose all errors (such as SerialisationError) in
some cases. this usually does not have any negative effects since
those errors will end up killing the process in another way. this
is not a reliable assumption though and we should be using proper
error handling (and closing connections more often, preferring to
close over keeping something open that might be in a weird state)

Change-Id: I1b792cd7ad8ba9ff0f6bd174945ab2575ff2208e
2024-03-31 16:42:40 +00:00
eldritch horrors f402c45cfa libutil: allow graceful dropping of Pool::Handle
not needed yet, but returning a resource from the exception handling
path that has ownership of a handle is currently not well-supported.
we could also add a default constructor to Handle, but then we would
also need to change the pool reference to a pointer. eventually that
should be done since now resources can be swapped between pools with
clever moves, but since that's not a problem yet we won't do it now.

Change-Id: I26eb06581f7be34569e9e67a33da736128d167af
2024-03-31 15:46:01 +00:00
eldritch horrors 2d3fd6b9ab libstore: using throwing finally in withFramedSink
the duplication of exception handling was added without justification,
so we can only assume that it was done like this because Finally could
not throw exceptions safely. since this has now been rectified we will
deduplicate this handler code again.

Change-Id: I40721f3378c0fd9f34e2914a16d383f6e2713b40
2024-03-31 14:23:15 +00:00
eldritch horrors dd06f9b792 libutil: make ~Finally noexcept(false)
this is supposed to act like a finally block does in other languages. a
finally block should be able to throw exceptions of its own rather than
just crashing the entire program when it throws it own exceptions. even
in the rare case of a finally throwing an unexpected exception it might
be better to report the exception from Finally instead of the original,
at least that can keep our program running instead of letting it crash.

Change-Id: Id42011e46b1df369152b4564938c0e93fa1acf32
2024-03-31 13:42:01 +00:00
eldritch horrors c777dcd1ae libstore: remove one Resource::good flag
usage of this flag previously kept connections open much longer than
necessary, and at the same time obscured that a connection was being
dropped when it *was* set. new variable names clarify this somewhat.

Change-Id: I11f6f08f37a5e4dc04ea6c6036ea589154b121c6
2024-03-31 00:52:41 +00:00
eldritch horrors 0b6d353474 libutil: remove Pool::Handle::bad
it was used incorrectly (not swapped on handle move), only used in one
place (that is now handled with exception handling detection in Handle
itself), and if ever reintroduced should be replaced with a different,
more understandable mechanism (like an explicit dropAsInvalid method).

Change-Id: Ie3e5d5cfa81d335429cb2ee5c3ad85c74a9df17b
2024-03-31 00:30:43 +00:00
eldritch horrors 862f20a4ba libutil: remove Pool::flushBad
this was never actually used, and bad design in the first place—why
should a bad resource be put back into the idle pool? just drop it.

Change-Id: Idab8774bee19dadae0209d404c4fb86dd4aeba1e
2024-03-31 00:07:09 +00:00
eldritch horrors 620de98d0c libutil: drop Pool resources on exceptional free
if a scope owning a resource does not gracefully drop that resource
while handling exceptions from deeper down the call stack we should
assume the resource is invalid state and drop it. currently it *is*
true that such cases do not cause resources to be freed, but thanks
to validator misuses this has so far not caused any larger problem.

Change-Id: Ie4f91bcd60a64d05c5ff9d22cc97954816d13b97
2024-03-30 23:40:45 +00:00
eldritch horrors 5956f509b6 meson: install libexec binaries
Change-Id: I149892bf081e1569d7786f085e890bc3d2eb50e5
2024-03-30 14:34:22 +00:00
eldritch horrors 351c1a9c52 meson: build and install nix3 manpages
this notably does *not* install the `nix3-manpages` manpage the old
system generated, mostly because that page was empty and just a bug
with a coat of documentation paint.

Change-Id: I7a4248a72e7bb5e0cc925a6311a33b6b72589569
2024-03-30 14:26:47 +00:00
eldritch horrors bfbcc3352b meson: add manpages for old cli and config files
Change-Id: I93384ec774d1945a649f6aaf7cd967c3fb7197f9
2024-03-30 14:26:47 +00:00
eldritch horrors 65d544b1a6 build: extract make "functions" into scripts
we'll want to use these for the meson builds, and probably eventually
rewrite them in something that isn't plain shell. diffoscope confirms
that out/share and doc/share are equal before and after these changes

Change-Id: I49aa418fc8615cad86d67328e08c28a7405ec952
2024-03-30 14:26:47 +00:00
eldritch horrors 5a54b0a20c meson: install systemd files
Change-Id: Idacf602fd379c82a051f00df2293cb02c8b286d4
2024-03-29 20:10:33 +00:00
eldritch horrors e28dc26084 meson: install shell files
Change-Id: I7c30690e5763d095cf7444333f7b687509051c5f
2024-03-29 20:10:33 +00:00
eldritch horrors 1da1f501fc meson: fix state-dir default value
the autoconf build system defaults to /nix/var, not /nix/var/nix. the
latter is only used in libstore, so we'll move the extra segment there.

Change-Id: Idfbc988ee302355982abdcd51d6d7b5d5d661c0d
2024-03-29 19:14:23 +00:00
eldritch horrors 6e5db5e4a2 meson: install missing/generated headers
one headers (args/root.hh) was simply missing, and the generated headers
were not installed. not all of them *should* be installed either, only a
select few (and sadly this needs a custom target for each one, it seems)

Change-Id: I37b25517895d0e5e521abc1202fa65624de57ed1
2024-03-29 02:45:48 +00:00
eldritch horrors 69bfd21e20 meson: install pkg-config files for libraries
Change-Id: I14b9d81d09f188eacfb9c68bcfb84751c18e3779
2024-03-29 02:45:48 +00:00
eldritch horrors 86b954a7af meson: increase functional test timeout
sometimes these fail with timeouts on loaded machines. let's up the
timeouts until we can pull the tests apart to more reasonable sizes

Change-Id: I2dfff2183cc1f3ff5e6107f43748ac046fe00d05
2024-03-29 02:19:36 +00:00
eldritch horrors 0436f4cfa6 flake: always build release notes in devshell
Change-Id: I0e02567fe8f102a8a8f1558aa094eefacdac9393
2024-03-27 03:09:14 +00:00
eldritch horrors 279e30e7ef build: replace changelog-d with local script
hacking changelog-d to support not just github but also forgejo and
gerrit is a lot more complicated than it's worth, even moreso since
the entire thing can just as well be done with ~60 lines of python.
this new script is also much cheaper to instantiate (being python),
so having it enabled in all shells is far less of a hassle.

we've also adjusted existing release notes that referenced a gerrit
cl to auto-link to the cl in question, making the diff a bit bigger

closes https://git.lix.systems/lix-project/lix/issues/176

Change-Id: I8ba7dd0070aad9ba4474401731215fcf5d9d2130
2024-03-27 03:09:14 +00:00