Commit Graph
86 Commits
Author SHA1 Message Date
Robert HensingandRebecca Turner ee0c195eba Split ignoreException to avoid suppressing CTRL-C
This splits `ignoreException` into `ignoreExceptionExceptInterrupt`
(which ignores all exceptions except `Interrupt`, which indicates a
SIGINT/CTRL-C) and `ignoreExceptionInDestructor` (which ignores all
exceptions, so that destructors do not throw exceptions).

This prevents many cases where Nix ignores CTRL-C entirely.
See: https://github.com/NixOS/nix/issues/7245

Upstream-PR: https://github.com/NixOS/nix/pull/11618
Change-Id: Ie7d2467eedbe840d1b9fa2e88a4e88e4ab26a87b
2024-10-01 15:49:56 -07:00
Jade LovelaceandRebecca Turner 04f8a14833 tree-wide: shuffle headers around for about 30s compile time
This didn't really feel so worth it afterwards, but I did untangle a
bunch of stuff that should not have been tangled.

The general gist of this change is that variant bullshit was causing a
bunch of compile time, and it seems like the only way to deal with
variant induced compile time is to keep variant types out of headers.
Explicit template instantiation seems to do nothing for them.

I also seem to have gotten some back-end time improvement from
explicitly instantiating regex, but I don't know why. There is no
corresponding front-end time improvement from it: regex is still at the
top of the sinners list.

**** Templates that took longest to instantiate:
 15231 ms: std::basic_regex<char>::_M_compile (28 times, avg 543 ms)
 15066 ms: std::__detail::_Compiler<std::regex_traits<char>>::_Compiler (28 times, avg 538 ms)
 12571 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_disjunction (28 times, avg 448 ms)
 12454 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_alternative (28 times, avg 444 ms)
 12225 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_term (28 times, avg 436 ms)
 11363 ms: nlohmann::basic_json<>::parse<const char *> (21 times, avg 541 ms)
 10628 ms: nlohmann::basic_json<>::basic_json (109 times, avg 97 ms)
 10134 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_atom (28 times, avg 361 ms)

Back-end time before messing with the regex:
**** Function sets that took longest to compile / optimize:
  8076 ms: void boost::io::detail::put<$>(boost::io::detail::put_holder<$> cons... (177 times, avg 45 ms)
  4382 ms: std::_Rb_tree<$>::_M_erase(std::_Rb_tree_node<$>*) (1247 times, avg 3 ms)
  3137 ms: boost::stacktrace::detail::to_string_impl_base<boost::stacktrace::de... (137 times, avg 22 ms)
  2896 ms: void boost::io::detail::mk_str<$>(std::__cxx11::basic_string<$>&, ch... (177 times, avg 16 ms)
  2304 ms: std::_Rb_tree<$>::_M_get_insert_hint_unique_pos(std::_Rb_tree_const_... (210 times, avg 10 ms)
  2116 ms: bool std::__detail::_Compiler<$>::_M_expression_term<$>(std::__detai... (112 times, avg 18 ms)
  2051 ms: std::_Rb_tree_iterator<$> std::_Rb_tree<$>::_M_emplace_hint_unique<$... (244 times, avg 8 ms)
  2037 ms: toml::result<$> toml::detail::sequence<$>::invoke<$>(toml::detail::l... (93 times, avg 21 ms)
  1928 ms: std::__detail::_Compiler<$>::_M_quantifier() (28 times, avg 68 ms)
  1859 ms: nlohmann::json_abi_v3_11_3::detail::serializer<$>::dump(nlohmann::js... (41 times, avg 45 ms)
  1824 ms: std::_Function_handler<$>::_M_manager(std::_Any_data&, std::_Any_dat... (973 times, avg 1 ms)
  1810 ms: std::__detail::_BracketMatcher<$>::_BracketMatcher(std::__detail::_B... (112 times, avg 16 ms)
  1793 ms: nix::fetchers::GitInputScheme::fetch(nix::ref<$>, nix::fetchers::Inp... (1 times, avg 1793 ms)
  1759 ms: std::_Rb_tree<$>::_M_get_insert_unique_pos(std::__cxx11::basic_strin... (281 times, avg 6 ms)
  1722 ms: bool nlohmann::json_abi_v3_11_3::detail::parser<$>::sax_parse_intern... (19 times, avg 90 ms)
  1677 ms: boost::io::basic_altstringbuf<$>::overflow(int) (194 times, avg 8 ms)
  1674 ms: std::__cxx11::basic_string<$>::_M_mutate(unsigned long, unsigned lon... (249 times, avg 6 ms)
  1660 ms: std::_Rb_tree_node<$>* std::_Rb_tree<$>::_M_copy<$>(std::_Rb_tree_no... (304 times, avg 5 ms)
  1599 ms: bool nlohmann::json_abi_v3_11_3::detail::parser<$>::sax_parse_intern... (19 times, avg 84 ms)
  1568 ms: void std::__detail::_Compiler<$>::_M_insert_bracket_matcher<$>(bool) (112 times, avg 14 ms)
  1541 ms: std::__shared_ptr<$>::~__shared_ptr() (531 times, avg 2 ms)
  1539 ms: nlohmann::json_abi_v3_11_3::detail::serializer<$>::dump_escaped(std:... (41 times, avg 37 ms)
  1471 ms: void std::__detail::_Compiler<$>::_M_insert_character_class_matcher<... (112 times, avg 13 ms)

After messing with the regex (notice std::__detail::_Compiler vanishes
here, but I don't know why):

**** Function sets that took longest to compile / optimize:
  8054 ms: void boost::io::detail::put<$>(boost::io::detail::put_holder<$> cons... (177 times, avg 45 ms)
  4313 ms: std::_Rb_tree<$>::_M_erase(std::_Rb_tree_node<$>*) (1217 times, avg 3 ms)
  3259 ms: boost::stacktrace::detail::to_string_impl_base<boost::stacktrace::de... (137 times, avg 23 ms)
  3045 ms: void boost::io::detail::mk_str<$>(std::__cxx11::basic_string<$>&, ch... (177 times, avg 17 ms)
  2314 ms: std::_Rb_tree<$>::_M_get_insert_hint_unique_pos(std::_Rb_tree_const_... (207 times, avg 11 ms)
  1923 ms: std::_Rb_tree_iterator<$> std::_Rb_tree<$>::_M_emplace_hint_unique<$... (216 times, avg 8 ms)
  1817 ms: bool nlohmann::json_abi_v3_11_3::detail::parser<$>::sax_parse_intern... (18 times, avg 100 ms)
  1816 ms: toml::result<$> toml::detail::sequence<$>::invoke<$>(toml::detail::l... (93 times, avg 19 ms)
  1788 ms: nlohmann::json_abi_v3_11_3::detail::serializer<$>::dump(nlohmann::js... (40 times, avg 44 ms)
  1749 ms: std::_Rb_tree<$>::_M_get_insert_unique_pos(std::__cxx11::basic_strin... (278 times, avg 6 ms)
  1724 ms: std::__cxx11::basic_string<$>::_M_mutate(unsigned long, unsigned lon... (248 times, avg 6 ms)
  1697 ms: boost::io::basic_altstringbuf<$>::overflow(int) (194 times, avg 8 ms)
  1684 ms: nix::fetchers::GitInputScheme::fetch(nix::ref<$>, nix::fetchers::Inp... (1 times, avg 1684 ms)
  1680 ms: std::_Rb_tree_node<$>* std::_Rb_tree<$>::_M_copy<$>(std::_Rb_tree_no... (303 times, avg 5 ms)
  1589 ms: bool nlohmann::json_abi_v3_11_3::detail::parser<$>::sax_parse_intern... (18 times, avg 88 ms)
  1483 ms: non-virtual thunk to boost::wrapexcept<$>::~wrapexcept() (181 times, avg 8 ms)
  1447 ms: nlohmann::json_abi_v3_11_3::detail::serializer<$>::dump_escaped(std:... (40 times, avg 36 ms)
  1441 ms: std::__shared_ptr<$>::~__shared_ptr() (496 times, avg 2 ms)
  1420 ms: boost::stacktrace::basic_stacktrace<$>::init(unsigned long, unsigned... (137 times, avg 10 ms)
  1396 ms: boost::basic_format<$>::~basic_format() (194 times, avg 7 ms)
  1290 ms: std::__cxx11::basic_string<$>::_M_replace_cold(char*, unsigned long,... (231 times, avg 5 ms)
  1258 ms: std::vector<$>::~vector() (354 times, avg 3 ms)
  1222 ms: std::__cxx11::basic_string<$>::_M_replace(unsigned long, unsigned lo... (231 times, avg 5 ms)
  1194 ms: std::_Rb_tree<$>::_M_get_insert_hint_unique_pos(std::_Rb_tree_const_... (49 times, avg 24 ms)
  1186 ms: bool tao::pegtl::internal::sor<$>::match<$>(std::integer_sequence<$>... (1 times, avg 1186 ms)
  1149 ms: std::__detail::_Executor<$>::_M_dfs(std::__detail::_Executor<$>::_Ma... (70 times, avg 16 ms)
  1123 ms: toml::detail::sequence<$>::invoke(toml::detail::location&) (69 times, avg 16 ms)
  1110 ms: nlohmann::json_abi_v3_11_3::basic_json<$>::json_value::destroy(nlohm... (55 times, avg 20 ms)
  1079 ms: std::_Function_handler<$>::_M_manager(std::_Any_data&, std::_Any_dat... (541 times, avg 1 ms)
  1033 ms: nlohmann::json_abi_v3_11_3::detail::lexer<$>::scan_number() (20 times, avg 51 ms)

Change-Id: I10af282bcd4fc39c2d3caae3453e599e4639c70b
2024-08-28 09:55:05 -07:00
Jade LovelaceandRebecca Turner 4d89844207 build: remove about 30 cpu-sec of compile time by explicit instantiation
Apparently the fmt contraption has some extremely popular overloads, and
the boost stuff in there gets built approximately infinite times in
every compilation unit.

Change-Id: Ideba2db7d6bf8559e4d91974bab636f5ed106198
2024-08-28 09:52:05 -07:00
Jade LovelaceandRebecca Turner 0cc285f87b treewide: fix a bunch of lints
Fixes:
- Identifiers starting with _ are prohibited
- Some driveby header dependency cleaning which wound up with doing some
  extra fixups.
- Fucking C style casts, man. C++ made these 1000% worse by letting you
  also do memory corruption with them with references.
  - Remove casts to Expr * where ExprBlackHole is an incomplete type by
    introducing an explicitly-cast eBlackHoleAddr as Expr *.
  - An incredibly illegal cast of the text bytes of the StorePath hash
    into a size_t directly. You can't DO THAT.

    Replaced with actually parsing the hash so we get 100% of the bits
    being entropy, then memcpying the start of the hash. If this shows
    up in a profile we should just make the hash parser faster with a
    lookup table or something sensible like that.
  - This horrendous bit of UB which I thankfully slapped a deprecation
    warning on, built, and it didn't trigger anywhere so it was dead
    code and I just deleted it. But holy crap you *cannot* do that.

    inline void mkString(const Symbol & s)
    {
        mkString(((const std::string &) s).c_str());
    }
- Some wrong lints. Lots of wrong macro lints, one wrong
  suspicious-sizeof lint triggered by the template being instantiated
  with only pointers, but the calculation being correct for both
  pointers and not-pointers.
- Exceptions in destructors strike again. I tried to catch the
  exceptions that might actually happen rather than all the exceptions
  imaginable. We can let the runtime hard-kill it on other exceptions
  imo.

Change-Id: I71761620846cba64d66ee7ca231b20c061e69710
2024-08-26 16:13:03 -07:00
eldritch horrors ef0de7c79f remove boost coroutine references
we no longer need these since sinkToSource and sourceToSink are gone.

Change-Id: Ibbf440e2cf71bf3e9f3b833af2d78a21fb1b3193
2024-07-16 01:50:16 +00:00
eldritch horrors dfedbc154f remove sourceToSink, sinkToSource, and boehm patch
Change-Id: I1379841299713175d0225b82a67f50660f9eb5e2
2024-07-16 01:50:16 +00:00
eldritch horrors 5eec6418de libutil: begin porting serialization to generators
generators are a better basis for serializers than streaming into sinks
as we do currently for many reasons, such as being usable as sources if
one wishes to (without requiring an intermediate sink to serialize full
data sets into memory, or boost coroutines to turn sinks into sources),
composing more naturally (as one can just yield a sub-generator instead
of being forced to wrap entire substreams into clunky functions or even
more clunky custom types to implement operator<< on), allowing wrappers
to transform data with clear ownership semantics (removing the need for
explicit memory allocations and Source wrappers), and many other things

Change-Id: I361d89ff556354f6930d9204f55117565f2f7f20
2024-07-03 11:46:53 +00:00
eldritch horrors 39a1e248c9 libutil: remove sinkToSource eof callback
this is only used in one place, and only to set a nicer error message on
EndOfFile. the only caller that actually *catches* this exception should
provide an error message in that catch block rather than forcing support
for setting error message so deep into the stack. copyStorePath is never
called outside of PathSubstitutionGoal anyway, which catches everything.

Change-Id: Ifbae8706d781c388737706faf4c8a8b7917ca278
2024-06-23 11:52:49 +00:00
Tom Hubrecht a39ba22ff7 util.hh: Delete remaining file and clean up headers
Change-Id: Ic1f68e6af658e94ef7922841dd3ad4c69551ef56
2024-05-29 12:38:51 +02:00
eldritch horrors 29f93e1e0d libutil: throw EndOfFile at sourceToSink end
... how has this never broken anything before

Change-Id: If3789c02028e8f929481514f63d76b0b46bfc182
2024-05-07 14:35:20 +00:00
eldritch horrors 821ad98beb Revert "libutil: drop Fs{Source,Sink}::good"
This reverts commit 1340807e30dba4b3972c31f02861bbaeaeb60e61.

Change-Id: I34d2a80eb3c3e9d79cb02b92cd1189da32d18cb6
2024-04-05 20:13:02 +00:00
eldritch horrors 45623f077f libutil: drop Fs{Source,Sink}::good
setting this only on exceptions caused by actual fd access is not
sufficient to diagnose all errors (such as SerialisationError) in
some cases. this usually does not have any negative effects since
those errors will end up killing the process in another way. this
is not a reliable assumption though and we should be using proper
error handling (and closing connections more often, preferring to
close over keeping something open that might be in a weird state)

Change-Id: I1b792cd7ad8ba9ff0f6bd174945ab2575ff2208e
2024-03-31 16:42:40 +00:00
Jade Lovelace 8be7030299 util.hh: split out signals stuff
Copies part of the changes of ac89bb064aeea85a62b82a6daf0ecca7190a28b7

Change-Id: I9ce601875cd6d4db5eb1132d7835c5bab9f126d8
2024-03-11 00:52:09 -07:00
eldritch horrors b221a14f0a Merge pull request #9925 from 9999years/fmt-cleanup
Cleanup `fmt.hh`

(cherry picked from commit 47a1dbb4b8e7913cbb9b4d604728b912e76e4ca0)
Change-Id: Id076a45cb39652f437fe3f8bda10c310a9894777
2024-03-09 07:00:13 -07:00
eldritch horrors 68f148ed45 Merge pull request #9798 from edolstra/remote-store-eof
Print a more helpful message if the daemon crashes

(cherry picked from commit 32706b14a7531c2c21b9f96da083a540a0031ec4)
Change-Id: Ief7c465bca7666e2b7e7c9d1dd0c01c5f9014146
2024-03-04 08:12:15 +01:00
eldritch horrors cd326a2aa4 Merge pull request #9673 from pennae/drv-parse-opts
optimize derivation parsing

(cherry picked from commit 3511430902941f0f26dc71313a54bb5096f57305)
Change-Id: I00f76dcd464a5811944613731501af504b6e8c29
2024-03-04 07:36:51 +01:00
afb55f36df Fix bad_format_string error when builder stdout contains % (#10063)
(cherry picked from commit e2b6821ca0147f36bcb9404aab080f80746984c8)

Co-authored-by: roblabla <unfiltered@roblab.la>
2024-02-22 23:12:29 +01:00
Eelco DolstraandGitHub 33fc09c2a5 Merge pull request #8176 from tweag/rename-confusing-write-method
Rename and protect `BufferedSink::write`
2023-04-14 10:44:36 +02:00
Yorick van PeltandThéophane Hufschmitt 58d24a4cb6 Always disable GC in a coroutine unless the patch is applied 2023-04-07 14:54:38 +02:00
Yorick van PeltandThéophane Hufschmitt 00bc34430b DisableGC: replace by CoroutineContext, std::shared_ptr<void> 2023-04-07 14:53:40 +02:00
Yorick van PeltandThéophane Hufschmitt 2c53ef1bfe Disable GC inside coroutines on mac OS 2023-04-07 14:52:59 +02:00
Théophane Hufschmitt 214f1d6791 Rename and protect BufferedSink::write
The `write` name is ambiguous and could lead to some funny bugs like
https://github.com/NixOS/nix/pull/8173#issuecomment-1500009480. So
rename it to the more explicit `writeUnbuffered`.
Besides, this method shouldn't be (and isn't) used outside of the class
implementation, so mark it `protected`.

This makes it more symetrical to `BufferedSource` which uses a
`protected readUnbuffered` method.
2023-04-07 09:21:50 +02:00
Théophane HufschmittandGitHub 39700c5cbe Revert "Disable GC during coroutine execution + test" 2023-03-08 20:47:52 +01:00
Théophane HufschmittandGitHub 4a6244dcf7 Merge pull request #7725 from yorickvP/check-coro-gc
Disable GC during coroutine execution + test
2023-03-08 09:51:46 +01:00
Eelco Dolstra 29abc8e764 Remove FormatOrString and remaining uses of format() 2023-03-02 15:57:54 +01:00
Yorick van Pelt 176005749c Always disable GC in a coroutine unless the patch is applied 2023-03-01 15:07:00 +01:00
Yorick van Pelt 4c73eab923 DisableGC: replace by CoroutineContext, std::shared_ptr<void> 2023-03-01 13:55:41 +01:00
Yorick van Pelt eaeb994d8b Disable GC inside coroutines on mac OS 2023-03-01 13:55:41 +01:00
Eelco Dolstra 703d863a48 Trivial changes from the lazy-trees branch 2022-12-07 14:06:34 +01:00
Eelco Dolstra 53e7b7e8ac Remove warnLargeDump()
This message was unhelpful (#1184) and probably misleading since
memory is O(1) in most cases now.
2022-08-17 11:32:01 +02:00
Eelco Dolstra 168ef9f3ab Remove unused Error.name field 2022-04-08 11:31:51 +02:00
Eelco Dolstra df552ff53e Remove std::string alias (for real this time)
Also use std::string_view in a few more places.
2022-02-25 16:13:02 +01:00
Eelco Dolstra d62a9390fc Get rid of std::shared_ptr<std::string> and ref<std::string>
These were needed back in the pre-C++11 era because we didn't have
move semantics. But now we do.
2022-01-18 11:12:30 +01:00
Eelco Dolstra 776eb97a43 serialise.hh: Use std::string_view 2022-01-17 22:20:23 +01:00
Eelco Dolstra 1ba993d07c Fix clang warning 2021-09-14 08:15:33 +02:00
Yorick van PeltandLars Jellema 8a0c00b856 Use libarchive for all compression 2021-03-10 22:34:29 +01:00
Eelco Dolstra 8d4268d190 Improve error formatting
Changes:

* The divider lines are gone. These were in practice a bit confusing,
  in particular with --show-trace or --keep-going, since then there
  were multiple lines, suggesting a start/end which wasn't the case.

* Instead, multi-line error messages are now indented to align with
  the prefix (e.g. "error: ").

* The 'description' field is gone since we weren't really using it.

* 'hint' is renamed to 'msg' since it really wasn't a hint.

* The error is now printed *before* the location info.

* The 'name' field is no longer printed since most of the time it
  wasn't very useful since it was just the name of the exception (like
  EvalError). Ideally in the future this would be a unique, easily
  googleable error ID (like rustc).

* "trace:" is now just "…". This assumes error contexts start with
  something like "while doing X".

Example before:

  error: --- AssertionError ---------------------------------------------------------------------------------------- nix
  at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix

       6|
       7|   x = assert false; 1;
        |       ^
       8|

  assertion 'false' failed
  ----------------------------------------------------- show-trace -----------------------------------------------------
  trace: while evaluating the attribute 'x' of the derivation 'hello-2.10'
  at: (192:11) in file: /home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/make-derivation.nix

     191|         // (lib.optionalAttrs (!(attrs ? name) && attrs ? pname && attrs ? version)) {
     192|           name = "${attrs.pname}-${attrs.version}";
        |           ^
     193|         } // (lib.optionalAttrs (stdenv.hostPlatform != stdenv.buildPlatform && !dontAddHostSuffix && (attrs ? name || (attrs ? pname && attrs ? version)))) {

Example after:

  error: assertion 'false' failed

         at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix

              6|
              7|   x = assert false; 1;
               |       ^
              8|

         … while evaluating the attribute 'x' of the derivation 'hello-2.10'

         at: (192:11) in file: /home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/make-derivation.nix

            191|         // (lib.optionalAttrs (!(attrs ? name) && attrs ? pname && attrs ? version)) {
            192|           name = "${attrs.pname}-${attrs.version}";
               |           ^
            193|         } // (lib.optionalAttrs (stdenv.hostPlatform != stdenv.buildPlatform && !dontAddHostSuffix && (attrs ? name || (attrs ? pname && attrs ? version)))) {
2021-01-21 11:02:09 +01:00
Eelco Dolstra 1b79b5b983 read(): Use char * instead of unsigned char *
This gets rid of some pointless casts.
2020-12-02 14:17:27 +01:00
Eelco Dolstra faa31f4084 Sink: Use std::string_view 2020-12-02 14:17:27 +01:00
Ricardo M. Correia 108a2dab7e Fix stack overflow introduced in #4206 2020-11-10 04:25:24 +01:00
Robert Hensing c4d903ddb0 Fix memory corruption caused by GC-invisible coroutine stacks
Crucially this introduces BoehmGCStackAllocator, but it also
adds a bunch of wiring to avoid making libutil depend on bdw-gc.

Part of the solutions for #4178, #4200
2020-10-30 21:21:59 +01:00
Eelco Dolstra be149acfda Serialize exceptions from the sandbox process to the parent
Fixes #4118.
2020-10-07 16:34:03 +02:00
Robert Hensing 29c82ccc77 Add Source.drainInto(Sink) 2020-09-17 20:21:04 +02:00
John Ericson 5602637d9e Revert "LocalStore::addToStoreFromDump copy in chunks"
This reverts commit 592851fb67. We don't
need this extra feature anymore
2020-07-15 23:37:49 +00:00
John Ericson bc109648c4 Get rid of LocalStore::addToStoreCommon
I got it to just become `LocalStore::addToStoreFromDump`, cleanly taking
a store and then doing nothing too fancy with it.

`LocalStore::addToStore(...Path...)` is now just a simple wrapper with a
bare-bones sinkToSource of the right dump command.
2020-07-15 23:14:30 +00:00
John Ericson 592851fb67 LocalStore::addToStoreFromDump copy in chunks
Rather than copying byte-by-byte, we let the coroutine know how much
data we would like it to send back to us.
2020-07-14 13:56:36 +00:00
Eelco Dolstra 1fb762d11f Get rid of explicit ErrorInfo constructors 2020-06-15 14:06:58 +02:00
Ben Burdette 958e81987b switch from printError warnings to logWarnings 2020-05-11 13:02:16 -06:00
Eelco Dolstra ef52ccf035 experimental/optional -> optional 2019-03-14 14:10:52 +01:00
Eelco Dolstra ed78582847 sinkToSource(): Start the coroutine lazily
In particular this causes copyStorePath() from HttpBinaryCacheStore to
only start a download if needed. E.g. if the destination LocalStore
goes to sleep waiting for the path lock and another process creates
the path, then LocalStore::addToStore() will never read from the
source so we don't have to do the download.
2018-09-26 21:19:34 +02:00