guiwf/lix
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
16,413 Commits 37 Branches 40 Tags
61146c73ce4e102bbf63f16caa32b0c60cd5a766
Commit Graph
42 Commits
Author SHA1 Message Date
eldritch horrors 982d049d3b libstore: remove FileTransferResult::data 
return it as a separate item in a pair instead. this will let us remove
enqueueDownload() in favor of returning metadata from download() itself

Change-Id: I74fad2ca15f920da1eefabc950c2baa2c360f2ba
 2024-10-27 21:44:38 +00:00
eldritch horrors 5cd7055044 libstore: de-future-ize FileTransfer::enqueueUpload 
it's only used once, and synchronously at that.

Change-Id: Ife9db15dd97bc0de8de59a25d27f3f7afeb8791b
 2024-10-27 21:44:38 +00:00
eldritch horrors 6f18e1ebde libstore: remove FileTransferRequest 
it's just a uri and some headers now. those can be function arguments
with no loss of clarity. *actual* additional arguments, for example a
TLS context with additional certificates, could be added on a new and
improved FileTransfer class that carries not just a backend reference
but some real, visible context for its transfers. curl not being very
multi-threading-friendly when using multi handles will make sharing a
bit hard anyway once we drop the single global download worker thread

Change-Id: Id2112c95cbd118c6d920488f38d272d7da926460
 2024-10-27 21:44:38 +00:00
eldritch horrors a839c31e6c libstore: remove FileTransferRequest::expectedETag 
just another http specific used in only one place.

Change-Id: I99361a7226f4e6cd8f18170d3683c0025657bcb3
 2024-10-27 21:44:38 +00:00
eldritch horrors 30bec83fa4 libstore: remove FileTransferResult::bodySize 
it's only used for internal bookkeeping in TransferItem.

Change-Id: I467c5be023488be4a8a76e5f98a4ef25762df6f3
 2024-10-27 21:44:38 +00:00
eldritch horrors d82b212d33 libstore: remove FileTransferRequest::verifyTLS 
it's never set to false.

Change-Id: I1e436c82f1097091a08faa1dfada75e51bd5edf9
 2024-10-27 21:44:38 +00:00
eldritch horrors 220251ba51 libstore: remove FiletransferRequest::head 
add a method to FileTransfer that provides this functionality instead.

Change-Id: Ic1933a5df76a109c248c9c5efea065356b20a6f9
 2024-10-27 21:44:38 +00:00
eldritch horrors 6c2609c5f9 libstore: remove FileTransferRequest::tries 
it's never set, and then only used internally. *once*.

Change-Id: I32585b1821e979f3ebb53b794ba0d1f576126b92
 2024-10-26 21:42:35 +00:00
eldritch horrors af27d1ecd8 libstore: make baseRetryTimeMs a FileTransfer property 
we don't even need this outside of tests. maybe we should not do
automatic retries at this level at all and use retrying wrappers
instead? at some point we may have to do this, but not just yet.

Change-Id: If0088aa55215be81f1770c25b3bb1b5268c65cf8
 2024-10-26 21:42:35 +00:00
eldritch horrors 1e3b45546c libstore: remove FileTransferRequest::parentAct 
never set explicitly, and transfers are never instantiated with one
current activity but submitted with a *different* current activity.

Change-Id: I1a3ec57c02013565aeb9e9398ea42d0c4279095e
 2024-10-26 21:42:35 +00:00
eldritch horrors ce3e1d1e7a libstore: remove FileTransferRequests::data 
use separate upload and download methods instead.

Change-Id: I5baa2177c8ddd70268c75ff074e361b2f17dddbd
 2024-10-26 21:42:35 +00:00
eldritch horrors 2d49efaa2e libstore: remove Filetransfer::transfer 
just use enqueueFileTransfer().get() insteaad.

Change-Id: I67a43c9d3d5f68ac3f9e8ba7973c243dd78b86a3
 2024-10-26 21:42:35 +00:00
eldritch horrors 98b55c3a1d libstore: move FileTransferRequest::verb to TransferItem 
this function is only used internally by curl wrapper.

Change-Id: I71d4c430cb069e2c949be769c17fede8dd04d480
 2024-10-26 21:42:35 +00:00
eldritch horrors a83bf24281 libstore: remove FileTransferRequest::mimeType 
it's only used by HttpBinaryCacheStore, and even there used in only on
place. this one place can set the header explicitly, which it now does

Change-Id: Id89228150669e25e7f59a3d6bd939e46059ce29e
 2024-10-26 21:42:35 +00:00
Jade LovelaceandRebecca Turner 04f8a14833 tree-wide: shuffle headers around for about 30s compile time 
This didn't really feel so worth it afterwards, but I did untangle a
bunch of stuff that should not have been tangled.

The general gist of this change is that variant bullshit was causing a
bunch of compile time, and it seems like the only way to deal with
variant induced compile time is to keep variant types out of headers.
Explicit template instantiation seems to do nothing for them.

I also seem to have gotten some back-end time improvement from
explicitly instantiating regex, but I don't know why. There is no
corresponding front-end time improvement from it: regex is still at the
top of the sinners list.

**** Templates that took longest to instantiate:
 15231 ms: std::basic_regex<char>::_M_compile (28 times, avg 543 ms)
 15066 ms: std::__detail::_Compiler<std::regex_traits<char>>::_Compiler (28 times, avg 538 ms)
 12571 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_disjunction (28 times, avg 448 ms)
 12454 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_alternative (28 times, avg 444 ms)
 12225 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_term (28 times, avg 436 ms)
 11363 ms: nlohmann::basic_json<>::parse<const char *> (21 times, avg 541 ms)
 10628 ms: nlohmann::basic_json<>::basic_json (109 times, avg 97 ms)
 10134 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_atom (28 times, avg 361 ms)

Back-end time before messing with the regex:
**** Function sets that took longest to compile / optimize:
  8076 ms: void boost::io::detail::put<$>(boost::io::detail::put_holder<$> cons... (177 times, avg 45 ms)
  4382 ms: std::_Rb_tree<$>::_M_erase(std::_Rb_tree_node<$>*) (1247 times, avg 3 ms)
  3137 ms: boost::stacktrace::detail::to_string_impl_base<boost::stacktrace::de... (137 times, avg 22 ms)
  2896 ms: void boost::io::detail::mk_str<$>(std::__cxx11::basic_string<$>&, ch... (177 times, avg 16 ms)
  2304 ms: std::_Rb_tree<$>::_M_get_insert_hint_unique_pos(std::_Rb_tree_const_... (210 times, avg 10 ms)
  2116 ms: bool std::__detail::_Compiler<$>::_M_expression_term<$>(std::__detai... (112 times, avg 18 ms)
  2051 ms: std::_Rb_tree_iterator<$> std::_Rb_tree<$>::_M_emplace_hint_unique<$... (244 times, avg 8 ms)
  2037 ms: toml::result<$> toml::detail::sequence<$>::invoke<$>(toml::detail::l... (93 times, avg 21 ms)
  1928 ms: std::__detail::_Compiler<$>::_M_quantifier() (28 times, avg 68 ms)
  1859 ms: nlohmann::json_abi_v3_11_3::detail::serializer<$>::dump(nlohmann::js... (41 times, avg 45 ms)
  1824 ms: std::_Function_handler<$>::_M_manager(std::_Any_data&, std::_Any_dat... (973 times, avg 1 ms)
  1810 ms: std::__detail::_BracketMatcher<$>::_BracketMatcher(std::__detail::_B... (112 times, avg 16 ms)
  1793 ms: nix::fetchers::GitInputScheme::fetch(nix::ref<$>, nix::fetchers::Inp... (1 times, avg 1793 ms)
  1759 ms: std::_Rb_tree<$>::_M_get_insert_unique_pos(std::__cxx11::basic_strin... (281 times, avg 6 ms)
  1722 ms: bool nlohmann::json_abi_v3_11_3::detail::parser<$>::sax_parse_intern... (19 times, avg 90 ms)
  1677 ms: boost::io::basic_altstringbuf<$>::overflow(int) (194 times, avg 8 ms)
  1674 ms: std::__cxx11::basic_string<$>::_M_mutate(unsigned long, unsigned lon... (249 times, avg 6 ms)
  1660 ms: std::_Rb_tree_node<$>* std::_Rb_tree<$>::_M_copy<$>(std::_Rb_tree_no... (304 times, avg 5 ms)
  1599 ms: bool nlohmann::json_abi_v3_11_3::detail::parser<$>::sax_parse_intern... (19 times, avg 84 ms)
  1568 ms: void std::__detail::_Compiler<$>::_M_insert_bracket_matcher<$>(bool) (112 times, avg 14 ms)
  1541 ms: std::__shared_ptr<$>::~__shared_ptr() (531 times, avg 2 ms)
  1539 ms: nlohmann::json_abi_v3_11_3::detail::serializer<$>::dump_escaped(std:... (41 times, avg 37 ms)
  1471 ms: void std::__detail::_Compiler<$>::_M_insert_character_class_matcher<... (112 times, avg 13 ms)

After messing with the regex (notice std::__detail::_Compiler vanishes
here, but I don't know why):

**** Function sets that took longest to compile / optimize:
  8054 ms: void boost::io::detail::put<$>(boost::io::detail::put_holder<$> cons... (177 times, avg 45 ms)
  4313 ms: std::_Rb_tree<$>::_M_erase(std::_Rb_tree_node<$>*) (1217 times, avg 3 ms)
  3259 ms: boost::stacktrace::detail::to_string_impl_base<boost::stacktrace::de... (137 times, avg 23 ms)
  3045 ms: void boost::io::detail::mk_str<$>(std::__cxx11::basic_string<$>&, ch... (177 times, avg 17 ms)
  2314 ms: std::_Rb_tree<$>::_M_get_insert_hint_unique_pos(std::_Rb_tree_const_... (207 times, avg 11 ms)
  1923 ms: std::_Rb_tree_iterator<$> std::_Rb_tree<$>::_M_emplace_hint_unique<$... (216 times, avg 8 ms)
  1817 ms: bool nlohmann::json_abi_v3_11_3::detail::parser<$>::sax_parse_intern... (18 times, avg 100 ms)
  1816 ms: toml::result<$> toml::detail::sequence<$>::invoke<$>(toml::detail::l... (93 times, avg 19 ms)
  1788 ms: nlohmann::json_abi_v3_11_3::detail::serializer<$>::dump(nlohmann::js... (40 times, avg 44 ms)
  1749 ms: std::_Rb_tree<$>::_M_get_insert_unique_pos(std::__cxx11::basic_strin... (278 times, avg 6 ms)
  1724 ms: std::__cxx11::basic_string<$>::_M_mutate(unsigned long, unsigned lon... (248 times, avg 6 ms)
  1697 ms: boost::io::basic_altstringbuf<$>::overflow(int) (194 times, avg 8 ms)
  1684 ms: nix::fetchers::GitInputScheme::fetch(nix::ref<$>, nix::fetchers::Inp... (1 times, avg 1684 ms)
  1680 ms: std::_Rb_tree_node<$>* std::_Rb_tree<$>::_M_copy<$>(std::_Rb_tree_no... (303 times, avg 5 ms)
  1589 ms: bool nlohmann::json_abi_v3_11_3::detail::parser<$>::sax_parse_intern... (18 times, avg 88 ms)
  1483 ms: non-virtual thunk to boost::wrapexcept<$>::~wrapexcept() (181 times, avg 8 ms)
  1447 ms: nlohmann::json_abi_v3_11_3::detail::serializer<$>::dump_escaped(std:... (40 times, avg 36 ms)
  1441 ms: std::__shared_ptr<$>::~__shared_ptr() (496 times, avg 2 ms)
  1420 ms: boost::stacktrace::basic_stacktrace<$>::init(unsigned long, unsigned... (137 times, avg 10 ms)
  1396 ms: boost::basic_format<$>::~basic_format() (194 times, avg 7 ms)
  1290 ms: std::__cxx11::basic_string<$>::_M_replace_cold(char*, unsigned long,... (231 times, avg 5 ms)
  1258 ms: std::vector<$>::~vector() (354 times, avg 3 ms)
  1222 ms: std::__cxx11::basic_string<$>::_M_replace(unsigned long, unsigned lo... (231 times, avg 5 ms)
  1194 ms: std::_Rb_tree<$>::_M_get_insert_hint_unique_pos(std::_Rb_tree_const_... (49 times, avg 24 ms)
  1186 ms: bool tao::pegtl::internal::sor<$>::match<$>(std::integer_sequence<$>... (1 times, avg 1186 ms)
  1149 ms: std::__detail::_Executor<$>::_M_dfs(std::__detail::_Executor<$>::_Ma... (70 times, avg 16 ms)
  1123 ms: toml::detail::sequence<$>::invoke(toml::detail::location&) (69 times, avg 16 ms)
  1110 ms: nlohmann::json_abi_v3_11_3::basic_json<$>::json_value::destroy(nlohm... (55 times, avg 20 ms)
  1079 ms: std::_Function_handler<$>::_M_manager(std::_Any_data&, std::_Any_dat... (541 times, avg 1 ms)
  1033 ms: nlohmann::json_abi_v3_11_3::detail::lexer<$>::scan_number() (20 times, avg 51 ms)

Change-Id: I10af282bcd4fc39c2d3caae3453e599e4639c70b
 2024-08-28 09:55:05 -07:00
eldritch horrors c55dcc6c13 filetransfer: return a Source from download() 
without this we will not be able to get rid of makeDecompressionSink,
which in turn will be necessary to get rid of sourceToSink (since the
libarchive archive wrapper *must* be a Source due to api limitations)

Change-Id: Iccd3d333ba2cbcab49cb5a1d3125624de16bce27
 2024-06-19 10:50:12 +00:00
eldritch horrors 0b9a72524a filetransfer: {up,down}load -> transfer 
even the transfer function is not all that necessary since there aren't
that many users, but we'll keep it for now. we could've kept both names
but we also kind of want to use `download` for something else very soon

Change-Id: I005e403ee59de433e139e37aa2045c26a523ccbf
 2024-06-18 23:58:25 +00:00
Raito Bezarius b8cb7abcf0 chore: rebrand Nix to Lix when it makes sense 
Here's my guide so far:

$ rg '((?!(recursive).*) Nix
(?!(daemon|store|expression|Rocks!|Packages|language|derivation|archive|account|user|sandbox|flake).*))'
-g '!doc/' --pcre2

All items from this query have been tackled. For the documentation side:
that's for https://git.lix.systems/lix-project/lix/issues/162.

Additionally, all remaining references to github.com/NixOS/nix which
were not relevant were also replaced.

Fixes: https://git.lix.systems/lix-project/lix/issues/148.
Fixes: https://git.lix.systems/lix-project/lix/issues/162.
Change-Id: Ib3451fae5cb8ab8cd9ac9e4e4551284ee6794545
Signed-off-by: Raito Bezarius <raito@lix.systems>
 2024-06-01 20:31:24 +02:00
Tom Hubrecht a39ba22ff7 util.hh: Delete remaining file and clean up headers 
Change-Id: Ic1f68e6af658e94ef7922841dd3ad4c69551ef56
 2024-05-29 12:38:51 +02:00
eldritch horrors b66451ae7f libstore: de-callback-ify FileTransfer 
also add a few more tests for exception propagation behavior. using
packaged_tasks and futures (which only allow a single call to a few
of their methods) introduces error paths that weren't there before.

Change-Id: I42ca5236f156fefec17df972f6e9be45989cf805
 2024-05-10 02:21:11 +02:00
eldritch horrors fb0996aaa8 filetransfer: remove dataCallback from interface 
this is highly questionable. single-arg download calls will misbehave
with it set, and two-arg download calls will just overwrite it. being
an implementation detail this should not have been in the API at all.

Change-Id: I613772951ee03d8302366085f06a53601d13f132
 2024-04-26 15:26:37 +00:00
eldritch horrors dfe3baea12 filetransfer: make two-arg download abstract 
this lets each implementation of FileTransfer (of which currently only
the one exists at all) implement appropriate handling for its internal
behaviours that are not otherwise exposed. in curl this lets us switch
the buffer-full handling method from "block the entire curl thread" to
"pause just the one transfer", move the non-libcurl body decompression
out of the actual curl wrapper (which will let us eventually morph the
curl wrapper intto an actual source of Sources), and some other things

Change-Id: Id6d3593cde6b4915aab3e90a43b175c103cc3f18
 2024-04-26 15:26:37 +00:00
eldritch horrors 38442e3123 filetransfer: remove decompress request parameter 
this is never read.

Change-Id: I4c46f140519843a21e452958900e81edd2f78be2
 2024-04-25 01:33:08 +02:00
Eelco Dolstra 1ad3328c5e Allow tarball URLs to redirect to a lockable immutable URL 
Previously, for tarball flakes, we recorded the original URL of the
tarball flake, rather than the URL to which it ultimately
redirects. Thus, a flake URL like
http://example.org/patchelf-latest.tar that redirects to
http://example.org/patchelf-<revision>.tar was not really usable. We
couldn't record the redirected URL, because sites like GitHub redirect
to CDN URLs that we can't rely on to be stable.

So now we use the redirected URL only if the server returns the
`x-nix-is-immutable` or `x-amz-meta-nix-is-immutable` headers in its
response.
 2023-06-13 14:17:45 +02:00
John Ericson f4ab297b31 Ensure all headers have #pragma once and are in API docs 
`///@file` makes them show up in the internal API dos. A tiny few were
missing `#pragma once`.
 2023-03-31 23:19:44 -04:00
John EricsonandEelco Dolstra abd5e7dec0 Extend internal API docs, part 2 
Picking up from #8111.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
 2023-03-31 23:01:40 -04:00
Eelco Dolstra ae5f62a894 Move isUri() and resolveUri() out of filetransfer.cc 
These are purely related to NIX_PATH / -I command line parsing, so put
them in libexpr.
 2022-12-12 14:05:35 +01:00
Alex AmeenGitHubCole Helbling
 		e5c934cd48 doc: rephrase connect-timeout help message 
Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
 2022-04-17 18:17:37 -05:00
Alex Ameen 25c85f5a0e doc: document nix.conf connect-timeout default 2022-04-17 17:14:38 -05:00
Eelco Dolstra 8bd9ebf52c Error: Remove unused sname() method 2022-04-08 11:31:51 +02:00
Eelco Dolstra df552ff53e Remove std::string alias (for real this time) 
Also use std::string_view in a few more places.
 2022-02-25 16:13:02 +01:00
Eelco Dolstra d62a9390fc Get rid of std::shared_ptr<std::string> and ref<std::string> 
These were needed back in the pre-C++11 era because we didn't have
move semantics. But now we do.
 2022-01-18 11:12:30 +01:00
Eelco Dolstra 0813350349 Add 'nix store prefetch-{file,tarball}' 
These replace nix-prefetch-url and nix-prefetch-url --unpack,
respectively.
 2021-01-07 21:51:46 +01:00
Eelco Dolstra faa31f4084 Sink: Use std::string_view 2020-12-02 14:17:27 +01:00
Eelco Dolstra 5999978a05 Make Headers an optional argument 2020-09-29 13:05:19 +02:00
Greg Hale a303c0b6dc Fetch commits from github/gitlab using Auth header 
`nix flake info` calls the github 'commits' API, which requires
authorization when the repository is private. Currently this request
fails with a 404.

This commit adds an authorization header when calling the 'commits' API.
It also changes the way that the 'tarball' API authenticates, moving the
user's token from a query parameter into the Authorization header.

The query parameter method is recently deprecated and will be disallowed
in November 2020. Using them today triggers a warning email.
 2020-09-16 13:46:48 -04:00
Eelco Dolstra c8fa39324a Generate the nix.conf docs from the source code 
This means we don't have two (divergent) sets of option descriptions
anymore.
 2020-08-19 18:28:04 +02:00
John Ericson 74b219ef6e Adjust FileTransferError message to use opt response 2020-06-18 14:48:45 +00:00
Carlo NuceraandJohn Ericson 004570a377 Add HTTP responses to FileTransferErrors 2020-06-17 22:34:31 +00:00
Ben Burdette b93c1bf3d6 fixes to merged code 2020-05-11 15:52:15 -06:00
Ben Burdette 59b1f5c701 Merge branch 'master' into errors-phase-2 2020-05-11 14:35:30 -06:00
Nikola Knezevic f5095594e7 datatransfer.{cc,hh} -> filetransfer.{cc,hh} 2020-04-08 22:26:57 +02:00