the rpc transition will require sync and async objects to share a single
io buffer (since defining serializers on async is an immense pain in the
tail, slow, and ultimately not necessary). a generic buffer class allows
us to reuse existing serializers more readily (reuse them at all, even).
Change-Id: I5ebba8449f26f2bb76016818928183c7e0123be0
remote store async io will need to set O_NONBLOCK on the connection fds,
and right now the number of fds can vary between connection types: local
connections have one one fd for the sink/source pair since they use unix
sockets, but ssh connections have two because ssh uses pipes. this makes
it rather hard to manage flags correctly, and even harder to wait for io
readiness on both directions using kj. using sockets for ssh fixes this.
Change-Id: I0f563ece7627cd3fbd0f5ce21c25140469729e5a
this could've just ignored exceptions thrown by the remote. in the
current implementation there's no way such an exception could have
propagated to the client though, so there's no change in behavior.
Change-Id: Ide03bda1cb0ad7fb5f27b4ee5d16efd6c2b635ba
mostly to make moving this to async writes easier. this won't have a
performance impact because it's only a single packet, that's written
to a BufferedSink, but the connection sink only gets a single write.
Change-Id: I9a5f1afe7d3e25f5f4502ef9520ff2f2529431ba
the test is for the map that usually wraps it though because it's the
bit we're interested in replacing, and it has custom serializer code.
Change-Id: If77a236dfca738b646ed2b7a5c65515dad6b7295
the old protocols are largely untested, mostly unused, and have design
problems that make the RPC transition a lot harder, if not impossible.
in theory we could ship a transparent protocol-converting proxy that'd
isolate the daemon itself from old protocol versions, but that's a lot
of code to maintain for presumably little gain or even no gain at all.
Change-Id: I4c3f3bb34d39044f6aeb07c10caaf13b8340a220
also remove all the documentation referencing it, or rewrite the docs
to make sense in the non-floating-content-addressed world we live in.
Change-Id: I724e67839f44cc9f1cfc7d6f1c05252b62752b42
we don't need to worry about leaving around old ca data in the database:
this was always a possiblity when enabling ca derivations, and disabling
them again some time later. behavior is unchanged, but we lose dead code
Change-Id: I8c10ff7fdcee08c3badf23d64403f5ee6452e41e
we no longer need placeholders to represent all derivation output paths
as string context, and thus will not need experimental features either.
Change-Id: I9e86ce86810e976cf8397b2c2f473af11390874c
neither are actually partial now, and the the non-Static variant has a
non-Partial wrapper which merely returns the Partial result unchanged.
Change-Id: I5fa86682883c2305cc12c711ccff58537b7a278d
derivation outpaths are now statically known at all times. the one snag
here is that the wires encode even statically known paths as optionals,
forcing us to check for this any time we receive an output map. remotes
answering with nullopt paths for derivations we still support now would
be a protocol error on its own though, so we do not diagnose it deeply.
Change-Id: Ib7080b2a0c45c3506233e87c8ef6842576f61050
we don't need to touch the schema of the cache here. keeping the table
around doesn't hurt (and avoids cppnix breakage) thanks to foreign key
constraints and the ca bits of the schema being independent enough for
us to just ignore them (and not having to do any maintenance on them).
Change-Id: Ib5d8eb1cd838826d88eb65bbf8f245703a2482da
only a daemon wire operation and the perl bindings could initiate these
queries at this point. the daemon ops can throw an error instead (as if
the daemon were older) and realistically should never be queries if the
client hasn't evaluated a ca derivation on a given store, and perl code
is best off dying early. nothing known except hydra uses these bdingins
anyway, and we control our hydra so we don't need backward compat code.
Change-Id: Ia7df27aba59a4a4a692ae014f407415f3bea63f2
it's only used by the RegisterDrvOutput daemon wire operation now, and
that one we can safely stub out to throw an error when called instead.
Change-Id: If29716976392c9c7a2a05b151dfe80b2c8d9c07d
this removes the ca-derivations system feature and, perhaps most
importantly, realisation closure copy support. the latter is not
needed any more and its existence blocks some more code removal.
Change-Id: I2931b03637e25d35252ae6bd5f34f0c0168d80e9
we can't create these any more except by reading an old json-formatted
derivation that used them. since we cannot do anything with a deferred
derivation even when read we will remove json support for them as well
Change-Id: I4f9ea0b7c6469f57977784037f7710f939e40a2c
now that we have no deferred hashes (since floating ca derivations were
the only way to create them) we can safely remove this enumeration too.
Change-Id: Ic72ed90500fcee7aa5b3b5a302477fa515acf1be
only FODs can be content-addressed now, and those are always fixed.
FODs are also never sandboxed, so we do not need that field either.
Change-Id: I1be62b3ec85e08ec003cc8769723328d19777728
this mostly takes the form of removes feature checks and the associated
"ca derivations enabled" branches, but for the realisation info command
turns into a stub. we keep it around for compatibility, but from now on
it will always throws "ca derivations not implemented" errors when run.
Change-Id: I0abea5f76262013415330adcca2b498c6dca555b
this goal is only involved for output paths that aren't known at initial
build time, which in turn can only happen if they are ca paths. since we
can no longer create ca derivations during eval *or* read them from disk
we can now assume that we will never run this goal. there are still some
vestiges like output known-ness we can't remove yet, so those must stay.
Change-Id: I989e5ad4600c628bcbe8e17e1b082ce8d73a3bd9
we remove not only support for *building* a ca derivation, but also
support for *resolving* ca derivations as part of a build. we never
have to resolve derivations from here on, so this code is now dead.
Change-Id: I0346442d5fa00eb927177545ae61315f588477cc
we no longer have any experimental features depending on ca derivations,
so we can start removing them. since ca derivations are very invasive we
will need a while to remove all of the explicitly experimental code, and
even then we will not have removed *all* code related to ca derivations.
especially in the derivation goals there is a lot of code that is not as
easy to disentangle from experimental features as some would have hoped.
Change-Id: Ia456aadc6164613ded343f571318494d9310a549
only dynamic derivations could produce a non-opaque drvPath. since
dynamic derivations are no longer supported we can have drvPath be
opaque at all times, simplifying downstream code significantly and
making quite a few methods unnecessary. discardOutputPath was only
called on drvPath members anyway and thus reduces to a copy, other
operations at the very least are no longer recursive. some vestige
of dynamic derivations remains in DerivedPathMap though (for now).
Change-Id: Ifb4ad53a3c67800be5a62540068c8279d4ae0046
string context doesn't need any tests because it's never persisted or
shown to the user. getting rid of recursive string context means that
the context string parsers can be a lot simpler from here on forward.
Change-Id: I58443679ad76c0f28ea5f4eb8bfb3874f270e764
as with impure derivations it is still possible to garbage-collect
existing xp-dyn-drv derivations. we once again don't introduce any
new kinds of errors, we only change the dynamic type of exceptions
from MissingExperimentalFeature to UnimplementedError (although we
do throw FormatError when reading xp-dyn-drv derivations now, that
seems to make a little more sense than "feature not implemented").
Change-Id: Ic26e5b6c9c9e2533093e27f6cf901dc9db57c83e
only dynamic derivation produce text-hashed derivation outputs. toFile
produces text-hashed store paths, so we cannot remove text hashing now
without breaking stores, but we can disallow it in derivation outputs.
Change-Id: I95ff9882a59153a7d5fd509f5c9fd85925f30d02
with impure derivations gone we move on to dynamic derivations. this too
is not done in a single commit because dynamic derivations are invasive,
modifying semantics of all references to derivation output paths and all
derivation dependency calculations. removing dynamic derivations cleanly
is made significantly harder by the multiple did-you-mean-sum types, aka
"wrappers for std::variant", holding all derivation outpath information.
Change-Id: Ice7a7700c7b54c6a6061d4beb322b4175923d27a
writing them is technically still supported because what makes a
derivation impure is entirely specified by magically named data,
but without derivationStrict being able to pass these through to
libstore there is no way (besides reading existing files) to get
any new impure derivations into an existing store. it will still
be possible to garbage-collect existing impure derivations since
the gc process does not need to read them as derivations, and we
are not introducing any new kinds of unsupported-feature errors.
Change-Id: I648f53129ce67ee2b48d0591219759812dd557da
we don't remove the entire feature in one go to make review easier.
impure derivations are rather unintrusive on their own, at least if
we compare them to dynamic or ca derivations in general, so we will
be done with this soon. as it stands impure derivations cannot work
without ca derivations, and those we *really* want to leave behind.
Change-Id: I4f01d8d758b2c85dcd6c3078304b5ee1b52f65b0
ca derivations are what we're really after, but dynamic derivations
must also go because they depend on ca derivations. we can't easily
implement dynamic derivations any other way, so we remove them too.
impure derivations build on the content-addressed infrastructure in
ways we cannot easily detangle, so they too must go for time being.
see #815
Change-Id: If61371736dfd89cc71a1b2ae5a005757c3cb9484
it's an eval-time only setting, the daemon doesn't use it anywhere. this
is a hack, but until we have a much better settings system we are stuck.
fixes#680
Change-Id: I532088b0279f13da0a0a65c2bd2e5f9d1dfb39da
this can be extremely distracting when running in a repl that has access
to all of nixpkgs from its static env. not only do we print all bindings
in the stack trace, we also print them *again* every time we enter *any*
but the outermost frames. duplicating this environment information isn't
all that useful and mostly just flushes useful information out of sight.
fixes#816
Change-Id: I027e6a565e82663275e21333bb1d00d498cd4127
this way we don't have to even check whether we need a debug frame when
the debugger isn't enabled. not doing this gives us an eval performance
improvement of roughly 7% on nixos system eval and 2% for `nix search`.
Change-Id: I1cdad3de61f865ea54d6e09d63a281688e828768
we'll want to wrap some exprs for debug purposes, and dynamic casts
cannot look through such wrappers. dedicated casting functions can.
Change-Id: I1fba0ec52d281a1b8de85a62e4948bfae536bcfc
add a position parameter to the autocaller instead, and pass it much
more accurate position information where we have it easily available
Change-Id: If2f1c3006ca3f2b413492842905d079a8b752542
errors for individual attrs should report the position of the attribute,
not the position of the set they were taken from (which we'll assume was
the intent here). this is already the case for most attributes, only the
the extra-special __structuredAttrs and __ignoreNulls weren't annotated.
this includes not calling atPos(v) on errors builders since the position
of v is always invalid. error messages are unchanged by this part of the
patch (and the caller adds a trace pointing to the derivation name too).
Change-Id: Ia3540f9ce1f2505275e49dd4e4336b2a59d5b336
replInitFilesFunction is always valid, but pointing to it is misleading
because the source is inaccessible. every replInit is better identified
by its path if it isn't a lambda; determinePos will return noPos anyway
Change-Id: I0b6fcd2f95f121a802c87cb1085d3a0b1c585aeb
it's not useful. nix-instantiate will always point to the wrong location
because "at the toplevel" is not a position we can point to. all nesting
conversions will point to strange places: the *original* definition of a
lambda value (or partial applications thereof), or nowhere at all (since
only lambda values can have their positions determined in the formatter)
Change-Id: I56bd9e1cf3cae11b4198f148a141ca758e4fe8ac
this is much more useful than a frame pointing to the set but claiming
it as a function being called. if the function is actually at fault we
will now point to its attribute, although the position may be slightly
wrong if __toString was defined from set updates or builtin functions.
Change-Id: Ib3eb237a276d94426d9c6fc0e26eea72382d34a2
by the time enumerateOutputs is called we've already established that
vFlake will evaluate to an attrset (as a locked flake it can't be any
other kind of value). the outputs attribute is likewise already a set
because the nix code that builds it says so. adding positions here is
merely an unnecessary debugging aid for developers working on flakes.
Change-Id: I2a10ddf1400f7650c6e303433f9fee8813f0188d
only attrsets, lambdas, and app nodes can have positions determined, and
none of these three types are incomplete. calling determinePos before we
force a value is thus only a convoluted copy operation for its argument.
Change-Id: I7e647aacca1fc8250f7bb11bb0071a4f7186047f
otherwise lix may crash when e.g. nix search receives invalid regex.
we now also give better error messages for regex errors during eval.
fixes#803
Change-Id: Icc7c578ff488ba520efac5d898572ccf4486e9a8
we no longer explicitly configure these ssh options, so we now
have *even less* reason to stick to old and awful terminology.
Change-Id: I8f38f5486e6918f69c61ddc2d6882b6bb50554c6
our implementation is mostly unused, completely untested, and simply
breaks when users configure connection sharing independently. we can
safely delete this "feature" and inherit user configuration instead.
also have the remote build test check that connection sharing works.
fixes#304, fixes#644
Change-Id: Iea44cc0f8e51a1d231ad186a95c7e310bbfeb303
This reverts commit bba678e5c5.
Reason for revert: didn't fix the bug and created new ones (fj#794)
Change-Id: I0450205d3041b6c876737151a4051081c1366f1d
448c7d50e1 was a bit over-eager and didn't
make the `ref(enable_shared_from_this &)` constructor explicit. this has
confused hydra maintainers, and is just generally bad practice since any
reference is allowed to implicitly convert, even if those references are
not associated with an active control block. we can't avoid this problem
entirely, but we can make what happens more explicit by ... well, making
the involved constructor explicit. enable_shared_from_this is statically
unsafe in principle and we really have to get rid of this nonsense soon.
Change-Id: I8b48ef4353e6301b61af3569083f42bc7379b0a4
this gives us a semblance of a stack trace if async tasks fail, like:
Async task trace (probably incomplete):
#0: virtual kj::Promise<Result<Goal::WorkResult>> nix::LocalDerivationGoal::tryLocalBuild() (lix/libstore/build/local-derivation-goal.cc:257:33)
building of '//0ppb5aj4p3vp0icnkq4yibajw1czvdja-bash52-009.drv^out' from .drv file
#1: kj::Promise<Result<Goal::WorkResult>> nix::DerivationGoal::outputsSubstitutionTried() (lix/libstore/build/derivation-goal.cc:381:47)
building of '//0ppb5aj4p3vp0icnkq4yibajw1czvdja-bash52-009.drv^out' from .drv file
#2: kj::Promise<Result<Goal::WorkResult>> nix::DerivationGoal::loadDerivation() (lix/libstore/build/derivation-goal.cc:213:41)
building of '//0ppb5aj4p3vp0icnkq4yibajw1czvdja-bash52-009.drv^out' from .drv file
#3: kj::Promise<Result<Goal::WorkResult>> nix::DerivationGoal::gaveUpOnSubstitution() (lix/libstore/build/derivation-goal.cc:453:62)
building of '//f8rkb1vam9y9vi9scsm1v213b01rm72h-bash-5.2p37.drv^out' from .drv file
#4: kj::Promise<Result<Goal::WorkResult>> nix::DerivationGoal::outputsSubstitutionTried() (lix/libstore/build/derivation-goal.cc:381:47)
building of '//f8rkb1vam9y9vi9scsm1v213b01rm72h-bash-5.2p37.drv^out' from .drv file
#5: kj::Promise<Result<Goal::WorkResult>> nix::DerivationGoal::loadDerivation() (lix/libstore/build/derivation-goal.cc:213:41)
building of '//f8rkb1vam9y9vi9scsm1v213b01rm72h-bash-5.2p37.drv^out' from .drv file
#6: kj::Promise<Result<Goal::WorkResult>> nix::DerivationGoal::gaveUpOnSubstitution() (lix/libstore/build/derivation-goal.cc:453:62)
building of '//qa9jvc4j6fphzwr63k5c829b2hr04q6z-foo.drv^*' from .drv file
#7: kj::Promise<Result<Goal::WorkResult>> nix::DerivationGoal::outputsSubstitutionTried() (lix/libstore/build/derivation-goal.cc:381:47)
building of '//qa9jvc4j6fphzwr63k5c829b2hr04q6z-foo.drv^*' from .drv file
#8: kj::Promise<Result<Goal::WorkResult>> nix::DerivationGoal::loadDerivation() (lix/libstore/build/derivation-goal.cc:213:41)
building of '//qa9jvc4j6fphzwr63k5c829b2hr04q6z-foo.drv^*' from .drv file
#9: virtual kj::Promise<Result<void>> nix::Store::buildPaths(const std::vector<DerivedPath> &, BuildMode, std::shared_ptr<Store>) (lix/libstore/build/entry-points.cc:20:11)
it's not much, but it's a lot better than nothing. sadly we're forced
to disable another clang-tidy warning now, but since that one's about
assert side effects being compiled out in release mode and we've long
since decided to just not do that we are not affected by this at all.
also see #724 for a related but different approach to origin tracing.
Change-Id: I8161f82ca39d0b271316ea4df80513acc1b06a03
we generally do not want to catch or throw these. catching them to print
and discard is fine, tests are largely exempt, and cases in which we can
be certain where the exception came from are also fine to *catch*. we'll
try to never *throw* (or rethrow) these if possible though because doing
so will make it impossible to construct async traces for the exceptions.
Change-Id: I3b71c32ecd16afc2246c946472f5629a1fa31f2c
this wraps a non-lix exception so we'll be able to add async traces. to
not interfere with existing infrastructure we also add a BaseException,
deriving from BaseError would have foreign exceptions match on the very
broad catches we have in libmain even though we want these logged only.
Change-Id: I5545788c299e9bbdb4d730458914bfd816870794
we'll also assert instead of throwing an exception because that just
seems more useful. this should never *ever* happen, and it is on the
callers of such conversions to ensure that the conversion *is* safe.
Change-Id: Ib0696af4f037046f2d45bf5b1b255393ea9b5f05
don't throw if a downcast fails, have that particular casting method
return an optional instead and check the result at the call site. we
also rename dynamic_pointer_cast to try_cast_shared for consistency.
Change-Id: I65ee0fb498b1b2b8b418935fce4358a654f984e5
we don't need to unsafely cast a shared_from_this pointer to create a
ref (the pointer must be non-null to not invoke nasal demons anyway).
Change-Id: I133fe5f07b2cff8ec2c925f0528f4c5288827261
don't store them as shared_ptrs and cast them back to ref when needed.
we have sufficiently few instances of this pattern that we can spare a
few bytes to make the code less implicitly unsafe (and a bit clearer).
Change-Id: Ied573ab2e116ae228730390ff4c6b0469c59ff6a
eldritch horrors