Commit Graph
281 Commits
Author SHA1 Message Date
Raito Bezarius f8ccd9d572 libutil/async-io: augment read type safety w.r.t to EOFs
Usually, EOFs are represented by returning 0 in the `read` APIs, at
least, this is what read(2) dictate.

As clever creature, we may sum zeroes sometimes (advanced form:
`buf->added(got)`) and forego handling the EOF condition.

To avoid the bug that lurked in remote-store.cc and caused busy looping
if the remote end disconnects suddenly, we return
`Result<Option<size_t>>` forcing the caller to perform a specific
processing for the EOF situation.

The conversion did not raise any other offending code path.

Change-Id: I185fdcb77aa82d87ab0802d66ac37c1363657a73
Signed-off-by: Raito Bezarius <raito@lix.systems>
2025-07-23 10:35:37 +00:00
Alois Wohlschlager 2090853b80 falsify the glibc store paths
Commits 205c59367c and
325e7e1824 introduced real glibc store paths from
current nixpkgs unstable into the source. Since nixpkgs `fetchFromGitea` (and
similar fixed-output derivations) depends on a C library, on x86_64-linux they
will fail with the forbidden reference error:

error: the fixed-output derivation '/nix/store/wnmnj3jzc82y89sfmyicr04kilg8zs2k-source.drv' must not reference store paths but 1 such references were found:
         /nix/store/q4wq65gl3r8fy746v9bbwgx4gzn0r2kl-glibc-2.40-66

Falsify the store path to prevent this failure.

Change-Id: I949033567bcad070f9a0a19cefdb33a79222e421
2025-07-13 10:56:47 +02:00
Philipp OtterbeinandRaito Bezarius fc3261c340 libexpr/primops: fix overflows and silent type casts
Avoids incorrect behavior with large integers in `elemAt`, `substring`,
`genList`, etc, which results into crashing the Lix interpreter.

At the same time, unit tests were added for these edge cases with 2^32
as an argument of these primops.

Port of https://github.com/NixOS/nix/pull/13309.
Prior art in https://github.com/NixOS/nix/pull/7222 (forgotten by the original project…).

Change-Id: I1c43ed64f26bcb60e51869e11a74e5de2b7db53a
Co-authored-by: Raito Bezarius <raito@lix.systems>
Signed-off-by: Raito Bezarius <raito@lix.systems>
2025-07-07 23:32:18 +00:00
Maximilian Bosch da0df360e1 libstore: print dependency in tree boldly
That way it's easier to spot whether a node is the "final" node in the
graph which is especially helpful for larger graphs.

Change-Id: I460a699f07f5455917792599f4247ebf8f430d93
2025-07-07 11:44:17 +00:00
Maximilian Bosch 205c59367c libstore: add genGraphString from why-depends
This will be useful for other things as well such as the
disallowedRequisites error in the builder code. Additionally, print the
dependencyPath in the tree bold to spot where a change terminates.

Also implemented some unit-tests for this code.

Change-Id: I8460f3f6c5095d5bfbe390f223bc0252800dca5e
2025-07-07 11:44:17 +00:00
Raito Bezarius c7976e63a3 libutil: guess or invent a path from file descriptors
This is useful for certain error recovery paths (no pun intended) that
does not thread through the original path name.

Change-Id: I2d800740cb4f9912e64c923120d3f977c58ccb7e
Signed-off-by: Raito Bezarius <raito@lix.systems>
2025-06-23 16:57:23 +02:00
Maximilian Bosch 242a228124 libutil: close file handle in async NAR parser
This bit us while upgrading Hydra[1]: when all the data was read into
the hashing sink while receinving NAR contents, the hash was never
created which lead to a test failing because file size was correct, but
the hash was std::nullopt.

[1] https://git.lix.systems/lix-project/hydra/src/commit/7a0dae579b53b4b96a829263b160c6dc9f42ce75/src/hydra-queue-runner/nar-extractor.cc#L70-L73

Change-Id: Ie71b5f1f17c926a2ab95fb2aabf23c7a575ff70b
2025-06-21 13:52:43 +02:00
eldritch horrors 1729c8ca3e libstore: asyncify curl return streams
making stores and their users fully async requires all data streams to
be async. the most notable data streams in common usage are curl first
and remote stores second. curl is much more contained today and easier
to asyncify (with the preparatory work we've done in the past commits)

Change-Id: I2d6ff4687ee2b47e4efaa6714827b7283bed941d
2025-06-15 13:36:31 +00:00
eldritch horrors de89c7f7c8 libstore: asyncify curl interface
Change-Id: I3fc93016b8ac5e59d9062d4f4aead19ae051a680
2025-06-15 13:36:31 +00:00
eldritch horrors d824753377 libutil: add async decompression support
it's a real mess, but it's also the best we can reasonably do.

Change-Id: I3b84840cede0363396bdf290d6e6b0e03ace513c
2025-06-15 15:35:51 +02:00
eldritch horrors 87fbc15938 libutil: make the pool element factory a promise
processStderr of RemoteStore wants to be a promise and it must be used
from connection setup, so the pool factory callback must be a promise.

Change-Id: I9ac742b6048ae6dba0bfa5dcb58971386229690b
2025-06-11 22:28:44 +02:00
eldritch horrors 7d681a5049 libutil: add io buffer abstraction
the rpc transition will require sync and async objects to share a single
io buffer (since defining serializers on async is an immense pain in the
tail, slow, and ultimately not necessary). a generic buffer class allows
us to reuse existing serializers more readily (reuse them at all, even).

Change-Id: I5ebba8449f26f2bb76016818928183c7e0123be0
2025-06-11 18:11:57 +00:00
eldritch horrors 4ebf79bc19 libstore: associate wire connection states with stores
why pass the stores as a distinct argument every time?

Change-Id: If529a49541483e8a3d33eb2b3532d66b3bb9738d
2025-06-06 18:09:46 +02:00
eldritch horrors dd31a23c31 libstore: add worker serializer for SubstitutablePathInfo
the test is for the map that usually wraps it though because it's the
bit we're interested in replacing, and it has custom serializer code.

Change-Id: If77a236dfca738b646ed2b7a5c65515dad6b7295
2025-06-06 18:09:46 +02:00
eldritch horrors fca0a30470 libstore: remove pre-2.18 protocols
the old protocols are largely untested, mostly unused, and have design
problems that make the RPC transition a lot harder, if not impossible.
in theory we could ship a transparent protocol-converting proxy that'd
isolate the daemon itself from old protocol versions, but that's a lot
of code to maintain for presumably little gain or even no gain at all.

Change-Id: I4c3f3bb34d39044f6aeb07c10caaf13b8340a220
2025-06-02 22:43:24 +00:00
eldritch horrors 5097c5db63 libstore: remove unused DownstreamPlaceholder
Change-Id: I3b71778aa9e92bf8ac0edef0ca929e92010f5c6a
2025-05-20 17:43:46 +00:00
eldritch horrors 34317c0081 libexpr: simplify EvalState::mkOutputString{,Raw}
we no longer need placeholders to represent all derivation output paths
as string context, and thus will not need experimental features either.

Change-Id: I9e86ce86810e976cf8397b2c2f473af11390874c
2025-05-20 17:43:46 +00:00
eldritch horrors e543ac686f libstore: remove DerivationOutput::Deferred
we can't create these any more except by reading an old json-formatted
derivation that used them. since we cannot do anything with a deferred
derivation even when read we will remove json support for them as well

Change-Id: I4f9ea0b7c6469f57977784037f7710f939e40a2c
2025-05-20 17:43:46 +00:00
eldritch horrors 2cc420c1e5 libstore: remove ca derivation read support
we can now no longer read ca store derivations from disk.

Change-Id: I233edea597b550dd7e8c78a555b7f12760a993dd
2025-05-20 13:27:04 +02:00
eldritch horrors dad28eca75 cli: disallow ca derivations
we no longer have any experimental features depending on ca derivations,
so we can start removing them. since ca derivations are very invasive we
will need a while to remove all of the explicitly experimental code, and
even then we will not have removed *all* code related to ca derivations.
especially in the derivation goals there is a lot of code that is not as
easy to disentangle from experimental features as some would have hoped.

Change-Id: Ia456aadc6164613ded343f571318494d9310a549
2025-05-20 13:27:04 +02:00
piegamesandLix Systems Gerrit eb18a90afb Merge "libutil: Introduce LinearMap" into main 2025-05-19 14:20:34 +00:00
piegames bd8ec106fa libutil: Introduce LinearMap
Change-Id: I68ce4c1dc17b0742690e49f62206c65f5a1a4a30
2025-05-19 15:35:52 +02:00
Lily BallardandLix Systems Gerrit b7ce00fc55 Merge "libutil: move filterANSIEscapes tests" into main 2025-05-15 01:42:34 +00:00
Lily Ballard bc5d7ad458 libutil: move filterANSIEscapes tests
This moves the original test suite for `filterANSIEscapes` into the same
file as the newer tests. There is some overlap between the old and new
tests but that doesn't hurt anything so I kept them as-is.

Change-Id: Id00000009919024a5f206ec9a7bc0022541ff612
2025-05-13 19:04:52 -07:00
Lily BallardandLix Systems Gerrit 6237c50161 Merge "libutil: handle OSC escapes in filterANSIEscapes()" into main 2025-05-14 02:02:08 +00:00
Lily Ballard 207b5d81bf libutil: handle OSC escapes in filterANSIEscapes()
This teaches `filterANSIEscapes()` how to find the end of an OSC
sequence. It also keeps OSC 8 (hyperlinks) when not instructed to filter
out all escapes, just as it keeps colors.

This also relaxes the parsing of CSI escapes to find the end of the
sequence for invalid sequences, and handles better escapes that don't
start CSI or OSC.

This fixes the repl output for `:doc builtins.fetchGit`.

Fixes: https://git.lix.systems/lix-project/lix/issues/160
Change-Id: Id0000000f2a6956c042c883a4545edf347fa1799
2025-05-13 18:59:08 -07:00
eldritch horrors ccdd916226 libstore: de-ref {Derived,Built}Path::Built drvPath
they're no longer recursive, so this is perfectly fine.

Change-Id: If565a557f2c2074e2a96a7f89c51ff1c51146b36
2025-05-12 13:37:54 +02:00
eldritch horrors 84c1df46ea libstore: remove DerivedPathMap
single-level maps suffice now that dynamic derivations are gone.

Change-Id: If29998b104b31255292ab0c789622d7d27040f69
2025-05-12 13:37:54 +02:00
eldritch horrors f5e2e78266 libexpr: remove more obsolete DerivedPath methods
split from the prior commit for easier review.

Change-Id: Iabf1bc759cb56a04211d4cb07769f53b0f302b39
2025-05-12 13:37:54 +02:00
eldritch horrors 5f723e96e6 libstore: flatten {,Single}{Built,Derived}Path
only dynamic derivations could produce a non-opaque drvPath. since
dynamic derivations are no longer supported we can have drvPath be
opaque at all times, simplifying downstream code significantly and
making quite a few methods unnecessary. discardOutputPath was only
called on drvPath members anyway and thus reduces to a copy, other
operations at the very least are no longer recursive. some vestige
of dynamic derivations remains in DerivedPathMap though (for now).

Change-Id: Ifb4ad53a3c67800be5a62540068c8279d4ae0046
2025-05-12 13:37:54 +02:00
eldritch horrors 8a539424c8 libexpr: drop support for dyn-drv string context
string context doesn't need any tests because it's never persisted or
shown to the user. getting rid of recursive string context means that
the context string parsers can be a lot simpler from here on forward.

Change-Id: I58443679ad76c0f28ea5f4eb8bfb3874f270e764
2025-05-12 13:37:54 +02:00
eldritch horrors 540071dd77 cli: disallow dynamic derivations
with impure derivations gone we move on to dynamic derivations. this too
is not done in a single commit because dynamic derivations are invasive,
modifying semantics of all references to derivation output paths and all
derivation dependency calculations. removing dynamic derivations cleanly
is made significantly harder by the multiple did-you-mean-sum types, aka
"wrappers for std::variant", holding all derivation outpath information.

Change-Id: Ice7a7700c7b54c6a6061d4beb322b4175923d27a
2025-05-12 13:37:54 +02:00
eldritch horrors 6599be1a9f cli: disallow impure derivations
we don't remove the entire feature in one go to make review easier.
impure derivations are rather unintrusive on their own, at least if
we compare them to dynamic or ca derivations in general, so we will
be done with this soon. as it stands impure derivations cannot work
without ca derivations, and those we *really* want to leave behind.

Change-Id: I4f01d8d758b2c85dcd6c3078304b5ee1b52f65b0
2025-05-11 17:26:55 +02:00
Jade Lovelace 69ba3c92db fix: Terminate daemons properly on Ctrl-C on macOS
This was an absolute nightmare to diagnose. It turns out there's a
kernel bug: poll with events = POLLHUP will receive an event for NOT
POLLHUP internally in the kernel, delete their event subscription, and
then not receive events for any HUP later. lol! lmao!!

We choose to use plain old EVFILT_READ because the watched fd can be
either a socket or a pipe and it's preferable to eat some spurious
wakeups than have separate paths for those. The alternative is using
EVFILT_SOCK, a private API that's existed for years and which netty
uses for its sockets, but that doesn't work on pipes.

Fixes: https://git.lix.systems/lix-project/lix/issues/729
Change-Id: If72b5d7a39f00320a9acccdbe81121cdb1a04c45
2025-05-01 12:22:27 -07:00
Raito BezariusandLix Systems Gerrit a133633ecc Merge "lixexpr: Declutter error traces" into main 2025-04-24 19:07:06 +00:00
2b22dae1ba lixexpr: Declutter error traces
Now that we can correctly point to all expressions, we can remove
redundant intermediate traces to reduce clutter.

Co-authored-by: eldritch horrors <pennae@lix.systems>
Co-authored-by: Raito Bezarius <raito@lix.systems>
Change-Id: I3e9d7c1c7a6599a8e68302448bbb961d051002b7
2025-04-24 20:44:57 +02:00
eldritch horrors feebecd60b treewide: wrap std::regex_error
otherwise lix may crash when e.g. nix search receives invalid regex.
we now also give better error messages for regex errors during eval.

fixes #803

Change-Id: Icc7c578ff488ba520efac5d898572ccf4486e9a8
2025-04-24 13:48:15 +00:00
gilice 341e6049a7 libutil: canonPath: error instead of panic on empty path
This could previously crash lix:

Before:
$ nix eval -E '{type="derivation"; drvPath="";}'
nix: lix/libutil/file-system.cc:45: Path nix::canonPath(PathView, bool): Assertion `path != ""' failed.
Aborted (core dumped)

After:
$ nix eval -E '{type="derivation"; drvPath="";}'
error:
       … while evaluating the drvPath of a derivation
         at «string»:1:21:
            1| {type="derivation"; drvPath="";}
             |                     ^

       error: path '' is not in the Nix store

Fixes #536

Change-Id: I406dc9e58047be8f263cf2e4bc3ed5da75a46602
2025-04-12 16:31:28 +02:00
jadeandLix Systems Gerrit 81d4769849 Merge "fix: make test/ headers compliant with styleguide" into main 2025-04-10 16:25:52 +00:00
FireFlyandLix Systems Gerrit 6b3162be9c Merge changes Idd20d965,I27e2c7d8 into main
* changes:
  tests: fix deprecated of googletest macro use
  tests: ignore deprecated uses in rapidcheck
2025-04-10 11:48:09 +00:00
Jade Lovelace c0de512e14 fix: make test/ headers compliant with styleguide
These were not included in that check. Beats me as to why.

Change-Id: Ib44c7f0b10672359026f95daeb088039cf6df4eb
2025-04-09 23:46:41 -07:00
jadeandLix Systems Gerrit 9c2dba4ee2 Merge "Revert "libstore: don't use curl decompression support"" into main 2025-04-09 16:01:37 +00:00
FireFly 478253b16a tests: fix deprecated of googletest macro use
Change-Id: Idd20d9659116c352f05876acd91f5c3d8325f097
2025-04-09 17:15:19 +02:00
FireFly a8c3fbf0cc tests: ignore deprecated uses in rapidcheck
There is a pull request [1] addressing these upstream--it doesn't appear
likely to be merged anytime soon though... this is a no-op til we enable
-Wdeprecated-declarations, but helps in the direction of #744

[1]: https://github.com/emil-e/rapidcheck/pull/325

Change-Id: I27e2c7d81df152de8674696f2a56d5f21c414ce3
2025-04-09 17:04:42 +02:00
eldritch horrorsandLix Systems Gerrit 350bf7e42d Revert "libstore: don't use curl decompression support"
This reverts commit bba678e5c5.

Reason for revert: didn't fix the bug and created new ones (fj#794)

Change-Id: I0450205d3041b6c876737151a4051081c1366f1d
2025-04-09 11:28:50 +00:00
eldritch horrors 8d8bb60796 treewide: lint against non-lix exceptions
we generally do not want to catch or throw these. catching them to print
and discard is fine, tests are largely exempt, and cases in which we can
be certain where the exception came from are also fine to *catch*. we'll
try to never *throw* (or rethrow) these if possible though because doing
so will make it impossible to construct async traces for the exceptions.

Change-Id: I3b71c32ecd16afc2246c946472f5629a1fa31f2c
2025-04-06 12:14:38 +02:00
eldritch horrors 63d550938b treewide: derive all lix exceptions from BaseError
even the non-errors. we should probably insert a BaseException here.

Change-Id: I1b1af8ba0bf49251fe9d1a24c6559db3ef4a59d2
2025-04-06 11:52:32 +02:00
jadeandLix Systems Gerrit 7bf3d0fa2a Merge changes Iaa962f04,I19480ab0 into main
* changes:
  MonitorFdHup: introduce a test
  tests: fix the fmt.cc test
2025-04-05 17:12:48 +00:00
FireFly c292fbc46d tests: actually test printing of paths in unit test
This seems to be a copypasta mistake that slipped through when writing the
initial tests.  Changes the ValuePrintingTests::tPath to actually test a path
value instead of a string.

Change-Id: I20fe72635a0b1a6a0d72f674a4619da0de9f3d44
2025-04-05 17:11:17 +02:00
Jade Lovelace 7aec313597 MonitorFdHup: introduce a test
Well, I was trying to figure out
https://git.lix.systems/lix-project/lix/issues/729 in which this feature
is clearly just broken on macOS, but frustratingly, it seems that it
*does* work, except for the daemon. um........ sure.

Change-Id: Iaa962f045c16fdfa82854151c90a03e5cf0eea47
2025-04-04 16:59:23 -07:00