Commit Graph
2336 Commits
Author SHA1 Message Date
jadeandLix Systems Gerrit 9c2dba4ee2 Merge "Revert "libstore: don't use curl decompression support"" into main 2025-04-09 16:01:37 +00:00
eldritch horrorsandLix Systems Gerrit 350bf7e42d Revert "libstore: don't use curl decompression support"
This reverts commit bba678e5c5.

Reason for revert: didn't fix the bug and created new ones (fj#794)

Change-Id: I0450205d3041b6c876737151a4051081c1366f1d
2025-04-09 11:28:50 +00:00
Commentator2.0 fb0ef6ca6b Made parser warnings use Error info
no other warn calls were found, which have position information

Change-Id: If9773b86a756e5b2e4096d3f6b3a5114bfed5bee
2025-04-08 10:21:53 +02:00
eldritch horrors 8d8bb60796 treewide: lint against non-lix exceptions
we generally do not want to catch or throw these. catching them to print
and discard is fine, tests are largely exempt, and cases in which we can
be certain where the exception came from are also fine to *catch*. we'll
try to never *throw* (or rethrow) these if possible though because doing
so will make it impossible to construct async traces for the exceptions.

Change-Id: I3b71c32ecd16afc2246c946472f5629a1fa31f2c
2025-04-06 12:14:38 +02:00
eldritch horrors 63d550938b treewide: derive all lix exceptions from BaseError
even the non-errors. we should probably insert a BaseException here.

Change-Id: I1b1af8ba0bf49251fe9d1a24c6559db3ef4a59d2
2025-04-06 11:52:32 +02:00
jadeandLix Systems Gerrit 7bf3d0fa2a Merge changes Iaa962f04,I19480ab0 into main
* changes:
  MonitorFdHup: introduce a test
  tests: fix the fmt.cc test
2025-04-05 17:12:48 +00:00
FireFly c292fbc46d tests: actually test printing of paths in unit test
This seems to be a copypasta mistake that slipped through when writing the
initial tests.  Changes the ValuePrintingTests::tPath to actually test a path
value instead of a string.

Change-Id: I20fe72635a0b1a6a0d72f674a4619da0de9f3d44
2025-04-05 17:11:17 +02:00
Jade Lovelace 7aec313597 MonitorFdHup: introduce a test
Well, I was trying to figure out
https://git.lix.systems/lix-project/lix/issues/729 in which this feature
is clearly just broken on macOS, but frustratingly, it seems that it
*does* work, except for the daemon. um........ sure.

Change-Id: Iaa962f045c16fdfa82854151c90a03e5cf0eea47
2025-04-04 16:59:23 -07:00
Jade Lovelace 6d8e4337e2 tests: fix the fmt.cc test
Yikes!! I wonder if we have any other ones of these that just .. didn't
get added to a meson file?

Change-Id: I19480ab03cdbecf608e523d5b6c3980233f4f445
2025-04-03 18:40:06 -07:00
Maximilian BoschandLix Systems Gerrit d169c092fc Merge "fetchers: don't consider a path locked if a rev is specified" into main 2025-03-29 18:51:45 +00:00
eldritch horrors 0d47773d76 treewide: handle JSON parse errors
or more accurately, wrap them in a nix::Error subclass so we can display
them properly without crashing, and add some error context if available.

fixes #642
fixes #753
fixes #759
fixes #769

Change-Id: I1aad0c0501fea83f9de3a1335eaa6adc20721616
2025-03-27 08:56:14 +00:00
eldritch horrors fc58e7b217 libexpr: mark eval error handling NeverAsync
technically it doesn't *have* to be NeverAsync, but not marking it as
such unconditionally requires templating DebugState over asyncness of
its callback (which then requires templating EvalState, which, *NO*.)

Change-Id: I4980d45b541c2e40328beac139b18c6c1ba0957c
2025-03-25 21:26:47 +01:00
eldritch horrors 8a90195cdb libexpr: don't debugThrow from async code
running a debugger is never-async since it holds an EvalState reference.

fixes #761

Change-Id: Ie1b4df6f81bb5614f11ea183eb9d1058fde73121
2025-03-25 21:23:05 +01:00
eldritch horrors c3929c78f3 libutil: disallow enum serialization by default
allow opting in to serialization as integers via a trait type instead,
and add string-list serializers for the feature flag set enumerations.

fixes #738

Change-Id: I2746eb5ef1f15c01b4e681f9ba1615b6c6e64f44
2025-03-25 10:44:52 +01:00
eldritch horrors 56df5ba164 libutil: remove json-utils.hh
we can merge it into json.hh instead.

Change-Id: Ic40c25fa759bf52bb69eae5d7c0597f260fd94a6
2025-03-23 22:29:58 +00:00
eldritch horrors 45017f7508 libutil: specialize nix::JSON
we want to own this specialization fully so we can change the default
serializer behavior without also forcing downstream users of our code
to use the same behavior. it'll also let us do things we cannot do in
regular nlohmann::json, such as selectively enabling serialization of
enums as integral types, or using `to_json`/`from_json` overloads for
not-default-constructible types instead of serializer specializations

Change-Id: I91a1db362e37d654090f1824b1cd3ce783d32134
2025-03-23 22:16:30 +00:00
eldritch horrors 19d7c8352e treewide: add nix::JSON
this will become a proper specialization of `nlohmann::basic_json` soon.
specialing basic_json will let us get rid of our `adl_serializer` hacks,
and it'll open the door to better enum serializing behavior without also
forcing all those who use lix as a library to set certain defines (which
may not even be possible depending on how those users use json already).

Change-Id: I5228d2b9df581a189552c993363207cfbd20f445
2025-03-23 20:42:39 +00:00
eldritch horrors a186bc5021 treewide: add json forwarder headers
this doesn't do much, just wrap a few nlohmann headers in headers of our
own (and delete includes we don't need because they're transitively seen
by other includes). doing this now will make the next change much nicer.

Change-Id: I166933102ea86bb5322ebbf9ba9411f96032a53b
2025-03-23 20:42:39 +00:00
eldritch horrors 09769bbf41 treewide: remove unused nlohmann includes
Change-Id: I261e4bd4c859eb3924202e690331c05887aa26b0
2025-03-23 20:42:39 +00:00
jadeandLix Systems Gerrit 3f45d5e020 Merge "fix!: file type flake inputs are always recursive hashed" into main 2025-03-23 17:45:41 +00:00
Jade Lovelace b22bee91f5 fix!: file type flake inputs are always recursive hashed
Well that is a mess. I don't like breaking derivation compatibility,
but I like non-deterministic evaluation much less.

We can break compat if we own up to it, which this does.

Fixes: https://git.lix.systems/lix-project/lix/issues/750

Change-Id: Ic9e2407393f1d42c2be604f80b4aa11bc872bc23
2025-03-21 13:19:15 -07:00
Maximilian Bosch 62fb98ae55 fetchers: don't consider a path locked if a rev is specified
It's possible to put a path into the store in pure mode by pretending
it's locked like this:

    $ echo 'lalala' > testfile
    $ nix eval --expr '(builtins.fetchTree { path = "/home/ma27/testfile"; rev = "0000000000000000000000000000000000000000"; type = "path"; })'
    { lastModified = 1723656303; lastModifiedDate = "20240814172503"; narHash = "sha256-hOMY06A0ohaaCLwnhpZIMoAqi/8kG2vk30NRiqi0dfc="; outPath = "/nix/store/lhfz259iipmv9ky995rml8018jvriynh-source"; rev = "0000000000000000000000000000000000000000"; shortRev = "0000000"; }
    $ cat /nix/store/lhfz259iipmv9ky995rml8018jvriynh-source
    lalala

There was a fix in CppNix[1], but Puck noted that it's breaking
backwards-compatibility because

> [...] a github fetch with a fully specified rev is no longer considered "locked"
> this is for "purity" reasons, but it breaks any existing flake.nix files

I tried a way smaller correctness fix here:

* Each scheme can denote whether a `rev` is enough to consider itself
  locked.
* If a `rev` is given and the scheme is OK with just a `rev` to be
  locked, the input is marked as locked.

For `path` this is not the case anymore, i.e. it requires a NAR hash to
be locked down.

[1] https://github.com/nixos/nix/commit/071dd2b3a4e6c0b2106f1b6f14ec26e153d97446

Change-Id: Ibbbf4733c82bcfa1c24dfe099a896d8aaecd81cc
2025-03-20 13:17:34 +00:00
jadeandLix Systems Gerrit 3fcf823a86 Merge "fix: do not test non-presence of flakes from a subdirectory of lix src" into main 2025-03-18 20:03:40 +00:00
Jade Lovelace ed7c89790e fix: do not test non-presence of flakes from a subdirectory of lix src
Yeah, we could have seen that one coming, it's kind of on the nose,
really.

Fixes: https://git.lix.systems/lix-project/lix/issues/748
Change-Id: I9e89f73ed89105338343fdecd615e398fc758e89
2025-03-18 12:40:10 -07:00
eldritch horrors 27d5209f4d libutil: fix copyNAR not reading the whole nar when dropped early
if a copyNAR generator was not drained to completion it would not read
the full nar data from its source. this could happen if the copier was
passed to parseAndDump wrapped as a source because copyNAR would yield
nar metadata *before* it had read it, and GeneratorSource will drain a
generator fully *only* if the source is allowed to throw EndOfFile. in
the parseAndDump case this never happened because parseAndDump expects
to be given an unterminated stream, and thus the combination left some
nar metadata in the input Source, breaking the remote store protocols.

fixes #732

Change-Id: Ia59a53375992bfcdb7bc6b37764ca779622bc8f7
2025-03-18 19:32:58 +00:00
jadeandLix Systems Gerrit be4ab653eb Merge "fetchTree: make invisible when not available, sorta document" into main 2025-03-18 19:32:12 +00:00
Jade LovelaceandJade Lovelace fcea7379d3 fetchTree: make invisible when not available, sorta document
Documenting fetchTree is an exercise in frustration because of the sheer
amount of stringly typed everything in it. I do not know which fields
exist without reading the entirety of libfetchers. However, we can write
something slightly perfunctory but at least perhaps somewhat helpful
documentation-wise.

Fixes: https://git.lix.systems/lix-project/lix/issues/609
Change-Id: I991391b53fcd69172dbc7efb9d384e62bc847b91
2025-03-18 11:34:28 -07:00
jadeandLix Systems Gerrit a20a1e9692 Merge "libstore: move case hacking to the FS interaction code" into main 2025-03-18 17:56:53 +00:00
jadeandLix Systems Gerrit 6e6ff58272 Merge "feat!: remove experimental repl-flakes" into main 2025-03-18 16:02:55 +00:00
jadeandLix Systems Gerrit ea4cf48879 Merge "post-build-hook: don't emit config settings that aren't overridden" into main 2025-03-17 23:31:32 +00:00
KFearsandJade Lovelace 2c21882767 feat!: remove experimental repl-flakes
This is a backport of CppNix PR 10299

Fixes #557

Change-Id: I848e8c5af4bde09afba053a5911fbbd54004fbb8
2025-03-17 16:01:53 -07:00
Jade Lovelace 523965697d post-build-hook: don't emit config settings that aren't overridden
This is a redesign from 9b1f3cbc13 where
this was introduced.

I deleted the AbstractConfig::toKeyValue since it was conspicuously and
obviously broken for two years since 450e5ec618.
I asked myself if anyone was using it, given that it only emitted
settings that were aliases (broken!), and found that nobody used it.

The motivation for this change is the same for only emitting overridden
settings to the protocol: the nix inside there may not be able to parse
our defaults, as is the case of CppNix since the consensual
accept-flake-config was added to Lix.

Fixes: https://git.lix.systems/lix-project/lix/issues/739
Change-Id: Ib9874a52137f1f22220c25bcfa2425a4802509c7
2025-03-17 15:25:52 -07:00
Jade Lovelace adf45deeca plugins: support nix_plugin_entry, do some minor reworks
Plugins are already allowed to fail to load, let's be even more tolerant
of them failing to load by ignoring inaccessible ones with a warning.

This change also significantly reworks the docs for plugins.

This is a tiny extremely partial backport of the C API at
https://github.com/NixOS/nix/pull/8699.

Fixes: https://git.lix.systems/lix-project/lix/issues/740
CC: https://git.lix.systems/lix-project/lix/issues/359

Change-Id: If4ee20c3daaf26c8184a415eef3e20ca5b5e7aef
2025-03-16 16:43:54 -07:00
Julian Stecklina 0e59e5b308 libfetchers: ensure that lastModified is a uint64_t
When lastModified comes via inputFromAttrs it ends up as string in the
Attrs map. This results in:

error: input attribute 'lastModified' is not an integer

Fix by handling it like revCount, which already does the right thing.
If added a test and confirmed that it catches the issue.

Also kudos to alexander.sieg@cyberus-technology.de for helping with
debugging this!

Change-Id: I8378fcaea986d798cb8458d4e6e15c2a92c2520a
2025-03-13 09:34:59 +01:00
Jade Lovelace b816ae4119 libstore: move case hacking to the FS interaction code
This makes it much harder for bugs to crop up where stuff is wrongly
case hacked.

Fixes the nix store ls problem described in
https://git.lix.systems/lix-project/lix/issues/633, but it is still a
global setting which still sucks.

Fixes: https://git.lix.systems/lix-project/lix/issues/726
Change-Id: Ieba18976d1fb490830db0299ee9922b4b03b453e
2025-03-12 13:42:48 -07:00
ZitroneandGerrit Code Review 85a140accb Merge "libexpr: show all missing and unexpected arguments instead of just one" into main 2025-03-11 21:54:32 +00:00
Zitrone f6c4034e2f libexpr: show all missing and unexpected arguments instead of just one
also only give suggestions for unused arguments

Change-Id: Iae8e72defbbe2571a803dfd7216745b39848ccb2
2025-03-11 20:21:24 +01:00
Linus Heckemann 29b07611b8 gc: allow continuing deletion even if some paths are still live
This adds a gcTryDeleteSpecific operation. This is similar to
gcDeleteSpecific, but will not fail if any of the given paths cannot
be deleted. Paths that could not be deleted are reported in the new
`kept` field of struct GCResults.

This also changes the behaviour of gcDeleteSpecific, such that it will
now continue deleting paths even if it fails to delete one along the
way, and only throw an error once deletion of all the given paths has
been attempted. This seems reasonable to me, because it makes its
behaviour somewhat less surprising -- previously, if some paths were
deletable and others weren't, the deletable ones would be deleted iff
they preceded the live ones in lexical sort order.

This also fixes a regression introduced in
8614cf1334, whereby nix-store --delete
failed to delete paths if they had any dependents -- even if none of
the dependents had GC roots.

The gcTryDeleteSpecific operation is surfaced via additional flags for
the `nix store delete` and `nix-store --delete` commands.

This makes custom garbage-collection logic a lot easier to implement
and experiment with:

- Paths known to be large can be thrown at `nix store delete` without
  having to manually filter out those that are still reachable from a
  root, e.g.
  `nix store delete /nix/store/*mbrola-voices*`

- The --delete-closure option allows extending this to paths that are
  not large themselves but do have a large closure size, e.g.
  `nix store delete /nix/store/*nixos-system-gamingpc*`
  Having an option for this is not strictly necessary, but convenient
  because it doesn't require the user to add an extra `nix-store
  -qR` (or `nix path-info -r`) into their command, nor to rewrite
  their command to use `--stdin` if the closure ends up too large to
  fit on a command line.

- Other heuristics like atime-based deletion can be applied more
  easily, because `nix store delete` once again takes over the task of
  working out which paths can't be deleted.

Change-Id: If345407fe7b11bdb3a8fdc04b0d56c32ab3d5928
2025-03-11 14:10:32 +01:00
Linus HeckemannandGerrit Code Review 423a343937 Merge "libstore/gc: restore deletion of referrers" into main 2025-03-09 21:41:49 +00:00
Linus Heckemann d6c871eb2d libstore/gc: restore deletion of referrers
Fixes: 8614cf1334
Fixes #495

Change-Id: I4d8d655c6254786ba1222a5b37551c68d9e370ee
2025-03-09 22:15:12 +01:00
9p4andGerrit Code Review 8f871210b5 Merge "Detect if in Nix3 shell" into main 2025-03-09 17:45:14 +00:00
eldritch horrors bba678e5c5 libstore: don't use curl decompression support
it's broken with http2 and transfer flow control. cf fj#662

Change-Id: Iaf6312bfcefa18d168faef47f57481199dd30b8d
2025-03-08 20:58:34 +01:00
eldritch horrors 93c3ca4e92 libutil: fix async copyNAR failing on slow io
well, oops. on slow io (as can happen with ssh remote builders) we could
have extended a nar read buffer past what was actually read, injecting a
span of zeroes into the read buffer where we requested some data but got
a partial result instead. also add some tests that would've caught this.

Change-Id: I67aa06b4715aeec6a5bdacaafa9b79849e664e2f
2025-03-07 15:45:51 +01:00
eldritch horrors 0f3aba83c5 tests: turn off unit test verbosity
there's no real usefulness to this output in build logs. in interactive
development builds -v can be passed to meson to restore the old output.

Change-Id: Iac4c0573fd1cac5fc9044b950ef52ee50448e6fc
2025-03-07 15:44:42 +01:00
Ersei Saggi 7928669ef9 Detect if in Nix3 shell
This replicates behaviour found in the Nix2 commands, where IN_NIX_SHELL
is set. This is for shells to determine whether they are inside of a
Nix3 shell, and set a custom prompt accordingly.

For example, Fish's Tide prompt framework checks for that environment
variable and displays it in the prompt, indicating that the shell is in
a Nix environment.

This is not new behaviour, and the old Nix2 commands set the variable.

If the shell that is created is a "pure" shell, ie --ignore-environment
is passed, then IN_NIX_SHELL will be set to "pure". However, "nix
develop" will always create an impure environment.

Replicated from my Nix PR: https://github.com/NixOS/nix/pull/8885

Change-Id: I695cdc336f76541940a302835124fe7d8f7f39b2
Signed-off-by: Ersei Saggi <vcs@ersei.net>
2025-03-07 07:45:08 -05:00
eldritch horrors 4e266a25d6 libutil/libstore: asyncify Pool
connection pools of remote stores can currently block. this is not a
problem when each request runs on a dedicated thread, but with async
code this is no longer true. if a remote store has exhausted all its
available connections on one executor and another job starts *on the
same executor* we'll deadlock if that job makes another remote store
request. unlike with sqlite previously it's not reasonable, not even
necessary, to make the pools unbounded: since we use pools only with
remote stores we can asyncify all of them at once, and since they're
leaves of all call stacks we do not have much code to change either.

Change-Id: I8c457e27893e22c2cfc35933307a5283c986805a
2025-03-05 23:07:20 +01:00
eldritch horrors 66d515a3a2 libutil: add an async nar indexer
Change-Id: I4c22364c94e2bea1f18b835d9514c4c65a530c80
2025-03-03 20:48:59 +01:00
eldritch horrors 2367521008 libutil: add an async nar parser
sadly this is a visitor-only interface; async generators are not yet a
thing and preliminary benchmarks say that overhead would be too large.

Change-Id: I0460d18eba94441cf3101d46bfffdb69cdda81d1
2025-03-03 20:48:59 +01:00
eldritch horrors 18a24393e8 libutil: add an async copyNAR overload
Change-Id: I046ff704a2d2cb5115f7bf610feac3ed3247d992
2025-03-03 20:48:59 +01:00
eldritch horrors 99bc932196 libutil: add an async io header
we'll need all of these before long. we're so, so sorry.

Change-Id: I7baab54cf8a112b74d52e3a53f82836bd9f7cb83
2025-03-03 20:48:59 +01:00