Commit Graph
17896 Commits
Author SHA1 Message Date
Raito BezariusandLix Systems Gerrit cfaeed469b Merge "libstore/ssh: remove echo started check" into main 2025-05-18 19:51:20 +00:00
Raito Bezarius 0dd8bf6c1c libstore/ssh: remove echo started check
The pre-flight `echo started` check over SSH was originally added in
577ebeaefb. As it is usual with these old
commits, understanding why is there a need for something is difficult.

The closest thing would be
> Fix a race starting the SSH master. We now wait synchronously for
> the SSH master to finish starting. This prevents the SSH clients
> from starting their own connections.

But, we removed SSH connection sharing, so this does not apply anymore.

Nonetheless, we believed this check was meant as a way to catch obvious
misconfigurations or SSH failures early, before handing off to
`nix-store`. However, this approach was not fruitful: it assumes the
remote has a `bash`-compatible shell, `echo` behaves in a standard way,
and no `ForceCommand` interferes—all of which are unreliable assumptions
in practice.

While the intent was to provide slightly better diagnostics (e.g. in
case of SSH hanging or returning an interactive shell), in practice it
does not meaningfully catch or improve real failure cases. The
underlying protocol or engine can and should handle those errors more
robustly anyway.

In contrast, this check *does* break several legitimate workflows,
including:

* remote builders using `ForceCommand` wrappers (e.g.
`nix-remote-build`-style setups), see
<https://discourse.nixos.org/t/wrapper-to-restrict-builder-access-through-ssh-worth-upstreaming/25834/15>,

* SSHing into minimal environments lacking `bash` (e.g. initrd,
busybox-based systems),

* configurations that don’t default to POSIX-like shells, e.g., nushell
enthusiasts.

As such, we’re removing this code. Protocol mismatch errors and SSH
failures can be rethought and handled more structurally elsewhere in the
engine.

Change-Id: I187f6881375d42ef83987a13a350c97964bbdb30
Signed-off-by: Raito Bezarius <raito@lix.systems>
2025-05-18 18:45:26 +02:00
Linus HeckemannandLix Systems Gerrit a7634f87aa Merge "libstore/local-derivation-goal: better debuggability for FOD mismatches" into main 2025-05-16 15:18:11 +00:00
eldritch horrors 8029cec3c0 libstore: remove some unused code
these must've been left around by accident if past cleanups.

Change-Id: I3458ae45e362e7912a6bbe1f31956c41094144d3
2025-05-16 13:16:27 +02:00
Rebecca Turner 3f355b8fd1 .editorconfig: json: init with 4 space indent
Relevant for `version.json`. Noticed while running the 2.93.0 release.

Change-Id: I9f740a3cd412c1d27a76c3feb4e05d89319de3b4
2025-05-15 15:11:57 -07:00
Rebecca Turner 61920dd663 releng: Update README.md documentation
These are (polished versions of) some notes I wrote down while preparing
the 2.93.0 release with Jade.

Fixes: https://git.lix.systems/lix-project/lix/issues/441

Change-Id: Ib7f0b83ce2984a86d3a0c354e707fc6ee569a7ea
2025-05-15 10:47:10 -07:00
Lily BallardandLix Systems Gerrit b7ce00fc55 Merge "libutil: move filterANSIEscapes tests" into main 2025-05-15 01:42:34 +00:00
rebecca “wiggles” turnerandLix Systems Gerrit fed92d56ee Merge changes Id732c31a,I022583e9,I2f08bd2f,I85090f4c into main
* changes:
  aws s3 cp: Note issue with `--checksum-algorithm=SHA256`
  releng/docker_assemble.py: fix empty `auths` error
  releng: remove unused variables
  releng: ignore `prev-git-branch.txt`
2025-05-14 19:15:53 +00:00
Lily BallardandLix Systems Gerrit 5ed5efe6fc Merge "libcmd: use correct stream for ANSI testing for markdown" into main 2025-05-14 02:15:18 +00:00
Lily Ballard bc5d7ad458 libutil: move filterANSIEscapes tests
This moves the original test suite for `filterANSIEscapes` into the same
file as the newer tests. There is some overlap between the old and new
tests but that doesn't hurt anything so I kept them as-is.

Change-Id: Id00000009919024a5f206ec9a7bc0022541ff612
2025-05-13 19:04:52 -07:00
Lily BallardandLix Systems Gerrit 6237c50161 Merge "libutil: handle OSC escapes in filterANSIEscapes()" into main 2025-05-14 02:02:08 +00:00
Lily Ballard 207b5d81bf libutil: handle OSC escapes in filterANSIEscapes()
This teaches `filterANSIEscapes()` how to find the end of an OSC
sequence. It also keeps OSC 8 (hyperlinks) when not instructed to filter
out all escapes, just as it keeps colors.

This also relaxes the parsing of CSI escapes to find the end of the
sequence for invalid sequences, and handles better escapes that don't
start CSI or OSC.

This fixes the repl output for `:doc builtins.fetchGit`.

Fixes: https://git.lix.systems/lix-project/lix/issues/160
Change-Id: Id0000000f2a6956c042c883a4545edf347fa1799
2025-05-13 18:59:08 -07:00
Rebecca Turner a3b3b06a21 aws s3 cp: Note issue with --checksum-algorithm=SHA256
There may or may not be a bug in `garage` here. Previously we added
`--checksum-algorithm=SHA256` here to fix it, but when @rbt was
running the release for 2.93.0, she found it actually made the S3
uploads fail.

If this command is failing, here are some links to investigate.

See: https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/963
See: https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1017
See: https://github.com/boto/boto3/issues/4392
See: https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-envvars.html#envvars-list-AWS_REQUEST_CHECKSUM_CALCULATION
Change-Id: Id732c31aa715191e78c7f0246e1b74cc202e675c
2025-05-13 12:45:21 -07:00
Rebecca Turner 4611d1f331 releng/docker_assemble.py: fix empty auths error
I had a `~/.docker/config.json` which was missing an `auths` key, which
caused an error. The release automation succesfully ignored the error,
but it was noisy. Using `json_obj.get('auths', {})` instead of
`json_obj['auths']` fixes this `KeyError`.

Change-Id: I022583e9e668bf8ad7bdc1fa5a3305aee2f18d85
2025-05-13 12:18:32 -07:00
Rebecca Turner 2e25580d8d releng: remove unused variables
`lib` and `config` were unused here. In the future maybe we should
integrate `deadnix` or something similar for linting.

Change-Id: I2f08bd2f87f74b90a5f76ea7db7e6d4db1663450
2025-05-13 12:18:31 -07:00
Rebecca Turner 53673b7331 releng: ignore prev-git-branch.txt
This file is created by `python -m releng tag`.

Change-Id: I85090f4c6d9c2ba9a991b47df6a7382b80f7ff52
2025-05-13 12:18:31 -07:00
Raito BezariusandLix Systems Gerrit 901940df00 Merge "libstore/profiles: do not dereference current generation if it doesn't exist" into main 2025-05-12 22:47:48 +00:00
eldritch horrors 3684c2f8a0 libutil: remove dynamic derivations feature
Change-Id: Id48c775197e89bfda711c5cb03752980785c3d26
2025-05-12 13:37:54 +02:00
eldritch horrors ccdd916226 libstore: de-ref {Derived,Built}Path::Built drvPath
they're no longer recursive, so this is perfectly fine.

Change-Id: If565a557f2c2074e2a96a7f89c51ff1c51146b36
2025-05-12 13:37:54 +02:00
eldritch horrors 84c1df46ea libstore: remove DerivedPathMap
single-level maps suffice now that dynamic derivations are gone.

Change-Id: If29998b104b31255292ab0c789622d7d27040f69
2025-05-12 13:37:54 +02:00
eldritch horrors 68dfcfc6a4 treewide: don't resolveDerivedPath opaque paths
resolution of opaque paths is just an expensive `->path`.

Change-Id: I0c8d8b908f358d0bd26d1dcba84b2de6cbdc7c29
2025-05-12 13:37:54 +02:00
eldritch horrors f5e2e78266 libexpr: remove more obsolete DerivedPath methods
split from the prior commit for easier review.

Change-Id: Iabf1bc759cb56a04211d4cb07769f53b0f302b39
2025-05-12 13:37:54 +02:00
eldritch horrors 5f723e96e6 libstore: flatten {,Single}{Built,Derived}Path
only dynamic derivations could produce a non-opaque drvPath. since
dynamic derivations are no longer supported we can have drvPath be
opaque at all times, simplifying downstream code significantly and
making quite a few methods unnecessary. discardOutputPath was only
called on drvPath members anyway and thus reduces to a copy, other
operations at the very least are no longer recursive. some vestige
of dynamic derivations remains in DerivedPathMap though (for now).

Change-Id: Ifb4ad53a3c67800be5a62540068c8279d4ae0046
2025-05-12 13:37:54 +02:00
eldritch horrors 8a539424c8 libexpr: drop support for dyn-drv string context
string context doesn't need any tests because it's never persisted or
shown to the user. getting rid of recursive string context means that
the context string parsers can be a lot simpler from here on forward.

Change-Id: I58443679ad76c0f28ea5f4eb8bfb3874f270e764
2025-05-12 13:37:54 +02:00
eldritch horrors a490e2d946 libstore: remove ability to read or write dyn-drvs
as with impure derivations it is still possible to garbage-collect
existing xp-dyn-drv derivations. we once again don't introduce any
new kinds of errors, we only change the dynamic type of exceptions
from MissingExperimentalFeature to UnimplementedError (although we
do throw FormatError when reading xp-dyn-drv derivations now, that
seems to make a little more sense than "feature not implemented").

Change-Id: Ic26e5b6c9c9e2533093e27f6cf901dc9db57c83e
2025-05-12 13:37:54 +02:00
eldritch horrors 5ae8ac84ea libstore: remove dyn-drvs remote store error hack
with dynamic derivations gone this code will never run again.

Change-Id: I7673a81269c33e62c6c184d33cbc9f3ba0079bee
2025-05-12 13:37:54 +02:00
eldritch horrors 936ac14f1a libexpr: disallow creation of text-hashed derivation outputs
only dynamic derivation produce text-hashed derivation outputs. toFile
produces text-hashed store paths, so we cannot remove text hashing now
without breaking stores, but we can disallow it in derivation outputs.

Change-Id: I95ff9882a59153a7d5fd509f5c9fd85925f30d02
2025-05-12 13:37:54 +02:00
eldritch horrors b8b05d4da4 libexpr: remove dynamic derivation eval support
Change-Id: I8bbdaa280f634bafd5abd7034a605564f45978c0
2025-05-12 13:37:54 +02:00
eldritch horrors 540071dd77 cli: disallow dynamic derivations
with impure derivations gone we move on to dynamic derivations. this too
is not done in a single commit because dynamic derivations are invasive,
modifying semantics of all references to derivation output paths and all
derivation dependency calculations. removing dynamic derivations cleanly
is made significantly harder by the multiple did-you-mean-sum types, aka
"wrappers for std::variant", holding all derivation outpath information.

Change-Id: Ice7a7700c7b54c6a6061d4beb322b4175923d27a
2025-05-12 13:37:54 +02:00
Lily Ballard 530532ca8a libcmd: use correct stream for ANSI testing for markdown
Rendering markdown tests if ANSI is supported in order to tell lowdown
to disable ANSI escapes. Unfortunately it was testing stderr and yet
nearly all rendered markdown output was printed to stdout.

Change-Id: Id0000000f0e667d235239c095330d355a9b7714a
2025-05-11 17:55:44 -07:00
Lily Ballard e4b48ca3f0 version.json: 2.93.0 -> 2.94.0-dev
Change-Id: Id00000000f2c82a2c5cedd4e962521a34436d45d
2025-05-11 16:51:25 -07:00
eldritch horrors 18aebab9b6 libstore: remove unused resolveDerivedPath overload
Change-Id: Iaaeb688d601a21c817fb0449faf9541ea8e0301a
2025-05-11 21:16:36 +00:00
Linus Heckemann d19593d00b libstore/local-derivation-goal: better debuggability for FOD mismatches
The expected and the obtained path are now printed as part of the
error message, making comparing them easier when they're both at hand.

The extra rethrow for the hash-mismatch exception in the bmCheck case
has been removed, allowing the path to be registered as in the
non-check case. This makes having both paths at hand a lot more likely!

The determinism check logic was incorrect for content-addressed paths,
since it only ever tried to compare the path produced, even if this
was not the path expected (in the case of fixed-output derivations) or
the path previously produced (in the case of non-fixed CA
derivations). This made little sense, because that would always be the
same path if it exists! The determinism check is therefore now
bypassed for CA paths. Having a correct determinism check for
non-fixed CA derivations and running the diff hook for fixed-output
derivations would be nice, but feels out of scope and bypassing the
inapplicable logic isn't a regression from the previous behaviour.

Change-Id: I5fc14fb477c8c7d2f5bdedad5591af916f72b128
2025-05-11 21:22:52 +02:00
YurekaandLix Systems Gerrit b2b519a3af Merge "lix-doc: remove meson pre-1.5 hacks" into main 2025-05-11 19:14:44 +00:00
eldritch horrors 7bbe6fc47f libstore: remove impure-drvs feature
no documentation seems to have existed for this feature.

Change-Id: I3ec8afd9aeedbff1cc5edabc9074df33dae7d357
2025-05-11 17:27:05 +02:00
eldritch horrors be07629820 libstore: remove DerivationType::Impure
all derivations are now pure again, making isPure a constant function.

Change-Id: I2c65a7b6c8255beb97e2bdd280e4b11e39cb4387
2025-05-11 17:27:05 +02:00
eldritch horrors ae98420772 libstore: drop support for representing impure outputs
nothing can create, or even handle, them any more.

Change-Id: I5ef80129d6b734e65633df6eb9f161ffd31d1327
2025-05-11 17:27:05 +02:00
eldritch horrors 2f9a4a71aa libstore: drop support for reading impure derivations
writing them is technically still supported because what makes a
derivation impure is entirely specified by magically named data,
but without derivationStrict being able to pass these through to
libstore there is no way (besides reading existing files) to get
any new impure derivations into an existing store. it will still
be possible to garbage-collect existing impure derivations since
the gc process does not need to read them as derivations, and we
are not introducing any new kinds of unsupported-feature errors.

Change-Id: I648f53129ce67ee2b48d0591219759812dd557da
2025-05-11 17:27:05 +02:00
eldritch horrors 75f234d84b libexpr: remove impure derivation creation support
Change-Id: I58f481c188c6a6e99d226b4862508cef853b7571
2025-05-11 17:27:05 +02:00
eldritch horrors 6599be1a9f cli: disallow impure derivations
we don't remove the entire feature in one go to make review easier.
impure derivations are rather unintrusive on their own, at least if
we compare them to dynamic or ca derivations in general, so we will
be done with this soon. as it stands impure derivations cannot work
without ca derivations, and those we *really* want to leave behind.

Change-Id: I4f01d8d758b2c85dcd6c3078304b5ee1b52f65b0
2025-05-11 17:26:55 +02:00
Raito BezariusandQyriad d19a9e3039 libstore/profiles: do not dereference current generation if it doesn't exist
If the profile inode is invalid, e.g. invalid symlink, the current
generation cannot be discovered.

Nonetheless, this should not be a reason for an assert failure, instead
of crashing, just raise an error.

Fixes fj#801.

Change-Id: I63937672173bc3bf37196de98307800adc5757e1
Signed-off-by: Raito Bezarius <raito@lix.systems>
Co-authored-by: Qyriad <qyriad@qyriad.me>
2025-05-11 01:07:18 +02:00
Commentator2.0andLix Systems Gerrit 1da9c0261e Merge changes I6830c2fc,Ib88565a1,I0b280587 into main
* changes:
  functional2: Added ruff formatter
  functional2: use loggers
  fix codestyle of functional2
2025-05-10 20:49:43 +00:00
Commentator2.0 b17502088d functional2: Added ruff formatter
Ruff is used to enforce our code-style for the python parts of the
reposity, similar to clang-tidy for the cpp parts.

This includes a pre-commit hook to format code before it is committed
When "unfixable" - i.e. no autoformatting is available - the commit is
rejected

resolves #812

Change-Id: I6830c2fc29ae86337ec18f2b0e3565fac66c5523
2025-05-10 22:14:10 +02:00
Commentator2.0 01985e5add functional2: use loggers
Use logger in favor over print statment.
This is explicitly supported and encuraged by pytest, which also allows
for capturing logs separate from stdout calls, which is handy for when
e.g. lix code calls out to stdout to keep those differentiated from test
output

Change-Id: Ib88565a1663da3b77ca6b95f8edf644eafb4a99d
2025-05-10 21:13:45 +02:00
Commentator2.0 427696a58d fix codestyle of functional2
Fixing up codestyle issues found within functional2 for later adding
ruff formatter

Change-Id: I0b280587c8243137184091a6d36df3dfe7568eb7
2025-05-10 21:13:45 +02:00
eldritch horrors d8e2f53d07 deprecate CA, dynamic, and impure derivations
ca derivations are what we're really after, but dynamic derivations
must also go because they depend on ca derivations. we can't easily
implement dynamic derivations any other way, so we remove them too.
impure derivations build on the content-addressed infrastructure in
ways we cannot easily detangle, so they too must go for time being.

see #815

Change-Id: If61371736dfd89cc71a1b2ae5a005757c3cb9484
2025-05-10 17:15:26 +00:00
benaryorg fcd967da16 libexpr: fetchGit output documentation
Documentation for the output of `builtins.fetchGit`.
In particular this includes details on the both `lastModifiedDate` and `shortRev` which were not readily apparent.
Recommendations are made on the use of `shortRev`; it is considered stable, yet use is discouraged to avoid compatibility and interoperablity issues.

Fixes fj#814

Change-Id: If65c4f84d8a1569dcab2db07f63e69e4053ab74b
Signed-off-by: benaryorg <binary@benary.org>
2025-05-10 13:43:38 +00:00
eldritch horrors 5917db84aa nix: don't send tarballTtl to the daemon
it's an eval-time only setting, the daemon doesn't use it anywhere. this
is a hack, but until we have a much better settings system we are stuck.

fixes #680

Change-Id: I532088b0279f13da0a0a65c2bd2e5f9d1dfb39da
2025-05-10 12:20:09 +00:00
Linus HeckemannandLix Systems Gerrit a10de1ee56 Merge "build-release-notes: ignore dotfiles" into main 2025-05-10 09:53:56 +00:00
Rebecca Turner dcb0a97000 release: merge release 2.93.0 back to mainline
This merge commit returns to the previous state prior to the release but leaves the tag in the branch history.
Release created with releng/create_release.xsh

Change-Id: I67db1020490da19f5cf35b04b9ccfe47cfe829d1
2025-05-09 14:48:06 -07:00