Commit Graph
17647 Commits
Author SHA1 Message Date
Jade Lovelace 0f4c5b3b8a releng: workaround awscli breaking garage by changing checksum algo
We use https://garagehq.deuxfleurs.fr for https://docs.lix.systems,
https://releases.lix.systems, https://cache.lix.systems. It's generally
great, but AWS doesn't, erm, care, about other implementations and broke
their client library.

We already ran into
https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/824, which was
mitigated by a garage upgrade to create a new error. These bugs were
what got us:
- https://github.com/boto/boto3/issues/4392
- https://github.com/aws/aws-cli/issues/9214

Error:
upload failed: release/manual/.nojekyll to s3://docs/manual/lix/nightly/.nojekyll An error occurred (InvalidRequest) when calling the PutObject operation: Bad request: invalid checksum algorithm

The missing checksum algorithm is CRC32NVME, with a bug filed here:
https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/963

Change-Id: Ib78a89034bf0f2a6773fc505a347b2aadb775e93
2025-03-13 15:08:43 -07:00
ToastandGerrit Code Review a4514e44f8 Merge "Repl: error out if :log is called without a derivation" into main 2025-03-13 19:15:47 +00:00
jadeandGerrit Code Review 20edd45ae8 Merge "releng: Document release validation checks" into main 2025-03-13 17:53:29 +00:00
Toast 84fbc9d644 Repl: error out if :log is called without a derivation
The repl crashes with an assertion error at lix/libutil/file-system.cc:45
if you run ':log' without any derivation, so I added a check to error out
gracefully

Change-Id: I7af70040f884e905f0514ea9883b2380fe148f15
2025-03-13 18:24:09 +01:00
Raito BezariusandGerrit Code Review b1a8c0f2e0 Merge "libfetchers: ensure that lastModified is a uint64_t" into main 2025-03-13 16:44:10 +00:00
Julian Stecklina 0e59e5b308 libfetchers: ensure that lastModified is a uint64_t
When lastModified comes via inputFromAttrs it ends up as string in the
Attrs map. This results in:

error: input attribute 'lastModified' is not an integer

Fix by handling it like revCount, which already does the right thing.
If added a test and confirmed that it catches the issue.

Also kudos to alexander.sieg@cyberus-technology.de for helping with
debugging this!

Change-Id: I8378fcaea986d798cb8458d4e6e15c2a92c2520a
2025-03-13 09:34:59 +01:00
Rebecca TurnerandJade Lovelace 761001f58d releng: Document release validation checks
After the release is created, we do some post-release validation for the
Lix packaging in Nixpkgs. This change documents those checks.

NB Most of these should probably be *pre*-release validation checks!

Closes #627.

Change-Id: I773e703582ccc8349987abd1e860a189a437cf3b
2025-03-12 13:52:13 -07:00
K900 40601ea3ab flake: don't override nix-prefetch-git when we don't need to
https://github.com/NixOS/nixpkgs/pull/384099 has landed.
Change-Id: Ic4034dcd3028f7f8e476bf436dd9e1cf174060c5
2025-03-12 22:21:20 +03:00
ZitroneandGerrit Code Review 85a140accb Merge "libexpr: show all missing and unexpected arguments instead of just one" into main 2025-03-11 21:54:32 +00:00
Zitrone f6c4034e2f libexpr: show all missing and unexpected arguments instead of just one
also only give suggestions for unused arguments

Change-Id: Iae8e72defbbe2571a803dfd7216745b39848ccb2
2025-03-11 20:21:24 +01:00
eldritch horrors a60a362d8a libstore: correctly move remote connection exception info
Change-Id: I264dab54502c178a362593b3c757110588e0548a
2025-03-11 16:15:20 +00:00
eldritch horrors ba9ab0c855 libutil: add in-place construction to Sync
only then can we Sync<> non-movable types.

Change-Id: I7835cdaf9fd2b90c842c9e054942524f36515d5a
2025-03-11 16:15:20 +00:00
Linus HeckemannandGerrit Code Review be1491fa6a Merge "gc: allow continuing deletion even if some paths are still live" into main 2025-03-11 13:42:16 +00:00
Linus Heckemann 29b07611b8 gc: allow continuing deletion even if some paths are still live
This adds a gcTryDeleteSpecific operation. This is similar to
gcDeleteSpecific, but will not fail if any of the given paths cannot
be deleted. Paths that could not be deleted are reported in the new
`kept` field of struct GCResults.

This also changes the behaviour of gcDeleteSpecific, such that it will
now continue deleting paths even if it fails to delete one along the
way, and only throw an error once deletion of all the given paths has
been attempted. This seems reasonable to me, because it makes its
behaviour somewhat less surprising -- previously, if some paths were
deletable and others weren't, the deletable ones would be deleted iff
they preceded the live ones in lexical sort order.

This also fixes a regression introduced in
8614cf1334, whereby nix-store --delete
failed to delete paths if they had any dependents -- even if none of
the dependents had GC roots.

The gcTryDeleteSpecific operation is surfaced via additional flags for
the `nix store delete` and `nix-store --delete` commands.

This makes custom garbage-collection logic a lot easier to implement
and experiment with:

- Paths known to be large can be thrown at `nix store delete` without
  having to manually filter out those that are still reachable from a
  root, e.g.
  `nix store delete /nix/store/*mbrola-voices*`

- The --delete-closure option allows extending this to paths that are
  not large themselves but do have a large closure size, e.g.
  `nix store delete /nix/store/*nixos-system-gamingpc*`
  Having an option for this is not strictly necessary, but convenient
  because it doesn't require the user to add an extra `nix-store
  -qR` (or `nix path-info -r`) into their command, nor to rewrite
  their command to use `--stdin` if the closure ends up too large to
  fit on a command line.

- Other heuristics like atime-based deletion can be applied more
  easily, because `nix store delete` once again takes over the task of
  working out which paths can't be deleted.

Change-Id: If345407fe7b11bdb3a8fdc04b0d56c32ab3d5928
2025-03-11 14:10:32 +01:00
eldritch horrors f21fd6b7e6 libstore: fix substitution failures on racy builds
if two derivations have different outputs of the same other derivation
as inputs there's a race window in which wanted output tracking broke:

 - start building depender-a
 - schedule dependency^a for depender-a
 - substitute dependency^a
 - dependency runs path validity checks, yields
 - start building depender-b
   - depender-b adds b to the wanted set of dependency
 - dependency resumes from validity checks with allValid
 - depender-b is unblocked, but dependency^b is not in the store

Change-Id: Id4dba517bbbdbae96bf430d2c68f63eccd3b99dd
2025-03-10 21:18:22 +00:00
Linus HeckemannandGerrit Code Review f9d91129de Merge "libcmd/markdown: avoid stripping ANSI escapes" into main 2025-03-10 21:05:40 +00:00
Linus Heckemann a954a5ea2f libcmd/markdown: avoid stripping ANSI escapes
Instead, we instruct lowdown not to emit them in the first place.

Fixes #622.

Change-Id: I596bb873943c2c0b5067c5357a90457f01307cfa
2025-03-10 19:32:00 +01:00
Linus HeckemannandGerrit Code Review f7a93d8ffb Merge "nix store delete: don't realise paths" into main 2025-03-10 18:27:29 +00:00
Linus HeckemannandGerrit Code Review 423a343937 Merge "libstore/gc: restore deletion of referrers" into main 2025-03-09 21:41:49 +00:00
Linus Heckemann d6c871eb2d libstore/gc: restore deletion of referrers
Fixes: 8614cf1334
Fixes #495

Change-Id: I4d8d655c6254786ba1222a5b37551c68d9e370ee
2025-03-09 22:15:12 +01:00
9p4andGerrit Code Review 8f871210b5 Merge "Detect if in Nix3 shell" into main 2025-03-09 17:45:14 +00:00
alois31andGerrit Code Review 3df901e6ed Merge changes Ic91ca404,I142cf38f into main
* changes:
  libmain/progress-bar: implement synchronized updates
  libmain/progress-bar: fix writeToStdout in multiline mode
2025-03-09 17:17:58 +00:00
Alois Wohlschlager a13521bb1f libmain/progress-bar: implement synchronized updates
If tearing occurs while the progress bar is rendering, flickering can occur.
This is particularly visible in multiline mode. Use the synchronized updates
specification [1] to make updates atomic on supported terminal emulators. On
unsupported terminals, the escape sequences should be ignored, leaving the
behaviour effectively unchanged (but there's nothing we could do better in this
situation).

[1] https://gitlab.com/gnachman/iterm2/-/wikis/synchronized-updates-spec

Change-Id: Ic91ca40422c73ac9f9e088c7cb78c1b8c28adc4e
2025-03-09 16:36:02 +01:00
Alois Wohlschlager 0b2d58a925 libmain/progress-bar: fix writeToStdout in multiline mode
The multiline progress bar would not clear itself before writing to stdout,
leading to interference with the redraw; the most visible symptom is after
building in the repl, where the lines indicating the build outputs would be
wiped instead of the progress bar header. Separate the steps of erasing and
redrawing the progress bar, so that arbitrary output can happen in between in
a non-awkward way. In addition to fixing the bug, the code is simplified.

Change-Id: I142cf38f5ba5cd672cd6016e996b2f7bf726e9cd
2025-03-09 16:36:01 +01:00
Linus Heckemann 6a4efb82bc nix store delete: don't realise paths
Change-Id: I9566974a3afa70c61a0c8185b4dbd24a6440473f
2025-03-09 13:07:03 +01:00
eldritch horrors bba678e5c5 libstore: don't use curl decompression support
it's broken with http2 and transfer flow control. cf fj#662

Change-Id: Iaf6312bfcefa18d168faef47f57481199dd30b8d
2025-03-08 20:58:34 +01:00
eldritch horrors 93c3ca4e92 libutil: fix async copyNAR failing on slow io
well, oops. on slow io (as can happen with ssh remote builders) we could
have extended a nar read buffer past what was actually read, injecting a
span of zeroes into the read buffer where we requested some data but got
a partial result instead. also add some tests that would've caught this.

Change-Id: I67aa06b4715aeec6a5bdacaafa9b79849e664e2f
2025-03-07 15:45:51 +01:00
eldritch horrors 0f3aba83c5 tests: turn off unit test verbosity
there's no real usefulness to this output in build logs. in interactive
development builds -v can be passed to meson to restore the old output.

Change-Id: Iac4c0573fd1cac5fc9044b950ef52ee50448e6fc
2025-03-07 15:44:42 +01:00
Ersei Saggi 7928669ef9 Detect if in Nix3 shell
This replicates behaviour found in the Nix2 commands, where IN_NIX_SHELL
is set. This is for shells to determine whether they are inside of a
Nix3 shell, and set a custom prompt accordingly.

For example, Fish's Tide prompt framework checks for that environment
variable and displays it in the prompt, indicating that the shell is in
a Nix environment.

This is not new behaviour, and the old Nix2 commands set the variable.

If the shell that is created is a "pure" shell, ie --ignore-environment
is passed, then IN_NIX_SHELL will be set to "pure". However, "nix
develop" will always create an impure environment.

Replicated from my Nix PR: https://github.com/NixOS/nix/pull/8885

Change-Id: I695cdc336f76541940a302835124fe7d8f7f39b2
Signed-off-by: Ersei Saggi <vcs@ersei.net>
2025-03-07 07:45:08 -05:00
eldritch horrors f61bcc6ca9 libstore: turn LocalStore::dbPool into a mutex
this reverts commit 749a323597. the pool
does hurt performance and concurreny, so let's revert now that we can.

Change-Id: I0b154cf04d14fa2cb4d2028f9cc865e463cd2265
2025-03-05 23:07:20 +01:00
eldritch horrors 166860fbc6 libsture: use processGraphAsync in Store
processGraph is not fully async-safe because it cannot yield, ever.

Change-Id: I1c988bc5113303a8a31a6f96c6edc08a9d73c336
2025-03-05 23:07:20 +01:00
eldritch horrors 4e266a25d6 libutil/libstore: asyncify Pool
connection pools of remote stores can currently block. this is not a
problem when each request runs on a dedicated thread, but with async
code this is no longer true. if a remote store has exhausted all its
available connections on one executor and another job starts *on the
same executor* we'll deadlock if that job makes another remote store
request. unlike with sqlite previously it's not reasonable, not even
necessary, to make the pools unbounded: since we use pools only with
remote stores we can asyncify all of them at once, and since they're
leaves of all call stacks we do not have much code to change either.

Change-Id: I8c457e27893e22c2cfc35933307a5283c986805a
2025-03-05 23:07:20 +01:00
eldritch horrors 2a1a78c025 libstore: turn Store::state into an async mutex
Change-Id: I9192203e036ee24cac09ed14f7a343a9595ebe70
2025-03-05 18:49:45 +01:00
eldritch horrors 3fc07e38c7 libstore: asyncify Store::getStats
nothing uses this at the moment, but we may as well keep it.

Change-Id: I9e0061fb03e0371f83071d08e6dba1275c727226
2025-03-05 18:49:45 +01:00
eldritch horrors c72d057ee3 libstore: asyncify LocalStore::invalidatePath
Change-Id: I9dc8704b183a48c10d5dfd211aac621764e0a483
2025-03-05 18:49:45 +01:00
eldritch horrors 81d5528646 libstore: asyncify Store::queryPathInfo{,Uncached}
Change-Id: I182c396ec0bfdfc8febd6752774e668b3d97dcc1
2025-03-05 18:49:45 +01:00
eldritch horrors 09727e15c5 libstore: asyncify Store::isValidPathUncached
Change-Id: I5a2df380490325282768b2edbe42656ffe40d10e
2025-03-05 18:49:45 +01:00
eldritch horrors 90caba8489 libstore: asyncify Store::isValidPath
Change-Id: I2b98781c04944282fd1219cbec0804fd1ceb6765
2025-03-05 18:49:45 +01:00
eldritch horrors fb2a808604 libstore: asyncify Store::narFromPath
Change-Id: Ia231b83d58adcf52fe1ba107a0eefea8111528df
2025-03-05 18:49:45 +01:00
eldritch horrors d3e435b2b9 libstore: asyncify BinaryCacheStore::writeNarInfo
Change-Id: Iad36b9541876a6a66030db357104ed189b45c4f0
2025-03-05 18:49:45 +01:00
eldritch horrors a29e3a4a58 libstore: asyncify FSAccessor, listNar
Change-Id: I804ac18c4b6b80699247bb885493056799bf3d0c
2025-03-05 18:49:45 +01:00
eldritch horrors 19b8502ced libstore: asyncify Store::read{,Invalid}Derivation
Change-Id: Id1af47e5f35ea81ede9c47dba8f50045a2b9bde2
2025-03-05 18:49:45 +01:00
eldritch horrors f6ce58ac72 libstore: asyncify hashDerivationModulo
Change-Id: I15285a5b833799c3e23e44229a28a7af1bcb3e8a
2025-03-05 18:49:45 +01:00
eldritch horrors f43dbf3035 libstore: asyncify Derivation::checkInvariants
Change-Id: I269d5d11af811e4ab2f099fdc4e1c56597f38678
2025-03-05 18:49:45 +01:00
eldritch horrors 9d52d0db79 libstore: asyncify LocalStore::addValidPath
Change-Id: Iada9f7647b6913d267b061421d7b3d7b1bed8343
2025-03-05 18:49:45 +01:00
eldritch horrors fc0bea42c2 libstore: asyncify staticOutputHashes
Change-Id: Idb0daa96021656258c8018bb47cdf3c77871be0d
2025-03-05 18:49:45 +01:00
eldritch horrors 03ab0fcb53 libstore: asyncify Store::queryRealisation{,Uncached}
Change-Id: I4c4d75c773493fcd63c131e30a2e4d6ec5d230ac
2025-03-05 18:49:45 +01:00
eldritch horrors e057497f3e libstore: asyncify LocalStore::invalidatePathChecked
Change-Id: I425b155e98d4d85f6d2f91841cbb8725e195a607
2025-03-05 18:49:45 +01:00
eldritch horrors d5d498ab22 libstore: asyncify Store::queryReferrers
Change-Id: Iec5c85a22738f1a427774936d369e5f30af1a8a5
2025-03-05 18:49:45 +01:00
eldritch horrors 85cd558b6d libstore: asyncify Store::queryValidDerivers
Change-Id: I9ac447cb57e0ba330eee216280eb3d7913c5782b
2025-03-05 18:49:45 +01:00