Commit Graph
16660 Commits
Author SHA1 Message Date
eldritch horrors f5f2e1537d libutil: add capability support to runProgram2
launching pasta to not run as root will ambient require capabilities.

Change-Id: I1dd2506a1fa3944a9d9062123ef8a74903c597ea
2025-06-24 10:50:36 +00:00
eldritch horrors 0ea8649445 libutil: add generic redirections runProgram2
explicit stderr redirection makes mergeStderrToStdout unnecessary also.

Change-Id: I63de929e6dc53f6c5ceb2d43c2ce288bfc04d872
2025-06-24 10:50:36 +00:00
eldritch horrors d0678a57f9 libutil: make RunningProgram more useful
make it moveable, make it killable, and add a stdout fd accessor.

Change-Id: I2387cbe8ac67b899a322cd6c7d306ef9ea7abcd0
2025-06-24 10:50:36 +00:00
Raito Bezarius 1d4ddb7e3b libutil: ensure that _deletePath does NOT use absolute paths with dirfds
When calling `_deletePath` with a parent file descriptor, `openat` is
made effective by using relative paths to the directory file descriptor.

To avoid the problem, the signature is changed to resist misuse with an
assert in the prologue of the function.

Fixes CVE-2025-46415.

Change-Id: I6b3fc766bad2afe54dc27d47d1df3873e188de96
Signed-off-by: Raito Bezarius <raito@lix.systems>
2025-06-24 10:50:36 +00:00
Raito Bezarius c7867e89f9 libstore: ensure that passAsFile is created in the original temp dir
This ensures that `passAsFile` data is created inside the expected
temporary build directory by `openat()` from the parent directory file
descriptor.

Fixes CVE-2025-52993.

Change-Id: Ie5273446c4a19403088d0389ae8e3f473af8879a
Signed-off-by: Raito Bezarius <raito@lix.systems>
2025-06-24 10:45:29 +00:00
Raito Bezarius deb5150c82 libutil: writeFile variant for file descriptors
`writeFile` lose its `sync` boolean flag to make things simpler.

A new `writeFileAndSync` function is created and all call sites are
converted to it.

Change-Id: Ib871a5283a9c047db1e4fe48a241506e4aab9192
Signed-off-by: Raito Bezarius <raito@lix.systems>
2025-06-23 16:50:44 +02:00
Raito Bezarius 92eb418a59 libstore: chown to builder variant for file descriptors
We use it immediately for the build temporary directory.

Change-Id: I180193c63a2b98721f5fb8e542c4e39c099bb947
Signed-off-by: Raito Bezarius <raito@lix.systems>
2025-06-23 16:50:44 +02:00
Raito Bezarius 7fceee3ce3 libstore: open build directory as a dirfd as well
We now keep around a proper AutoCloseFD around the temporary directory
which we plan to use for openat operations and avoiding the build
directory being swapped out while we are doing something else.

Change-Id: I18d387b0f123ebf2d20c6405cd47ebadc5505f2a
Signed-off-by: Raito Bezarius <raito@lix.systems>
2025-06-23 16:50:44 +02:00
Raito Bezarius 31f976dc19 libutil: guess or invent a path from file descriptors
This is useful for certain error recovery paths (no pun intended) that
does not thread through the original path name.

Change-Id: I2d800740cb4f9912e64c923120d3f977c58ccb7e
Signed-off-by: Raito Bezarius <raito@lix.systems>
2025-06-23 16:50:44 +02:00
jadeandLix Systems Gerrit d8db15010d Merge changes I577bdc15,I1c5f7a5c,I75f56c80 into release-2.92
* changes:
  version: 2.92.1
  release-notes: note the flake.lock fix
  libexpr/flake: user friendly error if parsing flake.lock fails
2025-03-17 22:26:40 +00:00
Jade Lovelace 253be7d2ba version: 2.92.1
Change-Id: I577bdc1549e80c0b7621624e2d390153581b3bf2
2025-03-17 12:11:24 -07:00
Jade Lovelace a34e583305 releng: workaround awscli breaking garage by changing checksum algo
CHERRY-PICK: required to release

We use https://garagehq.deuxfleurs.fr for https://docs.lix.systems,
https://releases.lix.systems, https://cache.lix.systems. It's generally
great, but AWS doesn't, erm, care, about other implementations and broke
their client library.

We already ran into
https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/824, which was
mitigated by a garage upgrade to create a new error. These bugs were
what got us:
- https://github.com/boto/boto3/issues/4392
- https://github.com/aws/aws-cli/issues/9214

Error:
upload failed: release/manual/.nojekyll to s3://docs/manual/lix/nightly/.nojekyll An error occurred (InvalidRequest) when calling the PutObject operation: Bad request: invalid checksum algorithm

The missing checksum algorithm is CRC32NVME, with a bug filed here:
https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/963

Change-Id: Ib78a89034bf0f2a6773fc505a347b2aadb775e93
(cherry picked from commit 0f4c5b3b8a)
2025-03-17 19:10:24 +00:00
Rebecca Turnerandjade a8fb008106 releng: use aws s3 sync --delete when uploading manual
CHERRY-PICK: prereq to fixing the aws flags for garage shenanigans

For the user manual, we don't delete things that are missing when doing
aws s3 sync. This doesn't seem wise. If you delete a page in the manual,
it will stay public and visible.

This adds `--delete` to the `aws s3 sync` commands to fix this issue.

Closes #396

Change-Id: I6d7fb97bcdab96c0115d6c66fea0310125207df4
(cherry picked from commit 16df34b295)
2025-03-17 19:09:10 +00:00
Jade Lovelace 32cbb66a69 release-notes: note the flake.lock fix
Change-Id: I1c5f7a5cba12da1bfb2896ffc04ec89862cb346d
2025-03-17 12:06:39 -07:00
giliceandjade 9631e9a30f libexpr/flake: user friendly error if parsing flake.lock fails
CHERRY-PICK: repeated bug reports

Fix: https://git.lix.systems/lix-project/lix/issues/559
Change-Id: I75f56c801d7f16a2e1ef8e702f16f1ef91f7b01f
(cherry picked from commit 68373f8664)
2025-03-17 19:01:23 +00:00
jadeandLix Systems Gerrit 98772c4a3b Merge "Revert "libstore: don't use curl decompression support"" into release-2.92 2025-03-17 18:17:41 +00:00
jadeandLix Systems Gerrit a7fd5c3867 Merge "libexpr: fix checkSourcePath purity regression" into release-2.92 2025-03-17 17:58:29 +00:00
jadeandLix Systems Gerrit 6f3a7bbeb0 Revert "libstore: don't use curl decompression support"
This reverts commit 0d1f794178.

Reason for revert: pennae says it makes the bug more visible, for some unknown reason. See https://git.lix.systems/lix-project/lix/issues/662

Change-Id: I91853cb6645f188a260c4c71cfff4de7bea99feb
2025-03-17 17:56:20 +00:00
eldritch horrorsandjade 0d1f794178 libstore: don't use curl decompression support
it's broken with http2 and transfer flow control. cf fj#662

BACKPORT: due to regression
Change-Id: Iaf6312bfcefa18d168faef47f57481199dd30b8d
2025-03-09 08:39:10 +00:00
Alois Wohlschlager ef5689dc1b libexpr: fix checkSourcePath purity regression
Starting with commit 0dbfa7b26e access would also
be allowed to ancestors of allowed paths. This is (ironically) a significant
purity regression, since several users of the purity checks will themselves
assume that arbitrary descent is allowed. For example, `builtins.readDir` and
`builtins.path` could now refer to the filesystem root, breaking purity
entirely in the latter case by allowing to read arbitrary files. Restore the
previous behaviour of only allowing access to explicitly allowed paths.

Change-Id: Ie64180733ab735da9873255e1ccbf95ba7c9161c
(cherry picked from commit 9d99a7c2cf)
2025-03-04 18:15:15 +01:00
Lunaphiedandjade db55ca9a2e hotfix for releng scripts, fixes #440
Change-Id: I33b29fbf3920a2d4fb53b58db477dff5bf7a1af1
2025-02-21 01:17:44 +00:00
eldritch horrorsandJade Lovelace 4cd618272a libstore: never return from lockFile without a lock
signals could cause lockFile to return without having locked the file.
the garbage collector didn't check for this, and then hilarity ensued.

Change-Id: If86d33595e8bf5510d2b032139342261dc6e07c9
(cherry picked from commit d186064c3d)
2025-02-09 12:54:48 -08:00
eldritch horrorsandJade Lovelace 73f0213500 libstore: split tryLockFile lockFile
lockFile is currently interruptible by signals like SIGCHLD. which
happen a lot in daemons. now imagine that daemon enabled automatic
garbage collection. observe that the local store does not actually
check whether its lock operations have succeeded ... get the idea?

Change-Id: Ibfd7ee786c4fee3add72d4456a7e95e73e09c73e
(cherry picked from commit 79f9c39e36)
2025-02-09 12:54:48 -08:00
eldritch horrorsandJade Lovelace 9c1db3cd8b libstore: extract unlockFile from lockFile
propagating the "unlock" lock type through this high-level api is
nonsense. it doesn't make sense to treat locking and unlocking as
similar operations; unlocking *must* not not interruptible by our
checkInterrupt machinery or it will just leave locks lying around
for a potentially very long time. unlock operations should not be
taking long enough to *want* them interrupted anyway. even on nfs
this makes very little sense because nfs waits *uninterruptibly*.

Change-Id: I10d605c8fe6c651bee64466eee1f8e20251d39f4
(cherry picked from commit aa87c8aa93)
2025-02-09 12:54:48 -08:00
Dusk Banksandeldritch horrors 5fa27057b2 libexpr: fix --debugger --ignore-try
a65e9e5828 did not inform `tryEval` that
(as far as it's concerned) `state.debug` moved to `state.errors.debug`
and changed types. this resulted in the REPL erroneously coming up, that
REPL having a non-debug state, and segfaulting after that REPL exited.

it's probably good that `state.debug` isn't mutated by `--ignore-try`
anymore.

Change-Id: I1918e93edacd626452aa423fc2eb825080738835
Fixes: a65e9e5828 ("libexpr: extract eval error creation into new type")
Signed-off-by: Dusk Banks <me@bb010g.com>
(cherry picked from commit 6a583136b7)
2025-01-31 12:36:14 +00:00
Dusk Banksandeldritch horrors d2b1af70ee doc: add bb010g to change-authors
Change-Id: Iddd3c21b2c42669cec394ac6b80f4e766e4cb81c
Signed-off-by: Dusk Banks <me@bb010g.com>
(cherry picked from commit 050cf17307)
2025-01-31 03:21:44 +00:00
Raito Bezariusandjade 50def3fa73 fix(gc): log sudden "path in use" exceptions and recover during GC gracefully
Original-Author: picnoir <picnoir@alternativebit.fr>

Inspired from
https://github.com/NixOS/nix/pull/11922/commits/ced8d311a593fcf9c3823e4e118474ac132d8e60
and adapted for Lix needs.

TL;DR: The topological sort should ensure that it is possible to delete
the path iterated upon. Nonetheless, in some cases,
`invalidatePathChecked` can still throw `PathInUse`, the exception
bubbles up and cancel the garbage collection procedure, leaving the rest
of the paths untouched. This change ensure that the error is logged for
further investigation but doesn't prevent the GC to continue when it
can.

After code review, we decided to make it a `printInfo` to inform the
user about sudden "in use" dependencies during garbage collection and
let them re-run garbage collection if they care about this.

References: https://github.com/NixOS/nix/issues/11923
References: https://git.lix.systems/lix-project/lix/issues/621
Change-Id: I5606c9afd16b5faa747b713fde2dc24016990ba3
Signed-off-by: Raito Bezarius <raito@lix.systems>
(cherry picked from commit 6a41dae49a)
2025-01-23 06:04:00 +00:00
eldritch horrorsandjade 8e2ab5532c FileTransfer: fix race condition on awaitData
There's a race condition where awaitData could early-return for data
coming from a 404 response or similar and thus not rethrow the exception
that is forthcoming, and a related race during transfer setup (which
could retry a transfer *twice* per retry round).

This would then cause substitution failures like below since the exception
isn't caught in HttpBinaryCacheStore::getFile as intended, but instead
by an exception handler downstream of `drain()` which would error out
the entire operation.

Symptom:

 » nix-build ./docs-service.nix -o "docs-service-result"
error: unable to download 'https://cache.nixos.org/7mr3fy8w66gi5inmf0jkkkl90lxy4jyg.narinfo': HTTP e
rror 404 ()

       response body:

This is kind of a hack in how it is implemented: it assumes that you
can't intentionally be receiving a large unsuccessful response since in
such a case, `awaitData` will wait for finish() to be called to throw an
exception and will never escape until the download finishes, while
continuing to buffer the entire response into memory, which could be bad
if an error response had a large payload.

That said, nobody is sending Lix 1GiB of 404, so meh I guess, and this
is how it is seemingly intended to work. That was a design flaw of the
thing before any of the Lix team got our paws on it.

I tested this by adding _exit(0) inside the expected exception catch and
then running the offending command repeatedly to see if the symptom ever
appeared again, and it did not.

Needs cherry-pick to 2.92 and a 2.92.1 release once reviewed.

Fixes: https://git.lix.systems/lix-project/lix/issues/635
Co-Authored-By: lix@jade.fyi
Change-Id: If54f6eeaad60b5ca9d5b77d4d9232da1d295e7d1
(cherry picked from commit de58cd6e80)
2025-01-22 21:06:58 +00:00
Jade Lovelace 079528098f release: merge release 2.92.0 back to mainline
This merge commit returns to the previous state prior to the release but leaves the tag in the branch history.
Release created with releng/create_release.xsh

Change-Id: I4e4650d96de82b46c35171b3a9fc1e3a6ca8e6a1
2025-01-18 13:04:46 -08:00
Jade Lovelace 9446732466 release: 2.92.0 "Bombe glacée"
Release produced with releng/create_release.xsh

Change-Id: Ife8c3ed7dc44b6ed334e53ec260fd17a931bf55f
2.92.0
2025-01-18 13:04:46 -08:00
Jade Lovelace b5c3c21792 release: release notes for 2.92.0
Release created with releng/create_release.xsh

Change-Id: Ie02e27b7328758727e78c85925e04629cd4d6d14
2025-01-18 13:04:35 -08:00
Jade Lovelace fc8bd88cf4 releng: fix for newer xonsh
Looks like a module rename

Change-Id: I281e41b8781fa5aa75a3bcb6e6907e582f46dcb7
2025-01-18 13:04:20 -08:00
Jade Lovelace 9b2761d66d version.json: codename and remove -dev suffix in prep for 2.92
Change-Id: I30e45aac44e7fd5bd5cfe56a9514cee95b1d4d8b
2025-01-17 19:03:49 -08:00
Jade Lovelace fe79b90055 doc: add some missing release notes for 2.92
Change-Id: I4861f8885aac53ce76322aae0387facfdd5f3e88
2025-01-17 19:03:49 -08:00
Justin ! 0d14c2b67a libmain: always print all information when calling nix --version
This commit remove a check for the log level in the `printVersion()`
function that was making `nix --version` behaving weirdly and
inconsistently compared to other `nix-*` commands.

The root cause is the following code in `lix/nix/main.cc` that change
the log level at runtime if nix is call interactively:

```cpp
if (isatty(STDERR_FILENO)) {
  verbosity = lvlNotice;
} else {
  verbosity = lvlInfo;
}
```

This should be removed, but since it has more implication it's gonna be
done in another CL.

Fix: https://git.lix.systems/lix-project/lix/issues/620
Change-Id: Id2c83c51d7ef799ee2f9b8dbdd2bfaeaf2df6188
2025-01-16 22:56:42 -05:00
Jade Lovelace 9760c00591 clang-tidy: Obliterate FixIncludes check
We have obliterated the support across the build system for these
include paths anywhere, so the codemod is now redundant and can go.

Change-Id: I54082f39752c6aac6429e1c24026211adae8221a
2025-01-16 00:12:57 -08:00
Jade Lovelace bf3ebde25f pkg-config: remove legacy include paths!
This is a breaking change for non-migrated external clients. External
users always need to use <lix/libcmd/foo.hh> type include paths now.

This is as was always planned with the include rearrangement.

Change-Id: I269be91ff9f9cc94d5d3043cf3e0bdf8db1d8edb
2025-01-16 00:12:57 -08:00
eldritch horrorsandGerrit Code Review 423d8b03c8 Merge "libutil: thread-pool: ensure threads finished on error" into main 2025-01-15 22:33:30 +00:00
Jade Lovelace acbb3cff2d Merge remote-tracking branch 'pennae/path-access' into HEAD
This fixes a bug where flakes do not actually do purity path checks
correctly.

Tested-By: Jade Lovelace <lix@jade.fyi>
Change-Id: If7d131a8e73a5874fb15cfaa0dea3b8811ba35d2
2025-01-13 17:19:54 -08:00
Maximilian BoschandGerrit Code Review 38dd196b03 Merge "libstore: fix "illegal reference specifier 'man'"-error in postgresql_14" into main 2025-01-12 11:12:12 +00:00
Jade Lovelaceandeldritch horrors d46adb45eb tests: validate that flakes do path traversal checking
Apparently we had zero test coverage of this, let's fix that.

Change-Id: I00c906daf5acfc01913562036ca88abbf63dd3d9
2025-01-11 20:42:30 +01:00
eldritch horrors 0dbfa7b26e libexpr: forbid allowed -> disallowed -> allowed links
this is more of a theoretical problem, but it does allow changing the
behavior of a flake depending on mutable machine state. it's unlikely
that this could be used to reliably do anything bad, but it does lead
to even more non-determinstic evaluation of (notionally) pure flakes.

Change-Id: I5bac7ed045046da08a36c764ab887bc9c7551542
2025-01-11 20:42:30 +01:00
Dominique Martinet 4737d8b65e libutil: thread-pool: ensure threads finished on error
This fixes segfaults with nix copy when there was an error processing
addMultipleToStore.

Running with ASAN/TSAN pointed at an use-after-free with threads from
the pool accessing the graph declared in processGraph after the function
was exiting and destructing the variables.

It turns out that if there is an error before pool.process() is called,
for example while we are still enqueueing tasks, then pool.process()
isn't called and threads are still left to run.

By creating the pool last we ensure that it is stopped first before
running other destructors even if an exception happens early.

fixes #618

Change-Id: I42a355f632aa0354df94c5d5d8cbe7ab5196c9a6
2025-01-12 04:14:28 +09:00
eldritch horrorsandJade Lovelace c948b350fb libutil: add CheckedSourcePath for accessing things
SourcePath only manipulates path names now. all accesses must go through
a checked path going forward to ensure we don't escape restriction lists
of pure and restricted evaluation. if a directory path is checked it can
safely be assumed that the directory itself is allowed, and its contents
will likewise be safe to access. it is tempting to assumed that contents
will also be fine, but that's only true if the content is not a symlink.

Change-Id: Icec3098d53fe9dce50997954ba958fe4f304d59b
2025-01-10 15:20:27 -08:00
eldritch horrorsandJade Lovelace 8db8ac9a67 always checkSource before accessing anything
like earlier, anything accessed during eval must be checked against the
list of path restrictions. this notably excludes `Pos::getSource` which
is run only from an unrestricted context (resolving line/column numbers
for expressions), but since positions require the parser to run and the
parser requires a checked input to produce positions this is not a leak

Change-Id: I337859e9c780590d4434885125a3ef70a11f6e93
2025-01-10 15:20:27 -08:00
eldritch horrorsandJade Lovelace 3f6a1e45c9 libexpr: always checkSourcePath in resolveExprPath
the purpose of resolveExprPath is to produce a parser input path. parser
input paths must be validated against the path allow list so they do not
escape the restricted/pure eval sandbox. checking the input path and any
intermediate paths during resolving makes this a lot harder to do badly.

Change-Id: Ib31b5bca63fe26a5e08458a871cdc9f92f9b6a10
2025-01-10 15:20:27 -08:00
eldritch horrorsandJade Lovelace ede0851fb4 libexpr: move resolveExprPath to EvalPaths
Change-Id: I4f8e27bb816d6498df4d73a57e10b654eb995c32
2025-01-10 15:20:27 -08:00
eldritch horrorsandJade Lovelace 5af069b248 libutil: remove SourcePath::resolveSymlinks
in pure mode it is entirely useless. in impure mode it's mostly useless
since the way in which it is used is either equivalent to not being run
at all, or is equivalent to turning the following lstat into a stat. we
add a stat method instead for all those who need final symlinks stat'd.

Change-Id: I801886d18eb34b26e62b4c05d53318c6421a69bf
2025-01-10 15:20:27 -08:00
eldritch horrorsandJade Lovelace f93af1db1f libutil: make SourcePath::path private
use canonical() to get the disk path, to_string() to get the string form.

Change-Id: I95bb6df53356f30290b487d1cca0aa2fb37249ed
2025-01-10 15:20:27 -08:00
eldritch horrorsandJade Lovelace 3acba7951a libexpr: use StorePath::to_string in string contexts
`path.abs()` does the same thing, but `to_string` communicates intent as well.

Change-Id: I9619a9f2e32317f0a1cc8e092ce8898471b980ac
2025-01-10 15:20:27 -08:00