We upgrade to 25.05 release, which contains the curl commit
https://github.com/curl/curl/commit/5fbd78eb2dc4afbd8884e8eed27147fc3d4318f6
done in
https://github.com/NixOS/nixpkgs/pull/396200#issuecomment-2795944006.
This fixes HTTP transfers generating arbitrary errors and possibly
failing unusually.
Users who are already depending on 25.05-small or a recent unstable
already had the fix.
Special mention to the Linux kernel who gave me the opportunity to get
on a 24 hours bisection side quest to fix the local release engineering
test.
Special thanks to everyone who had to endure me ranting.
Change-Id: I866caf65d5ea103f1fa5eccd57df8031c9eacda0
Co-authored-by: eldritch horrors <pennae@lix.systems>
Co-authored-by: helle <helle@h3l.li>
Signed-off-by: Raito Bezarius <raito@lix.systems>
Most of these are simple fixes and clarifications. One set of fixes will
come in the commit that actually upgrades nixpkgs and hence ruff as it
will otherwise cause errors here.
Change-Id: Ie857da0f6cf728478700ec2d24cf518f8c7b7815
we need a wrapper type for the remote exception because our Result type
does not deal well with its good type being the same as its error type.
we could have also return a `Result<Result<void>>` to fix this, but the
wrapper type clarifies via its name where the exception_ptr originates.
Change-Id: Ia6ce67b962cb8d6528b017f4cb682a55d6918939
the subframing layer is ... a bit of challenge. since the old code is
synchronous but wants to handle errors asynchronously anyway it is on
the subframing layer to *spawn a thread* that polls for errors on the
wire, while non-framed commands handle errors synchronously once they
have sent all their data. this encapsulation of the wires is far from
perfect (let alone legible), but hopefully it will be only temporary.
Change-Id: I26d8020549b767794cae121313360c488504995f
use a new helper method to send simple command data (that is, command
data that doesn't involve nested framing) to the daemon. this wraps a
large chunk of wire io, and once all wire io is wrapped thusly we can
replace the sink/source io model with new async input/output streams.
Change-Id: Ief9f520263c230a98403b8756bde917fd1cb236e
a size_t followed by as many pairs of things is exactly the format of a
vector of two-element tuples. it would also be the format of a map, but
Roots is a map of sets. rather than adding a serialization format fixed
to this map type (or some wrapper) we can deserialize the response as a
vector and convert it to the map-of-sets later as this is not run much.
Change-Id: I3950c0f7cc59661576170ace10b25a6f8af1464b
processStderr of RemoteStore wants to be a promise and it must be used
from connection setup, so the pool factory callback must be a promise.
Change-Id: I9ac742b6048ae6dba0bfa5dcb58971386229690b
async io for remote store connections needs some sync parts still for
serialization purposes, and those will have to reuse async io buffers
Change-Id: I05e066e3bf8c4318dc23306383f6a849d018ef91
the rpc transition will require sync and async objects to share a single
io buffer (since defining serializers on async is an immense pain in the
tail, slow, and ultimately not necessary). a generic buffer class allows
us to reuse existing serializers more readily (reuse them at all, even).
Change-Id: I5ebba8449f26f2bb76016818928183c7e0123be0
remote store async io will need to set O_NONBLOCK on the connection fds,
and right now the number of fds can vary between connection types: local
connections have one one fd for the sink/source pair since they use unix
sockets, but ssh connections have two because ssh uses pipes. this makes
it rather hard to manage flags correctly, and even harder to wait for io
readiness on both directions using kj. using sockets for ssh fixes this.
Change-Id: I0f563ece7627cd3fbd0f5ce21c25140469729e5a
If state records are not destroyed at destroy time, this might confuse a
new build that thinks there's a remnant of a cgroup when actually it was
destroyed.
This fixes a bunch of inoffensive and noisy warnings about cgroups being
deleted by someone else.
Reported-by: Ramses <@rvdp:infosec.exchange>
Change-Id: Ib3d33f4ecd6143f33e032c5107b288b4ecabaee1
Signed-off-by: Raito Bezarius <raito@lix.systems>
User locks are taken to avoid another build grabbing the same UID.
Under build user contention, it is possible to recycle the same UID from
another build which did not run the Goal destructor yet.
Prior to this change, cgroups were destroyed at Goal destruction time,
but user locks were released at `buildDone()` time.
Therefore, it was possible to have 2 builds fights for the same cgroup
and mess with it, resulting in confusion.
To avoid this, we override `cleanupHookFinally` in charge to release the
user locks and we destroy the cgroup before releasing the locks.
Statistics are kept in the `cgroup` object a bit longer and can be
obtained at `killSandbox(true)` time.
`AutoDestroyCgroup::kill` now ignore if the cgroup path has already been
destroyed, as kill is idempotent.
Reported-by: Ramses <@rvdp:infosec.exchange>
Reported-by: Frederico Schonborn <@fredericoschonborn:matrix.org>
Change-Id: Idfbf9aaf010c5f718f2c1c38548383d912d8ee95
Signed-off-by: Raito Bezarius <raito@lix.systems>
Such a RAII structure should NEVER be copyable or movable, otherwise:
```
AutoDelete x;
x = AutoDelete(p, false);
```
will trigger the immediate deletion of `p`!
This fixes an annoying bug where the state record for cgroups was
deleted immediately as soon as it was created.
Change-Id: I2bfbc0815706700a0a75b79d1059cc552119b2c9
Signed-off-by: Raito Bezarius <raito@lix.systems>
It's `delegated` and not `delgated`, also it's `DelegateSubgroup` and
not `DelegateSubtree` which I clearly hallucinated because of subtree
vs. sub(c)group.
Change-Id: Icfaa6116fa83416c431820978ef35aa8aa943feb
Signed-off-by: Raito Bezarius <raito@lix.systems>
We offer full cgroup delegation to our sandbox now, required for running
containers inside the sandbox.
To run systemd-nspawn or containers managers inside the sandbox, there
is a need for one extra ingredient now: control over your own cgroup
subtree inside the sandbox.
If, in addition, you need multiple UIDs, for e.g. rootless usecases, you
need to run with the `uid-range` system feature.
Therefore, when the daemon or Nix runs under the right condition, e.g.
systemd-style delegation of the cgroup subtree while placing the
nix-daemon in a supervisor sub-cgroup, we create a new sub-cgroup for
each build based on the build UID and delegate that sub-cgroup to the
builder's process.
Additionally, `uid-range` always request the `cgroups` feature now, as
`uid-range` builds would probably always benefit from having cgroups
delegated, but the converse is not true.
Inspired from https://github.com/NixOS/nix/pull/11412 with a different
design that does not use function-local statics to derive the root
cgroup.
Co-authored-by: Linus Heckemann <git@sphalerite.org>
Co-authored-by: Parker Hoyes <contact@parkerhoyes.com>
Change-Id: Ic8947c5adaf4b5bbd153386e05fad65a935274fa
Signed-off-by: Raito Bezarius <raito@lix.systems>
We drop it to re-introduce it via the concept of build context which
will control in which cgroup a certain build should be spawned.
Change-Id: I4b4705d768129a6d7c0f061dc2163ba116088b18
Signed-off-by: Raito Bezarius <raito@lix.systems>
Some source trees might not be representable inside of the NAR listing
format v1 as file paths (on Linux) are not guaranteed to be valid UTF-8.
When something like this happens on a large-scale build farm, a
mysterious "queued" but impossible to process job appears, this is
because we cannot write the NAR listing and serialization always fails.
Why did this work before? nlohmann was introduced _after_ such paths
were ingested, see: 09f00dd4d0.
What happened for such previously mis-serialized NAR listings?
```
curl -v 'https://cache.nixos.org/nz8p9hn00r6z7s57581c1hiv39pa1ia6.ls' |
brotli -d | jq .
```
This fixes the build of `sub-batch`
(https://github.com/kl/sub-batch/tree/master/tests/rename_invalid_utf8)
on ForkOS infrastructure.
Many thanks to Puck for the assistance on holding `rr` right on this one
and finding the history of these changes.
Change-Id: I2c2fbac70818e02810f9fd236c3a248187bf5fe7
Signed-off-by: Raito Bezarius <raito@lix.systems>
Instead of allocating a new Value and copy the symbol string
representation inside of it, we can pass along the underlying Value,
which avoids (garbage collected) allocations.
This results in:
* a ~8 % reduction for `gc.totalBytes` over
`nixos.ec2.closures.x86_64-linux` for NixOS 24.11. (920MiB → 842MiB)
* a slight reduction in CPU time due to less allocations being performed
at all
Change-Id: I097f586dbc98f889fbc62d0a5f80c9d76ddedfd2
Signed-off-by: Raito Bezarius <raito@lix.systems>
The backing storage for symbols becomes a class storing a Value and a
string.
The Value is itself a string which contents points to the owned string.
Recovering a `SymbolStr` is still possible.
Change-Id: I171151abc3c0a513f2150c4b54edd61dea256cce
Signed-off-by: Raito Bezarius <raito@lix.systems>
The symbol table will contain types that encloses a Value, thus, it
needs to depend upon the Value header, whereas the Value header depends
on `Symbol` for typedefs.
We move the typedefs in the place where they are used.
Change-Id: Ic533e5aad927b9bc4a9d1723430e90e86a4b5466
Signed-off-by: Raito Bezarius <raito@lix.systems>
This simplifies many call-sites where construction can take place
automatically.
Change-Id: I87f697d55375676345b388024eb8df900bf808de
Co-authored-by: Tom Hubrecht <github@mail.hubrecht.ovh>
Signed-off-by: Raito Bezarius <raito@lix.systems>
This fixes Meson's "Project does not target a minimum version but uses
feature introduced in '1.1': meson.options file" warning.
Silly Meson.
I also added a note in the top-level meson.build to indicate
`meson_version` is specified in more than one place.
Change-Id: I2c04278bb46a562a1c96cd2e5e4d9ce59ce8e125
Lix has a style guide:
https://wiki.lix.systems/books/lix-contributors/page/code but
contributors like me have been unable to enforce it, which is sad.
To avoid further violations of that style guide, we enable a pre-commit
hook for clang formatting of the changed lines.
Change-Id: I217452efa3ac8bd66b4d3a08a6fe9a241207790b
Signed-off-by: Raito Bezarius <raito@lix.systems>
This is useful to reformat only changed hunks of a file via
`clang-format`.
Change-Id: I9aa8526d75fd2301113ee57f3a2e595f3b03504f
Signed-off-by: Raito Bezarius <raito@lix.systems>
Currently, the typecheck for the config values is only done
half-heartedly only checking if something is either a list or non-list
item, but not checking what type the list items are
this commit fixes the typecheck and adds test for proper serialization
Change-Id: Ifd93842b19b1dd870bdb3af0c000243b4380e7aa
The error message used to only contain the last key of the merge failure
this commit changes the message to contain the full path to the merge
conflict, resolving ambiguity
Change-Id: I9848a559b1b888e50a548eef8609bf34506040de
currently, there is a small helper funciton in lang_util to check if
something is of a list type generic
to improve re-usability, this function is moved to utils and improved to
be also check for nested iterables and such
Change-Id: I92984daa4c4decf13d340a2ea5e52f724cee800e
this could've just ignored exceptions thrown by the remote. in the
current implementation there's no way such an exception could have
propagated to the client though, so there's no change in behavior.
Change-Id: Ide03bda1cb0ad7fb5f27b4ee5d16efd6c2b635ba
mostly to make moving this to async writes easier. this won't have a
performance impact because it's only a single packet, that's written
to a BufferedSink, but the connection sink only gets a single write.
Change-Id: I9a5f1afe7d3e25f5f4502ef9520ff2f2529431ba
the test is for the map that usually wraps it though because it's the
bit we're interested in replacing, and it has custom serializer code.
Change-Id: If77a236dfca738b646ed2b7a5c65515dad6b7295
the old protocols are largely untested, mostly unused, and have design
problems that make the RPC transition a lot harder, if not impossible.
in theory we could ship a transparent protocol-converting proxy that'd
isolate the daemon itself from old protocol versions, but that's a lot
of code to maintain for presumably little gain or even no gain at all.
Change-Id: I4c3f3bb34d39044f6aeb07c10caaf13b8340a220
eldritch horrors