The vast majority of changes was pretty straight-forward: we need an
AsyncIoRoot per thread to resolve promises we get from Lix. The `aio`
object is created per thread and passed as lvalue ref instead of making
it a field of the State struct since that struct is shared between the
threads.
The biggest change affects the NAR extractor: parseAndCopyDump has been
removed, instead the `nar::parse` and `nar::dump` parts are public now.
This won't work with the original parse visitor anymore, so instead the
NAR is parsed once and dumped again to copy it to the binary cache. In
between, the generator is walked through to search for relevant files
(such as build products) and after that the generators are recreated for
`AsyncGeneratorInputStream` to read again.
This also fixes#18: `nix-eval-jobs` isn't taken from the flake input,
but stems from the same overlay as Lix itself, so this won't cause two
Lix inside Hydra's closure anymore.
Most notably, this fixes the API change of `Store::narFromPath`[1] that
now returns a `box_ptr<Source>` instead of a Generator. I decided to do
the most minimal invasive thing here, i.e. dereferencing the source out
of the box in the NAR extraction code to fix the build again given that
the API will change even more in the future and thus the implementation
in here will require more work anyways.
To quote the upstream commit:
> [...] and would need to be changed much as they are now once async
> streams come around either way
[1] See Lix commit c82cc16754
I've been using the module on 24.11 and building Hydra from this repo
with 24.11 as well, so far it's looking good.
Making the upgrade since 24.05 is deprecated now.
Since ca1dc3f70b, the NAR parser has moved
the preallocate & receive steps into the file handle class to remove the
assumption that only one file can be handled at a time.
The third argument to `open()` in `-|` mode is passed to a shell if it's
a string. In my case the store URI contains
`?secret-key=${signingKey.directory}/secret&compression=zstd`
For the `nix store cat` case this means that
* until `&` the process will be started in the background. This fails
immediately because no path to cat is specified.
* `compression=zstd` is a variable assignment
* the `$path` argument to `store cat` is attempted to be executed as
another command
Passing just the list solves the problem.
`dev-notes` are severely outdated. I dropped everything except one note
that I moved to hacking.md. The parts about creating users are also
covered elsewhere.
The `update-dbix` part got a just command to make it discoverable again.
* justfile, inspired from Lix.
* let foreman use the stuff from outputs, similar to what Lix does>
* mess around with PERL5LIB[1] and PATH to get tests running locally.
[1] I don't really know how `Setup` was found before tbh.
When an artifact is requested from hydra the output is first copied
from the nix store into memory and then sent as a response, delaying
the download and taking up significant amounts of memory.
As reported in https://github.com/NixOS/hydra/issues/1357
Instead of calling a command and blocking while reading in the entire
output, this adds read_into_socket(). the function takes a
command, starting a subprocess with that command, returning a file
descriptor attached to stdout.
This file descriptor is then by responsebuilder of Catalyst to steam
the output directly
The primary host for this repo isn't github, and this is causing a whole
bunch of false indications in the Lix Forgejo UI as it tries to run the
.github/workflow with non-existing runners.
Only log issues/failures when something's actually up.
It has irked me for a long time that so much output came
out of running the tests, this seems to silence it.
It does hide some warnings, but I think it makes the output
so much more readable that it's worth the tradeoff.
Helps for highly parallel running of jobs, sometimes they'd not give output for a while.
Setting this timeout higher appears to help.
Not completely sure if this is the right place to do it, but it works fine for me.
We've seen many fails on ofborg, at lot of them ultimately appear to come down to
a timeout being hit, resulting in something like this:
Failure executing slapadd -F /<path>/slap.d -b dc=example -l /<path>/load.ldif.
Hopefully this resolves it for most cases.
I've done some endurance testing and this helps a lot.
some other commands also regularly time-out with high load:
- hydra-init
- hydra-create-user
- nix-store --delete
This should address most issues with tests randomly failing.
Used the following script for endurance testing:
```
import os
import subprocess
run_counter = 0
fail_counter = 0
while True:
try:
run_counter += 1
print(f"Starting run {run_counter}")
env = os.environ
env["YATH_JOB_COUNT"] = "20"
result = subprocess.run(["perl", "t/test.pl"], env=env)
if (result.returncode != 0):
fail_counter += 1
print(f"Finish run {run_counter}, total fail count: {fail_counter}")
except KeyboardInterrupt:
print(f"Finished {run_counter} runs with {fail_counter} fails")
break
```
In case someone else wants to do it on their system :).
Note that YATH_JOB_COUNT may need to be changed loosely based on your
cores.
I only have 4 cores (8 threads), so for others higher numbers might
yield better results in hashing out unstable tests.
Maximilian Bosch