guiwf/lix
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
18,653 Commits 37 Branches 40 Tags
6928c6f952af98a257ddd84eee551a3dda15ab1f
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Jörg ThalheimandRaito Bezarius 52e96bd421 fix crash in worker when opening the store 2024-05-11 17:41:27 +02:00
Jörg Thalheimandmergify[bot] 3c6e1234af flake.nix: return to nixpkgs-unstable 2023-12-17 12:15:15 +00:00
Jörg Thalheimandmergify[bot] 8ef37192bd README: add check-cache-status and nixpkgs evaluation examples 2023-12-17 12:15:15 +00:00
Jörg Thalheimandmergify[bot] 280de12085 README: update json and help output 2023-12-17 12:15:15 +00:00
Jörg Thalheimandmergify[bot] e3d71921c3 buffered-io: also check for interrupts in readline 
This allows us to more reliable interrupt nix-eval-jobs
 2023-12-17 12:15:15 +00:00
Jörg Thalheim b73f7ceff4 classify SIGSEV/SIGBUS as infinite recursion errors 2023-12-16 11:58:01 +01:00
Jörg Thalheim 093b8ce5cc apply treefmt 2023-12-16 11:44:08 +01:00
Jörg Thalheim b4f7eb3c77 fix ci for infinite recursion case 2023-12-16 11:40:34 +01:00
Jörg Thalheim 4d97e5a386 improve infinite recursion errors 2023-12-16 10:52:23 +01:00
Jörg Thalheim 83df9d4e24 fix exit status reporting when evaluation fails 2023-12-16 09:48:37 +01:00
Jörg Thalheim 89927c434c make sure we also define HAVE_STRUCT_DIRENT_D_TYPE outside of autotools 2023-12-16 09:24:42 +01:00
Jörg Thalheim 8daea338f3 flake.lock: Update 
Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/0838486e9183ce39398f01b221766e68b50f405c' (2023-12-10)
  → 'github:NixOS/nixpkgs/203ecda835bcf69633df7183459283543dd4a874' (2023-12-15)
• Updated input 'treefmt-nix':
    'github:numtide/treefmt-nix/afdd5e48a0869b389027307652a658051c0d2f96' (2023-12-10)
  → 'github:numtide/treefmt-nix/d06b70e5163a903f19009c3f97770014787a080f' (2023-12-13)
 2023-12-16 09:20:53 +01:00
Jörg Thalheim 4b6d34214d add include-what-you-use to devshell 2023-12-16 09:20:53 +01:00
Jörg Thalheim 1f4bbded61 optimize imports with "include-what-you-use" 2023-12-16 09:20:53 +01:00
Jörg Thalheimandmergify[bot] 521380076d release proc.pid properly before calling waitpid() 2023-12-10 20:36:24 +00:00
Jörg Thalheimandmergify[bot] e1ad62cef1 include orignal json in worker error message 2023-12-10 20:36:24 +00:00
Jörg Thalheimandmergify[bot] d48cfadb3d fix catching eval errors on macOS 2023-12-10 20:36:24 +00:00
Jörg Thalheimandmergify[bot] c00fcbba8d drop obsolete compiler warning suppression 2023-12-10 20:36:24 +00:00
Jörg Thalheimandmergify[bot] b6ec7d2ecf make MyArgs non-copyable 
this seems to have weird effects
 2023-12-10 20:36:24 +00:00
Jörg Thalheimandmergify[bot] b24c03e2de simplify collector function 
looks like the lambda doesn't buy us anything here.
 2023-12-10 20:36:24 +00:00
Jörg Thalheim b267eb917d apply treefmt 2023-12-10 17:00:31 +01:00
Jörg Thalheim f970ec3524 add test for evaluation errors 2023-12-10 16:59:14 +01:00
Jörg Thalheimandmergify[bot] e7c30b306d don't fail nix-eval-jobs on eval errors 
This error handling was copied by accident in https://github.com/nix-community/nix-eval-jobs/pull/277/commits/5c764d4a67feceae1848414fd9434e79120884e6#diff-a79ded172fd76747492a417a39848b6c25c14238e65971e6a05fe81706d5048fR622
 2023-12-10 15:57:18 +00:00
Jörg Thalheimandmergify[bot] 36a5368dfc include missing filesystem header 2023-12-10 14:52:52 +00:00
Jörg Thalheimandmergify[bot] 3be8c48f54 fix format 2023-12-10 14:52:52 +00:00
Jörg Thalheimandmergify[bot] a03f039a56 split nix-eval-jobs further into smaller files 2023-12-10 14:52:52 +00:00
Jörg Thalheim 880c66a7d1 move drvs ot its own class 2023-12-10 15:07:38 +01:00
Jörg Thalheim db3099bc8f move argument parsing to new file 2023-12-10 14:22:32 +01:00
Jörg Thalheimandmergify[bot] 6f4bee53f6 nix-github-action: fix value beeing interpreted as bool 2023-12-10 10:25:15 +00:00
Jörg Thalheimandmergify[bot] f49cb87963 enable asan/ubsan in ci 2023-12-10 10:25:15 +00:00
Jörg Thalheimandmergify[bot] 00d3f014e7 print error if worker receives invalid command 2023-12-10 10:25:15 +00:00
Jörg Thalheimandmergify[bot] 93972c0c18 handle broken evaluation worker pipes on write 2023-12-10 10:25:15 +00:00
Jörg Thalheimandmergify[bot] 36483b325c treefmt: replace prettier with deno 
we don't have toml files and deno has a smaller install closure.
 2023-12-10 10:25:15 +00:00
Jörg Thalheimandmergify[bot] 5ad4e7266f handle broken evaluation worker pipes more gracefully 
writeLine will throw a SysError exception, which obfuscates out-of-memory events where the eval worker is killed by the OS.
readLine is suffering from the same problem and will be handled in a subsequent commit.
 2023-12-10 10:25:15 +00:00
Jörg Thalheimandmergify[bot] 40ad828088 update flakes to fix tests on macos 2023-12-10 08:31:27 +00:00
Jörg Thalheimandmergify[bot] fad244725f print which derivation failed to evaluate 2023-12-10 08:31:27 +00:00
Jörg Thalheim 6f9a8d2cda bump to 2.19.0 2023-12-06 09:41:01 +01:00
Jörg Thalheimandmergify[bot] d4eeecf6b2 fix commandline parsing for nix 2.19 2023-12-06 08:30:10 +00:00
Jörg Thalheimandmergify[bot] c240e61481 fix missing includes for nix 2.19 2023-12-06 08:30:10 +00:00
Jörg Thalheimandmergify[bot] 34110992a8 flake.lock: Update 
Flake lock file updates:

• Updated input 'flake-parts':
    'github:hercules-ci/flake-parts/8c9fa2545007b49a5db5f650ae91f227672c3877' (2023-11-01)
  → 'github:hercules-ci/flake-parts/34fed993f1674c8d06d58b37ce1e0fe5eebcb9f5' (2023-12-01)
• Updated input 'nix-github-actions':
    'github:nix-community/nix-github-actions/4bb5e752616262457bc7ca5882192a564c0472d2' (2023-11-03)
  → 'github:nix-community/nix-github-actions/93e39cc1a087d65bcf7a132e75a650c44dd2b734' (2023-11-28)
• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/5171694860f185961daff3b1b413dabcab421300' (2023-11-27)
  → 'github:NixOS/nixpkgs/9ed8ade77aef706a03d8cc3a5ad4f60848ac59a7' (2023-12-06)
• Updated input 'treefmt-nix':
    'github:numtide/treefmt-nix/e82f32aa7f06bbbd56d7b12186d555223dc399d1' (2023-11-12)
  → 'github:numtide/treefmt-nix/affe7fc3f5790e1d0b5ba51bcff0f7ebe465e92d' (2023-12-04)
 2023-12-06 08:30:10 +00:00
Jörg Thalheim 70786f5ff4 fix mergify for normal non-bot merges 2023-11-20 06:50:56 +01:00
Jörg Thalheim 354fd8b93d README: mention nix-fast-build and buildbot-nix 2023-11-20 06:48:59 +01:00
Jörg Thalheim 0cdb4f4c63 renovate: add dependencies label 2023-11-14 09:49:42 +01:00
Jörg Thalheim 6ac0adcbb0 update version: 2.16.0 -> 2.18.0 2023-11-12 14:17:43 +01:00
Jörg Thalheim a53aaefbcc publish to flakestry 2023-11-12 14:17:19 +01:00
Jörg Thalheim cdbfafc2e6 flake.lock: Update 
Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/0d93ec62e06faec6c52331a8a87bd5721b38ce14' (2023-11-06)
  → 'github:NixOS/nixpkgs/44cf4801c0937b76cc6f416a0b160b5d1b3286af' (2023-11-12)
• Updated input 'treefmt-nix':
    'github:numtide/treefmt-nix/5deb8dc125a9f83b65ca86cf0c8167c46593e0b1' (2023-10-27)
  → 'github:numtide/treefmt-nix/e82f32aa7f06bbbd56d7b12186d555223dc399d1' (2023-11-12)
 2023-11-12 14:11:00 +01:00
Jörg Thalheim 56f0464288 use platform instead of querySystem() 
nix-build uses "system" from the derivation rather than the derivation attributes
 2023-10-07 22:52:14 +02:00
Jörg Thalheimandmergify[bot] 15ec2c4663 print derivation name if system attribute is missing 2023-09-12 07:07:07 +00:00
Jörg Thalheimandmergify[bot] a830f68be9 drop optional bear 2023-08-22 19:43:56 +00:00
Jörg Thalheimandmergify[bot] 68c4ce65a0 treefmt: fix eval warnings 2023-08-06 16:55:20 +00:00
Jörg ThalheimandMic92 21c6125232 update mergify-config 2023-06-20 04:13:52 +00:00
Jörg Thalheim 239f8d26c9 mergify: replace bors's merge queue 2023-06-12 06:41:43 +02:00
Jörg Thalheim 3af3106877 bump version to 2.16.0 2023-06-10 14:06:29 +02:00
Jörg Thalheim c51f3f1eb2 nix actually needs c++20 now 2023-06-10 13:56:05 +02:00
Jörg Thalheim 9e8320dbd6 fix c++20 warnings 2023-06-10 13:52:28 +02:00
Jörg Thalheim cace652bff fix build on darwin 2023-06-10 13:50:23 +02:00
Jörg Thalheim 7130e84e9e fix build 2023-06-10 13:34:42 +02:00
Jörg Thalheim e9c301bcbe document the worker/memory tradeoff 2023-04-13 23:23:51 +02:00
Jörg Thalheim 3170433774 switch to nix 2_14 2023-03-11 10:25:59 +01:00
Jörg Thalheim 4983b15121 bump version 2023-03-10 09:09:44 +01:00
Jörg Thalheim 5a7dc07ad1 bump version 2023-03-10 09:08:07 +01:00
Jörg Thalheim 54a9b26431 fix build with nix 2.14 2023-03-09 22:21:59 +01:00
Jörg Thalheim 94b3a3a43d update nix version 2023-03-09 21:01:15 +01:00
Jörg Thalheim 2f518e70ec flake.lock: Update 
Flake lock file updates:

• Updated input 'flake-parts':
    'github:hercules-ci/flake-parts/dc531e3a9ce757041e1afaff8ee932725ca60002' (2023-03-01)
  → 'github:hercules-ci/flake-parts/c13d60b89adea3dc20704c045ec4d50dd964d447' (2023-03-09)
• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/75c8abce0657f9981be937f2bc7b88125d98c03f' (2023-03-09)
  → 'github:NixOS/nixpkgs/8d8f5ede919dc60a86ac37311a63092411c72e90' (2023-03-09)
 2023-03-09 20:57:30 +01:00
Jörg ThalheimandThéophane Hufschmitt f20d3726dd advertise transport encoding in http transfers to 
tl;dr: With this 1 line change I was able to get a speedup of 1.5x on 1Gbit/s
wan connections by enabling zstd compression in nginx.

Also nix already supported all common compression format for http
transfer, webservers usually only enable them if they are advertised
through the Accept-Encoding header.

This pull requests makes nix advertises content compression support for
zstd, br, gzip and deflate.

It's particular useful to add transparent compression for binary caches
that serve packages from the host nix store in particular nix-serve,
nix-serve-ng and harmonia.

I tried so far gzip, brotli and zstd, whereas only zstd was able to bring
me performance improvements for 1Gbit/s WAN connections.

The following nginx configuration was used in combination with the
[zstd module](https://github.com/tokers/zstd-nginx-module) and
[harmonia](https://github.com/nix-community/harmonia/)

```nix
{
  services.nginx.virtualHosts."cache.yourhost.com" = {
    locations."/".extraConfig = ''
      proxy_pass http://127.0.0.1:5000;
      proxy_set_header Host $host;
      proxy_redirect http:// https://;
      proxy_http_version 1.1;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;

      zstd on;
      zstd_types application/x-nix-archive;
    '';
  };
}
```

For testing I unpacked a linux kernel tarball to the nix store using
this command `nix-prefetch-url --unpack https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.1.8.tar.gz`.

Before:

```console
$ nix build && rm -rf /tmp/hello  && time ./result/bin/nix copy --no-check-sigs --from https://cache.thalheim.io --to 'file:///tmp/hello?compression=none' '/nix/store/j42mahch5f0jvfmayhzwbb88sw36fvah-linux-6.1.8.tar.gz'
warning: Git tree '/scratch/joerg/nix' is dirty

real    0m18,375s
user    0m2,889s
sys     0m1,558s
```

After:

```console
$ nix build && rm -rf /tmp/hello  && time ./result/bin/nix copy --no-check-sigs --from https://cache.thalheim.io --to 'file:///tmp/hello?compression=none' '/nix/store/j42mahch5f0jvfmayhzwb
b88sw36fvah-linux-6.1.8.tar.gz'

real    0m11,884s
user    0m4,130s
sys     0m1,439s
```

Signed-off-by: Jörg Thalheim <joerg@thalheim.io>

Update src/libstore/filetransfer.cc

Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
 2023-02-03 12:33:38 +00:00
Jörg Thalheim cdd53c2782 bump nix-eval-jobs version 2023-01-22 11:19:24 +01:00
Jörg Thalheim 205f05ac02 fix build with 2.13 2023-01-22 11:19:24 +01:00
Jörg Thalheim 396a0d3045 bump nixpkgs and set version to 2_13 2023-01-22 11:19:24 +01:00
Jörg Thalheim 9d516a72d7 fix double-free caused by usage of shared pointer 2023-01-22 10:37:17 +01:00
Jörg Thalheim a42845708e add mergify rules for garnix 2023-01-02 11:51:50 +01:00
Jörg Thalheim be12f80cfa bors: include garnix in tests 2023-01-02 11:51:50 +01:00
Jörg Thalheim 24db95cbac ci: also build github actions in staging 2023-01-02 11:42:56 +01:00
Jörg Thalheim d2c00b8c84 fix build of devShell 2023-01-02 11:39:34 +01:00
Jörg Thalheim 01369215dc bors: update toml 2023-01-02 11:35:29 +01:00
Jörg Thalheim 0d5f97d617 switch to garnix 2023-01-02 11:35:29 +01:00
Jörg Thalheim 9d1abc0bc8 switch to flake-parts from flake-utils 2023-01-02 11:24:46 +01:00
Jörg Thalheim e6392759be ci/update-flake-lock: provide access to github token 2023-01-02 11:10:09 +01:00
Jörg Thalheim cccd57c022 getMaxCPU: fix cgroup path 
Given this typo I am not sure if it has been tested.
 2022-12-04 18:22:12 +01:00
Jörg Thalheim be849ae99f create bors.toml 2022-10-31 07:22:17 +01:00
Jörg Thalheim 724c8df419 also trigger ci on release branches 2022-10-31 07:14:24 +01:00
Jörg Thalheim ad9c0e0b70 flake.nix: bump version 2022-10-31 07:09:44 +01:00
Jörg Thalheim e00761af73 Also test github flakes when access tokens are provided 2022-10-30 07:10:58 +01:00
Jörg Thalheim 48f840cff8 assign fricklerhandwerk for documentation releated PRs 2022-10-27 18:13:24 +02:00
Jörg Thalheim b445d2c1ad README: update patchelf example 2022-10-06 22:20:43 +02:00
Jörg Thalheim d9abce4ad4 libfetchers: avoid api.github.com ratelimit if no github token is set 
If we don't have any github token, we won't be able to fetch private
repos, but we are also more likely to run into API limits since
we don't have a token. To mitigate this only ever use the github api
if we actually have a token.
 2022-10-01 10:38:09 +02:00
Jörg Thalheim 47a74fed83 also accept relative gc root directories 2022-09-20 20:12:55 +02:00
Jörg Thalheim 70afaeebca avoid shadowing metaName 2022-09-17 10:48:25 +02:00
Jörg Thalheim f8fa3ea069 README: update available options 2022-09-15 11:04:12 +02:00
Jörg Thalheim cb8126538f fix recurseIntoAttrs handling 2022-09-15 11:04:12 +02:00
Jörg Thalheim f6890b93e8 add check-cache-status option to query wether a binary cache has the build 2022-09-15 11:04:12 +02:00
Jörg Thalheim 3071792e91 README: link to ci configurations 2022-07-11 13:39:04 +02:00
Jörg Thalheim f293656be1 update mergify rules 2022-05-19 08:04:54 +02:00
Jörg Thalheim e67f2f8090 README: clarify branches 2022-05-09 09:53:59 +02:00
Jörg Thalheim 882d2376cf fixes for nixUnstable 2022-05-09 09:50:48 +02:00
Jörg Thalheim 5088049295 only target single nix version in repo 2022-05-09 09:21:53 +02:00
Jörg Thalheimandadisbladis c1b86245a1 fix for nixUnstable 2022-04-22 18:53:12 +12:00
Jörg Thalheimandadisbladis 3094e12732 flake: switch back to original nixpkgs 2022-04-22 18:53:12 +12:00
Jörg Thalheim d592323c6b fix build with nix 2.7 2022-03-13 17:47:15 +01:00
Jörg Thalheim f3bb3a1dd7 README: mention projects using nix-eval-jobs 2022-02-16 18:57:35 +01:00
Jörg Thalheim aa81824125 ci/update-flake-lock: switch back to upstream 2022-02-04 16:37:45 +01:00