Raito Bezarius
95f15cf94f
libarchive's xz offers single threaded xz compression which is very slow and provides ~10-20Mbps compression speed in addition to maxing a core. In exchange, it achieves optimal compression ratios among all our compression methods. Nonetheless, xz prevent the saturation of 1Gbps+ connections and slow down significantly decompression for end users. As these connections and faster hardware is becoming prevalent for cache servers and clients, we offer to default to zstd. Lix is a "compress once, decompress many times" application. To avoid incurring a high penalty to end users very sensitive to compress ratio (very slow Internet connections), we dampen the consequences of switching to zstd by increasing the default zstd level to 12. On one example, xz will compress a 4.4GB file to 632MB, zstd on 12 will compress it to 775MB, that is a ~18 % increase over the optimal xz compression. zstd took 18 seconds to produce this file. Increasing to level 14 leads to a 773MB file while taking 37s. Increasing to level 16 leads to 735MB file while taking 66s. Finally, xz took 77s, so a 50 % reduction in time taken to compress in exchange of an increase of 18 % of the compressed size. This change will reduce issues encountered in #945 but is probably not the root cause. References: - https://discourse.nixos.org/t/switch-cache-nixos-org-to-zstd-to-fix-slow-nixos-updates-nix-downloads/23961 Change-Id: I7beda2bf2c1fed146dcb797b8f85dc290c486ab2 Signed-off-by: Raito Bezarius <raito@lix.systems>
30 lines
888 B
Bash
30 lines
888 B
Bash
source common.sh
|
|
|
|
BINARY_CACHE=file://$cacheDir
|
|
|
|
build() {
|
|
nix-build --no-out-link "$@" --expr 'derivation {
|
|
name = "text";
|
|
system = builtins.currentSystem;
|
|
builder = "/bin/sh";
|
|
args = [ "-c" "echo some text to make the nar less empty > $out" ];
|
|
}'
|
|
}
|
|
|
|
path=$(build)
|
|
nix copy --to "$BINARY_CACHE" "$path"
|
|
nix-collect-garbage >/dev/null 2>&1
|
|
|
|
nar=0c3y7p42issm0ydjilwvk0drv958p4p4d2d6c7y5ksmzmbf7rfhg.nar.zst
|
|
|
|
[ -e $cacheDir/nar/$nar ] || fail "long nar missing?"
|
|
|
|
zstdcat $cacheDir/nar/$nar > $TEST_HOME/tmp
|
|
truncate -s $(( $(stat -c %s $TEST_HOME/tmp) - 10 )) $TEST_HOME/tmp
|
|
zstd - --stdout < $TEST_HOME/tmp > $cacheDir/nar/$nar
|
|
|
|
# Copying back '$path' from the binary cache. This should fail as it is truncated
|
|
if build --option substituters "$BINARY_CACHE" --option require-sigs false -j0; then
|
|
fail "Importing a truncated nar should fail"
|
|
fi
|