lix/mk/run-test.sh at 4f3a4b732cdc8a0179b0da9cccafef0b215a4e86

guiwf/lix

Files

T

John EricsonandRobert Hensing c11836126b Harden tests' bash

Use `set -u` and `set -o pipefail` to catch accidental mistakes and
failures more strongly.

 - `set -u` catches the use of undefined variables
 - `set -o pipefail` catches failures (like `set -e`) earlier in the
   pipeline.

This makes the tests a bit more robust. It is nice to read code not
worrying about these spurious success paths (via uncaught) errors
undermining the tests. Indeed, I caught some bugs doing this.

There are a few tests where we run a command that should fail, and then
search its output to make sure the failure message is one that we
expect. Before, since the `grep` was the last command in the pipeline
the exit code of those failing programs was silently ignored. Now with
`set -o pipefail` it won't be, and we have to do something so the
expected failure doesn't accidentally fail the test.

To do that we use `expect` and a new `expectStderr` to check for the
exact failing exit code. See the comments on each for why.

`grep -q` is replaced with `grepQuiet`, see the comments on that
function for why.

`grep -v` when we just want the exit code is replaced with `grepInverse,
see the comments on that function for why.

`grep -q -v` together is, surprise surprise, replaced with
`grepQuietInverse`, which is both combined.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>

2023-03-08 10:26:30 -05:00

1.1 KiB

Executable File

Raw Blame History

#!/usr/bin/env bash

set -eu -o pipefail

red=""
green=""
yellow=""
normal=""

test=$1

dir="$(dirname "${BASH_SOURCE[0]}")"
source "$dir/common-test.sh"

post_run_msg="ran test $test..."
if [ -t 1 ]; then
red=""
green=""
yellow=""
normal=""
fi

run_test () {
(init_test 2>/dev/null > /dev/null)
log="$(run_test_proper 2>&1)" && status=0 || status=$?
}

run_test

# Hack: Retry the test if it fails with “unexpected EOF reading a line” as these
# appear randomly without anyone knowing why.
# See https://github.com/NixOS/nix/issues/3605 for more info
if [[ $status -ne 0 && $status -ne 99 && \
"$(uname)" == "Darwin" && \
"$log" =~ "unexpected EOF reading a line" \
]]; then
echo "$post_run_msg [${yellow}FAIL$normal] (possibly flaky, so will be retried)"
echo "$log" | sed 's/^/ /'
run_test
fi

if [ $status -eq 0 ]; then
echo "$post_run_msg [${green}PASS$normal]"
elif [ $status -eq 99 ]; then
echo "$post_run_msg [${yellow}SKIP$normal]"
else
echo "$post_run_msg [${red}FAIL$normal]"
echo "$log" | sed 's/^/ /'
exit "$status"
fi

1.1 KiB Executable File Raw Blame History

1.1 KiB

Executable File

Raw Blame History