Problem
make test always exits 0, even when tests fail. This makes it unsuitable for CI pipelines that rely on exit codes to detect failures.
Root cause
The test target in base.mk:
.IGNORE: installcheck
...
.PHONY: test
test: testdeps install installcheck
@if [ -r $(TESTOUT)/regression.diffs ]; then cat $(TESTOUT)/regression.diffs; fi
.IGNORE: installcheck prevents make from aborting when installcheck fails — this is intentional so the recipe can display the regression diffs. But the recipe itself never exits non-zero: cat succeeds, and make reports success.
Verified locally
$ make test # with a deliberately failing test
# ... 19 "not ok" lines, regression.diffs displayed ...
$ echo $?
0
Impact
Any CI system that runs make test and checks the exit code will silently pass on test failures. We hit this in a Tekton CI pipeline — tests were broken for an unknown period because make test always reported success.
Suggested fix
Exit non-zero when regression.diffs exists:
test: testdeps install installcheck
@if [ -r $(TESTOUT)/regression.diffs ]; then cat $(TESTOUT)/regression.diffs; exit 1; fi
This preserves the current behavior of displaying diffs before exiting, but now correctly signals failure.
Problem
make testalways exits 0, even when tests fail. This makes it unsuitable for CI pipelines that rely on exit codes to detect failures.Root cause
The
testtarget inbase.mk:.IGNORE: installcheckprevents make from aborting wheninstallcheckfails — this is intentional so the recipe can display the regression diffs. But the recipe itself never exits non-zero:catsucceeds, and make reports success.Verified locally
Impact
Any CI system that runs
make testand checks the exit code will silently pass on test failures. We hit this in a Tekton CI pipeline — tests were broken for an unknown period becausemake testalways reported success.Suggested fix
Exit non-zero when
regression.diffsexists:This preserves the current behavior of displaying diffs before exiting, but now correctly signals failure.