Skip to content

make test exits 0 even when tests fail #35

@jnasbyupgrade

Description

@jnasbyupgrade

Problem

make test always exits 0, even when tests fail. This makes it unsuitable for CI pipelines that rely on exit codes to detect failures.

Root cause

The test target in base.mk:

.IGNORE: installcheck
...
.PHONY: test
test: testdeps install installcheck
	@if [ -r $(TESTOUT)/regression.diffs ]; then cat $(TESTOUT)/regression.diffs; fi

.IGNORE: installcheck prevents make from aborting when installcheck fails — this is intentional so the recipe can display the regression diffs. But the recipe itself never exits non-zero: cat succeeds, and make reports success.

Verified locally

$ make test   # with a deliberately failing test
# ... 19 "not ok" lines, regression.diffs displayed ...
$ echo $?
0

Impact

Any CI system that runs make test and checks the exit code will silently pass on test failures. We hit this in a Tekton CI pipeline — tests were broken for an unknown period because make test always reported success.

Suggested fix

Exit non-zero when regression.diffs exists:

test: testdeps install installcheck
	@if [ -r $(TESTOUT)/regression.diffs ]; then cat $(TESTOUT)/regression.diffs; exit 1; fi

This preserves the current behavior of displaying diffs before exiting, but now correctly signals failure.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions