fix(jest): automatic checks should not be triggered for failed tests

[mohanraj-r]

• Automatic check is triggered regardless of the test status which would result in the original test failure if any getting overwritten by a11y failures if any from automatic checks.

• Tests using the sa11y jest api would get tested twice with automatic checks - once as part of the sa11y API in the test and again as part of the automatic check

  • a11y issues from automatic checks would overwrite the a11y issues found by the API
  • If the sa11y API has been added to the test to check specific intermediate states of the DOM, enabling automatic checks could result in missed a11y issues

Workarounds

Current

• Enable automatic checks in a parallel CI workflow and not in the main CI workflow.

Future

• With the default Jasmine test runner in Jest v26 there doesn't seem to be clean way to detect test failures from Jest's teardown hooks
• But jest-circus offers event handlers which allows binding to test failure events among others
  • jest-circus is the default test runner starting Jest v27. After Jest v27 gains more ground and known issues (fix(jest): codecov, automatic check tests to work with Jest v27 #65) are fixed this could be a viable option
• But even if the test failure status can be detected reliably, the question then is if the automatic checks can be triggered from within the test context - from within a CustomEnvironment that extends JSDOM env e.g. where the event handler is bound
  • With custom reporter / results processor e.g., accessing DOM is not possible and the test context is no longer available (Jest: Custom reporter vs results processor #64).