HARNESS-RETRY override feature request.

[dkechag]

Currently, if I have thousands of test files and I only want to allow retry on a couple of them, I have to add HARNESS-NO-RETRY on all the others and start yath with a global retry.
I would like to have a harness directive that does the opposite, call it FORCE-RETRY or ALWAYS-RETRY or OVERRIDE-RETRY or something like that, so I only need to add it to a specific test that is flaky while I work on stabilizing it.
You told me after your talk it could be added to the new Yath update ;)
Thanks!

[kfix]

FWIW, I implemented this for one of my suites with a custom --finder plugin.

Would be cool to have a stronger annotation that makes this plugin unnecessary.

It went something like this:

yath test -Dlib --retry=2 --finder=+MyCoolProj::Yath::Finder stabilized_stuff/ t/

lib/MyCoolProj/Yath/Finder.pm

package MyCoolProj::Yath::Finder 1;

use strict;
use warnings;

sub find_project_files {
    my $self = shift;
    my ($plugins, $settings, $search) = @_;

    my $tests = [];
    for my $path (@$search) {
        my @files;
        push @files, glob "${path}/*.t" if -d $path;
        push @files, $path if -f $path =~ /\.t$/;

        for my $t_file (@files) {
            my $tf = Test2::Harness::TestFile->new(file => $t_file);

            if ($path =~ qr{(^|/)stabilized_stuff($|/)}) {
                # don't retry stabilized tests unless their annotation explicitly requests retries
                if (! defined $tf->headers()->{'retry'}) {
                    $tf->set_retry(0);
                }
            }

            push @$tests, $tf;
        }
    }
    return $tests;
}

1;

[kfix]

it looks like yath2.0 implements this with #HARNESS-RETRY-n https://github.com/Test-More/Test2-Harness/tree/2.0?tab=readme-ov-file#harness-retry-n

[Koan-Bot]

For anyone landing here: the # HARNESS-RETRY and # HARNESS-RETRY-N directives already exist in v1 (master branch) and work independently of the global --retry flag.

You can add # HARNESS-RETRY-2 to specific test files to enable retry for only those tests, without needing --retry globally and without adding # HARNESS-NO-RETRY to all other files.

The per-test directive takes priority over the global setting in the fallback chain (task → run), so:

• # HARNESS-RETRY-2 in a file → that file retries up to 2 times, regardless of global --retry
• # HARNESS-RETRY-ISO → retry in isolation mode
• No directive + no global --retry → no retries (default)

This is documented in App::Yath pod under "HARNESS-RETRY" and "HARNESS-RETRY-N".