MatchClosestOption maps an empty, whitespace or refusal judge reply to "Yes" (score 1.0) and, in pairwise mode, always to "Response 2" #1980

src/unitxt/processors.py:218-222

class MatchClosestOption(InstanceFieldOperator):
    options_field: str = "options"

    def process_instance_value(self, value: Any, instance: Dict[str, Any]):
        options = instance["task_data"][self.options_field]
        return get_close_matches(value, options, n=1, cutoff=0.0)[0]

processors.match_closest_option is the postprocessor on the option-selection ("answer") template of both LLMJudgeDirect and LLMJudgePairwise (llm_as_judge_chat_templates.py:39,77). With cutoff=0.0 every string matches something; when every option has a similarity ratio of 0 (empty reply, whitespace, a refusal with no shared characters) difflib breaks the tie in favour of the option that sorts last (get_close_matches ranks (ratio, option) tuples with heapq.nlargest, independent of list order). For every catalog direct criterion that is ["Yes", "No"] with option_map={"Yes": 1.0, "No": 0.0} (llm_as_judge_constants.py:385,437,458,536,551,572,660) that is "Yes", and llm_as_judge.py:392 then does criteria.option_map[selection] with no unparseable branch. For pairwise contests the options are ["Response 1", "Response 2"], so the tie always resolves to "Response 2", which llm_as_judge.py:1592 turns into "2" and counts as a win for the second system.

Measured

process_instance_value executed from the actual source at HEAD (1.26.10) with instance={"task_data": {"options": [...]}}:

Swapping the option order (['No','Yes'], ['Response 2','Response 1']) still returns Yes / Response 2 — the tie-break is lexicographic, not positional.

Consequence

• A direct judge whose completion is empty, whitespace, a refusal, or the inference engine's "-" error placeholder scores the evaluated response as a full pass (1.0) on every Yes/No criterion in the catalog. A judge that failed on every row reports a perfect metric.
• A pairwise judge with the same failures awards the contest to the second-listed response every time. Because the tie-break ignores option order, the positional-bias re-run (llm_as_judge.py:572-580, options reversed) yields the same selection, so positional_bias reports False while *_winrate is skewed toward whichever system was listed second.
• Nothing in results distinguishes these rows from genuine selections; selected_option reads "Yes" / "Response 2".
• The existing test (tests/library/test_postprocessors.py:158-199) only feeds near-exact strings (option1, choice, cheese, 2), so none of the above is pinned.

Suggested fix

Follow the precedent already in the same file: ExtractHarmRatingJudgement (processors.py:323-329) returns np.NaN when the judge reply does not parse, so the row is excluded rather than scored. Concretely, use a non-zero cutoff (or check SequenceMatcher(None, value, opt).ratio() > 0 for the winner) and return None/np.NaN when nothing matches; then in llm_as_judge.py:391-394 map a None selection to np.nan (and skip it in the pairwise contest accounting at :1592 and :810) instead of indexing option_map. The stray module-level process_instance_value at processors.py:226-229 already has the closest_match[0] if closest_match else None shape and can replace the class body once it is bound. Add the empty / whitespace / refusal / "-" cases to test_match_closest_option so the contract is pinned.

Adjacent, not duplicates: #679 introduced the processor; #1409 wired it into the judge templates; open #1960 touches ExtractMtBenchRatingJudgment only.

Happy to open the PR.