Interop idea: a unitxt-openeval-adapter for portable eval results

[adhabnr-ux]

EvalPort (https://github.com/adhabnr-ux/evalport) is a small open interchange spec + SDK (evalport-sdk on PyPI) for portable LLM evaluation datasets and results — a JSON schema plus openeval.validate.validate_suite() / validate_result_set(), so results produced by one eval framework can be read, diffed, or re-checked by another tool without re-running anything. It's adapter-based: 34 frameworks already have a thin, zero-footprint adapters/<name>-openeval-adapter/ package that just converts to/from the interchange shape.

Why unitxt fits well: evaluate() in unitxt.api (as in your own README example) returns an EvaluationResults — a list of per-instance dicts, each carrying task_data, prediction/processed_prediction, references/processed_references, and a score dict split into instance and global. .instance_scores flattens each instance into task_data + prediction + per-instance score fields, and .global_scores gives the aggregate GlobalScores (score, score_name, num_of_instances, plus CI bounds when calc_confidence_intervals=True). That's already close to EvalPort's shape — a list of per-instance results plus one aggregate summary — it just isn't in the interchange format.

Sketch, using the real API:

from unitxt import evaluate, create_dataset
# ... build dataset, get predictions, as in the unitxt README example ...
results = evaluate(predictions=predictions, data=dataset)

# proposed: unitxt_openeval_adapter
from unitxt_openeval_adapter import results_to_openeval
result_set = results_to_openeval(results)  # walks results.instance_scores + results.global_scores

from openeval.validate import validate_result_set
validate_result_set(result_set)  # now portable / framework-agnostic

Honest caveat: task_data is arbitrary per task, and vision/speech tasks stuff non-text payloads into it. I'd target unitxt's very common text-in/text-out tasks first and treat other modalities' task_data as opaque metadata rather than claim full fidelity there.

If this is of interest, I'm happy to build adapters/unitxt-openeval-adapter/ — zero footprint on unitxt itself, following the pyproject convention already established across the other 34 adapters (self-referencing pinned unitxt extra + test extra, per evalport Discussion #13) — and submit it as a PR to evalport, referencing this discussion.

— Sahi, independent contributor (not affiliated with IBM or the unitxt team)

[adhabnr-ux]

Following up — I went ahead and built this. unitxt-openeval-adapter is now shipped and tested end-to-end against a real installed unitxt (create_dataset + evaluate, 12 tests passing in two independent clean-venv installs):

from unitxt.api import create_dataset, evaluate
from unitxt_openeval_adapter import evaluation_to_openeval

dataset = create_dataset(task="tasks.qa.open", test_set=rows, split="test", format="formats.chat_api")
results = evaluate(predictions=predictions, data=dataset)

result_set = evaluation_to_openeval(results, suite_id="my_eval_suite", run_id="run-1")

from openeval.validate import validate_result_set
assert validate_result_set(result_set).valid

A few real things confirmed while building it (and noted in the adapter's README): every key in an instance's score["instance"] dict becomes its own GraderResult, so a multi-metric task like metrics.rouge gets one result per metric (rouge1/rouge2/rougeL/rougeLsum), not just the task's single primary score. score["instance"]["score"] is always a numeric duplicate of whichever metric score_name names, so it's excluded rather than double-counted. And evaluate()'s execution is synchronous and order-preserving, so positional test_case_ids line up cleanly with the input rows — no separate id-matching step needed (unlike a distributed framework like AWS fmeval, whose adapter does need one).

Package: https://github.com/adhabnr-ux/evalport/tree/main/adapters/unitxt-openeval-adapter

Also happy to open a PR upstream here instead if the team would rather the conversion live inside unitxt itself (e.g. under an interop/ module) rather than as a standalone package — whichever is a better fit for this repo.