[FR]: EvalPort import/export for Datasets and Experiments

[adhabnr-ux]

Proposal summary

Add to_openeval() / from_openeval() support for Opik's Dataset/Experiment objects, converting to and from EvalPort — an open interchange format (Apache 2.0) for portable LLM evaluation datasets (test cases, graders, eval suites, result sets as plain JSON). It's already integrated with UK AISI's Inspect AI (PR approved) and has standalone adapter packages for AutoGen, CrewAI, Ragas, LangSmith, Braintrust, and MLflow — Opik would join that list.

from opik import Opik
from opik_openeval_adapter import to_openeval, from_openeval

client = Opik()
dataset = client.get_dataset("my-dataset")
experiment_items = client.get_experiment_by_name("my-experiment").get_items()

# Export a dataset + its experiment results to an EvalPort suite
suite = to_openeval(dataset, experiment_items=experiment_items)

from openeval.validate import validate_suite
assert validate_suite(suite).valid

# ...and the other direction: load an EvalPort suite as a fresh Opik dataset
items = from_openeval(suite)
dataset = client.create_dataset(name="from-evalport")
dataset.insert(items)

Each Opik metric attached to an experiment item (Hallucination, AnswerRelevance, ContextPrecision, heuristic metrics, or a custom metric) would map to its own EvalPort grader, and the already-computed scores would be preserved per test case — so an experiment run exported to EvalPort keeps its results, not just its inputs.

Motivation

What problem are you trying to solve? Moving a dataset (and its already-computed experiment results) between Opik and another eval tool today means hand-translating it — there's no shared format. That matters when a team wants to compare Opik's LLM-as-a-judge metrics against another framework's graders on the exact same test cases, or wants to reuse an Opik-built dataset in a tool that doesn't speak Opik's format, or vice versa.

How are you currently solving this problem? Manually re-authoring the dataset per tool, or writing one-off glue scripts per pair of tools — which doesn't scale past two or three frameworks.

What are the benefits of this feature? Opik already integrates with more frameworks than almost anyone in this space (the integrations table in the README is enormous) — an EvalPort adapter extends that same "framework-agnostic" positioning to the evaluation data itself, not just tracing. Teams evaluating across Opik plus one of the other EvalPort-speaking tools (Inspect AI, AutoGen, CrewAI, Ragas, LangSmith, Braintrust, MLflow) get a shared dataset/result format for free instead of custom glue per pair.

Happy to build this as a standalone opik-openeval-adapter package if that's the better fit, or contribute it in-repo if maintainers would rather it live alongside the other integrations — open to discussing the mapping (especially how to round-trip Opik's feedback-score definitions) before writing code.

[github-actions]

I have enough to write a comprehensive response now.

Hi, I'm Scout 🦉, the comet-ml/opik repository agent. Let me look into this.

Solution / Workaround

No existing EvalPort/openeval adapter or equivalent interchange format integration exists in the codebase. However, the existing SDK already exposes all the building blocks needed to write this as a standalone external adapter package (opik-openeval-adapter) today, without any core changes required:

# Existing API surface the adapter can already use today

# Export a dataset
dataset = client.get_dataset("my-dataset")
items = dataset.get_items()          # List[Dict[str, Any]]
json_str = dataset.to_json()         # raw JSON string
df = dataset.to_pandas()             # DataFrame

# Export experiment items (via experiments_client)
experiment = client.get_experiment_by_name("my-experiment")
# experiment items carry: dataset_item_data, evaluation_task_output, feedback_scores

# Import into a dataset
dataset.insert(items)                # items is List[Dict[str, Any]]

An external package only needs to map between EvalPort's JSON schema and these dicts.

Code Investigation

The relevant SDK files are:

Dataset layer:

• sdks/python/src/opik/api_objects/dataset/dataset.py — DatasetExportOperations abstract base class exposes to_pandas(), to_json(), and get_items() (returns List[Dict[str, Any]]). Dataset.insert() accepts List[Dict[str, Any]] directly.
• sdks/python/src/opik/api_objects/dataset/dataset_item.py — DatasetItem Pydantic model: core fields are id, trace_id, span_id, source, description, evaluators, execution_policy; arbitrary extra fields are stored as the "data" payload via model_extra.
• sdks/python/src/opik/api_objects/dataset/converters.py — to_json(), from_json(), to_pandas(), from_pandas() converter functions. The EvalPort adapter could follow this same pattern.

Experiment layer:

• sdks/python/src/opik/api_objects/experiment/experiment_item.py — ExperimentItemContent dataclass: carries dataset_item_data, evaluation_task_output, and feedback_scores: List[FeedbackScoreDict]. This is the full "result set" that would map to EvalPort grader results.
• sdks/python/src/opik/types.py — FeedbackScoreDict TypedDict: fields name, value, category_name, reason. These would map to EvalPort grader scores per test case.
• sdks/python/src/opik/api_objects/experiment/experiment.py — Experiment class; batch_upload_items() and insert() are the write paths.

Integrations pattern:

• sdks/python/src/opik/integrations/ — all third-party integrations (langchain, crewai, openai, etc.) live here as subpackages. A first-party EvalPort adapter would fit in integrations/evalport/.

Key data mapping for a round-trip adapter:

Opik concept	EvalPort concept
DatasetItem.model_extra (the "data" dict)	Test case inputs + expected_output
ExperimentItemContent.evaluation_task_output	Test case output / result
FeedbackScoreDict (name, value, reason)	Grader result per test case
DatasetItem.evaluators	EvalPort grader definitions

No test coverage exists for a hypothetical converter, as the feature doesn't exist yet. The existing converters.py tests in sdks/python/tests/ would be the natural place to add them.

Next Steps

There are two paths, with different scope implications:

Option A (recommended starting point): Standalone external package
The author's proposal of opik-openeval-adapter as a third-party package is the natural fit — it requires zero changes to the Opik core SDK. It would depend on opik and openeval, implementing the mapping described above. This is also how the Ragas, LangSmith, and Braintrust EvalPort adapters were structured externally.

Option B: First-party integration in-repo
If maintainers want this in-repo (alongside integrations/langchain, integrations/crewai, etc.), the new module would live at sdks/python/src/opik/integrations/evalport/ with two public functions:

• to_openeval(dataset, experiment_items=None) -> dict — streams DatasetItem objects + ExperimentItemContent objects and serializes to an EvalPort suite dict.
• from_openeval(suite: dict) -> List[DatasetItem] — parses the EvalPort suite and constructs DatasetItem instances.

Key design question before committing to Option B: EvalPort is a relatively new project (the reporter is the author and appears to be a primary contributor). Maintainers should assess whether EvalPort has enough independent adoption to justify a first-party integration, or whether an external adapter package is the right boundary to keep for now — similar to how Opik doesn't bundle LangSmith or Braintrust code directly. Starting with Option A keeps the door open for Option B later without any API commitments.

If the team decides Option B is appropriate, this would be a good good first issue — the converter logic is straightforward given the existing converters.py pattern, and there are no breaking API changes required.

Was this helpful? React to this comment with 👍 or 👎 to rate my response.

---

View Scout trace in Opik

[adhabnr-ux]

Thanks for the thorough investigation, Scout — that's a genuinely useful breakdown of the SDK surface, and the DatasetItem/ExperimentItemContent/FeedbackScoreDict mapping is exactly right.

Going with Option A: I built opik-openeval-adapter as a standalone external package (same playbook as the AutoGen and CrewAI adapters), so it's zero footprint on opik core and no API commitment needed from the team. It's live and tested against the real opik SDK classes (20 tests, all passing against real DatasetItem/ExperimentItemContent, not fakes):

• Package: https://github.com/adhabnr-ux/evalport/tree/main/adapters/opik-openeval-adapter
• to_openeval()/from_openeval() for Dataset items ↔ an EvalPort suite
• experiment_to_openeval() for Experiment items (task output + FeedbackScoreDicts) → an EvalPort ResultSet

Both validate against the real openeval spec validator, and the input/expected-output field detection is heuristic (with explicit overrides) since Opik dataset items are schema-less by design — full mapping details are in the README.

If it's useful, happy to have this linked from Opik's own docs/integrations page, or to adapt the mapping if maintainers want to pursue Option B later — either way, there's now working code to look at rather than just a proposal.