Workspace-level feedback-score metrics never include online-scoring (authored) scores — WorkspaceMetricsDAO reads only legacy feedback_scores

[macblackstuff]

Component: Opik Server

Opik version: 2.2.13 (self-hosted, docker-compose deployment); also verified present at current HEAD (2.2.14-6215, commit d6c2776)

Describe the problem

Feedback scores written by online scoring (automation rule LLM-as-judge evaluators) never appear in workspace-level feedback-score metrics — summary and daily time-series both return empty, so score trends cannot be charted at workspace scope, even though the scores exist and are visible per-trace/per-span.

Root cause (from source at tag 2.2.13):

• The score write path is author-aware: FeedbackScoreDAOImpl.BULK_INSERT_FEEDBACK_SCORE (apps/opik-backend/.../FeedbackScoreDAO.java:81-120) renders INSERT INTO <if(author)>authored_feedback_scores<else>feedback_scores<endif>(...). The online-scoring consumer always sets an author (OnlineScoringBaseScorer.java:121-130), so all online-scoring rows land in authored_feedback_scores.
• Project-level metrics correctly UNION ALL both tables (ProjectMetricsDAO.java:164-206, SpanMetricsQueries.java:20-149) — these return the data fine.
• Workspace-level metrics read only the legacy table: WorkspaceMetricsDAOImpl.GET_FEEDBACK_SCORES_SUMMARY (WorkspaceMetricsDAO.java:78-98) and GET_FEEDBACK_SCORES_DAILY / GET_FEEDBACK_SCORES_DAILY_BY_PROJECT (WorkspaceMetricsDAO.java:115-185) use FROM feedback_scores fs final with no authored_feedback_scores branch.
• No materialized view or migration folds authored_feedback_scores → feedback_scores (checked the analytics Liquibase changelogs through 000090), so on a deployment where all scores are online-scoring-authored, feedback_scores has 0 rows and workspace score metrics are permanently empty.

Diffing FeedbackScoreDAO.java, WorkspaceMetricsDAO.java, and ProjectMetricsDAO.java between 2.2.13 and current HEAD shows no change — the gap is still present upstream.

Reproduction steps

1. Self-host Opik 2.2.13. Create an automation rule (LLM-as-judge, span scope) on a project; let it score some spans (source = online_scoring rows appear in ClickHouse opik.authored_feedback_scores; opik.feedback_scores stays at 0 rows).
2. POST /v1/private/projects/{projectId}/metrics with {"metric_type":"SPAN_FEEDBACK_SCORES","interval":"DAILY",...} → returns data (unioned read).
3. Call the workspace feedback-score summary/daily endpoints (WorkspacesResource) for the same window → empty for every score name.

Expected: workspace-level feedback-score metrics include authored (online-scoring) scores, consistent with the project-level queries.

Suggested fix: mirror the UNION ALL feedback_scores / authored_feedback_scores + dedup pattern already used in ProjectMetricsDAO/SpanMetricsQueries in the three WorkspaceMetricsDAO queries.

Error logs or stack trace

None — no errors anywhere; the queries simply read a table the write path no longer populates.

[github-actions]

I now have a thorough understanding of the bug. The issue is clearly confirmed and the fix pattern is well-established in the codebase. Let me write the response.

Hi, I'm Scout 🦉, the comet-ml/opik repository agent. Let me look into this.

Solution / Workaround

No existing solution or workaround found. The bug is confirmed at current HEAD. There is no migration that merges authored_feedback_scores into feedback_scores, and no alternative API endpoint that provides workspace-scope feedback score metrics from both tables.

Code Investigation

The reporter's analysis is accurate and fully confirmed. Here is the precise breakdown:

Write path — apps/opik-backend/src/main/java/com/comet/opik/domain/FeedbackScoreDAO.java

FeedbackScoreDAOImpl.BULK_INSERT_FEEDBACK_SCORE (lines ~81–120) routes inserts based on the presence of an author:

INSERT INTO <if(author)>authored_feedback_scores<else>feedback_scores<endif>(...)

The online-scoring consumer always sets an author, so all online-scoring rows land exclusively in authored_feedback_scores. The feedback_scores table is only written to by manual/SDK-authored scores.

---

Project-level metrics (correct) — apps/opik-backend/src/main/java/com/comet/opik/domain/ProjectMetricsDAO.java and SpanMetricsQueries.java

ProjectMetricsDAOImpl.TRACE_FILTERED_PREFIX and SpanMetricsQueries.SPAN_FILTERED_PREFIX_TEMPLATE both use a feedback_scores_deduped CTE that UNION ALLs both tables:

WITH feedback_scores_deduped AS (
    SELECT ... FROM feedback_scores WHERE entity_type = 'trace' AND ...
    UNION ALL
    SELECT ... FROM authored_feedback_scores WHERE entity_type = 'trace' AND ...
)
ORDER BY last_updated_at DESC
LIMIT 1 BY workspace_id, project_id, entity_id, name, author, source_queue_id

Project-level queries therefore correctly include online-scoring scores.

---

Workspace-level metrics (broken) — apps/opik-backend/src/main/java/com/comet/opik/domain/WorkspaceMetricsDAO.java

All three workspace feedback score query constants read only the legacy table:

• GET_FEEDBACK_SCORES_SUMMARY (~line 78): FROM feedback_scores fs final JOIN (SELECT id FROM traces final ...) t ON t.id = fs.entity_id
• GET_FEEDBACK_SCORES_DAILY_BY_PROJECT (~line 115): FROM feedback_scores fs final JOIN (SELECT id, start_time FROM traces final ...) t ON t.id = fs.entity_id
• GET_FEEDBACK_SCORES_DAILY (~line 148): FROM feedback_scores fs final JOIN (SELECT id, start_time FROM traces final ...) t ON t.id = fs.entity_id

None reference authored_feedback_scores. In deployments where all scores originate from online scoring, feedback_scores has 0 trace-type rows and workspace metrics return permanently empty results.

Test coverage gap: There is no WorkspaceMetricsDAOTest.java in apps/opik-backend/src/test/java/com/comet/opik/domain/ and no end-to-end test covering the workspace metrics + authored scores path.

Next Steps

The fix is a targeted SQL change in WorkspaceMetricsDAO.java. Each of the three query strings needs to replace the direct FROM feedback_scores fs final scan with a deduped UNION CTE, mirroring the pattern from TRACE_FILTERED_PREFIX in ProjectMetricsDAO.java.

Sketch for GET_FEEDBACK_SCORES_SUMMARY:

Replace:

FROM feedback_scores fs final
JOIN (SELECT id FROM traces final WHERE ...) t ON t.id = fs.entity_id
WHERE workspace_id = :workspace_id ...
  AND entity_type = 'trace'

With a CTE preamble:

WITH fs_union AS (
    SELECT workspace_id, project_id, entity_id, name, value, last_updated_at
    FROM feedback_scores
    WHERE entity_type = 'trace'
      AND workspace_id = :workspace_id
      <if(project_ids)> AND project_id IN :project_ids <endif>
    UNION ALL
    SELECT workspace_id, project_id, entity_id, name, value, last_updated_at
    FROM authored_feedback_scores
    WHERE entity_type = 'trace'
      AND workspace_id = :workspace_id
      <if(project_ids)> AND project_id IN :project_ids <endif>
),
fs_deduped AS (
    SELECT entity_id, name, avg(value) AS value
    FROM fs_union
    GROUP BY entity_id, name
)
SELECT
    AVGIf(fs.value, t.id >= :id_start AND t.id <= :id_end) AS current,
    AVGIf(fs.value, t.id >= :id_prior_start AND t.id < :id_start) AS previous,
    fs.name
FROM fs_deduped fs
JOIN (SELECT id FROM traces final WHERE workspace_id = :workspace_id ...) t ON t.id = fs.entity_id
GROUP BY fs.name;

Apply the same CTE substitution to GET_FEEDBACK_SCORES_DAILY and GET_FEEDBACK_SCORES_DAILY_BY_PROJECT. The exact dedup semantics (using last_updated_at DESC, LIMIT 1 BY) should mirror the project-level feedback_scores_deduped CTE exactly, to ensure consistent tie-breaking between the two tables.

A test covering WorkspaceMetricsDAOImpl with scores in authored_feedback_scores only (and none in feedback_scores) should be added to prevent regression.

Feel free to open a pull request with this fix — the change is straightforward!

Was this helpful? React to this comment with 👍 or 👎 to rate my response.

---

View Scout trace in Opik