Skip to content

docs: say examples support offline evaluation only - #795

Closed
fuergaosi233 wants to merge 1 commit into
sgl-project:mainfrom
fuergaosi233:docs/examples-offline-eval-wording
Closed

docs: say examples support offline evaluation only#795
fuergaosi233 wants to merge 1 commit into
sgl-project:mainfrom
fuergaosi233:docs/examples-offline-eval-wording

Conversation

@fuergaosi233

Copy link
Copy Markdown

Motivation

examples/README.md on main contradicts itself about online evaluation.

Same file, same page:

  1. Supported-looking claim (now fixed):

    Optional config sections provide online/offline evaluation with <run_id>-best selection

  2. Contradictory claim (left as-is; matches code):

    Online evaluation is also unsupported.

Code on main (specforge/config/schema.py) rejects online eval:

if self.data.eval_data_path:
    raise ValueError(
        "data.eval_data_path is unsupported; online evaluation is not "
        "supported by the server-only capture path"
    )
...
if self.data.eval_hidden_states_path and mode != "offline":
    raise ValueError(
        "data.eval_hidden_states_path requires an offline training data source"
    )

Tests: test_online_evaluation_is_explicitly_unsupported and
test_offline_eval_source_requires_offline_training in
tests/test_config/test_schema.py.

Modifications

One wording change in examples/README.md: online/offline evaluationoffline evaluation.

The earlier "Online evaluation is also unsupported." sentence is unchanged.

Related Issues

N/A

Accuracy Test

N/A (docs only)

Benchmark & Profiling

N/A (docs only)

Checklist

  • Format your code according to the Code Formatting with Pre-Commit.
  • Add unit tests as outlined in the Running Unit Tests.
  • Update documentation / docstrings / example tutorials as needed.
  • Provide throughput / latency benchmark results and accuracy evaluation results as needed.
  • For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
  • Please feel free to join our Slack channel to discuss your PR.

examples/README.md claimed optional configs provide "online/offline evaluation"
while the same page and schema reject online eval.

Signed-off-by: holegots <fuergaosi@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant