Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions guides/agent-skills.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,8 @@ or a checkpoint UUID from a previous training job.
Response: { "id": "uuid-of-training-job", "status": "requested" }

Job status values: requested | running | complete | failed | stopped
Metrics (on COMPLETED): { "f1": 0.94, "precision": 0.96, "recall": 0.92 }
Metrics (on COMPLETED): training loss only — { "final_training_loss": …, "final_validation_loss": …, "best_validation_loss": … }
After an evaluation runs, eval_f1_score / eval_precision / eval_recall are merged into the same metrics field.

## Evaluations

Expand All @@ -199,7 +200,7 @@ curl -X POST https://api.pioneer.ai/felix/evaluations \
}'
```

Results include: f1, precision, recall, per_entity breakdown
Results include: f1_score, precision_score, recall_score, accuracy

## Errors

Expand Down
12 changes: 6 additions & 6 deletions guides/fine-tune-classification.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -137,21 +137,21 @@ Text classification assigns one or more labels to a piece of text — sentiment,

Job status values: `requested` → `running` → `complete` (or `failed` / `stopped`).

When the job reaches `"complete"`, the response includes evaluation metrics:
When the job reaches `"complete"`, the response includes training loss metrics:

```json
{
"id": "YOUR_JOB_ID",
"status": "complete",
"metrics": {
"f1": 0.92,
"precision": 0.94,
"recall": 0.90
"final_training_loss": 0.18,
"final_validation_loss": 0.21,
"best_validation_loss": 0.20
}
}
```

A high F1 score (above 0.85) generally indicates a model ready for production. If scores are lower, consider adding more training examples — especially for any minority classes — or making your label definitions more distinct.
A low validation loss is a good first signal, but the headline F1 / precision / recall numbers come from running an evaluation on a held-out dataset (next step). Once you've run one, those scores are merged into the same `metrics` field as `eval_f1_score`, `eval_precision`, and `eval_recall`.
</Step>
<Step title="Run an evaluation">
Evaluate your trained model against a held-out dataset to get a more rigorous view of performance before deploying.
Expand All @@ -170,7 +170,7 @@ Text classification assigns one or more labels to a piece of text — sentiment,

</CodeGroup>

Retrieve evaluation results with `GET /felix/evaluations/:id`. Results include `f1`, `precision`, `recall`, and a per-label breakdown so you can see which classes need more training data.
Retrieve evaluation results with `GET /felix/evaluations/:id`. Results include `f1_score`, `precision_score`, `recall_score`, and `accuracy`. A high F1 score (above 0.85) generally indicates a model ready for production. If scores are lower, consider adding more training examples — especially for any minority classes — or making your label definitions more distinct.
</Step>
<Step title="Run inference with your trained model">
Use your job ID as the `model_id` to run predictions. Classification lives under the `classifications` key of the `schema` field — each entry defines one independent classification head.
Expand Down
12 changes: 6 additions & 6 deletions guides/fine-tune-extraction.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -138,21 +138,21 @@ Structured extraction pulls JSON-shaped records — invoices, contracts, product

Job status values: `requested` → `running` → `complete` (or `failed` / `stopped`).

When the job reaches `"complete"`, the response includes evaluation metrics:
When the job reaches `"complete"`, the response includes training loss metrics:

```json
{
"id": "YOUR_JOB_ID",
"status": "complete",
"metrics": {
"f1": 0.91,
"precision": 0.93,
"recall": 0.89
"final_training_loss": 0.22,
"final_validation_loss": 0.27,
"best_validation_loss": 0.25
}
}
```

For extraction, the metrics are computed per field across all structure instances. A high F1 score (above 0.85) generally indicates a model ready for production. If a particular field is dragging the score down, the most common fix is adding more training examples that contain that field — especially examples where the value is phrased differently from what you've already labelled.
A low validation loss is a good first signal, but the headline F1 / precision / recall numbers come from running an evaluation on a held-out dataset (next step). Once you've run one, those scores are merged into the same `metrics` field as `eval_f1_score`, `eval_precision`, and `eval_recall`.
</Step>
<Step title="Run an evaluation">
Evaluate your trained model against a held-out dataset for a more rigorous read on performance before deploying.
Expand All @@ -171,7 +171,7 @@ Structured extraction pulls JSON-shaped records — invoices, contracts, product

</CodeGroup>

Retrieve evaluation results with `GET /felix/evaluations/:id`. Results include `f1`, `precision`, `recall`, and a per-field breakdown so you can see which fields are accurate and which need more training data.
Retrieve evaluation results with `GET /felix/evaluations/:id`. Results include `f1_score`, `precision_score`, `recall_score`, and `accuracy`, computed per field across all structure instances. A high F1 score (above 0.85) generally indicates a model ready for production. If a particular field is dragging the score down, the most common fix is adding more training examples that contain that field — especially examples where the value is phrased differently from what you've already labelled.
</Step>
<Step title="Run inference with your trained model">
Use your job ID as the `model_id` to run predictions. Extraction lives under the `structures` key of the `schema` field — at inference time you describe each structure with a list of typed fields, and the model returns the values it extracts from the text.
Expand Down
12 changes: 6 additions & 6 deletions guides/fine-tune-ner.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -108,21 +108,21 @@ Named Entity Recognition (NER) lets you extract structured information — peopl

Job status values: `requested` → `running` → `complete` (or `failed` / `stopped`).

When the job reaches `"complete"`, the response includes evaluation metrics:
When the job reaches `"complete"`, the response includes training loss metrics:

```json
{
"id": "YOUR_JOB_ID",
"status": "complete",
"metrics": {
"f1": 0.94,
"precision": 0.96,
"recall": 0.92
"final_training_loss": 0.15,
"final_validation_loss": 0.18,
"best_validation_loss": 0.17
}
}
```

A high F1 score (above 0.85) generally indicates a model ready for production. If scores are lower, consider adding more training examples or adjusting your entity label definitions.
A low validation loss is a good first signal, but the headline F1 / precision / recall numbers come from running an evaluation on a held-out dataset (next step). Once you've run one, those scores are merged into the same `metrics` field as `eval_f1_score`, `eval_precision`, and `eval_recall`.
</Step>
<Step title="Run an evaluation">
Evaluate your trained model against a held-out dataset to get a more rigorous view of performance before deploying.
Expand All @@ -141,7 +141,7 @@ Named Entity Recognition (NER) lets you extract structured information — peopl

</CodeGroup>

Retrieve evaluation results with `GET /felix/evaluations/:id`. Results include `f1`, `precision`, `recall`, and a `per_entity` breakdown so you can see which entity types need more training data.
Retrieve evaluation results with `GET /felix/evaluations/:id`. Results include `f1_score`, `precision_score`, `recall_score`, and `accuracy`. A high F1 score (above 0.85) generally indicates a model ready for production. If scores are lower, consider adding more training examples or adjusting your entity label definitions.
</Step>
<Step title="Run inference with your trained model">
Use your job ID as the `model_id` to run predictions. The `schema` field controls what Pioneer extracts.
Expand Down