docs: evaluate Pioneer models with Braintrust - #43
Open
kfastino wants to merge 5 commits into
Open
Conversation
…llect Customers evaluating decoder LLMs had no documented path, even though the OpenAI-compatible API already works with these harnesses unchanged. Documents the verified commands for both, plus the gotchas that make a working setup look broken — chiefly that a low max_tokens truncates reasoning models into zero-scoring empty completions, and that GET /v1/models is public so it cannot be used to validate an API key. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
Prime Intellect competes with us in the same space, so pointing customers at their tooling from our own docs is not a trade we want to make. Drops that section and the verifiers-specific caveat, flattens the now-redundant vendor heading level, and renames the page to match its single subject. The page has not shipped yet, so the slug change needs no redirect. Co-authored-by: Cursor <cursoragent@cursor.com>
kfastino
changed the title
…ider Registering the endpoint once in Braintrust settings beats passing base_url per script: the models show up in the playground and prompt UI, and no Braintrust credential ever has to live on our side. Recommends project names as the registered model names. Pioneer resolves a project to its currently deployed model, so promoting a fine-tune retargets the entry with no reconfiguration — verified against the live API, where a request for project "All Inferences" resolved through active_model_id and served a completion. Co-authored-by: Cursor <cursoragent@cursor.com>
…ive org Our own Fastino workspace had a "Pioneer Prod" provider that never worked, and each of its three defects is an easy mistake to repeat: an endpoint URL without /v1 (Braintrust appends the route, so it 404s), streaming left off, and no model names registered at all, which makes a correct-looking provider serve nothing. Also notes the short config cache, which makes a freshly added provider 404 on the first call and look misconfigured when it is fine. Co-authored-by: Cursor <cursoragent@cursor.com>
Verified the training-job-ID row against production: one fine-tune served normally, another returned 409 "has no active inference deployment" because its deployment had been torn down. The ID resolved in both cases, so the failure reads as a bad model ID when it is really a deployment state — worth calling out since the guide recommends pinning IDs for comparison runs. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
guides/braintrust-evals.mdx, a guide for evaluating Pioneer decoder models with Braintrust. Our OpenAI-compatible API already works with Braintrust unchanged, but nothing documented that, so customers had no path for decoder LLM evaluation —concepts/evaluationscovers encoder metrics (F1/precision/recall) only.braintrust evalin both Python and TypeScript, with--no-send-logsshown first so customers can verify connectivity before creating an accountdocs.json, and a<Note>onconcepts/evaluationsso the encoder and decoder pages hand off to each otherThe guide also documents the LLM-as-judge scorer running on Pioneer, which means no OpenAI or Anthropic key is needed anywhere in the eval pipeline.
Gotchas documented
Each of these was hit while testing and makes a working setup look broken:
max_tokensbelow ~512 truncates reasoning models mid-thought; the empty completion scores as a wrong answerGET /v1/modelsis public and returns 200 without valid credentials, so it can't validate an API keypio_sk_prefix for the Bearer path to treat them as API keysbraintrust eval --listrequiresBRAINTRUST_API_KEY;--no-send-logsdoes notScope note
An earlier revision of this PR also covered Prime Intellect's
vf-eval. That was dropped — they compete with us in the same space, and pointing customers at their tooling from our own docs is not a trade we want to make. The page was renamed fromthird-party-evalstobraintrust-evalsto match its single subject; it has not shipped, so no redirect is needed.Test plan
mint broken-linkspassesdocs.jsonvalidates and the new page resolves in navigationchart-bar, already proven elsewhere in this repovf-eval/verifiersreferences anywhere in the repobraintrust eval evals/run verbatim from this guide against production (api.pioneer.ai/v1): discoveredeval_pioneer.py, created thepioneer-decoder-evalproject and an experiment, Factuality 100% / Levenshtein 91.67%, LLM-as-judge served by PioneerAll Inferences) resolves throughactive_model_id0ae6696b(Llama-3.1-8B-Instruct) served a completion and echoed the UUID as the model. A second fine-tune resolved but returned409 has no active inference deployment, now documented as a caveatMade with Cursor