docs(readme): add a skimmable 'How Midas compares' table - #19
Merged
Merged
Conversation
A skimmable side-by-side after 'Why Midas', built only from claims already measured and published in BENCHMARKS.md — including the aggregation weakness, stated in Midas's own table. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
vornicx
reviewed
Jul 6, 2026
vornicx
left a comment
vornicx
left a comment
Owner
There was a problem hiding this comment.
Verified every row against BENCHMARKS.md on main — they all check out:
- 0 LLM / $0 / no egress at ingest, and ~16–116 ms ingest (§ Cost/latency: ~16 ms p50 short records, ~116 ms/event), vs ~668 ms for the LLM-at-ingest class — ✅ matches.
- 0.84 LongMemEval-
sjudged answer (gpt-4o), tying Observational Memory — ✅ matches the § judged-answer table ("gpt-4o | 0.84 | 0.84 — match"). - Verbatim source-traceable recall / fully offline (Ollama e2e) / deterministic, one-command repro — ✅ all supported by the repo.
- The aggregation row is stated as a weakness, ❌ by design, and linked to the measurements — on-brand with the eval-first discipline and it defuses the obvious launch-day criticism before it's made.
Docs-only, scoped to one section, nothing new asserted. LGTM — recommend merging. (CI hasn't run because fork PRs need a maintainer to approve the workflow; the change is README-only, so that's expected.)
One optional nit, non-blocking: the header row | | **Midas** | … renders an empty first cell — some Markdown viewers prefer a label there (e.g. | Property | **Midas** | … |). Fine to leave as-is.
Generated by Claude Code
vornicx
added a commit
that referenced
this pull request
Jul 6, 2026
Addresses the "community surface" and troubleshooting items from the outside review in #18. All additive, no code touched: - ROADMAP.md: the semver contract, the design invariants (no-LLM/local/ traceable/governed) stated as non-negotiable, what shipped in 1.0.0, a help-wanted near-term list, and — on-brand — an explicit "not planned" section tied to the documented measured ceilings (naive distillation 0.37->0.08, whole-conversation aggregation). - .github/ISSUE_TEMPLATE/: a bug report that pre-fills the `midas doctor` block (config paths/versions only, safe to paste), a feature request that asks up front whether it adds an LLM to the ingest/query path, and a config.yml routing questions to Discussions and the benchmark repro. - .github/PULL_REQUEST_TEMPLATE.md: a checklist that separates the measure-everything core bar from the lighter docs/examples path. - README "Troubleshooting": surfaces `midas doctor` as the one-command diagnosis with a symptom -> fix table (the #18 P2 ask). - CONTRIBUTING "A lighter bar for docs & examples": explicitly relaxes the eval-first bar for docs/guides/examples so first contributions aren't gated by the retrieval-core rigor. Does not include the README comparison table — that's #19, already open. Claude-Session: https://claude.ai/code/session_01Deray2qcPRy1hZVQnboHp4 Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a How Midas compares section to the README, right after Why Midas — a single skimmable table for the visitor deciding "Midas or Mem0?" in the ~30 seconds they give a repo.
Why
All the material already exists, but it's spread across README prose and BENCHMARKS.md sections. The table is built only from claims already measured and published in this repo (ingest cost/latency, provenance, the 0.84 gpt-4o tie with Observational Memory, offline e2e run) — nothing new is asserted.
The last row deliberately states Midas's own weakness (whole-conversation aggregation, ❌ by design, linked to the measurements). That's on-brand with the eval-first discipline, and it defuses the criticism before a launch-day commenter makes it.
Context and rationale in #18. Happy to adjust wording, numbers, or placement — or close if you'd rather present this differently.
🤖 Generated with Claude Code