feat: make the library the source of truth; CLI becomes a presentation surface

[nwaughachukwuma]

Summary

Restructures mm so the library (mm.Context + support modules) owns all core computation and the CLI becomes a thin presentation surface (option parsing, stdin/pipe resolution, loading/success/error messages, Rich rendering, exit codes). This closes the documented feature-parity gaps between mm <cmd> and the public library surface, and makes the library self-contained / pluggable into any caller (notebook, web service, other agent) — not just the CLI.

The contract for every command is now:

parse options → resolve stdin → call library (all work) → render/serialize/print → exit

Two hard constraints from the design discussion are honored:

1. Performance via dependency injection, not regression. Hot-path commands keep their lean, pyarrow-free fast path by passing pre-built heavy objects into the library instead of having the library rebuild them. Library methods accept optional injected objects and construct a default only when the caller passes None:

   def grep(self, pattern, *, files: list[FileEntry] | None = None,
            regex: re.Pattern | None = None, ...) -> GrepResult:
       if files is None:                  # non-CLI caller → library builds it
           files = [f for f in self.filter(kind=kind).files if ...]
       return search_content(pattern, files=files, regex=regex, ...)

   Verified: mm find … --format json still imports 0 pyarrow / rich modules on the fast path (checked with python -X importtime).

2. Persistence is fully decoupled from the library. The core library never mandates a DB and Context has no DB awareness at all. Context.save() and the _db field were removed; the only persistence entry point is Context.to_records(), which exports plain dict records (directory-scan and incremental) so each caller writes to whatever backend it wants. The mm CLI writes those records to its own SQLite store entirely within the CLI/store surface:

   # caller-owned persistence (this is the mm CLI's workflow, not the library's)
   from mm.store.db import MmDatabase
   db = MmDatabase()
   db.upsert_records(ctx.to_records(refs=True), root=ctx.root)

What moved into the library

Capability	New source of truth	CLI now
wc / token+line stats	mm.stats.compute_wc → WcStats	renders WcStats
peek metadata	mm.peek.FileMetadata.from_path (also Context.peek)	renders rows
cat extraction pipeline	mm.cat_utils.extract.extract → CatResult; Context.cat(mode=…), Context.to_md(mode=…)	calls extract
find row selection + tree	Context.filter(...), Context.print_tree(layout=…)	passes options down
grep (smart-case, ctx, count, ext, FTS, semantic)	mm.search.search_content → GrepResult; Context.grep(...)	collects files, renders
sql table introspection	MmDatabase.list_tables() / mm.store.list_tables()	renders rows

New typed result objects live in mm/results.py (WcStats, GrepMatch, GrepFileCount, GrepResult, CatResult) — each with to_dict() so every --format serializes identically. These plus list_tables are re-exported from the top-level mm namespace.

config / profile / bench were already thin over mm.config / mm.profile / the benchmark harness and stay as-is (bench is intentionally CLI-only/operational).

One intentional behavior change: Context.grep is now smart-case (matching the CLI's single code path), with regex= / uppercase-pattern escape hatches.

Using the library

The library is the source of truth, so anything the CLI does is reproducible from Python. The same Context powers the CLI, a web service, and ad-hoc scripts.

1. Inside the CLI (presentation only)

The CLI parses options, calls the library, renders the typed result, and owns persistence — it never computes anything itself. Sketch of mm grep:

# python/mm/commands/grep.py (shape)
def grep_cmd(pattern: str, ...):
    ctx = Context(directory, no_ignore=no_ignore)
    files = _resolve_files(ctx, stdin_paths)          # CLI: dedupe scan + piped paths
    regex = compile_pattern(pattern, ignore_case=ignore_case)  # CLI fast path
    result: GrepResult = ctx.grep(                    # library does all the work
        pattern, files=files, regex=regex, ignore_case=ignore_case,
        context_lines=context_lines, count=count, semantic=do_semantic,
    )
    render_grep(result, fmt)                          # CLI: Rich/json/tsv rendering
    raise typer.Exit(0 if result.has_matches else 1)  # CLI: exit code

Persistence (mm sql, --pre-index) is also caller-owned, kept entirely in the CLI/store surface:

db = MmDatabase()
db.upsert_records(ctx.to_records(), root=ctx.root)    # CLI writes to its own store

2. In a FastAPI app

A web service can expose the same capabilities. Typed results have to_dict(), so they serialize directly to JSON. The service owns its own storage (here: none — pure compute).

from fastapi import FastAPI, HTTPException
from mm import Context

app = FastAPI()

@app.get("/wc")
def word_count(path: str, kind: str | None = None):
    ctx = Context(path)
    return ctx.wc(kind=kind).to_dict(by_kind=True)

@app.get("/grep")
def grep(path: str, pattern: str, ignore_case: bool = False, context_lines: int = 0):
    result = Context(path).grep(
        pattern, ignore_case=ignore_case, context_lines=context_lines,
    )
    return {"matches": result.to_dict()["matches"], "files": result.total_files}

@app.get("/cat")
def cat(path: str, file: str, mode: str = "fast"):
    try:
        return {"content": Context(path).cat(file, mode=mode)}
    except FileNotFoundError:
        raise HTTPException(404, f"{file} not found under {path}")

@app.get("/records")
def records(path: str):
    # storage-agnostic export; the service could persist these to Postgres,
    # object storage, a vector DB, etc. — the library never dictates a backend.
    return Context(path).to_records()

3. Generic Python (scripts, notebooks, agents)

Directory-scan mode for analyzing a corpus, plus DataFrame interop:

from mm import Context

ctx = Context("~/datasets/photos")
print(ctx.wc().to_dict())                                       # totals
df = ctx.to_polars()                                            # zero-copy from Arrow
big = df.filter(df["size"] > 1_000_000).select(["path", "size"])
hits = ctx.grep("TODO", kind="code", context_lines=2)           # GrepResult
meta = ctx.peek("hero.jpg")                                     # dims / EXIF / hash

Incremental role-aware mode for building a multimodal prompt and calling a VLM:

from pathlib import Path
from openai import OpenAI
from mm import Context

ctx = Context()                                  # auto-mints a session id
ctx.add("Describe these images and the report.", role="user")
ctx.add(Path("photo.jpg"), role="user", metadata={"note": "hero shot"})
ctx.add(Path("report.pdf"), role="user")

messages = ctx.to_messages(format="openai",      # VLM-ready, per-kind encoders
                           encoders={"image": "tile", "document": "rasterize"})
resp = OpenAI().chat.completions.create(model="gpt-4o", messages=messages)

Caller-owned persistence (any backend you like — example uses mm's SQLite store):

from mm import Context
from mm.store.db import MmDatabase                # the *caller* chooses this

ctx = Context("~/data", session_id="my-session")
records = ctx.to_records(refs=True)              # plain dicts; library's only persistence API
MmDatabase().upsert_records(records, root=ctx.root)

Link to Devin session: https://app.devin.ai/sessions/206deaa0c2c447ac9dc579ec4b5aa9ca
Requested by: @nwaughachukwuma

---

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[gemini-code-assist]

Code Review

This pull request refactors the codebase to decouple core computational logic from the CLI presentation layer, moving capabilities like content extraction, content search, and word count aggregation into library modules. It introduces structured result types in python/mm/results.py and enhances the Context class with new APIs and tree layout options. Feedback on the changes highlights two issues: a potential ValueError and infinite recursion loop in _build_tree_view when handling paths across different roots, and an inconsistency in _scan_files where absolute paths are used instead of relativized paths, leading to duplicate keys in the search results.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[devin-ai-integration]

Devin Review found 2 potential issues.

[nwaughachukwuma]

Hi Devin, please address all the comments from Gemini and Devin bots.

[nwaughachukwuma]

2. Persistence is decoupled from the library. The core library never mandates a DB. New Context.to_records() exports plain dict records (directory-scan and incremental) so each caller writes to whatever backend it wants. Context.save() is reframed as the mm CLI's own SQLite workflow (delegating to MmDatabase), and incremental save() now points users to to_records() rather than promising a built-in writer.

def save(self) -> None:
   ...
   ...
   self.db.upsert_files()

Hi Devin, the above is wrong. A true decoupling should remove the CLI code from the library/context. We don't want to burden anyone else using the library with CLI related stuff. The library/context should not be aware of the CLI internals. The CLI should call the library/context, get the required data or record and write to its DB self contained within the CLI surface. In summary, all users of the library surface should export with to_records and write the records themselves.

While you've done much of this, please self-review the code end-to-end based to ensure that the library/context surface is truly self contained, independent and completely decoupled from the CLI surface.

I hope this is clear?

[devin-ai-integration]

Done — the library/Context surface is now fully decoupled from persistence and CLI internals. Pushed in 1d1f7e7.

What changed

• Removed Context.save() entirely (along with the _db field and all DB-write code). The library no longer imports or instantiates MmDatabase in any executable path — context.py's only imports are json, pathlib, typing, and mm.refs.
• The sole persistence entry point is Context.to_records() (storage-agnostic dict records, both directory-scan and incremental modes). Callers write those records to whatever backend they want.
• Context.get() now resolves in-memory role-aware refs only; for directory-scan/cross-session lookups it raises and points callers to export + resolve via their own store.
• The mm CLI owns its persistence, self-contained in the CLI surface: commands/sql.py does MmDatabase().upsert_records(ctx.to_records(), root=ctx.root). The actual row-writing (upsert_files) lives only in mm/store/db.py (the store layer), never in Context.

End-to-end self-review result
I scanned the whole package. No library module forces a DB on consumers of Context:

• context.py — zero DB awareness; remaining MmDatabase mentions are docstring examples illustrating the caller-owned pattern.
• fts.py / semantic.py — these are search features that read an index; the index lives in mm.store, which is part of the library package (not the CLI). They're intra-library, optional, and only used when you call search.
• cat_utils/extract.py — uses shared_db() purely as an optional LLM-extraction result cache, guarded by content_hash/--no-cache; it's a perf optimization, not a mandated write.

So the invariant now holds: every consumer of the library exports with to_records() and writes records itself; the mm CLI is just one such consumer.

Also pinned the ty-check pre-commit hook to the project's ty (uv run --frozen ty) instead of unpinned uvx ty — the latest ty release started exiting non-zero on pre-existing optional-dep import warnings, which was breaking the hook repo-wide. CI/make typecheck already used the pinned ty, so this just aligns the hook.

All 1175 tests pass; lint + typecheck green.