Skip to content

feat(orm): Laravel-style chunk() and chunk_by_id() on the async ORM#166

Merged
tmgbedu merged 2 commits into
mainfrom
task/orm-chunk-865
Jul 11, 2026
Merged

feat(orm): Laravel-style chunk() and chunk_by_id() on the async ORM#166
tmgbedu merged 2 commits into
mainfrom
task/orm-chunk-865

Conversation

@tmgbedu

@tmgbedu tmgbedu commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Summary

Implements Laravel-equivalent chunking on the async Masonite ORM fork (fastapi_startkit/src/fastapi_startkit/masoniteorm/). Both methods are async generators and yield the ORM Collection of hydrated models per batch.

async for projects in Project.where("active", True).chunk(100):
    ...

async for projects in Project.chunk_by_id(100):
    for project in projects:
        print(project.id)

What changed

  • QueryBuilder.chunk(count) — offset/limit paging. Yields a Collection per batch and stops as soon as a batch comes back empty or shorter than count, so it never loops forever (mirrors Laravel chunk()).
  • QueryBuilder.chunk_by_id(count, column=None, alias=None) — keyset pagination. Orders by the primary key (or a given column) ascending and filters column > last_seen on each pass, so it stays correct when rows are inserted/deleted mid-iteration (mirrors Laravel chunkById()). alias reads the last-seen value from a differently named selected column when the ordering column is aliased.
  • Both guard against a non-positive size (ValueError).
  • Model.chunk() / Model.chunk_by_id() classmethod entry points, following the existing explicit cls.query() passthrough pattern — usable directly on the model and chainable after builder methods.

Testing

New sqlite-backed test module tests/masoniteorm/models/test_model_chunk.py (14 tests) covering: batch sizes covering all rows, hydrated Collection batches, exact-multiple termination, where(...) chaining, empty tables, keyset ordering, custom column, delete-safety mid-iteration, and size validation.

  • uv run pytest --ignore=tests/masoniteorm/postgres --cov → 1791 passed, 7 skipped, coverage 78.64% (>= fail_under 68).
  • ruff check + ruff format --check clean.

No changes to unrelated framework code.

Add async-generator chunking to the QueryBuilder with Model classmethod
entry points, both chainable after builder methods.

- chunk(size): offset/limit paging, yields a Collection per batch and
  stops on an empty or short batch (no infinite loop).
- chunk_by_id(size, column, alias): keyset pagination ordered by the
  primary key (or given column), filtering col > last_seen each pass so
  it stays correct when rows change mid-iteration.
- Both guard against a non-positive size.

Covered by sqlite-backed tests for batching, ordering, where-chaining,
empty tables, delete-safety, and size validation.
@codecov

codecov Bot commented Jul 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@tmgbedu

tmgbedu commented Jul 11, 2026

Copy link
Copy Markdown
Contributor Author

✅ Reviewer verdict: APPROVE (self-approval blocked — posting as comment)

Reviewed for correctness, Laravel fidelity, async-generator hygiene, termination, and test quality (task #867). Verified locally at c22a312.

Correctness & Laravel fidelity

  • chunk(count) — offset/limit paging; breaks on empty or short batch, so exact-multiple counts don't spin an extra empty query or loop forever. Matches Laravel chunk() (not delete-safe, by design).
  • chunk_by_id(count, column, alias) — true keyset pagination: snapshots base WHEREs, then each pass re-applies them + WHERE col > last_seen, ORDER BY col ASC, LIMIT count. No offset drift; safe under mid-iteration deletes. Matches chunkById().
  • Where-snapshot (base_wheres) prevents predicate stacking across passes; _order_by reset to () matches its init type; where(col, ">", last_id) hits the normal operator/value path; _model.__primary_key__ is the established access pattern.

Async-generator hygiene

  • Model.chunk/chunk_by_id classmethods return the async generator (not awaited) → correct async for and chainable after where(...). Yielded batches are the ORM Collection with hydrated models (asserted).

Termination

  • Both guard count <= 0 with ValueError; both terminate on empty/short batch. test_exact_multiple_does_not_loop_forever locks this in.

Test quality — genuinely behavioral, not hollow

  • 14 sqlite-backed tests with real assertions on batch sizes/ids ([3,3,3,1], ids == range(1,11)). No mock-call theater.
  • test_safe_when_rows_deleted_mid_iteration deletes a not-yet-seen row each pass and asserts seen == [1,2,4,5,7,8,10] — a real proof keyset skips deleted rows without missing others. Exactly the rigor the PR Coverage: masoniteorm relationships (BelongsToMany, Morph*) #159 audit asked for.

Scope / gates

  • Only builder.py (+53), model.py (+8), new test file touched — no unrelated framework code.
  • Local run: 1791 passed, 7 skipped; coverage 78.64% (≥ 68 fail_under); ruff check + ruff format --check clean. New file: 14 passed.

Non-blocking suggestions

  1. The alias param of chunk_by_id ships without a dedicated test. I verified it works (select("id as pk") + alias="pk" → ids 1..10), but a committed test would lock it in.
  2. chunk_by_id mutates builder state (_wheres/_order_by) in place, leaving the builder modified after iteration. Fine for one-shot builders; worth a note if builders are ever reused.

Verdict: APPROVE. Not merging (reviewer role).

@tmgbedu tmgbedu merged commit f5c4a3b into main Jul 11, 2026
6 checks passed
@tmgbedu tmgbedu deleted the task/orm-chunk-865 branch July 11, 2026 19:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant