Skip to content

perf: paginate wakatime/sync users query (#3161) - #3203

Merged
Priyanshu-byte-coder merged 1 commit into
Priyanshu-byte-coder:mainfrom
vedant7007:fix/wakatime-sync-pagination-3161
Jul 19, 2026
Merged

perf: paginate wakatime/sync users query (#3161)#3203
Priyanshu-byte-coder merged 1 commit into
Priyanshu-byte-coder:mainfrom
vedant7007:fix/wakatime-sync-pagination-3161

Conversation

@vedant7007

Copy link
Copy Markdown
Contributor

Summary

`/api/wakatime/sync` did one unbounded `.select()` and pulled every user with a wakatime API key (plus their encrypted keys) into serverless memory in a single response. At a few thousand users that would either:

  • hit Supabase's ~4.5 MB response cap,
  • blow the Vercel function timeout, or
  • get killed by the runtime memory limit,

and the nightly sync would silently fail.

What changed

Adopted the same pagination pattern already in use by `/api/cron/sync`:

  • `PAGE_SIZE = 50`
  • deterministic `.order("id")` so pagination doesn't drift
  • `.range(page * PAGE_SIZE, (page + 1) * PAGE_SIZE - 1)`
  • `while (hasMore)` loop that breaks out when a short page is returned

Kept the inner `Promise.allSettled` `CHUNK_SIZE = 5` for parallelism across the Wakatime API within each page — that part was already sensible, just needed to be nested inside the pagination.

What's unchanged

Decrypt-then-fetch-then-upsert logic per user is identical. No behavior change for the actual sync — only the fetch/scan strategy differs.

Test plan

  • `tsc --noEmit` clean
  • verified fix mirrors the sibling `/api/cron/sync` pagination line-for-line (same PAGE_SIZE, same `.order("id")` + `.range()` shape, same `if (users.length < PAGE_SIZE) hasMore = false` tail check)
  • once approved: manual smoke — confirm the cron completes across a few paginated batches without truncating

Fixes #3161

/api/wakatime/sync did one unbounded `.select()` and pulled every user
with a wakatime API key (plus their encrypted keys) into serverless
memory in a single response. At a few thousand users that would either
hit Supabase's ~4.5 MB response cap, blow the function timeout, or get
killed by the memory limit — and the nightly sync would silently fail.

Adopt the same pagination pattern already in use by /api/cron/sync:
PAGE_SIZE=50, deterministic .order("id"), .range(page*50, page*50+49),
break out when a short page is returned. Kept the inner Promise.allSettled
CHUNK_SIZE=5 for API-call parallelism inside each page.

Fixes Priyanshu-byte-coder#3161
@github-actions github-actions Bot added type:bug GSSoC type bonus: bug fix type:feature GSSoC type bonus: new feature type:performance GSSoC type bonus: performance (+15 pts) gssoc26 GSSoC 2026 contribution labels Jul 13, 2026
@github-actions

Copy link
Copy Markdown

GSSoC Label Checklist 🏷️

@Priyanshu-byte-coder — please apply the appropriate labels before merging:

Difficulty (pick one):

  • level:beginner — 20 pts
  • level:intermediate — 35 pts
  • level:advanced — 55 pts
  • level:critical — 80 pts

Quality (optional):

  • quality:clean — ×1.2 multiplier
  • quality:exceptional — ×1.5 multiplier

Validation (required to score):

  • gssoc:approved — counts for points
  • gssoc:invalid / gssoc:spam / gssoc:ai-slop — does not score

Type labels (type:*) are auto-detected from files and title. Review and adjust if needed.
Points formula: (difficulty × quality_multiplier) + type_bonus

@vedant7007

Copy link
Copy Markdown
Contributor Author

starred the repo — should turn the star-required gate green on the next re-run 🌟

@Priyanshu-byte-coder Priyanshu-byte-coder added gssoc:approved GSSoC: PR approved for scoring level2 GSSoC Level 2 - Medium complexity (25 points) quality:clean GSSoC: Clean quality multiplier (×1.2) labels Jul 19, 2026
@Priyanshu-byte-coder
Priyanshu-byte-coder merged commit f13be04 into Priyanshu-byte-coder:main Jul 19, 2026
6 of 7 checks passed
@github-actions

Copy link
Copy Markdown

🎉 Merged! Thanks for contributing to DevTrack.

If the project has been useful to you, a ⭐ star on the repo is the easiest way to support it — it helps DevTrack get discovered by more developers.

Keep an eye on open issues for your next contribution!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gssoc:approved GSSoC: PR approved for scoring gssoc26 GSSoC 2026 contribution level2 GSSoC Level 2 - Medium complexity (25 points) quality:clean GSSoC: Clean quality multiplier (×1.2) type:bug GSSoC type bonus: bug fix type:feature GSSoC type bonus: new feature type:performance GSSoC type bonus: performance (+15 pts)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Performance] /api/wakatime/sync loads every user with a wakatime key in one unbounded query

2 participants