Skip to content

Verify PreviewFullSyncAsync constant factors at Scale template (100K+) on a 20 GB+ host #1520

Description

@JayVDZ

Summary

The Sync Preview engine (#288) shipped with one verification outstanding. PreviewFullSyncAsync's bounding mechanics are proven, but its constant factors at realistic scale are not: the cloud sandbox could not host a Scale template (Scale100k50Groups needs 20+ GB; that host has ~15 GB). Recorded as a deferral in engineering/plans/done/SYNC_PREVIEW_ENGINE.md (Phase 4).

What is already proven

FullSyncPreviewScaleDatabaseTests runs against live PostgreSQL at 2,000 objects: two full walks, counts exact, samples bounded per category, the object cap truncating exactly, and the integrity tables byte-identical afterwards. That establishes that memory stays flat and the budget is honoured; it says nothing about wall-clock at 10^5.

What to run

From the devcontainer, which has the headroom the sandbox lacks.

  1. Bring up Scenario 8 at Scale100k50Groups with -SetupOnly. Population is far cheaper than the template menu's "~1 hour" suggests when the jim-openldap:s8-scale100k50groups snapshot image is present: the runner detects it and starts the directory from it rather than populating.
  2. Run a Full Import. -SetupOnly stops after configuring Connected Systems, Synchronisation Rules and Run Profiles; it imports nothing, so JIM's database holds zero Connected System Objects and a preview walk would find an empty population. The import has to run before any measurement, and it should run without a subsequent full synchronisation, so the population is unjoined and the preview exercises the projection path.
  3. Call PreviewFullSyncAsync with the default work budget (MaxObjects 10,000, no time budget). No caller exists: D3 shipped the engine and its JimApplication API only, so there is no portal, REST or PowerShell entry point and the measurement needs a harness (an opt-in database-backed test fixture beside FullSyncPreviewScaleDatabaseTests is the cheapest shape, and leaves the measurement repeatable if the defaults later change).
  4. Confirm bounded wall-clock and bounded memory, and that TotalObjectCount still reports the full population while Truncated and FullSyncPreviewTruncationReason are set correctly.
  5. Repeat with a TimeBudget set, to confirm the time-based truncation path behaves at scale as it does at 10^3.

Acceptance criteria

  • The run is performed and the figures recorded on this issue (wall-clock, peak memory, counts, truncation state).
  • If the constant factors are worse than the defaults assume, either the defaults change or the finding is written up so Deferred: Connected System full sync preview surface #1530 can set expectations.
  • Phase 4's deferral note in engineering/plans/done/SYNC_PREVIEW_ENGINE.md is not retro-edited; the outcome lives here (completed plans are not rewritten, per engineering/CLAUDE.md).

Sequencing

This gates #1530 only (the whole-system preview surface), not #1519. It is not overdue work; it is correctly deferred until #1530 is picked up, which is why it carries no milestone. The relationship is recorded natively in the sidebar.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    testingTesting infrastructure and coverage

    Type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions