Skip to content

ChangeSetWriter: order entries as journal narrative (uncommitted snapshot → dependents → commit) #214

Description

@knowledge-graphlet

Summary

ChangeSetWriterProvider currently emits a stamp's lifecycle in an order
that is correct on the wire but inverts the temporal narrative: when a
stamp commits, the committed chronology is written before the
uncommitted entities that were held back waiting for it. A reader of
the change set ZIP sees [commit, the-uncommitted-stuff-that-led-up-to-it]
rather than the natural [uncommitted-stuff-that-led-up-to-it, commit].

The on-disk format is unchanged, the loader is order-insensitive, and
no existing consumer breaks if we flip this. But the recent change-set
diagnostic tool surfaced the inversion as actively confusing during
manual review — the journal is now a thing humans read, not just a
machine transport.

This issue covers the writer-side reorder and the matching rollover
flush partition, plus a small ordering contract in javadoc.

Background

The relevant code is the service-thread loop in
ChangeSetWriterProvider.java#L230-L264.
Its current behavior:

  1. Uncommitted entities arrive at the queue → stashed in
    uncommittedEntitiesByStamp[stampNid] keyed by the stamp nid each
    uncommitted version references.
  2. A committed stamp arrives:
    • writeEntity(committed_stamp) — writes the now-2-version chronology.
    • Then uncommittedEntitiesByStamp.removeAll(stampNid).forEach(writeEntity)
      drains the held-back uncommitted entities (concept / semantic / pattern
      versions, plus any prior uncommitted snapshot of the stamp itself
      that was stashed against its own nid).
  3. Inactivity rollover / explicit save() / shutdown() triggers
    checkpoint() → state transitions from RUNNINGROTATING or
    STOPPED → loop exits → uncommittedEntitiesByStamp.forEachValue(writeEntity)
    flushes everything still held back, in undefined multimap-value order.

The per-stamp commit drain (step 2) writes commit first and the
uncommitted history second. The rollover flush (step 3) groups
nothing — stamp snapshots and their dependent entities are interleaved
arbitrarily across stamp nids.

Observed effect

A simple flow — create concept → reference some stamp uncommitted →
commit the stamp — produces:

StampChronology (2 versions: uncommitted, committed)
ConceptChronology (1 uncommitted version referencing the stamp)
SemanticChronology (1 uncommitted version referencing the stamp)
…

A natural reading of the journal would expect:

StampChronology (1 uncommitted version)
ConceptChronology (1 uncommitted version)
SemanticChronology (1 uncommitted version)
StampChronology (2 versions: uncommitted, committed)

The first matches "snapshot-of-state-at-flush-time". The second matches
"sequence of edits as they happened, terminated by the commit event."
The second is what humans expect when they read a change set as a
journal.

Proposed change

Two narrow edits in ChangeSetWriterProvider.startService():

1. Reorder the per-stamp commit drain

Drain held-back entities before writing the committed stamp.
Within the drain, write StampEntity snapshots first and other
entities (concepts / semantics / patterns) second, so each unit reads
as uncommitted-stamp → its-dependents → committed-stamp.

2. Partition the rollover/shutdown flush by stamp nid

Same partition strategy applied to the final flush, grouped per stamp
nid so each stamp's snapshot is followed by its own dependents instead
of all stamp snapshots clumped together followed by all dependent
entities. Mirrors the per-stamp commit-drain shape so the on-disk
narrative is consistent regardless of which path produced the entries.

3. Document the ordering contract

A short javadoc block on startService() (or a class-level note)
establishing the contract:

Within a change set ZIP, entries appear in a journal order where each
unit is uncommitted-stamp-snapshot → dependent-uncommitted-versions → committed-stamp, terminated either by the commit event itself or by
a rollover/shutdown flush. Importers should treat this ordering as
informational; correctness must not depend on it.

That clarifies that we have a contract for human readers without
trapping future contributors into thinking importers can rely on it.

Compatibility

  • Loader (LoadEntitiesFromProtobufFile): order-insensitive — merges
    by PublicId. Existing change set ZIPs (with the old order) and new
    change set ZIPs (with the new order) load identically.
  • Schema: unchanged.
  • Existing change set files on disk: unchanged. The reorder is
    forward-only; old files keep their old ordering and load just fine.
  • Downstream consumers: only consumer that cares about order is the
    change-set summary tool, which displays whatever order it finds. No
    changes needed there.

Acceptance

  • The commit-drain path emits [uncommitted-stamp(s)] [dependents] [committed-stamp]
    in that order.
  • The rollover-flush path emits one group per stamp nid:
    [uncommitted-stamp(s)] [dependents] per group, groups in any order.
  • Roundtrip test: write a transaction with held-back uncommitted
    dependents, close the writer, read back the ZIP, assert the new
    ordering message-by-message.
  • Roundtrip-via-loader test: same fixture, run through
    LoadEntitiesFromProtobufFile, assert the resulting in-store
    chronologies are equivalent to a baseline (i.e., the loader produces
    the same data regardless of writer order).
  • Javadoc contract block added; no existing javadoc removed.

Effort

~30 lines source + ~50 lines test, no schema or compatibility changes.
Ships in the next tinkar-core release.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions