Re-evaluate dependent queries on touches and streamline dependency bookkeeping

.github/workflows/flutter_test.yml

@@ -37,5 +37,19 @@ jobs:
     - name: Run Native tests
       run: flutter test test/native --concurrency=1
 
+    # Validate workload results without treating shared-runner timings as a baseline.
+    - name: Check profiling workloads
+      env:
+        PROFILE_WARMUPS: '1'
+        PROFILE_WARMUP_MS: '0'
+        PROFILE_TRIALS: '1'
+      run: flutter test benchmark/value_store --concurrency=1
+
+    - name: Check manager workloads
+      run: >-
+        dart run benchmark/value_store/run_core.dart --suite manager_core
+        --source workspace=dir:. --modes jit --passes 1 --trials 1
+        --warmups 1 --warmup-ms 0 --out build/ci-manager-core
+
     - name: Run Web tests
       run: flutter test test/web --platform chrome --concurrency=1

CHANGELOG.md

@@ -1,3 +1,18 @@
+## 6.0.0
+
+* Re-evaluate documents touched by a dependency change or `Document.rebroadcast()` in queries the way modified documents are, so a touched document can enter, leave or move within a query's results. Previously a query only re-emitted touched documents that were already in its results.
+* Touch the dependents of every document under a deleted document or collection, including documents in nested subcollections.
+* Remove deleted documents from the dependency graph eagerly, including documents in nested subcollections.
+* Make rebroadcasting a missing document do nothing.
+* Report an error thrown while a query processes a broadcast, such as by its filter or sort, through `FlutterError.reportError`. Previously it stopped all later broadcasts.
+* Speed up the value stores' reads, writes and deletes, and reduce document allocation and path lookup overhead.
+* Fix a data store keeping empty entries after documents move out of it, such as when their persistence key changes, which kept an emptied data store from being deleted.
+* [Breaking] Compare observable documents by path like plain documents, so an `ObservableDocument` equals other handles to the same document. Active observers are still tracked by identity.
+* [Breaking] Store dependencies only for documents that have some, so `Document.dependencies()` returns `null` instead of an empty set when the dependencies builder returns one.
+* [Breaking] Remove `PathRefStore` and `ObservableDocument.inspect()`, remove `deps` and `docDeps` from `ObservableQuery.inspect()`, and key its `docSnaps` by document ID. Dependency state is available through `Document.dependencies()` and `Document.dependents()`.
+* [Breaking] In `Loon.inspect()`, return dependency entries as objects whose `toJson()` produces `{doc, dependencies}` using document paths, return `dependentsStore` as a path tree, and remove `dependencyCache`.
+* [Breaking] Move the implementation under `lib/src` and export Loon's public API only from `package:loon/loon.dart`, which now also exports `FilePersistor`, `SqlitePersistor`, `IndexedDBPersistor`, `DataStoreEncrypter`, `generateSecureId` and `generateFastId`. Imports of other `package:loon/...` paths need to change to `package:loon/loon.dart`. Internal classes such as `ValueStore`, `ValueRefStore`, `BroadcastManager`, `BroadcastObserver`, `DependencyManager`, `PersistManager` and the persistor operations are no longer exported, and neither are the persistors' implementation details, such as `SqlitePersistor.initDB` and its table constants.
+
 ## 5.6.1
 
 * Update `flutter_secure_storage` to v10.

README.md

@@ -245,6 +245,9 @@ UserModel.store.streamChanges().listen((changes) {
       case BroadcastEvents.hydrated:
         print('${changeSnap.id} was hydrated from the persisted data.');
         break;
+      case BroadcastEvents.touched:
+        print('${changeSnap.id} was rebroadcast because state it depends on changed.');
+        break;
     }
   }
 });
@@ -275,11 +278,34 @@ final posts = Loon.collection<PostModel>(
 );
 ```
 
-In this example, whenever a post's associated user is updated, the post will also be rebroadcast to its active listeners.
+In this example, whenever a post's associated user is updated or deleted (including through the deletion of a parent document or
+collection), the post is rebroadcast to its active listeners and re-evaluated by any queries that observe it.
 
 Additionally, whenever a document is updated, it will rebuild its set of dependencies, allowing documents to support dynamic dependencies
 that can change in response to updated document data.
 
+### Rebroadcasting
+
+A document can also be rebroadcast manually. This is useful when a query's filter or sort reads state from outside of the store:
+
+```dart
+final blockedUserIds = <String>{};
+
+final visibleUsers = UserModel.store.where((snap) => !blockedUserIds.contains(snap.id));
+
+void blockUser(String userId) {
+  blockedUserIds.add(userId);
+
+  // Re-evaluates the user in queries like `visibleUsers`.
+  UserModel.store.doc(userId).rebroadcast();
+}
+```
+
+A rebroadcast notifies the document's listeners with a `BroadcastEvents.touched` event and has queries on its collection re-evaluate
+whether it belongs in their results and where it sorts. Query change listeners receive `added` or `removed` instead when the document enters
+or leaves the results. Rebroadcasting doesn't persist the document or rebuild its dependencies, and rebroadcasting a missing document does
+nothing. For state stored in another document, a data dependency rebroadcasts automatically instead.
+
 ## 🪴 Root collection
 
 Not all documents necessarily make sense to be grouped together under any particular collection. In this scenario, any one-off documents can be stored

analysis_options.yaml

@@ -1,4 +1,9 @@
 include: package:flutter_lints/flutter.yaml
 
+analyzer:
+  exclude:
+    # Profiling runs write generated programs and results here.
+    - build/**
+
 # Additional information about this file can be found at
 # https://dart.dev/guides/language/analysis-options

benchmark/.pubignore

@@ -0,0 +1,2 @@
+# Benchmarks are maintainer tooling and are not published with the package.
+*

benchmark/DECISIONS.md

@@ -0,0 +1,153 @@
+# Performance decisions
+
+The performance decisions behind the value store and the dependency and broadcast managers, newest
+first. Raw samples for the entries up to 2026-09-21 are in the history of PR #42. Fetch it with
+`git fetch origin pull/42/head`, then run, for example, `git show 5efc2e6:benchmark/value_store/results/`.
+
+## 2026-09-25: Keep inlining the store's hot helpers
+
+`_getParent`, `_PathCache.isMatch` and `_nextStoreDelimiter` are marked `@pragma('vm:prefer-inline')`,
+which asks the Dart compiler to inline them into their callers even when its size heuristics
+wouldn't. Inlining `_getParent` also lets each caller skip allocating the record it returns. Without
+the pragmas in AOT, empty-store gets took 46% longer, misses and existence checks 5–8% longer, and
+`manager_core` propagation and writes 5–14% longer, while a second copy with them stayed mostly
+within 5%.
+
+## 2026-09-24: Hold the parent cache in an immutable class
+
+The store's last-parent cache is a small immutable `_PathCache` holding the path, the index at which
+its final segment starts and the parent node, with the prefix check as its `isMatch` method. A walk
+that resolves replaces it, and anything that removes nodes sets it to null. Unlike the helper class
+of 2026-09-23, an empty cache is still one null check on the store, and empty-store gets took 1%
+longer. Against the three fields in AOT, every `store_core` operation stayed within its A/A noise of
+about 10%, including reads that resolve a new parent and allocate a new cache each time, and
+`manager_core` propagation and writes moved by -8% to +4%, also within noise.
+
+## 2026-09-24: Touch dependents without clearing their document observers
+
+The propagation benchmark wrote observer values at the dependents' and collections' own paths, so the
+first collection clear emptied the store and every later clear returned at once, and the 2026-09-22
+one-collection numbers left out the cost of clearing observer values. With values keyed under the
+observed paths, as observers key them, touching 20k dependents in one collection took 16.5 ms in AOT
+`manager_core`, most of it clearing observer values.
+
+A touch doesn't change a dependent's data, so its document observers keep their cached snapshots, and
+only its collection's cached query results are cleared, once per collection. For 20k dependents:
+
+- Without the document clears: 16.5 → 9.0 ms in one collection, and 22.3 → 15.1 ms across 5,000.
+- Then clearing each collection once: 9.1 → 5.2 ms in one collection, and 14.8 → 11.0 ms across
+  5,000 interleaved collections. Skipping only consecutive repeats matched the set in one collection
+  but didn't help interleaved dependents. The set costs 30–80 ns per fan-out of one or two dependents.
+
+## 2026-09-24: Settle the heap before manager samples
+
+Setup left tens of thousands of young objects that collections during the timed operation copied, so
+builds with the same algorithm differed by up to 24% on the 5,000-collection propagation cases, in
+either direction. Each `manager_core` sample now starts after `settleHeap()` promotes what its setup
+allocated. The same builds then agreed within 3–7%, as they did with a 512 MB young generation.
+
+## 2026-09-24: Write events in one lookup
+
+`writeDocument` writes an event directly, or with `putIfAbsent` for a touch, instead of reading the
+pending event first: 20k writes without dependents took 31–35% less time in AOT `manager_core`
+(3.9 → 2.7 ms), and chained writes 3–15% less. Merging the store's two parent lookups into one
+`_getParent` was neutral in `store_core` over 7 passes, with a 15-pass recheck of overwrites.
+`extractValues` collects through `forEachValue`, with no measurable difference.
+
+## 2026-09-24: Deletes through the parent cache (not adopted)
+
+Deletes that reused the cached parent, and kept it while the node stayed in the tree, took 22–53% less
+time deleting 20k siblings in `store_core`, with no cost on scattered deletes. They didn't change
+propagation once touches stopped clearing document observers, so they're left for bulk-delete work.
+
+## 2026-09-23: Visit deleted dependency entries in place
+
+Deleting a collection unlinks its documents from their dependencies' reverse indexes while visiting
+their entries with `ValueStore.forEachValue`, instead of extracting the entries into a set first.
+For 20k records in AOT `manager_core`, it takes 32–37% less time: 2.8 vs 4.5 ms when they share one
+dependency, and 5.5 vs 8.2 ms when each has its own.
+
+## 2026-09-23: Keep the parent cache in the value store
+
+Moving the last-parent state into a helper class was slower on the store's hot paths: empty-store
+gets took about 25% longer, and early misses and plain writes a few percent longer. It stays as
+three fields and two helpers in `_BaseValueStore`. Replaced on 2026-09-24 by an immutable cache
+class.
+
+## 2026-09-22: Reuse the last parent node in the value store
+
+Sending dependent touches through `writeDocument` read well, but made propagation to 20k dependents in
+one collection about 6× slower: 2.7 → 16.2 ms in AOT `manager_core`.
+
+`_BaseValueStore` now remembers the last parent node it resolved. Reads and writes under the same
+parent reuse it, and every operation that removes nodes forgets it. `ValueStore.putIfAbsent` records a
+touch in one call. This replaced the manager's own collection-map cache.
+
+AOT, against sending touches through `writeDocument`:
+
+- Propagation to 20k dependents: 16.2 → 4.9 ms in one collection, 15.4 → 10.1 ms across 5,000.
+- 20k writes without dependents: 4.4 → 3.8 ms. Store reads and writes under one parent take 40–80%
+  less time.
+- Costs: about 5% on early misses and deletes, and 12% on reads that never share a parent.
+- A 2- or 4-entry cache only helped writes that alternate between deep collections that fit in it
+  (about 15%), and cost 13–30% on scattered access. One entry was kept.
+
+## 2026-09-22: Read query snapshots by collection
+
+`ObservableQuery` reads a broadcast's snapshots with one `getChildValues` call and caches them by
+document ID, instead of building a `Document` per event. It is about 2× faster with 20k changed
+documents (JIT). Snapshots that aren't parsed yet, such as hydrated data, still go through
+`Document.get`.
+
+## 2026-09-21: Insert-if-absent primitives (not adopted)
+
+A single-walk insert-if-absent took about 46% less time for repeated insertion, and grouped bulk
+inserts 74% less. Propagation built on them lost to collection-map reuse on dense fan-out, and bulk
+batching changed query ordering. Superseded by the value store's parent reuse.
+
+## 2026-09-21: Collection-map helper (not adopted)
+
+A `_CollectionValues` helper and a document-access wrapper read better, but repeated path checks and
+slowed deletion: alternating-dependency deletion went from 4.7 to 10.2 ms. The engine stayed as it was.
+
+## 2026-09-21: Typed parent lookup and shared getter
+
+One `(Map, String)? _getParent(path)` helper replaced the access-mode enum, and `get()` uses it too.
+`ValueRefStore` shares one `_subtractRef` helper. The existing pruning stayed after two simpler
+versions cost more.
+
+## 2026-09-20: Scoped dependency propagation
+
+Propagation reused the last collection map of the event store and the reverse index during one
+synchronous walk, taking about 71% and 81% less time for shared and nested propagation. Deletion
+reused the last dependency set. Replaced on 2026-09-22 by the value store's parent reuse.
+
+## 2026-09-20: Parsed-path caches (not adopted)
+
+Parsed paths kept on document handles made reads about 40% faster but cost 14–29 MB per 100k handles.
+Bounded shared caches only helped when the working set fit, and a one-entry recent-path cache didn't
+help. The store kept its scanner.
+
+## 2026-09-20: Store core pass and delimiter scanning
+
+`hasPath` resolves in one walk, extraction gets the owning node and final segment together, traversal
+uses `Map.forEach` and stops at leaf buckets, and a code-unit delimiter scanner replaced
+multi-character `indexOf`. No node cache, flat index or new API was added.
+
+## 2026-09-16: Document allocation and dependency entries
+
+Adopted fixed-arity `Object.hash`, cached collection paths, membership loops instead of temporary
+difference sets, and reference-path parsing without splitting and rejoining. Retained
+`_DependencyEntry` handles stayed over rebuilding documents from paths: 2.6–3.1× faster bulk-deletion
+cleanup in AOT, at a memory cost.
+
+## 2026-09-15: Traversal and extraction APIs (not adopted)
+
+Callbacks, lazy iterables, hand-written iterators and shared bucket extraction had no consistent
+winner. A lazy iterator took about twice as long as a callback for simple sums in AOT, and extraction
+cost is mostly building the result set. `extractValues` stayed, with no public traversal API.
+
+## 2026-09-15: First store pass
+
+`get` walks segments incrementally and returns early on an empty store, empty-store deletes skip
+work, and `Document` caches its path and hash.

benchmark/README.md

@@ -0,0 +1,70 @@
+# Benchmarks
+
+Maintainer tooling for measuring Loon's performance. None of it is published with the package.
+
+| Tool | Measures | Runs in |
+| --- | --- | --- |
+| [`value_store/run_core.dart`](value_store/run_core.dart) | The value store built from any git ref or directory, and the dependency and broadcast managers from 6.0.0 on | Dart JIT and native AOT |
+| [`loon_benchmark.dart`](loon_benchmark.dart) | The whole library through its public API | `flutter test` (JIT) |
+
+## Compare versions
+
+`run_core.dart` compiles the store (and, for `manager_core`, the managers) from each source into a
+standalone program, runs the same workload against every source, and reports pooled medians with the
+range of per-process medians.
+
+```sh
+# origin/main, HEAD and the working tree, in JIT and AOT.
+dart run benchmark/value_store/run_core.dart
+
+# Dependency propagation, dependency updates and writes, in AOT.
+dart run benchmark/value_store/run_core.dart --suite manager_core --modes aot
+
+# Chosen sources and operations.
+dart run benchmark/value_store/run_core.dart --suite manager_core \
+  --source before=git:origin/main --source after=dir:. \
+  --filter '^propagation/' --modes aot --passes 3 --trials 9
+```
+
+- `store_core`, the default, covers reads, misses, existence checks, child values, ancestor queries,
+  writes, overwrites, deletes and extraction. Paths are shallow, deep, or each under a different
+  parent, with UUID-like, long, Unicode and underscore-heavy segments. It also covers
+  reference-counted stores.
+- `manager_core` covers registering and updating dependencies, deleting 20k documents that share one
+  dependency or each have their own, propagation to 20k dependents in one or 5,000 collections,
+  chains whose writes alternate between collections, and writes without dependents. Propagation
+  starts with an observer value under each dependent and its collection, keyed the way observers key
+  them. It uses the fixture documents in `manager_fixtures.dart`, so it leaves out persistence,
+  document data and observer delivery. Each sample starts after `settleHeap()` promotes what its
+  setup allocated, so collections during the sample don't copy it.
+- Results go to `build/value_store_profiles/<timestamp>-core`, or `--out`: `report.md`, raw samples,
+  source hashes and command receipts. Result checks run outside the timed interval. `summarize.dart`
+  rebuilds a report from a results directory.
+- Compare numbers within one run only, and check the load average first. On a busy machine,
+  differences under about 10% are not reliable.
+
+## End to end
+
+`loon_benchmark.dart` drives the real library: write and read throughput, observer setup, broadcast
+latency, sorted queries, dependency fan-out into a queried collection, dependency shapes (registering,
+propagating to and deleting 20k dependents) and sparse updates.
+
+```sh
+flutter test benchmark/loon_benchmark.dart
+```
+
+It only uses public API, so it runs on older versions: add a worktree for the ref, copy this file into
+its `benchmark/` directory, and run it there. Before 6.0.0, the ID generators also need
+`import 'package:loon/utils/id.dart';`. Timings are JIT and vary between runs, so run each version a few
+times.
+
+## CI
+
+CI runs the `store_core` workload in `flutter test` and `manager_core` through `run_core.dart`, one
+trial each, to check that both still compile and produce correct results. Timings from shared runners
+are not recorded.
+
+## Recording a decision
+
+Add an entry to [DECISIONS.md](DECISIONS.md): the question, what was chosen, the numbers that decided
+it and how to reproduce them. Don't commit raw results.