feat(persistence): snapshots by mrdevrobot · Pull Request #5 · EntglDb/EntglDb.Net

mrdevrobot · 2026-01-26T17:25:27Z

This pull request introduces major improvements to peer synchronization and maintenance in the EntglDb distributed database system. The key enhancements are the addition of periodic maintenance routines (such as oplog pruning), robust error handling and recovery during sync (including snapshot-based recovery), and exponential backoff for failed peer sync attempts. Several new methods and configuration options have been added to support these features.

Maintenance and Configuration Improvements

Added OplogRetentionHours and MaintenanceIntervalMinutes settings to PeerNodeConfiguration, allowing configurable periodic pruning of the oplog and maintenance intervals. These are now settable in both configuration and the Maui sample app. [1] [2]
Implemented periodic maintenance in SyncOrchestrator.SyncLoopAsync, which prunes the oplog based on the configured retention period.

Peer Synchronization Robustness

Introduced exponential backoff for failed sync attempts with peers, tracked via PeerStatus in SyncOrchestrator, to reduce repeated failed connection attempts. [1] [2] [3]
Enhanced handling of sync errors: now supports gap detection, chain integrity validation, and triggers full or merge snapshot syncs for recovery, including emergency database replacement if corruption is detected. [1] [2] [3] [4] [5] [6]

Peer Store and Snapshot Operations

Extended IPeerStore interface with new methods for snapshot creation, database replacement, merging snapshots, pruning oplog, and clearing all data. Also added a custom CorruptDatabaseException for error signaling. [1] [2]

SyncOrchestrator and TcpPeerClient Enhancements

Refactored SyncOrchestrator and TcpPeerClient to support new snapshot and chain recovery flows, including handling the SnapshotRequiredException when a peer requests a snapshot sync. [1] [2] [3] [4] [5]

These changes collectively make peer synchronization more reliable, maintainable, and resilient to data corruption and network

Copilot

Pull request overview

This PR adds snapshot-based recovery, periodic maintenance (oplog pruning), and exponential backoff to the EntglDb peer sync pipeline, plus the necessary persistence and protocol support. Overall it significantly extends the capabilities of both the SQLite and EF persistence providers and the network sync layer.

Changes:

Extend IPeerStore with oplog pruning and snapshot operations (PruneOplogAsync, CreateSnapshotAsync, ReplaceDatabaseAsync, MergeSnapshotAsync, ClearAllDataAsync) and implement them for the SQLite and EF Core stores, including new snapshot metadata tables/entities.
Enhance the sync protocol (sync.proto), TCP server/client, and SyncOrchestrator to support gap detection, snapshot-required signaling, snapshot streaming, emergency recovery on corruption, and exponential backoff for failing peers, plus periodic maintenance driven by new PeerNodeConfiguration options.
Update tests and the Maui sample app to adopt the new IPeerStore surface and configuration knobs for maintenance and retention.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 15 comments.

Show a summary per file

File	Description
tests/EntglDb.Network.Tests/VectorClockSyncTests.cs	Stub `IPeerStore` implementation extended with new snapshot/maintenance methods to keep tests compiling against the expanded interface.
tests/EntglDb.Network.Tests/HandshakeRegressionTests.cs	Network test stub store updated to implement the new `IPeerStore` methods (throwing `NotImplementedException`), ensuring handshake tests still isolate networking logic only.
tests/EntglDb.Network.Tests/ConnectionTests.cs	Connection test `StubStore` updated for the expanded `IPeerStore` contract, keeping these tests focused on connectivity rather than persistence.
tests/EntglDb.Core.Tests/PeerCollectionTests.cs	In-memory `IPeerStore` test double extended with `PruneOplogAsync` and new snapshot methods; `PruneOplogAsync` has a minimal in-memory implementation, others remain unimplemented.
src/EntglDb.Persistence.Sqlite/SqlitePeerStore.cs	SQLite store gains a `SnapshotMetadata` table, node-cache initialization from snapshots plus oplog, oplog pruning with metadata upserts, snapshot creation (file-level copy after WAL checkpoint), database replacement and merge-from-snapshot logic, cache-aware tail lookups; also introduces ClearAllData and extra corruption handling, but currently regresses query/count/index features and stops updating `_nodeCache` in `ApplyBatchAsync` (see stored comments).
src/EntglDb.Persistence.EntityFramework/Snapshot/SnapshotDto.cs	Defines EF snapshot JSON DTOs for documents, oplog, snapshot metadata, and remote peers, used by `EfCorePeerStore` snapshot/merge logic.
src/EntglDb.Persistence.EntityFramework/Entities/SnapshotMetadataEntity.cs	Adds EF entity for `SnapshotMetadata`, storing HLC physical/logical time and a hash per node for pruned-oplog checkpoints.
src/EntglDb.Persistence.EntityFramework/EntglDbContext.cs	Registers `SnapshotMetadataEntity` (`DbSet<SnapshotMetadataEntity>`) and configures its key and index on HLC components.
src/EntglDb.Persistence.EntityFramework/EfCorePeerStore.cs	Implements EF variants of `PruneOplogAsync`, JSON snapshot creation, full database replacement from snapshot, merge-snapshot with conflict resolution for documents, oplog, snapshot metadata, and peers, plus `ClearAllDataAsync` and bulk snapshot import.
src/EntglDb.Network/sync.proto	Extends protocol with snapshot messages (`GetSnapshotRequest`, `SnapshotChunk`), adds `snapshot_required` flags to `ChainRangeResponse` and `AckResponse`, and new `MessageType` enum values for the snapshot flow.
src/EntglDb.Network/TcpSyncServer.cs	Server now marks `ChainRangeResponse.SnapshotRequired` when gaps can’t be filled and implements `GetSnapshotReq` handling by streaming a snapshot over `SnapshotChunk` messages from a temp file produced by `_store.CreateSnapshotAsync`.
src/EntglDb.Network/TcpPeerClient.cs	Client detects `SnapshotRequired` flags on chain-range and push-ack responses (throwing `SnapshotRequiredException`) and adds `GetSnapshotAsync` to receive snapshot chunks and stream them into a destination.
src/EntglDb.Network/SyncOrchestrator.cs	Orchestrator adds per-peer failure state and exponential backoff, periodic maintenance via `_store.PruneOplogAsync` using `MaintenanceIntervalMinutes`/`OplogRetentionHours`, and a richer inbound-batch pipeline (batch integrity checks, gap detection with range-recovery, snapshot merge/full-replace flows on `SnapshotRequired` or `CorruptDatabaseException`).
src/EntglDb.Core/Storage/IPeerStore.cs	Core `IPeerStore` interface extended with pruning, snapshot creation, snapshot-merge, full replacement, and clear-all APIs, plus the `CorruptDatabaseException` type for signaling local persistence corruption to the sync layer.
src/EntglDb.Core/Network/PeerNodeConfiguration.cs	Adds `MaintenanceIntervalMinutes` and `OplogRetentionHours` configuration knobs that drive periodic maintenance in `SyncOrchestrator`.
samples/EntglDb.Test.Maui/MauiProgram.cs	Sample Maui app wires initial values for `OplogRetentionHours` and `MaintenanceIntervalMinutes` into its static peer configuration to exercise the new maintenance feature.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-26T17:34:19Z

+
+    public Task EnsureIndexAsync(string collection, string propertyPath, CancellationToken cancellationToken = default)
+    {
+        // Not implemented (Placeholder)


EnsureIndexAsync has been reduced to a no-op placeholder, which breaks the expectation established by both the IPeerStore contract and existing tests (SqlitePeerStoreTests.EnsureIndexAsync_ShouldCreateIndexWithoutError) that this method will actually create or ensure a backing SQLite index to optimize queries. If indexing is still required for production use, please reinstate the previous CREATE INDEX IF NOT EXISTS logic (with appropriate sanitization) or at least add logging and documentation to indicate that index creation is intentionally unsupported for the SQLite provider.

Suggested change

// Not implemented (Placeholder)

_logger.LogWarning(

"EnsureIndexAsync called for collection '{Collection}' and property path '{PropertyPath}', " +

"but index creation is not supported by the SQLite provider. This method is a no-op.",

collection,

propertyPath);

Se should reinstate previous behavior @copilot

@copilot correct the issue reinstating the missing part

Copilot · 2026-01-26T17:34:19Z

+    public Task<IEnumerable<string>> GetCollectionsAsync(CancellationToken cancellationToken = default)
+    {
+        using var connection = new SqliteConnection(_connectionString);
+        connection.Open();
+        return Task.FromResult<IEnumerable<string>>(GetKnownCollections(connection));


GetCollectionsAsync opens the SQLite connection synchronously (connection.Open()) and then immediately wraps the result in Task.FromResult, which is inconsistent with the rest of this class' async ADO usage and ignores the CancellationToken. For consistency and to avoid blocking threads under load, consider switching to await connection.OpenAsync(cancellationToken) and making the method fully async, or document why synchronous I/O is acceptable here.

Suggested change

public Task<IEnumerable<string>> GetCollectionsAsync(CancellationToken cancellationToken = default)

{

using var connection = new SqliteConnection(_connectionString);

connection.Open();

return Task.FromResult<IEnumerable<string>>(GetKnownCollections(connection));

public async Task<IEnumerable<string>> GetCollectionsAsync(CancellationToken cancellationToken = default)

{

using var connection = new SqliteConnection(_connectionString);

await connection.OpenAsync(cancellationToken);

return GetKnownCollections(connection);

Copilot · 2026-01-26T17:34:21Z

                            var rangeRes = new ChainRangeResponse();
-                            foreach (var e in rangeEntries)
+
+                            if (!rangeEntries.Any() && rangeReq.StartHash != rangeReq.EndHash)


Comparison of identical values.

Copilot · 2026-01-26T17:34:22Z

+                try 
+                {
+                    await PerformSnapshotSyncAsync(client, true, token);
+                    syncSuccessful = true;


This assignment to syncSuccessful is useless, since its value is never read.

@copilot make sure we are not missing the use of the variable and fix it

Copilot · 2026-01-26T17:34:22Z

+                {
+                    // EMERGENCY RECOVERY: Replace local DB with remote snapshot (mergeOnly: false)
+                    await PerformSnapshotSyncAsync(client, false, token);
+                    syncSuccessful = true;


This assignment to syncSuccessful is useless, since its value is never read.

Copilot · 2026-01-26T17:34:22Z

                _logger.LogDebug("Vector clocks are concurrent with {PeerNodeId}, but no divergence detected.", peer.NodeId);
            }
+
+            syncSuccessful = true;


This assignment to syncSuccessful is useless, since its value is never read.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot · 2026-01-26T19:19:51Z

@mrdevrobot I've opened a new pull request, #6, to work on those changes. Once the pull request is ready, I'll request review from you.

Copilot · 2026-01-26T19:20:07Z

@mrdevrobot I've opened a new pull request, #7, to work on those changes. Once the pull request is ready, I'll request review from you.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot · 2026-01-26T19:22:09Z

@mrdevrobot I've opened a new pull request, #8, to work on those changes. Once the pull request is ready, I'll request review from you.

Co-authored-by: mrdevrobot <12503462+mrdevrobot@users.noreply.github.com>

Copilot · 2026-01-26T22:05:40Z

@mrdevrobot I've opened a new pull request, #9, to work on those changes. Once the pull request is ready, I'll request review from you.

Co-authored-by: mrdevrobot <12503462+mrdevrobot@users.noreply.github.com>

Reinstate EnsureIndexAsync implementation with CREATE INDEX logic

Copilot · 2026-01-26T22:48:21Z

@mrdevrobot I've opened a new pull request, #10, to work on those changes. Once the pull request is ready, I'll request review from you.

Co-authored-by: mrdevrobot <12503462+mrdevrobot@users.noreply.github.com>

Reinstate QueryDocumentsAsync SQL implementation

feat(persistence): snapshots

05068ff

mrdevrobot requested a review from Copilot January 26, 2026 17:25

Copilot started reviewing on behalf of mrdevrobot January 26, 2026 17:25 View session

mrdevrobot self-assigned this Jan 26, 2026

Copilot AI reviewed Jan 26, 2026

View reviewed changes

mrdevrobot and others added 5 commits January 26, 2026 20:12

Update SqlitePeerStore.cs

e813ca3

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update SqlitePeerStore.cs

fd44a79

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update SqlitePeerStore.cs

48b7842

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update SqlitePeerStore.cs

28949bd

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update SyncOrchestrator.cs

e4e2fc7

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI mentioned this pull request Jan 26, 2026

Fix empty catch blocks in transaction rollback handlers #6

Closed

8 tasks

Copilot AI mentioned this pull request Jan 26, 2026

Fix empty catch blocks in SqlitePeerStore transaction rollback handling #7

Closed

13 tasks

mrdevrobot and others added 2 commits January 26, 2026 20:20

Update SqlitePeerStore.cs

09d8a4c

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Initial plan

75b92f7

Copilot AI mentioned this pull request Jan 26, 2026

Use syncSuccessful variable for logging sync outcomes #8

Merged

14 tasks

Copilot AI and others added 4 commits January 26, 2026 19:24

Use syncSuccessful variable for logging sync outcome

416008c

Co-authored-by: mrdevrobot <12503462+mrdevrobot@users.noreply.github.com>

Simplify sync outcome logging to avoid redundancy

69ba728

Co-authored-by: mrdevrobot <12503462+mrdevrobot@users.noreply.github.com>

Merge pull request #8 from EntglDb/copilot/sub-pr-5-another-one

1f26c06

Initial plan

0c44afd

Copilot AI mentioned this pull request Jan 26, 2026

Reinstate EnsureIndexAsync implementation with CREATE INDEX logic #9

Merged

13 tasks

Copilot AI and others added 4 commits January 26, 2026 22:10

Reinstate EnsureIndexAsync implementation with CREATE INDEX logic

960d27c

Co-authored-by: mrdevrobot <12503462+mrdevrobot@users.noreply.github.com>

Fix SQL injection vulnerability in EnsureIndexAsync WHERE clause

167c2f7

Co-authored-by: mrdevrobot <12503462+mrdevrobot@users.noreply.github.com>

Merge pull request #9 from EntglDb/copilot/sub-pr-5-yet-again

8a16dc7

Reinstate EnsureIndexAsync implementation with CREATE INDEX logic

Initial plan

4c09d0e

Copilot AI mentioned this pull request Jan 26, 2026

Reinstate QueryDocumentsAsync SQL implementation #10

Merged

14 tasks

Copilot AI and others added 3 commits January 26, 2026 22:51

Reinstate QueryDocumentsAsync with full SQL implementation

861ebb5

Co-authored-by: mrdevrobot <12503462+mrdevrobot@users.noreply.github.com>

Fix indentation in QueryDocumentsAsync implementation

d358869

Co-authored-by: mrdevrobot <12503462+mrdevrobot@users.noreply.github.com>

Merge pull request #10 from EntglDb/copilot/sub-pr-5-one-more-time

2723ead

Reinstate QueryDocumentsAsync SQL implementation

mrdevrobot merged commit 84956f1 into main Jan 26, 2026

-        // Not implemented (Placeholder)
+        _logger.LogWarning(
+            "EnsureIndexAsync called for collection '{Collection}' and property path '{PropertyPath}', " +
+            "but index creation is not supported by the SQLite provider. This method is a no-op.",
+            collection,
+            propertyPath);

Uh oh!

Conversation

mrdevrobot commented Jan 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

mrdevrobot Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrdevrobot Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

mrdevrobot Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Jan 26, 2026

Uh oh!

Copilot AI commented Jan 26, 2026

Uh oh!

Copilot AI commented Jan 26, 2026

Uh oh!

Copilot AI commented Jan 26, 2026

Uh oh!

Copilot AI commented Jan 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mrdevrobot Jan 26, 2026 •

edited

Loading