RFC: GitHub API Integration for Remote Snapshot Sync & Backup

[kavix]

Hi team!

I've put together a comprehensive architectural design and roadmap plan for adding remote sync and backup support to Eko directly using GitHub's REST/GraphQL APIs instead of relying on a local Git client.

We want to allow developers to seamlessly push and pull Eko snapshots directly to/from GitHub repositories without cluttering the local Git history or index.

Here is the proposed design overview:

1. Stateless Remote & SQLite Re-Indexing:
   To avoid concurrent merge conflicts (e.g. merging a single SQLite binary file), remote storage will be purely file-based. On eko push, manifests are uploaded as individual JSON files to a custom git namespace (refs/eko/snapshots/*). On eko pull, Eko fetches new manifests and dynamically indexes them to reconstruct the local database cache.

2. Storage Mechanisms:

   • Manifests & metadata: Custom remote refs namespace (refs/eko/snapshots/<id>).
   • CAS Object blobs: Packaged into compressed archives (.tar.gz) and uploaded as GitHub Release Assets or generic container packages.
   • Restoring: missing object blobs are lazily downloaded from GitHub on-demand during eko restore.

3. Keyring-based OAuth Device Flow:
   Authenticate using GitHub OAuth Device flow and store tokens securely in native OS keychains using zalando/go-keyring.

I've created a set of trackable issues matching each implementation phase:

• Authentication & Keychain (Phase 1): github-integrate: Implement OAuth Device Flow & Keychain Credential Storage (Phase 1) #162
• Remote Manifest Sync (Phase 2): github-integrate: Event-Sourced Remote Manifest Synchronization (Phase 2) #163
• Pack Assembly & Lazy CAS Blob Sync (Phase 3): github-integrate: Pack Assembly & Lazy CAS Blob Synchronization (Phase 3) #164
• Bootstrap Workspace via eko clone (Phase 4): github-integrate: Bootstrap Workspace via eko clone (Phase 4) #165

Please check out the full roadmap and let me know your thoughts/suggestions on this approach!