Skip to content

github-integrate: Pack Assembly & Lazy CAS Blob Synchronization (Phase 3) #164

Description

@kavix

Goal

Synchronize compressed CAS object files (.eko/objects/) efficiently by packaging and uploading them on-demand.

Steps to Outcome

  1. Blob Packaging (TAR.GZ Pack):
    • Identify local CAS object files referenced by new manifests.
    • Compress and bundle these objects into a single .eko_pack.tar.gz archive.
  2. Remote Pack Storage (Release Assets / Packages):
    • Upload the package archive using the GitHub Releases API (as a release asset) or GitHub Container Registry (as generic OCI artifacts).
  3. Lazy Object Download on eko restore:
    • When a user triggers eko restore <id>, inspect the local cache first.
    • If object blobs are missing, download the corresponding remote pack archive from GitHub and extract it into .eko/objects/.
  4. Progress and Performance UX:
    • Show a real-time progress bar (using github.com/schollz/progressbar) during upload and download phases.

Final Outcome

Eko packages and uploads object blobs to GitHub during push, and downloads missing blobs lazily on-demand during restoration.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    github-integrateGitHub remote storage integration

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions