Skip to content

feat(artifacts): fetch the artifacts a protocol pins #20

Description

@junwen94

Problem

This package can upload an artifact to PSDI and verify one already on disk. It
cannot fetch one.

goldilocks_ml.artifacts.resolve builds the path a pinned dependency should
occupy, and if the file is not there it raises, naming the record and telling
the reader to go and get it:

local_data/artifacts/ptc95-vbq12/atom_init.json is missing. The
metallicity_checkpoint feature dependency is file atom_init.json from PSDI
record ptc95-vbq12 (https://data-collections.psdi.ac.uk/records/ptc95-vbq12).
Download it to that path.

Every protocol that pins an artifact therefore begins with a manual download
into a path the user has to construct correctly. The QRF95 protocol pins two;
the metallicity protocol pins one. A new user following the training guide hits
this before their first run.

The digests are already in the protocol, so the fetch has everything it needs
to be safe — it is simply not implemented.

Proposed approach

Give resolve a way to obtain what it is currently only able to describe.

  • fetch a pinned dependency from its PSDI record into the artifact directory
    when it is absent, and verify the digest the protocol pins before the file is
    used for anything;
  • refuse to overwrite a file that is present and correct, and refuse to keep
    one that arrives wrong;
  • keep the current behaviour available: an explicit offline mode still fails
    with today's message, because a run on a plane should say what is missing
    rather than hang;
  • expose it as goldilocks-ml train fetch PROTOCOL, so a user can prepare
    everything a protocol needs before starting a long job.

PSDI serves record files without authentication, so this needs no token — which
keeps it well clear of the upload path and its --confirm-upload guard.

Goldilocks Core is building an asset store for the same class of problem in
stfc/goldilocks-core#168. This is deliberately smaller: a training dependency
resolver, not a shared cache, because the two have different lifetimes and
sharing one would couple the packages for little gain.

Acceptance criteria

  • A protocol's pinned dependencies can be fetched with one command
  • Every fetched file is digest-verified before use, and a mismatch fails
  • An offline mode reports what is missing without attempting a fetch
  • The training guide no longer opens with a manual download
  • Tests cover fetch, cache hit, digest mismatch, and offline

Written by an agent on behalf of Junwen Yin.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions