Problem
This package can upload an artifact to PSDI and verify one already on disk. It
cannot fetch one.
goldilocks_ml.artifacts.resolve builds the path a pinned dependency should
occupy, and if the file is not there it raises, naming the record and telling
the reader to go and get it:
local_data/artifacts/ptc95-vbq12/atom_init.json is missing. The
metallicity_checkpoint feature dependency is file atom_init.json from PSDI
record ptc95-vbq12 (https://data-collections.psdi.ac.uk/records/ptc95-vbq12).
Download it to that path.
Every protocol that pins an artifact therefore begins with a manual download
into a path the user has to construct correctly. The QRF95 protocol pins two;
the metallicity protocol pins one. A new user following the training guide hits
this before their first run.
The digests are already in the protocol, so the fetch has everything it needs
to be safe — it is simply not implemented.
Proposed approach
Give resolve a way to obtain what it is currently only able to describe.
- fetch a pinned dependency from its PSDI record into the artifact directory
when it is absent, and verify the digest the protocol pins before the file is
used for anything;
- refuse to overwrite a file that is present and correct, and refuse to keep
one that arrives wrong;
- keep the current behaviour available: an explicit offline mode still fails
with today's message, because a run on a plane should say what is missing
rather than hang;
- expose it as
goldilocks-ml train fetch PROTOCOL, so a user can prepare
everything a protocol needs before starting a long job.
PSDI serves record files without authentication, so this needs no token — which
keeps it well clear of the upload path and its --confirm-upload guard.
Goldilocks Core is building an asset store for the same class of problem in
stfc/goldilocks-core#168. This is deliberately smaller: a training dependency
resolver, not a shared cache, because the two have different lifetimes and
sharing one would couple the packages for little gain.
Acceptance criteria
Written by an agent on behalf of Junwen Yin.
Problem
This package can upload an artifact to PSDI and verify one already on disk. It
cannot fetch one.
goldilocks_ml.artifacts.resolvebuilds the path a pinned dependency shouldoccupy, and if the file is not there it raises, naming the record and telling
the reader to go and get it:
Every protocol that pins an artifact therefore begins with a manual download
into a path the user has to construct correctly. The QRF95 protocol pins two;
the metallicity protocol pins one. A new user following the training guide hits
this before their first run.
The digests are already in the protocol, so the fetch has everything it needs
to be safe — it is simply not implemented.
Proposed approach
Give
resolvea way to obtain what it is currently only able to describe.when it is absent, and verify the digest the protocol pins before the file is
used for anything;
one that arrives wrong;
with today's message, because a run on a plane should say what is missing
rather than hang;
goldilocks-ml train fetch PROTOCOL, so a user can prepareeverything a protocol needs before starting a long job.
PSDI serves record files without authentication, so this needs no token — which
keeps it well clear of the upload path and its
--confirm-uploadguard.Goldilocks Core is building an asset store for the same class of problem in
stfc/goldilocks-core#168. This is deliberately smaller: a training dependency
resolver, not a shared cache, because the two have different lifetimes and
sharing one would couple the packages for little gain.
Acceptance criteria
Written by an agent on behalf of Junwen Yin.