Repository navigation
fetchFromHuggingFace: add - #766
Conversation
Add a fetcher to get kernels/models/datasets from the Hub. This will be used in upcoming changes to fetch kernel-kernel dependencies.
Coverage report —
|
| Name | Stmts | Miss | Cover | Missing |
|---|---|---|---|---|
| src/kernels/__init__.py | 13 | 0 | 100% | |
| src/kernels/_system.py | 6 | 1 | 83% | 10 |
| src/kernels/_versions.py | 63 | 7 | 89% | 46, 49, 52-53, 56-57, 100 |
| src/kernels/backends.py | 212 | 62 | 71% | 40, 44, 48-51, 68, 90, 108, 117, 121, 125-127, 148, 157, 161, 165-167, 188, 199, 201, 208-211, 224, 228, 232-252, 260, 283-303 |
| src/kernels/compat.py | 8 | 1 | 88% | 5 |
| src/kernels/deps.py | 58 | 4 | 93% | 59-60, 101, 104 |
| src/kernels/hf_hub.py | 62 | 5 | 92% | 18, 20, 114, 136-137 |
| src/kernels/importer.py | 48 | 3 | 94% | 106, 110, 113 |
| src/kernels/install.py | 50 | 5 | 90% | 78, 116-117, 155, 173 |
| src/kernels/layer/__init__.py | 6 | 0 | 100% | |
| src/kernels/layer/_interval_tree.py | 103 | 4 | 96% | 23, 52, 147, 150 |
| src/kernels/layer/device.py | 48 | 14 | 71% | 42, 47-49, 91, 96-98, 101, 149, 152, 155-157 |
| src/kernels/layer/func.py | 82 | 7 | 91% | 83, 113, 185, 303, 309, 322, 340 |
| src/kernels/layer/globals.py | 5 | 0 | 100% | |
| src/kernels/layer/kernelize.py | 74 | 8 | 89% | 255, 281, 289-290, 296, 300, 316-318 |
| src/kernels/layer/layer.py | 211 | 16 | 92% | 169, 212, 218, 231, 339, 419-420, 432, 441, 449, 460, 489, 493, 506, 559, 589 |
| src/kernels/layer/mode.py | 14 | 0 | 100% | |
| src/kernels/layer/repos.py | 144 | 42 | 71% | 27, 33, 36-43, 63-64, 70, 73-76, 90, 94, 103-104, 110, 113-116, 123-124, 130, 133-136, 143-144, 150, 153-156, 163-164, 170, 173-176, 257 |
| src/kernels/load.py | 74 | 9 | 88% | 218, 224, 230-231, 252-264 |
| src/kernels/locking.py | 108 | 70 | 35% | 42-109, 113-136, 144, 148-155, 159-169, 173-180 |
| src/kernels/status.py | 49 | 2 | 96% | 23, 81 |
| src/kernels/variants.py | 278 | 22 | 92% | 64, 95, 116, 146, 255-256, 298-301, 303, 387-394, 400-406, 437-443, 455-461, 611-613 |
| src/kernels/verify.py | 88 | 1 | 99% | 32 |
| TOTAL | 1804 | 283 | 84% |
Updated by the Test kernels workflow on commit 2f2544837400b5249e1c64f027855a994574cb8f.
|
|
||
| # Downloading leaves per-file metadata (etags, timestamps) behind, which would | ||
| # make the hash of the snapshot unstable. | ||
| shutil.rmtree(Path(local_dir) / ".cache", ignore_errors=True) |
There was a problem hiding this comment.
nit/question: is this safe? is there a case where the fetched repo legitimately contains files under .cache? if so, this would delete them too. could we remove only .cache/huggingface instead?
**I think it should be safe to remove the whole dir but wanted to double check
There was a problem hiding this comment.
Good question. I'm a bit worried about impurities that would arise because dependencies of huggingface_hub are also allowed to write to .cache, so I think it's safer to yank the whole directory. But we should keep it in mind for the future.
sayakpaul
left a comment
sayakpaul
left a comment
There was a problem hiding this comment.
Nice. My only comment is maybe we should restrict this to "kernel" type repos for now.
| let | ||
| # `hf download` only accepts the `model`, `dataset` and `space` repository | ||
| # types, so we use `snapshot_download` (which the CLI wraps) directly. | ||
| python = python3.withPackages (ps: [ ps.huggingface-hub ]); | ||
| in | ||
|
|
||
| { | ||
| # Repository id, e.g. `kernels-community/relu`. | ||
| repoId, | ||
|
|
||
| # Repository type, e.g. `model`, `dataset` or `kernel`. | ||
| type ? "model", | ||
|
|
||
| # Tag or commit SHA. | ||
| revision, | ||
|
|
||
| # SRI hash of the repository snapshot. | ||
| hash, | ||
|
|
||
| # Derivation name, defaults to `<repoId>-<revision>`. | ||
| name ? null, | ||
| }: |
There was a problem hiding this comment.
Feels illegal to write code this way!
| stdenvNoCC, | ||
| }: | ||
|
|
||
| let |
There was a problem hiding this comment.
Should we filter only "kernel" type repos?
There was a problem hiding this comment.
I like having a generic fetchers, maybe we can get it upstreamed in nixpkgs at some point 😁 .
| # downloading the kernel repo type yet. | ||
| python ${./download.py} \ | ||
| ${lib.escapeShellArg repoId} \ | ||
| ${lib.escapeShellArg type} \ |
There was a problem hiding this comment.
Shouldn't we only allow "kernel" type repo downloads in this section?
Add a fetcher to get kernels/models/datasets from the Hub. This will be used in upcoming changes to fetch kernel-kernel dependencies.