[sayakpaul]

Fixes #648

Obsolete for now

Considerations:

• The Nix sandbox has no .git, so this information has to be passed in explicitly. So, builder-info related stuff become command-line arguments embedded in the derivation's build script at eval time. Is there a better way to handle it?
• nix build / nix flake show writes an untracked flake.lock, which makes the kernel tree dirty aka kernel.dirty=true unless the lock is committed. I have made a note about it in the docs. Is there a better way to handle it?

Example metadata.json with builder-info:

"provenance": {
  "kernel-builder": {
    "version": "0.17.0-dev0",
    "sha": "ebfedf0ffbf7b12da51cc336f90a26650090fdca",
    "dirty": false
  },
  "kernel": {
    "sha": "ebfedf0ffbf7b12da51cc336f90a26650090fdca",
    "dirty": false
  }
}

---

We will throw a warning during upload to the Hub and also show a warning on the Hub card that the corresponding builds were made with a "dirty" version. Kernel loading also logs similar warnings.

As discussed over Slack, this PR will only tackle the metadata-related stuff. Warnings and related things will be added in subsequent PRs.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[danieldk]

This can lead to incorrect metadata in case someone generates pyproject.toml and then adds more commits. So doing this from CMake (and then giving the kernel build sandbox access to git files) seems like the way to do it. For the Nix build (since we probably do not want to copy all the git objects to the sandbox), we could pass the SHA to CMake. Not sure how important this is though.

This is currently also an issue for the sha hash we use in the ops name. We are generating it when making the project files, but it should probably be done inside CMake (but would better be a separate PR to avoid PRs becoming too big).

[sayakpaul]

Will table it for later as mentioned.

We go with our approach and document the bad practice of running create-pyproject and committing after that.

[danieldk]

A local path can also be a git source. I think in flakes it must even be.

[sayakpaul]

Will break the PR into multiple parts, starting with appending the builder info, as discussed.

Will wait for the metadata versioning PR, first before continuing. This so that we don't have to make the provenance data optional,

[github-actions]

Coverage report — kernels/

Measured on: Python 3.10 / Torch 2.12.0.
Other CI configurations are not included in this number.
Hardware-gated code paths (ROCm/XPU/NPU/Darwin/Windows) are excluded or unreachable on the Linux+CUDA runner.

Total coverage: 84.6% — threshold: 80% — ✅

Per-file breakdown

Name	Stmts	Miss	Cover	Missing
src/kernels/__init__.py	10	0	100%
src/kernels/_system.py	6	1	83%	10
src/kernels/_versions.py	63	7	89%	46, 49, 52-53, 56-57, 100
src/kernels/backends.py	194	55	72%	40, 44, 48-51, 68, 90, 108, 117, 121, 125-127, 148, 170, 181, 188-191, 201, 205-225, 233, 256-276
src/kernels/compat.py	8	1	88%	5
src/kernels/deps.py	54	4	93%	58-59, 95, 98
src/kernels/layer/__init__.py	6	0	100%
src/kernels/layer/_interval_tree.py	103	4	96%	23, 52, 147, 150
src/kernels/layer/device.py	48	14	71%	42, 47-49, 91, 96-98, 101, 149, 152, 155-157
src/kernels/layer/func.py	81	7	91%	81, 111, 183, 301, 307, 320, 338
src/kernels/layer/globals.py	5	0	100%
src/kernels/layer/kernelize.py	74	8	89%	255, 281, 289-290, 296, 300, 316-318
src/kernels/layer/layer.py	210	16	92%	167, 210, 216, 229, 337, 417-418, 430, 439, 447, 458, 487, 491, 504, 557, 587
src/kernels/layer/mode.py	14	0	100%
src/kernels/layer/repos.py	130	34	74%	27, 33, 36-41, 61-62, 68, 71-74, 88, 92, 101-102, 108, 111-114, 121-122, 128, 131-134, 141-142, 148, 151-154, 235
src/kernels/lockfile.py	71	46	35%	37-104, 108-131
src/kernels/status.py	49	2	96%	23, 81
src/kernels/utils.py	301	55	82%	65, 77-81, 87-88, 218, 222, 225, 287, 295, 334-335, 373, 404, 409, 444, 673, 676, 678, 684, 697-698, 719-731, 735-742, 750, 754-764, 768-775, 813, 817, 836, 838
src/kernels/variants.py	262	19	93%	56, 87, 108, 138, 247-248, 289, 291, 371-378, 384-390, 421-427, 439-445, 534-536
src/kernels/verify.py	88	1	99%	32
TOTAL	1777	274	85%

Updated by the Test kernels workflow on commit c97b27d27dbc420544add10e6134b649318f4a55.

[danieldk]

Great! Let's merge this and then rebuild all kernels-community kernels once the Torch 2.13 RC 15 PR is also merged (working on it).