nix-builder: wire up kernel dependencies by danieldk · Pull Request #786 · huggingface/kernels

danieldk · 2026-08-25T15:55:01Z

The new lock-kernel-deps command creates a kernels.lock file with dependency locks and Nix output hashes. These are used to make fixed-output derivations of kernel dependencies.
fetchKernelDeps is a function that takes a kernels.lock file and generates a JSON file that is compatible with the KernelPaths data structure from kernels-data.
The kernel builder derivations take a new argument kernelDeps, which is used to pass the KernelPaths JSON file.
get-kernel-check is updated to read the JSON file and construct a KernelPathsResolver that is used by kernels to resolve the dependencies from the Nix store.

Aside from that, a new example kernel-deps to test building and loading with dependencies.

I have also split the deps.nix file into python-deps.nix and cpp-deps.nix for clarity.

- The new `lock-kernel-deps` command creates a `kernels.lock` file with dependency locks and Nix output hashes. These are used to make fixed-output derivations of kernel dependencies. - `fetchKernelDeps` is a function that takes a `kernels.lock` file and generates a JSON file that is compatible with the `KernelPaths` data structure from `kernels-data`. - The kernel builder derivations take a new argument `kernelDeps`, which is used to pass the `KernelPaths` JSON file. - `get-kernel-check` is updated to read the JSON file and construct a `KernelPathsResolver` that is used by `kernels` to resolve the dependencies from the Nix store. Aside from that, a new example `kernel-deps` to test building and loading with dependencies.

HuggingFaceDocBuilderDev · 2026-08-25T15:56:18Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

github-actions · 2026-08-25T16:05:01Z

Coverage report — `kernels/`

Measured on: Python 3.10 / Torch 2.13.0.
Other CI configurations are not included in this number.
Hardware-gated code paths (ROCm/XPU/NPU/Darwin/Windows) are excluded or unreachable on the Linux+CUDA runner.

Total coverage: 86.2% — threshold: 80% — ✅

Per-file breakdown

Name	Stmts	Miss	Cover	Missing
src/kernels/__init__.py	14	0	100%
src/kernels/_system.py	6	1	83%	10
src/kernels/_versions.py	78	9	88%	47, 53-54, 57-58, 97, 119, 130, 136
src/kernels/backends.py	212	62	71%	40, 44, 48-51, 68, 90, 108, 117, 121, 125-127, 148, 157, 161, 165-167, 188, 199, 201, 208-211, 224, 228, 232-252, 260, 283-303
src/kernels/compat.py	8	1	88%	5
src/kernels/deps.py	67	1	99%	57
src/kernels/hf_hub.py	62	3	95%	18, 20, 114
src/kernels/importer.py	50	3	94%	110, 114, 117
src/kernels/install.py	21	7	67%	78-102
src/kernels/layer/__init__.py	6	0	100%
src/kernels/layer/_interval_tree.py	103	4	96%	23, 52, 147, 150
src/kernels/layer/device.py	48	14	71%	42, 47-49, 91, 96-98, 101, 149, 152, 155-157
src/kernels/layer/func.py	86	6	93%	90, 121, 193, 313, 338, 366
src/kernels/layer/globals.py	5	0	100%
src/kernels/layer/kernelize.py	74	8	89%	255, 281, 289-290, 296, 300, 316-318
src/kernels/layer/layer.py	215	15	93%	177, 222, 247, 365, 445-446, 458, 467, 475, 486, 515, 519, 532, 585, 615
src/kernels/layer/mode.py	14	0	100%
src/kernels/layer/repos.py	144	42	71%	27, 33, 36-43, 63-64, 70, 73-76, 90, 94, 103-104, 110, 113-116, 123-124, 130, 133-136, 143-144, 150, 153-156, 163-164, 170, 173-176, 257
src/kernels/load.py	61	2	97%	297, 335
src/kernels/locking.py	89	64	28%	35-83, 91-98, 102-125, 137, 152-159, 165-175, 179-186
src/kernels/python_deps.py	58	6	90%	59-60, 64-65, 101, 104
src/kernels/resolver.py	135	2	99%	174, 180
src/kernels/status.py	49	2	96%	23, 81
src/kernels/variants.py	278	19	93%	64, 95, 116, 146, 255-256, 298-301, 303, 387-394, 400-406, 437-443, 455-461
src/kernels/verify.py	88	1	99%	32
TOTAL	1971	272	86%

Updated by the Test kernels workflow on commit d26104d01170b442d71839dda987949f38a95c6d.

sayakpaul · 2026-08-27T10:13:01Z

Haven't started reviewing yet but a question on

get-kernel-check is updated to read the JSON file and construct a KernelPathsResolver that is used by kernels to resolve the dependencies from the Nix store.

If kernels were to solely lie on the consumer side (no builder component), how is this surfaced?

sayakpaul

Left some small comments. Looks exceptionally clean!

sayakpaul · 2026-08-27T10:22:35Z

+kernel-depends = [
+    { repo-id = "kernels-community/einops", version = 1 },
+]


Do you think it's worth having one more depends here just to test that we're not limited to just one kernel dependency?

Additionally, is this expected to work seamlessly with python-depends? I know we want to get rid of that completely. But till the time we don't, how is that supposed to play with kernel-depends?

I can add another.

There should be no interaction with python-depends at all.

sayakpaul · 2026-08-27T10:24:47Z


 > [!TIP]
 > Refer to the [kernel requirements' page](../kernel-requirements.md) to
-> get an idea of what is expected from the kernel structure and content. 


Have we already made modifications to our docs to talk about kernel dependency?

No, still intentionally not there, we should play a little with it before making it an 'official' feature.

sayakpaul · 2026-08-27T10:31:03Z

@@ -0,0 +1,47 @@
+{


Very nice segregation!

sayakpaul · 2026-08-27T10:32:51Z

+from pathlib import Path
+
+from kernels.load import get_kernel_with_resolver
+from kernels.hf_hub import _get_hf_api
+from kernels.resolver import KernelPathsResolver, RepoPathsResolver, SequentialResolver
+from kernels_data import KernelDependency, KernelPaths, KernelVersion
+
+with open('${kernelDeps}') as f:
+  kernel_paths = KernelPaths.from_json(f.read())
+
+kernel = KernelDependency(repo_id='${out}', version=KernelVersion.Version(0))
+resolvers = [
+  RepoPathsResolver(local_kernels={'${out}': Path('${out}')}),
+  KernelPathsResolver(kernel_paths=kernel_paths)
+]
+
+get_kernel_with_resolver(
+  api=_get_hf_api(),
+  backend=None,
+  kernel=kernel,
+  resolver=SequentialResolver(resolvers=resolvers)
+)


Can we keep it in a separate Python file?

Yep, fixed!

sayakpaul · 2026-08-27T10:34:17Z

+`fetchFromHuggingFace` fixed-output derivation.
+"""
+
+import argparse


(nit): is there any merit in introducing a directory called py_utils here and keeping all the Python utilities inside it?

I think support files for Nix functions/derivations like this should be in the directory of the derivation itself. This also keeps things close to upstream nixpkgs.

danieldk · 2026-08-27T11:54:43Z

Haven't started reviewing yet but a question on

get-kernel-check is updated to read the JSON file and construct a KernelPathsResolver that is used by kernels to resolve the dependencies from the Nix store.

If kernels were to solely lie on the consumer side (no builder component), how is this surfaced?

It is not. In the future, when we open up the resolver API, downstream users can use the API to use and combine resolvers directly like we do in get-kernel-check.

sayakpaul reviewed Aug 27, 2026

View reviewed changes

kernel-deps: add another dependency as an example

ce0151d

nix-builder: move Python part of get-kernel-check-hook to separate file

d26104d

danieldk temporarily deployed to testpypi August 27, 2026 12:07 — with GitHub Actions Inactive

sayakpaul approved these changes Aug 27, 2026

View reviewed changes

danieldk merged commit 8c59f17 into main Aug 27, 2026
55 checks passed

danieldk deleted the kernel-deps-nix branch August 27, 2026 12:45

Conversation

danieldk commented Aug 25, 2026

Uh oh!

HuggingFaceDocBuilderDev commented Aug 25, 2026

Uh oh!

github-actions Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage report — kernels/

Uh oh!

sayakpaul commented Aug 27, 2026

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danieldk commented Aug 27, 2026

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented Aug 25, 2026 •

edited

Loading

Coverage report — `kernels/`