fix(skills): scope the GitHub import size cap to the skill subpath#394
Open
pjdoland wants to merge 1 commit into
Open
fix(skills): scope the GitHub import size cap to the skill subpath#394pjdoland wants to merge 1 commit into
pjdoland wants to merge 1 commit into
Conversation
_extract_skill applied MAX_EXTRACTED_BYTES to every file in the repo
tarball, but GitHub tarballs always contain the whole repo while only
the subpath subtree gets installed. Any skill hosted in a monorepo
whose total content exceeded 20 MB therefore failed to install with
"Extracted archive exceeds size limit", regardless of the skill's own
size. This hit every skill in an affected managed-skills manifest
identically.
Count sizes and extract members only within the {top_dir}/{subpath}
subtree. This also stops unrelated repo content from being written to
the temp dir and raises a clear "not found" error when the subpath
does not exist in the archive. The unsafe-path check still runs over
every member before the subtree filter, so a hostile name anywhere in
the archive fails the import loudly.
Regression tests pin the monorepo case, the still-enforced cap on the
skill subtree itself (with and without a subpath), sibling-directory
prefix confusion (skills/foobar vs skills/foo), a file-valued subpath,
and unsafe paths outside the subtree.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Installing a skill from a subpath of a large repo fails with
ValueError: Extracted archive exceeds size limit (20 MB)even when the skill itself is tiny._extract_skillcounted every file in the GitHub tarball againstMAX_EXTRACTED_BYTES, but GitHub tarballs always contain the whole repo while only the subpath subtree gets installed. In practice this means any skill hosted in a monorepo larger than 20 MB cannot be installed at all, and in a managed-skills manifest pointing at such a repo every entry fails identically on reconcile. This was hit in production with a skills monorepo where all four skills failed with the same trace throughskill_reconciler.reconcile->install_managed_from_github->stage_skill_from_github->_extract_skill.Solution
Scope both the size count and the extraction to the
{top_dir}/{subpath}subtree. The load-bearing change is the member filter in_extract_skill: members outside the subtree are skipped entirely, so they neither count against the cap nor get written to the temp dir (previously the whole repo was extracted, then everything but the subtree deleted). Two related behaviors were kept deliberately:..segments) still runs over every member before the subtree filter, so a hostile name anywhere in the archive fails the import loudly rather than being silently skipped.Path '<subpath>' not found in repoearly, before any extraction.Side benefits: unrelated repo content is never written to disk during an import, and extraction I/O for a small skill in a large monorepo drops accordingly. The cap semantics for whole-repo skills (no subpath) are unchanged.
Testing
tests/test_skill_github_import.py: the monorepo case (large sibling content no longer trips the cap, and is not extracted), the cap still enforced on the skill subtree itself (with and without a subpath), sibling-directory prefix confusion (skills/foobarmust not matchskills/foo), a subpath naming a file (clean "not found"), and an unsafe path outside the subtree (still rejected).jlpm tsc --noEmit,jlpm lint:check, andjlpm jestgreen (no TypeScript changes).Risks / follow-ups
parse_github_urlpermits.path segments, which now surface as a "not found" error; and duplicate tar entries can raiseOSErrorfromextractallinstead of a cleanValueError(cleanup still runs in both cases).Reported via user support with production stack traces; no GitHub issue exists for this.