Optimize Git.status() performance and fix numstat parsing by michalkahle · Pull Request #1531 · jupyterlab/jupyterlab-git

michalkahle · 2026-09-03T13:46:42Z

Credit to @dualc for diagnosing the bottleneck in #1431. This improves on #1432 - see below.

Problem

Git.status() runs this unconditionally on every status refresh (i.e. all the time):

git diff --numstat -z --cached 4b825dc642cb6eb9a060e54bf8d69288fbee4904

That diffs the whole index against the empty tree, so git reads every tracked file to produce line counts. On our 12,331-file / 1.4 GB repository it takes over 20 s on cold page cache, ~8 s once warm.

The point of all this is to find out if the files in git status output are binary or text.

Change

Two commits.

1. Compare against HEAD instead of the empty tree. git diff --numstat -z --no-renames HEAD reports only files that actually differ.

--no-renames forces renames to be reported as a delete plus an add. Keeps every line a plain three-field tab-separated record.

A repository with no commits has no HEAD, so the diff exits non-zero there and falls back to the empty-tree comparison.

2. Split numstat records from the left. Independent latent bug found while looking at the code. -z emits paths raw and unquoted, and filenames may contain tabs, so the last tab is not reliably the field separator:

line = "1\t0\tfile with\ttab.py"
diff, name = line.rsplit("\t", maxsplit=1)
diff # '1\t0\tfile with'
name # 'tab.py'

It fails silently rather than crashing and the result would be subtle: binary file with tab in its filename will be treated as text. But it is worth fixing IMO.

How is this different from #1432

Same core idea but three concrete differences:

	#1432 (dualc)	#1531 (michalkahle)
command	`git diff --numstat -z --cached`	`git diff --numstat -z --no-renames HEAD`
compares	index vs HEAD	working tree vs HEAD
no-HEAD case	`_is_first_commit()` probe every refresh	fallback on non-zero exit

--no-renames. Without it, rename detection turns on, and --numstat -z emits a rename as 0\t0\t\0\0\0 — three NUL records, two with no tab which breaks the parser.
HEAD, not --cached. Index-vs-HEAD only sees staged changes, so working-tree-only modifications are not covered. Working-tree-vs-HEAD matches main everywhere and gains a flag on staged deletions.
Exit code, not a probe. git rev-parse --verify HEAD on every refresh starts a subprocess to handle a state that exists only before a repo's first commit. Letting the diff fail costs the extra call only there.

Worth saying plainly: @dualc identified the bottleneck and had the right instinct about HEAD. This PR is that idea with some modifications.

Testing

packages/core/tests/test_status.py:

changed to mirror conditional invocation of git diff.
Case of a filename with tab added.

github-actions · 2026-09-03T13:46:53Z

👈 Launch a Binder on branch michalkahle/jupyterlab-git/fix%2Fstatus-pathspec

krassowski · 2026-09-03T14:01:54Z

+                "--cached",
+                "4b825dc642cb6eb9a060e54bf8d69288fbee4904",
+                "--",
+                *changed_paths,


This will work fine on Linux which has large ARG_MAX but I would expect this to fail quickly on Windows or macOS which have lower limits. Quick estimate from Gemini (I would not trust exact numbers other than 2MB Ubuntu that I verified but the order of magnitude sounds right):

Operating System Total Argument Space (ARG_MAX) Est. Max Files (at 50 chars/path)

Ubuntu / Modern Linux ~2,097,152 bytes (2 MB) ~40,000 files

macOS ~262,144 bytes (256 KB) ~5,000 files

Windows (cmd.exe) 8,191 characters ~160 files

Windows (PowerShell) 32,766 characters ~650 files

Maybe we could have a fast path using *changed_paths but also a fallback to the old way of doing things?

Yes, you are right. I'll try to fix this.

I've changed the implementation and description. Please reconsider.

michalkahle · 2026-09-07T06:57:09Z

I've changed the implementation and description. Please reconsider.

`Git.status()` ran `git diff --numstat -z --cached <empty-tree>` on every status refresh. Comparing the stage to the empty tree makes every tracked file an addition, so git reads every tracked blob to produce line counts. On a 12,331-file / 1.4 GB repository that is ~8.1 s, against 75 ms for the `git status` it accompanies. The cost compounds: `__execute` serializes on a process-wide lock and the frontend polls status every `refreshInterval` (3000 ms default), so the command never drains and every user action queues behind it until it hits `git_command_timeout`. Diff against HEAD instead, which only reports files that actually differ: git diff --numstat -z --no-renames HEAD 0.091 s git diff --numstat -z --cached <empty> 8.095 s `--no-renames` is required for correctness, not speed. With rename detection on, `--numstat -z` emits a rename as `0\t0\t\0<from>\0<to>\0` -- three NUL-separated records, two of which contain no tab -- and the parser below would fail to unpack them. Forcing renames to be reported as a delete plus an add keeps every record a plain three-field record, which is the invariant the empty-tree comparison used to provide. Comparing against HEAD rather than the index also widens coverage slightly: files deleted from the index now resolve to a flag instead of None. Nothing loses a flag. Untracked files remain None, as they were, being in neither the index nor HEAD. A repository with no commits has no HEAD, so the diff exits non-zero there; fall back to the empty-tree comparison in that case. That costs a second subprocess only before a repository's first commit, rather than spending a `git rev-parse --verify HEAD` on every refresh. Refs jupyterlab#1431

`--numstat -z` emits paths raw and unquoted (without `-z` git quotes them), and a filename may contain a tab, so the last tab is not reliably the field separator. `line.rsplit("\t", maxsplit=1)` therefore splits in the wrong place for such paths: line = "1\t0\tfile with\ttab.py" rsplit("\t", 1) -> diff = "1\t0\tfile with", name = "tab.py" split("\t", 2) -> ["1", "0", "file with\ttab.py"] The failure is silent rather than a crash: `startswith` only inspects the first three characters, so the flag itself stays correct, but `are_binary` is keyed on a truncated path, the later lookup misses, and the file falls back to `is_binary: None` -- a binary file with a tab in its name is then offered a text diff. Split from the left with `maxsplit=2`, which is what the format actually guarantees, and compare the two count columns directly now that they are separate values. This matches the sibling numstat parser in `diff()`, which already splits from the left.

michalkahle · 2026-09-09T07:59:11Z

If there is anything I could do to move this forward, please let me know. I'd be happy to do it. I sincerely believe this is a significant improvement and not only for gigantic repositories.

krassowski · 2026-09-09T10:03:50Z

Thanks, I will take a look. as a general note, fewer force pushes make it easier to review.

Co-authored-by: Michał Krassowski <5832902+krassowski@users.noreply.github.com>

krassowski reviewed Sep 3, 2026

View reviewed changes

krassowski added the enhancement label Sep 3, 2026

michalkahle force-pushed the fix/status-pathspec branch from 5984ecb to 59c14b8 Compare September 4, 2026 14:02

michalkahle added 2 commits September 9, 2026 09:55

michalkahle force-pushed the fix/status-pathspec branch from 59c14b8 to f20e84f Compare September 9, 2026 07:55

krassowski reviewed Sep 14, 2026

View reviewed changes

Comment thread packages/core/jupyterlab_git_core/git.py Outdated

michalkahle and others added 2 commits September 16, 2026 10:10

fix: terminate git diff with -- to handle files named HEAD

6136d35

Co-authored-by: Michał Krassowski <5832902+krassowski@users.noreply.github.com>

Merge branch 'main' into fix/status-pathspec

90ea3e2

michalkahle requested a review from krassowski September 16, 2026 08:17

Navigation Menu

Optimize Git.status() performance and fix numstat parsing - #1531

Optimize Git.status() performance and fix numstat parsing#1531
michalkahle wants to merge 4 commits into
jupyterlab:mainfrom
michalkahle:fix/status-pathspec

michalkahle commented Sep 3, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 3, 2026

Uh oh!

krassowski Sep 3, 2026

Uh oh!

michalkahle Sep 4, 2026

Uh oh!

michalkahle Sep 4, 2026

Uh oh!

michalkahle commented Sep 7, 2026

Uh oh!

michalkahle commented Sep 9, 2026

Uh oh!

krassowski commented Sep 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Operating System	Total Argument Space (ARG_MAX)	Est. Max Files (at 50 chars/path)
Ubuntu / Modern Linux	~2,097,152 bytes (2 MB)	~40,000 files
macOS	~262,144 bytes (256 KB)	~5,000 files
Windows (cmd.exe)	8,191 characters	~160 files
Windows (PowerShell)	32,766 characters	~650 files

Conversation

michalkahle commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Change

How is this different from #1432

Testing

Uh oh!

github-actions Bot commented Sep 3, 2026

Uh oh!

krassowski Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

michalkahle Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

michalkahle Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

michalkahle commented Sep 7, 2026

Uh oh!

michalkahle commented Sep 9, 2026

Uh oh!

krassowski commented Sep 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

michalkahle commented Sep 3, 2026 •

edited

Loading