perf: -prod builds write snapshots in quadratic time

[nepinhum]

A release build writes commits into the index far slower than a debug build of the same source and the gap grows with the repository. Debug stays linear. Under -prod, doubling the commits roughly quadruples the time.

I ran into this while measuring #5 and #6 and parked it.

Numbers

One repository, one author, gitlife sync into a fresh database:

commits	debug	-prod
2,000	0.03 s	0.26 s
4,000	0.06 s	0.82 s
8,000	0.10 s	2.43 s
16,000	0.19 s	8.56 s

Reading the history isn't the problem. Walking 16,000 commits takes about 0.1 s in both builds; the rest is writing.

It also reproduces without git or the syncer, as a test inside index that calls write_snapshot on synthetic commits:

commits	debug	-prod
4,000	84 ms	790 ms
8,000	186 ms	2,270 ms
16,000	381 ms	9,000 ms
32,000	789 ms	39,229 ms

Reproducing

# a repository with 16,000 commits and one author
mkdir r && cd r && git init -q
python3 -c '
n = 16000
print("reset refs/heads/master")
for i in range(n):
    print("commit refs/heads/master")
    print(f"mark :{i+1}")
    print(f"author Test <test@example.com> {1600000000+i} +0000")
    print(f"committer Test <test@example.com> {1600000000+i} +0000")
    m = f"commit {i}"; print(f"data {len(m)}"); print(m)
    if i: print(f"from :{i}")
    b = f"line {i}"; print("M 100644 inline f.txt"); print(f"data {len(b)}"); print(b)
print("done")
' | git fast-import --quiet
cd ..

v . -o gitlife-debug
v . -o gitlife-prod
for b in gitlife-debug gitlife-prod; do
  export GITLIFE_CONFIG_DIR=$(mktemp -d) GITLIFE_STATE_DIR=$(mktemp -d) GITLIFE_CACHE_DIR=$(mktemp -d)
  ./$b source add local "$PWD/r" >/dev/null
  ./$b sync | tail -1
done

Ruled out

• SQLite. The same statements through the sqlite3 CLI take 0.33 s for 8,000 commits, unprepared. Both builds link the same libsqlite3.
• Batching. Batch sends the same rows in the same number of calls in both builds: 8,000 rows in 4 calls.
• Garbage collection. GC_DONT_GC=1 and gc_disable() change nothing and RSS stays around 32 MB.
• Allocation in general. A small program that builds and keeps 128,000 rows of 14 strings runs 2.5 times faster under -prod.
• Threads. --jobs 1, 2 and 4 all take the same time.
• LTO. -prod -cflags -fno-lto is just as slow.
• -O3. A non prod build with -cflags -O3 is fast: 0.09 s for 8,000.

What's left

-prod also adds -DNDEBUG -DNO_DEBUGGING and links a different GC: thirdparty/libgc/gc.o built with ALL_INTERIOR_POINTERS, where other builds use the prebuilt libgc under thirdparty/tcc/lib. I haven't isolated any of those yet.

Open question

WORKING ON.