Skip to content
View shaurya416's full-sized avatar
🚀
building
🚀
building
5 followers · 0 following

Achievements

Achievement: Pull Shark

Achievements

Achievement: Pull Shark

Highlights

Block or report shaurya416

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Content in all repositories owned by your account will be closed.
Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. Markdown is supported. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
shaurya416/README.md

Shaurya Singh, Seattle, cs at UW, class of '27. A route through eight stops, drawn as icons: brains, quantum, classrooms, street safety, buildings, underwater robot, graders, music. A terminal: $ grep -rl "same field" ~/built/ prints no matches.

shauryasingh.com

i find what hurts, then build at it. so far: brains, quantum noise, classrooms, street safety, buildings, an underwater robot, and the code that grades things.

grading the graders

the habit: i read the code that decides whether an answer is right (eval harnesses, classroom autograders, forecasting and fairness metrics) and file what i find.

Since 2026-09-23: 238 issues filed in 229 repositories, 15 areas; 29 fixed by a merged pull request, 26 by other people and 3 by me; median 235.9 hours from filing to the merged pull request, over 26 fixes by others; 32 pull requests to other people's repositories, 3 merged and 25 open.

238 issues drawn as stars in 15 clusters, one per area: eval harnesses 42, judges & RAG 29, RL rewards 22, agent evals 21, metric libs 20, eval platforms 19, fairness & UQ 18, code & math 13, data pipelines 12, safety 12, autograders 8, forecasting 8, retrieval 7, explainability 4, speech & audio 3. Lit: 29 fixed by a merged pull request, 26 by other people and 3 by me (ringed). Rings: 8 closed without a fix. Dots: 201 open. 229 repositories in 15 areas.

one example, small enough to read: scoreprobe on a two-line grader, before and after a one-line fix.

scoreprobe on a grader that returns reference.lower() in answer.lower(): 5 of 9 checks reproduce, for example 'not Paris' earns credit against 'Paris'. With reference.lower() == answer.lower() instead: 0 of 9.

three more panels and the one-defect table

Issues filed per week in UTC, 2026-09-23 to 2026-11-02: 180 58 0 0 0 0 0. 238 filed, 29 fixed by a merged PR. State of each of the 238 issues: fixed by someone else's PR 26, by own PR 3, closed without a merged PR 8, open 201. Median 235.9 hours from report to a merged fix by someone else. 238 issues across 229 repositories in 15 areas; largest: eval harnesses & benchmarks, 42.

eight of the fixes share one defect: a missing, empty or undefined result counted as a score.

report, fixed bywhat was wrong
otter-grader#1020
shaurya416 #1021		grading produced no results and the log still said all tests passed
AReaL#1757
MohammadHijjawi97 #1758		a reply with no answer scored 0.9 against a "None" placeholder gold
evalscope#1783
Yunnglin #1785		a blank answer against a blank reference scored full credit
mlxtend#1202
MohammadHijjawi97 #1203		mcnemar returned p = 0.0 by default when there were no discordant pairs
utilsforecast#285
vgvr0 #286		smape scored a missing forecast as a perfect 0
trl#7364
adithya-s-k #7467		OpenReward's default GRPO reward could not tell never scored from scored 0.0
Curator#2443
1fanwang #2444		an empty document scored 1.0 on the repetition filter and was kept
darts#3221
adigulalkari #3223		grid search returned a NaN-scored combination as the best when it came first in the grid
28 merged pull requests closed 29 of the issues

Report to fix, drawn as a logic analyzer. Top channel: one edge per issue filed, 238 in all; filed per week: week of 09-21 180, week of 09-28 58, week of 10-05 0, week of 10-12 0, week of 10-19 0, week of 10-26 0, week of 11-02 0. Below, one channel per fixed issue (29), high from the report until the merged pull request that closed it: giskard-oss#2840 834.5 h, evalscope#1774 115.0 h, lmms-eval#1546 235.1 h, OSWorld#597 614.5 h, kaggle-benchmarks#217 187.5 h, Curator#2443 316.2 h, opik#8519 157.7 h, trl#7364 296.2 h, data-juicer#1077 101.5 h, EasyEdit#716 4.7 h, PrairieLearn#15852 26.8 h, otter-grader#1020 31.1 h, AReaL#1757 56.7 h, trulens#2823 109.5 h, agenta#7149 316.2 h, AutoRAG#1707 86.7 h, agentic_security#350 541.1 h, ai-gateway#447 532.8 h, verifywise#4783 776.5 h, evalscope#1783 16.6 h, MAPIE#993 236.8 h, mlxtend#1202 30.8 h, shapash#777 245.8 h, darts#3221 241.4 h, utilsforecast#285 105.9 h, aeon#3880 341.9 h, ir_measures#83 2.5 h, statsforecast#1247 27.9 h, rouge#77 780.9 h. Three are mine: data-juicer#1077 101.5 h, otter-grader#1020 31.1 h, ir_measures#83 2.5 h. Median wait for fixes by others: 235.9 h. As of 11-02 10:26 utc.

merged 2026, utc pull request by closes hours
09-24 zjunlp/EasyEdit#717 TengJiao33 #716 4.7
09-25 PrairieLearn/PrairieLearn#15854 jonatanschroeder #15852 26.8
09-25 ucbds-infra/otter-grader#1021 shaurya416 #1020 31.1
09-27 areal-project/AReaL#1758 MohammadHijjawi97 #1757 56.7
09-28 datajuicer/data-juicer#1080 shaurya416 #1077 101.5
09-28 modelscope/evalscope#1785 Yunnglin #1774, #1783 115.0 / 16.6
09-29 Marker-Inc-Korea/AutoRAG#1722 vkehfdl1 #1707 86.7
09-29 rasbt/mlxtend#1203 MohammadHijjawi97 #1202 30.8
09-30 truera/trulens#2836 feiiiiii5 #2823 109.5
09-30 comet-ml/opik#8528 desty #8519 157.7
09-30 terrierteam/ir_measures#84 shaurya416 #83 2.5
10-01 Kaggle/kaggle-benchmarks#220 yyl #217 187.5
10-01 Nixtla/statsforecast#1248 vgvr0 #1247 27.9
10-02 Nixtla/utilsforecast#286 vgvr0 #285 105.9
10-03 EvolvingLMMs-Lab/lmms-eval#1553 longzhenren #1546 235.1
10-06 huggingface/trl#7467 adithya-s-k #7364 296.2
10-07 NVIDIA-NeMo/Curator#2444 1fanwang #2443 316.2
10-07 scikit-learn-contrib/MAPIE#994 SashaMIT #993 236.8
10-08 unit8co/darts#3223 adigulalkari #3221 241.4
10-08 MAIF/shapash#794 aminemanai2003 #777 245.8
10-08 Agenta-AI/agenta#7161 unfitcoder101 #7149 316.2
10-12 aeon-toolkit/aeon#3887 anshuman83-40 #3880 341.9
10-18 ferro-labs/ai-gateway#455 MitulShah1 #447 532.8
10-18 msoedov/agentic_security#352 msoedov #350 541.1
10-19 xlang-ai/OSWorld#614 Timothyxxx #597 614.5
10-28 Giskard-AI/giskard-oss#2904 kevinmessiaen #2840 834.5
10-28 verifywise-ai/verifywise#5147 MuhammadKhalilzadeh #4783 776.5
11-02 pltrdy/rouge#78 DataDetective-Neel #77 780.9

238 dots, one per issue in the order filed; the 29 fixed by a merged pull request are lit.

field guide

A flawless 8-qubit device sampling a uniform target: one simulated run of 1,024 shots (splitmix64, seed 0) as 256 bars against the ideal of 4 each, bars clipped at 4 times the ideal. This run scores 0.9334 and leaves 3 outcomes empty. The curve is shotfloor's closed form E[BC]², a lower bound on the mean Hellinger fidelity: 0.3807, 0.5986, 0.8056 at 128, 256, 512 shots; 0.9239 at 1,024 shots (0.92 at 1,024 on the curve); by that bound, shotfloor puts 0.95 at 1,425 shots; 1.0 only at infinite shots. The curve runs from 64 to 65,536 shots.

one line per world i've wandered into.

  • brains · four essays, taught to myself in public, optogenetics to connectomics (2022–23).
  • quantum · shotfloor: the score a perfect quantum computer gets from shot noise alone. 8 qubits, uniform target, 1,024 shots: about 0.92 hellinger fidelity, not 1.
  • classrooms · co-founded SkillTern, its CTO (2022–23): computer science for 350+ students, built to run without me.
  • street safety · TurtleShell: K-means over 10,000+ LAPD crime datapoints and an iOS SOS app in Swift. Microsoft for Startups, then shut down on purpose (oct 2023).
  • buildings · a cabin designer that has to pass the Seattle residential code: rule engine, NSGA-II over cost, space and comfort, IFC export. runs locally, so no link.
  • earlier · H2OAquatics, an autonomous underwater vehicle aimed at ocean acidification; a co-written policy paper on phishing, runner-up at WatGov (Waterloo, 2023).
  • off the clock · music. made often, released never.

238 dots, one per issue in the order filed; the 29 fixed by a merged pull request are lit.

things i made

small python tools, each about one thing that quietly goes wrong. none is public yet, so no links: each name becomes one the day its repository is.

8 small python tools as cards on a shelf, each with an icon, under three headings. tests and repos: tolwatch, pytest plugin: flags allclose, isclose and approx checks that can't fail, or are far looser than they need to be; ziptrace, names the zip() and map() calls that silently dropped data to a length mismatch; whymodified, git status says modified and nothing changed: says why, and the one command that clears it. science data: shotfloor, the score a flawless quantum computer gets at a given shot count, and the shots it needs to reach a target. grading code: scoreprobe, feeds a grader the inputs known to slip through (a blank answer, "not Paris" against "Paris", 4200 against 42) and reports which earned credit; normladder, is "1,000" the same answer as "1000"? 8 of 14 answer-matching rules from lm-eval, HELM and SQuAD say yes; it shows which, and why; mcqlint, lints multiple-choice eval sets: keys that name no option, duplicate options, leaked answers, position bias; tabverify, recomputes the averages, deltas and bold in results tables at the printed precision, and flags what no rounding explains.

commit weather

A 3D contribution calendar for the past year, drawn by a GitHub Action from the public profile calendar, with a radar of commits, issues, pull requests, reviews and repositories and a language chart. On a phone it is a picture of the calendar's shape; its small labels are not meant to be read there. Its totals are the calendar's own.

A snake eats the public profile calendar of the past year, once, and leaves the grid as it found it.

drawn daily by two GitHub Actions from the public profile calendar: the shape of the year, not numbers to read. the totals in the 3D picture are the calendar's own.

method
  • counts are total_count from GitHub search at 2026-11-02 10:26 utc: author:shaurya416 is:public is:issue (238: 201 open, 37 closed) and author:shaurya416 is:public is:pr (32: 3 merged, 25 open, 4 closed unmerged; 32 to other people's repositories), excluding the owner's own organisations.
  • fixed means the issue is closed and a pull request in its repository that references it was merged before it closed; an issue closed as not planned never counts. all 37 closed issues were traced: 29 fixed, 8 closed without a merged pull request. the 25 pull requests by others come from 23 people.
  • an open issue whose fix did not close it is not counted, so fixed is a lower bound.
  • hours run from the issue being filed to the pull request merging.
  • areas: each of the 229 repositories is put in one of 15 areas by hand. that, the four kinds of code the side-quest sentence names, the sentence above the defect table and its one line per row are the hand-written inputs to the side quest.
  • the scoreprobe panel is a captured run of scoreprobe on the code it shows. its first public release is pending; with its source, python3 build.py capture-probe --scoreprobe-src PATH reruns the panel.
  • weeks start on monday, utc. a week with nothing is drawn as a mark, not left blank.
  • regenerate (python 3.12 or newer): python3 build.py fetch reads the api (it needs GITHUB_TOKEN), then python3 build.py renders. same input, same bytes.
  • the quantum card is computed, not measured: the curve is shotfloor's closed form E[BC]², a lower bound on the mean hellinger fidelity, and the bars one simulated flawless run (splitmix64, seed 0).
  • the header's and the shelf's icons are icons.
  • the constellation, divider and report-to-fix pictures are drawn from the same data as the tables, and every number on them is in CHECKS.md.
  • the 3D graph and the snake are drawn by two GitHub Actions pinned to full commit SHAs, from the public profile calendar; their totals are the calendar's.

side-quest numbers: github api, as of 2026-11-02 10:26 utc.

Popular repositories Loading

  1. open-instruct open-instruct Public

    Forked from allenai/open-instruct

    AllenAI's post-training codebase

    Python

  2. SkyRL SkyRL Public

    Forked from NovaSky-AI/SkyRL

    SkyRL: A Modular Full-stack RL Library for LLMs

    Python

  3. slime slime Public

    Forked from THUDM/slime

    slime is an LLM post-training framework for RL Scaling.

    Python

  4. LiveBench LiveBench Public

    Forked from LiveBench/LiveBench

    LiveBench: A Challenging, Contamination-Free LLM Benchmark

    Python

  5. SWE-bench SWE-bench Public

    Forked from SWE-bench/SWE-bench

    SWE-bench: Can Language Models Resolve Real-world Github Issues?

    Python

  6. open-r1 open-r1 Public

    Forked from huggingface/open-r1

    Fully open reproduction of DeepSeek-R1

    Python