webgpu-fly runs a whole-animal Drosophila nervous system inside a web browser with no installation and no server. The FlyWire FAFB whole-brain connectome (139,255 neurons, ~15 million synaptic connections) and the Janelia MANC ventral-nerve-cord connectome (23,188 neurons, 5.2 million connections) are each simulated as leaky integrate-and-fire (LIF) networks in fused WebGPU compute kernels — gather, integrate, threshold and reset in a single kernel, with presynaptic-neurotransmitter signs pre-baked into the connection weights so the inner loop never branches on excitatory/inhibitory type.
The brain's descending command neurons drive the spinal cord by cell-type name match (the same named cell on both sides of the brain–VNC boundary), and the spine's 369 leg motor neurons are averaged into a walking magnitude and a turn bias that scale a hand-written tripod gait, which in turn actuates a physically simulated 67-body, 111-actuator TuragaLab flybody model running in MuJoCo compiled to WebAssembly. The connectome scales that gait; it does not generate the stepping rhythm, which is an analytic sinusoid of simulation time. A 64×16 retina rendered each frame from the fly's own head pose is fed back into the brain's optic neurons, closing a sensorimotor loop. An optional trained reinforcement-learning walking policy (Vaxenburg et al. 2025) runs as a pure-TypeScript forward pass and walks the body from leg actuation and ground reaction alone, with the kinematic assist switched off: 2.004–2.021 cm per simulated second against a 2.0 cm/s command, uprightness +0.997, no capsize across three repetitions, versus 0.032 cm per simulated second with the policy disabled. That path bypasses the brain and the ventral nerve cord entirely — it is the published policy walking the fly, not the connectome. The forward pass is checked element-wise against an independent NumPy re-run of the same extracted weights; that check validates the port's arithmetic, not the assumed layer architecture against the original SavedModel.
The deployment is a game: the player fires real descending neurons with keypresses to steer the fly to a target, and a winning run produces a deterministic, shareable replay URL that re-executes the identical neuron cascade against the same connectome — a brain trace, not a video. Performance is reported honestly: the brain LIF kernel is memory-bandwidth-bound and runs at ~0.25 kHz of biological time on an Apple M2 Pro, benchmarked on the same machine against NEST 3.10 (0.67 kHz) and a hand-written multicore Rust port (0.45 kHz). The original 1 kHz target was unreachable for any of the three on that hardware; the contribution is reachability — a real connectome simulation behind a single URL — not raw throughput. Known limitations (the connectome scaling rather than generating the gait, the closed-loop visual-reflex approximation, kinematic-assist options, the unverified policy architecture) are enumerated in LIMITATIONS.md.", + "description": "
webgpu-fly runs a whole-animal Drosophila nervous system inside a web browser with no installation and no server. The FlyWire FAFB whole-brain connectome (139,255 neurons, ~15 million synaptic connections) and the Janelia MANC ventral-nerve-cord connectome (23,188 neurons, 5.2 million connections) are each simulated as leaky integrate-and-fire (LIF) networks in fused WebGPU compute kernels — gather, integrate, threshold and reset in a single kernel, with presynaptic-neurotransmitter signs pre-baked into the connection weights so the inner loop never branches on excitatory/inhibitory type.
The brain's descending command neurons drive the spinal cord by cell-type name match (the same named cell on both sides of the brain–VNC boundary), and the spine's 369 leg motor neurons are averaged into a walking magnitude and a turn bias that scale a hand-written tripod gait, which in turn actuates a physically simulated 67-body, 111-actuator TuragaLab flybody model running in MuJoCo compiled to WebAssembly. The connectome scales that gait; it does not generate the stepping rhythm, which is an analytic sinusoid of simulation time. A 64×16 retina rendered each frame from the fly's own head pose is fed back into the brain's optic neurons, closing a sensorimotor loop. An optional trained reinforcement-learning walking policy (Vaxenburg et al. 2025) runs as a pure-TypeScript forward pass and walks the body from leg actuation and ground reaction, with the kinematic assist switched off: 2.019 cm per simulated second against a 2.0 cm/s command, versus 0.032 cm per simulated second with the policy disabled. The forward translation is earned that way; the posture is not. A pitch/roll attitude damper runs every substep, and with it disabled the fly capsizes and stops walking — 0.068 cm per simulated second, uprightness -0.87 — so the +0.997 uprightness and the absence of capsizes across three repetitions are products of that damper, not of the policy. That path bypasses the brain and the ventral nerve cord entirely — it is the published policy walking the fly, not the connectome. The forward pass is checked element-wise against an independent NumPy re-run of the same extracted weights; that check validates the port's arithmetic, not the assumed layer architecture against the original SavedModel.
The deployment is a game: the player fires real descending neurons with keypresses to steer the fly to a target, and a winning run produces a deterministic, shareable replay URL that re-executes the identical neuron cascade against the same connectome — a brain trace, not a video. Performance is reported honestly: the brain LIF kernel is memory-bandwidth-bound and runs at ~0.25 kHz of biological time on an Apple M2 Pro, benchmarked on the same machine against NEST 3.10 (0.67 kHz) and a hand-written multicore Rust port (0.45 kHz). The original 1 kHz target was unreachable for any of the three on that hardware; the contribution is reachability — a real connectome simulation behind a single URL — not raw throughput. Known limitations (the connectome scaling rather than generating the gait, the closed-loop visual-reflex approximation, kinematic-assist options, the pitch/roll attitude damper, the unverified policy architecture) are enumerated in LIMITATIONS.md.", "keywords": [ "WebGPU", "WebAssembly", diff --git a/CITATION.cff b/CITATION.cff index 634e036..218bfae 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -42,11 +42,15 @@ abstract: >- rendered from the fly's head pose feeds back into the brain's optic neurons. An optional trained reinforcement-learning walking policy (Vaxenburg et al. 2025) runs as a pure-TypeScript forward pass and walks - the body from leg actuation and ground reaction alone, with the - kinematic assist off: 2.004-2.021 cm per simulated second against a - 2.0 cm/s command, uprightness +0.997, no capsize across three - repetitions, versus 0.032 cm per simulated second with the policy - disabled. That path bypasses the brain and the ventral nerve cord — it + the body from leg actuation and ground reaction, with the kinematic + assist off: 2.019 cm per simulated second against a 2.0 cm/s command, + versus 0.032 cm per simulated second with the policy disabled. The + forward translation is earned that way; the posture is not. A pitch/roll + attitude damper runs every substep, and with it disabled the fly + capsizes and stops walking — 0.068 cm per simulated second, uprightness + -0.87 — so the +0.997 uprightness and the absence of capsizes are + products of that damper, not of the policy. + That path bypasses the brain and the ventral nerve cord — it is the published policy walking the fly, not the connectome. The forward pass is checked element-wise against an independent NumPy re-run of the same extracted weights, which validates the port's diff --git a/LIMITATIONS.md b/LIMITATIONS.md index ecc6ddf..9753b70 100644 --- a/LIMITATIONS.md +++ b/LIMITATIONS.md @@ -15,9 +15,10 @@ the fly.** Roughly 20.3M connectome edges reach the body as about one scalar magnitude plus a turn bias per tick, and those two numbers scale a hand-written `sin(t × 10 Hz)` tripod. §8 is the complete shortcut inventory, with measured numbers for what the body does once the assist is off. A *published RL policy* -(Vaxenburg et al. 2025) does walk the body under physics with the assist off -(§4.1) — that policy bypasses the brain and the spine entirely, so it is not -the fly's own brain doing the walking either. +(Vaxenburg et al. 2025) does translate the body under physics with the assist +off (§4.1), though it stays upright only because a pitch/roll damper outside +the assist holds it there — and that policy bypasses the brain and the spine +entirely, so it is not the fly's own brain doing the walking either. --- @@ -56,31 +57,69 @@ the fly's own brain doing the walking either. simply out of scope for v1. Changing the confidence threshold requires rebuilding `brain.bin`. -## 3. Dynamics validation is qualitative, not quantitative +## 3. Dynamics validation is qualitative — and calibrated, not independent - We check the **shape** of the dynamics: no-input networks go silent (no - runaway), Kenyon cells fire sparsely (~5–15%) under sensory drive, - consistent with Shiu et al. 2024. + runaway), Kenyon cells fire sparsely (~5–15%) under sensory drive. +- **`w_syn` is not taken from Shiu et al. — it is fitted to the number we then + report.** Our own source names their free parameter as 0.275 mV per synapse + (`src/sim.ts:16`); we ship 0.005 (`src/sim.ts:39`), and the comment above it + says why in as many words (`src/sim.ts:25-30`): + + > w_syn is tuned EMPIRICALLY (not from peak-matching) to land KC at the + > canonical 5-15% on Mixed sensory. Alpha synapse integrates each spike + > over ~5ms so the cascade amplifies non-linearly vs old single-step + > direct injection — peak-matching gives way too hot a brain (73% KC). + > 0.005 keeps the dataset's natural cascade strength visible without + > runaway. + + So KC sparsity is a **calibration target, not an independent validation**: + the single free synaptic weight was tuned until the sparsity landed in the + canonical band, and that agreement is then reported as a result. What the + check actually shows is that a `w_syn` exists which puts this network in a + plausible regime — sparse rather than silent or saturated — not that the + network reproduces Shiu et al.'s dynamics. **The sparsity number is not + independent evidence.** - We have **not** done a quantitative, cell-type-resolved firing-rate match - against a published reference simulation across the whole brain. The - sparsity check is the strongest dynamics claim we stand behind. + against a published reference simulation across the whole brain. With + `w_syn` fitted to the sparsity band, there is no check here that is both + quantitative and independent of that fit. ## 4. The brain → spine → body path has documented approximations These are the "honest gaps" from the README, restated as limitations. §8 is the complete list; these four are the ones with the longest history. -1. **Trained RL walker walks — after four port fixes.** With the kinematic - assist explicitly off, the policy drives the body from actuator → ground - reaction alone: **2.004–2.021 cm per simulated second** against a 2.0 cm/s - command, 5.03–5.08 cm of travel per window, uprightness **+0.997** at the - end of the window, and **no capsize in 3 of 3 reps**, with |action| max - ≈ 5.9 while upright — inside flybody's native band (~6). Control condition, +1. **Trained RL walker translates — after four port fixes, and only with the + attitude damper on.** With the kinematic assist explicitly off and the + damper in its shipped on state, nothing on this path writes the body's + translational velocity (`qvel[0..2]`), so the forward motion is genuinely + actuator → ground reaction: **2.004–2.021 cm per simulated second** against + a 2.0 cm/s command, 5.03–5.08 cm of travel per window, uprightness + **+0.997** at the end of the window, and **no capsize in 3 of 3 reps**, with + |action| max ≈ 5.9 while upright — inside flybody's native band (~6). Control condition, policy never enabled and assist still off: 0.032 cm/sim s, uprightness - 0.999 — the fly just stands there, so the locomotion comes from the policy + 0.999 — the fly just stands there, so the translation comes from the policy and not from anything else. In the e2e suite (assist at its default) the walker's displacement is dx = +1.642 cm, previously −1.174 cm. Reproduce - with `.walkbench.mjs`. + with `tools/walkbench.mjs`. + + **The uprightness is not the policy's.** The pitch/roll damper (§8) is a + separate intervention that is not gated by the assist, and it is what keeps + this path on its feet. Measured A/B, trained policy, assist off, 3 reps + each: + + | Pitch/roll damper | cm/sim s | Uprightness at end | Min uprightness | + |---|---|---|---| + | **ON** (as shipped) | 2.019 | **+0.997** | +0.997 | + | **OFF** | 0.068 | **−0.87** | −0.913 | + + Commanded speed is 2.0 cm/s in both. With the damper off the fly capsizes + and stops walking — 0.068 cm/sim s is closer to the 0.032 cm/sim s of a + body with no controller at all (damper on) than to the commanded 2.0. So + the split is: **translation is earned, attitude is not.** A sentence of the + form "walks from leg actuation and ground reaction alone" is false about + this path; the honest version keeps the two halves apart. Until today this item read "does not walk", and it was accurate: with the assist off the fly capsized within ~1.5 s and stayed on its back @@ -107,9 +146,9 @@ the complete list; these four are the ones with the longest history. 59-dim `actuator_activation` observation real instead of all zeros and softens the plant ~5.5× per control tick. - **What this does not change:** the pitch/roll damper (§8) is not gated by - the assist and runs on this path too, so the uprightness numbers above are - not a damper-free result; the forward pass has still never been compared + **What this does not change:** the uprightness numbers above are not a + damper-free result, and the A/B says what the damper-free result is; the + forward pass has still never been compared against the published SavedModel (see the end of §8); and this is still not the connectome walking the fly — it is a published RL policy on a path that bypasses the brain and the spine. @@ -142,7 +181,7 @@ the complete list; these four are the ones with the longest history. synthetic, not a training clip. Real fly mocap from the Vaxenburg deposit is wired in as opt-in (`__walkingRefFromMocap`), and that branch is **not** fixed: it is baked at 50 Hz (`tools/bake_walking_ref.py:46`) and replayed - one frame per 2 ms control tick (`src/physics.ts:971`), a 10× rate error, + one frame per 2 ms control tick (`src/physics.ts:978`), a 10× rate error, and the 65-frame lookahead exceeds the 57-frame trajectory so the window wraps mid-observation. Measured before the item-1 fixes: enabling it takes |action| max from ~7 to 3097–3250 and the fly spins 944–1051° in ~1.8 @@ -180,15 +219,18 @@ the complete list; these four are the ones with the longest history. ## 8. Full shortcut inventory — what actually moves the body -The "Honest mode" button flips exactly three flags (`src/main.ts:669-683`). -The table below has thirteen rows, and nine of them are behind no toggle at -all. This section is all of them — the ones the button covers and the ones it -does not — so the button is not the only place they are disclosed. +The "Honest mode" button flips exactly three flags (`src/main.ts:772-774`). +The table below has fourteen rows, and ten of them are marked **no** — outside +the button entirely. This section is all of them — the ones the button covers +and the ones it does not — so the button is not the only place they are +disclosed. -Nothing here is a claim about the neural simulation. The FlyWire and MANC -connectomes are real, both LIF networks genuinely run on the GPU, and the -stimulus→cascade dynamics are connectome-derived. What follows is about how -the *body* is driven, which is a different and much weaker story. +Almost nothing here is a claim about the neural simulation. The FlyWire and +MANC connectomes are real, both LIF networks genuinely run on the GPU, and the +stimulus→cascade dynamics are connectome-derived. What follows is mostly about +how the *body* is driven, which is a different and much weaker story — one +exception is the deafferentation row, which is about what the cord is never +told. ### The information bottleneck @@ -204,7 +246,7 @@ MANC 23,188 neurons / 5,243,574 edges ↓ and the direction sign is discarded — it comes from the hand-wired 200-neuron synthetic spine main.ts:470 ↓ driveLegs(walk, turn) room.ts:640 - = sin(t · 10 Hz), 18 of 48 leg actuators physics.ts:663-719 + = sin(t · 10 Hz), 18 of 48 leg actuators physics.ts:734-790 ``` **Roughly 20.3 million connectome edges reach the body as about one scalar @@ -220,17 +262,31 @@ geometry (`src/vnc.ts:369-384`). |---|---|---|---| | **Kinematic assist** — writes freejoint `qvel[0]`, `qvel[1]`, `qvel[5]` from `fwdCmd`/`turnCmd`, re-asserted every substep before `mj_step`. Still what moves the body on the **CPG path**; the trained-policy path no longer needs it (§4.1) | ground reaction from leg contact | **yes** (`Physics.kinematicAssistEnabled`) | `src/physics.ts` `step()`, default on at `Physics.kinematicAssistEnabled` | | **Stale CPG command under the policy — resolved** — `fwdCmd`/`turnCmd` are written only by `driveLegs` and the policy path skips `driveLegs`; nothing used to zero them, so the last CPG command kept driving the body throughout "trained walking". The policy path now clears the stale command on takeover | — | n/a — no longer a live shortcut | `src/physics.ts`, `src/room.ts:633-647` | -| **Pitch/roll attitude damper** — `qvel[3] *= 0.85; qvel[4] *= 0.85` per substep, ×0.039 per 2 ms control tick | balance, and the body's ability to tip at all | **no** — it sits before and outside the assist guard | `src/physics.ts:656-659` | +| **Pitch/roll attitude damper** — `qvel[3] *= 0.85; qvel[4] *= 0.85` per substep, ×0.039 per 2 ms control tick. It is what keeps the trained policy upright: damper off, the policy capsizes and drops from 2.019 to 0.068 cm/sim s (§4.1) | balance, and the body's ability to tip at all | **no** — it sits before and outside the assist guard, and the button does not flip it. It is now gated by its own flag (`Physics.attitudeDamperEnabled`, default on) so it can be measured | `src/physics.ts:625`, `:663-666` | | **Boot stimulus drives itself** — science mode auto-runs `STIMULI[0]` at load, saturating the spine to `fwdCmd = 0.99999` for the length of its window; the drive is put back to rest when that window ends, so the residual no longer survives to the first user click. `decayDrive()` is defined and never called (one grep hit, the definition) | a brain whose drive responds to what you click | **no** | `src/main.ts:1284-1288`, `src/main.ts:499` | -| **Tripod CPG is the source of leg timing** — `phase = data.time · 10 Hz`, hard-coded gait constants, 3 of 8 DOFs driven per leg | motor-neuron output setting stance/swing | **no** | `src/physics.ts:718-725`, `:727-783`, actuator cache `:277-284` | +| **The cord is deafferented** — every one of MANC's 6,282 sensory neurons receives zero input. The only drive written into the VNC is the 7 DN types' brain rate; the rest of the `ext` vector stays zero (`src/main.ts:438-452`), and the offline rhythm script drives it the same way (`tools/vnc_rhythm.py:436-437`). Count from `public/vnc.meta.json` (`cell_classes.sensory`) | load, campaniform, hair-plate and chordotonal feedback — which is load-bearing for real insect leg coordination | **no** | `src/main.ts:438-452`, `tools/vnc_rhythm.py:436-437` | +| **Tripod CPG is the source of leg timing** — `phase = data.time · 10 Hz`, hard-coded gait constants, 3 of 8 DOFs driven per leg | motor-neuron output setting stance/swing | **no** | `src/physics.ts:725-732`, `:734-790`, actuator cache `:277-284` | | **Wing motion is hand-written** — 218 Hz analytic stroke, amplitude hard-capped at ×0.2 of flybody's canonical pattern because anything above ~0.25 launches the freejoint body | wing motor neurons (MANC's 66 are read for the readout only) | **no** | `src/physics.ts:558-582`, cap at `:568` | -| **`jumpImpulse` writes `qvel[2]` directly** | leg extension producing a takeoff | **no** | `src/physics.ts:1063-1065`, called from `src/main.ts:495` | -| **Adhesion clamped to 1.0** at init and whenever walk drive < 0.01 — a standing fly is glued to the floor | claw contact and friction holding a stationary fly | **no** | `src/physics.ts:285-291`, `:773-776` | +| **`jumpImpulse` writes `qvel[2]` directly** | leg extension producing a takeoff | **no** | `src/physics.ts:1074-1076`, called from `src/main.ts:495` | +| **Adhesion clamped to 1.0** at init and whenever walk drive < 0.01 — a standing fly is glued to the floor | claw contact and friction holding a stationary fly | **no** | `src/physics.ts:287-293`, `:780-783` | | **Visual-reflex angle bypass** — `turn ∝ retinal angle`, forward speed from retinal area | the brain's optic→DN contralateral cascade | **yes**, but the brain path falls back to the identical law when cascade asymmetry < 0.05, and the code records the cascade's sign as empirically **wrong** for tracking | `src/vnc.ts:369-378`; brain path `:352-368`; sign note `:322-326` | | **Sweep-mode spine bypass** — target lost for 4+ ticks writes a scripted alternating scan turn straight to the body | search behaviour emerging from the brain | **no** | `src/main.ts:998-1005` | -| **Walking reference** — synthetic world-space trajectory by default, now closed-loop against the live body pose (§4.1), but still synthetic rather than a training clip; the mocap opt-in is **still** baked at 50 Hz and replayed at 500 Hz, and the 65-frame lookahead still exceeds the 57-frame clip | the policy's training reference clip | switches to mocap, which measures **worse** (§4.4) | `src/physics.ts:974-1000`, `:971`; `tools/bake_walking_ref.py:46` | +| **Walking reference** — synthetic world-space trajectory by default, now closed-loop against the live body pose (§4.1), but still synthetic rather than a training clip; the mocap opt-in is **still** baked at 50 Hz and replayed at 500 Hz, and the 65-frame lookahead still exceeds the 57-frame clip | the policy's training reference clip | switches to mocap, which measures **worse** (§4.4) | `src/physics.ts:981-1008`, `:978`; `tools/bake_walking_ref.py:46` | | **"Evolve gait (WebGPU ARS)" does not evolve against MuJoCo** — the fitness is a 1-D point-mass rollout with analytic thrust and quadratic drag: no gravity, no ground contact, no body — and the winner is written into the live physics body | optimizing the gait against the actual simulated fly | **no** | `src/shaders/evolve.wgsl:4-6`, `:101-104`; applied at `src/main.ts:1054-1056` | -| **The speed readout displays the assist** — `bodySpeed` reads `qvel[0..1]`, the exact slots the assist writes immediately before `mj_step` | measured locomotion | **no** | `src/physics.ts:1069-1073`, rendered at `src/main.ts:739,746` | +| **The speed readout displays the assist** — `bodySpeed` reads `qvel[0..1]`, the exact slots the assist writes immediately before `mj_step` | measured locomotion | **no** | `src/physics.ts:1079-1083`, rendered at `src/main.ts:739,746` | + +The deafferentation row also scopes what `tools/vnc_rhythm.py` can and cannot +conclude. Any negative rhythm result it produces is a finding about **our +deafferented LIF model of MANC**, not about the MANC connectome. Per-neuron +state in the kernel is membrane voltage, a refractory counter and a two-state +alpha synapse — four read-write buffers besides the spike bitmask +(`src/shaders/lif.wgsl:36-37`, `:39-40`) — with no spike-frequency adaptation, no synaptic depression, no +conduction delay, no rebound current, and, per that row, no sensory feedback +into the cord. The mechanisms a half-centre oscillator relies on to terminate a +burst are therefore absent by construction, and a network built like this +failing to alternate is a property of the reduction before it is evidence about +the wiring. Read such a result as "this model does not oscillate," never as +"MANC has no CPG." One more, about evidence rather than physics: **the walking policy's forward pass is not verified against the published SavedModel.** @@ -301,11 +357,15 @@ Reading it: significant figures, because the other two flags are only read in code paths CPG mode never enters. -The pitch/roll damper is not in that table, because **the project has never -been run with it off.** It has been ×0.005 per CPG render frame for the entire -life of the codebase, in every mode including Honest mode, so no measurement -here — or in any commit message — describes a fly that could tip over. That -baseline is unmeasured. +The pitch/roll damper is not a column in that table, because every one of those +16 runs had it on. It was ×0.005 per CPG render frame in every mode including +Honest mode, and it was not switchable, so **no uprightness figure recorded +anywhere before the `Physics.attitudeDamperEnabled` experiment is damper-free** +— not in this table, not in §4.1's original numbers, not in any commit message. +The damper-off baseline is no longer unmeasured, but it has only been measured +on one path: the trained policy with the assist off (§4.1), where turning the +damper off takes the fly from 2.019 to 0.068 cm/sim s and from +0.997 +uprightness to −0.87. The CPG path has still never been run damper-free. --- diff --git a/README.md b/README.md index 22952c4..9fc4571 100644 --- a/README.md +++ b/README.md @@ -51,7 +51,7 @@ approximation, and every approximation and shortcut is inventoried in - **Not a scientific simulator replacement.** NEST / Brian2 / NEURON are faster, biophysically detailed, and validated. For real fly-brain dynamics research, use those. - **Not biophysically detailed.** Neurons are LIF with a two-state alpha synapse — no ion channels, dendritic compartments, or neuromodulation. -- **Not quantitatively validated** whole-brain. The dynamics check is qualitative (Kenyon-cell sparsity matches Shiu et al. 2024), not a cell-type-resolved rate match. +- **Not quantitatively validated** whole-brain. The one dynamics check is Kenyon-cell sparsity in the canonical 5–15% band — and `w_syn` was tuned to land it there (`src/sim.ts:25-30`), so it is a calibration target, not an independent validation of Shiu et al. 2024. - **Not faster than the reference.** It runs *slower* than real time. The win is reachability, not throughput. See [`LIMITATIONS.md`](./LIMITATIONS.md). @@ -107,7 +107,7 @@ mode, ARS evolver, raw spike-rate log. | **Spine** | [Janelia MANC](https://www.janelia.org/project-team/flyem/manc-connectome) connectome (Takemura et al. 2024) | 23,188 VNC neurons, 5.2M edges, second WebGPU LIF instance | | **Body** | [TuragaLab/flybody](https://github.com/TuragaLab/flybody) MJCF (Vaxenburg et al. 2025, *Nature*) | 67 bodies, 111 actuators, real physics in MuJoCo/WASM | | **Eyes** | offscreen render-to-texture from fly head pose | 64×16 retinal sample fed to brain optic neurons | -| **Walker** | trained RL policy ([Vaxenburg et al. 2025 Figshare](https://janelia.figshare.com/articles/dataset/25309105)) | LayerNormMLP, 741-dim obs → 59 actions, pure-TS forward pass. Walks the body under physics with the kinematic assist **off** — 2.004–2.021 cm per simulated second against a 2.0 cm/s command, upright +0.997, no capsize in 3/3 reps. Checked element-wise against a numpy re-run of the same extracted weights (`tools/verify_walking_policy.py`) — that validates the port's arithmetic, not the assumed architecture against the original SavedModel | +| **Walker** | trained RL policy ([Vaxenburg et al. 2025 Figshare](https://janelia.figshare.com/articles/dataset/25309105)) | LayerNormMLP, 741-dim obs → 59 actions, pure-TS forward pass. Translates the body under physics with the kinematic assist **off** — 2.004–2.021 cm per simulated second against a 2.0 cm/s command. It stays upright (+0.997) only with the pitch/roll damper on; with the damper off it capsizes and covers 0.068 cm/sim s. Checked element-wise against a numpy re-run of the same extracted weights (`tools/verify_walking_policy.py`) — that validates the port's arithmetic, not the assumed architecture against the original SavedModel | Brain → spine wiring is by **cell-type name match** (`DNa01` in the brain is the same neuron as `DNa01` in the VNC — brain side has the soma, VNC side the axon). @@ -123,14 +123,20 @@ the leg phase itself is `sin(sim_time · freq)`. (Caveat: it's a name join acros connectomes, not a reconstructed synaptic bridge — see [`LIMITATIONS.md`](./LIMITATIONS.md) §5.) -The **trained RL walking policy** is a separate path, and it does walk the body -from leg actuation and ground reaction alone: with the kinematic assist off it -covers 2.004–2.021 cm per simulated second against a 2.0 cm/s command and stays -upright (+0.997, no capsize in 3/3 reps), while a run with the policy never -enabled travels 0.032 cm/sim s. That path bypasses the brain and the spine -entirely — it is Vaxenburg et al.'s published policy walking the fly, not the -connectome. [`LIMITATIONS.md`](./LIMITATIONS.md) §4.1 has the numbers and the -four port defects that had to be fixed to get there. +The **trained RL walking policy** is a separate path, and its *translation* is +earned: with the kinematic assist off nothing on that path writes the body's +translational velocity, so the 2.004–2.021 cm per simulated second it covers +against a 2.0 cm/s command comes from leg actuation and ground reaction, while a +run with the policy never enabled travels 0.032 cm/sim s. Its *attitude* is not +earned. A pitch/roll damper that sits outside the assist multiplies the body's +pitch and roll angular velocity by 0.85 every substep (`src/physics.ts:663-666`, +×0.039 per control tick); with that damper off the same policy capsizes and +covers 0.068 cm/sim s. So the +0.997 uprightness is the damper's doing, not the +policy's. +That path also bypasses the brain and the spine entirely — it is Vaxenburg et +al.'s published policy walking the fly, not the connectome. +[`LIMITATIONS.md`](./LIMITATIONS.md) §4.1 has the damper A/B, the numbers, and +the four port defects that had to be fixed to get there. --- diff --git a/index.html b/index.html index dc21784..ed2afd2 100644 --- a/index.html +++ b/index.html @@ -471,21 +471,34 @@
- The brain and spine are joined the way real biology does it: a command
- neuron named DNa01 in the brain is the same cell as
- DNa01 in the spinal cord. Nothing in that path is learned
- or scripted — it is wire for wire from the real animal. Past the
- spinal cord it changes: the motor neurons set a walking speed and a
- turn, and a hand-written leg rhythm does the actual stepping.
+ The brain and the spine are joined by name, not by wire: a
+ command neuron called DNa01 in the brain is matched to the
+ neuron with the same name in the spinal cord. Biology motivates the
+ match — that cell really does keep its body in the brain and send its
+ axon down the cord — but the two maps come from two different flies (a
+ female brain, a male nerve cord), and no reconstructed connection
+ crosses between them. What crosses is a number: the brain neuron's
+ recent firing rate, averaged over its left and right copies and scaled
+ by a constant we picked, dropped into the spinal neurons that share its
+ name. The spiking on either side is real; the handoff is ours. Past the
+ spinal cord it changes again: the motor neurons set a walking speed and
+ a turn, the choice of forward or backward comes from a small hand-wired
+ spine rather than from the real one, and a hand-written leg rhythm does
+ the actual stepping.
There is also a second mode that hands the legs to a - trained walking policy published with the body model. That one - really does walk the fly on its own legs, at the speed you ask it - for, with the push we normally give the body switched off. It is - worth being clear about what it is: a neural network someone trained - with reinforcement learning, not the fly's own brain — the connectome - plays no part in it. + trained walking policy published with the body model. With the + push we normally give the body switched off, the forward motion is + genuinely the legs' doing — the fly covers 2.019 cm per simulated + second against a 2.0 cm/s command, pushing off the ground. Staying + upright is not: a stabiliser bleeds off the fly's pitch and roll on + every physics substep, and with that switched off too it topples over + and gets nowhere (0.068 cm per simulated second). The forward motion + is earned; the balance is propped up. It is also worth being clear + about what the policy is: a neural network someone trained with + reinforcement learning, not the fly's own brain — the connectome plays + no part in it.
diff --git a/package.json b/package.json index 93b4096..9ab1a01 100644 --- a/package.json +++ b/package.json @@ -23,7 +23,7 @@ "bench:brain": "playwright test tests/bench.spec.ts --reporter=list", "build:slim": "npm run build && rm -rf dist/flybody dist/flybody.bundle.bin dist/brain.bin dist/brain.meta.json dist/vnc.bin dist/vnc.meta.json dist/walking-policy.bin dist/walking-obs-norm.bin dist/walking-ref.bin dist/walking-policy-fixtures.json", "deploy": "npm run build:slim && npx --yes wrangler pages deploy dist --project-name=webgpu-fly --branch=main", - "deploy:hf": "npm run build:slim && cp space/README.md space/.gitattributes dist/ && rm -f dist/_headers && hf upload abgunaydin/webgpu-fly dist . --repo-type=space", + "deploy:hf": "npm run build:slim && cp space/README.md space/.gitattributes NOTICE LICENSE LICENSE-FLYBODY LIMITATIONS.md dist/ && rm -f dist/_headers && hf upload abgunaydin/webgpu-fly dist . --repo-type=space", "deploy:vercel": "npm run build && vercel --prod", "deploy:r2": "bash tools/upload_to_r2.sh" }, diff --git a/space/README.md b/space/README.md index 98a3407..af4cbbf 100644 --- a/space/README.md +++ b/space/README.md @@ -21,8 +21,13 @@ runs in the tab.** Two separate things move the body, and the difference matters: - A **trained RL walking policy** (Vaxenburg et al. 2025) genuinely walks the - fly from leg actuation and ground reaction alone. That path bypasses the - brain and the spine entirely. + fly forward from leg actuation and ground reaction — 2.019 cm per simulated + second against a 2.0 cm/s command, with the kinematic assist off. It does + not keep the fly upright, though: a pitch/roll damper bleeds off the body's + pitch and roll angular velocity every substep, and with that damper off the + fly capsizes and stops walking (0.068 cm/sim s, uprightness −0.87). The + translation is earned; the posture is not. That path bypasses the brain and + the spine entirely. - The **connectome** drives a hand-written tripod gait. It scales that gait but does not generate its rhythm — the leg phase is `sin(sim_time · freq)`. @@ -49,16 +54,21 @@ so the 314 MB cache may not survive between visits. ## Credits and licensing -Code is MIT. The data is not, and each piece keeps its own terms: +The `license: mit` in this Space's header describes the source code only. The +~314 MB of data the app downloads is not MIT, and each piece keeps its own +terms: | | | |---|---| -| Brain connectome | [FlyWire](https://flywire.ai) FAFB, CC-BY | -| Ventral nerve cord | [Janelia MANC](https://www.janelia.org/project-team/flyem/manc-connectome) (Takemura et al. 2024) | -| Body model + walking policy | [TuragaLab/flybody](https://github.com/TuragaLab/flybody) (Vaxenburg et al. 2025), Apache-2.0 | -| Physics | MuJoCo compiled to WebAssembly | +| Brain connectome | [FlyWire](https://flywire.ai) FAFB, CC-BY 4.0 | +| Ventral nerve cord | [Janelia MANC](https://www.janelia.org/project-team/flyem/manc-connectome) (Takemura et al. 2024), CC-BY 4.0 | +| Body model (MJCF + meshes) | [TuragaLab/flybody](https://github.com/TuragaLab/flybody) (Vaxenburg et al. 2025), Apache-2.0 | +| Walking policy | [Janelia Figshare deposit](https://janelia.figshare.com/articles/dataset/25309105) (Vaxenburg et al. 2025), CC-BY 4.0 | +| Physics | MuJoCo compiled to WebAssembly, Apache-2.0 | -Full attribution in `NOTICE`; every approximation and shortcut is inventoried -in `LIMITATIONS.md`. +Full attribution in [NOTICE](NOTICE); the Apache-2.0 text is in +[LICENSE-FLYBODY](LICENSE-FLYBODY) and the MIT text in [LICENSE](LICENSE). +Every approximation and shortcut is inventoried in +[LIMITATIONS.md](LIMITATIONS.md). Source: