A native iOS (Swift) + Android (Kotlin) app that recognizes flag-semaphore signals from the device camera and decodes them to text. Pose estimation runs on-device (Apple Vision / ML Kit); a small classifier maps two arm angles to a character.
Primarily a learning vehicle — the goal is to learn the full on-device ML pipeline (data → train → export → native inference → UX) on a tractable problem, while deliberately building parallel native implementations on both platforms against a single shared contract. See
docs/semaphore-translator-spec.md(the authoritative spec, v0.2).
Beta-track. The full camera → pose → adapter → decode → commit pipeline ships on both platforms (Epics 1–5):
- The shared contract is frozen and verified (Epic 1).
- The cross-platform parity harness is live (Epic 2, #14): both the
iOS (XCTest) and Android (JVM unit test) ports run
shared/test_vectors.jsonthrough their decode path and assert identical results. - Capture, on-device pose estimation, and the per-platform adapter run on each platform in lockstep (Epics 3–4).
- The app opens to a no-camera home fork into two modes (Epic 5): Learn (front camera, practice your own signing) and Interpret (rear camera, read another signer).
Current work (Epic 6) is release polish and the light Tier-A
distribution/legal layer toward a closed friends-&-family beta on both
app stores. Per-platform detail: ios/README.md,
android/README.md, and the architecture decision
records in docs/adr/.
- Publisher: Skyline Trail Computing LLC.
- License: MIT — see
LICENSE. - Distribution posture: Tier A (free, on-device, no PII of
consequence, no injury surface). The legal layer for a closed F&F beta
is light — a camera purpose string, a short on-device-only privacy
note, and a minimal EULA. See
DISTRO_CHECKLIST.md.
A single pipeline, reimplemented natively on each platform, consuming shared language-neutral artifacts:
Camera frame → pose estimation (native) → [adapter] → shared 6-keypoint
struct → feature extraction (arm angles) → classifier (rules | trained)
→ temporal smoothing/commit → decoded text
The adapter is the only place allowed to know platform-specific
skeleton schemas, coordinate origins, or the camera mirror. Everything
downstream operates on the shared representation and must be logically
identical across platforms — enforced by a parity test over
shared/test_vectors.json.
| Path | What |
|---|---|
docs/ |
The shared spec (source of truth for logic). |
shared/ |
Language-neutral source of truth for data: alphabet, config constants, parity test vectors. Both platforms load these; nothing here is hardcoded in Swift/Kotlin. |
model/ |
Off-device Python training + export to Core ML and LiteRT; checked-in exported artifacts. |
ios/ |
Swift app (SwiftUI, AVFoundation, Vision, Core ML). |
android/ |
Kotlin app (Compose, CameraX, ML Kit, LiteRT). |
- Freeze the shared contract (
shared/). - Stand up the adapter spec + parity-test harness for both platforms.
- Capture + pose + adapter on each platform, in lockstep.
- Rules-based classifier + decoder on each, gated by the parity test.
- Minimal UI on each, to parity.
- Tier-A legal + closed F&F on both stores together.
- ML pipeline (the learning payoff): train once, export to both runtimes, wire behind the classifier toggle, measure against the rules baseline.