Project-Her / VOForge

Batch neural voice-over generator with tones, emotional styles, and voice cloning. Feed it a list of dialog lines and a cast sheet; it renders one audio file per unique line. Project-agnostic, pip-installable, CI-friendly, with drop-in runtime players for web, Unity, and Godot.

Network is needed at build time only — the rendered pack ships with your project and plays offline. This is the generator behind VENTFALL's fully-voiced campaign (139 lines, seven actors).

Project-Her is the end-to-end harness layered on VOForge: versioned model and voice selection, explicit per-component licenses, clone-consent evidence, deterministic run manifests, upgrade workflows, and a local stdio MCP server that any MCP-capable AI can use. Start with docs/PROJECT_HER.md.

project-her init my-voice-project
project-her validate --config my-voice-project/project-her.json
project-her plan --config my-voice-project/project-her.json
project-her render --config my-voice-project/project-her.json --confirm

pip install ./tools/voforge          # or: pip install git+https://github.com/Protonmatter/ventfall#subdirectory=tools/voforge
voforge --lines lines.json --cast cast.json --out vo/

Choosing a backend for naturalness

Naturalness, intonation, and accuracy are properties of the model, not a knob — so pick the backend that matches how natural you need it:

Backend	Naturalness	Emotion path	Cost / setup
edge (default)	good neutral neural voices	tones (prosody deltas — an approximation of emotion, not acted)	free, no key
azure	good, with acted emotion	styles — true express-as (angry/sad/whispering...) with natural intonation	Azure Speech key
xtts	very natural, clones a real performance	inherits the reference recording's own prosody & intonation	free, local, one-time model download
f5	reference-conditioned, deterministic candidates	inherits timbre from a transcribed reference; fixed seeds support controlled bake-offs	free/local; official weights are CC-BY-NC
kokoro	natural local stock voices	speed and volume controls; no reference-speaker cloning	free/local, one-time model download
elevenlabs	most natural	contextual, per-line	paid API key

Honest guidance: prosody tones never produce natural acted emotion — they pitch/speed-shift a neutral read. For genuinely natural emotional delivery use azure (acted styles) or, best of all, clone a reference performance with xtts/elevenlabs — the model reproduces the intonation of the sample you give it.

Expression: tones vs styles

Two tiers, one schema — write your dialogue once, render on any backend:

Tones are prosody presets (rate/pitch/volume deltas layered on the speaker's base voice). They work everywhere, including the free edge backend. Built-ins: grim, urgent, soft, whisper, bright, cold, awe, fear, weary, neutral (voforge --list-tones), and projects can add their own via _tones in the cast sheet.

Styles are true acted emotions (angry, sad, whispering, terrified, ...) rendered by Microsoft's express-as engine — these need the azure backend (a real Azure Speech key). On backends without style support each style degrades automatically to the nearest tone, so the same lines file renders anywhere, just less theatrically.

[
  {"who": "mentor",  "text": "Careful, kid.", "tone": "grim"},
  {"who": "villain", "text": "That was your first mistake.",
   "style": "angry", "styledegree": "1.4"},
  {"who": "mentor",  "text": "Run. RUN!", "tone": "urgent", "rate": "+25%"}
]

Resolution order: speaker base → tone preset → per-line rate/pitch/ volume overrides. voforge --dry-run shows exactly how every line resolves before you spend a single network call.

Voice cloning from a microphone, audio, or video

Clone any voice you have the right to use — feed the model a short reference recording and it reproduces that person's timbre and their natural prosody/intonation.

1. Make a reference clip (6–30 s of one clear speaker, no music):

voforge --record-ref mara.wav --duration 20            # from the microphone
voforge --extract-ref interview.mp4 mara.wav --start 12 --duration 20   # from video
voforge --extract-ref memo.m4a mara.wav                # from an audio file

(--extract-ref/--record-ref use ffmpeg — found on PATH or via pip install imageio-ffmpeg.)

2. Point a cast entry at it and declare consent:

{ "mara": { "ref": "mara.wav", "consent": true } }

3. Render with a cloning backend:

voforge --lines lines.json --cast cast.json --backend xtts --out vo/

Cloned lines are written as .wav; the runtime players take an ext: "wav" option to match. Consent gate: voforge refuses to synthesize from a ref or voice_id unless that cast entry has "consent": true. Only clone voices you own or have explicit permission to use.

Setting up the local cloning engine (xtts)

XTTS needs PyTorch, which currently ships wheels only for Python 3.12 (not 3.13/3.14). Provision a dedicated env — verified working, including on Windows on Arm / Snapdragon:

# 1. a Python 3.12 environment (uv is the easy way on any OS/arch):
uv venv voforge-clone --python 3.12
# 2. native torch from the PyTorch CPU index (win-arm64 or win-amd64 wheels;
#    on Snapdragon the amd64 wheels also run fine under emulation):
uv pip install torch==2.7.0 torchaudio==2.7.0 --index-url https://download.pytorch.org/whl/cpu
# 3. voforge + the cloning stack (pins in requirements-clone.txt):
uv pip install ./tools/voforge -r ./tools/voforge/requirements-clone.txt

First render downloads ~1.8 GB of XTTS weights, then runs fully offline.

Setting up F5-TTS cloning

F5 requires the exact transcript of each reference and an integer seed on every line. This makes candidate generation reproducible and prevents an untranscribed reference or random seed from silently entering a build:

pip install ".[f5]"
voforge --lines f5-lines.json --cast f5-cast.json --backend f5 --name id --jobs 1 --out vo/

{
  "lieutenant": {
    "ref": "lieutenant-reference.wav",
    "ref_text": "The exact words spoken in the reference recording.",
    "consent": true
  }
}

[
  {"id": "briefing-01", "who": "lieutenant", "text": "Perimeter contact.", "seed": 4701}
]

F5 output is mastered before PCM16 quantization with a deterministic look-ahead limiter and delivered as mono 22.05 kHz WAV. The default ceiling is -1 dBFS. Projects can instantiate F5Backend directly to provide a pinned checkpoint, different mastering settings, and a machine-readable render record.

Setting up Kokoro stock voices

Kokoro is a local stock-voice backend, not a cloning backend. Use a dedicated Python 3.10–3.12 environment because Kokoro 0.9.4 does not support Python 3.9 or 3.13+. The first run downloads the selected model repository:

python -m venv voforge-kokoro
voforge-kokoro/Scripts/python -m pip install ".[kokoro]"   # Windows
voforge --lines lines.json --cast cast.json --backend kokoro --out vo/
voforge --backend kokoro --audition af_heart "Systems online." --out audition/

Kokoro output is delivered as mono 24 kHz PCM16 WAV with the same deterministic look-ahead limiter used by F5. VOForge maps percentage rate to Kokoro speed and percentage volume to pre-limiter gain. Kokoro has no pitch control; requested pitch deltas are listed in the backend render record as ignored. Voice IDs such as af_heart are Kokoro voices and do not require clone consent.

Backends

Backend	Cost	Expression	Setup
edge (default)	free, no key	tones (prosody)	pip install edge-tts
azure	Azure Speech pricing	tones + true styles + beat breaks (…/— → 350ms pauses)	AZURE_SPEECH_KEY + AZURE_SPEECH_REGION
xtts	free, local	clones a reference recording (natural prosody)	Python 3.12 + requirements-clone.txt
f5	free, local	seeded reference-conditioned cloning	pip install ".[f5]"; official weights are CC-BY-NC
kokoro	free, local	packaged stock voices, speed, volume	Python 3.10–3.12 + pip install ".[kokoro]"
elevenlabs	paid API	hosted synthesis / cloning by voice_id	ELEVENLABS_API_KEY

The azure and elevenlabs backends are implemented to their documented REST contracts but haven't been exercised against live subscriptions from this repo; treat the first run as a shakedown. The edge and xtts backends are verified end-to-end (xtts on Windows-on-Arm, x86-64-emulated).

Inputs

lines — JSON array (or CSV with a header) of:

key
who	required	speaker, must match the cast sheet (or _default exists)
text	required	the display/subtitle line
spokenText	optional	pronunciation-only text sent to TTS
id	optional	stable name for --name id
tone	optional	tone preset name
style, styledegree	optional	acted emotion (azure; degrades to tone elsewhere)
rate, pitch, volume	optional	explicit prosody overrides ("+6%", "-35Hz")
seed	required for F5	deterministic integer candidate seed

cast — voice per speaker plus specials:

{
  "_default": {"voice": "en-GB-ThomasNeural"},
  "_subs":    [["⋮", ","], ["[▸·—…]", ", "]],
  "_tones":   {"spooked": {"rate": "+10%", "pitch": "+9Hz", "volume": "-15%"}},
  "hero":     {"voice": "en-US-AriaNeural"},
  "villain":  {"voice": "en-GB-RyanNeural", "rate": "+6%", "style": "unfriendly"}
}

_subs is regex [pattern, replacement] text cleanup (typography → phonetics). Speakers can carry a default style/tone of their own. Use per-line spokenText when only one performance needs an accent-specific phonetic spelling. Content-addressed names use the spoken form so pronunciation changes cannot silently reuse stale audio.

For Kokoro, set voice to a packaged Kokoro ID such as af_heart. Project configuration should pin the selected model repository and voice IDs; do not describe Kokoro stock voices as clones of project characters.

F5 cast entries additionally require ref_text, the exact transcript of the authorized reference clip. VOForge validates this before loading the model.

Output naming and runtime lookup

Default is content-addressed: <djb2(who|text)>.mp3. Your app computes the same hash at runtime and requests the file — no manifest, and a 404 is the fallback signal (e.g. to live speechSynthesis). --name id and --name seq exist for stable/manual naming.

Hash: djb2, h = (h*33 XOR byte) mod 2^32, seed 5381, over UTF-8 of "who|text", hex-encoded to 8 chars.

Runtime players (runtime/)

Matching consumers so projects don't reimplement the hash:

• voforge.js — browser ES module: VoPlayer with WebAudio, caching, preload, cancel-on-advance, and a fallback hook. Tested.
• VoForge.cs — Unity reference implementation (hash + streaming play).
• voforge.gd — Godot 4 reference implementation.

CLI

voforge --lines FILE --cast FILE [--out DIR] [--name hash|id|seq]
        [--backend edge|azure|xtts|f5|kokoro|elevenlabs] [--jobs N] [--force] [--dry-run]
voforge --list-voices [FILTER]
voforge --list-tones
voforge --audition VOICE "TEXT" [--tone TONE] [--out DIR]
voforge --record-ref OUT.wav [--duration S] [--device NAME]
voforge --extract-ref MEDIA OUT.wav [--start S] [--duration S]

Build-ready behavior: inputs are validated up front with every problem listed; exit code 0 on success, 1 if any line failed to render, 2 on bad input — safe to gate CI on. Rendering is incremental (existing files are kept; --force re-renders).

Library

import asyncio, voforge
asyncio.run(voforge.render(lines, cast, "vo/"))          # edge backend
voforge.plan(lines, cast, "vo/")                          # dry-run rows
voforge.render(..., backend=voforge.make_backend("azure"))

Tests

python -m unittest discover -s tests    # fully offline (FakeBackend)

Cloned WAV examples

examples/audio/ contains a synthetic Kokoro reference and two F5-cloned WAV demonstrations with exact transcripts, seeds, checkpoint and artifact hashes, PCM measurements, and an explicit asset notice. They use no natural-person or game-character reference audio and are excluded from the MIT code license because the official F5 checkpoint is CC-BY-NC-4.0.

The separately licensed Cybernetic Commander profile demonstrates a hash-pinned multi-clip reference pack. Its user-supplied voice assets are not MIT-licensed, require the recorded authorization boundary, and remain restricted to evaluation and non-commercial use.

Fidelity and promotion evidence

See docs/VOICE_FIDELITY.md for the reusable findings from reference-cloning evaluation: clipping must be prevented in floating point, raw speaker-embedding cosine is diagnostic rather than a calibrated identity score, owner approval applies to exact hashes, and runtime promotion must retain a rollback artifact. The library exposes PcmGate, candidate_record, validate_pcm_artifact, canonical_source_sha256, and promote_artifact for that evidence path.

License

Project-Her and VOForge source are MIT under LICENSE. Voice audio is generated through third-party models or services; review the selected backend's terms. In particular, F5-TTS code is MIT but its official pretrained weights are CC-BY-NC. The Kokoro Python package declares Apache-2.0; model and packaged-voice terms must still be checked for the specific model release before distributing generated assets.