Score2ConVec

English · 简体中文 · 日本語

Website: https://utaisynthesizer.net

Give any ContentVec-based SVC voice model the ability to sing from a score. Score (MIDI notes + lyrics) → ContentVec → so-vits-svc 4.0 / 4.1 or RVC → singing.

Score2ConVec (Score-to-ContentVec) is a small, deterministic SVS front-end for SVC voice models. It reads a musical score — notes + lyrics — and produces ContentVec content features, which any ContentVec-based SVC backend decodes into a singing voice. It turns an SVC "cover" model into a full "sing-from-score" singer, without training a heavyweight end-to-end SVS model.

  score  ──G2P + arrays──▶  Score2ConVec  ──▶  ContentVec [T, D] @50fps ──┐
 (notes + lyrics)                              (deterministic content)     ├─▶  SVC backend ─▶ singing .wav
                                                                           │    (so-vits-svc / RVC = the voice)
  f0 stream (note pitch, from the DAW) ─────────────────────────────────────┘

Why decouple

Traditional score-based SVS (FFT/FastSpeech/DiffSinger-style singers) learns notes → spectrogram end to end, with pitch, content, and timbre all entangled inside one decoder. Score2ConVec instead splits singing into three independent axes:

• Pitch (f0) — a separate stream you supply from the DAW (exact note pitch + portamento/vibrato).
• Content ("what is sung") — produced by this model as speaker-invariant ContentVec.
• Timbre ("who sings") — comes entirely from the SVC backend. Same content → different backend → different singer.

The SVC backend already separates f0 from timbre; Score2ConVec factors content out on top of that. The payoff is a very low training barrier: an SVC voice needs only ~10–15 minutes of dry vocals and no manual phoneme labeling, yet gains score-driven singing. The content model is trained once, multilingually, and is backend-agnostic and deterministic (same score → same output, no sampling).

Supported backends

Backend	ContentVec	Model	Checkpoint
so-vits-svc 4.1	vec768l12 (768-d)	ScoreToCV 768	cv_final.pt
RVC v2	ContentVec 768	ScoreToCV 768	cv_final.pt (zero-retrain, feeds straight in)
so-vits-svc 4.0	vec256l9 (256-d)	ScoreToCV 256	cv256_final.pt

Same architecture, only the target ContentVec flavor differs. Full backend guides: docs/DEPLOY_768_sovits41_rvc.md and docs/DEPLOY_256_sovits40.md.

Languages (EAR-accepted): zh, ja, en, de, fr, es, it.

Install

git clone https://github.com/yasoukyoku/Score2ConVec.git
cd Score2ConVec

# Python 3.10 recommended. Install PyTorch for your CUDA build first (https://pytorch.org), then:
pip install -r requirements.txt

You also need, separately:

1. A ContentVec-based SVC backend — a so-vits-svc checkout (4.0 or 4.1) or RVC, plus a voice model (.pth + config.json). This is the voice.
2. Model weights for Score2ConVec (below).

Weights

The ScoreToCV checkpoints are not in the git tree (each ~188 MB). Download them from the Releases page and place them under checkpoints/:

File	Dim	For	det_floor
cv_final.pt	768	so-vits-svc 4.1, RVC v2	~0.795
cv256_final.pt	256	so-vits-svc 4.0	0.791

The weights are for research / non-commercial use — see Training data & licensing.

Quick start

1) Check the front-end (G2P) — no models needed:

python scripts/render_ust.py --ust your_song.ust --dump

This parses a UST (SynthV / UTAU export), maps each lyric to IPA phones, and prints the per-note result so you can verify the lyric → phoneme mapping before rendering.

2) Render (needs cv_final.pt + an SVC backend):

# point the glue at your local so-vits-svc checkout and voice model
export SOVITS_ROOT=/path/to/so-vits-svc
export SOVITS_MODEL=/path/to/your_voice.pth
export SOVITS_CONFIG=/path/to/config.json          # (Windows: use `set NAME=...`)

python scripts/render_ust.py --ust your_song.ust --out processed/out
# -> processed/out/render_noteonly.wav

Minimal Python (the I/O contract):

import torch, yaml
from src.model.score2cv import ScoreToCV

cfg = yaml.safe_load(open("configs/model_cv_final.yaml", encoding="utf-8"))
model = ScoreToCV(cfg).cuda().float().eval()
model.load_state_dict(torch.load("checkpoints/cv_final.pt", weights_only=False)["model"])

# build per-phone score arrays (see render_ust.build_arrays / DEPLOY §2-3):
# phonemes, note_pitch, phone_dur, note_dur, note_to_phone, speaker_id, lang_id, phone_mask, technique(zeros)
with torch.no_grad():
    out = model(**inputs)
    T   = int(out["frame_mask"][0].sum())
    cv  = model.infer_cv(out["frame_hidden"])[0, :T].cpu().numpy()   # [T, 768] ContentVec, de-normalized
# feed cv + an f0 stream to the SVC backend -> singing.

The full I/O contract (every input array, the f0 stream, chunking for long songs, the RVC recipe) is in docs/DEPLOY_768_sovits41_rvc.md.

Training / retargeting

The architecture is backend-agnostic — only the target feature changes. To train a voice or retarget to a new ContentVec flavor (e.g. the 256-d retarget for so-vits 4.0):

1. Extract features over your audio — scripts/extract_contentvec.py (768) or extract_contentvec256.py (256), extract_f0.py, extract_notes.py.
2. Pack aligned .npz — scripts/pack_npz.py (768) or build_npz256.py (256), then compute_cv_norm*.py.
3. Train — python scripts/train_cv.py --config configs/model_cv_final.yaml.
4. Validate by ear through the target backend (a cv-space metric is never a ship gate).

Alignment uses a forced aligner (this project used HubertFA; MFA works too) — not included here. Details: docs/DEPLOY_256_sovits40.md §4.

Repository layout

src/model/          ScoreToCV (score2cv.py), ScoreToF0, shared sub-modules
src/preprocessing/  IPA phoneme vocab (210 tokens, 9 langs), ContentVec / f0 / RMVPE extractors
src/training/       dataset + losses
configs/            model_cv_final.yaml (768), model_cv256.yaml (256), model_f0_single.yaml
scripts/            feature extraction, npz packing, training, and the render/inference front-end
docs/               per-backend deployment guides (768 / 256)
checkpoints/        (you place the downloaded .pt here)

Limitations (honest scope)

• 7 languages are ear-accepted (zh/ja/en/de/fr/es/it). ko/ru were dropped (alignment quality).
• The content model uses a deterministic conditional-mean head — clean and stable, but residual artifacts show up as slightly breathy/blurred consonants. This is a property of the head, not under-training.
• f0 is parametric in the DAW. The learned f0 model was retired (it undershot large interval jumps); use exact note pitch + portamento + tail vibrato. See DEPLOY §7.
• scripts/render_ust.py is a first-pass reference front-end — a real DAW supplies precise per-note timing and should expose the duration / grouping / f0 knobs itself.

Training data & licensing

Code is MIT (see LICENSE). Weights are research / non-commercial.

The shipped models are trained on public singing corpora: M4Singer (zh), GTSinger (en/de/fr/es/it), and Japanese singing databases (kiritan_singing, PJS, Ofuton-P, Oniku, Itako, Natsume). Several of these are released for non-commercial / research use only — for example, M4Singer is CC BY-NC-SA 4.0. The released weights are not an MIT relicensing of that data. Review each dataset's own license before any commercial use.

Community

• QQ group: 1058227212
• Discord: https://discord.gg/p3fGh942fJ

Acknowledgements

Built on ContentVec, RMVPE, so-vits-svc, RVC, and the singing corpora above.