Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@ static/vendor/*
*PitchLoader Output*.ustx
*output*.ustx
settings.json
expressive_config.json
Expressive-GUI.spec
.pytest_cache/
.coverage
Expand Down
88 changes: 67 additions & 21 deletions README.en.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@

# Expressive

**Expressive** is a [DiffSinger](https://github.com/openvpi/diffsinger) expression parameter importer developed for [OpenUtau](https://github.com/stakira/OpenUtau). It aims to extract emotional parameters from real human vocals and import them into the appropriate tracks of your project.
**Expressive** is a [DiffSinger](https://github.com/openvpi/diffsinger) expression parameter importer developed for [OpenUtau](https://github.com/stakira/OpenUtau). It aims to extract expression parameters from real human vocals and import them into the appropriate tracks of your project.

The current version supports importing the following expression parameters:

Expand All @@ -21,14 +21,14 @@ The current version supports importing the following expression parameters:
<img src="https://github.com/user-attachments/assets/268b44d4-528d-481e-acfb-3f7da7261c80" width="100%" />
</p>

> - *OpenUtau version used from [keirokeer/OpenUtau-DiffSinger-Lunai](https://github.com/keirokeer/OpenUtau-DiffSinger-Lunai)*
> - *OpenUtau version from [keirokeer/OpenUtau-DiffSinger-Lunai](https://github.com/keirokeer/OpenUtau-DiffSinger-Lunai)*
> - *Singer model from [yousa-ling-official-production/yousa-ling-diffsinger-v1](https://github.com/yousa-ling-official-production/yousa-ling-diffsinger-v1)*

> [!TIP]
> <details>
> <summary><b>👉 Click to expand the full voiced demo video 👈</b></summary>
>
> https://github.com/user-attachments/assets/4b5b7c15-947a-4f54-b80e-a14a9eefc86b
> <p align="center"><video src="https://github.com/user-attachments/assets/4b5b7c15-947a-4f54-b80e-a14a9eefc86b"></video></p>
>
> </details>

Expand All @@ -40,7 +40,7 @@ The current version supports importing the following expression parameters:

By default, this application uses [swift-f0](https://github.com/lars76/swift-f0) (based on ONNX Runtime) as the pitch extraction backend, which runs on CPU only and satisfies basic usage scenarios.

The classic [CREPE](https://github.com/marl/crepe) pitch extraction backend (depends on TensorFlow) is also available, which suits for scenarios with higher accuracy requirements. If your computer is equipped with an NVIDIA GPU and supports [CUDA 11.x](https://docs.nvidia.com/deploy/cuda-compatibility/minor-version-compatibility.html) (i.e., GPU driver version >= 450), the CREPE backend will automatically enable GPU acceleration.
The classic [CREPE](https://github.com/marl/crepe) pitch extraction backend (depends on TensorFlow) is also available for scenarios with higher accuracy requirements. If your computer is equipped with an NVIDIA GPU and supports [CUDA 11.x](https://docs.nvidia.com/deploy/cuda-compatibility/minor-version-compatibility.html) (i.e., GPU driver version >= 450), the CREPE backend will automatically enable GPU acceleration.

> \* On Windows, TensorFlow 2.10 is the last version that supports GPU acceleration, and Python 3.10 is the highest Python version supported by its `.whl` files.

Expand All @@ -52,13 +52,17 @@ When using a DiffSinger virtual singer for covers, users often already have an O

### Inputs

> [!TIP]
> Starting from `v0.6.0`, this application supports OpenUtau voice tracks with **multiple parts** and **multiple tempos**.

> [!TIP]
> Starting from `v0.5.0`, users can define a selection region independently within the full audio of both the **Utau vocal** and the **Reference vocal**. The selected audio segment will be used as the final input.

* **Utau vocal**: Emotionless synthesized vocal output from OpenUtau (WAV format). It's recommended to keep `Tempo` and segmentation as close to the reference vocal as possible.
* **Reference vocal**: Original human vocal recording (WAV format). You can use tools like [UVR](https://github.com/Anjok07/ultimatevocalremovergui) to remove instrumental and reverb.
* **Utau vocal**: Emotionless synthesized vocal output from OpenUtau (WAV format). It is recommended to keep the segmentation and tempo as close to the **Reference vocal** as possible, as large discrepancies may affect alignment quality.
* **Reference vocal**: Original human vocal recording (WAV format). You can use tools like [UVR](https://github.com/Anjok07/ultimatevocalremovergui) or [MSST](https://github.com/SUC-DriverOld/MSST-WebUI) to remove instrumentals, harmonies, and reverb.
* **Input project**: Original OpenUtau project file (USTX format).
* **Output path**: Where the new processed project file will be saved.
* **Track number**: The track number in the OpenUtau project where the **Utau vocal** resides (1-based). Expression parameters will be imported into this track.

### Output

Expand All @@ -74,10 +78,6 @@ A new USTX file with expression parameters added. The original project will not
* [x] `Dynamics` generation
* [x] `Tension` generation

## ⚠️ Known Issues

1. The current version does not support tempo changes within a single track. It’s recommended to use a consistent tempo throughout the project. This limitation will be addressed in future updates.

## 🚀 Direct Install

You can download pre-compiled executable files directly from the [Releases](https://github.com/NewComer00/expressive/releases) page:
Expand All @@ -100,7 +100,6 @@ Includes CUDA runtime libraries. When used on a computer with an NVIDIA GPU (dri

> [!IMPORTANT]
> This project uses [Git LFS](https://git-lfs.com/) to store large files such as example audio under `examples/`. Please ensure Git LFS is installed on your system before cloning.

```bash
git clone https://github.com/NewComer00/expressive.git --depth 1
cd expressive
Expand All @@ -109,7 +108,6 @@ cd expressive
### Install the application

Install the package and its dependencies in a virtual environment:

```bash
pip install -e ".[gpu,gui]"
```
Expand All @@ -129,13 +127,11 @@ After installation, you can use the `expressive` and `expressive-gui` entry poin
### Command Line Interface (CLI)

Display help:

```bash
expressive --help
```

Run example in Windows PowerShell:

```powershell
expressive `
--utau_wav "examples/明天会更好/utau.wav" `
Expand All @@ -150,7 +146,6 @@ expressive `
```

Run example in Linux shell:

```bash
expressive \
--utau_wav "examples/明天会更好/utau.wav" \
Expand All @@ -169,20 +164,24 @@ The output project file will be saved to `examples/明天会更好/output.ustx`.
### Graphical User Interface (GUI)

Launch in English:

```bash
expressive-gui --lang en
```

> [!IMPORTANT]
> Due to framework limitations, the GUI launched via the `expressive-gui` command currently **does not support drag-and-drop**. To use drag-and-drop, please install the GUI [directly](#-direct-install), or run `expressive_gui.py` as a script:
>
> Due to framework limitations, the GUI launched via the `expressive-gui` command currently **does not support drag-and-drop**. To use drag-and-drop, please [install directly](#-direct-install), or run `expressive_gui.py` as a script:
>
> ```bash
> python expressive_gui.py --lang en
> ```

## 🔬 Algorithm Workflow
## 📂 Examples

The [`examples/` directory](examples/) contains several sample projects. You can import the `expressive_config.json` file from any example into the GUI to automatically populate all parameters with the preset values.

If you installed the application from the installer, a shortcut named `Expressive-examples` pointing to the examples directory will appear on your desktop after installation — you can import the config files directly from there.

## 🔬 Algorithm Workflow
```mermaid
graph TB;
ustx_in[/"OpenUtau Project (USTX)"/]
Expand All @@ -191,11 +190,14 @@ graph TB;
refwav-->feat_pitd
ustx_in-.->|Export|utauwav
utauwav-->feat_pitd
ustx_in-->|Tempo|time_pitd

ustx_editor["USTX Editor"]
ustx_in-->ustx_editor
ustx_editor-->|UProject & Time Axis|PitdLoader

subgraph PitdLoader
direction TB
feat_pitd["Features Extraction<br>Pitch & MFCC"]
feat_pitd["Features Extraction<br>Pitch & MFCC & RMS"]

time_pitd["Time Alignment<br>FastDTW"]
feat_pitd-->time_pitd
Expand Down Expand Up @@ -232,3 +234,47 @@ graph TB;
time_tenc-->get_tenc
end
```

## ⚠️ Troubleshooting

### Drag-and-drop does not work on first launch after installation

#### Symptom
On Windows 10 / 11, after installing the application from the installer for the **first time** (reinstalling after a previous uninstall does not count), the drag-and-drop functionality does not work.

#### Possible Cause
The [NiceGUI](https://nicegui.io/) framework's support for drag-and-drop in native applications is not yet fully mature. The drag-and-drop feature in this application is currently implemented via the underlying library [pywebview](https://pywebview.flowrl.com/).

#### Solution
Relaunching the application should restore normal functionality, and this issue will not occur again on the same system afterward.

#### Future Plan
The NiceGUI framework has begun improving its drag-and-drop support and should resolve this in a future release.

### PITD expression curve is overall too flat

#### Symptom
The extracted PITD expression curve is too flat, with almost no significant variation overall. Pitch changes in the reference vocal are not reflected in the expression curve.

#### Possible Cause
The two confidence thresholds in the PITD extractor are set **too high**, causing many pitch changes to be discarded.

#### Solution
Try lowering both confidence thresholds. In general, the **Utau vocal** is relatively clean, so it is advisable to first adjust the confidence threshold for the **Reference vocal**.

#### Future Plan
Introduce a better PITD backend (e.g., [RMVPE](https://github.com/Dream-High/RMVPE)). Add visualization of intermediate results.

### PITD expression curve has sudden jumps or spikes at certain positions

#### Symptom
The PITD expression curve changes too rapidly at certain positions, with very large jumps or spikes that clearly do not match natural vocal behavior.

#### Possible Cause
The two confidence thresholds in the PITD extractor are set **too low**, causing erroneous detection results to be accepted.

#### Solution
Try increasing both confidence thresholds. In general, the **Utau vocal** is relatively clean, so it is advisable to first adjust the confidence threshold for the **Reference vocal**.

#### Future Plan
Introduce a better PITD backend (e.g., [RMVPE](https://github.com/Dream-High/RMVPE)). Add visualization of intermediate results.
73 changes: 63 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@
> <details>
> <summary><b>👉 点击展开完整有声演示视频 👈</b></summary>
>
> https://github.com/user-attachments/assets/4b5b7c15-947a-4f54-b80e-a14a9eefc86b
>
> <p align="center"><video src="https://github.com/user-attachments/assets/4b5b7c15-947a-4f54-b80e-a14a9eefc86b"></video></p>
>
> </details>

## ✅ 支持平台
Expand All @@ -52,13 +52,17 @@

### 输入

> [!TIP]
> 从 `v0.6.0` 开始,本应用支持带有**多分段**与**多曲速**的 OpenUtau 人声音轨。

> [!TIP]
> 从 `v0.5.0` 开始,用户可以分别在**歌姬音声**与**参考人声**的完整音频中划定选区,选区内的音频段落将作为最终输入。

* **歌姬音声**:由 OpenUtau 输出的无表情虚拟歌声音频(WAV 格式)。建议节奏 (`Tempo`) 和分段尽量与参考人声一致。
* **参考人声**:原始人声录音(WAV 格式),可使用 [UVR](https://github.com/Anjok07/ultimatevocalremovergui) 等工具去除伴奏与混响。
* **歌姬音声**:由 OpenUtau 输出的无表情虚拟歌声音频(WAV 格式)。建议分段与曲速尽量与**参考人声**相近,若相差过大可能影响对齐效果。
* **参考人声**:原始人声录音(WAV 格式),可使用 [UVR](https://github.com/Anjok07/ultimatevocalremovergui) 、[MSST](https://github.com/SUC-DriverOld/MSST-WebUI) 等工具去除伴奏、和声与混响。
* **输入工程**:原始 OpenUtau 工程文件(USTX 格式)。
* **输出路径**:处理完成后新工程文件的保存位置。
* **音轨编号**:OpenUtau 工程中**歌姬音声**所在的音轨编号(从 1 开始)。表情参数会被导入到该音轨中。

### 输出

Expand All @@ -74,10 +78,6 @@
* [x] `Dynamics` 参数生成
* [x] `Tension` 参数生成

## ⚠️ 已知问题

1. 当前版本尚不支持单一轨道中的 `Tempo` 变化,建议工程全程使用统一节奏。该限制将在未来版本中解决。

## 🚀 直接安装

您可以直接在 [Releases](https://github.com/NewComer00/expressive/releases) 页面下载预编译的可执行文件:
Expand Down Expand Up @@ -179,6 +179,12 @@ expressive-gui --lang zh_CN
> python expressive_gui.py --lang zh_CN
> ```

## 📂 示例工程

项目的 [`examples/` 目录](examples/)下存放有多个示例。您可以在图形用户界面中导入相应示例的 `expressive_config.json` 配置文件,将预设的参数一键填写到应用中。

若您是从安装包获取的本应用,安装完毕后示例目录的快捷方式 `Expressive-examples` 将出现在您的桌面,您也可以直接导入其中的配置文件。

## 🔬 算法流程
```mermaid
graph TB;
Expand All @@ -188,11 +194,14 @@ graph TB;
refwav-->feat_pitd
ustx_in-.->|Export|utauwav
utauwav-->feat_pitd
ustx_in-->|Tempo|time_pitd

ustx_editor["USTX Editor"]
ustx_in-->ustx_editor
ustx_editor-->|UProject & Time Axis|PitdLoader

subgraph PitdLoader
direction TB
feat_pitd["Features Extraction<br>Pitch & MFCC"]
feat_pitd["Features Extraction<br>Pitch & MFCC & RMS"]

time_pitd["Time Alignment<br>FastDTW"]
feat_pitd-->time_pitd
Expand Down Expand Up @@ -229,3 +238,47 @@ graph TB;
time_tenc-->get_tenc
end
```

## ⚠️ 常见问题

### 安装后首次运行图形界面,文件拖拽功能无法正常使用

#### 问题现象
在 Windows 10 / 11 平台下,通过安装包**首次**安装本应用后(先前安装过再卸载不算),应用的文件拖拽功能无法正常使用。

#### 可能原因
[NiceGUI](https://nicegui.io/) 框架对原生应用的文件拖拽功能支持尚不完善。目前本应用的文件拖拽功能是基于底层库 [pywebview](https://pywebview.flowrl.com/) 实现的。

#### 解决方案
重新打开应用后应当可以恢复正常,且该系统今后不会再出现此问题。

#### 未来计划
NiceGUI 框架已经开始着手改进文件拖拽支持,应该在未来的版本中能够解决此问题。

### PITD 表情曲线整体变化过于平缓

#### 问题现象
提取出的 PITD 表情曲线过于平缓,整体上几乎没有大的起伏,参考人声中的音高变化并没有反映到表情曲线上。

#### 可能原因
PITD 表情提取器中,两个置信度阈值设置**过高**,许多音高变化没有被采信。

#### 解决方案
尝试降低两个置信度阈值。一般来说,**歌姬音声**比较纯净,可以先调整**参考人声**的置信度阈值。

#### 未来计划
引入更好的 PITD 后端(如 [RMVPE](https://github.com/Dream-High/RMVPE))。添加中间结果的可视化功能。

### PITD 表情曲线在某些位置变化过快,出现跳跃或毛刺

#### 问题现象
PITD 表情曲线在某些位置变化过快,出现非常大的跳跃或毛刺,明显不符合人声的变化规律。

#### 可能原因
PITD 表情提取器中,两个置信度阈值设置**过低**,错误的识别结果被采信。

#### 解决方案
尝试增加两个置信度阈值。一般来说,**歌姬音声**比较纯净,可以先调整**参考人声**的置信度阈值。

#### 未来计划
引入更好的 PITD 后端(如 [RMVPE](https://github.com/Dream-High/RMVPE))。添加中间结果的可视化功能。
2 changes: 1 addition & 1 deletion build/installer.iss
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ LZMAUseSeparateProcess=yes
LZMANumBlockThreads=8
SetupIconFile=..\assets\icons\app.ico
PrivilegesRequired=lowest
PrivilegesRequiredOverridesAllowed=commandline dialog
PrivilegesRequiredOverridesAllowed=commandline

[Files]
Source: "..\dist\Expressive-GUI\*"; DestDir: "{app}"; Flags: recursesubdirs createallsubdirs
Expand Down
15 changes: 15 additions & 0 deletions examples/Прекрасное Далеко/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Example: Прекрасное Далеко

## Audio
- **Source:** Прекрасное Далеко / Wonderful Faraway (из к/ф «Гостья из будущего», 1985)
- **Music:** Евгений Крылатов
- **Lyrics:** Юрий Энтин

## Voicebank
- **Source:** [Ashera Lyre DiffSinger V110](https://vocadb.net/Ar/139321)
- **Voice Provider:** OtomeHime
- **Illustration:** OtomeHime

## OpenUtau
- **Phonemizer:** DiffSinger Russian
- **Tested on version:** OpenUtau LUNAI Edition v0.1.1.0
38 changes: 38 additions & 0 deletions examples/Прекрасное Далеко/expressive_config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
{
"utau_wav": "examples/Прекрасное Далеко/utau.wav",
"ref_wav": "examples/Прекрасное Далеко/reference.wav",
"ustx_input": "examples/Прекрасное Далеко/project.ustx",
"ustx_output": "examples/Прекрасное Далеко/output.ustx",
"track_number": 1,
"ref_start": "0:01.71",
"ref_end": "0:22.37",
"utau_start": "0:01.75",
"utau_end": "0:22.46",
"expressions": {
"dyn": {
"selected": true,
"trim_silence": true,
"align_radius": 1,
"smoothness": 2,
"scaler": 1.3
},
"pitd": {
"selected": true,
"backend": "crepe",
"confidence_utau": 0.8,
"confidence_ref": 0.6,
"align_radius": 1,
"semitone_shift": 0,
"smoothness": 4,
"scaler": 2.2
},
"tenc": {
"selected": true,
"trim_silence": true,
"align_radius": 1,
"smoothness": 6,
"scaler": 1.0,
"bias": 10
}
}
}
Loading
Loading