Gradio UI produces deterministic near-silent/noise output (repaint & text2music), while the identical generate_music() call succeeds when invoked directly outside the Gradio session

[mdhite1]

Title

Gradio UI produces deterministic near-silent/noise output (repaint & text2music), while the identical generate_music() call succeeds when invoked directly outside the Gradio session

Environment

• Image: ghcr.io/ace-step/ace-step-1.5:latest
• GPU: NVIDIA RTX 2000 Ada Laptop GPU, 8GB VRAM
• PyTorch 2.10.0+cu128, torchao 0.16.0, CUDA 12.8
• Config: acestep-v15-turbo DiT, acestep-5Hz-lm-1.7B LM (auto-downgraded from 4B by the app's own GPU-tier detection), pt LM backend, --quantization none (also reproduced with default int8_weight_only quantization)
• 8 inference steps (turbo), DCW enabled (double mode, default scalers)

Summary

Every generation submitted through the bundled Gradio web UI for task_type="repaint" or task_type="text2music" comes back as near-silent audio that decodes to broadband noise after loudness normalization — regardless of the caption, lyrics, seed, or batch content entered. Across 6+ distinct UI-submitted requests (different prompts, different seeds, both single and batch-size-2), the pre-normalization peak was bit-identical or near-identical every time: Peak=0.0034 (×4), Peak=0.0035, Peak=0.0036. After the app's own -1dB normalization step, this becomes audible full-volume noise.

Calling the exact same underlying generate_music() function directly in a fresh Python process — same checkpoints, same DiT/LM handlers, same parameters captured verbatim from a failing UI request (including the same caption, seed, src_audio, repaint settings) — consistently succeeds, with Peak=1.0000 before normalization and spectrally/perceptually correct musical output (verified via spectrogram: clear rhythmic structure, natural harmonic content, matching the requested instrumentation) in every one of 6+ direct-API attempts, including one that reproduced a task_type="text2music" batch and one that reproduced a task_type="repaint" request.

This strongly suggests the bug is not in generate_music(), the DiT/LM checkpoints, or GPU/precision handling, but somewhere in the Gradio server process's request handling — something that behaves differently for a long-running, previously-used server session vs. a fresh single-shot process.

What was ruled out (with evidence)

1. Gradio event-wiring / positional-argument mismatch. Traced the full call chain (generation_run_wiring.py → generate_with_batch_management → generate_with_progress → GenerationParams(...)) — all 78 positional arguments in the .click() wiring match their callee's signature 1:1, in order. The final GenerationParams(...) construction uses explicit keyword arguments throughout, not positional, eliminating ordering bugs at that layer.
2. Checkpoint/weight loading. Loaded acestep-v15-turbo's state dict directly and inspected weight statistics (mean/std/min/max across first and last layers, including detokenizer.proj_out.{weight,bias}) — all values look like genuinely trained parameters, no missing keys reported, nothing zeroed or randomly-initialized-looking.
3. --quantization (torchao AffineQuantizedTensor). Reproduced with quantization fully disabled (--quantization none) — identical failure (Peak=0.0034/0.0035).
4. guidance_scale / shift. The app already auto-corrects guidance_scale for turbo models (Turbo model detected: overriding guidance_scale 7.0 -> 1.0) regardless of what the UI sends. Explicitly testing shift=3.0 (the UI's actual default, vs. the dataclass default 1.0) via direct API still succeeded (Peak=1.0000).
5. DCW correction. Active with identical settings (mode=double, UI defaults) in both failing and succeeding runs — not differentiating.
6. The LM "thinking" step. A task_type="repaint" request with think_checkbox=False (LM skipped — confirmed via log: Skipping LM for task_type='repaint' - using DiT directly, use_lm=False) still failed identically (Peak=0.0035). Ruled out LM-state corruption in the long-lived llm_handler singleton as the sole cause.
7. Stale text2music_audio_code_string textbox. Confirmed via logs that the app's own defensive guard (if task_type != "text2music": text2music_audio_code_string = "") correctly clears this field for repaint/cover before use — the failing repaint request genuinely processed the fresh src_audio file ([generate_music] Processing source audio..., followed by a correct text_prompt reaching the DiT text encoder with the real caption).
8. custom_timesteps override. UI default is "" (empty), which safely no-ops in parse_and_validate_timesteps.

Diagnostic method

Patched a temporary debug block at the top of generate_with_progress (acestep/ui/gradio/events/results/generation_progress.py) logging every one of its ~55 raw incoming arguments before any processing. This let me capture, verbatim, what the live Gradio server actually received for both a succeeding (task_type="cover") and a failing (task_type="repaint") request, and confirmed the caption/lyrics/metadata text reaching the DiT's text encoder was correct in the failing case — i.e., the input to generation was fine, but the output was still degenerate.

Suspected (not confirmed) contributing factor: reference_audio

Every failing repaint/text2music request I captured or was shown had reference_audio set (in addition to src_audio). My only fully clean, successful direct-API reproduction of a repaint request never set reference_audio. When I attempted to reproduce the exact failing scenario — task_type="repaint", reference_audio and src_audio both pointing at the same real ~4.5-minute track — the direct-API call hung on _prepare_reference_and_source_audio's "Processing source audio..." step for 15+ minutes (131–151% CPU, no forward progress, no error), whereas the live Gradio server processed the identical file (both reference and source encode) in ~7 seconds. I killed the hung process rather than let it run indefinitely, so I can't confirm whether it would have eventually produced degenerate output, errored, or completed correctly.

This is offered as a lead for maintainers with more context on the reference_audio + repaint conditioning path, not a confirmed root cause — the hang itself is a distinct, second anomaly worth investigating alongside the primary near-silent-output bug.

Reproduction steps (primary bug)

1. Run the container per the official docker-compose.yml (nanobot-gateway-equivalent gateway/Gradio service).
2. In the Gradio UI, submit a text2music or repaint generation with any caption/lyrics, default settings otherwise.
3. Observe: generation completes without error, but the resulting audio is broadband noise. Check server logs for [Normalization] Audio N BEFORE: Peak= — expect a value in the 0.003–0.004 range regardless of prompt content.
4. For comparison, docker exec into the container and run the same request via acestep.inference.generate_music() directly in a fresh python3 process (see run_generate_test.py in the repo root for the pattern) — expect Peak≈1.0.

Impact

This affects the primary, documented way most users are expected to interact with the model (the bundled Gradio web UI via docker-compose.yml/docker run ... gateway). Anyone following the README's Docker instructions and using the default UI for text2music or repaint tasks will get unusable output every time, with no error surfaced — the UI reports success.