v2.1.5: remove the dark edge around subjects in AI mattes - #199
Merged
Merged
Conversation
The refiner additive guard trusted the model's alpha inside a fixed 4 pixel shell around the silhouette and the alpha hint everywhere outside it. When the hint covered more of the edge than that shell was wide, the switch left a dark band with opaque alpha on both sides, tracing the subject at a constant offset. Hint generators run at a fixed working resolution and are scaled up to the source, so the amount they overshoot the edge grows with the footage while the shell stayed at 4 pixels. The band therefore appeared on high resolution footage and not at 1080p, and no combination of despill, despeckle or refiner settings could remove it. The guard now grows the hint's contribution outward from the alpha the model itself predicted, instead of switching between the two at a fixed distance. Detail the model corroborates is restored, including a fly-away strand with no path back to the body; spill and stray blobs the model drove to zero have nothing to grow from and stay suppressed. There is no radius left to tune, so nothing needs to scale with resolution. ☼ Rebuild refiner_additive_guard around morphological reconstruction ☼ Seed the fill from every pixel the model gave alpha to, not just the body ☼ Drop the shell_px parameter, which no longer has meaning ☼ Clamp alpha to [0, 1] after the Lanczos upscale in both engines, since the filter overshoots at sharp edges and out of range alpha corrupts premultiplication ☼ Cover the matte invariants with tests: no dark band enclosed by opaque alpha at any hint width or resolution, interior holes still filled, wisps still protected The matte now sits marginally wider at the edge, following the hint, which is the same behaviour the reference implementation has. That is the milder of the two outcomes; the band it replaces was far more damaging in a composite. Verified on both reported clips at 5120x2700 and 1920x1080, across hint generators, colour spaces and model resolutions. The enclosed dark band drops to zero, no pixel loses more than 0.12 alpha, and a chroma keyed clip is unchanged to the bit.
☼ Bump 2.1.4 -> 2.1.5 in pyproject and all 17 READMEs, with localized release notes per language ☼ CHANGELOG entry: dark edge band around the subject in AI alpha mattes on high-resolution footage (#179), alpha clamped to the valid range after upscaling
The guard is a few full-frame passes plus the reconstruction loop, all plain max/min selections. Running them in numpy on the CPU cost a fixed ~250-350 ms per frame regardless of how fast inference got. The same work as one upload, on-device passes, and one download takes ~5 ms at 1080p and ~20 ms at 5K, under half a percent of per-frame inference at any resolution, with output identical to the CPU path on every test input including captured production frames. ☼ Add a torch path for the whole guard: elementwise passes, seed dilation via max_pool2d (exactly a rectangular dilate), the reconstruction loop, and the weight resample all stay on the device ☼ Keep the cv2/numpy implementation as the reference and as the fallback; any device failure falls back per frame instead of failing the render, and machines without a GPU are unaffected ☼ Probe CUDA first, then MPS, once per process ☼ Cover the dispatch and both paths with equivalence tests: forced-CPU fallback, reconstruction-loop parity, and full-guard parity on a fixture with a detached strand
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #179.
High-resolution footage could show a thin dark line tracing the subject at a small offset from its edge, and no despill, despeckle, or refiner setting could remove it.
The step that blends the model's matte with the alpha hint was drawing that line once the footage got sharp enough. It now blends the two without ever drawing a line, and the work runs on the GPU, so it adds no meaningful render time at any resolution.
☼ Dark edge gone on the footage that reproduced it; unaffected mattes stay unchanged
☼ Verified with every hint model, both color spaces, green and blue screens, 1080p through 5K
☼ New tests lock the fix in place
☼ Version 2.1.5 with changelog and localized release notes