Changes in Mooncake 0.5.25 that may affect DI users

[sunxd3]

Mooncake.jl v0.5.25 is about to be released.

Full changelog:

• All commits since v0.5.24

A few changes in v0.5.25 look relevant to DifferentiationInterface's Mooncake backend:

• If DI is used with Config(friendly_tangents=true), user-facing cotangents / gradients for structured matrix inputs are no longer returned as Symmetric, Hermitian, or SymTridiagonal. Mooncake now returns a plain Matrix{T} for these cases. This can affect gradient, value_and_gradient, and related pullback outputs that rely on friendly tangents. (#1103)

• tangent_to_primal!! is deprecated and will be removed in Mooncake 0.6. DI currently uses it in its Mooncake extension, so some DI code paths may emit a deprecation warning until DI is updated. This is mainly relevant when friendly_tangents=true. (#1103)

• Prepared Mooncake caches are now validated more strictly. Reusing a Mooncake-backed DI prep with different arity, top-level argument types, or array sizes than it was prepared for now throws PreparedCacheSpecError; previous versions could silently return incorrect results. (#1121)

• Config now has two additional fields: chunk_size::Union{Nothing,Int} and enable_nfwd::Bool. Both have defaults, so Config() and Config(friendly_tangents=true) still work. The compatibility risk is mostly for positional Config(...) construction. enable_nfwd=false is also the new opt-out for the nfwd-backed forward-cache path. (#1111, #1120)

[gdalle]

Thanks @sunxd3 for keeping us posted. As mentioned in chalk-lab/Mooncake.jl#1103 (comment), I would have liked this to happen in a breaking Mooncake release, but I was overruled. If you have time to take a crack at the DI fixes, I'll appreciate it, otherwise I'll add them to my to-do list

[gdalle]

Mooncake versions starting at 0.5.25 were excluded from DI's compat bounds in the General Registry:

• DifferentiationInterface: Fix broken Mooncake version JuliaRegistries/General#152662

[ChrisRackauckas-Claude]

Flagging the downstream impact since this cap is now blocking more than just DI users directly:

ComponentArrays 0.15.34 added Mooncake.friendly_tangent_cache(x::ComponentArray) to its Mooncake extension (SciML/ComponentArrays.jl@29ddd04), which requires Mooncake ≥ 0.5.25. Combined with the DI 0.7.16 cap at Mooncake 0.5.24 from JuliaRegistries/General#152662, any environment that pulls in both DI and ComponentArrays 0.15.34+ via the Mooncake ext becomes unsatisfiable:

• DI 0.7.16 forbids Mooncake > 0.5.24
• ComponentArrays 0.15.34+ requires Mooncake ≥ 0.5.25
• → resolver is forced to downgrade ComponentArrays to 0.15.33 (pre-friendly_tangent_cache), which defeats the reason 0.15.34 was released

This is currently blocking every Mooncake-loading test job in SciMLSensitivity.jl (see CI on SciML/SciMLSensitivity.jl#1421 and on master #1418), because ComponentArraysMooncakeExt fails to precompile with UndefVarError: friendly_tangent_cache not defined in Mooncake whenever the resolver picks 0.5.24.

No action needed on your side beyond what's already tracked here — just wanted to flag that the DI fix to Mooncake 0.5.25's new API is now on the critical path for downstream SciML CI, not only DI's own users. Happy to help with the tangent_to_primal!! migration if that'd be useful.

[thorek1]

Mooncake 0.5.25 fixed some type inference issues in MacroModelling but with the new DifferentiationInterface WeakCompat restriction (Mooncake 0.5.24) they become inaccessible. This is jsut to say that there are DifferentiationInterface users who would benefit from making DifferentiationInterface comptaible with Mooncake 0.5.25 onwards. Thank you for your efforts in any case

[gdalle]

I am acutely aware of the problem, and I plan to review #988 this week.

[gdalle]

First round of review done on #988, waiting for answers before going further