SPy features needed for a potential paper (loop fusion, SIMD, *args)

[paugier]

Personal tracking issue for the SPy-side prerequisites to a paper on SPy as a basis for the scientific Python ecosystem. Potentially targeting MPLR 2027 (Prague), work-in-progress research paper track?

The same content can be presented at EuroPython/EuroScipy 2027.

Scientific/technical question. Can SPy — a statically-compilable subset of Python — serve as the implementation language for the basis of the scientific Python ecosystem (NumPy/SciPy-like array libraries), rather than requiring a C/C++/Cython/Rust layer underneath?

Content. Demonstrations of SPy's suitability for numerical-kernel implementation:

• metaprogramming (blue, metafunc, blue unroll, variadic *args) as the mechanism behind array-API-style generic functions;
• loop fusion of array expressions;
• SIMD vectorization via vectorize, itself written in pure SPy rather than hand-written intrinsics;
• a short section on simple parallelism.

We do NOT want loop fusion itself landed in SPy, only what loop fusion needs.

See the existing checklist for the loop-fusion demo itself:
https://github.com/paugier/spy-demos/tree/loop-fusion/loop-fusion

Use more generic classes

• nonlocal A = ndarray[DTYPE, NDIM] syntax for generic struct (see draft: Generic class type alias spylang/spy#529)

Less repetitions

• __add__ = make_add_metafunc(DTYPE, NDIM) ?

Of course, we could also use Protocol but it has to be supported by SPy.

Cleaner redshift output

• Redshift: no current module name in FQN by default? spylang/spy#590
• format rs output

UnrollFor (blue unroll)

• Blue unroll scoping improvements (Antonio, in progress)
• Blue unroll itself (Antonio, next)

Starred / SpliceUnroll (*args)

Depends on UnrollFor landing. Relatively big but shouldn't need months once
UnrollFor is done. See https://fluiddyn.pages.heptapod.net/spy-book/design-more-metaprog/

• B1 — Parser: Starred in Tuple.elts / Call.args / Subscript args
• B2 — astcompile recognition (Starred / SpliceUnroll markers)
• B3 — m_args.types
• B4 — Blue-interpreted consumption
• B5 — Doppler consumption + *args: <type-tuple-expr> sugar

Use new meta programming in stdlib/array.spy

• Nicer implementation of array (multidimensional with metaprog)

SIMD / vectorize

• Aligned-pointers PRs: Aligned pointers and pointer reinterpret-cast spylang/spy#676, Cast align spylang/spy#686
• align_offset (https://github.com/paugier/spy/tree/align_offset)
• Clean SIMD implementation with the discussed API (draft MRs mostly exist);
  shouldn't take months once aligned pointers are merged

Parallelism

I don't think the paper should include a demo on parallelism but it can discussed in the conclusions/perspectives. There is an AI generated plan for simple parallelism strategy (not yet discussed with Antonio).