Skip to content

feat(sql): add configurable WebAssembly scalar UDFs - #2513

Open
ksushant6566 wants to merge 1 commit into
getdozer:mainfrom
ksushant6566:feat/1653-wasm-udf
Open

ksushant6566 wants to merge 1 commit into
getdozer:mainfrom
ksushant6566:feat/1653-wasm-udf

Conversation

@ksushant6566

Copy link
Copy Markdown

Adds configurable scalar WebAssembly UDFs to Dozer SQL. With the wasm feature enabled, a .wasm export can be registered in the existing UDF list and called from projections, aggregate expressions and WHERE predicates:

udfs:
  - name: increment
    config: !Wasm
      path: ./math.wasm
      function: addOne

SELECT increment(value) FROM input calls addOne; omitting function uses the SQL UDF name. The generated configuration schema and CLI feature forwarding are included.

The runtime uses Wasmi and infers i32/i64/f32/f64 signatures. It supports checked integer conversion (including Int8), float arguments, SQL NULL propagation, and AssemblyScript env.abort. Modules are compiled during planning and each row gets a fresh instance with fuel, memory and table limits. Invalid argument types are rejected during non-aggregate planning, including SelectionProcessorFactory builds; aggregate argument types are checked against the post-aggregation schema. The included AssemblyScript example has a pinned npm lockfile and a standalone SQL runner.

The supported scalar ABI has one numeric return. Strings/objects through linear memory, reference types, multiple returns and WASI are outside this implementation.

Demo

Watch the approximately 25-second MP4, or download the original terminal recording.

AssemblyScript compilation, YAML configuration and actual Dozer SQL results

The recording shows a real AssemblyScript build, integer/float results, NULL propagation and an assertion error through Dozer's SQL parser/planner/evaluator. It uses in-memory rows; an ingestion connector is not started. Reproduction instructions are in dozer-tests/wasm_udf/README.md.

Validation

  • cargo test --locked -p dozer-sql-expression -p dozer-sql --features wasm --no-default-features: 216 SQL tests, 35 expression unit tests and 5 SQL/WASM integration tests pass; one pre-existing SQL test remains ignored.
  • cargo test --locked -p dozer-sql-expression --no-default-features --test wasm_udf: feature-disabled regression passes.
  • cargo test -p dozer-types udf_yaml_deserialize: all 4 configuration tests pass.
  • cargo clippy --locked -p dozer-sql-expression -p dozer-sql --features wasm --all-targets --no-default-features: succeeds with warnings, including the existing large error types. Changed Rust files pass rustfmt; git diff --check passes.
  • cargo check --locked -p dozer-cli --features wasm --no-default-features: passes with Rust 1.78 on macOS arm64. The installed Rust 1.98 hits the unchanged time 0.3.34 E0282 error with or without WASM; no unrelated dependency upgrade is included.
  • npm ci and the AssemblyScript build/example run succeed.

Coverage includes numeric boundaries, invalid modules/exports/imports, arity and type errors, NULLs, traps, fuel/memory/table limits, module-state isolation, source-file removal after planning, cloned expressions, nested/aggregate SQL calls, and real WHERE processor insert/update behavior.

AI assistance: implemented and tested with Codex.

Fixes #1653
/claim #1653

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

WASM UDF support

1 participant