feat(sql): add configurable WebAssembly scalar UDFs - #2513
Open
ksushant6566 wants to merge 1 commit into
Open
ksushant6566 wants to merge 1 commit into
ksushant6566 wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds configurable scalar WebAssembly UDFs to Dozer SQL. With the
wasmfeature enabled, a.wasmexport can be registered in the existing UDF list and called from projections, aggregate expressions and WHERE predicates:SELECT increment(value) FROM inputcallsaddOne; omittingfunctionuses the SQL UDF name. The generated configuration schema and CLI feature forwarding are included.The runtime uses Wasmi and infers
i32/i64/f32/f64signatures. It supports checked integer conversion (including Int8), float arguments, SQL NULL propagation, and AssemblyScriptenv.abort. Modules are compiled during planning and each row gets a fresh instance with fuel, memory and table limits. Invalid argument types are rejected during non-aggregate planning, including SelectionProcessorFactory builds; aggregate argument types are checked against the post-aggregation schema. The included AssemblyScript example has a pinned npm lockfile and a standalone SQL runner.The supported scalar ABI has one numeric return. Strings/objects through linear memory, reference types, multiple returns and WASI are outside this implementation.
Demo
Watch the approximately 25-second MP4, or download the original terminal recording.
The recording shows a real AssemblyScript build, integer/float results, NULL propagation and an assertion error through Dozer's SQL parser/planner/evaluator. It uses in-memory rows; an ingestion connector is not started. Reproduction instructions are in
dozer-tests/wasm_udf/README.md.Validation
cargo test --locked -p dozer-sql-expression -p dozer-sql --features wasm --no-default-features: 216 SQL tests, 35 expression unit tests and 5 SQL/WASM integration tests pass; one pre-existing SQL test remains ignored.cargo test --locked -p dozer-sql-expression --no-default-features --test wasm_udf: feature-disabled regression passes.cargo test -p dozer-types udf_yaml_deserialize: all 4 configuration tests pass.cargo clippy --locked -p dozer-sql-expression -p dozer-sql --features wasm --all-targets --no-default-features: succeeds with warnings, including the existing large error types. Changed Rust files pass rustfmt;git diff --checkpasses.cargo check --locked -p dozer-cli --features wasm --no-default-features: passes with Rust 1.78 on macOS arm64. The installed Rust 1.98 hits the unchangedtime 0.3.34E0282 error with or without WASM; no unrelated dependency upgrade is included.npm ciand the AssemblyScript build/example run succeed.Coverage includes numeric boundaries, invalid modules/exports/imports, arity and type errors, NULLs, traps, fuel/memory/table limits, module-state isolation, source-file removal after planning, cloned expressions, nested/aggregate SQL calls, and real WHERE processor insert/update behavior.
AI assistance: implemented and tested with Codex.
Fixes #1653
/claim #1653