Add integer overflow detection to VM and compiler

[vbergeron]

Summary

This PR adds proper overflow detection for integer arithmetic in the VM and compiler. Previously, integer operations used wrapping arithmetic, which could silently produce incorrect results. Now, INT_ADD, INT_SUB, and INT_MUL detect when the exact result falls outside the 24-bit signed range [-2^23, 2^23 - 1] and return a VmError::IntOverflow error instead.

Key Changes

• VM (encore_vm/src/vm.rs): Replace wrapping arithmetic with checked arithmetic for INT_ADD, INT_SUB, and INT_MUL. Operations that overflow the 24-bit range now trap with VmError::IntOverflow { pc }.

• Value module (encore_vm/src/value.rs): Add INT_MIN and INT_MAX constants and int_in_range() helper function to define and check the valid 24-bit signed integer range.

• Error handling (encore_vm/src/error.rs): Add IntOverflow { pc: u16 } variant to VmError enum.

• Compiler constant folder (encore_compiler/src/pass/cps_simplify/simpl_03_constant_fold.rs): Update eval_int_binop() to use checked arithmetic and return Option<Val>. Expressions that would overflow are left unfolded, allowing the VM to trap at runtime with the same semantics.

• Code emitter (encore_compiler/src/pass/asm_emit.rs): Add assertion to reject integer literals outside the valid range at compile time, with a clear error message.

• Documentation (VM.md): Update the integer value description to clarify the exact range and overflow behavior.

• Tests: Add comprehensive test coverage:

  • VM tests for operations at the boundaries (test_int_add_at_max, test_int_sub_at_min, test_int_mul_at_min)
  • VM tests for overflow detection (test_int_add_overflow, test_int_sub_overflow, test_int_mul_overflow, test_int_mul_overflow_i32)
  • Compiler constant folder tests for boundary cases and non-folding of overflowing expressions
  • Emitter tests for literal bounds checking and rejection of out-of-range literals

Implementation Details

The overflow detection uses Rust's checked_* methods on i32, which return Option. For arithmetic operations, we check both that the operation doesn't overflow i32 and that the result fits in the 24-bit range. Comparison operations (INT_EQ, INT_LT) are unaffected since they always produce boolean results.

The compiler maintains the same contract: the constant folder leaves out-of-range arithmetic unfolded (so it traps at runtime), and the emitter rejects integer literals outside the range at compile time.

https://claude.ai/code/session_01L5Xv4Npfn2U5QBKUqV9Dbu