Simplify effects further - #4515
Merged
Merged
Conversation
Prims today declares PURE, GHOST and DIV as the primitive effects, with Tot/Pure, GTot/Ghost and Div/Dv as abbreviations of them. We would like to flip that, so that Tot, GTot and Div are primitive and the others are abbreviations. That cannot be done in one step: the fixed stage0 binary has to be able to process the flipped Prims before the flip can land, so the compiler must first stop caring which spelling is primitive. This does that, and nothing else: it is meant to be behaviour-preserving against today's Prims. Parser.Const gains two groups of definitions. Three *class* predicates, is_pure_effect_lid / is_ghost_effect_lid / is_div_effect_lid, accept any spelling of each effect, and are what a *classification* test should now use. Three aliases, primitive_pure_lid / primitive_ghost_lid / primitive_div_lid, name whichever spelling Prims actually declares, and are what a *construction* site should use. Flipping the primitives is then a change to those three aliases and to Prims. Roughly forty hardwired lident comparisons across the typechecker, the SMT encoder, extraction and the printer are routed through them. The distinction matters: a comp carries a specification, but an lcomp and a residual_comp do not, so a test on one of those cannot be widened from Tot/GTot to the whole class without silently discarding a specification. Parser.Const says so where the predicates are defined. Three things beyond the mechanical rewrite: - Env.is_erasable_effect tested Prims.GHOST alone, relying on GTot unfolding to it. It now tests the ghost class. Under the flip norm_eff_name lands on GTot instead and erasure silently stopped firing, which is how this was found. - Tot and GTot no longer reject a requires or ensures clause. They are the pure and ghost effects with an empty specification, so there is no reason they should not take one, and Prims needs to write Admit as Tot a (ensures False) once Tot is primitive. The dedicated Total and GTotal comps are now built only when the computation type has no further arguments at all. Bug250 and OptionalSpecs asserted the old rejection; they now assert that such a specification is type-checked and proved, which was Bug250's original complaint. - The TOTAL cflag is recomputed from the actual specification, rather than inherited. Sig_effect_abbrev stores the flags of the abbreviation's *body*, so once an abbreviation bottoms out at Tot every use of it inherits TOTAL, including uses that add a specification -- and Rel.solve_c_aux short-circuits on is_total_comp c1 && is_total_comp c2 and drops the specification on the floor. TOTAL is a property of an occurrence, not of an effect. Also adds Env.comp_to_comp_typ_with_univs. comp_to_comp_typ infers universes with env.universe_of, which needs the result type's free variables to be in scope; unfold_effect_abbrev was calling it on the body of an abbreviation, whose free variables need not be. That was harmless only because such a body is always a Comp today. Validated with make 2, make test-3 (stage 3, Pulse and examples) and make test fsharp-all boot-diff test-2-bare stage2-unit-tests, all green with no golden file changes. Separately, a copy of ulib with Prims, Pervasives, All and Tactics.Effect flipped so that Tot, GTot and Div are primitive now lax-checks with results identical to the unflipped copy across all 313 modules; before this commit it failed immediately in Prims. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Snapshot of stage2, so that stage0 knows that the choice of which spelling of the primitive effects Prims declares is not hardwired into the compiler. This is what lets the next step actually flip Prims: the fixed stage0 binary has to be able to desugar the flipped Prims before the flip can land. Rebuilt from the new stage0 from clean (make clean-1 clean-2 clean-3 && make 2), green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Flip the primitive effects in Prims: Tot, GTot and Div are now declared primitive, and Pure, Ghost and Dv become front-end-only abbreviations that ToSyntax unfolds. A computation type no longer carries a specification: a precondition desugars to a trailing implicit #(squash P) binder and a postcondition to a refinement of the result type, so all logical content lives in binders and in guard_t. Work in progress: stage 1 and stage 2 are green, Pulse (stage 3) is down to 14 errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Three places treated an unsolved expected type as authoritative: - close_x now flattens nested refinements before introducing the existential, so a fact stacked in the *sort* of an outer refinement can still be closed instead of falling back to substituting an effectful term into a type. - check_inner_let keeps the let's own result type when the expected type is a bare flex -- which is what a match branch is checked against. - drop_spec_args unfolds an abbreviated head type, so a precondition's implicit binder is erased consistently in types and in applications. Pulse (stage 3) goes from 18 to 11 errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
With preconditions desugared to trailing implicit #(squash P) binders and
postconditions to result-type refinements, comp_typ no longer needs to carry
comp_pre/comp_post: an arrow's meaning is entirely in its binders and its
result type, and every remaining obligation lives in a guard_t.
* drop comp_pre/comp_post from comp_typ and update Subst, Free, Hash,
VisitM, InstFV, CheckLN, Class.Binders, Positivity, NBE, Resugar,
Print.Ugly, Reflection.V2.Builtins and the SMT encoder;
* retire U.comp_pre/comp_post/is_trivial_post/mk_conj_post/apply_post and
the trivial_pre/trivial_post constructors' users;
* tc_comp no longer typechecks a specification, and reifying a computation
raises no precondition obligation of its own;
* set_expected_typ_of_comp reduces to comp_result;
* retire `effect Admit`: it is now `val admit: #a:Type -> unit -> Tot (_:a{l_False})`.
Also fix check_expected_effect: maybe_assume_result_eq_pure_term now refines
the *result type*, so under use_eq -- where the expected type must match
exactly -- adding an equation the caller did not ask for turns a successful
check into an unprovable obligation. Skip it in that case.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Both are fallout from preconditions becoming implicit #(squash P) binders.
1. Do not solve a flex under a squash from the left.
`squash A <: squash ?q` used to be harmless: a Lemma application had type
`unit`, which could not solve `?q`. Now it has type `squash A`, and
unifying commits `?q := A` on the strength of the left-hand side alone.
`?q` is normally determined by a *later* constraint -- the expected type of
the enclosing application -- which is exactly how FStar.Classical.Sugar's
`introduce`/`eliminate` elaboration fixes the metavariables of
`implies_intro` and friends; committing early made every such idiom fail
with "Failed to resolve implicit ... : prop".
Defer the problem instead, so the determining constraint runs first. The
guard is `defer_ok <> NoDefer` rather than `= DeferAny`, because the first
forcing point is solve_non_tactic_deferred_constraints at DeferFlexFlexOnly.
2. Solve single-valued implicits created under binders.
try_solve_single_valued_implicits only recognised implicits of type `unit`
or `x:unit{...}`. A `#(squash P)` implicit created under local binders --
inside a match branch, say -- is abstracted over them and has type
`bs -> squash P`. Eta-expand the unit solution for those.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Now that a Lemma's precondition is a trailing implicit #(squash P) binder and
its postcondition a refinement of the result type, a handful of Pulse proofs
need to say explicitly what the old comp-carried specification said for them.
The recurring patterns:
* a lemma called inside a subterm no longer exports its fact outward: hoist
it to a `let _ = ... in`, or end an `introduce ... with` block with an
explicit `assert` of the fact that must escape;
* an expected type is not propagated into a dtuple2 component, so a proof
component written as `(fun q m' -> ())` loses its implicit precondition
binder; give it its own Lemma-typed signature;
* point-free re-exports of interface-declared symbols (`let later = later`)
are opaque to SMT; transport across them by conversion instead, via a
`conv_squash` helper and `_ by (T.trefl ())`;
* inference sometimes coarsens or over-refines a type that used to be
pinned by a postcondition: ascribe it (`(SZ.v n <: nat) == cap`,
`let new_spec : table_spec = ...`);
* `Classical.move_requires` applied to a lemma with no `requires` no longer
typechecks, since the trivial precondition binder is suppressed; call
`forall_intro` directly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
An lcomp defers its comp to a thunk that also returns a guard. While a computation type still carried a specification, that guard was mostly bookkeeping and the obligations were in the comp itself; now the obligations *are* the guard. bind_cases in particular returns the match-exhaustiveness check that way. tc_tot_or_gtot_term_maybe_solve_deferred returned the lcomp unforced when it was already Tot/GTot, and typeof_tot_or_gtot_term reads only res_typ -- so the exhaustiveness check was silently discarded on that path. Pulse's check_match_complete goes through exactly there, and started accepting non-exhaustive matches (pulse/test/nolib/MatchRange.fst). Force the lcomp in that branch and conjoin its guard. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rk on lemmas A comp no longer stores a postcondition, so inspect_comp was returning a degenerate C_Lemma (True, fun _ -> True), and MApply0.apply_squash_or_lem could no longer see that a lemma's conclusion is an implication. Add U.post_of_result_typ, the inverse of U.refine_with_post, and use it in inspect_comp for C_Lemma and C_Eff; pack_comp rebuilds the result type with refine_with_post, so inspect o pack round-trips. Because a Lemma now carries the TOTAL flag, try_unify_by_application's "Codomain is effectful" bail no longer fires and plain apply succeeds on a lemma -- but then leaves a uvar for its _:unit argument. Fill unit-typed binders with (), exactly as t_apply_lemma has always done. A consequence is that mapply on a lemma discharges its precondition by SMT (the #(squash p) implicit is single-valued), rather than leaving a goal; the book's Part5.Mapply snippet no longer needs the focus/smt() scaffolding. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
An abbreviation's ensures becomes a refinement of the stored comp's result type, which survives unfolding, so there is no reason to reject it any more. A requires would have to become a binder on an arrow the abbreviation does not have, so it is still rejected (Error 184). TcEffect's "Result type of effect abbreviation does not match" check must unrefine the definition's result type before comparing it to the declared one. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…urce type find_coercion applies a candidate to exactly one explicit argument, so the coercion's source is its last *explicit* binder. A coercion written with a Pure ... (requires ...) type now ends in a #(squash _) binder, which was being mistaken for the source type. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A postcondition is a refinement of a result type now, so the shape let r = match ... with ... in lem r; r in a function whose result type is refined gives the match's result variable both lower bounds (the branches) and a refined upper bound. Flex_rigid outranks Rigid_flex, so the refinement was becoming part of the variable's definition and every branch was then asked to prove the postcondition, at the branch's own source position. Defer in that case so the lower bounds win and the refinement stays an obligation of the Flex_rigid problem. This generalises a rule that was previously restricted to typeclass variables; it also no longer requires the refined bound to be the one being solved, since the bounds are met pairwise. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pulse matches slprops syntactically, and a primop reduced SZ.v (SZ.uint_to_t 0) to 0 in the assertion but not in the context; bind the size explicitly. The norm/primops string-concatenation assertion in StringNormalization now genuinely succeeds, so drop its expect_failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
FStar.Tactics.NamedView's open_comp/close_comp took a *view* of a comp, but RD.Tv_Arrow hands out a comp that is still closed with respect to the arrow's binder. Taking the view introduces the postcondition's own [fun (_:unit) ->] binder, which then captures the free index 0 of the enclosing arrow, so a round-trip through the named view silently dropped an outer binder. There is no way to repair this after the fact: FStarC.Syntax.Subst can replace a name or open index 0, but has no de Bruijn shift, and none can be built out of subst_elt. So make open_comp/open_comp_with/open_comp_simple/close_comp/ close_comp_simple operate on the raw R.comp, inspecting and packing only at the boundary, once the comp is fully opened. The lossy subst_comp helper is deleted. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[Prims.nonempty] is discharged entirely by [clearly_inhabited]. Now that a
computation type has no postcondition to record the value of a pure term in,
[assume_result_eq_pure_term] states it as a refinement [x:t{x == e}] on the
result type instead, so that shape reaches [clearly_inhabited] for any
definition whose body ends in a literal. Accept it: such a refinement is
inhabited by [e] exactly when [t] is.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A postcondition is a refinement of a result type now, which puts refinements on
lower bounds where there used to be none, and the join was too eager to keep
them.
- Joining two genuinely different refinements over a common base produces a
disjunction that is not the type either side was written at, and that nobody
downstream can use. [assert (f x y == f y x)] is enough to hit it: [eq2]
ends up indexed by a disjunction of the two arguments' postconditions, and
[apply]/[apply_lemma] can no longer unify against it. Widen to the base
instead -- sound, since these are lower bounds. [False] is first treated as
the unit of the join, and syntactically equal refinements are kept, so the
ordinary cases are unaffected.
- A lower bound of the form [_:t{False}] -- the result type of a computation
that never returns, e.g. [raise] -- says nothing, but committing to it forces
every other lower bound to establish [False]. Drop the refinement.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Five changes, all consequences of a precondition being a trailing implicit [squash P] binder and a postcondition a refinement of the result type: - proc_guard now turns a stranded [squash]-typed implicit into a goal. Outside tactics such an implicit is solved with [()] and its [phi] discharged with the guard; Rel.try_solve_single_valued_implicits deliberately does nothing in tactic mode. Without this, a tactic that merely elaborates a lemma application with an unmet precondition reported an uninstantiated unification variable rather than the obligation. - __exact_now retries after stripping a top-level refinement, and then falls back on SMT-free subtyping. A term built by applying a function with an [ensures] clause now has a refined type even when the goal does not. - t_apply auto-fills an *anonymous* [unit] argument, so that plain [apply] meets the [_:unit ->] argument of a lemma. It must be [unit] on the nose -- a [squash p] argument is an obligation, not noise -- and anonymous, since a named [(u:unit)] is part of the caller's interface. - t_apply_lemma peels a trailing implicit [squash] binder off the arrow and uses it as the precondition goal, restoring the historical goal list. - do_subtype: SMT-free subtyping inside a UF transaction, for the above. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- assume_safe takes a [squash False -> Tac a]: TacF cannot carry a [requires] any more, since an effect abbreviation has no arrow to hang a precondition on. Callers must write [fun _ -> ...], as a unit *pattern* would force the binder's type to [unit] and erase the [False]. - pose_lemma is just [pose_apply], a new variant of [pose] that applies the term rather than using it exactly, so that a lemma's precondition -- now a trailing implicit argument -- becomes a goal instead of an unsolved implicit. All the machinery that took a [Lemma] computation apart is gone: a lemma application is an ordinary term of type [squash ens]. - Bump an rlimit in Pulse.Lib.HashTable.Spec, and adapt the tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
With `Lemma (ensures Q)` desugaring to `Tot (squash Q)`, the `Lemma` abbreviation's argument is instantiated with `squash Q`, which does not satisfy `hasEq`. Every lemma was therefore dragging a bogus `Prims.hasEq (Prims.squash ...)` proof obligation -- and, because F* chains obligations so that earlier ones become hypotheses for later ones, an accidental extra *hypothesis* mentioning the whole postcondition -- into its VC. Widen the argument to `Type`. `eqtype_u` becomes unused in-tree but is kept, since it is exported. Removing the accidental hypothesis perturbs the solver context, which pushes `FStar.Algebra.CommMonoid.Fold.Nested.double_fold_transpose_lemma` over the default rlimit; bump it locally. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
`Normalize.term_to_string`/`term_to_doc`/`comp_to_string`/`comp_to_doc` run a full, strong normalization purely to tidy a term up for display. A term that appears in an error message has no reason to terminate: for `let rec f x : Dv nat = f x in f`, the printer unfolded the fixpoint forever, and `Bug2876.fst` grew to 34GB of residency over 52 minutes before being killed. The existing handler catches `Stack_overflow` but not the OOM. Exclude Zeta in the steps used for printing. A folded `let rec` also prints better than an unfolded one. Non-recursive local lets are handled by an earlier case that does not consult `zeta`, so they are unaffected. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
An earlier commit removed the `focus (fun () -> ...; smt())` wrappers on the assumption that a lemma's precondition would be discharged as a single-valued implicit. Now that a tactic-stranded proof-obligation implicit becomes a goal again, the scaffolding is required. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When a flex variable is solved by joining its lower bounds, a bound may mention names that are not in the variable's scope, and can therefore never be assigned to it. A postcondition is a refinement of a result type now, so this arises for something as ordinary as match x with C y -> assert (p y) whose branch has type `squash (p y)` while the match's result type is a variable created before `y` was bound. Widen such a bound by dropping the offending refinement. The widened type is still above the bound, so the problem is still solved; we merely claim less about the match's result. Upper bounds are left alone, since dropping a refinement there would be unsound. This is a latent bug: the same failure is reachable today by writing the refinement out by hand. Fixes RecordFieldOperator, PatternMatch.IFuel, Bug3207c and OPLSS2021.IFC. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A definition's result computation type has its precondition turned into a trailing implicit binder, exactly as a `val` does. The clause was only looked for in tagged form, so let f (x:nat) : Pure nat (x > 0) (fun y -> y > 0) = x kept its precondition in the ascription, where it became an assertion the definition could not discharge. Classify the arguments the way `desugar_comp` does, and weaken a positional clause to a tagged `requires True` rather than dropping it, so that the remaining positional arguments still mean what they did. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
An inner let moves e2's logical obligations into c2's lcomp guard, so that bind puts them under the binding. It conjoined them *after* the guard c2 already carried, i.e. after the obligations e2's own nested binds had deferred -- so the conjuncts came out in reverse program order. The solver proves a conjunction of obligations left to right, assuming each conjunct while proving the ones that follow, so this lost every such hypothesis, and a chain of lets reported one error per obligation instead of one. Also drop magic_dump_t's trailing 'exact (`())': apply now auto-fills magic's anonymous unit argument, so there is no goal left for it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A function with a precondition now takes an extra trailing implicit `squash` argument, so a `primitive_step` written for the old arity fires one argument early and the leftover `#()` is applied to its *result*: `FStar.Int8.mul 11y 11y` reduced to `FStar.Int8.int_to_t 121 #()`, which the SMT encoder emits as an `ApplyTT` chain and cannot equate with `121y`. Filtering the surplus argument inside `reduce_primops` does not work -- `rebuild` applies arguments one at a time and re-invokes the step after each -- so instead give the steps their true arity, with `with_extra_args`. It bumps `arity` (which `NBE` reads too) and truncates the argument list before the interpretation sees it. `add`/`sub`/`mul` and, for the unsigned kinds, `shift_left`/`shift_right` are the operators with a nontrivial `requires`. Second, a closed application of a primitive operator is replaced by its value before the solver sees it, so the callee's typing axiom never fires and a refinement on its result is lost. That refinement can be the only statement relating the value to the operation -- `FStar.UInt32.lognot 0xff00ul` reduces to `0xffff00fful` and nothing records that these are complements -- so capture it in the enclosing application's result type, as we already do for data constructors. Third, when a top-level definition's effect is masked, drop an inferred refinement from its type: that refinement is the computation's postcondition, which under partial correctness holds only if the computation returned, and so cannot be claimed of a value. A written annotation is left alone; `check_nonempty_result` makes the user justify it. All of tests/extraction is green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The normalizer tracks the local scope in its own closure environment and never extends cfg.tcenv, so a type read off a residual comp or a monadic lift annotation may mention variables tcenv has never heard of. That was harmless while computation types carried no logical content; now that a result type is a refinement carrying the postcondition, such a type routinely mentions the binders the postcondition talks about, and reify_bind/reify_lift's calls to universe_of trip the defensive well-scopedness check (Bug3236, Error 290). Reintroduce the free variables from the sorts they already carry before asking for the universe. A universe is determined by sorts alone, so this changes no result -- it only tells the check what the normalizer already knew. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Now that a postcondition is a refinement of the result type, several
places that used to coarsen a result type freely -- because the type
carried no logical content -- silently throw away the specification.
1. value_check_expected_typ ended with an unconditional
[set_lcomp_result lc t'], overwriting whatever weaken_result_typ had
just decided to keep, even when t' was a bare unification variable.
A match branch is checked against a bare uvar, so a branch that was a
plain name came back with res_typ [?u]; tc_match's "all branches
agree" rule could then not fire and the match got a
guard-conditioned refinement over the base type instead of the
branches' shared type. Gate the overwrite on TcUtil.keep_res_typ.
2. check_no_escape weakened a result type mentioning an out-of-scope
variable by dropping the *whole* refinement. normalize_refinement
flattens nested refinements into one conjunction, so that also threw
away the user's own annotation: the body of
[let rec h ... in h 2] checked against [y:int{y>=0}] came back as
[int]. Drop only the conjuncts that mention an escaping variable.
3. tc_match's fallback combine_branch_res_typs read each branch's
result type with the match's own should_return, which is false for a
pure match. A pure branch then contributed no refinement at all,
where a WP-based bind_cases used to contribute its result equation
under the branch's guard. Read the branches with should_return set:
the enclosing [x == <the whole match>] equation is opaque to the
solver as soon as the scrutinee is symbolic.
4. check_top_level_let's masked-effect branch replaced the result type
with [U.unrefine], which is wrong when the definition has a val: the
val is the interface. check_let_bound_def now returns the opened
annotation rather than a boolean so the branch can use it.
Also ascribe an empty match's result type in phase 1 as well as phase 2:
the type is an unsolved metavariable there, and the ascription is what
carries phase 1's generalization into phase 2.
Fixes Bug016, Bug058, Bug379, Bug1097, Bug1362.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
be436d2 generalised "solve a flex variable from its lower bounds first" from typeclass variables to any flex with a refined upper bound. That is too broad: with ap : ('a -> Tot bool) -> 'a -> Tot bool evenb3 : i:int{i>0} -> Tot bool the application [ap evenb3 1] gives ?a the refined upper bound [i:int{i>0}] (from evenb3) and the bare lower bound [int] (from the literal). Preferring the lower bound solves ?a := int and the argument then fails, where solving from the upper bound leaves [1 <: i:int{i>0}], which the literal's own result equation discharges. Lower bounds can only be preferred when they say something: require one of them to be refined. The motivating shape -- a match whose branches carry their result equations, under a refined expected type -- still qualifies. Fixes Bug026. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A function with a precondition now takes a trailing implicit [squash] argument, so [Pulse.Simplify]'s syntactic matchers for [FStar.SizeT.add], [sub], [mul] and [uint_to_t] no longer see the argument list shape they expect. Filter to explicit arguments before matching. Also: annotate [Printers.ff_bnd] with [simple_binder] rather than [binder]. An annotation on a let with an effectful right-hand side is now authoritative -- the computation type has no postcondition left to restate the sharper type in -- and [check_inner_let] cannot keep the sharper type instead without growing the result type of a chain of effectful lets in proportion to the chain (it makes lax-checking FStarC.SMTEncoding.Encode diverge). Record that in a comment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[collect_arr] does not push the arrow's binders into the environment, so the codomain it returns is open and normalizing it raises "Variable n not found" for a dependent signature like [eq_to_bv]. That path is only reached when [apply] and [apply_lemma] have both failed, so the error replaced an honest "can't apply" with one pointing into the lemma. The [apply_lemma] failure it was masking is a real change: an interface's [Lemma] postcondition is now a refinement of the definition's result type, so it guides the elaboration of the definition's body. In X64.Poly1305.Bitvectors_i the body asserts [logand #64 x 0 == (0 <: uint_t 64)] while the interface says [logand #64 x 0 == 0]; the goal that reaches [bv_tac] is now [eq2 #int], which [eq_to_bv] cannot apply to. Ascribe in the interface as well. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[solve_t'_aux]'s local [equal] helper decides an equation between two
interpreted heads by normalising both sides and comparing the results.
That is unbounded, and it is the only place in the unifier where a single
equation can cost seconds: reducing [FStar.UInt.logand] at width 64
unfolds [to_vec] on a symbolic argument, ~1.5s per side, and cannot
succeed.
This branch made that reachable where it was not before. A postcondition
is a refinement of a result type now, so [U64.logand x y] has type
[_:U64.t{UInt.logand (v x) (v y) = v _}], and [assert (logand x y ==
logand y x)] gives [eq2 #?a] two refined lower bounds. [meet_or_join]'s
[combine_refinements] asks [same_formula] whether they are the same
formula, which reaches [equal] with both sides ground -- where upstream
every such problem still carries a uvar and the [no_free_uvars] gate
keeps [equal] out. tests/tactics/TestBV.fst went from 0.93s to 12.7s.
Worse, the answer is discarded: [combine_refinements] widens to the base
type either way.
Add [eq_norm_heuristic_ok] to the worklist, alongside the existing
[umax_heuristic_ok]. It defaults to true, so every existing caller is
unchanged; [same_formula], and only [same_formula], turns it off.
That call site is safe by construction: it asks a syntactic question --
are these the same formula, modulo universes? -- and both answers are
already handled, so a conservative "no" costs inference precision and
never an SMT obligation. Two earlier attempts were not safe this way and
were withdrawn; a third, keying off [smt_ok] instead, broke the [unify]
tactic in tests/micro-benchmarks/UnifyMatch.fst, which genuinely needs
the normalisation to relate [nat2unary 10] and [S (nat2unary 9)].
TestBV.fst is back to 0.91s against master's 0.92s. doc/ref records the
diagnosis, the measurements and the three rejected fixes.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Record the fourth alternative considered for the TestBV blow-up -- making [same_formula] purely syntactic again -- with its measurements: it removes the blow-up, `make ci` and the full kuiper regression are green, and an instrumented build shows the fallback firing zero times across ulib and the test suites. Explain why it is so hard to observe (two formulas differing only in universe uvars join to a redundant but logically equivalent disjunction; only the [may_widen] path actually drops a refinement), why the fallback was kept anyway, and what the better long-term shape would be. Also drop the note about the benchmarking bot's run: it was about one superseded commit and is not worth carrying. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Master's Custard was written against the old effect surface, where a comp could name an abbreviation and Env.norm_eff_name had to resolve it. A comp_typ.effect_name is always a root effect now, so Effects.of_lid and RegEmb's TAC comparison drop the call; and add_modul_to_env lost the erase_univs parameter that existed only to erase universes from eff_decl.binders, so Loader stops passing N.erase_universes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
key_of_comp built the monomorphization key from ct.comp_pre and ct.comp_post, and from the Total/GTotal comp nodes, none of which exist now: a comp is a single Comp node whose comp_typ is an effect name, a result type and flags. The key is now the effect name and the result type. source_effect_name is deliberately not in the key. It is presentation only, so keying a Lemma apart from the Tot it is an alias of would emit two identical definitions under two names. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Custard decides an arity twice from the same type and the two decisions
have to agree. A definition's signature is read by Mono.classify,
which deletes unit-shaped binders itself because only it has the
codomain to tell a thunk from a proof; an anonymous arrow is read by
ty_of_typ, which keeps them. Rule 1 (is_dropped_binder) therefore
exempted anything U.is_unit accepts -- and U.is_unit unrefines, so it
accepts squash p as well as unit.
On this branch a precondition elaborates to a trailing implicit
#_:squash p binder, so that exemption split the two worlds on every
function with a requires: classify deleted the binder by its own unit
rule and ty_of_typ kept it, and the two met at a call site as an
ill-typed partial application. It also reached Builtins through
prim_app: Warning_CustardRuleArity fired 257 times in one make ci.
A squash p binder can never be a thunk -- F* writes a thunk as
unit -> ..., never as squash p -> ... -- so it needs no codomain to be
decided and rule 1 can delete it directly. U.is_exactly_unit is the
test: unlike is_unit it does not unrefine, so it accepts Prims.unit and
rejects squash p and _:unit{p}.
tests/extraction/SquashArgErasure.ml.expected is regenerated: it was
written against the ML backend before master converted the directory to
Custard, and the output it pinned did not type-check.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
app_of_fv asked erasable_app -- 'a saturated pure or ghost call whose
result is non-informative is ()' -- before consulting the rule table.
A name a rule interprets is a name whose F* type does not describe what
it does at runtime, so a shortcut that reads exactly that type has no
standing over it.
That was safe on master only by accident. Prims.admit was declared in
the effect abbreviation Admit a = PURE a (ensures fun _ -> False), and
U.is_pure_or_ghost_comp resolves no abbreviation, so it answered no and
the call survived for the wrong reason. admit is honestly
Tot (_:a{False}) now, the shortcut fires, and every admit () -- including
the one Pulse emits for Tm_Admit -- was silently replaced by (). The
abort() disappeared from pulse/test/Bug356.c.expected.
The same would hold of Prims.magic and FStar.Pervasives.false_elim, all
of which have non-informative results by construction.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
prim_app filtered its call spine with Mono.erased_binders while checking
the resulting arity against Mono.erased_binders_unfold. The two differ
on the binder Mono.keep_thunk puts back, and on anything an abbreviation
hides.
FStar.Pervasives.false_elim is #a:Type -> unit{False} -> Tot a and its
rule has arity 1. Once the unit{False} binder became droppable, the
filter without keep_thunk deleted both binders, the rule was
under-applied, and prim_app eta-expanded it into a lambda of unknown
representation:
let CFalseElim.g__lam (eta: any) : any [Impure] = <abort>
let CFalseElim.g (sq: unit) : u32 [Pure] = CFalseElim.g__lam
Error 368: Custard lost the representation of 2 value(s)
The spine is now filtered by erased_binders_unfold, the same list the
warning counts. Mono.retained_sorts and retained_names -- which name
and type the binders an eta-expansion introduces, and so must index that
same list -- share a new retained_binders that derives its flags the
same way.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Filtering prim_app's spine with keep_thunk over-supplies the rules in the other direction. Pulse.Lib.Array.null is #a:Type0 -> array a: every binder is erased, so keep_thunk's becomes-a-value clause restores the last one -- which here is the type binder. app_of_fv' handles exactly this, passing () for any position Mono.unit_binders flags, because a binder keep_thunk restored is there for its arity and for nothing else. prim_app had no such step, so the restored argument was left over and applied to the rule's result: uint32_t *a = (uint32_t *)NULL(); // tests/custard/pulse/ArrTup let null_x : Prims.int ref = ((Obj.magic 0) ()) (* pulse/test/Null *) A rule replaces a name rather than calling a definition whose arity has to be preserved, so for a rule such a binder is not an argument at all. prim_app now drops the left-over arguments unit_binders flags before the left-over-argument warning considers the rest. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
const_of_arg reduces a Mono argument to the constant a C++ non-type
template parameter will see, peeling the wrappers a size index normally
arrives in -- Ghost.hide, uint_to_t. It peeled by taking the
application's last argument.
FStar.SizeT.uint_to_t is x:nat{fits x} -> Pure t (requires ...), so on
this branch 16sz is uint_to_t 16 () and the last argument is the squash
witness. tests/custard/TmplLet, TmplLet3 and TmplMono stopped with
error 390, 'this external type is applied to a constant that cannot be
a template argument ... and neither is unit', about an argument the
source never wrote.
const_of_arg now drops () arguments before it looks at the spine.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A specialization's readable suffix is built from its Mono arguments by hint_of_term, which rendered Const_unit as "unit". A precondition now arrives as a trailing implicit squash binder, so 16sz is FStar.SizeT.uint_to_t 16 () rather than FStar.SizeT.uint_to_t 16, and every specialization on a bounded-integer constant acquired a _unit component: MonoAttr_f__uint_to_t_16_unit. The component appears in every such name, distinguishes none of them from any other, and eats the width budget fit has for the components that do. Const_unit now yields no hint; when () is all a specialization has, hint_of_args falls back to the sequence number, which is the right answer for an argument that carries no information. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A lift's source and target must be root effects now, so sub_effect PURE ~> PLAIN is error 52, raised from lookup_effect_lid_for_lift. The lift functions keep their lift_PURE names, as in tests/micro-benchmarks/Erasable.fst. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A new subsection of section 10 for the whole interaction: the two worlds that decide an arity and the predicate that has to be exact about where they differ, the rule table's standing over the erasability shortcut, the binders keep_thunk restores and what a rule may do with them, the squash witness const_of_arg has to peel past, and the () that names nothing. Six new rows in the regression index. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ffects Resolved: pulse/src/ml/Pulse_Extract_CompilerLib.ml, where master's rename_let support adds lbattrs_of_binder to the letbindings this branch builds from primitive_pure_lid/primitive_div_lid rather than effect_PURE_lid/effect_DIV_lid. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Sep 24, 2026
Open
Merged
This was referenced Sep 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Make
Tot/GTot/Divprimitive, and move specifications out of computation typesThis replaces the design in #4508 / #4510 (pushing an expected postcondition
through the typechecker). That approach kept the Hoare specification inside a
comp_typand worked around the consequences; this one removes it fromcomp_typaltogether, so the consequences do not arise.120 commits, 331 files,
+10029 / −4170. Of that, 328 files and+6966 / −4170are code and tests; the remainder is this document,regression_questions.md(two accepted regressions worked out in detail) andrevise_primitive_effects.md(the original design brief, kept for the record —where it and this document disagree, this document is what was built).
The two representations that went away
F* had
PURE/GHOST/DIVas primitive effects, withTot/GTot/Divasabbreviations of them — and, separately, dedicated
Total/GTotalconstructors in
comp'carryingPrims.Tot/Prims.GTot. One concept, threerepresentations, each with its own hardwired lident comparisons (~140 of them).
Independently, a
comp_typcarriedcomp_preandcomp_post, so an arrow'smeaning was split between its binders and a specification buried in its
codomain. That split is the source of the "arrows compared without their
pre/post" bug class, and it is what forced the expected-postcondition machinery.
After this PR:
Pure,GhostandDvbecome ordinary front-end abbreviations that areunfolded and desugared away before the typechecker ever sees them, and
A computation type is now a label and a result type. Obligations live in
guard_t, where they were always meant to live. (source_effect_namecarriesno meaning of its own — see "An effect abbreviation is a bare alias" below.)
comp_univswent with them. It was there to carry the universe instance of apolymonadic effect's
wp, and a computation type has nowpany more: everyone of its ~50 read sites either passed the list straight back to a
mk_Compthat reconstructed the same comp, or fed it to a
wpcombinator that no longerexists. The universe of a comp is now recovered where it is needed, from
result_typ, which is the one place it was ever really recorded.Removing it is what made the next simplification possible.
lcompis goneTypeChecker.Common.lcompwas a computation type whosecompwas behind athunk:
It existed because building a
compused to be expensive — it meant composingwps — while the three fields callers usually wanted (the effect, the resulttype, the flags) were cheap. So the expensive part was deferred, and forced only
if someone actually needed it.
After the flip those three fields are the whole of a
comp. What is left ofan
lcompover acompis one thing: a deferredguard_t. So the type isreplaced throughout the typechecker by the pair it had become —
lcompcomplcompwith a deferred guardcomp & guard_tTcComm.lcomp_comp lclc, Env.trivial_guardlcomp_with_bindercomp_with_binder = option bv & comp & guard_tand 12 API functions (
mk_lcomp,apply_lcomp,lcomp_set_flags,is_total_lcomp,residual_comp_of_lcomp, …) collapse onto theirSyntax.Utilcounterparts on
comp. Three more retire outright, having become the identityafter the flip:
TypeChecker.Util.weaken_precondition,should_not_inline_lcand
lcomp_has_trivial_postcondition, together withNormalize's fourghost_to_pure_*_lcompvariants.The one thing that needs care is that a thunk was forced inside the scope of
the binders its guard mentions.
TcUtil.bindcloses a continuation's guard overthe bound variable and weakens it with
x == e; that used to happen to whateverthe continuation's thunk produced when
bindforced it. So an eager rewrite hasto hand those obligations to
bindexplicitly rather than conjoin them into theambient guard —
tc_matchpassesbind_cases' guard asbind's continuationguard, and
tc_eqnweakens and closes each branch's obligations over thepattern variables itself.
The resulting verification conditions are, if anything, cleaner: a chain of
forced thunks used to leave behind vacuous quantifiers like
forall (base: nat). base == base ==> P, which are simply absent now. ulibverifies in 1m21s wall / 12.6 CPU-minutes at
-j16, against 1m35s / 13.2 before.Two
expect_failureannotations change, both because error recovery got morehonest.
weaken_result_typused to record the expected type on thelcomp'sres_typfield alone, leaving thecompinside the thunk with the type that hadjust been rejected; the inconsistency then produced a second, spurious error.
Bug655.fstno longer reports a bogus "GTotandSTATEcannot be composed"after a subtyping failure, and
Bug3213.fstreports both of its offendingarguments instead of one plus a cascade.
The
cflaglist went from five constructors to two. It wasand it is now, in full:
TOTAL,MLEFFECTandLEMMAare all gone. Each was a restatement of theeffect name, which is now always reliable, so each had a reader that tested the
name anyway:
MLEFFECTwas set exactly wheneffect_namewas alreadyFStar.All.ML.LEMMAis nowsource_effect_name = Prims.Lemma, which is whatis_lemma_compandis_smt_lemmaread.TOTALis nowPC.is_pure_effect_lid (comp_effect_name c), which is the wholeof
Syntax.Util.is_total_comp.The last one took two steps and is the reason the other two could go.
TOTALwassprinkled on every
Tot-named comp, residual comp andbindresult, but it hadone use that was not redundant: it recorded that a comp's effect was an
abbreviation rooted at
Tot, such asLemma— something the effect name didnot say, and which
is_total_comphas no env to look up. So it was firstnarrowed to that single job, and only deleted once the desugarer began resolving
abbreviations away, making
effect_nameunconditionally a root effect. See "Aneffect abbreviation is a bare alias" below.
Two things fell out of the narrowing:
TypeChecker.Util.weaken_flagsbecamedead, and
mk_bindlost itsflagsparameter along with the standingTODOabout
bind's flags being inconsistent with the comp it returns.Where the specification went
In the only two positions where a computation type may appear:
E t (requires P) (ensures Q)becomes... -> #(_ : squash P) -> E (x:t{Q x})— the implicit binder goes last, soPmay mention the explicit bindersPhere, and ascribeE (x:t{Q x})The precondition becomes a proof argument: the caller must supply it, F*
instantiates it by unification, and the obligation is raised at the call site
with the caller's hypotheses in scope. The postcondition becomes a refinement of
the result type, which is exactly what a caller learns.
Both are suppressed when trivial, so the overwhelming majority of code is
untouched.
Lemma
Lemmais the unit-result instance of the same rule, and is no longer special:Since
squash Qis_:unit{Q}, this is the general rule att = unit. Twothings fall out:
Lemma's postcondition was thunkedprecisely so the precondition could be assumed while checking the post's
well-formedness (Requires clause not assumed when checking well-formedness of ensures clause #57). With
#(_:squash P)bound to the left of the codomain,Pis in scope for free.thunk_ens,unthunkandunthunk_lemma_postaredeleted.
Tot (squash phi)andLemma (ensures phi)are now the same type, so thebespoke subtyping rule for that pair is deleted too.
The SMT encoding of a
Lemmais unchangedThis was the main risk: ~5300
Lemmaoccurrences, ~1080 withrequires. Iftrigger selection or the quantified-binder set shifted, proofs would fail
diffusely and far from the cause.
It does not shift. The comp still records that the user wrote
Lemma(
source_effect_name) and still carries itsSMTPATflag, and the post iswritten with the
squashfvar, so the encoder recovers everythingstructurally:
prefrom the trailing squash-typed implicit binder,postfrom the argument ofsquash, and the quantifier ranges over the realbinders only. For
the emitted axiom is
— byte-for-byte the shape emitted before. Verified across no-
requireslemmas, multi-binder lemmas with
SMTPatOr, universe-polymorphic lemmas withfuel instrumentation, and lemmas with a quantified
ensures.An effect abbreviation is a bare alias
Before this PR an effect abbreviation could take binders and give its
right-hand side a specification:
Neither could mean anything. A computation type supplies exactly one argument —
its result type — so every binder but the first was already dead, and once a
comp carries no specification the
ensuresabove is silently dropped:x -> MyTot bchecks asx -> Tot b. (An earlier commit on this branch,7e71460e09, added support for anensureshere; this reverses it. Making itmean what it says would require refining the result type at every use site of
the abbreviation, and a
requireswould have to become an implicit binder on anarrow the abbreviation does not have.)
The machinery keeping that shape alive was substantial:
Env.norm_eff_name(~50 call sites),
lookup_effect_abbrev,unfold_effect_abbrev,TcEffect.tc_effect_abbrev,eff_decl.univs/binders, and theTOTALandLEMMAcomp flags, which existed only to record env-free facts about anot-yet-unfolded abbreviation.
An abbreviation is now what it always was in substance: another name for an
effect.
ToSyntaxresolves it away, socomp_typ.effect_nameis always aroot effect and the typechecker never unfolds anything.
comp_typgainssource_effect_name, which records the name the user wrote so that errormessages, IDE hovers,
Syntax.Resugarandinspect_compcan still sayLemma,TacorSt. It is presentation only, with one exception:Lemmaroots atTot, soU.is_lemma_comp/is_smt_lemma— and hence whetherSMTEncoding.Encodeemits a lemma's axiom — read it.Sig_effect_abbrevshrinks tokept only so that a module read from a
.checkedfile can rebuild itsDsEnv.The canonical surface form is
effect M = N. The eta-expanded spellingeffect M (a:Type) = N ais still accepted, becauseulibhas to stayparseable by the bootstrap compiler in
stage0; everything else is now rejected(Error 316) rather than silently misinterpreted.
tests/bug-reports/closed/Bug1370b.fstpins down the accepted and refusedforms.
Two hand-built-syntax sites named an abbreviation where a root effect is
required, and only worked before because
norm_eff_namecleaned up after them:Pulse.Extract.CompilerLib(DIV,PURE) andis_ml_comp/ thefail_expletbinding (
ML).Three neighbouring pieces of surface syntax go with it:
redefine_effect(effect M = N <: ...) is gone from the grammar. It wasthe only other production for
NEW_EFFECT.[attributes ...]clause on an effect abbreviation or redefinition isgone — the
ATTRIBUTEStoken, the production, theAttributessurface-ASTnode and the
cattributesplumbing it fed inToSyntax. The only flag itever produced was
CPS, which went away with Dijkstra Monads for Free(
7e468aa485), leaving a match with nothing but a wildcard raising "Unknownattribute". Nothing in
ulib,examples,tests,docorpulsewrites it.effect lattice and an abbreviation is not a node of it;
sub_effect PURE ~> Mworked only because
ToSyntaxquietly resolved it first. Now thatPURE,GHOSTandDIVare abbreviations ofTot,GTotandDiv, write theeffect. The error message names the effect the abbreviation stands for, so the
fix is in the message.
A fourth, from the same clean-up of how a computation type's arguments are read:
a universe application on an effect, as in
Tot u#0 int, is now rejectedrather than accepted and dropped. A computation is an effect applied to its
result type, so its universe is that type's and there is nothing an annotation
could add. It was recorded in
comp_univsbefore this series and has beensilently discarded since. The commit that does this (
7c9426d2c8) alsointroduces
sort_comp_args, a single classifier for "which argument is theresult type, which the pre, which the post".
comp_requires— which lifts aprecondition out of a codomain into an implicit binder — used to scan for an
index in a way that had to agree with
desugar_comp's own classification butshared no code with it, so a definition could acquire a binder that its
valdoes not have; and
desugar_compclassified twice. Both are now driven fromsort_comp_args, andLemmais simply the effect that has no result type andmay carry SMT patterns.
A total effect's universe comes from its representation
TcUtil.universe_of_compdecided the universe ofM tbywhich is unsound for any total effect whose
reprdoes not preserve universes.Given
M boolis inhabited by a(t:Type u#0 & bool), so it belongs inType u#1;answering
u#0letunit -> M boolpass as aType u#0while really carrying aType u#1value — an embedding ofType u#0intoType u#0.FStarC.TypeChecker.Core.check_compalready had this right: for a total effectit built
repr tand took its universe. The main typechecker and the corechecker disagreed, and the main one was wrong. They now share
Env.effect_universe.Rather than re-derive the representation's universe at every arrow,
TcEffect.tc_eff_declreads it offrepronce, when the effect is declared, andstores it in
eff_combinators.repr_universeas the schemeso that instantiating it at the universe of a result type gives the universe of
the computation type. This is a function of
u_aalone:repr's codomainuniverse is fixed by its type.
The rule cuts both ways. A
reprthat lowers the universe — sayrepr (a:Type u#a) : Type u#0 = bool— makesM tsmaller thant, where theold rule wrongly reported
u_res;unit -> M (Type u#5)is now correctly aType u#0.Unchanged: a partial effect still answers
u#0, since an arrow into one is not atype of values (
unit -> Dv t : Type0for anyt); andTot,GTotand anyother
total assume effecthave no representation to consult, so they stillanswer with the universe of the result type.
This bug is not one the surrounding refactor introduced —
masterhas thesame three lines — but it is one the refactor's own test effects walk straight
into. Nothing in ulib or pulse declares a total effect with a representation, so
nothing there moves.
tests/micro-benchmarks/SimpleEffects_ReprUniverse.fstpins it.
The size of an elaborated term
A postcondition is now a refinement of the result type, and a result type is part
of the term. That is fine in the two places a specification is written, and it
was a serious problem in one place it is inferred.
Meta_monadic/Meta_monadic_liftannotate a monadicletor application withits result type, as a hint for reification and extraction —
tc_termdrops thetype when it re-checks such a term, and extraction ignores it. Recording the
inferred type there meant recording a postcondition that embeds the very terms
it describes: the definiens of a pure let, the result of every branch of a match.
Effectful code binds at every step, so the copies nested, and the elaborated term
grew multiplicatively with the nesting depth. Reducing
FStar.Tactics.Visit.visit_tmover a term of size n took time exponential inn:
tests/bug-reports/closed/Bug3210.fstwent from 0.52s to 1214s, andFStar.Tactics.Visit.fst.checkedfrom 151KB to 546KB.Recording the bare type (
5f60b4c352) puts Bug3210 back to 0.57s, makesvisit_tmflat in the size of the visited term again, and brings the checkedfile to 255KB. Before specifications moved into the result type this information
lived in the WP, which was never part of the term, so this restores the size
annotated terms used to have.
Getting a variable out of a type
The other consequence of an inferred postcondition being part of the type: it
mentions the terms it is about, so it routinely mentions variables that are
about to go out of scope — a
let-bound name, amatchpattern variable, thenames of a
let rec. Five commits converge on a single discipline here, and itis worth reading them together.
8f70eb8b3f). An inferred refinement that mentions anescaping variable used to have the offending conjuncts deleted. Quantify the
escaping variables existentially instead: they witness the existential
themselves, so this is still a weakening, but simplification then applies the
one-point rule and the fact survives.
_ == xwithx : natused to leavenothing behind and now yields
_ >= 0;_ == f x /\ x == 3is recovered as_ == f 3. The whole formula is closed at once rather than conjunct byconjunct: with
yescaping,x == y /\ y == zis recovered asx == z, whichclosing separately would reduce to nothing. The quantified binders' sorts are
normalized, since the one-point rule restates the eliminated binder's typing
hypothesis and cannot see it through an abbreviation — that is what turns
natinto
_ >= 0.let rec(3557bce2d2). For the names bound bya
let rec, the recovery above says nothing:exists (f: a -> b). _ == f niswitnessed by any constant function, while putting a higher-order quantifier in
every type derived from this one. So those conjuncts are not introduced in the
first place.
env.rec_namesrecords the names bound by thelet recwhosebody is being checked, and the four points in
TypeChecker.Utilthat would puta term in a type consult it:
should_return,bind_result_subst, thepure-substitution branch of
eliminate_binder_from_typ, andcaptured_typing.4c798eb6f4).check_no_escapeis that authority, but itlived in
TcTerm, out ofTypeChecker.Util's reach — soeliminate_binder_from_typhad a last case that returned its argument withxstill free and relied on
TcTermto notice, breaking the contract its namestates. Moving
check_no_escapeandescape_causeintoTypeChecker.Utildeletes logic: the case used to drop refinements with
U.unrefinewhen thathappened to suffice, and
check_no_escapedoes better — it normalizes first,so it sees through
squashand other abbreviations, closes what it canexistentially, and discards conjunct by conjunct rather than wholesale.
42f039a3c1). That lastresort used to substitute the bound term, which is exact for a pure or ghost
term and wrong for an effectful one, which may diverge and need not produce the
same value twice. Instrumenting the branch finds it reachable: one hit across
ulib and the test suite, at
tests/extraction/Micro.fstwithc1 = Div, whereit produced
squash (f11 (g11 x) == g11 x)— a type mentioning aDivapplication, which no source program could write.
d62b4d6194).bind_maybe_capturehad grown to ~500lines conflating four jobs: closing the binder, deciding how much of what
e1established is worth restating, simplifying degenerate binds, and building the
composite result type together with the
x == e1hypothesis. The driver is now34 lines.
composite_result_typis the sole authority on the result type, andits two ways of getting rid of the binder are separated:
bind_result_substsubstitutes
e1,eliminate_binder_from_typclosesxexistentially when itcannot. This is the type-side counterpart of the guard-side elimination, which
quantifies instead — types are closed by substitution, formulas by
quantification — and the two do not conflict: the substitution rewrites the
result type, where
xis not bound, while thex == e1equation goes on theguard under
Env.close_guard, wherexdeliberately stays.Smaller compiler fixes carried by this branch
Several of these are latent on
masterand were surfaced, not caused, by therefactor.
(
dc401f3935).check_implicit_solution_and_discharge_guarddischarged theguard with whatever range the environment happened to carry when the implicit
was finally resolved, which is typically the enclosing definition. The range is
now the implicit's own introduction site. This matters directly for the
squashimplicits that preconditions desugar to.binders (
a198fab809). The normalizer tracks the local scope in its ownclosure environment and never extends
cfg.tcenv, so a type read off aresidual comp or a monadic lift annotation may mention variables
tcenvhasnever heard of. That was harmless while computation types carried no logical
content; now that a result type carries the postcondition, such a type
routinely mentions the binders the postcondition talks about, and
reify_bind/reify_lift's calls touniverse_oftrip the defensivewell-scopedness check (Bug3236, Error 290). The free variables are reintroduced
from the sorts they already carry before asking for the universe; a universe is
determined by sorts alone, so no result changes.
has_typewas instantiated atu#0twice (691d7c8598), with a standingTODO. OnlyRel.guard_of_probwas still on that path, and the SMT encoderdoes encode universe arguments, so a formula about
x <: tat any otheruniverse was encoded against a symbol nothing else mentions. Both universes are
now computed at that site and
mk_has_typetakes them.fc8dbb0d71).examples/native_tactics/Registers.List.Testwas OOM-killed in CI (34 GB andstill climbing locally). When a native plugin cannot unembed its arguments —
because they are still symbolic —
arrow_as_prim_step_Nfalls back to a"shadow" application rebuilt from the arguments its generated wrapper handed
it, which exclude the universes and leading type arguments the wrapper stripped
off. The result is a strictly partial application of the same head:
sel #int r 1comes back assel r 1.reduce_primopsaccepted that as a reduction,after which the term could never reach the primitive step again — the plugin
was silently disabled for that occurrence even once its arguments became
concrete. Latent on
master; the primitive-effect flip made it reachable..cmxswas never rebuilt (f31316706f).load_native_tacticscompiles a plugin's extracted.mlonly when the.cmxsis absent; an existing one is dynlinked however old it is. After a compiler
rebuild every test in that directory failed with Error 353 ("interface mismatch
on
FStarC_TypeChecker_Util") or an undefined symbol, and the only cure was toknow to delete the objects by hand. The stamps already depend on
$(FSTAR_EXE),so the objects are dropped there now.
--ext optimize_let_vcis now inert (f8a8e05784). Keeping a let-boundvariable opaque in the VC —
forall x. x == e ==> phirather thanphi[e/x]—is no longer optional, and there are no layered effects left in
bindtoaccommodate. The key defaulted to true in
Options.Ext.defaultsand nothing inthe tree set it to false, so the disjunct it guarded was constantly false; the
flags still passed by pulse, examples and karamel become inert rather than
wrong, and are left alone. Two neighbouring dead branches go with it
(
is_layeredwas the literalfalse; anelsewas unreachable because theguard of the case above it contains
not is_let_binding).Tot/GTottest goes through aParser.Constpredicate(
8b19adb704). CollapsingTotal/GTotalintoCompturned every match onthose constructors into an open-coded
lid_equals ct.effect_name PC.effect_Tot_lid— 20-odd copies of the knowledge that generation 1 existedto remove.
is_tot_lid,is_gtot_lidandis_tot_or_gtot_lidare deliberatelydistinct from the class predicates (
PureandPUREare in the pure classbut are not
Tot), andSyntax.Utilgainsis_named_gtot/is_named_tot_or_gtotso a caller holding acompnever reaches for theeffect name. This found a latent inconsistency:
Normalizegave a reifieddivergent let-binding
lbeff = Dv.FStar.Rational.Gcd(14351b8174). The module headeralready warns that
is_gcdanddividesreliably produce matching loops withnonlinear arithmetic, and the module is written to keep them apart; one
assertwas proved with the recursive call's
is_gcdpostcondition in scope, and z3fired
primitive_Prims.op_Star22k times. Raising the rlimit does not help — itis a loop, not a marginal proof. Hoisting the arithmetic into a private lemma,
where the
is_gcdfact is not in scope, brings the module to 4.5s.Two generations, and a stage0 bump
src/is only ever lax-checked, so the sole hard bootstrap question is whetherthe fixed stage0 binary can desugar a flipped
Prims. It cannot: thecompiler hardwires
Prims.GHOSTinEnv.is_erasable_effect, which relies onGTot → GHOSTunfolding, so makingGTotprimitive silently stops erasure fromfiring.
So the flip could not land in one generation:
0444fb29c6) makes the compiler name-agnostic about whichspelling
Primsdeclares — one canonical classification of the pure, ghostand divergent effect classes, with every hardwired comparison routed through
it. No behaviour change. Then
make bump-stage0(0cdb18b5a5).Primsand removes specifications fromcomp_typ.Testing against EverParse
ciis not a big enough sample for a change this broad, so the branch was alsorun against EverParse's
fstar2branch — two clean clones built side by side, one with EverParse's pinned
toolchain to establish that the tree is green to begin with, one with this
branch's
stage3compiler. The pinned build reported zero errors, so everyfailure in the other build is a genuine difference attributable to this PR.
The experiment ran to a green build over several rounds (
-konly ever exposesone layer of failures at a time, since dependents of a failing module are
skipped). It found four more typechecker bugs and one extraction bug, all fixed
here:
Subtyping could not eta-expand across an arity mismatch. A precondition is
a trailing implicit binder, so
Pure t (requires p)has one binder more thanTot t.tc_absinserts a missing implicit for a lambda, but a point-freeterm had no way to bridge the gap.
try_eta_expand_to_expected_typinTypeChecker.Utilnow handles both directions — the term's type havingfewer binders than expected and having more, all of them implicit (which is
where an application lands).
eis applied to the shorter of the twoarities' worth of arguments, taken from the term's own type — whose sorts are
concrete, where the expected type's may still be uvars — while the
abstraction binds all of the expected type's binders, since
tc_absonlyever inserts leading implicits and the ones at issue are trailing.
It has to run before the subtyping check, not only in its failure branch:
relating
x:a -> Tot btox:a -> #_:squash p -> Tot bdoes not fail, itsucceeds with an unprovable
has_type b (#_:squash p -> Tot b)obligation. Soweaken_result_typtries it up front, on types that are already syntacticallyarrows (so the common case costs nothing), and again after subtyping has
failed, that time normalizing first. Eta-expanding an effectful term would
delay, duplicate or drop its effect, so both hooks are guarded by
is_pure_or_ghost_comp. This closes the follow-up that the "point-freedefinition" regression below asked for.
A refinement was dropped when joining two lower bounds under unsolved
universes. Two structurally identical refinements can differ only in the
universe uvar of an
eq2;U.term_eqcompares universe uvars by identity, socombine_refinementsconcluded the two bounds were genuinely different andwidened to the base type, silently losing the refinement. It now falls back to
try_eqon the two refinement formulas whenterm_eqsays no.try_eqruns with
smt_ok=false, so it can only unify structurally-equal formulasmodulo universe solving — applying it to the whole types instead would wrongly
identify
twitht{phi}.TypeChecker.Corerejected an unelaboratedletinside a type. Core'sTm_letcase typecheckedlb.lbtypunconditionally, but aletthat occursinside a type — e.g. the binder sort
(x:nat) -> squash (let y = x + 1 in y > 0)of a Pulse
fnargument — can still carry theTm_unknownthe desugarer leftthere. Core then failed with
Unexpected term: Tm_unknown. It now falls backto the definition's inferred type when the annotation is absent, which is
sound: an unannotated
let's type is its definition's type, and thesubtyping check it would otherwise perform is then reflexive.
It is worth being precise about where that hole comes from, because "Pulse
hands Core an unelaborated term" would be a much more alarming statement than
what is actually happening. Pulse does elaborate binder sorts:
Pulse.Checker.Abs.arrow_of_abssends each one throughPulse.Checker.Pure.tc_type_phase1, which callstc_tot_or_gtot_termwithphase1=trueandadmit=true. That call setsinstantiate_imp, and runssolve_deferred_constraintsandresolve_implicitsbefore returning, soimplicit arguments are inserted and solved;
let y = id 0 in y >= 0comesback fully applied. The one field phase 1 deliberately leaves blank is
lb.lbtyp, and it is this branch's own phase-1 code that leaves it blank:TcTerm.check_inner_letkeepslbtyp = tunwhen the source had no annotation(see the comment there), because phase 1 discards specifications and phase 2
reads
lbtypback as if it were a source annotation — recording phase 1'scoarser type would throw away the postcondition, which is now a refinement on
the result. So the hole is intentional, it is confined to that one field, and
the two consumers of phase-1 output are phase 2, which re-infers it by design,
and Pulse, which does not. Patching Pulse would mean asking it not to use
phase-1 elaboration at all; tolerating a missing annotation in Core is both
smaller and independently correct, since Core is a checker for arbitrary
well-scoped terms and an unannotated
letis one. Reached in practice onlythrough Pulse; the original repro was a
fnbinder ofLemmatype whoseensurescontained alet. Regression test:pulse/test/LetInLemmaBinder.fst.A
let recwhose result is a function lost itsensures. Anensuresisnow a refinement on the result type, so a definition returning a function is
annotated with a refinement of an arrow.
Syntax.Util.arrow_formals_compdeliberately looks through such a refinement to find the binders underneath,
and throws the predicate away — harmless for a caller that only counts
binders, fatal for one that rebuilds a type from what it got back. Two did:
TcUtil.extract_let_rec_annotation, which moves the annotation onto the bodyand so was checking the body against the unrefined arrow, and
TcTerm.guard_letrecs, which gives the recursive occurrence its type and sowas hiding the definition's own postcondition from its recursive calls. The
postcondition was then left to a single subtyping check on the whole
definition, discharged with none of the body's facts in scope, and typically
unprovable.
Normalize.get_n_binders_no_unrefinesplits with the strictsplitter, falling back to the old one only when that finds too few binders, so
it can never see less than before; the four sites in
extract_let_rec_annotationand the one inguard_letrecsuse it.Regression test:
tests/micro-benchmarks/LetRecRefinedFunctionResult.fst.Extraction left a precondition's proof argument behind. A
requiresis atrailing implicit
squashbinder, and extraction erases it:is_spec_binderrecognises it,
binders_as_ml_bindersdrops it from a lambda anddrop_spec_argsdrops the matching argument from an application. Butdrop_spec_argslooked for the binders in onearrow_formalsof the head'stype, unfolding it once if that produced too few.
It is deliberately liberal, and the liberality is not observable.
squash pis
x:unit{p}, so an argument of that type carries no information whateverits provenance; erasing it can only ever be right. Concretely, a use of such
a variable in the body extracts to
()whether or not its binder was kept:and the higher-order case stays consistent because the type is erased by the
same predicate:
#s:squash (1 == 1) -> int -> intextracts toPrims.int -> Prims.int, so a lambda, an application, and a value of thattype all agree.
Attributing the desugarer's binder is a one-line change at
ToSyntax.fst:1337— it is the only place an implicit
squashbinder is built — but it wouldmake erasure depend on provenance rather than on type, and provenance is the
thing that is easy to lose. Every path that rebuilds an arrow would have to
preserve the attribute:
Syntax.Util's arrow constructors, Pulse'sPulse_Extract_CompilerLib, the reflection API'smk_arrow, andTcUtil.extract_let_rec_annotation, which already demonstrably drops arefinement it does not know about (see the
let recfinding above). A singlemiss is silent: that one definition keeps the argument while its callers drop
it, which is exactly the ABI inconsistency the type-directed predicate cannot
produce. It would also need
cache_version_numberbumped, since avalchecked before the change and a
letchecked after would disagree.So: not done, and not because it is hard. If the attribute is wanted anyway,
the right form is a marker in
Prims(arequiresinsidePrims.fstitselfmust be able to mention it) plus a check in
is_spec_binderthat keeps thetype test as a fallback, so that a lost attribute degrades to today's
behaviour rather than to a mismatch.
That is not enough when the
squashbinder is inside the head type's result: forcallee : t_t -> Tot t_twheret_t = x:int -> y:int -> Pure r (requires ...),the visible arity is 1 and one unfolding of the whole type still exposes only
the outer arrow. The
()proof then survived into the generated OCaml as areal argument, and the ML typechecker rejected it with
Error 76: Ill-typed application.drop_spec_argsnow unfolds the result ofthe arrow it found, repeatedly, until it has as many formals as there are
arguments — bounded by fuel and by the unfolding reaching a fixpoint, so a type
that genuinely has fewer binders than arguments still costs one step.
Regression test:
tests/extraction/SquashArgErasure.fst.(A sixth problem, in the SMT encoding rather than the typechecker, was
root-caused but deliberately not fixed; see below.)
An open bug: obligations escaping a
letRel.try_solve_single_valued_implicitssolves anyunit- orsquash-typedimplicit with
()unconditionally and defers the proof tocheck_implicit_solution_and_discharge_guard, which re-typechecks the solutionunder
{env with gamma = imp_uvar.ctx_uvar_gamma}and discharges the guardthere.
gammacarries binder sorts and nothing else — no let-equations, nobranch hypotheses. So an obligation that a precondition raises can be discharged
in a context that has lost the very equation that proves it:
fails with
Failed to prove: m > 129, in a context that bindsmbut notm == n + 130. An annotated inner let is what loses it:check_inner_lettakes
x.sortfromU.comp_result c1, and the annotation has already forcedthat through
weaken_result_typ, discarding the refinement thatmaybe_assume_result_eq_pure_termwould otherwise have attached. Dropping theannotation, or writing
let m : (q:nat{q == n + 130}) = n + 130, or assertingthe equation (
assertis alet _ : squash p, which putspin a binder sort)all make it go through.
This is pre-existing, but this PR makes it far easier to hit, because every
precondition is now a
squashimplicit and so takes this path. It is left openon purpose: enriching an annotated let's binder sort would change the SMT
encoding of every annotated inner let in every F* program, which is not a change
to make blind at the end of a refactor. The workarounds are local and cheap.
A second open bug: a
squash pbinder is a weak SMT hypothesisPrims.squash pis_:unit{p}, but the encoder treats the two spellingsdifferently. A refinement type gets a
refinement_interpretationaxiom, so ahypothesis
HasTypeFuel f x _:unit{p}yieldsValid pin one E-matching step.Prims.squash pis an application of an uninterpreted symbol, so reachingValid pobliges the solver to first rewrite withequation_Prims.squashandthen match the refinement axiom up to congruence. On small goals it manages;
on large ones it sometimes does not, and the hypothesis is then silently useless.
Side by side, at the same call site:
This is not new — upstream F* fails identically on a hand-written
squashbinder — but it was rare, because upstream rarely produces one. This PR makes
every precondition such a binder, so the weakness is now reachable from ordinary
code. Its sharpest form is not a precondition at all but a typing hypothesis.
Checking
serialize (serialize_dsum_cases t f sr g sg tg) yh, whereyhisdeclared at
dsum_type t, leavessquash (has_type yh (dsum_cases t tg))inscope; the solver then cannot see that
serialize ... yhis aSeq.seq, and socannot prove
Seq.length (serialize ... yh) >= 0--- a goal that is true by theresult type of
Seq.length. That isLowParse.PulseParse.Sum.l2r_safe_writer_dsum_noroom_lemma, the one EverParsedefinition that hits this.
The workarounds all amount to putting the fact back into a binder's type,
where the refinement interpretation reaches it:
Three ways to close it in the encoder were tried and all three were rejected,
because each traded this rare failure for a different one:
squash pto the refinement it denotes, before encodingTm_refine_<hash>symbol and three axioms per distinct precondition shape; timed outCBOR.Spec.API.FormatHasType e unit /\ pfor a squash binder guardLowParse.Spec.Base.serializer_injectiveHasTypeFuel f x (Prims.squash p) ==> Valid pFStar.Tactics.CanonMonoidandFStar.Algebra.CommMonoid.Fold.Nestedin ulibEvery variant is a net-neutral trade of one rare instability for another, so the
encoding is left alone. Closing this properly means making the hypothesis
available lazily, in a way that does not also strengthen unrelated
squash-typed hypotheses — a change to make on its own, with its own measurement,
not at the end of a refactor.
A fourth attempt was made and also rejected: closing the query over a
squash pbinding asp ==> qrather thanforall (x: squash p). q(
Encode.encode_query). That is exactly the shape upstream produces, and it doesput
pin the solver's hypothesis set directly — but it fixed neitherl2r_safe_writer_dsum_noroom_lemmanor theMapGroupfailure below, whilerestating every precondition in every query. It was reverted.
A third finding: the content of a proof argument is not restated
CDDL.Pulse.Parse.MapGroup.impl_zero_copy_map_zero_or_more_auxwas the lastEverParse regression, and it is worth recording because the diagnosis is
counter-intuitive: the goal term and the hypothesis list are byte-identical
to upstream's, the axiom sets emitted for every symbol involved are identical,
and the proof still fails. The difference is a single extra ground fact.
The proof asserts
where
i.ser2 : erased (dfst (mk_spec r2) -> bool)andsp2.serializable : tvalue -> bool. The two arrow types are differentTm_arrow_<hash>symbols in the encoding — the domain is inside the abstraction,not an argument to it — so no amount of congruence on
dfst (mk_spec r2) == tvaluerelates them. The hypothesis in scope is
i.ser2 == hide (tvalue -> bool) sp2.serializable,and
lemma_FStar.Ghost.reveal_hidetriggers onreveal a (hide a x): it can onlyfire if the two
erasedtype indices are the same E-graph term. So the proofneeds the equation between the two arrow types, and nothing else will do.
That equation is exactly the
squash (a == b)argument the user's tactic solves.Taking the unsat core of upstream's query names it directly (
@hypothesis_135):upstream restates a bound term's type at every
bind, so the coercion's proofobligation is also published as a fact. This branch's
captured_typingrestatesonly what a binder's elimination would lose, and a tactic-solved implicit is not
that, so the fact is dropped.
The workaround is to state the equation the coercion rests on, once:
which is the same tactic already written inline for the coercion. The definition
then verifies in 32s, against 45s for the failing attempt.
Testing against kuiper
EverParse exercises parsing and low-level imperative code; it says little about
type-level computation, typeclasses, or Pulse's implicit-heavy style. So the
branch was run a second time, against
kuiper at
c1cd3c2d, using the same A/Bmethod: one clone built with the F* fork kuiper is developed against, one with
this branch merged with that fork (the merge is conflict-free and touches
nothing this PR touches). The baseline verifies all 396 modules with zero
errors, so again every difference is attributable to this PR. With the changes
below, the revised tree verifies all 396 modules too.
The interesting thing about kuiper is where it broke. EverParse's failures
were about specifications — an
ensuresthat went missing, a precondition thatthe solver could not use. Kuiper's were almost all about unification: a
requiresis now a binder, so it changes the shape of types, and fourseparate places in
Relturned out to handle refinements and proof-irrelevantuvars in ways that only worked because those shapes did not arise before.
A typeclass-constrained variable was solved from an upper bound. An
instance head never mentions a refinement, so committing the variable to a
refined upper bound makes the constraint unsolvable whatever the lower bounds
say. Upstream had a rule preferring lower bounds for exactly this; generalising
prefer_lower_boundsfor the postcondition-as-refinement shapes had droppedit. Restored as a disjunct, so the
Bug026case that motivated the extraconditions is unaffected.
Kuiper.Seq.Common.fsti'sseq_replace, whose++is
Kuiper.Monoid's typeclass-dispatchedmplus.refinement_of_flexfired on a bound whose base is the variable beingsolved. A recursive function with an implicit argument of inferred type —
Pulse's
(#[full_default ()] f: _)idiom — bounds that type byx:?u (n-1) {decreases ...}. Treating it as a head match makescombinebuild an equation that fails the occurs check; meet/join then gives up and the
caller widens the bound all the way to its base, dropping the refinement the
other bound asked for, so
permbecamereal. Leaving it aMisMatchkeeps the other bound intact.
Kuiper.SHMem.fsti'slive_c_shmems.Joining two lower bounds widened to a base neither side was written at.
combine_refinementswidens to the base type when the joined predicate isneither input's — the right thing when the two bounds' bases were already the
same type, since the disjunction of two refinements is rarely what a later
upper bound needs. But when the bases agreed only after delta-unfolding —
natlt n1andnatlt n2both reducing to a refinement ofnat— the base isa type neither side was written at, and widening to it throws away the very
information the bounds carry: joining them to
i:nat{i < n1 \/ i < n2}is whatlets the result meet a later upper bound of
natlt (max n1 n2). The wideningrule now applies only on the
try_eqpath, where the bases really were equal.Kuiper.IView.fsti'smerge_either, whose result was inferred at-> GTot nat. Regression test:tests/micro-benchmarks/JoinRefinedLowerBounds.fst.A flex-flex problem at a proof-irrelevant type invented a uvar.
solve_t_flex_flex's quasi-pattern rule allocates a fresh variable over theintersected binders and solves both sides to functions of it. When the shared
result type is
squash phithere is nothing to determine —()is its onlyinhabitant — and the fresh variable is simply never solved. This looked like a
fifth bug for a while and it is not: the
Error 217it produced came from anexperiment elsewhere, and with that reverted the rule is unnecessary. Recorded
here only because the shape is tempting: "solve both sides with
()" alsobreaks
tests/tactics/SolvedWitness.fst, whose whole point is thatassert True by (dup (); flip (); trefl (); qed ())does leave a witnessuninstantiated.
A goal that was open only in proof-irrelevant uvars was resolved too late.
resolve_implicits'defers a meta arg — a typeclass goal, in practice — whosetype or context mentions a free uvar, on the grounds that solving something
else may instantiate it (Order sensitivity for typeclass resolution? #3130). When nothing else can progress it gives up
and runs the tactic on the open goals anyway, in the reverse of the order it
first saw them, which is a much worse position to guess from. Since a
requiresnow desugars to an implicitsquashbinder, uvars that carry noinformation at all are everywhere, and both halves of that test started
misfiring:
has_pts_to (array2 et l) (frac (chest2 et (v (rows +^ 2sz)) d))counts asopen purely because of a
squashuvar in one of its arguments.Kuiper.Kernel.Stencil.fst'skpre.Kuiper.Sparse.Common.fst'sis_ematrix_tile_atis aPure prop (requires offset_chunk et j k nthr < cols), so its ownrequiresbinder is in scope while its body is checked — and the call it mentions has a
requires trueof its own, hence asquash trueuvar. That singleuninformative uvar makes
gamma_has_free_uvarstrue, so every typeclassgoal in the definition is deferred to the eager pass, where they are then
attempted in dependency-violating order:
has_vec_cpy et #?sruns before?s : sized etis solved, and instance search declines to guess?s.Before deciding whether a meta arg's goal is open, the loop now solves the
single-valued uvars of that goal and its context — the same
()-for-squash phistep the loop already performs, just targeted andearlier; their
phiis still discharged when the loop reaches their ownimplicit. Restricting it to the goal's own uvars is load-bearing: running the
general pass early instead re-broke
Kuiper.Seq.Common, because solvingunrelated single-valued implicits instantiated
monoid0 ?tto the refinedresult type before instance search ever saw it. Regression test:
pulse/test/PtsToSquashImplicit.fst.squash p <: squash qwas decided by equality, and diverged. This is themost serious defect the branch had, and it is the one that a downstream
campaign is uniquely good at finding: it needs no unusual feature, only a
proposition whose proof term is expensive to unfold.
Lemma (ensures p)is nowTot (squash p), so a lemma whose body is itself alemma call produces a subtyping problem between two squashed propositions —
what the body proves against what the enclosing lemma promises. Upstream that
problem did not exist: a lemma call had type
unit, and the postconditionarrived as a guard from the computation type. Both sides now have head
Prims.squash, sohead_matchesreported a match and the applicationcongruence rule fired, decomposing the problem into
p == q— an equalitybetween the two propositions — and then delta-unfolding both of them looking
for a syntactic match.
For arithmetic propositions that merely wastes a little time. For bitvector
propositions it does not terminate:
FStar.UInt.nth,logandandshift_rightunfold intoto_vec/from_vecrecursion, and the typecheckerallocates until the machine dies.
Kuiper.Bitmask.fst— 288 lines, 12s andunder a gigabyte upstream — took a single
fstar.exepast 561 GB ofresident memory before the kernel OOM-killer stopped it. It never once
completed on this branch, and because the failure surfaced as a killed process
rather than an error message it hid behind
make -k's exit status for severalrounds.
squash pis by definition_:unit{p}, so the two sides are related byimplication, not equality. The fix makes
squashtransparent to subtyping:the problem is unfolded to its refinement form and handed to the existing
Tm_refine, Tm_refinerule, which already knows to emitp ==> q— andalready knows how to treat uvars in
pandq, which is why the rewrite isdelegated rather than open-coded. Gating it on both sides being uvar-free was
tried first and does not fire: the
eq2on the right of a typicalensuresstill carries an unresolved universe. Reduced to ten lines of ordinary F* in
tests/micro-benchmarks/SquashSubtypingDivergence.fst; the fixed compilerchecks it in 1.01s against master's 0.99s.
Pulse reaches the same conclusion by a different route, and needed the same
rule again in
FStarC.TypeChecker.Core. There acalcjustification hasexpected type
unit -> Tot (squash (p y z)), the body has typesquash A,and
check_relation''sTm_app/Tm_appcongruence demandedA == Bviacheck_relation_args … EQUALITY. This one fails fast rather than diverging —it reports
A == true == B, which iseq2 (b2t A) Bprinted — but it is thesame confusion of proof irrelevance with syntactic identity.
Kuiper.Sparse.Matrix.PtsTo.fstneeded no downstream edit once it was fixed.Regression test:
pulse/test/CalcSquashSubtyping.fst.Downstream, kuiper needed 22 files, +99/-32 lines of code (+260/-36 with the
explanatory comments each change now carries). Most are the familiar
kind — an explicit type ascription, a dropped
Classical.move_requiresthat isnow redundant because the precondition is a binder, a calc justification
restated as the library lemma it was open-coding, a missing
lemma_divides_exactthat the old encoding happened to supply anyway, and an arithmetic hint or an
SMTPatlemma where afitsobligation is no longer a ground fact (see thefourth finding below). Six are more interesting:
Kuiper.Kernel.LogSoftmax.fsti'slog_softmax_realhad no result annotation,and its body sequences a
Lemmacall before returning. That postcondition isnow a refinement on the
Lemma'sunitresult, andcaptured_typingpropagates it onto the type of the
let-body, so the inferred result typebecame
chest1 real n {forall i. acc (softmax_real ra) i >. 0.0R}. Nocan_approximateinstance head mentions a refinement, so downstream resolutionfailed. Annotating the result type is the fix. This is the most general
downstream hazard in the PR: an unannotated definition whose body sequences a
Lemmanow acquires a refined type, which is usually harmless but is fatalto typeclass resolution.
Kuiper.Kernel.SDPA.Naive.fst'sscaled_add_approxproved aapprox2 (fun x y -> ...) (fun x y -> ...)goal withintroduce forall ... with introduce _ ==> _ with aux x y rx ry, where thetwo
_s of the implication are inferred fromaux's type — which is now... -> #_:squash (x %~ rx /\ y %~ ry) -> Tot (_:unit{...})rather than anarrow into
Lemma. The two holes are left deferred andtc_declreportsError 54.Classical.forall_intro_4 (Classical.move_requires_4 aux)provesthe same thing in one line and does not depend on inferring them; the
neighbouring
comb2_approx, whoseapprox2arguments are named rather thanlambdas, was unaffected. This one is a genuine inference regression rather
than a design consequence, but it resisted a small reproduction, so it is
recorded rather than fixed.
Kuiper.Example.ArrayView.Test.EvenOdds3.fst'sit_of_nat_lem_1carries anSMTPatmentioningit_of_nat vw i, whose second argument is refined byin_image vw.iview.step.imap.f i. Upstream proves that refinement bybrute-force unfolding — the baseline's unsat core names no lemma at all, just
merge_either,sum_aiview,even_view,odd_viewand friends. Here it mustbe said:
all_in_image, which already existed twenty lines further down, movesabove the two lemmas and loses its dependency on them, and the two lemmas
take the fact as a
requires. That is strictly better factored than what wasthere, but it is a real edit.
Kuiper.Tensor.Layout.Alg.fsti'sl4_batched_row_major_imapstates itsright-hand side in
SZ.tarithmetic, fourSZ.muls and threeSZ.adds deep.Every one of them is partial, so the well-typedness of the statement is a
fitsobligation over the whole nest. It is now stated innatarithmeticinstead, which has no obligation at all. Why the original stopped working is
worth recording precisely; see the next section.
Kuiper.Sparse.Load.fst'sload_cellstates its postcondition asCell (x <: array et) (SZ.v i) |-> Seq.index s j. Thehas_pts_toinstanceis
has_pts_to (cell (array a) nat) a, so the index type has to be literallynat;SZ.v iused to elaborate to exactly that, but its result type is nowreached through
SizeT.v's refinement and comes out asnat{fits …}, whichno instance head matches. Ascribing the index
(SZ.v i <: nat)— kuiper's ownidiom, e.g.
Kuiper.Kernel.HReduce.Block.Max.fst:374— fixes it. This is thesame hazard as
LogSoftmaxabove, reached from the other direction: there arefinement was added to an inferred type, here one that was always there
stopped being erased.
Kuiper.Sparse.SPMM.Compute.fstneeds the same fact asblock_lemma_offatfour separate places —
cntdivides bothkandnandk < n, sok + cnt <= n— once in a pureTotfunction, once in a Pulsefn, once asa
fitsbound inside awhileinvariant, and once inside apropdefinition, where there is no statement position to put a hint in. A local
__divides_nextlemma covers the first three. Giving it anSMTPatto coverthe fourth is a trap: it discharges that goal but breaks an unrelated
decreasescheck forty lines earlier, which is the usual cost of a pattern ona predicate as common as
divides. Inside thepropthe fact is scopedinstead,
k2 < n ==> (let _ = __divides_next cnt k2 n in …)— which worksprecisely because of this PR: sequencing a
Lemmanow puts its conclusion inscope as a binder rather than as an effect.
Kuiper.Sparse.SPMM.LoadSparse.fstcallsforevery_rw_sizetwice with thesame equation,
v (n /^ nthr /^ chunk et) == v n / (v nthr * v chunk et),once before a
foreachand once after. The first still goes through; thesecond, in the much larger context the
foreachleaves behind, times out.FStar.Math.Lemmas.division_multiplication_lemmasupplied explicitly fixesit. Both halves of the fourth finding are visible here at once: the
SizeT.divequations are no longer ground, and what that costs depends on how much else
is in the context.
Kuiper.Sparse.SPMM.Defs.fst'sblock_lemma_offprovedk * block + off < wholeby(), fromblock /? whole,k * block < wholeand
off < block. The lemma immediately above it,block_lemma, alreadystates the missing step (
k * block + block <= whole) and still proves by(); only the composite one needs it spelled out now. Calling it is the wholefix. Nothing here is about
squash: it is a divisibility fact whose proofneeds one nonlinear step, and the encoding change moved it across the
threshold.
Auditing the downstream changes, and what happened to the
z3rlimitbumpsEvery one of the 22 edits was re-tested individually, by restoring the original
text of just that change — in the multi-part files, of just that hunk — and
rechecking the module against the current compiler. All of them are still
required: none is left over from an intermediate state of the branch. The
harness is a scratch
--includedirectory that shadowssrc/, so a singlemodule can be rechecked in about a minute against the already-built
obj.That audit also revised the three
z3rlimitbumps, which are the changes mostlikely to hide a future regression. Two of the three are gone, and the
downstream diff now contains no rlimit increase at all beyond one relocated
#push-options "--z3rlimit 20"that simply follows a moved lemma and matchesits two neighbours.
Kuiper.Math.OnlineSoftmax.fst'sabcd_adcb— the fifth finding below — wascarrying
--z3rlimit 30. The real fix is to state the two non-zero sideconditions as a
requiresinstead of as refinements onbandd. Reducedto six lines over
FStar.Realand nothing else, the refinement form takes11.1s and the
requiresform 0.30s, both at the default rlimit; inthe module itself the change replaces
--z3rlimit 30and 22s with no optionat all and 17s. The refinement form makes each of the four divisions in the
conclusion re-derive its own guard, and those guards now survive into the
goal's context, where nlsat case-splits every one of them; a single
requiresis one hypothesis instead.
Kuiper.Kernel.GEMM.SHMem.fst'sbkfhad been raised from 40 to 100. Whatactually fails is one
assert (pure (2 * (!bk + 1) == 2 * !bk + 1 + 1))inthe loop body — linear, trivial, and timing out only because of how much else
is in scope by that point. Proving it as a two-line top-level lemma in an
empty context and calling it instead restores the original rlimit of 40.
(50, 60 and 80 all still fail without the lemma, so this was a real 2.5x bump,
not a rounding-up.)
Kuiper.Kernel.GEMM.FlipFlopBarrier2.fst'sodd_barrier_p_to_qis the onecase where a raise is genuinely the right answer, and it is lowered from 100
to 80. Here the failing goal is
it / 2 >= 0withit : natlt (2 * (shared/bk))in scope. It is not a hint that is missing: asking for the fact as the very
first
assert pureof the body fails in 54s just as it does at the point ofuse, so the cost is the ambient VC — the function's slprops mention the
concrete k-tile
it/2where the neighbouringeven_barrier_p_to_q, whichneeds no raise, uses an existential. A sequenced
Lemmadoes not help either:it arrives at the query as a
Prims.unitbinder with its conclusion dropped.Measured, 20 and 40 fail while 60, 80 and 100 succeed, so 80 leaves a 2x
margin over the last failing value without carrying the original number.
The two lemma-in-a-clean-context fixes above are worth generalising: when a
trivial arithmetic fact times out inside a large Pulse function, hoisting it to
a top-level lemma is almost always better than raising the budget, because it
is the context and not the goal that is expensive. It only fails when the
ambient VC is itself over budget, which is what distinguishes the
FlipFlopBarrier2case from the other two.A fourth finding: a postcondition now takes two instantiations, behind a guard
This is the same
squash pweakness as above, seen from the other end, andkuiper gives it a sharper measurement than EverParse did.
Upstream, an application of a partial function inside a specification publishes
its postcondition as a ground fact:
Pureis a computation type, so VCgeneration for the enclosing
bindrestatesv (mul a b) == v a * v bfor everysubterm. Here
mulis aTotfunction with a refined result type and animplicit
squashargument, so the equation is not stated anywhere; the solverhas to derive it, from
typing_FStar.SizeT.mul(which yieldsHasType (mul x y u) (Tm_refine_c477 x y), guarded byHasType u (Prims.squash (fits (v x * v y)))) and thenrefinement_interpretation_Tm_refine_c477. Two instantiations, the first behinda
squash-typed guard.Taking the failing goal — the
fitsobligation above — out of--log_queriesand editing the axioms directly separates the two costs:
typing_+refinement_interpretation,squashguardunknownin 2.8s(mul x y u),squashguardunknownin 2.8styping_+refinement_interpretation, guard rewritten toValid (fits …)unknownin 2.6s(mul x y u), guardValid (fits …)unsatin 0.6sunsatin 0.6sSo both costs are load-bearing: the goal is provable, and neither halving the
instantiation depth nor fixing the guard is enough on its own. For completeness,
raising the rlimit does not substitute for either — 20M gives
unknownafter93s, 100M was still running after ten minutes — nor do
smt.arith.nl false,arith.solver 2,relevancy 0,case_split 0|1, four random seeds, or--fuel 2 --ifuel 2 --z3rlimit 80in the source. (:produce-unsat-cores truedoes turn it
unsat, which is a fact about z3's search, not about the goal.)The clean fix follows directly: emit, for a
val f : bs -> Tot (r:t{phi}), anaxiom
forall bs. {:pattern (f bs)} guards ==> phi[f bs/r], with a squashbinder's guard given as
Valid prather thanHasType u (squash p). That isone new axiom per function with a refined result — measurably not free — and the
second half of it is the very rewrite that the table in the previous section
records as having broken
LowParse.Spec.Base.serializer_injective. It is thesame trade-off, and it wants the same treatment: a change of its own, with its
own measurement across ulib, EverParse and kuiper, not a patch at the end of a
refactor. Downstream, the workaround is the one applied above — say it in
unrefined arithmetic, or supply the equation with an
SMTPatlemma.A fifth finding: a discharged side condition is now a live hypothesis
Kuiper.Math.OnlineSoftmaxwas the last regression kuiper produced, and theonly one that is purely about proof performance. Baseline checks the module in
40s; this branch spent half an hour on it and had not finished.
It reduces to six lines with no kuiper in them at all:
The query is identical —
--log_queriesgives byte-for-byte the same@queryassertion on both. What differs is the assumption stack it is askedunder.
( /. ) : real -> d:real{d =!= 0.0R} -> Tot real, so each of the fourdivisions in the statement raises a
d =!= 0.0Robligation; those are goals 1-4and they are trivial on both sides. On master they are discharged inside their
own
push/popframes and are gone by the time goal 5 is asked, which seesfour hypotheses, all of them
HasTypefacts. Here the same obligations surviveinto goal 5's frame, which sees eight:
Two of those are exact duplicates of the other two, and two of them are
tautologies. None of them carries information the refinement on
sk_2andsk_4did not already carry. But they are ground disequalities over reals,and nlsat case-splits a disequality into
< \/ >: four redundant atoms are upto sixteen extra branches through a nonlinear decision procedure. Nothing about
the goal got harder; the context got noisier in exactly the way this one theory
cannot absorb.
The reason they survive is the shape of the VC. A
requiresis a binder now, sothe obligation attached to an implicit
squashargument is closed over thebinders in scope and conjoined into the same VC as the body's obligation, rather
than being solved and discharged in a nested frame. That the two copies are
identical says the closure happens twice, once per elaboration path.
This is worth fixing, but the fix is in VC construction — deduplicating and
scoping the guards that
Env.push_guardaccumulates for implicit arguments —not in anything this PR touches, and it needs its own measurement: every
Lemmain ulib is affected by how those guards are framed, and most theoriesare far less sensitive to redundant hypotheses than nonlinear reals are.
Downstream the workaround is not an rlimit bump but a restatement: writing the
two side conditions as a
requiresrather than as refinements onbanddproduces one hypothesis instead of four guards, and takes 0.30s against the
refinement form's 11.1s at the same default budget. That is a useful rule of
thumb for anyone hitting this — if a lemma's arguments are refined and its
conclusion uses each of them under a partial operation, prefer a
requires—and it is also a hint about the eventual fix: the
requirespath already doesthe scoping that the implicit-argument path does not.
The fifth finding, resolved: deduplicating VC conjuncts
Merging
origin/masterturned the fifth finding from a performance note intotwo hard failures. Upstream landed a new SMT encoding for
prop, which adds aBoxPropconstructor toTermalong withis-BoxPropis a datatype tester, so every prop-typed term in the context is apotential constructor case-split. Master's VCs absorb that; ours do not, because
of exactly the duplication described above.
FStar.Math.Lemmas.lemma_div_plusand
FStar.Math.Fermatbegan failing at the default budget. The failing goalwas instructive: the SMT text of the query was byte-identical before and after
the merge, and bare
z3still solved it in 0.9s, but the goal went from 0.087rlimit to exhausting 5.000 — a purely contextual, ~57x blow-up. Its VC carried
32 syntactically identical copies of the guard
n > 0 ==> n <> 0emitted bythe divisions in the statement, nested under seven layers of
forall (_: Prims.unit).So the fix is the one this section already predicted, and it is now implemented:
dedup_vcinFStarC.TypeChecker.Rel. It walks the conjunctive structure of aVC and replaces a conjunct by
Truewhen a syntactically identical conjunct hasalready been seen in a goal position that dominates it. That is sound because
the retained occurrence is proved outright, so the dropped one follows from it.
The set of known conjuncts only ever travels downwards — into the right of a
conjunction, the conclusion of an implication, and the body of a quantifier — so
a conjunct found under a binder is never assumed known outside it. Pushing the
outer set under a binder is fine: those conjuncts are well scoped in the
enclosing context and therefore mention none of the bound variables, and
SS.open_term_1picks globally fresh names, so capture is impossible.Membership uses
FStarC.Syntax.Hash's structuralequal_term, not a hashcomparison, so a collision costs a missed opportunity and never an unsound drop.
It runs at the single point in
do_discharge_vcwhere a goal is handed toenv.solver.solve— after tactic preprocessing, after normalisation, and aftercheck_trivial. Nothing upstream of the solver can observe it, so it cannotperturb unification, inference or tactics.
On
FStar.Math.Lemmas, against the pre-merge build of this branch:lemma_div_plusThe worst single goal in the module sits at rlimit 4.0 in both the pre-merge
baseline and the deduplicated merge — it merely moves between lemmas, which is
ordinary Z3 luck rather than a change in difficulty.
This is a narrower fix than the section above asks for: it removes the
duplicates at the end rather than avoiding their construction, so
Env.push_guardstill does redundant work and the compile-time cost of buildingthose conjuncts remains. Scoping the guards at construction is still worth
doing. But it removes the duplicates from every query, which is what the solver
was actually paying for, and it does so without changing a single downstream
proof.
The rest of the merge fallout
Three tests moved, and it is worth separating what the dedup did from what the
merge did.
FSTAR_NO_DEDUP_VC=1turnsdedup_vcoff, which makes theattribution mechanical.
tests/bug-reports/closed/Bug3213b.fstis the only one caused by the dedup,and it is the intended behaviour rather than a regression. The test asserts
expect_failure [19; 19; 19]; it now raises two. Its twoforall_elimcallsdiffer only in their explicit argument, and
forall_elim's preconditionforall (x:a). p xdoes not mention that argument — so the two obligations arethe same formula, and are now reported once. The annotation is now
[19; 19].The cost is real, if small: two failing obligations at two source lines can
collapse to one message. Labelled goals are unaffected, since
equal_termcompares the range inside
Meta_labeled, so only unlabelled duplicates merge.The other two are fallout from #4519, which stopped emitting the term
equation
f x == bodyfor a prop-valued definition, leaving only the formulaequation
Valid (f x) <==> body. Both fail with the dedup off as well.examples/data_structures/BinomialQueue.fst—find_max_emp_repr_l'svacuous branch. The encoded query is byte-identical to the pre-merge one and the
goal is still provable, but z3 now returns
unknown because (incomplete quantifiers)in 0.01s having used 0.049 of its budget: it saturates rather thanrunning out of resources, and
--z3rlimit 200,--fuel 4and--ifuel 2allleave it exactly where it was. The unsat core from a run without a resource
bound shows why — the new proof needs
prop_inversion,prop_validity,true_interpandfunction_token_typing_Prims.l_True, none of which the oldone used. Naming the intermediate fact (
assert (S.mem k (keys l).ms_elems))restores it. That is the right shape of fix for a saturation failure; an rlimit
bump would not have worked at any size.
examples/dsls/dependent_bool_refinement/DependentBoolRefinement.fst—soundness'sT_Appcase. This one is resource exhaustion, and--z3rlimit_factor 2on the enclosing#push-optionsblock is enough; 4 and 8were also tried and are not needed. It is the one rlimit change in this merge.
Re-testing EverParse and kuiper against the merged compiler
Both downstream trees were wiped of every
.checkedfile and rebuilt fromscratch against the merged compiler. EverParse revised is green again at 417
.checkedafter two changes; kuiper revised is green again at 396.checkedafter five. Every failure below was attributed with
FSTAR_NO_DEDUP_VC=1first:none of them is caused by
dedup_vc.EverParse,
LowParse.Pulse.Combinators: an implicit that used to be solved tothe other side's spelling.
split_nondep_thenandghost_split_nondep_thenpass
nondep_then_eq_dtuple2where a(x: bytes) -> Lemma (parse p1 x == parse p2 x)is expected. The lemma proves exactly that, and the error printed thegoal and the hypothesis identically — even with
--print_implicits. Theencoded query showed the real difference:
The call site writes the type implicit as
#(_: t1 & t2), which elaborates todtuple2 t1 (fun _ -> t2), whilenondep_then_eq_dtuple2states itspostcondition with
dtuple2 t1 (const_fun t2). The two are equal only bydelta-unfolding
const_funand eta — which the unifier does and the SMTencoding of a closure cannot. Pre-merge, the implicit was solved to the
const_funspelling and the obligation never reached the solver at all: thepre-merge query for this definition has two goals, both mentioning
const_funand neither mentioning the closure token. Post-merge the user's spelling
survives, so the obligation is emitted, and z3 saturates on it
(
incomplete quantifiers, 0.01s, 0.07 of a budget of 5 — no rlimit helps).Only two upstream commits in the merge touch the typechecker
(
790da6baa1, which makeseq_tmcompare binder qualifiers on arrows, andbd499fb784), and I did not pin it to either; what is verified is that thepre-merge build of this branch checks the module and the merged one does not.
The fix is to write the same spelling on both sides:
#(dtuple2 t1 (const_fun t2)).EverParse,
CBOR.Pulse.Raw.Format.Serialize.map_peek: the subterm orderingfst (List.Tot.hd (Map?.v r)) << r, needed fordepth_cb_pos's last binder,now exhausts the default rlimit (
canceled, exactly 5.000). The identicalproof still succeeds unaided in
CBOR.Pulse.Raw.Read.map_peek, so the cost isthe ambient context of this module rather than the goal.
--z3rlimit 10,scoped to that one
ghost fn, is well below the 32 and 64 already usedthroughout the file.
The eight kuiper failures are all arithmetic — nonlinear multiplication,
division and modulus — and six of the eight are better fixed by naming the
missing step than by raising a limit:
Kuiper.Divides.lemma_divides_trans—x * f1 == yandy * f2 == znolonger give
x * (f1 * f2) == zon their own;M.paren_mul_right x f1 f2supplies the reassociation. A second step in the same file
(
c == (c/a) * afroma * (c/a) == c) needsM.swap_mul.Kuiper.Kahan.kahan_sum— the invariant'snew_c %~ 0.0Rwas costing61 seconds and exhausting rlimit 20. The real-arithmetic core is
(s1 -. s0) -. (y -. 0.0R) == 0.0Rgivens1 == s0 +. y. Hoisted to atop-level
kahan_delta_zeroproved in an empty context, the module dropsfrom a 61s failure to a 4s success. The ambient context inside the loop is
saturated with the
_approx_patSMT patterns ofKuiper.Approximates.Base, every one of which fires on thesubs in thebody; that is what made an otherwise trivial goal expensive.
Kuiper.Kernel.GEMM.Copy.Vec2.cp_array2_vec— thewhilemeasure. Thenew index is
(git + 1) * nthr * chunk_etand stays undermlenbecausechunk_et * nthrdividesmlen; chasing that through division, commutationand reassociation inside the loop body took 303 seconds and exhausted an
already generous rlimit of 120. A top-level
cp_measure_helperdoing thesame four
FStar.Math.Lemmassteps in an empty context is instant.Kuiper.Sparse.Array.PtsTo.thread_gather_chunksandKuiper.Kernel.SDPA.Naive.sdpa_probs_spec_slice— the two that did getan rlimit. Both are resource-bound (
canceledat exactly the limit, notincomplete quantifiers), both areforall-quantified nonlinear indexgoals with no per-element proof hook to hang a lemma on, and
--z3rlimit_factor 2scoped to the single definition is enough for each. Inthe
PtsTocase I first tried the structural route — a quantifiedchunk_cell_offset_forall— and it discharged the stated goal but simplymoved the cost onto the accompanying
Seqbounds obligation, so the scopedfactor is the honest fix.
Kuiper.Sparse.SPMM.LoadSparse.load_array_vec—n / (nthr * chunk et) == n / nthr / chunk et, a singledivision_multiplication_lemma, wasexhausting rlimit 30 inside the
thread_live_chunksunfolding. A top-levelload_array_vec_sizeproved in an empty context is instant.Kuiper.Sparse.SPMM.Compute.seq_load_vmprod_cell_lemma— the recursivecase has to recombine
(k1 / chunk et, k1 % chunk et)back intok1to turnthe
_prop_form of the invariant into the_propform. The author hadalready written the bridging call to
seq_load_vmprod_row_cell_prop_equivand left it commented out because SMT had been finding it; uncommenting it is
the whole fix.
Kuiper.Sparse.SPMM.Barrier.barrier_p_to_q_transform— the third andlast rlimit, and the least satisfying.
barrier_in's implicit divisibilitysquashes are spelled
(chunk et * p.blockWidth) /? p.blockItemsKwhile theparametersrecord refinesblockWidthwith the commuted(k * chunk et) /? blockItemsK; discharging one from the other misses the default budget by alittle (
canceledat 5.000; rlimit 8 suffices). Respelling would touch 69binders across the SPMM sources, so this is a scoped
--z3rlimit_factor 2onthe single declaration.
Two measurement notes came out of this round. First,
--admit_exceptis not asound way to size an rlimit:
seq_load_vmprod_cell_lemmapasses under--admit_exceptand fails in the full-module run, because F* reuses one z3process across a module and the earlier queries change how the later ones
perform. Sizes have to be measured in a full-module run. Second, the
distinction between
canceledandincomplete quantifiersin--query_statsdecided every one of these:
canceledat exactly the limit means a bump willwork, and
incomplete quantifiersin a fraction of a second means no bump everwill.
Testing against pulse-verified-gc, and a three-way A/B/C
EverParse is parsing and low-level imperative code; kuiper is type-level
computation and typeclasses. The third round was run against
pulse-verified-gc, a verified
OCaml-style garbage collector: a very large body of first-order arithmetic
spec code — heap addresses, word alignment, header bit-fields — with Pulse
implementations on top. It is the most SMT-bound of the three, and it exercises
a part of the system the first two rounds barely touched.
It also forced a change in method. By this point the branch had merged
origin/masterseveral times, while pulse-verified-gc pins F* nightlyae858eacbd07. A two-way A/B can no longer distinguish "this PR broke it" from"upstream broke it in the meantime". So this round is an A/B/C: the pinned
baseline, this branch, and a third tree built with plain
origin/masterat52f17ab8fd. Anything that fails in tree C is upstream drift and is not thisPR's to fix.
The result is worth stating plainly. Against the 241 modules of the baseline:
ae858eacbd07)origin/master52f17ab8fdGC.Spec.Allocator.fstimerely to get that farPlain master needs the same mechanical
op_Subtraction→op_Minusrename thisbranch does (upstream's "uniform operator name mangling"), then still fails in
eight places, including every one of the two hardest failures this branch hit —
GC.Spec.SweepCoalesce.Helpers.combine_extract_nthandGC.Gen.CheneyPreservation.Forwarding— plus four sites inGC.Gen.MinorCollectForwardingand two inGC.Spec.Allocator.Lemmasthat thisbranch verifies without complaint. The
SweepCoalesce.Helpersslowdown inparticular (a ~4x regression on a bit-blasting-heavy
logand/shift_rightproof) is attributable to upstream
9c919fce78, "Encode prop like bool, boxingto SMT Bool", which introduces the
BoxPropconstructor and shows up as aliteral diff in the generated
.smt2. None of it is this PR.A finding that changes how a regression should be read: gensym instability
Two F*-library modules —
Pulse.Lib.PriorityQueueandPulse.Lib.Array.Core—started failing after a
Rel.fstchange that could not possibly affect them.Dumping
--log_queriesfrom both compilers and normalising showed the two.smt2files differ only in the numbering of gensym'd universe variables(
uu___79→uu___83,uu___91→uu___95). Replayed offline through z3, theold file gives zero
unknownand the new one gives exactly one, at the samegoal; renaming part of the symbol set does not flip it back, so the effect
depends on the whole set.
That is not a semantic regression. It is a proof that was passing with no margin,
knocked over by a shifted fresh-name counter. Any perturbation of the compiler
can do this, so it will happen again, and the diagnostic is worth writing down:
--log_queries(the file lands in the cwd asqueries-<Module>.smt2).diff <(sed 's/uu___[0-9]*/UU/g;s/@x[0-9]*/@X/g' A) <(sed ... B). If the onlyremaining difference is the
; STATUS:comment, the inputs are equivalent andthe compiler change is not the cause.
z3 -smt2and counting^unknown. F*embeds the per-goal
(set-option :rlimit N)in the logged file, so an offlinereplay is faithful.
The right response is to fix the proof, not to revert the compiler change,
and both were fixed at the source:
almost_to_full_heap's induction on sequencelength was deleted outright (
almost_up_implies_heap_downalready givesheap_down_at s iat every index, so a singleClassical.forall_introdoes it),and
pcm_sharegot them1-side permission bound that was already present,asymmetrically, for
m2.Two compiler fixes
Uvars in implicit positions are not logical content. Under this PR a
Lemma postis checked by subtyping betweensquashtypes.Relhas a rulethat rewrites
squash p <: squash qinto(_:unit{p}) <: (_:unit{q}), which iswhat makes such a check cheap; it was guarded by "neither side contains a uvar".
An incidental implicit uvar — the
#a:eqtypeofop_Equals— was enough todisable it, sending the problem to
Tm_appcongruence instead, whose localequalhelper normalises with[UnfoldUntil delta_constant; ...]; unfoldingto_vec/from_vecat width 64 then consumed 32 GB and did not terminate. Theguard is now
has_uvar_needing_congruence: a uvar that is an implicit argumentof an interpreted head can be ignored, while every other uvar is logical
content and must still block the rewrite. (That distinction matters: an earlier
"no flex at all" formulation broke
introduce _ ==> _, becauseFStar.Classical.Sugar.implies_intro'spandqare explicit.)The restriction to interpreted heads was not the first attempt, and the
intermediate version — ignore a uvar in any implicit position — is worth
recording, because it broke EverParse in a way that no
make cirun wouldhave caught.
ASN1.Syntaxhasproj2_of_3has an implicit#c : a -> b -> Type. In the type ofpf_wfthelist is empty, so
#coccurs nowhere else and nothing local determines it. Theone thing that does determine it is checking the body:
pf_wfis passed toASN1_ANY_DEFINED_BY, whose expected type for that argument mentions the sameList.map proj2_of_3 []with#calready solved, and congruence on thatsquash <: squashproblem commits it. Rewriting the problem into refinementsubtyping instead hands it to the SMT solver as an implication, which solves
nothing;
#cthen survived typechecking and was generalized, givingasn1_any_oida spurious leading#_: Typebinder. Every call site inASN1.X509then failed withError 66: Failed to resolve implicit argument.Two things about this are worth remembering. First, the symptom appeared three
commits away from its cause, in a file whose
.checkedhad been reused acrosscompilers — a stale
ASN1.Syntax.fst.checkedalso masked the fix on the firstattempt, which sent the diagnosis down a blind alley. When a regression is about
inference rather than proof, the caches of the dependencies have to be wiped
too. Second, the useful oracle was not the error but the inferred type: running
under the branch and under
origin/mastershowed#_: Type ->present in oneand absent in the other, and reduced a 3000-line EverParse module to a
fifteen-line test case.
Regression tests:
tests/bug-reports/closed/SquashSubtypingDivergence.fst, whichnow covers both directions — the
GC.Lib.Headershape that must fire, and theasn1_any_oidshape that must not.The unbounded normalisation inside that
equalhelper is the more fundamentalproblem and is left as a follow-up:
Env.stephas no fuel constructor, sobounding it is not a one-line change.
Eta-expansion across a missing
requiresbinder.ToSyntaxomits the#(_:squash pre)binder whenpreis syntacticallyTrue, soLemma (ensures q)has one binder fewer thanLemma (requires p) (ensures q).Classical.move_requires' argument binder is$_:, i.e.Equality, whichforces
use_eqand rules out ordinary subtyping, so the gap has to be bridgedin
try_eta_expand_to_expected_typ. It now rebinds a trailing expected binderwhose sort is
squash ?pwith?puvar-headed atsquash True, letting?p := Truefall out of the ordinary check. A concrete expected preconditionis left alone, so genuinely strengthening a precondition is still rejected.
Regression test:
tests/bug-reports/closed/MoveRequiresNoPrecondition.fst.The source changes in pulse-verified-gc
Every one of them is either an improvement or a documented stabilisation; none
is a large rlimit bump. The pattern that dominates is the one kuiper already
suggested, and pulse-verified-gc makes overwhelming:
GC.Gen.CheneyPreservation.Forwardingneeded two:(a + k*8) % 8 == 0froma % 8 == 0, andb + ((a-b)/8)*8 == afroma % 8 == b % 8 == 0. Both areone-line consequences of
FStar.Math.Lemmas. Inline they werecanceledatrlimit 120; hoisted, the whole module verifies with a maximum used rlimit of
7.1.
GC.Gen.Promote.promote_preserves_field_atandGC.Gen.MinorHeap.minor_reset_tag_zero: same treatment, both back to themodule's base rlimit. The
MinorHeapone is also a small lesson inassert_norm:the fact was
U64.v (U64.logand 0UL 0xFFUL) == 0, and normalising it drives theevaluator through
UInt.to_vec/from_vecat width 64. Deriving it fromUInt.logand_leinstead is both cheaper and context-independent. It has to beparameterised over the header, though — as a closed fact Z3 will not do the
congruence step from
hdr == 0ULunder--ifuel 0.GC.Gen.Cheney.SimOne: twoUInt64facts hoisted; the module went fromfailing after ~130 s to verifying in 8 s.
GC.Gen.TwoPassEquiv.two_pass_pointwise: an ascription bug this PR makesvisible. The proof writes
let obj : obj_addr = IndDesc.indefinite_description_ghost obj_addr (fun obj -> ...).Under this PR
indefinite_description_ghostreturns a refined resultx:a{p x}; ascribing the unrefinedobj_addrthrows the refinement away andleaves Z3 to re-derive
p objfrom the definitional equation. Deleting the twoascriptions fixes it. This is the general shape to look for when a
Pure/Ghostresult stops carrying its postcondition: an ascription that used to be free now
weakens the type.
GC.Impl.Allocator.init_heap_normal_lemmais the same storyread in the other direction — there an ascription had to be added, to strip
write_word's new result refinement where the unrefinedheapwas wanted.GC.Spec.Sweep.sweep_object_preserves_other_header: the shared conclusion isnow asserted at the end of each of the four branches rather than once after
the
if. A minimal test confirmed that lemma postconditions are notgenerally lost across a join, on this branch or the baseline, so this is proof
robustness rather than a compiler workaround: the branches reach the conclusion
through different intermediates and the join keeps only what is stated.
--query_statsmeasurement recorded ina comment next to it:
GC.Gen.CheneyBFS.forward_one_queue_prefix10 → 20 andGC.Spec.Allocator.Lemmas.Part1.alloc_split_facts_part1(canceledat exactly5.000; 6.925 used at 10). Nothing larger was needed.
drowning them.
GC.Gen.PromoteUpdate.Fieldis the sharpest: theensuresofupdate_all_objects_aux_field_effectappliedU64.uint_to_ttoU64.v obj + j * 8, soFStar.UInt.size _ 64and thehp_addrrefinementwere being discharged in that lemma's full context — 21 s and 34.6 rlimit units
against a budget of 12. Adding the bound as an extra
requiresconjunct didnot help; the context, not the goal, was the problem. The fix is a total
function with a junk value: a private
field_addr : U64.t -> nat -> GTot hp_addrreturningzero_addrwhen theaddress is out of range, so the obligation is discharged once, at the
definition, in an empty context, plus a
field_addr_vlemma naming theequation under the real precondition. The lemma now uses 3.3 rlimit units.
(A first attempt returned
U64.t; the caller then demandedhp_addrand theproblem simply moved. The return type has to be the refined one.)
GC.Impl.MarkBounded.wosize_offset_fitsandGC.Gen.MinorHeap.infix_parent_beloware the same idiom applied toU64.mul wz mwordinside a Pulsefnand toaddr >= infix_parent minor addrin all four infix branches of
CheneyPreservation.Frame.GC.Spec.SweepCoalesce.Helpers.combine_extract_nthis a bit-level proof — an8-way
select_byte, ashift_rightby the nonlinear8 * k, sixteenUInt.nthlemmas — and it wascanceledat rlimit 200, at 400, and at 800.For each byte
mabove the extracted bytek, them-th shifted bytecontributes nothing at bit
j; that follows fromj >= 56 - 8*kandm > k,but only after a case analysis with both
8*kand8*msymbolic. Writingthe seven instances out, with
ma literal so8*mis a constant, takes thelemma from timing out at 800 to using 42 of its declared 200. Worth
stressing: this lemma also fails on plain
origin/master, so it is not a costof this branch — it is where the ~4× slowdown from upstream
9c919fce78"Encode prop like bool, boxing to SMT Bool" surfaces. The fix is upstreamable
as-is.
Forwarding.fwd_classified_weakens(fwd_valid_or_infixisfwd_classifiedwith the existential witness dropped, but the weakening is under a
quantifier) and
Allocator.Lemmas.Part2.hd_address_v. The first is the bestillustration in the whole campaign of why isolated probes are not evidence:
the goal took 0.1 s and 0.34 rlimit units when the module was checked on
its own, and timed out at rlimit 20 in a full build. Whether the solver finds
the instantiation depends on the rest of the module, so "it passes in
isolation" means nothing. Every fix here was confirmed by a clean rebuild.
GC.Gen.MinorHeap.minor_zero_header_fields: decoding a zero minor header intowosize 0 / tag 0 needs the bit-vector encoding of
shift_rightandlogand.All three SPOT nurseries were doing that inside a proof whose context already
fixes several other header words, and all three timed out. Proving it once
for an arbitrary
minor_statefixes all three call sites.Method notes
--query_stats' reason-unknown is the classifier, and it was right every time:canceledat exactly the limit means a bump may work;incomplete quantifiersin a fraction of a second means a fact is missing and no bump ever will.
--admit_exceptremains unsuitable for sizing an rlimit — F* reuses one z3process per module, so earlier queries change how later ones perform — but it is
fine for extracting a single query with
--log_queries. And--admit_excepttakes exactly one name: a comma-separated list silently admits the whole module
and reports success.
Two benchmark outliers, and what they were
The benchmarking bot on this PR reports the change as roughly neutral overall
(geometric mean 1.003x memory, 0.989x time, 308s less wall clock in total), with
some large wins —
ExtUIntMask-55.8%,BVExtend-94.4%,Lib.Sequence.Lemmas-49% — and two large outliers. Both turned out to be worthchasing: neither is really about this branch's design, and one of them is a
long-standing performance bug in
Rel.Bug3800.fst:forall x. phi ==> Truetests/bug-reports/closed/Bug3800.fstwent from 0.47s/94MB to 6.18s/330MB. Asize-parameterised family of the same shape shows why: the cost is exponential
in the nesting depth of the test's sixteen chained
let v = if ... then ... else v in,while on master it is linear. The SMT query is not the problem — it is in fact
smaller on this branch.
--profileputs 5.9 of the 6.2 seconds insideRel.sub_comp->Rel.simplify_vc->Normalize.normalize.The guard being normalized is
It comes from the refinement/refinement case of
solve_t'. The left-hand sideof the subtyping problem is the definition's computed type, which on this branch
carries the definitional equation as a refinement (on master the same fact
lives in a
Purewp and is already CPS-flattened, so it normalizes linearly).The right-hand side is the annotated
Tot u32, which is unrefined —force_refinementturns it intox:u32{True}purely so that the two sides havethe same shape. The case then builds
forall x. phi1 ==> True.That guard is trivial, but nothing noticed:
mk_conj/mk_impdo not simplify,so
simplify_vcdutifully normalized the antecedent first, and normalizing achain of sixteen
lets over amatchduplicates the continuation into bothbranches.
The fix is two lines of
mk_imp_simp/mk_conj_simp(which already existed inSyntax.Utiland short-circuit onis_t_true) plus anis_t_truetest beforeguard_on_element, which also avoids a needlessuniverse_ofcall on thebinder's sort.
EQis deliberately left alone:phi1 <==> Trueisphi1, notTrue.This is not a regression this branch introduced so much as one it exposed —
master reaches the same code, just with an antecedent that happens to be cheap to
normalize — and the fix is independent of everything else here. After it,
Bug3800.fstruns in 0.31s/84MB, i.e. faster than master's 0.47s/94MB.Quicksort.Base.fst: a proof that was passing by luckpulse/share/pulse/examples/Quicksort.Base.fstwent from 22s to 87s. Profilingputs all of the delta in Z3 (9.8s -> 45.8s of aggregate query time), and
--query_statsnarrows it to two lemmas,transfer_larger_sliceandtransfer_smaller_slice, under a#push-options "--retry 10".Both compilers fail the same goal — the third
assert, which re-indexes alower bound on
sinto a lower bound onSeq.slice s (l - shift) (r - shift).Master happens to succeed on its second retry; this branch exhausts all ten
(~2.8s each) and then succeeds only once F* escalates
ifuelto 2. Extractingthe goal with
--log_queriesand running it standalone confirms it: with a freshsolver the goal is
unknownatifuel 1on both compilers, under everyhypothesis configuration I tried. The three-
assertproof was never actuallyworking; it was winning a race against
--retry.The missing step is that the goal mentions
Seq.index (Seq.slice s (l - shift) (r - shift)) k, which theSMTPatonSeq.lemma_index_slicerewrites toSeq.index s (k + (l - shift)), whereas thehypothesis has to be instantiated at
k + l, givingSeq.index s ((k + l) - shift). The two index terms are equal only by lineararithmetic, so whether E-matching bridges them depends on whether the arithmetic
solver has already merged their congruence classes.
Replacing the three
asserts with anintroduce forall ... with introduce _ ==> _that names the witness
j = k + lexplicitly — which putsSeq.index s (j - shift)in scope and makes the instantiation immediate — makes the goal go through
deterministically, and the
--retry 10and#restart-solverare no longerneeded. The file now takes 7.6s on this branch and 7.7s on master, against
14.4s for master before the change.
Re-measured locally after the
NDETmerge,Bug3800.fstis unchanged: 0.28sand 86.0 MB peak RSS on this branch against 0.48s and 94.2 MB on
master(best of three each, same machine).
NDET's added declarations inFStar.Pervasivesdid not erode the margin.Merging master's
NDETeffectWhile this branch was in review, master landed
NDET: a primitive effect that isnondeterministic but terminating, so the lattice becomes
PURE ~> NDET ~> DIVwith an explicitNDET ~> TAClift. That is the sameterritory this branch rewrites, so the merge is worth describing.
Most of the nine conflicts were mechanical. Master extended hardwired lists like
src = PURE || src = NDETat exactly the sites where this branch had introducedthe class predicates of "An effect abbreviation is a bare alias".
NDETis botha lift source and a lift target, so it cannot be folded into either neighbouring
class; it gets its own
PC.is_ndet_effect_lid— coveringNDET,NdetandNd— with
PC.primitive_ndet_lidandU.is_ndet_effectrouted through it, exactlyas the other three classes are, and each site becomes a disjunction of two class
predicates. Two of master's hunks call
Env.norm_eff_name, which this branchdeleted:
ToSyntaxresolves abbreviations now, solbeffandcomp_effect_namealready name a root effect and there is nothing to normalize.
FStar.Pervasives.fstineeded a fix that was not in a conflict hunk, and somerged silently into something the compiler rejects. Master writes
sub_effect PURE ~> NDETandNDET ~> DIV, but on this branchPUREandDIVare abbreviations and a lift must name the effect itself. These become
Tot ~> NDETandNDET ~> Div, and the directTot ~> Divedge is dropped:Env.update_effect_latticecloses the lattice transitively as each edge isadded, so composing the two gives it back.
The one real decision is at the top level. Master replaced
check_top_level'sboolresult with a three-way action so that a terminating effect is maskedsilently — no warning 272, no
nonemptyobligation — while this branch hadindependently changed the same function from
lcomptocomp. Both apply. Butthis branch also drops the refinement it infers for the result type when an
effect is masked, on the grounds that a postcondition under partial correctness
only holds if the computation returned.
Mask_effect_silentlyis precisely thecase where it does return, so the refinement is kept there and dropped only for
Mask_effect_and_warn.That is safe because it cannot leak a defining equation
_ == e, which is whatwould let the solver identify two separate calls of a nondeterministic
computation. Such an equation is only ever introduced by
maybe_assume_result_eq_pure_term, andshould_returngates it on thecomputation being pure or ghost — which
NDETis not. Checked rather thanargued: with
assume val f : unit -> Nd (x:int{x > 0})andlet g1 = f (),assert (g1 > 0)proves, whileassert (g1 == g2)andassert (g1 == f ())both fail as they must. Master's own
TestNd.fstpasses unchanged, includingits universe test —
NDETistotalwith no representation, so the rule of"A total effect's universe comes from its representation" answers
u_resandunit -> Nd (Type u#0)is stillType u#1.One inconsistency is left deliberately. Master makes
NDETthe primitivespelling with
Ndet/Ndas abbreviations, which is the opposite of theconvention here, where the short name is primitive (
Tot/GTot/Div) and theall-caps name is the abbreviation (
PURE/GHOST/DIV). Renaming a feature thathas just landed is churn that belongs in its own change, not in a merge.
User-visible changes
assume_safe's argument is nowsquash False -> Tac a, notunit -> Tac a.Write
assume_safe (fun _ -> ...), notassume_safe (fun () -> ...).applynow works on lemmas;pose_lemmais joined bypose_apply.()-against-squashcheck reports "Assertion failed" rather than"Subtyping check failed" — the obligation really is an assertion now.
#(squash P) -> Tot (x:t{Q x})back intoLemma (requires P) (ensures Q), so error messages and IDE hovers read asbefore. Squash binders print as hypotheses rather than as arguments.
effect M = Nis canonical; theeta-expanded
effect M (a:Type) = N ais still accepted. Anything else — extrabinders, a right-hand side that is not an eta-expansion of an effect name, or a
requires/ensureson the right-hand side — is now rejected with Error 316instead of being silently dropped. See "An effect abbreviation is a bare alias".
effect M = N <: ...(redefine_effect) form is gone from the grammar.[attributes ...]clause on an effect declaration is gone. It has beenimpossible to write since Dijkstra Monads for Free removed the
CPSflag.sub_effectmust name effects, not abbreviations: writesub_effect Tot ~> M,not
sub_effect PURE ~> M. The error message names the effect to write.Tot u#0 int) is rejected rather thanaccepted and discarded.
--ext optimize_let_vcis inert. The behaviour it selected is now the onlybehaviour; existing flags in downstream Makefiles need no change.
introduceandeliminateno longer bind a name for the hypothesis: writewith e, notwith h. e. The hypothesis is an implicitsquashbinder thatF* puts in the proof context of
eitself, so there is nothing to name.with h. eis rejected with a message saying so.Classical.move_requires*applied to a lemma that has norequiresclauseis now a no-op rather than an error. Such a lemma has no
squashbinder tomove, so it has one binder fewer than
move_requiresexpects; the gap isbridged by
try_eta_expand_to_expected_typ, which binds the missingprecondition binder at
squash Truewhen the expected precondition is stillan unresolved uvar (a concrete expected precondition is left alone, so a
genuine strengthening is still checked). This keeps a very common idiom
working. Note, though, that the wrapper is not wanted:
Lemma (ensures Q)is now literally
Tot (squash Q), which is whatClassical.forall_intro*expects, so the lemma can be passed directly. Several vacuous
move_requireswrappers in ulib were deleted.
failure is localized to the alias rather than to the call.
Pure/Ghostwith anensuresnow returns arefined type, so an implicit solved from such a result picks up the
refinement — most visibly for polymorphic equality, where
SZ.v n == capneeds
(SZ.v n <: nat) == cap.Prims.eq2already carries the[@@@unrefine]binder attribute that fixes this; promoting it from--ext __unrefineto the default is proposed as a follow-up. Likewise, alemma's statement is now part of its type and so participates in
unification, which can pin an implicit that used to be left to the expected
result type. See
regression_questions.mdfor both, worked out in detail.The same thing bites a container:
Ghost.hide (cbor_map_sub m s)infersGhost.hide's implicit atcbor_map_sub's refined result, giving aGhost.erased (m:cbor_map{...})where aGhost.erased cbor_mapwas meant, andthe mismatch surfaces later as an unprovable
l_True == <the ensures>. Givethe implicit explicitly:
Ghost.hide #cbor_map (...).arrow that has one has one binder more than an otherwise identical arrow that
does not. Subtyping now eta-expands to bridge that gap (see "Testing against
EverParse"), so a point-free definition whose implementation is more general
than its interface still typechecks. The eta-expansion is only attempted for
pure and ghost computations and only when the surplus binders are implicit, so
a few point-free idioms still need to be written out: passing
( + )where atwo-argument arrow is expected may need
(fun a b -> a + b).the constraint from an earlier one is processed. If argument
ngives?utherigid lower bound
t{phi}while argument 1 only wantst <: ?u,solve_flex_rigid_meetfires with a single bound in hand, sets?u := t{phi},and turns the earlier constraint into an SMT obligation that cannot be proved.
This PR makes it more reachable because a lemma's statement is now part of its
type. Instantiate the implicit explicitly at the call site.
match/ifscrutinee's refinement is not alwaysavailable in the branches, so
if strong_excluded_middle p then ...may nolonger see
b = true <==> p. Bind the scrutinee with an explicit refinedannotation.
(
x `logand` lognot ((lognot 0uL `shift_right` a) `shift_left` b)),only the outermost result's refinement is now attached; the intermediate ones
are lost. Let-bind each intermediate operand — the idiom EverParse already used
for its
UInt8instances of the same code — and the refinements come back.assertelaborates==at the refined type ofits operands, which can add a side condition that did not exist before
(
assert (a *. (b /. a) == b)fora b : permnow carries>. 0.0R).necessarily SMT-unfoldable to it when the module's interface has a
valforthe alias.
assert_normof the equation restores it.intro (Trade.trade A B) #emp fn _ {...}no longer resolves itsintroducableconstraint; call
Trade.intro_trade A B emp fn _ {...}directly.coerce_eq () xinfers its source type fromx, sowhen
xis the result of a function with anensuresit is the refinedtype, and the
()is then asked to prove that a refinement equals its ownunderlying type. Ascribe the argument at the type intended
(
coerce_eq () (parse_nlist n p <: parser _ (nlist n t))) --- the sameascription EverParse already wrote for the neighbouring serializer.
tip over it, because every lemma called in a Pulse block leaves its
postcondition — now a refinement, and so a hypothesis — in scope, and the
goal is buried among them. Two EverParse proofs needed the same remedy: state
the obligation as a small standalone lemma, whose context contains only what
the proof needs (
LowParse.PulseParse.Sum.dsum_tag_is_strong_prefix,CDDL.Pulse.Parse.ArrayGroup.half_plus_half_eq). Both then verify fasterthan before, and two
--z3rlimitbumps that had looked necessary turned outnot to be.
requires(ensures (inj (f x)), wheref xis a partial applicationawaiting the squash binder) is eta-expanded at each use, and two eta-expansions
of the same term are two distinct closures to the solver, so the lemma's
conclusion no longer matches the goal. Removing the
requiresin favour of arefinement on the argument's own type removes the eta-expansion and the
problem: this is what
ASN1.Spec.SequenceandASN1.Spec.Anydo.squash-typed argument proves isno longer published as a fact to the enclosing goal, so a
coerce_eq (_ by tac) xwhose two types are only equal after normalisation leaves the solver unable to
relate them. State the equation once, with the same tactic, before the use:
assert (a == b) by tac. See the section above for the full diagnosis; this isCDDL.Pulse.Parse.MapGroup.impl_zero_copy_map_zero_or_more_aux.a scrutinee that the body then
matches, the branch may no longer see whatthe precondition says about the branch's pattern variables. The
squashhypothesis is in scope, but as an opaque
HasTypefact it does not drive thesolver to unfold the predicate at the refined scrutinee. Restate the
consequence with a
Lemmataking the precondition and concluding what thebranch needs, called with
[@@inline_let] let _ = ... inat the head of thebranch — the idiom EverParse already uses elsewhere. This is
CDDL.Pulse.AST.Bundle.impl_bundle_wf_map_group_zero_or_more, which neededtyp_bounded ... keyand... valuein itsWfMZeroOrMorebranch.let x = assert pnow has typesquash p, sopbecomes a factfor the rest of the module. Ascribe
: unitwhere that is not wanted --in particular
let _ : unit = assert False, which otherwise poisonseverything after it.
asserts that used to be discharged inside asquash (...)argument nolonger contribute to the enclosing definition's own refinement; hoist the
lemma call out of the
squash.apply (magic)fills inmagic's anonymousunitargument itself; a followingexact (())now fails with "no more goals".failreturns a refinedunit, so an unannotated tactic whose body ends in amatch ... | [] -> fail ...infers a refined result type. Annotate: Tac unit.Costs
#(squash P)binder carries no computational content,so extraction drops it — both the binder and the matching argument — and the
ABI of a function with a
requiresclause is unchanged. The two sides have tostay in agreement, which is where the extraction bug found by the EverParse
run came from; see above.
examplesanddoc. In aggregate there is no regression: a from-scratch verification ofulib's 319 modules takes 1m35s wall at
-j16, or 13.2 CPU-minutes, againstthe 14m58 recorded for the previous design. The baseline's measurement
conditions are not documented, so read this as "no regression" rather than as
a precise speedup.
comp_viewkeeps its constructors;C_Lemma/C_Effreportpre = True, since a precondition is now a binder on the arrow and out of theview's reach. The postcondition is recovered from the result-type
refinement, and
inspect_comp/pack_compround-trip. Giving the view anhonest precondition means changing the view type, which needs its own stage0
bump and is deliberately left to a follow-up.
A documented limitation
tests/micro-benchmarks/Positivity.fst'sneg_matchnow also raises a spuriousError 19 on a definition that is rejected anyway. When a closed scrutinee makes
subst_pat_bvs_in_res_typfire and a branch builds an arrow, the branch musttransport its result type across
t == Some?.v g— and F*'s SMT encoding givesarrow types no congruence, since each arrow is encoded as its own constant. This
is unprovable on the pre-refactor compiler too. Every parameterized form of the
same type-level match verifies.
Validation
make ci -j48 -kfrom a fully wiped tree —stage{1,2}/{ulib,fstarc}.checked,pulse/build/lib.pulse.checked, and every_outputand_cachedirectory undertests,pulse,docandexamples— exits 0. That coversmake 1,make 2,make 3andmake test(which istests,examplesanddoc, atstage 3, with Pulse), plus
boot-diff,test-2-bare,stage2-unit-testsandfsharp-all. Note that test.checkedfiles live in_cacheas well as_output; wiping only the latter is what let several failures hide.cialready runs stage 3,examplesanddocvia_test, so it needed nochange.
Both benchmark outliers reported by the PR's benchmarking bot are fixed and the
fixes are in that run:
Bug3800.fstis 0.31s / 84MB againstmaster's 0.47s /94MB, and
Quicksort.Base.fstis 7.6s againstmaster's 7.7s (masterwas14.4s before the same change was applied to it). See "Two benchmark outliers".
Beyond
ci, EverParse'sfstar2branch verifies and extracts end to endagainst this compiler, from a clean tree, after the downstream edits catalogued
above. The A/B baseline build with EverParse's pinned toolchain reported zero
errors, so that catalogue is the complete list of differences this PR makes to a
large external codebase: 32 files, +246/-102 lines, made up of explicit
implicit arguments and type ascriptions,
asserts restating a fact the solverused to be handed, four small helper
Lemmas, oneGhost.hide, two implicittype annotations respelled to match the lemma they are passed to, and three
rlimit bumps. Each of the five load-bearing workarounds was re-tested against the final
compiler with the pristine source restored, and each is still required; none is
masking a bug that has since been fixed.
Kuiper is the second such run, and the same statement holds for it: 396 modules,
green from a clean tree, against a baseline of 396 green modules built with the
F* fork kuiper pins; 27 files, +354/-48 lines of downstream difference,
catalogued above, of which a good part is the comment on each change explaining
why it is there. Both downstream trees were re-verified from scratch against the
final compiler, after the last typechecker fix and after the earlier merge with
origin/master, not against the compiler each regression was found on. Thefinal numbers are EverParse 417
.checkedand kuiper 396.checked, both atexit 0, matching their baselines exactly.
pulse-verified-gc is the third, and the largest of the three: 241
.checkedplus the
spotsub-build, both at exit 0 from a clean tree, against abaseline of the same 241 built with the F* nightly it pins. The downstream
difference is 8 commits, all of them named lemmas and case analyses rather
than budget increases -- the two scoped rlimit bumps listed above are the only
ones, and one reduction came out of it (
combine_extract_nthwent fromtiming out at rlimit 800 to using 42 of its declared 200).
A caution that this run produced and the earlier two did not: an isolated
module check is not evidence.
Forwarding.cheney_promote_fwd_valid_or_infixtook 0.1 s and 0.34 rlimit units when its module was checked on its own, and
timed out at rlimit 20 in a full build of the same tree, with the same
dependency
.checkedfiles. Fixing one blocker also exposes the next: a-kbuild stops at ~176
.checkedwhen an early spec module fails, so error countsbetween runs are not comparable. Every fix reported here was confirmed by a
clean rebuild, not by a probe.
All three downstream trees were rebuilt one final time, from clean, against the
compiler that includes the two benchmark fixes: EverParse 417
.checked,exit 0; kuiper 396
.checked, exit 0; pulse-verified-gc exit 0 on boththe main build and
spot. Those are the same counts as their respectivebaselines.
Those three numbers were taken at
5209ef174b, immediately before master'sNDETeffect was merged in. After that merge,make ci -j48 -kis again exit 0from a fully wiped tree, and EverParse was re-verified end to end against the
merged compiler — verification and extraction to C, Rust and OCaml, exit 0 with
no F* errors, at the same 417
.checkedas its baseline. SinceFStar.Pervasiveschanged, every downstream.checkedfile was invalidated bydependency hash, so that run re-checked the tree rather than replaying a cache.
Kuiper and pulse-verified-gc were not re-run against the merged compiler; their
numbers stand as of
5209ef174b.