Fix checker when empty blocks result in unchanged-from-Top entry state. (#113)

The checker works by keeping a worklist of blocks to process, and adds a
block to the worklist when its entry state changes. Every entry state is
initially `Top` (in a lattice). The entry block is explicitly added to
the worklist to kick off the processing.

In ordinary cases, the entry block has some instructions that change
state from `Top` to something else (lower in the lattice), and this is
propagated to its successors; its successors are added to the worklist;
and so on. No other state is `Top` from then on (because of
monotonicity) so every reachable block is processed.

However, if the entry block is completely empty except for the
terminating branch, the state remains `Top`; then the entry state of its
successors, even when updated, is still `Top`; and the state didn't
change so the blocks are not added to the worklist. (Nevermind that they
were not processed in the first place!) The bug is that the invariant
"has been processed already with current state" is not true initially,
when the current state is set to `Top` but nothing has been processed.

This PR makes a simple fix: it adds every block to the worklist
initially to be processed, in input order (which is usually RPO order in
practice) as a good first heuristic; then if after processing the input
state changes again, it can be reprocessed until fixpoint as always.

Fixes bytecodealliance/wasmtime#5791.

Author: cfallin

src/checker.rs

@@ -987,8 +987,19 @@ impl<'a, F: Function> Checker<'a, F> {
         let mut queue = Vec::new();
         let mut queue_set = FxHashSet::default();
 
-        queue.push(self.f.entry_block());
-        queue_set.insert(self.f.entry_block());
+        // Put every block in the queue to start with, to ensure
+        // everything is visited even if the initial state remains
+        // `Top` after preds update it.
+        //
+        // We add blocks in reverse order so that when we process
+        // back-to-front below, we do our initial pass in input block
+        // order, which is (usually) RPO order or at least a
+        // reasonable visit order.
+        for block in (0..self.f.num_blocks()).rev() {
+            let block = Block::new(block);
+            queue.push(block);
+            queue_set.insert(block);
+        }
 
         while !queue.is_empty() {
             let block = queue.pop().unwrap();