Skip to content

Add packaging mechanism to automatically build SV-Comp zip - #30

Draft
nils-loose wants to merge 373 commits into
mainfrom
dev
Draft

nils-loose wants to merge 373 commits into
mainfrom
dev

Conversation

@nils-loose

Copy link
Copy Markdown
Contributor

No description provided.

Comment thread .github/workflows/gradle.yml Outdated
Comment on lines +9 to +10
SVCOMP_REFERENCE_URL: https://zenodo.org/api/records/17748741/files/swat-verify.zip/content
SVCOMP_REFERENCE_MD5: 8c776d19ac33ecf5c1c7441d16cc18a6

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@k0lja Can you propose a different mechanism, perhaps another repository here that publishes the required files, I dont want to reference zenodo here in the CI

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In addition, the venv we are using is built on the specific ubuntu version required by sv-comp. updating the dependencies is a hassle atm. Can you check if it could be possible either here or in the supporting repo (for the above) you can actually build the venv based on the requirements (pip freeze the current venv) as part of the ci there? this would make updating the venv much easier

Comment thread .github/workflows/gradle.yml Outdated
Comment on lines +30 to +31
- name: Build WitnessCreator JAR
run: ./gradlew --no-daemon :targets:sv-comp:WitnessCreator:shadowJar

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed could you also move the full witness creator into its own repo that publishes an artifact we can use here?

Comment thread .github/workflows/gradle.yml Outdated
@@ -18,8 +23,142 @@ jobs:
distribution: 'temurin'
- name: Setup Gradle
uses: gradle/actions/setup-gradle@017a9effdb900e5b5b2fddfb590a105619dca3c3 # v4

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While you are at it, could you also check why the setup-gradle v4 is referenced using this hash I assume? is that the proper way to do that? (I know this is older then your changes but I was wondering for some time)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This entire package of supporting file should then also move into the supporting repo I guess?

solid-eureka and others added 29 commits October 16, 2026 20:21
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two branches of one trace can share an iid (switch case index wraps at 16,
loop counter at 64), and one got the other's constraint: false SAFE.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…inside)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…call tree

The graph's only targets are assert sites, so pruning by it under any
other property (no-runtime-exception) drops paths to the violation. The
explorer now runs no pre-analysis unless the property is valid-assert,
and logs why; the harness no longer passes --sa-path for other
properties.

The walk follows main's call tree only, so an assert run by a shutdown
hook, another thread or an uncaught-exception handler is on no path it
knows. A graph with a library call that hands application code to such
an entry (Thread.start, addShutdownHook, set*UncaughtExceptionHandler,
executor submission, Timer, *Async) is not used. When a callback reaches
an assert, leaving main normally or by an exception is interesting too,
since such a callback may still run after main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the trace's branch key or a frame of its call stack is missing,
check_branch cannot tell whether the walk is still in step. The walk
still follows the graph there, so later branches can be checked, but no
side of that branch is pruned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DfsTest trees now carry branch keys and stack deltas, since unkeyed
pairings no longer prune.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FCMP built its formula from greaterThan and FP equality only, so a NaN
operand compared as 'less' in the formula, while the JVM pushes 1 (FCMPG)
or -1 (FCMPL). The solver could then pick NaN to satisfy an ordered
comparison such as 0.0f <= x, and the run took the other side. The
concrete value used Float.compareTo, which orders -0.0f below 0.0f and NaN
above everything. Mirror DoubleValue.DCMP: an explicit NaN case in the
formula and IEEE comparisons for the concrete value. A concrete NaN
operand no longer drops the formula.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The nondet redirect assumed a ()X descriptor and produced an invalid
descriptor for nondetObject(Class, ObjectFactory), so instrumenting the
calling class failed. nondetObject is defined as factory.createObject(),
and the factory makes its choices with the primitive nondet methods, so
the call is now rewritten to the factory call in the instrumented
caller: the factory's nondet values are symbolic and every object it
can return is explored.

A factory that is not instrumented makes concrete choices; its
createObject call is recorded as a context loss so that SAFE is
downgraded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A run that does not take the side its solution was solved for used to
leave the branch solved, so its unexplored side was never tried again
and SAFE was ruled out (unexpected_divergence). Most such runs end in an
OutOfMemoryError before the branch: Z3 picks a size such as 2^31-1 and
the allocation fails, though small sizes reach the branch.

Now the branch becomes a candidate again, up to --divergence-retries
times (default 3). A retry excludes every solution that failed so far
and looks for small values first: integral inputs bounded by 16, 256 and
65536 in turn, then unbounded. UNSAT under these extra constraints
proves nothing, so it is never recorded as an infeasible branch. Only
when all retries diverge, or no other solution is left, is the branch
given up on and recorded as unexpected_divergence (or as an expected
divergence for an approximate model, as before). A given-up branch
rules out SAFE unless another run reached its unexplored side by the
end of the search.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… be added

Every Database method now holds the lock in a with-block. An exception in
add_trace (a RecursionError on a long trace) used to leave it held, and
the driver then waited for it until the time limit. get_violations also
returned with the lock held for an unknown endpoint.

add_trace records the soundness flags before it adds the trace to the
tree, so they hold even if that fails. ConstraintService catches a
failure, logs it and records a trace_ingest_failed execution error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- An exception in the exploration ends the run at once with UNKNOWN and
  an explorer_crash execution error. Before, the worker thread died, the
  HTTP server kept the process alive and the run hung until the time
  limit. run_sv_comp prints DONT-KNOW for anything that fails around the
  driver and exits.
- A trace that could not be added ends the search with UNKNOWN: the tree
  lacks its path. A violation in the same run still counts.
- The divergence check only runs when the round sent a trace. Before, it
  checked the previous round's trace. A round without a trace records
  no_trace: its branch counts as solved and is never tried again.
- tree_mismatch, trace_ingest_failed, no_trace and explorer_crash rule
  out SAFE.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The walk was recursive and visited shared subterms once per path to them,
which is exponential in the depth of a DAG.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
solid-eureka and others added 30 commits October 28, 2026 11:12
- The explorer logs relayed executor output (log.RELAYED) as it is, without its
  own timestamp, thread and level in front of the executor's.
- The harness reads a task's output while it runs and, in single mode, prints
  each line when it arrives instead of all of them at the end.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
get_verdict_logger() returned the explorer logger, so verdict.log stayed empty.
The verdict still goes to explorer.log too, and is printed once on the console.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ThreadHandler.init() created a Z3 SolverContext that nothing ever read, and
never closed the previous one. Six test classes called it before every test
case, so each case leaked a whole Z3 context (about 17 MB): the executor tests
peaked above 22 GB, and GitHub's runner (16 GB) was shut down in the middle of
:symbolic-executor:test. Without it the tests peak at 1.9 GB, and the executor
no longer creates an unused Z3 context at the start of every run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ThreadHandler.init() created a Z3 SolverContext that nothing ever read, and
never closed the previous one. Six test classes called it before every test
case, so each case leaked a whole Z3 context (about 17 MB): the executor tests
peaked above 22 GB, and GitHub's runner (16 GB) was shut down in the middle of
:symbolic-executor:test. Without it the tests peak at 1.9 GB, and the executor
no longer creates an unused Z3 context at the start of every run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit c17c0ef)
ThreadHandler.init() created a Z3 SolverContext that nothing ever read, and
never closed the previous one. Six test classes called it before every test
case, so each case leaked a whole Z3 context (about 17 MB): the executor tests
peaked above 22 GB, and GitHub's runner (16 GB) was shut down in the middle of
:symbolic-executor:test. Without it the tests peak at 1.9 GB, and the executor
no longer creates an unused Z3 context at the start of every run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit c17c0ef)
ThreadHandler.init() created a Z3 SolverContext that nothing ever read, and
never closed the previous one. Six test classes called it before every test
case, so each case leaked a whole Z3 context (about 17 MB): the executor tests
peaked above 22 GB, and GitHub's runner (16 GB) was shut down in the middle of
:symbolic-executor:test. Without it the tests peak at 1.9 GB, and the executor
no longer creates an unused Z3 context at the start of every run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit c17c0ef)
Since 373384d the marker is no tree level: Tree.structural keeps it as the
unmodelled_exception flag on the node after it, and Tree.add is the only
place that builds nodes. The walk's check for a marker node could only be
reached by hand-built trees in test_sa_walk, which now set the flag instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dfs (strategy/DFS.py) now reads as its three steps per branch: pair it
with the graph (_pair), prune, slice (_classify). How a branch was paired
is one of verified, resynced, unverified, lost or masked, and the walk
counts those (Tree.walk) and logs one line per selection
("[DFS] Walk: ..."), also written to stats.json (static_analysis.last_walk).
The walk state is a named tuple instead of an 8-tuple; dfs loses the
parameters only it used itself (visited, clinit_depth, sa_stack, sa_index,
trace_stack). Merged and sliced branches are one classification.

SAGraph: target points are SANode.isTarget, reachesTargetSameLevel and
targetDistance (no longer named after asserts). Dropped what nothing
needs any more: graphs without return sites (and their
context-insensitive marking), isPhantomGuard, the RETURN edge lists, the
child accessors, and the call site's fall-through to its one callee.
SANode.handler_for replaces three copies of the handler lookup.

Relevance: the unused constructor parameter is gone, the closure of
methods running a relevant branch is computed by one helper.

Node and Leaf have class defaults for every per-node flag, so neither
the walk nor tests need getattr fallbacks.

Removed --sa-retry-without (explorer and harness): a guard against false
SAFE from SA bugs from before keys checked the walk; desyncs now turn
SAFE into UNKNOWN instead.

The analysis results are unchanged: every node's summaries, relevance,
regions and distances are the same as before on all 2201 graphs of
sv-benchmarks and swat-benchmarks (valid-assert and
no-runtime-exception).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…m too

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
StrategyService used a module logger, which reaches neither explorer.log
nor the console: its walk summary (and, before, the relevance and
loop-state lines) were never seen. It now uses the explorer's logger;
is_symbolic_branch's two INFO lines per new candidate are one DEBUG line.

The walk line uses the documented terms:
  [DFS] Walk: 34 branches (33 verified, 0 resynced, 0 unverified, 0 lost,
  1 masked), 12 sides pruned; 32 sliced, 0 merged, 0 covered, 0 poisoned

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The SA modules (DFS, StrategyService, SAGraph, Relevance, the tree
classes, StaticPreAnalysis) pass mypy with --check-untyped-defs
--disallow-untyped-defs --disallow-any-generics.

Named types instead of bare tuple/dict/set:
- SAGraph: Location and Effect (the data-flow facts: dataflow is
  tuple[Effect, ...], control tuple[Location, ...]), CallFacts (a
  TypedDict for the `call` field), ExceptionChain, TraceFrames and
  TraceStack. LOST is an enum sentinel, so `is LOST` narrows.
- Relevance: CfgNode (a graph node or a virtual EXIT/EXIT_EXC/END),
  Method (str | None: nodes without method metadata).
- DFS: Pairing and Role are Literal types, OpenRegions names the
  region list; dfs's walk state is fully typed.
- Node: Ident and BranchName (a candidate's identity); Branch: Position
  and PositionedState (loop-head states).
- StrategyService.select_branch takes named parameters instead of
  **kwargs; `-> list()` return annotations (a call, not a type) are
  list[str].

Where a value is Optional in general but always set where it is read,
there is now one accessor with an assertion instead of an unchecked
read: SANode.sides() (a branch's two successors) and resume_site() (a
call site's return site). dfs keeps the verified graph node in one
variable (None unless the pairing was verified), which is what the
pruning and slicing checks.

Also: the Leaf docstring is its __doc__ again, Node caches its depth in
a declared field (depth_cache) instead of a getattr'd _depth, and
loop variables that were reused with another type in Relevance are
renamed.

Every node's analysis results are unchanged on the 2201 graphs from
sv-benchmarks and swat-benchmarks; all explorer tests pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Relevance (slicing) and loop summaries; rebased onto svcomp27/overnight-solver-perf
Brings in the overnight solver performance work (#96, #98, #99), the
property pre-check in cfg-extraction (#108), and relevance slicing and
loop summaries (#109).

The one conflict, in Transformer.java, comes from the #110 fix having
been committed on both branches; resolved as on the overnight branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts:
#	symbolic-executor/src/main/java/de/uzl/its/swat/instrument/Transformer.java
Brings in the SA cleanup (one vocabulary for the walk, legacy graph
format and --sa-retry-without removed, walk counts in the log and in
stats.json), full type annotations of the SA code, and the console log
changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Against dfs (300 s, with SA, 2026-11-03), coverage found 6 more
violations on swat-benchmarks and 4 more on sv-benchmarks valid-assert
that dfs did not find in five runs, lost only a flaky one, and left every
SAFE verdict as it was. On no-runtime-exception it missed one real bug
dfs finds (WeibullGenerator_exceptionless, whose label is wrong and is
fixed locally in the sv-benchmarks checkout).

The harness now always passes --search-order, so that an explicit
--search-order dfs is not lost to the explorer's default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fault

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The approximate Math models were exact only at "nice" inputs (small
numbers, multiples of pi) and at inputs giving nice outputs, so a result
that overflows to infinity or becomes NaN could not be reached:
Math.pow(0.69, (int) (1 / beta)) is +Infinity only for an exponent of
about -1940 or less, which is no sample. sv-benchmarks
WeibullGenerator_exceptionless throws for such a beta, and its throw
branch was UNSAT, or out of the resource limit, in every round; dfs
found the bug only through a lucky model of another branch, coverage not
at all.

The tables now also sample Integer and Long MIN/MAX_VALUE, +-Double.MAX_VALUE,
+-Double.MIN_VALUE, +-Infinity and NaN. The task's throw branch is then
solved directly: VIOLATION in 11 rounds with coverage, 16 with dfs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The archive only shipped the property pre-check, and run_swat.py passed no --sa-path, so SA never
ran there. Now:

- CI builds cfg-extraction's fatJar next to precheckJar, and package-svcomp.sh ships it as
  cfg-extraction/cfg-extractor.jar.
- The explorer takes the extractor jar itself with --sa-jar (--sa-path still takes a checkout).
- run_swat.py passes --sa-jar under valid-assert only (as the harness does) and --target Main.
  Without --target the graph's entry was None and the extractor failed, silently leaving the run
  unpruned; StaticPreAnalysis now also defaults the entry to Main, the class the executor runs.
- third_party_licenses.py lists the fat jar's components (needs its META-INF/swat-third-party.json
  from cfg-extraction's fatJarComponentList) and maps SootUp's and JGraphT's license names.
- check-svcomp-package.sh fails unless the smoke test's static pre-analysis loaded its graph.

Needs a cfg-extraction release with fatJarComponentList in cfg-extraction.ref (v0.2.0 has none).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… static pre-analysis

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since ca0f2a5 a run reports no more loop-head states after a call into
untracked code other than a few pure methods. The static initializer of
every class with an assert calls Class.desiredAssertionStatus (javac's
$assertionsDisabled), so every valid-assert task stopped reporting
before main started, and loop-state matching pruned nothing in any task
of sv-benchmarks or swat-benchmarks (2026-11-03).

desiredAssertionStatus reads the JVM's assertion setting, which is the
same in every run, and its result lands in a static field of the
application class, which is hashed. float_unboundedloop/
Batch-Conveyor-Counter, whose state cycles, is SAFE again in 4 rounds
(UNKNOWN at the timeout without states).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With the extreme samples of the Math models (2a9ea29), Z3 started to pick
huge doubles (2^67 and beyond) for coral45 and coral53: there the model
of exp() is the value of the previous run, not the real one, so every
such solution diverged, and after the retries the branch was given up
(UNKNOWN in 17 s; both were found in every run before).

A solution with a floating-point input beyond 65536 in magnitude, or an
infinite one, is now re-solved with the inputs bounded first, as for
integral inputs, and the divergence retries bound floating-point inputs
too (excluding NaN and infinities in the bounded tries; the unbounded
try comes last). coral45 and coral53 are found again, and
EquidistantConicProjection_true, whose violation needs infinite inputs,
still is (its label is wrong, fixed locally).

The bounds are built in the value's own context: z3.fpGEQ, z3.fpAbs and
unary minus use the global one, which crashed or failed with the
solver's shared context.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DREM and FREM returned the concrete remainder without a formula and
without a soundness flag, so every branch on a remainder looked concrete:
a search that never ran into the violation by chance reported SAFE. With
the floating-point small-model preference (c8cdcc2) the early runs of
sv-benchmarks PieSegment_false/_true took other inputs, and both came out
SAFE although both violate valid-assert (angle % 360 is negative for a
negative angle).

The remainder is now a model like the transcendental Math functions
(SampledMath#define, now public and typed by its arguments): exact where
the quotient is at most 1 in magnitude (x for |x| < |y|; |x| - |y| with
x's sign for |y| <= |x| < 2|y|, exact by Sterbenz), the concrete result
elsewhere, and declared approximate, so a branch on it is a precision
loss. PieSegment_false and _true are VIOLATION again, in 4 rounds.

The exact encoding (fp.rem, adjusted by one |y| where its sign differs
from x's) is correct but too slow: Z3 did not decide
angle % 360 < -180 for a symbolic double within 120 s; the model takes
milliseconds. JavaRemainderSpec checks the model against Java's % bit for
bit on edge values of doubles and floats.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The small-model preference and the bounded divergence retries only look
for a nicer solution, but ran without a resource limit. Bounding
floating-point inputs (c8cdcc2) can make a query far harder:
ReverseInterpolator_false and _false2 spent 261 s on one bounded query,
until the CPU deadline, instead of running the original solution, which
reaches the violation. Each bounded query now gets at most 10M resource
units (about 4 s, BOUNDED_RLIMIT); running out moves on to the next bound
or keeps the original solution. Both tasks are VIOLATION again in 3
rounds; coral45 and coral53 stay found.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixed 10M-unit cap (4caea36) was too small where the query is
expensive anyway: jbmc-regression/calc needs 22M units unbounded and 25M
bounded; capped, the bounded query ran out, the run took the original
solution with an array size of 2^31 - 1 and ran out of memory (SAFE ->
UNKNOWN), and coral61/coral62 lost their violations the same way. The
limit is now twice what the branch's unbounded query used, at least 10M
(Z3Handler.last_rlimit_used, recorded per branch). calc is SAFE again,
coral61/62 and ReverseInterpolator_false are VIOLATION.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ate matching

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…alysis

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Extreme samples for the Math models, floating-point % as an approximate
model (it was silently concrete: false SAFE), small models for
floating-point inputs with bounded re-solves capped, and loop-head state
matching working again for programs with asserts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants