Add doctest workflow

[renezander90]

Description

Implementation strategy: run only on src/qrisp/qtypes, and expand incrementally throughout the code base as flagged issues are resolved.

Related Issues

Closes #
Related to #830

Type of Change

• Feature (new functionality)
• Change Request (modification of existing functionality)
• Bug Fix
• Refactoring (no behavior change)
• Performance improvement
• Documentation
• CI / Build

Breaking Change?

• Yes
• No

If yes, describe the impact and migration path:

What was changed?

How was it tested?

Test-ID	Status
T-001	✅ / ❌
T-002	✅ / ❌
T-003	✅ / ❌
T-004	✅ / ❌

Screenshots / Output (if applicable)

Checklist

• My code follows the project's coding standards
• I have performed a self-review
• I have added/updated tests (referencing issue Test-IDs)
• All tests pass locally and in CI
• I have updated the documentation
• I have added a changelog entry to changelog-dev.rst
• Breaking changes are documented with migration path

Reviewer Notes

[PietropaoloFrisoni]

Thanks @renezander90 . Should we start adding <BLANKLINE> in docstrings?

[purva-thakre]

@PietropaoloFrisoni Feel free to edit the example docstring example in #830 if you think it needs something additional.

we likely won't be turning on doctest for quite a while. I was planning to take a look at this myself after QCE. Thankfully, Rene jumped on this :)

[purva-thakre]

@renezander90 Since you started looking into this, could you check if jupyter-sphinx is a better alternative to doctest? Pietro's comment reminded me that we would have to spend a lot of time making sure everything in the docstring examples output is formatted correctly to make doctest happy.

With jupyter-sphinx, you simply add the code in the examples section (minus the output) and the code block gets executed when you run the sphinx build. I have used this in a past project before the project moved to a md based docs workflow. https://jupyter-sphinx.readthedocs.io/en/latest/

[renezander90]

It is recommended to use a combination of doctest and jupyter-sphinx:

Using only jupyter-sphinx could work, but I would not recommend it for Qrisp.

The main issue is that jupyter-sphinx is primarily an executable documentation renderer, whereas pytest doctest is also a regression-testing mechanism. With only jupyter-sphinx:

• existing >>> docstring examples must be rewritten as jupyter-execute directives;
• docstrings are no longer automatically tested as Python source examples;
• examples that merely run without exceptions may still produce incorrect quantum results unless they contain explicit assert statements;
• documentation builds require a working Jupyter kernel and all runtime dependencies;
• build time and reproducibility become more sensitive to simulations, measurements, randomness, and backend behavior;
• Qrisp already uses nbsphinx for complete notebook tutorials, so there would be overlap.

For example, this is weaker as a regression test:

.. jupyter-execute::

   qf = QuantumFloat(3, -1)
   qf[:] = 2.5
   print(qf)

It verifies that the code runs, but an incorrect printed value may not fail the build in a meaningful way because the output is generated automatically.

This is stronger:

.. jupyter-execute::

   qf = QuantumFloat(3, -1)
   qf[:] = 2.5
   assert qf.get_measurement() == {2.5: 1.0}

My recommendation remains:

pytest doctest       short API/docstring regression tests
jupyter-sphinx       executable RST/Markdown examples with generated output
nbsphinx             complete notebook tutorials

Using only jupyter-sphinx is reasonable only if the project intentionally wants to migrate all examples out of docstrings and accepts that documentation execution, rather than exact doctest expectations, will be the primary verification mechanism.

[renezander90]

@PietropaoloFrisoni <BLANKLINE> should be used wherever it is appropriate:

Use <BLANKLINE> when the actual program output contains an empty line:

>>> print("start")
>>> print()
>>> print("end")
start
<BLANKLINE>
end

Do not use it for formatting the docstring or separating examples:

>>> x = 1

>>> x + 1
2

That blank line separates two doctest interactions; it is not output.

Also do not use it when the output has no blank line:

>>> print("start")
start
>>> print("end")
end

So the rule is: use <BLANKLINE> only where the executed code actually prints an empty line.

[PietropaoloFrisoni]

It should even appear in the rendered sphinx version if used properly 👍

[renezander90]

@PietropaoloFrisoni Feel free to edit the example docstring example in #830 if you think it needs something additional.

we likely won't be turning on doctest for quite a while. I was planning to take a look at this myself after QCE. Thankfully, Rene jumped on this :)

Actually, I would like to turn it on as soon as possible. We could first keep it restricted to src/qrisp/qtypes as soon as the issues in this module are fixed (after #846, #848). This would ensure that all documentation examples in this module will work perfectly:) Afterwards, we can have a series of follow-up PR's to extend it gradually to more modules of Qrisp.

Regarding the necessary formatting changes, this can be done quite fast with AI. Only bugs or more fundamental problems have to be resolved manually (e.g. the arithmetic bugs I discovered when using doctest on the qtypes module).

[purva-thakre]

@renezander90 If that's the case, before turning it on, could you compare sphinx doctest to pytest doctest? From experience, sphinx doctest takes a loooot of time. The LLM I am using recommended pytest doctest as a better alternative.

Regarding the necessary formatting changes, this can be done quite fast with AI.

If you say so! I am a bit cautious that the LLM could make unnecessary changes that I would fail to catch.

[renezander90]

Some recommendation on dealing with randomness:

Recommendation for nondeterministic examples

The goal should be to keep examples natural while avoiding assertions about values that are inherently unstable.

Situation	Recommended doctest approach
Deterministic output	Show and verify the exact output
Floating-point variation	Check a tolerance instead of printing the value
Variable formatting or generated names	Use ELLIPSIS or NORMALIZE_WHITESPACE
Probabilistic quantum result	Check an invariant or statistical property
Random, backend-dependent, unavailable, or expensive example	Use SKIP with a clear reason

Deterministic output

>>> qf = QuantumFloat(3, -1)
>>> qf[:] = 2.5
>>> qf.get_measurement()
{2.5: 1.0}

Floating-point results

Instead of maintaining a fragile decimal representation:

>>> computed_value = calculate_value()
>>> abs(computed_value - expected_value) < 1e-8
True

This keeps the example readable while checking the meaningful behavior.

Variable formatting

For generated names or partially unstable textual output:

>>> print(circuit)  # doctest: +ELLIPSIS
QuantumCircuit: ...

Use this only when the omitted portion is genuinely irrelevant. Do not hide the actual result behind ....

For insignificant whitespace differences:

>>> print(description)  # doctest: +NORMALIZE_WHITESPACE
A description with normalized whitespace.

Probabilistic quantum results

Avoid showing one exact measurement result when several outcomes are valid:

>>> qv = QuantumVariable(1)
>>> h(qv)
>>> measurement = measure(qv)
>>> set(measurement).issubset({"0", "1"})
True
>>> sum(measurement.values())
1.0

For statistical behavior, use a sufficiently large number of shots and test a broad property rather than exact frequencies. Detailed statistical checks are usually better placed in unit tests than in doctests.

Random or unsuitable examples

SKIP is acceptable when you genuinely want to show a random result, but do not want the documentation build to verify it:

>>> random_result()  # doctest: +SKIP
0.731842...

The result is displayed as illustrative documentation, but the code is not executed or checked. Add prose making this clear:

The following output is an example; the actual result varies between runs.

SKIP is also appropriate for examples that are expensive, backend-specific, require unavailable credentials, or depend on external services:

>>> run_on_hardware_backend()  # doctest: +SKIP
...

Use it sparingly, because skipped examples provide no automated correctness guarantee.

The resulting policy is:

deterministic output  -> exact doctest output
floating-point drift  -> tolerance check
variable formatting   -> ELLIPSIS or NORMALIZE_WHITESPACE
probabilistic result  -> invariant or statistical property
random example to show -> SKIP with an explicit explanation
unavailable/expensive  -> SKIP with a clear reason