Skip to content

Make examples work when compiled #630

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
redorlik wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Make examples work when compiled #630

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3ecbf18
Try to test that the examples works when compiled. Currently marked a…
redorlik Jul 19, 2026
ca427a1
To fix the exit code example the build and execute are split into two…
redorlik Jul 19, 2026
b6ccd73
Merge branch 'spylang:main' into main
redorlik Jul 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 25 additions & 4 deletions examples/test_examples.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,16 @@ def _run(spy_file: Path) -> subprocess.CompletedProcess:
check=False,
)

def _build_run(spy_file: Path) -> subprocess.CompletedProcess:
cmd = ["spy","build", str(spy_file)]
subprocess.run(cmd)
cmd = [str(spy_file.parent)+'/build/'+str(spy_file.stem)]
return subprocess.run(
cmd,
capture_output=True,
text=True,
check=False,
)

def test_all_examples_have_expected_output(request) -> None:
"""Catch the case where a new .spy file was added without a saved expected output.
Expand All @@ -76,11 +86,11 @@ def test_all_examples_have_expected_output(request) -> None:
)


@pytest.mark.parametrize("spy_file", _spy_files, ids=lambda f: f.stem)
def test_example(spy_file: Path, request) -> None:
expected_file = EXPECTED_OUTPUT_DIR / f"{spy_file.stem}.txt"
result = _run(spy_file)

def run_example(spy_file: Path, request,runner = _run) -> None:
expected_file = EXPECTED_OUTPUT_DIR / f"{spy_file.stem}.txt"
result = runner(spy_file)
print(result.returncode,expected_returncode(spy_file))
assert result.returncode == expected_returncode(spy_file), (
f"spy exited with code {result.returncode}\n"
f"--- stdout ---\n{result.stdout}"
Expand All @@ -107,3 +117,14 @@ def test_example(spy_file: Path, request) -> None:
f"Output of {spy_file.name} differs from {expected_file.relative_to(EXAMPLES_DIR)}\n\n"
"Update the expected result with: pytest --update-examples"
)

@pytest.mark.parametrize("spy_file", _spy_files, ids=lambda f: f.stem)
def test_example_interp(spy_file: Path, request) -> None:
run_example(spy_file,request)

@pytest.mark.parametrize("spy_file", _spy_files, ids=lambda f: f.stem)
def test_example_build(spy_file: Path, request) -> None:
# There are four examples which fails in the c compiling stage - so they are marked as XFAIL
if spy_file.stem in ['collections','annotated','unroll_nested_loops','convert']:
pytest.xfail()
Comment on lines +128 to +129

@antocuni antocuni Jul 21, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure about this approach.
The biggest issue is that it doesn't solve the problem that you encountered during the sprint, i.e. that you opened an example, tried to compile and see it was broken.

I think that a better way is the following:

  1. add a special "marker" at the beginning of the docstring of examples which are known not to compile
  2. detected the marker here (could be something as simple as if MARKER in spy_file.read()).

What to use as a marker is open to discussion, but I propose something like this in the first line of the docstring:

"""
SPY BUILD: xfail

Things to notice:
  - Tuples, dicts, and lists work as in Python.
[...]
"""

def main() -> None:
    ...

run_example(spy_file,request,_build_run)
Loading