Add option to support unit tests

[stefandrissen]

Currently I am writing unit tests for every module I write.
From pyz80 the tests are wrapped in conditional assembly blocks witth if defined(unittest).
The tests indicate success or failure by putting either 200 (ok) or 500 (failure) in the bc register. This allows print usr 32768 to show the status of the test.

But... this requires:

• build
• load 1
• print usr 32768

Which is fine when writing the code initially, but no good for continuous automated testing.

A command-line option like -unittest which returns the value of bc (or something else) could really help.

[simonowen]

I've been thinking about the best approach to adding Python bindings to the emulator for some time, but hadn't really settled on an approach. Ideally I'd like to be able to launch a session from Python, perhaps load snapshots, set breakpoints, etc. all from the same script. That would give much more flexibility in testing the emulator itself, as well as exposing it for developers to test their own code.

There's a certain amount of overlap with what would be needed for a debug server for VS Code, so that is a tempting starting point. It would launch the emulator with options to behave as a debugger, and provide services to set breakpoints and view symbol values (mostly registers, probably memory too).

There's quite a lot of work needed to add full bindings, and the codebase isn't really in a state yet where it'd be worth starting. However, it may still be worth a simple version in the meantime, much like you're suggesting. Launching the emulator with an auto-executing disk would get the test run, leaving just the hook on return from the code (and BC returned as an exit code) to add.

[simonowen]

I've been experimenting with Lua scripting as an alternative to this, which looks very promising so far. Rather than starting and controlling the emulator from the outside (perhaps from Python), the scripting is driven from the inside by the emulation. I'm currently using a single global script from the settings directory, but may also add disk-specific scripts too. A script passed on the command-line seems ideal for the type of unit testing you're wanting to do.

I currently have CPU and ASIC registers plus RAM exposed to the scripting for testing, but that can be expanded to include many other devices and emulation features. Emulator events call event-specific functions in the script, giving it temporary control to do whatever it wants. That could be to update some state within the script, change the emulation state, etc. I've got basic screen bindings too, so the script can draw shapes and text to the screen each frame.

Attaching scripting directly to high-frequency events would probably hurt performance too much, even if the hooks aren't active. My initial approach will be to add actions to the existing breakpoints functionality. So you could create a conditional breakpoint that called a script function, and if no action is specified the default will be to break into the debugger as always. If you need it you're still free to create breakpoints against a high frequency events, with each trigger calling a function in the script.

Why Lua? The Lua engine implementation I'm using is stand-alone and much easier to portably integrate with the emulation than Python bindings would be. Then 3rd party Lua/C++ bindings make exposing variables and code to Lua very easy, often just a few lines of code. Performance has seemed very good so far and there's a LuaJIT option to look into if more performance is needed. Lua scripting is popular with games because it's performant and flexible, which is good for us too.

Your unit test script could be as simple as setting an execution breakpoint and quitting the emulator with an exit code from BC when it triggered. Scripting should offer users much more flexibility than any debugger enhancements could provide.

[simonowen]

Here's a working script example with the code I've got at the moment:

cpu = emu.cpu

function on_frame_end()
    if cpu.halted and not cpu.iff1 then
        emu.exit(cpu.bc)
    end
end

At the end of every frame it checks if the CPU is halted with interrupts disabled. If so it exits the emulator returning the value in BC.

This example requires your test code end in DI ; HALT and use the autoexec feature of pyz80 to start the test executing when loaded. Though with other features exposed you can have it do whatever you need. It could set a breakpoint at the end of the test and exit when that triggers, or keyin some BASIC text to load the program, etc.

[petemoore]

@stefandrissen Does #109 cover your use case now?

[stefandrissen]

Maybe, how are you detecting that tests have passed?

[simonowen]

It might need an exit code to be set from a register to complete the chain, when -exitonhalt is being used. Launch environments have different limits on how that's seen but it should be good enough to get a meaningful value out. To copy the existing USR pattern I'd be tempted to exit with the C register value, which should survive as an unsigned 8-bit value. It's all still a temporary band-aid until there's something better.

My experimental Lua bindings from above are no longer compatible with my current development code, as they sit fairly tightly over internal structures that have since been refactored. I still think it has good potential for scripted control for testing like this, and more. Though a debug adapter compatible with VSCode could be nice too, in addition to the replacement built-in debugger.

[petemoore]

Maybe, how are you detecting that tests have passed?

I launch SimCoupe with -exitonhalt 1 -parallel1 1 -outpath <dir> -nextfile 0, and the tests log their output to the printer and then execute di; halt to cause SimCoupe to exit. This means the test logs are published in <dir> and they can output whether they pass or fail together with any other useful context.