Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/mdoc
/mdoc.exe
/modelstackcheck
/modelstackcheck.exe
/report.json
/report.md
.DS_Store
31 changes: 0 additions & 31 deletions BUILD.md

This file was deleted.

93 changes: 55 additions & 38 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,92 +1,109 @@
# ModelStackCheck

![ModelStackCheck mark](docs/images/modelstackcheck-mark.svg)
![Generated illustration showing a model connected to a provider and a coding tool](docs/images/modelstackcheck-overview.jpg)

ModelStackCheck helps you find where a coding model setup breaks, and it points to the next useful check. It tests the provider directly, and it can send a read only fixture through OpenCode as well.
ModelStackCheck finds where an AI coding setup breaks and points you toward the next useful check. It tests the provider directly, then it can send a safe fixture through OpenCode so you can see whether the full path works.

![A sample ModelStackCheck report](docs/images/doctor-report.svg)
![Generated illustration of provider, tool call, file edit, context, and image checks](docs/images/modelstackcheck-diagnostic-path.jpg)

## Try it
![Sample report from a local ModelStackCheck run](docs/images/doctor-report.svg)

Download [version 0.1.3 alpha](https://github.com/Praket7/modelstackcheck/releases/tag/v0.1.3-alpha) for your computer, and run a private demo with no model account.
## Download version 0.1.3 alpha

Choose the download that matches your system and processor. Each archive contains the command line program for that platform. The [release page](https://github.com/Praket7/modelstackcheck/releases/tag/v0.1.3-alpha) also includes SHA 256 checksums.

| System | Processor | Download |
| --- | --- | --- |
| macOS | Apple silicon | [Download](https://github.com/Praket7/modelstackcheck/releases/download/v0.1.3-alpha/modelstackcheck_v0.1.3-alpha_darwin_arm64.tar.gz) |
| macOS | Intel | [Download](https://github.com/Praket7/modelstackcheck/releases/download/v0.1.3-alpha/modelstackcheck_v0.1.3-alpha_darwin_amd64.tar.gz) |
| Linux | Intel or AMD 64 bit | [Download](https://github.com/Praket7/modelstackcheck/releases/download/v0.1.3-alpha/modelstackcheck_v0.1.3-alpha_linux_amd64.tar.gz) |
| Linux | ARM 64 bit | [Download](https://github.com/Praket7/modelstackcheck/releases/download/v0.1.3-alpha/modelstackcheck_v0.1.3-alpha_linux_arm64.tar.gz) |
| Windows | Intel or AMD 64 bit | [Download](https://github.com/Praket7/modelstackcheck/releases/download/v0.1.3-alpha/modelstackcheck_v0.1.3-alpha_windows_amd64.tar.gz) |

After downloading, open a terminal in the folder containing the archive and extract it with this command.

```sh
modelstackcheck doctor --provider mock
tar -xzf modelstackcheck_v0.1.3-alpha_darwin_arm64.tar.gz
```

The mock provider is local and repeatable, so it lets you see the report before connecting a real model.
Use the archive name that matches your download. On Windows, extract the archive with File Explorer or 7 Zip. The program inside is named `modelstackcheck_v0.1.3-alpha_windows_amd64.exe`.

On macOS or Linux, run a private demo with the local mock provider. For another system, replace the program name with the one inside its archive.

```sh
./modelstackcheck_v0.1.3-alpha_darwin_arm64 doctor --provider mock
```

The mock provider runs on your computer and needs no model account. To see the installed version, run the program with the `version` command.

## What it checks

The quick check tests the provider connection and streaming, and then checks basic and nested tool calls. It also tests enum choices, multiple calls, recovery after a tool error, a file edit in a temporary project, and a restricted file listing.
The quick profile checks the provider connection, streaming, basic and nested tool calls, multiple calls, recovery after a tool error, a file edit in a temporary project, and a restricted file listing.

Add `--harness opencode` to run the same fixture task through the provider and the installed OpenCode command. ModelStackCheck compares the results, creates a temporary project, reads one fixture file, and removes the project afterward. Use `--timeout` for provider requests and `--harness-timeout` for the OpenCode task.
Add `--harness opencode` to send the same fixture through the installed OpenCode command. ModelStackCheck creates a temporary project, asks the model to read and edit a fixture file, checks the result, and removes the project afterward.

Add `--profile context` to test retrieval at about 16K and 32K tokens. It repeats each check three times, and these sizes are estimates based on four characters per token. The large request may use more tokens with a hosted provider.
Add `--profile context` to check retrieval at about 16K and 32K tokens. Each check runs three times, and token counts are estimates based on four characters per token. Hosted providers may charge for these requests.

Add `--profile vision` to send a generated red image and check whether the model identifies it. Add `--profile full` to run both context and image checks.
Add `--profile vision` to send a generated red image and check whether the model can identify it. Add `--profile full` to run both context and image checks.

```sh
modelstackcheck doctor --harness opencode --profile full
./modelstackcheck_v0.1.3-alpha_darwin_arm64 doctor --harness opencode --profile full
```

Use the built in mock scenarios to see how ModelStackCheck reports a broken tool call, an OpenCode parser failure, a timeout, or a context limit without relying on a live model.
The mock scenarios can show how a failed tool call, an OpenCode parser problem, a timeout, or a context limit appears in the report.

```sh
modelstackcheck doctor --provider mock --mock-scenario malformed-tool
modelstackcheck doctor --provider mock --harness opencode --mock-scenario harness-parser
modelstackcheck doctor --provider mock --harness opencode --mock-scenario harness-timeout --harness-timeout 300ms
modelstackcheck doctor --provider mock --profile context --mock-scenario context-degradation
./modelstackcheck_v0.1.3-alpha_darwin_arm64 doctor --provider mock --mock-scenario malformed-tool
./modelstackcheck_v0.1.3-alpha_darwin_arm64 doctor --provider mock --harness opencode --mock-scenario harness-parser
./modelstackcheck_v0.1.3-alpha_darwin_arm64 doctor --provider mock --profile context --mock-scenario context-degradation
```

ModelStackCheck sends prompts to the provider you select, but it does not save them. Review reports before sharing, and remember that a successful probe covers only the tested request.

## Connect a provider
## Connect a model provider

Ollama is detected at its usual local address, and other OpenAI compatible services use these environment settings.
Ollama is detected at its usual local address. Other services that use the OpenAI compatible API can be set up with these environment values.

```sh
export OPENAI_BASE_URL=https://api.example.com/v1
export OPENAI_API_KEY=your-key
export OPENAI_MODEL=your-model
modelstackcheck doctor
./modelstackcheck_v0.1.3-alpha_darwin_arm64 doctor
```

## Read or save a report
ModelStackCheck sends prompts only to the provider you select. It does not save prompts or upload reports. Review a report before sharing because provider errors can include local details.

## Save and read reports

Reports can be shown in the terminal, or saved as JSON and Markdown. Keys, email addresses, and home folder names are redacted from report text.
Reports can be shown in the terminal or saved as JSON and Markdown. Keys, email addresses, and home folder names are redacted from report text.

```sh
modelstackcheck doctor --format json --output report.json
modelstackcheck doctor --format markdown --output report.md
modelstackcheck diagnose report.json
./modelstackcheck_v0.1.3-alpha_darwin_arm64 doctor --format json --output report.json
./modelstackcheck_v0.1.3-alpha_darwin_arm64 doctor --format markdown --output report.md
./modelstackcheck_v0.1.3-alpha_darwin_arm64 diagnose report.json
```

## Apply a supported fix

ModelStackCheck previews a provider timeout change, and it applies the change only when you add `--apply`. The report must include an OpenCode timeout finding, and the original file is saved beside the config as a backup.
ModelStackCheck can suggest a provider timeout change when an OpenCode report includes that finding. It shows a preview first, and it changes the configuration only when you add `--apply`. The original file is saved beside the configuration as a backup.

```sh
modelstackcheck fix --report report.json --config opencode.json --provider ollama
modelstackcheck fix --report report.json --config opencode.json --provider ollama --apply
./modelstackcheck_v0.1.3-alpha_darwin_arm64 fix --report report.json --config opencode.json --provider ollama
./modelstackcheck_v0.1.3-alpha_darwin_arm64 fix --report report.json --config opencode.json --provider ollama --apply
```

Fixes support standard JSON configuration files. JSON with comments is left untouched, and an existing backup is never overwritten.

## Current limits

This is an early alpha, so context sizes are estimates and the image check tests one simple generated image. The OpenCode check has been verified with a deterministic local provider, and real provider behavior can vary. The tool does not certify a model for every project, and it does not change files without the explicit apply option.
Fixes support standard JSON files. JSON with comments is left untouched, and an existing backup is never overwritten.

## Build from source

Install Go 1.27 or newer, and build the command from this folder.
Install Go 1.27 or newer, then run these commands from the project folder.

```sh
go test ./...
go vet ./...
go build -o modelstackcheck ./cmd/modelstackcheck
```

## Project
## Project status

Version 0.1.3 alpha is an early preview. The OpenCode path has been checked with a deterministic local provider, and real provider behavior can vary. Context sizes are estimates, and the vision profile checks one simple generated image. A passing report covers only the requests that were tested, so it cannot certify a model for every project.

ModelStackCheck is open source under the MIT license, and contributions are welcome. Read the [build notes](BUILD.md), [contribution guide](CONTRIBUTING.md), and [security policy](SECURITY.md) before sending a change.
ModelStackCheck is available under the MIT license. Read the [contribution guide](CONTRIBUTING.md) before proposing a change, and use the [security policy](SECURITY.md) to report a security issue privately.
Binary file added docs/images/modelstackcheck-diagnostic-path.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
9 changes: 0 additions & 9 deletions docs/images/modelstackcheck-mark.svg

This file was deleted.

Binary file added docs/images/modelstackcheck-overview.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading