Skip to content

feat(scenarios): add firmware update success scenario - #133

Merged
sepehr-safari merged 4 commits into
ocpp-debugkit:mainfrom
YANGCHUNHONG3000:feat/firmware-update-success-scenario
Jul 28, 2026
Merged

feat(scenarios): add firmware update success scenario#133
sepehr-safari merged 4 commits into
ocpp-debugkit:mainfrom
YANGCHUNHONG3000:feat/firmware-update-success-scenario

Conversation

@YANGCHUNHONG3000

Copy link
Copy Markdown
Contributor

Add a new scenario to verify that a station undergoing a successful firmware update does not trigger any false positives:

  • expectedFailures: [] — The trace describes a clean firmware update flow (BootNotification → FirmwareStatusNotification(Dowloaded) → FirmwareStatusNotification(Installed) → Heartbeat), which should not trigger any detection rule.
  • assertions: [{ type: 'no_failures', params: {} }] — Verifies that no failures are detected, acting as a regression guard against overly aggressive detection rules.
  • tests/external-fixture/test.mjs was also updated (line 161: 1516) because it has a hard-coded scenario count assertion.
  • All 520 tests pass (19 test files).

@sepehr-safari

Copy link
Copy Markdown
Member

Thanks for this, and for writing up the reasoning in the description. The scenario
does the job it was asked to do: FIRMWARE_UPDATE_FAILURE is a real rule in
detection.ts, and a clean Downloaded then Installed flow is a sensible negative
control against it firing where it should not. I ran the branch locally and lint,
typecheck, build, the 520 unit tests, and the external fixture test all pass.

Four things before I merge. The first two are on the PR, the last two are on me
for leaving them out of the issue.

  1. Prettier. pnpm format:check fails on
    packages/toolkit/src/scenarios/__scenarios__/firmware-update-success.ts.
    Three of the message arrays are wrapped at 80 columns and this repo is set to
    100. pnpm format fixes it. CI runs the format check right after lint, so it
    would have stopped there. That step is missing from the contributing guide's
    local verification list, which is my gap and I am fixing it separately.

  2. Changeset. The PR adds a new public export (firmwareUpdateSuccessScenario)
    and changes the length of the exported scenarios array, so it needs a
    changeset. pnpm changeset, pick a minor bump for @ocpp-debugkit/toolkit,
    and commit the generated file.

  3. Station ID. CS-SYNTHETIC-012 is already used by meter-anomaly.ts. The
    existing scenarios each use a distinct ID, currently 004 through 015, so this
    one wants CS-SYNTHETIC-016. The issue told you to use the CS-SYNTHETIC-NNN
    shape without saying they have to be unique, so this one is on the issue text.

  4. Counts in the docs. Adding a scenario also makes these two lines stale:
    README.md:25 and packages/toolkit/README.md:23, both of which say
    "15 predefined scenarios". Happy for you to bump them here since it is a
    one-word edit in each. CONTRIBUTING.md needs the same bump plus an unrelated
    correction, so leave that file alone and I will take it. The counts in
    CURRENT_STATE.md are historical milestone entries and should stay as they are.

I will approve the workflow run so you get CI output on the next push.

One note on scope, separate from this PR. You claimed #104, #106, #107 and #108
together. I would rather assign these one at a time, so #104 is yours and I will
hold the others open. Once this merges, tell me which one you want next and I
will assign it. That is a limit in how I worded the claim instructions rather
than anything you did wrong, and I am rewording them.

sepehr-safari added a commit that referenced this pull request Jul 26, 2026
…e state (#135)

PR #133, the first external contribution to a good-first-issue, failed CI and
needed four review items. Three of the four came from gaps in the contributor
guide rather than from anything the contributor did.

CONTRIBUTING.md now documents `pnpm format:check`, which CI enforces and the
guide never named, with both command lists ordered to match the CI job step for
step and a note that CI stops at the first failure. The hard-coded counts are out
of the architecture table: the detection entry claimed 15 rules and there have
been 16 since REPEATED_BOOT_NOTIFICATION landed in #114, and those counts have
now drifted twice. "Adding a Scenario" names all four files carrying the scenario
count and flags that only two of them fail the suite when missed. Station IDs are
documented as unique per scenario. Good-first-issues carry a claim policy of one
open claim at a time.

Also corrects the v0.4.x release state in CURRENT_STATE.md, which reported 0.4.1
with a 0.4.2 patch in progress. 0.4.2 is published and holds the latest dist-tag,
its tag and GitHub release exist, and both bugs it tracked (#127, #128) are
closed. The active milestone now reads v0.5.0 with v0.4.x complete, What's Done
gains the v0.4.x release record it never had, and What's Next drops the request
to cut v0.4.0, which shipped on 2026-07-17.

No changeset: neither file is in the package `files` list, so nothing here ships
to npm.

Closes #134
Closes #136
@YANGCHUNHONG3000

Copy link
Copy Markdown
Contributor Author

@sepehr-safari Thanks for the thorough review!

I've pushed a fix commit addressing items 2–4:

  • ✅ Station ID: CS-SYNTHETIC-012CS-SYNTHETIC-016
  • ✅ Changeset added (minor bump for @ocpp-debugkit/toolkit)
  • ✅ README counts bumped: "15 predefined scenarios" → "16"

For item 1 (Prettier formatting), I noticed the new scenario file doesn't seem to have message arrays wrapped at 80 columns in my local view — could you point me to the specific lines that need pnpm format? Happy to push another commit once I know exactly what needs fixing.

Also, thanks for clarifying the claim workflow — noted for next time. I'd like to take #104 next once this merges.

@sepehr-safari

Copy link
Copy Markdown
Member

Thanks, that is items 2, 3 and 4 done. I checked each against the branch: the
changeset is a minor on the right package, CS-SYNTHETIC-016 is free and 012
is gone from the file, and both READMEs read 16. Lint, typecheck, build, the 520
tests and the external fixture test all pass locally on your head commit.

On item 1, here is why you cannot see it. The file is not Prettier-clean under
any print width. I checked 80, 100 and 120 and all three differ, so this is not
your editor disagreeing with the repo's printWidth: 100. Whatever the file was
checked with, it was not Prettier.

Three message arrays are hand-wrapped where Prettier wants them on one line, in
packages/toolkit/src/scenarios/__scenarios__/firmware-update-success.ts:

  • lines 46 to 51, the Downloaded notification
  • lines 61 to 66, the Installed notification
  • lines 81 to 85, the heartbeat response

Each collapses to a single line inside 100 characters. The fix is one command,
which rewrites in place using the repo's .prettierrc:

pnpm format

Rather than send you round a third time for one command, I ran it and pushed the
result to your branch, along with a merge of current main, since main had
moved ahead and the branch protection wants branches current before merge. Your
commits are untouched and the diff is otherwise exactly what you wrote. I hope
that is welcome rather than presumptuous; say so if you would rather push these
yourself in future and I will leave them to you.

For next time, the thing worth carrying over is to run pnpm format from the
repo rather than relying on an editor integration, because the repo script is
what CI runs. pnpm format:check then prints "All matched files use Prettier
code style!" and you know the first CI step will pass.

Worth a look at the CONTRIBUTING.md change that came in with that merge, since
it is the fix for the gap that tripped you: pnpm format:check is now part of
the documented local check, ordered the same as the CI job.

On what is next: #104 is this issue, the one this PR closes. I think you meant
#107, the --verbose flag, which is yours as soon as this merges. If you would
rather stay with scenarios, I opened three that are close to what you have just
done, each with the exact trace spelled out and a station ID reserved so there is
nothing to collide with: #137 heartbeat-timeout, #138 firmware-update-failure,
and #139 repeated-boot-notification. #138 in particular is the failing
counterpart to the scenario you wrote here. Say which one and it is yours.

@sepehr-safari
sepehr-safari merged commit 9e1a294 into ocpp-debugkit:main Jul 28, 2026
2 checks passed
sepehr-safari added a commit that referenced this pull request Jul 28, 2026
The changeset that came in with #133 was a `minor`, so the pending release PR
proposed `0.5.0`. That number is reserved for OCPP 2.0.1 in both `ROADMAP.md` and
`CURRENT_STATE.md`. Spending it on one new scenario would push the 2.0.1 work to
`0.6.0` and break the milestone-to-version mapping the project has held since
v0.1.

Retags the pending changeset to `patch`, so the release recomputes to `0.4.3`.
Nothing had published yet, so the change was free to make.

Adds the convention to the "Adding a Scenario" section, which did not mention
changesets at all. That omission is how the wrong size arrived: #133 was asked
for a minor during review, and #137, #138 and #139 are queued behind it. A new
scenario does add a public export, but scenarios are test corpus rather than
consumer API surface: nothing is built against an individual scenario constant,
and the registry is consumed as a whole through `scenarios` and `getScenario()`.
Minor and major stay reserved for changes to the analysis engine.

Closes #142
@sepehr-safari

sepehr-safari commented Jul 28, 2026

Copy link
Copy Markdown
Member

Merged, and it went out in @ocpp-debugkit/toolkit@0.4.3 a few minutes ago.
Thanks for sticking with it through two rounds of review.

#138, the firmware-update-failure scenario, is held for you. Leave a comment on
it and I will assign it: GitHub only lets me assign people who have commented on
that particular issue, which is why #104 could be assigned and this one needs the
extra step. Nobody else will be given it in the meantime.

It is the counterpart to the one you just wrote: same message type, the statuses the rule
does treat as failures. The rule counts exactly four of them, DownloadFailed,
DownloadPaused, InstallFailed and InstallRebootingFailed, and nothing in
the corpus covers any of them today.

Two things are different from last time, both meant to save you the round trips.
The issue spells out the whole trace, timestamps included, and I checked that
trace against the detection engine before writing it up, so it fires
FIRMWARE_UPDATE_FAILURE and nothing else. It also reserves CS-SYNTHETIC-019
for you, so there is no ID to go hunting for.

One correction from the guide since your PR: scenario additions take a patch
changeset, not minor. Scenarios are test corpus rather than API surface, and the
minor line is reserved for the analysis engine. That is written into
CONTRIBUTING now, and the issue says it too.

If you would rather do something other than another scenario, #107 is still open
and unclaimed. Just say so and I will switch it.

sepehr-safari added a commit that referenced this pull request Jul 28, 2026
…us work (#146)

The living document drifted through two paths that bypass a normal feature PR:
the release PR is created by the changesets action and never touches this file,
and issue-only work leaves no PR at all. #137 through #140, #144 and the 0.4.3
release all landed without a record here.

Records 0.4.3 (2026-07-28) and 0.3.2 (2026-07-14), the latter never logged at
all, and corrects the package status table, which had read 0.3.1 since the
v0.4.0 release.

Adds a section covering the external contribution pipeline: the second
good-first-issue completing, the patch-not-minor changeset decision (#142, #143),
the detection-rule coverage audit that found 3 of 16 rules with no scenario, the
three issues opened to close it, the retarget of #108 after its expectedFailures
turned out to be unsatisfiable, and the station ID allocation across in-flight
issues. It closes with the arithmetic to the v1.0 target of 20+ scenarios.

Corrects an overclaim from #135, which described PR #133 as the first external
contribution to a good-first-issue. It is the second: #105 carried the label and
Develop-KIM completed it in #114, shipped in 0.3.1.

Replaces "None currently. All design decisions resolved in ADRs" under pending
decisions, since #144 carries four open questions.

Closes #145
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants