Skip to content

no-cdk-api-above-floor cannot see test files or banned imports, so floor violations in tests surface only in the CI floor matrix #408

Description

@laazyj

Problem / use case

cdk-floors:enforce runs each package's whole suite — typecheck included — against its declared floor, so test files are bound by the floor exactly as src/ is. The cloudformation floor entry says as much:

Src itself only needs the root entry (2.0.0), but the enforce model requires the suite to run, and aws-cdk-lib/assertions is table stakes for builder tests.

But composurecdk/no-cdk-api-above-floor — the rule that exists to catch precisely this — cannot fire on a test file, for two independent reasons. So a floor violation in a test is invisible locally (npm run verify is green) and surfaces ~10 minutes into a 12-job CI matrix as a tsc error in an unrelated-looking job.

This just happened on #352. Two assertions in packages/cloudformation/test/template-text-policy.test.ts reached above that package's 2.1.0 floor:

API Available from Rule could catch it?
Match.stringLikeRegexp 2.9.0 Only if it ran on tests and had a FORBIDDEN entry
Annotations from aws-cdk-lib/assertions absent at 2.1.0 entirely No — it's a named import, not a member access

Gap 1 — scope

eslint.config.mjs applies composurecdk.configs.recommended to packages/*/src/**/*.ts only:

{
  files: ["packages/*/src/**/*.ts"],
  plugins: { composurecdk },
  rules: composurecdk.configs.recommended.rules,
},

Widening that glob is not the fix: recommended also carries builder-must-be-tagged, builder-must-implement-copy-state, lifecycle-build-context-required and friends, which are written for library source and would misfire on tests. recommended.ts says so explicitly. The fix is a separate config block enabling only no-cdk-api-above-floor on packages/*/test/**/*.ts.

Gap 2 — it only checks member accesses

no-cdk-api-above-floor.ts has a single MemberExpression visitor. rootIsCdkImport already handles aws-cdk-lib/ subpaths correctly, so Match.stringLikeRegexp would be caught once a FORBIDDEN entry existed — but a banned symbol (import { Annotations } from "aws-cdk-lib/assertions") is structurally invisible. That needs an ImportDeclaration visitor and a second ban list keyed by (module, specifier).

Gap 3 — the ban list is global, and it cannot be

This is the part that makes the enhancement more than a glob change. FORBIDDEN is a single flat list applied identically to every package, but floors differ by an order of magnitude across the graph (2.1.0 → 2.216.0). A global entry for Match.stringLikeRegexp would fire on 10 packages that legitimately use it, every one of which sits above 2.9.0:

Package Floor Uses Match.stringLikeRegexp in tests
cloudformation 2.1.0 ← the only actual violation
apigateway, budgets, cloudwatch, events, sqs 2.93.0 legal
acm 2.119.0 legal
cloudfront 2.124.0 legal
s3 2.123.0 legal
ec2 2.140.0 legal

addWarningV2 (2.93.0) has the same shape: illegal in cloudformation, legal in the six packages that already call it.

So the rule has to compare a banned API's since against the floor of the package the file lives in, read from cdk-floors.json — which is already the single source of truth for floors and already validated by cdk-floors:check. Today since is documentation only; nothing compares it to anything.

Proposed solution

Make no-cdk-api-above-floor floor-aware, then point it at tests:

  1. Resolve the current package's floor from the filename. Map packages/<name>/...cdk-floors.json.floors[<name>].floor. Load and cache once per lint run.
  2. Compare rather than blanket-ban. Report only when semver.lt(packageFloor, entry.since). This retires the current all-or-nothing list and lets entries be added freely — an entry becomes a fact about aws-cdk-lib, not a policy decision about every package. The existing isCfn* entry keeps working unchanged (2.231.0 is above every floor in the repo).
  3. Add an ImportDeclaration visitor with a (module, specifier, since) ban list, so Annotations from aws-cdk-lib/assertions is catchable.
  4. Enable the one rule on packages/*/test/**/*.ts as its own config block, leaving recommended's src-only scope alone.
  5. Seed the list with what has already bitten: Annotations (aws-cdk-lib/assertions, absent at 2.1.0 — needs an exact introducing version establishing), Match.stringLikeRegexp (2.9.0), Annotations.addWarningV2 (2.93.0).

Step 2 is the load-bearing one. Steps 1 and 2 also make the rule useful in src/, where it has the same latent problem: the FORBIDDEN list can only ever hold APIs above the highest floor in the repo, which is why it has exactly one entry today.

Alternatives considered

  • Exclude tests from the floor typecheck. Makes the symptom vanish and guts the guard — the enforce model deliberately runs the suite because that is what proves a package works at its floor rather than merely compiling. Rejected.
  • Raise the floor of whichever package trips. What feat(cloudformation): add templateTextPolicy, an opt-in guard for template text #352's failure would have "fixed". Wrong lever for cloudformation especially: it is the base package every other one peer-depends on, so raising it drags the whole library up. Rejected.
  • Run the full floor matrix in npm run verify. Correct but unaffordable: each leg is an npm ci against a pinned aws-cdk-lib, ~45s per floor × 12 floors, and it rewrites node_modules under the developer. Lint is the right layer for a static fact.
  • A per-package FORBIDDEN override in each package's eslint config. Duplicates floor data that already lives in cdk-floors.json and would drift from it. Rejected in favour of reading the manifest.
  • Status quo — let CI catch it. Tenable, and it does work. The cost is a slow, badly-located signal: the failure names a tsc error in one of a dozen Enforce aws-cdk-lib@X jobs, and the fix is often "rewrite the assertion", which is much cheaper to learn before pushing.

Related: #377 (cdk-floors check does not enforce floor monotonicity across the peer graph) — a different gap in the same guard, also about cdk-floors.json being under-used as a source of truth.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions