Skip to content

fix(pii): redact third-party and document PII in remarks - #79

Merged
armona merged 1 commit into
masterfrom
fix/remark-free-text-pii-gap
Aug 12, 2026
Merged

fix(pii): redact third-party and document PII in remarks#79
armona merged 1 commit into
masterfrom
fix/remark-free-text-pii-gap

Conversation

@armona

@armona armona commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Why

A customer reported PII surviving anonymization, "mainly in remarks". I replayed two captured
responses (Amadeus PNR_Reply, Sabre GetReservationRS) through the shipped baked ruleset with the
real engine. The passenger's own identity is handled correctly — structured name, phone, email,
DOCS/APIS entries, frequent-flyer number and card all scrubbed. Two classes still leaked, for two
different structural reasons.

1. PII that exists only in free text. The phase-2 reference pass can only scrub values a phase-1
field rule already collected, so a value with no structured counterpart is invisible to it. That is
exactly the third-party population: emergency contact, authority-to-charge holder, travel arranger,
and the Amadeus orderer. The clearest evidence was a single node that redacted to
EMER-<given name> <ENC_ token> — the passenger surname beside it was scrubbed by the reference
rule, while the emergency contact's own given name survived because nothing had collected it.

2. Sabre history mirrors, for one-way types. Every coded passenger remark is re-emitted three
times — RemarkLine/Text, HistoryAssociationElement, and AssociationChild/AssociationParent
Content, with @code turned into a P!/V!/E! text prefix. ReferenceRule accepts only
EncryptAction, so date of birth, passport number, nationality and card data — all one-way by policy
— can never be propagated into those mirrors by the reference pass.

Two further gaps were structural: no rule covered the Amadeus FP (form of payment) or OS (other
service information) elements at all, leaving a card number in cleartext; and person-linked
pseudonymous identifiers were unclassified.

What changed

24 new rules in rules_fallback.json. No engine change, no schema change, no new PiiType.

  • Names are reached by a marker-anchored field rule at their introducing site. That both redacts
    the value and seeds the collector, so the existing reference rules cover the remaining renderings
    for free.
  • One-way types (dob, gender, passport_id, nationality, payment) name the two history
    mirror paths explicitly, since phase 2 cannot carry a non-encrypt action.
  • Identity-document data in free text uses format-preserving sentinels (01JAN00, 00000000,
    31DEC99, ZZ, M) rather than REDACTED, extending the existing typed-field sentinel policy so
    consumers parsing the surrounding operational line keep working.
  • New Amadeus paths: FP card + expiry, OS traveller contract number.
  • Every extract_patterns entry anchors on a literal operational marker observed in a payload. No
    generic shape matching — an unanchored date or digit-run pattern would redact fares and flight
    dates.

Scope decisions

  • In scope: traveller-adjacent third parties (emergency contact, authority-to-charge, travel
    arranger, orderer) and person-linked pseudonymous identifiers (employee id, traveller profile id,
    traveller-attached loyalty/contract numbers, encrypted so clients can still correlate).
  • Deliberately preserved: agency-agent personal names and sign-in codes, so the remark audit trail
    stays legible; and organisation-level corporate account, tour and discount codes, since redacting
    those breaks fare reissue and policy display. Both are asserted by tests, not left to chance.

Two load-bearing ordering constraints

Rules mutate the tree in list order, so an extraction rule must precede any rule that overwrites the
span it reads. A regression test fails if either is reordered:

  • the arranger rules must precede the phone rules that mask those nodes;
  • *_card_expiry must precede *_card_fragment, whose replacement destroys the X run the expiry
    pattern depends on.

Latent round-trip bug fixed

Replaying the real payload surfaced a pre-existing defect (present under the old ruleset too):
sabre.res.received_from_phone encrypted a phone number that is followed directly by -<agent name>.
ENC_ payloads are base64url, so - reads as a token character, the greedy token scan re-consumed
the suffix, and the value never de-anonymized — meaning the relay would have forwarded a Wenrix
token to the channel
, breaking transparency. Phones are one-way per the action policy, so it now
uses a sentinel, which contains no token characters. The spec delta records the general rule.

Testing

  • Two new sanitized fixtures, built value-for-value from the real captures (both remark mirrors, all
    three Sabre history renderings, the ¤/¥ delimiters preserved). Every token in them was diffed
    against the raw payloads to confirm nothing real carried over.
  • 17 new golden tests plus 2 relay-level integration cases: sentinels in all three renderings,
    token-identity across renderings, the ordering regression, round-trip, and operational-survival
    negatives.
  • Locked one new catch on a pre-existing fixture that had carried a plaintext traveller profile id all
    along — the suite that owns it asserts values rather than counts, so it would otherwise have been
    invisible.
  • 692 tests pass; coverage 95.8% (gate 85%); just types clean; pre-commit's ruff / ruff-format /
    mypy-strict / pylint / secret-detection all pass on the changed files.
  • Acceptance on the real payloads: every leaked value gone, all operational text byte-identical, and
    zero ENC_ tokens left after deanonymize_request_body — i.e. the response still round-trips.

⚠️ just lint and just fmt-check are red, but identically so on pristine master — 40
ruff check violations and 4 unformatted files, none of them in code this PR touches. Pre-existing
and unrelated; this PR adds none.

The raw captured payloads were deleted and gitignored — they contained live passenger data and a live
Sabre BinarySecurityToken, and were never committed.

OpenSpec change close-remark-free-text-pii-gap is included; sync + archive after merge.

🤖 Generated with Claude Code

Customer reported PII surviving anonymization, mainly in remarks. Replaying two
captured responses (Amadeus PNR_Reply, Sabre GetReservationRS) through the shipped
ruleset showed the passenger's own identity is handled correctly; two classes leak.

Free-text-only PII. The phase-2 reference pass can only scrub values a phase-1 field
rule already collected, so values with no structured counterpart are invisible to it.
That is exactly the third-party population: emergency contact, authority-to-charge
holder, travel arranger, and the Amadeus orderer. Fix by reaching each at its
introducing site with a marker-anchored field rule, which also seeds the collector so
the reference pass covers the remaining renderings.

Sabre history mirrors. Every coded passenger remark is re-emitted as
HistoryAssociationElement and AssociationChild/AssociationParent Content. Reference
rules accept only an encrypt action, so date of birth, passport, nationality and card
data (all one-way) can never reach those mirrors via the reference pass; their field
rules now name the mirror paths explicitly.

Also: first-time coverage of the Amadeus FP (card) and OS (contract number) elements,
and person-linked pseudonymous identifiers (employee id, traveller profile id) are now
treated as PII. Agency-agent names and sign-in codes stay plaintext by design, so the
remark audit trail remains legible; organisation-level corporate, tour and discount
codes stay too, since redacting them breaks fare reissue.

Two ordering constraints are load-bearing and covered by a regression test: the
arranger rules must precede the phone rules that overwrite the spans they read, and
the card-expiry rule must precede the card-fragment rule.

Fixes a latent round-trip bug found by replaying the real payload:
sabre.res.received_from_phone encrypted a phone that is followed by "-<agent name>",
and since "-" is a base64url character the greedy token scan re-consumed the suffix,
so the value never de-anonymized and the relay would have forwarded a Wenrix token to
the channel. Phones are one-way per the action policy, so it now uses a sentinel.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@armona
armona merged commit 2585d59 into master Aug 12, 2026
11 of 12 checks passed
@armona
armona deleted the fix/remark-free-text-pii-gap branch August 12, 2026 20:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant