fix(pii): redact third-party and document PII in remarks - #79
Merged
Conversation
Customer reported PII surviving anonymization, mainly in remarks. Replaying two captured responses (Amadeus PNR_Reply, Sabre GetReservationRS) through the shipped ruleset showed the passenger's own identity is handled correctly; two classes leak. Free-text-only PII. The phase-2 reference pass can only scrub values a phase-1 field rule already collected, so values with no structured counterpart are invisible to it. That is exactly the third-party population: emergency contact, authority-to-charge holder, travel arranger, and the Amadeus orderer. Fix by reaching each at its introducing site with a marker-anchored field rule, which also seeds the collector so the reference pass covers the remaining renderings. Sabre history mirrors. Every coded passenger remark is re-emitted as HistoryAssociationElement and AssociationChild/AssociationParent Content. Reference rules accept only an encrypt action, so date of birth, passport, nationality and card data (all one-way) can never reach those mirrors via the reference pass; their field rules now name the mirror paths explicitly. Also: first-time coverage of the Amadeus FP (card) and OS (contract number) elements, and person-linked pseudonymous identifiers (employee id, traveller profile id) are now treated as PII. Agency-agent names and sign-in codes stay plaintext by design, so the remark audit trail remains legible; organisation-level corporate, tour and discount codes stay too, since redacting them breaks fare reissue. Two ordering constraints are load-bearing and covered by a regression test: the arranger rules must precede the phone rules that overwrite the spans they read, and the card-expiry rule must precede the card-fragment rule. Fixes a latent round-trip bug found by replaying the real payload: sabre.res.received_from_phone encrypted a phone that is followed by "-<agent name>", and since "-" is a base64url character the greedy token scan re-consumed the suffix, so the value never de-anonymized and the relay would have forwarded a Wenrix token to the channel. Phones are one-way per the action policy, so it now uses a sentinel. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A customer reported PII surviving anonymization, "mainly in remarks". I replayed two captured
responses (Amadeus
PNR_Reply, SabreGetReservationRS) through the shipped baked ruleset with thereal engine. The passenger's own identity is handled correctly — structured name, phone, email,
DOCS/APIS entries, frequent-flyer number and card all scrubbed. Two classes still leaked, for two
different structural reasons.
1. PII that exists only in free text. The phase-2 reference pass can only scrub values a phase-1
field rule already collected, so a value with no structured counterpart is invisible to it. That is
exactly the third-party population: emergency contact, authority-to-charge holder, travel arranger,
and the Amadeus orderer. The clearest evidence was a single node that redacted to
EMER-<given name> <ENC_ token>— the passenger surname beside it was scrubbed by the referencerule, while the emergency contact's own given name survived because nothing had collected it.
2. Sabre history mirrors, for one-way types. Every coded passenger remark is re-emitted three
times —
RemarkLine/Text,HistoryAssociationElement, andAssociationChild/AssociationParentContent, with@codeturned into aP!/V!/E!text prefix.ReferenceRuleaccepts onlyEncryptAction, so date of birth, passport number, nationality and card data — all one-way by policy— can never be propagated into those mirrors by the reference pass.
Two further gaps were structural: no rule covered the Amadeus
FP(form of payment) orOS(otherservice information) elements at all, leaving a card number in cleartext; and person-linked
pseudonymous identifiers were unclassified.
What changed
24 new rules in
rules_fallback.json. No engine change, no schema change, no newPiiType.the value and seeds the collector, so the existing reference rules cover the remaining renderings
for free.
dob,gender,passport_id,nationality,payment) name the two historymirror paths explicitly, since phase 2 cannot carry a non-encrypt action.
01JAN00,00000000,31DEC99,ZZ,M) rather thanREDACTED, extending the existing typed-field sentinel policy soconsumers parsing the surrounding operational line keep working.
FPcard + expiry,OStraveller contract number.extract_patternsentry anchors on a literal operational marker observed in a payload. Nogeneric shape matching — an unanchored date or digit-run pattern would redact fares and flight
dates.
Scope decisions
arranger, orderer) and person-linked pseudonymous identifiers (employee id, traveller profile id,
traveller-attached loyalty/contract numbers, encrypted so clients can still correlate).
stays legible; and organisation-level corporate account, tour and discount codes, since redacting
those breaks fare reissue and policy display. Both are asserted by tests, not left to chance.
Two load-bearing ordering constraints
Rules mutate the tree in list order, so an extraction rule must precede any rule that overwrites the
span it reads. A regression test fails if either is reordered:
*_card_expirymust precede*_card_fragment, whose replacement destroys theXrun the expirypattern depends on.
Latent round-trip bug fixed
Replaying the real payload surfaced a pre-existing defect (present under the old ruleset too):
sabre.res.received_from_phoneencrypted a phone number that is followed directly by-<agent name>.ENC_payloads are base64url, so-reads as a token character, the greedy token scan re-consumedthe suffix, and the value never de-anonymized — meaning the relay would have forwarded a Wenrix
token to the channel, breaking transparency. Phones are one-way per the action policy, so it now
uses a sentinel, which contains no token characters. The spec delta records the general rule.
Testing
three Sabre history renderings, the
¤/¥delimiters preserved). Every token in them was diffedagainst the raw payloads to confirm nothing real carried over.
token-identity across renderings, the ordering regression, round-trip, and operational-survival
negatives.
along — the suite that owns it asserts values rather than counts, so it would otherwise have been
invisible.
just typesclean; pre-commit's ruff / ruff-format /mypy-strict / pylint / secret-detection all pass on the changed files.
zero
ENC_tokens left afterdeanonymize_request_body— i.e. the response still round-trips.just lintandjust fmt-checkare red, but identically so on pristinemaster— 40ruff checkviolations and 4 unformatted files, none of them in code this PR touches. Pre-existingand unrelated; this PR adds none.
The raw captured payloads were deleted and gitignored — they contained live passenger data and a live
Sabre
BinarySecurityToken, and were never committed.OpenSpec change
close-remark-free-text-pii-gapis included; sync + archive after merge.🤖 Generated with Claude Code