[fix] strip NUL bytes before tdengine history insert - #4310
Open
orangeCatDeveloper wants to merge 1 commit into
Open
[fix] strip NUL bytes before tdengine history insert#4310orangeCatDeveloper wants to merge 1 commit into
orangeCatDeveloper wants to merge 1 commit into
Conversation
Windows snmp interface descriptions carry NUL padding that made the hand-built INSERT fail with error 216 and lose the whole batch (apache#1481). Remove NUL from string values before escaping, and truncate the logical value before escaping so the cut cannot split an escape sequence.
orangeCatDeveloper
force-pushed
the
fix/1481-tdengine-control-chars
branch
from
August 12, 2026 05:06
1c6978c to
677547e
Compare
Aias00
approved these changes
Aug 17, 2026
Aias00
left a comment
Contributor
There was a problem hiding this comment.
Review: [fix] strip NUL bytes before tdengine history insert — APPROVED
Small, well-scoped fix for issue #1481 (snmp octet strings carrying NUL padding that broke the INSERT SQL).
What's correct
- Added
NUL_CHAR_PATTERN = Pattern.compile("\\u0000")and strip NUL bytes first informatStringValue. - Bonus correctness fix: the truncation is now done before escaping (
SQL_SPECIAL_STRING_PATTERN), with the comment explaining why — the old code truncated the already-escaped string, which could split an escape sequence (e.g. cut\'into\+ bare'). The new order avoids that class of bug entirely. replaceAll("")result is never null, so removing the!= nullguard is safe.- Test
testSaveDataStripsControlCharacterspins the contract: NUL stripped, tab (legitimate data) preserved, quote still escaped, and no raw/escaped NUL reaches the SQL — solid regression guard.
Non-blocking suggestions
- Only
\u0000is stripped. Other control chars (CR/LF, vertical tab) still flow into the SQL text. SNMP octet strings can carry those too; consider a broader[\p{Cntrl}&&[^\t]]strip if you want to be thorough, but tab-preservation is a deliberate choice here so this is optional. - Truncating the logical value before escaping means the post-escape length can slightly exceed
tableStrColumnDefineMaxLengthby the number of inserted backslashes. Acceptable tradeoff given the escape-split fix, just worth noting.
Verdict: APPROVED. Clean fix with a meaningful secondary improvement and a real test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's changed?
Fixes #1481: Windows SNMP interface descriptions carry NUL padding bytes (
\^@). The TDengine history storage builds its INSERT by hand, so the NUL reached the SQL text, TDengine rejected it with error 216, and the whole batch was lost — including the healthy rows.Fix in
TdEngineDataStorage.formatStringValue, the single choke point all string values, labels json and the instance tag pass through:\u0000before anything else (only NUL — tabs/newlines are legitimate data)\'in half, leaving a dangling backslash that escapes the closing quote and kills the batch the same wayOnly TDengine is affected: the other history backends use typed clients / JSON / line protocol instead of hand-built SQL.
Verified against a real TDengine 3.3.5.0 with a metric whose string value ends in
\u0000(one poisoned row, one healthy row per batch):Before — every batch rejected, both rows lost:
After — no errors, both rows stored, NUL stripped:
Checklist
Add or update API