Skip to content

[fix] strip NUL bytes before tdengine history insert - #4310

Open
orangeCatDeveloper wants to merge 1 commit into
apache:masterfrom
orangeCatDeveloper:fix/1481-tdengine-control-chars
Open

[fix] strip NUL bytes before tdengine history insert#4310
orangeCatDeveloper wants to merge 1 commit into
apache:masterfrom
orangeCatDeveloper:fix/1481-tdengine-control-chars

Conversation

@orangeCatDeveloper

@orangeCatDeveloper orangeCatDeveloper commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

What's changed?

Fixes #1481: Windows SNMP interface descriptions carry NUL padding bytes (\^@). The TDengine history storage builds its INSERT by hand, so the NUL reached the SQL text, TDengine rejected it with error 216, and the whole batch was lost — including the healthy rows.

Fix in TdEngineDataStorage.formatStringValue, the single choke point all string values, labels json and the instance tag pass through:

  • strip \u0000 before anything else (only NUL — tabs/newlines are legitimate data)
  • truncate the logical value before SQL-escaping: the old escape-then-truncate order could cut an escape pair like \' in half, leaving a dangling backslash that escapes the closing quote and kills the batch the same way

Only TDengine is affected: the other history backends use typed clients / JSON / line protocol instead of hand-built SQL.

Verified against a real TDengine 3.3.5.0 with a metric whose string value ends in \u0000 (one poisoned row, one healthy row per batch):

Before — every batch rejected, both rows lost:

ERROR TdEngineDataStorage - TDengine ERROR (216): sql: INSERT INTO `rows1481_rows_null_80_v2`
USING `rows1481_rows_super_v2` TAGS ('null:80') VALUES
(1786324976558, '{}', 1.0, 'Software Loopback Interface 1', 10.0)
(1786324976559, '{}', 2.0, 'normal-item', 20.0),
desc: syntax error near ''Software Loopback Interface 1' (invalid data or symbol)

taos> select * from hertzbeat.rows1481_rows_super_v2 limit 5;
Query OK, 0 row(s) in set

After — no errors, both rows stored, NUL stripped:

taos> select * from hertzbeat.rows1481_rows_super_v2 limit 5;
           ts            | seq |             name               | value |
==========================================================================
 2026-08-10 01:25:43.746 | 1.0 | Software Loopback Interface 1  | 10.0  |
 2026-08-10 01:25:43.747 | 2.0 | normal-item                    | 20.0  |
Query OK, 4 row(s) in set

Checklist

  • I have read the Contributing Guide
  • I have written the necessary doc or comment.
  • I have added the necessary unit tests and all cases have passed.

Add or update API

  • I have added the necessary e2e tests and all cases have passed.

Windows snmp interface descriptions carry NUL padding that made the
hand-built INSERT fail with error 216 and lose the whole batch (apache#1481).
Remove NUL from string values before escaping, and truncate the logical
value before escaping so the cut cannot split an escape sequence.
@orangeCatDeveloper
orangeCatDeveloper force-pushed the fix/1481-tdengine-control-chars branch from 1c6978c to 677547e Compare August 12, 2026 05:06

@Aias00 Aias00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: [fix] strip NUL bytes before tdengine history insert — APPROVED

Small, well-scoped fix for issue #1481 (snmp octet strings carrying NUL padding that broke the INSERT SQL).

What's correct

  • Added NUL_CHAR_PATTERN = Pattern.compile("\\u0000") and strip NUL bytes first in formatStringValue.
  • Bonus correctness fix: the truncation is now done before escaping (SQL_SPECIAL_STRING_PATTERN), with the comment explaining why — the old code truncated the already-escaped string, which could split an escape sequence (e.g. cut \' into \ + bare '). The new order avoids that class of bug entirely.
  • replaceAll("") result is never null, so removing the != null guard is safe.
  • Test testSaveDataStripsControlCharacters pins the contract: NUL stripped, tab (legitimate data) preserved, quote still escaped, and no raw/escaped NUL reaches the SQL — solid regression guard.

Non-blocking suggestions

  • Only \u0000 is stripped. Other control chars (CR/LF, vertical tab) still flow into the SQL text. SNMP octet strings can carry those too; consider a broader [\p{Cntrl}&&[^\t]] strip if you want to be thorough, but tab-preservation is a deliberate choice here so this is optional.
  • Truncating the logical value before escaping means the post-escape length can slightly exceed tableStrColumnDefineMaxLength by the number of inserted backslashes. Acceptable tradeoff given the escape-split fix, just worth noting.

Verdict: APPROVED. Clean fix with a meaningful secondary improvement and a real test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] <using tdengine 3.1.1.12 error>

2 participants