GH-50830: [C++][Parquet] Use JsonWriter for LogicalType::ToJSON() - #50877
GH-50830: [C++][Parquet] Use JsonWriter for LogicalType::ToJSON()#50877NathanChung4 wants to merge 2 commits into
Conversation
I changed the 8 ToJSON() functions so that they now use JsonWriter instead of the old hand build strings. I deleted WriteCrsKeyAndValue since it was no longer needed, and I updated the test files to match the changes made in types.cc
There was a problem hiding this comment.
Pull request overview
Refactors Parquet C++ LogicalType::ToJSON() implementations to consistently use ::arrow::json::JsonWriter rather than manual string/stream construction, resulting in compact JSON output and reduced risk of producing invalid JSON for unescaped values.
Changes:
- Replaced hand-built JSON in 8
LogicalType::Impl::*::ToJSON()implementations withJsonWriter-based construction. - Removed the now-unneeded
WriteCrsKeyAndValuehelper and simplified CRS emission toStringField. - Updated Parquet schema and reader tests to match the new compact JSON formatting.
Reviewed changes
Copilot reviewed 1 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| cpp/src/parquet/types.cc | Converts LogicalType::ToJSON() methods to JsonWriter and removes manual/stream JSON helper code. |
| cpp/src/parquet/schema_test.cc | Updates expected ToJSON() strings for logical types to match compact JSON output. |
| cpp/src/parquet/reader_test.cc | Updates expected JSON substrings containing embedded logical-type JSON to match compact formatting. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| writer.StringField("Type", "Decimal"); | ||
| writer.Key("precision"); | ||
| writer.Int(precision_); | ||
| writer.Key("scale"); |
There was a problem hiding this comment.
Could we add an IntField() helper to JsonWriter and use it here, similar to StringField() and BoolField()? It would make these conversions a bit cleaner.
There was a problem hiding this comment.
Hey Reranko05, thank you for the suggestion and I totally agree with you. I have committed the proposed change, please let me know if you want to change anything or anything
Added IntField() to JsonWriter per Reranko05's PR review suggestion. Updated the 3 different functions: Decimal, Int, and Variant that had the Key() + Int() two call pattern
|
@kou Can you review this when you have time? |
| { "Id": "0", "Name": "id", "PhysicalType": "INT32", "ConvertedType": "NONE", "LogicalType": {"Type": "None"} }, | ||
| { "Id": "1", "Name": "bool_col", "PhysicalType": "BOOLEAN", "ConvertedType": "NONE", "LogicalType": {"Type": "None"} }, | ||
| { "Id": "2", "Name": "tinyint_col", "PhysicalType": "INT32", "ConvertedType": "NONE", "LogicalType": {"Type": "None"} }, | ||
| { "Id": "3", "Name": "smallint_col", "PhysicalType": "INT32", "ConvertedType": "NONE", "LogicalType": {"Type": "None"} }, | ||
| { "Id": "4", "Name": "int_col", "PhysicalType": "INT32", "ConvertedType": "NONE", "LogicalType": {"Type": "None"} }, | ||
| { "Id": "5", "Name": "bigint_col", "PhysicalType": "INT64", "ConvertedType": "NONE", "LogicalType": {"Type": "None"} }, | ||
| { "Id": "6", "Name": "float_col", "PhysicalType": "FLOAT", "ConvertedType": "NONE", "LogicalType": {"Type": "None"} }, | ||
| { "Id": "7", "Name": "double_col", "PhysicalType": "DOUBLE", "ConvertedType": "NONE", "LogicalType": {"Type": "None"} }, | ||
| { "Id": "8", "Name": "date_string_col", "PhysicalType": "BYTE_ARRAY", "ConvertedType": "NONE", "LogicalType": {"Type": "None"} }, | ||
| { "Id": "9", "Name": "string_col", "PhysicalType": "BYTE_ARRAY", "ConvertedType": "NONE", "LogicalType": {"Type": "None"} }, | ||
| { "Id": "10", "Name": "timestamp_col", "PhysicalType": "INT96", "ConvertedType": "NONE", "LogicalType": {"Type": "None"} } | ||
| { "Id": "0", "Name": "id", "PhysicalType": "INT32", "ConvertedType": "NONE", "LogicalType": {"Type":"None"} }, | ||
| { "Id": "1", "Name": "bool_col", "PhysicalType": "BOOLEAN", "ConvertedType": "NONE", "LogicalType": {"Type":"None"} }, | ||
| { "Id": "2", "Name": "tinyint_col", "PhysicalType": "INT32", "ConvertedType": "NONE", "LogicalType": {"Type":"None"} }, | ||
| { "Id": "3", "Name": "smallint_col", "PhysicalType": "INT32", "ConvertedType": "NONE", "LogicalType": {"Type":"None"} }, | ||
| { "Id": "4", "Name": "int_col", "PhysicalType": "INT32", "ConvertedType": "NONE", "LogicalType": {"Type":"None"} }, | ||
| { "Id": "5", "Name": "bigint_col", "PhysicalType": "INT64", "ConvertedType": "NONE", "LogicalType": {"Type":"None"} }, | ||
| { "Id": "6", "Name": "float_col", "PhysicalType": "FLOAT", "ConvertedType": "NONE", "LogicalType": {"Type":"None"} }, | ||
| { "Id": "7", "Name": "double_col", "PhysicalType": "DOUBLE", "ConvertedType": "NONE", "LogicalType": {"Type":"None"} }, | ||
| { "Id": "8", "Name": "date_string_col", "PhysicalType": "BYTE_ARRAY", "ConvertedType": "NONE", "LogicalType": {"Type":"None"} }, | ||
| { "Id": "9", "Name": "string_col", "PhysicalType": "BYTE_ARRAY", "ConvertedType": "NONE", "LogicalType": {"Type":"None"} }, | ||
| { "Id": "10", "Name": "timestamp_col", "PhysicalType": "INT96", "ConvertedType": "NONE", "LogicalType": {"Type":"None"} } |
There was a problem hiding this comment.
Could you use JsonWriter in
arrow/cpp/src/parquet/printer.cc
Lines 254 to 435 in f64e90a
It seems that we can use https://github.com/simdjson/simdjson/blob/master/doc/builder.md#pretty-formatted-fractured-json for it.
Rationale for this change
This change was made because it refactored ToJSON() to build JSON through a proper writer instead of hand-built strings, which removes a correctness risk. It also aligns the codebase's ongoing transition away from manual JSON construction.
What changes are included in this PR?
This PR focused on changing the 8 ToJSON() functions from hand built strings to using JsonWriter. In addition, the test files (schema_test.cc, reader_test.cc) were also updated to reflect these changes. Finally, WriteCrsKeyAndValue were no longer needed so it was deleted.
Are these changes tested?
These changes are tested and verified. There was a parquet-schema-test in which 41/41 testcases passed. There was a parquet-reader-test in which 149/154 passed and 5 were skipped as they were unrelated. A Full ctest -R "^parquet-": 11/11 suites passed. And finally the pre-commit (C++ Format + C++ Lint) came out clean.
Are there any user-facing changes?
Yes, there are user-facing changes. ToJSON() now produces compact JSON instead of spaced JSON. It's the same valid JSON, but different exact bytes:
AI Disclosure
Per the AI-generated code guidance: the 8 ToJSON() transitions to JsonWriter, test file changes, the merge conflict resolutions, and the test were produced with Claude Code, and reviewed and verified by me. Correctness was checked by rebuilding and running the relevant test after every single function conversion, not just once at the end. In addition, catching two mistakes early via failing tests (the Decimal attempt that dropped fields, the Int attempt that did the same).
JsonWriterconsistently inLogicalType::ToJSON()#50830