Skip to content

V1.4.21 - #641

Merged
flarco merged 42 commits into
mainfrom
v1.4.21
Sep 15, 2025
Merged

V1.4.21#641
flarco merged 42 commits into
mainfrom
v1.4.21

Conversation

@flarco

@flarco flarco commented Sep 6, 2025 •
edited
Loading

Copy link
Copy Markdown
Collaborator

New Features & Enhancements:

  1. MongoDB ObjectID Support - Added proper ObjectID filtering capabilities for MongoDB connections with automatic hex string conversion
  2. ClickHouse Improvements - Enhanced merge operations with retry logic for "Too many unfinished mutations" errors and lightweight delete support for v23+
  3. Prometheus Enhancements - Better transform support and column type handling for Prometheus data sources
  4. API Spec Fetching - New FetchSpec function for loading API specifications
  5. Databricks Volume Management - Improved volume path handling and creation logic

Bug Fixes:

  1. SQL Server Money Type - Fixed precision issues with MONEY and SMALLMONEY data types when replicating to StarRocks
  2. Decimal Precision Handling - Improved decimal column typing with better precision and scale calculations
  3. Snowflake Stage Management - Better internal stage creation and validation
  4. Redshift Credentials - Enhanced AWS credential handling for COPY operations including support for IAM roles
  5. PostgreSQL Timeout Support - Added statement_timeout parameter support

flarco added 30 commits September 4, 2025 07:50
@flarco
feat: add MongoDB ObjectID filtering support and corresponding test
0fe9bb0
@flarco
feat: refactor column retrieval and validation in CopyViaStage method
5561e9f
@flarco
feat: initialize sample size from environment variable in SetSampleSi…
9308044 
…ze function
@flarco
feat: add type checking for new columns in parseRecords method
af590f2
@flarco
feat: add JSON to Snowflake nested import test and corresponding YAML…
f6a6836 
… configuration
@flarco
feat: enhance DropTable method to attempt drop after purge failure
25329ea
@flarco
feat: improve error message for unexpected mapping structure in GetTa…
e476203 
…bleColumns method
@flarco
feat: update GetTableColumns and StreamRowsContext methods to use 'an…
a99b76f 
…y' type for improved type handling
@flarco
refactor: move evaluator initialization outside of test loop in TestE…
ca5e40b 
…valuatorCheckExpression
@flarco
feat: add version handling to ClickhouseConn and update connection logic
84f4da8
@flarco
fix: correct SQL delete statement syntax in GenerateMergeSQL method
bbb42d3
@flarco
fix: update internal stage handling to require fully qualified stage …
4f9e864 
…name format
@flarco
fix: clarify warning message for fully qualified stage name requirement
ede70dc
@flarco
feat: add Merge method to ClickhouseConn for handling upserts with re…
729968d 
…try logic
@flarco
fix: update error messages to use 'merge' terminology in Merge functions
955f244
@flarco
fix: update LoadAPISpec to use specIdentifier and add FetchSpec for f…
59484e2 
…etching API specs
@flarco
fix: enhance GenerateMergeConfig to handle primary key scenarios in s…
9669a03 
…etFields
@flarco
test: add validation check for email hash length in staged transforms
ff5e541
@flarco
fix: update Prometheus YAML configurations to ensure proper date form…
544c74d 
…atting and SQL column definitions
@flarco
fix: update error messages in ErrorHelper for clarity and improved do…
840e59c 
…cumentation links
@flarco
Revert prometheus changes
18cfd57
@flarco
fix: update transform field names to use PascalCase for consistency
97875b0
@flarco
feat: enhance Prometheus data handling with transform support and con…
3d85691 
…figuration updates
@flarco
fix: add fields parameter to SQL queries in replication and task exec…
bd6ef47 
…ution
@flarco
fix: add statement_timeout support in connection URL and enhance erro…
f48dd98 
…r messages for Postgres timeout
@flarco
feat: add create_schema template to Databricks YAML configuration
26f442b
@flarco
fix: update internal volume handling to store and retrieve volume pat…
10b1994 
…hs correctly
@flarco
feat: add test for SQL Server money type to StarRocks replication
295232a
@flarco
fix: add strict parameter to StreamLoad for enhanced data validation
5809100
@flarco
fix: enforce minimum precision of 15 for SQL column types to address …
c82e1b7 
…precision issues
flarco added 12 commits September 13, 2025 14:14
@flarco
feat: enhance SQL replication tests by increasing decimal precision a…
4d74d0c 
…nd adding checks for SQL target rows
@flarco
fix: update jinzhu/copier dependency to remove indirect requirement
3e74e24
@flarco
feat: enhance Redshift credential handling by consolidating AWS crede…
92e0fe0 
…ntials into a single method and updating YAML templates
@flarco
fix: update minimum precision check for SQL columns to use configurab…
88db8a5 
…le environment variable
@flarco
fix: remove redundant streams entry in MSSQL test configuration
09aae46
@flarco
feat: add YellowString function for colored output in terminal
42e5fdc
@flarco
feat: update Prometheus buffer configuration to use parquet format an…
a7d7470 
…d enhance transform handling
@flarco
feat: add Casted method to Transform interface and enhance type handl…
3566a1b 
…ing in stream processor
@flarco
fix: ensure col.Sourced is set to false when conditions are not met i…
ba10bb3 
…n SQLColumns function
@flarco
fix: remove redundant Stringer case in CastVal and update CastToStrin…
49a7183 
…gE for model.Sample handling
@flarco
fix: update precision and scale handling for DecimalType across datab…
8b7c91a 
…ase and arrow modules
@flarco
fix: enhance decimal precision and scale handling in ColumnTyping and…
0ef315f 
… SQLColumns, add tests for ColumnTyping
@flarco
flarco merged commit d9d39a3 into main Sep 15, 2025
9 checks passed
@flarco
flarco deleted the v1.4.21 branch September 15, 2025 10:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@flarco