Currently the iceberg is updated by kafka connect.
Optimizing job failed due to below error
Caused by: org.apache.iceberg.exceptions.ValidationException: Cannot commit, found new position delete for replaced data file:
Is there any way to fix it?
I also have already added below table properties:
self-optimizing.filte = created_at < '2025-01-01T00:00:00Z'
self-optimizing.allow-partial-commit =true
'self-optimizing.allow-partial-commit' = 'true',
'self-optimizing.max-file-count' = '3000',
'self-optimizing.max-task-size-bytes' = '67108864',
'self-optimizing.minor.trigger.file-count' = '8',
'self-optimizing.minor.trigger.interval' = '900000',
'self-optimizing.major.trigger.duplicate-ratio' = '0.2',
'self-optimizing.target-size' = '134217728',
'self-optimizing.fragment-ratio' = '8',
'self-optimizing.min-target-size-ratio' = '0.75',
'self-optimizing.min-plan-interval' = '60000'
Currently the iceberg is updated by kafka connect.
Optimizing job failed due to below error
Caused by: org.apache.iceberg.exceptions.ValidationException: Cannot commit, found new position delete for replaced data file:
Is there any way to fix it?
I also have already added below table properties:
self-optimizing.filte = created_at < '2025-01-01T00:00:00Z'
self-optimizing.allow-partial-commit =true
'self-optimizing.allow-partial-commit' = 'true',
'self-optimizing.max-file-count' = '3000',
'self-optimizing.max-task-size-bytes' = '67108864',
'self-optimizing.minor.trigger.file-count' = '8',
'self-optimizing.minor.trigger.interval' = '900000',
'self-optimizing.major.trigger.duplicate-ratio' = '0.2',
'self-optimizing.target-size' = '134217728',
'self-optimizing.fragment-ratio' = '8',
'self-optimizing.min-target-size-ratio' = '0.75',
'self-optimizing.min-plan-interval' = '60000'