Skip to content

Add DuckDB partitioned Parquet cleaning and export recipe #495

Description

@JohnnyWilson16

Context

As outlined in the Contributor Roadmap, DuckDB is widely used for analytical pipelines querying partitioned Parquet datasets. Providing a standalone recipe makes it easy for developers to clean and export partitioned datasets.

Task

Add a runnable example script in examples/integrations/duckdb_parquet_export.py showing how to read a partitioned Parquet dataset, clean it using FreshData, and write the cleaned result back out using DuckDB.

Acceptance Criteria

  • Create examples/integrations/duckdb_parquet_export.py with standalone, executable code.
  • Demonstrates creating/loading a multi-partition Parquet directory, executing fd.clean(), and exporting back to DuckDB table or Parquet destination.
  • Document required dependency (pip install "freshdata-cleaner[duckdb]") in script docstring.
  • Register the new recipe in examples/README.md.
  • Script runs cleanly without errors via python examples/integrations/duckdb_parquet_export.py.

Pointers


New to the codebase? Comment here and we'll help you get set up! You can also check our First Contribution Guide.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions