diff --git a/_freeze/index/execute-results/html.json b/_freeze/index/execute-results/html.json index 2a495cd44..4b94b2cdb 100644 --- a/_freeze/index/execute-results/html.json +++ b/_freeze/index/execute-results/html.json @@ -1,10 +1,10 @@ { - "hash": "b7a72b381238301376a19441bb7d071e", + "hash": "36e95974dbc94a4c655f2b4b9cdebd2f", "result": { "engine": "jupyter", - "markdown": "---\ntitle: \"\"\ntoc: false\nbody-classes: \"gd-homepage\"\n---\n\n\n\n```{=html}\n
\n\"Logo\"\n

Pointblank

\n

Find out if your data is what you think it is.

\n
\n\n```\n\n::: {.column-margin}\n#### Links\n\nView on PyPI
\n\n#### AI / Agents\n\nSkills
\n[llms.txt](llms.txt)
\n[llms-full.txt](llms-full.txt)
\n\n#### Developers\n\n

Richard Iannone

Maintainer

Posit, PBC
\n

Posit Software, PBC

Copyright holder, funder
\n\n#### Community\n\n[Contributing guide](contributing.qmd)
\n[Code of conduct](code-of-conduct.qmd)
\n[Project roadmap](roadmap.qmd)
\n[Security policy](security.qmd)
\nFull license MIT
\n[Citing pointblank](citation.qmd)
\n\n#### Meta\n\n**Requires:** Python `>=3.10`
\n**Provides-Extra:** `pd`, `pl`, `pyspark`, `generate`, `mcp`, `otel`, `excel`, `cdisc`, `cdisc-core`, `bigquery`, `databricks`, `duckdb`, `mysql`, `mssql`, `postgres`, `snowflake`, `sqlite`, `docs`
\n[Package Info](package-info.html)\n:::\n\nPointblank is a data validation framework for Python that makes data quality checks beautiful,\npowerful, and stakeholder-friendly. Instead of cryptic error messages, get stunning interactive\nreports that turn data issues into conversations.\n\nHere's what a validation looks like (click \"Show the code\" to see how it's done):\n\n::: {#cdd1005c .cell execution_count=1}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"Show the code\"}\nimport pointblank as pb\nimport polars as pl\n\nvalidation = (\n pb.Validate(\n data=pb.load_dataset(dataset=\"game_revenue\", tbl_type=\"polars\"),\n tbl_name=\"game_revenue\",\n label=\"Comprehensive validation of game revenue data\",\n thresholds=pb.Thresholds(warning=0.10, error=0.25, critical=0.35),\n brief=True\n )\n .col_vals_regex(columns=\"player_id\", pattern=r\"^[A-Z]{12}[0-9]{3}$\")\n .col_vals_gt(columns=\"session_duration\", value=20)\n .col_vals_ge(columns=\"item_revenue\", value=0.20)\n .col_vals_in_set(columns=\"item_type\", set=[\"iap\", \"ad\"])\n .col_vals_in_set(\n columns=\"acquisition\",\n set=[\"google\", \"facebook\", \"organic\", \"crosspromo\", \"other_campaign\"]\n )\n .col_vals_not_in_set(columns=\"country\", set=[\"Mongolia\", \"Germany\"])\n .col_vals_between(\n columns=\"session_duration\",\n left=10, right=50,\n pre=lambda df: df.select(pl.median(\"session_duration\")),\n brief=\"Expect that the median of `session_duration` should be between `10` and `50`.\"\n )\n .rows_distinct(columns_subset=[\"player_id\", \"session_id\", \"time\"])\n .row_count_match(count=2000)\n .col_count_match(count=11)\n .col_vals_not_null(columns=\"item_type\")\n .col_exists(columns=\"start_day\")\n .interrogate()\n)\n\nvalidation.get_tabular_report(title=\"Game Revenue Validation Report\")\n```\n\n::: {.cell-output .cell-output-display execution_count=1}\n```{=html}\n
\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n

Game Revenue Validation Report

\n
Comprehensive validation of game revenue data
Polarsgame_revenueWARNING0.1ERROR0.25CRITICAL0.35
STEPCOLUMNSVALUESTBLEVALUNITSPASSFAILWECEXT
#4CA64C1\n
\n \n\n col_vals_regex\n \n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_regex()
\n
\n

Expect that values in player_id should match the regular expression: ^[A-Z]{12}[0-9]{3}$.

\n
\n 		player_id^[A-Z]{12}[0-9]{3}$\n \n \n \n \n \n \n \n✓20002000
1.00		0
0.00		○○○—
#EBBC142\n
\n \n\n col_vals_gt\n \n \n \n \n \n \n\n
\n
\n
col_vals_gt()
\n
\n

Expect that values in session_duration should be > 20.

\n
\n 		session_duration20\n \n \n \n \n \n \n \n✓20001418
0.71		582
0.29		●●○
#FF33003\n
\n \n\n col_vals_ge\n \n \n \n \n \n \n\n
\n
\n
col_vals_ge()
\n
\n

Expect that values in item_revenue should be >= 0.2.

\n
\n 		item_revenue0.2\n \n \n \n \n \n \n \n✓20001192
0.60		808
0.40		●●●
#4CA64C4\n
\n \n\n col_vals_in_set\n \n \n \n \n \n \n\n
\n
\n
col_vals_in_set()
\n
\n

Expect that values in item_type should be in the set of iap, ad.

\n
\n 		item_typeiap, ad\n \n \n \n \n \n \n \n✓20002000
1.00		0
0.00		○○○—
#4CA64C665\n
\n \n\n col_vals_in_set\n \n \n \n \n \n \n\n
\n
\n
col_vals_in_set()
\n
\n

Expect that values in acquisition should be in the set of google, facebook, organic, and 2 more.

\n
\n 		acquisitiongoogle, facebook, organic, crosspromo, other_campaign\n \n \n \n \n \n \n \n✓20001975
0.99		25
0.01		○○○
#AAAAAA6\n
\n \n\n col_vals_not_in_set\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_in_set()
\n
\n

Expect that values in country should not be in the set of Mongolia, Germany.

\n
\n 		countryMongolia, Germany\n \n \n \n \n \n \n \n✓20001775
0.89		225
0.11		●○○
#4CA64C7\n
\n \n\n col_vals_between\n \n \n \n \n \n \n\n
\n
\n
col_vals_between()
\n
\n

Expect that the median of session_duration should be between 10 and 50.

\n
\n 		session_duration[10, 50]\n \n \n \n \n \n \n \n \n \n✓11
1.00		0
0.00		○○○—
#4CA64C668\n
\n \n\n rows_distinct\n \n \n \n \n \n \n \n \n \n \n\n
\n
\n
rows_distinct()
\n
\n

Expect entirely distinct rows across player_id, session_id, time.

\n
\n 		player_id, session_id, time—\n \n \n \n \n \n \n \n✓20001978
0.99		22
0.01		○○○
#4CA64C9\n
\n \n\n row_count_match\n \n \n \n \n \n \n \n \n \n \n \n\n
\n
\n
row_count_match()
\n
\n

Expect that the row count is exactly 2000.

\n
\n 		—2000\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		○○○—
#4CA64C10\n
\n \n\n col_count_match\n \n \n \n \n \n \n \n \n \n \n \n\n
\n
\n
col_count_match()
\n
\n

Expect that the column count is exactly 11.

\n
\n 		—11\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		○○○—
#4CA64C11\n
\n \n\n col_vals_not_null\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_null()
\n
\n

Expect that all values in item_type should not be Null.

\n
\n 		item_type—\n \n \n \n \n \n \n \n✓20002000
1.00		0
0.00		○○○—
#4CA64C12\n
\n \n\n col_exists\n \n \n \n \n \n \n \n\n
\n
\n
col_exists()
\n
\n

Expect that column start_day exists.

\n
\n 		start_day—\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		○○○—
2026-07-22 23:17:43 UTC< 1 s2026-07-22 23:17:43 UTC
\nNotes\n

Step 7 (pre_applied) Precondition applied: table dimensions [2,000 rows, 11 columns] → [1 row, 1 column].

\n\n
\n```\n:::\n:::\n\n\nThat's the kind of report you get from Pointblank: clear, interactive, and designed for everyone on\nyour team.\n\n### What is Data Validation?\n\nData validation makes sure your data is what you think it is before it reaches analysis, reports, or downstream systems. Pointblank gives you a structured way to declare what good data looks like, run those checks against a real table, and communicate the results to technical and non-technical audiences alike. You build a plan with a fluent, chainable API that draws on more than 45 validation methods, set warning, error, and critical thresholds, attach actions that fire when a threshold is crossed, and get back a report anyone on the team can read. Because Pointblank runs on [Narwhals](https://narwhals-dev.github.io/narwhals/) and [Ibis](https://ibis-project.org) under the hood, the same plan executes unchanged across Polars, Pandas, DuckDB, Spark, Snowflake, BigQuery, Databricks, PostgreSQL, MySQL, SQLite, and Parquet.\n\n### More than a checker\n\nThe reporting goes well beyond a single pass or fail. Any step can be opened in a focused [step report](https://posit-dev.github.io/pointblank/user-guide/post-interrogation/step-reports.html) that drills into the exact rows that failed, and those [failing rows can be pulled out](https://posit-dev.github.io/pointblank/user-guide/post-interrogation/extracts.html) as their own table for debugging. The source data can even be [split into passing and failing pieces](https://posit-dev.github.io/pointblank/user-guide/post-interrogation/sundering.html) for quarantine or reprocessing. Reports are localized in 40 languages, and results roll up into [quality dimensions and a single health score](https://posit-dev.github.io/pointblank/user-guide/post-interrogation/quality-dimensions-and-scoring.html), so completeness, validity, uniqueness, consistency, timeliness, and volume become one number you can watch over time.\n\n![A step report drills into the specific rows behind a failing validation step.](https://posit-dev.github.io/pointblank/assets/pointblank-step-report.png)\n\nAuthoring a plan does not have to start from an empty file. Pointblank can [draft a starting plan from a natural-language prompt](https://posit-dev.github.io/pointblank/user-guide/advanced-validation/draft-validation.html), and from there you can [revise and iterate on it in plain English](https://posit-dev.github.io/pointblank/user-guide/advanced-validation/ai-validation-editor.html) or ask it to suggest improvements. When your standards already live somewhere else, you can bring them with you. Pointblank will [import contracts](https://posit-dev.github.io/pointblank/user-guide/contracts-and-pipelines/importing-contracts.html) written as JSON Schema or Frictionless, pull column metadata straight from [SPSS, SAS, and Stata](https://posit-dev.github.io/pointblank/user-guide/metadata-import/statistical-packages.html) files, and for clinical work validate against [CDISC SDTM and ADaM](https://posit-dev.github.io/pointblank/user-guide/metadata-import/cdisc-validation.html) templates or read a Define-XML specification. It can also model structured missingness, encoding [why a value is absent](https://posit-dev.github.io/pointblank/user-guide/data-inspection/missing-vals-tbl.html) instead of treating every gap identically.\n\n![Point Pointblank at a table and let it draft a starting validation plan for you.](https://posit-dev.github.io/pointblank/assets/pointblank-draft-validation-report.png)\n\nGetting all of this into production is where Pointblank earns its place. You can define reusable [data contracts](https://posit-dev.github.io/pointblank/user-guide/contracts-and-pipelines/contracts.html) and enforce them at [both the source and target](https://posit-dev.github.io/pointblank/user-guide/contracts-and-pipelines/pipelines.html) of a transformation, keep plans as [YAML](https://posit-dev.github.io/pointblank/user-guide/yaml/yaml-validation-workflows.html) for version control and review, and run the whole thing from a [command-line interface](https://posit-dev.github.io/pointblank/user-guide/the-pointblank-cli/cli-data-validation.html) inside CI. Pointblank also speaks to machines: it ships an [MCP server](https://posit-dev.github.io/pointblank/user-guide/mcp-server/mcp-quick-start.html) and llms.txt files for AI agents, emits [OpenTelemetry](https://posit-dev.github.io/pointblank/user-guide/integrations/otel-integration.html) traces and metrics for observability, and can [generate synthetic test data](https://posit-dev.github.io/pointblank/user-guide/test-data-generation/test-data-generation.html) when you need something to validate against.\n\n", + "markdown": "---\ntitle: \"\"\ntoc: false\nbody-classes: \"gd-homepage\"\n---\n\n\n\n```{=html}\n
\n\"Logo\"\n

Pointblank

\n

Find out if your data is what you think it is.

\n
\n\n```\n\n::: {.column-margin}\n#### Links\n\nView on PyPI
\n\n#### AI / Agents\n\nSkills
\n[llms.txt](llms.txt)
\n[llms-full.txt](llms-full.txt)
\n\n#### Developers\n\n

Richard Iannone

Maintainer

Posit, PBC
\n

Posit Software, PBC

Copyright holder, funder
\n\n#### Community\n\n[Contributing guide](contributing.qmd)
\n[Code of conduct](code-of-conduct.qmd)
\n[Project roadmap](roadmap.qmd)
\n[Security policy](security.qmd)
\nFull license MIT
\n[Citing pointblank](citation.qmd)
\n\n#### Meta\n\n**Requires:** Python `>=3.10`
\n**Provides-Extra:** `pd`, `pl`, `pyspark`, `generate`, `mcp`, `otel`, `excel`, `cdisc`, `cdisc-core`, `bigquery`, `databricks`, `duckdb`, `mysql`, `mssql`, `postgres`, `snowflake`, `sqlite`, `docs`
\n[Package Info](package-info.html)\n:::\n\nPointblank is a data validation framework for Python that makes data quality checks beautiful,\npowerful, and stakeholder-friendly. Instead of cryptic error messages, get stunning interactive\nreports that turn data issues into conversations.\n\nHere's what a validation looks like (click \"Show the code\" to see how it's done):\n\n::: {#f1314558 .cell execution_count=1}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"Show the code\"}\nimport pointblank as pb\nimport polars as pl\n\nvalidation = (\n pb.Validate(\n data=pb.load_dataset(dataset=\"game_revenue\", tbl_type=\"polars\"),\n tbl_name=\"game_revenue\",\n label=\"Comprehensive validation of game revenue data\",\n thresholds=pb.Thresholds(warning=0.10, error=0.25, critical=0.35),\n brief=True\n )\n .col_vals_regex(columns=\"player_id\", pattern=r\"^[A-Z]{12}[0-9]{3}$\")\n .col_vals_gt(columns=\"session_duration\", value=20)\n .col_vals_ge(columns=\"item_revenue\", value=0.20)\n .col_vals_in_set(columns=\"item_type\", set=[\"iap\", \"ad\"])\n .col_vals_in_set(\n columns=\"acquisition\",\n set=[\"google\", \"facebook\", \"organic\", \"crosspromo\", \"other_campaign\"]\n )\n .col_vals_not_in_set(columns=\"country\", set=[\"Mongolia\", \"Germany\"])\n .col_vals_between(\n columns=\"session_duration\",\n left=10, right=50,\n pre=lambda df: df.select(pl.median(\"session_duration\")),\n brief=\"Expect that the median of `session_duration` should be between `10` and `50`.\"\n )\n .rows_distinct(columns_subset=[\"player_id\", \"session_id\", \"time\"])\n .row_count_match(count=2000)\n .col_count_match(count=11)\n .col_vals_not_null(columns=\"item_type\")\n .col_exists(columns=\"start_day\")\n .interrogate()\n)\n\nvalidation.get_tabular_report(title=\"Game Revenue Validation Report\")\n```\n\n::: {.cell-output .cell-output-display execution_count=1}\n```{=html}\n
\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n

Game Revenue Validation Report

\n
Comprehensive validation of game revenue data
Polarsgame_revenueWARNING0.1ERROR0.25CRITICAL0.35
STEPCOLUMNSVALUESTBLEVALUNITSPASSFAILWECEXT
#4CA64C1\n
\n \n\n col_vals_regex\n \n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_regex()
\n
\n

Expect that values in player_id should match the regular expression: ^[A-Z]{12}[0-9]{3}$.

\n
\n 		player_id^[A-Z]{12}[0-9]{3}$\n \n \n \n \n \n \n \n✓20002000
1.00		0
0.00		○○○—
#EBBC142\n
\n \n\n col_vals_gt\n \n \n \n \n \n \n\n
\n
\n
col_vals_gt()
\n
\n

Expect that values in session_duration should be > 20.

\n
\n 		session_duration20\n \n \n \n \n \n \n \n✓20001418
0.71		582
0.29		●●○
#FF33003\n
\n \n\n col_vals_ge\n \n \n \n \n \n \n\n
\n
\n
col_vals_ge()
\n
\n

Expect that values in item_revenue should be >= 0.2.

\n
\n 		item_revenue0.2\n \n \n \n \n \n \n \n✓20001192
0.60		808
0.40		●●●
#4CA64C4\n
\n \n\n col_vals_in_set\n \n \n \n \n \n \n\n
\n
\n
col_vals_in_set()
\n
\n

Expect that values in item_type should be in the set of iap, ad.

\n
\n 		item_typeiap, ad\n \n \n \n \n \n \n \n✓20002000
1.00		0
0.00		○○○—
#4CA64C665\n
\n \n\n col_vals_in_set\n \n \n \n \n \n \n\n
\n
\n
col_vals_in_set()
\n
\n

Expect that values in acquisition should be in the set of google, facebook, organic, and 2 more.

\n
\n 		acquisitiongoogle, facebook, organic, crosspromo, other_campaign\n \n \n \n \n \n \n \n✓20001975
0.99		25
0.01		○○○
#AAAAAA6\n
\n \n\n col_vals_not_in_set\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_in_set()
\n
\n

Expect that values in country should not be in the set of Mongolia, Germany.

\n
\n 		countryMongolia, Germany\n \n \n \n \n \n \n \n✓20001775
0.89		225
0.11		●○○
#4CA64C7\n
\n \n\n col_vals_between\n \n \n \n \n \n \n\n
\n
\n
col_vals_between()
\n
\n

Expect that the median of session_duration should be between 10 and 50.

\n
\n 		session_duration[10, 50]\n \n \n \n \n \n \n \n \n \n✓11
1.00		0
0.00		○○○—
#4CA64C668\n
\n \n\n rows_distinct\n \n \n \n \n \n \n \n \n \n \n\n
\n
\n
rows_distinct()
\n
\n

Expect entirely distinct rows across player_id, session_id, time.

\n
\n 		player_id, session_id, time—\n \n \n \n \n \n \n \n✓20001978
0.99		22
0.01		○○○
#4CA64C9\n
\n \n\n row_count_match\n \n \n \n \n \n \n \n \n \n \n \n\n
\n
\n
row_count_match()
\n
\n

Expect that the row count is exactly 2000.

\n
\n 		—2000\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		○○○—
#4CA64C10\n
\n \n\n col_count_match\n \n \n \n \n \n \n \n \n \n \n \n\n
\n
\n
col_count_match()
\n
\n

Expect that the column count is exactly 11.

\n
\n 		—11\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		○○○—
#4CA64C11\n
\n \n\n col_vals_not_null\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_null()
\n
\n

Expect that all values in item_type should not be Null.

\n
\n 		item_type—\n \n \n \n \n \n \n \n✓20002000
1.00		0
0.00		○○○—
#4CA64C12\n
\n \n\n col_exists\n \n \n \n \n \n \n \n\n
\n
\n
col_exists()
\n
\n

Expect that column start_day exists.

\n
\n 		start_day—\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		○○○—
2026-08-10 16:43:38 UTC< 1 s2026-08-10 16:43:39 UTC
\nNotes\n

Step 7 (pre_applied) Precondition applied: table dimensions [2,000 rows, 11 columns] → [1 row, 1 column].

\n\n
\n```\n:::\n:::\n\n\nThat's the kind of report you get from Pointblank: clear, interactive, and designed for everyone on\nyour team.\n\n### What is Data Validation?\n\nData validation makes sure your data is what you think it is before it reaches analysis, reports, or downstream systems. Pointblank gives you a structured way to declare what good data looks like, run those checks against a real table, and communicate the results to technical and non-technical audiences alike. You build a plan with a fluent, chainable API that draws on more than 45 validation methods, set warning, error, and critical thresholds, attach actions that fire when a threshold is crossed, and get back a report anyone on the team can read. Because Pointblank runs on [Narwhals](https://narwhals-dev.github.io/narwhals/) and [Ibis](https://ibis-project.org) under the hood, the same plan executes unchanged across Polars, Pandas, DuckDB, Spark, Snowflake, BigQuery, Databricks, PostgreSQL, MySQL, SQLite, and Parquet.\n\n### More than a checker\n\nThe reporting goes well beyond a single pass or fail. Any step can be opened in a focused [step report](https://posit-dev.github.io/pointblank/user-guide/post-interrogation/step-reports.html) that drills into the exact rows that failed, and those [failing rows can be pulled out](https://posit-dev.github.io/pointblank/user-guide/post-interrogation/extracts.html) as their own table for debugging. The source data can even be [split into passing and failing pieces](https://posit-dev.github.io/pointblank/user-guide/post-interrogation/sundering.html) for quarantine or reprocessing. Reports are localized in 40 languages, and results roll up into [quality dimensions and a single health score](https://posit-dev.github.io/pointblank/user-guide/post-interrogation/quality-dimensions-and-scoring.html), so completeness, validity, uniqueness, consistency, timeliness, and volume become one number you can watch over time.\n\n![A step report drills into the specific rows behind a failing validation step.](https://posit-dev.github.io/pointblank/assets/pointblank-step-report.png)\n\nAuthoring a plan does not have to start from an empty file. Pointblank can [draft a starting plan from a natural-language prompt](https://posit-dev.github.io/pointblank/user-guide/advanced-validation/draft-validation.html), and from there you can [revise and iterate on it in plain English](https://posit-dev.github.io/pointblank/user-guide/advanced-validation/ai-validation-editor.html) or ask it to suggest improvements. When your standards already live somewhere else, you can bring them with you. Pointblank will [import contracts](https://posit-dev.github.io/pointblank/user-guide/contracts-and-pipelines/importing-contracts.html) written as JSON Schema or Frictionless, pull column metadata straight from [SPSS, SAS, and Stata](https://posit-dev.github.io/pointblank/user-guide/metadata-import/statistical-packages.html) files, and for clinical work validate against [CDISC SDTM and ADaM](https://posit-dev.github.io/pointblank/user-guide/metadata-import/cdisc-validation.html) templates or read a Define-XML specification. It can also model structured missingness, encoding [why a value is absent](https://posit-dev.github.io/pointblank/user-guide/data-inspection/missing-vals-tbl.html) instead of treating every gap identically.\n\n![Point Pointblank at a table and let it draft a starting validation plan for you.](https://posit-dev.github.io/pointblank/assets/pointblank-draft-validation-report.png)\n\nGetting all of this into production is where Pointblank earns its place. You can define reusable [data contracts](https://posit-dev.github.io/pointblank/user-guide/contracts-and-pipelines/contracts.html) and enforce them at [both the source and target](https://posit-dev.github.io/pointblank/user-guide/contracts-and-pipelines/pipelines.html) of a transformation, keep plans as [YAML](https://posit-dev.github.io/pointblank/user-guide/yaml/yaml-validation-workflows.html) for version control and review, and run the whole thing from a [command-line interface](https://posit-dev.github.io/pointblank/user-guide/the-pointblank-cli/cli-data-validation.html) inside CI. Pointblank also speaks to machines: it ships an [MCP server](https://posit-dev.github.io/pointblank/user-guide/mcp-server/mcp-quick-start.html) and llms.txt files for AI agents, emits [OpenTelemetry](https://posit-dev.github.io/pointblank/user-guide/integrations/otel-integration.html) traces and metrics for observability, and can [generate synthetic test data](https://posit-dev.github.io/pointblank/user-guide/test-data-generation/test-data-generation.html) when you need something to validate against.\n\n", "supporting": [ - "index_files" + "index_files/figure-html" ], "filters": [], "includes": { diff --git a/_freeze/reference/Validate/execute-results/html.json b/_freeze/reference/Validate/execute-results/html.json index 65cc593a8..83b2d2466 100644 --- a/_freeze/reference/Validate/execute-results/html.json +++ b/_freeze/reference/Validate/execute-results/html.json @@ -1,10 +1,10 @@ { - "hash": "fccf1c521741d090b700bb57d418380e", + "hash": "369ed6bd1f7b129cf01287d787fea696", "result": { "engine": "jupyter", - "markdown": "---\ntitle: \"[Validate]{.doc-object-name .doc-class .doc-label .doc-label-dataclass}\"\nbody-classes: doc-api-page\npage-navigation: false\nhtml-table-processing: none\n---\n\n::: {.doc-subject}\n\nWorkflow for defining a set of validations on a table and interrogating for results.\n:::\n\n::: {.doc-usage-source}\nUsage\n\n[Source](https://github.com/posit-dev/pointblank/blob/main/pointblank/validate.py#L5066-L21280){target=\"_blank\" rel=\"noopener\"}\n:::\n\n::: {.doc-signature .doc-Kind.CLASS}\n```python\nValidate(\n data,\n reference=None,\n tbl_name=None,\n label=None,\n thresholds=None,\n actions=None,\n final_actions=None,\n brief=None,\n lang=None,\n locale=None,\n owner=None,\n consumers=None,\n version=None\n)\n```\n:::\n\n::: {.doc-text}\n\nThe `Validate` class is used for defining a set of validation steps on a table and interrogating\nthe table with the *validation plan*. This class is the main entry point for the *data quality\nreporting* workflow. The overall aim of this workflow is to generate comprehensive reporting\ninformation to assess the level of data quality for a target table.\n\nWe can supply as many validation steps as needed, and having a large number of them should\nincrease the validation coverage for a given table. The validation methods (e.g.,\n[`col_vals_gt()`](`pointblank.Validate.col_vals_gt`),\n[`col_vals_between()`](`pointblank.Validate.col_vals_between`), etc.) translate to discrete\nvalidation steps, where each step will be sequentially numbered (useful when viewing the\nreporting data). This process of calling validation methods is known as developing a\n*validation plan*.\n\nThe validation methods, when called, are merely instructions up to the point the concluding\n[`interrogate()`](`pointblank.Validate.interrogate`) method is called. That kicks off the\nprocess of acting on the *validation plan* by querying the target table getting reporting\nresults for each step. Once the interrogation process is complete, we can say that the workflow\nnow has reporting information. We can then extract useful information from the reporting data\nto understand the quality of the table. Printing the `Validate` object (or using the\n[`get_tabular_report()`](`pointblank.Validate.get_tabular_report`) method) will return a table\nwith the results of the interrogation and\n[`get_sundered_data()`](`pointblank.Validate.get_sundered_data`) allows for the splitting of the\ntable based on passing and failing rows.\n:::\n\n## Parameters {.doc-parameters}\n\n::: {.doc-definition-items}\n[data]{.doc-parameter-name}[:]{.doc-parameter-annotation-sep} [IntoDataFrame]{.doc-parameter-annotation}\n\n: The table to validate, which could be a DataFrame object, an Ibis table object, a CSV\n file path, a Parquet file path, a GitHub URL pointing to a CSV or Parquet file, or a\n database connection string. When providing a CSV or Parquet file path (as a string or\n `pathlib.Path` object), the file will be automatically loaded using an available DataFrame\n library (Polars or Pandas). Parquet input also supports glob patterns, directories\n containing .parquet files, and Spark-style partitioned datasets. GitHub URLs are\n automatically transformed to raw content URLs and downloaded. Connection strings enable\n direct database access via Ibis with optional table specification using the `::table_name`\n suffix. Read the *Supported Input Table Types* section for details on the supported table\n types.\n\n[tbl_name]{.doc-parameter-name}[:]{.doc-parameter-annotation-sep} [str | None]{.doc-parameter-annotation} [=]{.doc-parameter-default-sep .op} [None]{.doc-parameter-default}\n\n: An optional name to assign to the input table object. If no value is provided, a name will\n be generated based on whatever information is available. This table name will be displayed\n in the header area of the tabular report.\n\n[label]{.doc-parameter-name}[:]{.doc-parameter-annotation-sep} [str | None]{.doc-parameter-annotation} [=]{.doc-parameter-default-sep .op} [None]{.doc-parameter-default}\n\n: An optional label for the validation plan. If no value is provided, a label will be\n generated based on the current system date and time. Markdown can be used here to make the\n label more visually appealing (it will appear in the header area of the tabular report).\n\n[thresholds]{.doc-parameter-name}[:]{.doc-parameter-annotation-sep} [int | float | bool | tuple | dict | Thresholds | None]{.doc-parameter-annotation} [=]{.doc-parameter-default-sep .op} [None]{.doc-parameter-default}\n\n: Generate threshold failure levels so that all validation steps can report and react\n accordingly when exceeding the set levels. The thresholds are set at the global level and\n can be overridden at the validation step level (each validation step has its own\n `thresholds=` parameter). The default is `None`, which means that no thresholds will be set.\n Look at the *Thresholds* section for information on how to set threshold levels.\n\n[actions]{.doc-parameter-name}[:]{.doc-parameter-annotation-sep} [Actions | None]{.doc-parameter-annotation} [=]{.doc-parameter-default-sep .op} [None]{.doc-parameter-default}\n\n: The actions to take when validation steps meet or exceed any set threshold levels. These\n actions are paired with the threshold levels and are executed during the interrogation\n process when there are exceedances. The actions are executed right after each step is\n evaluated. Such actions should be provided in the form of an `Actions` object. If `None`\n then no global actions will be set. View the *Actions* section for information on how to set\n actions.\n\n[final_actions]{.doc-parameter-name}[:]{.doc-parameter-annotation-sep} [FinalActions | None]{.doc-parameter-annotation} [=]{.doc-parameter-default-sep .op} [None]{.doc-parameter-default}\n\n: The actions to take when the validation process is complete and the final results are\n available. This is useful for sending notifications or reporting the overall status of the\n validation process. The final actions are executed after all validation steps have been\n processed and the results have been collected. The final actions are not tied to any\n threshold levels, they are executed regardless of the validation results. Such actions\n should be provided in the form of a `FinalActions` object. If `None` then no finalizing\n actions will be set. Please see the *Actions* section for information on how to set final\n actions.\n\n[brief]{.doc-parameter-name}[:]{.doc-parameter-annotation-sep} [str | bool | None]{.doc-parameter-annotation} [=]{.doc-parameter-default-sep .op} [None]{.doc-parameter-default}\n\n: A global setting for briefs, which are optional brief descriptions for validation steps\n (they be displayed in the reporting table). For such a global setting, templating elements\n like `\"{step}\"` (to insert the step number) or `\"{auto}\"` (to include an automatically\n generated brief) are useful. If `True` then each brief will be automatically generated. If\n `None` (the default) then briefs aren't globally set.\n\n[lang]{.doc-parameter-name}[:]{.doc-parameter-annotation-sep} [str | None]{.doc-parameter-annotation} [=]{.doc-parameter-default-sep .op} [None]{.doc-parameter-default}\n\n: The language to use for various reporting elements. By default, `None` will select English\n (`\"en\"`) as the but other options include French (`\"fr\"`), German (`\"de\"`), Italian\n (`\"it\"`), Spanish (`\"es\"`), and several more. Have a look at the *Reporting Languages*\n section for the full list of supported languages and information on how the language setting\n is utilized.\n\n[locale]{.doc-parameter-name}[:]{.doc-parameter-annotation-sep} [str | None]{.doc-parameter-annotation} [=]{.doc-parameter-default-sep .op} [None]{.doc-parameter-default}\n\n: An optional locale ID to use for formatting values in the reporting table according the\n locale's rules. Examples include `\"en-US\"` for English (United States) and `\"fr-FR\"` for\n French (France). More simply, this can be a language identifier without a designation of\n territory, like `\"es\"` for Spanish.\n\n[owner]{.doc-parameter-name}[:]{.doc-parameter-annotation-sep} [str | None]{.doc-parameter-annotation} [=]{.doc-parameter-default-sep .op} [None]{.doc-parameter-default}\n\n: An optional string identifying the owner of the data being validated. This is useful for\n governance purposes, indicating who is responsible for the quality and maintenance of the\n data. For example, `\"data-platform-team\"` or `\"analytics-engineering\"`.\n\n[consumers]{.doc-parameter-name}[:]{.doc-parameter-annotation-sep} [str | list[str] | None]{.doc-parameter-annotation} [=]{.doc-parameter-default-sep .op} [None]{.doc-parameter-default}\n\n: An optional string or list of strings identifying who depends on or consumes this data.\n This helps document data dependencies and can be useful for impact analysis when data\n quality issues are detected. For example, `\"ml-team\"` or `[\"ml-team\", \"analytics\"]`.\n\n[version]{.doc-parameter-name}[:]{.doc-parameter-annotation-sep} [str | None]{.doc-parameter-annotation} [=]{.doc-parameter-default-sep .op} [None]{.doc-parameter-default}\n\n: An optional string representing the version of the validation plan or data contract. This\n supports semantic versioning (e.g., `\"1.0.0\"`, `\"2.1.0\"`) and is useful for tracking changes\n to validation rules over time and for organizational governance.\n:::\n\n## Returns {.doc-returns}\n\n::: {.doc-definition-items}\n [Validate]{.doc-parameter-annotation}\n\n: A `Validate` object with the table and validations to be performed.\n:::\n\n## Supported Input Table Types {.doc-supported-input-table-types}\n\nThe `data=` parameter can be given any of the following table types:\n\n- Polars DataFrame (`\"polars\"`)\n- Pandas DataFrame (`\"pandas\"`)\n- PySpark table (`\"pyspark\"`)\n- DuckDB table (`\"duckdb\"`)*\n- MySQL table (`\"mysql\"`)*\n- PostgreSQL table (`\"postgresql\"`)*\n- SQLite table (`\"sqlite\"`)*\n- Microsoft SQL Server table (`\"mssql\"`)*\n- Snowflake table (`\"snowflake\"`)*\n- Databricks table (`\"databricks\"`)*\n- BigQuery table (`\"bigquery\"`)*\n- Parquet table (`\"parquet\"`)*\n- CSV files (string path or `pathlib.Path` object with `.csv` extension)\n- Parquet files (string path, `pathlib.Path` object, glob pattern, directory with `.parquet`\nextension, or partitioned dataset)\n- Database connection strings (URI format with optional table specification)\n\nThe table types marked with an asterisk need to be prepared as Ibis tables (with type of\n`ibis.expr.types.relations.Table`). Furthermore, the use of `Validate` with such tables requires\nthe Ibis library v9.5.0 and above to be installed. If the input table is a Polars or Pandas\nDataFrame, the Ibis library is not required.\n\nTo use a CSV file, ensure that a string or `pathlib.Path` object with a `.csv` extension is\nprovided. The file will be automatically detected and loaded using the best available DataFrame\nlibrary. The loading preference is Polars first, then Pandas as a fallback.\n\nConnection strings follow database URL formats and must also specify a table using the\n`::table_name` suffix. Examples include:\n\n```\n\"duckdb:///path/to/database.ddb::table_name\"\n\"sqlite:///path/to/database.db::table_name\"\n\"postgresql://user:password@localhost:5432/database::table_name\"\n\"mysql://user:password@localhost:3306/database::table_name\"\n\"bigquery://project/dataset::table_name\"\n\"snowflake://user:password@account/database/schema::table_name\"\n```\n\nWhen using connection strings, the Ibis library with the appropriate backend driver is required.\n\n## Thresholds {.doc-thresholds}\n\nThe `thresholds=` parameter is used to set the failure-condition levels for all validation\nsteps. They are set here at the global level but can be overridden at the validation step level\n(each validation step has its own local `thresholds=` parameter).\n\nThere are three threshold levels: 'warning', 'error', and 'critical'. The threshold values can\neither be set as a proportion failing of all test units (a value between `0` to `1`), or, the\nabsolute number of failing test units (as integer that's `1` or greater).\n\nThresholds can be defined using one of these input schemes:\n\n1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create\nthresholds)\n2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is the\n'error' level, and position `2` is the 'critical' level\n3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and\n'critical'\n4. a single integer/float value denoting absolute number or fraction of failing test units for\nthe 'warning' level only\n\nIf the number of failing test units for a validation step exceeds set thresholds, the validation\nstep will be marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need\nto be set, you're free to set any combination of them.\n\nAside from reporting failure conditions, thresholds can be used to determine the actions to take\nfor each level of failure (using the `actions=` parameter).\n\n## Actions {.doc-actions}\n\nThe `actions=` and `final_actions=` parameters provide mechanisms to respond to validation\nresults. These actions can be used to notify users of validation failures, log issues, or\ntrigger other processes when problems are detected.\n\n*Step Actions*\n\nThe `actions=` parameter allows you to define actions that are triggered when validation steps\nexceed specific threshold levels (warning, error, or critical). These actions are executed\nduring the interrogation process, right after each step is evaluated.\n\nStep actions should be provided using the [`Actions`](`pointblank.Actions`) class, which lets\nyou specify different actions for different severity levels:\n\n```python\n# Define an action that logs a message when warning threshold is exceeded\ndef log_warning():\n metadata = pb.get_action_metadata()\n print(f\"WARNING: Step {metadata['step']} failed with type {metadata['type']}\")\n\n# Define actions for different threshold levels\nactions = pb.Actions(\n warning = log_warning,\n error = lambda: send_email(\"Error in validation\"),\n critical = \"CRITICAL FAILURE DETECTED\"\n)\n\n# Use in Validate\nvalidation = pb.Validate(\n data=my_data,\n actions=actions # Global actions for all steps\n)\n```\n\nYou can also provide step-specific actions in individual validation methods:\n\n```python\nvalidation.col_vals_gt(\n columns=\"revenue\",\n value=0,\n actions=pb.Actions(warning=log_warning) # Only applies to this step\n)\n```\n\nStep actions have access to step-specific context through the\n[`get_action_metadata()`](`pointblank.get_action_metadata`) function, which provides details\nabout the current validation step that triggered the action.\n\n*Final Actions*\n\nThe `final_actions=` parameter lets you define actions that execute after all validation steps\nhave completed. These are useful for providing summaries, sending notifications based on\noverall validation status, or performing cleanup operations.\n\nFinal actions should be provided using the [`FinalActions`](`pointblank.FinalActions`) class:\n\n```python\ndef send_report():\n summary = pb.get_validation_summary()\n if summary[\"status\"] == \"CRITICAL\":\n send_alert_email(\n subject=f\"CRITICAL validation failures in {summary['tbl_name']}\",\n body=f\"{summary['critical_steps']} steps failed with critical severity.\"\n )\n\nvalidation = pb.Validate(\n data=my_data,\n final_actions=pb.FinalActions(send_report)\n)\n```\n\nFinal actions have access to validation-wide summary information through the\n[`get_validation_summary()`](`pointblank.get_validation_summary`) function, which provides a\ncomprehensive overview of the entire validation process.\n\nThe combination of step actions and final actions provides a flexible system for responding to\ndata quality issues at both the individual step level and the overall validation level.\n\n## Reporting Languages {.doc-reporting-languages}\n\nVarious pieces of reporting in Pointblank can be localized to a specific language. This is done\nby setting the `lang=` parameter in `Validate`. Any of the following languages can be used (just\nprovide the language code):\n\n- English (`\"en\"`)\n- French (`\"fr\"`)\n- German (`\"de\"`)\n- Italian (`\"it\"`)\n- Spanish (`\"es\"`)\n- Portuguese (`\"pt\"`)\n- Dutch (`\"nl\"`)\n- Swedish (`\"sv\"`)\n- Danish (`\"da\"`)\n- Norwegian Bokmål (`\"nb\"`)\n- Icelandic (`\"is\"`)\n- Finnish (`\"fi\"`)\n- Polish (`\"pl\"`)\n- Czech (`\"cs\"`)\n- Romanian (`\"ro\"`)\n- Greek (`\"el\"`)\n- Russian (`\"ru\"`)\n- Turkish (`\"tr\"`)\n- Arabic (`\"ar\"`)\n- Hindi (`\"hi\"`)\n- Simplified Chinese (`\"zh-Hans\"`)\n- Traditional Chinese (`\"zh-Hant\"`)\n- Japanese (`\"ja\"`)\n- Korean (`\"ko\"`)\n- Vietnamese (`\"vi\"`)\n- Indonesian (`\"id\"`)\n- Ukrainian (`\"uk\"`)\n- Bulgarian (`\"bg\"`)\n- Croatian (`\"hr\"`)\n- Estonian (`\"et\"`)\n- Hungarian (`\"hu\"`)\n- Irish (`\"ga\"`)\n- Latvian (`\"lv\"`)\n- Lithuanian (`\"lt\"`)\n- Maltese (`\"mt\"`)\n- Slovak (`\"sk\"`)\n- Slovenian (`\"sl\"`)\n- Hebrew (`\"he\"`)\n- Thai (`\"th\"`)\n- Persian (`\"fa\"`)\n\nAutomatically generated briefs (produced by using `brief=True` or `brief=\"...{auto}...\"`) will\nbe written in the selected language. The language setting will also used when generating the\nvalidation report table through\n[`get_tabular_report()`](`pointblank.Validate.get_tabular_report`) (or printing the `Validate`\nobject in a notebook environment).\n\n## Examples {.doc-examples}\n\n### Creating a validation plan and interrogating\n\nLet's walk through a data quality analysis of an extremely small table. It's actually called\n`\"small_table\"` and it's accessible through the [`load_dataset()`](`pointblank.load_dataset`)\nfunction.\n\n::: {#415e51eb .cell execution_count=1}\n``` {.python .cell-code}\nimport pointblank as pb\n\n# Load the `small_table` dataset\nsmall_table = pb.load_dataset(dataset=\"small_table\", tbl_type=\"polars\")\n\n# Preview the table\npb.preview(small_table)\n```\n\n::: {.cell-output .cell-output-display execution_count=1}\n```{=html}\n
\n\n\n\n \n \n \n \n \n \n \n \n \n\n\n\n\n \n \n \n\n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
PolarsRows13Columns8
date_time
Datetime
date
Date
a
Int64
b
String
c
Int64
d
Float64
e
Boolean
f
String
12016-01-04 11:00:002016-01-0421-bcd-34533423.29Truehigh
22016-01-04 00:32:002016-01-0435-egh-16389999.99Truelow
32016-01-05 13:32:002016-01-0568-kdg-93832343.23Truehigh
42016-01-06 17:23:002016-01-0625-jdo-903None3892.4Falsemid
52016-01-09 12:36:002016-01-0983-ldm-0387283.94Truelow
92016-01-20 04:30:002016-01-2035-bce-6429837.93Falsehigh
102016-01-20 04:30:002016-01-2035-bce-6429837.93Falsehigh
112016-01-26 20:07:002016-01-2642-dmx-0107833.98Truelow
122016-01-28 02:51:002016-01-2827-dmx-0108108.34Falselow
132016-01-30 11:23:002016-01-3013-dka-303None2230.09Truehigh
\n\n
\n```\n:::\n:::\n\n\nWe ought to think about what's tolerable in terms of data quality so let's designate\nproportional failure thresholds to the 'warning', 'error', and 'critical' states. This can be\ndone by using the [`Thresholds`](`pointblank.Thresholds`) class.\n\n::: {#adfc5c73 .cell execution_count=2}\n``` {.python .cell-code}\nthresholds = pb.Thresholds(warning=0.10, error=0.25, critical=0.35)\n```\n:::\n\n\nNow, we use the `Validate` class and give it the `thresholds` object (which serves as a default\nfor all validation steps but can be overridden). The static thresholds provided in `thresholds=`\nwill make the reporting a bit more useful. We also need to provide a target table and we'll use\n`small_table` for this.\n\n::: {#42f5e608 .cell execution_count=3}\n``` {.python .cell-code}\nvalidation = (\n pb.Validate(\n data=small_table,\n tbl_name=\"small_table\",\n label=\"`Validate` example.\",\n thresholds=thresholds\n )\n)\n```\n:::\n\n\nThen, as with any `Validate` object, we can add steps to the validation plan by using as many\nvalidation methods as we want. To conclude the process (and actually query the data table), we\nuse the [`interrogate()`](`pointblank.Validate.interrogate`) method.\n\n::: {#800b206e .cell execution_count=4}\n``` {.python .cell-code}\nvalidation = (\n validation\n .col_vals_gt(columns=\"d\", value=100)\n .col_vals_le(columns=\"c\", value=5)\n .col_vals_between(columns=\"c\", left=3, right=10, na_pass=True)\n .col_vals_regex(columns=\"b\", pattern=r\"[0-9]-[a-z]{3}-[0-9]{3}\")\n .col_exists(columns=[\"date\", \"date_time\"])\n .interrogate()\n)\n```\n:::\n\n\nThe `validation` object can be printed as a reporting table.\n\n::: {#7f6cf9b4 .cell execution_count=5}\n``` {.python .cell-code}\nvalidation\n```\n\n::: {.cell-output .cell-output-display execution_count=5}\n```{=html}\n
\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
Pointblank Validation
`Validate` example.
Polarssmall_tableWARNING0.1ERROR0.25CRITICAL0.35
STEPCOLUMNSVALUESTBLEVALUNITSPASSFAILWECEXT
#4CA64C1\n
\n \n\n col_vals_gt\n \n \n \n \n \n \n\n
\n
\n
col_vals_gt()
\n
\n \n 		d100\n \n \n \n \n \n \n \n✓1313
1.00		0
0.00		○○○—
#FF33002\n
\n \n\n col_vals_le\n \n \n \n \n \n \n\n
\n
\n
col_vals_le()
\n
\n \n 		c5\n \n \n \n \n \n \n \n✓135
0.38		8
0.62		●●●
#4CA64C663\n
\n \n\n col_vals_between\n \n \n \n \n \n \n\n
\n
\n
col_vals_between()
\n
\n \n 		c[3, 10]\n \n \n \n \n \n \n \n✓1312
0.92		1
0.08		○○○
#4CA64C4\n
\n \n\n col_vals_regex\n \n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_regex()
\n
\n \n 		b[0-9]-[a-z]{3}-[0-9]{3}\n \n \n \n \n \n \n \n✓1313
1.00		0
0.00		○○○—
#4CA64C5\n
\n \n\n col_exists\n \n \n \n \n \n \n \n\n
\n
\n
col_exists()
\n
\n \n 		date—\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		○○○—
#4CA64C6\n
\n \n\n col_exists\n \n \n \n \n \n \n \n\n
\n
\n
col_exists()
\n
\n \n 		date_time—\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		○○○—
2026-07-22 23:20:59 UTC< 1 s2026-07-22 23:20:59 UTC
\n\n
\n```\n:::\n:::\n\n\nThe report could be further customized by using the\n[`get_tabular_report()`](`pointblank.Validate.get_tabular_report`) method, which contains\noptions for modifying the display of the table.\n\n### Adding briefs\n\nBriefs are short descriptions of the validation steps. While they can be set for each step\nindividually, they can also be set globally. The global setting is done by using the\n`brief=` argument in `Validate`. The global setting can be as simple as `True` to have\nautomatically-generated briefs for each step. Alternatively, we can use templating elements\nlike `\"{step}\"` (to insert the step number) or `\"{auto}\"` (to include an automatically generated\nbrief). Here's an example of a global setting for briefs:\n\n::: {#e194c1a1 .cell execution_count=6}\n``` {.python .cell-code}\nvalidation_2 = (\n pb.Validate(\n data=pb.load_dataset(),\n tbl_name=\"small_table\",\n label=\"Validation example with briefs\",\n brief=\"Step {step}: {auto}\",\n )\n .col_vals_gt(columns=\"d\", value=100)\n .col_vals_between(columns=\"c\", left=3, right=10, na_pass=True)\n .col_vals_regex(\n columns=\"b\",\n pattern=r\"[0-9]-[a-z]{3}-[0-9]{3}\",\n brief=\"Regex check for column {col}\"\n )\n .interrogate()\n)\n\nvalidation_2\n```\n\n::: {.cell-output .cell-output-display execution_count=6}\n```{=html}\n
\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
Pointblank Validation
Validation example with briefs
Polarssmall_table
STEPCOLUMNSVALUESTBLEVALUNITSPASSFAILWECEXT
#4CA64C1\n
\n \n\n col_vals_gt\n \n \n \n \n \n \n\n
\n
\n
col_vals_gt()
\n
\n

Step 1: Expect that values in d should be > 100.

\n
\n 		d100\n \n \n \n \n \n \n \n✓1313
1.00		0
0.00		————
#4CA64C662\n
\n \n\n col_vals_between\n \n \n \n \n \n \n\n
\n
\n
col_vals_between()
\n
\n

Step 2: Expect that values in c should be between 3 and 10.

\n
\n 		c[3, 10]\n \n \n \n \n \n \n \n✓1312
0.92		1
0.08		———
#4CA64C3\n
\n \n\n col_vals_regex\n \n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_regex()
\n
\n

Regex check for column b

\n
\n 		b[0-9]-[a-z]{3}-[0-9]{3}\n \n \n \n \n \n \n \n✓1313
1.00		0
0.00		————
2026-07-22 23:20:59 UTC< 1 s2026-07-22 23:20:59 UTC
\n\n
\n```\n:::\n:::\n\n\nWe see the text of the briefs appear in the `STEP` column of the reporting table. Furthermore,\nthe global brief's template (`\"Step {step}: {auto}\"`) is applied to all steps except for the\nfinal step, where the step-level `brief=` argument provided an override.\n\nIf you should want to cancel the globally-defined brief for one or more validation steps, you\ncan set `brief=False` in those particular steps.\n\n### Post-interrogation methods\n\nThe `Validate` class has a number of post-interrogation methods that can be used to extract\nuseful information from the validation results. For example, the\n[`get_data_extracts()`](`pointblank.Validate.get_data_extracts`) method can be used to get\nthe data extracts for each validation step.\n\n::: {#2f0955e3 .cell execution_count=7}\n``` {.python .cell-code}\nvalidation_2.get_data_extracts()\n```\n\n::: {.cell-output .cell-output-display execution_count=7}\n```\n{1: shape: (0, 9)\n ┌───────────┬──────────────┬──────┬─────┬───┬─────┬─────┬──────┬─────┐\n │ _row_num_ ┆ date_time ┆ date ┆ a ┆ … ┆ c ┆ d ┆ e ┆ f │\n │ --- ┆ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │\n │ u32 ┆ datetime[μs] ┆ date ┆ i64 ┆ ┆ i64 ┆ f64 ┆ bool ┆ str │\n ╞═══════════╪══════════════╪══════╪═════╪═══╪═════╪═════╪══════╪═════╡\n └───────────┴──────────────┴──────┴─────┴───┴─────┴─────┴──────┴─────┘,\n 2: shape: (1, 9)\n ┌───────────┬─────────────────────┬────────────┬─────┬───┬─────┬─────────┬───────┬─────┐\n │ _row_num_ ┆ date_time ┆ date ┆ a ┆ … ┆ c ┆ d ┆ e ┆ f │\n │ --- ┆ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │\n │ u32 ┆ datetime[μs] ┆ date ┆ i64 ┆ ┆ i64 ┆ f64 ┆ bool ┆ str │\n ╞═══════════╪═════════════════════╪════════════╪═════╪═══╪═════╪═════════╪═══════╪═════╡\n │ 8 ┆ 2016-01-17 11:27:00 ┆ 2016-01-17 ┆ 4 ┆ … ┆ 2 ┆ 1035.64 ┆ false ┆ low │\n └───────────┴─────────────────────┴────────────┴─────┴───┴─────┴─────────┴───────┴─────┘,\n 3: shape: (0, 9)\n ┌───────────┬──────────────┬──────┬─────┬───┬─────┬─────┬──────┬─────┐\n │ _row_num_ ┆ date_time ┆ date ┆ a ┆ … ┆ c ┆ d ┆ e ┆ f │\n │ --- ┆ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │\n │ u32 ┆ datetime[μs] ┆ date ┆ i64 ┆ ┆ i64 ┆ f64 ┆ bool ┆ str │\n ╞═══════════╪══════════════╪══════╪═════╪═══╪═════╪═════╪══════╪═════╡\n └───────────┴──────────────┴──────┴─────┴───┴─────┴─────┴──────┴─────┘}\n```\n:::\n:::\n\n\nWe can also view step reports for each validation step using the\n[`get_step_report()`](`pointblank.Validate.get_step_report`) method. This method adapts to the\ntype of validation step and shows the relevant information for a step's validation.\n\n::: {#f9080fbe .cell execution_count=8}\n``` {.python .cell-code}\nvalidation_2.get_step_report(i=2)\n```\n\n::: {.cell-output .cell-output-display execution_count=8}\n```{=html}\n
\n\n\n\n \n \n \n \n \n \n \n \n \n\n\n\n\n \n \n \n\n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n\n\n
Report for Validation Step 2
ASSERTION 3 ≤ c ≤ 10
1 / 13 TEST UNIT FAILURES IN COLUMN 5
EXTRACT OF ALL 1 ROWS (WITH TEST UNIT FAILURES IN RED):
date_time
Datetime
date
Date
a
Int64
b
String
c
Int64
d
Float64
e
Boolean
f
String
82016-01-17 11:27:002016-01-1745-boe-63921035.64Falselow
\n\n
\n```\n:::\n:::\n\n\nThe `Validate` class also has a method for getting the sundered data, which is the data that\npassed or failed the validation steps. This can be done using the\n[`get_sundered_data()`](`pointblank.Validate.get_sundered_data`) method.\n\n::: {#e9a2116b .cell execution_count=9}\n``` {.python .cell-code}\npb.preview(validation_2.get_sundered_data())\n```\n\n::: {.cell-output .cell-output-display execution_count=9}\n```{=html}\n
\n\n\n\n \n \n \n \n \n \n \n \n \n\n\n\n\n \n \n \n\n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
PolarsRows12Columns8
date_time
Datetime
date
Date
a
Int64
b
String
c
Int64
d
Float64
e
Boolean
f
String
12016-01-04 11:00:002016-01-0421-bcd-34533423.29Truehigh
22016-01-04 00:32:002016-01-0435-egh-16389999.99Truelow
32016-01-05 13:32:002016-01-0568-kdg-93832343.23Truehigh
42016-01-06 17:23:002016-01-0625-jdo-903None3892.4Falsemid
52016-01-09 12:36:002016-01-0983-ldm-0387283.94Truelow
82016-01-20 04:30:002016-01-2035-bce-6429837.93Falsehigh
92016-01-20 04:30:002016-01-2035-bce-6429837.93Falsehigh
102016-01-26 20:07:002016-01-2642-dmx-0107833.98Truelow
112016-01-28 02:51:002016-01-2827-dmx-0108108.34Falselow
122016-01-30 11:23:002016-01-3013-dka-303None2230.09Truehigh
\n\n
\n```\n:::\n:::\n\n\nThe sundered data is a DataFrame that contains the rows that passed or failed the validation.\nThe default behavior is to return the rows that failed the validation, as shown above.\n\n### Working with CSV Files\n\nThe `Validate` class can directly accept CSV file paths, making it easy to validate data stored\nin CSV files without manual loading:\n\n::: {#d7cc7b60 .cell execution_count=10}\n``` {.python .cell-code}\n# Get a path to a CSV file from the package data\ncsv_path = pb.get_data_path(\"global_sales\", \"csv\")\n\nvalidation_3 = (\n pb.Validate(\n data=csv_path,\n label=\"CSV validation example\"\n )\n .col_exists([\"customer_id\", \"product_id\", \"revenue\"])\n .col_vals_not_null([\"customer_id\", \"product_id\"])\n .col_vals_gt(columns=\"revenue\", value=0)\n .interrogate()\n)\n\nvalidation_3\n```\n\n::: {.cell-output .cell-output-display execution_count=10}\n```{=html}\n
\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
Pointblank Validation
CSV validation example
Polars
STEPCOLUMNSVALUESTBLEVALUNITSPASSFAILWECEXT
#4CA64C1\n
\n \n\n col_exists\n \n \n \n \n \n \n \n\n
\n
\n
col_exists()
\n
\n \n 		customer_id—\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		————
#4CA64C2\n
\n \n\n col_exists\n \n \n \n \n \n \n \n\n
\n
\n
col_exists()
\n
\n \n 		product_id—\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		————
#4CA64C3\n
\n \n\n col_exists\n \n \n \n \n \n \n \n\n
\n
\n
col_exists()
\n
\n \n 		revenue—\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		————
#4CA64C664\n
\n \n\n col_vals_not_null\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_null()
\n
\n \n 		customer_id—\n \n \n \n \n \n \n \n✓50.0K49.7K
0.99		334
0.01		———
#4CA64C665\n
\n \n\n col_vals_not_null\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_null()
\n
\n \n 		product_id—\n \n \n \n \n \n \n \n✓50.0K49.7K
0.99		335
0.01		———
#4CA64C6\n
\n \n\n col_vals_gt\n \n \n \n \n \n \n\n
\n
\n
col_vals_gt()
\n
\n \n 		revenue0\n \n \n \n \n \n \n \n✓50.0K50.0K
1.00		0
0.00		————
2026-07-22 23:20:59 UTC< 1 s2026-07-22 23:20:59 UTC
\n\n
\n```\n:::\n:::\n\n\nYou can also use a Path object to specify the CSV file. Here's an example of how to do that:\n\n::: {#dbc8d126 .cell execution_count=11}\n``` {.python .cell-code}\nfrom pathlib import Path\n\ncsv_file = Path(pb.get_data_path(\"game_revenue\", \"csv\"))\n\nvalidation_4 = (\n pb.Validate(data=csv_file, label=\"Game Revenue Validation\")\n .col_exists([\"player_id\", \"session_id\", \"item_name\"])\n .col_vals_regex(\n columns=\"session_id\",\n pattern=r\"[A-Z0-9]{8}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{12}\"\n )\n .col_vals_gt(columns=\"item_revenue\", value=0, na_pass=True)\n .interrogate()\n)\n\nvalidation_4\n```\n\n::: {.cell-output .cell-output-display execution_count=11}\n```{=html}\n
\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
Pointblank Validation
Game Revenue Validation
Polars
STEPCOLUMNSVALUESTBLEVALUNITSPASSFAILWECEXT
#4CA64C1\n
\n \n\n col_exists\n \n \n \n \n \n \n \n\n
\n
\n
col_exists()
\n
\n \n 		player_id—\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		————
#4CA64C2\n
\n \n\n col_exists\n \n \n \n \n \n \n \n\n
\n
\n
col_exists()
\n
\n \n 		session_id—\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		————
#4CA64C3\n
\n \n\n col_exists\n \n \n \n \n \n \n \n\n
\n
\n
col_exists()
\n
\n \n 		item_name—\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		————
#4CA64C664\n
\n \n\n col_vals_regex\n \n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_regex()
\n
\n \n 		session_id[A-Z0-9]{8}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{12}\n \n \n \n \n \n \n \n✓20000
0.00		2000
1.00		———
#4CA64C5\n
\n \n\n col_vals_gt\n \n \n \n \n \n \n\n
\n
\n
col_vals_gt()
\n
\n \n 		item_revenue0\n \n \n \n \n \n \n \n✓20002000
1.00		0
0.00		————
2026-07-22 23:20:59 UTC< 1 s2026-07-22 23:20:59 UTC
\n\n
\n```\n:::\n:::\n\n\nThe CSV loading is automatic, so when a string or Path with a `.csv` extension is provided,\nPointblank will automatically load the file using the best available DataFrame library (Polars\npreferred, Pandas as fallback). The loaded data can then be used with all validation methods\njust like any other supported table type.\n\n### Working with Parquet Files\n\nThe `Validate` class can directly accept Parquet files and datasets in various formats. The\nfollowing examples illustrate how to validate Parquet files:\n\n::: {#fed5c098 .cell execution_count=12}\n``` {.python .cell-code}\n# Single Parquet file from package data\nparquet_path = pb.get_data_path(\"nycflights\", \"parquet\")\n\nvalidation_5 = (\n pb.Validate(\n data=parquet_path,\n tbl_name=\"NYC Flights Data\"\n )\n .col_vals_not_null([\"carrier\", \"origin\", \"dest\"])\n .col_vals_gt(columns=\"distance\", value=0)\n .interrogate()\n)\n\nvalidation_5\n```\n\n::: {.cell-output .cell-output-display execution_count=12}\n```{=html}\n
\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
Pointblank Validation
2026-07-22|23:20:59
PolarsNYC Flights Data
STEPCOLUMNSVALUESTBLEVALUNITSPASSFAILWECEXT
#4CA64C1\n
\n \n\n col_vals_not_null\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_null()
\n
\n \n 		carrier—\n \n \n \n \n \n \n \n✓337K337K
1.00		0
0.00		————
#4CA64C2\n
\n \n\n col_vals_not_null\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_null()
\n
\n \n 		origin—\n \n \n \n \n \n \n \n✓337K337K
1.00		0
0.00		————
#4CA64C3\n
\n \n\n col_vals_not_null\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_null()
\n
\n \n 		dest—\n \n \n \n \n \n \n \n✓337K337K
1.00		0
0.00		————
#4CA64C4\n
\n \n\n col_vals_gt\n \n \n \n \n \n \n\n
\n
\n
col_vals_gt()
\n
\n \n 		distance0\n \n \n \n \n \n \n \n✓337K337K
1.00		0
0.00		————
2026-07-22 23:20:59 UTC< 1 s2026-07-22 23:20:59 UTC
\n\n
\n```\n:::\n:::\n\n\nYou can also use glob patterns and directories. Here are some examples for how to:\n\n1. load multiple Parquet files\n2. load a Parquet-containing directory\n3. load a partitioned Parquet dataset\n\n```python\n# Multiple Parquet files with glob patterns\nvalidation_6 = pb.Validate(data=\"data/sales_*.parquet\")\n\n# Directory containing Parquet files\nvalidation_7 = pb.Validate(data=\"parquet_data/\")\n\n# Partitioned Parquet dataset\nvalidation_8 = (\n pb.Validate(data=\"sales_data/\") # Contains year=2023/quarter=Q1/region=US/sales.parquet\n .col_exists([\"transaction_id\", \"amount\", \"year\", \"quarter\", \"region\"])\n .interrogate()\n)\n```\n\nWhen you point to a directory that contains a partitioned Parquet dataset (with subdirectories\nlike `year=2023/quarter=Q1/region=US/`), Pointblank will automatically:\n\n- discover all Parquet files recursively\n- extract partition column values from directory paths\n- add partition columns to the final DataFrame\n- combine all partitions into a single table for validation\n\nBoth Polars and Pandas handle partitioned datasets natively, so this works seamlessly with\neither DataFrame library. The loading preference is Polars first, then Pandas as a fallback.\n\n### Working with Database Connection Strings\n\nThe `Validate` class supports database connection strings for direct validation of database\ntables. Connection strings must specify a table using the `::table_name` suffix:\n\n::: {#37c4a2ad .cell execution_count=13}\n``` {.python .cell-code}\n# Get path to a DuckDB database file from package data\nduckdb_path = pb.get_data_path(\"game_revenue\", \"duckdb\")\n\nvalidation_9 = (\n pb.Validate(\n data=f\"duckdb:///{duckdb_path}::game_revenue\",\n label=\"DuckDB Game Revenue Validation\"\n )\n .col_exists([\"player_id\", \"session_id\", \"item_revenue\"])\n .col_vals_gt(columns=\"item_revenue\", value=0)\n .interrogate()\n)\n\nvalidation_9\n```\n\n::: {.cell-output .cell-output-display execution_count=13}\n```{=html}\n
\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
Pointblank Validation
DuckDB Game Revenue Validation
DuckDB
STEPCOLUMNSVALUESTBLEVALUNITSPASSFAILWECEXT
#4CA64C1\n
\n \n\n col_exists\n \n \n \n \n \n \n \n\n
\n
\n
col_exists()
\n
\n \n 		player_id—\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		————
#4CA64C2\n
\n \n\n col_exists\n \n \n \n \n \n \n \n\n
\n
\n
col_exists()
\n
\n \n 		session_id—\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		————
#4CA64C3\n
\n \n\n col_exists\n \n \n \n \n \n \n \n\n
\n
\n
col_exists()
\n
\n \n 		item_revenue—\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		————
#4CA64C4\n
\n \n\n col_vals_gt\n \n \n \n \n \n \n\n
\n
\n
col_vals_gt()
\n
\n \n 		item_revenue0\n \n \n \n \n \n \n \n✓20002000
1.00		0
0.00		————
2026-07-22 23:21:00 UTC< 1 s2026-07-22 23:21:00 UTC
\n\n
\n```\n:::\n:::\n\n\nFor comprehensive documentation on supported connection string formats, error handling, and\ninstallation requirements, see the [`connect_to_table()`](`pointblank.connect_to_table`)\nfunction. This function handles all the connection logic and provides helpful error messages\nwhen table specifications are missing or backend dependencies are not installed.\n\n", + "markdown": "---\ntitle: \"[Validate]{.doc-object-name .doc-class .doc-label .doc-label-dataclass}\"\nbody-classes: doc-api-page\npage-navigation: false\nhtml-table-processing: none\n---\n\n::: {.doc-subject}\n\nWorkflow for defining a set of validations on a table and interrogating for results.\n:::\n\n::: {.doc-usage-source}\nUsage\n\n[Source](https://github.com/posit-dev/pointblank/blob/main/pointblank/validate.py#L5066-L21284){target=\"_blank\" rel=\"noopener\"}\n:::\n\n::: {.doc-signature .doc-Kind.CLASS}\n```python\nValidate(\n data,\n reference=None,\n tbl_name=None,\n label=None,\n thresholds=None,\n actions=None,\n final_actions=None,\n brief=None,\n lang=None,\n locale=None,\n owner=None,\n consumers=None,\n version=None\n)\n```\n:::\n\n::: {.doc-text}\n\nThe `Validate` class is used for defining a set of validation steps on a table and interrogating\nthe table with the *validation plan*. This class is the main entry point for the *data quality\nreporting* workflow. The overall aim of this workflow is to generate comprehensive reporting\ninformation to assess the level of data quality for a target table.\n\nWe can supply as many validation steps as needed, and having a large number of them should\nincrease the validation coverage for a given table. The validation methods (e.g.,\n[`col_vals_gt()`](`pointblank.Validate.col_vals_gt`),\n[`col_vals_between()`](`pointblank.Validate.col_vals_between`), etc.) translate to discrete\nvalidation steps, where each step will be sequentially numbered (useful when viewing the\nreporting data). This process of calling validation methods is known as developing a\n*validation plan*.\n\nThe validation methods, when called, are merely instructions up to the point the concluding\n[`interrogate()`](`pointblank.Validate.interrogate`) method is called. That kicks off the\nprocess of acting on the *validation plan* by querying the target table getting reporting\nresults for each step. Once the interrogation process is complete, we can say that the workflow\nnow has reporting information. We can then extract useful information from the reporting data\nto understand the quality of the table. Printing the `Validate` object (or using the\n[`get_tabular_report()`](`pointblank.Validate.get_tabular_report`) method) will return a table\nwith the results of the interrogation and\n[`get_sundered_data()`](`pointblank.Validate.get_sundered_data`) allows for the splitting of the\ntable based on passing and failing rows.\n:::\n\n## Parameters {.doc-parameters}\n\n::: {.doc-definition-items}\n[data]{.doc-parameter-name}[:]{.doc-parameter-annotation-sep} [IntoDataFrame]{.doc-parameter-annotation}\n\n: The table to validate, which could be a DataFrame object, an Ibis table object, a CSV\n file path, a Parquet file path, a GitHub URL pointing to a CSV or Parquet file, or a\n database connection string. When providing a CSV or Parquet file path (as a string or\n `pathlib.Path` object), the file will be automatically loaded using an available DataFrame\n library (Polars or Pandas). Parquet input also supports glob patterns, directories\n containing .parquet files, and Spark-style partitioned datasets. GitHub URLs are\n automatically transformed to raw content URLs and downloaded. Connection strings enable\n direct database access via Ibis with optional table specification using the `::table_name`\n suffix. Read the *Supported Input Table Types* section for details on the supported table\n types.\n\n[tbl_name]{.doc-parameter-name}[:]{.doc-parameter-annotation-sep} [str | None]{.doc-parameter-annotation} [=]{.doc-parameter-default-sep .op} [None]{.doc-parameter-default}\n\n: An optional name to assign to the input table object. If no value is provided, a name will\n be generated based on whatever information is available. This table name will be displayed\n in the header area of the tabular report.\n\n[label]{.doc-parameter-name}[:]{.doc-parameter-annotation-sep} [str | None]{.doc-parameter-annotation} [=]{.doc-parameter-default-sep .op} [None]{.doc-parameter-default}\n\n: An optional label for the validation plan. If no value is provided, a label will be\n generated based on the current system date and time. Markdown can be used here to make the\n label more visually appealing (it will appear in the header area of the tabular report).\n\n[thresholds]{.doc-parameter-name}[:]{.doc-parameter-annotation-sep} [int | float | bool | tuple | dict | Thresholds | None]{.doc-parameter-annotation} [=]{.doc-parameter-default-sep .op} [None]{.doc-parameter-default}\n\n: Generate threshold failure levels so that all validation steps can report and react\n accordingly when exceeding the set levels. The thresholds are set at the global level and\n can be overridden at the validation step level (each validation step has its own\n `thresholds=` parameter). The default is `None`, which means that no thresholds will be set.\n Look at the *Thresholds* section for information on how to set threshold levels.\n\n[actions]{.doc-parameter-name}[:]{.doc-parameter-annotation-sep} [Actions | None]{.doc-parameter-annotation} [=]{.doc-parameter-default-sep .op} [None]{.doc-parameter-default}\n\n: The actions to take when validation steps meet or exceed any set threshold levels. These\n actions are paired with the threshold levels and are executed during the interrogation\n process when there are exceedances. The actions are executed right after each step is\n evaluated. Such actions should be provided in the form of an `Actions` object. If `None`\n then no global actions will be set. View the *Actions* section for information on how to set\n actions.\n\n[final_actions]{.doc-parameter-name}[:]{.doc-parameter-annotation-sep} [FinalActions | None]{.doc-parameter-annotation} [=]{.doc-parameter-default-sep .op} [None]{.doc-parameter-default}\n\n: The actions to take when the validation process is complete and the final results are\n available. This is useful for sending notifications or reporting the overall status of the\n validation process. The final actions are executed after all validation steps have been\n processed and the results have been collected. The final actions are not tied to any\n threshold levels, they are executed regardless of the validation results. Such actions\n should be provided in the form of a `FinalActions` object. If `None` then no finalizing\n actions will be set. Please see the *Actions* section for information on how to set final\n actions.\n\n[brief]{.doc-parameter-name}[:]{.doc-parameter-annotation-sep} [str | bool | None]{.doc-parameter-annotation} [=]{.doc-parameter-default-sep .op} [None]{.doc-parameter-default}\n\n: A global setting for briefs, which are optional brief descriptions for validation steps\n (they be displayed in the reporting table). For such a global setting, templating elements\n like `\"{step}\"` (to insert the step number) or `\"{auto}\"` (to include an automatically\n generated brief) are useful. If `True` then each brief will be automatically generated. If\n `None` (the default) then briefs aren't globally set.\n\n[lang]{.doc-parameter-name}[:]{.doc-parameter-annotation-sep} [str | None]{.doc-parameter-annotation} [=]{.doc-parameter-default-sep .op} [None]{.doc-parameter-default}\n\n: The language to use for various reporting elements. By default, `None` will select English\n (`\"en\"`) as the but other options include French (`\"fr\"`), German (`\"de\"`), Italian\n (`\"it\"`), Spanish (`\"es\"`), and several more. Have a look at the *Reporting Languages*\n section for the full list of supported languages and information on how the language setting\n is utilized.\n\n[locale]{.doc-parameter-name}[:]{.doc-parameter-annotation-sep} [str | None]{.doc-parameter-annotation} [=]{.doc-parameter-default-sep .op} [None]{.doc-parameter-default}\n\n: An optional locale ID to use for formatting values in the reporting table according the\n locale's rules. Examples include `\"en-US\"` for English (United States) and `\"fr-FR\"` for\n French (France). More simply, this can be a language identifier without a designation of\n territory, like `\"es\"` for Spanish.\n\n[owner]{.doc-parameter-name}[:]{.doc-parameter-annotation-sep} [str | None]{.doc-parameter-annotation} [=]{.doc-parameter-default-sep .op} [None]{.doc-parameter-default}\n\n: An optional string identifying the owner of the data being validated. This is useful for\n governance purposes, indicating who is responsible for the quality and maintenance of the\n data. For example, `\"data-platform-team\"` or `\"analytics-engineering\"`.\n\n[consumers]{.doc-parameter-name}[:]{.doc-parameter-annotation-sep} [str | list[str] | None]{.doc-parameter-annotation} [=]{.doc-parameter-default-sep .op} [None]{.doc-parameter-default}\n\n: An optional string or list of strings identifying who depends on or consumes this data.\n This helps document data dependencies and can be useful for impact analysis when data\n quality issues are detected. For example, `\"ml-team\"` or `[\"ml-team\", \"analytics\"]`.\n\n[version]{.doc-parameter-name}[:]{.doc-parameter-annotation-sep} [str | None]{.doc-parameter-annotation} [=]{.doc-parameter-default-sep .op} [None]{.doc-parameter-default}\n\n: An optional string representing the version of the validation plan or data contract. This\n supports semantic versioning (e.g., `\"1.0.0\"`, `\"2.1.0\"`) and is useful for tracking changes\n to validation rules over time and for organizational governance.\n:::\n\n## Returns {.doc-returns}\n\n::: {.doc-definition-items}\n [Validate]{.doc-parameter-annotation}\n\n: A `Validate` object with the table and validations to be performed.\n:::\n\n## Supported Input Table Types {.doc-supported-input-table-types}\n\nThe `data=` parameter can be given any of the following table types:\n\n- Polars DataFrame (`\"polars\"`)\n- Pandas DataFrame (`\"pandas\"`)\n- PySpark table (`\"pyspark\"`)\n- DuckDB table (`\"duckdb\"`)*\n- MySQL table (`\"mysql\"`)*\n- PostgreSQL table (`\"postgresql\"`)*\n- SQLite table (`\"sqlite\"`)*\n- Microsoft SQL Server table (`\"mssql\"`)*\n- Snowflake table (`\"snowflake\"`)*\n- Databricks table (`\"databricks\"`)*\n- BigQuery table (`\"bigquery\"`)*\n- Parquet table (`\"parquet\"`)*\n- CSV files (string path or `pathlib.Path` object with `.csv` extension)\n- Parquet files (string path, `pathlib.Path` object, glob pattern, directory with `.parquet`\nextension, or partitioned dataset)\n- Database connection strings (URI format with optional table specification)\n\nThe table types marked with an asterisk need to be prepared as Ibis tables (with type of\n`ibis.expr.types.relations.Table`). Furthermore, the use of `Validate` with such tables requires\nthe Ibis library v9.5.0 and above to be installed. If the input table is a Polars or Pandas\nDataFrame, the Ibis library is not required.\n\nTo use a CSV file, ensure that a string or `pathlib.Path` object with a `.csv` extension is\nprovided. The file will be automatically detected and loaded using the best available DataFrame\nlibrary. The loading preference is Polars first, then Pandas as a fallback.\n\nConnection strings follow database URL formats and must also specify a table using the\n`::table_name` suffix. Examples include:\n\n```\n\"duckdb:///path/to/database.ddb::table_name\"\n\"sqlite:///path/to/database.db::table_name\"\n\"postgresql://user:password@localhost:5432/database::table_name\"\n\"mysql://user:password@localhost:3306/database::table_name\"\n\"bigquery://project/dataset::table_name\"\n\"snowflake://user:password@account/database/schema::table_name\"\n```\n\nWhen using connection strings, the Ibis library with the appropriate backend driver is required.\n\n## Thresholds {.doc-thresholds}\n\nThe `thresholds=` parameter is used to set the failure-condition levels for all validation\nsteps. They are set here at the global level but can be overridden at the validation step level\n(each validation step has its own local `thresholds=` parameter).\n\nThere are three threshold levels: 'warning', 'error', and 'critical'. The threshold values can\neither be set as a proportion failing of all test units (a value between `0` to `1`), or, the\nabsolute number of failing test units (as integer that's `1` or greater).\n\nThresholds can be defined using one of these input schemes:\n\n1. use the [`Thresholds`](`pointblank.Thresholds`) class (the most direct way to create\nthresholds)\n2. provide a tuple of 1-3 values, where position `0` is the 'warning' level, position `1` is the\n'error' level, and position `2` is the 'critical' level\n3. create a dictionary of 1-3 value entries; the valid keys: are 'warning', 'error', and\n'critical'\n4. a single integer/float value denoting absolute number or fraction of failing test units for\nthe 'warning' level only\n\nIf the number of failing test units for a validation step exceeds set thresholds, the validation\nstep will be marked as 'warning', 'error', or 'critical'. All of the threshold levels don't need\nto be set, you're free to set any combination of them.\n\nAside from reporting failure conditions, thresholds can be used to determine the actions to take\nfor each level of failure (using the `actions=` parameter).\n\n## Actions {.doc-actions}\n\nThe `actions=` and `final_actions=` parameters provide mechanisms to respond to validation\nresults. These actions can be used to notify users of validation failures, log issues, or\ntrigger other processes when problems are detected.\n\n*Step Actions*\n\nThe `actions=` parameter allows you to define actions that are triggered when validation steps\nexceed specific threshold levels (warning, error, or critical). These actions are executed\nduring the interrogation process, right after each step is evaluated.\n\nStep actions should be provided using the [`Actions`](`pointblank.Actions`) class, which lets\nyou specify different actions for different severity levels:\n\n```python\n# Define an action that logs a message when warning threshold is exceeded\ndef log_warning():\n metadata = pb.get_action_metadata()\n print(f\"WARNING: Step {metadata['step']} failed with type {metadata['type']}\")\n\n# Define actions for different threshold levels\nactions = pb.Actions(\n warning = log_warning,\n error = lambda: send_email(\"Error in validation\"),\n critical = \"CRITICAL FAILURE DETECTED\"\n)\n\n# Use in Validate\nvalidation = pb.Validate(\n data=my_data,\n actions=actions # Global actions for all steps\n)\n```\n\nYou can also provide step-specific actions in individual validation methods:\n\n```python\nvalidation.col_vals_gt(\n columns=\"revenue\",\n value=0,\n actions=pb.Actions(warning=log_warning) # Only applies to this step\n)\n```\n\nStep actions have access to step-specific context through the\n[`get_action_metadata()`](`pointblank.get_action_metadata`) function, which provides details\nabout the current validation step that triggered the action.\n\n*Final Actions*\n\nThe `final_actions=` parameter lets you define actions that execute after all validation steps\nhave completed. These are useful for providing summaries, sending notifications based on\noverall validation status, or performing cleanup operations.\n\nFinal actions should be provided using the [`FinalActions`](`pointblank.FinalActions`) class:\n\n```python\ndef send_report():\n summary = pb.get_validation_summary()\n if summary[\"status\"] == \"CRITICAL\":\n send_alert_email(\n subject=f\"CRITICAL validation failures in {summary['tbl_name']}\",\n body=f\"{summary['critical_steps']} steps failed with critical severity.\"\n )\n\nvalidation = pb.Validate(\n data=my_data,\n final_actions=pb.FinalActions(send_report)\n)\n```\n\nFinal actions have access to validation-wide summary information through the\n[`get_validation_summary()`](`pointblank.get_validation_summary`) function, which provides a\ncomprehensive overview of the entire validation process.\n\nThe combination of step actions and final actions provides a flexible system for responding to\ndata quality issues at both the individual step level and the overall validation level.\n\n## Reporting Languages {.doc-reporting-languages}\n\nVarious pieces of reporting in Pointblank can be localized to a specific language. This is done\nby setting the `lang=` parameter in `Validate`. Any of the following languages can be used (just\nprovide the language code):\n\n- English (`\"en\"`)\n- French (`\"fr\"`)\n- German (`\"de\"`)\n- Italian (`\"it\"`)\n- Spanish (`\"es\"`)\n- Portuguese (`\"pt\"`)\n- Dutch (`\"nl\"`)\n- Swedish (`\"sv\"`)\n- Danish (`\"da\"`)\n- Norwegian Bokmål (`\"nb\"`)\n- Icelandic (`\"is\"`)\n- Finnish (`\"fi\"`)\n- Polish (`\"pl\"`)\n- Czech (`\"cs\"`)\n- Romanian (`\"ro\"`)\n- Greek (`\"el\"`)\n- Russian (`\"ru\"`)\n- Turkish (`\"tr\"`)\n- Arabic (`\"ar\"`)\n- Hindi (`\"hi\"`)\n- Simplified Chinese (`\"zh-Hans\"`)\n- Traditional Chinese (`\"zh-Hant\"`)\n- Japanese (`\"ja\"`)\n- Korean (`\"ko\"`)\n- Vietnamese (`\"vi\"`)\n- Indonesian (`\"id\"`)\n- Ukrainian (`\"uk\"`)\n- Bulgarian (`\"bg\"`)\n- Croatian (`\"hr\"`)\n- Estonian (`\"et\"`)\n- Hungarian (`\"hu\"`)\n- Irish (`\"ga\"`)\n- Latvian (`\"lv\"`)\n- Lithuanian (`\"lt\"`)\n- Maltese (`\"mt\"`)\n- Slovak (`\"sk\"`)\n- Slovenian (`\"sl\"`)\n- Hebrew (`\"he\"`)\n- Thai (`\"th\"`)\n- Persian (`\"fa\"`)\n\nAutomatically generated briefs (produced by using `brief=True` or `brief=\"...{auto}...\"`) will\nbe written in the selected language. The language setting will also used when generating the\nvalidation report table through\n[`get_tabular_report()`](`pointblank.Validate.get_tabular_report`) (or printing the `Validate`\nobject in a notebook environment).\n\n## Examples {.doc-examples}\n\n### Creating a validation plan and interrogating\n\nLet's walk through a data quality analysis of an extremely small table. It's actually called\n`\"small_table\"` and it's accessible through the [`load_dataset()`](`pointblank.load_dataset`)\nfunction.\n\n::: {#83e28644 .cell execution_count=1}\n``` {.python .cell-code}\nimport pointblank as pb\n\n# Load the `small_table` dataset\nsmall_table = pb.load_dataset(dataset=\"small_table\", tbl_type=\"polars\")\n\n# Preview the table\npb.preview(small_table)\n```\n\n::: {.cell-output .cell-output-display execution_count=1}\n```{=html}\n
\n\n\n\n \n \n \n \n \n \n \n \n \n\n\n\n\n \n \n \n\n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
PolarsRows13Columns8
date_time
Datetime
date
Date
a
Int64
b
String
c
Int64
d
Float64
e
Boolean
f
String
12016-01-04 11:00:002016-01-0421-bcd-34533423.29Truehigh
22016-01-04 00:32:002016-01-0435-egh-16389999.99Truelow
32016-01-05 13:32:002016-01-0568-kdg-93832343.23Truehigh
42016-01-06 17:23:002016-01-0625-jdo-903None3892.4Falsemid
52016-01-09 12:36:002016-01-0983-ldm-0387283.94Truelow
92016-01-20 04:30:002016-01-2035-bce-6429837.93Falsehigh
102016-01-20 04:30:002016-01-2035-bce-6429837.93Falsehigh
112016-01-26 20:07:002016-01-2642-dmx-0107833.98Truelow
122016-01-28 02:51:002016-01-2827-dmx-0108108.34Falselow
132016-01-30 11:23:002016-01-3013-dka-303None2230.09Truehigh
\n\n
\n```\n:::\n:::\n\n\nWe ought to think about what's tolerable in terms of data quality so let's designate\nproportional failure thresholds to the 'warning', 'error', and 'critical' states. This can be\ndone by using the [`Thresholds`](`pointblank.Thresholds`) class.\n\n::: {#6db0dd40 .cell execution_count=2}\n``` {.python .cell-code}\nthresholds = pb.Thresholds(warning=0.10, error=0.25, critical=0.35)\n```\n:::\n\n\nNow, we use the `Validate` class and give it the `thresholds` object (which serves as a default\nfor all validation steps but can be overridden). The static thresholds provided in `thresholds=`\nwill make the reporting a bit more useful. We also need to provide a target table and we'll use\n`small_table` for this.\n\n::: {#90babf95 .cell execution_count=3}\n``` {.python .cell-code}\nvalidation = (\n pb.Validate(\n data=small_table,\n tbl_name=\"small_table\",\n label=\"`Validate` example.\",\n thresholds=thresholds\n )\n)\n```\n:::\n\n\nThen, as with any `Validate` object, we can add steps to the validation plan by using as many\nvalidation methods as we want. To conclude the process (and actually query the data table), we\nuse the [`interrogate()`](`pointblank.Validate.interrogate`) method.\n\n::: {#2bb2902a .cell execution_count=4}\n``` {.python .cell-code}\nvalidation = (\n validation\n .col_vals_gt(columns=\"d\", value=100)\n .col_vals_le(columns=\"c\", value=5)\n .col_vals_between(columns=\"c\", left=3, right=10, na_pass=True)\n .col_vals_regex(columns=\"b\", pattern=r\"[0-9]-[a-z]{3}-[0-9]{3}\")\n .col_exists(columns=[\"date\", \"date_time\"])\n .interrogate()\n)\n```\n:::\n\n\nThe `validation` object can be printed as a reporting table.\n\n::: {#1511c989 .cell execution_count=5}\n``` {.python .cell-code}\nvalidation\n```\n\n::: {.cell-output .cell-output-display execution_count=5}\n```{=html}\n
\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
Pointblank Validation
`Validate` example.
Polarssmall_tableWARNING0.1ERROR0.25CRITICAL0.35
STEPCOLUMNSVALUESTBLEVALUNITSPASSFAILWECEXT
#4CA64C1\n
\n \n\n col_vals_gt\n \n \n \n \n \n \n\n
\n
\n
col_vals_gt()
\n
\n \n 		d100\n \n \n \n \n \n \n \n✓1313
1.00		0
0.00		○○○—
#FF33002\n
\n \n\n col_vals_le\n \n \n \n \n \n \n\n
\n
\n
col_vals_le()
\n
\n \n 		c5\n \n \n \n \n \n \n \n✓135
0.38		8
0.62		●●●
#4CA64C663\n
\n \n\n col_vals_between\n \n \n \n \n \n \n\n
\n
\n
col_vals_between()
\n
\n \n 		c[3, 10]\n \n \n \n \n \n \n \n✓1312
0.92		1
0.08		○○○
#4CA64C4\n
\n \n\n col_vals_regex\n \n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_regex()
\n
\n \n 		b[0-9]-[a-z]{3}-[0-9]{3}\n \n \n \n \n \n \n \n✓1313
1.00		0
0.00		○○○—
#4CA64C5\n
\n \n\n col_exists\n \n \n \n \n \n \n \n\n
\n
\n
col_exists()
\n
\n \n 		date—\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		○○○—
#4CA64C6\n
\n \n\n col_exists\n \n \n \n \n \n \n \n\n
\n
\n
col_exists()
\n
\n \n 		date_time—\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		○○○—
2026-08-10 16:44:28 UTC< 1 s2026-08-10 16:44:28 UTC
\n\n
\n```\n:::\n:::\n\n\nThe report could be further customized by using the\n[`get_tabular_report()`](`pointblank.Validate.get_tabular_report`) method, which contains\noptions for modifying the display of the table.\n\n### Adding briefs\n\nBriefs are short descriptions of the validation steps. While they can be set for each step\nindividually, they can also be set globally. The global setting is done by using the\n`brief=` argument in `Validate`. The global setting can be as simple as `True` to have\nautomatically-generated briefs for each step. Alternatively, we can use templating elements\nlike `\"{step}\"` (to insert the step number) or `\"{auto}\"` (to include an automatically generated\nbrief). Here's an example of a global setting for briefs:\n\n::: {#8056522a .cell execution_count=6}\n``` {.python .cell-code}\nvalidation_2 = (\n pb.Validate(\n data=pb.load_dataset(),\n tbl_name=\"small_table\",\n label=\"Validation example with briefs\",\n brief=\"Step {step}: {auto}\",\n )\n .col_vals_gt(columns=\"d\", value=100)\n .col_vals_between(columns=\"c\", left=3, right=10, na_pass=True)\n .col_vals_regex(\n columns=\"b\",\n pattern=r\"[0-9]-[a-z]{3}-[0-9]{3}\",\n brief=\"Regex check for column {col}\"\n )\n .interrogate()\n)\n\nvalidation_2\n```\n\n::: {.cell-output .cell-output-display execution_count=6}\n```{=html}\n
\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
Pointblank Validation
Validation example with briefs
Polarssmall_table
STEPCOLUMNSVALUESTBLEVALUNITSPASSFAILWECEXT
#4CA64C1\n
\n \n\n col_vals_gt\n \n \n \n \n \n \n\n
\n
\n
col_vals_gt()
\n
\n

Step 1: Expect that values in d should be > 100.

\n
\n 		d100\n \n \n \n \n \n \n \n✓1313
1.00		0
0.00		————
#4CA64C662\n
\n \n\n col_vals_between\n \n \n \n \n \n \n\n
\n
\n
col_vals_between()
\n
\n

Step 2: Expect that values in c should be between 3 and 10.

\n
\n 		c[3, 10]\n \n \n \n \n \n \n \n✓1312
0.92		1
0.08		———
#4CA64C3\n
\n \n\n col_vals_regex\n \n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_regex()
\n
\n

Regex check for column b

\n
\n 		b[0-9]-[a-z]{3}-[0-9]{3}\n \n \n \n \n \n \n \n✓1313
1.00		0
0.00		————
2026-08-10 16:44:28 UTC< 1 s2026-08-10 16:44:28 UTC
\n\n
\n```\n:::\n:::\n\n\nWe see the text of the briefs appear in the `STEP` column of the reporting table. Furthermore,\nthe global brief's template (`\"Step {step}: {auto}\"`) is applied to all steps except for the\nfinal step, where the step-level `brief=` argument provided an override.\n\nIf you should want to cancel the globally-defined brief for one or more validation steps, you\ncan set `brief=False` in those particular steps.\n\n### Post-interrogation methods\n\nThe `Validate` class has a number of post-interrogation methods that can be used to extract\nuseful information from the validation results. For example, the\n[`get_data_extracts()`](`pointblank.Validate.get_data_extracts`) method can be used to get\nthe data extracts for each validation step.\n\n::: {#1c0ff79f .cell execution_count=7}\n``` {.python .cell-code}\nvalidation_2.get_data_extracts()\n```\n\n::: {.cell-output .cell-output-display execution_count=7}\n```\n{1: shape: (0, 9)\n ┌───────────┬──────────────┬──────┬─────┬───┬─────┬─────┬──────┬─────┐\n │ _row_num_ ┆ date_time ┆ date ┆ a ┆ … ┆ c ┆ d ┆ e ┆ f │\n │ --- ┆ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │\n │ u32 ┆ datetime[μs] ┆ date ┆ i64 ┆ ┆ i64 ┆ f64 ┆ bool ┆ str │\n ╞═══════════╪══════════════╪══════╪═════╪═══╪═════╪═════╪══════╪═════╡\n └───────────┴──────────────┴──────┴─────┴───┴─────┴─────┴──────┴─────┘,\n 2: shape: (1, 9)\n ┌───────────┬─────────────────────┬────────────┬─────┬───┬─────┬─────────┬───────┬─────┐\n │ _row_num_ ┆ date_time ┆ date ┆ a ┆ … ┆ c ┆ d ┆ e ┆ f │\n │ --- ┆ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │\n │ u32 ┆ datetime[μs] ┆ date ┆ i64 ┆ ┆ i64 ┆ f64 ┆ bool ┆ str │\n ╞═══════════╪═════════════════════╪════════════╪═════╪═══╪═════╪═════════╪═══════╪═════╡\n │ 8 ┆ 2016-01-17 11:27:00 ┆ 2016-01-17 ┆ 4 ┆ … ┆ 2 ┆ 1035.64 ┆ false ┆ low │\n └───────────┴─────────────────────┴────────────┴─────┴───┴─────┴─────────┴───────┴─────┘,\n 3: shape: (0, 9)\n ┌───────────┬──────────────┬──────┬─────┬───┬─────┬─────┬──────┬─────┐\n │ _row_num_ ┆ date_time ┆ date ┆ a ┆ … ┆ c ┆ d ┆ e ┆ f │\n │ --- ┆ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │\n │ u32 ┆ datetime[μs] ┆ date ┆ i64 ┆ ┆ i64 ┆ f64 ┆ bool ┆ str │\n ╞═══════════╪══════════════╪══════╪═════╪═══╪═════╪═════╪══════╪═════╡\n └───────────┴──────────────┴──────┴─────┴───┴─────┴─────┴──────┴─────┘}\n```\n:::\n:::\n\n\nWe can also view step reports for each validation step using the\n[`get_step_report()`](`pointblank.Validate.get_step_report`) method. This method adapts to the\ntype of validation step and shows the relevant information for a step's validation.\n\n::: {#b14f0b91 .cell execution_count=8}\n``` {.python .cell-code}\nvalidation_2.get_step_report(i=2)\n```\n\n::: {.cell-output .cell-output-display execution_count=8}\n```{=html}\n
\n\n\n\n \n \n \n \n \n \n \n \n \n\n\n\n\n \n \n \n\n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n\n\n
Report for Validation Step 2
ASSERTION 3 ≤ c ≤ 10
1 / 13 TEST UNIT FAILURES IN COLUMN 5
EXTRACT OF ALL 1 ROWS (WITH TEST UNIT FAILURES IN RED):
date_time
Datetime
date
Date
a
Int64
b
String
c
Int64
d
Float64
e
Boolean
f
String
82016-01-17 11:27:002016-01-1745-boe-63921035.64Falselow
\n\n
\n```\n:::\n:::\n\n\nThe `Validate` class also has a method for getting the sundered data, which is the data that\npassed or failed the validation steps. This can be done using the\n[`get_sundered_data()`](`pointblank.Validate.get_sundered_data`) method.\n\n::: {#8cf891cb .cell execution_count=9}\n``` {.python .cell-code}\npb.preview(validation_2.get_sundered_data())\n```\n\n::: {.cell-output .cell-output-display execution_count=9}\n```{=html}\n
\n\n\n\n \n \n \n \n \n \n \n \n \n\n\n\n\n \n \n \n\n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
PolarsRows12Columns8
date_time
Datetime
date
Date
a
Int64
b
String
c
Int64
d
Float64
e
Boolean
f
String
12016-01-04 11:00:002016-01-0421-bcd-34533423.29Truehigh
22016-01-04 00:32:002016-01-0435-egh-16389999.99Truelow
32016-01-05 13:32:002016-01-0568-kdg-93832343.23Truehigh
42016-01-06 17:23:002016-01-0625-jdo-903None3892.4Falsemid
52016-01-09 12:36:002016-01-0983-ldm-0387283.94Truelow
82016-01-20 04:30:002016-01-2035-bce-6429837.93Falsehigh
92016-01-20 04:30:002016-01-2035-bce-6429837.93Falsehigh
102016-01-26 20:07:002016-01-2642-dmx-0107833.98Truelow
112016-01-28 02:51:002016-01-2827-dmx-0108108.34Falselow
122016-01-30 11:23:002016-01-3013-dka-303None2230.09Truehigh
\n\n
\n```\n:::\n:::\n\n\nThe sundered data is a DataFrame that contains the rows that passed or failed the validation.\nThe default behavior is to return the rows that failed the validation, as shown above.\n\n### Working with CSV Files\n\nThe `Validate` class can directly accept CSV file paths, making it easy to validate data stored\nin CSV files without manual loading:\n\n::: {#3adaa774 .cell execution_count=10}\n``` {.python .cell-code}\n# Get a path to a CSV file from the package data\ncsv_path = pb.get_data_path(\"global_sales\", \"csv\")\n\nvalidation_3 = (\n pb.Validate(\n data=csv_path,\n label=\"CSV validation example\"\n )\n .col_exists([\"customer_id\", \"product_id\", \"revenue\"])\n .col_vals_not_null([\"customer_id\", \"product_id\"])\n .col_vals_gt(columns=\"revenue\", value=0)\n .interrogate()\n)\n\nvalidation_3\n```\n\n::: {.cell-output .cell-output-display execution_count=10}\n```{=html}\n
\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
Pointblank Validation
CSV validation example
Polars
STEPCOLUMNSVALUESTBLEVALUNITSPASSFAILWECEXT
#4CA64C1\n
\n \n\n col_exists\n \n \n \n \n \n \n \n\n
\n
\n
col_exists()
\n
\n \n 		customer_id—\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		————
#4CA64C2\n
\n \n\n col_exists\n \n \n \n \n \n \n \n\n
\n
\n
col_exists()
\n
\n \n 		product_id—\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		————
#4CA64C3\n
\n \n\n col_exists\n \n \n \n \n \n \n \n\n
\n
\n
col_exists()
\n
\n \n 		revenue—\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		————
#4CA64C664\n
\n \n\n col_vals_not_null\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_null()
\n
\n \n 		customer_id—\n \n \n \n \n \n \n \n✓50.0K49.7K
0.99		334
0.01		———
#4CA64C665\n
\n \n\n col_vals_not_null\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_null()
\n
\n \n 		product_id—\n \n \n \n \n \n \n \n✓50.0K49.7K
0.99		335
0.01		———
#4CA64C6\n
\n \n\n col_vals_gt\n \n \n \n \n \n \n\n
\n
\n
col_vals_gt()
\n
\n \n 		revenue0\n \n \n \n \n \n \n \n✓50.0K50.0K
1.00		0
0.00		————
2026-08-10 16:44:28 UTC< 1 s2026-08-10 16:44:29 UTC
\n\n
\n```\n:::\n:::\n\n\nYou can also use a Path object to specify the CSV file. Here's an example of how to do that:\n\n::: {#513bfbc5 .cell execution_count=11}\n``` {.python .cell-code}\nfrom pathlib import Path\n\ncsv_file = Path(pb.get_data_path(\"game_revenue\", \"csv\"))\n\nvalidation_4 = (\n pb.Validate(data=csv_file, label=\"Game Revenue Validation\")\n .col_exists([\"player_id\", \"session_id\", \"item_name\"])\n .col_vals_regex(\n columns=\"session_id\",\n pattern=r\"[A-Z0-9]{8}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{12}\"\n )\n .col_vals_gt(columns=\"item_revenue\", value=0, na_pass=True)\n .interrogate()\n)\n\nvalidation_4\n```\n\n::: {.cell-output .cell-output-display execution_count=11}\n```{=html}\n
\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
Pointblank Validation
Game Revenue Validation
Polars
STEPCOLUMNSVALUESTBLEVALUNITSPASSFAILWECEXT
#4CA64C1\n
\n \n\n col_exists\n \n \n \n \n \n \n \n\n
\n
\n
col_exists()
\n
\n \n 		player_id—\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		————
#4CA64C2\n
\n \n\n col_exists\n \n \n \n \n \n \n \n\n
\n
\n
col_exists()
\n
\n \n 		session_id—\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		————
#4CA64C3\n
\n \n\n col_exists\n \n \n \n \n \n \n \n\n
\n
\n
col_exists()
\n
\n \n 		item_name—\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		————
#4CA64C664\n
\n \n\n col_vals_regex\n \n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_regex()
\n
\n \n 		session_id[A-Z0-9]{8}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{12}\n \n \n \n \n \n \n \n✓20000
0.00		2000
1.00		———
#4CA64C5\n
\n \n\n col_vals_gt\n \n \n \n \n \n \n\n
\n
\n
col_vals_gt()
\n
\n \n 		item_revenue0\n \n \n \n \n \n \n \n✓20002000
1.00		0
0.00		————
2026-08-10 16:44:29 UTC< 1 s2026-08-10 16:44:29 UTC
\n\n
\n```\n:::\n:::\n\n\nThe CSV loading is automatic, so when a string or Path with a `.csv` extension is provided,\nPointblank will automatically load the file using the best available DataFrame library (Polars\npreferred, Pandas as fallback). The loaded data can then be used with all validation methods\njust like any other supported table type.\n\n### Working with Parquet Files\n\nThe `Validate` class can directly accept Parquet files and datasets in various formats. The\nfollowing examples illustrate how to validate Parquet files:\n\n::: {#d53f0354 .cell execution_count=12}\n``` {.python .cell-code}\n# Single Parquet file from package data\nparquet_path = pb.get_data_path(\"nycflights\", \"parquet\")\n\nvalidation_5 = (\n pb.Validate(\n data=parquet_path,\n tbl_name=\"NYC Flights Data\"\n )\n .col_vals_not_null([\"carrier\", \"origin\", \"dest\"])\n .col_vals_gt(columns=\"distance\", value=0)\n .interrogate()\n)\n\nvalidation_5\n```\n\n::: {.cell-output .cell-output-display execution_count=12}\n```{=html}\n
\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
Pointblank Validation
2026-08-10|16:44:29
PolarsNYC Flights Data
STEPCOLUMNSVALUESTBLEVALUNITSPASSFAILWECEXT
#4CA64C1\n
\n \n\n col_vals_not_null\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_null()
\n
\n \n 		carrier—\n \n \n \n \n \n \n \n✓337K337K
1.00		0
0.00		————
#4CA64C2\n
\n \n\n col_vals_not_null\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_null()
\n
\n \n 		origin—\n \n \n \n \n \n \n \n✓337K337K
1.00		0
0.00		————
#4CA64C3\n
\n \n\n col_vals_not_null\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_null()
\n
\n \n 		dest—\n \n \n \n \n \n \n \n✓337K337K
1.00		0
0.00		————
#4CA64C4\n
\n \n\n col_vals_gt\n \n \n \n \n \n \n\n
\n
\n
col_vals_gt()
\n
\n \n 		distance0\n \n \n \n \n \n \n \n✓337K337K
1.00		0
0.00		————
2026-08-10 16:44:29 UTC< 1 s2026-08-10 16:44:29 UTC
\n\n
\n```\n:::\n:::\n\n\nYou can also use glob patterns and directories. Here are some examples for how to:\n\n1. load multiple Parquet files\n2. load a Parquet-containing directory\n3. load a partitioned Parquet dataset\n\n```python\n# Multiple Parquet files with glob patterns\nvalidation_6 = pb.Validate(data=\"data/sales_*.parquet\")\n\n# Directory containing Parquet files\nvalidation_7 = pb.Validate(data=\"parquet_data/\")\n\n# Partitioned Parquet dataset\nvalidation_8 = (\n pb.Validate(data=\"sales_data/\") # Contains year=2023/quarter=Q1/region=US/sales.parquet\n .col_exists([\"transaction_id\", \"amount\", \"year\", \"quarter\", \"region\"])\n .interrogate()\n)\n```\n\nWhen you point to a directory that contains a partitioned Parquet dataset (with subdirectories\nlike `year=2023/quarter=Q1/region=US/`), Pointblank will automatically:\n\n- discover all Parquet files recursively\n- extract partition column values from directory paths\n- add partition columns to the final DataFrame\n- combine all partitions into a single table for validation\n\nBoth Polars and Pandas handle partitioned datasets natively, so this works seamlessly with\neither DataFrame library. The loading preference is Polars first, then Pandas as a fallback.\n\n### Working with Database Connection Strings\n\nThe `Validate` class supports database connection strings for direct validation of database\ntables. Connection strings must specify a table using the `::table_name` suffix:\n\n::: {#6f402424 .cell execution_count=13}\n``` {.python .cell-code}\n# Get path to a DuckDB database file from package data\nduckdb_path = pb.get_data_path(\"game_revenue\", \"duckdb\")\n\nvalidation_9 = (\n pb.Validate(\n data=f\"duckdb:///{duckdb_path}::game_revenue\",\n label=\"DuckDB Game Revenue Validation\"\n )\n .col_exists([\"player_id\", \"session_id\", \"item_revenue\"])\n .col_vals_gt(columns=\"item_revenue\", value=0)\n .interrogate()\n)\n\nvalidation_9\n```\n\n::: {.cell-output .cell-output-display execution_count=13}\n```{=html}\n
\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
Pointblank Validation
DuckDB Game Revenue Validation
DuckDB
STEPCOLUMNSVALUESTBLEVALUNITSPASSFAILWECEXT
#4CA64C1\n
\n \n\n col_exists\n \n \n \n \n \n \n \n\n
\n
\n
col_exists()
\n
\n \n 		player_id—\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		————
#4CA64C2\n
\n \n\n col_exists\n \n \n \n \n \n \n \n\n
\n
\n
col_exists()
\n
\n \n 		session_id—\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		————
#4CA64C3\n
\n \n\n col_exists\n \n \n \n \n \n \n \n\n
\n
\n
col_exists()
\n
\n \n 		item_revenue—\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		————
#4CA64C4\n
\n \n\n col_vals_gt\n \n \n \n \n \n \n\n
\n
\n
col_vals_gt()
\n
\n \n 		item_revenue0\n \n \n \n \n \n \n \n✓20002000
1.00		0
0.00		————
2026-08-10 16:44:29 UTC< 1 s2026-08-10 16:44:29 UTC
\n\n
\n```\n:::\n:::\n\n\nFor comprehensive documentation on supported connection string formats, error handling, and\ninstallation requirements, see the [`connect_to_table()`](`pointblank.connect_to_table`)\nfunction. This function handles all the connection logic and provides helpful error messages\nwhen table specifications are missing or backend dependencies are not installed.\n\n", "supporting": [ - "Validate_files" + "Validate_files/figure-html" ], "filters": [], "includes": { diff --git a/_freeze/user-guide/contracts-and-pipelines/custom-adapters/execute-results/html.json b/_freeze/user-guide/contracts-and-pipelines/custom-adapters/execute-results/html.json index 63423fa0d..606560a8d 100644 --- a/_freeze/user-guide/contracts-and-pipelines/custom-adapters/execute-results/html.json +++ b/_freeze/user-guide/contracts-and-pipelines/custom-adapters/execute-results/html.json @@ -1,10 +1,10 @@ { - "hash": "dfdfed9f77d77097d04ed66e42496791", + "hash": "2269fb81839e7614a787c50fda67686f", "result": { "engine": "jupyter", - "markdown": "---\ntitle: Custom Adapters\njupyter: python3\nhtml-table-processing: none\nbread-crumbs: false\n---\n\n\n\nPointblank's contract import/export system is designed to be extensible. If your organization uses\na proprietary schema format, an internal data catalog, or any other schema definition tool that\nisn't covered by the built-in adapters, you can write a **custom adapter** and register it with\nthe framework.\n\nOnce registered, your custom adapter works seamlessly with `import_contract()` and\n`export_contract()`, the same API surface your team already uses for JSON Schema and Frictionless.\n\n## The Adapter Architecture\n\nEvery adapter follows the same pattern:\n\n1. **Subclass** `ContractAdapter` and set a few class attributes\n2. **Implement** `detect()` (for auto-detection), `import_contract()`, and optionally\n `export_contract()`\n3. **Register** the adapter with the `@register_adapter` decorator\n\nHere's a minimal example to illustrate the structure:\n\n::: {#4f05bb5e .cell execution_count=2}\n``` {.python .cell-code}\nfrom pointblank.adapters import ContractAdapter, ContractImport, MappedConstraint, register_adapter\n\n\n@register_adapter(\"my_format\")\nclass MyFormatAdapter(ContractAdapter):\n \"\"\"Adapter for My Company's internal schema format.\"\"\"\n\n format_name = \"my_format\"\n file_extensions = [\".myschema\"]\n supports_import = True\n supports_export = False # export not implemented yet\n\n @staticmethod\n def detect(source) -> bool:\n \"\"\"Return True if this adapter can handle the source.\"\"\"\n if isinstance(source, dict):\n return \"my_format_version\" in source\n return False\n\n def import_contract(self, source, **kwargs) -> ContractImport:\n \"\"\"Parse the source and return a ContractImport.\"\"\"\n # Your parsing logic here\n columns = [(\"id\", \"Int64\"), (\"value\", \"Float64\")]\n constraints = [\n MappedConstraint(\n method=\"col_vals_not_null\",\n kwargs={\"columns\": \"id\"},\n source_description=\"id is required\",\n ),\n ]\n return ContractImport(\n source_format=\"my_format\",\n columns=columns,\n constraints=constraints,\n )\n```\n:::\n\n\nAfter registration, it's immediately usable:\n\n::: {#42407e51 .cell execution_count=3}\n``` {.python .cell-code}\n# Now this works\nresult = pb.import_contract({\"my_format_version\": \"1.0\", \"fields\": []}, format=\"my_format\")\nprint(result)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nContractImport(format='my_format', columns=2, constraints=1, coverage=100%)\n```\n:::\n:::\n\n\nThe adapter is now part of the Pointblank ecosystem. Any call to `import_contract()` with\n`format=\"my_format\"` will route through this adapter, and the auto-detection system will call\n`detect()` when no format is specified.\n\n::: {#367e6fd4 .cell execution_count=4}\n``` {.python .cell-code}\n# And it shows up in the adapter list\npb.list_adapters()\n```\n\n::: {.cell-output .cell-output-display execution_count=4}\n```\n{'frictionless': {'class': 'FrictionlessAdapter',\n 'file_extensions': ['.resource.json', '.datapackage.json'],\n 'supports_import': True,\n 'supports_export': True},\n 'json_schema': {'class': 'JSONSchemaAdapter',\n 'file_extensions': ['.schema.json'],\n 'supports_import': True,\n 'supports_export': True},\n 'my_format': {'class': 'MyFormatAdapter',\n 'file_extensions': ['.myschema'],\n 'supports_import': True,\n 'supports_export': False}}\n```\n:::\n:::\n\n\nThe `list_adapters()` output confirms your adapter is registered alongside the built-in ones,\nshowing its supported file extensions and whether it handles import, export, or both.\n\n## The `ContractAdapter` Base Class\n\nHere are the class attributes and methods you can define:\n\n| Attribute | Type | Purpose |\n|---|---|---|\n| `format_name` | `str` | Short identifier (e.g., `\"json_schema\"`, `\"my_format\"`) |\n| `file_extensions` | `list[str]` | File extensions for auto-detection (e.g., `[\".schema.json\"]`) |\n| `supports_import` | `bool` | Whether `import_contract()` is implemented |\n| `supports_export` | `bool` | Whether `export_contract()` is implemented |\n\n| Method | Required? | Purpose |\n|---|---|---|\n| `detect(source)` | Recommended | Returns `True` if this adapter handles the given source |\n| `import_contract(source, **kwargs)` | If `supports_import` | Parses source, returns `ContractImport` |\n| `export_contract(obj, destination, **kwargs)` | If `supports_export` | Exports to the format |\n\n## Building an Import Adapter\n\nLet's build a more realistic adapter, one that reads a simple YAML-based schema format used\ninternally at a hypothetical company:\n\n```yaml\n# company_schema.yaml\nversion: \"2.0\"\ntable: user_events\ncolumns:\n - name: event_id\n type: string\n required: true\n unique: true\n - name: user_id\n type: integer\n required: true\n - name: event_type\n type: string\n values: [click, view, purchase, signup]\n - name: amount\n type: float\n min: 0\n```\n\nHere's the adapter that handles this format:\n\n::: {#4cb3909d .cell execution_count=5}\n``` {.python .cell-code}\nimport yaml\nfrom pointblank.adapters import ContractAdapter, ContractImport, MappedConstraint, register_adapter\n\n\n@register_adapter(\"company_schema\")\nclass CompanySchemaAdapter(ContractAdapter):\n \"\"\"Adapter for our company's internal YAML schema format.\"\"\"\n\n format_name = \"company_schema\"\n file_extensions = [\".company.yaml\", \".company.yml\"]\n supports_import = True\n supports_export = False\n\n # Type mapping from our format to Pointblank dtypes\n TYPE_MAP = {\n \"string\": \"String\",\n \"integer\": \"Int64\",\n \"float\": \"Float64\",\n \"boolean\": \"Boolean\",\n \"date\": \"Date\",\n \"datetime\": \"Datetime\",\n }\n\n @staticmethod\n def detect(source) -> bool:\n \"\"\"Detect our format by looking for the 'version' + 'columns' keys.\"\"\"\n if isinstance(source, dict):\n return \"version\" in source and \"columns\" in source and \"table\" in source\n return False\n\n def import_contract(self, source, **kwargs) -> ContractImport:\n \"\"\"Import from our company schema format.\"\"\"\n # Load from file or use dict directly\n if isinstance(source, str):\n from pathlib import Path\n\n with open(Path(source)) as f:\n doc = yaml.safe_load(f)\n elif isinstance(source, dict):\n doc = source\n else:\n raise TypeError(f\"Expected str or dict, got {type(source).__name__}\")\n\n columns = []\n constraints = []\n warnings = []\n total = 0\n\n for col_def in doc.get(\"columns\", []):\n col_name = col_def[\"name\"]\n col_type = col_def.get(\"type\", \"string\")\n dtype = self.TYPE_MAP.get(col_type)\n columns.append((col_name, dtype))\n\n if col_def.get(\"required\", False):\n total += 1\n constraints.append(\n MappedConstraint(\n method=\"col_vals_not_null\",\n kwargs={\"columns\": col_name},\n source_description=f\"{col_name} is required\",\n )\n )\n\n if col_def.get(\"unique\", False):\n total += 1\n constraints.append(\n MappedConstraint(\n method=\"rows_distinct\",\n kwargs={\"columns_subset\": col_name},\n source_description=f\"{col_name} must be unique\",\n )\n )\n\n if \"values\" in col_def:\n total += 1\n constraints.append(\n MappedConstraint(\n method=\"col_vals_in_set\",\n kwargs={\"columns\": col_name, \"set\": col_def[\"values\"]},\n source_description=f\"{col_name} allowed values: {col_def['values']}\",\n )\n )\n\n if \"min\" in col_def:\n total += 1\n constraints.append(\n MappedConstraint(\n method=\"col_vals_ge\",\n kwargs={\"columns\": col_name, \"value\": col_def[\"min\"]},\n source_description=f\"{col_name} >= {col_def['min']}\",\n )\n )\n\n if \"max\" in col_def:\n total += 1\n constraints.append(\n MappedConstraint(\n method=\"col_vals_le\",\n kwargs={\"columns\": col_name, \"value\": col_def[\"max\"]},\n source_description=f\"{col_name} <= {col_def['max']}\",\n )\n )\n\n coverage = 1.0 if total == 0 else (total - len(warnings)) / total\n\n return ContractImport(\n source_format=\"company_schema\",\n source_path=source if isinstance(source, str) else None,\n source_version=doc.get(\"version\"),\n columns=columns,\n constraints=constraints,\n metadata={\"table\": doc.get(\"table\")},\n warnings=warnings,\n coverage=coverage,\n )\n```\n:::\n\n\nNow let's use it:\n\n::: {#954a597a .cell execution_count=6}\n``` {.python .cell-code}\nimport polars as pl\n\n# Simulate a company schema document\ncompany_schema = {\n \"version\": \"2.0\",\n \"table\": \"user_events\",\n \"columns\": [\n {\"name\": \"event_id\", \"type\": \"string\", \"required\": True, \"unique\": True},\n {\"name\": \"user_id\", \"type\": \"integer\", \"required\": True},\n {\"name\": \"event_type\", \"type\": \"string\", \"values\": [\"click\", \"view\", \"purchase\", \"signup\"]},\n {\"name\": \"amount\", \"type\": \"float\", \"min\": 0},\n ],\n}\n\n# Import using our custom adapter\nresult = pb.import_contract(company_schema, format=\"company_schema\")\nprint(result.summary())\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nContract Import Summary\n Format: company_schema\n Format version: 2.0\n Columns detected: 4\n Constraints mapped: 5\n Coverage: 100%\n```\n:::\n:::\n\n\nThe summary shows four columns detected and five constraints mapped (two `required` fields, one\n`unique` field, one `values` check, and one `min` bound). All constraints were successfully\ntranslated, giving 100% coverage.\n\n::: {#7b24c54f .cell execution_count=7}\n``` {.python .cell-code}\n# Validate some data\nevents = pl.DataFrame(\n {\n \"event_id\": [\"E001\", \"E002\", \"E003\", \"E004\", \"E005\"],\n \"user_id\": [101, 102, 101, 103, 104],\n \"event_type\": [\"click\", \"view\", \"purchase\", \"signup\", \"click\"],\n \"amount\": [0.0, 0.0, 49.99, 0.0, 0.0],\n }\n)\n\nresult.to_validate(data=events).interrogate()\n```\n\n::: {.cell-output .cell-output-display execution_count=7}\n```{=html}\n
\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
Pointblank Validation
2026-07-22|23:23:50
Polars
STEPCOLUMNSVALUESTBLEVALUNITSPASSFAILWECEXT
#4CA64C1\n
\n \n\n col_schema_match\n \n \n \n \n \n \n \n \n \n \n \n\n
\n
\n
col_schema_match()
\n
\n \n 		—SCHEMA\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		————
#4CA64C2\n
\n \n\n col_vals_not_null\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_null()
\n
\n \n 		event_id—\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C3\n
\n \n\n rows_distinct\n \n \n \n \n \n \n \n \n \n \n\n
\n
\n
rows_distinct()
\n
\n \n 		event_id—\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C4\n
\n \n\n col_vals_not_null\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_null()
\n
\n \n 		user_id—\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C5\n
\n \n\n col_vals_in_set\n \n \n \n \n \n \n\n
\n
\n
col_vals_in_set()
\n
\n \n 		event_typeclick, view, purchase, signup\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C6\n
\n \n\n col_vals_ge\n \n \n \n \n \n \n\n
\n
\n
col_vals_ge()
\n
\n \n 		amount0\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
\nNotes\n

Step 1 (schema_check) ✓ Schema validation passed.

\n
\nSchema Comparison\n
\n
\n\n\n\n \n \n \n \n \n \n \n \n\n\n\n \n \n\n\n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
\n TARGET\n \n EXPECTED\n
COLUMNDATA TYPECOLUMNDATA TYPE
1event_idString1event_id✓String✓
2user_idInt642user_id✓Int64✓
3event_typeString3event_type✓String✓
4amountFloat644amount✓Float64✓
Supplied Column Schema:
[('event_id', 'String'), ('user_id', 'Int64'), ('event_type', 'String'), ('amount', 'Float64')]
\n
Schema Match Settings
\n
COMPLETE
IN ORDER
COLUMN ≠ column
DTYPE ≠ dtype
float ≠ float64
\n
\n
\n
\n
\n
\n\n
\n```\n:::\n:::\n\n\nThe validation report shows each imported constraint as a separate step, just as if you had written\nthe validation by hand. From the user's perspective, there is no difference between validation steps\nthat came from a custom adapter and those written directly in Python.\n\n## The `MappedConstraint` Class\n\nEach constraint from the external format gets mapped to a `MappedConstraint`, which is a simple\ndata container holding:\n\n- `method`: the Pointblank `Validate` method name (e.g., `\"col_vals_gt\"`)\n- `kwargs`: the keyword arguments to pass to that method\n- `source_description`: optional human-readable note about what this was in the source format\n\n::: {#f81bfe35 .cell execution_count=8}\n``` {.python .cell-code}\n# Creating constraints manually\nc1 = MappedConstraint(\n method=\"col_vals_between\",\n kwargs={\"columns\": \"temperature\", \"left\": -40, \"right\": 60},\n source_description=\"Temperature must be in physical range\",\n)\nprint(c1)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nMappedConstraint('col_vals_between', columns='temperature', left=-40, right=60)\n```\n:::\n:::\n\n\nThe `source_description` is stored for debugging and documentation but doesn't affect validation.\nWhen users call `.summary()` or inspect the `ContractImport` object, these descriptions help them\nunderstand the provenance of each validation step. This is especially useful when debugging why a\nparticular check was generated or when comparing the import output against the original schema.\n\n## Handling Unmappable Constraints\n\nNot every constraint in every format has a clean Pointblank equivalent. When you encounter something\nthat can't be translated, add it to the warnings list rather than silently dropping it:\n\n```python\n# In your import_contract() method:\nif \"custom_check\" in col_def:\n total += 1\n warnings.append(\n f\"Column '{col_name}': 'custom_check' has no Pointblank equivalent, skipped.\"\n )\n```\n\nThis follows Pointblank's design principle of **best-effort translation**: generate everything you\ncan, be transparent about what was skipped, and never silently lose information. Users can then\nreview the warnings list and decide whether to add manual validation steps for the missing\nconstraints or whether the gap is acceptable for their use case.\n\n## Auto-Detection Tips\n\nThe `detect()` method enables format auto-detection. Good detection should be:\n\n- **Fast**: don't load the entire file just to check if it's your format\n- **Specific**: avoid false positives that could conflict with other adapters\n- **Graceful**: return `False` (never raise) if the source isn't your format\n\nThe detection system iterates through all registered adapters and calls `detect()` on each one.\nBecause of this, your detection logic should be as lightweight as possible. Checking for the\npresence of a few distinctive keys in a dict is ideal. Avoid expensive operations like parsing\nlarge files or making network requests inside `detect()`.\n\n```python\n@staticmethod\ndef detect(source) -> bool:\n if isinstance(source, dict):\n # Check for a distinctive key combination\n return \"my_format_version\" in source and \"tables\" in source\n\n if isinstance(source, str):\n # Check file extension first (cheapest check)\n return source.lower().endswith(\".myformat.yaml\")\n\n return False\n```\n\n## Best Practices\n\n1. **Map as much as possible**: users expect high coverage. If a constraint is *close* to\n something Pointblank supports, map it (possibly with reduced precision) rather than skipping it.\n\n2. **Use descriptive source_description**: this helps users understand what each generated\n validation step corresponds to in their original schema.\n\n3. **Set coverage accurately**: track the total number of source constraints and how many were\n successfully mapped. This gives users confidence in the import quality.\n\n4. **Handle both file paths and dicts**: users should be able to pass either a path string or\n pre-loaded data. Most adapters check `isinstance(source, str)` for file paths and\n `isinstance(source, dict)` for pre-parsed content.\n\n5. **Fail clearly on bad input**: raise `TypeError` for wrong source types, `FileNotFoundError`\n for missing files, and `ValueError` for malformed content. Don't return partial results\n silently.\n\n6. **Keep dependencies optional**: if your adapter needs a third-party library, check for it at\n import time and give a clear installation hint if it's missing.\n\n## Conclusion\n\nCustom adapters let you extend Pointblank's import/export system to handle any schema format your\norganization uses. The plugin architecture is intentionally simple: subclass `ContractAdapter`,\nimplement one or two methods, and register it with a decorator. From that point forward, your\nformat participates in the same `import_contract()` and `export_contract()` workflow that the\nbuilt-in adapters use.\n\nThis extensibility means that Pointblank can serve as a universal validation layer regardless of\nwhere your data contracts originate. Whether your schemas live in a proprietary YAML format, an\ninternal data catalog API, or a custom metadata store, a short adapter class is all you need to\nbring them into the Pointblank ecosystem and benefit from its validation reporting, threshold\nsystem, and pipeline integration.\n\n", + "markdown": "---\ntitle: Custom Adapters\njupyter: python3\nhtml-table-processing: none\nbread-crumbs: false\n---\n\n\n\nPointblank's contract import/export system is designed to be extensible. If your organization uses\na proprietary schema format, an internal data catalog, or any other schema definition tool that\nisn't covered by the built-in adapters, you can write a **custom adapter** and register it with\nthe framework.\n\nOnce registered, your custom adapter works seamlessly with `import_contract()` and\n`export_contract()`, the same API surface your team already uses for JSON Schema, Frictionless, dbt,\nand ODCS.\n\n## The Adapter Architecture\n\nEvery adapter follows the same pattern:\n\n1. **Subclass** `ContractAdapter` and set a few class attributes\n2. **Implement** `detect()` (for auto-detection), `import_contract()`, and optionally\n `export_contract()`\n3. **Register** the adapter with the `@register_adapter` decorator\n\nHere's a minimal example to illustrate the structure:\n\n::: {#5ebfad4a .cell execution_count=2}\n``` {.python .cell-code}\nfrom pointblank.adapters import ContractAdapter, ContractImport, MappedConstraint, register_adapter\n\n\n@register_adapter(\"my_format\")\nclass MyFormatAdapter(ContractAdapter):\n \"\"\"Adapter for My Company's internal schema format.\"\"\"\n\n format_name = \"my_format\"\n file_extensions = [\".myschema\"]\n supports_import = True\n supports_export = False # export not implemented yet\n\n @staticmethod\n def detect(source) -> bool:\n \"\"\"Return True if this adapter can handle the source.\"\"\"\n if isinstance(source, dict):\n return \"my_format_version\" in source\n return False\n\n def import_contract(self, source, **kwargs) -> ContractImport:\n \"\"\"Parse the source and return a ContractImport.\"\"\"\n # Your parsing logic here\n columns = [(\"id\", \"Int64\"), (\"value\", \"Float64\")]\n constraints = [\n MappedConstraint(\n method=\"col_vals_not_null\",\n kwargs={\"columns\": \"id\"},\n source_description=\"id is required\",\n ),\n ]\n return ContractImport(\n source_format=\"my_format\",\n columns=columns,\n constraints=constraints,\n )\n```\n:::\n\n\nAfter registration, it's immediately usable:\n\n::: {#f4e3cd8a .cell execution_count=3}\n``` {.python .cell-code}\n# Now this works\nresult = pb.import_contract({\"my_format_version\": \"1.0\", \"fields\": []}, format=\"my_format\")\nprint(result)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nContractImport(format='my_format', columns=2, constraints=1, coverage=100%)\n```\n:::\n:::\n\n\nThe adapter is now part of the Pointblank ecosystem. Any call to `import_contract()` with\n`format=\"my_format\"` will route through this adapter, and the auto-detection system will call\n`detect()` when no format is specified.\n\n::: {#6aec1cae .cell execution_count=4}\n``` {.python .cell-code}\n# And it shows up in the adapter list\npb.list_adapters()\n```\n\n::: {.cell-output .cell-output-display execution_count=4}\n```\n{'dbt': {'class': 'DbtAdapter',\n 'file_extensions': ['.yml', '.yaml'],\n 'supports_import': True,\n 'supports_export': True},\n 'frictionless': {'class': 'FrictionlessAdapter',\n 'file_extensions': ['.resource.json', '.datapackage.json'],\n 'supports_import': True,\n 'supports_export': True},\n 'json_schema': {'class': 'JSONSchemaAdapter',\n 'file_extensions': ['.schema.json'],\n 'supports_import': True,\n 'supports_export': True},\n 'my_format': {'class': 'MyFormatAdapter',\n 'file_extensions': ['.myschema'],\n 'supports_import': True,\n 'supports_export': False},\n 'odcs': {'class': 'ODCSAdapter',\n 'file_extensions': ['.odcs.yml', '.odcs.yaml', '.odcs.json'],\n 'supports_import': True,\n 'supports_export': True}}\n```\n:::\n:::\n\n\nThe `list_adapters()` output confirms your adapter is registered alongside the built-in ones,\nshowing its supported file extensions and whether it handles import, export, or both.\n\n## The `ContractAdapter` Base Class\n\nHere are the class attributes and methods you can define:\n\n| Attribute | Type | Purpose |\n|---|---|---|\n| `format_name` | `str` | Short identifier (e.g., `\"json_schema\"`, `\"my_format\"`) |\n| `file_extensions` | `list[str]` | File extensions for auto-detection (e.g., `[\".schema.json\"]`) |\n| `supports_import` | `bool` | Whether `import_contract()` is implemented |\n| `supports_export` | `bool` | Whether `export_contract()` is implemented |\n\n| Method | Required? | Purpose |\n|---|---|---|\n| `detect(source)` | Recommended | Returns `True` if this adapter handles the given source |\n| `import_contract(source, **kwargs)` | If `supports_import` | Parses source, returns `ContractImport` |\n| `export_contract(obj, destination, **kwargs)` | If `supports_export` | Exports to the format |\n\n## Building an Import Adapter\n\nLet's build a more realistic adapter, one that reads a simple YAML-based schema format used\ninternally at a hypothetical company:\n\n```yaml\n# company_schema.yaml\nversion: \"2.0\"\ntable: user_events\ncolumns:\n - name: event_id\n type: string\n required: true\n unique: true\n - name: user_id\n type: integer\n required: true\n - name: event_type\n type: string\n values: [click, view, purchase, signup]\n - name: amount\n type: float\n min: 0\n```\n\nHere's the adapter that handles this format:\n\n::: {#0c0b98ac .cell execution_count=5}\n``` {.python .cell-code}\nimport yaml\nfrom pointblank.adapters import ContractAdapter, ContractImport, MappedConstraint, register_adapter\n\n\n@register_adapter(\"company_schema\")\nclass CompanySchemaAdapter(ContractAdapter):\n \"\"\"Adapter for our company's internal YAML schema format.\"\"\"\n\n format_name = \"company_schema\"\n file_extensions = [\".company.yaml\", \".company.yml\"]\n supports_import = True\n supports_export = False\n\n # Type mapping from our format to Pointblank dtypes\n TYPE_MAP = {\n \"string\": \"String\",\n \"integer\": \"Int64\",\n \"float\": \"Float64\",\n \"boolean\": \"Boolean\",\n \"date\": \"Date\",\n \"datetime\": \"Datetime\",\n }\n\n @staticmethod\n def detect(source) -> bool:\n \"\"\"Detect our format by looking for the 'version' + 'columns' keys.\"\"\"\n if isinstance(source, dict):\n return \"version\" in source and \"columns\" in source and \"table\" in source\n return False\n\n def import_contract(self, source, **kwargs) -> ContractImport:\n \"\"\"Import from our company schema format.\"\"\"\n # Load from file or use dict directly\n if isinstance(source, str):\n from pathlib import Path\n\n with open(Path(source)) as f:\n doc = yaml.safe_load(f)\n elif isinstance(source, dict):\n doc = source\n else:\n raise TypeError(f\"Expected str or dict, got {type(source).__name__}\")\n\n columns = []\n constraints = []\n warnings = []\n total = 0\n\n for col_def in doc.get(\"columns\", []):\n col_name = col_def[\"name\"]\n col_type = col_def.get(\"type\", \"string\")\n dtype = self.TYPE_MAP.get(col_type)\n columns.append((col_name, dtype))\n\n if col_def.get(\"required\", False):\n total += 1\n constraints.append(\n MappedConstraint(\n method=\"col_vals_not_null\",\n kwargs={\"columns\": col_name},\n source_description=f\"{col_name} is required\",\n )\n )\n\n if col_def.get(\"unique\", False):\n total += 1\n constraints.append(\n MappedConstraint(\n method=\"rows_distinct\",\n kwargs={\"columns_subset\": col_name},\n source_description=f\"{col_name} must be unique\",\n )\n )\n\n if \"values\" in col_def:\n total += 1\n constraints.append(\n MappedConstraint(\n method=\"col_vals_in_set\",\n kwargs={\"columns\": col_name, \"set\": col_def[\"values\"]},\n source_description=f\"{col_name} allowed values: {col_def['values']}\",\n )\n )\n\n if \"min\" in col_def:\n total += 1\n constraints.append(\n MappedConstraint(\n method=\"col_vals_ge\",\n kwargs={\"columns\": col_name, \"value\": col_def[\"min\"]},\n source_description=f\"{col_name} >= {col_def['min']}\",\n )\n )\n\n if \"max\" in col_def:\n total += 1\n constraints.append(\n MappedConstraint(\n method=\"col_vals_le\",\n kwargs={\"columns\": col_name, \"value\": col_def[\"max\"]},\n source_description=f\"{col_name} <= {col_def['max']}\",\n )\n )\n\n coverage = 1.0 if total == 0 else (total - len(warnings)) / total\n\n return ContractImport(\n source_format=\"company_schema\",\n source_path=source if isinstance(source, str) else None,\n source_version=doc.get(\"version\"),\n columns=columns,\n constraints=constraints,\n metadata={\"table\": doc.get(\"table\")},\n warnings=warnings,\n coverage=coverage,\n )\n```\n:::\n\n\nNow let's use it:\n\n::: {#56066385 .cell execution_count=6}\n``` {.python .cell-code}\nimport polars as pl\n\n# Simulate a company schema document\ncompany_schema = {\n \"version\": \"2.0\",\n \"table\": \"user_events\",\n \"columns\": [\n {\"name\": \"event_id\", \"type\": \"string\", \"required\": True, \"unique\": True},\n {\"name\": \"user_id\", \"type\": \"integer\", \"required\": True},\n {\"name\": \"event_type\", \"type\": \"string\", \"values\": [\"click\", \"view\", \"purchase\", \"signup\"]},\n {\"name\": \"amount\", \"type\": \"float\", \"min\": 0},\n ],\n}\n\n# Import using our custom adapter\nresult = pb.import_contract(company_schema, format=\"company_schema\")\nprint(result.summary())\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nContract Import Summary\n Format: company_schema\n Format version: 2.0\n Columns detected: 4\n Constraints mapped: 5\n Coverage: 100%\n```\n:::\n:::\n\n\nThe summary shows four columns detected and five constraints mapped (two `required` fields, one\n`unique` field, one `values` check, and one `min` bound). All constraints were successfully\ntranslated, giving 100% coverage.\n\n::: {#40280ea6 .cell execution_count=7}\n``` {.python .cell-code}\n# Validate some data\nevents = pl.DataFrame(\n {\n \"event_id\": [\"E001\", \"E002\", \"E003\", \"E004\", \"E005\"],\n \"user_id\": [101, 102, 101, 103, 104],\n \"event_type\": [\"click\", \"view\", \"purchase\", \"signup\", \"click\"],\n \"amount\": [0.0, 0.0, 49.99, 0.0, 0.0],\n }\n)\n\nresult.to_validate(data=events).interrogate()\n```\n\n::: {.cell-output .cell-output-display execution_count=7}\n```{=html}\n
\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
Pointblank Validation
2026-08-10|16:45:25
Polars
STEPCOLUMNSVALUESTBLEVALUNITSPASSFAILWECEXT
#4CA64C1\n
\n \n\n col_schema_match\n \n \n \n \n \n \n \n \n \n \n \n\n
\n
\n
col_schema_match()
\n
\n \n 		—SCHEMA\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		————
#4CA64C2\n
\n \n\n col_vals_not_null\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_null()
\n
\n \n 		event_id—\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C3\n
\n \n\n rows_distinct\n \n \n \n \n \n \n \n \n \n \n\n
\n
\n
rows_distinct()
\n
\n \n 		event_id—\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C4\n
\n \n\n col_vals_not_null\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_null()
\n
\n \n 		user_id—\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C5\n
\n \n\n col_vals_in_set\n \n \n \n \n \n \n\n
\n
\n
col_vals_in_set()
\n
\n \n 		event_typeclick, view, purchase, signup\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C6\n
\n \n\n col_vals_ge\n \n \n \n \n \n \n\n
\n
\n
col_vals_ge()
\n
\n \n 		amount0\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
\nNotes\n

Step 1 (schema_check) ✓ Schema validation passed.

\n
\nSchema Comparison\n
\n
\n\n\n\n \n \n \n \n \n \n \n \n\n\n\n \n \n\n\n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
\n TARGET\n \n EXPECTED\n
COLUMNDATA TYPECOLUMNDATA TYPE
1event_idString1event_id✓String✓
2user_idInt642user_id✓Int64✓
3event_typeString3event_type✓String✓
4amountFloat644amount✓Float64✓
Supplied Column Schema:
[('event_id', 'String'), ('user_id', 'Int64'), ('event_type', 'String'), ('amount', 'Float64')]
\n
Schema Match Settings
\n
COMPLETE
IN ORDER
COLUMN ≠ column
DTYPE ≠ dtype
float ≠ float64
\n
\n
\n
\n
\n
\n\n
\n```\n:::\n:::\n\n\nThe validation report shows each imported constraint as a separate step, just as if you had written\nthe validation by hand. From the user's perspective, there is no difference between validation steps\nthat came from a custom adapter and those written directly in Python.\n\n## The `MappedConstraint` Class\n\nEach constraint from the external format gets mapped to a `MappedConstraint`, which is a simple\ndata container holding:\n\n- `method`: the Pointblank `Validate` method name (e.g., `\"col_vals_gt\"`)\n- `kwargs`: the keyword arguments to pass to that method\n- `source_description`: optional human-readable note about what this was in the source format\n\n::: {#617e78bb .cell execution_count=8}\n``` {.python .cell-code}\n# Creating constraints manually\nc1 = MappedConstraint(\n method=\"col_vals_between\",\n kwargs={\"columns\": \"temperature\", \"left\": -40, \"right\": 60},\n source_description=\"Temperature must be in physical range\",\n)\nprint(c1)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nMappedConstraint('col_vals_between', columns='temperature', left=-40, right=60)\n```\n:::\n:::\n\n\nThe `source_description` is stored for debugging and documentation but doesn't affect validation.\nWhen users call `.summary()` or inspect the `ContractImport` object, these descriptions help them\nunderstand the provenance of each validation step. This is especially useful when debugging why a\nparticular check was generated or when comparing the import output against the original schema.\n\n## Handling Unmappable Constraints\n\nNot every constraint in every format has a clean Pointblank equivalent. When you encounter something\nthat can't be translated, add it to the warnings list rather than silently dropping it:\n\n```python\n# In your import_contract() method:\nif \"custom_check\" in col_def:\n total += 1\n warnings.append(\n f\"Column '{col_name}': 'custom_check' has no Pointblank equivalent, skipped.\"\n )\n```\n\nThis follows Pointblank's design principle of **best-effort translation**: generate everything you\ncan, be transparent about what was skipped, and never silently lose information. Users can then\nreview the warnings list and decide whether to add manual validation steps for the missing\nconstraints or whether the gap is acceptable for their use case.\n\n## Auto-Detection Tips\n\nThe `detect()` method enables format auto-detection. Good detection should be:\n\n- **Fast**: don't load the entire file just to check if it's your format\n- **Specific**: avoid false positives that could conflict with other adapters\n- **Graceful**: return `False` (never raise) if the source isn't your format\n\nThe detection system iterates through all registered adapters and calls `detect()` on each one.\nBecause of this, your detection logic should be as lightweight as possible. Checking for the\npresence of a few distinctive keys in a dict is ideal. Avoid expensive operations like parsing\nlarge files or making network requests inside `detect()`.\n\n```python\n@staticmethod\ndef detect(source) -> bool:\n if isinstance(source, dict):\n # Check for a distinctive key combination\n return \"my_format_version\" in source and \"tables\" in source\n\n if isinstance(source, str):\n # Check file extension first (cheapest check)\n return source.lower().endswith(\".myformat.yaml\")\n\n return False\n```\n\n## Best Practices\n\n1. **Map as much as possible**: users expect high coverage. If a constraint is *close* to\n something Pointblank supports, map it (possibly with reduced precision) rather than skipping it.\n\n2. **Use descriptive source_description**: this helps users understand what each generated\n validation step corresponds to in their original schema.\n\n3. **Set coverage accurately**: track the total number of source constraints and how many were\n successfully mapped. This gives users confidence in the import quality.\n\n4. **Handle both file paths and dicts**: users should be able to pass either a path string or\n pre-loaded data. Most adapters check `isinstance(source, str)` for file paths and\n `isinstance(source, dict)` for pre-parsed content.\n\n5. **Fail clearly on bad input**: raise `TypeError` for wrong source types, `FileNotFoundError`\n for missing files, and `ValueError` for malformed content. Don't return partial results\n silently.\n\n6. **Keep dependencies optional**: if your adapter needs a third-party library, check for it at\n import time and give a clear installation hint if it's missing.\n\n## Conclusion\n\nCustom adapters let you extend Pointblank's import/export system to handle any schema format your\norganization uses. The plugin architecture is intentionally simple: subclass `ContractAdapter`,\nimplement one or two methods, and register it with a decorator. From that point forward, your\nformat participates in the same `import_contract()` and `export_contract()` workflow that the\nbuilt-in adapters use.\n\nThis extensibility means that Pointblank can serve as a universal validation layer regardless of\nwhere your data contracts originate. Whether your schemas live in a proprietary YAML format, an\ninternal data catalog API, or a custom metadata store, a short adapter class is all you need to\nbring them into the Pointblank ecosystem and benefit from its validation reporting, threshold\nsystem, and pipeline integration.\n\n", "supporting": [ - "custom-adapters_files" + "custom-adapters_files/figure-html" ], "filters": [], "includes": { diff --git a/_freeze/user-guide/contracts-and-pipelines/importing-contracts/execute-results/html.json b/_freeze/user-guide/contracts-and-pipelines/importing-contracts/execute-results/html.json index fb5d90be2..d57cdbdfd 100644 --- a/_freeze/user-guide/contracts-and-pipelines/importing-contracts/execute-results/html.json +++ b/_freeze/user-guide/contracts-and-pipelines/importing-contracts/execute-results/html.json @@ -1,10 +1,10 @@ { - "hash": "bde327e365d0758dff3be65749f22dc4", + "hash": "b2f1fc1ea0a74e923d3c5200df57d2c8", "result": { "engine": "jupyter", - "markdown": "---\ntitle: Importing External Schemas\njupyter: python3\nhtml-table-processing: none\nbread-crumbs: false\n---\n\n\n\nMany teams already have data schemas defined in other tools: JSON Schema files for API validation,\nFrictionless Table Schemas for open data, dbt `schema.yml` files for analytics pipelines, or\nPandera/Pydantic models in application code. Rather than manually rewriting these specifications as\nPointblank validation steps, you can **import** them directly.\n\nThe `import_contract()` function reads an external schema definition and produces a `ContractImport`\nobject containing everything Pointblank needs to validate data: column types, constraints, and\nmapped validation steps. From there you can create a `Validate` workflow, a `Contract` object,\ngenerate equivalent Python code, or produce a YAML definition, all with a single function call.\n\n## Quick Start\n\nThe fastest path from an external schema to running validation:\n\n::: {#97c51258 .cell execution_count=2}\n``` {.python .cell-code}\nimport pointblank as pb\nimport polars as pl\n\n# Define a JSON Schema (could also be loaded from a file)\nuser_schema = {\n \"type\": \"object\",\n \"properties\": {\n \"user_id\": {\"type\": \"integer\"},\n \"email\": {\"type\": \"string\", \"format\": \"email\"},\n \"age\": {\"type\": \"integer\", \"minimum\": 0, \"maximum\": 150},\n \"status\": {\"type\": \"string\", \"enum\": [\"active\", \"inactive\", \"pending\"]},\n },\n \"required\": [\"user_id\", \"email\"],\n}\n\n# Import the schema\nresult = pb.import_contract(user_schema, format=\"json_schema\")\n\n# Create sample data and validate\nusers = pl.DataFrame(\n {\n \"user_id\": [1, 2, 3, 4, 5],\n \"email\": [\n \"alice@example.com\",\n \"bob@corp.io\",\n \"charlie@mail.org\",\n \"dave@startup.co\",\n \"eve@company.net\",\n ],\n \"age\": [28, 34, 45, 22, 31],\n \"status\": [\"active\", \"active\", \"inactive\", \"pending\", \"active\"],\n }\n)\n\nresult.to_validate(data=users).interrogate()\n```\n\n::: {.cell-output .cell-output-display execution_count=2}\n```{=html}\n
\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
Pointblank Validation
2026-07-22|23:23:53
Polars
STEPCOLUMNSVALUESTBLEVALUNITSPASSFAILWECEXT
#4CA64C1\n
\n \n\n col_schema_match\n \n \n \n \n \n \n \n \n \n \n \n\n
\n
\n
col_schema_match()
\n
\n \n 		—SCHEMA\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		————
#4CA64C2\n
\n \n\n col_vals_not_null\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_null()
\n
\n \n 		user_id—\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C3\n
\n \n\n col_vals_not_null\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_null()
\n
\n \n 		email—\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C4\n
\n \n\n col_vals_within_spec\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_within_spec()
\n
\n \n 		emailemail\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C5\n
\n \n\n col_vals_ge\n \n \n \n \n \n \n\n
\n
\n
col_vals_ge()
\n
\n \n 		age0\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C6\n
\n \n\n col_vals_le\n \n \n \n \n \n \n\n
\n
\n
col_vals_le()
\n
\n \n 		age150\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C7\n
\n \n\n col_vals_in_set\n \n \n \n \n \n \n\n
\n
\n
col_vals_in_set()
\n
\n \n 		statusactive, inactive, pending\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
\nNotes\n

Step 1 (schema_check) ✓ Schema validation passed.

\n
\nSchema Comparison\n
\n
\n\n\n\n \n \n \n \n \n \n \n \n\n\n\n \n \n\n\n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
\n TARGET\n \n EXPECTED\n
COLUMNDATA TYPECOLUMNDATA TYPE
1user_idInt641user_id✓Int64✓
2emailString2email✓String✓
3ageInt643age✓Int64✓
4statusString4status✓String✓
Supplied Column Schema:
[('user_id', 'Int64'), ('email', 'String'), ('age', 'Int64'), ('status', 'String')]
\n
Schema Match Settings
\n
COMPLETE
IN ORDER
COLUMN ≠ column
DTYPE ≠ dtype
float ≠ float64
\n
\n
\n
\n
\n
\n\n
\n```\n:::\n:::\n\n\nThat's it. The JSON Schema `minimum`, `maximum`, `enum`, `format`, and `required` keywords were\nautomatically translated into the appropriate Pointblank validation steps. Each keyword becomes a\ndedicated validation check: `minimum` becomes `col_vals_ge()`, `maximum` becomes `col_vals_le()`,\n`enum` becomes `col_vals_in_set()`, and so on. The schema's `required` array generates\n`col_vals_not_null()` steps for each listed field, ensuring that null values are caught at\nvalidation time.\n\n## How It Works\n\nThe import process has three stages:\n\n1. **Parse**: the external schema is read (from a file path, a dict, or a Python object)\n2. **Map**: each constraint in the source format is translated to a Pointblank validation method\n3. **Package**: the results are stored in a `ContractImport` object with multiple output options\n\n```{mermaid}\nflowchart LR\n A[External Schema] --> B[import_contract]\n B --> C[ContractImport]\n C --> D[.to_validate‹data›]\n C --> E[.to_contract‹›]\n C --> F[.to_python‹›]\n C --> G[.to_yaml‹›]\n```\n\nThe `ContractImport` object is your bridge between the external world and Pointblank. It doesn't\nexecute anything on its own. Rather, it holds the *translated specification* and lets you choose\nhow to use it. This separation is intentional: you can inspect the translation results, check\nfor any warnings, and decide how to proceed before committing to a particular output format.\n\n## The `ContractImport` Object\n\nAfter calling `import_contract()`, you get back a `ContractImport` with these key attributes and\nmethods:\n\n| Attribute / Method | Description |\n|---|---|\n| `.columns` | List of `(column_name, dtype)` tuples detected from the source |\n| `.constraints` | List of `MappedConstraint` objects (method + kwargs) |\n| `.warnings` | Messages about constraints that couldn't be translated |\n| `.coverage` | Fraction of source constraints successfully mapped (0.0–1.0) |\n| `.metadata` | Extra metadata (title, description) from the source |\n| `.to_validate(data)` | Build a `Validate` object ready for `.interrogate()` |\n| `.to_contract(name)` | Build a `Contract` object for pipeline use |\n| `.to_python()` | Generate equivalent Python code as a string |\n| `.to_yaml()` | Generate Pointblank YAML configuration |\n| `.summary()` | Return a human-readable summary |\n\n### Inspecting What Was Imported\n\nBefore running validation, it's often useful to inspect what the import produced:\n\n::: {#a1086592 .cell execution_count=3}\n``` {.python .cell-code}\nresult = pb.import_contract(user_schema, format=\"json_schema\")\nprint(result.summary())\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nContract Import Summary\n Format: json_schema\n Columns detected: 4\n Constraints mapped: 6\n Coverage: 100%\n```\n:::\n:::\n\n\nIf any constraints couldn't be mapped, they appear in `.warnings`:\n\n::: {#9b6c35a3 .cell execution_count=4}\n``` {.python .cell-code}\n# A schema with an unmappable format\nschema_with_date = {\n \"type\": \"object\",\n \"properties\": {\n \"created_at\": {\"type\": \"string\", \"format\": \"date-time\"},\n },\n}\nresult = pb.import_contract(schema_with_date, format=\"json_schema\")\n\nif result.warnings:\n for w in result.warnings:\n print(f\"⚠ {w}\")\n\nprint(f\"\\nCoverage: {result.coverage:.0%}\")\n```\n\n::: {.cell-output .cell-output-stdout}\n```\n⚠ Column 'created_at': JSON Schema format 'date-time' has no Pointblank equivalent — skipped.\n\nCoverage: 0%\n```\n:::\n:::\n\n\nThe `coverage` metric tells you what fraction of the source constraints were successfully\ntranslated. A coverage of 100% means everything mapped cleanly; lower values mean some constraints\nwere skipped (with details in `warnings`). This transparency is important because no translation\nbetween formats is perfect. By checking `coverage` and `warnings` before running validation, you\ncan be confident about exactly which parts of your original schema are being enforced and which\nparts might need manual attention.\n\n## Supported Formats\n\nPointblank ships with adapters for the two most universal tabular schema formats. Additional\nadapters (dbt, Pydantic, Pandera) are planned for future releases.\n\n::: {#8d3dbe27 .cell execution_count=5}\n``` {.python .cell-code}\npb.list_adapters()\n```\n\n::: {.cell-output .cell-output-display execution_count=5}\n```\n{'frictionless': {'class': 'FrictionlessAdapter',\n 'file_extensions': ['.resource.json', '.datapackage.json'],\n 'supports_import': True,\n 'supports_export': True},\n 'json_schema': {'class': 'JSONSchemaAdapter',\n 'file_extensions': ['.schema.json'],\n 'supports_import': True,\n 'supports_export': True}}\n```\n:::\n:::\n\n\n### JSON Schema\n\n[JSON Schema](https://json-schema.org/) is a widely used format for describing the structure of JSON\ndata. Because tabular data (DataFrames) can be modeled as arrays of JSON objects, JSON Schema is a\nnatural fit for defining column-level constraints.\n\n**Constraint mapping:**\n\n| JSON Schema Keyword | Pointblank Method |\n|---|---|\n| `type: \"integer\"` / `\"number\"` / `\"string\"` / `\"boolean\"` | Schema dtype check |\n| `minimum` | `col_vals_ge()` |\n| `maximum` | `col_vals_le()` |\n| `exclusiveMinimum` | `col_vals_gt()` |\n| `exclusiveMaximum` | `col_vals_lt()` |\n| `enum` | `col_vals_in_set()` |\n| `pattern` | `col_vals_regex()` |\n| `format: \"email\"` | `col_vals_within_spec(spec=\"email\")` |\n| `format: \"uri\"` | `col_vals_within_spec(spec=\"url\")` |\n| `format: \"ipv4\"` / `\"ipv6\"` | `col_vals_within_spec(spec=\"ipv4\"/\"ipv6\")` |\n| `const` | `col_vals_eq()` |\n| `required` (array of field names) | `col_vals_not_null()` |\n\n**Importing from a file:**\n\n```python\n# File-based import (auto-detects .schema.json extension)\nresult = pb.import_contract(\"models/user_profile.schema.json\")\n```\n\n**Importing from a dict:**\n\n::: {#a448d260 .cell execution_count=6}\n``` {.python .cell-code}\nproduct_schema = {\n \"$schema\": \"https://json-schema.org/draft/2020-12/schema\",\n \"title\": \"Product Catalog\",\n \"description\": \"Expected structure for product data\",\n \"type\": \"object\",\n \"properties\": {\n \"sku\": {\"type\": \"string\", \"pattern\": \"^[A-Z]{3}-[0-9]{4}$\"},\n \"price\": {\"type\": \"number\", \"exclusiveMinimum\": 0},\n \"category\": {\"type\": \"string\", \"enum\": [\"electronics\", \"clothing\", \"food\", \"other\"]},\n \"in_stock\": {\"type\": \"boolean\"},\n },\n \"required\": [\"sku\", \"price\", \"category\"],\n}\n\nresult = pb.import_contract(product_schema, format=\"json_schema\")\n\n# Inspect what was mapped\nfor c in result.constraints:\n print(f\" {c.method}({c.kwargs})\")\n```\n\n::: {.cell-output .cell-output-stdout}\n```\n col_vals_not_null({'columns': 'sku'})\n col_vals_regex({'columns': 'sku', 'pattern': '^[A-Z]{3}-[0-9]{4}$'})\n col_vals_not_null({'columns': 'price'})\n col_vals_gt({'columns': 'price', 'value': 0})\n col_vals_not_null({'columns': 'category'})\n col_vals_in_set({'columns': 'category', 'set': ['electronics', 'clothing', 'food', 'other']})\n```\n:::\n:::\n\n\nEach constraint from the JSON Schema has been translated into the corresponding Pointblank method\ncall. The `pattern` keyword on the `sku` field became a `col_vals_regex()` step, `exclusiveMinimum`\nbecame `col_vals_gt()` (note the strict inequality), and the three `required` fields each generated\na `col_vals_not_null()` step. You can iterate over `result.constraints` like this to verify the\ntranslation before running any validation.\n\n### Frictionless Data Table Schema\n\n[Frictionless Data](https://frictionlessdata.io/) is a set of standards for describing and\npackaging data. The **Table Schema** format is particularly well-suited for tabular data validation,\nwith explicit support for column types, constraints, and primary/foreign keys.\n\n**Constraint mapping:**\n\n| Frictionless Feature | Pointblank Method |\n|---|---|\n| `type` | Schema dtype check |\n| `constraints.required: true` | `col_vals_not_null()` |\n| `constraints.unique: true` | `rows_distinct()` |\n| `constraints.minimum` / `maximum` | `col_vals_ge()` / `col_vals_le()` |\n| `constraints.enum` | `col_vals_in_set()` |\n| `constraints.pattern` | `col_vals_regex()` |\n| `primaryKey` | `col_vals_not_null()` + `rows_distinct()` |\n| `foreignKeys` | ⚠ Warning (cross-table not yet supported) |\n\n**Importing a standalone Table Schema:**\n\n::: {#fce6e6c5 .cell execution_count=7}\n``` {.python .cell-code}\ninventory_schema = {\n \"fields\": [\n {\"name\": \"item_id\", \"type\": \"integer\", \"constraints\": {\"required\": True, \"unique\": True}},\n {\"name\": \"name\", \"type\": \"string\", \"constraints\": {\"required\": True}},\n {\"name\": \"quantity\", \"type\": \"integer\", \"constraints\": {\"minimum\": 0}},\n {\"name\": \"warehouse\", \"type\": \"string\", \"constraints\": {\"enum\": [\"NYC\", \"LAX\", \"ORD\"]}},\n ],\n \"primaryKey\": \"item_id\",\n}\n\nresult = pb.import_contract(inventory_schema, format=\"frictionless\")\nprint(result.summary())\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nContract Import Summary\n Format: frictionless\n Columns detected: 4\n Constraints mapped: 7\n Coverage: 100%\n```\n:::\n:::\n\n\nNotice how the `primaryKey` field generates both a not-null check and a uniqueness check for\n`item_id`. This is the correct semantic interpretation: a primary key must always be present and\nmust uniquely identify each row. The field-level `constraints.required` and `constraints.unique`\nalso contribute their own checks, so the adapter deduplicates where appropriate.\n\n**Importing from a Data Package:**\n\nData Packages bundle multiple resources (tables) together. You can select which resource to import\nby name or index:\n\n::: {#04ed1d79 .cell execution_count=8}\n``` {.python .cell-code}\ndata_package = {\n \"name\": \"ecommerce-data\",\n \"resources\": [\n {\n \"name\": \"customers\",\n \"path\": \"customers.csv\",\n \"schema\": {\n \"fields\": [\n {\"name\": \"id\", \"type\": \"integer\", \"constraints\": {\"required\": True}},\n {\"name\": \"email\", \"type\": \"string\"},\n ],\n },\n },\n {\n \"name\": \"orders\",\n \"path\": \"orders.csv\",\n \"schema\": {\n \"fields\": [\n {\"name\": \"order_id\", \"type\": \"integer\", \"constraints\": {\"required\": True}},\n {\"name\": \"amount\", \"type\": \"number\", \"constraints\": {\"minimum\": 0}},\n ],\n },\n },\n ],\n}\n\n# Import a specific resource by name\norders_import = pb.import_contract(data_package, format=\"frictionless\", resource=\"orders\")\nprint(f\"Columns: {[name for name, _ in orders_import.columns]}\")\nprint(f\"Constraints: {len(orders_import.constraints)}\")\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nColumns: ['order_id', 'amount']\nConstraints: 2\n```\n:::\n:::\n\n\nThe `resource=` parameter accepts either a string (the resource name) or an integer (the resource\nindex). When omitted, the first resource in the package is used. This makes it straightforward to\nwork with multi-table data packages where each table has its own schema definition.\n\n## Output Options\n\nOnce you have a `ContractImport`, you can use it in several ways depending on your workflow.\n\n### Direct Validation with `.to_validate()`\n\nThe most common path is to get a `Validate` object, pass your data, and run it:\n\n::: {#8fa85f35 .cell execution_count=9}\n``` {.python .cell-code}\nschema = {\n \"type\": \"object\",\n \"properties\": {\n \"temperature\": {\"type\": \"number\", \"minimum\": -50, \"maximum\": 60},\n \"humidity\": {\"type\": \"number\", \"minimum\": 0, \"maximum\": 100},\n \"station_id\": {\"type\": \"string\"},\n },\n \"required\": [\"temperature\", \"humidity\", \"station_id\"],\n}\n\nweather_data = pl.DataFrame(\n {\n \"temperature\": [22.5, 18.3, -5.1, 35.0, 28.7],\n \"humidity\": [45.0, 78.2, 30.0, 92.5, 55.0],\n \"station_id\": [\"WX-001\", \"WX-002\", \"WX-003\", \"WX-001\", \"WX-004\"],\n }\n)\n\nimported = pb.import_contract(schema, format=\"json_schema\")\nimported.to_validate(data=weather_data).interrogate()\n```\n\n::: {.cell-output .cell-output-display execution_count=9}\n```{=html}\n
\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
Pointblank Validation
2026-07-22|23:23:53
Polars
STEPCOLUMNSVALUESTBLEVALUNITSPASSFAILWECEXT
#4CA64C1\n
\n \n\n col_schema_match\n \n \n \n \n \n \n \n \n \n \n \n\n
\n
\n
col_schema_match()
\n
\n \n 		—SCHEMA\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		————
#4CA64C2\n
\n \n\n col_vals_not_null\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_null()
\n
\n \n 		temperature—\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C3\n
\n \n\n col_vals_ge\n \n \n \n \n \n \n\n
\n
\n
col_vals_ge()
\n
\n \n 		temperature-50\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C4\n
\n \n\n col_vals_le\n \n \n \n \n \n \n\n
\n
\n
col_vals_le()
\n
\n \n 		temperature60\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C5\n
\n \n\n col_vals_not_null\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_null()
\n
\n \n 		humidity—\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C6\n
\n \n\n col_vals_ge\n \n \n \n \n \n \n\n
\n
\n
col_vals_ge()
\n
\n \n 		humidity0\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C7\n
\n \n\n col_vals_le\n \n \n \n \n \n \n\n
\n
\n
col_vals_le()
\n
\n \n 		humidity100\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C8\n
\n \n\n col_vals_not_null\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_null()
\n
\n \n 		station_id—\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
\nNotes\n

Step 1 (schema_check) ✓ Schema validation passed.

\n
\nSchema Comparison\n
\n
\n\n\n\n \n \n \n \n \n \n \n \n\n\n\n \n \n\n\n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
\n TARGET\n \n EXPECTED\n
COLUMNDATA TYPECOLUMNDATA TYPE
1temperatureFloat641temperature✓Float64✓
2humidityFloat642humidity✓Float64✓
3station_idString3station_id✓String✓
Supplied Column Schema:
[('temperature', 'Float64'), ('humidity', 'Float64'), ('station_id', 'String')]
\n
Schema Match Settings
\n
COMPLETE
IN ORDER
COLUMN ≠ column
DTYPE ≠ dtype
float ≠ float64
\n
\n
\n
\n
\n
\n\n
\n```\n:::\n:::\n\n\nThe `.to_validate()` method returns a fully configured `Validate` object with all imported\nconstraints already applied as validation steps. You get the familiar validation report showing\npass/fail counts for each check. Because the `Validate` object is not yet interrogated when\ncreated, you also have the option of adding additional validation steps before calling\n`.interrogate()`.\n\nYou can also pass additional arguments to the `Validate` constructor:\n\n```python\nimported.to_validate(\n data=weather_data,\n tbl_name=\"weather_readings\",\n label=\"Daily sensor check\",\n thresholds=pb.Thresholds(warning=0.05, error=0.10),\n)\n```\n\nAny keyword argument accepted by the `Validate` class can be passed through here, including\n`tbl_name`, `label`, `thresholds`, `owner`, and `consumers`. This gives you full control over\nhow the validation is configured without needing to modify the import result.\n\n### Creating a Reusable Contract with `.to_contract()`\n\nIf you want to store the imported schema as a Pointblank `Contract` for use in pipelines or\nrepeated validation:\n\n::: {#616ee628 .cell execution_count=10}\n``` {.python .cell-code}\nimported = pb.import_contract(inventory_schema, format=\"frictionless\")\ncontract = imported.to_contract(name=\"inventory_check\", version=\"1.0.0\", owner=\"warehouse-team\")\nprint(contract)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nContract(name='inventory_check', direction='source', version='1.0.0', schema=, steps=7)\n```\n:::\n:::\n\n\nThe resulting `Contract` can be serialized to YAML, used in `Pipeline`, or shared with other teams.\nThis is particularly valuable when you want to maintain a stable contract definition that outlives\nthe original external schema file. The `Contract` object carries all the metadata (version, owner,\ndescription) that makes it suitable for team workflows and CI/CD pipelines.\n\n### Generating Python Code with `.to_python()`\n\nWhen you want to see (or save) the equivalent Pointblank Python code that would be generated from\nthe import:\n\n::: {#ec1d077e .cell execution_count=11}\n``` {.python .cell-code}\nimported = pb.import_contract(user_schema, format=\"json_schema\")\nprint(imported.to_python())\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nimport pointblank as pb\n\nvalidation = (\n pb.Validate(data=data)\n .col_schema_match(schema=pb.Schema(user_id=\"Int64\", email=\"String\", age=\"Int64\", status=\"String\"))\n .col_vals_not_null(columns='user_id')\n .col_vals_not_null(columns='email')\n .col_vals_within_spec(columns='email', spec='email')\n .col_vals_ge(columns='age', value=0)\n .col_vals_le(columns='age', value=150)\n .col_vals_in_set(columns='status', set=['active', 'inactive', 'pending'])\n)\n\nvalidation.interrogate()\n```\n:::\n:::\n\n\nThe generated code is syntactically valid Python that you can copy directly into a script or\nnotebook. It uses the standard Pointblank method-chaining style, making it easy to read and modify.\nThis is especially useful for:\n\n- understanding exactly what validation steps an import produces\n- generating starter code that you can then customize\n- documentation and onboarding (show teams what their schema \"means\" in validation terms)\n\nOnce you have the generated code, you can paste it into your project and modify it freely. Add\nextra validation steps, remove checks that don't apply, or adjust parameter values. The generated\ncode has no dependency on the original schema file, so it serves as a clean handoff point between\nthe schema world and your Python codebase.\n\n### Generating YAML with `.to_yaml()`\n\nFor workflows that use Pointblank's YAML-based validation:\n\n::: {#36221395 .cell execution_count=12}\n``` {.python .cell-code}\nimported = pb.import_contract(user_schema, format=\"json_schema\")\nprint(imported.to_yaml())\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nvalidation:\n steps:\n - col_schema_match:\n schema:\n user_id: Int64\n email: String\n age: Int64\n status: String\n - col_vals_not_null:\n columns: user_id\n - col_vals_not_null:\n columns: email\n - col_vals_within_spec:\n columns: email\n spec: email\n - col_vals_ge:\n columns: age\n value: 0\n - col_vals_le:\n columns: age\n value: 150\n - col_vals_in_set:\n columns: status\n set:\n - active\n - inactive\n - pending\n\n```\n:::\n:::\n\n\nThe YAML output follows Pointblank's validation YAML format, with each constraint appearing as a\nseparate step entry. You can save this output to a file and use it with `pb.yaml_interrogate()` or\n`pb.validate_yaml()` for configuration-driven workflows where validation rules are managed as YAML\nfiles rather than Python code.\n\n## Exporting Contracts\n\nThe reverse operation (taking a Pointblank `Contract` or `Validate` object and writing it out in\nan external format) is handled by `export_contract()`:\n\n::: {#9cdecb27 .cell execution_count=13}\n``` {.python .cell-code}\n# Create a contract\ncontract = pb.Contract(\n name=\"sensor_data\",\n schema=pb.Schema(temperature=\"Float64\", humidity=\"Float64\", station_id=\"String\"),\n steps=[\n pb.Step(\"col_vals_ge\", columns=\"temperature\", value=-50),\n pb.Step(\"col_vals_le\", columns=\"temperature\", value=60),\n pb.Step(\"col_vals_ge\", columns=\"humidity\", value=0),\n pb.Step(\"col_vals_le\", columns=\"humidity\", value=100),\n pb.Step(\"col_vals_not_null\", columns=[\"temperature\", \"humidity\", \"station_id\"]),\n ],\n)\n\n# Export to JSON Schema\njson_schema = pb.export_contract(contract, format=\"json_schema\")\njson_schema\n```\n\n::: {.cell-output .cell-output-display execution_count=13}\n```\n{'$schema': 'https://json-schema.org/draft/2020-12/schema',\n 'type': 'object',\n 'title': 'sensor_data',\n 'properties': {'temperature': {'type': 'number',\n 'minimum': -50,\n 'maximum': 60},\n 'humidity': {'type': 'number', 'minimum': 0, 'maximum': 100},\n 'station_id': {'type': 'string'}},\n 'required': ['temperature', 'humidity', 'station_id']}\n```\n:::\n:::\n\n\n::: {#d97d8e72 .cell execution_count=14}\n``` {.python .cell-code}\n# Export to Frictionless Table Schema\ntable_schema = pb.export_contract(contract, format=\"frictionless\")\ntable_schema\n```\n\n::: {.cell-output .cell-output-display execution_count=14}\n```\n{'fields': [{'name': 'temperature',\n 'type': 'number',\n 'constraints': {'minimum': -50, 'maximum': 60, 'required': True}},\n {'name': 'humidity',\n 'type': 'number',\n 'constraints': {'minimum': 0, 'maximum': 100, 'required': True}},\n {'name': 'station_id', 'type': 'string', 'constraints': {'required': True}}]}\n```\n:::\n:::\n\n\nEach format produces the output structure that is native to that standard. JSON Schema export\ncreates a valid `$schema`-annotated document with `properties`, `type`, and `required` fields.\nFrictionless export creates a Table Schema with `fields` and `constraints` entries. Both formats\ncan be fed directly into tools that consume those standards, such as form validators, data catalogs,\nor documentation generators.\n\nYou can also write directly to a file:\n\n```python\npb.export_contract(contract, \"output/sensor_data.schema.json\", format=\"json_schema\")\npb.export_contract(contract, \"output/sensor_data.resource.json\", format=\"frictionless\")\n```\n\nWhen a `destination` path is provided, the output is written to that file (creating parent\ndirectories as needed) and also returned from the function. This makes it convenient to both\npersist the output and inspect it in the same call.\n\n## Round-Trip Fidelity\n\nImporting a schema and then exporting it back should produce an equivalent result. This is\nimportant for workflows where you maintain schemas in an external format but want to validate with\nPointblank:\n\n::: {#5cfaa71f .cell execution_count=15}\n``` {.python .cell-code}\n# Start with a JSON Schema\noriginal = {\n \"type\": \"object\",\n \"properties\": {\n \"score\": {\"type\": \"integer\", \"minimum\": 0, \"maximum\": 100},\n \"grade\": {\"type\": \"string\", \"enum\": [\"A\", \"B\", \"C\", \"D\", \"F\"]},\n },\n \"required\": [\"score\"],\n}\n\n# Import → Contract → Export\nimported = pb.import_contract(original, format=\"json_schema\")\ncontract = imported.to_contract(name=\"grades\")\nexported = pb.export_contract(contract, format=\"json_schema\")\n\n# The exported schema preserves the constraints\nprint(f\"Original constraints on 'score': minimum={original['properties']['score']['minimum']}, \"\n f\"maximum={original['properties']['score']['maximum']}\")\nprint(f\"Exported constraints on 'score': minimum={exported['properties']['score'].get('minimum')}, \"\n f\"maximum={exported['properties']['score'].get('maximum')}\")\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nOriginal constraints on 'score': minimum=0, maximum=100\nExported constraints on 'score': minimum=0, maximum=100\n```\n:::\n:::\n\n\nRound-trip fidelity is tested as part of Pointblank's test suite. The general guarantee is that\nany constraint that can be expressed in both Pointblank and the target format will survive the\nround trip. Constraints that are unique to one format (like JSON Schema's `$ref` or Pointblank's\n`pre=` argument) may not survive, but the core numeric bounds, enum checks, null checks, and\npattern constraints will always round-trip cleanly.\n\n## Auto-Detection\n\nWhen the format is obvious from the source content, you can omit the `format=` parameter:\n\n::: {#42cbebf9 .cell execution_count=16}\n``` {.python .cell-code}\n# JSON Schema: detected by presence of \"$schema\" or \"type\" + \"properties\"\nresult = pb.import_contract({\"type\": \"object\", \"properties\": {\"x\": {\"type\": \"integer\"}}})\nprint(f\"Detected: {result.source_format}\")\n\n# Frictionless: detected by presence of \"fields\" list\nresult = pb.import_contract({\"fields\": [{\"name\": \"x\", \"type\": \"integer\"}]})\nprint(f\"Detected: {result.source_format}\")\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nDetected: json_schema\nDetected: frictionless\n```\n:::\n:::\n\n\nFor file-based imports, the extension is also used for detection (`.schema.json` maps to JSON\nSchema, `.resource.json` or `.datapackage.json` maps to Frictionless). Auto-detection is a\nconvenience feature that works well for common cases. When working with ambiguous files or dict\ninputs that could match multiple formats, it is best to specify `format=` explicitly to avoid\nany possibility of misdetection.\n\n## Combining Imports with Extra Checks\n\nAn imported schema gives you a baseline, but you can always add more Pointblank checks on top:\n\n::: {#9d80e62c .cell execution_count=17}\n``` {.python .cell-code}\nimported = pb.import_contract(user_schema, format=\"json_schema\")\n\n# Start from the import but add custom checks\nvalidation = (\n imported\n .to_validate(data=users, tbl_name=\"enriched_check\")\n .col_vals_regex(columns=\"email\", pattern=r\".*\\.(com|io|org|net|co)$\")\n .rows_distinct(columns_subset=\"user_id\")\n .interrogate()\n)\n\nvalidation\n```\n\n::: {.cell-output .cell-output-display execution_count=17}\n```{=html}\n
\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
Pointblank Validation
2026-07-22|23:23:53
Polarsenriched_check
STEPCOLUMNSVALUESTBLEVALUNITSPASSFAILWECEXT
#4CA64C1\n
\n \n\n col_schema_match\n \n \n \n \n \n \n \n \n \n \n \n\n
\n
\n
col_schema_match()
\n
\n \n 		—SCHEMA\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		————
#4CA64C2\n
\n \n\n col_vals_not_null\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_null()
\n
\n \n 		user_id—\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C3\n
\n \n\n col_vals_not_null\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_null()
\n
\n \n 		email—\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C4\n
\n \n\n col_vals_within_spec\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_within_spec()
\n
\n \n 		emailemail\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C5\n
\n \n\n col_vals_ge\n \n \n \n \n \n \n\n
\n
\n
col_vals_ge()
\n
\n \n 		age0\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C6\n
\n \n\n col_vals_le\n \n \n \n \n \n \n\n
\n
\n
col_vals_le()
\n
\n \n 		age150\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C7\n
\n \n\n col_vals_in_set\n \n \n \n \n \n \n\n
\n
\n
col_vals_in_set()
\n
\n \n 		statusactive, inactive, pending\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C8\n
\n \n\n col_vals_regex\n \n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_regex()
\n
\n \n 		email.*\\.(com|io|org|net|co)$\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C9\n
\n \n\n rows_distinct\n \n \n \n \n \n \n \n \n \n \n\n
\n
\n
rows_distinct()
\n
\n \n 		user_id—\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
\nNotes\n

Step 1 (schema_check) ✓ Schema validation passed.

\n
\nSchema Comparison\n
\n
\n\n\n\n \n \n \n \n \n \n \n \n\n\n\n \n \n\n\n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
\n TARGET\n \n EXPECTED\n
COLUMNDATA TYPECOLUMNDATA TYPE
1user_idInt641user_id✓Int64✓
2emailString2email✓String✓
3ageInt643age✓Int64✓
4statusString4status✓String✓
Supplied Column Schema:
[('user_id', 'Int64'), ('email', 'String'), ('age', 'Int64'), ('status', 'String')]
\n
Schema Match Settings
\n
COMPLETE
IN ORDER
COLUMN ≠ column
DTYPE ≠ dtype
float ≠ float64
\n
\n
\n
\n
\n
\n\n
\n```\n:::\n:::\n\n\nThis pattern works well when the external schema covers structural and type constraints, but your\nteam has additional business rules that only make sense in the Pointblank context. The imported\nconstraints form the foundation, and your additional `.col_vals_*()` or `.rows_*()` calls layer\non top. Because `.to_validate()` returns a standard `Validate` object, you have full access to\nthe entire Pointblank API for adding checks, setting thresholds, or attaching actions.\n\n## Migration from Other Tools\n\nA key use case for `import_contract()` is **migration**: bringing existing validation definitions\nfrom other tools into Pointblank without manual rewriting.\n\n### Coming from JSON Schema\n\nIf your team uses JSON Schema for API validation and you want the same rules applied to DataFrames:\n\n```python\n# Your existing JSON Schema (maybe generated by your API framework)\nresult = pb.import_contract(\"api/schemas/user.schema.json\")\n\n# Now use it for DataFrame validation in your data pipeline\nvalidation = result.to_validate(data=raw_users_df).interrogate()\n```\n\nThis approach is particularly powerful when your API team already maintains JSON Schema definitions\nfor request/response validation. Those same schemas can now serve double duty: validating API\npayloads at the service boundary and validating the resulting DataFrames in your analytics pipeline.\nYou get consistent enforcement across both layers without writing the rules twice.\n\n### Coming from Frictionless\n\nIf you have data packages from open data sources or research datasets:\n\n```python\n# Import from an existing data package descriptor\nresult = pb.import_contract(\"data/datapackage.json\", resource=\"observations\")\n\n# Validate the actual CSV data against the declared schema\nvalidation = result.to_validate(data=observations_df).interrogate()\n```\n\nFrictionless Data Packages are common in open data portals, government datasets, and academic\nresearch repositories. By importing their Table Schemas directly, you can validate downloaded data\nagainst its declared structure without needing to manually inspect the descriptor file and rewrite\neach constraint. This is especially valuable when working with unfamiliar datasets where the schema\ndescriptor is your primary documentation of what the data should contain.\n\n### Generating a Starting Point\n\nEven if you don't plan to keep using the external format, importing is a great way to bootstrap\na Pointblank contract:\n\n```python\n# Import from your existing schema\nimported = pb.import_contract(\"legacy_schema.json\", format=\"json_schema\")\n\n# Save as a YAML contract you'll maintain going forward\ncontract = imported.to_contract(name=\"my_table\", version=\"1.0.0\")\ncontract.to_yaml(\"contracts/my_table.yaml\")\n```\n\nNow you have a Pointblank-native contract that you can extend and evolve independently of the\noriginal source. You can add new validation steps, adjust thresholds, or incorporate business rules\nthat go beyond what the original schema format could express.\n\n## Conclusion\n\nThe contract import/export system lets you bridge the gap between external schema definitions and\nPointblank's validation engine. Rather than maintaining duplicate specifications across tools, you\ncan keep your source of truth in whichever format suits your team and import it into Pointblank\nwhenever you need runtime validation. The key points to remember:\n\n- Use `pb.import_contract()` to read external schemas and translate them into Pointblank checks\n- The `ContractImport` object gives you multiple output options: direct validation, reusable\n contracts, generated Python code, or YAML\n- Check `.coverage` and `.warnings` to understand how completely the translation covered your\n original schema\n- Use `pb.export_contract()` to write Pointblank contracts back to external formats for sharing\n with other tools\n- Combine imports with additional Pointblank-specific checks for the most thorough validation\n coverage\n\nWhether you are migrating from another validation tool, bootstrapping contracts from existing\nschemas, or maintaining interoperability with external systems, the adapter framework gives you a\nclean path between external specifications and Pointblank's validation engine. As new adapters are\nadded in future releases, the same `import_contract()` interface will continue to work, so any code\nyou write today will gain new format support automatically.\n\n", + "markdown": "---\ntitle: Importing External Schemas\njupyter: python3\nhtml-table-processing: none\nbread-crumbs: false\n---\n\n\n\nMany teams already have data schemas defined in other tools: JSON Schema files for API validation,\nFrictionless Table Schemas for open data, dbt `schema.yml` files for analytics pipelines, or ODCS\ndata contracts for cross-team governance. Rather than manually rewriting these specifications as\nPointblank validation steps, you can **import** them directly.\n\nThe `import_contract()` function reads an external schema definition and produces a `ContractImport`\nobject containing everything Pointblank needs to validate data: column types, constraints, and\nmapped validation steps. From there you can create a `Validate` workflow, a `Contract` object,\ngenerate equivalent Python code, or produce a YAML definition, all with a single function call.\n\n## Quick Start\n\nThe fastest path from an external schema to running validation:\n\n::: {#7e8b55f0 .cell execution_count=2}\n``` {.python .cell-code}\nimport pointblank as pb\nimport polars as pl\n\n# Define a JSON Schema (could also be loaded from a file)\nuser_schema = {\n \"type\": \"object\",\n \"properties\": {\n \"user_id\": {\"type\": \"integer\"},\n \"email\": {\"type\": \"string\", \"format\": \"email\"},\n \"age\": {\"type\": \"integer\", \"minimum\": 0, \"maximum\": 150},\n \"status\": {\"type\": \"string\", \"enum\": [\"active\", \"inactive\", \"pending\"]},\n },\n \"required\": [\"user_id\", \"email\"],\n}\n\n# Import the schema\nresult = pb.import_contract(user_schema, format=\"json_schema\")\n\n# Create sample data and validate\nusers = pl.DataFrame(\n {\n \"user_id\": [1, 2, 3, 4, 5],\n \"email\": [\n \"alice@example.com\",\n \"bob@corp.io\",\n \"charlie@mail.org\",\n \"dave@startup.co\",\n \"eve@company.net\",\n ],\n \"age\": [28, 34, 45, 22, 31],\n \"status\": [\"active\", \"active\", \"inactive\", \"pending\", \"active\"],\n }\n)\n\nresult.to_validate(data=users).interrogate()\n```\n\n::: {.cell-output .cell-output-display execution_count=2}\n```{=html}\n
\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
Pointblank Validation
2026-08-10|16:45:27
Polars
STEPCOLUMNSVALUESTBLEVALUNITSPASSFAILWECEXT
#4CA64C1\n
\n \n\n col_schema_match\n \n \n \n \n \n \n \n \n \n \n \n\n
\n
\n
col_schema_match()
\n
\n \n 		—SCHEMA\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		————
#4CA64C2\n
\n \n\n col_vals_not_null\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_null()
\n
\n \n 		user_id—\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C3\n
\n \n\n col_vals_not_null\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_null()
\n
\n \n 		email—\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C4\n
\n \n\n col_vals_within_spec\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_within_spec()
\n
\n \n 		emailemail\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C5\n
\n \n\n col_vals_ge\n \n \n \n \n \n \n\n
\n
\n
col_vals_ge()
\n
\n \n 		age0\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C6\n
\n \n\n col_vals_le\n \n \n \n \n \n \n\n
\n
\n
col_vals_le()
\n
\n \n 		age150\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C7\n
\n \n\n col_vals_in_set\n \n \n \n \n \n \n\n
\n
\n
col_vals_in_set()
\n
\n \n 		statusactive, inactive, pending\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
\nNotes\n

Step 1 (schema_check) ✓ Schema validation passed.

\n
\nSchema Comparison\n
\n
\n\n\n\n \n \n \n \n \n \n \n \n\n\n\n \n \n\n\n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
\n TARGET\n \n EXPECTED\n
COLUMNDATA TYPECOLUMNDATA TYPE
1user_idInt641user_id✓Int64✓
2emailString2email✓String✓
3ageInt643age✓Int64✓
4statusString4status✓String✓
Supplied Column Schema:
[('user_id', 'Int64'), ('email', 'String'), ('age', 'Int64'), ('status', 'String')]
\n
Schema Match Settings
\n
COMPLETE
IN ORDER
COLUMN ≠ column
DTYPE ≠ dtype
float ≠ float64
\n
\n
\n
\n
\n
\n\n
\n```\n:::\n:::\n\n\nThat's it. The JSON Schema `minimum`, `maximum`, `enum`, `format`, and `required` keywords were\nautomatically translated into the appropriate Pointblank validation steps. Each keyword becomes a\ndedicated validation check: `minimum` becomes `col_vals_ge()`, `maximum` becomes `col_vals_le()`,\n`enum` becomes `col_vals_in_set()`, and so on. The schema's `required` array generates\n`col_vals_not_null()` steps for each listed field, ensuring that null values are caught at\nvalidation time.\n\n## How It Works\n\nThe import process has three stages:\n\n1. **Parse**: the external schema is read (from a file path, a dict, or a Python object)\n2. **Map**: each constraint in the source format is translated to a Pointblank validation method\n3. **Package**: the results are stored in a `ContractImport` object with multiple output options\n\n```{mermaid}\nflowchart LR\n A[External Schema] --> B[import_contract]\n B --> C[ContractImport]\n C --> D[.to_validate‹data›]\n C --> E[.to_contract‹›]\n C --> F[.to_python‹›]\n C --> G[.to_yaml‹›]\n```\n\nThe `ContractImport` object is your bridge between the external world and Pointblank. It doesn't\nexecute anything on its own. Rather, it holds the *translated specification* and lets you choose\nhow to use it. This separation is intentional: you can inspect the translation results, check\nfor any warnings, and decide how to proceed before committing to a particular output format.\n\n## The `ContractImport` Object\n\nAfter calling `import_contract()`, you get back a `ContractImport` with these key attributes and\nmethods:\n\n| Attribute / Method | Description |\n|---|---|\n| `.columns` | List of `(column_name, dtype)` tuples detected from the source |\n| `.constraints` | List of `MappedConstraint` objects (method + kwargs) |\n| `.warnings` | Messages about constraints that couldn't be translated |\n| `.coverage` | Fraction of source constraints successfully mapped (0.0–1.0) |\n| `.metadata` | Extra metadata (title, description) from the source |\n| `.to_validate(data)` | Build a `Validate` object ready for `.interrogate()` |\n| `.to_contract(name)` | Build a `Contract` object for pipeline use |\n| `.to_python()` | Generate equivalent Python code as a string |\n| `.to_yaml()` | Generate Pointblank YAML configuration |\n| `.summary()` | Return a human-readable summary |\n\n### Inspecting What Was Imported\n\nBefore running validation, it's often useful to inspect what the import produced:\n\n::: {#4e857938 .cell execution_count=3}\n``` {.python .cell-code}\nresult = pb.import_contract(user_schema, format=\"json_schema\")\nprint(result.summary())\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nContract Import Summary\n Format: json_schema\n Columns detected: 4\n Constraints mapped: 6\n Coverage: 100%\n```\n:::\n:::\n\n\nIf any constraints couldn't be mapped, they appear in `.warnings`:\n\n::: {#56b30e45 .cell execution_count=4}\n``` {.python .cell-code}\n# A schema with an unmappable format\nschema_with_date = {\n \"type\": \"object\",\n \"properties\": {\n \"created_at\": {\"type\": \"string\", \"format\": \"date-time\"},\n },\n}\nresult = pb.import_contract(schema_with_date, format=\"json_schema\")\n\nif result.warnings:\n for w in result.warnings:\n print(f\"⚠ {w}\")\n\nprint(f\"\\nCoverage: {result.coverage:.0%}\")\n```\n\n::: {.cell-output .cell-output-stdout}\n```\n⚠ Column 'created_at': JSON Schema format 'date-time' has no Pointblank equivalent — skipped.\n\nCoverage: 0%\n```\n:::\n:::\n\n\nThe `coverage` metric tells you what fraction of the source constraints were successfully\ntranslated. A coverage of 100% means everything mapped cleanly; lower values mean some constraints\nwere skipped (with details in `warnings`). This transparency is important because no translation\nbetween formats is perfect. By checking `coverage` and `warnings` before running validation, you\ncan be confident about exactly which parts of your original schema are being enforced and which\nparts might need manual attention.\n\n## Supported Formats\n\nPointblank ships with adapters for four widely-used schema and contract formats.\n\n::: {#643e3523 .cell execution_count=5}\n``` {.python .cell-code}\npb.list_adapters()\n```\n\n::: {.cell-output .cell-output-display execution_count=5}\n```\n{'dbt': {'class': 'DbtAdapter',\n 'file_extensions': ['.yml', '.yaml'],\n 'supports_import': True,\n 'supports_export': True},\n 'frictionless': {'class': 'FrictionlessAdapter',\n 'file_extensions': ['.resource.json', '.datapackage.json'],\n 'supports_import': True,\n 'supports_export': True},\n 'json_schema': {'class': 'JSONSchemaAdapter',\n 'file_extensions': ['.schema.json'],\n 'supports_import': True,\n 'supports_export': True},\n 'odcs': {'class': 'ODCSAdapter',\n 'file_extensions': ['.odcs.yml', '.odcs.yaml', '.odcs.json'],\n 'supports_import': True,\n 'supports_export': True}}\n```\n:::\n:::\n\n\n### JSON Schema\n\n[JSON Schema](https://json-schema.org/) is a widely used format for describing the structure of JSON\ndata. Because tabular data (DataFrames) can be modeled as arrays of JSON objects, JSON Schema is a\nnatural fit for defining column-level constraints.\n\n**Constraint mapping:**\n\n| JSON Schema Keyword | Pointblank Method |\n|---|---|\n| `type: \"integer\"` / `\"number\"` / `\"string\"` / `\"boolean\"` | Schema dtype check |\n| `minimum` | `col_vals_ge()` |\n| `maximum` | `col_vals_le()` |\n| `exclusiveMinimum` | `col_vals_gt()` |\n| `exclusiveMaximum` | `col_vals_lt()` |\n| `enum` | `col_vals_in_set()` |\n| `pattern` | `col_vals_regex()` |\n| `format: \"email\"` | `col_vals_within_spec(spec=\"email\")` |\n| `format: \"uri\"` | `col_vals_within_spec(spec=\"url\")` |\n| `format: \"ipv4\"` / `\"ipv6\"` | `col_vals_within_spec(spec=\"ipv4\"/\"ipv6\")` |\n| `const` | `col_vals_eq()` |\n| `required` (array of field names) | `col_vals_not_null()` |\n\n**Importing from a file:**\n\n```python\n# File-based import (auto-detects .schema.json extension)\nresult = pb.import_contract(\"models/user_profile.schema.json\")\n```\n\n**Importing from a dict:**\n\n::: {#27d0a42f .cell execution_count=6}\n``` {.python .cell-code}\nproduct_schema = {\n \"$schema\": \"https://json-schema.org/draft/2020-12/schema\",\n \"title\": \"Product Catalog\",\n \"description\": \"Expected structure for product data\",\n \"type\": \"object\",\n \"properties\": {\n \"sku\": {\"type\": \"string\", \"pattern\": \"^[A-Z]{3}-[0-9]{4}$\"},\n \"price\": {\"type\": \"number\", \"exclusiveMinimum\": 0},\n \"category\": {\"type\": \"string\", \"enum\": [\"electronics\", \"clothing\", \"food\", \"other\"]},\n \"in_stock\": {\"type\": \"boolean\"},\n },\n \"required\": [\"sku\", \"price\", \"category\"],\n}\n\nresult = pb.import_contract(product_schema, format=\"json_schema\")\n\n# Inspect what was mapped\nfor c in result.constraints:\n print(f\" {c.method}({c.kwargs})\")\n```\n\n::: {.cell-output .cell-output-stdout}\n```\n col_vals_not_null({'columns': 'sku'})\n col_vals_regex({'columns': 'sku', 'pattern': '^[A-Z]{3}-[0-9]{4}$'})\n col_vals_not_null({'columns': 'price'})\n col_vals_gt({'columns': 'price', 'value': 0})\n col_vals_not_null({'columns': 'category'})\n col_vals_in_set({'columns': 'category', 'set': ['electronics', 'clothing', 'food', 'other']})\n```\n:::\n:::\n\n\nEach constraint from the JSON Schema has been translated into the corresponding Pointblank method\ncall. The `pattern` keyword on the `sku` field became a `col_vals_regex()` step, `exclusiveMinimum`\nbecame `col_vals_gt()` (note the strict inequality), and the three `required` fields each generated\na `col_vals_not_null()` step. You can iterate over `result.constraints` like this to verify the\ntranslation before running any validation.\n\n### Frictionless Data Table Schema\n\n[Frictionless Data](https://frictionlessdata.io/) is a set of standards for describing and\npackaging data. The **Table Schema** format is particularly well-suited for tabular data validation,\nwith explicit support for column types, constraints, and primary/foreign keys.\n\n**Constraint mapping:**\n\n| Frictionless Feature | Pointblank Method |\n|---|---|\n| `type` | Schema dtype check |\n| `constraints.required: true` | `col_vals_not_null()` |\n| `constraints.unique: true` | `rows_distinct()` |\n| `constraints.minimum` / `maximum` | `col_vals_ge()` / `col_vals_le()` |\n| `constraints.enum` | `col_vals_in_set()` |\n| `constraints.pattern` | `col_vals_regex()` |\n| `primaryKey` | `col_vals_not_null()` + `rows_distinct()` |\n| `foreignKeys` | ⚠ Warning (cross-table not yet supported) |\n\n**Importing a standalone Table Schema:**\n\n::: {#a5392d80 .cell execution_count=7}\n``` {.python .cell-code}\ninventory_schema = {\n \"fields\": [\n {\"name\": \"item_id\", \"type\": \"integer\", \"constraints\": {\"required\": True, \"unique\": True}},\n {\"name\": \"name\", \"type\": \"string\", \"constraints\": {\"required\": True}},\n {\"name\": \"quantity\", \"type\": \"integer\", \"constraints\": {\"minimum\": 0}},\n {\"name\": \"warehouse\", \"type\": \"string\", \"constraints\": {\"enum\": [\"NYC\", \"LAX\", \"ORD\"]}},\n ],\n \"primaryKey\": \"item_id\",\n}\n\nresult = pb.import_contract(inventory_schema, format=\"frictionless\")\nprint(result.summary())\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nContract Import Summary\n Format: frictionless\n Columns detected: 4\n Constraints mapped: 7\n Coverage: 100%\n```\n:::\n:::\n\n\nNotice how the `primaryKey` field generates both a not-null check and a uniqueness check for\n`item_id`. This is the correct semantic interpretation: a primary key must always be present and\nmust uniquely identify each row. The field-level `constraints.required` and `constraints.unique`\nalso contribute their own checks, so the adapter deduplicates where appropriate.\n\n**Importing from a Data Package:**\n\nData Packages bundle multiple resources (tables) together. You can select which resource to import\nby name or index:\n\n::: {#ffebb1b4 .cell execution_count=8}\n``` {.python .cell-code}\ndata_package = {\n \"name\": \"ecommerce-data\",\n \"resources\": [\n {\n \"name\": \"customers\",\n \"path\": \"customers.csv\",\n \"schema\": {\n \"fields\": [\n {\"name\": \"id\", \"type\": \"integer\", \"constraints\": {\"required\": True}},\n {\"name\": \"email\", \"type\": \"string\"},\n ],\n },\n },\n {\n \"name\": \"orders\",\n \"path\": \"orders.csv\",\n \"schema\": {\n \"fields\": [\n {\"name\": \"order_id\", \"type\": \"integer\", \"constraints\": {\"required\": True}},\n {\"name\": \"amount\", \"type\": \"number\", \"constraints\": {\"minimum\": 0}},\n ],\n },\n },\n ],\n}\n\n# Import a specific resource by name\norders_import = pb.import_contract(data_package, format=\"frictionless\", resource=\"orders\")\nprint(f\"Columns: {[name for name, _ in orders_import.columns]}\")\nprint(f\"Constraints: {len(orders_import.constraints)}\")\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nColumns: ['order_id', 'amount']\nConstraints: 2\n```\n:::\n:::\n\n\nThe `resource=` parameter accepts either a string (the resource name) or an integer (the resource\nindex). When omitted, the first resource in the package is used. This makes it straightforward to\nwork with multi-table data packages where each table has its own schema definition.\n\n### dbt `schema.yml`\n\n[dbt](https://www.getdbt.com/) (data build tool) is the standard for transformation workflows in\nmodern analytics stacks. dbt models declare column-level tests in `schema.yml` files, and Pointblank\ncan import these definitions directly.\n\n**Constraint mapping:**\n\n| dbt Feature | Pointblank Method |\n|---|---|\n| `data_type` | Schema dtype check |\n| `not_null` test | `col_vals_not_null()` |\n| `unique` test | `rows_distinct()` |\n| `accepted_values` test | `col_vals_in_set()` |\n| `relationships` test | Warning (cross-table not yet supported) |\n\nBoth the newer `data_tests` key (dbt v1.8+) and the legacy `tests` key are supported.\n\n**Importing from a dict:**\n\n::: {#8f761f44 .cell execution_count=9}\n``` {.python .cell-code}\ndbt_schema = {\n \"version\": 2,\n \"models\": [\n {\n \"name\": \"orders\",\n \"description\": \"Cleaned order data\",\n \"columns\": [\n {\n \"name\": \"order_id\",\n \"data_type\": \"integer\",\n \"data_tests\": [\"not_null\", \"unique\"],\n },\n {\n \"name\": \"status\",\n \"data_type\": \"string\",\n \"data_tests\": [\n {\"accepted_values\": {\"values\": [\"pending\", \"shipped\", \"delivered\"]}}\n ],\n },\n {\n \"name\": \"amount\",\n \"data_type\": \"float\",\n },\n ],\n }\n ],\n}\n\nresult = pb.import_contract(dbt_schema, format=\"dbt\")\nprint(result.summary())\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nContract Import Summary\n Format: dbt\n Columns detected: 3\n Constraints mapped: 3\n Coverage: 100%\n```\n:::\n:::\n\n\n**Importing from a file:**\n\n```python\n# File-based import (auto-detects .yml / .yaml with dbt structure)\nresult = pb.import_contract(\"models/schema.yml\", format=\"dbt\")\n```\n\n**Selecting a specific model:**\n\nWhen a `schema.yml` file contains multiple models, use the `model=` parameter to pick one:\n\n::: {#262a7a7c .cell execution_count=10}\n``` {.python .cell-code}\nmulti_model = {\n \"version\": 2,\n \"models\": [\n {\"name\": \"users\", \"columns\": [{\"name\": \"id\", \"data_tests\": [\"not_null\"]}]},\n {\"name\": \"orders\", \"columns\": [{\"name\": \"order_id\", \"data_tests\": [\"not_null\", \"unique\"]}]},\n ],\n}\n\nresult = pb.import_contract(multi_model, format=\"dbt\", model=\"orders\")\nprint(f\"Imported model: {result.metadata.get('title')}\")\nprint(f\"Constraints: {len(result.constraints)}\")\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nImported model: orders\nConstraints: 2\n```\n:::\n:::\n\n\ndbt sources are also supported. When a file contains `sources` with `tables`, each table is treated\nas a model candidate:\n\n```python\nresult = pb.import_contract(\"sources.yml\", format=\"dbt\", model=\"raw_events\")\n```\n\n### ODCS (Open Data Contract Standard)\n\nThe [Open Data Contract Standard](https://github.com/bitol-io/open-data-contract-standard) (ODCS)\nis a vendor-neutral format for defining data contracts between producers and consumers. Pointblank\nsupports both ODCS v2.x and v3.x documents.\n\n**Constraint mapping:**\n\n| ODCS Feature | Pointblank Method |\n|---|---|\n| `logicalType` | Schema dtype check |\n| `isNullable: false` | `col_vals_not_null()` |\n| `isUnique: true` | `rows_distinct()` |\n| `isPrimaryKey: true` | `col_vals_not_null()` + `rows_distinct()` |\n| `enum` / `values` | `col_vals_in_set()` |\n| `pattern` | `col_vals_regex()` |\n| `minimum` / `maximum` | `col_vals_ge()` / `col_vals_le()` |\n| `minLength` / `maxLength` | Warning (no direct equivalent) |\n| Custom checks / SodaCL | Warning (no automatic mapping) |\n\n**Importing an ODCS v3 contract:**\n\n::: {#7630b5b1 .cell execution_count=11}\n``` {.python .cell-code}\nodcs_contract = {\n \"kind\": \"DataContract\",\n \"apiVersion\": \"v3.0.0\",\n \"info\": {\n \"title\": \"Customer Data\",\n \"description\": \"Contract for customer records\",\n },\n \"dataset\": [\n {\n \"table\": \"customers\",\n \"columns\": [\n {\n \"column\": \"customer_id\",\n \"logicalType\": \"integer\",\n \"isNullable\": False,\n \"isUnique\": True,\n },\n {\n \"column\": \"email\",\n \"logicalType\": \"string\",\n \"isNullable\": False,\n \"pattern\": r\"^[^@]+@[^@]+\\.[^@]+$\",\n },\n {\n \"column\": \"tier\",\n \"logicalType\": \"string\",\n \"enum\": [\"free\", \"pro\", \"enterprise\"],\n },\n {\n \"column\": \"lifetime_value\",\n \"logicalType\": \"float\",\n \"minimum\": 0,\n },\n ],\n }\n ],\n}\n\nresult = pb.import_contract(odcs_contract, format=\"odcs\")\nprint(result.summary())\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nContract Import Summary\n Format: odcs\n Format version: v3.0.0\n Columns detected: 4\n Constraints mapped: 6\n Coverage: 100%\n```\n:::\n:::\n\n\n**Selecting a specific table:**\n\nFor contracts with multiple dataset tables, use `table=`:\n\n```python\nresult = pb.import_contract(\"contract.odcs.yml\", format=\"odcs\", table=\"orders\")\n```\n\n**Importing from a file:**\n\n```python\n# YAML files\nresult = pb.import_contract(\"contracts/customer_data.odcs.yml\", format=\"odcs\")\n\n# JSON files\nresult = pb.import_contract(\"contracts/customer_data.odcs.json\", format=\"odcs\")\n```\n\n## Output Options\n\nOnce you have a `ContractImport`, you can use it in several ways depending on your workflow.\n\n### Direct Validation with `.to_validate()`\n\nThe most common path is to get a `Validate` object, pass your data, and run it:\n\n::: {#e9faa4b8 .cell execution_count=12}\n``` {.python .cell-code}\nschema = {\n \"type\": \"object\",\n \"properties\": {\n \"temperature\": {\"type\": \"number\", \"minimum\": -50, \"maximum\": 60},\n \"humidity\": {\"type\": \"number\", \"minimum\": 0, \"maximum\": 100},\n \"station_id\": {\"type\": \"string\"},\n },\n \"required\": [\"temperature\", \"humidity\", \"station_id\"],\n}\n\nweather_data = pl.DataFrame(\n {\n \"temperature\": [22.5, 18.3, -5.1, 35.0, 28.7],\n \"humidity\": [45.0, 78.2, 30.0, 92.5, 55.0],\n \"station_id\": [\"WX-001\", \"WX-002\", \"WX-003\", \"WX-001\", \"WX-004\"],\n }\n)\n\nimported = pb.import_contract(schema, format=\"json_schema\")\nimported.to_validate(data=weather_data).interrogate()\n```\n\n::: {.cell-output .cell-output-display execution_count=12}\n```{=html}\n
\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
Pointblank Validation
2026-08-10|16:45:28
Polars
STEPCOLUMNSVALUESTBLEVALUNITSPASSFAILWECEXT
#4CA64C1\n
\n \n\n col_schema_match\n \n \n \n \n \n \n \n \n \n \n \n\n
\n
\n
col_schema_match()
\n
\n \n 		—SCHEMA\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		————
#4CA64C2\n
\n \n\n col_vals_not_null\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_null()
\n
\n \n 		temperature—\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C3\n
\n \n\n col_vals_ge\n \n \n \n \n \n \n\n
\n
\n
col_vals_ge()
\n
\n \n 		temperature-50\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C4\n
\n \n\n col_vals_le\n \n \n \n \n \n \n\n
\n
\n
col_vals_le()
\n
\n \n 		temperature60\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C5\n
\n \n\n col_vals_not_null\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_null()
\n
\n \n 		humidity—\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C6\n
\n \n\n col_vals_ge\n \n \n \n \n \n \n\n
\n
\n
col_vals_ge()
\n
\n \n 		humidity0\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C7\n
\n \n\n col_vals_le\n \n \n \n \n \n \n\n
\n
\n
col_vals_le()
\n
\n \n 		humidity100\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C8\n
\n \n\n col_vals_not_null\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_null()
\n
\n \n 		station_id—\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
\nNotes\n

Step 1 (schema_check) ✓ Schema validation passed.

\n
\nSchema Comparison\n
\n
\n\n\n\n \n \n \n \n \n \n \n \n\n\n\n \n \n\n\n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
\n TARGET\n \n EXPECTED\n
COLUMNDATA TYPECOLUMNDATA TYPE
1temperatureFloat641temperature✓Float64✓
2humidityFloat642humidity✓Float64✓
3station_idString3station_id✓String✓
Supplied Column Schema:
[('temperature', 'Float64'), ('humidity', 'Float64'), ('station_id', 'String')]
\n
Schema Match Settings
\n
COMPLETE
IN ORDER
COLUMN ≠ column
DTYPE ≠ dtype
float ≠ float64
\n
\n
\n
\n
\n
\n\n
\n```\n:::\n:::\n\n\nThe `.to_validate()` method returns a fully configured `Validate` object with all imported\nconstraints already applied as validation steps. You get the familiar validation report showing\npass/fail counts for each check. Because the `Validate` object is not yet interrogated when\ncreated, you also have the option of adding additional validation steps before calling\n`.interrogate()`.\n\nYou can also pass additional arguments to the `Validate` constructor:\n\n```python\nimported.to_validate(\n data=weather_data,\n tbl_name=\"weather_readings\",\n label=\"Daily sensor check\",\n thresholds=pb.Thresholds(warning=0.05, error=0.10),\n)\n```\n\nAny keyword argument accepted by the `Validate` class can be passed through here, including\n`tbl_name`, `label`, `thresholds`, `owner`, and `consumers`. This gives you full control over\nhow the validation is configured without needing to modify the import result.\n\n### Creating a Reusable Contract with `.to_contract()`\n\nIf you want to store the imported schema as a Pointblank `Contract` for use in pipelines or\nrepeated validation:\n\n::: {#5a672971 .cell execution_count=13}\n``` {.python .cell-code}\nimported = pb.import_contract(inventory_schema, format=\"frictionless\")\ncontract = imported.to_contract(name=\"inventory_check\", version=\"1.0.0\", owner=\"warehouse-team\")\nprint(contract)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nContract(name='inventory_check', direction='source', version='1.0.0', schema=, steps=7)\n```\n:::\n:::\n\n\nThe resulting `Contract` can be serialized to YAML, used in `Pipeline`, or shared with other teams.\nThis is particularly valuable when you want to maintain a stable contract definition that outlives\nthe original external schema file. The `Contract` object carries all the metadata (version, owner,\ndescription) that makes it suitable for team workflows and CI/CD pipelines.\n\n### Generating Python Code with `.to_python()`\n\nWhen you want to see (or save) the equivalent Pointblank Python code that would be generated from\nthe import:\n\n::: {#56ac6816 .cell execution_count=14}\n``` {.python .cell-code}\nimported = pb.import_contract(user_schema, format=\"json_schema\")\nprint(imported.to_python())\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nimport pointblank as pb\n\nvalidation = (\n pb.Validate(data=data)\n .col_schema_match(schema=pb.Schema(user_id=\"Int64\", email=\"String\", age=\"Int64\", status=\"String\"))\n .col_vals_not_null(columns='user_id')\n .col_vals_not_null(columns='email')\n .col_vals_within_spec(columns='email', spec='email')\n .col_vals_ge(columns='age', value=0)\n .col_vals_le(columns='age', value=150)\n .col_vals_in_set(columns='status', set=['active', 'inactive', 'pending'])\n)\n\nvalidation.interrogate()\n```\n:::\n:::\n\n\nThe generated code is syntactically valid Python that you can copy directly into a script or\nnotebook. It uses the standard Pointblank method-chaining style, making it easy to read and modify.\nThis is especially useful for:\n\n- understanding exactly what validation steps an import produces\n- generating starter code that you can then customize\n- documentation and onboarding (show teams what their schema \"means\" in validation terms)\n\nOnce you have the generated code, you can paste it into your project and modify it freely. Add\nextra validation steps, remove checks that don't apply, or adjust parameter values. The generated\ncode has no dependency on the original schema file, so it serves as a clean handoff point between\nthe schema world and your Python codebase.\n\n### Generating YAML with `.to_yaml()`\n\nFor workflows that use Pointblank's YAML-based validation:\n\n::: {#83d8101e .cell execution_count=15}\n``` {.python .cell-code}\nimported = pb.import_contract(user_schema, format=\"json_schema\")\nprint(imported.to_yaml())\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nvalidation:\n steps:\n - col_schema_match:\n schema:\n user_id: Int64\n email: String\n age: Int64\n status: String\n - col_vals_not_null:\n columns: user_id\n - col_vals_not_null:\n columns: email\n - col_vals_within_spec:\n columns: email\n spec: email\n - col_vals_ge:\n columns: age\n value: 0\n - col_vals_le:\n columns: age\n value: 150\n - col_vals_in_set:\n columns: status\n set:\n - active\n - inactive\n - pending\n\n```\n:::\n:::\n\n\nThe YAML output follows Pointblank's validation YAML format, with each constraint appearing as a\nseparate step entry. You can save this output to a file and use it with `pb.yaml_interrogate()` or\n`pb.validate_yaml()` for configuration-driven workflows where validation rules are managed as YAML\nfiles rather than Python code.\n\n## Exporting Contracts\n\nThe reverse operation (taking a Pointblank `Contract` or `Validate` object and writing it out in\nan external format) is handled by `export_contract()`:\n\n::: {#19530e8f .cell execution_count=16}\n``` {.python .cell-code}\n# Create a contract\ncontract = pb.Contract(\n name=\"sensor_data\",\n schema=pb.Schema(temperature=\"Float64\", humidity=\"Float64\", station_id=\"String\"),\n steps=[\n pb.Step(\"col_vals_ge\", columns=\"temperature\", value=-50),\n pb.Step(\"col_vals_le\", columns=\"temperature\", value=60),\n pb.Step(\"col_vals_ge\", columns=\"humidity\", value=0),\n pb.Step(\"col_vals_le\", columns=\"humidity\", value=100),\n pb.Step(\"col_vals_not_null\", columns=[\"temperature\", \"humidity\", \"station_id\"]),\n ],\n)\n\n# Export to JSON Schema\njson_schema = pb.export_contract(contract, format=\"json_schema\")\njson_schema\n```\n\n::: {.cell-output .cell-output-display execution_count=16}\n```\n{'$schema': 'https://json-schema.org/draft/2020-12/schema',\n 'type': 'object',\n 'title': 'sensor_data',\n 'properties': {'temperature': {'type': 'number',\n 'minimum': -50,\n 'maximum': 60},\n 'humidity': {'type': 'number', 'minimum': 0, 'maximum': 100},\n 'station_id': {'type': 'string'}},\n 'required': ['temperature', 'humidity', 'station_id']}\n```\n:::\n:::\n\n\n::: {#1b4474cd .cell execution_count=17}\n``` {.python .cell-code}\n# Export to Frictionless Table Schema\ntable_schema = pb.export_contract(contract, format=\"frictionless\")\ntable_schema\n```\n\n::: {.cell-output .cell-output-display execution_count=17}\n```\n{'fields': [{'name': 'temperature',\n 'type': 'number',\n 'constraints': {'minimum': -50, 'maximum': 60, 'required': True}},\n {'name': 'humidity',\n 'type': 'number',\n 'constraints': {'minimum': 0, 'maximum': 100, 'required': True}},\n {'name': 'station_id', 'type': 'string', 'constraints': {'required': True}}]}\n```\n:::\n:::\n\n\n::: {#d93aa4dd .cell execution_count=18}\n``` {.python .cell-code}\n# Export to dbt schema.yml\ndbt_doc = pb.export_contract(contract, format=\"dbt\")\ndbt_doc\n```\n\n::: {.cell-output .cell-output-display execution_count=18}\n```\n{'version': 2,\n 'models': [{'name': 'sensor_data',\n 'columns': [{'name': 'temperature',\n 'data_type': 'float',\n 'data_tests': ['not_null']},\n {'name': 'humidity', 'data_type': 'float', 'data_tests': ['not_null']},\n {'name': 'station_id',\n 'data_type': 'string',\n 'data_tests': ['not_null']}]}]}\n```\n:::\n:::\n\n\n::: {#60fe6bb8 .cell execution_count=19}\n``` {.python .cell-code}\n# Export to ODCS\nodcs_doc = pb.export_contract(contract, format=\"odcs\")\nodcs_doc\n```\n\n::: {.cell-output .cell-output-display execution_count=19}\n```\n{'kind': 'DataContract',\n 'apiVersion': 'v3.0.0',\n 'info': {'title': 'sensor_data'},\n 'dataset': [{'table': 'sensor_data',\n 'columns': [{'column': 'temperature',\n 'logicalType': 'number',\n 'minimum': -50,\n 'maximum': 60,\n 'isNullable': False},\n {'column': 'humidity',\n 'logicalType': 'number',\n 'minimum': 0,\n 'maximum': 100,\n 'isNullable': False},\n {'column': 'station_id', 'logicalType': 'string', 'isNullable': False}]}]}\n```\n:::\n:::\n\n\nEach format produces the output structure that is native to that standard. JSON Schema export\ncreates a valid `$schema`-annotated document with `properties`, `type`, and `required` fields.\nFrictionless export creates a Table Schema with `fields` and `constraints` entries. dbt export\ncreates a `version: 2` document with models and column-level `data_tests`. ODCS export creates a\n`kind: DataContract` document with typed columns and constraint properties. All formats can be fed\ndirectly into tools that consume those standards.\n\nYou can also write directly to a file:\n\n```python\npb.export_contract(contract, \"output/sensor_data.schema.json\", format=\"json_schema\")\npb.export_contract(contract, \"output/sensor_data.resource.json\", format=\"frictionless\")\npb.export_contract(contract, \"output/schema.yml\", format=\"dbt\")\npb.export_contract(contract, \"output/sensor_data.odcs.yml\", format=\"odcs\")\n```\n\nWhen a `destination` path is provided, the output is written to that file (creating parent\ndirectories as needed) and also returned from the function. This makes it convenient to both\npersist the output and inspect it in the same call.\n\n## Round-Trip Fidelity\n\nImporting a schema and then exporting it back should produce an equivalent result. This is\nimportant for workflows where you maintain schemas in an external format but want to validate with\nPointblank:\n\n::: {#222e8424 .cell execution_count=20}\n``` {.python .cell-code}\n# Start with a JSON Schema\noriginal = {\n \"type\": \"object\",\n \"properties\": {\n \"score\": {\"type\": \"integer\", \"minimum\": 0, \"maximum\": 100},\n \"grade\": {\"type\": \"string\", \"enum\": [\"A\", \"B\", \"C\", \"D\", \"F\"]},\n },\n \"required\": [\"score\"],\n}\n\n# Import → Contract → Export\nimported = pb.import_contract(original, format=\"json_schema\")\ncontract = imported.to_contract(name=\"grades\")\nexported = pb.export_contract(contract, format=\"json_schema\")\n\n# The exported schema preserves the constraints\nprint(f\"Original constraints on 'score': minimum={original['properties']['score']['minimum']}, \"\n f\"maximum={original['properties']['score']['maximum']}\")\nprint(f\"Exported constraints on 'score': minimum={exported['properties']['score'].get('minimum')}, \"\n f\"maximum={exported['properties']['score'].get('maximum')}\")\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nOriginal constraints on 'score': minimum=0, maximum=100\nExported constraints on 'score': minimum=0, maximum=100\n```\n:::\n:::\n\n\nRound-trip fidelity is tested as part of Pointblank's test suite. The general guarantee is that\nany constraint that can be expressed in both Pointblank and the target format will survive the\nround trip. Constraints that are unique to one format (like JSON Schema's `$ref` or Pointblank's\n`pre=` argument) may not survive, but the core numeric bounds, enum checks, null checks, and\npattern constraints will always round-trip cleanly.\n\n## Auto-Detection\n\nWhen the format is obvious from the source content, you can omit the `format=` parameter:\n\n::: {#a28b62b5 .cell execution_count=21}\n``` {.python .cell-code}\n# JSON Schema: detected by presence of \"$schema\" or \"type\" + \"properties\"\nresult = pb.import_contract({\"type\": \"object\", \"properties\": {\"x\": {\"type\": \"integer\"}}})\nprint(f\"Detected: {result.source_format}\")\n\n# Frictionless: detected by presence of \"fields\" list\nresult = pb.import_contract({\"fields\": [{\"name\": \"x\", \"type\": \"integer\"}]})\nprint(f\"Detected: {result.source_format}\")\n\n# dbt: detected by \"models\" or \"sources\" key\nresult = pb.import_contract({\"version\": 2, \"models\": [{\"name\": \"t\", \"columns\": []}]})\nprint(f\"Detected: {result.source_format}\")\n\n# ODCS: detected by \"kind\": \"DataContract\"\nresult = pb.import_contract({\n \"kind\": \"DataContract\", \"apiVersion\": \"v3.0.0\", \"info\": {\"title\": \"x\"},\n \"dataset\": [{\"table\": \"t\", \"columns\": []}],\n})\nprint(f\"Detected: {result.source_format}\")\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nDetected: json_schema\nDetected: frictionless\nDetected: dbt\nDetected: odcs\n```\n:::\n:::\n\n\nFor file-based imports, the extension is also used for detection (`.schema.json` maps to JSON\nSchema, `.resource.json` or `.datapackage.json` maps to Frictionless, `.odcs.yml` or `.odcs.json`\nmaps to ODCS). Note that `.yml` and `.yaml` extensions are shared between dbt and ODCS, so for YAML\nfiles without a distinguishing extension, auto-detection falls through to content-based inspection.\nWhen working with ambiguous files or dict inputs that could match multiple formats, it is best to\nspecify `format=` explicitly to avoid any possibility of misdetection.\n\n## Combining Imports with Extra Checks\n\nAn imported schema gives you a baseline, but you can always add more Pointblank checks on top:\n\n::: {#4c87274c .cell execution_count=22}\n``` {.python .cell-code}\nimported = pb.import_contract(user_schema, format=\"json_schema\")\n\n# Start from the import but add custom checks\nvalidation = (\n imported\n .to_validate(data=users, tbl_name=\"enriched_check\")\n .col_vals_regex(columns=\"email\", pattern=r\".*\\.(com|io|org|net|co)$\")\n .rows_distinct(columns_subset=\"user_id\")\n .interrogate()\n)\n\nvalidation\n```\n\n::: {.cell-output .cell-output-display execution_count=22}\n```{=html}\n
\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
Pointblank Validation
2026-08-10|16:45:28
Polarsenriched_check
STEPCOLUMNSVALUESTBLEVALUNITSPASSFAILWECEXT
#4CA64C1\n
\n \n\n col_schema_match\n \n \n \n \n \n \n \n \n \n \n \n\n
\n
\n
col_schema_match()
\n
\n \n 		—SCHEMA\n \n \n \n \n \n \n \n✓11
1.00		0
0.00		————
#4CA64C2\n
\n \n\n col_vals_not_null\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_null()
\n
\n \n 		user_id—\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C3\n
\n \n\n col_vals_not_null\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_not_null()
\n
\n \n 		email—\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C4\n
\n \n\n col_vals_within_spec\n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_within_spec()
\n
\n \n 		emailemail\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C5\n
\n \n\n col_vals_ge\n \n \n \n \n \n \n\n
\n
\n
col_vals_ge()
\n
\n \n 		age0\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C6\n
\n \n\n col_vals_le\n \n \n \n \n \n \n\n
\n
\n
col_vals_le()
\n
\n \n 		age150\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C7\n
\n \n\n col_vals_in_set\n \n \n \n \n \n \n\n
\n
\n
col_vals_in_set()
\n
\n \n 		statusactive, inactive, pending\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C8\n
\n \n\n col_vals_regex\n \n \n \n \n \n \n \n \n \n\n
\n
\n
col_vals_regex()
\n
\n \n 		email.*\\.(com|io|org|net|co)$\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
#4CA64C9\n
\n \n\n rows_distinct\n \n \n \n \n \n \n \n \n \n \n\n
\n
\n
rows_distinct()
\n
\n \n 		user_id—\n \n \n \n \n \n \n \n✓55
1.00		0
0.00		————
\nNotes\n

Step 1 (schema_check) ✓ Schema validation passed.

\n
\nSchema Comparison\n
\n
\n\n\n\n \n \n \n \n \n \n \n \n\n\n\n \n \n\n\n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n
\n TARGET\n \n EXPECTED\n
COLUMNDATA TYPECOLUMNDATA TYPE
1user_idInt641user_id✓Int64✓
2emailString2email✓String✓
3ageInt643age✓Int64✓
4statusString4status✓String✓
Supplied Column Schema:
[('user_id', 'Int64'), ('email', 'String'), ('age', 'Int64'), ('status', 'String')]
\n
Schema Match Settings
\n
COMPLETE
IN ORDER
COLUMN ≠ column
DTYPE ≠ dtype
float ≠ float64
\n
\n
\n
\n
\n
\n\n
\n```\n:::\n:::\n\n\nThis pattern works well when the external schema covers structural and type constraints, but your\nteam has additional business rules that only make sense in the Pointblank context. The imported\nconstraints form the foundation, and your additional `.col_vals_*()` or `.rows_*()` calls layer on\ntop. Because `.to_validate()` returns a standard `Validate` object, you have full access to the\nentire Pointblank API for adding checks, setting thresholds, or attaching actions.\n\n## Migration from Other Tools\n\nA key use case for `import_contract()` is **migration**: bringing existing validation definitions\nfrom other tools into Pointblank without manual rewriting.\n\n### Coming from JSON Schema\n\nIf your team uses JSON Schema for API validation and you want the same rules applied to DataFrames:\n\n```python\n# Your existing JSON Schema (maybe generated by your API framework)\nresult = pb.import_contract(\"api/schemas/user.schema.json\")\n\n# Now use it for DataFrame validation in your data pipeline\nvalidation = result.to_validate(data=raw_users_df).interrogate()\n```\n\nThis approach is particularly powerful when your API team already maintains JSON Schema definitions\nfor request/response validation. Those same schemas can now serve double duty: validating API\npayloads at the service boundary and validating the resulting DataFrames in your analytics pipeline.\nYou get consistent enforcement across both layers without writing the rules twice.\n\n### Coming from Frictionless\n\nIf you have data packages from open data sources or research datasets:\n\n```python\n# Import from an existing data package descriptor\nresult = pb.import_contract(\"data/datapackage.json\", resource=\"observations\")\n\n# Validate the actual CSV data against the declared schema\nvalidation = result.to_validate(data=observations_df).interrogate()\n```\n\nFrictionless Data Packages are common in open data portals, government datasets, and academic\nresearch repositories. By importing their Table Schemas directly, you can validate downloaded data\nagainst its declared structure without needing to manually inspect the descriptor file and rewrite\neach constraint. This is especially valuable when working with unfamiliar datasets where the schema\ndescriptor is your primary documentation of what the data should contain.\n\n### Coming from dbt\n\nIf your analytics team maintains dbt models with schema tests:\n\n```python\n# Import the tests from your dbt schema.yml\nresult = pb.import_contract(\"models/schema.yml\", format=\"dbt\", model=\"orders\")\n\n# Now validate source data before it reaches dbt\nvalidation = result.to_validate(data=raw_orders_df).interrogate()\n```\n\nThis is really valuable for \"shift-left\" validation: catching data quality issues in raw source data\nbefore it enters the dbt transformation pipeline. Your dbt schema tests define what the transformed\ndata should look like. Importing those same rules into Pointblank lets you apply them at the\ningestion boundary.\n\n### Coming from ODCS\n\nIf your organization uses the Open Data Contract Standard for cross-team data governance:\n\n```python\n# Import a data contract\nresult = pb.import_contract(\"contracts/customer_data.odcs.yml\", format=\"odcs\")\n\n# Validate data against the contract\nvalidation = result.to_validate(data=customer_df).interrogate()\n```\n\nODCS contracts are increasingly used in data mesh architectures where domain teams publish contracts\nfor their data products. Importing these contracts into Pointblank lets consumers validate incoming\ndata against the producer's declared contract, without needing to manually rewrite the constraints.\n\n### Generating a Starting Point\n\nEven if you don't plan to keep using the external format, importing is a great way to bootstrap\na Pointblank contract:\n\n```python\n# Import from your existing schema\nimported = pb.import_contract(\"legacy_schema.json\", format=\"json_schema\")\n\n# Save as a YAML contract you'll maintain going forward\ncontract = imported.to_contract(name=\"my_table\", version=\"1.0.0\")\ncontract.to_yaml(\"contracts/my_table.yaml\")\n```\n\nNow you have a Pointblank-native contract that you can extend and evolve independently of the\noriginal source. You can add new validation steps, adjust thresholds, or incorporate business rules\nthat go beyond what the original schema format could express.\n\n## Conclusion\n\nThe contract import/export system lets you bridge the gap between external schema definitions and\nPointblank's validation engine. Rather than maintaining duplicate specifications across tools, you\ncan keep your source of truth in whichever format suits your team and import it into Pointblank\nwhenever you need runtime validation. The key points to remember:\n\n- Use `pb.import_contract()` to read external schemas and translate them into Pointblank checks\n- The `ContractImport` object gives you multiple output options: direct validation, reusable\n contracts, generated Python code, or YAML\n- Check `.coverage` and `.warnings` to understand how completely the translation covered your\n original schema\n- Use `pb.export_contract()` to write Pointblank contracts back to external formats for sharing\n with other tools\n- Combine imports with additional Pointblank-specific checks for the most thorough validation\n coverage\n\nWhether you are migrating from another validation tool, bootstrapping contracts from existing\nschemas, or maintaining interoperability with external systems, the adapter framework gives you a\nclean path between external specifications and Pointblank's validation engine. The four built-in\nadapters (JSON Schema, Frictionless, dbt, and ODCS) cover the most common schema formats in the data\necosystem. And if you need to support a proprietary format, the custom adapter system (covered in\nthe next section) makes it straightforward to extend the framework.\n\n", "supporting": [ - "importing-contracts_files" + "importing-contracts_files/figure-html" ], "filters": [], "includes": { diff --git a/pointblank/adapters/__init__.py b/pointblank/adapters/__init__.py index eaadb2e6c..8545a553d 100644 --- a/pointblank/adapters/__init__.py +++ b/pointblank/adapters/__init__.py @@ -1,9 +1,10 @@ from __future__ import annotations -import pointblank.adapters._frictionless # noqa: F401 - # Import adapter modules to trigger registration via @register_adapter +import pointblank.adapters._dbt # noqa: F401 +import pointblank.adapters._frictionless # noqa: F401 import pointblank.adapters._json_schema # noqa: F401 +import pointblank.adapters._odcs # noqa: F401 from pointblank.adapters._api import export_contract, import_contract from pointblank.adapters._base import ContractAdapter, ContractImport, MappedConstraint from pointblank.adapters._registry import ( diff --git a/pointblank/adapters/_api.py b/pointblank/adapters/_api.py index 9110edc0c..09a824c99 100644 --- a/pointblank/adapters/_api.py +++ b/pointblank/adapters/_api.py @@ -9,19 +9,17 @@ def import_contract(source: Any, *, format: str | None = None, **kwargs: Any) -> ContractImport: """Import a contract/schema from an external format. - Reads an external schema definition (JSON Schema, Frictionless Table Schema, dbt schema.yml, - Pandera schema, Pydantic model, etc.) and produces a `ContractImport` with validation steps - mapped to Pointblank methods. + Reads an external schema definition and produces a `ContractImport` with validation steps + mapped to Pointblank methods. Use `pb.list_adapters()` to see all registered formats. Parameters ---------- source - The source to import from. Can be: (1) a file path (str) to a schema/contract file, (2) a - Python dict with schema content already loaded, or (3) a Python object (e.g., a Pandera - `DataFrameSchema` or Pydantic model class). + The source to import from. Can be: (1) a file path (str) to a schema/contract file, or + (2) a Python dict with schema content already loaded. format - The format identifier (e.g., `"json_schema"`, `"frictionless"`, `"dbt"`, etc.). If `None`, - the format is auto-detected from file extension or content. + The format identifier (e.g., `"json_schema"`, `"frictionless"`, `"dbt"`, `"odcs"`). If + `None`, the format is auto-detected from file extension or content. **kwargs Format-specific options passed to the adapter. @@ -87,7 +85,7 @@ def export_contract( Optional file path to write the output. If None, the result is returned without writing to disk. format - The target format identifier (e.g., `"json_schema"`, `"frictionless"`, `"dbt"`, etc.). + The target format identifier (e.g., `"json_schema"`, `"frictionless"`, `"dbt"`, `"odcs"`). **kwargs Format-specific options passed to the adapter. diff --git a/pointblank/adapters/_dbt.py b/pointblank/adapters/_dbt.py new file mode 100644 index 000000000..4786fe4d4 --- /dev/null +++ b/pointblank/adapters/_dbt.py @@ -0,0 +1,471 @@ +from __future__ import annotations + +from pathlib import Path +from typing import Any + +import yaml + +from pointblank.adapters._base import ContractAdapter, ContractImport, MappedConstraint +from pointblank.adapters._registry import register_adapter + +_DBT_TYPE_MAP: dict[str, str] = { + "integer": "Int64", + "int": "Int64", + "bigint": "Int64", + "smallint": "Int64", + "tinyint": "Int64", + "float": "Float64", + "double": "Float64", + "numeric": "Float64", + "decimal": "Float64", + "number": "Float64", + "real": "Float64", + "string": "String", + "text": "String", + "varchar": "String", + "char": "String", + "character varying": "String", + "boolean": "Boolean", + "bool": "Boolean", + "date": "Date", + "datetime": "Datetime", + "timestamp": "Datetime", + "timestamp_ntz": "Datetime", + "timestamp_tz": "Datetime", + "time": "Time", +} + + +def _normalize_dbt_type(raw_type: str) -> str | None: + raw_lower = raw_type.lower().strip() + # Strip precision/scale suffixes like "varchar(256)" or "numeric(10,2)" + base_type = raw_lower.split("(")[0].strip() + return _DBT_TYPE_MAP.get(base_type) + + +@register_adapter("dbt") +class DbtAdapter(ContractAdapter): + """Adapter for dbt schema.yml (models and sources). + + Supports import from dbt `schema.yml` files (or equivalent dicts), and export of Pointblank + validations back to dbt schema.yml format. + + Handles both the legacy `tests` key and the newer `data_tests` key for column-level tests. + """ + + format_name = "dbt" + file_extensions = [".yml", ".yaml"] + supports_import = True + supports_export = True + + @staticmethod + def detect(source: Any) -> bool: + if isinstance(source, dict): + return _is_dbt_schema(source) + + if isinstance(source, str): + path = Path(source) + if path.suffix in (".yml", ".yaml") and path.exists(): + try: + with open(path) as f: + data = yaml.safe_load(f) + return isinstance(data, dict) and _is_dbt_schema(data) + except (yaml.YAMLError, OSError): + return False + + return False + + def import_contract(self, source: Any, **kwargs: Any) -> ContractImport: + """Import a dbt schema.yml document. + + Parameters + ---------- + source + A file path (str) to a .yml/.yaml file, or a dict with the schema content. + model + For files with multiple models/sources, the name of the model to import. If `None`, the + first model (or source table) is used. + **kwargs + Additional options. + + Returns + ------- + ContractImport + The import result. + """ + source_path = None + + if isinstance(source, str): + source_path = source + path = Path(source) + if not path.exists(): + raise FileNotFoundError(f"dbt schema file not found: {source}") + with open(path) as f: + doc = yaml.safe_load(f) + elif isinstance(source, dict): + doc = source + else: + raise TypeError( + f"dbt source must be a file path (str) or dict, got {type(source).__name__}" + ) + + if not isinstance(doc, dict): + raise ValueError("dbt schema.yml must be a YAML mapping at the top level.") + + model_def = self._extract_model(doc, **kwargs) + return self._parse_model(model_def, source_path=source_path) + + def export_contract( + self, + validation_or_contract: Any, + destination: str | None = None, + **kwargs: Any, + ) -> str | dict[str, Any]: + """Export a Validate or Contract to dbt schema.yml format. + + Parameters + ---------- + validation_or_contract + A `Validate` or `Contract` object. + destination + Optional file path to write the YAML. + **kwargs + Not currently used. + + Returns + ------- + dict + The dbt schema.yml document as a dict. + """ + from pointblank.contract import Contract + from pointblank.validate import Validate + + if isinstance(validation_or_contract, Contract): + doc = self._export_from_contract(validation_or_contract) + elif isinstance(validation_or_contract, Validate): + doc = self._export_from_validate(validation_or_contract) + else: + raise TypeError( + f"Expected a Validate or Contract object, " + f"got {type(validation_or_contract).__name__}" + ) + + if destination is not None: + path = Path(destination) + path.parent.mkdir(parents=True, exist_ok=True) + with open(path, "w") as f: + yaml.dump(doc, f, default_flow_style=False, sort_keys=False) + + return doc + + # ── helpers ────────────────────────────────────────────────────────── + + def _extract_model(self, doc: dict[str, Any], **kwargs: Any) -> dict[str, Any]: + model_name = kwargs.get("model") + + # Collect candidate model/source definitions + candidates: list[dict[str, Any]] = [] + + for model in doc.get("models", []): + candidates.append(model) + + for src in doc.get("sources", []): + for table in src.get("tables", []): + candidates.append(table) + + if not candidates: + raise ValueError("No models or source tables found in this dbt schema.yml document.") + + if model_name is None: + return candidates[0] + + for candidate in candidates: + if candidate.get("name") == model_name: + return candidate + + available = [c.get("name", "") for c in candidates] + raise ValueError(f"Model '{model_name}' not found. Available: {available}") + + def _parse_model( + self, model_def: dict[str, Any], source_path: str | None = None + ) -> ContractImport: + columns: list[tuple[str, str | None]] = [] + constraints: list[MappedConstraint] = [] + warnings: list[str] = [] + metadata: dict[str, Any] = {} + + if "name" in model_def: + metadata["title"] = model_def["name"] + if "description" in model_def: + metadata["description"] = model_def["description"] + + total_constraints = 0 + + for col_def in model_def.get("columns", []): + col_name = col_def.get("name", "") + raw_type = col_def.get("data_type", "") + dtype = _normalize_dbt_type(raw_type) if raw_type else None + columns.append((col_name, dtype)) + + # dbt uses "tests" (legacy) or "data_tests" (v1.8+) + tests = col_def.get("data_tests") or col_def.get("tests") or [] + + for test in tests: + total_constraints += 1 + + if isinstance(test, str): + self._map_simple_test(test, col_name, constraints, warnings) + elif isinstance(test, dict): + self._map_dict_test(test, col_name, constraints, warnings) + else: + warnings.append( + f"Column '{col_name}': unrecognized test format {type(test).__name__} — skipped." + ) + + # Model-level tests + model_tests = model_def.get("data_tests") or model_def.get("tests") or [] + for test in model_tests: + total_constraints += 1 + if isinstance(test, dict): + self._map_model_level_test(test, constraints, warnings) + else: + warnings.append(f"Model-level test {test!r} — skipped (not yet supported).") + + coverage = 1.0 + if total_constraints > 0: + mapped_count = total_constraints - len(warnings) + coverage = mapped_count / total_constraints + + return ContractImport( + source_format="dbt", + source_path=source_path, + source_version=str(model_def.get("version", "")), + columns=columns, + constraints=constraints, + metadata=metadata, + warnings=warnings, + coverage=coverage, + ) + + def _map_simple_test( + self, + test_name: str, + col_name: str, + constraints: list[MappedConstraint], + warnings: list[str], + ) -> None: + if test_name == "not_null": + constraints.append( + MappedConstraint( + method="col_vals_not_null", + kwargs={"columns": col_name}, + source_description=f"test: not_null on {col_name}", + ) + ) + elif test_name == "unique": + constraints.append( + MappedConstraint( + method="rows_distinct", + kwargs={"columns_subset": col_name}, + source_description=f"test: unique on {col_name}", + ) + ) + else: + warnings.append( + f"Column '{col_name}': dbt test '{test_name}' has no Pointblank equivalent — skipped." + ) + + def _map_dict_test( + self, + test: dict[str, Any], + col_name: str, + constraints: list[MappedConstraint], + warnings: list[str], + ) -> None: + if "not_null" in test: + constraints.append( + MappedConstraint( + method="col_vals_not_null", + kwargs={"columns": col_name}, + source_description=f"test: not_null on {col_name}", + ) + ) + elif "unique" in test: + constraints.append( + MappedConstraint( + method="rows_distinct", + kwargs={"columns_subset": col_name}, + source_description=f"test: unique on {col_name}", + ) + ) + elif "accepted_values" in test: + config = test["accepted_values"] + values = config.get("values", []) + constraints.append( + MappedConstraint( + method="col_vals_in_set", + kwargs={"columns": col_name, "set": values}, + source_description=f"test: accepted_values {values} on {col_name}", + ) + ) + elif "relationships" in test: + config = test["relationships"] + ref_model = config.get("to", "?") + ref_field = config.get("field", "?") + warnings.append( + f"Column '{col_name}': relationship test ({col_name} → {ref_model}.{ref_field}) " + f"skipped (cross-table validation not supported)." + ) + else: + test_name = next(iter(test), "unknown") + warnings.append( + f"Column '{col_name}': dbt test '{test_name}' has no Pointblank equivalent — skipped." + ) + + def _map_model_level_test( + self, + test: dict[str, Any], + constraints: list[MappedConstraint], + warnings: list[str], + ) -> None: + if "unique" in test: + config = test["unique"] + combo = config.get("combination_of_columns") or config.get("columns", []) + if combo: + constraints.append( + MappedConstraint( + method="rows_distinct", + kwargs={"columns_subset": combo}, + source_description=f"model test: unique combination {combo}", + ) + ) + else: + warnings.append("Model-level unique test with no columns — skipped.") + else: + test_name = next(iter(test), "unknown") + warnings.append( + f"Model-level test '{test_name}' has no Pointblank equivalent — skipped." + ) + + def _export_from_contract(self, contract: Any) -> dict[str, Any]: + columns: list[dict[str, Any]] = [] + + if contract.schema is not None and contract.schema.columns is not None: + for col_name, col_dtype in contract.schema.columns: + col_def: dict[str, Any] = {"name": col_name} + if col_dtype: + col_def["data_type"] = _pb_dtype_to_dbt_type(str(col_dtype)) + columns.append(col_def) + + col_map = {c["name"]: c for c in columns} + + for step in contract.steps: + _apply_step_to_dbt_columns(step.method, step.kwargs, col_map, columns) + + model: dict[str, Any] = {"name": contract.name} + if contract.description: + model["description"] = contract.description + model["columns"] = columns + + return {"version": 2, "models": [model]} + + def _export_from_validate(self, validation: Any) -> dict[str, Any]: + columns: list[dict[str, Any]] = [] + col_map: dict[str, dict[str, Any]] = {} + + for step in validation.validation_info: + col = step.column + if col and col not in col_map: + col_def: dict[str, Any] = {"name": col} + columns.append(col_def) + col_map[col] = col_def + + kwargs = _extract_validate_step_kwargs(step) + _apply_step_to_dbt_columns(step.assertion_type, kwargs, col_map, columns) + + model_name = "" + if hasattr(validation, "_tbl_name") and validation._tbl_name: + model_name = validation._tbl_name + + model: dict[str, Any] = {"name": model_name, "columns": columns} + return {"version": 2, "models": [model]} + + +def _is_dbt_schema(data: dict[str, Any]) -> bool: + if "models" in data and isinstance(data.get("models"), list): + return True + if "sources" in data and isinstance(data.get("sources"), list): + return True + return False + + +def _pb_dtype_to_dbt_type(dtype: str) -> str: + dtype_lower = dtype.lower() + if "int" in dtype_lower: + return "integer" + if "float" in dtype_lower or "double" in dtype_lower or "decimal" in dtype_lower: + return "float" + if "str" in dtype_lower or "utf8" in dtype_lower or "object" in dtype_lower: + return "string" + if "bool" in dtype_lower: + return "boolean" + if "datetime" in dtype_lower or "timestamp" in dtype_lower: + return "timestamp" + if "date" in dtype_lower: + return "date" + if "time" in dtype_lower: + return "time" + return "string" + + +def _apply_step_to_dbt_columns( + method: str, + kwargs: dict[str, Any], + col_map: dict[str, dict[str, Any]], + columns: list[dict[str, Any]], +) -> None: + target_columns = kwargs.get("columns", kwargs.get("column", kwargs.get("columns_subset"))) + if target_columns is None: + return + + if isinstance(target_columns, str): + col_list = [target_columns] + elif isinstance(target_columns, list): + col_list = target_columns + else: + return + + for col in col_list: + if col not in col_map: + col_def: dict[str, Any] = {"name": col} + columns.append(col_def) + col_map[col] = col_def + + col_def = col_map[col] + if "data_tests" not in col_def: + col_def["data_tests"] = [] + + tests: list[Any] = col_def["data_tests"] + + if method == "col_vals_not_null": + if "not_null" not in tests: + tests.append("not_null") + elif method == "rows_distinct": + if "unique" not in tests: + tests.append("unique") + elif method == "col_vals_in_set": + values = kwargs.get("set", []) + tests.append({"accepted_values": {"values": values}}) + + +def _extract_validate_step_kwargs(step_info: Any) -> dict[str, Any]: + kwargs: dict[str, Any] = {} + if hasattr(step_info, "column") and step_info.column: + kwargs["columns"] = step_info.column + if hasattr(step_info, "values") and step_info.values is not None: + val = step_info.values + if isinstance(val, (list, tuple)): + kwargs["set"] = list(val) + else: + kwargs["value"] = val + return kwargs diff --git a/pointblank/adapters/_odcs.py b/pointblank/adapters/_odcs.py new file mode 100644 index 000000000..cecd24367 --- /dev/null +++ b/pointblank/adapters/_odcs.py @@ -0,0 +1,507 @@ +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any + +import yaml + +from pointblank.adapters._base import ContractAdapter, ContractImport, MappedConstraint +from pointblank.adapters._registry import register_adapter + +_ODCS_TYPE_MAP: dict[str, str] = { + "string": "String", + "text": "String", + "varchar": "String", + "char": "String", + "integer": "Int64", + "int": "Int64", + "bigint": "Int64", + "smallint": "Int64", + "tinyint": "Int64", + "number": "Float64", + "float": "Float64", + "double": "Float64", + "decimal": "Float64", + "numeric": "Float64", + "boolean": "Boolean", + "bool": "Boolean", + "date": "Date", + "datetime": "Datetime", + "timestamp": "Datetime", + "timestamp_ntz": "Datetime", + "timestamp_tz": "Datetime", + "time": "Time", +} + + +def _normalize_odcs_type(raw_type: str) -> str | None: + raw_lower = raw_type.lower().strip() + base_type = raw_lower.split("(")[0].strip() + return _ODCS_TYPE_MAP.get(base_type) + + +@register_adapter("odcs") +class ODCSAdapter(ContractAdapter): + """Adapter for the Open Data Contract Standard (ODCS). + + Supports import from ODCS v2.x and v3.x YAML/JSON documents, and export of Pointblank + validations back to ODCS v3 format. + + See https://github.com/bitol-io/open-data-contract-standard for the specification. + """ + + format_name = "odcs" + file_extensions = [".odcs.yml", ".odcs.yaml", ".odcs.json"] + supports_import = True + supports_export = True + + @staticmethod + def detect(source: Any) -> bool: + if isinstance(source, dict): + return _is_odcs(source) + + if isinstance(source, str): + path = Path(source) + if path.suffix in (".yml", ".yaml", ".json") and path.exists(): + try: + with open(path) as f: + if path.suffix == ".json": + data = json.load(f) + else: + data = yaml.safe_load(f) + return isinstance(data, dict) and _is_odcs(data) + except (yaml.YAMLError, json.JSONDecodeError, OSError): + return False + + return False + + def import_contract(self, source: Any, **kwargs: Any) -> ContractImport: + """Import an ODCS data contract. + + Parameters + ---------- + source + A file path (str) to a YAML/JSON file, or a dict with the contract content. + table + For contracts with multiple tables/datasets, the name of the table to import. If `None`, + the first table is used. + **kwargs + Additional options. + + Returns + ------- + ContractImport + The import result. + """ + source_path = None + + if isinstance(source, str): + source_path = source + path = Path(source) + if not path.exists(): + raise FileNotFoundError(f"ODCS contract file not found: {source}") + with open(path) as f: + if path.suffix == ".json": + doc = json.load(f) + else: + doc = yaml.safe_load(f) + elif isinstance(source, dict): + doc = source + else: + raise TypeError( + f"ODCS source must be a file path (str) or dict, got {type(source).__name__}" + ) + + if not isinstance(doc, dict): + raise ValueError("ODCS document must be a YAML/JSON mapping at the top level.") + + return self._parse_contract(doc, source_path=source_path, **kwargs) + + def export_contract( + self, + validation_or_contract: Any, + destination: str | None = None, + **kwargs: Any, + ) -> str | dict[str, Any]: + """Export a Validate or Contract to ODCS v3 format. + + Parameters + ---------- + validation_or_contract + A `Validate` or `Contract` object. + destination + Optional file path to write the YAML/JSON output. + **kwargs + Not currently used. + + Returns + ------- + dict[str, Any] + The ODCS document as a dict. + """ + from pointblank.contract import Contract + from pointblank.validate import Validate + + if isinstance(validation_or_contract, Contract): + doc = self._export_from_contract(validation_or_contract) + elif isinstance(validation_or_contract, Validate): + doc = self._export_from_validate(validation_or_contract) + else: + raise TypeError( + f"Expected a Validate or Contract object, " + f"got {type(validation_or_contract).__name__}" + ) + + if destination is not None: + path = Path(destination) + path.parent.mkdir(parents=True, exist_ok=True) + with open(path, "w") as f: + if path.suffix == ".json": + json.dump(doc, f, indent=2) + else: + yaml.dump(doc, f, default_flow_style=False, sort_keys=False) + + return doc + + # ── parsing ────────────────────────────────────────────────────────── + + def _parse_contract( + self, + doc: dict[str, Any], + source_path: str | None = None, + **kwargs: Any, + ) -> ContractImport: + metadata: dict[str, Any] = {} + api_version = doc.get("apiVersion", "") + + # Extract metadata from top-level or info block + info = doc.get("info", {}) + title = info.get("title") or doc.get("datasetName") or doc.get("title", "") + description = info.get("description") or doc.get("description", "") + if title: + metadata["title"] = title + if description: + metadata["description"] = description + + # Find the dataset/schema entries — supports both v2 and v3 structures + table_def = self._extract_table(doc, **kwargs) + + columns, constraints, warnings, total_constraints = self._parse_table(table_def) + + coverage = 1.0 + if total_constraints > 0: + mapped_count = total_constraints - len(warnings) + coverage = mapped_count / total_constraints + + return ContractImport( + source_format="odcs", + source_path=source_path, + source_version=api_version or None, + columns=columns, + constraints=constraints, + metadata=metadata, + warnings=warnings, + coverage=coverage, + ) + + def _extract_table(self, doc: dict[str, Any], **kwargs: Any) -> dict[str, Any]: + table_name = kwargs.get("table") + dataset = doc.get("dataset", []) + + # v3: "schema" as a list of column dicts at top level (flat) + schema = doc.get("schema", []) + if isinstance(schema, list) and schema and not dataset: + if isinstance(schema[0], dict) and "column" in schema[0]: + return {"columns": schema} + + if not dataset: + raise ValueError( + "No 'dataset' section found in this ODCS document. " + "Expected a list of table definitions under 'dataset'." + ) + + if table_name is None: + return dataset[0] + + for table in dataset: + tname = table.get("table") or table.get("name", "") + if tname == table_name: + return table + + available = [t.get("table") or t.get("name", "") for t in dataset] + raise ValueError(f"Table '{table_name}' not found. Available: {available}") + + def _parse_table( + self, table_def: dict[str, Any] + ) -> tuple[ + list[tuple[str, str | None]], + list[MappedConstraint], + list[str], + int, + ]: + columns: list[tuple[str, str | None]] = [] + constraints: list[MappedConstraint] = [] + warnings: list[str] = [] + total_constraints = 0 + + col_defs = table_def.get("columns", []) + + for col_def in col_defs: + col_name = col_def.get("column") or col_def.get("name", "") + logical_type = col_def.get("logicalType") or col_def.get("type", "") + dtype = _normalize_odcs_type(logical_type) if logical_type else None + columns.append((col_name, dtype)) + + # isNullable: false → col_vals_not_null + is_nullable = col_def.get("isNullable") + if is_nullable is False: + total_constraints += 1 + constraints.append( + MappedConstraint( + method="col_vals_not_null", + kwargs={"columns": col_name}, + source_description=f"isNullable: false on {col_name}", + ) + ) + + # isUnique: true → rows_distinct + is_unique = col_def.get("isUnique") + if is_unique is True: + total_constraints += 1 + constraints.append( + MappedConstraint( + method="rows_distinct", + kwargs={"columns_subset": col_name}, + source_description=f"isUnique: true on {col_name}", + ) + ) + + # isPrimaryKey: true → not_null + distinct + is_pk = col_def.get("isPrimaryKey") + if is_pk is True: + total_constraints += 1 + constraints.append( + MappedConstraint( + method="col_vals_not_null", + kwargs={"columns": col_name}, + source_description=f"isPrimaryKey: true on {col_name} (not null)", + ) + ) + total_constraints += 1 + constraints.append( + MappedConstraint( + method="rows_distinct", + kwargs={"columns_subset": col_name}, + source_description=f"isPrimaryKey: true on {col_name} (unique)", + ) + ) + + # enum / values list + enum_values = col_def.get("enum") or col_def.get("values") + if enum_values and isinstance(enum_values, list): + total_constraints += 1 + constraints.append( + MappedConstraint( + method="col_vals_in_set", + kwargs={"columns": col_name, "set": enum_values}, + source_description=f"enum: {enum_values} on {col_name}", + ) + ) + + # pattern + pattern = col_def.get("pattern") + if pattern: + total_constraints += 1 + constraints.append( + MappedConstraint( + method="col_vals_regex", + kwargs={"columns": col_name, "pattern": pattern}, + source_description=f"pattern: {pattern} on {col_name}", + ) + ) + + # minimum / maximum (numeric range) + if "minimum" in col_def: + total_constraints += 1 + constraints.append( + MappedConstraint( + method="col_vals_ge", + kwargs={"columns": col_name, "value": col_def["minimum"]}, + source_description=f"minimum: {col_def['minimum']} on {col_name}", + ) + ) + if "maximum" in col_def: + total_constraints += 1 + constraints.append( + MappedConstraint( + method="col_vals_le", + kwargs={"columns": col_name, "value": col_def["maximum"]}, + source_description=f"maximum: {col_def['maximum']} on {col_name}", + ) + ) + + # minLength / maxLength → warnings (no direct Pointblank equivalent yet) + if "minLength" in col_def or "maxLength" in col_def: + total_constraints += 1 + warnings.append( + f"Column '{col_name}': minLength/maxLength constraints have no " + f"Pointblank equivalent — skipped." + ) + + # Column-level quality/checks (custom SodaCL etc.) → warnings + checks = col_def.get("checks") or col_def.get("quality", []) + if checks and isinstance(checks, list): + for check in checks: + total_constraints += 1 + warnings.append( + f"Column '{col_name}': custom check {check!r} — skipped " + f"(no automatic mapping)." + ) + + return columns, constraints, warnings, total_constraints + + # ── export ─────────────────────────────────────────────────────────── + + def _export_from_contract(self, contract: Any) -> dict[str, Any]: + col_defs: list[dict[str, Any]] = [] + + if contract.schema is not None and contract.schema.columns is not None: + for col_name, col_dtype in contract.schema.columns: + col_def: dict[str, Any] = {"column": col_name} + if col_dtype: + col_def["logicalType"] = _pb_dtype_to_odcs_type(str(col_dtype)) + col_defs.append(col_def) + + col_map = {c["column"]: c for c in col_defs} + + for step in contract.steps: + _apply_step_to_odcs_columns(step.method, step.kwargs, col_map, col_defs) + + doc: dict[str, Any] = { + "kind": "DataContract", + "apiVersion": "v3.0.0", + "info": { + "title": contract.name, + }, + "dataset": [ + { + "table": contract.name, + "columns": col_defs, + } + ], + } + + if contract.description: + doc["info"]["description"] = contract.description + + return doc + + def _export_from_validate(self, validation: Any) -> dict[str, Any]: + col_defs: list[dict[str, Any]] = [] + col_map: dict[str, dict[str, Any]] = {} + + for step in validation.validation_info: + col = step.column + if col and col not in col_map: + col_def: dict[str, Any] = {"column": col} + col_defs.append(col_def) + col_map[col] = col_def + + kwargs = _extract_validate_step_kwargs(step) + _apply_step_to_odcs_columns(step.assertion_type, kwargs, col_map, col_defs) + + title = "" + if hasattr(validation, "_tbl_name") and validation._tbl_name: + title = validation._tbl_name + + return { + "kind": "DataContract", + "apiVersion": "v3.0.0", + "info": {"title": title}, + "dataset": [ + { + "table": title, + "columns": col_defs, + } + ], + } + + +def _is_odcs(data: dict[str, Any]) -> bool: + if data.get("kind") == "DataContract": + return True + if "apiVersion" in data and ("dataset" in data or "schema" in data): + return True + return False + + +def _pb_dtype_to_odcs_type(dtype: str) -> str: + dtype_lower = dtype.lower() + if "int" in dtype_lower: + return "integer" + if "float" in dtype_lower or "double" in dtype_lower or "decimal" in dtype_lower: + return "number" + if "str" in dtype_lower or "utf8" in dtype_lower or "object" in dtype_lower: + return "string" + if "bool" in dtype_lower: + return "boolean" + if "datetime" in dtype_lower or "timestamp" in dtype_lower: + return "timestamp" + if "date" in dtype_lower: + return "date" + if "time" in dtype_lower: + return "time" + return "string" + + +def _apply_step_to_odcs_columns( + method: str, + kwargs: dict[str, Any], + col_map: dict[str, dict[str, Any]], + col_defs: list[dict[str, Any]], +) -> None: + target_columns = kwargs.get("columns", kwargs.get("column", kwargs.get("columns_subset"))) + if target_columns is None: + return + + if isinstance(target_columns, str): + col_list = [target_columns] + elif isinstance(target_columns, list): + col_list = target_columns + else: + return + + for col in col_list: + if col not in col_map: + col_def: dict[str, Any] = {"column": col} + col_defs.append(col_def) + col_map[col] = col_def + + col_def = col_map[col] + + if method == "col_vals_not_null": + col_def["isNullable"] = False + elif method == "rows_distinct": + col_def["isUnique"] = True + elif method == "col_vals_in_set": + col_def["enum"] = kwargs.get("set", []) + elif method == "col_vals_regex": + col_def["pattern"] = kwargs.get("pattern", "") + elif method == "col_vals_ge": + col_def["minimum"] = kwargs.get("value") + elif method == "col_vals_le": + col_def["maximum"] = kwargs.get("value") + + +def _extract_validate_step_kwargs(step_info: Any) -> dict[str, Any]: + kwargs: dict[str, Any] = {} + if hasattr(step_info, "column") and step_info.column: + kwargs["columns"] = step_info.column + if hasattr(step_info, "values") and step_info.values is not None: + val = step_info.values + if isinstance(val, (list, tuple)): + kwargs["set"] = list(val) + else: + kwargs["value"] = val + return kwargs diff --git a/tests/test_adapters.py b/tests/test_adapters.py index d62eca504..e6d3eab4d 100644 --- a/tests/test_adapters.py +++ b/tests/test_adapters.py @@ -125,6 +125,8 @@ def test_builtin_adapters_registered(self): adapters = list_adapters() assert "json_schema" in adapters assert "frictionless" in adapters + assert "dbt" in adapters + assert "odcs" in adapters def test_get_adapter_json_schema(self): adapter = get_adapter("json_schema") @@ -662,3 +664,603 @@ def _hashable_kwargs(kwargs): original_methods = {(c.method, _hashable_kwargs(c.kwargs)) for c in imported.constraints} roundtrip_methods = {(c.method, _hashable_kwargs(c.kwargs)) for c in reimported.constraints} assert original_methods == roundtrip_methods + + +# ── dbt adapter fixtures ───────────────────────────────────────────────────── + + +@pytest.fixture +def dbt_schema_dict(): + """A dbt schema.yml document as a dict.""" + return { + "version": 2, + "models": [ + { + "name": "users", + "description": "User accounts table", + "columns": [ + { + "name": "id", + "data_type": "integer", + "data_tests": ["not_null", "unique"], + }, + { + "name": "name", + "data_type": "varchar", + "data_tests": ["not_null"], + }, + { + "name": "age", + "data_type": "integer", + }, + { + "name": "status", + "data_type": "string", + "data_tests": [ + {"accepted_values": {"values": ["active", "inactive", "pending"]}} + ], + }, + { + "name": "email", + "data_type": "varchar(256)", + }, + ], + } + ], + } + + +@pytest.fixture +def dbt_schema_legacy_tests(): + """A dbt schema.yml using the legacy 'tests' key.""" + return { + "version": 2, + "models": [ + { + "name": "orders", + "columns": [ + { + "name": "order_id", + "data_type": "integer", + "tests": ["not_null", "unique"], + }, + { + "name": "user_id", + "data_type": "integer", + "tests": [ + {"relationships": {"to": "ref('users')", "field": "id"}}, + ], + }, + ], + } + ], + } + + +@pytest.fixture +def dbt_sources_dict(): + """A dbt schema.yml with sources instead of models.""" + return { + "version": 2, + "sources": [ + { + "name": "raw", + "tables": [ + { + "name": "events", + "columns": [ + { + "name": "event_id", + "data_type": "bigint", + "data_tests": ["not_null", "unique"], + }, + { + "name": "event_type", + "data_type": "string", + "data_tests": [ + {"accepted_values": {"values": ["click", "view", "purchase"]}} + ], + }, + ], + } + ], + } + ], + } + + +class TestDbtImport: + def test_import_from_dict(self, dbt_schema_dict): + result = import_contract(dbt_schema_dict, format="dbt") + + assert result.source_format == "dbt" + assert len(result.columns) == 5 + assert result.metadata.get("title") == "users" + assert result.metadata.get("description") == "User accounts table" + + def test_import_column_types(self, dbt_schema_dict): + result = import_contract(dbt_schema_dict, format="dbt") + col_map = dict(result.columns) + assert col_map["id"] == "Int64" + assert col_map["name"] == "String" + assert col_map["age"] == "Int64" + assert col_map["status"] == "String" + assert col_map["email"] == "String" # varchar(256) -> String + + def test_import_not_null(self, dbt_schema_dict): + result = import_contract(dbt_schema_dict, format="dbt") + methods = [(c.method, c.kwargs) for c in result.constraints] + assert ("col_vals_not_null", {"columns": "id"}) in methods + assert ("col_vals_not_null", {"columns": "name"}) in methods + + def test_import_unique(self, dbt_schema_dict): + result = import_contract(dbt_schema_dict, format="dbt") + methods = [(c.method, c.kwargs) for c in result.constraints] + assert ("rows_distinct", {"columns_subset": "id"}) in methods + + def test_import_accepted_values(self, dbt_schema_dict): + result = import_contract(dbt_schema_dict, format="dbt") + methods = [(c.method, c.kwargs) for c in result.constraints] + assert ( + "col_vals_in_set", + {"columns": "status", "set": ["active", "inactive", "pending"]}, + ) in methods + + def test_import_legacy_tests_key(self, dbt_schema_legacy_tests): + result = import_contract(dbt_schema_legacy_tests, format="dbt") + methods = [(c.method, c.kwargs) for c in result.constraints] + assert ("col_vals_not_null", {"columns": "order_id"}) in methods + assert ("rows_distinct", {"columns_subset": "order_id"}) in methods + + def test_import_relationship_warning(self, dbt_schema_legacy_tests): + result = import_contract(dbt_schema_legacy_tests, format="dbt") + assert any( + "relationship" in w.lower() or "cross-table" in w.lower() for w in result.warnings + ) + assert result.coverage < 1.0 + + def test_import_from_sources(self, dbt_sources_dict): + result = import_contract(dbt_sources_dict, format="dbt") + assert result.source_format == "dbt" + assert len(result.columns) == 2 + col_map = dict(result.columns) + assert col_map["event_id"] == "Int64" + + def test_import_specific_model(self): + doc = { + "version": 2, + "models": [ + {"name": "first", "columns": [{"name": "a"}]}, + {"name": "second", "columns": [{"name": "b"}, {"name": "c"}]}, + ], + } + result = import_contract(doc, format="dbt", model="second") + assert len(result.columns) == 2 + col_names = [name for name, _ in result.columns] + assert "b" in col_names + + def test_import_model_not_found(self, dbt_schema_dict): + with pytest.raises(ValueError, match="not found"): + import_contract(dbt_schema_dict, format="dbt", model="nonexistent") + + def test_import_from_file(self, dbt_schema_dict): + import yaml as _yaml + + with tempfile.NamedTemporaryFile(mode="w", suffix=".yml", delete=False) as f: + _yaml.dump(dbt_schema_dict, f) + f.flush() + result = import_contract(f.name, format="dbt") + + assert result.source_format == "dbt" + assert result.source_path == f.name + + def test_import_file_not_found(self): + with pytest.raises(FileNotFoundError): + import_contract("/nonexistent/schema.yml", format="dbt") + + def test_import_invalid_type(self): + with pytest.raises(TypeError, match="must be a file path"): + import_contract(12345, format="dbt") + + def test_auto_detect_dbt(self, dbt_schema_dict): + result = import_contract(dbt_schema_dict) + assert result.source_format == "dbt" + + def test_to_validate_end_to_end(self, dbt_schema_dict, simple_df): + result = import_contract(dbt_schema_dict, format="dbt") + validation = result.to_validate(data=simple_df) + validation.interrogate() + + def test_no_models_or_sources_raises(self): + with pytest.raises(ValueError, match="No models or source tables"): + import_contract({"version": 2}, format="dbt") + + +class TestDbtExport: + def test_export_from_contract(self): + contract = pb.Contract( + name="test_model", + description="A test model", + schema=pb.Schema(id="Int64", name="String", age="Int64"), + steps=[ + pb.Step("col_vals_not_null", columns="id"), + pb.Step("rows_distinct", columns="id"), + pb.Step("col_vals_in_set", columns="name", set=["Alice", "Bob"]), + ], + ) + result = export_contract(contract, format="dbt") + + assert result["version"] == 2 + assert len(result["models"]) == 1 + model = result["models"][0] + assert model["name"] == "test_model" + assert model["description"] == "A test model" + + col_map = {c["name"]: c for c in model["columns"]} + assert "not_null" in col_map["id"]["data_tests"] + assert "unique" in col_map["id"]["data_tests"] + assert col_map["id"]["data_type"] == "integer" + + def test_export_to_file(self): + import yaml as _yaml + + contract = pb.Contract( + name="file_test", + schema=pb.Schema(x="Int64"), + steps=[], + ) + with tempfile.NamedTemporaryFile(suffix=".yml", delete=False) as f: + export_contract(contract, f.name, format="dbt") + + with open(f.name) as fh: + data = _yaml.safe_load(fh) + assert data["version"] == 2 + assert data["models"][0]["name"] == "file_test" + + def test_export_invalid_type_raises(self): + with pytest.raises(TypeError, match="Expected a Validate or Contract"): + export_contract("not a contract", format="dbt") + + +class TestDbtRoundTrip: + def test_dbt_roundtrip(self): + original = { + "version": 2, + "models": [ + { + "name": "users", + "columns": [ + { + "name": "id", + "data_type": "integer", + "data_tests": ["not_null", "unique"], + }, + { + "name": "status", + "data_type": "string", + "data_tests": [{"accepted_values": {"values": ["a", "b"]}}], + }, + ], + } + ], + } + imported = import_contract(original, format="dbt") + contract = imported.to_contract(name="roundtrip") + exported = export_contract(contract, format="dbt") + reimported = import_contract(exported, format="dbt") + + def _hashable_kwargs(kwargs): + items = [] + for k, v in sorted(kwargs.items()): + items.append((k, tuple(v) if isinstance(v, list) else v)) + return tuple(items) + + original_methods = {(c.method, _hashable_kwargs(c.kwargs)) for c in imported.constraints} + roundtrip_methods = {(c.method, _hashable_kwargs(c.kwargs)) for c in reimported.constraints} + assert original_methods == roundtrip_methods + + +# ── ODCS adapter fixtures ──────────────────────────────────────────────────── + + +@pytest.fixture +def odcs_v3_dict(): + """An ODCS v3 data contract as a dict.""" + return { + "kind": "DataContract", + "apiVersion": "v3.0.0", + "info": { + "title": "User Accounts", + "description": "Contract for user account data", + }, + "dataset": [ + { + "table": "users", + "columns": [ + { + "column": "id", + "logicalType": "integer", + "isNullable": False, + "isUnique": True, + }, + { + "column": "name", + "logicalType": "string", + "isNullable": False, + }, + { + "column": "age", + "logicalType": "integer", + "minimum": 0, + "maximum": 150, + }, + { + "column": "status", + "logicalType": "string", + "enum": ["active", "inactive", "pending"], + }, + { + "column": "email", + "logicalType": "string", + "pattern": r"^[^@]+@[^@]+\.[^@]+$", + }, + ], + } + ], + } + + +@pytest.fixture +def odcs_v2_dict(): + """An ODCS v2-style data contract as a dict.""" + return { + "kind": "DataContract", + "apiVersion": "v2.2.2", + "datasetName": "orders", + "description": "Order data contract", + "dataset": [ + { + "table": "orders", + "columns": [ + { + "column": "order_id", + "logicalType": "integer", + "isNullable": False, + "isPrimaryKey": True, + }, + { + "column": "amount", + "logicalType": "float", + "minimum": 0, + }, + ], + } + ], + } + + +class TestODCSImport: + def test_import_from_dict_v3(self, odcs_v3_dict): + result = import_contract(odcs_v3_dict, format="odcs") + + assert result.source_format == "odcs" + assert result.source_version == "v3.0.0" + assert len(result.columns) == 5 + assert result.metadata.get("title") == "User Accounts" + assert result.metadata.get("description") == "Contract for user account data" + + def test_import_column_types(self, odcs_v3_dict): + result = import_contract(odcs_v3_dict, format="odcs") + col_map = dict(result.columns) + assert col_map["id"] == "Int64" + assert col_map["name"] == "String" + assert col_map["age"] == "Int64" + assert col_map["status"] == "String" + assert col_map["email"] == "String" + + def test_import_not_null(self, odcs_v3_dict): + result = import_contract(odcs_v3_dict, format="odcs") + methods = [(c.method, c.kwargs) for c in result.constraints] + assert ("col_vals_not_null", {"columns": "id"}) in methods + assert ("col_vals_not_null", {"columns": "name"}) in methods + + def test_import_unique(self, odcs_v3_dict): + result = import_contract(odcs_v3_dict, format="odcs") + methods = [(c.method, c.kwargs) for c in result.constraints] + assert ("rows_distinct", {"columns_subset": "id"}) in methods + + def test_import_min_max(self, odcs_v3_dict): + result = import_contract(odcs_v3_dict, format="odcs") + methods = [(c.method, c.kwargs) for c in result.constraints] + assert ("col_vals_ge", {"columns": "age", "value": 0}) in methods + assert ("col_vals_le", {"columns": "age", "value": 150}) in methods + + def test_import_enum(self, odcs_v3_dict): + result = import_contract(odcs_v3_dict, format="odcs") + methods = [(c.method, c.kwargs) for c in result.constraints] + assert ( + "col_vals_in_set", + {"columns": "status", "set": ["active", "inactive", "pending"]}, + ) in methods + + def test_import_pattern(self, odcs_v3_dict): + result = import_contract(odcs_v3_dict, format="odcs") + methods = [(c.method, c.kwargs) for c in result.constraints] + assert ( + "col_vals_regex", + {"columns": "email", "pattern": r"^[^@]+@[^@]+\.[^@]+$"}, + ) in methods + + def test_import_v2(self, odcs_v2_dict): + result = import_contract(odcs_v2_dict, format="odcs") + assert result.source_format == "odcs" + assert result.source_version == "v2.2.2" + assert result.metadata.get("title") == "orders" + + def test_import_primary_key(self, odcs_v2_dict): + result = import_contract(odcs_v2_dict, format="odcs") + methods = [(c.method, c.kwargs) for c in result.constraints] + assert ("col_vals_not_null", {"columns": "order_id"}) in methods + assert ("rows_distinct", {"columns_subset": "order_id"}) in methods + + def test_import_specific_table(self): + doc = { + "kind": "DataContract", + "apiVersion": "v3.0.0", + "info": {"title": "Multi"}, + "dataset": [ + {"table": "first", "columns": [{"column": "a", "logicalType": "string"}]}, + {"table": "second", "columns": [{"column": "b"}, {"column": "c"}]}, + ], + } + result = import_contract(doc, format="odcs", table="second") + assert len(result.columns) == 2 + + def test_import_table_not_found(self, odcs_v3_dict): + with pytest.raises(ValueError, match="not found"): + import_contract(odcs_v3_dict, format="odcs", table="nonexistent") + + def test_import_from_file_yaml(self, odcs_v3_dict): + import yaml as _yaml + + with tempfile.NamedTemporaryFile(mode="w", suffix=".odcs.yml", delete=False) as f: + _yaml.dump(odcs_v3_dict, f) + f.flush() + result = import_contract(f.name, format="odcs") + + assert result.source_format == "odcs" + assert result.source_path == f.name + + def test_import_from_file_json(self, odcs_v3_dict): + with tempfile.NamedTemporaryFile(mode="w", suffix=".odcs.json", delete=False) as f: + json.dump(odcs_v3_dict, f) + f.flush() + result = import_contract(f.name, format="odcs") + + assert result.source_format == "odcs" + + def test_import_file_not_found(self): + with pytest.raises(FileNotFoundError): + import_contract("/nonexistent/contract.odcs.yml", format="odcs") + + def test_import_invalid_type(self): + with pytest.raises(TypeError, match="must be a file path"): + import_contract(12345, format="odcs") + + def test_auto_detect_odcs(self, odcs_v3_dict): + result = import_contract(odcs_v3_dict) + assert result.source_format == "odcs" + + def test_to_validate_end_to_end(self, odcs_v3_dict, simple_df): + result = import_contract(odcs_v3_dict, format="odcs") + validation = result.to_validate(data=simple_df) + validation.interrogate() + + def test_no_dataset_raises(self): + with pytest.raises(ValueError, match="No 'dataset' section"): + import_contract({"kind": "DataContract", "apiVersion": "v3.0.0"}, format="odcs") + + def test_minlength_warning(self): + doc = { + "kind": "DataContract", + "apiVersion": "v3.0.0", + "info": {"title": "test"}, + "dataset": [ + { + "table": "t", + "columns": [{"column": "name", "logicalType": "string", "minLength": 1}], + } + ], + } + result = import_contract(doc, format="odcs") + assert any("minLength" in w for w in result.warnings) + assert result.coverage < 1.0 + + +class TestODCSExport: + def test_export_from_contract(self): + contract = pb.Contract( + name="test_contract", + description="A test contract", + schema=pb.Schema(id="Int64", name="String", age="Int64"), + steps=[ + pb.Step("col_vals_not_null", columns="id"), + pb.Step("rows_distinct", columns="id"), + pb.Step("col_vals_ge", columns="age", value=0), + ], + ) + result = export_contract(contract, format="odcs") + + assert result["kind"] == "DataContract" + assert result["apiVersion"] == "v3.0.0" + assert result["info"]["title"] == "test_contract" + assert result["info"]["description"] == "A test contract" + + table = result["dataset"][0] + col_map = {c["column"]: c for c in table["columns"]} + assert col_map["id"]["isNullable"] is False + assert col_map["id"]["isUnique"] is True + assert col_map["id"]["logicalType"] == "integer" + assert col_map["age"]["minimum"] == 0 + + def test_export_to_file_yaml(self): + import yaml as _yaml + + contract = pb.Contract(name="file_test", schema=pb.Schema(x="Int64"), steps=[]) + with tempfile.NamedTemporaryFile(suffix=".odcs.yml", delete=False) as f: + export_contract(contract, f.name, format="odcs") + + with open(f.name) as fh: + data = _yaml.safe_load(fh) + assert data["kind"] == "DataContract" + + def test_export_to_file_json(self): + contract = pb.Contract(name="file_test", schema=pb.Schema(x="Int64"), steps=[]) + with tempfile.NamedTemporaryFile(suffix=".odcs.json", delete=False) as f: + export_contract(contract, f.name, format="odcs") + + with open(f.name) as fh: + data = json.load(fh) + assert data["kind"] == "DataContract" + + def test_export_invalid_type_raises(self): + with pytest.raises(TypeError, match="Expected a Validate or Contract"): + export_contract("not a contract", format="odcs") + + +class TestODCSRoundTrip: + def test_odcs_roundtrip(self): + original = { + "kind": "DataContract", + "apiVersion": "v3.0.0", + "info": {"title": "test"}, + "dataset": [ + { + "table": "users", + "columns": [ + { + "column": "id", + "logicalType": "integer", + "isNullable": False, + "isUnique": True, + }, + {"column": "age", "logicalType": "integer", "minimum": 0, "maximum": 150}, + {"column": "status", "logicalType": "string", "enum": ["a", "b"]}, + ], + } + ], + } + imported = import_contract(original, format="odcs") + contract = imported.to_contract(name="roundtrip") + exported = export_contract(contract, format="odcs") + reimported = import_contract(exported, format="odcs") + + def _hashable_kwargs(kwargs): + items = [] + for k, v in sorted(kwargs.items()): + items.append((k, tuple(v) if isinstance(v, list) else v)) + return tuple(items) + + original_methods = {(c.method, _hashable_kwargs(c.kwargs)) for c in imported.constraints} + roundtrip_methods = {(c.method, _hashable_kwargs(c.kwargs)) for c in reimported.constraints} + assert original_methods == roundtrip_methods diff --git a/user_guide/10-contracts-and-pipelines/04-importing-contracts.qmd b/user_guide/10-contracts-and-pipelines/04-importing-contracts.qmd index 3666cc8a0..f0e549711 100644 --- a/user_guide/10-contracts-and-pipelines/04-importing-contracts.qmd +++ b/user_guide/10-contracts-and-pipelines/04-importing-contracts.qmd @@ -12,8 +12,8 @@ pb.config(report_incl_footer_timings=False) ``` Many teams already have data schemas defined in other tools: JSON Schema files for API validation, -Frictionless Table Schemas for open data, dbt `schema.yml` files for analytics pipelines, or -Pandera/Pydantic models in application code. Rather than manually rewriting these specifications as +Frictionless Table Schemas for open data, dbt `schema.yml` files for analytics pipelines, or ODCS +data contracts for cross-team governance. Rather than manually rewriting these specifications as Pointblank validation steps, you can **import** them directly. The `import_contract()` function reads an external schema definition and produces a `ContractImport` @@ -148,8 +148,7 @@ parts might need manual attention. ## Supported Formats -Pointblank ships with adapters for the two most universal tabular schema formats. Additional -adapters (dbt, Pydantic, Pandera) are planned for future releases. +Pointblank ships with adapters for four widely-used schema and contract formats. ```{python} pb.list_adapters() @@ -298,6 +297,174 @@ The `resource=` parameter accepts either a string (the resource name) or an inte index). When omitted, the first resource in the package is used. This makes it straightforward to work with multi-table data packages where each table has its own schema definition. +### dbt `schema.yml` + +[dbt](https://www.getdbt.com/) (data build tool) is the standard for transformation workflows in +modern analytics stacks. dbt models declare column-level tests in `schema.yml` files, and Pointblank +can import these definitions directly. + +**Constraint mapping:** + +| dbt Feature | Pointblank Method | +|---|---| +| `data_type` | Schema dtype check | +| `not_null` test | `col_vals_not_null()` | +| `unique` test | `rows_distinct()` | +| `accepted_values` test | `col_vals_in_set()` | +| `relationships` test | Warning (cross-table not yet supported) | + +Both the newer `data_tests` key (dbt v1.8+) and the legacy `tests` key are supported. + +**Importing from a dict:** + +```{python} +dbt_schema = { + "version": 2, + "models": [ + { + "name": "orders", + "description": "Cleaned order data", + "columns": [ + { + "name": "order_id", + "data_type": "integer", + "data_tests": ["not_null", "unique"], + }, + { + "name": "status", + "data_type": "string", + "data_tests": [ + {"accepted_values": {"values": ["pending", "shipped", "delivered"]}} + ], + }, + { + "name": "amount", + "data_type": "float", + }, + ], + } + ], +} + +result = pb.import_contract(dbt_schema, format="dbt") +print(result.summary()) +``` + +**Importing from a file:** + +```python +# File-based import (auto-detects .yml / .yaml with dbt structure) +result = pb.import_contract("models/schema.yml", format="dbt") +``` + +**Selecting a specific model:** + +When a `schema.yml` file contains multiple models, use the `model=` parameter to pick one: + +```{python} +multi_model = { + "version": 2, + "models": [ + {"name": "users", "columns": [{"name": "id", "data_tests": ["not_null"]}]}, + {"name": "orders", "columns": [{"name": "order_id", "data_tests": ["not_null", "unique"]}]}, + ], +} + +result = pb.import_contract(multi_model, format="dbt", model="orders") +print(f"Imported model: {result.metadata.get('title')}") +print(f"Constraints: {len(result.constraints)}") +``` + +dbt sources are also supported. When a file contains `sources` with `tables`, each table is treated +as a model candidate: + +```python +result = pb.import_contract("sources.yml", format="dbt", model="raw_events") +``` + +### ODCS (Open Data Contract Standard) + +The [Open Data Contract Standard](https://github.com/bitol-io/open-data-contract-standard) (ODCS) +is a vendor-neutral format for defining data contracts between producers and consumers. Pointblank +supports both ODCS v2.x and v3.x documents. + +**Constraint mapping:** + +| ODCS Feature | Pointblank Method | +|---|---| +| `logicalType` | Schema dtype check | +| `isNullable: false` | `col_vals_not_null()` | +| `isUnique: true` | `rows_distinct()` | +| `isPrimaryKey: true` | `col_vals_not_null()` + `rows_distinct()` | +| `enum` / `values` | `col_vals_in_set()` | +| `pattern` | `col_vals_regex()` | +| `minimum` / `maximum` | `col_vals_ge()` / `col_vals_le()` | +| `minLength` / `maxLength` | Warning (no direct equivalent) | +| Custom checks / SodaCL | Warning (no automatic mapping) | + +**Importing an ODCS v3 contract:** + +```{python} +odcs_contract = { + "kind": "DataContract", + "apiVersion": "v3.0.0", + "info": { + "title": "Customer Data", + "description": "Contract for customer records", + }, + "dataset": [ + { + "table": "customers", + "columns": [ + { + "column": "customer_id", + "logicalType": "integer", + "isNullable": False, + "isUnique": True, + }, + { + "column": "email", + "logicalType": "string", + "isNullable": False, + "pattern": r"^[^@]+@[^@]+\.[^@]+$", + }, + { + "column": "tier", + "logicalType": "string", + "enum": ["free", "pro", "enterprise"], + }, + { + "column": "lifetime_value", + "logicalType": "float", + "minimum": 0, + }, + ], + } + ], +} + +result = pb.import_contract(odcs_contract, format="odcs") +print(result.summary()) +``` + +**Selecting a specific table:** + +For contracts with multiple dataset tables, use `table=`: + +```python +result = pb.import_contract("contract.odcs.yml", format="odcs", table="orders") +``` + +**Importing from a file:** + +```python +# YAML files +result = pb.import_contract("contracts/customer_data.odcs.yml", format="odcs") + +# JSON files +result = pb.import_contract("contracts/customer_data.odcs.json", format="odcs") +``` + ## Output Options Once you have a `ContractImport`, you can use it in several ways depending on your workflow. @@ -433,17 +600,32 @@ table_schema = pb.export_contract(contract, format="frictionless") table_schema ``` +```{python} +# Export to dbt schema.yml +dbt_doc = pb.export_contract(contract, format="dbt") +dbt_doc +``` + +```{python} +# Export to ODCS +odcs_doc = pb.export_contract(contract, format="odcs") +odcs_doc +``` + Each format produces the output structure that is native to that standard. JSON Schema export creates a valid `$schema`-annotated document with `properties`, `type`, and `required` fields. -Frictionless export creates a Table Schema with `fields` and `constraints` entries. Both formats -can be fed directly into tools that consume those standards, such as form validators, data catalogs, -or documentation generators. +Frictionless export creates a Table Schema with `fields` and `constraints` entries. dbt export +creates a `version: 2` document with models and column-level `data_tests`. ODCS export creates a +`kind: DataContract` document with typed columns and constraint properties. All formats can be fed +directly into tools that consume those standards. You can also write directly to a file: ```python pb.export_contract(contract, "output/sensor_data.schema.json", format="json_schema") pb.export_contract(contract, "output/sensor_data.resource.json", format="frictionless") +pb.export_contract(contract, "output/schema.yml", format="dbt") +pb.export_contract(contract, "output/sensor_data.odcs.yml", format="odcs") ``` When a `destination` path is provided, the output is written to that file (creating parent @@ -497,13 +679,25 @@ print(f"Detected: {result.source_format}") # Frictionless: detected by presence of "fields" list result = pb.import_contract({"fields": [{"name": "x", "type": "integer"}]}) print(f"Detected: {result.source_format}") + +# dbt: detected by "models" or "sources" key +result = pb.import_contract({"version": 2, "models": [{"name": "t", "columns": []}]}) +print(f"Detected: {result.source_format}") + +# ODCS: detected by "kind": "DataContract" +result = pb.import_contract({ + "kind": "DataContract", "apiVersion": "v3.0.0", "info": {"title": "x"}, + "dataset": [{"table": "t", "columns": []}], +}) +print(f"Detected: {result.source_format}") ``` For file-based imports, the extension is also used for detection (`.schema.json` maps to JSON -Schema, `.resource.json` or `.datapackage.json` maps to Frictionless). Auto-detection is a -convenience feature that works well for common cases. When working with ambiguous files or dict -inputs that could match multiple formats, it is best to specify `format=` explicitly to avoid -any possibility of misdetection. +Schema, `.resource.json` or `.datapackage.json` maps to Frictionless, `.odcs.yml` or `.odcs.json` +maps to ODCS). Note that `.yml` and `.yaml` extensions are shared between dbt and ODCS, so for YAML +files without a distinguishing extension, auto-detection falls through to content-based inspection. +When working with ambiguous files or dict inputs that could match multiple formats, it is best to +specify `format=` explicitly to avoid any possibility of misdetection. ## Combining Imports with Extra Checks @@ -526,9 +720,9 @@ validation This pattern works well when the external schema covers structural and type constraints, but your team has additional business rules that only make sense in the Pointblank context. The imported -constraints form the foundation, and your additional `.col_vals_*()` or `.rows_*()` calls layer -on top. Because `.to_validate()` returns a standard `Validate` object, you have full access to -the entire Pointblank API for adding checks, setting thresholds, or attaching actions. +constraints form the foundation, and your additional `.col_vals_*()` or `.rows_*()` calls layer on +top. Because `.to_validate()` returns a standard `Validate` object, you have full access to the +entire Pointblank API for adding checks, setting thresholds, or attaching actions. ## Migration from Other Tools @@ -570,6 +764,39 @@ against its declared structure without needing to manually inspect the descripto each constraint. This is especially valuable when working with unfamiliar datasets where the schema descriptor is your primary documentation of what the data should contain. +### Coming from dbt + +If your analytics team maintains dbt models with schema tests: + +```python +# Import the tests from your dbt schema.yml +result = pb.import_contract("models/schema.yml", format="dbt", model="orders") + +# Now validate source data before it reaches dbt +validation = result.to_validate(data=raw_orders_df).interrogate() +``` + +This is really valuable for "shift-left" validation: catching data quality issues in raw source data +before it enters the dbt transformation pipeline. Your dbt schema tests define what the transformed +data should look like. Importing those same rules into Pointblank lets you apply them at the +ingestion boundary. + +### Coming from ODCS + +If your organization uses the Open Data Contract Standard for cross-team data governance: + +```python +# Import a data contract +result = pb.import_contract("contracts/customer_data.odcs.yml", format="odcs") + +# Validate data against the contract +validation = result.to_validate(data=customer_df).interrogate() +``` + +ODCS contracts are increasingly used in data mesh architectures where domain teams publish contracts +for their data products. Importing these contracts into Pointblank lets consumers validate incoming +data against the producer's declared contract, without needing to manually rewrite the constraints. + ### Generating a Starting Point Even if you don't plan to keep using the external format, importing is a great way to bootstrap @@ -607,6 +834,7 @@ whenever you need runtime validation. The key points to remember: Whether you are migrating from another validation tool, bootstrapping contracts from existing schemas, or maintaining interoperability with external systems, the adapter framework gives you a -clean path between external specifications and Pointblank's validation engine. As new adapters are -added in future releases, the same `import_contract()` interface will continue to work, so any code -you write today will gain new format support automatically. +clean path between external specifications and Pointblank's validation engine. The four built-in +adapters (JSON Schema, Frictionless, dbt, and ODCS) cover the most common schema formats in the data +ecosystem. And if you need to support a proprietary format, the custom adapter system (covered in +the next section) makes it straightforward to extend the framework. diff --git a/user_guide/10-contracts-and-pipelines/05-custom-adapters.qmd b/user_guide/10-contracts-and-pipelines/05-custom-adapters.qmd index f0cd3761f..439c35478 100644 --- a/user_guide/10-contracts-and-pipelines/05-custom-adapters.qmd +++ b/user_guide/10-contracts-and-pipelines/05-custom-adapters.qmd @@ -17,7 +17,8 @@ isn't covered by the built-in adapters, you can write a **custom adapter** and r the framework. Once registered, your custom adapter works seamlessly with `import_contract()` and -`export_contract()`, the same API surface your team already uses for JSON Schema and Frictionless. +`export_contract()`, the same API surface your team already uses for JSON Schema, Frictionless, dbt, +and ODCS. ## The Adapter Architecture