Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 49 additions & 0 deletions docs/feature-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,55 @@ print(result.quality.to_markdown())
assert result.passed_gate
```

### `freshdata clean --config` files

`--config` takes a JSON or YAML object with two optional sections, `clean` and
`enterprise`. An unknown section or key, including a typo, stops the run before
any data is read: a one-line error with a "did you mean" hint, exit 1.

```yaml
clean:
strategy: balanced
enterprise:
fail_under_trust: 80
masking:
- {name: pii, columns: [email], strategy: hash}
enable_privacy_detection: true
privacy: {min_score: 0.6}
```

`clean` accepts any `CleanConfig` option. `enterprise` accepts these keys:

| Key | Value | Builds |
|---|---|---|
| `actor` | string or null | `EnterpriseConfig.actor` |
| `fail_under_trust` | number from 0 to 100, or null | the trust gate; `--fail-under-trust` overrides it |
| `enable_masking`, `enable_clustering`, `enable_validation`, `enable_lineage`, `enable_privacy_detection`, `enable_entity_resolution` | `true` or `false` | the toggle of the same name |
| `masking` | list of objects | one `MaskingRule` each; `--mask` adds more |
| `semantic` | list of objects | one `SemanticValidatorConfig` each |
| `clustering` | object | `ClusterConfig`; `--cluster` replaces it |
| `trust_weights` | object | `TrustScoreWeights` |
| `lineage` | object | `LineageConfig` |
| `privacy` | object | `PIIDetectionConfig`, applied when `enable_privacy_detection` is true |
| `k_anonymity` | object | `KAnonymityConfig` |
| `entity_resolution` | object, with `blocking_rules` and `comparisons` as lists of objects | `EntityResolutionConfig` (with `BlockingRule` and `ComparisonLevel`), applied when `enable_entity_resolution` is true |

Nested objects take the field names of the class they build, and unknown names
are rejected the same way. Three `EnterpriseConfig` fields do nothing in
`freshdata clean`: `enable_contracts` and `drift` (the command takes no baseline or
data contract to check against) and `anonymization` (no pipeline applies it; use
`masking`, or `privacy` with `enable_privacy_detection`). They are accepted and
ignored when null or set to their default (`enable_contracts: false`, `drift: {}`
or an object of `DriftConfig` defaults, `anonymization: []`). Any other value is
rejected with that explanation.

The config file is validated the same way on every `--engine`. With a native
engine (`polars`, `duckdb`, `spark`, `freshcore`, `auto`) the `clean` section is
applied under the command-line options, as on pandas, except `context` and
`policy`, which only the pandas engine supports. Native engines do not run the
enterprise stage, so an `enterprise` section that sets anything other than the
defaults exits 1 and names the keys; drop them or use `--engine pandas`.

## Compliance reports

The `freshdata.compliance` subpackage turns a `CleanReport` into a regulatory
Expand Down
4 changes: 3 additions & 1 deletion docs/integrations.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,9 @@ gates it, and exits non-zero (with `--fail`) if any model is below the threshold
Ephemeral and disabled models are not read; they are listed under `"skipped"`. If
no model is gated at all (an empty manifest, or only ephemeral models), `all_passed`
is `false` and `--fail` exits 1. A file that is not a dbt manifest (for example
`run_results.json`) is reported as a one-line error with exit 1. For
`run_results.json`) is reported as a one-line error with exit 1. Stdout carries
only the JSON summary, so it can be piped straight into `jq` or `json.loads`;
cleaning warnings and other messages go to stderr. For
a single model — or to write per-model `<model>_audit.json` files — use
`FreshDataDbtTransform`:

Expand Down
Loading
Loading