diff --git a/CHANGELOG.md b/CHANGELOG.md index ffe1734..37d2a82 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,11 @@ ## 1.9.0 +### Agent skills + +- **Build and edit Lightdash charts and dashboards through your AI assistant.** With `dj.codingAgent` enabled, the new `dj-create-lightdash-yaml` skill scaffolds and uploads chart/dashboard YAML for a deployed explore — resolving exact field IDs, honoring required filters, and confirming which project to target so prod isn't changed by mistake. The companion `dj-edit-lightdash-yaml` skill clarifies which filter edits are safe in place versus which need a real field ID. +- **Scaffold marts that back a Lightdash dashboard.** The `dj-create-new-model` skill now bundles mart-to-Lightdash recipes — default dashboard time windows (`lightdash.table.required_filters`), summable metrics on passthrough marts, and which framework-column exclude flag keeps your date dimension intact. All three skills treat the model's `.model.json` as the single source of truth for required filters. + ### Materialization - **Bucket and sort incremental tables.** Add `bucket` and `sorted_by` to a model's `materialization` to control how large incremental tables are bucketed and sorted on disk for faster reads. DJ generates the correct table properties for your storage format (Iceberg, Hive/Glue, or Delta Lake) and flags combinations a format can't support — such as bucketing on Delta Lake or uneven bucket counts on Hive — in the Problems tab. diff --git a/templates/skills/dj-create-lightdash-yaml/_SKILL.md b/templates/skills/dj-create-lightdash-yaml/_SKILL.md new file mode 100644 index 0000000..be5ec1e --- /dev/null +++ b/templates/skills/dj-create-lightdash-yaml/_SKILL.md @@ -0,0 +1,124 @@ +--- +name: dj-create-lightdash-yaml +description: >- + Create brand-new Lightdash chart and dashboard YAML (Dashboards-as-Code) from + scratch for a dbt model/explore managed by the DJ extension. Use when the user + wants to build, author, or scaffold a chart or dashboard that does not exist + yet (no prior `lightdash download`) and then upload it. For tweaking an + already-downloaded chart/dashboard, use `dj-edit-lightdash-yaml` instead. +compatibility: DJ (Data JSON) Framework extension workspace with .agents/dj/AGENTS.md, a deployed/preview Lightdash project, and the chart/dashboard JSON schemas bound in .vscode/settings.json +metadata: + dj-skill: '1.0' +--- + +# Create Lightdash YAML (Dashboards as Code) + +Author **net-new** `lightdash/charts/.yml` and `lightdash/dashboards/.yml` +files for an existing explore, then upload them with `--force`. This is the +"build me a dashboard from this model" path. To edit YAML that already exists +locally (downloaded), use `dj-edit-lightdash-yaml`. + +The hard part is not the YAML shape (the bound schema guides that) -- it is using +the **exact** field IDs, honoring the model's **required filters**, and knowing +that the **schema cannot validate references**. The detailed, copy-pasteable +recipes live in `references/lightdash-as-code-authoring.md`; load it before +writing. + +## Prerequisites (the explore must already exist) + +A chart references field IDs that only exist once the model is deployed as a +Lightdash **explore**. Before authoring: + +1. The dbt model has `tags: ["lightdash", "lightdash-explore"]` and a built + relation (dev view is fine). +2. The model is deployed to a project you can target -- usually a **preview**. + Create it through the DJ extension's Lightdash preview tooling, or **offer to + run the command yourself** (after confirming with the user) -- do not tell the + user to run it directly. The equivalent CLI is: + `lightdash start-preview --name "" --profiles-dir ~/.dbt --project-dir -s ` + - Do NOT pass `--defer` to `start-preview` (it routes to `dbt ls`, which + rejects `--defer`). Pass `-y` to skip the credential prompt so the preview + stores warehouse creds that can read your dev relation. + - The prod project is typically `block`ed in `dj.lightdash.restrictedProjects` + (`.vscode/settings.json`). That only stops the DJ **Upload tab** from pushing + to prod by mistake -- it does **not** stop a direct `lightdash upload` + (Lightdash allows it if you have permission), so deliberately target a + non-prod project, usually your preview. +3. Capture the **preview project UUID** (printed as `.../projects//tables`). + +## Workflow + +1. **Resolve the explore + exact field IDs.** This is read-only, so just run the + helper yourself: + `scripts/get_explore_fields.py --project --explore `. + It needs the Lightdash base URL, an API token, and the project UUID -- pass + them as `--url` / `--api-key` / `--project` flags, or rely on the standard + `LIGHTDASH_URL` / `LIGHTDASH_API_KEY` / `LIGHTDASH_PROJECT` env vars (the same + names the Lightdash CLI and DJ extension use). If those are not set, ask the + user for the values rather than guessing. **Confirm which project to target + instead of assuming one** -- the same `--project ` is used here and at + upload, so settle it up front. Lay out the choices and let the user pick: a + **preview** project you found or just created (usually the safe choice), the + configured `LIGHTDASH_PROJECT` / settings project (say whether it's prod or + listed in `dj.lightdash.restrictedProjects`), or another project UUID they name. + Pass the chosen `--project ` + explicitly rather than the env default. It prints the + `baseTable` (= `exploreName`/`tableName`), every dimension/metric field ID + (including per-interval date variants like `__week`), and + -- with `--spaces` -- valid `spaceSlug` values. If the API is unavailable, open the + project Tables page and read IDs there. **Never guess IDs from labels.** +2. **Read** `references/lightdash-as-code-authoring.md` for required keys, the + field-ID convention, the canonical date filter, the pivot recipe, the + cartesian `chartConfig` skeleton, and dashboard tile types. +3. **Author the chart** at `lightdash/charts/.yml` with the + `# yaml-language-server: $schema=...` header (the `yaml.schemas` binding in the + workspace `.vscode/settings.json`). Include the model's **required date filter**, + and keep mapping keys **sorted alphabetically** (the CLI warns on unsorted keys + -- reference §9; the header comment stays first, lists keep their order). +4. **Author the dashboard** at `lightdash/dashboards/.yml` (mapping keys + sorted too); reference the chart via a `saved_chart` tile's + `properties.chartSlug`. Add a `markdown` tile for context if helpful. +5. **Upload.** Do not instruct the user to run the upload CLI directly. Offer one + of two paths: (a) they upload from the `DJ: Lightdash - Dashboards as Code` + webview Upload tab (paste the chosen project UUID, enable `--force`, and + `--include-charts` for the dashboard), or (b) you **offer to run the command + yourself** after confirming the target project UUID. The equivalent command is + `lightdash upload --force --include-charts --validate --project -c -d `. + Always `--validate` -- it is the only thing that catches bad field IDs / missing + chart slugs. + +## Hard rules + +- **Honor model-level required filters.** The DJ framework can auto-add a date + `required_filters` (e.g. `: inThePast N days`) to the generated model + YAML. Every chart on that explore MUST filter on that base field + (`_`, not an interval variant) or the query errors. The + source is the mart's `.model.json` under `lightdash.table.required_filters`, not + the generated YAML -- to add/relax the default window, edit the model + (`dj-create-new-model`). +- **`exploreName` and `tableName` are the dbt model name** (the explore + `baseTable`), never the slugified `lightdash.table.label`. +- **Net-new files require `--force`.** Without it, upload silently skips files + Lightdash has never seen. +- **The bound YAML schema validates structure, not meaning.** A clean editor + does not prove field IDs exist -- only `lightdash upload --validate` does. +- **Sort YAML keys alphabetically.** Emit mapping keys sorted at every level so + `lightdash upload` doesn't warn about unsorted keys; keep list items in order + and the `# yaml-language-server` header on top (reference §9). +- **Never hand-edit `.sql`/`.yml` under `models/`** -- those are DJ's JSON-sync + output. To add a metric/dimension a chart needs, edit the upstream + `.model.json` first (use `dj-create-new-model`), let DJ regenerate, redeploy + the explore, then reference it. +- **Don't tell the user to run `lightdash` commands directly.** For any CLI step + (preview, upload), either point them to the matching DJ webview, or offer to run + the command yourself after confirming the target project -- the webview keeps + auth, cwd, and schema bindings in sync, and running it yourself keeps the user + out of the terminal. + +## References + +- [references/lightdash-as-code-authoring.md](references/lightdash-as-code-authoring.md) + -- required keys, field-ID rules, filter/pivot recipes, chartConfig skeleton, + tile types, and upload flags. +- [scripts/get_explore_fields.py](scripts/get_explore_fields.py) -- read-only + field-ID and space-slug lookup against the Lightdash API. diff --git a/templates/skills/dj-create-lightdash-yaml/references/lightdash-as-code-authoring.md b/templates/skills/dj-create-lightdash-yaml/references/lightdash-as-code-authoring.md new file mode 100644 index 0000000..8c67e59 --- /dev/null +++ b/templates/skills/dj-create-lightdash-yaml/references/lightdash-as-code-authoring.md @@ -0,0 +1,263 @@ +# Lightdash Dashboards-as-Code: authoring reference + +Detailed recipes for hand-authoring net-new chart/dashboard YAML in a DJ +workspace. Placeholders in `` (e.g. ``, ``, +``) stand in for your project's real names. + +The chart/dashboard JSON schemas are bound via the `yaml.schemas` setting in the +workspace `.vscode/settings.json`: + +- charts -> `chart-as-code-1.0.json` +- dashboards -> `dashboard-as-code-1.0.json` + +Both schemas use `version: 1`. + +## 1. Minimum required keys + +Validate against the bound schema; do not invent fields. + +- **Chart:** `name`, `slug`, `version: 1`, `spaceSlug`, `tableName`, + `metricQuery`, `chartConfig`, `tableConfig.columnOrder`. + `metricQuery` requires: `exploreName`, `dimensions`, `metrics`, `sorts`, + `limit`, `tableCalculations`, `filters`. +- **Dashboard:** `name`, `slug`, `version: 1`, `spaceSlug`, `tabs` (use `[]` if + untabbed), `tiles`, `filters`. +- Always start the file with the schema header so the editor validates as you + type: + `# yaml-language-server: $schema=https://raw.githubusercontent.com/lightdash/lightdash/refs/heads/main/packages/common/src/schemas/json/chart-as-code-1.0.json` + +## 2. Field-ID convention (the #1 source of errors) + +Lightdash field IDs are derived mechanically -- derive them, do not guess from +labels: + +- **Dimension:** `_` +- **Time-interval dimension:** append the granularity to the base time column: + `__day`, `_week`, `_month`, `_year`, + `_day_of_week_name`. The bare `_` is the raw timestamp. +- **Metric:** `_` where `` is the `name` + under the model's `lightdash.metrics` (custom) or the framework default (e.g. + `metric__sum`). +- **`exploreName` and `tableName`** both equal the **dbt model name** (= the + explore `baseTable`), NOT the slugified `lightdash.table.label`. The label is + display-only. + +Fastest lookup: `scripts/get_explore_fields.py --project --explore ` +or `GET /api/v1/projects/{projectUuid}/explores/{exploreName}` (read +`results.baseTable` and `results.tables[baseTable].dimensions|metrics`). + +## 3. Required filters (do not skip) + +The DJ framework can auto-add a date `required_filters` to the generated model +YAML, e.g.: + +```yaml +meta: + required_filters: + - : inThePast 14 days +``` + +Any chart on that explore MUST include a filter on that base field +(`_`), or the query errors. Target the base timestamp, not an +interval variant. + +Source of truth: the required filter lives in the mart's `.model.json` under +`lightdash.table.required_filters` (author it with `dj-create-new-model`). The +`meta.required_filters` shown above is only the generated dbt-YAML view -- to +add/relax a default window, edit the model, not the generated YAML. + +## 4. Canonical date filter ("in the last N days") + +```yaml +metricQuery: + filters: + dimensions: + id: + and: + - id: + target: + fieldId: _ + operator: inThePast + values: + - 30 + settings: + unitOfTime: days + completed: false +``` + +- Generate fresh UUIDs for brand-new rules; preserve them when editing later + (the edit skill matches rules by `id`). +- Widening the window (e.g. 30 -> 90) is purely a filter change; no dbt rebuild + is needed when the model is a view over a full-history upstream. + +## 5. Cartesian chart (bars/lines) + +`chartConfig.type: cartesian`. Required: `config.layout` and +`config.eChartsConfig`. Each series needs `type` and `encode` (with `xRef.field` +and `yRef.field`). + +```yaml +chartConfig: + type: cartesian + config: + layout: + xField: _ + yField: + - _ + showGridY: true + eChartsConfig: + series: + - type: bar # line | bar | scatter | area + encode: + xRef: + field: _ + yRef: + field: _ +``` + +## 6. Time series stacked/split by a category (pivot) + +To put time on the x-axis and split a metric by a category. The simplest and most +robust form does **not** enumerate category values: set `pivotDimensions` + +`pivotConfig.columns` and a single stacking series, and Lightdash expands one +series per category automatically. This needs no knowledge of the distinct +values, so you never have to run a (credentialed) query to list them: + +```yaml +metricQuery: + dimensions: + - __week + - _ + metrics: + - _ + pivotDimensions: + - _ + # ... sorts (by the time field, ascending), limit (>= periods x categories), filters ... +pivotConfig: + columns: + - _ +chartConfig: + type: cartesian + config: + layout: + xField: __week + yField: + - _ + stack: true + eChartsConfig: + series: + - type: bar + stack: # shared string -> stacked + encode: + xRef: + field: __week + yRef: + field: _ +``` + +Notes: +- `pivotDimensions` (in `metricQuery`) and `pivotConfig.columns` (top-level) must + both name the category field. +- Set `limit` high enough for `periods x categories` so no series is truncated. + +### Optional: enumerate series to highlight specific values + +Only if you want to style/stack specific categories (e.g. the top N), add +`pivotValues` to the series. This requires knowing the values -- take them from a +known set; do not run a credentialed query just to enumerate them. Values you do +not list still render with default styling. + +```yaml + eChartsConfig: + series: + - type: bar + stack: + encode: + xRef: + field: __week + yRef: + field: _ + pivotValues: + - field: _ + value: # one series per enumerated value +``` + +## 7. Dashboard tiles + +Tiles need `type`, `x`, `y`, `w`, `h` (36-column grid) and `properties`. Common +types: + +```yaml +tiles: + - type: markdown # context / titles + x: 0 + y: 0 + w: 36 + h: 4 + tabUuid: null + properties: + title: "" + content: | + ### + + - type: saved_chart + x: 0 + y: 4 + w: 36 + h: 12 + tabUuid: null + properties: + chartSlug: # must match a local chart slug or an existing chart + title: "" +``` + +Other tile types: `sql_chart`, `loom`, `heading`, `data_app`. Keep `x/y/w/h` +rectangles non-overlapping. + +## 8. Uploading + +Don't tell the user to run the upload CLI directly. Either point them to the +`DJ: Lightdash - Dashboards as Code` webview (paste the project UUID in the +required Project UUID field), or offer to run the command yourself after +confirming the target project. The equivalent command: + +``` +lightdash upload --force --include-charts --validate \ + --project \ + -c -d +``` + +- `--force`: required for files Lightdash has not seen before (net-new). +- `--include-charts`: when uploading a dashboard that references newly-added + local charts. +- `--validate`: runs Lightdash's content validator against the live explore + (catches bad field IDs, missing chart slugs). The YAML schema cannot do this. +- Missing spaces are auto-created unless `--skip-space-create`; pass an existing + `spaceSlug` (from `get_explore_fields.py --spaces`) to avoid creating one. + +## 9. Gotchas + +- **Schema validates structure, not meaning.** Clean lint != valid chart. Always + `--validate`. +- **Sort YAML keys alphabetically.** `lightdash upload` warns + ` has unsorted YAML keys`. Author/emit mapping keys sorted at every level + so the warning never fires (this also matches what `lightdash download` + produces). Sort **mapping keys only -- never reorder list items** (`tiles`, + `dimensions`, `sorts`, `series`, `columnOrder` are order-sensitive). Keep the + `# yaml-language-server` header comment as the first line. Sort with a key + sorter if one is available (e.g. `yq 'sort_keys(..)'` or your editor's YAML + formatter); otherwise author keys in alphabetical order. +- **`lightdash download -c ` can return 0 files** for some slugs; do not + rely on it to fetch a template. Author from this reference + the explore API + instead. +- **Prod is guarded, not locked.** A project listed in + `dj.lightdash.restrictedProjects` (workspace `.vscode/settings.json`) with + `mode: block` makes the DJ **Upload tab** refuse to push to it -- a guardrail + against changing prod by mistake. It does **not** block a direct + `lightdash upload` (Lightdash permits it if you have access), so target a + preview project UUID deliberately rather than relying on the guardrail. +- **Verify data before relying on a chart.** Confirm the chart returns rows with + `POST /api/v1/projects/{uuid}/explores/{exploreName}/runQuery` (same + dimensions/metrics/filters as the chart) so the rendered view is not empty. + Some agent environments block credentialed data-extraction calls; if yours + does, don't retry-loop -- verify from the DJ webview or ask the user instead. diff --git a/templates/skills/dj-create-lightdash-yaml/scripts/get_explore_fields.py b/templates/skills/dj-create-lightdash-yaml/scripts/get_explore_fields.py new file mode 100644 index 0000000..895f2bf --- /dev/null +++ b/templates/skills/dj-create-lightdash-yaml/scripts/get_explore_fields.py @@ -0,0 +1,136 @@ +#!/usr/bin/env python3 +"""Read-only Lightdash lookup for authoring Dashboards-as-Code YAML. + +Prints the exact field IDs (dimensions + metrics, including per-interval date +variants) for an explore, or the valid space slugs for a project. Use the output +to fill `exploreName`/`tableName`, dimension/metric IDs, and `spaceSlug` in chart +and dashboard YAML -- never guess these from labels. + +Credentials can be passed as flags or read from environment variables. The env +var names below are the **standard Lightdash CLI / DJ extension convention** (the +same vars the CLI and the extension's preview tooling use), so they are not +specific to any one project: + + --url / LIGHTDASH_URL e.g. https://lightdash.example.com (required) + --api-key / LIGHTDASH_API_KEY personal access token (required) + --project / LIGHTDASH_PROJECT project UUID (required) + +Flags take precedence over env vars, so you can run this without exporting +anything (e.g. when the values live elsewhere or you want to avoid env state). + +Usage: + python get_explore_fields.py --explore + python get_explore_fields.py --url --api-key --project --explore + python get_explore_fields.py --spaces + python get_explore_fields.py --explore --json # raw-ish field dump + +Read-only: issues only HTTP GETs. Safe to run against prod or preview projects. +""" +import argparse +import json +import os +import sys +import urllib.error +import urllib.request + + +def _api_get(base, key, path): + if not base or not key: + sys.exit( + "error: provide Lightdash credentials via --url/--api-key flags or the " + "LIGHTDASH_URL/LIGHTDASH_API_KEY env vars." + ) + url = base.rstrip("/") + path + req = urllib.request.Request(url, headers={"Authorization": f"ApiKey {key}"}) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + return json.load(resp) + except urllib.error.HTTPError as exc: + body = exc.read().decode("utf-8", "replace")[:400] + sys.exit(f"error: HTTP {exc.code} for {path}\n{body}") + except urllib.error.URLError as exc: + sys.exit(f"error: could not reach {url}: {exc}") + + +def _results(payload): + return payload.get("results", payload) if isinstance(payload, dict) else payload + + +def list_spaces(base, key, project): + data = _results(_api_get(base, key, f"/api/v1/projects/{project}/spaces")) + print("spaceSlug\tname") + for sp in data: + if isinstance(sp, dict): + print(f"{sp.get('slug')}\t{sp.get('name')}") + + +def show_explore(base, key, project, explore, as_json): + data = _results( + _api_get(base, key, f"/api/v1/projects/{project}/explores/{explore}") + ) + base_table = data.get("baseTable") + table = data.get("tables", {}).get(base_table, {}) + dims = table.get("dimensions", {}) + metrics = table.get("metrics", {}) + + if as_json: + out = { + "exploreName": base_table, + "label": table.get("label"), + "dimensions": {f"{base_table}_{k}": v.get("type") for k, v in dims.items()}, + "metrics": {f"{base_table}_{k}": v.get("type") for k, v in metrics.items()}, + } + print(json.dumps(out, indent=2)) + return + + print(f"exploreName / tableName: {base_table}") + print(f"label: {table.get('label')}") + print("\nDIMENSIONS (fieldId | type | hidden)") + for k, v in sorted(dims.items()): + print(f" {base_table}_{k} | {v.get('type')} | hidden={v.get('hidden')}") + print("\nMETRICS (fieldId | type | label)") + for k, v in sorted(metrics.items()): + print(f" {base_table}_{k} | {v.get('type')} | {v.get('label')}") + + +def main(): + parser = argparse.ArgumentParser( + description="Read-only Lightdash explore field-ID / space-slug lookup." + ) + parser.add_argument( + "--url", + default=os.environ.get("LIGHTDASH_URL"), + help="Lightdash base URL (defaults to LIGHTDASH_URL).", + ) + parser.add_argument( + "--api-key", + default=os.environ.get("LIGHTDASH_API_KEY"), + help="Lightdash personal access token (defaults to LIGHTDASH_API_KEY).", + ) + parser.add_argument( + "--project", + default=os.environ.get("LIGHTDASH_PROJECT"), + help="Project UUID (defaults to LIGHTDASH_PROJECT).", + ) + parser.add_argument("--explore", help="Explore (dbt model) name.") + parser.add_argument( + "--spaces", action="store_true", help="List space slugs instead of fields." + ) + parser.add_argument( + "--json", action="store_true", help="Emit field IDs as JSON." + ) + args = parser.parse_args() + + if not args.project: + sys.exit("error: pass --project or set LIGHTDASH_PROJECT.") + + if args.spaces: + list_spaces(args.url, args.api_key, args.project) + return + if not args.explore: + sys.exit("error: pass --explore (or --spaces).") + show_explore(args.url, args.api_key, args.project, args.explore, args.json) + + +if __name__ == "__main__": + main() diff --git a/templates/skills/dj-create-new-model/_SKILL.md b/templates/skills/dj-create-new-model/_SKILL.md index d033519..54aa6bb 100644 --- a/templates/skills/dj-create-new-model/_SKILL.md +++ b/templates/skills/dj-create-new-model/_SKILL.md @@ -33,6 +33,8 @@ One clarifying question if source vs existing model is unclear. **Path:** `models////______.model.json` (`stg_*`→`staging`, etc.). +**Locate the project first:** the dbt project may be nested, not the workspace root — find its `dbt_project.yml` and treat `models/...` paths as relative to that directory. `.dj/schemas/` lives at the DJ/workspace root, which can differ from the dbt project dir. If more than one dbt project exists, use the one named in `dj.dbtProjectNames`. + **Checklist:** - [ ] `type` from table; read **`.dj/schemas/model.type..schema.json`**; if CTE / subquery / **`from.model.rollup`** / hooks / **`agg`** / materialization, also **`model.cte`**, **`model.subquery`**, **`model.from.rollup`**, **`model.sql_hooks`**, **`model.materialization`**, **`model.select.*.with.agg`** as needed @@ -65,7 +67,7 @@ The layer directory is derived from the type prefix: `stg_*` -> `staging`, `int_ - [ ] Step 4: Refer to the AGENTS.md "Model Types" section for the example structure of the selected type - [ ] Step 5: Read upstream model/source files to verify available columns before writing `select` - [ ] Step 6: Create the `.model.json` file at the correct path using JSONC format (comments and trailing commas allowed) -- [ ] Step 7: Verify the file is valid against the schema +- [ ] Step 7: Verify validity via the editor's bound schema + DJ's on-save regeneration/diagnostics — do not assume standalone validators (`jsonschema`, `pyyaml`, `pip`) are installed (see [references/mart-lightdash-recipes.md](references/mart-lightdash-recipes.md) §4) ## Important Conventions @@ -86,6 +88,7 @@ The layer directory is derived from the type prefix: `stg_*` -> `staging`, `int_ - Source freshness can be disabled with `"freshness": null` at source or table level - Free-form `meta` keys are allowed at both model and column level on `.model.json` (e.g., `owner`, `pii`, `compliance`). See AGENTS.md "Custom Meta" section, `model.meta.schema.json`, `column.meta.schema.json` - For Lightdash column config, author `select[i].lightdash.dimension`, `.metrics`, `.metrics_merge`, `.case_sensitive` — not `meta.dimension` etc. The framework surfaces a Warning-severity diagnostic in the Problems tab if authored under `meta` +- **Marts that back a Lightdash explore/dashboard** — for a default time window (`lightdash.table.required_filters`), a summable metric on a `mart_select_model` passthrough, the right framework-column exclude flag, and how validation works, see [references/mart-lightdash-recipes.md](references/mart-lightdash-recipes.md) ## Gotchas @@ -111,3 +114,7 @@ The layer directory is derived from the type prefix: `stg_*` -> `staging`, `int_ - `materialization` structured form allows `"format": "iceberg"` for Iceberg storage -- partitioning keyword changes automatically based on format - Both `materialized` (legacy) and `materialization` (preferred) are accepted; when both are present, `materialization` takes precedence - **`meta` is free-form but has a few reserved keys**. Column `type`, `dimension`, `metrics`, `case_sensitive`, `origin` and model `metrics`, `local_tags`, `case_sensitive`, and any key on `lightdash.table` are framework-owned — author via the structured sibling field (`type`, `lightdash.*`, `tags: [{ type: "local", tag }]`, etc.). Collisions trigger Warning diagnostics in the Problems tab + +## References + +- [references/mart-lightdash-recipes.md](references/mart-lightdash-recipes.md) — recipes for marts that back a Lightdash explore: default time window via `lightdash.table.required_filters`, a summable metric on a `mart_select_model` passthrough, the right framework-column exclude flag, and how validation works. diff --git a/templates/skills/dj-create-new-model/references/mart-lightdash-recipes.md b/templates/skills/dj-create-new-model/references/mart-lightdash-recipes.md new file mode 100644 index 0000000..781baef --- /dev/null +++ b/templates/skills/dj-create-new-model/references/mart-lightdash-recipes.md @@ -0,0 +1,90 @@ +# Marts that back a Lightdash explore: recipes + +Recipes for `.model.json` marts that surface as a Lightdash explore/dashboard. +Placeholders in `` (e.g. ``, ``, +``) stand in for your project's real names. Author everything in the +`.model.json` source of truth -- DJ regenerates the `.sql`/`.yml`. + +## 1. Default time window for dashboard-facing marts (`required_filters`) + +If an explore should default to a recent window (e.g. "last 30 days" in every +chart), declare a **model-level required date filter**. Author it under +`lightdash.table.required_filters` in the `.model.json` -- NOT under `meta`. (The +`meta.required_filters` you may see in generated dbt YAML is the compiled output; +the source of truth is `lightdash.table.required_filters`.) + +```json +{ + "lightdash": { + "table": { + "label": "", + "required_filters": [ + { "": "inThePast 30 days" } + ] + } + } +} +``` + +- Every chart on the explore inherits a filter on ``, so you do not + have to re-add it per chart (individual charts may still narrow the window). +- Schema: `.dj/schemas/lightdash.required_filters.schema.json` (an array of + `{ "": "" }`), referenced from + `.dj/schemas/lightdash.table.schema.json`. + +## 2. Summable metric on a passthrough mart (`mart_select_model`) + +`mart_select_model` (and `int_union_models`) cannot use `agg`/`aggs` in `select`. +To expose a summable measure, keep the fact column as a passthrough/`expr` column +and define the metric under that column's `lightdash.metrics`: + +```json +{ + "name": "", + "type": "fct", + "data_type": "number", + "expr": "", + "description": "", + "lightdash": { + "dimension": { "hidden": true }, + "metrics": [ + { + "name": "", + "type": "sum", + "label": "", + "format": "usd", + "round": 0, + "sql": "coalesce(, 0)" + } + ] + } +} +``` + +- Hide the raw fact dimension (`dimension.hidden: true`) so only the metric shows + in the explore. +- The metric's Lightdash field ID becomes `_` (used when + authoring charts -- see the `dj-create-lightdash-yaml` skill). + +## 3. Framework columns on a Lightdash mart (which exclude flag to use) + +- **Time-series mart** (keeps a `datetime`/date dimension): set only the + individual flags you want, e.g. `exclude_portal_source_count` and/or + `exclude_portal_partition_columns`. Do **not** use the + `exclude_framework_artifacts` bundle here -- `"columns"`/`"all"` also drop + `datetime`, which you need as the date dimension. +- **Pure lookup / dimension mart** (no time grain): the + `exclude_framework_artifacts: "columns"` (or `"all"`) shortcut is the clean way + to drop all framework columns at once. + +## 4. How validation works (do not assume external validators) + +- **Structure** is checked by the editor's bound JSON schema -- a DJ workspace + binds `*.model.json` to `.dj/schemas/model.schema.json` -- and by the DJ + extension regenerating the `.sql`/`.yml` and surfacing diagnostics in the + Problems tab on save. +- **Do not assume** standalone validators (`jsonschema`, `pyyaml`, `pip`) are + installed in your environment; they often are not. If you need to inspect a + file from the shell, use a tool you have confirmed is present. +- Lightdash field-ID/semantic validation happens later, at + `lightdash upload --validate` (see the Lightdash skills). diff --git a/templates/skills/dj-edit-lightdash-yaml/_SKILL.md b/templates/skills/dj-edit-lightdash-yaml/_SKILL.md index 54e2e3a..4476b2c 100644 --- a/templates/skills/dj-edit-lightdash-yaml/_SKILL.md +++ b/templates/skills/dj-edit-lightdash-yaml/_SKILL.md @@ -1,11 +1,12 @@ --- name: dj-edit-lightdash-yaml description: >- - Edit Lightdash chart or dashboard YAML files managed by the DJ extension's - Dashboards-as-Code workflow. Use when the user wants to tweak a chart's - filters, sorts, axes, table config, or dashboard tiles/filters locally before - re-uploading to Lightdash via the `DJ: Lightdash — Dashboards as Code` - webview. + Edit existing Lightdash chart or dashboard YAML files managed by the DJ + extension's Dashboards-as-Code workflow. Use when the user wants to tweak a + chart's filters, sorts, axes, table config, or dashboard tiles/filters in YAML + that already exists locally (downloaded or previously authored) before + re-uploading via the `DJ: Lightdash - Dashboards as Code` webview. To create a + brand-new chart/dashboard from scratch, use `dj-create-lightdash-yaml` instead. compatibility: DJ (Data JSON) Framework extension workspace with .agents/dj/AGENTS.md and a populated `lightdash/` directory metadata: dj-skill: '1.0' @@ -13,10 +14,12 @@ metadata: # Edit Lightdash YAML (Dashboards as Code) -Modify chart and dashboard YAML files written by `lightdash download` so the -user can re-`lightdash upload` them. Do **not** run the CLI yourself — the user -runs it from the `DJ: Lightdash — Dashboards as Code` webview, which also keeps -the YAML schema bindings in sync. +Modify chart and dashboard YAML files that already exist locally (written by a +download step, or previously authored) so they can be re-uploaded. Don't tell the +user to run the upload CLI directly: either point them to the +`DJ: Lightdash - Dashboards as Code` webview (which keeps auth and YAML schema +bindings in sync), or **offer to run the command yourself** after confirming the +target project. ## When this skill applies @@ -25,8 +28,10 @@ the YAML schema bindings in sync. - The user wants to add/remove/rearrange tiles on a dashboard, or change dashboard-level filters. - The user has already downloaded YAML for the asset (`lightdash/charts/.yml` - or `lightdash/dashboards/.yml`). If the file does not yet exist, ask - them to run the Download tab (entire project, or specific assets) first. + or `lightdash/dashboards/.yml`). If the file does not yet exist, either + point them to the webview's Download tab (or offer to fetch it yourself) first, + or — for a genuinely new asset — switch to the **`dj-create-lightdash-yaml`** + skill. ## Workflow @@ -43,16 +48,18 @@ the YAML schema bindings in sync. and verify the `lightdash` block exposes the dimensions/metrics the chart references. If the chart needs a new metric or dimension, add it to the `.model.json` first (so DJ regenerates the dbt YAML), then reference it from - the chart YAML. + the chart YAML. For exact field-ID derivation, see + `references/lightdash-as-code-fields.md`. 5. **For dashboard edits referencing charts**, ensure every - `properties.savedChartSlug` (or equivalent) matches a chart slug that exists + `properties.chartSlug` (or equivalent) matches a chart slug that exists locally under `charts/` or already exists on Lightdash. 6. **Make the smallest possible diff.** Edit only the fields the user asked about and leave everything else byte-identical. -7. **Tell the user to re-upload via the Upload tab** of the - `DJ: Lightdash — Dashboards as Code` webview. Selection-driven upload sends - only the files they pick (the edited ones); selecting all (or nothing) runs - an entire-project upload. +7. **Re-upload — but don't tell the user to run the CLI directly.** Either point + them to the Upload tab of the `DJ: Lightdash - Dashboards as Code` webview + (selection-driven upload sends only the files they pick; selecting all or + nothing runs an entire-project upload), or **offer to run the upload command + yourself** after confirming the target project. ## Hard rules @@ -60,9 +67,10 @@ the YAML schema bindings in sync. the local file to the remote resource. Renaming creates a duplicate on upload. - **Never change `version`.** It pins the schema; bumping by hand breaks upload. - **Never delete a top-level key you do not recognize.** Preserve it. -- **Never run `lightdash download` / `lightdash upload` directly.** Use the - extension's webview command so auth, working directory, and YAML schema - bindings stay in sync. +- **Don't instruct the user to run `lightdash download` / `lightdash upload` + directly.** Either point them to the extension's webview (so auth, working + directory, and YAML schema bindings stay in sync), or offer to run the command + yourself after confirming the target project. - **Never edit `.sql` or `.yml` files under `models/`** as part of this skill — those belong to DJ's JSON-sync flow, not Dashboards-as-Code. @@ -71,6 +79,7 @@ the YAML schema bindings in sync. | Intent | Where in the YAML | | --- | --- | | Change a chart's row limit | `metricQuery.limit` | +| Change a date window (e.g. last 30 -> 90 days) | `metricQuery.filters.dimensions.and[].values` (keep the rule `id` UUID) | | Add/remove a chart filter | `metricQuery.filters` (preserve `id` UUIDs on existing filter rules) | | Re-order chart sorts | `metricQuery.sorts` (each entry has `fieldId` and `descending`) | | Add a custom table calc | `metricQuery.tableCalculations` | @@ -86,17 +95,40 @@ the YAML schema bindings in sync. - **Tile layout is grid-based.** Lightdash dashboards use a 36-column grid. Overlapping `x/y/w/h` rectangles will render in unexpected stacking order; shift other tiles before adding a new one. -- **`exploreName` is the dbt model's Lightdash table name**, not the - `____` model name. It usually maps to the model's - `lightdash.table.label` slugified, but confirm via the model's generated YAML - if uncertain. +- **`exploreName` and `tableName` are the dbt model name** - i.e. the explore's + `baseTable`, NOT the slugified `lightdash.table.label`. The `label` is only the display name shown + in the UI; it never appears in field IDs or `exploreName`. When in doubt, read + the explore's `baseTable` from + `GET /api/v1/projects/{projectUuid}/explores/{exploreName}` (or the Tables + page URL), not the model's label. - **`additionalMetrics` is local to the chart.** If the user wants this metric available across multiple charts, add it to the model's `lightdash` block instead and reference it from `metricQuery.metrics`. -- **`--force` is required for net-new files.** If the user authored a chart or - dashboard YAML by hand (rather than downloading), they must enable the - `--force` toggle in the Upload tab so Lightdash creates the new resource. -- **`--include-charts` only matters for dashboard uploads.** It tells Lightdash - to also upload any charts referenced by the selected dashboard. Mention it - if the user is uploading a dashboard that references newly-added local - charts. +- **Editing a value vs editing references.** Changing filter `values`, `limit`, + `sorts`, labels, or tile layout is safe in-place. Changing any `fieldId` + requires a real field ID — see `references/lightdash-as-code-fields.md`. +- **The bound YAML schema catches structure, not meaning.** It flags unknown + keys, wrong types, and missing required fields in-editor, but it cannot know + whether a `fieldId` exists in the explore, whether a `chartSlug` resolves, or + whether filter UUIDs are consistent. Those only fail at upload - always upload + with `--validate` and treat a clean editor as necessary, not sufficient. +- **Sort YAML keys alphabetically.** `lightdash upload` warns + ` has unsorted YAML keys`. Keep mapping keys sorted at every level when you + edit (this matches what `lightdash download` emits). Sort **mapping keys only -- + never reorder list items** (`tiles`, `dimensions`, `sorts`, `series`, + `columnOrder` are order-sensitive). Keep the `# yaml-language-server` header on + top. Use a key sorter if available (e.g. `yq 'sort_keys(..)'` / your editor's + formatter), else keep keys in alphabetical order. +- **A round-tripped file is normalized.** After `lightdash upload`/`download` the + local file is rewritten: mapping keys alphabetized, server fields added + (`contentType`, `chartName`, `tileSlug`, `verification`, `hideFrame`, ...), the + `# yaml-language-server` header stripped, and block scalars reflowed (`|` -> + `>`). Re-read the file before each edit (it won't match what you last authored) + and match on minimal unique substrings. The missing header is harmless -- the + workspace `yaml.schemas` path binding still validates. + +## References + +- [references/lightdash-as-code-fields.md](references/lightdash-as-code-fields.md) + -- field-ID derivation, the canonical filter-rule shape, and upload flags + (shared with `dj-create-lightdash-yaml`). diff --git a/templates/skills/dj-edit-lightdash-yaml/references/lightdash-as-code-fields.md b/templates/skills/dj-edit-lightdash-yaml/references/lightdash-as-code-fields.md new file mode 100644 index 0000000..842d189 --- /dev/null +++ b/templates/skills/dj-edit-lightdash-yaml/references/lightdash-as-code-fields.md @@ -0,0 +1,79 @@ +# Lightdash Dashboards-as-Code: field IDs, filters, upload + +Shared reference for editing chart/dashboard YAML in a DJ workspace. +Placeholders in `` stand in for your project's real names. (The +full authoring guide, including chartConfig and pivot recipes, lives in +`dj-create-lightdash-yaml`.) + +## Field-ID convention + +Lightdash field IDs are derived mechanically. Derive them; do not guess from +labels. + +- **Dimension:** `_` +- **Time-interval dimension:** append the granularity to the base time column: + `__day`, `_week`, `_month`, `_year`, `_day_of_week_name`. + The bare `_` is the raw timestamp. +- **Metric:** `_` where `` is the `name` + under the model's `lightdash.metrics` (custom) or the framework default (e.g. + `metric__sum`). +- **`exploreName` / `tableName`** = the dbt model name (= explore `baseTable`), + never the slugified `lightdash.table.label`. + +Lookup (read-only): `GET /api/v1/projects/{projectUuid}/explores/{exploreName}` +and read `results.baseTable` + `results.tables[baseTable].dimensions|metrics`. +The `dj-create-lightdash-yaml` skill ships `scripts/get_explore_fields.py` for +this. + +## Canonical date filter ("in the last N days") + +```yaml +metricQuery: + filters: + dimensions: + id: + and: + - id: + target: + fieldId: _ + operator: inThePast + values: + - 30 + settings: + unitOfTime: days + completed: false +``` + +- **Keep the `id` UUIDs** when editing an existing rule (Lightdash matches rules + by `id`). Only generate new UUIDs for brand-new rules. +- To change the window, edit only `values` (e.g. `30` -> `90`). No dbt rebuild is + needed when the model is a view over full-history upstream data -- the filter is + applied at query time. +- Honor the model's required date filter: a chart must keep a filter on that base + field or its query errors. The source is the mart's `.model.json` under + `lightdash.table.required_filters` (authored via `dj-create-new-model`); + `meta.required_filters` is only the generated dbt-YAML view -- change it in the + model, not here. + +## Upload flags + +Don't tell the user to run the upload CLI directly: either point them to the +`DJ: Lightdash - Dashboards as Code` webview, or offer to run it yourself after +confirming the target project. The equivalent command: + +```bash +lightdash upload --force --include-charts --validate \ + --project -c -d +``` + +- `--force`: needed for files Lightdash has not seen before (net-new). +- `--include-charts`: when a dashboard references newly-added local charts. +- `--validate`: runs the content validator against the live explore -- the only + check for bad field IDs / missing chart slugs. +- Spaces are auto-created unless `--skip-space-create`; use an existing + `spaceSlug` to avoid creating one. +- A `mode: block` entry in `dj.lightdash.restrictedProjects` (workspace + `.vscode/settings.json`) only makes the DJ **Upload tab** refuse the push -- a + guardrail against editing prod by mistake. A direct `lightdash upload` still + works if you have Lightdash access, so target the intended (usually preview) + project UUID deliberately.