docs: schemas and diagrams

docs/README.md

@@ -1,16 +1,47 @@
 # Topographic System
 
-The topographic system is a collection of components that are used to create New Zealand's Topo50 https://www.linz.govt.nz/products-services/maps/new-zealand-topographic-maps
+The topographic system is a collection of components that are used to create New Zealand's Topographic mapping products such as NZTopo50 https://www.linz.govt.nz/products-services/maps/new-zealand-topographic-maps
 
-## Components
+## Topographic Map Production
 
-- [linz/topographic-system](https://github.com/linz/topographic-system) - Source code and scripts to control the entire process
-- [linz/topographic-qgis](https://github.com/linz/topographic-qgis) - QGIS project files and symbology for editing and creating topographic data and maps
+Topographic maps are produced with [QGIS](https://github.com/qgis/qgis). All of the QGIS project files and assets are stored in [linz/topographic-qgis](https://github.com/linz/topographic-qgis)
 
-## Topographic Datasets
+The map production workflow automates the production of all map sheets rendering the maps with a Headless QGIS inside of docker.
 
-topographic data is stored as kart repos roughly broken down into a similar groups, some large datasets (contours) are in seperate repositories due to performance impacts of their size
+![](./assets/system.map.render.excalidraw.png)
+
+- [map prepare](../packages/map/README.md) - Prepare a map export run, creating immutable STAC based version of the dataset in S3, suitable for long term storage and versioning
+- [map export](../packages/map/README.md) - Export one or many map sheets from a map preparation
+- [stac push](../packages/stac/README.md) - Push the exported assets into a STAC Catalog
+
+## Topographic Data Editing
+
+![](./assets/system.data.edit.excalidraw.png)
+
+All editable topographic data is stored as [kart](https://github.com/koordinates/kart) repositories roughly broken down into similar groups, some large datasets (contours) are in separate repositories due to performance impacts of their size
+
+### Topographic datasets
 
 - [linz/topographic-data](https://github.com/linz/topographic-data) - Topographic data eg `water` or `airport`
 - [linz/topographic-product-data](https://github.com/linz/topographic-product-data) - Product specific datasets (eg `nz_topo50_map_sheet`)
 - [linz/topographic-contour-data](https://github.com/linz/topographic-contour-data) - Topo50 Contour lines
+
+### Data flow
+
+![](./assets/system.data.flow.excalidraw.png)
+
+Before data is merged into these kart repositories, Github actions is used to ensure data quality and consistent map production using a standard pull request based git flow.
+
+- [map visual-diff](../packages/map/README.md) - Export NZTopo50 map sheets and diff the results
+- [kart validate](../packages/kart/README.md) - Validate parquet data matches the topographic schemas
+- [kart to-parquet](../packages/kart/README.md) - Convert and optimize the parquet datasets
+
+Once the data is merged into master the data is exported as geoparquet and stored into S3 for use in the [map production system](#topographic-map-production)
+
+- [stac push](../packages/stac/README.md) - Push the STAC and assets into the STAC catalogue in S3
+
+### Data Storage
+
+Data is stored as a immutable date based system, `/data/airport/year=2026/date=2026-09-03T00-00-00Z/airport.parquet`. and also a mutable latest folder `/data/airport/latest/` that points to the latest date based version. More information on the [storage structure](./storage.structure.md)
+
+![](./assets/storage.structure.date.excalidraw.png)

docs/storage.schema.md

@@ -0,0 +1,101 @@
+# Dataset Schema
+
+All datasets have a corresponding schema defined as [Typespec](https://typespec.io)
+
+## Why Typespec
+
+Typespec allows us to create the schema once, and export it into many formats we use:
+
+- JSONSchema - Defines the shape of the data
+- Parquet - Physical storage format
+- Markdown - Documentation
+- Typescript - NPM module to import types
+- Pydantic (Future) - Python module to import types
+
+## Key fields
+
+Ideally all datasets should contain three key meta fields
+
+- id - `uuidv7` - Unique ID with the creation time of the feature
+- created_at - `datetime` - Creation time of the feature
+- updated_at - `datetime` - Last modified time of the feature
+
+## Versioning
+
+Schemas are versioned using Typespec's versioning structure, with all versions of the schemas stored in the topographic [schema](../packages/schema/README.md) package.
+
+Historical schema releases are immutable, to ensure data integrity and prevent breaking downstream consumers. Any structural modifications to released versions (updating fields or changing data types) are prohibited and will fail CI, Non-breaking documentation updates (such as docstrings, descriptions, or comments) are allowed.
+
+## Emitters
+
+### JSON Schema
+
+JSON schemas are stored in a public https accessible location with basic STAC metadata.
+
+```yml
+/schema/catalog.json # Root catalog pointing to every release
+/schema/latest/collection.json # Links to all schemas within latest (airport, building, etc)
+/schema/latest/airport.json # points to v2.1
+
+/schema/release=v1.1/collection.json
+/schema/release=v1.1/airport.json
+
+/schema/release=v1.2/collection.json
+/schema/release=v1.2/airport.json
+
+/schema/release=v2.1/collection.json
+/schema/release=v2.1/airport.json
+```
+
+### Typescript
+
+All schemas and their types are published into npm `@linzjs/topographic-schema`
+
+```typescript
+import type {Airport} from '@linzjs/topographic-schema'; // Latest
+
+const airport: Airport = { type: "airport", ... };
+
+import type {Airport as AirportV1_1} from '@linzjs/topographic-schema/v1.1'; // Specific version
+```
+
+### Parquet JSON
+
+JSON schema is not expressive enough to specify the parquet layout of the dataset, so a Parquet JSON is created, based off the [typespec proposal](https://github.com/microsoft/typespec/issues/10334)
+
+```json
+{
+  "type": "message",
+  "name": "airport",
+  "fields": [
+    {
+      "name": "id",
+      "repetition": "REQUIRED",
+      "physical_type": "FIXED_LEN_BYTE_ARRAY",
+      "type_length": 16,
+      "logical_type": "UUID"
+    },
+    { "name": "created_at", "repetition": "REQUIRED", "physical_type": "INT64", "logical_type": "TIMESTAMP_MILLIS" },
+    { "name": "updated_at", "repetition": "OPTIONAL", "physical_type": "INT64", "logical_type": "TIMESTAMP_MILLIS" },
+    { "name": "name", "repetition": "REQUIRED", "physical_type": "BYTE_ARRAY", "logical_type": "UTF8" },
+    { "name": "geometry", "repetition": "REQUIRED", "physical_type": "BYTE_ARRAY", "logical_type": "GEOMETRY" }
+  ],
+  "row_group_size": 8000,
+  "compression": "zstd",
+  "compression_level": 3
+}
+```
+
+## Kart integration
+
+Datasets are attached to their schemas using `kart meta`
+
+```bash
+# Set the schema for a dataset
+kart meta set ${dataset_name} schema ${schema_host}/schema/release=v0.0.1/${dataset_name}.json
+
+# Get the schema for a dataset
+kart meta get ${dataset_name} schema
+```
+
+These schemas are critical in ensuring the dataset structure is aligned to LINZ best practices and that all data published is aligned to the schema itself, The schemas also store valuable metadata about the dataset

docs/storage.structure.md

@@ -27,7 +27,7 @@ To keep the date based versions immutable a full ISO timestamp of the published
 year=2026/date=2026-01-01T12_00_00Z/collection.json
 ```
 
-![Date storage structure](./storage.structure.date.excalidraw.png)
+![Date storage structure](./assets/storage.structure.date.excalidraw.png)
 
 An example structure of a vector date based versioning structure
 
@@ -59,7 +59,7 @@ example-bucket:
 
 For components that have a fixed release process such as semver or a quarterly release, it is recommended to store in a similar structure as date based versioning with the addition of `next/` to store the next development release.
 
-![Release storage structure](./storage.structure.release.excalidraw.png)
+![Release storage structure](./assets/storage.structure.release.excalidraw.png)
 
 ```yaml
 example-bucket: