long-black turns Australia's public business-entity data into a single flat file of pre-joined records — one NDJSON document per ABN. It spins up an ephemeral Postgres, streams in the ABR ABN Bulk Extract (~20M ABNs), joins it with nineteen other public registers — ASIC Company, ASIC Business Names, ACNC charity + AIS financials, ASIC AFS & credit licence, ASIC banned & disqualified, AusTender government-contract spend, GrantConnect grant awards, ATO corporate-tax-transparency + R&D-incentive data, ASIC AFS + credit authorised representatives, WGEA gender-equality reporters, ASIC SMSF auditors, National Pollutant Inventory (per-ABN ANZSIC industry + facilities), the Aged Care provider register (registration + enforcement), NGER corporate emissions, and APRA authorised deposit-taking-institution financials — flattens to one document per ABN, and writes per-state gzipped NDJSON. Then it dies. Postgres is a build tool, not infrastructure.
The pipeline spine lives in crema — a shared streaming-pipeline core
extracted from its sister project flat-white (addresses).
long-black is the thin ABN domain layer on top.
A real document from the live v2026.07.03 release — Orora Limited, a listed packaging
manufacturer. Every value below is verbatim from the dataset; only the repetitive
business/other-name arrays are trimmed (marked …). No single ABN populates all twenty
sources — a bank isn't a charity isn't a polluter — so the fields Orora doesn't have
(apra for banks, charity/agedCare for care providers, the ASIC licences for financial
firms) are null here and populate for those entity types. Full field reference:
docs/DOCUMENT-SCHEMA.md (schema 0.22.0, 20 sources).
npm install
docker compose up db -d # ephemeral postgres:16
./scripts/build-fixture-only.sh # seed → flatten → verify → byte-for-byte (<30s)A complete build loads all 19 enrichment sources and fails fast if any is incomplete
(the coverage gate — the data must be complete before shipping). One source, GrantConnect,
needs a free account, so export its credentials first; a data.gov.au-only build can skip them
with ALLOW_PARTIAL=true, which ships degraded data and disables the coverage gate.
export GRANTCONNECT_USERNAME=you@example.com GRANTCONNECT_PASSWORD=… # for the grants source
LONG_BLACK_VERSION=2026.06.25 ./scripts/build-local.sh
# download → load (saxes XML → COPY) → finalize → enrich (19 sources + coverage gate)
# → flatten → verify → per-state .ndjson.gz + metadata.json
# …or a partial build without GrantConnect creds (degraded — for local iteration only):
ALLOW_PARTIAL=true LONG_BLACK_VERSION=2026.06.25 ./scripts/build-local.shdownload (ABR ZIPs) → load (saxes XML → COPY) → finalize (PK + indexes)
→ enrich (19 registers: data.gov.au CSV/XLSX + AusTender + GrantConnect
+ agency-direct XLSX (ACQSC/CER/APRA) → COPY, then coverage gate)
→ flatten (Postgres → NDJSON) → verify (schema + ABN mod-89)
→ split per state → gzip → metadata.json
Everything except load, enrich, compose, the schema, and the SQL comes from
crema.
Each weekly GitHub Release
ships the per-state long-black-<version>-<state>.ndjson.gz shards, metadata.json
(per-state counts + CC-BY attribution), and manifest.json (per-shard sha256 +
record counts + build provenance). A browsable
release catalogue is generated and
published to GitHub Pages after each build. Releases are also mirrored to a shared
S3 bucket (data/abn/<version>/ + manifests/abn-<version>.json) when configured,
where a consolidated all-ABN all.ndjson.gz is published alongside the shards —
see docs/RELEASING.md.
# grab one state's file from the latest release:
gh release download --repo jbejenar/long-black --pattern 'long-black-*-nsw.ndjson.gz'Twenty sources: ABR ABN Bulk Extract + ASIC Company + ASIC Business Names + ACNC
Charities + ACNC Annual Information Statement + ASIC AFS Licensees + ASIC Credit
Licensees + ASIC Banned & Disqualified Orgs (all data.gov.au) + AusTender contracts
(OCDS, via the OCP Data Registry) + GrantConnect Grant Awards + ATO Corporate Tax
Transparency + ATO R&D Tax Incentive + ASIC AFS Authorised Representatives + ASIC Credit
Representatives + WGEA Reporting Organisations + ASIC SMSF Auditors + National Pollutant
Inventory + Aged Care Provider Register (ACQSC) + NGER Corporate Emissions (Clean Energy
Regulator) + APRA ADI Centralised Publication — all CC-BY (mostly 3.0 AU; the ATO R&D
dataset is CC-BY 2.5 AU; NPI, Aged Care, NGER + APRA are CC-BY 4.0), joined on the ABN
(the banned register on the ACN; several sources fall back to the ACN via asic_number).
See docs/DATA-SOURCES.md for the per-source licence + attribution.
| Layer | Choice |
|---|---|
| Database | PostgreSQL 16 (ephemeral, no PostGIS) |
| Loader | saxes streaming XML (ABR) + COPY CSV/TSV (ASIC/ACNC) + exceljs XLSX + direct-URL fetch (APRA/NGER/Aged Care/GrantConnect) |
| Pipeline core | crema (flatten, split, compress, verify, download, metadata, manifest, catalogue) |
| Language | Node 22 / TypeScript (ESM, strict) |
| Output | NDJSON (.ndjson.gz) — per-state on the Release, full all.ndjson.gz on S3 |
Code: Apache-2.0. Derived data: CC-BY per source — mostly CC-BY 3.0 AU, with the
ATO R&D Tax Incentive dataset under CC-BY 2.5 AU and the National Pollutant Inventory, Aged
Care Provider Register, NGER corporate emissions + APRA ADI statistics under CC-BY 4.0. Each
source's exact licence + attribution is recorded per-dataset in metadata.json (sources[])
and docs/DATA-SOURCES.md.
{ "_id": "55004275165", "_version": "2026.07.03", "abnStatus": "ACT", "abnStatusFromDate": "2000-04-11", "entityName": "ORORA LIMITED", "entityTypeCode": "PUB", "entityTypeText": "Australian Public Company", "givenName": null, "familyName": null, "acn": "004275165", "acnType": null, "gstRegistered": true, "gstStatusFromDate": "2000-07-01", "recordLastUpdated": "2026-04-25", "state": "VIC", "postcode": "3122", "businessNames": ["Orora Australasia", "Orora Beverage Cans", "Orora Glass", "Saverglass", "…"], "tradingNames": [], "otherNames": ["AMCOR AUSTRALASIA", "AMCOR GLASS AUSTRALASIA", "…"], "dgr": [], "registeredBusinessNames": [ { "name": "Orora Australasia", "status": "Registered", "registrationDate": "2026-04-24", "cancellationDate": null, }, { "name": "Saverglass", "status": "Registered", "registrationDate": "2026-04-24", "cancellationDate": null, }, // …10 more ], "company": { "acn": "004275165", "name": "ORORA LIMITED", "currentName": null, "type": "APUB", "class": "LMSH", "subClass": "LIST", "status": "Registered", "registrationDate": "1949-11-07", "deregistrationDate": null, "previousState": "VIC", "stateRegistrationNumber": "C0026894N", }, "charity": null, "financialServicesLicence": null, "creditLicence": null, "bannedDisqualified": [], "govSpend": { "totalValueAud": 188210, "contractCount": 3, "firstContractDate": "2015-01-04", "lastContractDate": "2015-11-06", }, "govGrants": { "totalValueAud": 13777500, "grantCount": 2, "firstGrantDate": "2021-05-20", "lastGrantDate": "2023-01-19", }, "taxTransparency": { "incomeYear": "2023-24", "totalIncome": 1025591959, "taxableIncome": 51636878, "taxPayable": null, }, "rdTaxIncentive": { "incomeYear": "2022-23", "totalRdExpenditure": 23174468 }, "afsAuthorisedRep": { "number": "000333362", "licenceNumber": "000222444", "status": "Current", "startDate": "2009-01-07", "endDate": null, }, "creditRep": null, "wgeaReporter": { "primaryAbn": "55004275165", "primaryOrganisation": "Orora Limited" }, "smsfAuditor": null, "npi": { "anzsicCode": "2239", "anzsicName": "Other Metal Container Manufacturing", "facilityCount": 20, "latestReportYear": "2024/2025", }, "agedCare": null, "nger": { "scope1Emissions": 147960.03, "scope2Emissions": 72187.06, "netEnergyGj": 3018717.74, "reportYear": "2024-25", }, "apra": null, "ageYears": 26, "isActive": true, "flags": { "isIndividual": false, "isCompany": true, "isCharity": false, "isLicensed": false, "hasEnforcementAction": false, "isDgr": false, "hasGovContracts": true, "receivesGovGrants": true, "isLargeCorporateTaxpayer": true, "claimsRdTaxIncentive": true, "isAfsAuthorisedRep": true, "isCreditRep": false, "isWgeaReporter": true, "isSmsfAuditor": false, "isExternalAdministration": false, "isStrikeOffInProgress": false, "isDeregistered": false, "isNpiReporter": true, "isAgedCareProvider": false, "isNgerReporter": true, "isApraRegulatedAdi": false, }, }