diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 4cdb733..aba3484 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -15,7 +15,7 @@ pnpm build ## HTTP runtime tests -`pnpm test:http` runs the suites in `api/tests/http` against the activity, badge-definition, badge-query, acknowledgement, collection, context attachment and evaluation, contributor-information, funding, location, profile, organization, feature, work-scope-tag, vocabulary-tag, graph, and contribution query XRPC endpoints installed from this checkout. These tests exercise real HTTP behavior against PostgreSQL, not only Lua handlers with a fake database. +`pnpm test:http` runs the suites in `api/tests/http` against the activity, badge-definition, badge-query, acknowledgement, collection, context-measurement, context attachment and evaluation, contributor-information, funding, location, profile, organization, feature, work-scope-tag, vocabulary-tag, graph, and contribution query XRPC endpoints installed from this checkout. These tests exercise real HTTP behavior against PostgreSQL, not only Lua handlers with a fake database. The local runner requires a local Docker Compose daemon, `psql`, and the pinned PostgreSQL and HappyView images already cached locally. Set `PSQL_PATH` to the absolute path of a trusted `psql` executable: @@ -36,6 +36,7 @@ The HTTP gate fails if it discovers no suites, executes no `node:test` cases, or - Feature-query coverage in `api/tests/http/features.http.test.js` exercises exact retrieval and hydrated or absent author sidecars, author/type and organization-presence filters, bidirectional `createdAt`/URI pagination ties, and named request errors. - Contributor-information retrieval by exact AT-URI and listing with repeated-author filters, cursor pagination, hydrated and missing author sidecars, and named runtime errors. - Work-scope-tag exact-URI retrieval, repeated-author filtering with an empty-result case, hydrated and null publisher sidecars, middle-position `indexedAt` fallback, tied pagination in both directions, and named error responses. +- Context-measurement exact retrieval and publisher-sidecar hydration, repeated author/subject filters and negative results, timestamp/URI tie pagination in both directions, missing sidecars, and named runtime errors. - Acknowledgement exact retrieval and full-record preservation, hydrated and absent publisher sidecars, repeated author/subject filters, ascending and descending pagination across timestamp ties, and named errors. - Badge-query baseline-aware definition feeds and discriminating filters, exact-version award/response lookups, recipient status, raw response history, bidirectional tied pagination, nullable sidecars, and named runtime errors. - Profile and organization queries across all four endpoints for each record type, including batch null results, profile-sidecar hydration, filters, `createdAt`/URI pagination ties, and named errors. @@ -47,7 +48,7 @@ The HTTP gate fails if it discovers no suites, executes no `node:test` cases, or - Contribution exact-record retrieval, repeated publisher filters, tied ascending/descending cursor pagination, nullable publisher sidecars, and named errors. - Badge and contribution fixtures with CBOR-derived record CIDs; contribution DIDs are distinct from baseline fixture identities. - Location retrieval with nullable sidecars, repeated author/URI/location-type filters, unsupported-search errors, tied pagination in both directions, malformed/absent `createdAt` handling, and named errors. -- HTTP fixtures for activity, collection, funding, badge-definition, badge-query, acknowledgement, feature, work-scope-tag, contribution, contributor-information, actor, context attachment and evaluation, graph, location, and vocabulary-tag records use CBOR-derived CIDs. +- HTTP fixtures for activity, collection, funding, badge-definition, badge-query, acknowledgement, feature, work-scope-tag, contribution, contributor-information, actor, context-measurement, context attachment and evaluation, graph, location, measurement, and vocabulary-tag records use CBOR-derived CIDs. For the pinned HappyView release, ordinary Lua `error()` exceptions return HTTP 500 JSON with `error: "script_error"` and `errorType: "runtime"`; the error name appears in `message`. Negative HTTP tests assert this observed behavior. It is not a statement of the ideal public HTTP status contract, and does not guarantee 4xx mapping for `RecordNotFound` or `InvalidRequest`. diff --git a/README.md b/README.md index 76ab522..24f6693 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Hypercerts API workspace -This repository combines the shared HappyView installer, pinned Lexicon dependencies, reusable Lua projections, fixtures, offline checks, and independently owned capability modules. This composed checkout includes public query modules for actor profiles and organizations, activity, badge definitions and queries, collections, context attachments and evaluations, actor and entity follows, recent follows, funding receipts, locations, features, work-scope tags, contribution records, contributor information, vocabulary tags, and acknowledgements. The `org.hypercerts.vocab.getVocabTag` and `org.hypercerts.vocab.listVocabTags` handlers are bundled and registered through `api/modules/vocab/manifest.json`. +This repository combines the shared HappyView installer, pinned Lexicon dependencies, reusable Lua projections, fixtures, offline checks, and independently owned capability modules. This composed checkout includes public query modules for actor profiles and organizations, activity, badge definitions and queries, collections, context measurements, attachments and evaluations, actor and entity follows, recent follows, funding receipts, locations, features, work-scope tags, contribution records, contributor information, vocabulary tags, and acknowledgements. The `org.hypercerts.vocab.getVocabTag` and `org.hypercerts.vocab.listVocabTags` handlers are bundled and registered through `api/modules/vocab/manifest.json`. This checkout is one of the additive local sibling branches used to compose the API: @@ -43,6 +43,7 @@ pnpm build | Actor organizations | `app.certified.actor.getOrganization`, `app.certified.actor.getOrganizations`, `app.certified.actor.listOrganizations`, `app.certified.actor.searchOrganizations` | Single and batch retrieval, batch null results, profile-sidecar hydration, filters, `createdAt`/URI pagination ties, and named errors | | Activity | `org.hypercerts.claim.getActivity`, `org.hypercerts.claim.listActivities`, `org.hypercerts.claim.searchActivities` | Contributor-sidecar hydration, author/organization/contributor/URI filters, tied timestamp pagination, and literal wildcard search | | Collections | `org.hypercerts.collection.getCollection`, `org.hypercerts.collection.listCollections`, `org.hypercerts.collection.searchCollections`, `org.hypercerts.collection.listCollectionItems` | CBOR-derived CIDs, location/tag projections, author, organization, item and tag filters, title/shortDescription search, tied pagination, and source-order item pagination with exact-version resolution | +| Context measurements | `org.hypercerts.context.getMeasurement`, `org.hypercerts.context.listMeasurements` | Exact retrieval, publisher sidecars, author/subject filters, tied pagination in both directions, and named runtime errors | | Context attachments and evaluations | `org.hypercerts.context.getAttachment`, `org.hypercerts.context.listAttachments`, `org.hypercerts.context.getEvaluation`, `org.hypercerts.context.listEvaluations` | Publisher/evaluator sidecars, list filters, pagination across tied `createdAt` values, and named runtime errors | | Actor follows | `app.certified.graph.getFollow`, `app.certified.graph.listActorFollowers`, `app.certified.graph.listActorFollowing` | Actor lookup and lists, tied-key pagination, and nullable profile/organization sidecars | | Entity follows | `app.certified.graph.getEntityFollow`, `app.certified.graph.listEntityFollowers`, `app.certified.graph.listEntityFollowing` | Entity lookup and lists, tied-key pagination, nullable sidecars, and entity target resolution | diff --git a/api/README.md b/api/README.md index fa626d3..fdc5a94 100644 --- a/api/README.md +++ b/api/README.md @@ -20,7 +20,7 @@ PSQL_PATH="$(command -v psql)" pnpm test:http ## HTTP runtime tests -HTTP suites live in `api/tests/http` and exercise the funding, badge-definition, badge-query, feature, work-scope-tag, contribution, profile, organization, context attachment and evaluation, activity, collection, acknowledgement, vocabulary-tag, graph, and location XRPC endpoints installed from the current checkout. `pnpm test:http` discovers `*.http.test.js` suites and fixture modules named `*.fixture.js`, then creates a random Compose project with loopback-only dynamic ports, PostgreSQL data on tmpfs, and a task-owned default bridge network. Bridge networking permits container egress. HappyView receives loopback placeholder upstream URLs and proxy variables pointing to `127.0.0.1:9`; these are application-level settings, not network-hard egress isolation. The installer and HTTP suites target only the task-owned loopback service. The runner installs this checkout's manifest, seeds shared and HTTP fixtures, runs the suites, and tears down only that generated Compose project and its temporary credentials. Locally, Compose uses `--pull never`; missing cached images fail before service startup. +HTTP suites live in `api/tests/http` and exercise the funding, badge-definition, badge-query, feature, work-scope-tag, contribution, context-measurement, profile, organization, context attachment and evaluation, activity, collection, acknowledgement, vocabulary-tag, graph, and location XRPC endpoints installed from the current checkout. `pnpm test:http` discovers `*.http.test.js` suites and fixture modules named `*.fixture.js`, then creates a random Compose project with loopback-only dynamic ports, PostgreSQL data on tmpfs, and a task-owned default bridge network. Bridge networking permits container egress. HappyView receives loopback placeholder upstream URLs and proxy variables pointing to `127.0.0.1:9`; these are application-level settings, not network-hard egress isolation. The installer and HTTP suites target only the task-owned loopback service. The runner installs this checkout's manifest, seeds shared and HTTP fixtures, runs the suites, and tears down only that generated Compose project and its temporary credentials. Locally, Compose uses `--pull never`; missing cached images fail before service startup. The HTTP gate fails when it discovers zero suites, executes zero `node:test` cases, or runs only skipped cases. These checks cover real HTTP behavior against PostgreSQL, not just Lua handlers with a fake database. Funding coverage exercises record retrieval, repeated filters, and pagination. Badge-definition coverage exercises retrieval with an icon and allowed-issuer list, publisher-sidecar hydration, author and badge-type filters, createdAt/URI pagination ties, and named error responses. Badge-query coverage exercises baseline-aware definition feeds and discriminating filters, exact-version award/response lookups, recipient status, raw response history, bidirectional tied pagination, nullable sidecars, and named runtime errors. Vocabulary-tag coverage exercises exact retrieval, author filters, hydrated and nullable sidecars, tied pagination, and named errors. Acknowledgement coverage exercises exact retrieval, hydrated and absent publisher sidecars, repeated author/subject filters, tied pagination in both directions, and named errors. Feature coverage exercises exact retrieval and author hydration, list filters and sidecars, tied createdAt/URI pagination, and named errors. Contribution coverage exercises exact-record retrieval, repeated publisher filters, tied ascending/descending cursor pagination, nullable publisher sidecars, and named errors. Fixtures use CBOR-derived record CIDs and are seeded only into the task-owned disposable database. @@ -49,6 +49,25 @@ Listing accepts repeated, unbracketed `authors` DID parameters with OR matching The handlers require the PostgreSQL HappyView records backend. The shared module registers the tag record Lexicon for backfill; the workscope-tags module registers both query Lexicons and generated Lua scripts. `pnpm build` refreshes the checked-in handler bundles. Installing assets with `pnpm install:api` contacts a HappyView service; use it only with an explicitly approved target and token. +## Measurement queries + +Both queries are publicly readable and require no authentication: + +- `org.hypercerts.context.getMeasurement` accepts a full measurement AT-URI with a DID authority. It returns `RecordNotFound` when that exact URI is not indexed and `InvalidRequest` for an invalid or wrong-collection URI. +- `org.hypercerts.context.listMeasurements` accepts repeated, unbracketed `authors` and `subjects` query parameters. Each accepts at most 100 values; duplicate values are removed. `limit` defaults to 25 and is bounded from 1 through 100. `sortDirection` defaults to `desc`. + +Example: + +```text +/xrpc/org.hypercerts.context.listMeasurements?authors=did%3Aplc%3Apublisher-a&authors=did%3Aplc%3Apublisher-b&subjects=at%3A%2F%2Fdid%3Aplc%3Aproject%2Forg.hypercerts.claim.activity%2F3jzfcijpj2z2a&limit=25 +``` + +`authors` matches the repository owner (`did`), not `record.measurers`. `subjects` matches a supplied AT-URI against any `record.subjects[].uri`; the subject's CID is ignored. Values within one filter are ORed, while `authors` and `subjects` are ANDed. With neither filter, results are global and include measurements without subjects. + +Results sort by `(createdAt, uri)` in the requested direction. If `createdAt` is missing or invalid, ordering falls back to `indexed_at`, then the stored row creation time; this never rewrites the returned record. A next-page cursor is opaque and tied to `sortDirection`; keep the filter parameters unchanged when continuing pagination. The cursor is omitted when there is no next page. Each result includes its complete original record, including `value` as a numeric string. `indexedAt` is always present and is JSON `null` when the indexed row has SQL `NULL` in `indexed_at`. The publisher's Certified profile and organization sidecar are hydrated; missing sidecars are `null`. Subjects, locations, and measurers remain unexpanded. Database and hydration failures propagate as operational errors rather than being converted into missing records. + +The Lua handlers query `happyview_records` on PostgreSQL. Exact lookup binds the full AT-URI; listing uses repository DID matching, JSONB array expansion for subject URI matching, and keyset pagination over `(createdAt, uri)`. Actor hydration uses the shared `actorView.lua` projection. The `measurementView` Lexicon is owned by `getMeasurement` and reused by `listMeasurements`. Safe fallback validation uses PostgreSQL `pg_input_is_valid`, so the listing endpoint requires PostgreSQL 16 or newer; the target runtime version was not verified in this checkout. + ## Acknowledgement queries The public `org.hypercerts.context.getAcknowledgement` query accepts one exact acknowledgement AT-URI with a DID authority. It returns `RecordNotFound` when the acknowledgement is not indexed. `org.hypercerts.context.listAcknowledgements` lists indexed acknowledgements globally or with repeated, unbracketed `authors` (DIDs) and `subjects` (AT-URIs) parameters. Values within either filter use OR; the filters combine with AND. Subject matching compares only `record.subject.uri`, not its CID. diff --git a/api/lexicons/org.hypercerts.context.getMeasurement.json b/api/lexicons/org.hypercerts.context.getMeasurement.json new file mode 100644 index 0000000..4a634d8 --- /dev/null +++ b/api/lexicons/org.hypercerts.context.getMeasurement.json @@ -0,0 +1,62 @@ +{ + "lexicon": 1, + "id": "org.hypercerts.context.getMeasurement", + "defs": { + "main": { + "type": "query", + "description": "Gets an indexed measurement by exact AT-URI and hydrates its publisher. Authentication is not required.", + "parameters": { + "type": "params", + "required": ["uri"], + "properties": { + "uri": { + "type": "string", + "format": "at-uri", + "description": "Full measurement record AT-URI using a DID authority." + } + } + }, + "output": { + "encoding": "application/json", + "schema": { + "type": "ref", + "ref": "#output" + } + }, + "errors": [ + { + "name": "InvalidRequest", + "description": "The URI is invalid or is not a measurement record AT-URI." + }, + { + "name": "RecordNotFound", + "description": "No indexed measurement exists at this AT-URI." + } + ] + }, + "measurementView": { + "type": "object", + "description": "Indexed measurement with its publisher actor and original record.", + "required": ["uri", "cid", "indexedAt", "did", "author", "record"], + "nullable": ["indexedAt"], + "properties": { + "uri": { "type": "string", "format": "at-uri" }, + "cid": { "type": "string", "format": "cid" }, + "indexedAt": { "type": "string", "format": "datetime" }, + "did": { "type": "string", "format": "did" }, + "author": { "type": "ref", "ref": "org.hypercerts.api.defs#actorView" }, + "record": { "type": "ref", "ref": "org.hypercerts.context.measurement" } + } + }, + "output": { + "type": "object", + "required": ["measurement"], + "properties": { + "measurement": { + "type": "ref", + "ref": "#measurementView" + } + } + } + } +} diff --git a/api/lexicons/org.hypercerts.context.listMeasurements.json b/api/lexicons/org.hypercerts.context.listMeasurements.json new file mode 100644 index 0000000..afdefcd --- /dev/null +++ b/api/lexicons/org.hypercerts.context.listMeasurements.json @@ -0,0 +1,76 @@ +{ + "lexicon": 1, + "id": "org.hypercerts.context.listMeasurements", + "defs": { + "main": { + "type": "query", + "description": "Lists measurements with optional publisher and subject filters. Authentication is not required.", + "parameters": { + "type": "params", + "properties": { + "authors": { + "type": "array", + "maxLength": 100, + "description": "Publisher repository DIDs, not record.measurers.", + "items": { "type": "string", "format": "did" } + }, + "subjects": { + "type": "array", + "maxLength": 100, + "description": "Subject AT-URIs matched against record.subjects[].uri, ignoring CID.", + "items": { "type": "string", "format": "at-uri" } + }, + "sortDirection": { + "type": "string", + "enum": ["asc", "desc"], + "default": "desc", + "description": "Sort by createdAt and URI; default desc." + }, + "limit": { + "type": "integer", + "minimum": 1, + "maximum": 100, + "default": 25, + "description": "Maximum page size; default 25." + }, + "cursor": { + "type": "string", + "maxLength": 8192, + "description": "Opaque cursor tied to sortDirection; retain other filters between pages." + } + } + }, + "output": { + "encoding": "application/json", + "schema": { + "type": "ref", + "ref": "#output" + } + }, + "errors": [ + { + "name": "InvalidRequest", + "description": "A filter value, sort direction, page bound, cursor, repeated scalar, or query parameter is invalid." + } + ] + }, + "output": { + "type": "object", + "required": ["measurements"], + "properties": { + "measurements": { + "type": "array", + "items": { + "type": "ref", + "ref": "org.hypercerts.context.getMeasurement#measurementView" + } + }, + "cursor": { + "type": "string", + "maxLength": 8192, + "description": "Next-page cursor; omitted when no page follows." + } + } + } + } +} diff --git a/api/lua/endpoints/getMeasurement.lua b/api/lua/endpoints/getMeasurement.lua new file mode 100644 index 0000000..835b0a3 --- /dev/null +++ b/api/lua/endpoints/getMeasurement.lua @@ -0,0 +1,157 @@ +local function invalid(message) + error("InvalidRequest: " .. message, 0) +end + +local function keys_only(values, allowed, unknown_message_prefix) + for key in pairs(values) do + if not allowed[key] then + invalid(unknown_message_prefix and (unknown_message_prefix .. key) or "unknown query parameter") + end + end +end + +local function scalar(params, key) + local value = params[key] + if value == nil then return nil end + if type(value) ~= "string" and type(value) ~= "number" then + invalid(key .. " must occur once") + end + return tostring(value) +end + +local function valid_did(value) + if type(value) ~= "string" or #value > 2048 then return false end + local method, specific = value:match("^did:([a-z]+):(.+)$") + if not method or not specific or specific:sub(-1) == ":" or specific:sub(-1) == "%" + or value:find("[^%w%.:_%%%-]") then return false end + return true +end + +local function valid_record_key(value) + return #value >= 1 and #value <= 512 and value ~= "." and value ~= ".." + and not value:find("[^%w_~%.:%-]") +end + +local function valid_record_uri(value) + if type(value) ~= "string" or value:find("[?#]") then return false end + local authority, collection, rkey = value:match("^at://([^/]+)/([^/]+)/([^/]+)$") + if not authority or not valid_did(authority) or not valid_record_key(rkey) then return false end + return true, collection, authority +end + +local NULL = json.decode("null") + +local function record_view(row) + return { + uri = row.uri, + cid = row.cid, + indexedAt = row.indexed_at == nil and NULL or row.indexed_at, + did = row.did, + record = json.decode(row.record), + } +end + +local PROFILE = "app.certified.actor.profile" +local ORGANIZATION = "app.certified.actor.organization" + +local function hydrate_actor_views(actors, run_query) + if #actors == 0 then return end + local dids, seen = {}, {} + for _, actor in ipairs(actors) do + if not seen[actor.did] then + seen[actor.did] = true + dids[#dids + 1] = actor.did + end + end + local profiles, organizations = {}, {} + local function load(collection, target) + local params, marks = { collection }, {} + for _, did in ipairs(dids) do + params[#params + 1] = did + marks[#marks + 1] = "$" .. #params + end + local rows = run_query("SELECT uri, did, cid, indexed_at::text AS indexed_at, record::text AS record FROM happyview_records WHERE collection = $1 AND rkey = 'self' AND did IN (" .. table.concat(marks, ",") .. ")", params) + for _, row in ipairs(rows) do target[row.did] = row end + end + load(PROFILE, profiles) + load(ORGANIZATION, organizations) + for _, actor in ipairs(actors) do + actor.profile = profiles[actor.did] and record_view(profiles[actor.did]) or NULL + actor.organization = organizations[actor.did] and record_view(organizations[actor.did]) or NULL + end +end + +local MEASUREMENT = "org.hypercerts.context.measurement" +local MEASUREMENT_NULL = json.decode("null") + +local function measurement_valid_did(value) + if not valid_did(value) then return false end + local position = 1 + while true do + local percent = value:find("%", position, true) + if not percent then return true end + local hex = value:sub(percent + 1, percent + 2) + if #hex ~= 2 or hex:find("[^0-9A-Fa-f]") then return false end + position = percent + 3 + end +end + +local function measurement_valid_record_uri(value) + local valid, collection, authority = valid_record_uri(value) + if not valid then return false end + return measurement_valid_did(authority), collection +end + +local function measurement_query(sql, values) + if db.backend() ~= "postgres" then + error("MeasurementQueryFailed: measurement queries require PostgreSQL", 0) + end + local rows = db.raw(sql, values) + if type(rows) ~= "table" then + error("MeasurementQueryFailed: database returned an invalid result", 0) + end + return rows +end + +local function measurement_view(row) + local view = record_view(row) + if row.indexed_at == nil then view.indexedAt = MEASUREMENT_NULL end + view.author = { did = row.did } + return view +end + +local function omit_null_sidecar_indexed_at(sidecar) + if type(sidecar) == "table" and sidecar.indexedAt == MEASUREMENT_NULL then + sidecar.indexedAt = nil + end +end + +local function hydrate_measurement_views(views) + local authors = {} + for _, view in ipairs(views) do authors[#authors + 1] = view.author end + hydrate_actor_views(authors, measurement_query) + -- Measurement sidecars historically omitted SQL-NULL timestamps; top-level views retain JSON null. + for _, author in ipairs(authors) do + omit_null_sidecar_indexed_at(author.profile) + omit_null_sidecar_indexed_at(author.organization) + end +end + +function handle() + keys_only(params, { uri = true }) + local uri = scalar(params, "uri") + local valid, collection = measurement_valid_record_uri(uri) + if not uri or not valid or collection ~= MEASUREMENT then + invalid("uri must be a full " .. MEASUREMENT .. " AT-URI with a DID authority") + end + + local rows = measurement_query( + "SELECT uri, did, cid, indexed_at::text AS indexed_at, record::text AS record " .. + "FROM happyview_records WHERE collection = $1 AND uri = $2 LIMIT 1", + { MEASUREMENT, uri }) + if #rows == 0 then error("RecordNotFound: measurement record is not indexed", 0) end + + local view = measurement_view(rows[1]) + hydrate_measurement_views({ view }) + return { measurement = view } +end diff --git a/api/lua/endpoints/listMeasurements.lua b/api/lua/endpoints/listMeasurements.lua new file mode 100644 index 0000000..c59f7d3 --- /dev/null +++ b/api/lua/endpoints/listMeasurements.lua @@ -0,0 +1,361 @@ +local function invalid(message) + error("InvalidRequest: " .. message, 0) +end + +local function keys_only(values, allowed, unknown_message_prefix) + for key in pairs(values) do + if not allowed[key] then + invalid(unknown_message_prefix and (unknown_message_prefix .. key) or "unknown query parameter") + end + end +end + +local function scalar(params, key) + local value = params[key] + if value == nil then return nil end + if type(value) ~= "string" and type(value) ~= "number" then + invalid(key .. " must occur once") + end + return tostring(value) +end + +local function valid_did(value) + if type(value) ~= "string" or #value > 2048 then return false end + local method, specific = value:match("^did:([a-z]+):(.+)$") + if not method or not specific or specific:sub(-1) == ":" or specific:sub(-1) == "%" + or value:find("[^%w%.:_%%%-]") then return false end + return true +end + +local function valid_record_key(value) + return #value >= 1 and #value <= 512 and value ~= "." and value ~= ".." + and not value:find("[^%w_~%.:%-]") +end + +local function valid_record_uri(value) + if type(value) ~= "string" or value:find("[?#]") then return false end + local authority, collection, rkey = value:match("^at://([^/]+)/([^/]+)/([^/]+)$") + if not authority or not valid_did(authority) or not valid_record_key(rkey) then return false end + return true, collection, authority +end + +local function valid_datetime(value) + if type(value) ~= "string" then return false end + local year, month, day, hour, minute, second, suffix = value:match( + "^(%d%d%d%d)%-(%d%d)%-(%d%d)T(%d%d):(%d%d):(%d%d)(.*)$") + if not year then return false end + year, month, day = tonumber(year), tonumber(month), tonumber(day) + hour, minute, second = tonumber(hour), tonumber(minute), tonumber(second) + if month < 1 or month > 12 or hour > 23 or minute > 59 or second > 59 then return false end + local leap = year % 4 == 0 and (year % 100 ~= 0 or year % 400 == 0) + local month_days = { 31, leap and 29 or 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31 } + if day < 1 or day > month_days[month] then return false end + local fraction, zone = suffix:match("^(%.%d+)(Z)$") + if not fraction then fraction, zone = suffix:match("^(%.%d+)([+-]%d%d:%d%d)$") end + if not fraction then zone = suffix:match("^(Z)$") end + if not zone then zone = suffix:match("^([+-]%d%d:%d%d)$") end + if not zone or zone == "-00:00" then return false end + if zone ~= "Z" then + local zh, zm = zone:match("^[+-](%d%d):(%d%d)$") + if not zh or tonumber(zh) > 23 or tonumber(zm) > 59 then return false end + end + return true +end + +local function parse_list_limit(params) + local limit_value = scalar(params, "limit") + if limit_value and not limit_value:match("^%d+$") then invalid("limit must be an integer from 1 through 100") end + local limit = limit_value and tonumber(limit_value) or 25 + if not limit or limit % 1 ~= 0 or limit < 1 or limit > 100 then invalid("limit must be an integer from 1 through 100") end + return limit +end + +local function parse_sort_direction(params) + local direction = scalar(params, "sortDirection") or "desc" + if direction ~= "asc" and direction ~= "desc" then invalid("sortDirection must be 'asc' or 'desc'") end + return direction +end + +local function cursor_encode(value) + local encoded = json.encode(value) + return (encoded:gsub(".", function(char) return string.format("%02x", string.byte(char)) end)) +end + +local NULL = json.decode("null") + +local function record_view(row) + return { + uri = row.uri, + cid = row.cid, + indexedAt = row.indexed_at == nil and NULL or row.indexed_at, + did = row.did, + record = json.decode(row.record), + } +end + +local PROFILE = "app.certified.actor.profile" +local ORGANIZATION = "app.certified.actor.organization" + +local function hydrate_actor_views(actors, run_query) + if #actors == 0 then return end + local dids, seen = {}, {} + for _, actor in ipairs(actors) do + if not seen[actor.did] then + seen[actor.did] = true + dids[#dids + 1] = actor.did + end + end + local profiles, organizations = {}, {} + local function load(collection, target) + local params, marks = { collection }, {} + for _, did in ipairs(dids) do + params[#params + 1] = did + marks[#marks + 1] = "$" .. #params + end + local rows = run_query("SELECT uri, did, cid, indexed_at::text AS indexed_at, record::text AS record FROM happyview_records WHERE collection = $1 AND rkey = 'self' AND did IN (" .. table.concat(marks, ",") .. ")", params) + for _, row in ipairs(rows) do target[row.did] = row end + end + load(PROFILE, profiles) + load(ORGANIZATION, organizations) + for _, actor in ipairs(actors) do + actor.profile = profiles[actor.did] and record_view(profiles[actor.did]) or NULL + actor.organization = organizations[actor.did] and record_view(organizations[actor.did]) or NULL + end +end + +local MEASUREMENT = "org.hypercerts.context.measurement" +local MEASUREMENT_NULL = json.decode("null") + +local function measurement_valid_did(value) + if not valid_did(value) then return false end + local position = 1 + while true do + local percent = value:find("%", position, true) + if not percent then return true end + local hex = value:sub(percent + 1, percent + 2) + if #hex ~= 2 or hex:find("[^0-9A-Fa-f]") then return false end + position = percent + 3 + end +end + +local function measurement_valid_record_uri(value) + local valid, collection, authority = valid_record_uri(value) + if not valid then return false end + return measurement_valid_did(authority), collection +end + +local function measurement_query(sql, values) + if db.backend() ~= "postgres" then + error("MeasurementQueryFailed: measurement queries require PostgreSQL", 0) + end + local rows = db.raw(sql, values) + if type(rows) ~= "table" then + error("MeasurementQueryFailed: database returned an invalid result", 0) + end + return rows +end + +local function measurement_view(row) + local view = record_view(row) + if row.indexed_at == nil then view.indexedAt = MEASUREMENT_NULL end + view.author = { did = row.did } + return view +end + +local function omit_null_sidecar_indexed_at(sidecar) + if type(sidecar) == "table" and sidecar.indexedAt == MEASUREMENT_NULL then + sidecar.indexedAt = nil + end +end + +local function hydrate_measurement_views(views) + local authors = {} + for _, view in ipairs(views) do authors[#authors + 1] = view.author end + hydrate_actor_views(authors, measurement_query) + -- Measurement sidecars historically omitted SQL-NULL timestamps; top-level views retain JSON null. + for _, author in ipairs(authors) do + omit_null_sidecar_indexed_at(author.profile) + omit_null_sidecar_indexed_at(author.organization) + end +end + +local function valid_nsid(value) + if type(value) ~= "string" or #value > 317 then return false end + local segments = {} + for segment in value:gmatch("[^%.]+") do segments[#segments + 1] = segment end + if #segments < 3 or table.concat(segments, ".") ~= value then return false end + + for index = 1, #segments - 1 do + local segment = segments[index] + if #segment > 63 or segment:find("[^A-Za-z0-9%-]") then return false end + local first, last = segment:sub(1, 1), segment:sub(-1) + if not first:match(index == 1 and "^[A-Za-z]$" or "^[A-Za-z0-9]$") + or not last:match("^[A-Za-z0-9]$") then + return false + end + end + + local name = segments[#segments] + return #name <= 63 and name:match("^[A-Za-z][A-Za-z0-9]*$") ~= nil +end + +local function measurement_array(key, format) + local value = params[key] + if value == nil then return nil end + + local supplied = {} + if type(value) == "string" then + supplied[1] = value + elseif type(value) == "table" then + local count = 0 + for index in pairs(value) do + if type(index) ~= "number" or index < 1 or index % 1 ~= 0 then + invalid(key .. " must use repeated query values") + end + count = count + 1 + end + if count ~= #value then invalid(key .. " must use repeated query values") end + for index = 1, count do + if type(value[index]) ~= "string" then invalid(key .. " entries must be strings") end + supplied[#supplied + 1] = value[index] + end + else + invalid(key .. " must be a string or repeated string parameter") + end + if #supplied > 100 then invalid(key .. " accepts at most 100 values") end + + local unique, seen = {}, {} + for _, item in ipairs(supplied) do + if format == "did" and not measurement_valid_did(item) then + invalid("each " .. key .. " value must be a valid DID") + elseif format == "at-uri" then + local valid, collection = measurement_valid_record_uri(item) + if not valid or not valid_nsid(collection) then + invalid("each " .. key .. " value must be a full AT-URI with a DID authority and valid collection NSID") + end + end + if not seen[item] then + seen[item] = true + unique[#unique + 1] = item + end + end + return unique +end + +local function bind_values(values, items) + local placeholders = {} + for _, item in ipairs(items) do + values[#values + 1] = item + placeholders[#placeholders + 1] = "$" .. #values + end + return placeholders +end + +local function decode_measurement_cursor(token, direction) + if token == nil then return nil end + if #token == 0 or #token > 8192 or #token % 2 ~= 0 or token:find("[^0-9a-f]") then + invalid("cursor is malformed") + end + local decoded = token:gsub("..", function(pair) return string.char(tonumber(pair, 16)) end) + local ok, value = pcall(json.decode, decoded) + if not ok or type(value) ~= "table" or value.v ~= 1 or value.d ~= direction + or type(value.t) ~= "string" or type(value.u) ~= "string" then + invalid("cursor is malformed or belongs to another sortDirection") + end + for key in pairs(value) do + if key ~= "v" and key ~= "d" and key ~= "t" and key ~= "u" then invalid("cursor is malformed") end + end + local valid, collection = measurement_valid_record_uri(value.u) + -- PostgreSQL timestamptz has no year zero, so reject it before binding the cursor. + if value.t:sub(1, 4) == "0000" or not valid_datetime(value.t) or not valid or collection ~= MEASUREMENT then + invalid("cursor is malformed") + end + return value +end + +local function measurement_sort_expression() + local created = "measurement.record::jsonb->>'createdAt'" + local zoned = "^[0-9]{4}-[0-9]{2}-[0-9]{2}T([01][0-9]|2[0-3]):[0-5][0-9]:[0-5][0-9]([.][0-9]+)?(Z|[+-]([01][0-9]|2[0-3]):[0-5][0-9])$" + return "CASE WHEN jsonb_typeof(measurement.record::jsonb->'createdAt') = 'string' AND " .. created .. + " ~ '" .. zoned .. "' AND " .. created .. " !~ '-00:00$' AND pg_input_is_valid(" .. created .. + ", 'timestamptz') THEN (" .. created .. ")::timestamptz ELSE " .. + "COALESCE(measurement.indexed_at::timestamptz, measurement.created_at::timestamptz) END" +end + +local function measurement_list_query(authors, subjects, limit, cursor, direction) + local where, values = { "measurement.collection = $1" }, { MEASUREMENT } + if authors then + if #authors == 0 then + where[#where + 1] = "FALSE" + else + where[#where + 1] = "measurement.did IN (" .. table.concat(bind_values(values, authors), ", ") .. ")" + end + end + if subjects then + if #subjects == 0 then + where[#where + 1] = "FALSE" + else + local source = "CASE WHEN jsonb_typeof(measurement.record::jsonb->'subjects') = 'array' " .. + "THEN measurement.record::jsonb->'subjects' ELSE '[]'::jsonb END" + local placeholders = bind_values(values, subjects) + where[#where + 1] = "EXISTS (SELECT 1 FROM jsonb_array_elements(" .. source .. ") AS subject(value) " .. + "WHERE subject.value->>'uri' IN (" .. table.concat(placeholders, ", ") .. "))" + end + end + + local page_values, page_where = {}, {} + for _, value in ipairs(values) do page_values[#page_values + 1] = value end + for _, clause in ipairs(where) do page_where[#page_where + 1] = clause end + if cursor then + page_values[#page_values + 1] = cursor.t + local timestamp = "$" .. #page_values + page_values[#page_values + 1] = cursor.u + local uri = "$" .. #page_values + local operator = direction == "asc" and ">" or "<" + page_where[#page_where + 1] = "(sorted.sort_at, measurement.uri) " .. operator .. + " ((" .. timestamp .. ")::timestamptz, " .. uri .. ")" + end + + page_values[#page_values + 1] = limit + 1 + local ordering = direction == "asc" and "ASC" or "DESC" + local sql = "SELECT measurement.uri, measurement.did, measurement.cid, " .. + "measurement.indexed_at::text AS indexed_at, measurement.record::text AS record, " .. + "to_char(sorted.sort_at AT TIME ZONE 'UTC', 'YYYY-MM-DD\"T\"HH24:MI:SS.US\"Z\"') AS sort_timestamp " .. + "FROM happyview_records AS measurement CROSS JOIN LATERAL (SELECT " .. + measurement_sort_expression() .. " AS sort_at) AS sorted WHERE " .. table.concat(page_where, " AND ") .. + " ORDER BY sorted.sort_at " .. ordering .. ", measurement.uri " .. ordering .. " LIMIT $" .. #page_values + local rows = measurement_query(sql, page_values) + + local measurements = {} + local included = math.min(#rows, limit) + for index = 1, included do measurements[#measurements + 1] = measurement_view(rows[index]) end + + local next_cursor + if #rows > limit then + local last = rows[limit] + next_cursor = cursor_encode({ v = 1, d = direction, t = last.sort_timestamp, u = last.uri }) + end + hydrate_measurement_views(measurements) + return measurements, next_cursor +end + +function handle() + keys_only(params, { + authors = true, + subjects = true, + sortDirection = true, + limit = true, + cursor = true, + }) + + local authors = measurement_array("authors", "did") + local subjects = measurement_array("subjects", "at-uri") + local direction = parse_sort_direction(params) + local limit = parse_list_limit(params) + local cursor = decode_measurement_cursor(scalar(params, "cursor"), direction) + local measurements, next_cursor = measurement_list_query(authors, subjects, limit, cursor, direction) + + local response = { measurements = toarray(measurements) } + if next_cursor then response.cursor = next_cursor end + return response +end diff --git a/api/lua/shared/measurementQuery.lua b/api/lua/shared/measurementQuery.lua new file mode 100644 index 0000000..01b165d --- /dev/null +++ b/api/lua/shared/measurementQuery.lua @@ -0,0 +1,55 @@ +local MEASUREMENT = "org.hypercerts.context.measurement" +local MEASUREMENT_NULL = json.decode("null") + +local function measurement_valid_did(value) + if not valid_did(value) then return false end + local position = 1 + while true do + local percent = value:find("%", position, true) + if not percent then return true end + local hex = value:sub(percent + 1, percent + 2) + if #hex ~= 2 or hex:find("[^0-9A-Fa-f]") then return false end + position = percent + 3 + end +end + +local function measurement_valid_record_uri(value) + local valid, collection, authority = valid_record_uri(value) + if not valid then return false end + return measurement_valid_did(authority), collection +end + +local function measurement_query(sql, values) + if db.backend() ~= "postgres" then + error("MeasurementQueryFailed: measurement queries require PostgreSQL", 0) + end + local rows = db.raw(sql, values) + if type(rows) ~= "table" then + error("MeasurementQueryFailed: database returned an invalid result", 0) + end + return rows +end + +local function measurement_view(row) + local view = record_view(row) + if row.indexed_at == nil then view.indexedAt = MEASUREMENT_NULL end + view.author = { did = row.did } + return view +end + +local function omit_null_sidecar_indexed_at(sidecar) + if type(sidecar) == "table" and sidecar.indexedAt == MEASUREMENT_NULL then + sidecar.indexedAt = nil + end +end + +local function hydrate_measurement_views(views) + local authors = {} + for _, view in ipairs(views) do authors[#authors + 1] = view.author end + hydrate_actor_views(authors, measurement_query) + -- Measurement sidecars historically omitted SQL-NULL timestamps; top-level views retain JSON null. + for _, author in ipairs(authors) do + omit_null_sidecar_indexed_at(author.profile) + omit_null_sidecar_indexed_at(author.organization) + end +end diff --git a/api/lua/src/getMeasurement.lua b/api/lua/src/getMeasurement.lua new file mode 100644 index 0000000..bee7ec2 --- /dev/null +++ b/api/lua/src/getMeasurement.lua @@ -0,0 +1,18 @@ +function handle() + keys_only(params, { uri = true }) + local uri = scalar(params, "uri") + local valid, collection = measurement_valid_record_uri(uri) + if not uri or not valid or collection ~= MEASUREMENT then + invalid("uri must be a full " .. MEASUREMENT .. " AT-URI with a DID authority") + end + + local rows = measurement_query( + "SELECT uri, did, cid, indexed_at::text AS indexed_at, record::text AS record " .. + "FROM happyview_records WHERE collection = $1 AND uri = $2 LIMIT 1", + { MEASUREMENT, uri }) + if #rows == 0 then error("RecordNotFound: measurement record is not indexed", 0) end + + local view = measurement_view(rows[1]) + hydrate_measurement_views({ view }) + return { measurement = view } +end diff --git a/api/lua/src/listMeasurements.lua b/api/lua/src/listMeasurements.lua new file mode 100644 index 0000000..1d142e7 --- /dev/null +++ b/api/lua/src/listMeasurements.lua @@ -0,0 +1,180 @@ +local function valid_nsid(value) + if type(value) ~= "string" or #value > 317 then return false end + local segments = {} + for segment in value:gmatch("[^%.]+") do segments[#segments + 1] = segment end + if #segments < 3 or table.concat(segments, ".") ~= value then return false end + + for index = 1, #segments - 1 do + local segment = segments[index] + if #segment > 63 or segment:find("[^A-Za-z0-9%-]") then return false end + local first, last = segment:sub(1, 1), segment:sub(-1) + if not first:match(index == 1 and "^[A-Za-z]$" or "^[A-Za-z0-9]$") + or not last:match("^[A-Za-z0-9]$") then + return false + end + end + + local name = segments[#segments] + return #name <= 63 and name:match("^[A-Za-z][A-Za-z0-9]*$") ~= nil +end + +local function measurement_array(key, format) + local value = params[key] + if value == nil then return nil end + + local supplied = {} + if type(value) == "string" then + supplied[1] = value + elseif type(value) == "table" then + local count = 0 + for index in pairs(value) do + if type(index) ~= "number" or index < 1 or index % 1 ~= 0 then + invalid(key .. " must use repeated query values") + end + count = count + 1 + end + if count ~= #value then invalid(key .. " must use repeated query values") end + for index = 1, count do + if type(value[index]) ~= "string" then invalid(key .. " entries must be strings") end + supplied[#supplied + 1] = value[index] + end + else + invalid(key .. " must be a string or repeated string parameter") + end + if #supplied > 100 then invalid(key .. " accepts at most 100 values") end + + local unique, seen = {}, {} + for _, item in ipairs(supplied) do + if format == "did" and not measurement_valid_did(item) then + invalid("each " .. key .. " value must be a valid DID") + elseif format == "at-uri" then + local valid, collection = measurement_valid_record_uri(item) + if not valid or not valid_nsid(collection) then + invalid("each " .. key .. " value must be a full AT-URI with a DID authority and valid collection NSID") + end + end + if not seen[item] then + seen[item] = true + unique[#unique + 1] = item + end + end + return unique +end + +local function bind_values(values, items) + local placeholders = {} + for _, item in ipairs(items) do + values[#values + 1] = item + placeholders[#placeholders + 1] = "$" .. #values + end + return placeholders +end + +local function decode_measurement_cursor(token, direction) + if token == nil then return nil end + if #token == 0 or #token > 8192 or #token % 2 ~= 0 or token:find("[^0-9a-f]") then + invalid("cursor is malformed") + end + local decoded = token:gsub("..", function(pair) return string.char(tonumber(pair, 16)) end) + local ok, value = pcall(json.decode, decoded) + if not ok or type(value) ~= "table" or value.v ~= 1 or value.d ~= direction + or type(value.t) ~= "string" or type(value.u) ~= "string" then + invalid("cursor is malformed or belongs to another sortDirection") + end + for key in pairs(value) do + if key ~= "v" and key ~= "d" and key ~= "t" and key ~= "u" then invalid("cursor is malformed") end + end + local valid, collection = measurement_valid_record_uri(value.u) + -- PostgreSQL timestamptz has no year zero, so reject it before binding the cursor. + if value.t:sub(1, 4) == "0000" or not valid_datetime(value.t) or not valid or collection ~= MEASUREMENT then + invalid("cursor is malformed") + end + return value +end + +local function measurement_sort_expression() + local created = "measurement.record::jsonb->>'createdAt'" + local zoned = "^[0-9]{4}-[0-9]{2}-[0-9]{2}T([01][0-9]|2[0-3]):[0-5][0-9]:[0-5][0-9]([.][0-9]+)?(Z|[+-]([01][0-9]|2[0-3]):[0-5][0-9])$" + return "CASE WHEN jsonb_typeof(measurement.record::jsonb->'createdAt') = 'string' AND " .. created .. + " ~ '" .. zoned .. "' AND " .. created .. " !~ '-00:00$' AND pg_input_is_valid(" .. created .. + ", 'timestamptz') THEN (" .. created .. ")::timestamptz ELSE " .. + "COALESCE(measurement.indexed_at::timestamptz, measurement.created_at::timestamptz) END" +end + +local function measurement_list_query(authors, subjects, limit, cursor, direction) + local where, values = { "measurement.collection = $1" }, { MEASUREMENT } + if authors then + if #authors == 0 then + where[#where + 1] = "FALSE" + else + where[#where + 1] = "measurement.did IN (" .. table.concat(bind_values(values, authors), ", ") .. ")" + end + end + if subjects then + if #subjects == 0 then + where[#where + 1] = "FALSE" + else + local source = "CASE WHEN jsonb_typeof(measurement.record::jsonb->'subjects') = 'array' " .. + "THEN measurement.record::jsonb->'subjects' ELSE '[]'::jsonb END" + local placeholders = bind_values(values, subjects) + where[#where + 1] = "EXISTS (SELECT 1 FROM jsonb_array_elements(" .. source .. ") AS subject(value) " .. + "WHERE subject.value->>'uri' IN (" .. table.concat(placeholders, ", ") .. "))" + end + end + + local page_values, page_where = {}, {} + for _, value in ipairs(values) do page_values[#page_values + 1] = value end + for _, clause in ipairs(where) do page_where[#page_where + 1] = clause end + if cursor then + page_values[#page_values + 1] = cursor.t + local timestamp = "$" .. #page_values + page_values[#page_values + 1] = cursor.u + local uri = "$" .. #page_values + local operator = direction == "asc" and ">" or "<" + page_where[#page_where + 1] = "(sorted.sort_at, measurement.uri) " .. operator .. + " ((" .. timestamp .. ")::timestamptz, " .. uri .. ")" + end + + page_values[#page_values + 1] = limit + 1 + local ordering = direction == "asc" and "ASC" or "DESC" + local sql = "SELECT measurement.uri, measurement.did, measurement.cid, " .. + "measurement.indexed_at::text AS indexed_at, measurement.record::text AS record, " .. + "to_char(sorted.sort_at AT TIME ZONE 'UTC', 'YYYY-MM-DD\"T\"HH24:MI:SS.US\"Z\"') AS sort_timestamp " .. + "FROM happyview_records AS measurement CROSS JOIN LATERAL (SELECT " .. + measurement_sort_expression() .. " AS sort_at) AS sorted WHERE " .. table.concat(page_where, " AND ") .. + " ORDER BY sorted.sort_at " .. ordering .. ", measurement.uri " .. ordering .. " LIMIT $" .. #page_values + local rows = measurement_query(sql, page_values) + + local measurements = {} + local included = math.min(#rows, limit) + for index = 1, included do measurements[#measurements + 1] = measurement_view(rows[index]) end + + local next_cursor + if #rows > limit then + local last = rows[limit] + next_cursor = cursor_encode({ v = 1, d = direction, t = last.sort_timestamp, u = last.uri }) + end + hydrate_measurement_views(measurements) + return measurements, next_cursor +end + +function handle() + keys_only(params, { + authors = true, + subjects = true, + sortDirection = true, + limit = true, + cursor = true, + }) + + local authors = measurement_array("authors", "did") + local subjects = measurement_array("subjects", "at-uri") + local direction = parse_sort_direction(params) + local limit = parse_list_limit(params) + local cursor = decode_measurement_cursor(scalar(params, "cursor"), direction) + local measurements, next_cursor = measurement_list_query(authors, subjects, limit, cursor, direction) + + local response = { measurements = toarray(measurements) } + if next_cursor then response.cursor = next_cursor end + return response +end diff --git a/api/manifest.json b/api/manifest.json index ad911b8..8ca60e0 100644 --- a/api/manifest.json +++ b/api/manifest.json @@ -53,7 +53,9 @@ "getBadgeResponse": "implemented", "listBadgeResponses": "implemented", "getFeature": "implemented", - "listFeatures": "implemented" + "listFeatures": "implemented", + "getMeasurement": "implemented", + "listMeasurements": "implemented" }, "authentication": { "decision": "public", @@ -464,6 +466,18 @@ { "id": "org.hypercerts.entity.listFeatures", "path": "lexicons/org.hypercerts.entity.listFeatures.json" + }, + { + "id": "org.hypercerts.context.measurement", + "packagePath": "lexicons/org/hypercerts/context/measurement.json" + }, + { + "id": "org.hypercerts.context.getMeasurement", + "path": "lexicons/org.hypercerts.context.getMeasurement.json" + }, + { + "id": "org.hypercerts.context.listMeasurements", + "path": "lexicons/org.hypercerts.context.listMeasurements.json" } ], "modules": [ @@ -486,7 +500,8 @@ "modules/contribution/manifest.json", "modules/vocab/manifest.json", "modules/location/manifest.json", - "modules/features/manifest.json" + "modules/features/manifest.json", + "modules/context-measurements/manifest.json" ], "luaBuild": { "endpointSources": [ diff --git a/api/modules/context-measurements/manifest.json b/api/modules/context-measurements/manifest.json new file mode 100644 index 0000000..e4a2b56 --- /dev/null +++ b/api/modules/context-measurements/manifest.json @@ -0,0 +1,69 @@ +{ + "name": "context-measurements", + "assets": [ + { + "kind": "lexicon", + "id": "org.hypercerts.context.getMeasurement", + "path": "../../lexicons/org.hypercerts.context.getMeasurement.json", + "config": { + "backfill": false, + "target_collection": "org.hypercerts.context.measurement" + }, + "dependsOn": [ + "org.hypercerts.context.measurement", + "org.hypercerts.api.defs" + ] + }, + { + "kind": "script", + "id": "xrpc.query:org.hypercerts.context.getMeasurement", + "path": "../../lua/endpoints/getMeasurement.lua", + "sourcePath": "../../lua/src/getMeasurement.lua", + "sharedSourcePaths": [ + "../../lua/shared/query.lua", + "../../lua/shared/didValidation.lua", + "../../lua/shared/recordIdentifier.lua", + "../../lua/shared/recordView.lua", + "../../lua/shared/actorView.lua", + "../../lua/shared/measurementQuery.lua" + ], + "config": { + "script_type": "lua" + }, + "dependsOn": ["org.hypercerts.context.getMeasurement"] + }, + { + "kind": "lexicon", + "id": "org.hypercerts.context.listMeasurements", + "path": "../../lexicons/org.hypercerts.context.listMeasurements.json", + "config": { + "backfill": false, + "target_collection": "org.hypercerts.context.measurement" + }, + "dependsOn": [ + "org.hypercerts.context.measurement", + "org.hypercerts.context.getMeasurement" + ] + }, + { + "kind": "script", + "id": "xrpc.query:org.hypercerts.context.listMeasurements", + "path": "../../lua/endpoints/listMeasurements.lua", + "sourcePath": "../../lua/src/listMeasurements.lua", + "sharedSourcePaths": [ + "../../lua/shared/query.lua", + "../../lua/shared/didValidation.lua", + "../../lua/shared/recordIdentifier.lua", + "../../lua/shared/listValidation.lua", + "../../lua/shared/listQuery.lua", + "../../lua/shared/recordView.lua", + "../../lua/shared/actorView.lua", + "../../lua/shared/measurementQuery.lua" + ], + "config": { + "script_type": "lua" + }, + "dependsOn": ["org.hypercerts.context.listMeasurements"] + } + ] +} diff --git a/api/modules/shared/manifest.json b/api/modules/shared/manifest.json index fa2fb48..5395dde 100644 --- a/api/modules/shared/manifest.json +++ b/api/modules/shared/manifest.json @@ -16,6 +16,7 @@ { "kind": "lexicon", "id": "org.hypercerts.collection", "packagePath": "lexicons/org/hypercerts/collection.json", "config": { "backfill": true }, "dependsOn": ["org.hypercerts.defs", "app.certified.signature.defs"] }, { "kind": "lexicon", "id": "org.hypercerts.context.attachment", "packagePath": "lexicons/org/hypercerts/context/attachment.json", "config": { "backfill": true }, "dependsOn": [] }, { "kind": "lexicon", "id": "org.hypercerts.context.evaluation", "packagePath": "lexicons/org/hypercerts/context/evaluation.json", "config": { "backfill": true }, "dependsOn": ["app.certified.defs", "app.certified.signature.defs", "org.hypercerts.defs"] }, + { "kind": "lexicon", "id": "org.hypercerts.context.measurement", "packagePath": "lexicons/org/hypercerts/context/measurement.json", "config": { "backfill": true }, "dependsOn": ["app.certified.defs", "app.certified.signature.defs", "org.hypercerts.defs"] }, { "kind": "lexicon", "id": "org.hypercerts.defs", "packagePath": "lexicons/org/hypercerts/defs.json", "config": { "backfill": false }, "dependsOn": [], "registrationGroup": "location-record-schema-sources" }, { "kind": "lexicon", "id": "org.hypercerts.entity.feature", "packagePath": "lexicons/org/hypercerts/entity/feature.json", "config": { "backfill": true }, "dependsOn": ["org.hypercerts.defs", "app.certified.signature.defs"] }, { "kind": "lexicon", "id": "org.hypercerts.funding.receipt", "packagePath": "lexicons/org/hypercerts/funding/receipt.json", "config": { "backfill": true }, "dependsOn": [] }, diff --git a/api/tests/http/context-measurements.http.test.js b/api/tests/http/context-measurements.http.test.js new file mode 100644 index 0000000..40dd326 --- /dev/null +++ b/api/tests/http/context-measurements.http.test.js @@ -0,0 +1,139 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { contractUrl, requireContractTarget } from './helpers.js'; + +const endpoint = 'org.hypercerts.context.getMeasurement'; +const listEndpoint = 'org.hypercerts.context.listMeasurements'; +const collection = 'org.hypercerts.context.measurement'; +const publisher = 'did:plc:measurementhttpfixtureaa'; +const publisherB = 'did:plc:measurementhttpfixturebb'; +const measurer = 'did:plc:oooooooooooooooooooooooo'; +const measurementUri = `at://${publisher}/${collection}/3jzfcijpj2z2a`; +const measurementUris = { + a: measurementUri, + b: `at://${publisher}/${collection}/3jzfcijpj2z2b`, + c: `at://${publisher}/${collection}/3jzfcijpj2z2c`, + d: `at://${publisherB}/${collection}/3jzfcijpj2z2d`, +}; +const subjectUri = 'at://did:plc:gggggggggggggggggggggggg/org.hypercerts.claim.activity/3jzfcijpj2z2i'; +const otherSubjectUri = 'at://did:plc:gggggggggggggggggggggggg/org.hypercerts.claim.activity/3jzfcijpj2z2j'; +const absentSubjectUri = 'at://did:plc:gggggggggggggggggggggggg/org.hypercerts.claim.activity/3jzfcijpj2z2k'; +const subjectCid = 'bafyreih2gywmzfterjcd3egdsheuloawhwykojkrxb2tnfsjjuou5nu5va'; + +async function callEndpoint(nsid, params = {}) { + const url = contractUrl(requireContractTarget(), nsid, params); + const response = await fetch(url, { + headers: { accept: 'application/json' }, + signal: AbortSignal.timeout(10_000), + }); + const text = await response.text(); + let body; + try { + body = JSON.parse(text); + } catch { + body = text; + } + return { response, body }; +} + +const getMeasurement = (uri) => callEndpoint(endpoint, { uri }); +const listMeasurements = (params) => callEndpoint(listEndpoint, params); + +test('getMeasurement returns the exact indexed record and hydrates both publisher sidecars', async () => { + const { response, body } = await getMeasurement(measurementUri); + assert.equal(response.status, 200, JSON.stringify(body)); + + const { measurement } = body; + assert.equal(measurement.uri, measurementUri); + assert.equal(measurement.cid, 'bafyreihu2rszmv7k3uhufix4rh2ksbuq7pddhaic7jn6g2s6lbur3dl254'); + assert.equal(measurement.indexedAt, '2025-03-01T00:00:00.000Z'); + assert.equal(measurement.did, publisher); + assert.deepEqual(measurement.record, { + $type: collection, + metric: 'trees planted', + unit: 'tree', + value: '00012.3400', + createdAt: '2025-02-01T00:00:00.000Z', + subjects: [{ uri: subjectUri, cid: subjectCid }], + measurers: [{ did: 'did:plc:oooooooooooooooooooooooo' }], + }); + assert.equal(measurement.author.did, publisher); + assert.equal(measurement.author.profile.record.displayName, 'Measurement Publisher'); + assert.deepEqual(measurement.author.organization.record.organizationType, ['nonprofit']); +}); + +test('listMeasurements applies repeated author and subject filters and excludes nonmatches', async () => { + const { response, body } = await listMeasurements({ + authors: [publisher, publisherB], + subjects: [subjectUri, otherSubjectUri], + sortDirection: 'asc', + }); + assert.equal(response.status, 200, JSON.stringify(body)); + assert.deepEqual(body.measurements.map(({ uri }) => uri), [measurementUris.a, measurementUris.b, measurementUris.d]); + assert.equal(body.measurements[0].record.metric, 'trees planted'); + assert.equal(body.measurements[2].author.profile, null); + assert.equal(body.measurements[2].author.organization, null); + + const byMeasurerOnly = await listMeasurements({ authors: [measurer] }); + assert.equal(byMeasurerOnly.response.status, 200, JSON.stringify(byMeasurerOnly.body)); + assert.deepEqual(byMeasurerOnly.body.measurements, []); + + const byAbsentSubject = await listMeasurements({ subjects: [absentSubjectUri] }); + assert.equal(byAbsentSubject.response.status, 200, JSON.stringify(byAbsentSubject.body)); + assert.deepEqual(byAbsentSubject.body.measurements, []); +}); + +test('listMeasurements paginates timestamp ties by URI in both directions', async () => { + for (const [sortDirection, firstPage, secondPage] of [ + ['asc', [measurementUris.a, measurementUris.b], [measurementUris.c, measurementUris.d]], + ['desc', [measurementUris.d, measurementUris.c], [measurementUris.b, measurementUris.a]], + ]) { + const first = await listMeasurements({ sortDirection, limit: 2 }); + assert.equal(first.response.status, 200, JSON.stringify(first.body)); + assert.deepEqual(first.body.measurements.map(({ uri }) => uri), firstPage); + assert.deepEqual(first.body.measurements.map(({ record }) => record.createdAt), sortDirection === 'asc' + ? ['2025-02-01T00:00:00.000Z', '2025-02-01T00:00:00.000Z'] + : ['2025-02-02T00:00:00.000Z', '2025-02-01T00:00:00.000Z']); + assert.equal(typeof first.body.cursor, 'string'); + + const second = await listMeasurements({ sortDirection, limit: 2, cursor: first.body.cursor }); + assert.equal(second.response.status, 200, JSON.stringify(second.body)); + assert.deepEqual(second.body.measurements.map(({ uri }) => uri), secondPage); + assert.equal(Object.hasOwn(second.body, 'cursor'), false); + const collected = [...firstPage, ...secondPage]; + assert.equal(new Set(collected).size, 4); + assert.deepEqual(collected, sortDirection === 'asc' + ? [measurementUris.a, measurementUris.b, measurementUris.c, measurementUris.d] + : [measurementUris.d, measurementUris.c, measurementUris.b, measurementUris.a]); + } +}); + +test('getMeasurement returns null when the publisher has no sidecar records', async () => { + const { response, body } = await getMeasurement(measurementUris.d); + assert.equal(response.status, 200, JSON.stringify(body)); + assert.equal(body.measurement.author.profile, null); + assert.equal(body.measurement.author.organization, null); +}); + +function assertLuaRuntimeError({ response, body }, method, errorName) { + assert.equal(response.status, 500, JSON.stringify(body)); + assert.equal(body.error, 'script_error'); + assert.equal(body.errorType, 'runtime'); + assert.equal(body.method, method); + assert.match(body.message, new RegExp(errorName)); +} + +test('getMeasurement returns the named RecordNotFound runtime error for an unindexed URI', async () => { + const result = await getMeasurement(`at://${publisher}/${collection}/3jzfcijpj2z2z`); + assertLuaRuntimeError(result, endpoint, 'RecordNotFound'); +}); + +test('getMeasurement returns the named InvalidRequest runtime error for a wrong collection', async () => { + const result = await getMeasurement(`at://${publisher}/org.hypercerts.context.evaluation/3jzfcijpj2z2z`); + assertLuaRuntimeError(result, endpoint, 'InvalidRequest'); +}); + +test('listMeasurements returns the named InvalidRequest runtime error for an oversized limit', async () => { + const result = await listMeasurements({ limit: 101 }); + assertLuaRuntimeError(result, listEndpoint, 'InvalidRequest'); +}); diff --git a/api/tests/http/fixtures/context-measurements.fixture.js b/api/tests/http/fixtures/context-measurements.fixture.js new file mode 100644 index 0000000..fc011ec --- /dev/null +++ b/api/tests/http/fixtures/context-measurements.fixture.js @@ -0,0 +1,85 @@ +import { encode } from '@atcute/cbor'; +import * as CID from '@atcute/cid'; +import { activityRecord, staleOnlyActivityRecord } from '../../fixtures/activities.js'; + +const collection = 'org.hypercerts.context.measurement'; +const profileCollection = 'app.certified.actor.profile'; +const organizationCollection = 'app.certified.actor.organization'; +const publisherA = 'did:plc:measurementhttpfixtureaa'; +const publisherB = 'did:plc:measurementhttpfixturebb'; +const measurer = 'did:plc:oooooooooooooooooooooooo'; +const indexedAt = '2025-03-01T00:00:00.000Z'; +const tiedCreatedAt = '2025-02-01T00:00:00.000Z'; + +const measurementSpecs = [ + { + did: publisherA, + rkey: '3jzfcijpj2z2a', + fields: { + metric: 'trees planted', + unit: 'tree', + value: '00012.3400', + createdAt: tiedCreatedAt, + subjects: [{ uri: activityRecord.uri, cid: activityRecord.cid }], + measurers: [{ did: measurer }], + }, + }, + { + did: publisherA, + rkey: '3jzfcijpj2z2b', + fields: { + metric: 'water restored', + unit: 'litre', + value: '0.0000007', + createdAt: tiedCreatedAt, + subjects: [{ uri: staleOnlyActivityRecord.uri, cid: staleOnlyActivityRecord.cid }], + }, + }, + { + did: publisherA, + rkey: '3jzfcijpj2z2c', + fields: { + metric: 'soil carbon', + unit: 'tonne', + value: '3.1', + createdAt: tiedCreatedAt, + }, + }, + { + did: publisherB, + rkey: '3jzfcijpj2z2d', + fields: { + metric: 'canopy cover', + unit: 'percent', + value: '56.2', + createdAt: '2025-02-02T00:00:00.000Z', + subjects: [{ uri: activityRecord.uri, cid: activityRecord.cid }], + }, + }, +]; + +async function seedRow(collectionName, did, rkey, fields) { + const record = { $type: collectionName, ...fields }; + return { + uri: `at://${did}/${collectionName}/${rkey}`, + did, + collection: collectionName, + rkey, + cid: CID.toString(await CID.create(0x71, encode(record))), + indexedAt, + record, + }; +} + +export const seedRows = await Promise.all([ + ...measurementSpecs.map(({ did, rkey, fields }) => seedRow(collection, did, rkey, fields)), + seedRow(profileCollection, publisherA, 'self', { + displayName: 'Measurement Publisher', + createdAt: indexedAt, + }), + seedRow(organizationCollection, publisherA, 'self', { + organizationType: ['nonprofit'], + visibility: 'public', + createdAt: indexedAt, + }), +]); diff --git a/api/tests/unit/context-measurements/measurements.lua b/api/tests/unit/context-measurements/measurements.lua new file mode 100644 index 0000000..9fbe36a --- /dev/null +++ b/api/tests/unit/context-measurements/measurements.lua @@ -0,0 +1,420 @@ +local MEASUREMENT = "org.hypercerts.context.measurement" +local GET_URI = "at://did:plc:measurements.test/org.hypercerts.context.measurement/3jzfcijpj2z2a" +local URI_A = GET_URI +local URI_B = "at://did:plc:publisher-b/org.hypercerts.context.measurement/3jzfcijpj2z2b" +local URI_C = "at://did:plc:publisher-c/org.hypercerts.context.measurement/3jzfcijpj2z2c" +local SUBJECT_A = "at://did:plc:subject-a/org.hypercerts.claim.activity/3jzfcijpj2z2d" +local SUBJECT_B = "at://did:plc:subject-b/org.hypercerts.collection/3jzfcijpj2z2e" +local DID_A = "did:plc:measurements.test" +local DID_B = "did:plc:publisher-b" +local DID_C = "did:plc:publisher-c" +local NULL = { __test_json_null = true } + +local measurement_a_json = [[{"$type":"org.hypercerts.context.measurement","metric":"trees planted","unit":"tree","value":"00012.3400","createdAt":"2025-01-02T03:04:05.000Z","subjects":[{"uri":"at://did:plc:subject-a/org.hypercerts.claim.activity/3jzfcijpj2z2d","cid":"bafy-old-cid"},{"uri":"at://did:plc:subject-b/org.hypercerts.collection/3jzfcijpj2z2e","cid":"bafy-another-cid"}],"locations":[{"uri":"at://did:plc:place/app.certified.location/3jzfcijpj2z2f"}],"measurers":[{"did":"did:plc:publisher-b"}]}]] +local measurement_b_json = [[{"$type":"org.hypercerts.context.measurement","metric":"water restored","unit":"litre","value":"0.0000007","createdAt":"2025-01-02T03:04:05.000Z"}]] +local measurement_c_json = [[{"$type":"org.hypercerts.context.measurement","metric":"soil carbon","unit":"tonne","value":"3.1","createdAt":"2025-01-03T00:00:00.000Z","subjects":[{"uri":"at://did:plc:subject-a/org.hypercerts.claim.activity/3jzfcijpj2z2d","cid":"bafy-current-cid"}]}]] +local measurement_d_json = [[{"$type":"org.hypercerts.context.measurement","metric":"canopy cover","unit":"percent","value":"56.2","createdAt":"2025-01-04T00:00:00.000Z","subjects":[{"uri":"at://did:plc:subject-b/org.hypercerts.collection/3jzfcijpj2z2e","cid":"bafy-other"}]}]] +local record_json = { + [measurement_a_json] = { + ["$type"] = MEASUREMENT, + metric = "trees planted", + unit = "tree", + value = "00012.3400", + createdAt = "2025-01-02T03:04:05.000Z", + subjects = { + { uri = SUBJECT_A, cid = "bafy-old-cid" }, + { uri = SUBJECT_B, cid = "bafy-another-cid" }, + }, + locations = { { uri = "at://did:plc:place/app.certified.location/3jzfcijpj2z2f" } }, + measurers = { { did = DID_B } }, + }, + [measurement_b_json] = { + ["$type"] = MEASUREMENT, + metric = "water restored", + unit = "litre", + value = "0.0000007", + createdAt = "2025-01-02T03:04:05.000Z", + }, + [measurement_c_json] = { + ["$type"] = MEASUREMENT, + metric = "soil carbon", + unit = "tonne", + value = "3.1", + createdAt = "2025-01-03T00:00:00.000Z", + subjects = { { uri = SUBJECT_A, cid = "bafy-current-cid" } }, + }, + [measurement_d_json] = { + ["$type"] = MEASUREMENT, + metric = "canopy cover", + unit = "percent", + value = "56.2", + createdAt = "2025-01-01T00:00:00.000Z", + subjects = { { uri = SUBJECT_B, cid = "bafy-other" } }, + }, +} + +local function copy_json(value) + if type(value) ~= "table" or value == NULL then return value end + local copy = {} + for key, item in pairs(value) do copy[key] = copy_json(item) end + return copy +end + +json = {} +function json.decode(value) + if value == "null" then return NULL end + if record_json[value] then return copy_json(record_json[value]) end + local direction, timestamp, uri = value:match( + '^{"v":1,"d":"([^"]+)","t":"([^"]+)","u":"([^"]+)"}$') + if direction then return { v = 1, d = direction, t = timestamp, u = uri } end + error("test JSON decoder received an unexpected value: " .. tostring(value)) +end + +function json.encode(value) + return string.format('{"v":%d,"d":"%s","t":"%s","u":"%s"}', value.v, value.d, value.t, value.u) +end + +toarray = function(values) return values end +params = {} +db = { calls = {}, listRows = {}, exactRows = {}, sidecars = {}, failCollection = nil } +function db.backend() return "postgres" end +function db.raw(sql, values) + db.calls[#db.calls + 1] = { sql = sql, values = values } + local collection = values[1] + if db.failCollection == collection then error("test database unavailable", 0) end + if sql:find("WHERE collection = %$1 AND uri = %$2", 1) then + return db.exactRows + end + if sql:find("WHERE collection = %$1 AND rkey = 'self'", 1) then + return db.sidecars[collection] or {} + end + if sql:find("FROM happyview_records AS measurement", 1) then + return db.listRows + end + error("unexpected query in measurement contract test: " .. sql) +end + +local function reset_db() + db.calls = {} + db.listRows = {} + db.exactRows = {} + db.sidecars = {} + db.failCollection = nil +end + +local row_a = { + uri = URI_A, + cid = "bafy-measurement-a", + indexed_at = "2025-01-03T04:00:00.000Z", + did = DID_A, + record = measurement_a_json, + sort_timestamp = "2025-01-02T03:04:05.000000Z", +} +local row_b = { + uri = URI_B, + cid = "bafy-measurement-b", + indexed_at = "2025-01-04T04:00:00.000Z", + did = DID_B, + record = measurement_b_json, + sort_timestamp = "2025-01-02T03:04:05.000000Z", +} +local row_c = { + uri = URI_C, + cid = "bafy-measurement-c", + indexed_at = "2025-01-05T04:00:00.000Z", + did = DID_C, + record = measurement_c_json, + sort_timestamp = "2025-01-03T00:00:00.000000Z", +} +local row_d = { + uri = "at://did:plc:publisher-d/org.hypercerts.context.measurement/3jzfcijpj2z2g", + cid = "bafy-measurement-d", + indexed_at = "2025-01-01T04:00:00.000Z", + did = "did:plc:publisher-d", + record = measurement_d_json, + sort_timestamp = "2025-01-01T00:00:00.000000Z", +} + +local row_missing_date = { + uri = "at://did:plc:publisher-e/org.hypercerts.context.measurement/3jzfcijpj2z2h", + cid = "bafy-measurement-e", + indexed_at = "2025-02-01T00:00:00.000Z", + created_at = "2025-03-01T00:00:00.000Z", + did = "did:plc:publisher-e", + record = '{"$type":"org.hypercerts.context.measurement","metric":"legacy missing date","unit":"item","value":"1"}', + sort_timestamp = "2025-02-01T00:00:00.000000Z", +} +local row_malformed_date = { + uri = "at://did:plc:publisher-f/org.hypercerts.context.measurement/3jzfcijpj2z2i", + cid = "bafy-measurement-f", + indexed_at = nil, + created_at = "2025-02-02T00:00:00.000Z", + did = "did:plc:publisher-f", + record = '{"$type":"org.hypercerts.context.measurement","metric":"legacy malformed date","unit":"item","value":"2","createdAt":"not-a-timestamp"}', + sort_timestamp = "2025-02-02T00:00:00.000000Z", +} +record_json[row_missing_date.record] = { ["$type"] = MEASUREMENT, metric = "legacy missing date", unit = "item", value = "1" } +record_json[row_malformed_date.record] = { ["$type"] = MEASUREMENT, metric = "legacy malformed date", unit = "item", value = "2", createdAt = "not-a-timestamp" } + +local profile_row = { + uri = "at://did:plc:measurements.test/app.certified.actor.profile/self", + did = DID_A, + cid = "bafy-profile-a", + indexed_at = "2025-01-01T00:00:00.000Z", + record = '{"$type":"app.certified.actor.profile","displayName":"Measurement publisher"}', +} +record_json[profile_row.record] = { ["$type"] = "app.certified.actor.profile", displayName = "Measurement publisher" } +local organization_row = { + uri = "at://did:plc:measurements.test/app.certified.actor.organization/self", + did = DID_A, + cid = "bafy-organization-a", + indexed_at = "2025-01-02T00:00:00.000Z", + record = '{"$type":"app.certified.actor.organization","organizationType":["nonprofit"],"visibility":"public"}', +} +record_json[organization_row.record] = { + ["$type"] = "app.certified.actor.organization", + organizationType = { "nonprofit" }, + visibility = "public", +} + +local function expect_error(callback, prefix) + local ok, message = pcall(callback) + assert(not ok, "expected error beginning with " .. prefix) + assert(tostring(message):find(prefix, 1, true), "unexpected error: " .. tostring(message)) +end + +local function call_get(values) + params = values + return handle() +end + +local function call_list(values) + params = values + return handle() +end + +-- getMeasurement uses the complete AT-URI as authority and preserves the original record. +dofile("lua/endpoints/getMeasurement.lua") +reset_db() +db.exactRows = { row_a } +db.sidecars["app.certified.actor.profile"] = { profile_row } +local exact = call_get({ uri = GET_URI }).measurement +assert(exact.uri == GET_URI and exact.cid == row_a.cid and exact.did == DID_A) +assert(exact.record.value == "00012.3400", "numeric-string value must be preserved verbatim") +assert(#exact.record.subjects == 2 and exact.record.subjects[2].cid == "bafy-another-cid") +assert(exact.author.did == DID_A and exact.author.profile.record.displayName == "Measurement publisher") +assert(exact.author.profile.indexedAt == profile_row.indexed_at, "populated profile timestamps must remain unchanged") +assert(exact.author.organization == NULL, "missing organization sidecar must be nullable") +assert(db.calls[1].values[1] == MEASUREMENT and db.calls[1].values[2] == GET_URI) +assert(db.calls[1].sql:find("uri = %$2", 1) ~= nil, "lookup must compare the exact supplied URI") + +reset_db() +db.exactRows = { row_a } +db.sidecars["app.certified.actor.profile"] = { profile_row } +db.sidecars["app.certified.actor.organization"] = { organization_row } +local get_with_populated_sidecars = call_get({ uri = GET_URI }).measurement +assert(get_with_populated_sidecars.author.profile.indexedAt == profile_row.indexed_at) +assert(get_with_populated_sidecars.author.organization.indexedAt == organization_row.indexed_at) + +local profile_indexed_at, organization_indexed_at = profile_row.indexed_at, organization_row.indexed_at +profile_row.indexed_at, organization_row.indexed_at = nil, nil +reset_db() +db.exactRows = { row_a } +db.sidecars["app.certified.actor.profile"] = { profile_row } +db.sidecars["app.certified.actor.organization"] = { organization_row } +local get_with_nil_sidecar_timestamps = call_get({ uri = GET_URI }).measurement +assert(rawget(get_with_nil_sidecar_timestamps.author.profile, "indexedAt") == nil, "SQL-NULL profile indexedAt must be omitted") +assert(rawget(get_with_nil_sidecar_timestamps.author.organization, "indexedAt") == nil, "SQL-NULL organization indexedAt must be omitted") + +profile_row.indexed_at, organization_row.indexed_at = profile_indexed_at, organization_indexed_at + +local indexed_at = row_a.indexed_at +row_a.indexed_at = nil +reset_db() +db.exactRows = { row_a } +local get_without_indexed_at = call_get({ uri = GET_URI }).measurement +assert(get_without_indexed_at.indexedAt == NULL, "getMeasurement must include JSON null for a SQL NULL indexed_at") +row_a.indexed_at = indexed_at + +reset_db() +expect_error(function() call_get({ uri = "at://publisher.example/org.hypercerts.context.measurement/3jzfcijpj2z2a" }) end, "InvalidRequest:") +expect_error(function() call_get({ uri = "at://did:plc:measurements.test/org.hypercerts.context.evaluation/3jzfcijpj2z2a" }) end, "InvalidRequest:") +expect_error(function() call_get({ uri = GET_URI, extra = "not allowed" }) end, "InvalidRequest:") +for _, malformed_did in ipairs({ "did:plc:publisher%ZZ", "did:plc:publisher%A" }) do + reset_db() + db.exactRows = { row_a } + expect_error(function() + call_get({ uri = "at://" .. malformed_did .. "/org.hypercerts.context.measurement/3jzfcijpj2z2a" }) + end, "InvalidRequest:") + assert(#db.calls == 0, "malformed DID escapes in a get URI must fail before SQL") +end +reset_db() +expect_error(function() call_get({ uri = GET_URI }) end, "RecordNotFound:") +reset_db() +db.exactRows = { row_a } +db.failCollection = "app.certified.actor.profile" +expect_error(function() call_get({ uri = GET_URI }) end, "test database unavailable") + +-- listMeasurements includes subjectless rows without filters and defaults to a 25-item descending page. +dofile("lua/endpoints/listMeasurements.lua") +reset_db() +db.listRows = { row_a } +db.sidecars["app.certified.actor.profile"] = { profile_row } +db.sidecars["app.certified.actor.organization"] = { organization_row } +local list_with_populated_sidecars = call_list({}).measurements[1] +assert(list_with_populated_sidecars.author.profile.indexedAt == profile_indexed_at) +assert(list_with_populated_sidecars.author.organization.indexedAt == organization_indexed_at) + +profile_row.indexed_at, organization_row.indexed_at = nil, nil +reset_db() +db.listRows = { row_a } +db.sidecars["app.certified.actor.profile"] = { profile_row } +db.sidecars["app.certified.actor.organization"] = { organization_row } +local list_with_nil_sidecar_timestamps = call_list({}).measurements[1] +assert(rawget(list_with_nil_sidecar_timestamps.author.profile, "indexedAt") == nil, "listMeasurements must omit SQL-NULL profile indexedAt") +assert(rawget(list_with_nil_sidecar_timestamps.author.organization, "indexedAt") == nil, "listMeasurements must omit SQL-NULL organization indexedAt") +profile_row.indexed_at, organization_row.indexed_at = profile_indexed_at, organization_indexed_at + +for _, malformed_did in ipairs({ "did:plc:publisher%ZZ", "did:plc:publisher%A" }) do + for _, filters in ipairs({ + { authors = { malformed_did } }, + { subjects = { "at://" .. malformed_did .. "/org.hypercerts.claim.activity/3jzfcijpj2z2d" } }, + }) do + reset_db() + expect_error(function() call_list(filters) end, "InvalidRequest:") + assert(#db.calls == 0, "malformed DID escapes in a list filter must fail before SQL") + end +end + +-- Well-formed percent escapes remain valid DIDs after using the generic validator. +reset_db() +db.listRows = { row_a } +local escaped_author = "did:plc:publisher%20one" +assert(#call_list({ authors = { escaped_author } }).measurements == 1) +assert(db.calls[1].values[2] == escaped_author, "valid DID escapes must remain a bound author value") + +-- Measurement subject URIs still require a syntactically valid collection NSID. +reset_db() +expect_error(function() call_list({ subjects = { "at://did:plc:subject-a/invalid/3jzfcijpj2z2d" } }) end, "InvalidRequest:") +assert(#db.calls == 0, "invalid collection NSID must be rejected before SQL") + +reset_db() +db.listRows = { row_c, row_b } +local global = call_list({}) +assert(#global.measurements == 2, "unfiltered results include every measurement, including subjectless records") +assert(global.measurements[2].record.subjects == nil, "subjectless record must remain in the global result") +local global_query = db.calls[1] +assert(global_query.values[1] == MEASUREMENT and global_query.values[#global_query.values] == 26) +assert(not global_query.sql:find("jsonb_array_elements", 1, true), "no subject filter should not exclude subjectless rows") +assert(global_query.sql:find("ORDER BY sorted.sort_at DESC, measurement.uri DESC", 1, true)) + +-- Legacy records sort by a valid createdAt, then indexed_at, then the stored row creation time. +reset_db() +db.listRows = { row_missing_date, row_malformed_date } +local fallback_ascending = call_list({ sortDirection = "asc", limit = "1" }) +assert(fallback_ascending.measurements[1].record.createdAt == nil, "missing createdAt must remain missing in the source record") +assert(fallback_ascending.measurements[1].indexedAt == row_missing_date.indexed_at) +assert(fallback_ascending.cursor ~= nil) +local fallback_sql = db.calls[1].sql +local input_check = fallback_sql:find("pg_input_is_valid", 1, true) +local guarded_cast = fallback_sql:find("THEN (measurement.record::jsonb->>'createdAt')::timestamptz", 1, true) +assert(input_check and guarded_cast and input_check < guarded_cast, "untrusted createdAt must be validated before casting") +assert(fallback_sql:find("ELSE COALESCE(measurement.indexed_at::timestamptz, measurement.created_at::timestamptz) END", 1, true)) +assert(fallback_sql:find("ORDER BY sorted.sort_at ASC, measurement.uri ASC", 1, true)) + +reset_db() +db.listRows = { row_malformed_date } +local fallback_ascending_next = call_list({ sortDirection = "asc", limit = "1", cursor = fallback_ascending.cursor }) +assert(fallback_ascending_next.measurements[1].record.createdAt == "not-a-timestamp", "malformed source date must not be rewritten") +assert(fallback_ascending_next.measurements[1].indexedAt == NULL, "listMeasurements must include JSON null for SQL NULL indexed_at") +assert(db.calls[1].sql:find("(sorted.sort_at, measurement.uri) > (($2)::timestamptz, $3)", 1, true)) +assert(db.calls[1].values[2] == row_missing_date.sort_timestamp and db.calls[1].values[3] == row_missing_date.uri) + +reset_db() +db.listRows = { row_malformed_date, row_missing_date } +local fallback_descending = call_list({ sortDirection = "desc", limit = "1" }) +assert(fallback_descending.measurements[1].record.createdAt == "not-a-timestamp") +assert(fallback_descending.cursor ~= nil) +reset_db() +db.listRows = { row_missing_date } +call_list({ sortDirection = "desc", limit = "1", cursor = fallback_descending.cursor }) +assert(db.calls[1].sql:find("ORDER BY sorted.sort_at DESC, measurement.uri DESC", 1, true)) +assert(db.calls[1].sql:find("(sorted.sort_at, measurement.uri) < (($2)::timestamptz, $3)", 1, true)) +assert(db.calls[1].values[2] == row_malformed_date.sort_timestamp and db.calls[1].values[3] == row_malformed_date.uri) + +-- Filters use repository DID and any subject URI, with OR within arrays and AND between them. +reset_db() +db.listRows = { row_c, row_b, row_a } +local filtered = call_list({ + authors = { DID_A, DID_B, DID_A }, + subjects = { SUBJECT_A, SUBJECT_B, SUBJECT_A }, + sortDirection = "asc", + limit = "2", +}) +assert(#filtered.measurements == 2) +local filter_query = db.calls[1] +local sql, values = filter_query.sql, filter_query.values +assert(sql:find("measurement.did IN ($2, $3)", 1, true), "author filter must match repository owners and deduplicate OR values") +assert(sql:find("jsonb_array_elements", 1, true), "subject filter must inspect every subjects[] entry") +assert(sql:find("subject.value->>'uri' IN ($4, $5)", 1, true), "subject values must use OR and compare URI only") +assert(not sql:find("measurers", 1, true), "authors must not match named measurers") +assert(not sql:find("->>'cid'", 1, true), "subject matching must not compare CID") +assert(sql:find("measurement.did IN ($2, $3)", 1, true) < sql:find("jsonb_array_elements", 1, true), "distinct filters must both constrain one query") +assert(values[1] == MEASUREMENT and values[2] == DID_A and values[3] == DID_B) +assert(values[4] == SUBJECT_A and values[5] == SUBJECT_B and values[6] == 3) +assert(sql:find("ORDER BY sorted.sort_at ASC, measurement.uri ASC", 1, true)) +assert(type(filtered.cursor) == "string" and #filtered.cursor > 0, "overfetch must return an opaque next-page cursor") + +local function unhex(value) + return (value:gsub("..", function(pair) return string.char(tonumber(pair, 16)) end)) +end +local cursor_payload = json.decode(unhex(filtered.cursor)) +assert(cursor_payload.v == 1 and cursor_payload.d == "asc") +assert(cursor_payload.t == row_b.sort_timestamp and cursor_payload.u == row_b.uri) + +-- Cursor direction is bound; a valid cursor continues after the last returned (createdAt, uri) tuple. +reset_db() +expect_error(function() call_list({ cursor = filtered.cursor, sortDirection = "desc" }) end, "InvalidRequest:") +assert(#db.calls == 0, "direction mismatch must be rejected before querying") +reset_db() +db.listRows = { row_a } +local continued = call_list({ + authors = { DID_A, DID_B, DID_A }, + subjects = { SUBJECT_A, SUBJECT_B, SUBJECT_A }, + sortDirection = "asc", + limit = "2", + cursor = filtered.cursor, +}) +assert(#continued.measurements == 1 and continued.cursor == nil) +local continuation_query = db.calls[1] +assert(continuation_query.sql:find("(sorted.sort_at, measurement.uri) > (($6)::timestamptz, $7)", 1, true)) +assert(continuation_query.values[6] == row_b.sort_timestamp and continuation_query.values[7] == row_b.uri) + +-- Repeated-value and scalar validation protects bounds and avoids issuing malformed database queries. +reset_db() +local too_many = {} +for index = 1, 101 do too_many[index] = "did:plc:publisher-" .. index end +expect_error(function() call_list({ authors = too_many }) end, "InvalidRequest:") +expect_error(function() call_list({ limit = "101" }) end, "InvalidRequest:") +expect_error(function() call_list({ limit = { "2", "3" } }) end, "InvalidRequest:") +expect_error(function() call_list({ unexpected = "value" }) end, "InvalidRequest:") +expect_error(function() call_list({ cursor = "not-a-cursor" }) end, "InvalidRequest:") +local year_zero_json = json.encode({ v = 1, d = "asc", t = "0000-01-01T00:00:00.000000Z", u = GET_URI }) +local year_zero_cursor = year_zero_json:gsub(".", function(char) return string.format("%02x", string.byte(char)) end) +expect_error(function() call_list({ sortDirection = "asc", cursor = year_zero_cursor }) end, "InvalidRequest:") +assert(#db.calls == 0, "year-zero cursor must be rejected before SQL") +for _, malformed_did in ipairs({ "did:plc:publisher%ZZ", "did:plc:publisher%A" }) do + local cursor_json = json.encode({ + v = 1, + d = "asc", + t = "2025-01-01T00:00:00.000000Z", + u = "at://" .. malformed_did .. "/org.hypercerts.context.measurement/3jzfcijpj2z2a", + }) + local cursor = cursor_json:gsub(".", function(char) return string.format("%02x", string.byte(char)) end) + reset_db() + expect_error(function() call_list({ sortDirection = "asc", cursor = cursor }) end, "InvalidRequest:") + assert(#db.calls == 0, "cursor with malformed DID escapes must be rejected before SQL") +end +assert(#db.calls == 0, "invalid list requests must not reach the database") + +print("measurement behavior contracts passed") diff --git a/api/tests/unit/context-measurements/measurements.test.js b/api/tests/unit/context-measurements/measurements.test.js new file mode 100644 index 0000000..4945de0 --- /dev/null +++ b/api/tests/unit/context-measurements/measurements.test.js @@ -0,0 +1,16 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { spawnSync } from 'node:child_process'; +import { fileURLToPath } from 'node:url'; + +const root = fileURLToPath(new URL('../../../', import.meta.url)); + +test('measurement handlers preserve the public lookup, filtering, pagination, and hydration contract', () => { + const result = spawnSync('lua5.4', ['tests/unit/context-measurements/measurements.lua'], { + cwd: root, + encoding: 'utf8', + }); + assert.equal(result.error, undefined, result.error?.message); + assert.equal(result.status, 0, `${result.stdout}${result.stderr}`); + assert.match(result.stdout, /measurement behavior contracts passed/); +}); diff --git a/api/tests/unit/fixtures/http-seed-rows.test.js b/api/tests/unit/fixtures/http-seed-rows.test.js index 1c991a3..04fce43 100644 --- a/api/tests/unit/fixtures/http-seed-rows.test.js +++ b/api/tests/unit/fixtures/http-seed-rows.test.js @@ -86,6 +86,14 @@ test('shared and discovered HTTP seed rows preserve acknowledgement sidecar isol `${rkey} publisher must have no profile or organization sidecars in shared or HTTP fixtures`); } + const measurementWithoutSidecars = rows.find(({ row }) => + row.collection === 'org.hypercerts.context.measurement' && row.rkey === '3jzfcijpj2z2d'); + assert.ok(measurementWithoutSidecars, 'expected a measurement fixture with absent publisher sidecars'); + const measurementSidecars = rows.filter(({ row }) => row.did === measurementWithoutSidecars.row.did + && ['app.certified.actor.profile', 'app.certified.actor.organization'].includes(row.collection)); + assert.deepEqual(measurementSidecars.map(({ row }) => row.uri), [], + 'the measurement fixture without sidecars must not share its publisher DID with another fixture'); + const byUri = new Map(); const byIdentity = new Map(); const duplicateUris = []; diff --git a/api/tests/unit/tooling/lexicons.test.js b/api/tests/unit/tooling/lexicons.test.js index 013cec9..1b3ee88 100644 --- a/api/tests/unit/tooling/lexicons.test.js +++ b/api/tests/unit/tooling/lexicons.test.js @@ -39,6 +39,7 @@ test('the full validation Lexicon closure resolves locally while only selected p 'org.hypercerts.context.acknowledgement', 'org.hypercerts.context.attachment', 'org.hypercerts.context.evaluation', + 'org.hypercerts.context.measurement', 'org.hypercerts.defs', 'org.hypercerts.entity.feature', 'org.hypercerts.funding.receipt', @@ -174,6 +175,34 @@ if (hasModule('modules/actor-follow/manifest.json')) test('follow query Lexicons }); }); +if (hasModule('modules/context-measurements/manifest.json')) test('measurement query Lexicons preserve the typed publisher and full-record contract', async () => { + const { lexicons, documents } = await validatePackageLexicons(); + const byId = new Map(documents.map((document) => [document.id, document])); + const getMeasurement = byId.get('org.hypercerts.context.getMeasurement'); + const listMeasurements = byId.get('org.hypercerts.context.listMeasurements'); + const view = lexicons.getDefOrThrow('org.hypercerts.context.getMeasurement#measurementView'); + assert.ok(getMeasurement && listMeasurements); + assert.deepEqual(getMeasurement.defs.main.parameters.required, ['uri']); + assert.equal(getMeasurement.defs.main.parameters.properties.uri.format, 'at-uri'); + assert.deepEqual(getMeasurement.defs.main.errors.map(({ name }) => name), ['InvalidRequest', 'RecordNotFound']); + assert.equal(getMeasurement.defs.output.properties.measurement.ref, 'lex:org.hypercerts.context.getMeasurement#measurementView'); + assert.equal(view.properties.author.ref, 'lex:org.hypercerts.api.defs#actorView'); + assert.equal(view.properties.record.ref, 'lex:org.hypercerts.context.measurement'); + assert.deepEqual(view.required, ['uri', 'cid', 'indexedAt', 'did', 'author', 'record']); + assert.deepEqual(view.nullable, ['indexedAt']); + + const properties = listMeasurements.defs.main.parameters.properties; + assert.equal(properties.authors.maxLength, 100); + assert.equal(properties.authors.items.format, 'did'); + assert.equal(properties.subjects.maxLength, 100); + assert.equal(properties.subjects.items.format, 'at-uri'); + assert.deepEqual(properties.sortDirection.enum, ['asc', 'desc']); + assert.equal(properties.sortDirection.default, 'desc'); + assert.deepEqual([properties.limit.minimum, properties.limit.maximum, properties.limit.default], [1, 100, 25]); + assert.equal(listMeasurements.defs.output.properties.measurements.items.ref, 'lex:org.hypercerts.context.getMeasurement#measurementView'); + assert.deepEqual(listMeasurements.defs.main.errors.map(({ name }) => name), ['InvalidRequest']); +}); + test('contributor-information view is owned by getContributorInformation and keeps nullable indexedAt required', async () => { const { lexicons, documents } = await validatePackageLexicons(); const byId = new Map(documents.map((document) => [document.id, document]));