feat/public-access - Public Data Access

[KinshukSS2]

What

Implements Path A of the ODRL dual-path architecture: unauthenticated users can read datasets flagged is_public = true. All access control is enforced at the PostgreSQL RLS level — no application-layer guards.

Why

The previous blanket anonymous_datastream policy used USING (true), exposing all rows to guests. This PR replaces it with a per-row is_public flag and fine-grained RLS policies.

Key Changes

database/migrations/004_public_access.sql

• ADD COLUMN is_public BOOLEAN NOT NULL DEFAULT false on "Datastream" (existing rows default to private)
• Drops old blanket guest policies; creates new ones:
  • anonymous_datastream: USING (is_public = true)
  • anonymous_observation: USING (datastream_id IN (SELECT id FROM "Datastream" WHERE is_public = true))
• GRANT INSERT ON "AuditLog" TO "guest" for audit events

api/app/oauth.py

• Adds get_optional_current_user() — returns user dict or None (never raises 401/403). Handles missing header, expired/revoked/invalid tokens, and pending accounts silently.

api/app/v1/endpoints/read/read.py

• Replaces the AUTHORIZATION/ANONYMOUS_VIEWER conditional with a universal Depends(get_optional_current_user)
• When current_user is None: logs a PUBLIC_READ audit event (before role switch), then falls back to SET LOCAL ROLE "guest" to activate RLS
• Fix (c73ba6e): Removed orphaned RESET ROLE inside the $value path — SET LOCAL ROLE is transaction-scoped and auto-reverts; calling RESET ROLE mid-transaction would have prematurely re-escalated privileges

All 10 read endpoints (datastream.py, observation.py, etc.) migrated to get_optional_current_user.

Reviewer Focus

• RLS policies in migration 004 are the security boundary — verify USING clauses
• get_optional_current_user must never raise HTTPException — verify all branches return None
• Audit INSERT runs before SET LOCAL ROLE guest — intentional (pool-user privileges needed)
• Migration is idempotent: IF NOT EXISTS / DROP … IF EXISTS guards throughout

Testing

# Anonymous — sees only public datastreams
curl http://localhost:8018/istsos4/v1.1/Datastreams

# Authenticated — sees all
curl http://localhost:8018/istsos4/v1.1/Datastreams -H "Authorization: Bearer $TOKEN"

# Invalid token — degrades to guest (HTTP 200, not 401)
curl http://localhost:8018/istsos4/v1.1/Datastreams -H "Authorization: Bearer INVALID"

# Audit log
SELECT action_type, actor_id FROM sensorthings."AuditLog" WHERE action_type = 'PUBLIC_READ';

---

Summary

This PR implements Path A of the ODRL dual-path architecture: unauthenticated users can now read datasets that an administrator has explicitly marked as is_public = true, while all other datasets remain securely hidden. Access control is enforced entirely at the PostgreSQL kernel level via Row-Level Security (RLS) — no application-layer if/else guards.

---

Motivation

Before this PR:

• The system had a blanket anonymous_datastream RLS policy using USING (true), which exposed all Datastreams and Observations to unauthenticated callers.
• Public access required the ANONYMOUS_VIEWER environment flag and a binary conditional in the read pipeline.
• There was no audit trail for unauthenticated data access.

After this PR:

• Anonymous users see only rows where is_public = true.
• Authenticated users see the full row set their database role permits (unchanged).
• Every anonymous read generates a PUBLIC_READ entry in sensorthings."AuditLog".
• The feature works regardless of the ANONYMOUS_VIEWER / AUTHORIZATION flags — the optional-auth dependency gracefully degrades for any token state.

---

Changes

database/migrations/004_public_access.sql (new)

DDL	Purpose
ALTER TABLE "Datastream" ADD COLUMN IF NOT EXISTS is_public BOOLEAN NOT NULL DEFAULT false	Per-datastream public flag; defaults to private for all existing rows
DROP POLICY IF EXISTS anonymous_datastream ON "Datastream"	Remove blanket allow-all guest policy
DROP POLICY IF EXISTS anonymous_observation ON "Observation"	Remove blanket allow-all guest policy
CREATE POLICY anonymous_datastream … USING (is_public = true)	Guest sees only public datastreams
CREATE POLICY anonymous_observation … USING (datastream_id IN (SELECT id … WHERE is_public = true))	Observations inherit parent datastream visibility — no per-observation flag drift
GRANT INSERT ON "AuditLog" TO "guest"	Allows the guest session to write PUBLIC_READ audit events

The entire migration is guarded by IF current_setting('custom.authorization', true)::boolean (consistent with migrations 001–003) and is idempotent via the IF NOT EXISTS / DROP … IF EXISTS pattern.

---

api/app/oauth.py

Added oauth2_scheme_optional

oauth2_scheme_optional = OAuth2PasswordBearer(tokenUrl="Login", auto_error=False)

auto_error=False prevents FastAPI from auto-rejecting requests with no Authorization header before our handler runs.

Added get_optional_current_user()

A soft version of get_current_user that returns None instead of raising 401/403 for:

Condition	Behaviour
No Authorization header	return None
Malformed / invalid signature	except InvalidTokenError → return None
Expired token	except InvalidTokenError → return None
Redis-revoked token	return None
Unknown sub claim	return None
Account in pending state	return None (prevents pending users accessing guest data)

---

api/app/v1/endpoints/read/read.py

Module-level dependency — removed the AUTHORIZATION/ANONYMOUS_VIEWER conditional:

# Before
user = Header(default=None, include_in_schema=False)
if AUTHORIZATION and not ANONYMOUS_VIEWER:
    user = Depends(get_current_user)

# After
user = Depends(get_optional_current_user)

asyncpg_stream_results execution engine — updated anonymous branch:

if current_user is not None:
    await set_role(connection, current_user)   # authenticated path
else:
    # 1. Audit INSERT while still at pool-user privilege level
    await log_audit_event(conn, AUDIT_ACTION_PUBLIC_READ, actor_id=None, ...)
    # 2. Downgrade to guest → activates is_public RLS
    current_user = {"username": "guest"}
    await set_role(connection, current_user)

Fix (c73ba6e): Removed an orphaned RESET ROLE call in the $value early-return path. set_role() uses SET LOCAL ROLE (transaction-scoped), so an explicit RESET ROLE mid-transaction would have prematurely escalated back to pool-user privileges before the transaction committed.

---

api/app/v1/endpoints/read/{datastream,observation,thing,...}.py

All 10 specific-entity read endpoints now use:

from app.oauth import get_optional_current_user
user = Depends(get_optional_current_user)

The old AUTHORIZATION/ANONYMOUS_VIEWER conditionals are removed from all of them.

---

Security Invariants

Invariant	Enforcement Layer
Private datastreams hidden from guests	PostgreSQL RLS (kernel)
Observation visibility derived from parent Datastream	Correlated sub-query in RLS policy
No privilege leak on $value path	SET LOCAL ROLE auto-reverts; RESET ROLE removed
Expired/revoked tokens → guest fallback, not 401	get_optional_current_user
Pending users cannot use guest path as bypass	PENDING_ROLE check in optional dep
Anonymous reads are audited	log_audit_event(AUDIT_ACTION_PUBLIC_READ) before role downgrade
Audit trail is append-only	UPDATE/DELETE revoked from all roles (migration 003)
Pool connection role never leaks	SET LOCAL ROLE scoped to connection.transaction()

---

Test Plan

Unit / Manual

BASE="http://localhost:8018/istsos4/v1.1"
TOKEN=$(curl -s -X POST "${BASE%/v1.1}/Login" \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "username=admin&password=admin" | jq -r .access_token)

# 1. Mark a datastream public
curl -s -X PATCH "$BASE/Datastreams(1)" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"is_public": true}'

# 2. Anonymous fetch — should return only public rows
curl -s "$BASE/Datastreams" | jq '.value | length'

# 3. Authenticated fetch — should return ALL rows
curl -s "$BASE/Datastreams" -H "Authorization: Bearer $TOKEN" | jq '.value | length'

# 4. Invalid token → graceful guest fallback (HTTP 200, not 401)
curl -s "$BASE/Datastreams" -H "Authorization: Bearer INVALID" | jq .

# 5. Observations — only from public Datastreams
curl -s "$BASE/Observations" | jq '.value | length'

# 6. Audit log
docker exec istsos4-database-1 psql -U postgres -d istsos -c \
  "SELECT action_type, actor_id, dataset_id FROM sensorthings.\"AuditLog\" WHERE action_type='PUBLIC_READ' LIMIT 5;"

Expected assertions

• Anonymous: only is_public = true Datastreams visible; corresponding Observations visible; all else returns 0 rows or 404.
• Authenticated admin: full row count.
• Invalid token: HTTP 200 with guest-scoped rows.
• AuditLog: PUBLIC_READ row with actor_id IS NULL for every anonymous read.

---

Migration Instructions

# Apply on a running stack
docker exec -i istsos4-database-1 psql -U postgres -d istsos \
  < database/migrations/004_public_access.sql

# Verify column exists
docker exec istsos4-database-1 psql -U postgres -d istsos -c \
  "\d sensorthings.\"Datastream\"" | grep is_public

# Verify new policies
docker exec istsos4-database-1 psql -U postgres -d istsos -c \
  "SELECT policyname, roles, qual FROM pg_policies WHERE tablename IN ('Datastream','Observation');"

---

Review Notes

• CREATE POLICY is idempotent via DROP POLICY IF EXISTS before each CREATE POLICY.
• ADD COLUMN IF NOT EXISTS is safe to re-run.
• The GRANT INSERT ON "AuditLog" TO "guest" is guarded by an existence check on pg_roles.
• No environment variable changes required — the feature activates whenever AUTHORIZATION=1.