Skip to content

Lineage observability module points its web UI and lineage-sink contract at a hostname no compose service answers #1029

Closed

Description

@RonaldHensbergen

Problem statement

Found while implementing #1010 (PR #1028): the experimental lineage
observability module is a multi-service module (marquez-api,
marquez-web), so no compose service named after the instance id is
ever rendered. Yet three places substitute ${service.host} — the
instance id — as if it were a resolvable DNS name:

  1. modules-experimental/observability/marquez/module.yaml lineage-sink
    provide: host: ${service.host} and
    connectionUri: http://${service.host}:${config.apiPort} — a
    consumer of the lineage-sink contract would try to reach
    http://marquez:5000, which nothing answers on the Compose network.
  2. The marquez-web service's MARQUEZ_HOST: "${service.host}"
    environment variable — the web UI's link to the API backend — points
    at the same non-existent hostname.
  3. marquez-api's hostname: "${service.host}" (cosmetic; the container
    hostname only, not a DNS name).

The multi-service half of the ${service.dns} / serviceName mechanism
from #1010 does not help here: ${service.dns} also keeps the instance
id for multi-service modules, and serviceName is currently only
honored for http-service contracts (E136 route generation and the new
E147 host check). The lineage-sink kind is not E147-checked at all, so
this fails silently.

Proposed solution

  • Point the web UI's MARQUEZ_HOST (and the lineage-sink contract's
    host/connectionUri) at the API service's actual DNS name. The
    reference must go through the renderer's service-naming rule
    (_compose_service_name(id, service)) so it survives non-conventional
    instance ids, mirroring how http-service providers advertise
    serviceName.
  • Consider extending the serviceName mechanism (and E147 coverage) to
    the lineage-sink contract kind, so this class of mismatch fails
    loudly instead of silently.

Reproduction

Render profiles/local-dagster-postgres-marquez/profile.yaml (it
instantiates the module as id: marquez) and inspect the plan: the
lineage-sink contract advertises host: marquez; the rendered compose
file contains only marquez-api and marquez-web services, so
http://marquez does not resolve from any sibling container.

Impact

Module-level, experimental: the shipped lineage profile's web UI cannot
reach its API backend, and any consumer binding the lineage-sink
contract gets an unresolvable endpoint. No silent production impact
today (the module is explicitly not production-suitable), but it is the
same contract-hostname class of bug as #1010, found one layer deeper.

Metadata

Metadata

Labels

area:configurationConfiguration, wiring, and settings behaviorbugSomething isn't workingpriority:lowNice to have, low urgencysize:smallRoughly up to 3 hours

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions