Skip to content

feat(fly-io,hetzner,railway): MachineIdentity, tagged createServer errors, slim GraphQL - #477

Merged
sam-goodwin merged 9 commits into
mainfrom
feat/fly-machine-identity
Aug 29, 2026
Merged

feat(fly-io,hetzner,railway): MachineIdentity, tagged createServer errors, slim GraphQL#477
sam-goodwin merged 9 commits into
mainfrom
feat/fly-machine-identity

Conversation

@sam-goodwin

@sam-goodwin sam-goodwin commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Add Fly MachineIdentity so Machines PetSem encrypt/sign/decrypt/verify can talk to /.fly/api as the machine. Org-token GetSecret on the same fiber is unchanged.

Slim Railway GraphQL list/get selections so nuke list()s don't pull workspace/team/members/baseEnvironment/canvasGroupRefs/subscriptionType — those fields fail decode on live rows and blow the rate budget. Stop paginateRelay when endCursor does not advance. Type GraphQL INTERNAL_SERVER_ERROR "not found" / "creating too quickly" as RailwayNotFound / RailwayRateLimited, and parse retry-after from "per N seconds" or "every N s" (default 31s when the gateway sends none). Identify GraphQL as source=alchemy so Cloudflare's anonymous 10 RPS cap does not apply.

Type Hetzner createServer quota/placement failures as ServerLimitExceeded / ServerPlacementError instead of Forbidden / PreconditionFailed plus message text. convert.ts applies /shapes/ patches after OpenAPI conversion.

Fly MachineIdentity

FlyIoProtocol still sends Authorization: Bearer by default. A per-call MachineIdentity service drops that header and sets Connection: close. The protocol does not sniff an empty API key.

export class MachineIdentity extends Context.Service<
  MachineIdentity,
  true
>()("FlyIoMachineIdentity") {}

Generated machines ops are unchanged. Callers provide MachineIdentity plus an HttpClient with socketPath: "/.fly/api".

Hetzner createServer errors

// 403 resource_limit_exceeded, message includes "server limit"
ServerLimitExceeded
// 412 resource_unavailable, message includes "placement"
ServerPlacementError

Railway GraphQL

# projects list — identity scalars
project { id name workspaceId baseEnvironmentId primaryEnvironmentId }

# project({id}) — identity plus services / buckets / groups
services { id name groupId templateId }
buckets  { id name groupId }
groups   { id name groupId }

# environments list — identity plus volumeInstances (no environment({id}))
# volumeInstance GET — no nested environment
# volumeCreate — returns volumeInstances

# Boolean! mutations — no subfields
mutation deploymentCancel($id: String!) { deploymentCancel(id: $id) }

environment({id}) still selects canvasGroupRefs / meta. List paths must not call it.

const retryAfterForRateLimit = (message, headers) => {
  const fromHeader = parseRetryAfterForStatus(429, headers);
  if (fromHeader !== undefined) return fromHeader;
  const match =
    /per (\d+) seconds/.exec(message) ?? /every (\d+)\s*s/.exec(message);
  if (match !== null) return Duration.seconds(Number(match[1]) + 1);
  return Duration.seconds(31);
};

FlyIoProtocol still sends Bearer by default. When MachineIdentity is
provided on the calling fiber, encode drops Authorization and sets
Connection: close so encrypt/sign/decrypt/verify work as the machine
over the unix socket without sniffing an empty token.
@alchemy-version-bot

alchemy-version-bot Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Install the packages built from this commit:

Distilled

@distilled.cloud/core

bun add https://pkg.ing/@distilled.cloud/core/88b462a

@distilled.cloud/aws

bun add https://pkg.ing/@distilled.cloud/aws/88b462a

@distilled.cloud/axiom

bun add https://pkg.ing/@distilled.cloud/axiom/88b462a

@distilled.cloud/azure

bun add https://pkg.ing/@distilled.cloud/azure/88b462a

@distilled.cloud/cloudflare

bun add https://pkg.ing/@distilled.cloud/cloudflare/88b462a

@distilled.cloud/coinbase

bun add https://pkg.ing/@distilled.cloud/coinbase/88b462a

@distilled.cloud/discord

bun add https://pkg.ing/@distilled.cloud/discord/88b462a

@distilled.cloud/expo-eas

bun add https://pkg.ing/@distilled.cloud/expo-eas/88b462a

@distilled.cloud/fly-io

bun add https://pkg.ing/@distilled.cloud/fly-io/88b462a

@distilled.cloud/gcp

bun add https://pkg.ing/@distilled.cloud/gcp/88b462a

@distilled.cloud/github

bun add https://pkg.ing/@distilled.cloud/github/88b462a

@distilled.cloud/hetzner

bun add https://pkg.ing/@distilled.cloud/hetzner/88b462a

@distilled.cloud/kubernetes

bun add https://pkg.ing/@distilled.cloud/kubernetes/88b462a

@distilled.cloud/mongodb-atlas

bun add https://pkg.ing/@distilled.cloud/mongodb-atlas/88b462a

@distilled.cloud/neon

bun add https://pkg.ing/@distilled.cloud/neon/88b462a

@distilled.cloud/planetscale

bun add https://pkg.ing/@distilled.cloud/planetscale/88b462a

@distilled.cloud/posthog

bun add https://pkg.ing/@distilled.cloud/posthog/88b462a

@distilled.cloud/prisma-postgres

bun add https://pkg.ing/@distilled.cloud/prisma-postgres/88b462a

@distilled.cloud/railway

bun add https://pkg.ing/@distilled.cloud/railway/88b462a

@distilled.cloud/stripe

bun add https://pkg.ing/@distilled.cloud/stripe/88b462a

@distilled.cloud/supabase

bun add https://pkg.ing/@distilled.cloud/supabase/88b462a

@distilled.cloud/turso

bun add https://pkg.ing/@distilled.cloud/turso/88b462a

@distilled.cloud/typesense

bun add https://pkg.ing/@distilled.cloud/typesense/88b462a

@distilled.cloud/vercel

bun add https://pkg.ing/@distilled.cloud/vercel/88b462a

@distilled.cloud/workos

bun add https://pkg.ing/@distilled.cloud/workos/88b462a

The GraphQL `projects` connection was selecting nested workspace/team/members/baseEnvironment at maxDepth 2. That pulled subscriptionType and canvasGroupRefs, failed decode on live rows, and made listOwnedProjects hang under nuke.

Also terminate paginateRelay when endCursor does not advance.
Nuke scans ~20 resource list()s that each call project({id}) and
environments. The default maxDepth-2 selection pulled workspace/team/
members/baseEnvironment and canvasGroupRefs, then distilled retried
429s silently for minutes.

project({id}) now selects identity plus services/buckets/groups.
environments list is identity scalars. HttpClient GraphQlHttpGate
caps in-flight GraphQL POSTs at 2 (DISTILLED_RAILWAY_GRAPHQL_CONCURRENCY).
Do not serialize GraphQL behind a process lock. Slim project({id}) and
environments selections are the nuke-scan fix; wrapping HttpClient broke
.items() streams.
Volume.list can enumerate from the slim environments connection.
Nuke must not call environment({id}) — that GET still selects
canvasGroupRefs/meta and is what made the scan spin.
Select volumeInstances on volumeCreate and environments list, drop the
nested environment from volumeInstance GET, stamp service.templateId on
the slim project query, and map ServiceInstance/BucketInstance-not-found
plus "creating too quickly" onto RailwayNotFound/RailwayRateLimited.
GraphQL rate limits without Retry-After default to 31s.
@sam-goodwin sam-goodwin changed the title feat(fly-io): MachineIdentity protocol for PetSem over /.fly/api fix: fly and railway patches Aug 28, 2026
@sam-goodwin sam-goodwin changed the title fix: fly and railway patches feat(fly-io,railway): MachineIdentity protocol and slim GraphQL queries Aug 28, 2026
S3 (and Tigris) report a missing object as NoSuchKey. deleteObject was
patched to NotFound, so Fly.Website.AssetDeployment could not catchTag
the same error AWS S3 getObject uses.
…nCreateFailed

GraphQL INTERNAL_SERVER_ERROR "Failed to create service domain" is a
transient instance-not-ready. Also map environment-create rate limits.
…loyment mutations

Hetzner createServer now surfaces ServerLimitExceeded and ServerPlacementError instead of Forbidden/PreconditionFailed message checks. convert.ts applies /shapes/ patches after OpenAPI conversion.

Railway deploymentCancel/Remove/Stop/Restart queries match Boolean! (no subfields). GraphQL requests send source=alchemy so the public edge does not apply the anonymous 10 RPS cap, and retry-after also parses "every N s".
@sam-goodwin sam-goodwin changed the title feat(fly-io,railway): MachineIdentity protocol and slim GraphQL queries feat(fly-io,hetzner,railway): MachineIdentity, tagged createServer errors, slim GraphQL Aug 28, 2026
@sam-goodwin
sam-goodwin merged commit e7f12dd into main Aug 29, 2026
3 checks passed
@sam-goodwin
sam-goodwin deleted the feat/fly-machine-identity branch August 29, 2026 02:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant