Skip to content

feat(gcp): add GCP resource providers, HTTP bindings, and nuke - #1336

Draft
Mkassabov wants to merge 25 commits into
mainfrom
gcp-resources
Draft

feat(gcp): add GCP resource providers, HTTP bindings, and nuke#1336
Mkassabov wants to merge 25 commits into
mainfrom
gcp-resources

Conversation

@Mkassabov

@Mkassabov Mkassabov commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

GCP resource factory coverage: 1051 Resource exports (each with a dedicated {Resource}.test.ts), 192 HTTP Binding.Services, live tests under packages/alchemy/test/GCP, and pnpm nuke:gcp.

Effect-native Function with bindings

Cloud Run is the GCP analog of Lambda/Worker. Bindings grant IAM onto the runtime service account (AWS policyStatements / Cloudflare AccountApiToken) and inject env. Memorystore speaks RESP through the shared alchemy/Redis client.

const tweets = yield* GCP.PubSub.Topic("tweets", {});
const cache = yield* GCP.Redis.Instance("Cache", { memorySizeGb: 1 });

export class TweetBot extends GCP.Function<TweetBot>()(
  "TweetBot",
  { main: import.meta.url },
  Effect.gen(function* () {
    const publish = yield* GCP.PubSub.Publish(tweets);
    const redis = yield* GCP.Redis.ReadWriteRedis(cache);
    return {
      fetch: Effect.gen(function* () {
        const text = (yield* redis.get("last-tweet")) ?? "hello twitter";
        yield* publish({
          body: { messages: [{ data: btoa(text) }] },
        });
        return HttpServerResponse.text("queued");
      }),
    };
  }).pipe(
    Effect.provide(GCP.PubSub.PublishHttp),
    Effect.provide(GCP.Redis.ReadWriteRedisHttp),
  ),
) {}

GCP.Function is GCP.Run.Service. Jobs are the same shape with GCP.Run.Job (runs to completion). Zip-source Cloud Functions stay on GCP.CloudFunctions.Function and still receive binding env + IAM on the runtime SA.

const bucket = yield* GCP.Storage.Bucket("data", { forceDestroy: true });
const topic = yield* GCP.PubSub.Topic("events", {});

Observe-ensure-sync reconcilers, idempotent delete, list() of alchemy-labeled rows. HTTP bindings close over Credentials + HttpClient on GCP.providers() and grant least-privilege IAM on the host. Distilled pin: cecf35fe9597699b66e8e5ad3e3db6b135e2a44c.

Catalog

Every distilled create+get+delete trio is implemented or out-of-scope ({SCRATCH}/gcp-catalog.txt): 1127 implemented rows, 0 missing, 3 out-of-scope (Cloud Tasks tasks, Spanner sessions, Storage objects). Vertex/Apigee/consumer APIs are implemented with skipIf-gated live tests, not catalogued away.

Every registered Resource has packages/alchemy/test/GCP/{Service}/{Resource}.test.ts (1051/1051).

Live recapture (188 dirs)

Per-service timeout 240 pnpm test test/GCP/{dir} --profile testing --retry 0. The full test/GCP tree cannot finish in a 240s wall (~2124 tests).

  • 188/188 dirs PASS after targeted retests (skipIf FAST/creds/entitlement todos are not failures)
  • Compute whole-dir hits the 240s wall. File-batch recapture: 70 PASS / 2 GATED / 0 FAIL of 72 files
  • Skeptic Compute recapture (no GCP_TEST_*): NetworkFirewallPolicy 2/2, RegionNetworkFirewallPolicy 2/2, NetworkAttachment 2/2, RegionCompositeHealthCheck 3/3

Bindings census

192 Binding.Service exports, each with a live Action/HTTP invocation in Bindings.test.ts. Memorystore also has ReadRedis / WriteRedis / ReadWriteRedis over alchemy/Redis.

Nuke residue (catalogued)

pnpm nuke:gcp --dry-run after the recapture:

GCP.Analyticshub.DataExchange  2 to delete
GCP.Analyticshub.DataExchangesListing  2 to delete
GCP.KMS.CryptoKey  2 to delete
6 resource(s) to delete.
  • Analytics Hub listings/exchanges: GCP HTTP 500 INTERNAL when replicaState: UNAVAILABLE
  • KMS crypto keys: DESTROY_SCHEDULED (24h; no immediate delete API)

Introduce Alchemy GCP providers for Compute, GKE, Storage, Pub/Sub, Cloud Run, SQL, and related services, with HTTP bindings, a gcp-gke Kubernetes adapter, and `pnpm nuke:gcp`.
@alchemy-version-bot

alchemy-version-bot Bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Install the packages built from this commit:

Alchemy

alchemy

bun add https://pkg.ing/alchemy/c5229a6

@alchemy.run/better-auth

bun add https://pkg.ing/@alchemy.run/better-auth/c5229a6

@alchemy.run/cloudflare-runtime

bun add https://pkg.ing/@alchemy.run/cloudflare-runtime/c5229a6

@alchemy.run/frontend-frameworks

bun add https://pkg.ing/@alchemy.run/frontend-frameworks/c5229a6

@alchemy.run/node-utils

bun add https://pkg.ing/@alchemy.run/node-utils/c5229a6

@alchemy.run/pr-package

bun add https://pkg.ing/@alchemy.run/pr-package/c5229a6

@alchemy.run/floci

bun add https://pkg.ing/@alchemy.run/floci/c5229a6

Distilled

@distilled.cloud/core

bun add https://pkg.ing/@distilled.cloud/core/cecf35f

@distilled.cloud/aws

bun add https://pkg.ing/@distilled.cloud/aws/cecf35f

@distilled.cloud/axiom

bun add https://pkg.ing/@distilled.cloud/axiom/cecf35f

@distilled.cloud/cloudflare

bun add https://pkg.ing/@distilled.cloud/cloudflare/cecf35f

@distilled.cloud/hetzner

bun add https://pkg.ing/@distilled.cloud/hetzner/cecf35f

@distilled.cloud/neon

bun add https://pkg.ing/@distilled.cloud/neon/cecf35f

@distilled.cloud/planetscale

bun add https://pkg.ing/@distilled.cloud/planetscale/cecf35f

@sam-goodwin

Copy link
Copy Markdown
Contributor

Please update PR description with DX (heading + code snippet showing resources, effectful sevices, bindings) etc.

Point the submodule at alchemy-run/distilled#475 so Alchemy can consume the GCP credentials DX without waiting for merge.
@sam-goodwin

Copy link
Copy Markdown
Contributor

Can you show me a GCP.Function with bindings (something for twitter)

Comment thread packages/alchemy/src/GCP/Redis/Instance.ts
Factory coverage for distilled GCP CRUD: observe-ensure-sync resources,
HTTP Binding.Service layers, live tests, and pnpm nuke:gcp.
@Mkassabov Mkassabov changed the title feat(gcp): add first-wave resource providers feat(gcp): add GCP resource providers, HTTP bindings, and nuke Aug 26, 2026
Vertex missing-resource probes return BadRequest (and list on the
global endpoint); gate Vertex lifecycles behind GCP_TEST_AIPLATFORM.
Network wait now uses the operation id, surfaces quota errors, and
retries NETWORKS quota. Regional backend buckets default to
EXTERNAL_MANAGED and omit CDN/header fields GCP rejects. Instant
snapshots retry the 30s-too-recent insert.
Labels cannot be set on insert; a crash before setLabels left an
LbEdgeExtension forwarding rule invisible to list. Stamp ownership in
the description, treat rules that target an Alchemy HTTP(S) proxy as
owned, and delete forwarding rules before proxies/url maps. Listing
delete always sends deleteCommercial so Analytics Hub commercial
listings can be torn down when the API allows it.
Delete was using persisted ACTIVE and retrying BadRequest while the
bucket was already DELETE_REQUESTED or still UPDATING after a link
change. Observe live lifecycle, wait out pending, and treat
DELETE_REQUESTED as gone. Billing saved-query probes now catch
BadRequest when the billing account type does not support them.
…s gone

Regional Secret Manager needs a regional endpoint the global SDK cannot
reach, so lifecycle tests stay behind GCP_TEST_REGIONAL_SECRETS with a
probe that asserts BadRequest. Backup-schedule GET after the parent
database is deleted returns BadRequest instead of NotFound; treat that
as gone and retry create while the database is still coming up.
Treat missing DLP/CCI parents as gone, prefix DLP job ids with i-/r-,
drop invented Service Management error tags, and interpolate Dataplex
bucket names as Outputs. Gate API Hub, Integrations, Tag Manager, and
Datastream private-connection lifecycles behind entitlement env vars;
Pub/Sub Lite probes accept sunset NotFound.
Treat Translate glossary-entry list BadRequest as empty, skip malformed
Wasm plugin version names on delete, and keep ValidateAttestation from
re-yielding Output. Skip Dataplex lake, NCC hub/spoke, intercept/mirror,
Workstations cluster, and wasm version lifecycles unless the matching
entitlement env is set so per-service timeout 240 stays skip-clean.
Data taxonomy sunset probes also accept TooManyRequests. DataScan and
remaining Dataplex create tests stay behind GCP_TEST_DATAPLEX. Client
TLS policy delete can exceed 90s; wait treats BadRequest as gone.
catchTag('BadRequest') on get/list ops whose union is only NotFound |
Forbidden | GcpOpError failed tsc. Keep those as NotFound/Forbidden and
leave BadRequest on create where the SDK actually types it.
Comment thread packages/alchemy/src/GCP/AIPlatform/BindingHttp.ts Outdated
Comment thread packages/alchemy/src/GCP/AIPlatform/BindingHttp.ts Outdated
Comment thread packages/alchemy/src/GCP/Run/Service.ts
Comment thread packages/alchemy/src/GCP/Run/Job.ts
Stop GET-after-destroy on Firestore backup schedules (parent DB gone
returns untyped BadRequest). Wait out Logging locations-link LROs,
gate billing saved queries behind GCP_TEST_BILLING_SAVED_QUERY, skip
incomplete/global glossary parents on Translate read, and give TlsRoute
delete 180s.
Treat a missing bucket as gone during force-destroy, wait out backend
bucket insert LROs, retry instance delete while STOPPING, and give
network replace-delete 180s. Gate slow gen2 functions, instance stop
bindings, and zone-capacity reservations behind env flags with probes
kept ungated.
Catch OperationPending on BackendService global ops (TargetSslProxy
create) and give Subnetwork/TargetSslProxy 180s so VPC teardown can
finish.
…obes

Vertex GET/list on us-central1 often returns BadRequest (global-only
endpoint or AgentService v1). Assert that tag and skip the follow-up
list so the probes stay typed without catching untyped list errors.
Phrase matcher ids are server-assigned; GET of a generated name returns
BadRequest. Observe via output name or displayName. Gate Service
Management delete LROs behind GCP_TEST_SERVICEMANAGEMENT.
Create+destroy already completed inside 120s; waitUntilGone after
destroy blew the timeout. Isolated retest passed in 117.8s.
Add Action-driven live calls for the Binding.Service ops that had no
test presence, including skipIf/entitlement probes that record the
typed GCP error when the parent resource cannot be created.
Network insert now retries any NetworkOperationFailed (quota/race),
not only quota-shaped messages. Instant snapshots wait longer for the
30s source-disk cooldown. FAST skips HA/Classic VPN, firewall,
instant snapshot, and Artifact Registry attachment lifecycles.
Wait longer for Compute image, instance, router, and snapshot
operations, and retry Analytics Hub deletes on replica 500s.
Bump slow lifecycle test timeouts so per-service recapture stays
under the 240s wall.
Every Providers.ts-registered GCP Resource now has a dedicated
{Service}/{Resource}.test.ts: probe GET on a missing row, plus
create/update/delete skipIf-gated on creds, FAST, or entitlement.
RegionTargetTcpProxy has no setBackendService API. Reconcile now
deletes and recreates when the observed backend, proxy header, or
other immutable fields drift so an engine update still converges.
Network firewall insert cannot send rules; syncRules now leaves GCP's
reserved goto_next priorities (2147483548–2147483647) alone. Network
attachments derive name/selfLink when GET omits them and replace when
connectionPreference changes. Health sources expand to regional
backend-service URLs.
Cloud Run Service/Job (and Cloud Functions) are Platforms with a
{ env, iam } binding contract. HTTP bindings yield the distilled op
once (no double provide) and grant a role on the runtime service
account, matching AWS policyStatements / Cloudflare AccountApiToken.

Memorystore Read/Write/ReadWrite uses alchemy/Redis over REDIS_URL.
GCP.Function is an alias of Run.Service for the Effect-native DX.
@Mkassabov

Copy link
Copy Markdown
Contributor Author

Updated the PR description with an Effect-native `GCP.Function` (Cloud Run Service) that binds Pub/Sub + Memorystore Redis — the twitter-bot shape. `GCP.Function` is an alias of `GCP.Run.Service`.

Sam's spot-check was the standard for the whole GCP surface, not three
files. HTTP bindings no longer double-provide Credentials/HttpClient,
and every Binding.Service host.binds a role onto Cloud Run / Function
(AWS policyStatements / Cloudflare AccountApiToken). WorkerPool is the
same effectful host as Service/Job.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants