slim: remove every connector and dependency live-ingestion does not use - #6
Open
almostintuitive wants to merge 3 commits into
Open
almostintuitive wants to merge 3 commits into
almostintuitive wants to merge 3 commits into
Conversation
live-ingestion (aperiodic-io/live-ingestion) runs this fork as its raw and metrics archivers, and nothing else: its 133 rendered configs use the kafka_franz input; the aws_s3, broker, switch and reject_errored outputs; the protobuf_parquet_encode and json_parquet_encode processors; the prometheus metrics exporter; and Bloblang's meta(), timestamp_unix_nano(), uuid_v4(), split(), index() and has_suffix(). The 51 internal/impl packages none of those live in go, with their public/components packages, and so does the Lambda build (cmd/serverless, its Makefile target and goreleaser builds). What is left: aws, io, kafka, parquet, prometheus, protobuf and pure. go.mod falls from 456 lines to 187. Tests of what went, went with it. Tests of what stays that used what went: - the config linter's deprecation warnings were checked against amqp_1's url field and the sql output; no component left is deprecated, so the tests register a test input with a deprecated field and a deprecated test output; - the AWS MSK IAM SASL test goes with kafka/aws, which provided the mechanism; - a public/service Bloblang benchmark made its rows with lang's fake(), now uuid_v4(). Component examples that named removed components (sql_select, redis, memcached, nats_jetstream, amqp_1, gcp_pubsub, gcp_bigquery, redis_streams, the avro scanner) name kept ones instead, as the docs tests require. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The packages that hold what live-ingestion uses held much else besides: - aws: only the aws_s3 output (and the session config it reads) is left; the S3 input, cache and processor, SQS, SNS, Kinesis, Firehose, DynamoDB, Lambda, Bedrock and CloudWatch go. - kafka: only the kafka_franz input is left; the sarama kafka input and output and the kafka_franz output go. Its SASL keeps PLAIN, OAUTHBEARER and SCRAM, which franz-go carries; AWS_MSK_IAM, whose implementation went with kafka/aws, and GSSAPI (Kerberos) go. - protobuf: only protobuf_parquet_encode is left; the protobuf processor and its Buf Schema Registry watcher go. - parquet: the parquet input and parse_parquet go. parquet_encode and parquet_decode stay: json_parquet_encode's parity tests run parquet_encode, and parquet_encode's tests read it back with parquet_decode. - io: stdin, stdout, the file input and output and the env/file/hostname functions are left, as the defaults and building blocks the core and its tests use; HTTP, sockets, websockets, subprocesses, fsnotify, CSV, dynamic and the file cache and processor go. - pure: the components that bring a library of their own go (the lru and ttlru caches, grok, jmespath, jq, json_schema, parse_log, template, unarchive and workflow processors, the xml scanner, the logger metrics exporter, pure/extended's zstd). What is left is dependency-free and holds broker, switch and reject_errored, which live-ingestion uses, and the building blocks the core's own tests are written with. - internal/httpclient, internal/tracing/v2, internal/docs/interop and internal/component/input/config, used only by what went, go. go.mod is down to 132 lines from 456; sarama, gokrb5 and SPNEGO, the Buf registry client, gojq, jmespath, sprig, mxj and the websocket library are gone. Tests of what went, went with it. The core's own tests that used what went as a fixture use kept components instead: reject and file outputs for the http_client ones (drop_on, fallback, reject_errored), a file input with the csv scanner for the csv input (sequence), mapping for jmespath (try), memory caches for file caches (the stream API's resources), broker for dynamic (stream builder walk), and sleep for jmespath as an empty processor. Where no kept component could stand in, the test registers its own: a processor that runs a test function (parallel), one that splits a JSON array (cached), and an output that never connects (the stream API's readiness). The kafka_franz integration test that produced through the kafka_franz output goes, and so do the drop_on back-pressure tests, skipped and written against websocket. Component examples naming removed components name kept ones, or go where they only showed a removed one (HTTP enrichment, Lambda). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The 192 generated component pages of components that went are deleted; bento_docs_gen regenerates the rest, and the Bloblang function and method references. - Guides, configuration pages and component overviews whose config snippets named removed components name kept ones (kafka_franz for kafka, aws_s3, file or stdout for outputs, cache for http processors, memory for memcached and redis caches, none for jaeger). Pages and sections that only documented something removed go: the sync responses, GCP and Azure guides, dynamic inputs and outputs, Bloblang message expansion (unarchive), streams mode's HTTP endpoints. Links to removed pages are gone; a scan of website/, internal/, public/ and cmd/ finds none left other than three the upstream docs already had broken. - Cookbooks and example configs built on removed connectors (custom metrics, enrichments, Kafka topic mirroring, joining streams, the Discord bot, Redshift, site analytics, tracking downloads) go; the landing page's snippets use kept components. - Template examples: the SQS list template becomes a kafka_franz topic list, the dead letter one wraps aws_s3, the alias one aliases sleep. - bento test configs (config/test): the awk, lru, ttlru, protobuf and message expansion tests go; the mock tests mock a sleep instead of an http processor, which they never reach either way. - resources/: the Lambda deployment files and the schema registry and tracing docker setups go; the profiling setup drops to a drop output. schemastore/bento.json is regenerated (1.97 MB -> 201 KB). - public/wasm, the guest SDK for the wasm processor, goes. - README says up front what this fork carries. make docs' lint of config/examples, cookbooks and docs, template lint and bento test ./config/test/... all pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #5. This fork exists to run live-ingestion's archivers, so this PR removes every connector and dependency that live-ingestion doesn't use, together with their tests and docs.
What live-ingestion uses. This was taken from all 133 Bento configs rendered in its
deploy/gitops(raw and metrics archivers, on staging, production and local):kafka_franzaws_s3,broker,switch,reject_erroredprotobuf_parquet_encode,json_parquet_encodeprometheusmeta(),timestamp_unix_nano(),uuid_v4(),split(),index(),has_suffix()What's left:
aws,kafka,protobuf,parquet);prometheus;pure(which holdsbroker,switchandreject_errored) andio(stdin,stdout,file, and theenv/file/hostnamefunctions).parquet_encodeandparquet_decodestay only becausejson_parquet_encode's parity tests runparquet_encodeas the reference, andparquet_encode's round-trip tests read back throughparquet_decode.Result: the image goes from 265 MB to 69 MB,
go.modfrom 456 lines to 132, and 989 files change (about 252k lines deleted). Among the dependencies dropped: sarama, gokrb5/SPNEGO, the Buf registry client, the AWS SDK services other than S3/STS, the GCP and Azure SDKs, and the database drivers.Commits
internal/implpackages go, with theirpublic/componentswrappers and the Lambda build.aws: only theaws_s3output (plus its session config).kafka: only thekafka_franzinput. Its SASL keeps PLAIN, OAUTHBEARER and SCRAM;AWS_MSK_IAMand GSSAPI go.protobuf: onlyprotobuf_parquet_encode.io: HTTP, sockets, websockets, subprocesses and the like go.pure: components that bring their own library go (lru/ttlrucaches;grok,jmespath,jq,json_schema,parse_log,template,unarchive,workflowprocessors;xmlscanner;loggermetrics).config/test,resources/andschemastoreare updated to match.Tests
Tests of removed components are removed. Core tests that used removed components only as fixtures now use kept ones:
drop_on,fallback,reject_erroredhttp_clientoutputrejectandfileoutputssequencecsvinputfileinput with thecsvscannertryjmespathmappingfilecachesmemorycachesdynamicinputbrokerinputWhere no kept component could stand in, the test registers a small one of its own through the public API: a processor that runs a test function (for
parallel), one that splits a JSON array (forcached), and an output that never connects (for the stream API's readiness check). The config linter's deprecation-warning tests register a deprecated test input and output, since no remaining component is deprecated.Two groups of tests went because they can't run without removed components:
kafka_franzintegration test that produced through thekafka_franzoutput (Docker-only, never run in CI);drop_onback-pressure tests, which were already skipped and relied onwebsocket.Test plan
go build ./...,go vet ./...go test ./...: all pass.TestRetryParallel(timing-based, untouched) failed once under full-suite load, then passed 20 of 20 runs here and 20 of 20 on the base branch../...: 0 issuesmake docs' lints (bento lintofconfig/examples, cookbooks andwebsite/docs),bento template lint, andbento test ./config/test/...(27 tests): all passwebsite/,internal/,public/andcmd/: nothing links to a removed page, apart from three links upstream already had brokenTestComponentExamples)perf/**trigger:ghcr.io/aperiodic-io/bento:3af13c4a9fis 68.8 MB, against 265.2 MB for parquet: json_parquet_encode allocates a third as often, reuses writers, caches by metadata #5'sb7cc98399f.bento listin the image shows only the kept components.bento lintis clean for all 133 rendered configs on its main, and for the 133 in the pending integration branch, 64 of which usecache_by(withR2_ENDPOINTset, as production sets it).🤖 Generated with Claude Code