Omit the Avro namespace from Storage Read API schemas - #508
Open
laughingman7743 wants to merge 1 commit into
Open
Omit the Avro namespace from Storage Read API schemas#508laughingman7743 wants to merge 1 commit into
laughingman7743 wants to merge 1 commit into
Conversation
The schema a read session returns carried a namespace built from the
project and dataset ids. A BigQuery project id may contain a hyphen,
which is not a legal character in an Avro name, so every read against
such a project failed in the client's schema parser before a row was
decoded:
Record ought to have valid name: schema name ought to have second
and remaining characters contain only [A-Za-z0-9_]: test-project
BigQuery itself returns no namespace, so omitting it also brings the
emulator closer to the service. The value encoder already handled an
empty namespace: avroRecordUnionKey falls back to the bare record name,
which is the union branch key goavro expects.
Fixes goccy#221
This was referenced Aug 9, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #221.
What
TableToAVRObuilt the schema'snamespacefrom the project and dataset ids. A BigQuery project idmay contain a hyphen, which is not a legal character in an Avro name, so a read session against such
a project returned a schema no Avro client could parse — the failure landed in the client's schema
parser, before any row was decoded:
The namespace is now omitted, which is also what BigQuery does: a real read session answers with
{"type":"record","name":"__root__","fields":[…]}and no namespace (checked against the service on2026-08-09).
Why this is safe for the value encoder
AVROSchema.Namespaceis read in one other place,(*TableCell).AVROValue, which passes it toavroRecordUnionKeyfor a nullableRECORDfield. That helper already returns the bare record namewhen the namespace is empty — which is the union branch key goavro expects — so nullable STRUCT
columns keep working.
TestStorageReadAVROcovers that path and still passes.I left the record
Nameas the table id rather than also matching BigQuery's__root__, since thatis a separate behavioural change and not what breaks here; happy to add it if you would prefer the
closer match.
Test
TestIssue221ReadSessionSchemaParsesForHyphenatedProjectloads a project namedtest-project,creates a read session, parses the returned schema with
goavroand then decodes the rows thestream delivers. Reverting the change makes it fail with the message quoted above.
go test ./server ./types ./internal/...passes.