Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/release.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ jobs:
run: gh release create "$GITHUB_REF_NAME" --title "Release $GITHUB_REF_NAME" --generate-notes
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# The BSR remote plugin, so consumers can use `remote: buf.build/roderm/
# The BSR remote plugin, so consumers can use `remote: buf.build/snaerverk/
# go-sqlmap` instead of installing the binary. Runs after `release` because
# buf.plugin.yaml's license_url points at the tag's LICENSE.
plugin:
Expand Down
8 changes: 4 additions & 4 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,22 +6,22 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co

`protoc-gen-go-sqlmap` is a `protoc`/`buf` code-generator plugin. Given `.proto` messages annotated with `sqlmap` extensions (table/column metadata, primary keys, foreign keys), it emits a `<file>.sqlmap.go` per proto file containing `schema.v1.SchemaTable`/`SchemaColumn` values plus row-scanning helpers. `pkg/migration` then converts those values into `ariga/atlas` schema types to create tables or write migration files.

The goal is a set of independently switchable plugins over one shared `TableRepo`: (1) atlas migrations — **done**; (2) queries with eager loading of children + FieldMask support — **done**; (3) SQL filters built from the filtrify AST (`github.com/roderm/filtrify`, `proto/filtrify/ast/v1/ast.proto`). Note `docs/plans/sqlmap-generator/PLAN-SQLMAP-GENERATOR.md` is stale on (3): its Phase 5 sketches ad-hoc per-column builder funcs rather than translating an external AST.
The goal is a set of independently switchable plugins over one shared `TableRepo`: (1) atlas migrations — **done**; (2) queries with eager loading of children + FieldMask support — **done**; (3) SQL filters built from the filtrify AST (`github.com/snaerverk/filtrify`, `proto/filtrify/ast/v1/ast.proto`). Note `docs/plans/sqlmap-generator/PLAN-SQLMAP-GENERATOR.md` is stale on (3): its Phase 5 sketches ad-hoc per-column builder funcs rather than translating an external AST.

Plugins are toggled with the `plugins=` protoc parameter, a **`+`-separated** list (protoc already uses `,` between parameters): `--go-sqlmap_out=plugins=schema+scanner:.`. Omitting it enables everything. `plugins` entries live in one slice in `generator.go`; `query` declares `Requires: ["scanner"]` because its output calls the scanner's `Result` type, and enabling it alone is a generation-time error rather than uncompilable Go.

### BSR remote plugin

`buf.plugin.yaml` + `Dockerfile` publish the generator as `buf.build/roderm/go-sqlmap`, so consumers need no local binary:
`buf.plugin.yaml` + `Dockerfile` publish the generator as `buf.build/snaerverk/go-sqlmap`, so consumers need no local binary:

```yaml
plugins:
- remote: buf.build/roderm/go-sqlmap:v0.1.0
- remote: buf.build/snaerverk/go-sqlmap:v0.1.0
out: pkg/generated/
opt: [paths=source_relative, plugins=schema+scanner+query]
```

`buf.plugin.yaml` pins the runtime module the generated code imports (`pkg/query`, `pkg/generated/schema/v1`), so `plugin_version` and `registry.go.deps` must both match the git tag — the release workflow fails if they drift. Publishing happens in CI on tag (`buf beta registry plugin push`, needs `BUF_TOKEN`); it is a separate resource from the `buf.build/roderm/protoc-gen-go-sqlmap` *module* that carries the protos. Test the image locally with `docker build -t x . && protoc --plugin=protoc-gen-y=<script running 'docker run --rm -i x'>`.
`buf.plugin.yaml` pins the runtime module the generated code imports (`pkg/query`, `pkg/generated/schema/v1`), so `plugin_version` and `registry.go.deps` must both match the git tag — the release workflow fails if they drift. Publishing happens in CI on tag (`buf beta registry plugin push`, needs `BUF_TOKEN`); it is a separate resource from the `buf.build/snaerverk/protoc-gen-go-sqlmap` *module* that carries the protos. Test the image locally with `docker build -t x . && protoc --plugin=protoc-gen-y=<script running 'docker run --rm -i x'>`.

**Dependency rule: only `ariga.io/atlas` and `google.golang.org/protobuf` are allowed as direct dependencies.** Ask before adding any other, test-only ones included — prefer plain stdlib loops over utility libraries. Target Go 1.27.

Expand Down
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Image for the BSR remote plugin (buf.build/roderm/go-sqlmap).
# Image for the BSR remote plugin (buf.build/snaerverk/go-sqlmap).
# Built and pushed by `buf beta registry plugin push`, which requires
# linux/amd64, a minimal base and a non-root user.
FROM --platform=$BUILDPLATFORM golang:1.27-alpine AS build
Expand Down
195 changes: 136 additions & 59 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,75 +1,152 @@
# protoc-gen-go-sqlmap:
[![test](https://github.com/roderm/protoc-gen-go-sqlmap/actions/workflows/test.yaml/badge.svg)](https://github.com/roderm/protoc-gen-go-sqlmap/actions/workflows/test.yaml)
This is a protobuf-plugin to generate SQL-Select mappers in go to insert, update and query protbuf messages directly from tables or views.
# protoc-gen-go-sqlmap

[![test](https://github.com/snaerverk/protoc-gen-go-sqlmap/actions/workflows/test.yaml/badge.svg)](https://github.com/snaerverk/protoc-gen-go-sqlmap/actions/workflows/test.yaml)
[![Go Reference](https://pkg.go.dev/badge/github.com/snaerverk/protoc-gen-go-sqlmap.svg)](https://pkg.go.dev/github.com/snaerverk/protoc-gen-go-sqlmap)

`protoc`/`buf` plugin that turns `.proto` messages annotated with `sqlmap` extensions (table/column metadata, primary keys, foreign keys) into a `<file>.sqlmap.go` per proto file, containing:

- **`schema`** — `schema.v1.SchemaTable`/`SchemaColumn` values describing every table, consumed by [`pkg/migration`](pkg/migration) to create tables (via [ariga/atlas](https://atlasgo.io)) or generate migration files.
- **`scanner`** — a `<Message>Result` struct per table with `Scan`, `GetColValue`, and column-list helpers, for reading rows into Go without reflection.
- **`query`** — FieldMask-aware `Load<Message>`/`Load<Message>Rows` functions that select only requested columns and eagerly load related messages (one batched `IN (...)` per relation, stitched in Go), consumed by [`pkg/query`](pkg/query).

These are independently switchable via the `plugins=` parameter (see below); `schema`, `scanner`, and `query` are the three plugins that exist today.

## Install

## Usage:
Install with:
```bash
go install github.com/roderm/protoc-gen-go-sqlmap
go install github.com/snaerverk/protoc-gen-go-sqlmap/cmd/protoc-gen-go-sqlmap@latest
```
### protoc

### protoc

```bash
protoc --go-sqlmap_out=paths=source_relative:. your.proto
```

Optional `plugins=` parameter (comma-separated, protoc-style) selects which of `schema`, `scanner`, `query` to emit, `+`-separated:

```bash
protoc --go-sqlmap_out=. [your-protobuf-definition].proto
protoc --go-sqlmap_out=plugins=schema+scanner+query,paths=source_relative:. your.proto
```

Omit `plugins=` to enable all three. `query` requires `scanner` to also be enabled, since its output calls the scanner's `Result` type. The schema writer emits a `SchemaType` entry for every dialect (mysql/postgres/sqlite3) regardless — the actual dialect is chosen at runtime by `migration.New`, not at generation time.

### buf
* add dependency `buf.build/roderm/protoc-gen-go-sqlmap`
* add `go-sqlmap` and `go` to your `buf.gen.yaml`

To make use of this plugin, add following extensions to you .proto-file:
Add the plugin to `buf.gen.yaml`, either from BSR (no local install needed):

```yaml
plugins:
- remote: buf.build/snaerverk/go-sqlmap:v0.1.0
out: pkg/generated/
opt: [paths=source_relative, plugins=schema+scanner+query]
```

or as a local binary after `go install`:

```yaml
plugins:
- local: protoc-gen-go-sqlmap
out: pkg/generated/
opt: [paths=source_relative]
```

## Annotating messages

Import `sqlmap/v1/sqlmap.proto` and annotate tables and columns:

```proto
import "github.com/roderm/protoc-gen-go-sqlmap/proto/sqlgen/sqlgen.proto" // Import of the extensions
// or with buf
import "proto/sqlgen/sqlgen.proto"

message Some {
option (sqlgen.table) = {name:"tbl_some", crud:[R]}; // table name and generate read-function
string Id = 1 [(sqlgen.col) = {pk: AUTO, name = "some_id" }]; // It's an autogenerated key in the field "some_id"
repeated Attribute Attributes = 2; [(sqlgeb.col) = {name "some_id",fk = "tbl_some_attributes.some_id"}]; // Make a one-to-many link to load
syntax = "proto3";

package example;

import "sqlmap/v1/sqlmap.proto";

option go_package = "example/examplepb";

message Author {
option (sqlmap.v1.table) = { name: "tbl_author" };
int64 id = 1 [(sqlmap.v1.col) = { fieldname: "author_id", pk: PK_AUTO }];
string name = 2 [(sqlmap.v1.col) = { fieldname: "author_name" }];

// has-many: the key lives on tbl_book (author_id), not here.
repeated Book books = 3 [(sqlmap.v1.col) = {
foreign_key: { entity: "Book", fieldnames: ["author_id"] }
}];
}

message Attribute {
option (sqlgen.table) = {name:"tbl_some_attributes", crud:[C,R,D]}; // genereate insert,select and delete function
string Id = 1 [(sqlgen.col) = {pk= AUTO, name = "attributes_id" }];
string Name = 2 [(sqlgeb.col).name = "attributes_name"];
bytes Value = 3 [(sqlgeb.col).name = "attributes_value"];
Some Parent = 4 [(sqlgeb.col) = {name: "some_id", fk = "tbl_some.some_id"}]; // Make a one-to-one link to load
message Book {
option (sqlmap.v1.table) = {
name: "tbl_book"
foreign_keys: [{
fieldnames: ["author_id"]
to: { entity: "Author", fieldnames: ["author_id"] }
on_delete: ON_DELETE_CASCADE
}]
};
int64 id = 1 [(sqlmap.v1.col) = { fieldname: "book_id", pk: PK_AUTO }];
string title = 2 [(sqlmap.v1.col) = { fieldname: "book_title" }];
// proto3 optional gives this field presence, so it's nullable; a bare
// scalar has no presence and would generate NOT NULL instead.
optional int64 author_id = 3 [(sqlmap.v1.col) = { fieldname: "author_id" }];
}
```

After generation of the .sqlmap.go-file you should be able to load the data as following:
- `pk: PK_AUTO` for a database-generated key, `PK_MAN` for one the application supplies.
- A `message`-kind field with `foreign_key` is a relation: `repeated` is has-many (relation only, no column); a singular field is belongs-to (both a column holding the key and a relation).
- Nullability: an explicit `nullable` on the column wins; otherwise PKs are `NOT NULL` and every other column follows the proto field's presence — a `proto3 optional` or message-kind field has presence and is nullable, a bare proto3 scalar does not and becomes `NOT NULL`.
- A `message`-kind field *without* a `foreign_key` is stored as an embedded `JSON` column.

See [`proto/sqlmap/v1/sqlmap.proto`](proto/sqlmap/v1/sqlmap.proto) for the full extension surface (`ForeignKeyDefinition`, per-dialect `type` overrides) and [`docs/design/DESIGN-SUBTYPE-TABLES.md`](docs/design/DESIGN-SUBTYPE-TABLES.md) for joined-table subtype hierarchies (`oneof` + `(sqlmap.v1.subtypes)`).

## Using the generated code

```go
store = package.NewStore(db_conn)
rows, err = store.Some(context.TODO(), package.SomeWithAttributes())
db, _ := sql.Open("postgres", dsn)

// Create tables from the generated schema (or use m.Diff/ApplyPending for
// migration files instead).
m, err := migration.New(db, migration.DialectPostgres)
if err != nil {
return err
}
if err := m.Create(ctx, examplepb.AuthorTable, examplepb.BookTable); err != nil {
return err
}

// Load authors with their books' titles eagerly loaded, selecting only the
// columns the FieldMask asks for.
conn := query.Conn{DB: db, Dialect: query.Postgres}
authors, err := examplepb.LoadAuthor(ctx, conn,
&fieldmaskpb.FieldMask{Paths: []string{"name", "books.title"}})
```

`LoadAuthor` also takes `...query.Cond` filters (build one with `query.In(col, values)`, which reports whether the value slice was non-empty). For row-level access without eager loading, use `<Message>Result.Scan` directly against `GetAuthorColumns()`/`GetAuthorPKColumns()`.

## Commands

```bash
task install # go mod tidy && go install ./cmd/protoc-gen-go-sqlmap/
make install # buf generate && go install (regenerates pkg/generated/** from proto/)
buf generate # regenerate pkg/generated/**.pb.go from proto/
go build ./...
go test ./... # golden + unit tests
go test ./pkg/generator/sqlmap/ -update # accept new golden output
SQLMAP_E2E=1 go test ./pkg/generator/sqlmap/ -run TestE2E # dockerized-Postgres end-to-end
```

Additional there can be given extra parameters for filtering (Currently only forks for Postgres/Cockroach):
`store.Some(context.TODO(), SomeFilter(pg.EQ("some_id", "some_id"))`
or for the sub-query:
`store.Some(context.TODO(), package.SomeWithAttributes(AttributeFilter(pg.EQ("name", "FirstArgument")))`

## [Docs](https://pkg.go.dev/github.com/roderm/protoc-gen-go-sqlmap)
## Roadmap:
- [-] Queries
- [x] loading of one-to-many
- [x] loading of one-to-one
- [ ] add parameter to not resolve references (problem with json-encoding)
- [-] Insert
- [x] Insert single message to table
- [ ] resolve foreign-key (and create)
- [x] Delete
- [x] delete a data-row
- [ ] Implement update
- Should have use replace and conflict handling parameter
- [x] Decouple gogo
- [x] Write generator test
- [ ] Support multiple PKs
- [ ] Add multiple FKs for single Message
- [ ] Support `oneOf` type in proto3
- [ ] Improve filter to no use column-names
- [x] Load external messages to resolve foreign keys
- [ ] Implement more Database-Engines & syntax's
- [ ] CockroachDB
- [x] PostgreSQL
- [ ] MySQL / MariaDB
- [ ] MSSQL
- [ ] SQLite
## Docs

- [pkg.go.dev](https://pkg.go.dev/github.com/snaerverk/protoc-gen-go-sqlmap)
- [`docs/design/DESIGN-SUBTYPE-TABLES.md`](docs/design/DESIGN-SUBTYPE-TABLES.md) — joined-table subtype hierarchies.
- [`docs/design/DESIGN-FILTRIFY-CONDITIONS.md`](docs/design/DESIGN-FILTRIFY-CONDITIONS.md) — planned: translating a [filtrify](https://github.com/snaerverk/filtrify) AST into SQL filters at runtime.

## Roadmap

- [x] Schema generation + atlas-backed migrations (create tables, diff, migration files)
- [x] FieldMask-aware queries with eager loading (has-many and belongs-to, arbitrary depth)
- [x] Joined-table subtypes (`oneof` enforced as a DB-level FK/CHECK constraint)
- [x] Multiple dialects: PostgreSQL, MySQL/MariaDB, SQLite
- [x] Published as a BSR remote plugin (`buf.build/snaerverk/go-sqlmap`)
- [ ] SQL filters translated at runtime from a [filtrify](https://github.com/snaerverk/filtrify) AST, replacing ad-hoc `query.Cond` construction
- [ ] Insert/update helpers (currently out of scope; `pkg/migration` only creates schema, `pkg/query` only reads)
4 changes: 2 additions & 2 deletions buf.gen.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,15 @@ managed:
enabled: true
override:
- file_option: go_package_prefix
value: github.com/roderm/protoc-gen-go-sqlmap/pkg/generated
value: github.com/snaerverk/protoc-gen-go-sqlmap/pkg/generated
plugins:
- remote: buf.build/protocolbuffers/go
out: pkg/generated
opt:
- paths=source_relative
- Mgoogle/type/datetime.proto=google.golang.org/genproto/googleapis/type/datetime

# - remote: buf.build/roderm/sqlmap
# - remote: buf.build/snaerverk/sqlmap
# out: pkg/generated/sqlmap
# opt:
# - dialect=mysql
8 changes: 4 additions & 4 deletions buf.plugin.yaml
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
version: v1
name: buf.build/roderm/go-sqlmap
name: buf.build/snaerverk/go-sqlmap
# Keep in step with the git tag: registry.go.deps below pins the runtime
# packages the generated code imports, and they ship from this repo.
plugin_version: v0.1.0

description: Generates SQL schema definitions, row scanners and FieldMask-aware queries with eager loading from sqlmap-annotated messages.
source_url: https://github.com/roderm/protoc-gen-go-sqlmap
source_url: https://github.com/snaerverk/protoc-gen-go-sqlmap
spdx_license_id: MIT
license_url: https://github.com/roderm/protoc-gen-go-sqlmap/blob/v0.1.0/LICENSE
license_url: https://github.com/snaerverk/protoc-gen-go-sqlmap/blob/v0.1.0/LICENSE
output_languages:
- go

Expand All @@ -22,7 +22,7 @@ registry:
deps:
# pkg/generated/schema/v1 (SchemaTable/SchemaColumn) and pkg/query
# (Mask, Conn) are imported by the generated code.
- module: github.com/roderm/protoc-gen-go-sqlmap
- module: github.com/snaerverk/protoc-gen-go-sqlmap
version: v0.1.0
- module: google.golang.org/protobuf
version: v1.36.11
2 changes: 1 addition & 1 deletion buf.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
version: v2
modules:
- path: proto/
name: buf.build/roderm/protoc-gen-go-sqlmap
name: buf.build/snaerverk/protoc-gen-go-sqlmap
deps:
- buf.build/googleapis/googleapis
lint:
Expand Down
2 changes: 1 addition & 1 deletion cmd/protoc-gen-go-sqlmap/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import (
"io"
"os"

"github.com/roderm/protoc-gen-go-sqlmap/pkg/generator/sqlmap"
"github.com/snaerverk/protoc-gen-go-sqlmap/pkg/generator/sqlmap"
"google.golang.org/protobuf/compiler/protogen"
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/types/pluginpb"
Expand Down
2 changes: 1 addition & 1 deletion docs/design/DESIGN-SUBTYPE-TABLES.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Status: **implemented** for the schema half — the discriminator column, CHECK constraints, unique index and composite foreign key are generated and enforced. Eager loading of subtypes (the section near the end) is **not** implemented yet.

Consumers pulling the extensions from `buf.build/roderm/protoc-gen-go-sqlmap` need a `buf push` before they can use `subtypes` / `subtype_of`.
Consumers pulling the extensions from `buf.build/snaerverk/protoc-gen-go-sqlmap` need a `buf push` before they can use `subtypes` / `subtype_of`.

## The problem

Expand Down
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
module github.com/roderm/protoc-gen-go-sqlmap
module github.com/snaerverk/protoc-gen-go-sqlmap

go 1.27

Expand Down
4 changes: 2 additions & 2 deletions pkg/generated/schema/v1/schema.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 3 additions & 3 deletions pkg/generated/sqlmap/v1/sqlmap.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading