From 93cf876534e779c74ad28f725e5c7fdc518afd5d Mon Sep 17 00:00:00 2001 From: Yogthos Date: Tue, 11 Aug 2026 19:51:26 -0400 Subject: [PATCH] Use the real clojure.jdbc instead of reimplementing it The switchover the shim was built for. clj/jdbc/core.clj is gone and clojure.jdbc comes in as a git dependency, so jdbc.core is now the published library running on db.jdbc-shim over the native drivers, and its own documentation and semantics apply as written rather than being approximated here. Two dependencies. clojure.jdbc at v0.9.3, and jolt-lang/time because jdbc.util/lower-case calls (Locale/US), which that library provides rather than jolt core, and which sits in the path of every insert. The time dependency is pinned past its README's example sha: that one predates "Declare the two forward-referenced vars", without which the library no longer compiles, since jolt stopped late-binding a forward reference inside a nested scope. The suite moved to clojure.jdbc's contract where it differed from ours. insert! returns one result per row rather than a bare id, which is a seq of generated keys where the driver has them and of update counts where it does not, so the blob and bytea round-trips ask for the id with {:returning true} instead. Everything else was already the same API and needed no change. Fixed a real bug in the shim while checking next.jdbc still worked: the instance check answered false for values it did not own, where jolt's contract is true, false or nil with nil meaning "not mine, keep looking" and the first non-nil winning. next.jdbc registers its own check so its connection wrapper answers instance? java.sql.Connection, which is how migratus picks its Connection branch, and ours was silently overruling it. It answers true or nil now. Requires jolt v0.7.3 or newer, and the README says why: the shim leans on three host fixes from that release, so an older jolt fails at load or first connection. 93 checks pass on sqlite and against postgres 16, and next.jdbc still round-trips through the new stack. --- README.md | 50 +++++--- clj-test/jdbc/core_test.clj | 32 ++++-- clj/db/jdbc_shim.clj | 10 +- clj/jdbc/core.clj | 219 ------------------------------------ clj/next/jdbc.clj | 5 +- deps.edn | 12 ++ 6 files changed, 85 insertions(+), 243 deletions(-) delete mode 100644 clj/jdbc/core.clj diff --git a/README.md b/README.md index 1567874..67aeaf9 100644 --- a/README.md +++ b/README.md @@ -2,24 +2,41 @@ A SQLite and PostgreSQL database library for [jolt](https://github.com/jolt-lang/jolt) (Clojure on Chez Scheme). It binds the system **libsqlite3** and **libpq** -directly through `jolt.ffi` — jolt's foreign-function interface — and exposes the -[clojure.jdbc](https://github.com/yogthos/clojure.jdbc) API plus a small -[next.jdbc](https://github.com/seancorfield/next-jdbc) surface. No jolt built-in, -no JVM: the native binding lives in this library. +directly through `jolt.ffi` — jolt's foreign-function interface — and runs the +real [clojure.jdbc](https://github.com/yogthos/clojure.jdbc) on top of them, plus +a small [next.jdbc](https://github.com/seancorfield/next-jdbc) surface. No jolt +built-in, no JVM: the native binding lives here, and the API is the published +library rather than a copy of it. + +`jdbc.core` is clojure.jdbc itself. This library supplies the `java.sql` surface +it drives (`db.jdbc-shim`) over the native drivers, so its own documentation and +semantics apply as written. ```clojure +(require '[db.jdbc]) ; registers the shim, once (require '[jdbc.core :as jdbc]) (with-open [conn (jdbc/connection "sqlite::memory:")] ; or "postgres://user:pw@host/db" (jdbc/execute! conn "create table p (id integer primary key, name text)") - (jdbc/insert! conn :p {:name "ada"}) ; -> generated id + (jdbc/insert! conn :p {:name "ada"}) ; -> (1), one result per row (jdbc/fetch conn ["select * from p where name = ?" "ada"])) ``` +Require `db.jdbc` once before `jdbc.core`, and before anything else that pulls it +in. It has to be loaded first because clojure.jdbc's namespaces resolve the +`java.sql` constants as they compile, and it is what points connection +construction at the native drivers instead of `DriverManager`. + `fetch`/`fetch-one`, `execute!`, `insert!`/`insert-multi!`/`update!`/`delete!`, -`last-insert-id`, and `atomic` (transactions with nested savepoints) are +`prepared-statement`, and `atomic` (transactions with nested savepoints) are supported on both backends. Queries are strings or sqlvecs (`[sql & params]`, JDBC `?` placeholders — rewritten to `$N` for postgres). +Generated keys come back through `RETURNING`, since neither driver has a JDBC +generated-keys channel. `{:returning true}` (or `:all`) asks for the whole row, +which is what postgres' own driver gives; a sequence of column names asks for +those. Without it there are no generated keys to report, so `insert!` falls back +to the update count, exactly as clojure.jdbc does on a driver that has none. + ## Binary values A byte array parameter binds as a SQLite `blob` / postgres `bytea`, and those @@ -39,23 +56,30 @@ bytes rather than inferring text. ## Errors -Database errors are `ex-info` values carrying `:jdbc/sql-error true` in their -`ex-data`, and they also satisfy `(catch java.sql.SQLException ...)` so code -written against the JDBC contract works unchanged. Migratus depends on this: its +Database errors satisfy `(catch java.sql.SQLException ...)`, so code written +against the JDBC contract works unchanged. Migratus depends on this: its `table-exists?` probe catches `SQLException` to decide whether it still needs to -create `schema_migrations`. +create `schema_migrations`. Errors raised by the drivers themselves also carry +`:jdbc/sql-error true` in their `ex-data`. ## Layout - `db.sqlite` / `db.pg` — the native drivers (jolt.ffi bindings). -- `jdbc.core` — the clojure.jdbc API over them. +- `db.jdbc-shim` — the `java.sql` surface clojure.jdbc drives, over those drivers. +- `db.jdbc` — the entry point: loads the shim, then clojure.jdbc on top of it. +- `jdbc.core` — clojure.jdbc itself, pulled in as a dependency. - `next.jdbc` (+ `.sql`/`.prepare`/`.result-set`/`.transaction`) — the next.jdbc surface migratus and similar tools use. ## Requirements -`jolt` on PATH; the system `libsqlite3` (preinstalled on macOS and most Linux -distros). PostgreSQL support additionally needs `libpq` at runtime. +`jolt` **v0.7.3 or newer** on PATH; the system `libsqlite3` (preinstalled on macOS +and most Linux distros). PostgreSQL support additionally needs `libpq` at runtime. + +The version floor is not cosmetic. The shim needs three host fixes that landed in +v0.7.3: `with-open` on a `reify`, a parenthesised `(Class/FIELD)` reading the +field, and a protocol extended to a library-declared class actually dispatching. +On an older jolt this library fails at load or at the first connection. ## Test diff --git a/clj-test/jdbc/core_test.clj b/clj-test/jdbc/core_test.clj index 0d0373b..41d34e2 100644 --- a/clj-test/jdbc/core_test.clj +++ b/clj-test/jdbc/core_test.clj @@ -1,5 +1,10 @@ (ns jdbc.core-test - (:require [jdbc.core :as jdbc] + ;; db.jdbc first: it registers the java.sql shim, which clojure.jdbc's + ;; namespaces resolve against as they compile, and points connection + ;; construction at the native drivers. jdbc.core below is the published + ;; clojure.jdbc running on top of it. + (:require [db.jdbc] + [jdbc.core :as jdbc] ;; the placeholder rewriter lives in the pg driver; requiring it here ;; lets the sqlite-only run cover the lexer table. Loading db.pg does ;; not need libpq present, only calling into it does. @@ -18,10 +23,17 @@ (with-open [conn (jdbc/connection "sqlite::memory:")] (check "execute! ddl" 0 (jdbc/execute! conn "create table person (id integer primary key, name text, zip integer)")) - (check "insert! returns the id" 1 (jdbc/insert! conn :person {:name "ada" :zip 94546})) - (check "insert-multi! ids" [2 3] + ;; clojure.jdbc's insert! returns one result per row. Without :returning the + ;; shim hands back an empty generated-keys set, so each row reports its update + ;; count, which is how the API behaves on a driver with no generated keys. + (check "insert! returns a result per row" '(1) + (jdbc/insert! conn :person {:name "ada" :zip 94546})) + (check "insert-multi! returns a result per row" '(1 1) (jdbc/insert-multi! conn :person [{:name "grace" :zip 94546} {:name "alan" :zip 10001}])) + (check "insert! :returning gives the inserted row" {:id 4 :name "edsger" :zip 1} + (first (jdbc/insert! conn :person {:name "edsger" :zip 1} {:returning true}))) + (jdbc/delete! conn :person ["name = ?" "edsger"]) (check "fetch sqlvec with params" [{:id 1 :name "ada" :zip 94546}] (jdbc/fetch conn ["select * from person where name = ?" "ada"])) (check "fetch-one" {:id 3 :name "alan" :zip 10001} @@ -40,7 +52,8 @@ (doseq [[label payload] [["embedded NULs" (byte-array [65 0 66 0 67])] ["non-UTF-8 bytes" (byte-array [-1 -2])] ["empty payload" (byte-array [])]]] - (let [id (jdbc/insert! conn :payload {:content payload}) + (let [id (:id (first (jdbc/insert! conn :payload {:content payload} + {:returning true}))) actual (:content (jdbc/fetch-one conn ["select content from payload where id = ?" id]))] @@ -82,7 +95,7 @@ (count (jdbc/fetch conn ["select * from person where name = ?" "marked"])))) (println "dbspec parsing") - (check "map spec works" 1 + (check "map spec works" '(1) (with-open [c (jdbc/connection {:vendor "sqlite" :name ":memory:"})] (jdbc/execute! c "create table t (x integer)") (jdbc/insert! c :t {:x 7}))) @@ -144,7 +157,11 @@ (with-open [conn (jdbc/connection pg-uri)] (jdbc/execute! conn "drop table if exists jolt_person") (jdbc/execute! conn "create table jolt_person (id serial primary key, name text, zip integer)") - (check "pg insert! returns the id" 1 (jdbc/insert! conn :jolt_person {:name "ada" :zip 94546})) + (check "pg insert! returns a result per row" '(1) + (jdbc/insert! conn :jolt_person {:name "ada" :zip 94546})) + (check "pg insert! :returning gives the inserted row" {:id 2 :name "hopper" :zip 3} + (first (jdbc/insert! conn :jolt_person {:name "hopper" :zip 3} {:returning true}))) + (jdbc/delete! conn :jolt_person ["name = ?" "hopper"]) (check "pg fetch with ? params" [{:id 1 :name "ada" :zip 94546}] (jdbc/fetch conn ["select * from jolt_person where name = ?" "ada"])) (jdbc/insert! conn :jolt_person {:name "grace" :zip 94546}) @@ -212,7 +229,8 @@ ["every byte value" (byte-array (mapv (fn [i] (if (> i 127) (- i 256) i)) (range 256)))] ["empty payload" (byte-array [])]]] - (let [id (jdbc/insert! conn :jolt_payload {:content payload}) + (let [id (:id (first (jdbc/insert! conn :jolt_payload {:content payload} + {:returning true}))) actual (:content (jdbc/fetch-one conn ["select content from jolt_payload where id = ?" id]))] diff --git a/clj/db/jdbc_shim.clj b/clj/db/jdbc_shim.clj index d48b3a8..e559dfd 100644 --- a/clj/db/jdbc_shim.clj +++ b/clj/db/jdbc_shim.clj @@ -348,12 +348,18 @@ "java.sql.DatabaseMetaData" :jdbc/dbmeta "java.sql.Savepoint" :jdbc/savepoint}) +;; Answer true or nil, never false. nil means "not one of mine, keep looking", +;; while false settles the question for every other library's check as well: the +;; first non-nil answer wins. next.jdbc registers its own check so its connection +;; wrapper answers instance? java.sql.Connection, which is how migratus picks its +;; Connection branch, and returning false here silently overruled it. (clojure.core/__register-instance-check! (fn [cn val] (when-let [tag (get class-tags cn)] ;; a PreparedStatement is a Statement too - (boolean (or (tagged? val tag) - (and (= cn "java.sql.Statement") (tagged? val :jdbc/prepared))))))) + (when (or (tagged? val tag) + (and (= cn "java.sql.Statement") (tagged? val :jdbc/prepared))) + true)))) ;; Report the java.sql class name for (class x) and, more importantly, so a ;; protocol extended to java.sql.Connection dispatches on these values. diff --git a/clj/jdbc/core.clj b/clj/jdbc/core.clj deleted file mode 100644 index bdb3270..0000000 --- a/clj/jdbc/core.clj +++ /dev/null @@ -1,219 +0,0 @@ -(ns jdbc.core - "clojure.jdbc's API (https://github.com/yogthos/clojure.jdbc) for jolt, - over native database drivers (bound through jolt.ffi) instead of java.sql: - - - SQLite via db.sqlite (the system libsqlite3) - - PostgreSQL via db.pg (the system libpq) - - Connections are plain maps carrying the driver handle plus a :close fn, so - `with-open` works. Queries are strings or sqlvecs ([sql & params], JDBC ? - placeholders — rewritten to $N for postgres). Rows come back as vectors of - keyword-keyed maps. - - (require '[jdbc.core :as jdbc]) - (with-open [conn (jdbc/connection \"sqlite::memory:\")] - (jdbc/execute! conn \"create table p (id integer primary key, name text)\") - (jdbc/insert! conn :p {:name \"ada\"}) - (jdbc/fetch conn [\"select * from p where name = ?\" \"ada\"]))" - (:require [clojure.string :as str] - [db.sqlite :as sqlite] - [db.pg :as pg])) - -;;; dbspec - -(defn- parse-uri-spec [s] - (cond - (str/starts-with? s "postgres") {:vendor "postgresql" :uri s} - (str/starts-with? s "sqlite:") {:vendor "sqlite" :name (subs s 7)} - ;; bare path = sqlite file - :else {:vendor "sqlite" :name s})) - -(defn- pg-uri [{:keys [uri name host port user password]}] - (or uri - (str "postgres://" - (when user (str user (when password (str ":" password)) "@")) - (or host "127.0.0.1") - (when port (str ":" port)) - "/" name))) - -(defn- normalize-spec [spec] - (cond - (string? spec) (parse-uri-spec spec) - (map? spec) (let [vendor (or (:vendor spec) (:subprotocol spec)) - spec (assoc spec :vendor (case vendor - ("postgresql" "postgres" "pgsql") "postgresql" - ("sqlite" "sqlite3") "sqlite" - (throw (ex-info (str "unknown vendor: " vendor) {:spec spec}))))] - (if (:subname spec) (assoc spec :name (:subname spec)) spec)) - :else (throw (ex-info "dbspec must be a string or a map" {:spec spec})))) - -;;; connection - -(defn connection - "Open a connection. spec is a uri string (\"sqlite:path\", a bare sqlite - path, or \"postgres://user:pass@host:port/db\") or a dbspec map with - :vendor (or :subprotocol) + :name/:subname [:host :port :user :password]. - The returned conn map has a :close fn — use with-open." - [spec] - (let [{:keys [vendor] :as spec} (normalize-spec spec)] - (case vendor - "sqlite" - (let [h (sqlite/open (:name spec))] - (sqlite/query h "PRAGMA foreign_keys=1;" []) - {:vendor :sqlite - :handle h - :depth (atom 0) - :rollback (atom false) - :close (fn [] (sqlite/close h))}) - "postgresql" - ;; db.pg is required statically so an AOT build has a dependency edge to the - ;; postgres implementation. A sqlite-only app still does not need libpq: the - ;; FFI bindings resolve its symbols on first call, not on load. - (let [h (pg/connect (pg-uri spec))] - {:vendor :postgresql - :handle h - :depth (atom 0) - :rollback (atom false) - :close (fn [] (pg/close h))})))) - -;;; queries - -(defn- sqlvec [q] - (cond - (string? q) [q []] - (vector? q) [(first q) (vec (rest q))] - :else (throw (ex-info "query must be a string or sqlvec" {:q q})))) - -(defn- sqlite-eval [conn sql params] - (sqlite/query (:handle conn) sql params)) - -(defn- pg-eval [conn sql params] - (pg/exec (:handle conn) sql params)) - -(defn fetch - "Run a query (string or sqlvec), return a vector of keyword-keyed row maps." - ([conn q] (fetch conn q {})) - ([conn q opts] - (let [[sql params] (sqlvec q) - rows (case (:vendor conn) - :sqlite (sqlite-eval conn sql params) - :postgresql (pg/all (:handle conn) sql params))] - (if-let [n (:max-rows opts)] (vec (take n rows)) rows)))) - -(defn fetch-one - "Run a query, return the first row map (or nil)." - ([conn q] (fetch-one conn q {})) - ([conn q opts] (first (fetch conn q (merge {:max-rows 1} opts))))) - -(defn execute! - "Execute a statement (string or sqlvec). Returns rows affected." - ([conn q] (execute! conn q {})) - ([conn q opts] - (let [[sql params] (sqlvec q)] - (case (:vendor conn) - :sqlite (do (sqlite-eval conn sql params) - (sqlite/changes (:handle conn))) - :postgresql (pg-eval conn sql params))))) - -(defn last-insert-id - "Driver-specific id of the last inserted row (sqlite: last_insert_rowid, - postgres: lastval, which needs the session to have used a sequence)." - [conn] - (case (:vendor conn) - :sqlite (sqlite/last-insert-rowid (:handle conn)) - :postgresql (:id (first (pg/all (:handle conn) "select lastval() as id" []))))) - -;;; insert! / update! / delete! — the clojure.jdbc convenience surface - -(defn- entity-str [entities x] (entities (if (keyword? x) (name x) (str x)))) - -(defn insert! - "Insert one row map and return the generated id, from last_insert_rowid on - sqlite and lastval on postgres. Inserting into a postgres table with no - sequence therefore throws, since lastval has nothing to report — use - \"... returning ...\" with execute!/fetch for that case." - ([conn table row] (insert! conn table row {})) - ([conn table row opts] - (let [entities (get opts :entities identity) - cols (vec (keys row)) - sql (str "INSERT INTO " (entity-str entities table) - " (" (str/join ", " (map #(entity-str entities %) cols)) ")" - " VALUES (" (str/join ", " (repeat (count cols) "?")) ")")] - (execute! conn (into [sql] (map #(get row %) cols)) opts) - (last-insert-id conn)))) - -(defn insert-multi! - "Insert a sequence of row maps; returns a vector of generated ids." - ([conn table rows] (insert-multi! conn table rows {})) - ([conn table rows opts] - (mapv #(insert! conn table % opts) rows))) - -(defn update! - "(update! conn :person {:zip 94540} [\"zip = ?\" 94546])" - ([conn table set-map where-clause] (update! conn table set-map where-clause {})) - ([conn table set-map where-clause opts] - (let [entities (get opts :entities identity) - cols (vec (keys set-map)) - [where & wparams] where-clause - sql (str "UPDATE " (entity-str entities table) - " SET " (str/join ", " (map #(str (entity-str entities %) " = ?") cols)) - (when-not (str/blank? (or where "")) (str " WHERE " where)))] - (execute! conn (into (into [sql] (map #(get set-map %) cols)) wparams) opts)))) - -(defn delete! - "(delete! conn :person [\"zip = ?\" 94546])" - ([conn table where-clause] (delete! conn table where-clause {})) - ([conn table where-clause opts] - (let [entities (get opts :entities identity) - [where & params] where-clause - sql (str "DELETE FROM " (entity-str entities table) - (when-not (str/blank? (or where "")) (str " WHERE " where)))] - (execute! conn (into [sql] params) opts)))) - -;;; transactions: BEGIN at depth 0, SAVEPOINTs when nested (both drivers). - -(defn set-rollback! - "Mark the current transaction to roll back at the end of the atomic block." - [conn] - (reset! (:rollback conn) true) - conn) - -(defn atomic-apply - "Run (func conn) in a transaction; nested calls use savepoints." - ([conn func] (atomic-apply conn func {})) - ([conn func opts] - (let [depth @(:depth conn) - sp (str "jdbc_sp_" depth) - begin (if (zero? depth) "BEGIN" (str "SAVEPOINT " sp)) - commit (if (zero? depth) "COMMIT" (str "RELEASE SAVEPOINT " sp)) - rollback (if (zero? depth) "ROLLBACK" (str "ROLLBACK TO SAVEPOINT " sp))] - (execute! conn begin) - (swap! (:depth conn) inc) - (try - (let [ret (func conn)] - (swap! (:depth conn) dec) - (if (and (zero? @(:depth conn)) @(:rollback conn)) - (do (reset! (:rollback conn) false) - (execute! conn rollback)) - (execute! conn commit)) - ret) - (catch Throwable t - (swap! (:depth conn) dec) - (execute! conn rollback) - (when (zero? @(:depth conn)) (reset! (:rollback conn) false)) - (throw t)))))) - -(defmacro atomic - "(atomic conn body...) — body runs in a transaction bound to conn." - [conn & body] - (if (map? (first body)) - `(atomic-apply ~conn (fn [c#] (let [~conn c#] ~@(next body))) ~(first body)) - `(atomic-apply ~conn (fn [c#] (let [~conn c#] ~@body))))) - -;; SQL errors satisfy (catch java.sql.SQLException ...) — migratus's -;; table-exists? probe and friends rely on that contract. -(clojure.core/__register-instance-check! - (fn [cn val] - (if (= cn "java.sql.SQLException") - (boolean (:jdbc/sql-error (ex-data val))) - nil))) diff --git a/clj/next/jdbc.clj b/clj/next/jdbc.clj index c8261a1..4b8a9ae 100644 --- a/clj/next/jdbc.clj +++ b/clj/next/jdbc.clj @@ -1,6 +1,7 @@ (ns next.jdbc - "A next.jdbc compatibility layer for jolt, over jdbc.core (which binds the - system db drivers via jolt.ffi). Just the surface migratus uses: get-connection, + "A next.jdbc compatibility layer for jolt, over jdbc.core — which is clojure.jdbc + itself, running on the java.sql shim in db.jdbc-shim above the native drivers. + Just the surface migratus uses: get-connection, execute!, execute-batch!, and the with-transaction macro. See next.jdbc.sql for insert!/delete!/query and next.jdbc.prepare for statement batching. diff --git a/deps.edn b/deps.edn index 431e057..511f791 100644 --- a/deps.edn +++ b/deps.edn @@ -1,5 +1,17 @@ {:paths ["clj"] + ;; clojure.jdbc is the API this library exposes: the published library running on + ;; the java.sql shim in db.jdbc-shim, rather than a reimplementation of it here. + ;; Require db.jdbc once before jdbc.core so the shim is registered first. + ;; + ;; jolt-lang/time is needed even though nothing here asks for java.time: + ;; jdbc.util/lower-case calls (Locale/US), which that library provides rather + ;; than jolt core (RFC 0008), and it sits in the path of every insert. + :deps {io.github.yogthos/clojure.jdbc {:git/url "https://github.com/yogthos/clojure.jdbc.git" + :git/sha "7b19a2caaa59bf73e083b8ba1943a770a58c4c07"} + io.github.jolt-lang/time {:git/url "https://github.com/jolt-lang/time.git" + :git/sha "70dfb7981ef4ed70c5d142109d56a90edfb79cef"}} + ;; SQLite (libsqlite3) and PostgreSQL (libpq) are bound natively via jolt.ffi ;; (db.sqlite / db.pg). The shared libraries are declared below; jolt loads them ;; before the namespaces are required. libsqlite3 ships on macOS and most Linux