diff --git a/README.md b/README.md index 1567874..67aeaf9 100644 --- a/README.md +++ b/README.md @@ -2,24 +2,41 @@ A SQLite and PostgreSQL database library for [jolt](https://github.com/jolt-lang/jolt) (Clojure on Chez Scheme). It binds the system **libsqlite3** and **libpq** -directly through `jolt.ffi` — jolt's foreign-function interface — and exposes the -[clojure.jdbc](https://github.com/yogthos/clojure.jdbc) API plus a small -[next.jdbc](https://github.com/seancorfield/next-jdbc) surface. No jolt built-in, -no JVM: the native binding lives in this library. +directly through `jolt.ffi` — jolt's foreign-function interface — and runs the +real [clojure.jdbc](https://github.com/yogthos/clojure.jdbc) on top of them, plus +a small [next.jdbc](https://github.com/seancorfield/next-jdbc) surface. No jolt +built-in, no JVM: the native binding lives here, and the API is the published +library rather than a copy of it. + +`jdbc.core` is clojure.jdbc itself. This library supplies the `java.sql` surface +it drives (`db.jdbc-shim`) over the native drivers, so its own documentation and +semantics apply as written. ```clojure +(require '[db.jdbc]) ; registers the shim, once (require '[jdbc.core :as jdbc]) (with-open [conn (jdbc/connection "sqlite::memory:")] ; or "postgres://user:pw@host/db" (jdbc/execute! conn "create table p (id integer primary key, name text)") - (jdbc/insert! conn :p {:name "ada"}) ; -> generated id + (jdbc/insert! conn :p {:name "ada"}) ; -> (1), one result per row (jdbc/fetch conn ["select * from p where name = ?" "ada"])) ``` +Require `db.jdbc` once before `jdbc.core`, and before anything else that pulls it +in. It has to be loaded first because clojure.jdbc's namespaces resolve the +`java.sql` constants as they compile, and it is what points connection +construction at the native drivers instead of `DriverManager`. + `fetch`/`fetch-one`, `execute!`, `insert!`/`insert-multi!`/`update!`/`delete!`, -`last-insert-id`, and `atomic` (transactions with nested savepoints) are +`prepared-statement`, and `atomic` (transactions with nested savepoints) are supported on both backends. Queries are strings or sqlvecs (`[sql & params]`, JDBC `?` placeholders — rewritten to `$N` for postgres). +Generated keys come back through `RETURNING`, since neither driver has a JDBC +generated-keys channel. `{:returning true}` (or `:all`) asks for the whole row, +which is what postgres' own driver gives; a sequence of column names asks for +those. Without it there are no generated keys to report, so `insert!` falls back +to the update count, exactly as clojure.jdbc does on a driver that has none. + ## Binary values A byte array parameter binds as a SQLite `blob` / postgres `bytea`, and those @@ -39,23 +56,30 @@ bytes rather than inferring text. ## Errors -Database errors are `ex-info` values carrying `:jdbc/sql-error true` in their -`ex-data`, and they also satisfy `(catch java.sql.SQLException ...)` so code -written against the JDBC contract works unchanged. Migratus depends on this: its +Database errors satisfy `(catch java.sql.SQLException ...)`, so code written +against the JDBC contract works unchanged. Migratus depends on this: its `table-exists?` probe catches `SQLException` to decide whether it still needs to -create `schema_migrations`. +create `schema_migrations`. Errors raised by the drivers themselves also carry +`:jdbc/sql-error true` in their `ex-data`. ## Layout - `db.sqlite` / `db.pg` — the native drivers (jolt.ffi bindings). -- `jdbc.core` — the clojure.jdbc API over them. +- `db.jdbc-shim` — the `java.sql` surface clojure.jdbc drives, over those drivers. +- `db.jdbc` — the entry point: loads the shim, then clojure.jdbc on top of it. +- `jdbc.core` — clojure.jdbc itself, pulled in as a dependency. - `next.jdbc` (+ `.sql`/`.prepare`/`.result-set`/`.transaction`) — the next.jdbc surface migratus and similar tools use. ## Requirements -`jolt` on PATH; the system `libsqlite3` (preinstalled on macOS and most Linux -distros). PostgreSQL support additionally needs `libpq` at runtime. +`jolt` **v0.7.3 or newer** on PATH; the system `libsqlite3` (preinstalled on macOS +and most Linux distros). PostgreSQL support additionally needs `libpq` at runtime. + +The version floor is not cosmetic. The shim needs three host fixes that landed in +v0.7.3: `with-open` on a `reify`, a parenthesised `(Class/FIELD)` reading the +field, and a protocol extended to a library-declared class actually dispatching. +On an older jolt this library fails at load or at the first connection. ## Test diff --git a/clj-test/jdbc/core_test.clj b/clj-test/jdbc/core_test.clj index 0d0373b..41d34e2 100644 --- a/clj-test/jdbc/core_test.clj +++ b/clj-test/jdbc/core_test.clj @@ -1,5 +1,10 @@ (ns jdbc.core-test - (:require [jdbc.core :as jdbc] + ;; db.jdbc first: it registers the java.sql shim, which clojure.jdbc's + ;; namespaces resolve against as they compile, and points connection + ;; construction at the native drivers. jdbc.core below is the published + ;; clojure.jdbc running on top of it. + (:require [db.jdbc] + [jdbc.core :as jdbc] ;; the placeholder rewriter lives in the pg driver; requiring it here ;; lets the sqlite-only run cover the lexer table. Loading db.pg does ;; not need libpq present, only calling into it does. @@ -18,10 +23,17 @@ (with-open [conn (jdbc/connection "sqlite::memory:")] (check "execute! ddl" 0 (jdbc/execute! conn "create table person (id integer primary key, name text, zip integer)")) - (check "insert! returns the id" 1 (jdbc/insert! conn :person {:name "ada" :zip 94546})) - (check "insert-multi! ids" [2 3] + ;; clojure.jdbc's insert! returns one result per row. Without :returning the + ;; shim hands back an empty generated-keys set, so each row reports its update + ;; count, which is how the API behaves on a driver with no generated keys. + (check "insert! returns a result per row" '(1) + (jdbc/insert! conn :person {:name "ada" :zip 94546})) + (check "insert-multi! returns a result per row" '(1 1) (jdbc/insert-multi! conn :person [{:name "grace" :zip 94546} {:name "alan" :zip 10001}])) + (check "insert! :returning gives the inserted row" {:id 4 :name "edsger" :zip 1} + (first (jdbc/insert! conn :person {:name "edsger" :zip 1} {:returning true}))) + (jdbc/delete! conn :person ["name = ?" "edsger"]) (check "fetch sqlvec with params" [{:id 1 :name "ada" :zip 94546}] (jdbc/fetch conn ["select * from person where name = ?" "ada"])) (check "fetch-one" {:id 3 :name "alan" :zip 10001} @@ -40,7 +52,8 @@ (doseq [[label payload] [["embedded NULs" (byte-array [65 0 66 0 67])] ["non-UTF-8 bytes" (byte-array [-1 -2])] ["empty payload" (byte-array [])]]] - (let [id (jdbc/insert! conn :payload {:content payload}) + (let [id (:id (first (jdbc/insert! conn :payload {:content payload} + {:returning true}))) actual (:content (jdbc/fetch-one conn ["select content from payload where id = ?" id]))] @@ -82,7 +95,7 @@ (count (jdbc/fetch conn ["select * from person where name = ?" "marked"])))) (println "dbspec parsing") - (check "map spec works" 1 + (check "map spec works" '(1) (with-open [c (jdbc/connection {:vendor "sqlite" :name ":memory:"})] (jdbc/execute! c "create table t (x integer)") (jdbc/insert! c :t {:x 7}))) @@ -144,7 +157,11 @@ (with-open [conn (jdbc/connection pg-uri)] (jdbc/execute! conn "drop table if exists jolt_person") (jdbc/execute! conn "create table jolt_person (id serial primary key, name text, zip integer)") - (check "pg insert! returns the id" 1 (jdbc/insert! conn :jolt_person {:name "ada" :zip 94546})) + (check "pg insert! returns a result per row" '(1) + (jdbc/insert! conn :jolt_person {:name "ada" :zip 94546})) + (check "pg insert! :returning gives the inserted row" {:id 2 :name "hopper" :zip 3} + (first (jdbc/insert! conn :jolt_person {:name "hopper" :zip 3} {:returning true}))) + (jdbc/delete! conn :jolt_person ["name = ?" "hopper"]) (check "pg fetch with ? params" [{:id 1 :name "ada" :zip 94546}] (jdbc/fetch conn ["select * from jolt_person where name = ?" "ada"])) (jdbc/insert! conn :jolt_person {:name "grace" :zip 94546}) @@ -212,7 +229,8 @@ ["every byte value" (byte-array (mapv (fn [i] (if (> i 127) (- i 256) i)) (range 256)))] ["empty payload" (byte-array [])]]] - (let [id (jdbc/insert! conn :jolt_payload {:content payload}) + (let [id (:id (first (jdbc/insert! conn :jolt_payload {:content payload} + {:returning true}))) actual (:content (jdbc/fetch-one conn ["select content from jolt_payload where id = ?" id]))] diff --git a/clj/db/jdbc_shim.clj b/clj/db/jdbc_shim.clj index d48b3a8..e559dfd 100644 --- a/clj/db/jdbc_shim.clj +++ b/clj/db/jdbc_shim.clj @@ -348,12 +348,18 @@ "java.sql.DatabaseMetaData" :jdbc/dbmeta "java.sql.Savepoint" :jdbc/savepoint}) +;; Answer true or nil, never false. nil means "not one of mine, keep looking", +;; while false settles the question for every other library's check as well: the +;; first non-nil answer wins. next.jdbc registers its own check so its connection +;; wrapper answers instance? java.sql.Connection, which is how migratus picks its +;; Connection branch, and returning false here silently overruled it. (clojure.core/__register-instance-check! (fn [cn val] (when-let [tag (get class-tags cn)] ;; a PreparedStatement is a Statement too - (boolean (or (tagged? val tag) - (and (= cn "java.sql.Statement") (tagged? val :jdbc/prepared))))))) + (when (or (tagged? val tag) + (and (= cn "java.sql.Statement") (tagged? val :jdbc/prepared))) + true)))) ;; Report the java.sql class name for (class x) and, more importantly, so a ;; protocol extended to java.sql.Connection dispatches on these values. diff --git a/clj/jdbc/core.clj b/clj/jdbc/core.clj deleted file mode 100644 index bdb3270..0000000 --- a/clj/jdbc/core.clj +++ /dev/null @@ -1,219 +0,0 @@ -(ns jdbc.core - "clojure.jdbc's API (https://github.com/yogthos/clojure.jdbc) for jolt, - over native database drivers (bound through jolt.ffi) instead of java.sql: - - - SQLite via db.sqlite (the system libsqlite3) - - PostgreSQL via db.pg (the system libpq) - - Connections are plain maps carrying the driver handle plus a :close fn, so - `with-open` works. Queries are strings or sqlvecs ([sql & params], JDBC ? - placeholders — rewritten to $N for postgres). Rows come back as vectors of - keyword-keyed maps. - - (require '[jdbc.core :as jdbc]) - (with-open [conn (jdbc/connection \"sqlite::memory:\")] - (jdbc/execute! conn \"create table p (id integer primary key, name text)\") - (jdbc/insert! conn :p {:name \"ada\"}) - (jdbc/fetch conn [\"select * from p where name = ?\" \"ada\"]))" - (:require [clojure.string :as str] - [db.sqlite :as sqlite] - [db.pg :as pg])) - -;;; dbspec - -(defn- parse-uri-spec [s] - (cond - (str/starts-with? s "postgres") {:vendor "postgresql" :uri s} - (str/starts-with? s "sqlite:") {:vendor "sqlite" :name (subs s 7)} - ;; bare path = sqlite file - :else {:vendor "sqlite" :name s})) - -(defn- pg-uri [{:keys [uri name host port user password]}] - (or uri - (str "postgres://" - (when user (str user (when password (str ":" password)) "@")) - (or host "127.0.0.1") - (when port (str ":" port)) - "/" name))) - -(defn- normalize-spec [spec] - (cond - (string? spec) (parse-uri-spec spec) - (map? spec) (let [vendor (or (:vendor spec) (:subprotocol spec)) - spec (assoc spec :vendor (case vendor - ("postgresql" "postgres" "pgsql") "postgresql" - ("sqlite" "sqlite3") "sqlite" - (throw (ex-info (str "unknown vendor: " vendor) {:spec spec}))))] - (if (:subname spec) (assoc spec :name (:subname spec)) spec)) - :else (throw (ex-info "dbspec must be a string or a map" {:spec spec})))) - -;;; connection - -(defn connection - "Open a connection. spec is a uri string (\"sqlite:path\", a bare sqlite - path, or \"postgres://user:pass@host:port/db\") or a dbspec map with - :vendor (or :subprotocol) + :name/:subname [:host :port :user :password]. - The returned conn map has a :close fn — use with-open." - [spec] - (let [{:keys [vendor] :as spec} (normalize-spec spec)] - (case vendor - "sqlite" - (let [h (sqlite/open (:name spec))] - (sqlite/query h "PRAGMA foreign_keys=1;" []) - {:vendor :sqlite - :handle h - :depth (atom 0) - :rollback (atom false) - :close (fn [] (sqlite/close h))}) - "postgresql" - ;; db.pg is required statically so an AOT build has a dependency edge to the - ;; postgres implementation. A sqlite-only app still does not need libpq: the - ;; FFI bindings resolve its symbols on first call, not on load. - (let [h (pg/connect (pg-uri spec))] - {:vendor :postgresql - :handle h - :depth (atom 0) - :rollback (atom false) - :close (fn [] (pg/close h))})))) - -;;; queries - -(defn- sqlvec [q] - (cond - (string? q) [q []] - (vector? q) [(first q) (vec (rest q))] - :else (throw (ex-info "query must be a string or sqlvec" {:q q})))) - -(defn- sqlite-eval [conn sql params] - (sqlite/query (:handle conn) sql params)) - -(defn- pg-eval [conn sql params] - (pg/exec (:handle conn) sql params)) - -(defn fetch - "Run a query (string or sqlvec), return a vector of keyword-keyed row maps." - ([conn q] (fetch conn q {})) - ([conn q opts] - (let [[sql params] (sqlvec q) - rows (case (:vendor conn) - :sqlite (sqlite-eval conn sql params) - :postgresql (pg/all (:handle conn) sql params))] - (if-let [n (:max-rows opts)] (vec (take n rows)) rows)))) - -(defn fetch-one - "Run a query, return the first row map (or nil)." - ([conn q] (fetch-one conn q {})) - ([conn q opts] (first (fetch conn q (merge {:max-rows 1} opts))))) - -(defn execute! - "Execute a statement (string or sqlvec). Returns rows affected." - ([conn q] (execute! conn q {})) - ([conn q opts] - (let [[sql params] (sqlvec q)] - (case (:vendor conn) - :sqlite (do (sqlite-eval conn sql params) - (sqlite/changes (:handle conn))) - :postgresql (pg-eval conn sql params))))) - -(defn last-insert-id - "Driver-specific id of the last inserted row (sqlite: last_insert_rowid, - postgres: lastval, which needs the session to have used a sequence)." - [conn] - (case (:vendor conn) - :sqlite (sqlite/last-insert-rowid (:handle conn)) - :postgresql (:id (first (pg/all (:handle conn) "select lastval() as id" []))))) - -;;; insert! / update! / delete! — the clojure.jdbc convenience surface - -(defn- entity-str [entities x] (entities (if (keyword? x) (name x) (str x)))) - -(defn insert! - "Insert one row map and return the generated id, from last_insert_rowid on - sqlite and lastval on postgres. Inserting into a postgres table with no - sequence therefore throws, since lastval has nothing to report — use - \"... returning ...\" with execute!/fetch for that case." - ([conn table row] (insert! conn table row {})) - ([conn table row opts] - (let [entities (get opts :entities identity) - cols (vec (keys row)) - sql (str "INSERT INTO " (entity-str entities table) - " (" (str/join ", " (map #(entity-str entities %) cols)) ")" - " VALUES (" (str/join ", " (repeat (count cols) "?")) ")")] - (execute! conn (into [sql] (map #(get row %) cols)) opts) - (last-insert-id conn)))) - -(defn insert-multi! - "Insert a sequence of row maps; returns a vector of generated ids." - ([conn table rows] (insert-multi! conn table rows {})) - ([conn table rows opts] - (mapv #(insert! conn table % opts) rows))) - -(defn update! - "(update! conn :person {:zip 94540} [\"zip = ?\" 94546])" - ([conn table set-map where-clause] (update! conn table set-map where-clause {})) - ([conn table set-map where-clause opts] - (let [entities (get opts :entities identity) - cols (vec (keys set-map)) - [where & wparams] where-clause - sql (str "UPDATE " (entity-str entities table) - " SET " (str/join ", " (map #(str (entity-str entities %) " = ?") cols)) - (when-not (str/blank? (or where "")) (str " WHERE " where)))] - (execute! conn (into (into [sql] (map #(get set-map %) cols)) wparams) opts)))) - -(defn delete! - "(delete! conn :person [\"zip = ?\" 94546])" - ([conn table where-clause] (delete! conn table where-clause {})) - ([conn table where-clause opts] - (let [entities (get opts :entities identity) - [where & params] where-clause - sql (str "DELETE FROM " (entity-str entities table) - (when-not (str/blank? (or where "")) (str " WHERE " where)))] - (execute! conn (into [sql] params) opts)))) - -;;; transactions: BEGIN at depth 0, SAVEPOINTs when nested (both drivers). - -(defn set-rollback! - "Mark the current transaction to roll back at the end of the atomic block." - [conn] - (reset! (:rollback conn) true) - conn) - -(defn atomic-apply - "Run (func conn) in a transaction; nested calls use savepoints." - ([conn func] (atomic-apply conn func {})) - ([conn func opts] - (let [depth @(:depth conn) - sp (str "jdbc_sp_" depth) - begin (if (zero? depth) "BEGIN" (str "SAVEPOINT " sp)) - commit (if (zero? depth) "COMMIT" (str "RELEASE SAVEPOINT " sp)) - rollback (if (zero? depth) "ROLLBACK" (str "ROLLBACK TO SAVEPOINT " sp))] - (execute! conn begin) - (swap! (:depth conn) inc) - (try - (let [ret (func conn)] - (swap! (:depth conn) dec) - (if (and (zero? @(:depth conn)) @(:rollback conn)) - (do (reset! (:rollback conn) false) - (execute! conn rollback)) - (execute! conn commit)) - ret) - (catch Throwable t - (swap! (:depth conn) dec) - (execute! conn rollback) - (when (zero? @(:depth conn)) (reset! (:rollback conn) false)) - (throw t)))))) - -(defmacro atomic - "(atomic conn body...) — body runs in a transaction bound to conn." - [conn & body] - (if (map? (first body)) - `(atomic-apply ~conn (fn [c#] (let [~conn c#] ~@(next body))) ~(first body)) - `(atomic-apply ~conn (fn [c#] (let [~conn c#] ~@body))))) - -;; SQL errors satisfy (catch java.sql.SQLException ...) — migratus's -;; table-exists? probe and friends rely on that contract. -(clojure.core/__register-instance-check! - (fn [cn val] - (if (= cn "java.sql.SQLException") - (boolean (:jdbc/sql-error (ex-data val))) - nil))) diff --git a/clj/next/jdbc.clj b/clj/next/jdbc.clj index c8261a1..4b8a9ae 100644 --- a/clj/next/jdbc.clj +++ b/clj/next/jdbc.clj @@ -1,6 +1,7 @@ (ns next.jdbc - "A next.jdbc compatibility layer for jolt, over jdbc.core (which binds the - system db drivers via jolt.ffi). Just the surface migratus uses: get-connection, + "A next.jdbc compatibility layer for jolt, over jdbc.core — which is clojure.jdbc + itself, running on the java.sql shim in db.jdbc-shim above the native drivers. + Just the surface migratus uses: get-connection, execute!, execute-batch!, and the with-transaction macro. See next.jdbc.sql for insert!/delete!/query and next.jdbc.prepare for statement batching. diff --git a/deps.edn b/deps.edn index 431e057..511f791 100644 --- a/deps.edn +++ b/deps.edn @@ -1,5 +1,17 @@ {:paths ["clj"] + ;; clojure.jdbc is the API this library exposes: the published library running on + ;; the java.sql shim in db.jdbc-shim, rather than a reimplementation of it here. + ;; Require db.jdbc once before jdbc.core so the shim is registered first. + ;; + ;; jolt-lang/time is needed even though nothing here asks for java.time: + ;; jdbc.util/lower-case calls (Locale/US), which that library provides rather + ;; than jolt core (RFC 0008), and it sits in the path of every insert. + :deps {io.github.yogthos/clojure.jdbc {:git/url "https://github.com/yogthos/clojure.jdbc.git" + :git/sha "7b19a2caaa59bf73e083b8ba1943a770a58c4c07"} + io.github.jolt-lang/time {:git/url "https://github.com/jolt-lang/time.git" + :git/sha "70dfb7981ef4ed70c5d142109d56a90edfb79cef"}} + ;; SQLite (libsqlite3) and PostgreSQL (libpq) are bound natively via jolt.ffi ;; (db.sqlite / db.pg). The shared libraries are declared below; jolt loads them ;; before the namespaces are required. libsqlite3 ships on macOS and most Linux