diff --git a/doc/http-router.md b/doc/http-router.md new file mode 100644 index 000000000..5d39db8ff --- /dev/null +++ b/doc/http-router.md @@ -0,0 +1,450 @@ +# Datahike HTTP Router + +The `datahike.http.router` namespace provides route generation and handlers for the Datahike HTTP API, allowing you to embed Datahike's HTTP interface in your own applications without pulling in server dependencies. + +## Motivation + +Previously, using Datahike's HTTP API required running the full HTTP server with all its dependencies (Jetty, Ring, etc.). This prevented: +- GraalVM native image compilation due to heavy server dependencies +- Embedding Datahike routes in existing applications +- Using alternative server implementations + +The router namespace solves these issues by separating route definitions from server implementation. + +## Architecture + +``` +┌─────────────────────────────────────────────────────────┐ +│ Main Library │ +│ │ +│ datahike.http.router │ +│ - Route generation from API spec │ +│ - Handler functions │ +│ - Minimal middleware │ +│ - No server dependencies │ +└─────────────────────────────────────────────────────────┘ + │ + │ Uses + ▼ +┌─────────────────────────────────────────────────────────┐ +│ HTTP Server │ +│ │ +│ datahike.http.server │ +│ - Full server implementation │ +│ - Jetty, Ring, Reitit dependencies │ +│ - Authentication, CORS, Swagger UI │ +│ - Distributed mode support │ +└─────────────────────────────────────────────────────────┘ +``` + +## Usage Modes + +### 1. Embedded Mode (With Authentication) + +The standard way to embed Datahike routes in your application with proper authentication: + +```clojure +(require '[datahike.http.router :as router]) + +;; Standard approach: With authentication (same config format as server!) +(def config + {:token "securerandompassword" ; Required for production + :dev-mode false ; Set to true only for development + :level :info}) ; Optional logging level + +(def handler + (router/create-ring-handler + :config config ; Pass the security config + :prefix "/datahike" ; Mount under /datahike (optional) + :include-writers? false ; Don't expose internal writer routes + :middleware [your-middleware])) + +;; Use with any Ring-compatible server +(run-jetty handler {:port 3000}) + +;; Connect from REPL with authentication (include prefix in URL) +(d/connect {:store {:backend :datahike-server + :url "http://localhost:3000/datahike" + :token "securerandompassword"}}) +``` + +**Custom Prefix Examples:** +```clojure +;; Mount under /api/db +(def handler (router/create-ring-handler + :config config + :prefix "/api/db")) + +;; Mount under /v1/database +(def handler (router/create-ring-handler + :config config + :prefix "/v1/database")) + +;; No prefix (root level) +(def handler (router/create-ring-handler + :config config)) ; :prefix defaults to "" +``` + +**Development Mode**: For local development only, you can bypass authentication: +```clojure +(def dev-config {:dev-mode true}) ; WARNING: Never use in production! + +(def dev-handler + (router/create-ring-handler :config dev-config)) +``` + +### 2. Distributed Mode + +Run the full HTTP server for distributed deployments: + +```bash +# Run the standalone server +clojure -A:http-server -m datahike.http.server config.edn +``` + +## Complete Minimal Example + +Save this as `server.clj` and run with `clojure server.clj`: + +```clojure +(ns server + (:require [datahike.api :as d] + [datahike.http.router :as router] + [ring.adapter.jetty :refer [run-jetty]])) + +(def config + {:token "securerandompassword" ; Required for production + :dev-mode false}) ; Set to true for local development only + +(defn -main [] + ;; Create a test database for demo + (d/create-database {:store {:backend :mem :id "demo"}}) + + ;; Create handler with authentication and mount under /datahike + (let [handler (router/create-ring-handler + :config config + :prefix "/datahike")] + + (println "\n=== Datahike Embedded Server ===") + (println "URL: http://localhost:8080/datahike") + (println "Auth: token required (see config)") + (println "\nTest with:") + (println " curl -X GET http://localhost:8080/datahike/database-exists? \\") + (println " -H \"Authorization: token securerandompassword\" \\") + (println " -d '[{\"store\":{\"backend\":\"mem\",\"id\":\"demo\"}}]'") + + (run-jetty handler {:port 8080 :join? false}))) + +;; Connect from REPL +(comment + ;; Start server + (-main) + + ;; Connect with authentication (include prefix in URL) + (def conn (d/connect {:store {:backend :datahike-server + :url "http://localhost:8080/datahike" + :token "securerandompassword"}})) + + ;; Use normally + (d/transact conn [{:name "Alice"}]) + (d/q '[:find ?name :where [_ :name ?name]] @conn)) +``` + +## API Functions + +### `create-routes` + +Main function for getting routes in various formats: + +```clojure +(create-routes & {:keys [format prefix include-writers? middleware]}) +``` + +Options: +- `:format` - Output format (`:raw`, `:reitit`, `:compojure`, `:ring`) +- `:prefix` - URL prefix for all routes (e.g., `"/datahike"` or `"/api/db"`) +- `:include-writers?` - Include internal writer routes (default `false`) +- `:middleware` - Additional middleware to apply + +### `create-ring-handler` + +Main function for creating a secured Ring handler: + +```clojure +(create-ring-handler & {:keys [config prefix include-writers? middleware not-found-handler]}) +``` + +**Required for Production:** +- `:config` - Security configuration (same format as server config): + - `:token` - Authentication token **(required for production)** + - `:dev-mode` - If true, bypasses authentication **(never use in production)** + - `:level` - Log level (optional) + +**Other Options:** +- `:prefix` - URL prefix for all routes (e.g., `"/datahike"` or `"/api/db"`) +- `:include-writers?` - Include internal writer routes (default `false` - keep it false for embedded mode) +- `:middleware` - Additional middleware to apply +- `:not-found-handler` - Handler for unmatched routes + +**Standard Usage:** +```clojure +;; Production configuration with prefix +(def handler + (router/create-ring-handler + :config {:token "securerandompassword" + :dev-mode false} + :prefix "/datahike")) + +;; Without prefix (routes at root) +(def handler + (router/create-ring-handler + :config {:token "securerandompassword" + :dev-mode false})) +``` + +### `generate-api-routes` / `generate-writer-routes` + +Lower-level functions that return raw route definitions: + +```clojure +;; Get all API routes as data +(generate-api-routes) +;; => [{:path "/create-database" :method :post :handler fn ...} ...] + +;; Get internal writer routes (for distributed mode) +(generate-writer-routes) +``` + +## Route Formats + +### Raw Format +```clojure +{:path "/create-database" + :method :post + :handler #function[...] + :name :create-database + :doc "Creates a new database..."} +``` + +### Reitit Format +```clojure +["/create-database" + {:post {:handler #function[...] + :name :create-database + :summary "Creates a new database" + :middleware []}}] +``` + +### Ring Format +Returns a handler function that matches routes internally. + +## Integration Examples + +### With Existing Reitit App + +```clojure +;; Standard: with authentication +(def config {:token "your-api-token" :dev-mode false}) + +(def app + (ring/ring-handler + (ring/router + (concat + ;; Your routes + [["/health" {:get health-handler}]] + ;; Secured Datahike routes + (router/create-routes :format :reitit))) + ;; Apply authentication at router level + {:middleware [(fn [handler] + (router/wrap-token-auth handler config))]})) +``` + +### With Compojure + +```clojure +;; Standard: with authentication +(def datahike-config {:token "your-api-token" :dev-mode false}) + +(defroutes app-routes + ;; Your routes + (GET "/" [] "Home") + + ;; Mount secured Datahike under /api + (context "/api" [] + (let [handler (router/create-ring-handler :config datahike-config)] + (fn [req] (handler req))))) +``` + +## Authentication + +### Standard Token Authentication + +The router uses token-based authentication that matches the Datahike server configuration format. This is the recommended approach for production deployments: + +```clojure +;; Configuration (same format as server config.edn) +(def config + {:token "securerandompassword" ; Required for production + :dev-mode false ; Never true in production + :level :info}) ; Optional logging + +(def handler + (router/create-ring-handler + :config config + :middleware [wrap-json-response])) + +;; Clients must include the token +(d/connect {:store {:backend :datahike-server + :url "http://localhost:8080/datahike" + :token "securerandompassword"}}) +``` + +**Supported Authentication Headers:** +- `Authorization: token ` (Datahike format) +- `Authorization: Bearer ` (OAuth/JWT compatible) + +### Alternative: Custom Authentication + +For specialized requirements (OAuth, JWT, mTLS, etc.), you can implement custom authentication: + +```clojure +(defn wrap-custom-auth [handler] + (fn [request] + (if (valid-user? request) + (handler request) + {:status 401 :body "Unauthorized"}))) + +;; Use custom auth instead of built-in +(def handler + (-> (router/create-ring-handler) ; Omit :config for custom auth + wrap-custom-auth + wrap-json-response)) +``` + +### Development Mode (Local Only) + +For local development, you can temporarily disable authentication: + +```clojure +;; WARNING: Never deploy with dev-mode enabled! +(def dev-handler + (router/create-ring-handler + :config {:dev-mode true})) +``` + +## GraalVM Native Image + +The router namespace is compatible with GraalVM native image compilation: + +1. Include only `datahike.http.router` in your app +2. Exclude HTTP server dependencies +3. Compile to native image: + +```clojure +;; deps.edn +{:deps {io.replikativ/datahike {:mvn/version "..."}} + ;; No :http-server alias needed for embedded mode + + :aliases + {:native-image + {:main-opts ["-m" "clj.native-image" "your.main" + "--no-fallback" + "--initialize-at-build-time"]}}} +``` + +## Migration Guide + +### From Standalone Server to Embedded + +Before (running separate server): +```bash +clojure -A:http-server -m datahike.http.server config.edn +``` + +After (embedded in your app): +```clojure +(ns your.app + (:require [datahike.http.router :as router] + [ring.adapter.jetty :refer [run-jetty]])) + +(def handler + (router/create-ring-handler + :middleware [wrap-cors wrap-json])) + +(defn -main [] + (run-jetty handler {:port 8080})) +``` + +### From Embedded to Distributed + +Simply run the full server alongside your embedded app: + +```bash +# Development: embedded mode +lein run + +# Production: distributed mode +docker run -p 8080:8080 datahike/http-server +``` + +## Performance Considerations + +- Route generation happens once at startup +- Handlers are pre-compiled, no runtime overhead +- Minimal middleware stack in embedded mode +- Full caching and optimization in distributed mode + +## Security Best Practices + +### Required for Production + +1. **Always configure authentication:** + ```clojure + :config {:token "use-a-strong-random-token" + :dev-mode false} ; NEVER true in production + ``` + +2. **Use environment variables for tokens:** + ```clojure + :config {:token (System/getenv "DATAHIKE_TOKEN") + :dev-mode false} + ``` + +3. **Never expose writer routes in embedded mode:** + ```clojure + :include-writers? false ; Always false for embedded + ``` + +4. **Use HTTPS in production:** + - Deploy behind a reverse proxy with TLS termination + - Never send tokens over unencrypted connections + +5. **Rotate tokens regularly:** + - Change authentication tokens periodically + - Use different tokens for different environments + +### Security Checklist + +- [ ] Authentication token configured +- [ ] `:dev-mode` is `false` +- [ ] `:include-writers?` is `false` +- [ ] Token stored securely (env var or secret manager) +- [ ] HTTPS enabled in production +- [ ] Different tokens for dev/staging/production + +## Troubleshooting + +### Routes not working +- Check that body params are properly formatted +- Ensure content-type is set (application/json, application/edn) +- Verify middleware ordering + +### GraalVM compilation fails +- Ensure no server dependencies in classpath +- Check for reflection warnings +- Add necessary reflection config + +### Performance issues +- Use distributed mode for high load +- Add caching middleware in embedded mode +- Consider connection pooling \ No newline at end of file diff --git a/src/datahike/http/router.clj b/src/datahike/http/router.clj new file mode 100644 index 000000000..94902f4ae --- /dev/null +++ b/src/datahike/http/router.clj @@ -0,0 +1,494 @@ +(ns datahike.http.router + "Route generation and handlers for Datahike HTTP API. + This namespace can be used in embedded mode without pulling in server dependencies." + (:require [datahike.api :as d] + [datahike.api.specification :refer [api-specification ->url]] + [datahike.tools :as dt] + [datahike.store :as ds] + [datahike.transit :as transit] + [datahike.json :as json] + [datahike.readers :refer [edn-readers]] + [clojure.edn :as edn] + [clojure.string :as string] + [cognitect.transit :as t] + [jsonista.core :as j]) + (:import [java.io ByteArrayOutputStream ByteArrayInputStream])) + +;; ----------------------------------------------------------------------------- +;; Connection Registry +;; ----------------------------------------------------------------------------- + +;; Atom holding all active connections managed by the router. +;; Key: store identity vector [backend scope identifier], Value: connection object. +(defonce router-connections (atom {})) + +(defn get-connection + "Retrieves a connection for the given config or store-identity from the router's registry. + Accepts either: + - A config map: extracts store identity and looks up connection + - A store identity vector: directly looks up connection + Returns nil if no connection exists." + [config-or-store-id] + (let [store-id (if (vector? config-or-store-id) + config-or-store-id + (ds/store-identity (:store config-or-store-id)))] + (get @router-connections store-id))) + +(defn list-connections + "Returns a map of all active connections in the router's registry." + [] + @router-connections) + +(defn clear-connections! + "Clears all connections from the registry. Useful for testing." + [] + (reset! router-connections {})) + +;; ----------------------------------------------------------------------------- +;; Handler functions +;; ----------------------------------------------------------------------------- + +(defn wrap-api-handler + "Wraps a Datahike API function as a Ring handler." + [api-fn fn-name] + (fn [request] + (let [;; Support both direct body and Reitit-style parameters + body (or (get-in request [:parameters :body]) + (:body-params request) + (:body request) + []) + ;; Support both vector (direct args) and map format + [config args] (if (map? body) + [(:config body) (:args body [])] + [nil body]) + result (try + (case fn-name + create-database + (let [cfg (or config (first body)) + rest-args (if config args (rest body)) + res (apply api-fn cfg rest-args)] + (if (:remote-peer cfg) + (assoc res :remote-peer (:remote-peer cfg)) + res)) + + delete-database + (let [cfg (or config (first body)) + rest-args (if config args (rest body))] + (apply api-fn (dissoc cfg :remote-peer) rest-args)) + + connect + (let [cfg (or config (first body)) + rest-args (if config args (rest body)) + conn (apply api-fn cfg rest-args) + store-id (ds/store-identity (:store cfg))] + ;; Register the connection in the router's registry using store identity + (swap! router-connections assoc store-id conn) + conn) + + release-connection + (let [conn (first body) + ;; Find and remove the connection from registry + _ (swap! router-connections + (fn [conns] + (into {} (remove (fn [[_ v]] (identical? v conn)) conns))))] + (apply api-fn body)) + + (apply api-fn body)) + (catch Exception e + {:error (str "Error in " fn-name ": " (.getMessage e)) + :type (str (type e))}))] + {:status (if (:error result) 500 200) + :body result}))) + +(defn wrap-writer-handler + "Wraps writer operations for distributed mode only." + [api-fn fn-name] + (fn [request] + (let [body (or (get-in request [:parameters :body]) + (:body-params request) + (:body request) + []) + result (try + (case fn-name + delete-database-writer + (let [cfg (dissoc (first body) :remote-peer :writer)] + (apply d/delete-database cfg (rest body))) + + create-database-writer + (let [cfg (dissoc (first body) :remote-peer :writer)] + (apply d/create-database cfg (rest body))) + + transact!-writer + (let [cfg (dissoc (first body) :remote-peer :writer)] + (let [conn (d/connect cfg) + [_ tx-data tx-meta] body] + @(apply d/transact! conn tx-data (when tx-meta [tx-meta])))) + + (apply api-fn body)) + (catch Exception e + {:error (str "Error in writer " fn-name ": " (.getMessage e)) + :type (str (type e))}))] + {:status (if (:error result) 500 200) + :body result}))) + +;; ----------------------------------------------------------------------------- +;; Route generation +;; ----------------------------------------------------------------------------- + +(defn api-fn-name->path + "Converts function name to URL path (e.g., database-exists? -> /database-exists)." + [fn-name] + (str "/" (->url fn-name))) + +(defn generate-api-routes + "Generates routes for all remotely-supported API functions. + Each route has :path, :method, :handler, :name, and :doc keys." + [] + (vec + (for [[fn-name {:keys [supports-remote? referentially-transparent? doc]}] api-specification + :when supports-remote?] + (let [api-fn (resolve (symbol "datahike.api" (name fn-name))) + method (if referentially-transparent? :get :post)] + {:path (api-fn-name->path fn-name) + :method method + :handler (wrap-api-handler api-fn fn-name) + :name (keyword fn-name) + :doc doc})))) + +(defn generate-writer-routes + "Generates writer routes for distributed mode only." + [] + [{:path "/delete-database-writer" + :method :post + :handler (wrap-writer-handler d/delete-database 'delete-database-writer) + :name :delete-database-writer + :doc "Internal endpoint for distributed writer - DO NOT USE DIRECTLY"} + + {:path "/create-database-writer" + :method :post + :handler (wrap-writer-handler d/create-database 'create-database-writer) + :name :create-database-writer + :doc "Internal endpoint for distributed writer - DO NOT USE DIRECTLY"} + + {:path "/transact!-writer" + :method :post + :handler (wrap-writer-handler d/transact! 'transact!-writer) + :name :transact!-writer + :doc "Internal endpoint for distributed writer - DO NOT USE DIRECTLY"}]) + +;; ----------------------------------------------------------------------------- +;; Minimal middleware for embedded use +;; ----------------------------------------------------------------------------- + +(defn- detect-content-type + "Detects content type from request headers." + [request] + (or (get-in request [:headers "content-type"]) + (get-in request [:headers "Content-Type"]) + "application/edn")) + +(defn- detect-accept-type + "Detects accept type from request headers, defaults to content-type or edn." + [request] + (or (get-in request [:headers "accept"]) + (get-in request [:headers "Accept"]) + (detect-content-type request) + "application/edn")) + +(defn- parse-edn-body + "Parses EDN format body." + [body] + (cond + (instance? java.io.InputStream body) + (edn/read {:readers edn-readers} (java.io.PushbackReader. (java.io.InputStreamReader. body))) + + (string? body) + (edn/read-string {:readers edn-readers} body) + + :else body)) + +(defn- parse-transit-body + "Parses transit+json format body." + [body] + (cond + (instance? java.io.InputStream body) + (let [reader (t/reader body :json {:handlers transit/read-handlers})] + (t/read reader)) + + (string? body) + (let [in (ByteArrayInputStream. (.getBytes ^String body)) + reader (t/reader in :json {:handlers transit/read-handlers})] + (t/read reader)) + + (bytes? body) + (let [in (ByteArrayInputStream. body) + reader (t/reader in :json {:handlers transit/read-handlers})] + (t/read reader)) + + :else body)) + +(defn- parse-json-body + "Parses JSON format body." + [body] + (cond + (instance? java.io.InputStream body) + (j/read-value body json/mapper) + + (string? body) + (j/read-value body json/mapper) + + (bytes? body) + (j/read-value body json/mapper) + + :else body)) + +(defn wrap-parse-body + "Parses request body based on content-type." + [handler] + (fn [request] + (if-let [body (:body request)] + (let [content-type (detect-content-type request) + parsed-body (cond + (string/includes? content-type "application/edn") + (parse-edn-body body) + + (string/includes? content-type "application/transit+json") + (parse-transit-body body) + + (string/includes? content-type "application/json") + (parse-json-body body) + + :else + (parse-edn-body body))] + (handler (assoc request :body parsed-body))) + (handler request)))) + +(defn- serialize-edn + "Serializes response body to EDN." + [body] + (pr-str body)) + +(defn- serialize-transit + "Serializes response body to transit+json." + [body] + (let [out (ByteArrayOutputStream.) + writer (t/writer out :json {:handlers transit/write-handlers})] + (t/write writer body) + (.toByteArray out))) + +(defn- serialize-json + "Serializes response body to JSON." + [body] + (j/write-value-as-bytes body json/mapper)) + +(defn wrap-format-response + "Handles response formatting and serialization based on Accept header." + [handler] + (fn [request] + (let [response (handler request) + body (:body response) + accept-type (detect-accept-type request)] + (try + (let [[serialized-body content-type] + (cond + (string/includes? accept-type "application/transit+json") + [(serialize-transit body) "application/transit+json"] + + (string/includes? accept-type "application/json") + [(serialize-json body) "application/json"] + + :else + [(serialize-edn body) "application/edn"])] + (if (and (map? response) (contains? response :body)) + (assoc response + :body serialized-body + :headers (merge {"Content-Type" content-type} + (:headers response))) + {:status 200 + :body serialized-body + :headers {"Content-Type" content-type}})) + (catch Exception e + ;; If serialization fails, fall back to EDN + {:status 500 + :body (pr-str {:error (str "Serialization error: " (.getMessage e)) + :type (str (type e))}) + :headers {"Content-Type" "application/edn"}}))))) + +(defn wrap-error-handling + "Basic error handling middleware." + [handler] + (fn [request] + (try + (handler request) + (catch Exception e + {:status 500 + :body {:error (.getMessage e) + :type (str (type e))}})))) + +;; ----------------------------------------------------------------------------- +;; Authentication middleware (compatible with server config) +;; ----------------------------------------------------------------------------- + +(defn extract-token + "Extracts token from Authorization header (supports 'token' and 'Bearer' formats)." + [request] + (when-let [auth-header (get-in request [:headers "authorization"])] + (let [parts (clojure.string/split auth-header #" ")] + (when (>= (count parts) 2) + (case (clojure.string/lower-case (first parts)) + "token" (second parts) + "bearer" (second parts) + nil))))) + +(defn wrap-token-auth + "Token authentication middleware. Uses :token from config, bypasses if :dev-mode is true." + [handler config] + (if (or (:dev-mode config) (nil? (:token config))) + handler + (fn [request] + (let [token (extract-token request)] + (if (= token (:token config)) + (handler request) + {:status 401 + :body {:error "Not authorized"}}))))) + +;; ----------------------------------------------------------------------------- +;; Route compilation for different routers +;; ----------------------------------------------------------------------------- + +(defn routes-for-reitit + "Converts routes to Reitit format. + Options: + - :include-writers? - Include internal writer routes (default false) + - :prefix - URL prefix for all routes (e.g., \"/datahike\" or \"/api/db\") + - :middleware - Additional middleware to apply" + [& {:keys [include-writers? prefix middleware] + :or {include-writers? false + prefix "" + middleware []}}] + (let [api-routes (generate-api-routes) + writer-routes (when include-writers? (generate-writer-routes)) + all-routes (concat api-routes writer-routes)] + (vec + (for [{:keys [path method handler name doc]} all-routes] + [(str prefix path) + {method handler + :name name + :summary doc + :middleware middleware}])))) + +(defn routes-for-compojure + "Converts routes to Compojure format. + Returns a function that can be used with defroutes. + Options: + - :include-writers? - Include internal writer routes (default false) + - :prefix - URL prefix for all routes (e.g., \"/datahike\" or \"/api/db\")" + [& {:keys [include-writers? prefix] + :or {include-writers? false + prefix ""}}] + (let [api-routes (generate-api-routes) + writer-routes (when include-writers? (generate-writer-routes)) + all-routes (concat api-routes writer-routes)] + (fn [] + (vec + (for [{:keys [path method handler]} all-routes] + (let [full-path (str prefix path)] + (case method + :get `(~'GET ~full-path request# (~handler request#)) + :post `(~'POST ~full-path request# (~handler request#))))))))) + +(defn routes-for-ring + "Returns a simple Ring handler that matches routes. + Options: + - :include-writers? - Include internal writer routes (default false) + - :prefix - URL prefix for all routes (e.g., \"/datahike\" or \"/api/db\") + - :not-found-handler - Handler for unmatched routes" + [& {:keys [include-writers? prefix not-found-handler] + :or {include-writers? false + prefix "" + not-found-handler (fn [_] {:status 404 :body "Not found"})}}] + (let [api-routes (generate-api-routes) + writer-routes (when include-writers? (generate-writer-routes)) + all-routes (concat api-routes writer-routes) + route-map (reduce + (fn [m {:keys [path method handler]}] + (assoc-in m [(str prefix path) method] handler)) + {} + all-routes)] + (fn [request] + (if-let [handler (get-in route-map [(:uri request) (:request-method request)])] + (handler request) + (not-found-handler request))))) + +;; ----------------------------------------------------------------------------- +;; Public API +;; ----------------------------------------------------------------------------- + +(defn create-routes + "Creates routes for Datahike HTTP API. + + Options: + - :format - Router format (:reitit, :compojure, :ring, or :raw) + - :prefix - URL prefix for all routes (e.g., \"/datahike\" or \"/api/db\") + - :include-writers? - Include internal writer routes (for distributed mode) + - :middleware - Additional middleware (for formats that support it) + + Returns routes in the specified format: + - :raw - Vector of route maps (prefix added to path field) + - :reitit - Reitit-compatible route data + - :compojure - Compojure-compatible routes + - :ring - Simple Ring handler function" + [& {:keys [format prefix include-writers? middleware] + :or {format :raw + prefix "" + include-writers? false + middleware []}}] + (case format + :raw (let [routes (concat (generate-api-routes) + (when include-writers? (generate-writer-routes)))] + (if (empty? prefix) + routes + (map #(update % :path (fn [p] (str prefix p))) routes))) + :reitit (routes-for-reitit :include-writers? include-writers? + :prefix prefix + :middleware middleware) + :compojure (routes-for-compojure :include-writers? include-writers? + :prefix prefix) + :ring (routes-for-ring :include-writers? include-writers? + :prefix prefix) + (let [routes (concat (generate-api-routes) + (when include-writers? (generate-writer-routes)))] + (if (empty? prefix) + routes + (map #(update % :path (fn [p] (str prefix p))) routes))))) + +(defn create-ring-handler + "Creates a Ring handler for embedded use. + + Options: + - :config - Auth config (:token, :dev-mode, :level) + - :prefix - URL prefix (e.g., \"/datahike\", \"/api/db\") + - :include-writers? - Include writer routes (default false) + - :middleware - Additional Ring middleware + - :not-found-handler - Custom 404 handler + + Supports EDN, Transit+JSON, and JSON serialization based on Content-Type and Accept headers." + [& {:keys [config prefix include-writers? middleware not-found-handler] + :or {config {} + prefix "" + include-writers? false + middleware [] + not-found-handler (fn [_] {:status 404 :body "Not found"})}}] + (let [base-handler (routes-for-ring :include-writers? include-writers? + :prefix prefix + :not-found-handler not-found-handler) + handler-with-auth (if (or (:token config) (:dev-mode config)) + (wrap-token-auth base-handler config) + base-handler) + handler-with-format (-> handler-with-auth + wrap-parse-body + wrap-error-handling + wrap-format-response)] + (reduce (fn [h mw] (mw h)) + handler-with-format + (reverse middleware)))) diff --git a/test/datahike/test/http/router_test.clj b/test/datahike/test/http/router_test.clj new file mode 100644 index 000000000..fbebaac0d --- /dev/null +++ b/test/datahike/test/http/router_test.clj @@ -0,0 +1,525 @@ +(ns datahike.test.http.router-test + (:require + [clojure.test :refer :all] + [clojure.set :as set] + [datahike.http.router :as router] + [datahike.http.server :as server] + [datahike.http.client :as client] + [datahike.store :as ds] + [datahike.api.specification :refer [api-specification]] + [datahike.api :as d] + [ring.adapter.jetty :as jetty])) + +(deftest test-route-generation + (testing "API routes are generated correctly" + (let [routes (router/create-routes :format :raw :include-writers? false)] + (is (seq routes) "Routes should not be empty") + + ;; Check for essential routes + (let [route-paths (set (map :path routes))] + (is (contains? route-paths "/create-database")) + (is (contains? route-paths "/delete-database")) + (is (contains? route-paths "/connect")) + (is (contains? route-paths "/transact")) + (is (contains? route-paths "/q")) + (is (contains? route-paths "/db")) + (is (contains? route-paths "/pull"))) + + ;; Check route structure + (doseq [route routes] + (is (contains? route :path)) + (is (contains? route :method)) + (is (contains? route :handler)) + (is (contains? route :name)) + (is (fn? (:handler route))) + (is (#{:get :post} (:method route))))))) + +(deftest test-writer-routes + (testing "Writer routes are generated when requested" + (let [routes (router/generate-writer-routes)] + (is (= 3 (count routes)) "Should have 3 writer routes") + + (let [route-paths (set (map :path routes))] + (is (contains? route-paths "/create-database-writer")) + (is (contains? route-paths "/delete-database-writer")) + (is (contains? route-paths "/transact!-writer")))))) + +(deftest test-reitit-format + (testing "Routes can be converted to Reitit format" + (let [routes (router/create-routes :format :reitit :include-writers? false)] + (is (vector? routes)) + (is (seq routes)) + + ;; Check Reitit structure + (doseq [route routes] + (is (vector? route)) + (is (string? (first route))) + (is (map? (second route))))))) + +(deftest test-ring-handler + (testing "Ring handler processes requests correctly" + (let [handler (router/create-ring-handler :include-writers? false)] + + ;; Test 404 for unknown route + (let [response (handler {:uri "/unknown" :request-method :get})] + (is (= 404 (:status response)))) + + ;; Test that database-exists? route exists + (let [response (handler {:uri "/database-exists" + :request-method :post + :body [{:store {:backend :mem + :id "test"}}]})] + ;; Will return false or error, but should not be 404 + (is (not= 404 (:status response))))))) + +(deftest test-handler-wrapper + (testing "Handler wrapper extracts parameters correctly" + (let [routes (router/create-routes :format :raw) + create-db-route (first (filter #(= "/create-database" (:path %)) routes)) + handler (:handler create-db-route) + test-id-1 (str "test-db-" (System/currentTimeMillis)) + test-id-2 (str "test-db-" (System/currentTimeMillis) "-2") + test-id-3 (str "test-db-" (System/currentTimeMillis) "-3")] + + ;; Test with vector format (original format) + (let [response (handler {:body [{:store {:backend :mem + :id test-id-1}}]})] + (is (= 200 (:status response))) + (d/delete-database {:store {:backend :mem :id test-id-1}})) + + ;; Test with map format (for compatibility) + (let [response (handler {:body-params {:config {:store {:backend :mem + :id test-id-2}} + :args []}})] + (is (= 200 (:status response))) + (d/delete-database {:store {:backend :mem :id test-id-2}})) + + ;; Test with Reitit parameters format + (let [response (handler {:parameters {:body [{:store {:backend :mem + :id test-id-3}}]}})] + (is (= 200 (:status response))) + (d/delete-database {:store {:backend :mem :id test-id-3}}))))) + +(deftest test-router-works-without-server + (testing "Router can create routes and handlers without server" + ;; The important thing is that we can use the router functionality + ;; without needing the server namespace + (is (fn? (router/create-ring-handler)) + "Should be able to create ring handler") + (is (seq (router/create-routes)) + "Should be able to create routes") + + ;; Test that the router namespace itself is loaded + (is (resolve 'datahike.http.router/create-routes) + "Router namespace should be loaded") + + ;; Verify we can handle a request without a server + (let [handler (router/create-ring-handler :config {:dev-mode true})] + (is (map? (handler {:uri "/unknown" :request-method :get})) + "Handler should return a response map")))) + +(deftest test-middleware-application + (testing "Middleware can be applied to handlers" + (let [call-count (atom 0) + test-middleware (fn [handler] + (fn [request] + (swap! call-count inc) + (handler request))) + handler (router/create-ring-handler + :middleware [test-middleware])] + + (handler {:uri "/unknown" :request-method :get}) + (is (= 1 @call-count) "Middleware should be called")))) + +(deftest test-authentication + (testing "Token authentication works as expected" + (let [config {:token "test-token" + :dev-mode false} + handler (router/create-ring-handler :config config)] + + ;; Test without token - should fail + (let [response (handler {:uri "/database-exists" + :request-method :post + :body [{:store {:backend :mem + :id "test"}}]})] + (is (= 401 (:status response)) "Should require authentication") + (is (= {:error "Not authorized"} (read-string (:body response))))) + + ;; Test with wrong token - should fail + (let [response (handler {:uri "/database-exists" + :request-method :post + :headers {"authorization" "token wrong-token"} + :body [{:store {:backend :mem + :id "test"}}]})] + (is (= 401 (:status response)) "Wrong token should fail")) + + ;; Test with correct token - should succeed + (let [response (handler {:uri "/database-exists" + :request-method :post + :headers {"authorization" "token test-token"} + :body [{:store {:backend :mem + :id "test"}}]})] + (is (not= 401 (:status response)) "Correct token should succeed")) + + ;; Test with Bearer format - should also work + (let [response (handler {:uri "/database-exists" + :request-method :post + :headers {"authorization" "Bearer test-token"} + :body [{:store {:backend :mem + :id "test"}}]})] + (is (not= 401 (:status response)) "Bearer format should work"))))) + +(deftest test-dev-mode + (testing "Dev mode bypasses authentication" + (let [config {:token "test-token" + :dev-mode true} ; Dev mode enabled + handler (router/create-ring-handler :config config)] + + ;; Even without token, should succeed in dev mode + (let [response (handler {:uri "/database-exists" + :request-method :post + :body [{:store {:backend :mem + :id "test"}}]})] + (is (not= 401 (:status response)) "Dev mode should bypass auth"))))) + +(deftest test-prefix-routing + (testing "Prefix parameter correctly prepends to all routes" + (let [handler (router/create-ring-handler + :config {:dev-mode true} ; Skip auth for simplicity + :prefix "/datahike")] + + ;; Test that route without prefix returns 404 + (let [response (handler {:uri "/database-exists" + :request-method :post + :body [{:store {:backend :mem + :id "test"}}]})] + (is (= 404 (:status response)) "Route without prefix should not exist")) + + ;; Test that route with prefix works + (let [response (handler {:uri "/datahike/database-exists" + :request-method :post + :body [{:store {:backend :mem + :id "test"}}]})] + (is (not= 404 (:status response)) "Route with prefix should work")))) + + (testing "Nested prefix works correctly" + (let [handler (router/create-ring-handler + :config {:dev-mode true} + :prefix "/api/v1/db")] + + ;; Test with nested prefix + (let [response (handler {:uri "/api/v1/db/database-exists" + :request-method :post + :body [{:store {:backend :mem + :id "test"}}]})] + (is (not= 404 (:status response)) "Nested prefix should work")))) + + (testing "Empty prefix works (routes at root)" + (let [handler (router/create-ring-handler + :config {:dev-mode true})] ; No prefix specified + + ;; Test at root level + (let [response (handler {:uri "/database-exists" + :request-method :post + :body [{:store {:backend :mem + :id "test"}}]})] + (is (not= 404 (:status response)) "Routes should work at root with no prefix"))))) + +;; Integration test +(deftest test-embedded-usage + (testing "Routes work in embedded mode" + (let [cfg {:store {:backend :mem + :id "embedded-test"}} + handler (router/create-ring-handler :include-writers? false)] + + ;; Create database + (d/create-database cfg) + + ;; Test database-exists? through handler + (let [response (handler {:uri "/database-exists" + :request-method :post + :body [cfg]})] + (is (= 200 (:status response))) + (is (true? (read-string (:body response))))) + + ;; Clean up + (d/delete-database cfg)))) + +(deftest test-content-negotiation + (testing "Router supports multiple serialization formats" + (let [cfg {:store {:backend :mem :id "content-test"}} + handler (router/create-ring-handler :include-writers? false)] + + ;; Create database for testing + (d/create-database cfg) + + ;; Test EDN format (default) + (let [response (handler {:uri "/database-exists" + :request-method :post + :headers {"content-type" "application/edn" + "accept" "application/edn"} + :body (pr-str [cfg])})] + (is (= 200 (:status response))) + (is (= "application/edn" (get-in response [:headers "Content-Type"]))) + (is (string? (:body response))) + (is (true? (read-string (:body response))))) + + ;; Test Transit+JSON format + (let [out (java.io.ByteArrayOutputStream.) + writer (cognitect.transit/writer out :json {:handlers datahike.transit/write-handlers}) + _ (cognitect.transit/write writer [cfg]) + response (handler {:uri "/database-exists" + :request-method :post + :headers {"content-type" "application/transit+json" + "accept" "application/transit+json"} + :body (.toByteArray out)})] + (is (= 200 (:status response)) "Transit request should succeed") + (when (not= 200 (:status response)) + (println "Transit error:" (:body response))) + (is (= "application/transit+json" (get-in response [:headers "Content-Type"]))) + (is (bytes? (:body response))) + (let [in (java.io.ByteArrayInputStream. (:body response)) + reader (cognitect.transit/reader in :json {:handlers datahike.transit/read-handlers})] + (is (true? (cognitect.transit/read reader))))) + + ;; Test JSON format - use the datahike JSON mapper + (let [json-body (jsonista.core/write-value-as-bytes [cfg] datahike.json/mapper) + response (handler {:uri "/database-exists" + :request-method :post + :headers {"content-type" "application/json" + "accept" "application/json"} + :body json-body})] + (is (= 200 (:status response)) "JSON request should succeed") + (when (not= 200 (:status response)) + (println "JSON error body:" (jsonista.core/read-value (:body response) datahike.json/mapper))) + (is (= "application/json" (get-in response [:headers "Content-Type"]))) + (is (bytes? (:body response))) + (is (true? (jsonista.core/read-value (:body response) datahike.json/mapper)))) + + ;; Clean up + (d/delete-database cfg)))) + +;; Compatibility test with server +(deftest test-router-matches-server-routes + (testing "Router routes match server routes exactly" + (let [config {} + ;; Get routes from server (uses eval approach) + server-routes (server/create-routes config) + + ;; Extract route info from server (Reitit format) + server-route-info (into #{} + (map (fn [[path route-data]] + (let [method (if (:get route-data) :get :post)] + {:path path + :method method + :operation-id (get-in route-data [method :operationId])})) + server-routes)) + + ;; Get routes from router (raw format) + router-routes (router/generate-api-routes) + + ;; Extract route info from router + router-route-info (into #{} + (map (fn [{:keys [path method name]}] + {:path path + :method method + :operation-id (str (clojure.core/name name))}) + router-routes))] + + ;; Check counts match + (is (= (count server-route-info) (count router-route-info)) + (str "Route counts should match. Server: " (count server-route-info) + ", Router: " (count router-route-info))) + + ;; Check all server routes exist in router + (is (set/subset? server-route-info router-route-info) + "All server routes should exist in router routes") + + ;; Check all router routes exist in server + (is (set/subset? router-route-info server-route-info) + "All router routes should exist in server routes") + + ;; Verify they are identical + (is (= server-route-info router-route-info) + "Server and router routes should be identical"))) + + (testing "Writer routes match between router and server" + (let [server-connections (atom {}) + ;; Get writer routes from server + server-writer-routes (server/internal-writer-routes server-connections) + + ;; Extract info from server writer routes + server-writer-info (into #{} + (map (fn [[path route-data]] + {:path path + :method :post + :operation-id (get-in route-data [:post :operationId])}) + server-writer-routes)) + + ;; Get writer routes from router + router-writer-routes (router/generate-writer-routes) + + ;; Extract info from router writer routes + router-writer-info (into #{} + (map (fn [{:keys [path method name]}] + {:path path + :method method + :operation-id (clojure.core/name name)}) + router-writer-routes))] + + ;; Check counts match + (is (= 3 (count server-writer-info) (count router-writer-info)) + "Should have 3 writer routes in both") + + ;; Check paths match + (is (= (set (map :path server-writer-info)) + (set (map :path router-writer-info))) + "Writer route paths should match")))) + +(deftest test-connection-registry-sync + (testing "Connection registry maintains consistency across all connection types" + (let [test-port 3030 + db-path (str "/tmp/router-test-sync-" (System/currentTimeMillis)) + handler (router/create-ring-handler :config {:dev-mode true}) + server (jetty/run-jetty handler {:port test-port :join? false})] + + (try + (Thread/sleep 500) ; Let server start + + ;; Define configs for different connection types + (let [http-client-cfg {:store {:backend :file :path db-path} + :remote-peer {:backend :datahike-server + :url (str "http://localhost:" test-port)} + :allow-unsafe-config true} + writer-cfg {:store {:backend :file :path db-path} + :writer {:backend :self} + :allow-unsafe-config true} + remote-peer-cfg {:store {:backend :file :path db-path} + :remote-peer {:backend :datahike-server + :url (str "http://localhost:" test-port)} + :allow-unsafe-config true} + local-cfg {:store {:backend :file :path db-path} + :allow-unsafe-config true}] + + (try + ;; Clean up + (try (client/delete-database http-client-cfg) (catch Exception _)) + (router/clear-connections!) + + ;; Phase 1: HTTP CLIENT creates DB and writes + (client/create-database http-client-cfg) + (let [http-conn (client/connect http-client-cfg)] + + (client/transact http-conn [{:db/ident :test/source + :db/valueType :db.type/string + :db/cardinality :db.cardinality/one} + {:db/ident :test/value + :db/valueType :db.type/long + :db/cardinality :db.cardinality/one}]) + + (client/transact http-conn [{:test/source "http-client" :test/value 1}]) + + ;; Phase 2: MAIN PROCESS accesses router connection + (let [main-conn (router/get-connection local-cfg)] + (is (some? main-conn) "Should retrieve connection from router registry") + + (let [main-db @main-conn + main-data (d/q '[:find ?source ?val + :where [?e :test/source ?source] + [?e :test/value ?val]] + main-db)] + (is (= #{["http-client" 1]} main-data) + "MAIN should see HTTP CLIENT data")) + + ;; Phase 3: WRITER writes + (let [writer-conn (d/connect writer-cfg)] + (d/transact writer-conn [{:test/source "writer" :test/value 2}]) + + ;; Verify MAIN sees WRITER data + (let [main-db-2 @main-conn + main-data-2 (d/q '[:find ?source ?val + :where [?e :test/source ?source] + [?e :test/value ?val]] + main-db-2)] + (is (= 2 (count main-data-2)) + "MAIN should see both HTTP CLIENT and WRITER data")) + + ;; Verify HTTP CLIENT sees WRITER data + (let [http-db-2 (client/db http-conn) + http-data-2 (client/q '[:find ?source ?val + :where [?e :test/source ?source] + [?e :test/value ?val]] + http-db-2)] + (is (= 2 (count http-data-2)) + "HTTP CLIENT should see WRITER data")) + + ;; Phase 4: REMOTE-PEER writes + (let [remote-peer-conn (d/connect remote-peer-cfg)] + (d/transact remote-peer-conn [{:test/source "remote-peer" :test/value 3}]) + + ;; Verify all processes see all data + (let [main-db-3 @main-conn + http-db-3 (client/db http-conn) + writer-db-3 @writer-conn + remote-db-3 @remote-peer-conn + + main-data (d/q '[:find ?source ?val + :where [?e :test/source ?source] + [?e :test/value ?val]] + main-db-3) + http-data (client/q '[:find ?source ?val + :where [?e :test/source ?source] + [?e :test/value ?val]] + http-db-3) + writer-data (d/q '[:find ?source ?val + :where [?e :test/source ?source] + [?e :test/value ?val]] + writer-db-3) + remote-data (d/q '[:find ?source ?val + :where [?e :test/source ?source] + [?e :test/value ?val]] + remote-db-3)] + + (is (= 3 (count main-data)) "MAIN should see 3 records") + (is (= 3 (count http-data)) "HTTP CLIENT should see 3 records") + (is (= 3 (count writer-data)) "WRITER should see 3 records") + (is (= 3 (count remote-data)) "REMOTE-PEER should see 3 records") + + ;; All should see identical data + (is (= main-data http-data writer-data remote-data) + "All connection types should see identical data")) + + ;; Phase 5: Cross-process writes + (d/transact main-conn [{:test/source "main" :test/value 100}]) + (client/transact http-conn [{:test/source "http-round2" :test/value 200}]) + (d/transact writer-conn [{:test/source "writer-round2" :test/value 300}]) + (d/transact remote-peer-conn [{:test/source "remote-round2" :test/value 400}]) + + ;; Final verification: all see all 7 records + (let [final-main (d/q '[:find ?source :where [_ :test/source ?source]] @main-conn) + final-http (client/q '[:find ?source :where [_ :test/source ?source]] + (client/db http-conn)) + final-writer (d/q '[:find ?source :where [_ :test/source ?source]] @writer-conn) + final-remote (d/q '[:find ?source :where [_ :test/source ?source]] @remote-peer-conn)] + + (is (= 7 (count final-main)) "MAIN should see all 7 records") + (is (= 7 (count final-http)) "HTTP CLIENT should see all 7 records") + (is (= 7 (count final-writer)) "WRITER should see all 7 records") + (is (= 7 (count final-remote)) "REMOTE-PEER should see all 7 records") + + (is (= final-main final-http final-writer final-remote) + "All processes maintain perfect consistency")) + + ;; Test store identity vector access + (let [store-id (ds/store-identity (:store local-cfg)) + conn-by-id (router/get-connection store-id)] + (is (some? conn-by-id) "Should retrieve connection by store identity vector") + (is (identical? main-conn conn-by-id) + "Both access methods should return same connection object")))))) + + (finally + (try (client/delete-database http-client-cfg) (catch Exception _)) + (router/clear-connections!)))) + + (finally + (.stop server)))))) \ No newline at end of file