Dictionary pack mechanism + lazy loading (1.0.0)

.ruby-version

@@ -0,0 +1 @@
+4.0.1

README.md

@@ -240,9 +240,91 @@ MyBrand
 SpecialTerm
 ```
 
+## Dictionary Packs
+
+SpellKit still bundles no dictionaries. A **pack** is a separate gem that carries the data
+and registers itself when it loads, so you choose packs in your Gemfile:
+
+```ruby
+gem "spellkit"
+gem "spellkit-general-medical"   # drug, condition and gene vocabulary + general English
+```
+
+```ruby
+SpellKit.enable_dictionary(:general_medical, lazy: true)
+SpellKit.correct("acetaminphen")   # => "acetaminophen"
+```
+
+A pack ships the tuning that was measured against its own data, so you don't have to
+rediscover `edit_distance` and `frequency_threshold` yourself. Override anything:
+
+```ruby
+SpellKit.enable_dictionary(:general_medical, edit_distance: 1)
+```
+
+SpellKit knows the name of no pack — it provides only the mechanism, so a pack can ship
+new terms or new tuning without SpellKit releasing anything.
+
+### If you load this from a Rails initializer, read this
+
+**An initializer runs in every process that boots the app** — web, `rails db:migrate`,
+`rake`, `console`, sidecars — not just the one that searches. Loading a pack eagerly makes
+all of them pay, and a ~200k-term pack at `edit_distance: 2` measures **~2.1 GB resident
+and ~5.5s**. That has already OOM-killed a memory-constrained migrate init container in
+production.
+
+Pass `lazy: true`. Registration then costs nothing and the index is built on first real
+use, which a migrate or rake process never reaches:
+
+| | time | RSS |
+|---|---|---|
+| `enable_dictionary(..., lazy: true)` | 0.000s | +0 MB |
+| first `correct()` after that | ~5.5s | +2,076 MB |
+| every later call | ~0.0001s | +0 MB |
+
+**Lazy moves the cost, it doesn't remove it.** For a web server you usually want the
+server to pay it rather than the first user, so warm it at worker boot:
+
+```ruby
+# config/puma.rb
+on_worker_boot { SpellKit.load_dictionary! }
+```
+
+`stats` and `healthcheck` deliberately **do not** trigger the load — they report
+`{"loaded" => false, "deferred" => true}`. A liveness probe must not be able to build the
+index in the very process `lazy` protects.
+
+```ruby
+SpellKit.dictionary_loaded?   # false until something forces the load
+SpellKit.load_dictionary!     # force it now; idempotent, no-op when eager
+```
+
+Memory scales steeply with `edit_distance`: the same pack is **484 MB at 1** versus
+**2.1 GB at 2**, because SymSpell's deletion index grows sharply with distance. If you are
+memory-constrained, that knob matters more than lazy loading does.
+
+### Writing a pack gem
+
+A pack gem ships its data and one registration call:
+
+```ruby
+# lib/spellkit-my-domain.rb
+require "spellkit"
+
+SpellKit::Packs.register(:my_domain,
+  dictionary: File.expand_path("../data/dictionary.tsv", __dir__),
+  protected_path: File.expand_path("../data/protected.txt", __dir__),
+  defaults: {edit_distance: 1, frequency_threshold: 10.0},
+  summary: "What this pack covers")
+```
+
+Bundler requires it automatically, so `SpellKit.enable_dictionary(:my_domain)` then works.
+Registration verifies the files exist, so a packaging mistake fails while the stack still
+points at your gem.
+
 ## Dictionary Sources
 
-SpellKit doesn't bundle dictionaries, but works with several sources:
+SpellKit works with several raw dictionary sources directly:
 
 ### Use the Default Dictionary (Recommended)
 ```ruby

lib/spellkit.rb

@@ -17,13 +17,20 @@
   require "spellkit/spellkit"
 end
 
+require_relative "spellkit/packs"
+require_relative "spellkit/lazy_checker"
+
 module SpellKit
   class Error < StandardError; end
   class NotLoadedError < Error; end
   class FileNotFoundError < Error; end
   class InvalidArgumentError < Error; end
   class DownloadError < Error; end
 
+  # Raised when a pack name is not registered - almost always a data gem missing from
+  # the Gemfile, since a pack registers itself when its gem loads.
+  class UnknownPackError < Error; end
+
   # Default dictionary: SymSpell English 80k frequency dictionary
   DEFAULT_DICTIONARY_URL = "https://raw.githubusercontent.com/wolfgarbe/SymSpell/master/SymSpell.FrequencyDictionary/en-80k.txt"
 
@@ -98,6 +105,83 @@ def stats
     def healthcheck
       default.healthcheck
     end
+
+    # ----------------------------------------------------------------------------------
+    # Dictionary packs
+    #
+    # SpellKit still bundles no dictionaries. A pack lives in its own gem, registers
+    # itself on load (see SpellKit::Packs), and brings the tuning that was measured
+    # against its own data. These methods are the mechanism only - SpellKit knows the
+    # name of no pack.
+    # ----------------------------------------------------------------------------------
+
+    # Configure the DEFAULT checker from a registered pack, so plain SpellKit.correct
+    # becomes domain-aware.
+    #
+    #   SpellKit.enable_dictionary(:general_medical)
+    #   SpellKit.enable_dictionary(:general_medical, lazy: true)     # defer the index build
+    #   SpellKit.enable_dictionary(:general_medical, edit_distance: 1)
+    #   SpellKit.enable_dictionary(dictionary: "my.tsv")             # your own files
+    #
+    # `lazy: true` is strongly recommended from a Rails initializer - see LazyChecker for
+    # why. Eager stays the default so this is non-breaking.
+    def enable_dictionary(pack = nil, lazy: false, **options)
+      # Resolve eagerly even when lazy, so an unregistered pack (usually a data gem
+      # missing from the Gemfile) fails at boot rather than on a user's first search.
+      @dictionary_pack = pack.nil? ? nil : Packs.fetch(pack)
+
+      self.default = if lazy
+        LazyChecker.new(pack, options)
+      else
+        load!(**pack_load_options(pack, **options))
+      end
+    end
+
+    # An INDEPENDENT checker, for running more than one pack at once. Does not touch the
+    # default checker.
+    def dictionary_checker(pack = nil, lazy: false, **options)
+      Packs.fetch(pack) unless pack.nil?
+      return LazyChecker.new(pack, options) if lazy
+
+      Checker.new.load!(**pack_load_options(pack, **options))
+    end
+
+    # True once the default checker holds a real index.
+    def dictionary_loaded?
+      checker = @default
+      return false if checker.nil?
+      return checker.loaded? if checker.is_a?(LazyChecker)
+
+      true
+    end
+
+    # Force a deferred load now; no-op when already loaded or configured eagerly.
+    def load_dictionary!
+      checker = @default
+      checker.load_now! if checker.is_a?(LazyChecker)
+      checker
+    end
+
+    # The pack backing the default checker, or nil when configured from raw files.
+    attr_reader :dictionary_pack
+
+    private
+
+    # Internal: resolve a pack name (or a bare options hash) into load! keyword arguments.
+    # Private on purpose - see LazyChecker#resolved_options.
+    def pack_load_options(pack = nil, **overrides)
+      if pack.nil?
+        unless overrides.key?(:dictionary)
+          raise InvalidArgumentError,
+            "Pass a registered pack name or a dictionary: of your own. " \
+            "Registered packs: #{Packs.names.inspect}"
+        end
+
+        return overrides
+      end
+
+      Packs.fetch(pack).load_options(**overrides)
+    end
   end
 end
 
@@ -113,10 +197,9 @@ class SpellKit::Checker
   alias_method :_rust_healthcheck, :healthcheck
 
   def load!(dictionary: nil, protected_path: nil, protected_patterns: [],
-            edit_distance: 1, frequency_threshold: 10.0,
-            skip_urls: false, skip_emails: false, skip_hostnames: false,
-            skip_code_patterns: false, skip_numbers: false, **_options)
-
+    edit_distance: 1, frequency_threshold: 10.0,
+    skip_urls: false, skip_emails: false, skip_hostnames: false,
+    skip_code_patterns: false, skip_numbers: false, **_options)
     # Validate dictionary parameter
     raise SpellKit::InvalidArgumentError, "dictionary parameter is required" if dictionary.nil?
 
@@ -320,7 +403,7 @@ def download_dictionary(url)
     raise SpellKit::InvalidArgumentError, "Invalid URL: #{url} (#{e.message})"
   rescue Timeout::Error => e
     raise SpellKit::DownloadError, "Download timed out: #{url} (#{e.message})"
-  rescue StandardError => e
+  rescue => e
     raise SpellKit::DownloadError, "Failed to download dictionary: #{e.message}"
   end
 
@@ -356,13 +439,13 @@ def fetch_with_redirects(url, max_redirects: 5, open_timeout: 10, read_timeout:
         raise SpellKit::DownloadError, "HTTP #{response.code}: #{response.message} (#{url})"
       end
     end
-  rescue Net::OpenTimeout => e
+  rescue Net::OpenTimeout
     raise Timeout::Error, "Connection timeout after #{open_timeout}s: #{url}"
-  rescue Net::ReadTimeout => e
+  rescue Net::ReadTimeout
     raise Timeout::Error, "Read timeout after #{read_timeout}s: #{url}"
   rescue SocketError => e
     raise SpellKit::DownloadError, "Network error: #{e.message} (#{url})"
   rescue OpenSSL::SSL::SSLError => e
     raise SpellKit::DownloadError, "SSL verification failed: #{e.message} (#{url})"
   end
-end
+end

lib/spellkit/lazy_checker.rb

@@ -0,0 +1,91 @@
+# frozen_string_literal: true
+
+module SpellKit
+  # A checker that has not built its index yet.
+  #
+  # WHY. A Rails initializer runs in EVERY process that boots the app - web, db:migrate,
+  # rake, console, sidecars - but usually only the web server ever spell-checks anything.
+  # Loading eagerly makes all of them pay, and the cost is not small: a ~200k-term pack at
+  # edit_distance 2 measures ~2.1 GB resident and several seconds to index. That is not
+  # hypothetical - it OOM-killed a memory-constrained migrate init container in production.
+  #
+  # WHAT DOES NOT TRIGGER A LOAD. `stats` and `healthcheck` deliberately do not, and that
+  # matters more than it first appears: a liveness probe hitting a health endpoint would
+  # otherwise build the whole index in exactly the process this class exists to protect,
+  # silently reintroducing the bug. They report the deferred state instead.
+  #
+  # THREAD SAFETY. A threaded server can take two concurrent first requests; without the
+  # mutex both would build a multi-gigabyte index. The build happens at most once.
+  class LazyChecker
+    # Lookups need a real index. Introspection must not build one.
+    FORCES_LOAD = %i[correct correct? suggestions correct_tokens].freeze
+
+    attr_reader :pack_name
+
+    def initialize(pack_name, options)
+      @pack_name = pack_name
+      @options = options
+      @mutex = Mutex.new
+      @checker = nil
+    end
+
+    def loaded?
+      !@checker.nil?
+    end
+
+    # Build the index now. Idempotent and thread-safe.
+    #
+    # Lazy loading MOVES the cost rather than removing it, so for a web process you
+    # usually want the server to pay it instead of the first user:
+    #
+    #   # config/puma.rb
+    #   on_worker_boot { SpellKit.load_dictionary! }
+    def load_now!
+      return @checker if @checker
+
+      @mutex.synchronize do
+        # Re-check inside the lock; another thread may have loaded while we waited.
+        @checker ||= Checker.new.load!(**resolved_options)
+      end
+    end
+
+    FORCES_LOAD.each do |name|
+      define_method(name) { |*args| load_now!.public_send(name, *args) }
+    end
+
+    def stats
+      loaded? ? @checker.stats : deferred_report
+    end
+
+    def healthcheck
+      loaded? ? @checker.healthcheck : deferred_report
+    end
+
+    def method_missing(name, *args, &block)
+      if Checker.method_defined?(name)
+        load_now!.public_send(name, *args, &block)
+      else
+        super
+      end
+    end
+
+    def respond_to_missing?(name, include_private = false)
+      Checker.method_defined?(name) || super
+    end
+
+    private
+
+    # Resolved here rather than through a module-level helper so that helper can stay
+    # private: 1.0.0 freezes the public surface, and an internal seam that only exists to
+    # let one collaborator reach a private method should not be part of it.
+    def resolved_options
+      return @options if @pack_name.nil?
+
+      Packs.fetch(@pack_name).load_options(**@options)
+    end
+
+    def deferred_report
+      {"loaded" => false, "deferred" => true, "pack" => @pack_name&.to_s}
+    end
+  end
+end