sml-robots

A pure Standard ML parser and matcher for robots.txt — the Robots Exclusion Protocol standardised as RFC 9309.

No dependencies, no IO, no networking: parse turns the text of a robots.txt file into a value, and the queries answer access and crawl-delay questions deterministically. The same inputs always produce the same outputs under MLton and Poly/ML.

What it does

• Parsing follows RFC 9309 grouping: one or more consecutive User-agent lines begin a group; the Allow/Disallow/Crawl-delay lines that follow belong to it, until the next User-agent line starts a new group. Sitemap: directives are collected file-globally. Comments (#) and surrounding whitespace are stripped.
• Matching follows RFC 9309 section 2.2:
  • user-agent selection is case-insensitive; a group token matches when it is a prefix of the crawler's product token; the longest matching token wins; * is the fallback group;
  • among the selected group's rules, the rule with the longest matching path pattern wins; ties go to Allow;
  • path patterns support * (any run of characters) and a trailing $ (end-of-path anchor); an empty Disallow imposes no restriction;
  • paths are matched case-sensitively; with no matching rule, access is allowed.

Scope note. This library is intentionally dependency-free and covers robots.txt only. It does not parse sitemap.xml (which would require an XML dependency) — a deliberate simplification versus a plan that mentioned sitemap parsing. Sitemap: directives are returned as plain URL strings.

API

structure Robots : sig
  datatype rule = Allow of string | Disallow of string
  type group = { agents : string list
               , rules  : rule list
               , crawlDelay : real option }
  type robots
  val parse      : string -> robots
  val groups     : robots -> group list
  val sitemaps   : robots -> string list
  val isAllowed  : robots -> { userAgent : string, path : string } -> bool
  val crawlDelay : robots -> string -> real option
end

Example

val r = Robots.parse "User-agent: *\nDisallow: /private/\nAllow: /private/public\n"

val false = Robots.isAllowed r {userAgent = "anybot", path = "/private/secret"}
val true  = Robots.isAllowed r {userAgent = "anybot", path = "/private/public/x"}

val g = Robots.parse "User-agent: Googlebot\nDisallow: /*.gif$\nCrawl-delay: 10\n"
val false = Robots.isAllowed g {userAgent = "Googlebot", path = "/a/b.gif"}
val SOME 10.0 = Robots.crawlDelay g "Googlebot"

Running examples/demo.sml with make example prints:

Parsed 2 groups, 1 sitemap(s).

Sitemaps:
  https://example.com/sitemap.xml

Access decisions:
  anybot /private/secret -> blocked
  anybot /private/public/page -> allowed
  anybot /photos/cat.gif -> blocked
  anybot /index.html -> allowed
  Googlebot /nogoogle/x -> blocked
  Googlebot /photos/cat.gif -> allowed

Crawl-delay:
  anybot -> 10.0s
  Googlebot -> none

Build & test

Requires MLton and/or Poly/ML.

make test        # build + run the suite under MLton
make test-poly   # run the suite under Poly/ML
make all-tests   # both
make example     # build + run the demo
make clean

Installing with smlpkg

smlpkg add github.com/sjqtentacles/sml-robots
smlpkg sync

Reference lib/github.com/sjqtentacles/sml-robots/robots.mlb from your own .mlb (MLton / MLKit), or feed sources.mlb to tools/polybuild (Poly/ML).

Layout

sml.pkg                                      smlpkg manifest
Makefile                                     MLton + Poly/ML targets
.github/workflows/ci.yml                     CI: MLton + Poly/ML
lib/github.com/sjqtentacles/sml-robots/
  robots.sig    ROBOTS signature
  robots.sml    parser + RFC 9309 matcher
  sources.mlb / robots.mlb
examples/
  demo.sml      parse + access/crawl-delay/sitemap walkthrough
test/
  harness.sml   shared assertion harness
  test.sml      RFC 9309 parsing + matching vectors (31 checks)
  entry.sml / main.sml
tools/polybuild Poly/ML build wrapper

Tests

31 deterministic checks against real robots.txt fragments and RFC 9309 access decisions: longest-match precedence (specific Allow over broad Disallow), Allow-wins-ties, user-agent selection (specific over *, case-insensitive, prefix product tokens), * and $ path patterns, the / vs /$ root case, empty-Disallow allow-all, Crawl-delay (integer and fractional, per-group), Sitemap collection, comment/whitespace handling, and ruleless files. Run make all-tests to verify byte-identical output under both compilers.

License

MIT. See LICENSE.