Editor syntax highlighting

CHANGELOG.md

@@ -1,5 +1,10 @@
 # Changelog
 
+## Unreleased
+
+- Add optional Prism, CodeMirror, and highlight.js adapters, with the website using the shared CodeMirror stream parser.
+- Add VS Code and Sublime Text syntax packages for Knap Markdown, with shared generated grammars, template comments, and snippets.
+
 ## 0.4.2
 
 - Add `{# ... #}` template comments, including multiline comments, with syntax errors for unclosed comments.

README.md

@@ -180,6 +180,10 @@ Filters that deliberately preserve their input after invalid runtime data can re
 
 ## Syntax
 
+VS Code and Sublime Text packages are available in [editors/](editors/README.md). They highlight Knap inside Markdown and include template snippets. Use `.knap` for Knap templates. The packages also recognize `.knap.md`, or you can select **Knap Markdown** for an existing template.
+
+JavaScript applications can use the optional [Prism, CodeMirror, and highlight.js adapters](editors/web.md) through `knap/prism`, `knap/codemirror`, and `knap/highlightjs`.
+
 The syntax is inspired by Twig and Liquid.
 
 ```liquid

editors/README.md

@@ -0,0 +1,60 @@
+# Editor support
+
+For JavaScript applications, see the [Prism, CodeMirror, and highlight.js integration guide](web.md).
+
+Knap Markdown highlights Knap expressions alongside the editor's built-in Markdown syntax. Use `.knap` for Knap templates. The packages also recognize the legacy-compatible `.knap.md` suffix, and existing `.md` templates can use the syntax by selecting **Knap Markdown** manually. Installing the package does not change the language of ordinary Markdown files.
+
+## VS Code and compatible editors
+
+For local development, launch the editor with this repository's extension folder:
+
+```sh
+code --extensionDevelopmentPath="$PWD/editors/vscode" "$PWD/editors/examples/article.knap"
+```
+
+Use `cursor` instead of `code` for Cursor. The extension contains only declarative grammars, language configuration, and snippets; it does not run the template engine.
+
+To associate an existing template folder, add this to workspace settings:
+
+```json
+{
+  "files.associations": {
+    "templates/*.md": "knap"
+  }
+}
+```
+
+To build an installable VSIX, run `pnpm editors:build` from the repository root, then run `pnpm dlx @vscode/vsce package` from `editors/vscode`. Install the resulting file with **Extensions: Install from VSIX**. Marketplace publication is a separate maintainer action; the manifest's `obsidianmd` publisher must be verified before publishing.
+
+## Sublime Text 4
+
+Choose **Preferences → Browse Packages**, create a `Knap` folder there, and copy the contents of `editors/sublime` into it. Open the example template or select **View → Syntax → Knap Markdown** for an existing template.
+
+Sublime inherits its built-in Markdown grammar and uses a separate generated Knap expression grammar. Dedicated Markdown, YAML, HTML, and JavaScript adapters highlight templates inside frontmatter, HTML attributes, inline code, and JavaScript fences. Inheritance avoids recursive `with_prototype` expansion across embedded languages. Host scopes are removed temporarily inside Knap tags and restored afterward, so an expression in a quoted YAML value is not colored as YAML string content. A native grammar is necessary because Sublime cannot include TextMate grammars inside native syntax definitions.
+
+## Included features
+
+- Output expressions, logic tags, whitespace trimming, and multiline comments.
+- Strings and escapes, numbers, constants, variables, property access, filters, operators, and nested map expressions.
+- Markdown highlighting, including Knap inside frontmatter, HTML, and code spans.
+- Template comment toggling and snippets starting with `knap-` in both editors.
+- Template delimiter pairing and selection wrapping in VS Code.
+
+The syntax highlighter follows the website's convention of styling bare filter arguments as strings, with expression highlighting for `map`. Those arguments can still resolve to data at runtime; color does not determine their meaning. Validation, data-aware completions, rendering, and Markdown preview integration are outside this initial package. The `knap` language mode does not automatically inherit features contributed specifically for VS Code's `markdown` language ID.
+
+## Maintaining the grammars
+
+```sh
+pnpm editors:build
+pnpm check
+```
+
+`editors/build.mjs` is the source for the TextMate and core Sublime grammars and both sets of snippets. `Knap Expressions.sublime-syntax` contains the generated expression contexts; `Knap.sublime-syntax` selects the Markdown host. The Markdown, YAML, HTML, and JavaScript host adapters are hand-maintained. Generated files are checked in so editor packages can be used without installing Node.js. `pnpm editors:check`, included in `pnpm check`, rejects stale generated files.
+
+`src/language-syntax.json` shares keyword and constant definitions with the website's existing highlighters. Their colors, Markdown punctuation handling, CodeMirror integration, and HTML rendering remain in the website. The tests port its Knap fixtures and exercise VS Code's actual TextMate/Oniguruma engine with standard Markdown, YAML, HTML, and JavaScript grammars, including HTML's derivative grammar. They also check VS Code's encoded string/comment token types and verify that host highlighting resumes after tags and fences close. TextMate retains parent scopes; `meta.embedded` resets the editor token type inside a template.
+
+Sublime context references and YAML structure are checked separately; use Sublime's **Tools → Build System → Syntax Tests** on each of `sublime/syntax_test_knap.knap` and `sublime/syntax_test_knap_yaml.knap` after installation for native positive and negative scope assertions. These native checks require Sublime; the Node test suite does not execute its syntax engine.
+
+The host integration was compared against [Shopify Liquid](https://github.com/Shopify/liquid-tm-grammar), [Twig for VS Code](https://github.com/mblode/vscode-twig-language-2), and [BetterJinja](https://github.com/Sublime-Instincts/BetterJinja). The YAML adapter incorporates BetterJinja's scalar and mapping-key integration techniques; its MIT notice is included in `sublime/ThirdPartyNotices.txt`.
+
+Use **Developer: Inspect Editor Tokens and Scopes** in VS Code or **Show Scope Name** in Sublime to inspect theme behavior in `examples/article.knap`.

editors/build.mjs

@@ -0,0 +1,185 @@
+// The website's token categories and filter-argument conventions are the basis
+// for these grammars. Keep editor-specific state machines out of the runtime.
+import { existsSync, readFileSync, writeFileSync, mkdirSync } from 'node:fs';
+import { fileURLToPath } from 'node:url';
+import { dirname } from 'node:path';
+import { stringify } from 'yaml';
+
+const root = new URL('../', import.meta.url);
+const syntax = JSON.parse(readFileSync(new URL('src/language-syntax.json', root), 'utf8'));
+const check = process.argv.includes('--check');
+function output(path, contents) {
+  const file = fileURLToPath(new URL(path, root));
+  if (check) {
+    if (!existsSync(file) || readFileSync(file, 'utf8') !== contents) throw new Error(`${path} is stale. Run pnpm editors:build.`);
+  } else {
+    mkdirSync(dirname(file), { recursive: true });
+    writeFileSync(file, contents);
+  }
+}
+const json = (path, value) => output(path, JSON.stringify(value, null, 2) + '\n');
+const include = name => ({ include: `#${name}` });
+const match = (match, name) => ({ match, name });
+const capture = name => ({ 0: { name } });
+const identifier = '[A-Za-z_$][\\w$]*';
+const words = values => `\\b(?:${values.join('|')})\\b`;
+const boundary = '(?=-?\\}\\}|-?%\\}|\\{[{%])';
+const pipe = '\\|(?!\\|)';
+const filterEnd = `(?=${pipe}|-?\\}\\}|-?%\\}|\\{[{%]|\\))`;
+const repository = {
+  tags: { patterns: [include('comment'), include('output'), include('logic')] },
+  comment: {
+    name: 'comment.block.knap', begin: '\\{#', end: '#\\}',
+    beginCaptures: capture('punctuation.definition.comment.begin.knap'),
+    endCaptures: capture('punctuation.definition.comment.end.knap'),
+  },
+  output: {
+    name: 'meta.embedded.inline.knap', begin: '\\{\\{-?', end: '-?\\}\\}|(?=\\{[{%])',
+    beginCaptures: capture('punctuation.section.embedded.begin.knap'),
+    endCaptures: capture('punctuation.section.embedded.end.knap'),
+    patterns: [include('expression')],
+  },
+  logic: {
+    name: 'meta.embedded.block.knap', begin: '\\{%-?', end: '-?%\\}|(?=\\{[{%])',
+    beginCaptures: capture('punctuation.section.embedded.begin.knap'),
+    endCaptures: capture('punctuation.section.embedded.end.knap'),
+    patterns: [include('expression')],
+  },
+  expression: { patterns: [
+    include('strings'),
+    // A double pipe is boolean OR, never the start of a filter.
+    match('\\|\\|', 'keyword.operator.knap'),
+    include('map-filter'), include('filter'),
+    include('group'), include('array'), include('object'),
+    include('atoms'), match(identifier, 'variable.other.knap'),
+  ] },
+  strings: { patterns: ['"', "'"].map(quote => ({
+    name: `string.quoted.${quote === '"' ? 'double' : 'single'}.knap`,
+    begin: quote, end: quote,
+    beginCaptures: capture('punctuation.definition.string.begin.knap'),
+    endCaptures: capture('punctuation.definition.string.end.knap'),
+    patterns: [match('\\\\.', 'constant.character.escape.knap')],
+  })) },
+  'map-filter': {
+    begin: `(${pipe})\\s*(map)\\b`, end: filterEnd,
+    beginCaptures: { 1: { name: 'keyword.operator.pipe.knap' }, 2: { name: 'support.function.filter.knap' } },
+    patterns: [include('expression')],
+  },
+  filter: {
+    begin: `(${pipe})\\s*(${identifier})?`, end: filterEnd,
+    beginCaptures: { 1: { name: 'keyword.operator.pipe.knap' }, 2: { name: 'support.function.filter.knap' } },
+    patterns: [
+      // Support a filter name on the next line, including unfinished input.
+      { begin: ':', beginCaptures: capture('punctuation.separator.knap'), end: filterEnd, patterns: [include('arguments')] },
+      match(identifier, 'support.function.filter.knap'),
+    ],
+  },
+  arguments: { patterns: [
+    include('strings'), include('argument-group'),
+    match(`(\\.)\\s*(${identifier})`, 'string.unquoted.argument.knap'),
+    include('atoms'),
+    // Like the website, style bare argument fallbacks as strings. They may
+    // resolve to data at runtime; highlighting does not validate their value.
+    match(identifier, 'string.unquoted.argument.knap'),
+  ] },
+  atoms: { patterns: [
+    match(`(\\.)\\s*(${identifier})`, 'variable.other.property.knap'),
+    match(words(syntax.tags), 'keyword.control.knap'),
+    match(words(syntax.wordOperators), 'keyword.operator.word.knap'),
+    match(words(syntax.constants), 'constant.language.knap'),
+    match('(?<![\\w$])-?\\d+(?:\\.\\d+)?\\b', 'constant.numeric.knap'),
+    match('=>|==|!=|>=|<=|&&|\\|\\||\\?\\?|[=<>!+*/-]', 'keyword.operator.knap'),
+    match('\\\\.', 'constant.character.escape.knap'),
+    match('[.,:]', 'punctuation.separator.knap'),
+  ] },
+};
+for (const [name, begin, end, patterns] of [
+  ['group', '\\(', '\\)', 'expression'],
+  ['array', '\\[', '\\]', 'expression'],
+  ['object', '\\{(?![{%#])', '\\}', 'expression'],
+  ['argument-group', '\\(', '\\)', 'arguments'],
+]) repository[name] = {
+  begin, end: `${end}|${boundary}`,
+  beginCaptures: capture('punctuation.section.group.begin.knap'),
+  endCaptures: capture('punctuation.section.group.end.knap'),
+  patterns: [include(patterns)],
+};
+
+const core = { name: 'Knap', scopeName: 'source.knap', patterns: [include('tags')], repository };
+json('editors/vscode/syntaxes/knap.tmLanguage.json', core);
+json('editors/vscode/syntaxes/knap-markdown.tmLanguage.json', {
+  name: 'Knap Markdown', scopeName: 'text.html.markdown.knap',
+  patterns: [{ include: 'source.knap' }, { include: 'text.html.markdown' }],
+  injections: {
+    'L:text.html.markdown.knap - meta.embedded.inline.knap - meta.embedded.block.knap - comment.block.knap': {
+      patterns: [{ include: 'source.knap' }],
+    },
+    // Markdown normally consumes an entire inline-code span in one match,
+    // leaving no opportunity to inject template tokens inside it.
+    'L:text.html.markdown.knap - meta.embedded - comment - string - markup.inline.raw - markup.fenced_code - meta.tag': {
+      patterns: [
+        // Give the host's fenced-code rules precedence over inline backticks.
+        { include: 'text.html.markdown#fenced_code_block' },
+        {
+        name: 'markup.inline.raw.string.markdown', begin: '(`+)', end: '(?<!`)\\1(?!`)',
+        beginCaptures: capture('punctuation.definition.raw.markdown'),
+        endCaptures: capture('punctuation.definition.raw.markdown'),
+        patterns: [{ include: 'source.knap' }],
+        },
+      ],
+    },
+  },
+});
+
+// Translate the deliberately small TextMate subset above into native Sublime
+// contexts, so the inherited Markdown host can include the expression grammar.
+const contexts = {};
+const captures = value => value && new Map(Object.entries(value).map(([key, value]) => [Number(key), value.name]));
+function sublimePatterns(patterns, prefix) {
+  return patterns.map((rule, index) => {
+    if (rule.include) return { include: rule.include.slice(1) };
+    if (rule.match) return { match: rule.match, scope: rule.name };
+    const name = `${prefix}-${index}`;
+    // Unlike TextMate's meta.embedded token-type reset, Sublime can remove
+    // host scopes while a tag is active. Restore them automatically on pop.
+    const embedded = ['comment', 'output', 'logic'].includes(prefix);
+    contexts[name] = [
+      { meta_include_prototype: false },
+      ...(embedded ? [{ clear_scopes: true }] : []),
+      ...(rule.name ? [{ meta_scope: `${embedded ? 'source.knap ' : ''}${rule.name}` }] : []),
+      { match: rule.end, ...(rule.endCaptures ? { captures: captures(rule.endCaptures) } : {}), pop: true },
+      ...sublimePatterns(rule.patterns ?? [], name),
+    ];
+    return { match: rule.begin, ...(rule.beginCaptures ? { captures: captures(rule.beginCaptures) } : {}), push: name };
+  });
+}
+for (const [name, rule] of Object.entries(repository)) {
+  contexts[name] = sublimePatterns(rule.begin ? [rule] : rule.patterns, name);
+}
+output('editors/sublime/Knap Expressions.sublime-syntax', '%YAML 1.2\n---\n# Generated by editors/build.mjs. Do not edit directly.\n' + stringify({
+  name: 'Knap expressions', scope: 'source.knap', version: 2, hidden: true,
+  contexts: {
+    main: [{ include: 'tags' }],
+    ...contexts,
+  },
+}, { lineWidth: 0 }));
+output('editors/sublime/Knap.sublime-syntax', '%YAML 1.2\n---\n# Generated by editors/build.mjs. Do not edit directly.\n' + stringify({
+  name: 'Knap Markdown', scope: 'text.html.markdown.knap', version: 2,
+  file_extensions: ['knap', 'knap.md'],
+  extends: 'Packages/Knap/Knap Markdown.sublime-syntax',
+}, { lineWidth: 0 }));
+
+// Snippet bodies are shared; each editor gets its own package format.
+const snippets = {
+  'Output a variable': { prefix: 'knap-var', body: '{{ ${1:title} }}', description: 'Output a value' },
+  'Apply a filter': { prefix: 'knap-filter', body: '{{ ${1:title} | ${2:upper} }}', description: 'Output a filtered value' },
+  'Conditional block': { prefix: 'knap-if', body: '{% if ${1:condition} %}\n$0\n{% endif %}', description: 'Conditional block' },
+  'Conditional with fallback': { prefix: 'knap-ifelse', body: '{% if ${1:condition} %}\n\t${2}\n{% else %}\n\t$0\n{% endif %}', description: 'Conditional block with fallback' },
+  'Loop': { prefix: 'knap-for', body: '{% for ${1:item} in ${2:items} %}\n- {{ ${1:item} }}$0\n{% endfor %}', description: 'Loop over a collection' },
+  'Set a variable': { prefix: 'knap-set', body: '{% set ${1:name} = ${2:value} %}$0', description: 'Assign a local variable' },
+  'Template comment': { prefix: 'knap-comment', body: '{# ${1:comment} #}$0', description: 'Comment removed from rendered output' },
+};
+json('editors/vscode/snippets/knap.json', snippets);
+for (const snippet of Object.values(snippets)) {
+  output(`editors/sublime/${snippet.prefix}.sublime-snippet`, `<snippet>\n  <content><![CDATA[${snippet.body}]]></content>\n  <tabTrigger>${snippet.prefix}</tabTrigger>\n  <scope>(text.html.markdown.knap | source.knap) - comment - string</scope>\n  <description>${snippet.description}</description>\n</snippet>\n`);
+}

editors/examples/article.knap

@@ -0,0 +1,42 @@
+---
+title: "{{ title | trim }}"
+source: "{{ url }}"
+{{ tags | yaml_property:"tags" }}
+---
+
+# {{ title | trim }}
+
+{# This comment spans multiple lines.
+   {{ ignored }} and {% if ignored %} are not evaluated here.
+#}
+
+{% if author %}
+By **{{ author.name }}**
+{% elseif site %}
+From [{{ site }}]({{ url }})
+{% else %}
+Unknown source
+{% endif %}
+
+{% set heading = title | upper %}
+{{ published | date:"YYYY-MM-DD" }}
+{{ First name | trim }}
+{{ summary ?? "No summary" }}
+
+{% for tag in tags %}
+- #{{ tag | kebab }}
+{% endfor %}
+
+{{ text | highlight:blue }}
+{{ people | sort:(details.rank, desc) }}
+{{ people | map:item => ({name: item.name}) | first }}
+
+Inline code: `{{ title }}`
+
+```text
+{{ content }}
+```
+
+<a href="{{ url }}">{{ title }}</a>
+
+{{- "Literal delimiters: {# comment #} and }}" -}}

editors/sublime/Comments.tmPreferences

@@ -0,0 +1,10 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0"><dict>
+  <key>name</key><string>Knap comments</string>
+  <key>scope</key><string>text.html.markdown.knap, source.knap</string>
+  <key>settings</key><dict><key>shellVariables</key><array>
+    <dict><key>name</key><string>TM_COMMENT_START</string><key>value</key><string>{# </string></dict>
+    <dict><key>name</key><string>TM_COMMENT_END</string><key>value</key><string> #}</string></dict>
+  </array></dict>
+</dict></plist>