Make bracketed span conditional on valid attributes

[dereuromark]

Follow-up to the discussion in #137: per the docs, "Text in square brackets that is not a link or image and is followed immediately by an attribute is treated as a generic span." The parser committed the span matches as soon as it saw [...]{, so when the attribute parse subsequently failed (or ran into end of input), the braces were reparsed as literal text but the span itself remained:

[x]{#a<b}

rendered as

<p><span>x</span>{#a&lt;b}</p>

With this change the span is tracked as speculative and its open/close matches are reverted to literal brackets when the attribute parse fails, so the whole construct stays literal:

<p>[x]{#a&lt;b}</p>

Inline content between the brackets is still parsed normally ([*x*]{#a<b} gives [<strong>x</strong>]{#a&lt;b}), and the end-of-input case ([x]{#a) is covered via the same reparse path. Valid attributes are unaffected.

Tests: three new cases in test/spans.test (invalid attribute, invalid attribute with inline formatting, unterminated attribute at end of input). Full suite passes.

Also resolves the discussed bug in jgm/djot#399 here.

[jgm]

On reflexion, there may have been a reason I didn't originally do this, connected to performance.
It means that determining whether we are closing a span can depend, theoretically, an an unlimited amount of lookahead parsing. I suspect there are some evil pathological cases herein. For this reason we might consider relaxing the rule to say that a span must be bracketed text followed by {.

[dereuromark]

I think there are two separate questions here: what the rule costs as a spec rule, and what it costs in djot.js. They come out differently.

On the spec level you are right: "span only when followed by a valid attribute" means a ] cannot be classified until the whole attribute parse resolves, and attributes can be arbitrarily long (multi-line, quoted strings, comments). A strictly streaming implementation would have to buffer the span decision for that whole stretch. That is a real cost of the rule as written.

For djot.js specifically, though, this PR does not add any lookahead. The full attribute parse plus backtracking already happens today for every element followed by { - that is the existing attributeParser / attributeSlices / reparseAttributes() machinery. The PR only records the tentative span and, on failure, flips the two bracket matches back to str inside the already-existing reparseAttributes() path. That is O(1) extra work per failed attribute; no new scanning.

The pathological cases are also already guarded against, independent of this PR: reparseAttributes() re-feeds the failed slices with allowAttributes = false, so a failed attribute region cannot spawn nested attribute attempts inside itself. Every character is parsed at most twice, which keeps the whole thing linear. I benchmarked adversarial inputs (repeated [x]{a=", i.e. a failing attribute with an open quoted string after every span candidate) at 14 KB / 28 KB / 56 KB:

input	main	this PR
[x]{a=" x 2000	17 ms	19 ms
[x]{a=" x 4000	21 ms	20 ms
[x]{a=" x 8000	30 ms	35 ms

Growth is linear on both branches and the difference is within noise. So whatever performance reason there originally was, I do not think it applies to the current code base: the expensive part (attempt + reparse) has been there all along.

That leaves the question as a purely semantic one, and there I see the trade like this:

• Relaxing to "bracketed text followed by {" buys a clean invariant: an attribute parse failure is local to the braces and never reclassifies the element before it, exactly like word{a= today. The close decision at ] becomes one character of lookahead.
• The cost is spans that carry no information: a bare <span> without attributes has no meaning, unlike a word, and prose like [optional]{...} or bracketed text in front of a stray { would silently become a span with literal braces after it.

Worth noting the relaxation would also reverse the guidance from #137 and jgm/djot#399 (where the fully-literal rendering was called the intended one), and djot-php and the cdjot-derived tests would then need to change to match djot.js instead of the other way around.

I am fine with either outcome - just giving it some more context.

Personally, I think the stricter approach makes more sense and is more clear to the writer.
I do not think that simplification is worth the semantic regression.

[jgm]

OK, good, thanks for the additional analysis.