Quote strings that begin or end with non-ASCII whitespace

[youdie006]

The inconsistency

The decoder strips quoteless strings with strings.TrimSpace, which is
Unicode-aware (decode.go:511, under the comment "remove any whitespace at the
end (ignored in quoteless strings)" at :510).

The encoder decides whether to quote with needsQuotes (encode.go:82), which
anchors on ^\s and \s$. Go's RE2 \s is ASCII only — [\t\n\f\r ].

So a value that starts or ends with U+00A0, U+1680, U+2000–U+200A, U+205F or
U+3000 is written quoteless by quote() (encode.go:136) and then trimmed away
when it is read back.

needsQuotes already enumerates non-ASCII code points by hand for exactly this
reason — \x{2028}-\x{202f}, \x{feff} and friends are in there precisely
because \s cannot reach them. The list just stops short of the spaces
TrimSpace removes.

What a caller sees

b, _ := hjson.Marshal(map[string]interface{}{"k": " ab"})
// {
//   k:  ab          <- quoteless
// }

var dst map[string]interface{}
hjson.Unmarshal(b, &dst)
// dst["k"] == "ab"       <- the leading U+00A0 is gone

Silent data loss through the library's own round trip. Measured on master, for
{rune}+"ab" and "ab"+{rune}:

rune	round-trips
U+0020, U+0085, U+2028, U+202F	yes (already quoted)
U+00A0, U+1680, U+2000, U+200A, U+205F, U+3000	no — comes back "ab"
U+200B, U+180E	yes (not unicode.IsSpace, correctly left quoteless)

hjson-js quotes every one of the failing cases, so the two implementations
disagree here.

The change

One condition, using the same function the decoder uses, so the two cannot drift
apart again.

quoteName (encode.go:189) is a separate helper for keys and is unaffected —
the key reader does not TrimSpace, and I checked that  k and 　k
as keys already round-trip correctly on master.

The alternative

Widening the ^\s/\s$ anchors in needsQuotes to an explicit class
([\t\n\v\f\r \x{0085}\x{00a0}\x{1680}\x{2000}-\x{200a}\x{2028}\x{2029}\x{202f}\x{205f}\x{3000}])
also passes every test and mutation row here. I went with TrimSpace because it
is one line and it is literally the function the parser calls, so the encoder and
decoder cannot disagree in future. Both were benchmarked and neither is
distinguishable from master (prebuilt binaries, rotating order, 30 reps,
Marshal of assets/pass1_test.json): master min 115581 ns/op, regex-widen
114906, TrimSpace 112578 — all inside this box's run-to-run drift. Happy to
swap to the regex form if you prefer keeping the decision in one regexp.

I did not touch the decoder: making it trim ASCII-only would change the meaning
of documents already on disk, and hjson-js settles the direction by quoting on
output.

Tests

TestQuoteUnicodeWhitespace asserts both the quoting decision and the round
trip, for each rune in both leading and trailing position.

The U+200B and U+180E rows are the ones that matter for correctness of the
fix: those are not unicode.IsSpace, TrimSpace does not strip them, and
they must stay quoteless. Over-correcting to a cutset that also trims U+200B
fails that row; dropping U+3000 from the class fails the U+3000 row; reverting
fails the six uncovered runes.

Verification

go test ./... green, including all 155 asset conformance cases. gofmt -l .
clean. The fix uses only strings.TrimSpace and len, and the test only
bytes.Contains and a struct-literal table, so the go 1.12 CI row is fine. I
also ran the four hjson-cli | diff steps from test.yml locally after the fix
— all four match.

Separately, I re-ran a full structural differential of hjson-go against
hjson-js@5734a70 over all 180 shared asset documents after the fix: 0
divergences.

Note

Open PR #64 also touches quote() (it reflows the signature). It has been idle
since 2025-06-20, and my insertion sits directly under a line it keeps, so the
conflict should be trivial — but flagging it in case you want them ordered.

Precedent for the bug class: your #42, "Marshal \r to quoted strings" — a
character the encoder emitted unquoted that did not survive Unmarshal.

Disclosure

I used an AI assistant to help find and prepare this change. I reviewed and
tested it myself, and the numbers above are from runs I performed.