`Unmarshal` decodes a `\u` surrogate pair to two U+FFFD instead of the character

[DRMacIver]

This bug was found by an automated program we have running that writes tests with hegel (but I have personally reviewed it). More details at the bottom, but the short version is that I am not personally affected by this bug, and if that's not welcome then please feel free to disregard it.

A quoted string whose character outside the BMP is written as a \u surrogate pair, the way JSON encoders write it, comes out of hjson.Unmarshal as two U+FFFD replacement characters; encoding/json gives the character, and a single \u escape for a BMP character is fine:

package main

import (
	"encoding/json"
	"fmt"

	"github.com/hjson/hjson-go/v4"
)

func main() {
	for _, text := range []string{`"\u00e9"`, `"\ud83d\ude00"`} {
		var fromJSON, fromHjson interface{}
		json.Unmarshal([]byte(text), &fromJSON)
		err := hjson.Unmarshal([]byte(text), &fromHjson)
		fmt.Printf("%-16s encoding/json: %+q   hjson: %+q  err=%v\n", text, fromJSON, fromHjson, err)
	}
}

Output:

"\u00e9"         encoding/json: "\u00e9"   hjson: "\u00e9"  err=<nil>
"\ud83d\ude00"   encoding/json: "\U0001f600"   hjson: "\ufffd\ufffd"  err=<nil>

"\ud83d\ude00" is the JSON spelling of U+1F600 and encoding/json decodes it to that one character; hjson.Unmarshal returns two U+FFFD with no error. The same happens with the string as an object value, into a Node, an OrderedMap or a string destination.

Tested on hjson-go v4.7.1 and on current master (033fae8, the same commit).

As mentioned at the top, this was found by an automated program that writes property-based tests for various open source projects using hegel, and the report was reviewed by hand before filing. We've also potentially found (but not yet hand validated) 18 other bugs in hjson. You can see the tests at https://github.com/hegeldev/hegel-zoo/tree/main/targets/go/hjson. Let us know if you would like us to file the other bugs found and/or contribute the tests. NB the tests are currently LLM generated and probably not yet suitable for inclusion as is, but we're happy to help get them into a better state if you want them.