fix(core): drop matches that overlap any earlier kept match in tokenize

[RaphaelFakhri]

Summary

tokenize removes overlapping matches by comparing each match with the previous entry in the sorted list, not with the last match it kept. When a long match contains two or more shorter matches, the first nested match is dropped but the second one starts after it ends, so it passes the check. The output then contains the nested text a second time.

For the grammar { outer: /abcdefghij/g, first: /cd/g, second: /gh/g } and the input xx abcdefghij yy, the tokens join to xx abcdefghijghij yy.

This change tracks the end of the last kept match and drops any match that starts before it.

Testing

• Adds tokenizer.test.ts with a basic split case and two overlap cases.
• The two overlap cases fail on main and pass with this change.
• The default email and URL grammar does not trigger the bug in the cases tried, so the tests use a custom grammar.

Changeset

Adds a patch changeset for @livekit/components-core.

[changeset-bot]

🦋 Changeset detected

Latest commit: 0f0fe49

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 7 packages

Name	Type
@livekit/components-core	Patch
@livekit/components-react	Patch
@livekit/agents-ui	Patch
@livekit/component-example-next	Patch
@livekit/components-js-docs	Patch
@livekit/component-docs-storybook	Patch
@livekit/components-docs-gen	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

@RaphaelFakhri is attempting to deploy a commit to the LiveKit Team on Vercel.

A member of the Team first needs to authorize it.

[CLAassistant]

All committers have signed the CLA.

[1egoman]

@lukasIO Can you take a look at this one? I'm not fully sure where tokenize is used in the stack (it looks like by formatChatMessageLinks?) and if this is a fix which is worth reopening and partially / fully merging?

[lukasIO]

yeah, it's an edge case but worth fixing. Changes look good

[RaphaelFakhri]

Thanks for taking a look. This PR can't be reopened because its fork was deleted, so the same change is now in #1467, rebased onto current main.