feat: AST-aware chunk boundaries for semantic splitting

[clafollett]

Related Issue

Description

Updates the Rust chunking logic to prefer semantic (AST-like) chunk boundaries so that substantive, differently-named top-level definitions are split into separate chunks even when they still fit under the soft token budget.

• chunk_spans() now prefers splitting at function boundaries instead of greedily packing unrelated definitions together
• Split triggers when both accumulated and incoming spans are named definitions (both Some) with different names and accumulated content is substantive (≥25% of soft token limit)
• current_kind/current_name are updated when accumulating definition spans, ensuring boundary splits work correctly even when non-definition spans (e.g., imports/comments) precede a definition in the same chunk
• Name comparison requires both names to be Some — None vs Some(_) no longer forces an unintended split for unnamed definitions
• Threshold uses current_tokens.saturating_mul(4) >= soft instead of integer division to avoid rounding errors in the ≥25% check
• Small helpers (<25% threshold) still pack with the next function — they're likely related context
• Same-named spans (e.g., multiple impl MyStruct) stay packed

Type of Change

• 🐛 Bug fix (non-breaking change which fixes an issue) → Label: bug
• ✨ New feature (non-breaking change which adds functionality) → Label: feature
• 🛠️ Refactoring (code improvement without changing functionality) → Label: refactor
• 📚 Documentation update → Label: documentation
• 🎨 Style/UI update → Label: style
• ⚡ Performance improvement → Label: performance
• 🧪 Tests (adding missing tests or correcting existing tests) → Label: tests
• 💥 Breaking change (fix or feature that would cause existing functionality to change) → Label: breaking
• 🔒 Security fix → Label: security
• 🏗️ Infrastructure/Build/CI → Label: infrastructure

Visuals

N/A

Test Plan

• test_separate_named_functions_get_own_chunks — two substantive functions → 2 chunks
• test_small_helper_packed_with_next_function — tiny helper packs with large function
• test_non_definition_spans_still_pack — imports/comments pack as before
• test_same_name_definitions_pack — same-named impl blocks stay together
• Updated test_chunking_service_simple integration test for new 3-chunk behavior
• 49 total tests passing, clippy clean

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my code
• I have updated documentation as needed
• New and existing tests pass with my changes

Additional Notes

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Updates the Rust chunking logic to prefer semantic (AST-like) chunk boundaries so that substantive, differently-named top-level definitions are split into separate chunks even when they still fit under the soft token budget.

Changes:

• Added AST-aware split logic to ChunkingService::chunk_spans() to separate substantive top-level definitions with different names.
• Introduced unit tests in codetriever-parsing validating the new boundary behavior.
• Updated the codetriever-indexing integration test to reflect the new 3-chunk behavior for consecutive functions.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
crates/codetriever-parsing/src/chunking/service.rs	Adds AST-aware split heuristic and new unit tests for definition-boundary chunking behavior.
crates/codetriever-indexing/tests/chunking_tests.rs	Updates integration test expectations/comments for new per-function chunking behavior.

[clafollett]

@copilot re-review please — addressed all feedback: current_kind/current_name now updated when accumulating definition spans, name comparison requires both Some, threshold uses multiplication (tokens*4 >= soft) to avoid integer division rounding.

[Copilot]

@copilot re-review please — addressed all feedback: current_kind/current_name now updated when accumulating definition spans, name comparison requires both Some, threshold uses multiplication (token...

All three issues have been addressed in commit 9b25445:

1. current_kind/current_name staleness — updated in the accumulate branch whenever is_new_definition is true (lines 191–194), so boundary splits work even when non-definition spans precede a definition.
2. None vs Some name comparison — names_differ now uses a pattern match requiring both names to be Some before splitting; None vs Some(_) returns false (lines 134–137).
3. Integer division rounding — threshold now uses current_tokens.saturating_mul(4) >= self.budget.soft (line 142), eliminating any rounding errors.

All 49 tests pass.