Performance: Context builder crashes on massive minified JS files

[pradeep0153]

Closes #239.

Description

This PR resolves a catastrophic performance bottleneck within the global Repository Ingestion architecture that was actively crashing the agent. The context_builder.py module enforces a 512KB cap on file size, but it was highly vulnerable to memory/CPU spikes when processing minified files.

If a repository contained a 400KB bundle.min.js file consisting of a single, continuous line of text, the keyword scoring algorithm (context_builder.py) attempted to tokenize and score the file. Because the regex/tokenizer wasn't optimized for single-line files of this magnitude, it violently choked the main thread. The agent took over 45 seconds just to score the file before even contacting the LLM, completely destroying the developer experience and artificially inflating execution times.

Changes Made

• Algorithm Audit: Profiled the tokenization and regex matching functions within context_builder.py.
• The Core Vulnerability: Discovered a catastrophic failure to identify and ignore generated, minified, or non-human-readable source code during the scoring phase.
• Minification Pruning Pipeline: Refactored the ingestion architecture to implement a strict Minification Pruning pipeline.
• Execution: repo_ingestion.py now intelligently detects minified or generated files (e.g., checking if the average line length exceeds 5,000 characters or checking for common .min.js extensions). It automatically flags these files as is_generated=True and mathematically excludes them from the LLM context payload entirely, regardless of the 512KB size cap.

Impact

• Agent Performance & Robustness: Forcing the CPU to tokenize half a megabyte of single-line, minified JavaScript is a fatal algorithmic failure that guarantees thread starvation. By engineering this minification pruning pipeline, we mathematically eradicate the tokenizer bottleneck. We drastically accelerate the context compilation phase and protect the agent from hanging on modern frontend repositories, as the LLM does not need to read Webpack output to fix source code.

[vercel]

@pradeep0153 is attempting to deploy a commit to the sreerevanth's projects Team on Vercel.

A member of the Team first needs to authorize it.