-
Notifications
You must be signed in to change notification settings - Fork 1.3k
feat(llm): support configurable output language #425
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -71,6 +71,37 @@ | |
| STRUCTURED_RESPONSE_MAX_ATTEMPTS = STRUCTURED_RESPONSE_MAX_RETRIES + 1 | ||
| STRUCTURED_RESPONSE_RETRY_DELAYS_SECONDS = API_CONNECTION_RETRY_DELAYS_SECONDS | ||
| LLM_BATCH_MAX_ATTEMPTS = STRUCTURED_RESPONSE_MAX_ATTEMPTS + API_CONNECTION_MAX_RETRIES | ||
| OUTPUT_LANGUAGE_MAX_LENGTH = 64 | ||
|
|
||
|
|
||
| def resolve_output_language() -> str | None: | ||
| """Return the configured language for human-readable LLM finding text.""" | ||
| raw_language = os.environ.get("SKILLSPECTOR_OUTPUT_LANGUAGE", "") | ||
| language = raw_language.strip() | ||
| if not language: | ||
| return None | ||
| if ( | ||
| "\r" in raw_language | ||
| or "\n" in raw_language | ||
| or len(language) > OUTPUT_LANGUAGE_MAX_LENGTH | ||
| or not all(character.isalnum() or character in " -_" for character in language) | ||
| ): | ||
| return None | ||
| return language | ||
|
|
||
|
|
||
| def append_output_language_instruction(prompt: str) -> str: | ||
| """Append the optional output-language contract to an analyzer prompt.""" | ||
| language = resolve_output_language() | ||
| if language is None: | ||
| return prompt | ||
| return ( | ||
| f"{prompt}\n\n## Output language\n\n" | ||
| "Write human-readable finding text (including message, finding, explanation, " | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Small one: finding shouldn't be in this list. In this codebase Finding.finding is the "short matched snippet" (models.py:121) — actual text copied from the scanned file, i.e. evidence, not something to translate. No LLM schema returns a finding field today so nothing breaks, but if one ever does, this line would tell the model to translate evidence. Safer to just drop the word. |
||
| f"remediation, and intent fields when present) in {language}. Keep rule IDs, " | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This line asks the model to write intent in the target language. But the only place intent comes from an LLM is MetaAnalyzerFinding (meta_analyzer.py:101), where it must be exactly "malicious", "negligent", or "benign" — Pydantic rejects anything else. So if the model actually follows this instruction and writes, say, "悪意のある", validation fails, the call retries, and after max attempts the whole meta-analyzer batch fails. OpenAI's structured output would block this server-side, but claude_cli (JSON parsed from plain text) and Ollama via OPENAI_BASE_URL won't — and those are supported setups. Fix is one word: remove intent from this list (or move it to the "keep unchanged" list). Maybe mention impact there too, since it's also a fixed-choice field. |
||
| "severity values, categories, file paths, code, and other machine-readable values " | ||
| "unchanged." | ||
| ) | ||
|
|
||
|
|
||
| class _StructuredResponseValidationError(Exception): | ||
|
|
@@ -623,10 +654,12 @@ def build_prompt(self, batch: Batch, **kwargs: object) -> str: | |
| Override in subclasses that need a custom prompt layout. | ||
| """ | ||
| numbered = number_lines(batch.content, batch.start_line) | ||
| return BASE_ANALYSIS_PROMPT.format( | ||
| analyzer_prompt=self.base_prompt, | ||
| file_label=batch.file_label, | ||
| numbered_content=numbered, | ||
| return append_output_language_instruction( | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Suggestion, not a blocker: instead of wrapping the prompt in all 3 build_prompt methods, wrap it once where the prompt is actually used — the two prompt = self.build_prompt(batch, **kwargs) lines in the run loops (around lines 826 and 925 after this PR). That covers all current analyzers plus any future subclass automatically. Right now, anyone who writes a new build_prompt override has to remember to add this call, and nothing reminds them — TP4 shows overrides do happen. Only cost is adjusting the tests that call build_prompt directly. |
||
| BASE_ANALYSIS_PROMPT.format( | ||
| analyzer_prompt=self.base_prompt, | ||
| file_label=batch.file_label, | ||
| numbered_content=numbered, | ||
| ) | ||
| ) | ||
|
|
||
| def parse_response(self, response: object, batch: Batch) -> list[Finding]: | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -42,6 +42,7 @@ | |
| Batch, | ||
| LLMAnalyzerBase, | ||
| LLMRuntimeLimitError, | ||
| append_output_language_instruction, | ||
| estimate_tokens, | ||
| ) | ||
| from skillspector.model_info import get_max_input_tokens | ||
|
|
@@ -895,7 +896,7 @@ def __init__( | |
|
|
||
| def build_prompt(self, batch: Batch, **_kwargs: object) -> str: | ||
| """Use TP4's purpose-built prompt without the generic file wrapper.""" | ||
| return batch.content | ||
| return append_output_language_instruction(batch.content) | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Adding the language instruction here only translates what the LLM returns — but TP4's final message is built in Python from an English template (_tp4_finding, around line 1047): f"Description-behavior mismatch: declared purpose is '{declared}' but code also performs: {mismatched_text}.", and remediation a few lines below is a hardcoded English string the LLM never touches. So with SKILLSPECTOR_OUTPUT_LANGUAGE=Japanese the user gets an English sentence with Japanese pieces stuffed inside, plus an always-English remediation. Either translate those wrapper templates too, or note in the docs that TP4 messages stay in English. |
||
|
|
||
| def parse_response( # type: ignore[override] # TP4 returns its typed assessment. | ||
| self, response: object, _batch: Batch | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -46,6 +46,7 @@ | |
| BatchFailure, | ||
| LLMAnalyzerBase, | ||
| LLMRuntimeLimitError, | ||
| append_output_language_instruction, | ||
| estimate_tokens, | ||
| ) | ||
| from skillspector.llm_utils import run_async | ||
|
|
@@ -353,11 +354,13 @@ def _estimate_extra_overhead(self, findings: list[Finding]) -> int: | |
| def build_prompt(self, batch: Batch, **kwargs: object) -> str: | ||
| metadata_text = kwargs.get("metadata_text", "No metadata available") | ||
| findings_text = _format_findings_for_prompt(batch.findings) | ||
| return self.base_prompt.format( | ||
| metadata=metadata_text, | ||
| file_label=batch.file_label, | ||
| file_content=batch.content, | ||
| static_findings=findings_text, | ||
| return append_output_language_instruction( | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nit: The meta-analyzer prompt ends with "Analyze the findings now:", and this appends the language section after that. Models cope fine, but it reads oddly — a final instruction after the "go" line. Fine to leave (good to fix as well 🙃) ; just flagging. |
||
| self.base_prompt.format( | ||
| metadata=metadata_text, | ||
| file_label=batch.file_label, | ||
| file_content=batch.content, | ||
| static_findings=findings_text, | ||
| ) | ||
| ) | ||
|
|
||
| def parse_response( # type: ignore[override] # Base class permits custom parsed values. | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: this adds ~60 tokens to every prompt, but get_batches budgets file content using only estimate_tokens(self.base_prompt), so the extra text isn't counted. It's noise in practice (the base wrapper isn't fully counted either) — a short comment here would be enough.