fix: fall back to agent CLI when OpenAI conversion fails

[Chenxi1818]

Issue

convert_to_yaml() previously had only two paths: the OpenAI API, or a template-based conversion that drops paper citations, truncates the hypothesis to 500 chars, and keyword-guesses the domain. An OPENAI_API_KEY quota error therefore silently degraded every fetch.

What's done

Added a middle path: on any OpenAI failure (quota, auth, missing key, unparseable response), retry the same prompt through the local agent CLI for --provider, which authenticates with its own login rather than OPENAI_API_KEY. The template converter remains as the last resort.

• Refactored Prompt & OpenAI logic: Split prompt construction into _build_conversion_prompt() so both LLM paths share one prompt, and isolated OpenAI API execution into _convert_with_openai().
• Added CLI Fallback: Implemented _convert_with_cli(), mirroring the subprocess pattern in agents/manifest_trimmer.py.
• Robust Output Parsing: Added _extract_yaml() to strip CLI banners/markdown fences, and _parse_idea_yaml() to trim trailing agent chatter until the document successfully parses and contains an idea key.
• Provider Propagation: Threaded --provider through to the conversion step (previously used only for repo naming and --run).

Behavior changes

• When --provider is omitted, the conversion fallback now uses codex.
• A malformed LLM response no longer returns a partially-parsed object. _parse_idea_yaml() requires a dict containing idea, so bad output falls through to the next path instead of reaching submit_idea().

[Frankbest18]

Thanks for the great work! Here are some concern that hopefully can be addressed:

[P1] Converted YAML is accepted before it is validated as a complete NeuriCo idea

_parse_idea_yaml() considers any mapping containing an idea key successful. A syntactically valid but incomplete response—such as an idea containing only title—therefore prevents the CLI or template fallback from running.

NeuriCo has stronger validation in IdeaManager, but that validator is only reached when --submit is used, and it runs after the converted file has already been saved. Without --submit, an incomplete idea can be saved while the command reports success. This also covers unexpected YAML fields or provider commentary: the underlying issue is that conversion currently validates YAML syntax, not the resulting NeuriCo idea.

[P2] Model output can override authoritative provenance

source and source_url are described as auto-generated metadata, but _apply_source_metadata() uses setdefault(). If the model emits either field, its value is retained even though the converter already knows the actual source and fetched URL. This can silently save or submit incorrect provenance.

Trust-boundary note

I understand that IdeaHub is maintained internally, so the CLI-content concern depends on the project’s trust model. However, the current command accepts any URL beginning with http, and service ownership does not necessarily guarantee that every idea’s content is maintainer-authored. I would treat this as a documented trust-boundary concern rather than a confirmed blocker if the project can guarantee that all accepted content is intentionally trusted. Otherwise, defense-in-depth safeguards may be warranted to reduce the risk of prompt injection.

[Chenxi1818]

Thanks @Frankbest18 for the insightful review! I’ve addressed your feedback in the latest commit:

[P1] Validation: Updated the fallback logic to validate the full NeuriCo idea schema (instead of just checking for the idea key). If the schema validation fails, it now properly falls through to the CLI / Template fallback paths.

[P2] Metadata Override: Changed setdefault() to enforce/overwrite source and source_url with authoritative metadata, preventing any model-emitted provenance from overriding it.

[Trust-boundary]: Appreciate the security note! I created a new issue #155 to mention this.

Please let me know if everything looks good now!

[Frankbest18]

Thanks for the update. There is only one remaining validation gap:

[P2] The shared validator does not enforce the full NeuriCo schema

validate_idea_spec() centralizes the existing manual validation, but it still accepts values that violate ideas/schema.yaml. For example, constraints: none is reported as valid even though constraints must be an object. That value is later treated as a mapping by the prompt generator, which can cause a runtime failure.

The validator similarly accepts invalid types for background and metadata, as well as strings shorter than the schema minimums. Consequently, the required-field fallback issue is fixed, but structurally invalid optional fields can still be saved and reported as a successful conversion.

[Frankbest18]

Thanks for the update. The previous validation concern is improved but not fully resolved.

[P2] Schema-invalid null metadata passes validation and fails during finalization

The validator treats an explicitly present null structured field as absent. For example:

idea:
  title: A sufficiently long title
  domain: machine_learning
  hypothesis: This is a sufficiently long hypothesis.
  metadata:

This is reported as valid even though metadata, when present, must be an object according to ideas/schema.yaml. The converter then assumes that metadata is a mapping while applying authoritative provenance and raises:

AttributeError: 'NoneType' object has no attribute 'get'

This means a structurally invalid model response passes the stated validation boundary and only fails later during conversion finalization. The validator also continues to accept some schema-invalid nested values, such as non-string paper URLs and descriptions.