HIP author skill

[gjenkins8]

First pass at a HIP authoring skill:

• Aim to refine, scope and investigate an the idea first, and promotes validating the idea in #helm-dev or developer call before creating the full HIP
• Then a section, by section drafting and refinement process

The HIP template is duplicated and embedded in the skill - enables providing much more detailed guidance for the sections than what is currently in HIP-0001

closes: #436

[TerryHowe]

/lgtm

[scottrigby]

a few thoughts:

1. H4HIP as outlined in https://github.com/helm/community/blob/main/hips/hip-0012.md is missing from this
2. rather than targeted word length for each section, is it better to have instructions that focus more on the spirit of the goal - such as "keep proposals as simple as possible. do not add unnecessary detail other than what's needed" etc?
3. meta question - what is this ultimately adding for an LLM above what a copy of https://github.com/helm/community/blob/main/hips/hip-0001.md would provide? If there are missing pieces from hip 001 (eg, H4HIP defined in 0012) should we instead amend hip 0001 with any new/missing/clarifying information?

[gjenkins8]

1. H4HIP as outlined in https://github.com/helm/community/blob/main/hips/hip-0012.md is missing from this

H4HIP were HIPs targeting Helm 4's development process. We are beyond that now - we now just have "HIPs" IMHO.

Also to be discussed: the HIP frontmatter actually already has a helm-version: field. Which I think we didn't realize when writing the Helm 4 HIP and proposing H4HIPs. So the H4HIP prefix was actually a little redundant.

Finally, H4HIP was simply a thing we prepended to the HIP's PR title. So we can keep doing that with e.g. H5HIP: , without the skill for writing the HIP actually know/caring I think.

---

rather than targeted word length for each section, is it better to have instructions that focus more on the spirit of the goal - such as "keep proposals as simple as possible. do not add unnecessary detail other than what's needed" etc?

Given LLMs tendency to write overly verbosely, "as simple as possible" are much more ambiguous. In fact, when I updated the skill to remove the word counts and add such language, the eval reported:

My interpretation

Removing the inline Target ~N words anchors backfired. A concrete number is a cheap, strong constraint the model actually applies as it drafts. Replacing it with "as simple as possible, the counts live in the template, treat them as a ceiling" is a softer signal — and it looks like the runs didn't re-derive the numbers from the template, so sections drifted longer. The prose principle "the range is a guide, not a hard limit" may even have licensed expansion.

That said, an update to the skill to use the word counts as guides (not targets, and not hard ceilings) helped a little in evaluations:

e.g. 746af84#diff-61b24769e609a18d3995dcd65eecd2cc83143d97c8b2faee5c4637b8e6f37069R74

---

meta question - what is this ultimately adding for an LLM above what a copy of https://github.com/helm/community/blob/main/hips/hip-0001.md would provide? If there are missing pieces from hip 001 (eg, H4HIP defined in 0012) should we instead amend hip 0001 with any new/missing/clarifying information?

The "template" is included inline with the skill so it can be enriched with more detailed instructions. So yes, these could (perhaps should) be fed back into the original template - any instructions should be applicable to both people and LLMs after all.

But I have purposefully inlined into the skill initially, in order to get the skill created without affecting existing process. It is hard enough as it is to get changes into this repo as is without proposing changes to existing HIPs!

[scottrigby]

1. Re H4HIPs:

   I mentioned H4HIPs becuase hip-0012 is the only other place than hip-0001 that defines HIP processes, and I wanted to be comprehensive with a skill for writing HIPs.

   Regarding the helm-version field, yes I agree it addresses this one aspect of the H4HIP spec:

   to allow categorization of changes specific to the Helm 4 process and changes

   But that's all. I don't think it fully satisfies the spirit of the H4HIP process proposed in hip-0001, which was to have a lightweight version of a standard HIP to minimize Helm 4 development friction.

   We didn't really say that H4HIPs were only useful leading up to the initial release of Helm 4 or if they're still useful throughout the lifecycle of developing Helm 4. I wasn't thinking this was debatable, but I can see we should clarify this before adding to a skill to help people write HIPs. So until we clarify that I agree we should leave it out of this PR. I have added this as a question to the next Helm dev meeting agenda.

2. re max words and ceiling language

   I can't find authoritative documentation about how to prompt for this successfully, and it may even differ per model.

   Only one place in the HIP spec gives an actual word count:

   Abstract - a short (~200 word) description of the technical issue being addressed

   is it necessary to set either a word count minimum or maximum for any of the other sections? I'm not sure taking an average of what was done in the past is a good indicator. I feel like this could be good to experiment with individually and share results, but it feels odd to do for a skill that we maintain at a project level.

3. re duplicating the template and instructions

   it sounds like you agree ideally we'd update the spec to make sense for both humans and LLMs rather than duplicating most of it with some extra flourishes just for LLMs.

   My feeling is we should probably follow up on reevaluating HIPs separately from this skill PR, and keep the skill relevant to what we ask for now. Then whenever we update the HIP process details we can update the skill.

   About this:

   It is hard enough as it is to get changes into this repo as is without proposing changes to existing HIPs!

   I do want to help get changes into this repo, and happy top put more effort into that again (I acknowledge I've been slacking off on that lately, and also waiting for specific people to reply who have limited time, and because of both of those things some PRs have lingered longer than they should. Will work to remedy that).

   I'm only pushing back on this because I'm concerned about maintainability. Is it helping you to write HIPs more clearly? Do we need this? What problem is it trying to solve, and is it increasing or decreasing our maintenance burden?

   re point 2 above, I think it would be better to focus our efforts on making the HIP process better where we see areas that are not useful or causing more friction than it's meant to help reduce.

[scottrigby]

apart from questions in the PR comments, a quick note about file location - does Claude support .agents directory? I had been following this more in previous months, and so far don't see this in the docs, but do see issues like this: anthropics/claude-code#31005

[gjenkins8]

Claude doesn't (directly) support .agents/ to my understanding, and I think we also want to remain vendor agnostic.

(personally, I have a startup hook for Claude that symlinks e.g. CLAUDE.md -> AGENTS.md)