Repository navigation
Conversation
Individual-level belief reasoning (Li et al., EMNLP 2026, arXiv:2510.15144): 54 interview participants, 1,742 items, three domains. One agent routed by data_source: hugagent_bsi (two-option belief inference, exact match) and hugagent_bdu (post-scenario rating, tolerance accuracy as in the paper). Val parquets in verl format are published with the dataset at social-atoms/hugagent under odyssim/.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds HugAgent (Li et al., EMNLP 2026, arXiv:2510.15144) to the eval suite: individual-level belief reasoning, 54 interview participants, 1,742 items, three domains (healthcare, surveillance, zoning).
Two single-turn tasks, same agent, routed by
data_source:hugagent_bsi(356 items): given a participant's demographics and their own interview answers, pick which of two options states a belief they hold. Reward: exact match. Three items accept either letter.hugagent_bdu(1,386 items): predict the participant's own post-scenario rating (1-10 stance or 1-5 reason weight). Reward: tolerance accuracy as in the paper (within 1 on a 5-point scale, within 2 on a 10-point scale);1 - |err|/spanis logged ashugagent_bdu/norm_acc.Files:
agents/hugagent/agent.py: prompt builder (paper prompts, Appendix K.1),<answer>tag extraction, reward.agents/base_agent.py: routeshugagent,hugagent_bsi,hugagent_bdu.recipe/ditto/eval.shand READMEs: the two val parquets added to the suite.Data: the verl-format parquets live in the dataset repo https://huggingface.co/datasets/social-atoms/hugagent under
odyssim/(hugagent_bsi_val.parquet,hugagent_bdu_val.parquet). The dataset is gated with automatic approval (CC BY-NC 4.0, human participant data), so it is not insunweiwei/sim-eval-data; happy to mirror the two files there if you prefer.Tested offline: prompt construction on all 1,742 rows, extraction and reward on hand-written cases (A/B gold, tolerance bands, unparsed -> 0), and perfect predictions score 1.0 on every row. I do not have the verl stack set up, so the
agent_loopitself has not been run end to end here. Human test-retest ceiling is 84.8% (BSI) and 85.7% (BDU); GPT-4o scores 78.8 / 62.2 with our standalone scorer (https://github.com/jajamoa/HugAgent).