ConvokitAI PR

[ClaireOka]

Description

This PR adds ConvoKit AI, a set of packages for representing, simulating, and evaluating conversations that involve AI agents, together with three new datasets in the ConvoKit AI format. Type of change: new feature, new datasets, and documentation.

NOTE:

Before approving this PR change DEFAULT_IMAGE in convokitai-simulation/dl-container.py to be 'ghcr.io/cornellnlp/convoarena-backend:v0.3' and make the convoarena package on CornellNLP to be public.

1. convokitai: the ConvoKit AI data model (convokit/convokitai/)

• Extends convokit.model with AI-specific fields while staying a drop-in replacement: convokitai.Corpus works anywhere convokit.Corpus does, including with existing transformers.
• AI speakers: Speaker.is_ai, and Speaker.ai_meta with a generation config and a role (e.g. "participant", "public assistant" for a mediator).
• Private assistants and supports: PrivateAssistant and Support represent AI assistants that privately help speakers outside the main conversation, and the messages they send.
• Generation: Speaker.generate() and PrivateAssistant.generate() produce new messages through convokit.genai.
• Transcripts: Conversation.get_transcript(supports=True) interleaves supports with the utterances they reply to.
• Backward compatible on disk: AI fields are stored inside regular ConvoKit metadata, so every ConvoKit AI corpus also loads with convokit.Corpus.
• Packaging: bundled with ConvoKit through setup.py (package_dir) and imported as a top-level package: pip install convokit, then import convokitai.

2. convokitai_evaluation: transformers for AI-mediated conversations (convokit/convokitai_evaluation/)

• CIG: conversational information gain, an LLM rating (1–4) of how much each utterance adds to what the group already knew.
• ClaimTracker: tracks each speaker's atomic claims through the conversation, then measures how far participants converge and whether their final claims came from the mediator or another participant.
• MediatorRedirection: applies the existing convokit.redirection to multi-party conversations with a mediator.
• Includes a small sample corpus and three demo notebooks.

3. convokitai-simulation: AI agent simulations (convokit/convokitai-simulation/)

• Runs agent conversations described by a simulation YAML on ConvoArena, a platform built on Google PAIR's Deliberate Lab, and returns them as a convokitai Corpus.
• Backends: runs locally by default, on the backend's Firebase emulators in a Docker container (falling back to udocker on Colab). It can also run on a user's own Firebase deployment. Currently takes docker image off of ClaireOka.
• Vendored client: includes the Deliberate Lab Python client (deliberate_lab/), unmodified.
• Includes a demo notebook.

4. New datasets (in the ConvoKit AI format, with download entries in download_config.json and demo notebooks under datasets/)

• habermas-machine-corpus
• mathdial-corpus
• llm-facilitation-corpus

5. Documentation and versioning

• New Sphinx pages, convokitai.rst, convokitAISimulation.rst and convokitAIEvaluation.rst, in a new "ConvoKit AI" section of the docs index.
• All new modules have Sphinx-style docstrings.
• docs/source/conf.py adds the package folders to sys.path so autodoc can import them.
• Version bumped to 4.2.0 in setup.py and docs/source/conf.py. convokitai_evaluation and convokitai-simulation declare convokit>=4.2, the first version that bundles convokitai.

Motivation and Context

Conversations increasingly include AI participants: LLM agents, AI mediators that speak in a conversation, and private assistants that coach individual speakers. ConvoKit's data model has no way to mark which speakers are AI, record how their messages were generated, or represent assistant messages that only some speakers see. ConvoKit AI adds this while keeping full compatibility with existing corpora and transformers.

The simulation and evaluation packages build on the format: one to generate AI-mediated conversations at scale, the other to measure how mediators affect them. The three datasets give researchers ready-to-use AI and human–AI conversation data in the new format.

Other information

• For the maintainers at release time:
  • Publish 4.2.0 to PyPI; pip install convokit only includes convokitai after that.
  • Update the "latest version is 4.1.2" lines in README.md, docs/source/index.rst and website/docs/source/index.rst, and release in website/docs/source/conf.py.
  • Until the release, the packages install from GitHub: pip install "git+https://github.com/CornellNLP/ConvoKit.git@master".
• Separately installed packages: convokitai_evaluation and convokitai-simulation are not included in the convokit install. They install from their subfolders, e.g. pip install "git+https://github.com/CornellNLP/ConvoKit.git@master#subdirectory=convokit/convokitai_evaluation".
• Backend image: the local simulation backend pulls ghcr.io/claireoka/deliberate-lab-backend:v0.2. You may want to host it under the CornellNLP organization.
• Vendored code: convokitai-simulation/deliberate_lab/ is copied from Deliberate Lab. Its license notice should be kept with it.
• Demo notebook links in the docs point to CornellNLP/ConvoKit on master, so they work once this is merged.

[ethanxia4]

Most of the PR looks good to me. The main convokit website will need to be modified, which can be a separate PR. The new datasets should be straightforward and I assume we will add convokitai in some form to the tools page as well. I'm happy to do that in this PR or a separate.

[ethanxia4]

Do we have a way of making the python version requirement clear to users, given that the requirement is different from standard convokit? I didn't see anything in the documentation, but I could have missed it.

[ethanxia4]

Same thing with convokitai-simulation, do we have a way of making the python version requirement clear to users, given that the requirement is different from standard convokit?

[ClaireOka]

I don't think the requirement is clear at the moment. I can add clearer instructions to the documentation later today.

Edit: updated documentation