I am opening this issue for putting together our full analysis framework. Certainly, some of it will change as some parts may not work as envisioned.
@e-yen07, @mkarim-usc, @sungjunleeee, and @trinav0711, I am relying on your expertise here in terms of what you have explored so far to improve what I am describing here, come up with alternative ideas, etc. So please chime in, and feel free to let me know if anything is unclear.
1. Python Script as Orchestrator
A Python script (or other script) running on our local computers or servers acts as the orchestrator of the different parts of the framework. That is similar to our existing approach.
2. Small Language Model (SLM) and Benchmark
The script calls the SLM when it needs the benchmark instances parsed. The script constructs a prompt containing only the raw question text (i.e., a LEET-Arg question) and passes it to the SLM engine (e.g., vLLM, Ollama, or Outlines running locally or as a local API endpoint).
3. Format + Solver
That is where the formatting comes in. We need it for two tasks:
- to make sure that the extracted question text is always in the correct format so that the solver can reason about it, and
- to enable the solver to, in fact, reason.
In the plain model example from the paper, we just gave a prompt to the model to "provide explanations for your solution," but now we actually want to force the model into reasoning. Thus. we need to make sure that all text is in a reliable format that can be reasoned about.
3.1 Making Sure Extracted Text Is in Correct Format
Just telling the SLM in the prompt "use format xyz" may not reliably work 100% of the time. So, we try constrained decoding, maybe in combination with domain specific languages, to ensure that the format is 100% of the time correct. Maybe, there are other techniques as well, but Constrained Decoding seems good.
3.2 Enable the Solver to Reason
Assuming that we now have all extracted text from the question in the format we want, we need to go from syntax to semantics. How does the solver (which could be a deterministic solver as part of our Python script or could be part of the SLM as well; we will have to see) understand the semantics of the text to reason about? The basic idea is to use abstract argumentation schemas, such as PyArg.
For example, PyArg implements Dung’s Abstract Argumentation Framework (AAF) (Arguments and Attack edges). Above our SLM parses text into formatted instances representing arguments and conflict relations. Here we feed those instances into PyArg to compute their semantics.
af = ArgumentationFramework()
for arg in graph_data.arguments:
af.add_argument(Argument(arg))
...
In other words, our Python script now processes the formatted text excerpts for reasoning using PyArg. Maybe, it uses the SLM in that process; maybe not. It may also be necessary to pass in already some formatting information necessary for PyArg in the formatting step.
There are other abstract argumentation schemas beyond PyArg, but that is a staring point.
4. Variations
So, this is the model + solver framework, which is our core point. It should return better results (both in terms of final results as well as in terms of reasoning steps) than just the plain model from the LEET-Arg paper.
For our model + solver We can try zero-shot, one-shot, and few-shot prompting techniques. In other words, zero-shot is a "cold start" while in one-shot we give our SLM one example reasoning.
Model + chain-of-thought reasoning would also be a good variation to try.
I am writing out more details in the roadmap. But these are the core implementation points.
@e-yen07, @mkarim-usc, @sungjunleeee, and @trinav0711, please start working on this implementation. Once you have a sense of how it is going, we can allocate more specific work to everyone.
(cc'ing @pooyanjamshidi)
I am opening this issue for putting together our full analysis framework. Certainly, some of it will change as some parts may not work as envisioned.
@e-yen07, @mkarim-usc, @sungjunleeee, and @trinav0711, I am relying on your expertise here in terms of what you have explored so far to improve what I am describing here, come up with alternative ideas, etc. So please chime in, and feel free to let me know if anything is unclear.
1. Python Script as Orchestrator
A Python script (or other script) running on our local computers or servers acts as the orchestrator of the different parts of the framework. That is similar to our existing approach.
2. Small Language Model (SLM) and Benchmark
The script calls the SLM when it needs the benchmark instances parsed. The script constructs a prompt containing only the raw question text (i.e., a LEET-Arg question) and passes it to the SLM engine (e.g., vLLM, Ollama, or Outlines running locally or as a local API endpoint).
3. Format + Solver
That is where the formatting comes in. We need it for two tasks:
In the plain model example from the paper, we just gave a prompt to the model to "provide explanations for your solution," but now we actually want to force the model into reasoning. Thus. we need to make sure that all text is in a reliable format that can be reasoned about.
3.1 Making Sure Extracted Text Is in Correct Format
Just telling the SLM in the prompt "use format xyz" may not reliably work 100% of the time. So, we try constrained decoding, maybe in combination with domain specific languages, to ensure that the format is 100% of the time correct. Maybe, there are other techniques as well, but Constrained Decoding seems good.
3.2 Enable the Solver to Reason
Assuming that we now have all extracted text from the question in the format we want, we need to go from syntax to semantics. How does the solver (which could be a deterministic solver as part of our Python script or could be part of the SLM as well; we will have to see) understand the semantics of the text to reason about? The basic idea is to use abstract argumentation schemas, such as PyArg.
For example, PyArg implements Dung’s Abstract Argumentation Framework (AAF) (Arguments and Attack edges). Above our SLM parses text into formatted instances representing arguments and conflict relations. Here we feed those instances into PyArg to compute their semantics.
In other words, our Python script now processes the formatted text excerpts for reasoning using PyArg. Maybe, it uses the SLM in that process; maybe not. It may also be necessary to pass in already some formatting information necessary for PyArg in the formatting step.
There are other abstract argumentation schemas beyond PyArg, but that is a staring point.
4. Variations
So, this is the model + solver framework, which is our core point. It should return better results (both in terms of final results as well as in terms of reasoning steps) than just the plain model from the LEET-Arg paper.
For our model + solver We can try zero-shot, one-shot, and few-shot prompting techniques. In other words, zero-shot is a "cold start" while in one-shot we give our SLM one example reasoning.
Model + chain-of-thought reasoning would also be a good variation to try.
I am writing out more details in the roadmap. But these are the core implementation points.
@e-yen07, @mkarim-usc, @sungjunleeee, and @trinav0711, please start working on this implementation. Once you have a sense of how it is going, we can allocate more specific work to everyone.
(cc'ing @pooyanjamshidi)