@trinav0711, can you explore the use of Constrained Decoding (also known as Structured Outputs)? Very concretely, if we want to use Constrained Decoding with one of the models @e-yen07 identified, what is the practical setup? Can you set it up on your own computer and get it to run (just in general; no worries at this point about our exact use case)?
Here is a general setup article. Constrained Decoding allows us to extract text segments from a free-form text and put them into predicate format, e.g., Has_Symptom(x). The reason for using Constrained Decoding for doing so is that, as it is my understanding, it guarantees 100% syntactically correct output. If we just tell a model to adhere to a particular format via prompting, the model may occasionally produce a different format.
There are different techniques and libraries for Constrained Decoding. It would be good if you can explore the mainstream way(s) that are well-supported and state of the art and recommend one for us.
As a reference, Google Gemini recommends defining predicate structures as a Pydantic model in Python. Then pass this schema to the local inference engine (such as vLLM, SGLang, or Outlines), which are agnostic to the formalization of the law. Though, I read somewhere that Pydantic is not 100% reliable to generate correct syntax.
Once we have ensured the predicate output, we can then apply a DSL to reason over the predicates. That is the connection to your DSL exploration.
Also, in addition, @trinav0711, can you check (and, ideally, try out) using Constrained Decoding in frontier models via their API? The reason is that we may want to compare performance between local and frontier model. So, the question is also, how to run Constrained Decoding in the latter. Here is an explanation from Google Gemini (using Pydantic for Constrained Decoding):
All major frontier LLMs natively support Structured Outputs / JSON Schema Enforcement directly in their API calls. We pass the Pydantic schema as a parameter in the API call, and their cloud backend enforces token-level state machine masking before returning the response.
@trinav0711, can you explore the use of Constrained Decoding (also known as Structured Outputs)? Very concretely, if we want to use Constrained Decoding with one of the models @e-yen07 identified, what is the practical setup? Can you set it up on your own computer and get it to run (just in general; no worries at this point about our exact use case)?
Here is a general setup article. Constrained Decoding allows us to extract text segments from a free-form text and put them into predicate format, e.g.,
Has_Symptom(x). The reason for using Constrained Decoding for doing so is that, as it is my understanding, it guarantees 100% syntactically correct output. If we just tell a model to adhere to a particular format via prompting, the model may occasionally produce a different format.There are different techniques and libraries for Constrained Decoding. It would be good if you can explore the mainstream way(s) that are well-supported and state of the art and recommend one for us.
As a reference, Google Gemini recommends defining predicate structures as a Pydantic model in Python. Then pass this schema to the local inference engine (such as vLLM, SGLang, or Outlines), which are agnostic to the formalization of the law. Though, I read somewhere that Pydantic is not 100% reliable to generate correct syntax.
Once we have ensured the predicate output, we can then apply a DSL to reason over the predicates. That is the connection to your DSL exploration.
Also, in addition, @trinav0711, can you check (and, ideally, try out) using Constrained Decoding in frontier models via their API? The reason is that we may want to compare performance between local and frontier model. So, the question is also, how to run Constrained Decoding in the latter. Here is an explanation from Google Gemini (using Pydantic for Constrained Decoding):