Explore Domain Specific Language (DSL) as intermediary step for legal and causal reasoning

[trinav0711]

Check research specification languages to extract privacy rules from AI acts in the legal documents. One such example would be F star, developed by Microsoft Research. Relevant works regarding this are:

1. https://sebastianzimmeck.de/zimmeckAndBellovin2014Privee.pdf -> Section 2.1
2. https://www.cs.cmu.edu/~breaux/publications/tdbreaux-re13.pdf
3. https://cs.brown.edu/media/filer_public/d1/de/d1deb154-8d65-48b5-93ef-385c73555a09/zechcarolyn.pdf

Check these out with a focus on three scenarios in the work, namely Healthcare, Employment and Finance. Think about how we intend to use it since this is going to be a long-term plan.

This takes cue from existing issue xai-privacy/compliance-audit-benchmark#5 about exploring existing privacy laws in the US and across the globe.

[trinav0711]

All research papers and associated working groups have been covered in the sub-issue xai-privacy/research-planing#4.

[trinav0711]

The recommendation was to check F star (https://fstar-lang.org/tutorial/). Even though the language, developed by MS Research, is widely used as a type-safe language for mathematical proofs and implementation of TLS 1.3, it remains a functional language designed to implemented mathematically correct proofs. According to my understanding, it is not a good choice for our use case of translating natural language privacy policies to application compliance standards.

Open Digital Rights Language (ODRL) seems to be a better choice for declaring and translating privacy policies. It supports a variety of formats as well. A higher and more purpose-specific language choice could be the deontic LegalRuleML (https://github.com/oasis-tcs/legalruleml), an extension of RuleML. Even though its more complex than ODRL, we can express the legal terms in a better way, as per my understanding.

We have the candidates as ODRL, LegalRuleML and Paralegal, for now. Amongst the three, ODRL is the most intermediate choice which can generate the policies in JSON, easily parse-able by LLMs, and easy to check compliance. Even though LegalRuleML is tighter and more expressive, it will be difficult to implement and knit with LLMs. Paralegal needs more research at this point.

[SebastianZimmeck]

I am reopening this issue because, according to our discussion, a Domain Specific Language (DSL) for mapping natural language prompts into a more formalized legal language could be a good intermediary step to improve the legal (and causal) reasoning of AI models.

@trinav0711 could you create a list of legal DSLs that may be suitable for our purposes?

A few additional questions:

1. Is there already a de-facto standard for legal DSLs that is generally considered the default approach?
2. How would we practically make use of such DSL? For example, would we just give the DSL specification to the model in a system prompt, would we train it on a corpus of scenarios in which the DSL has been applied, ...?
3. More generally, is using a DSL for legal (or logical reasoning or a similar task) the actual approach in deployed AI models (e.g., ChatGPT)?

As @pooyanjamshidi said in our chat:

The approach of combining solvers and other tools is typically used to build modern systems such as ChatGPT, and it has mostly been used in the code domain. So we must distinguish and control for such distinctions in our evaluations. I mean, Plain LLM (Whether the model can reason directly from text), LLM + chain-of-thought/explanation (Whether prompting alone improves consistency), LLM → DSL → solver (Whether formalization fixes inference errors), LLM → DSL → solver → explanation (Whether the system can give a legally meaningful explanation), LLM self-check against solver (Whether the model can detect and revise its own wrong answer).

Here, at least in such scenarios, the key challenge is not the solver, but a reliable translation to the underlying representation. In our project, the challenge is converting messy legal text into a faithful formal representation: variables, predicates, etc.