Set up models etc. according to the paper "When correct is not enough"

[SebastianZimmeck]

We can use the paper When_correct_is_not_enough.pdf as a blueprint and extend it for our research. The first order of business is to get the setup the authors describe working with a local model. Can you do that @e-yen07, @sungjunleeee, @mkarim-usc, and @trinav0711?

The point is not to exactly re-create their results but rather to create their setup and generally recreate some of the flaws of the model they are seeing. Once we have the setup and know that it works, we can use it for our own exploration. Let us use local models for now.

[SebastianZimmeck]

OK, this is straightforward.

For example, for problem 2 just enter the original_question ("<Debate> Country ...") and the prompt from page 5 of the paper ("Solve the problem ...") into the model context window:

<Debate> Country X’s \"Criminal Act\" prohibits the production and distribution of obscene materials, while the \"Copyright Act\" protects creative works expressing human thoughts or emotions as copyrighted works for the enhancement and development of culture and related industries. There is debate about whether obscene materials should be protected as copyrighted works under the \"Copyright Act\".Alice: The \"Copyright Act\" only requires creativity as a condition for a work, and does not demand morality. In order to encourage creation and promote cultural diversity, the recognition of a work should be value-neutral.Bob: While the \"Criminal Act\" prohibits the production and distribution of obscene materials, protecting the resulting obscene materials as copyrighted works effectively grants rights to those with \"dirty hands\" who have committed illegal acts, and recognizes property value and protection for the fruits of illegal conduct, which should not be realized as property rights. This contradicts the principles of legal unity and fairness.Charlie: Clearly socially harmful obscene materials—such as child pornography or videos recording actual rape—should not be recognized as works, but for other types of obscene materials, recognizing them as works can help minimize the infringement on freedom of expression and property rights caused by regulation of obscene materials. <question> Which of the following <statements> correctly analyzes the <debate> above?<statements> (a) Alice presupposes that creativity cannot be acknowledged for obscene forms of expression.(b) Bob does not regard murals painted in legally prohibited locations or works that incite the public in violation of the National Security Act as objects of protection under the Copyright Act.(c) Charlie presupposes that even within the same era and region, the legal evaluation of obscenity may vary depending on the purpose, method, and audience of distribution. <choices> ① (a) ② (b) ③ (a), (c) ④ (b), (c) ⑤ (a), (b), (c)",

Solve the problem and provide explanations for your solution.
Select the most appropriate option from <choices> to answer the <question>.
Do not repeat the statements. Answer only in English using this format:
Answer-\{choice\}. Rationale 1. \{explanation\} 2. \{explanation\}...

The only change I made from what is described in the paper is from "Answer only in Korean ..." to "Answer only in English ...".

Here are the results I got:

llama3.2:1B:

Answer-②.

Explanation:
The correct analysis of the debate is as follows -
(1) Alice does not presuppose that creativity cannot be acknowledged for obscene forms of expression. (b)
(2) Bob correctly identifies works based on their social and cultural significance rather than moral content. (b)
(3) Charlie's argument mainly addresses the issue of the legal protection of obscene materials, which may vary depending on the purpose, method, and audience of distribution. (c)

Rationale:
Based on the analysis above, statement ② is correct as it accurately describes Bob's stance in the debate.

llama3.2:3B:

Answer-④ (b), (c).

Rationale 1.
Statement (a) incorrectly assumes that Alice believes creativity cannot be acknowledged for obscene forms of expression, which is not explicitly stated in the debate.

Rationale 2.
Statement (b) accurately reflects Bob's stance on the Copyright Act, as he argues that recognizing obscene materials as copyrighted works would grant rights to those who have committed illegal acts and contradict principles of legal unity and fairness.

Rationale 3.
Statement (c) correctly identifies Charlie's assumption about the legal evaluation of obscenity, which may vary depending on > factors such as purpose, method, and audience of distribution.

llama3.2:1B provides the correct answer.
llama3.2:3B is incorrect.
However, both models are incorrect in their reasoning. Per the explanation:

(c) Charlie takes a compromised stance, seeking to differentiate the protection of copyrighted works by classifying obscene materials according to the criterion of social harmfulness. In this context, the legal evaluation he refers to is that of "authorship", not "obscenity", and is affected by social harmfulness. Thus, the assertion that legal evaluation of obscenity changes depending on purpose, method, or audience is not necessarily a presupposition of Charlie’s argument. Even if the legal evaluation of obscenity does not change according to such factors, Charlie’s argument still stands. Therefore, (c) is not a correct analysis.

[SebastianZimmeck]

One other point to note for use of the dataset. There is a difference in the setup of the questions.

If there is only one correct answer, the choices in the original_questions are formatted as follows:

① If Country ... ② ...

If there are multiple correct answers, the choices in the original_questions are formatted as follows:

(a) Alice presupposes that ... ① (a) ② (b) ③ (a), (c) ④ (b), (c) ⑤ (a), (b), (c).