Overview
DebateJudge runs structured policy debates between two LLM debater agents, each embodying an economic school of thought. A third LLM agent -- the judge -- scores each exchange across 6 weighted dimensions, detects 7 types of logical fallacies via regex patterns, and produces a final verdict with full transcript. The system enforces turn structure according to one of 3 debate formats.
Capability Boundaries
What this tool does:
- Orchestrate structured debates between two persona-driven debater agents.
- Assign 8 economic school personas to debaters, each with defined key assumptions, value priorities, rhetorical style, constraints, notable thinkers, and signature argument moves:
1. Neoclassical Economist
2. Institutional Economist
3. Behavioral Economist
4. Public Choice Theorist
5. Austrian Economist
6. Marxist Economist
7. Keynesian Economist
8. Development Economist
- Enforce 3 debate formats with turn-by-turn structure (PolicyDebateFormat and others in debate_judge/formats.py).
- Score exchanges across 6 dimensions via weighted rubric aggregation.
- Detect 7 fallacy types during scoring using regex patterns:
1. Ad Hominem
2. Straw Man
3. False Dichotomy
4. Appeal to Authority
5. Slippery Slope
6. (additional types in _FALLACY_PATTERNS)
7. (see scoring.py for full list)
- Track per-round scores and accumulate them into a final verdict.
- Produce a complete DebateTranscript of all turns and scores.
What this tool does NOT do:
- It does NOT teach debate skills or provide debate tips.
- It does NOT support real-time human-vs-human judging (it only judges LLM-vs-LLM debates).
- It does NOT generate debate topics automatically -- the user must provide a topic/resolution.
- It is NOT a general-purpose argumentation tool -- personas are limited to economic schools of thought.
- It does NOT guarantee perfectly neutral judging; the rubric rewards substantive engagement over rhetorical flourish, not ideological alignment.
Trigger Conditions
Use this skill when the user:
- Wants to run a simulated debate between economic perspectives on a policy question.
- Needs structured scoring of argumentation quality.
- Wants to see how different economic schools would argue a particular issue.
- Mentions phrases like "debate between economists", "judge this argument", "policy debate", "economic schools debate", or "辩论评分".
Usage
The central orchestrator is DebateArena in debate_judge/arena.py:
from debate_judge import DebateArena, Persona
arena = DebateArena(
topic="Should central banks adopt nominal GDP targeting?",
format="policy_debate",
)
arena.set_debater_a(Persona.from_library("neoclassical_economist"))
arena.set_debater_b(Persona.from_library("keynesian_economist"))
arena.set_judge("neoclassical_economist")
result = arena.run()
print(result.verdict)
print(result.transcript)
Personas
Personas are defined in config/personas.yaml. Each persona has:
- school: The economic school.
- key_assumptions: Foundational beliefs about how the economy works.
- value_priorities: What this tradition considers most important.
- rhetorical_style: How arguments are framed and delivered.
- constraints: Rules the persona must follow (e.g., must cite at least one economic mechanism).
- notable_thinkers: Key intellectual figures.
- signature_moves: Characteristic argument patterns.
Scoring Rubric
The ScoringRubric in debate_judge/scoring.py provides:
- Multi-criterion scoring with weighted aggregation.
- Fallacy detection via regex patterns.
- Evidence quality assessment.
- Round-by-round tracking.
Dependencies
- LLM backend: OpenAI-compatible endpoint (Ollama, OpenAI, etc.)
- PyYAML for persona configuration.
Output Example
A debate run produces:
- DebateTranscript: Full turn-by-turn record with speaker, content, and per-round scores.
- Final Verdict: Aggregate scores across all rounds with the judge's reasoning.
- Scoring Breakdown: Per-dimension scores for each exchange.
FAQ
Q: Can I add new economic personas?
A: Yes. Add a new entry to config/personas.yaml following the existing structure (school, key_assumptions, value_priorities, rhetorical_style, constraints, notable_thinkers, signature_moves). They will be available via Persona.from_library().
Q: Can I use this for non-economic debates?
A: The persona library is economics-focused. To use it for other domains, you would need to create new persona definitions and potentially adjust the scoring rubric.
Q: How are fallacies detected?
A: Fallacies are detected via regex pattern matching in scoring.py (_FALLACY_PATTERNS). Each fallacy type has a list of regex patterns that match common linguistic markers. This is a heuristic approach, not guaranteed to catch all fallacies.
Q: Is the judge biased toward certain schools?
A: The scoring rubric explicitly states it "rewards substantive engagement over rhetorical flourish" and "does NOT reward ideological alignment with the judge." However, any LLM-based judge carries inherent biases from its training data.
