Beyond epistemic collapse: disagreement-aware scientific RAG
The failure mode: one fluent answer
TL;DR. Ask a scientific RAG system whether homework improves achievement, whether statins help in primary prevention, or whether a higher minimum wage costs jobs, and you get one confident paragraph, even though the literature is genuinely split. We call that epistemic collapse. EVIRAG keeps the disagreement instead: it retrieves adversarially, builds a signed claim graph, attributes why sources disagree, and answers with source-linked views and an evidence-structured confidence. EVIRAG-BENCH, 1,250 queries across five contested domains, measures whether a system preserved the disagreement.

The paper is careful about what this claim is. It is not that every query needs many answers; it is that questions with persistent disagreement need an output interface that preserves viewpoint structure, provenance and uncertainty. Improving retrieval recall alone does not fix it if synthesis still compresses the evidence into one paragraph. “Epistemic collapse” is used as a technical term here and should not be confused with its meaning in modal epistemology.
What counts as collapse
Definition 1 makes the failure behavioural and testable: a system exhibits epistemic collapse on a query with ground-truth view set V* if its answer covers at most one view, ViewpointCoverage ≤ 1 / |V*|. A long single answer that does lay out several views does not collapse under this definition, which is exactly why the paper includes a structured-prompt baseline that does just that.
There is also a mechanism. Topical retrieval tends to oversupply the dominant view, a “consensus attractor”, and a single-vector summarisation objective presses toward it. The homework query shows the difference in practice:
| Vanilla RAG | EVIRAG Full | |
|---|---|---|
| Answer to “Does homework improve achievement?” | “Research consistently shows that homework improves academic achievement, with stronger effects for older students (grades 7–12)… Homework builds self-regulation and reinforces classroom learning.” | View 1 (dominant, 6 sources, medium): conditional positive effect for grades 7–12, near zero for primary. View 2 (alternative, 3 sources, medium): null effect once SES and teacher quality are controlled. View 3 (minority, 2 sources, low): heavy loads harm wellbeing with no academic benefit. |
| CDA-7 attribution | — | View 1: methodological (grade-level scope) · View 2: statistical + methodological · View 3: operational (achievement vs wellbeing) |
| Gold viewpoints covered (VC_emb) | 0.33 (1 of 3) | 1.00 (3 of 3) |
| Controversy class | — | stable |
Both answers are fluent and locally accurate. Only one of them tells you the finding depends on grade level, that the effect vanishes once socio-economic status and teacher quality are controlled, and that the outcome you measure (achievement or wellbeing) decides the answer.
EVIRAG: seven stages, one principle
The design principle is that disagreement must be modelled before synthesis: once the evidence is a paragraph, the viewpoint structure cannot be recovered. The pipeline therefore does its structural work first and writes last.
| Stage | What the released code does | LLM calls |
|---|---|---|
| 1 · Intent | Asks “contested or resolved?”. Resolved: one Precision query, top 10 passages, one concise answer, no graph. | 1 |
| 2 · Retrieval | For each of four roles the model writes one search query; dense top-3, 5, 4, 3 chunks; merged and de-duplicated (≤ 15). | 4 |
| 3 · Claims & edges | Up to five atomic claims per chunk (JSON, with a sentence-split fallback). Every claim pair whose cosine is ≥ 0.35 and that is not from the same chunk is labelled supports / contradicts / neutral; neutral edges are dropped. | ≤ 15 + one per candidate pair |
| 4 · CDA-7 | For each contradiction edge, one word: methodological, population, temporal, operational, statistical, theoretical or replication. | one per contradiction |
| 5 · Temporal | A per-year table: claim counts and contradiction links. | 0 |
| 6 · Views | Signed Louvain over the claim graph; one JSON call per community: position, summary, weaknesses. | one per view |
| 7 · Confidence | A weighted score per view, mapped to high / medium / low (details below). | 0 |
Stage 2: retrieval that argues with itself
A single query retrieves the consensus. EVIRAG runs four role agents with different objectives (Precision, Recall, Skeptic, Counterfactual), each writing its own search query, with a fixed budget of at most 15 chunks. The Skeptic is the only agent whose objective is explicitly anti-consensus.
Stages 3 to 6: a signed graph of claims
Retrieved passages are broken into atomic claims, each carrying a provenance pointer. Every plausible pair is labelled supports, contradicts or neutral, and every contradiction is given a cause. The graph is partitioned with signed Louvain modularity, implemented from scratch in the released Rust; each community becomes one view.
CDA-7: why do they disagree?
NLI tells you that two claims conflict; it does not tell you why. CDA-7 adds that dimension, with seven causes, one primary cause stored on each contradiction edge.
| # | Cause | Definition (benchmark guide) |
|---|---|---|
| 1 | Replication | A later attempt repeats the earlier design and fails to recover its finding. |
| 2 | Population | Samples, age groups, settings or inclusion criteria differ. |
| 3 | Operational | The studies use different definitions or outcome measures for the same named concept. |
| 4 | Methodological | The study design, intervention, protocol or control strategy differs. |
| 5 | Statistical | Estimates or interpretations of uncertainty differ despite comparable designs. |
| 6 | Temporal | Evidence from a later period differs from evidence from an earlier period. |
| 7 | Theoretical | Competing causal or explanatory frameworks account for the same observations. |
Stage 7: confidence from evidence, not fluency
Each view gets a confidence tier. The class of the controversy (below) scales how confident a view can be, so a polarized topic is never presented with the confidence of a settled one, however well the prose is written.
| CDA-7 cause on the contradictions | Severity weight | Reading |
|---|---|---|
| replication | 1.0 | a failed replication is the most damaging kind of conflict |
| statistical · temporal | 0.8 | conflicting uncertainty readings; a later period disagreeing |
| methodological · operational | 0.6 | different designs or definitions |
| population · theoretical | 0.5 | different samples or frameworks |
| missing label | 0.7 | unclassified conflicts get a middle penalty |
EVIRAG-BENCH: measuring what systems erase
Standard retrieval metrics reward topical relevance to a single answer. EVIRAG-BENCH measures epistemic fidelity: how much of the structured disagreement in a literature a response preserves. The corpus is 989 papers across five domains with persistent, well-documented controversy.
| Domain | Topic | Papers | Chunks | Claims |
|---|---|---|---|---|
| Education | homework effectiveness | 201 | 8,751 | 13,082 |
| Biomedicine | statins in primary prevention | 198 | 8,612 | 12,847 |
| Economics | minimum wage and employment | 195 | 7,963 | 11,924 |
| Earth sciences | climate feedbacks | 203 | 9,347 | 14,013 |
| Nutrition science | dietary fat and cardiovascular risk | 192 | 8,482 | 12,089 |
| Total | 989 | 43,155 | 63,955 |
| Annotation layer | Content | Per query |
|---|---|---|
| Ground-truth viewpoints | distinct epistemic positions, each with provenance | 2–4 |
| Contradiction pairs | explicit pairs of atomic claims that cannot both hold | 1–8 |
| CDA-7 labels | the cause of disagreement for each contradicting pair | 1 per pair |
| Controversy class | Resolved / Emerging / Stable / Polarized | 1 |
| Human rating | epistemic completeness, 1–5 (250-query subset) | 3 annotators |
- Freeze the corpusSave each paper’s DOI, title, year, venue, access date and licence; mark full text versus abstract.
- Draft questions independently of system outputs250 per domain; record the source papers behind each.
- Annotate viewpointsTwo annotators independently identify 2–4 positions, each with a proposition and a passage ID, distinguished by conclusion, population, definition or theory.
- Record contradiction pairs1–8 pairs of incompatible atomic claims per query, each labelled with a CDA-7 cause. Compatible claims are not contradictions.
- Assign a controversy classAfter inspecting the evidence over time, with a written rationale.
- Adjudicate and freeze before any system runsKeep both independent files, the adjudicated file, notes, annotator IDs and a checksum of the frozen JSONL.
The paper reports Cohen’s κ = 0.71 for viewpoint identity and κ = 0.68 for CDA-7 conflict types on a 200-query re-annotation by an external consultant. The gold file was version-locked before any system was run, and every system was scored against it unchanged. The annotation was done by the authors with one external domain consultant per domain.
One gold record
{"id": "edu_001", "domain": "education",
"question": "Does homework improve academic achievement?",
"viewpoints": [{"id": "v1", "summary": "…", "sources": [{"doc_id": "DOI or corpus ID", "passage_id": "chunk ID"}]}, …],
"contradictions": [{"left": "Atomic claim from one source", "right": "Incompatible atomic claim from another source", "cda7": "population"}],
"controversy_class": "stable", "annotation_status": "unadjudicated"}The metrics, and the one the paper threw away
| Metric | Measures | Needs gold? | Direction |
|---|---|---|---|
| ContradictionRecall (CR) | fraction of gold contradiction pairs the output recovers as contradicting claim pairs | yes | higher |
| ViewpointCoverage (VC_emb) | fraction of gold viewpoints surfaced, by cosine ≥ 0.65 (all-MiniLM-L6-v2) | yes | higher |
| Single-View Concentration (SVC) | how much claim mass sits on one gold view (normalised entropy) | yes | lower |
| Confidence Calibration Error (CCE) | distance between emitted confidence tier and gold controversy class | yes | lower |
| FaithfulnessScore (FS) | fraction of statements supported by their cited passages (NLI judge) | no | higher |
| EC@K, ED, PI | retrieval-side coverage and the typology inputs; diagnostics, not system rankings | partly | – |
Because SVC depends on which gold view each claim is nearest to (an argmax) rather than on the size of a cosine, it is invariant to the inflation of absolute cosines that encoder anisotropy induces. That makes it a better collapse measure than anything that averages similarities.
A metric audit, and a negative result
A metric can be plausible, publishable and still measure the opposite of its intent. The paper audits its own suite for direction validity: on instances where the right ordering is known independently of the metric, does the metric agree?
Results
| System | SVC ↓ | VC_emb | CR | CCE ↓ | FS |
|---|---|---|---|---|---|
| Vanilla RAG | 0.783 | 0.423 | 0.572 | 0.125 | 0.824 |
| Single-agent | 0.731 | 0.438 | 0.580 | 0.121 | 0.834 |
| EVIRAG-NoGraph | 0.487 | 0.512 | 0.604 | 0.138 | 0.847 |
| EVIRAG-NoMultiView | 0.487 | 0.289 | 0.428 | 0.108 | 0.813 |
| MADAM-RAG | 0.531 | 0.508 | 0.618 | 0.132 | 0.856 |
| PaperQA2 | 0.584 | 0.467 | 0.553 | 0.109 | 0.903 |
| MMR-RAG | 0.647 | 0.534 | 0.595 | 0.129 | 0.839 |
| EVIRAG Full | 0.287 | 0.847 | 0.742 | 0.231 | 0.891 |
Which part earns its keep?
Dropping multi-view output costs the most; dropping the retrieval ensemble or the claim graph costs less but still hurts. Note what the NoGraph row says: with the same evidence and the same multi-view format but no graph, contradiction recall reaches only 0.604, below the structured prompt’s 0.613. What the graph buys is contradiction structure, not output shape.
Controls: is it just more compute?
EVIRAG makes about 14 calls where baselines make one, so a skeptic can say the gains are just more inference. The paper adds three controls: a closed-book run, a budget-matched run that retrieves 15 chunks like EVIRAG, and a single call with a structured prompt that asks for every viewpoint, its evidence, its weaknesses and why the sources disagree.
| Control (stratified 250-query subset) | LLM calls | VC_emb | CR | FS |
|---|---|---|---|---|
| Closed-book (no retrieval) | 1 | 0.392 | 0.214 | – |
| Vanilla RAG (k = 10) | 1 | 0.423 | 0.572 | 0.824 |
| Vanilla RAG (k = 15), budget-matched | 1 | 0.521 | 0.612 | – |
| Structured-prompt (k = 15) | 1 | 0.736 | 0.613 | 0.812 |
| EVIRAG-NoGraph (k = 15) | 5 | 0.512 | 0.604 | 0.847 |
| EVIRAG Full (k = 15) | 14 | 0.847 | 0.742 | 0.891 |
Across domains, backbones and encoders
| Backbone (education subset, 50 queries) | Vanilla CR | Vanilla VC | EVIRAG CR | EVIRAG VC |
|---|---|---|---|---|
| Qwen2.5-7B | 0.541 | 0.391 | 0.648 | 0.703 |
| Llama-3.1-70B | 0.563 | 0.418 | 0.712 | 0.821 |
| Qwen3.6-35B (main) | 0.580 | 0.423 | 0.760 | 0.847 |
What humans think
| Score | Rubric anchor |
|---|---|
| 1 | Single-answer collapse with no meaningful acknowledgment of disagreement. |
| 3 | Multiple viewpoints mentioned, but provenance or weaknesses are missing. |
| 5 | All identified viewpoints represented, each traceable to its sources and stated with its weaknesses, and the response conveys why the sources disagree, in any form: prose, a contrast of designs or populations, or an explicit label. |
Speed: the warm path
| Query | EVIRAG warm | Vanilla RAG | Speed-up |
|---|---|---|---|
| Effectiveness | 0.553 s | 12.268 s | 22.18× |
| Science outcomes | 0.411 s | 7.841 s | 19.08× |
| Achievement and attitudes | 0.330 s | 11.761 s | 35.60× |
| Mean | 0.431 s | 10.623 s | 25.62× |
Where it breaks
| Failure mode | What happens | Fix suggested |
|---|---|---|
| False contradiction | related claims with different emphasis are labelled as conflicting (prompt-based NLI noise) | a stronger cross-encoder verifier |
| Missing minority view | rare terminology prevents retrieval of a dissenting paper | query expansion and terminology substitution |
| Over-amplified weak dissent | a single-source minority view gets more salience than its evidence deserves | stronger confidence display and source-count emphasis |
The paper’s own list of limits is long and specific, and worth reading as the other half of the result:
- Benchmark circularity. The benchmark, the metrics and the system share authorship. The paper’s defences are the cross-system ordering (the structured prompt, which shares none of EVIRAG’s components, beats the NoGraph ablation on coverage), the version-locked gold with external consultants, and metrics adapted from SciFact, ExpertQA and FActScore. It then says it does not claim neutrality: the decisive test is a disagreement-aware system built by another group, evaluated on EVIRAG-BENCH by that group.
- Compute is not fully matched. The clean test, a call- and token-matched version of EVIRAG’s own strategy, has not been run. The paper states its prediction in advance and says it would report a failure.
- Louvain’s resolution limit. A small enough minority view can be absorbed into a larger community rather than surfaced as its own view, understating coverage exactly where minority positions are rarest.
- Publication bias. Null and negative results are underrepresented in academic corpora; the Skeptic cannot recover evidence that was never published.
- Scope. Five contested domains; not evidence that the gains transfer to settled factual QA or to arbitrary web RAG. On settled queries the system can produce unnecessary multi-view output, and the intent gate is imperfect: false positives give mild over-complexity, false negatives give collapse.
- Model dependence. One open-weight backbone for the main results; prompt-based NLI remains the major source of error; temporal drift is not ablated.
Reading the released code against the paper
The repository is a Rust re-implementation released with the camera-ready. It is a faithful skeleton of the pipeline, and in a few places simpler than the prose. For anyone reproducing it, these are the differences worth knowing:
| Topic | Paper | Released Rust implementation |
|---|---|---|
| Polarization Index | signed-Laplacian measure of conflict separation | share of contradiction edges whose endpoints fall in different communities |
| Temporal drift | a temporal disagreement curve | a per-year count table of claims and contradiction links |
| Confidence weights | names the ingredients, no numbers | weights 0.25 / 0.25 / 0.20 / 0.15 / 0.15 and class factors 1.0 / 0.9 / 0.85 / 0.75 |
| Skeptic prompt | quotes a passage-finding instruction | asks the model to write a search query for “null results, criticism, and contradictory findings” |
| Typology gaps | four classes with four threshold rules | combinations not covered by a rule return “stable” |
| Faithfulness | atomic-claim NLI verification (FActScore style) | sentence-level judgement by the same LLM |
| Claim extraction caps | – | at most 5 claims per chunk and per response text; generation capped at 256 new tokens |
| CCE | listed for every system | computed only where a system emits a confidence tier |
Run it
Rust, Ollama and a chat and an embedding model are all it needs. The smoke script exercises every command on the fixture; the full commands take your own corpus and gold file in the formats above.
git clone https://github.com/amethystani/evirag-bench && cd evirag-bench
cargo build --release
./scripts/smoke.sh # chunk, index, run, validate-gold, evaluate on a 1-query fixture
ollama serve # scripts/fetch_models.sh pulls qwen3.6:35b-a3b and all-minilm
target/release/evirag-bench validate-gold data/gold/pilot.jsonl # add --full for the 1,250-query layout
target/release/evirag-bench chunk data/corpus/documents.jsonl data/corpus/chunks.jsonl
target/release/evirag-bench index data/corpus/chunks.jsonl data/index.json
target/release/evirag-bench run data/index.json data/gold/benchmark.jsonl runs/full.jsonl --system full
target/release/evirag-bench evaluate runs/full.jsonl data/gold/benchmark.jsonl runs/full.metrics.json
scripts/run_all.sh # every system, then paired comparisonsEvery LLM call goes through Ollama’s /api/generate with temperature 0 and a 256-token cap, and embeddings are L2-normalised in code. The test suite (cargo test) covers the chunker’s exact overlap, three signed-Louvain cases (two camps are recovered exactly), the typology boundaries, the bootstrap and an exact Wilcoxon test, and needs no model. CI runs formatting, tests and clippy with warnings denied on Linux and macOS.
@inproceedings{mishra2026evirag,
title = {Beyond Epistemic Collapse: Disagreement-Aware Scientific Retrieval-Augmented Generation},
author = {Mishra, Animesh and Sharma, Krishang and Khetarpaul, Sonia},
booktitle = {Proceedings of EMNLP},
year = {2026}
}Related work this one sits next to: MADAM-RAG (structured debate that resolves to one answer; CR 0.618 here against EVIRAG’s 0.742 because its moderator collapses viewpoints at synthesis), PaperQA2 (scientific RAG with the best faithfulness, 0.903, but one answer), ConflictQA, SciFact and FActScore. EVIRAG shares the multi-agent architecture of the first but inverts its objective: it preserves the conflict.


