Beyond epistemic collapse: disagreement-aware scientific RAG

Listen
Share

The failure mode: one fluent answer

TL;DR. Ask a scientific RAG system whether homework improves achievement, whether statins help in primary prevention, or whether a higher minimum wage costs jobs, and you get one confident paragraph, even though the literature is genuinely split. We call that epistemic collapse. EVIRAG keeps the disagreement instead: it retrieves adversarially, builds a signed claim graph, attributes why sources disagree, and answers with source-linked views and an evidence-structured confidence. EVIRAG-BENCH, 1,250 queries across five contested domains, measures whether a system preserved the disagreement.

1,250queries across five contested domains
989peer-reviewed papers, 63,955 atomic claims
7pipeline stages, 7 causes of disagreement
0.742 vs 0.572ContradictionRecall, EVIRAG vs vanilla RAG
Paper Figure 1: standard RAG compresses three viewpoints into one consensus centroid; EVIRAG keeps them apart in a multi-view answer.
Figure 1 The paper’s own picture of the problem. Left: standard RAG pulls viewpoints V1, V2 and V3 toward one consensus centroid μ and returns a single averaged answer. Right: EVIRAG keeps the viewpoints separate and answers with all of them, with provenance. Source: paper · Figure 1

The paper is careful about what this claim is. It is not that every query needs many answers; it is that questions with persistent disagreement need an output interface that preserves viewpoint structure, provenance and uncertainty. Improving retrieval recall alone does not fix it if synthesis still compresses the evidence into one paragraph. “Epistemic collapse” is used as a technical term here and should not be confused with its meaning in modal epistemology.

What counts as collapse

Definition 1 makes the failure behavioural and testable: a system exhibits epistemic collapse on a query with ground-truth view set V* if its answer covers at most one view, ViewpointCoverage ≤ 1 / |V*|. A long single answer that does lay out several views does not collapse under this definition, which is exactly why the paper includes a structured-prompt baseline that does just that.

A single answer vector cannot sit near several separated viewpointsThree viewpoint clusters V1, V2 and V3 are far apart in embedding space; a single answer point has a ball of radius delta that contains at most one cluster centroid.V1dominant viewV2alternative viewV3minority viewaδ-ball around the answer aembedding space (unit sphere in ℝ³⁸⁴, all-MiniLM-L6-v2)if ‖cᵢ − cⱼ‖ > 2δ for all i ≠ j, a is within δ of at most one centroid
Figure 2 Why a single-vector objective presses toward the dominant view. If the gold viewpoint centroids are more than 2δ apart, no single answer can be close to more than one of them (triangle inequality). The paper is explicit about the scope: a real long-form answer is not a single point, so this explains the pressure, not an impossibility. Source: paper · Remark 1, §3.2

There is also a mechanism. Topical retrieval tends to oversupply the dominant view, a “consensus attractor”, and a single-vector summarisation objective presses toward it. The homework query shows the difference in practice:

Vanilla RAGEVIRAG Full
Answer to “Does homework improve achievement?”“Research consistently shows that homework improves academic achievement, with stronger effects for older students (grades 7–12)… Homework builds self-regulation and reinforces classroom learning.”View 1 (dominant, 6 sources, medium): conditional positive effect for grades 7–12, near zero for primary. View 2 (alternative, 3 sources, medium): null effect once SES and teacher quality are controlled. View 3 (minority, 2 sources, low): heavy loads harm wellbeing with no academic benefit.
CDA-7 attribution—View 1: methodological (grade-level scope) · View 2: statistical + methodological · View 3: operational (achievement vs wellbeing)
Gold viewpoints covered (VC_emb)0.33 (1 of 3)1.00 (3 of 3)
Controversy class—stable
The paper’s case study for one query, as printed in its appendix. “Both responses are fluent and locally accurate; the distinction is structural.” Source: paper Table 20, Appendix H

Both answers are fluent and locally accurate. Only one of them tells you the finding depends on grade level, that the effect vanishes once socio-economic status and teacher quality are controlled, and that the outcome you measure (achievement or wellbeing) decides the answer.

EVIRAG: seven stages, one principle

The design principle is that disagreement must be modelled before synthesis: once the evidence is a paragraph, the viewpoint structure cannot be recovered. The pipeline therefore does its structural work first and writes last.

The seven-stage EVIRAG pipelineA snake-shaped flow: query, intent analysis, adversarial retrieval, claim graph on the top row; CDA-7 attribution, temporal drift, multi-view synthesis and confidence on the bottom row, ending in source-linked views.Queryinput1 Intentanalysiscontested orresolved?2 Adversarialretrieval4 role agents≤ 15 chunks3 Claim graphatomic claims,signed edges7 Evidenceconfidencenever fromfluency6 Multi-viewsynthesissigned Louvain,one view per group5 Temporaldriftclaims andconflicts by year4 CDA-7attributioncause for eachcontradiction↓ source-linked views: position, evidence, weaknesses, why sources disagree, confidence tierDesign principle: model the disagreement before synthesis; once it is a paragraph it cannot be recovered.
Figure 3 EVIRAG, redrawn from the paper’s Figure 2 and Algorithm 1. Stages 2 and 6 (filled) carry the main design choices: adversarial retrieval to break the consensus attractor, and one view per community of the signed claim graph instead of one blended answer. Source: paper · Figure 2, Algorithm 1
StageWhat the released code doesLLM calls
1 · IntentAsks “contested or resolved?”. Resolved: one Precision query, top 10 passages, one concise answer, no graph.1
2 · RetrievalFor each of four roles the model writes one search query; dense top-3, 5, 4, 3 chunks; merged and de-duplicated (≤ 15).4
3 · Claims & edgesUp to five atomic claims per chunk (JSON, with a sentence-split fallback). Every claim pair whose cosine is ≥ 0.35 and that is not from the same chunk is labelled supports / contradicts / neutral; neutral edges are dropped.≤ 15 + one per candidate pair
4 · CDA-7For each contradiction edge, one word: methodological, population, temporal, operational, statistical, theoretical or replication.one per contradiction
5 · TemporalA per-year table: claim counts and contradiction links.0
6 · ViewsSigned Louvain over the claim graph; one JSON call per community: position, summary, weaknesses.one per view
7 · ConfidenceA weighted score per view, mapped to high / medium / low (details below).0
The stages mapped to the Rust implementation. All generation is deterministic (temperature 0), capped at 256 new tokens, run through Ollama with the chain-of-thought switch off. Source: src/pipeline.rs, src/retrieval.rs, src/graph.rs

Stage 2: retrieval that argues with itself

A single query retrieves the consensus. EVIRAG runs four role agents with different objectives (Precision, Recall, Skeptic, Counterfactual), each writing its own search query, with a fixed budget of at most 15 chunks. The Skeptic is the only agent whose objective is explicitly anti-consensus.

Adversarial retrieval budgetsA stacked bar showing the four retrieval roles: Precision 3 chunks, Recall 5, Skeptic 4 and Counterfactual 3, totalling at most 15, against a ten-chunk single-query baseline.Precision · 3Recall · 5Skeptic · 4Counterfactual · 3role agents, one search query written by the model per role (up to 15 chunks before de-duplication)Vanilla RAG · top-10 of one querybaselineOnly the Skeptic is explicitly anti-consensus. Its aim in the released code: “null results, criticism, and contradictory findings”.
Figure 4 Four agents with orthogonal objectives break the “consensus attractor”: topical retrieval oversupplies the dominant view, so EVIRAG spends a fixed budget on dissent, minority positions and breadth rather than on more of the same. Source: paper · §4 Stage 2; Appendix B

Stages 3 to 6: a signed graph of claims

Retrieved passages are broken into atomic claims, each carrying a provenance pointer. Every plausible pair is labelled supports, contradicts or neutral, and every contradiction is given a cause. The graph is partitioned with signed Louvain modularity, implemented from scratch in the released Rust; each community becomes one view.

Signed claim graph and its communitiesSix claims in two camps: supporting edges inside each camp, contradiction edges between camps labelled with CDA-7 causes; each camp becomes one view.view 1 · community Aview 2 · community Bcontradicts · populationcontradicts · methodologicalcontradicts · statisticalc1c2c3c4c5c6solid line = SUPPORTS dashed line = CONTRADICTS (+ CDA-7 cause) neutral edges are droppedSigned Louvain maximises modularity where supports pull claims together and contradictions push them apart.
Figure 5 A schematic of Stages 3 to 6 (not real data). Claims become nodes; supports and contradictions become signed edges; the number of views is whatever the signed partition yields, with no fixed cap in the released Rust. Source: github.com/amethystani/evirag-bench · src/graph.rs

CDA-7: why do they disagree?

NLI tells you that two claims conflict; it does not tell you why. CDA-7 adds that dimension, with seven causes, one primary cause stored on each contradiction edge.

#CauseDefinition (benchmark guide)
1ReplicationA later attempt repeats the earlier design and fails to recover its finding.
2PopulationSamples, age groups, settings or inclusion criteria differ.
3OperationalThe studies use different definitions or outcome measures for the same named concept.
4MethodologicalThe study design, intervention, protocol or control strategy differs.
5StatisticalEstimates or interpretations of uncertainty differ despite comparable designs.
6TemporalEvidence from a later period differs from evidence from an earlier period.
7TheoreticalCompeting causal or explanatory frameworks account for the same observations.
CDA-7, the seven causes of disagreement. A pair is contradictory only if both claims concern comparable propositions and cannot both hold under the stated conditions; one primary cause is stored on the edge. The taxonomy is motivated by philosophy-of-science accounts of how disagreements are sustained (Longino, Solomon). Source: docs/cda7.md
CDA-7 classification F1 (system-predicted vs gold, 250 queries)CDA-7 classification F1 (system-predicted vs gold, 250 queries)Temporal0.79Methodological0.72Theoretical0.69Replication0.68Operational0.65Population0.63Statistical0.60Macro average0.68
Figure 6 How well the pipeline recovers the cause of a disagreement. Temporal is easiest; Statistical is hardest. The main confusions are at the Methodological–Population and Statistical–Methodological boundaries. The paper calls 0.68 macro-F1 “thin” for a module the causal story leans on and advises reading individual labels as indicative. Source: paper · Table 19

Stage 7: confidence from evidence, not fluency

Each view gets a confidence tier. The class of the controversy (below) scales how confident a view can be, so a polarized topic is never presented with the confidence of a settled one, however well the prose is written.

Controversy typology on the ED–PI planeA plane with Epistemic Divergence on the x axis and Polarization Index on the y axis, divided into Resolved, Emerging, Stable and Polarized regions by thresholds.ResolvedED < 0.15 · PI < 0.3Emerging0.15 ≤ ED < 0.35 · PI < 0.4StableED ≥ 0.35 · PI < 0.4PolarizedED ≥ 0.35 · PI ≥ 0.4no class in the paper; the released code returns “stable”00.150.350.500.30.41Epistemic Divergence (mean pairwise distance between view centroids)Polarization Index
Figure 7 The four-way controversy typology that calibrates confidence. Resolved evidence is shown with high confidence, Emerging and Stable with medium, Polarized with low. Thresholds were set by inspecting a 25-query pilot; fewer than 8% of queries change class under ±0.05 perturbation. The dashed block is a gap in the paper’s rules that the code fills with “stable”. Source: paper · Appendix A · src/pipeline.rs
Confidence formula as implementedA stacked bar of five component weights (agreement 25, source diversity 25, one minus severity 20, temporal stability 15, provenance 15) multiplied by a class factor, then mapped to high, medium or low.25%agreement25%source diversity20%1 − severity15%temporal stability15%provenanceweighted sum of five signals per view× class factorResolved 1.00Emerging 0.90Stable 0.85Polarized 0.75→ tierlow < 0.45medium 0.45–0.75high ≥ 0.75
Figure 8 The released Stage 7. The paper names the ingredients (claim agreement, source diversity, contradiction severity, CDA-7 priors, temporal trajectory, never fluency); the numeric weights shown here come from the Rust code, not the paper. Source: github.com/amethystani/evirag-bench · src/pipeline.rs:120-200
CDA-7 cause on the contradictionsSeverity weightReading
replication1.0a failed replication is the most damaging kind of conflict
statistical · temporal0.8conflicting uncertainty readings; a later period disagreeing
methodological · operational0.6different designs or definitions
population · theoretical0.5different samples or frameworks
missing label0.7unclassified conflicts get a middle penalty
How contradiction severity enters the confidence score: the mean weight over the contradictions touching a view. Agreement is (supports + 1) / (supports + conflicts + 1); diversity is the number of distinct source documents capped at three, over three; temporal stability falls as recent conflicts accumulate. Source: src/pipeline.rs

EVIRAG-BENCH: measuring what systems erase

Standard retrieval metrics reward topical relevance to a single answer. EVIRAG-BENCH measures epistemic fidelity: how much of the structured disagreement in a literature a response preserves. The corpus is 989 papers across five domains with persistent, well-documented controversy.

DomainTopicPapersChunksClaims
Educationhomework effectiveness2018,75113,082
Biomedicinestatins in primary prevention1988,61212,847
Economicsminimum wage and employment1957,96311,924
Earth sciencesclimate feedbacks2039,34714,013
Nutrition sciencedietary fat and cardiovascular risk1928,48212,089
Total98943,15563,955
The EVIRAG-BENCH corpus: 989 peer-reviewed papers chosen through domain-expert recommendation and forward and backward citation tracing, chunked into 256-token passages with 32-token overlap. Source: paper Table 7
EVIRAG-BENCH queries by controversy class (1,250 total, 250 per domain)EVIRAG-BENCH queries by controversy class (1,250 total, 250 per domain)Resolved190Emerging335Stable445Polarized280
Figure 9 The mix of queries. Resolved queries (15%) are deliberate negatives: settled questions where multi-view output should be suppressed. The intent gate suppresses it on 155 of the 190 (81.6%); the other 35 get mild unnecessary multi-view output. Source: paper · §5.2, Appendix D
Annotation layerContentPer query
Ground-truth viewpointsdistinct epistemic positions, each with provenance2–4
Contradiction pairsexplicit pairs of atomic claims that cannot both hold1–8
CDA-7 labelsthe cause of disagreement for each contradicting pair1 per pair
Controversy classResolved / Emerging / Stable / Polarized1
Human ratingepistemic completeness, 1–5 (250-query subset)3 annotators
What each benchmark query carries. Annotations are written around the literature, not around any system’s output. Source: paper Table 13
  1. Freeze the corpusSave each paper’s DOI, title, year, venue, access date and licence; mark full text versus abstract.
  2. Draft questions independently of system outputs250 per domain; record the source papers behind each.
  3. Annotate viewpointsTwo annotators independently identify 2–4 positions, each with a proposition and a passage ID, distinguished by conclusion, population, definition or theory.
  4. Record contradiction pairs1–8 pairs of incompatible atomic claims per query, each labelled with a CDA-7 cause. Compatible claims are not contradictions.
  5. Assign a controversy classAfter inspecting the evidence over time, with a written rationale.
  6. Adjudicate and freeze before any system runsKeep both independent files, the adjudicated file, notes, annotator IDs and a checksum of the frozen JSONL.

The paper reports Cohen’s κ = 0.71 for viewpoint identity and κ = 0.68 for CDA-7 conflict types on a 200-query re-annotation by an external consultant. The gold file was version-locked before any system was run, and every system was scored against it unchanged. The annotation was done by the authors with one external domain consultant per domain.

One gold record

data/schema.example.json (shape of one gold record)
{"id": "edu_001", "domain": "education",
 "question": "Does homework improve academic achievement?",
 "viewpoints": [{"id": "v1", "summary": "…", "sources": [{"doc_id": "DOI or corpus ID", "passage_id": "chunk ID"}]}, …],
 "contradictions": [{"left": "Atomic claim from one source", "right": "Incompatible atomic claim from another source", "cda7": "population"}],
 "controversy_class": "stable", "annotation_status": "unadjudicated"}

The metrics, and the one the paper threw away

MetricMeasuresNeeds gold?Direction
ContradictionRecall (CR)fraction of gold contradiction pairs the output recovers as contradicting claim pairsyeshigher
ViewpointCoverage (VC_emb)fraction of gold viewpoints surfaced, by cosine ≥ 0.65 (all-MiniLM-L6-v2)yeshigher
Single-View Concentration (SVC)how much claim mass sits on one gold view (normalised entropy)yeslower
Confidence Calibration Error (CCE)distance between emitted confidence tier and gold controversy classyeslower
FaithfulnessScore (FS)fraction of statements supported by their cited passages (NLI judge)nohigher
EC@K, ED, PIretrieval-side coverage and the typology inputs; diagnostics, not system rankingspartly–
The metric suite. Statistics are bootstrap 95% intervals and paired Wilcoxon signed-rank tests with Bonferroni correction. Source: paper §3.3, §5; src/eval.rs
Single-View Concentration on four example answersFour small bar triplets showing the share of an answer’s claims on each of three gold views, with the resulting SVC: 1.00, 0.37, 0.18 and 0.V1V2V3All on one viewSVC = 1.00p = (1, 0, 0)V1V2V3Two of threeSVC = 0.37p = (0.5, 0.5, 0)V1V2V3SkewedSVC = 0.18p = (0.6, 0.3, 0.1)V1V2V3BalancedSVC = 0.00p = (0.33, 0.33, 0.33)SVC = 1 + Σ pᵢ log pᵢ / log |V*| lower is better: 1 is the collapse endpoint, 0 is an even spread
Figure 10 The collapse metric, worked by hand. Each claim in the answer is assigned to the nearest gold view by cosine; p is the share of claim mass on each view. These four distributions are illustrative (computed from the formula, not from data). Source: paper · Eq. 1, §3.3

Because SVC depends on which gold view each claim is nearest to (an argmax) rather than on the size of a cosine, it is invariant to the inflation of absolute cosines that encoder anisotropy induces. That makes it a better collapse measure than anything that averages similarities.

A metric audit, and a negative result

A metric can be plausible, publishable and still measure the opposite of its intent. The paper audits its own suite for direction validity: on instances where the right ordering is known independently of the metric, does the metric agree?

Direction check: orderings called correctly on 8 gold instancesDirection check: orderings called correctly on 8 gold instancesCCS (reference-free)0 / 8SVC8 / 8ContradictionRecall8 / 8VC_emb8 / 8
Figure 11 A negative result inside the paper. The first headline collapse metric, CCS, never consults the gold annotation, and on every one of eight known-collapsed-versus-known-multi-view pairs it ranked them the wrong way round (sign test p = 0.008). A response that genuinely covers several separated views pushes its dominant cluster away from the mean of its clusters, raising CCS: it was measuring cluster tightness and calling it viewpoint balance. The paper replaced it with SVC. On the homework query, CCS rated vanilla RAG 0.38 against EVIRAG 0.52 (wrong way round); SVC rated the pair in the expected direction. Source: paper · Table 15, Appendix F

Results

ViewpointCoverage and ContradictionRecall, eight systemsViewpointCoverage and ContradictionRecall, eight systems0.000.250.500.751.000.420.57VanillaRAG0.440.58Single-agent0.510.60NoGraph0.290.43NoMulti-View0.510.62MADAM-RAG0.470.55PaperQA20.530.59MMR-RAG0.850.74EVIRAGFullViewpointCoverage (VC_emb)ContradictionRecall (CR)
Figure 12 The headline result. On 1,250 queries with a 35B open-weight backbone, EVIRAG Full reaches CR 0.742 against vanilla RAG’s 0.572 and VC_emb 0.847 against 0.423, and the paper reports these differences as significant against every other system (paired Wilcoxon, Bonferroni-corrected). Removing multi-view output (NoMultiView) is the worst result of all. Source: paper · Table 2
SystemSVC ↓VC_embCRCCE ↓FS
Vanilla RAG0.7830.4230.5720.1250.824
Single-agent0.7310.4380.5800.1210.834
EVIRAG-NoGraph0.4870.5120.6040.1380.847
EVIRAG-NoMultiView0.4870.2890.4280.1080.813
MADAM-RAG0.5310.5080.6180.1320.856
PaperQA20.5840.4670.5530.1090.903
MMR-RAG0.6470.5340.5950.1290.839
EVIRAG Full0.2870.8470.7420.2310.891
Main results (paper Table 2). Best per column highlighted for the quality metrics. CR and VC_emb carry ±0.02 and ±0.01 bootstrap 95% half-widths in the paper. CCE is listed for completeness but is not comparable across systems: only EVIRAG emits an explicit confidence tier, and the paper says plainly that it does not claim calibration as an advantage (under partial dissent EVIRAG under-claims certainty, and CCE penalises that). Source: paper Table 2 (backbone Qwen3.6-35B)
Single-View Concentration by system (lower is better)Single-View Concentration by system (lower is better)Vanilla RAG0.783Single-agent0.731MMR-RAG0.647PaperQA20.584MADAM-RAG0.531NoGraph0.487NoMultiView0.487EVIRAG Full0.287
Figure 13 How concentrated each system’s answer is on one gold view. Every system that returns several views (the two NoGraph/NoMultiView ablations included) is well below single-answer systems; EVIRAG Full is lowest at 0.287. Source: paper · Table 2, Appendix F.2

Which part earns its keep?

Change in ContradictionRecall when a component is removedChange in ContradictionRecall when a component is removedRetrieval ensemble−0.162Claim graph−0.138Multi-view output−0.314
Change in ViewpointCoverage when a component is removedChange in ViewpointCoverage when a component is removedRetrieval ensemble−0.409Claim graph−0.335Multi-view output−0.558
Figure 14 Ablations. Collapsing the final answer back to a single response does the most damage on both metrics, more than removing the retrieval ensemble or the claim graph. Stage 1 (intent gating) and Stage 5 (temporal drift) are not ablated in the paper; it says the temporal component’s marginal contribution “remains an open question”. Source: paper · Table 14, Appendix D

Dropping multi-view output costs the most; dropping the retrieval ensemble or the claim graph costs less but still hurts. Note what the NoGraph row says: with the same evidence and the same multi-view format but no graph, contradiction recall reaches only 0.604, below the structured prompt’s 0.613. What the graph buys is contradiction structure, not output shape.

Controls: is it just more compute?

EVIRAG makes about 14 calls where baselines make one, so a skeptic can say the gains are just more inference. The paper adds three controls: a closed-book run, a budget-matched run that retrieves 15 chunks like EVIRAG, and a single call with a structured prompt that asks for every viewpoint, its evidence, its weaknesses and why the sources disagree.

Control (stratified 250-query subset)LLM callsVC_embCRFS
Closed-book (no retrieval)10.3920.214–
Vanilla RAG (k = 10)10.4230.5720.824
Vanilla RAG (k = 15), budget-matched10.5210.612–
Structured-prompt (k = 15)10.7360.6130.812
EVIRAG-NoGraph (k = 15)50.5120.6040.847
EVIRAG Full (k = 15)140.8470.7420.891
Controls that separate architecture from cheaper explanations. Closed-book says a topic is contested but cannot recover specific contradiction pairs. The structured prompt’s faithfulness slips (0.812 vs 0.824) as the heavier instruction pulls toward unsupported synthesis. The paper states a prediction in advance: under a call- and token-matched control the coverage gap should close further and a substantial CR gap should remain; “if the CR gap also closes, the central claim of this paper is wrong, and we would report that.” That control has not been run. Source: paper Table 3, §5.6, Limitations
Share of the gap to EVIRAG Full that each control closesShare of the gap to EVIRAG Full that each control closes0%25%50%75%100%23%24%Vanilla k=15(budget-matched)1 call21%19%EVIRAG-NoGraph(same evidence,no graph) 5 calls74%24%Structured prompt(k=15)1 callViewpointCoverage gap closedContradictionRecall gap closed
Figure 15 The most informative control. A single call with a structured prompt closes about three quarters of the viewpoint-coverage gap (so coverage is largely promptable and “should not be credited to architecture”), but only about a quarter of the contradiction-recall gap, the same share as simply retrieving more. Contradiction recall needs pairs of claims to be compared; generating more structured text does not do that. (Gap closed = (control − vanilla) / (EVIRAG − vanilla), computed from the paper’s Table 3 values.) Source: paper · Table 3, §5.6 · derived from the paper’s table
Average tokens per query (input + output), thousandsAverage tokens per query (input + output), thousandsClosed-book1 call0.92kVanilla RAG1 call5.24kMMR-RAG1 call5.86kSingle-agent1 call7.14kMADAM-RAG9 calls18.7kPaperQA2~6 calls31.6kEVIRAG-NoGraph5 calls46.8kEVIRAG Full14 calls68.9k
Figure 16 The price of structure. EVIRAG Full uses 14 calls and about 68,900 tokens per query: roughly 14× the calls and 13× the tokens of a single-call baseline. The paper treats the unmatched compute as a real confound and says that controls bound it from both sides without removing it. Tokens for the k=15 and structured-prompt controls were not logged. Source: paper · Table 5

Across domains, backbones and encoders

ContradictionRecall by domain: vanilla RAG vs EVIRAG FullContradictionRecall by domain: vanilla RAG vs EVIRAG Full0.000.250.500.751.000.580.76Education0.570.73Biomedicine0.590.79Economics0.560.72Earth sci.0.550.71NutritionVanilla RAGEVIRAG Full
Figure 17 No domain reverses. Gains are larger where the controversy is bipolar (economics +0.20, education +0.18) than in the multi-factorial domains (biomedicine, earth sciences and nutrition, +0.16 each). All five per-domain improvements are significant (p < 0.05, 250 queries each). Only CR is reported per domain. Source: paper · Table 6
Backbone (education subset, 50 queries)Vanilla CRVanilla VCEVIRAG CREVIRAG VC
Qwen2.5-7B0.5410.3910.6480.703
Llama-3.1-70B0.5630.4180.7120.821
Qwen3.6-35B (main)0.5800.4230.7600.847
Cross-backbone robustness. The advantage holds across model families and sizes. Only the education domain was checked. Source: paper Table 17, Appendix G
ViewpointCoverage under two encodersViewpointCoverage under two encoders0.0000.2500.5000.7501.0000.4230.847all-MiniLM-L6-v2(used in the system)0.4620.781bge-base-en-v1.5(disjoint encoder)Vanilla RAGEVIRAG Full
Figure 18 The metric uses the same encoder family the system retrieves with, which could flatter it. Re-scoring with a disjoint encoder narrows the gap from +0.424 to +0.319; the paper reads the 0.105 difference as its estimate of the shared-encoder advantage and notes the ordering is unchanged. Checked for these two systems only. Source: paper · Table 16, Appendix F.5

What humans think

Human-rated epistemic completeness (1–5), 250 queries, 3 annotatorsHuman-rated epistemic completeness (1–5), 250 queries, 3 annotators0.001.252.503.755.002.104.30All scores2.103.91Scores censored at 4Vanilla RAGEVIRAG Full
Figure 19 The encoder-free check on the automatic metrics: EVIRAG Full 4.3 against vanilla RAG 2.1 (Krippendorff’s α = 0.72, Mann–Whitney p < 0.001, Cohen’s d = 2.71). To guard against the top anchor flattering a structured format, every score is censored at 4: the ordering survives (3.91 vs 2.10). Annotators saw the gold viewpoint set; a reference-free replication is future work. Source: paper · §5.5, Appendix C
ScoreRubric anchor
1Single-answer collapse with no meaningful acknowledgment of disagreement.
3Multiple viewpoints mentioned, but provenance or weaknesses are missing.
5All identified viewpoints represented, each traceable to its sources and stated with its weaknesses, and the response conveys why the sources disagree, in any form: prose, a contrast of designs or populations, or an explicit label.
The rubric (scores 2 and 4 interpolate). Anchor 5 is deliberately implementation-neutral so that no system can reach the top of the scale by output format alone. Source: paper Table 10

Speed: the warm path

Warm-query latency (seconds)Warm-query latency (seconds)03.5710.5140.55312.268Effectiveness0.4117.841Science outcomes0.3311.761Achievement0.43110.623MeanEVIRAG warm pathVanilla RAG
Figure 20 Precomputing the claim graph offline lets a query slice a retrieved subgraph, replacing the O(|C|²) relation labelling with O(k) work and a short synthesis over precomputed structure, so multi-view output is preserved at about 0.43 s per query against about 10.6 s for vanilla RAG’s long-form synthesis. The speed-up amortises an offline cost; it does not avoid it. Source: paper · Table 8, §5.8
QueryEVIRAG warmVanilla RAGSpeed-up
Effectiveness0.553 s12.268 s22.18×
Science outcomes0.411 s7.841 s19.08×
Achievement and attitudes0.330 s11.761 s35.60×
Mean0.431 s10.623 s25.62×
Warm-query latency over the three queries in the paper’s latency benchmark. Both columns are end-to-end (retrieval, graph operations and generation) on the same hardware. The mean speed-up is the mean of per-query ratios; the ratio of mean times is 24.65×. Source: paper Table 8
The one-time offline buildA stacked bar of the offline build: chunking and embedding 0.6 hours, claim extraction 3.9 hours, pairwise relation labelling and CDA-7 6.5 hours, about 11 hours total.0.6 hChunking + embedding43,155 chunks3.9 hClaim extraction63,955 claims6.5 hPairwise labelling + CDA-7the O(|C|²) stepone-time cost for the 989-paper corpus on a single RTX 4000 Ada (20 GB)≈ 11 h total
Figure 21 Where the warm path’s speed comes from: it is paid for here, once. The per-query warm path then spends 0.04 s on dense retrieval, 0.07 s on subgraph slicing and 0.32 s on synthesis (0.43 s total); vanilla RAG spends 0.04 s on retrieval and 10.58 s on one long-form generation. Source: paper · Table 9

Where it breaks

Why do the 22 low-recall queries fail? (EVIRAG Full, CR < 0.5)Why do the 22 low-recall queries fail? (EVIRAG Full, CR < 0.5)NLI noise41% of the 229Claim-granularity mismatch32%7Retrieval coverage gaps18%4CDA-7 boundary confusion9%2
Figure 22 Error analysis. Prompt-based NLI is the largest single source of error (false contradictions between related claims), followed by claim-granularity mismatch against the gold pairs and retrieval gaps (rare terminology hides a dissenting paper). The paper’s expectation is that future gains are likelier from stronger verifiers than from structural change. Source: paper · §5.8, Appendices G and I
Failure modeWhat happensFix suggested
False contradictionrelated claims with different emphasis are labelled as conflicting (prompt-based NLI noise)a stronger cross-encoder verifier
Missing minority viewrare terminology prevents retrieval of a dissenting paperquery expansion and terminology substitution
Over-amplified weak dissenta single-source minority view gets more salience than its evidence deservesstronger confidence display and source-count emphasis
Documented failure modes of the pipeline. Source: paper Table 18

The paper’s own list of limits is long and specific, and worth reading as the other half of the result:

  • Benchmark circularity. The benchmark, the metrics and the system share authorship. The paper’s defences are the cross-system ordering (the structured prompt, which shares none of EVIRAG’s components, beats the NoGraph ablation on coverage), the version-locked gold with external consultants, and metrics adapted from SciFact, ExpertQA and FActScore. It then says it does not claim neutrality: the decisive test is a disagreement-aware system built by another group, evaluated on EVIRAG-BENCH by that group.
  • Compute is not fully matched. The clean test, a call- and token-matched version of EVIRAG’s own strategy, has not been run. The paper states its prediction in advance and says it would report a failure.
  • Louvain’s resolution limit. A small enough minority view can be absorbed into a larger community rather than surfaced as its own view, understating coverage exactly where minority positions are rarest.
  • Publication bias. Null and negative results are underrepresented in academic corpora; the Skeptic cannot recover evidence that was never published.
  • Scope. Five contested domains; not evidence that the gains transfer to settled factual QA or to arbitrary web RAG. On settled queries the system can produce unnecessary multi-view output, and the intent gate is imperfect: false positives give mild over-complexity, false negatives give collapse.
  • Model dependence. One open-weight backbone for the main results; prompt-based NLI remains the major source of error; temporal drift is not ablated.

Reading the released code against the paper

The repository is a Rust re-implementation released with the camera-ready. It is a faithful skeleton of the pipeline, and in a few places simpler than the prose. For anyone reproducing it, these are the differences worth knowing:

TopicPaperReleased Rust implementation
Polarization Indexsigned-Laplacian measure of conflict separationshare of contradiction edges whose endpoints fall in different communities
Temporal drifta temporal disagreement curvea per-year count table of claims and contradiction links
Confidence weightsnames the ingredients, no numbersweights 0.25 / 0.25 / 0.20 / 0.15 / 0.15 and class factors 1.0 / 0.9 / 0.85 / 0.75
Skeptic promptquotes a passage-finding instructionasks the model to write a search query for “null results, criticism, and contradictory findings”
Typology gapsfour classes with four threshold rulescombinations not covered by a rule return “stable”
Faithfulnessatomic-claim NLI verification (FActScore style)sentence-level judgement by the same LLM
Claim extraction caps–at most 5 claims per chunk and per response text; generation capped at 256 new tokens
CCElisted for every systemcomputed only where a system emits a confidence tier
Reading the code against the paper. The paper is the reference for what is claimed; the code is the reference for what a re-run does. Places where the released implementation is simpler than the prose are listed here for anyone reproducing the pipeline. Source: src/pipeline.rs, src/eval.rs, src/graph.rs vs the paper

Run it

Rust, Ollama and a chat and an embedding model are all it needs. The smoke script exercises every command on the fixture; the full commands take your own corpus and gold file in the formats above.

shell
git clone https://github.com/amethystani/evirag-bench && cd evirag-bench
cargo build --release
./scripts/smoke.sh                      # chunk, index, run, validate-gold, evaluate on a 1-query fixture
ollama serve                            # scripts/fetch_models.sh pulls qwen3.6:35b-a3b and all-minilm

target/release/evirag-bench validate-gold data/gold/pilot.jsonl          # add --full for the 1,250-query layout
target/release/evirag-bench chunk data/corpus/documents.jsonl data/corpus/chunks.jsonl
target/release/evirag-bench index data/corpus/chunks.jsonl data/index.json
target/release/evirag-bench run data/index.json data/gold/benchmark.jsonl runs/full.jsonl --system full
target/release/evirag-bench evaluate runs/full.jsonl data/gold/benchmark.jsonl runs/full.metrics.json
scripts/run_all.sh                      # every system, then paired comparisons

Every LLM call goes through Ollama’s /api/generate with temperature 0 and a 256-token cap, and embeddings are L2-normalised in code. The test suite (cargo test) covers the chunker’s exact overlap, three signed-Louvain cases (two camps are recovered exactly), the typology boundaries, the bootstrap and an exact Wilcoxon test, and needs no model. CI runs formatting, tests and clippy with warnings denied on Linux and macOS.

BibTeX (from the repository README)
@inproceedings{mishra2026evirag,
  title     = {Beyond Epistemic Collapse: Disagreement-Aware Scientific Retrieval-Augmented Generation},
  author    = {Mishra, Animesh and Sharma, Krishang and Khetarpaul, Sonia},
  booktitle = {Proceedings of EMNLP},
  year      = {2026}
}

Related work this one sits next to: MADAM-RAG (structured debate that resolves to one answer; CR 0.618 here against EVIRAG’s 0.742 because its moderator collapses viewpoints at synthesis), PaperQA2 (scientific RAG with the best faithfulness, 0.903, but one answer), ConflictQA, SciFact and FActScore. EVIRAG shares the multi-agent architecture of the first but inverts its objective: it preserves the conflict.

All articles

Research and building cool stuff

© 2026, Animesh Mishra

GitHub|LinkedIn

New Delhi, India