How do you test whether an AI PDF research-paper summarizer is accurate?

1. Define accuracy before judging how complete the answer looks

A summary can be grammatical and well organized while getting the population, comparison baseline, or scope of a conclusion wrong. Make the unit of evaluation a claim that can be checked against the paper, not an overall impression of the prose.

FENICE follows this direction by extracting atomic claims and aligning them to the source document. You do not need to reproduce the full automatic metric for a product test, but the same principle is useful: every important claim should lead back to evidence.

2. Use a small, varied paper set—not one clean PDF

Choose 5–10 papers that you or a colleague already understand well enough to check manually. One example only establishes performance on that file; it cannot represent publisher layouts, disciplines, or scan quality.

Include at least four cases: a claim in continuous prose, a decisive result in a table, a method that depends on equations and symbol definitions, and a PDF with two columns, footnotes, image text, or OCR problems. Add one question whose evidence is deliberately absent to test abstention.

• Prose case: check the main claim and the scope of the conclusion.
• Table case: check values, units, dataset, and baseline.
• Equation case: check symbol definitions, conditions, and dependencies.
• Difficult PDF: check reading order, garbled text, footnotes, and figure text.
• Missing-evidence case: check whether the tool says “not found” or invents an answer.

3. Fix the questions and output format before comparing tools

Use the same PDF, conversation context, and questions for every tool. Do not repeatedly coach one tool after a failure and compare its repaired answer with another tool's first attempt.

A minimum test asks for four things: the central claim in one sentence; the exact page, section, table, or equation supporting it; the strongest limitation explicitly stated by the authors; and “not found” when the paper does not provide the evidence. Record the model version, date, web access, and OCR setting because these conditions change the result.

4. Score separate failure modes instead of prose quality

A simple 0–2 scale works for each item: 0 for wrong or unsupported, 1 for partly correct with an inaccurate scope or location, and 2 for supported and directly verifiable. Keep claim support, evidence location, omission, and abstention as separate scores before considering any total.

LongDocFACTScore and FactPICO both show why long or specialized document summaries cannot be represented by traditional text-overlap scores or a single automatic factuality metric alone. For a small buying decision, manual inspection of important claims is usually more interpretable than a polished composite score.

• Claim support: does the paper actually say or establish it?
• Evidence location: does the cited page, section, table, or equation exist and support it?
• Scope control: did correlation become causation, or a local result become a universal claim?
• Critical omission: were method conditions, baselines, or author-stated limitations skipped?
• Abstention: does the tool admit when the evidence is absent?

5. Use clean text as a control for PDF extraction failures

If a tool scrambles a table, merges two columns, or drops an equation, the model can reason correctly over the wrong input and still produce a wrong answer. When a result fails, provide a manually corrected version of the same page or table and repeat the question.

Correct with clean text but wrong with the original PDF points mainly to parsing or OCR. Wrong under both conditions points more toward retrieval, context use, or reasoning. This control will not identify every cause, but it avoids treating the whole pipeline as one vague “model quality” problem.