A PaperBridge long-tail question
Does Chain of Draft really save tokens, and how should you evaluate it?
Reproduce a Chain-of-Thought baseline with the same model, tasks, decoding settings, and tools, then report accuracy and token use together. The most useful primary metric is `tokens per correct answer = total output tokens ÷ number of correct answers`. Stratify easy and hard tasks, repeat each condition, and include latency, retries, and failure handling. A headline such as “74% fewer tokens” does not prove lower production cost if one brittle failure triggers retries, human review, or an incorrect action.
1. Make the two conditions genuinely comparable
Hold the model version, system prompt, task order, tool permissions, output limit, and grader constant. The reasoning instruction should be the only planned change: Chain-of-Thought in one arm and Chain of Draft in the other.
Do not treat temperature zero as a guarantee of identical runs. Repeat each condition and retain task-level results instead of a single average.
2. Use tokens per correct answer as the primary metric
Raw token count measures output length, not useful work. Dividing total output tokens by correct answers puts efficiency and quality in the same denominator. If concise reasoning solves 80 of 100 tasks and the rest require retries, it may cost more than a longer but steadier baseline.
Keep accuracy and token totals visible alongside the combined metric. Readers should be able to tell whether the gain came from shorter outputs, more correct answers, or both.
- Total output tokens and provider-reported reasoning tokens.
- Accuracy, failures, refusals, and tasks that needed a retry.
- Tokens per correct answer and actual dollar cost per successful task.
- Median and P95 latency so averages do not hide slow requests.
3. Stratify by difficulty and task type
The original Chain of Draft paper reports very large token reductions on its evaluations, while a software-engineering extension finds a more modest saving. Code changes, open-ended analysis, and long-horizon state can require more intermediate information than grade-school arithmetic.
At minimum, separate easy single-step, hard multi-step, open-ended generation, and tool-using tasks. Concise reasoning can be enabled only for the strata where it passes, rather than forcing one five-word-step prompt onto every request.
4. Include the cost of recovering from failure
Production cost includes the first inference, automatic retries, a second-model check, human review, and repair after an incorrect action. Define the recovery path for each failure type, then calculate total cost per 100 successful tasks.
If Chain of Draft is cheaper on easy work but brittle on hard work, use a lightweight classifier or rules to route low-risk tasks to concise reasoning and keep fuller reasoning plus validation for high-risk tasks.
Minimum reproducible evaluation
- Prepare at least two difficulty strata with independent grading rules.
- Lock the model, version, tasks, non-prompt settings, and tool permissions.
- Run both CoT and CoD several times per condition.
- Record task-level accuracy, output tokens, reasoning tokens, latency, and failure type.
- Compute tokens per correct answer and total cost per 100 successful tasks.
- Publish failures and scope limits, not only the average token reduction.
- Route by risk and difficulty instead of replacing every prompt.
Primary research and official documentation
These sources support the facts. Workflow and comparison guidance is PaperBridge's synthesis of research, official documentation, and engineering practice.