A PaperBridge long-tail question
How do you estimate the inference cost of a method described in an AI paper?
Count every model, retriever, and tool call triggered by one user request, then record input and output length, hardware, batch size, and precision for each call. FLOPs, parameter count, or one latency number is not enough for product cost. Throughput, concurrency, cache hits, and retries also matter.
Map the full call chain for one request
Agents, RAG systems, and multi-stage methods often call models several times. List query rewriting, retrieval, reranking, generation, verification, and retries, and mark which steps run in parallel or sequence.
An average call count hides failure paths. Inspect whether tail requests need more retrieval rounds, tool calls, or human intervention.
Compare latency and throughput under matching conditions
Record model version, hardware, numerical precision, batch size, context length, and output length. Millisecond numbers from different configurations are not directly comparable.
Online products often care about time to first token, generation speed, and tail latency under concurrency. Offline jobs care more about total throughput. Use the metric that matches your scenario.
Convert system cost into cost per accepted result
Add GPU time, API fees, retrieval, storage, and human review for one request, then divide by successfully completed tasks. A high failure rate raises effective cost.
A small test on real inputs is safer than extrapolating one paper number. Hold a quality threshold fixed when comparing cost so cheaper output is not merely worse output.
Six variables in inference cost
- How many model and tool calls does one request trigger?
- How long are the inputs and outputs?
- Which hardware, precision, and batch size were used?
- What are first-token latency, total latency, and throughput?
- How do caching, concurrency, and retries affect the tail?
- What does each result that passes quality checks cost?
Primary research and official documentation
These sources support the facts. Workflow and comparison guidance is PaperBridge's synthesis of research, official documentation, and engineering practice.