Measure token savings only after the answer passes.
A local run that uses fewer hosted-model tokens can still be a failed run. Compare both outputs with an independently chosen expected answer before you credit any avoided tokens.
If both outputs match expected:
credited avoided tokens = max(0, baseline ChatGPT tokens − comparison ChatGPT tokens)
Otherwise: 0
Keep local-model tokens in a separate field. Do not subtract them from ChatGPT token counts or convert them to dollars without verified rates and billing records.
1. Choose one task and its expected answer
Write down the result the task requires before comparing runs. Use a short, deterministic example first, such as extracting a fixed list or returning a known status. If the expected answer is wrong or incomplete, the comparison is not meaningful.
2. Record the two outputs and their usage receipts
Save the hosted-model baseline and the local/alternate run. Copy input and output token counts from the provider's usage record. Record any ChatGPT calls used by the alternate workflow separately. Set that value to zero only when the usage record supports zero.
3. Apply the same quality gate to both
The Jev Evidence Kit's current checker uses exact text matching after trimming outer whitespace and normalizing Windows line endings. This is intentionally narrow: it does not judge semantic equivalence or general task quality. If either output fails the expected-answer check, credited avoided ChatGPT tokens are zero even when one run used fewer tokens.
Baseline: 145 observed ChatGPT tokens. Alternate run: 12 observed ChatGPT tokens and 88 local tokens. If both answers match the expected result, the checker credits 133 avoided ChatGPT tokens and lists the 88 local tokens separately. If the alternate answer fails, the credit is zero.
These numbers illustrate the arithmetic only; they are not a benchmark or a cost-saving result.
What this method cannot prove
- It does not run either model, select a route, or intercept ChatGPT.
- It does not verify invoices, provider billing, or monetary savings.
- Exact string equality is not a substitute for expert review when meaning, safety, or completeness matter.
- A single successful pair does not establish performance across a workload. Repeat the comparison on representative tasks and keep failed cases.
The offline Python CLI reads a local JSON file and makes no network requests. The browser example also checks values locally. The optional site counter is separate and remains disabled until configured; neither the demo nor this guide sends prompt text, answers, or token values.