savings
diapason.server.savings
¶
Savings calculation — compare local inference cost against cloud providers.
FLOPs and energy use a no-KV-cache model: P * N * (N+1) where P = params, N = total tokens. This reflects full recompute without cached attention.
ProviderSavings
dataclass
¶
Savings compared to a single cloud provider.
SavingsSummary
dataclass
¶
Overall savings summary across all cloud providers.
compute_savings(prompt_tokens, completion_tokens, total_calls=0, session_start=0.0, prompt_tokens_evaluated=0)
¶
Compute savings vs cloud providers given token counts.
Two token counts are used:
prompt_tokens— full prompt size (system prompt + all history). Used for dollar cost comparison since cloud providers bill for every input token on every request.prompt_tokens_evaluated— actual tokens processed (KV-cache- aware). In multi-turn conversations, subsequent turns only evaluate new tokens; the system prompt and prior context are served from KV cache. Used for FLOPs and energy calculations since these reflect actual compute.
When prompt_tokens_evaluated is 0 we used to fall back to
prompt_tokens. That's wrong in multi-turn: routes.py aggregates
by summing each turn's full prompt — which counts the system prompt
N times for an N-turn conversation — so the fallback turned the FLOPs
and energy estimates into N×-too-high numbers. That was the dominant
contributor to the bimodal Wh/token distribution on the local
savings dashboard.
Conservative behaviour now: when the KV-cache-aware count is
missing, treat prompt_tokens_evaluated as 0 — so the FLOPs/energy
denominator becomes just completion_tokens. That under-estimates
rather than over-estimates compute.
savings_to_dict(summary)
¶
Convert SavingsSummary to a JSON-serializable dict.