Skip to content

savings

diapason.server.savings

Savings calculation — compare local inference cost against cloud providers.

FLOPs and energy use a no-KV-cache model: P * N * (N+1) where P = params, N = total tokens. This reflects full recompute without cached attention.

ProviderSavings dataclass

Savings compared to a single cloud provider.

SavingsSummary dataclass

Overall savings summary across all cloud providers.

compute_savings(prompt_tokens, completion_tokens, total_calls=0, session_start=0.0, prompt_tokens_evaluated=0)

Compute savings vs cloud providers given token counts.

Two token counts are used:

  • prompt_tokens — full prompt size (system prompt + all history). Used for dollar cost comparison since cloud providers bill for every input token on every request.
  • prompt_tokens_evaluated — actual tokens processed (KV-cache- aware). In multi-turn conversations, subsequent turns only evaluate new tokens; the system prompt and prior context are served from KV cache. Used for FLOPs and energy calculations since these reflect actual compute.

When prompt_tokens_evaluated is 0 we used to fall back to prompt_tokens. That's wrong in multi-turn: routes.py aggregates by summing each turn's full prompt — which counts the system prompt N times for an N-turn conversation — so the fallback turned the FLOPs and energy estimates into N×-too-high numbers. That was the dominant contributor to the bimodal Wh/token distribution on the local savings dashboard.

Conservative behaviour now: when the KV-cache-aware count is missing, treat prompt_tokens_evaluated as 0 — so the FLOPs/energy denominator becomes just completion_tokens. That under-estimates rather than over-estimates compute.

savings_to_dict(summary)

Convert SavingsSummary to a JSON-serializable dict.