efficiency
diapason.telemetry.efficiency
¶
MFU/MBU efficiency calculator for GPU inference telemetry.
Computes Model FLOPs Utilization (MFU) and Model Bandwidth Utilization (MBU) to quantify how efficiently a model uses available GPU compute and memory bandwidth.
EfficiencyMetrics
dataclass
¶
Results of an MFU/MBU efficiency calculation.
compute_efficiency(param_count_b, active_params_b, gpu_peak_tflops, gpu_peak_bandwidth_gb_s, tokens_per_sec, num_gpus=1, energy_joules=0.0, accuracy=0.0, bytes_per_param=2.0)
¶
Compute MFU, MBU, and derived efficiency metrics.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
param_count_b
|
float
|
Total parameter count in billions. |
required |
active_params_b
|
float | None
|
Active parameters per token in billions (None for dense). |
required |
gpu_peak_tflops
|
float
|
Peak theoretical TFLOPS per GPU (e.g. 312 for A100 SXM FP16). |
required |
gpu_peak_bandwidth_gb_s
|
float
|
Peak memory bandwidth per GPU in GB/s (e.g. 2039 for A100 SXM). |
required |
tokens_per_sec
|
float
|
Measured generation throughput (tokens/second). |
required |
num_gpus
|
int
|
Number of GPUs used for inference. |
1
|
energy_joules
|
float
|
Total energy consumed in joules (for IPJ calculation). |
0.0
|
accuracy
|
float
|
Accuracy score in [0, 1] (for IPJ calculation). |
0.0
|
bytes_per_param
|
float
|
Bytes per parameter (default 2.0 for FP16). |
2.0
|
Returns:
| Type | Description |
|---|---|
EfficiencyMetrics
|
class: |
estimate_model_bytes_per_token(param_count_b, bytes_per_param=2.0)
¶
Estimate bytes of memory loaded per decode step.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
param_count_b
|
float
|
Total parameter count in billions. |
required |
bytes_per_param
|
float
|
Bytes per parameter (default 2.0 for FP16). |
2.0
|
Returns:
| Type | Description |
|---|---|
float
|
Bytes loaded per token. |
estimate_model_flops_per_token(param_count_b, active_params_b=None)
¶
Estimate FLOPs for one forward-pass token of a dense transformer.
For dense models, FLOPs per token ≈ 2 * params. For MoE models, pass
active_params_b (the number of active parameters per token).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
param_count_b
|
float
|
Total parameter count in billions. |
required |
active_params_b
|
float | None
|
Active parameters per token in billions. If None,
defaults to |
None
|
Returns:
| Type | Description |
|---|---|
float
|
Estimated FLOPs per token. |