Skip to content

security_scanner

diapason.evals.scorers.security_scanner

security_scanner scorer — vulnerability detection evaluation.

Tier 1: Pattern-match model output against vulnerability manifest. Tier 2: Binary checklist for severity correctness and fix quality.

Score formula

(vulns_found/total) * 0.6 + (severity_correct/found) * 0.2 + max(0, 1 - FP/3) * 0.1 + checklist_score * 0.1

SecurityScannerScorer

Bases: Scorer

Score security scanner output by vulnerability detection accuracy.