AI Patch Validation Analytics: Exploit-Variant Resistance and Root-Cause Conformance in LLM-Based Vulnerability Repair
Main article
Abstract
Large language models are increasingly used to generate candidate security patches, yet a patch that appears successful under one validation signal may still be incomplete, exploit-specific, or unsafe when stronger evidence is applied. This article develops an analytics-oriented framework for evaluating LLM-based vulnerability repair through two security-critical lenses: exploit-variant resistance and root-cause conformance. Instead of treating patch validation as a binary pass/fail decision, the study models validation as a layered evidence profile that combines build validity, functional preservation, original proof-of-concept blocking, exploit-variant resistance, CWE-specific root-cause assessment, static-warning rechecking, and regression-safety review. A controlled patch-validation dataset containing 400 candidate repairs across SQL injection, path traversal, cross-site scripting, and missing authorization scenarios is analyzed to show how weak validation oracles inflate apparent repair success. The analysis demonstrates that original exploit blocking and static-warning disappearance provide useful but incomplete signals. In the primary model configuration, 79.5% of patches blocked the original exploit, whereas 49.0% satisfied the full validation profile. CWE-level results further show that path traversal exhibits strong variant-bypass risk, SQL injection exhibits root-cause conformance gaps, and missing authorization remains difficult because correct repair depends on trust-boundary placement rather than local code edits. The paper contributes a patch-validation analytics model, a reporting structure for oracle divergence, and practical guidance for validation-aware repair workflows. The findings indicate that LLM-based vulnerability repair should be evaluated through layered security evidence rather than by a single permissive oracle.
