Study finds compile checks can mislead AI vulnerability-repair benchmarks

Compile checks can mislead AI repair benchmarks.

Compile checks can mislead AI repair benchmarks. A September 22 preprint reports five controlled experiments on 203 vulnerable Big-Vul functions and three open-source code language models, evaluated in an isolated-function setting. In that isolated-function setup, the authors attributed about 64% of compilation failures to harness and dataset artifacts rather than model-generated output. The study did not validate generated repairs using execution tests or exploit checks.

Show more