VulnBump: a dependency fix only counts if the tests still pass

01What happened

Snyk posted VulnBump to arXiv on 2026-10-12 and published the scores in a write-up on 2026-10-15. Gabija Pasiunaite, Ryan McMorrow, Brian Clark, Liran Tal and Luke Tu built 307 dependency upgrade tasks from 43 real repositories, covering 2,066 fixable vulnerabilities across six package managers. The harness and all 43 evaluation cases are released under Apache 2.0.

02How it works

Each VulnBump task runs in a fresh container and passes only when four gates clear: dependencies resolve, a rescan shows the vulnerabilities fixed, the project builds, and previously passing tests still pass. Every entrant ran on Claude Sonnet 5.5 with a 110-turn limit, so the comparison measures the harness around the model rather than the model itself. The authors report that Claude Code's failures usually leave the vulnerability in place, while the Snyk agent's failures usually break the build or the tests.

03Who is affected

Teams that let coding agents raise dependency upgrades are the audience, across Maven, Gradle, npm, Yarn, pip and Poetry. The authors report that 163 of the 307 tasks needed more than a version bump, and that major upgrades and large framework migrations stay hard for every approach tested.

04What to do

Check an agent's dependency fix by rescanning, rebuilding and rerunning the previously passing tests, rather than accepting a version bump as done. Weigh the scores against the limits Snyk states: one model, one run per upgrade, Snyk Open Source as the judge, and no penalty for new vulnerabilities an upgrade introduced.

Measured results

Sources 3 links, all fetched

1. 1snyk.ioSnyk Write-Up
2. 2arxiv.orgarXiv Paper
3. 3github.comBenchmark Code