# VulnBump: a dependency fix only counts if the tests still pass

> VulnBump is Snyk's open benchmark that counts a dependency fix only when the vulnerability is gone and the tests still pass.

- **Severity:** Info: Research, tools, evals or lab claims with no direct exposure. Context, not an alert.
- **Category:** Benchmark, Research
- **Published by:** Snyk
- **Disclosed:** 2026-10-12
- **Affects:** Claude Code, Claude Sonnet 5.5, Snyk Evo Remediation Agent
- **Research by:** Gabija Pasiunaite, Ryan McMorrow, Brian Clark, Liran Tal, Luke Tu
- **Primary source:** https://snyk.io/blog/vulnbump/
- **Page:** https://theinjection.dev/items/snyk-vulnbump/

VulnBump is an open benchmark from Snyk that counts a dependency upgrade as fixed only when the vulnerabilities are gone and the build and tests still pass. Four approaches scored 46.9% to 90.2% across 307 tasks.

## What happened

Snyk posted VulnBump to arXiv on 2026-10-12 and published the scores in a write-up on 2026-10-15. Gabija Pasiunaite, Ryan McMorrow, Brian Clark, Liran Tal and Luke Tu built 307 dependency upgrade tasks from 43 real repositories, covering 2,066 fixable vulnerabilities across six package managers. The harness and all 43 evaluation cases are released under Apache 2.0.

## How it works

Each VulnBump task runs in a fresh container and passes only when four gates clear: dependencies resolve, a rescan shows the vulnerabilities fixed, the project builds, and previously passing tests still pass. Every entrant ran on Claude Sonnet 5.5 with a 110-turn limit, so the comparison measures the harness around the model rather than the model itself. The authors report that Claude Code's failures usually leave the vulnerability in place, while the Snyk agent's failures usually break the build or the tests.

## Who is affected

Teams that let coding agents raise dependency upgrades are the audience, across Maven, Gradle, npm, Yarn, pip and Poetry. The authors report that 163 of the 307 tasks needed more than a version bump, and that major upgrades and large framework migrations stay hard for every approach tested.

## What to do

Check an agent's dependency fix by rescanning, rebuilding and rerunning the previously passing tests, rather than accepting a version bump as done. Weigh the scores against the limits Snyk states: one model, one run per upgrade, Snyk Open Source as the judge, and no penalty for new vulnerabilities an upgrade introduced.

## VulnBump pass rate, 307 upgrade tasks

A task passes only if the vulnerabilities are gone and the build and tests still pass. Snyk's own agent is one of the four entrants.

| Result | Score |
|---|---|
| Snyk Evo Remediation Agent | 90.2% |
| Claude Code with Snyk scan | 77.5% |
| Claude Code unaided | 59.6% |
| Package manager bump | 46.9% |

Source: https://snyk.io/blog/vulnbump/

## Quick facts

- **Scope:** 307 upgrade tasks, 43 repositories, 2,066 fixable vulnerabilities
- **Setup:** Claude Sonnet 5.5 for every entrant, 110-turn limit, one run per task
- **Package managers:** Maven, Gradle, npm, Yarn, pip, Poetry
- **Code:** Harness and all 43 cases released under Apache 2.0
- **Published:** arXiv v1 on 2026-10-12, Snyk write-up on 2026-10-15

## Sources

1. [Snyk Write-Up](https://snyk.io/blog/vulnbump/)
2. [arXiv Paper](https://arxiv.org/abs/2610.17374)
3. [Benchmark Code](https://github.com/snyk-labs/snyk-vulnbump)