Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ Jeffy was run against <!-- count:tested -->132<!-- /count --> open-source projec

| Projects tested | Converged | Failed | PRs merged | PRs open | Issues filed |
|:---:|:---:|:---:|:---:|:---:|:---:|
| **<!-- count:tested -->132<!-- /count -->** | **<!-- count:converged -->103<!-- /count -->** | **<!-- count:failed -->28<!-- /count -->** | **<!-- count:merged -->46<!-- /count -->** | **<!-- count:prs-open -->28<!-- /count -->** | **<!-- count:issues -->4<!-- /count -->** |
| **<!-- count:tested -->132<!-- /count -->** | **<!-- count:converged -->103<!-- /count -->** | **<!-- count:failed -->28<!-- /count -->** | **<!-- count:merged -->47<!-- /count -->** | **<!-- count:prs-open -->28<!-- /count -->** | **<!-- count:issues -->4<!-- /count -->** |

<div align="center">

Expand All @@ -45,14 +45,19 @@ Jeffy was run against <!-- count:tested -->132<!-- /count --> open-source projec

## Independent Validation

A merged pull request is the one result Jeffy cannot award itself. It takes an independent maintainer, someone with no stake in this project, to review the patch and accept it into their own codebase. Maintainers have done exactly that <!-- count:merged -->46<!-- /count --> times, across <!-- count:merged-projects -->35<!-- /count --> open source projects, including those below.
A merged pull request is the one result Jeffy cannot award itself. It takes an independent maintainer, someone with no stake in this project, to review the patch and accept it into their own codebase. Maintainers have done exactly that <!-- count:merged -->47<!-- /count --> times, across <!-- count:merged-projects -->36<!-- /count --> open source projects, including those below.

<table>
<tr>
<th align="left">Merged by</th>
<th align="left">Finding</th>
<th align="left">Merged in</th>
</tr>
<tr>
<td><img src="https://github.com/NVIDIA.png" width="20" height="20" alt="" align="absmiddle"> NVIDIA</td>
<td><a href="https://github.com/NVIDIA/go-nvml/pull/207">go-nvml #207</a><br>The buffer handed to <code>dlinfo</code> for a library's directory was allocated with zero bytes, so the first <code>Path()</code> on a library opened by soname wrote past it and the next <code>dlclose</code> crashed</td>
<td>12 days</td>
</tr>
<tr>
<td rowspan="2"><img src="https://github.com/google.png" width="20" height="20" alt="" align="absmiddle"> Google</td>
<td><a href="https://github.com/google/snappy/pull/257">snappy #257</a><br>Every release build compressed a 4 GiB input into a stream whose header claimed 0 bytes</td>
Expand Down
5 changes: 3 additions & 2 deletions evals/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,13 @@ Empirical evidence of how an autonomous coding agent performs on real software:

| Projects tested | Fixed | Failed | PRs merged | PRs open | Issues filed |
|:---:|:---:|:---:|:---:|:---:|:---:|
| **<!-- count:tested -->132<!-- /count -->** | **<!-- count:fixed -->104<!-- /count -->** | **<!-- count:failed -->28<!-- /count -->** | **<!-- count:merged -->46<!-- /count -->** | **<!-- count:prs-open -->28<!-- /count -->** | **<!-- count:issues -->4<!-- /count -->** |
| **<!-- count:tested -->132<!-- /count -->** | **<!-- count:fixed -->104<!-- /count -->** | **<!-- count:failed -->28<!-- /count -->** | **<!-- count:merged -->47<!-- /count -->** | **<!-- count:prs-open -->28<!-- /count -->** | **<!-- count:issues -->4<!-- /count -->** |

**Fixed** means the loop's closing audit came back clean and an independent evaluator countersigned it: <!-- count:converged -->103<!-- /count --> loop runs converged, plus one audit (PapaParse) held to the same method. That is a standard this repository set and checked itself. A merged pull request is the one outcome it cannot award itself, which is why those rows come first. **Failed** means the project's pre-registered run budget ran out without convergence, or, for one project (libuv), the run was abandoned before it had a budget or a receipt; every one is published. Counted per attempt rather than per project, [ATTEMPTS.md](./ATTEMPTS.md) holds more non-converged rows than this table holds Failed rows, because a project that failed more than once, or converged only on a retry, contributes each attempt.

## Merged upstream

<!-- count:merged -->46<!-- /count --> patches this loop wrote are merged across <!-- count:merged-projects -->35<!-- /count --> projects, because a maintainer with no stake in this project reviewed each one and said yes, and three more findings were fixed upstream by others who read them:
<!-- count:merged -->47<!-- /count --> patches this loop wrote are merged across <!-- count:merged-projects -->36<!-- /count --> projects, because a maintainer with no stake in this project reviewed each one and said yes, and three more findings were fixed upstream by others who read them:

- **[bat](https://github.com/sharkdp/bat/pull/3862) - merged.** A just-merged security flag did nothing when piped; caught before it ever shipped.
- **[fasthttp](https://github.com/valyala/fasthttp/pull/2343) - merged.** A `Content-Length` no parser should accept became a wrong number.
Expand Down Expand Up @@ -52,6 +52,7 @@ Empirical evidence of how an autonomous coding agent performs on real software:
- **Microsoft, [GSL](https://github.com/microsoft/GSL/pull/1272) - merged.** `dyn_array_iterator` declared `random_access_iterator_tag` but had no `<`, `>`, `<=`, `>=`, `->` or `n + it`, and its default constructor was guarded on `__cpp_lib_ranges`, so `std::sort` over a `dyn_array` did not compile. The relational operators check the same contract as `operator==`, `->` checks the same as `*`, `n + it` forwards to `it + n`, and the default constructor is unconditional; the rework the maintainer asked for makes the iterator satisfy `std::random_access_iterator` and `std::contiguous_iterator` outright, which the tests now assert. Filed on 6 September as a follow-up to #1271; Copilot flagged `nullptr + 0` in `to_address` as undefined behaviour, the reply cited [expr.add], Carson Radtke agreed that Copilot was wrong and merged it on 15 September for the 5.0.1 patch release.
- **[money](https://github.com/RubyMoney/money/pull/1227) - merged.** `import_rates(:json, ...)` parsed the file with `JSON.load`, which honours `json_class` keys and instantiates any class carrying a `json_create` hook, while the method's own deprecation warning offered `:json` as the safe alternative to `:ruby`. It now parses with `JSON.parse`, and the new spec fails on main and passes with the change. Filed on 3 September from the corpus run; one maintainer approved the next day and a second approved and merged it on 16 September.
- **[money](https://github.com/RubyMoney/money/pull/1228) - merged.** `Currency.reset!` reloaded the table and cleared the instance and iso_numeric caches but not the memoized key set that `Currency.new` gates on, so after a `register`, a lookup and a `reset!`, the removed id was still admitted and `initialize_data!` raised `NoMethodError` where `find` expects `UnknownCurrency`. `reset!` now clears the key set too, and the new spec fails on main and passes with the change. Filed on 3 September beside #1227; the maintainer merged main into the branch herself after #1227 landed and merged it eight minutes later on 16 September.
- **NVIDIA, [go-nvml](https://github.com/NVIDIA/go-nvml/pull/207) - merged.** `DynamicLibrary.Path()` allocated the `dlinfo(RTLD_DI_ORIGIN)` buffer with `C.CBytes(make([]byte, 0, C.PATH_MAX))`, and `C.CBytes` copies `len(b)` bytes, which is zero, so the first `Path()` call on a library opened by soname wrote the library's directory past a zero-byte allocation; with `libdl.so.2` under a 224-character directory on `LD_LIBRARY_PATH`, the process died with SIGSEGV in the next `dlclose`. The slice now carries its full length, and the new `TestPathFromSoname` crashes on main and passes with the change. Filed on 6 September; a maintainer cleared CI on 15 September, tariq1890 asked on 17 September for the commit to be signed as well as signed off, it was signed and rebased with the patch unchanged, he approved the same day and merged it on 18 September.
- **[chalk](https://github.com/chalk/chalk/pull/687) - fixed upstream.** The maintainer reproduced the finding, then wrote and merged his own fix, shipped in v6.0.0.
- **Cisco, [libsrtp](https://github.com/cisco/libsrtp/issues/822) - fixed upstream.** The committed autotools script appended `--static` to `PKG_CONFIG` unconditionally, so the `AC_SEARCH_LIBS` probes linked every entry of libcrypto's `Libs.private`; on stock Ubuntu that names `-l:libjitterentropy.a`, which no package installs, and `./configure` aborted with `can't find compatible openssl crypto lib` for every OpenSSL build. Filed as an issue rather than a pull request because removing the line narrows the generated `libsrtp3.pc` and the NSS backend takes the same path, which is a maintainer's call, with either patch offered. Another contributor opened [#823](https://github.com/cisco/libsrtp/pull/823) seven hours later citing the issue, deleting the line it named, and the maintainer merged it three days after the report.
- **Apache, [casbin](https://github.com/apache/casbin/issues/1752) - fixed upstream.** `SetEquals`, `SetEqualsInt` and `Set2DEquals` sorted both arguments in place, so comparing live policy rows reordered the model itself and later `Enforce` calls returned the wrong answer. Reported with the reproduction and a patch that copies before sorting; the maintainer closed both the issue and the pull request pointing at his own commit 071dce14, which drops the sort entirely for a counting map and is cheaper than what we sent.
Expand Down
Loading