Skip to content

releaseWizard: summarize dependency changes from solr/licenses - #4966

Open
dsmiley wants to merge 7 commits into
apache:mainfrom
dsmiley:remove-addDepsToChanges
Open

dsmiley wants to merge 7 commits into
apache:mainfrom
dsmiley:remove-addDepsToChanges

Conversation

@dsmiley

@dsmiley dsmiley commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

🤖 Replaces addDepsToChanges.py, which bulk-converted solrbot commits into changelog entries (duplicating the entries solrbot PRs already create), with dev-tools/scripts/dependencyChanges.py. Context: dev@ thread "changelog and dependencies".

The new script diffs the jar checksum files in solr/licenses/ between the previous release tag and HEAD (git only; no build needed):

  • Jars that moved between the same two versions are listed together (e.g. jetty-* (23 jars) 12.0.27 → 12.1.12), since they're typically one project.
  • Only jars with a LICENSE file are included (those are required only for shipped jars), minus test artifacts. Removals from a release predating SOLR-15465 may include jars we didn't actually ship.
  • By default it prints Added / Upgraded / Removed lines (semicolon-separated); --write writes one dependency_update entry per category to changelog/unreleased/dependency-changes-{1-added,2-upgraded,3-removed}.yml (numbered to order them), attributed to "various contributors". Re-running overwrites them.
  • It's a stopgap until Solr publishes an SBOM per release.

The release wizard step now runs it with --write. Try it: python3 dev-tools/scripts/dependencyChanges.py (on main, compares to 10.0.0: 11 added, 83 upgraded, 49 removed).

Also removes the workflows (and script) that generated a changelog entry for each solrbot (Renovate) PR; they're obsolete. Removing the existing solrbot entries is separate: #4944.

🤖 Generated with Claude Code

Each solrbot PR now creates its own changelog entry via the renovate-changelog workflows, so the bulk step that built entries from git log only produced duplicates. Remove the step and the addDepsToChanges.py script it called.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace addDepsToChanges.py (which bulk-converted solrbot commits into changelog entries, duplicating the per-PR entries) with dependencyChanges.py. It diffs the jar checksum files in solr/licenses/ between the previous release tag and HEAD, groups jars that moved between the same versions, and prints a summary or writes one dependency_update changelog entry. The wizard step now runs it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dsmiley dsmiley changed the title releaseWizard: remove solrbot addDepsToChanges step releaseWizard: summarize dependency changes from solr/licenses Sep 30, 2026
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Sep 30, 2026
Obsolete now that dependencyChanges.py summarizes dependency changes at release time. Removes the renovate-changelog-prepare/push workflows and generate-renovate-changelog.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dsmiley

dsmiley commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

Dependency changes from releases/solr/9.10.1 to apache/branch_9_11:
apache-client, arns, auth, aws-core, aws-query-protocol, aws-xml-protocol, checksums, checksums-spi, crt-core, endpoints-spi, http-auth, http-auth-aws, http-auth-aws-eventstream, http-auth-spi, http-client-spi, identity-spi, json-utils, metrics-spi, profiles, protocol-core, regions, retries, retries-spi, s3, sdk-core, sts, third-party-jackson-core, url-connection-client, utils 2.31.77 → 2.53.1
apache5-client, utils-lite 2.53.1 (new)
api-common 2.33.0 → 2.64.0
auto-value-annotations 1.10.4 → 1.11.0
avatica-core, avatica-metrics 1.25.0 → 1.28.0
bcpkix-jdk18on, bcprov-jdk18on, bcutil-jdk18on 1.82 → 1.85
biz.aQute.bnd.annotation 7.1.0 → 7.3.0
byte-buddy, byte-buddy-agent 1.17.7 → 1.18.11
caffeine 3.1.8 → 3.2.4
calcite-core, calcite-linq4j 1.37.0 → 1.42.0
checker-qual 3.44.0 → 3.49.0
commons-beanutils 1.9.4 → 1.11.0
commons-cli 1.10.0 → 1.11.0
commons-codec 1.19.0 → 1.22.1
commons-collections4 4.5.0 → 4.6.0
commons-compiler, janino, jersey-client, jersey-common, jersey-entity-filtering, jersey-hk2, jersey-media-json-jackson, jersey-server 3.1.11 → 3.1.12
commons-configuration2 2.12.0 → 2.15.1
commons-exec 1.5.0 → 1.6.0
commons-io 2.20.0 → 2.22.0
commons-lang3 3.19.0 → 3.20.0
commons-text 1.13.1 → 1.15.0
commons-validator 1.7 → 1.10.1
curvesapi 1.07 → 1.08
disruptor 3.4.4 → 4.0.0
docker-java-api, docker-java-transport, docker-java-transport-zerodep 3.4.0 → 3.7.1
failureaccess 1.0.2 → 1.0.3
gapic-google-cloud-storage-v2, grpc-google-cloud-storage-v2, proto-google-cloud-storage-v2 2.40.1-alpha → 2.69.0
gax, gax-grpc, gax-httpjson 2.50.0 → 2.81.0
google-* (5 jars) 1.44.2 → 2.1.0
google-api-client 2.6.0 → 2.7.2
google-api-services-storage-v1-rev20260204 2.0.0 (new)
google-auth-library-credentials, google-auth-library-oauth2-http 1.23.0 → 1.48.0
google-cloud-core, google-cloud-core-grpc, google-cloud-core-http 2.40.0 → 2.71.0
google-cloud-nio 0.127.20 → 0.133.0
google-cloud-storage 2.40.1 → 2.69.0
google-oauth-client 1.36.0 → 1.39.0
grpc-* (17 jars) 1.65.1 → 1.82.1
grpc-opentelemetry 1.82.1 (new)
gson 2.11.0 → 2.13.2
guava 33.1.0-jre → 33.6.0-jre
hadoop-* (8 jars) 3.4.1 → 3.4.3
hadoop-shaded-guava 1.4.0 → 1.5.0
httpclient5 5.2.1 → 5.6.4
httpcore5 5.2.3 → 5.4.3
httpcore5-h2 5.2 → 5.4.3
j2objc-annotations 3.0.0 → 3.1
jackson-* (12 jars) 2.18.0 → 2.22.2
jackson-annotations 2.18.0 → 2.22
jackson-jq 0.0.13 → 1.6.2
jcl-over-slf4j, jul-to-slf4j, slf4j-api 2.0.17 → 2.0.18
jctools-core 4.0.5 → 4.0.6
jersey-container-jetty-http 2.39.1 → 2.48
jetty-servlet-api 4.0.6 → 4.0.9
jna 5.13.0 → 5.19.1
joou-java-6 0.9.5 (new)
json-path 2.9.0 → 2.10.0
kafka-clients, kafka-group-coordinator, kafka-group-coordinator-api, kafka-metadata, kafka-raft, kafka-server, kafka-server-common, kafka-storage, kafka-storage-api, kafka-streams, kafka-tools-api, kafka-transaction-coordinator, kafka_2.13 3.9.1 → 3.9.2
kerb-admin, kerb-client, kerb-common, kerb-core, kerb-crypto, kerb-identity, kerb-server, kerb-simplekdc, kerb-util, kerby-asn1, kerby-config, kerby-pkix, kerby-util 2.1.0 → 2.1.2
kotlin-reflect 1.8.22 → 2.1.21
kotlin-stdlib 2.2.0 → 2.2.21
log4j-* (6 jars) 2.25.3 → 2.26.1
lz4-java 1.8.0 → 1.10.1
metrics-* (9 jars) 4.2.26 → 4.2.39
mockito-core, mockito-subclass 5.19.0 → 5.23.0
mockwebserver 4.11.0 → 5.4.0
mockwebserver3 5.4.0 (new)
netty-* (14 jars) 4.2.6.Final → 4.2.15.Final
netty-tcnative-boringssl-static, netty-tcnative-classes 2.0.73.Final → 2.0.77.Final
okhttp-jvm 5.1.0 → 5.4.0
okhttp-sse 4.12.0 → 5.4.0
okio-jvm 3.15.0 → 3.17.0
opennlp-tools 1.9.4 → 1.9.5
opentelemetry-* (15 jars) 1.53.0 → 1.65.0
opentelemetry-semconv 1.29.0-alpha (new)
osgi-resource-locator 1.0.3 → 1.0.4
paranamer 2.8 → 2.8.3
poi, poi-ooxml, poi-ooxml-lite 5.2.2 → 5.5.1
prometheus-metrics-config, prometheus-metrics-exposition-textformats 1.8.0 (new)
prometheus-metrics-model 1.1.0 → 1.8.0
proto-google-common-protos 2.41.0 → 2.72.0
proto-google-iam-v1 1.36.0 → 1.67.0
protobuf-java, protobuf-java-util 3.25.8 → 4.35.1
randomizedtesting-runner 2.8.3 → 2.9.1
scala-library 2.13.15 → 2.13.18
SparseBitSet 1.2 → 1.3
stax2-api 4.2.2 → 4.3.0
swagger-annotations-jakarta 2.2.22 → 2.2.53
testcontainers 1.20.4 → 2.0.5
threetenbp 1.6.9 → 1.7.0
tika-core 1.28.5 → 3.3.2
tika-langdetect-tika 3.3.2 (new)
value-annotations 2.11.3 → 2.12.2
woodstox-core 7.0.0 → 7.2.0
xmlbeans 5.0.3 → 5.3.0
zookeeper, zookeeper-jute 3.9.4 → 3.9.5

dsmiley and others added 3 commits September 30, 2026 01:17
Print one line per category, with changes separated by semicolons, and with --write produce one changelog entry per category. Removals are included again, limited to jars that had a LICENSE file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop the --author option (defaulting to git user.name); the entries summarize many people's changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, removed

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dsmiley

dsmiley commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

🤖 A thought for a follow-up, based on generating the 9.11 dependency entries by hand. Comparing the jars actually inside the binary distribution is more accurate than comparing solr/licenses/*.sha1 plus the LICENSE-file filter.

For 9.10.1 → 9.11, I compared the output against the jars in the official 9.10.1 tarball and in a fresh 9.11 build (gradlew :solr:packaging:devFull):

  • 32 jars appeared that we don't ship, e.g. mockito, testcontainers, docker-java, the hadoop minicluster/minikdc/kerby jars, byte-buddy and bc*-jdk18on. They have .sha1 files, and on 9.x also LICENSE files.
  • 10 real removals were missed, e.g. bc*-jdk15on, jaxb, jna and joda-time. They left the distribution (likely with the in-process Tika backend) but still have .sha1 files because tests use them.

Net effect: upgraded went from 87 to 74 entries and removed from 43 to 48. That's what I committed for 9.11.

Proposal: take the jar list from each distribution instead.

  • Old side: the previous release tarball from archive.apache.org (or Docker container). Listing the tarball's contents is enough; nothing has to be unpacked.
  • New side: the jars from :solr:packaging:devFull, or the RC tarball when one exists.
  • Parsing and grouping stay as they are.
  • Cost: a build or a download of roughly 300 MB, instead of pure git. It also makes the LICENSE-file heuristic and the test-artifact name filter unnecessary.

This could go in this PR or a follow-up. The current sha1-based version is still a big improvement over the old per-PR entries.

Note: looking at JARs is less than perfect as well, as it doesn't include non-JAR dependencies such as jQuery.

Finally... what we really want is likely a Cyclone DX SBOM to do comparisons. CC @epugh

🤖 Generated with Claude Code and reviewed/edited by Smiley

@epugh

epugh commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

Finally... what we really want is likely a Cyclone DX SBOM to do comparisons. CC @epugh

Agreed. The WIP is this file: https://github.com/apache/solr-site/blob/main/plugins/vex/solr-dependency-versions.json, it's only listing mappings for jars that have a vex file, so it doesn't cover everything. What I learned was that you have to look at the dist, not the source, or get you get a lot of noise.

Do you think starting to publish SBOM now, even though it won't exist for previous Solrs would be useful? @ppkarwasz has #4690 which adds SBOM. Maybe worth a discussion at Glasgow?

@dsmiley

dsmiley commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

I used this to augment the changelog on branch_9_11 last night (pushed). Rendered as HTML, it looks like so:
image

@dsmiley

dsmiley commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

Again, the sha1 file approach should become stable/reliable on the 10x line, and it's efficient to do the processing that way instead of comparing JARs in multiple distributions. Nonetheless I could switch this to the latter; Claude did so locally for my 9.11 analysis.

(SBOM): it's only listing mappings for jars that have a vex file

Sounds like a non-starter?

@ppkarwasz

ppkarwasz commented Sep 30, 2026 •

Copy link
Copy Markdown
Member

(SBOM): it's only listing mappings for jars that have a vex file

Sounds like a non-starter?

solr-dependency-versions.json contains only the dependencies with at least one VEX entry. The SBOM in #4690 contains all the dependencies, even the vendored JS dependencies and those compiled into the new UI.

The licenses are also there, although in “raw” form (i.e. as declared in the POM) and a script would be necessary to add a “concluded” license. For example I imagine that Jetty inside Solr is relicensed under Apache-2.0, not EPL-2.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation no-changelog scripts

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants