fix: prevent intermittent digest-mismatch errors during archive updates by gruyaume · Pull Request #296 · canonical/chisel

gruyaume · 2026-05-15T15:45:12Z

Chisel intermittently fails with expected digest X, got Y when Ubuntu's archive is mid-publication and InRelease is temporarily inconsistent with Packages.gz at the named path (a window we observed lasting at least 39 minutes against the same publication in real CI runs for Ella Core). Fetch the package index by content hash using apt's Acquire-By-Hash , which Ubuntu archives have advertised since 16.04 (2016), so the URL itself encodes the expected bytes; falls back to the named path when an archive doesn't advertise the feature or a specific hash has been garbage-collected.

Fixes #295

Reference

https://wiki.debian.org/DebianRepository/Format#Acquire-By-Hash

Have you signed the CLA?

Signed-off-by: Guillaume Belanger <guillaume.belanger27@gmail.com>

upils · 2026-06-01T12:26:12Z

Hey @gruyaume, thanks for your PR! Is this ready for review?

gruyaume · 2026-06-01T13:10:17Z

Hey @gruyaume, thanks for your PR! Is this ready for review?

Yes.

If you find the code changes mostly up to you standards, I am happy to address minor issues. However if there are major issues and you have a completely different idea of the solution, please go ahead and implement it your way. I only want the issue fixed as soon as possible, I don't care about getting the attribution.

upils · 2026-06-01T13:59:04Z

Yes.

If you find the code changes mostly up to you standards, I am happy to address minor issues. However if there are major issues and you have a completely different idea of the solution, please go ahead and implement it your way. I only want the issue fixed as soon as possible, I don't care about getting the attribution.

Thanks! I explored another approach in parallel but I ended up circling back to yours so there is no point just having another PR for the sake of it. I will review yours then, because I expect only minor changes will be requested.

Signed-off-by: Guillaume Belanger <guillaume.belanger27@gmail.com>

gruyaume · 2026-06-09T11:53:12Z

@upils , @lczyk can you or another admin please review this PR?

upils

Thanks @gruyaume for the PR. Overall I think it looks great and the approach seems right to me. I have a few suggestions that should be very quick to fix.

Signed-off-by: Guillaume Belanger <guillaume.belanger27@gmail.com>

upils

Thanks @gruyaume! Once the CI is done I will move it to the next review stage.

niemeyer

Thanks for the feature @gruyaume. There are a couple of minor details about the organization only. Let us know how you'd prefer to evolve this... might take it over from here as well if you want. /cc @upils

niemeyer · 2026-06-15T19:33:19Z


 	body := resp.Body
-	if strings.HasSuffix(path, ".gz") {
+	if flags&fetchGzip != 0 {


This is a behavioral change. Has anyone double checked that we don't have any other cases that were handled before and are not handled now?

Yes I checked. We only had one call site with a path ending with .gz, in fetchIndex, which is reworked in this PR to use the fetchGzip flag.

niemeyer · 2026-06-15T19:37:33Z

+	// are content-addressed and so are immune to the inconsistent view of
+	// InRelease vs Packages.gz that mirrors and CDNs can serve while a
+	// publication is propagating. See https://wiki.ubuntu.com/AptByHash.
+	gzPath := packagesPath + ".gz"


This whole function needs to have its variable names reviewed and polished, as it's becoming unreadable. We have digests, digest, gzPath, gzDigest, byHashPath, etc. These things all talk about their types and suffixes, and not about what they are. And they are not the same thing.

/cc @upils

niemeyer · 2026-06-15T19:44:12Z

+func (s *httpSuite) sawByHashRequest() bool {
+	for _, req := range s.requests {
+		if strings.Contains(req.URL.Path, "/by-hash/SHA256/") {
+			return true


If the archive only has the right file in the right place, the only way for us to get the right content is to use that file. I might be missing something, but it looks like at the moment all we are looking at is a request being made inside a directory, possibly in the wrong place considering we fallback.

niemeyer · 2026-06-15T19:51:49Z

+		}
+		if r.AcquireByHash && itemPath != r.Path() && !skipByHash[itemPath] {
+			byHashPath := path.Join(prefix, "dists", r.Suite, path.Dir(itemPath), "by-hash", "SHA256", makeSha256(itemContent))
+			content[byHashPath] = itemContent


This used to be a very clean and simple function, and it'd be nice to keep it that way. From these options, having a ByHash bool (not "Acquire", that's on the fetch side) seems in line with the type semantics. The other options look like custom hacks that require reading both the test and the implementation to be made sense of, so we need to think a bit about how to represent these while keeping that clean and simple aspect.

/cc @upils

This function is now simpler. The remaining additional complexity is due to adding the byHash path.

Tests are now responsible for tweaking the responses to simulate a bad mirror.

niemeyer

👆

gruyaume · 2026-06-15T20:04:12Z

Thanks for the feature @gruyaume. There are a couple of minor details about the organization only. Let us know how you'd prefer to evolve this... might take it over from here as well if you want. /cc @upils

Yes feel free to take it over from here.

Thank you.

Signed-off-by: Paul Mars <paul.mars@canonical.com>

upils · 2026-06-17T08:38:55Z

@gruyaume would you mind giving me permissions on your branch in your fork so I can push commits to it? I do not have the "maintainer" role on the main repo so I do not have this permissions by default. I could create another branch in my fork but I would like to keep the history of this PR.

gruyaume · 2026-06-17T11:31:12Z

@gruyaume would you mind giving me permissions on your branch in your fork so I can push commits to it? I do not have the "maintainer" role on the main repo so I do not have this permissions by default. I could create another branch in my fork but I would like to keep the history of this PR.

Sure, I just added you.

upils · 2026-06-17T11:51:31Z


 func (index *ubuntuIndex) fetchIndex() error {
-	digests := index.release.Get("SHA256")
+	releaseDigests := index.release.Get("SHA256")


[Note to reviewer]: I did some renaming to hopefully clarify what these variables are.

upils · 2026-06-17T11:54:31Z

+	base           string
+	request        *http.Request
+	requests       []*http.Request
+	requestResults []requestResult


[Note to reviewer]: This line is the only new one in this struct. It enables recording submitted requests and associated status. The additional tests were then reworked to properly check when the "by-hash" URL was used and what was the result.

I expect this additional capability will be useful in the future to test other fallback mechanisms.

Signed-off-by: Paul Mars <paul.mars@canonical.com>

fix: prevent intermittent digest-mismatch errors during archive updates

31bcd80

Signed-off-by: Guillaume Belanger <guillaume.belanger27@gmail.com>

lczyk added the Bug An undesired feature ;-) label May 29, 2026

lczyk mentioned this pull request May 29, 2026

fix: avoid digest mismatches during archive publication #300

Closed

1 task

gruyaume marked this pull request as ready for review June 1, 2026 13:10

gruyaume added 2 commits June 2, 2026 12:36

Merge branch 'canonical:main' into dev-fixhash

4cb9d90

chore: fix lint error

0c9b69b

Signed-off-by: Guillaume Belanger <guillaume.belanger27@gmail.com>

upils suggested changes Jun 15, 2026

View reviewed changes

Comment thread internal/archive/testarchive/testarchive.go Outdated

Comment thread internal/archive/testarchive/testarchive.go Outdated

Comment thread internal/archive/archive_test.go Outdated

fix: replace gzipBytes with MakeGzip

4837940

Signed-off-by: Guillaume Belanger <guillaume.belanger27@gmail.com>

gruyaume requested a review from upils June 15, 2026 11:47

upils approved these changes Jun 15, 2026

View reviewed changes

upils added the Priority Look at me first label Jun 15, 2026

niemeyer reviewed Jun 15, 2026

View reviewed changes

niemeyer requested changes Jun 15, 2026

View reviewed changes

upils added 2 commits June 16, 2026 11:40

style: polish names in fetchIndex

5f1bc3b

Signed-off-by: Paul Mars <paul.mars@canonical.com>

tests: rework approach to test by-hash URL usage

aa82b97

Signed-off-by: Paul Mars <paul.mars@canonical.com>

upils reviewed Jun 17, 2026

View reviewed changes

test: simplify Render

f80cc1b

Signed-off-by: Paul Mars <paul.mars@canonical.com>

Conversation

gruyaume commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reference

Uh oh!

upils commented Jun 1, 2026

Uh oh!

gruyaume commented Jun 1, 2026

Uh oh!

upils commented Jun 1, 2026

Uh oh!

gruyaume commented Jun 9, 2026

Uh oh!

upils left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

upils left a comment

Choose a reason for hiding this comment

Uh oh!

niemeyer left a comment

Choose a reason for hiding this comment

Uh oh!

niemeyer Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

upils Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

niemeyer Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

niemeyer Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

niemeyer Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

upils Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

niemeyer left a comment

Choose a reason for hiding this comment

Uh oh!

gruyaume commented Jun 15, 2026

Uh oh!

upils commented Jun 17, 2026

Uh oh!

gruyaume commented Jun 17, 2026

Uh oh!

upils Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

upils Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gruyaume commented May 15, 2026 •

edited

Loading

upils Jun 16, 2026 •

edited

Loading

niemeyer Jun 15, 2026 •

edited

Loading

niemeyer Jun 15, 2026 •

edited

Loading

upils Jun 17, 2026 •

edited

Loading