Manual build requests of charm/snap recipes never clear is_stale

[Loïc Gomez]

Setting automatic build on branch change did not take last merged commit into account.

https://launchpad.net/~canonical-is-sre/canonical-is-hr-automation/+charm/canonical-is-hr-automation-charm

I had just pushed a new version of requirements.txt and charmcraft.yaml files in this merge:
https://git.launchpad.net/~canonical-is/canonical-is-hr-automation/+git/hr-automation-charm/commit/?id=d099e640a2da8ecb140c74e460ac23ed126adbc0

This should have triggered a build with charmhelpers dependency and only on amd64 but instead I got a charm without the required lib and built on all architectures:
Successfully built 1 hour ago buildlog (16.8 KiB) arm64
Successfully built 1 hour ago buildlog (16.7 KiB) armhf
Successfully built 1 hour ago buildlog (16.7 KiB) s390x
Successfully built 1 hour ago buildlog (16.8 KiB) ppc64el
Successfully built 1 hour ago buildlog (16.8 KiB) amd64
https://launchpad.net/~canonical-is-sre/canonical-is-hr-automation/+charm/canonical-is-hr-automation-charm/+build/7251/+files/buildlog_charm_ubuntu_jammy_amd64_canonical-is-hr-automation-charm_BUILDING.txt.gz

Realized that after scratching my head about why the lib was still missing from my newly built charm. Manually requesting a build after a while did trigger a build only on amd64 with the proper lib pulled into the charm.

I don't know how it came to this, but it seems latest commit was not taken into account when building on branch change.

[Loïc Gomez (kotodama)]

Happened again, twice in a row on that same project.

[Colin Watson (cjwatson)]

I think I can see why you were confused, but your description of the timeline isn't quite right. This is the sequence of events:

  2023-04-03 07:46:45: commit 489497f56193dea47dbc738d3ba0c37483fe297c pushed to ~canonical-is/canonical-is-hr-automation/+git/hr-automation-charm
  2023-04-03 07:49:43: you manually requested builds of ~canonical-is-sre/canonical-is-hr-automation/+charm/canonical-is-hr-automation-charm; this had the effect of requesting builds for amd64, armhf, arm64, ppc64el, and s390x, though you cancelled all but amd64 shortly afterwards
  2023-04-04 00:15:10: our request-daily-builds job noticed that the charm recipe was "stale" (i.e. had been pushed to since the last automatically-requested set of builds), and requested new builds, including 7251 above
  2023-04-04 00:15:15: build 7251 started
  2023-04-04 00:23:27: d099e640a2da8ecb140c74e460ac23ed126adbc0 pushed to ~canonical-is/canonical-is-hr-automation/+git/hr-automation-charm

You can see from this that build 7251 couldn't possibly have used the commit you were referring to, because it was started _before_ that commit was pushed. The next occurrence is similar - you didn't wait for the automatic build request to happen, but instead requested builds manually, and then a little later the automatic build request job came round and repeated the job. The rest is all just timeline confusion due to builds happening to be automatically requested shortly before a push.

So, as far as I can tell, the bug as described isn't valid. However, a lot of this confusion arose because if you request builds manually in a way that exactly matches what an automatic build request would do, the "stale" flag on the recipe nevertheless isn't cleared. This tends to cause this sort of confusion, and it also wastes resources. I think a manual build request of a charm recipe (or, similarly, a snap recipe) should clear the "stale" flag if and only if the parameters of the manual build request exactly match what an automatic build request would do.

[Loïc Gomez (kotodama)]

Thanks for elaborating on this, I understand what happened now.

I did not realize the `Automatically build when branch changes` option was limited to one per hour. I realize this is described in the build schedule on the recipe page, but then it might be useful to add it also to the recipe configuration page to avoid misunderstandings. Indeed, current description had me thinking it is on (every single) change:
`Whether this charm recipe is built automatically when its branch changes.`

It must have been a coincidence that I checked on the builds not too long after a merged commit, expecting the last build to match the merge :)

I guess this bug is actually mostly a wording issue then.
Thank you!