Packages may be removed from -proposed before being published in -release

[Graham Inggs]

a very small library transition, with 3 dependency levels:
https://ubuntu-archive-team.ubuntu.com/transitions/html/auto-libstreamvbyte.html#good,bad,partial,unknown,!notintesting

* package libstreamvbyte was sync'd, switching the library from libstreamvbyte0 to libstreamvbyte1
https://launchpad.net/ubuntu/+source/libstreamvbyte/0.5.1-2

* then a no-change rebuild of libslow5lib
https://launchpad.net/ubuntu/+source/libslow5lib/1.3.1+dfsg-1build1
to pick up the new library name

* and finally, a no-change rebuild of nanopolish
https://launchpad.net/ubuntu/+source/nanopolish/0.14.0-2build1

libslowlib5 built successfully against the new libstreamvbyte1, but the later build of nanopolish built against libstreamvbyte0, e.g.
https://launchpad.net/ubuntu/+source/nanopolish/0.14.0-2build1/+build/31510649

[Sebastien Bacher (seb128)]

So

https://launchpad.net/ubuntu/+source/libstreamvbyte/0.5.1-2/+build/31504922 was build on 2025-11-19

https://launchpad.net/ubuntu/+source/nanopolish/0.14.0-2build1/+build/31510649 was built on 2025-11-21 around the time the build migrated from resolute-proposed to resolute and was served the old version:

```
Get:124 http://ftpmaster.internal/ubuntu resolute/universe amd64 libstreamvbyte0 amd64 0.4.1-4ubuntu1 [6244 B]
```

The expectation there is that from the point on where 0.5.1-2 was published to resolute-proposed a no point a builder should be getting 0.4.1-4ubuntu1 anymore.

The code of the tool doing the pocket move is https://git.launchpad.net/ubuntu-archive-tools/tree/promote-to-release#n61

        options.archive.copyPackage(
            move=True, source_name=name, version=version,
            from_archive=options.archive,
            from_series=options.series.name, from_pocket="Proposed",
            to_series=options.series.name, to_pocket=options.pocket.title(),
            include_binaries=True, sponsored=options.requestor,
            phased_update_percentage=percentage,
            auto_approve=True)

that's using copyPackage() with move=True and including binaries

[Graham Inggs (ginggs)]

This may have been another occurrence.

automake 1:1.18.1-2 migrated on 2025-11-01, and was removed from -proposed
https://launchpad.net/ubuntu/+source/automake/1:1.18.1-2

and for several hours, many builds failed against the old automake 1:1.17-4ubuntu1 found in -release
https://launchpad.net/ubuntu/+source/automake-1.17/1:1.17-4ubuntu1

fortunately, these builds only needed retrying after the new automake was published and available in -release

[Graham Inggs (ginggs)]

glibc 2.42-0ubuntu1 migrated on 2025-08-21
https://launchpad.net/ubuntu/+source/glibc/2.42-0ubuntu1

from my notes, libc6 2.42-0ubuntu1 was neither in questing nor questing-proposed from around 13:05 UTC until around 17:45 UTC

https://launchpad.net/ubuntu/+source/glibc/+publishinghistory for 2.42-0ubuntu1 shows:

Created on 2025-08-21 11:35:32 UTC
Published on 2025-08-21 13:59:39 UTC

Removal requested on 2025-08-21 11:29:13 UTC
Deleted on 2025-08-21 13:32:14 UTC

[Tushar Gupta (tushar5526)]

We investigated the issue, and it appears to be related to timing. If the copy operation occurs between step A/A2 (publishing) and step B (domination) of the publisher, the domination step deletes the package, but the new package is not published until the next run.

Since the publisher’s run time has been steadily increasing, this issue has become more evident, as it can take up to ~4 hours before the next run starts, during which the archive is in an unexpected state.

* From the logs (libstreamvbyte):

- Publisher run started before 4:52:00
- Copy operation occurred around 04:57:59
- Domination for resolute/proposed started at 04:59:36

* For the glibc incident:

- Publishing starts at 11:21:0
- Copy between 11:35:17 and 11:36:05
- Domination for questing/proposed starts at 11:38:11