-security vs. -updates/-proposed version comparison needs to be removed

[Martin Pitt]

<cprov> == dapper-security ==
<cprov> * linux-source-2.6.15_2.6.15-28.51.dsc REJECTED
<cprov> -> Version older than that in the archive. 2.6.15-28.51 <= 2.6.15-50.61
<cprov> * linux-source-2.6.15_2.6.15-28.51_{ia64, amd64, hppa, i386, powerpc, sparc}.changes REJECTED
<cprov> == edgy-security ==
<cprov>
<cprov> * linux-source-2.6.17_2.6.17.1-11.35.dsc REJECTED:
<cprov> -> Version older than that in the archive. 2.6.17.1-11.35 <= 2.6.17.1-50.50
<cprov> * linux-source-2.6.17_2.6.17.1-11.35_{amd64, amd64, hppa, i386, powerpc, sparc}.changes REJECTED

This means we cannot upload new -security kernels ATM. -updates/-proposed and -security versions cannot not have any constraints wrt. each other. Either might be smaller or greater than the other.

[Martin Pitt (pitti)]

This should be marked critical, since ATM dapper and edgy have uninstallable dist-upgrade due to the held-back kernel packages.

[Celso Providelo (cprov)]

wip

[Colin Watson (cjwatson)]

Edited log from #canonical today:

<cjwatson> cprov: as a matter of higher urgency, I'd rather that the constraint be fixed
<cjwatson> -security should not be required to be > -proposed or > -updates
<cjwatson> (-updates is slightly more questionable, but I think that makes sense; in any event this debate must not hold up fixing -security)
<cprov> cjwatson: nascentuploads is still a wild land, but the fix is feasible.
<cprov> cjwatson: pitti: we already keep BACKPORTS uploads version independent from RELEASE/UPDATES/SECURITY, I think we can do the same for PROPOSED
<pitti> cprov: well, all non-release pockets should be > release, this check makes sense
<cprov> in feel words, after this fix versions in SECURITY/UPDATES version won't be required to be higher than those in BACKPORTS/PROPOSED
<cjwatson> cprov: the only constraints that IMO make obvious sense are: uploads to !RELEASE must be > RELEASE; uploads to UPDATES must be > PROPOSED
<cjwatson> cprov: none of the other constraints make any kind of sense to me
<cprov> cjwatson: it's not that simple, I guess, working with post-release pockets(always higher than RELEASE): uploads to UPDATES/SECURITY > (ALL - BACKPORTS)[-1] AND uploads to BACKPORTS > ALL[-1]
<cprov> cjwatson: I will just exclude PROPOSED version checks for UPDATES/SECURITY
<cprov> I mean, exclude *also* PROPOSED ..
<cjwatson> cprov: I see no justification for bothering with the extra constraints you mention there
<cprov> cjwatson: it prevents uploads to updates < than what we have in security and vice-versa.
<cprov> cjwatson: but I agree it could be caught in pkg-review time
<cjwatson> cprov: I don't see why we should bother; it just causes panics like this
<cjwatson> and doesn't really buy us anything important
<cprov> cjwatson: it'd if was correct, no ?
<cjwatson> cprov: there are so many corner cases in how particular packages are handled that I strongly believe the versioning constraints enforced by the archive software on pockets should err on the
                  side of being relaxed
<cjwatson> cprov: for a start, your "vice-versa" above (preventing uploads to security < updates) is definitely wrong
<cjwatson> cprov: given that the current constraints are so wrong, I think they should be stripped back to the bare essentials and that every additional constraint should have clear thought and
                  justification
<cprov> cjwatson: yes, the reality proves your right.
<cjwatson> nothing scary is going to happen if those constraints are removed
<cprov> cjwatson: you made you point, could you comment it on 83976 ?
<cjwatson> but we're better off defending against that sort of thing by post-facto checking (i.e. send warning mails and stuff)
<cjwatson> cprov: sure
<cprov> cjwatson: thanks

To elaborate slightly on the -security vs. -updates check, the thing that the constraint is presumably intended to defend against is an upload to -security never making it out to users because there's already a newer version in -updates (which we assume does not contain the security fix). However, note that this can just as easily happen the other way round - an upload to -updates could be made with a higher version number than that ...

[Christian Reis (kiko)]

I've looked over Celso's patch twice now and what makes me most uneasy about it is that we replace comments which look authoritative with ugly XXXs. One of the comments we are removing says:

- # When looking for published sources, to verify that an uploaded
- # file has a usable version number, we must consider the special
- # case of the backports pocket.
- # Across the release, security and uploads pockets, we have one
- # sequence of versions, and any new upload must have a higher
- # version than the currently highest version across these pockets.
- # Backports has its own version sequence, all higher than the
- # highest we'll ever see in other pockets. So, it's not a problem
- # that the upload is a lower version than can be found in backports,
- # unless the upload is going to backports.
- # See bug 34089.

What I want to know is whether this comment is completely off-base.

[Celso Providelo (cprov)]

Ok, I've updated the replacement of the comment mentioned to:

{{{
        # Only lookup uploads ancestries in RELEASE pocket.
        # Upload ancestries found here will guide the auto-override
        # procedure and the version consistency check:
        #
        # * uploaded_version > RELEASE_version
        #
        # which is the *only right* check we can do automatically.
        # Post-release history and proposed content may diverge and can't
        # be properly automatically overridden.
        #
        # We are relaxing version constraints when processing uploads since
        # there are many corner cases when checking version consistency
        # against post-release pockets, like:
        #
        # * SECURITY/UPDATES can be lower than PROPOSED/BACKPORTS
        # * UPDATES can be lower than SECURITY
        # * ...
        #
        # And they really depends more on the package contents than the
        # version number itself.
        # Version inconsistencies will (should) be identified during the
        # mandatory review in queue, anyway.
        # See bug #83976
}}}

It's not a small dirty XXX anymore.

I hope this comment can clarify a bit more what is the patch goal and help you guys to decide if it is going in the right direction or not.

Thanks for pointing it.

[Paolo Sammicheli (xdatap1)]

Added two duplicated bugs

[Celso Providelo (cprov)]

Ok, demoted to High priority, since we have *cowboyed* the required uploads into security.
Nonetheless, we have to fix the code before this issues shows up again (100 % sure it will happen again in the next kernel security upload, so we need to be fast)

[towsonu2003 (towsonu2003)]

http://ubuntuforums.org/showthread.php?t=356408 (and a post that links to this) suggests that problems related to this are experienced by many ubuntu users.

[Sarah Kowalik (hobbsee-deactivatedaccount)]

Rejecting ubuntu task, as this is not a bug in ubuntu, it's a bug in soyuz.

Please only the soyuz guys edit this?

[Celso Providelo (cprov)]

RF 3811 (new tree) cherrypicked in drescher.

Version checker looks for ancestry inside the post-release pocket and/or fallback to RELEASE only.
Conflicts across pockets should be not identified automatically. We rely on reviewers.