gnupg2 autopkgtest 'simple-tests' should be included in x/b/c

[Dan Streetman]

[impact]

b/c currently only have gpgv-win32 test, which is limited in what it tests and what archs it runs on. additionally, it always fails (see bug 1825186).

x currently has no tests at all for gnupg or gnupg2.

[test case]

run autopkgtests for gnupg2 on b/c.

[regession potential]

adding a testcase may result in the testcase incorrectly failing in the future.

[other info]

this test case is cherry-picked from gnupg2 in disco.

the test case required slight modification for gnupg v1, as 'Key-Type: default' only works with v2. Note that Xenial is the last release that carries gnupg v1; Bionic and later carry only gnupg v2.

[Robie Basak (racb)]

Thank you for working on improving our test suites.

This upload is only to fix dep8 tests, right? If so, please could you assess this against the concerns raised in the thread here: https://lists.ubuntu.com/archives/ubuntu-release/2018-March/004352.html ? It seems to me that this upload falls on the wrong side of that discussion and on balance it's not worth accepting it, but perhaps there's some other aspect or opinion I'm missing?

[Dan Streetman (ddstreet)]

> please could you assess this against the concerns raised in the thread here

Please correct me if I'm mischaracterizing your argument, but basically you are saying that this update will cause all users to perform a deb download that has no impact on the code they are running.

I'm afraid I don't see the merit of your argument at all, if it is in fact simply about additional end-user downloads. Unless I'm misunderstanding it, that has no impact at all to end-users besides download bandwidth and time; end-users will not have to spend any additional time actually doing anything because of this update.

Compare that to what sru uploaders (and sru approvers) must do each time this package's autopkgtests fail - it wastes extremely valuable time looking at autopkgtest failures, figuring out why they failed, finding this bug (if they don't already know about it) and justifying why the autopkgtest regression should be ignored.

Alternately, this test could be hinted to completely ignore it for all reverse-depends autopkgtest runs, but IMO that's even worse - you're intentionally preventing the reverse-depends autopkgtest run from actually doing its job, of making sure that other package changes don't cause regressions in this package.

Can you explain in more detail why you believe it's better to leave broken autopkgtests broken?

[Robie Basak (racb)]

I was asking about how the points raised in the mailing list thread
specifically apply to this particular case.

It sounds like you object more in principle to the general points
raised. In that case I suggest you participate in the thread in
ubuntu-release@ instead, so we don't silo that discussion here. I will
be happy to respond to you there.

[Robie Basak (racb)]

On Wed, May 01, 2019 at 02:10:52PM -0000, Dan Streetman wrote:
> Can you explain in more detail why you believe it's better to leave
> broken autopkgtests broken?

Please don't mischaracterise my position like this. I find it upsetting,
disrespectful and unconstructive. For an example of how this your
characterisation of my position is inaccurate, see my post at
https://lists.ubuntu.com/archives/ubuntu-release/2018-March/004368.html

I hope that you participate in the mailing list discussion but do it in
a more constructive tone.

[Dan Streetman (ddstreet)]

> Please don't mischaracterise my position like this

That's why I specifically said:

"Please correct me if I'm mischaracterizing your argument"

I read your email, and that was truly the position that I took from it. If I'm misunderstood it, as I said, please correct me.

> I find it upsetting, disrespectful and unconstructive

I meant no disrespect, which is why I specifically asked you to correct me if I misunderstood you - apparently I have.

I'll read the entire mailing list thread to see where I misunderstood your position.

> It sounds like you object more in principle to the general points raised

For this bug specifically, the sru absolutely is just a fix to the autopkgtests, if that is what you're asking. Please let me know if you are asking for more information than that from me.

[Dimitri John Ledkov (xnox)]

Adding new autopkgtests in devel series - good.

Adding new autopkgtests in stable series - not good.

Fixing regressions that got introduced during the lifetime of a stable series - good.

The only times when we introduced new autopkgtests in stable series, is when we upload fixes for regressions / SRU bugs and attempt at preventing them from regressing again in the stable series.

This development does not appear to be that. Uploading new smoke-tests is out of the scope of the SRU policy. What prompted this bug report?

I feel like rejecting the series tasks of this bug report.

[Dan Streetman (ddstreet)]

> Adding new autopkgtests in stable series - not good.

Uh, why? Personal opinion?

> I feel like rejecting the series tasks of this bug report.

I'm going to undo this, and suggest that if this is, in fact, SRU policy, the actual SRU policy needs to be updated to indicate it.

If ~ubuntu-sru decides to actually reject my gnupg/gnupg2 uploads, then please feel free to wontfix this at that time. Until then please leave it open.

[Dimitri John Ledkov (xnox)]

SRU policy, Section 2 "When", has three major categories listing in bullet points cases that are eligible for SRU.

Which one is this SRU submitted under?

https://wiki.ubuntu.com/StableReleaseUpdates#When

[Steve Langasek (vorlon)]

"Other safe cases" mentions:

FTBFS(Fails To Build From Source) can also be considered. Please note that in main the release process ensures that there are no binaries which are not built from a current source. Usually those bugs should only be SRUed in conjunction with another bug fix.

When this was written, autopkgtests did not exist. I think it would be reasonable to extend the policy to include autopkgtests by analogy.

However, as this says, "usually those bugs should only be SRUed in conjunction with another bug fix". When there is no history of regressions being introduced in this package as a result of NOT having autopkgtests in place, I don't think we should do an SRU only to add a working test.

I won't mark these bugs wontfix, but I am rejecting the upload in the meantime until there is another change needed to the package that would warrant an SRU.

[Steve Langasek (vorlon)]

An upload of gnupg2 to bionic-proposed has been rejected from the upload queue for the following reason: "fixing tests only does not fit the SRU policy".

[Steve Langasek (vorlon)]

An upload of gnupg2 to cosmic-proposed has been rejected from the upload queue for the following reason: "fixing tests only does not fit the SRU policy".

[Steve Langasek (vorlon)]

An upload of gnupg2 to xenial-proposed has been rejected from the upload queue for the following reason: "fixing tests only does not fit the SRU policy".

[Steve Langasek (vorlon)]

An upload of gnupg to xenial-proposed has been rejected from the upload queue for the following reason: "fixing tests only does not fit the SRU policy".