[SRU] Backport letsencrypt from bionic

[Peter Eckersley]

[Impact]

Certbot (formerly called Let's Encrypt, as released in Xenial) will stop working on 13 March 2019 when TLS-SNI-01 validation is turned off by the primary Let's Encrypt CA. This will make the package effectively useless for just about all users.

[Development Fix]

Newer validation options are present in the packages in Bionic onwards, including Disco.

[Stable Fix]

For Xenial, we are backporting the version of Certbot in Bionic.

Note that this update includes two important functional changes:

1) Automatic renewal is being enabled.

2) Log rotation is switching to being handled by logrotate.

See the discussion in this bug for details.

Since the upstream project has been renamed from "Let's Encrypt" to "Certbot" to better differentiate between the tooling and the CA, the /usr/bin/certbot command will become available. However, a compatibility symlink is provided under the old name /usr/bin/letsencrypt.

[Test Case]

Upstream have an extensive test suite and are participating in this SRU to help us validate and land it.

[Test Plan]

See https://wiki.ubuntu.com/StableReleaseUpdates/Certbot#SRU_Verification_Process and https://wiki.ubuntu.com/StableReleaseUpdates/Certbot/TestScript

In addition, we will test the upgrade path from the Xenial release pocket to proposed explicitly.

[Regression Potential]

The Certbot team has viewed breakage of existing workflows (especially ones that may be automated) as a serious issue, has strived to avoid them, and has treated workflow changes as regressions where it has occurred.

We have the following test suites in place for Certbot:

* Nosetest unit tests with coverage for each module between 97% and 100%; *test.py in the relevant tree.
* Integration tests that run Certbot against the current copy of Let's Encrypt's serverside boulder codebase. These require docker and are a little more involved to run. See tests/boulder_integration.sh for instructions.
* "Compatibility tests" that run the Apache and Nginx plugins against corpora of configuration files for those webservers; these live in certbot-compatibility-test/
* Test farm tests, which we use to check that our releases run correctly on a wide range of platforms. These spin up Amazon EC2 instances for numerous OSes and run various tests on them. They live in tests/letstest

We recommend that Ubuntu run the first of these test suites during build (but we believe the Debian packages already do that).

All of these tests mitigate the risk of regressions in our releases; nonetheless, some regressions do slip past. Because many of our users auto-update, these tend to be reported and fixed quickly in point releases. For instance, regressions in 0.9.0 were fixed in 0.9.1, 0.9.2 and 0.9.3. Certbot 0.9.3 has been used to issue hundreds of thousands of Certs in the field, so we are fairly confident that no further significant regressions exist in it, and that release is likely to be safe as a Xenial SRU.

At least two changes in functionality between 0.4.1 and 0.9.3 do bear specific consideration for Xenial though:

Debian has added a "certbot renew" twice-daily cron job to their packages between 0.4.1 and 0.9.3; we believe this is low regression risk (having secondary renewal mechanisms in place is a NOOP) but Xenial packages may want to increase the debconf verbosity to get consent for this from Xenial users who are upgrading?

We had a custom log rotation scheme (rotate logs after every run), we now act like a more typical daemon, so packages need to be rotating our logs:
https://github.com/certbot/certbot/issues/3382

[Peter Eckersley (pde-lists)]

Also fixes: Launchpad bug #1608214

[Peter Eckersley (pde-lists)]

On IRC, rbasak asked:

<rbasak> From an SRU review perspective, what I most want spelled out the exact list of any behaviour changes you expect Xenial users to receive, and your confirmation that you don't believe that there are any other changes.
<rbasak> Your word as upstream carries a great deal of weight on this.
<rbasak> What you've written is great and sound like it covers most of this already - it's just a clear summary of what the final upload to Xenial will do is what I feel is missing - though of course I know you haven't finalised that yet.

Debian's certbot 0.9.3 packages are a close approximation to what we expect users to get. They Provide: and Replace: letsencrypt, and include aliases for all previous "letsencrypt" commands.

With the exception of the two (an auto-renewal cron job and slightly different log rotation) points noted above, we believe that the functionality and workflows supported by certbot 0.9.3 should be an but less buggy, super-set of the functionality and workflows supported by letsencrypt 0.4.1.

[Peter Eckersley (pde-lists)]

* "a compatible but less buggy"

I should also note that most of our users have been running the certbot-auto script which automatically upgrades them to the latest release of Certbot as soon as it's public, so the odds of us getting bug reports about workflow changes is fairly high.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Harlan Lieberman-Berg (hlieberman)]

Hello everyone -- I'm (the primary, but not only) Debian maintainer of the certbot suite of packages. pde asked me to weigh in here to help answer some of rbasak's questions in a definitive manner.

Since the 0.4.1 release, there have been a handful of user-facing changes, but only a few are workflow changing ones.

The largest is the change in package name from letsencrypt to certbot. This was done for trademark reasons, however, we continue to ship both a transitional package and a symlink for the binary that should preserve users' workflows unmodified.

The second largest change is the addition of an automatic renewal of any certificates, through both a cronjob installed into cron.d and a systemd timer. The certbot binary will only act to renew the certificate, however, when it is "near" to expiring (<15 days). Users who have manually setup a cron already shouldn't experience any failures.

Other than that, there is new functionality available in upstream, but pde has already mentioned all of those. I'm happy to answer any questions or concerns the stable release managers have.

[Peter Eckersley (pde-lists)]

@hlieberman My understanding is that the package name change shouldn't affect any workflows, because "sudo apt-get install letsencrypt" will still work and the "letsencrypt" command will still work. Anything I might have missed?

[Robie Basak (racb)]

Hi Harlan,

Thank you for taking the time to help us out!

On Sun, Nov 20, 2016 at 02:47:14AM -0000, Harlan Lieberman-Berg wrote:
> The second largest change is the addition of an automatic renewal of any
> certificates, through both a cronjob installed into cron.d and a systemd
> timer.

Does this mean that users who currently won't get auto-renewed will
start getting auto-renewed after this proposed update? If so, will that
also include users who currently have expired (languishing) old
certificates?

All,

Is there any progress on the preparation of the update itself? I hope
you're not waiting on my approval. To be clear, my opinion is that
we should update letsencrypt/certbot in 16.04, and I appreciate all the
interest in helping us do this; it's just a matter of the details
(documenting, understanding and mitigating any behavioural changes as
appropriate, and agreeing on the QA plan).

Robie

[Peter Eckersley (pde-lists)]

> Does this mean that users who currently won't get auto-renewed will
> start getting auto-renewed after this proposed update? If so, will that
> also include users who currently have expired (languishing) old
> certificates?

Yes. Each certificate (or more precisely, each lineage of certificates, where a lineage is a series of certificates that replace each other with new validity dates and possibly new domains added) gets a renewal configuration file in /etc/letsencrypt/renewal/ ; the "certbot renew" command walks through those and tries to renew any that are within 30 days of expiry. The Debian packages run that task twice a day out of the box.

I think we've concluded that we'll add a note-upon-installation telling the sys admin that that's going to start happening, and point to where it can be turned off or tweaked.

[Robie Basak (racb)]

On Mon, Nov 28, 2016 at 09:22:28PM -0000, Peter Eckersley wrote:
> I think we've concluded that we'll add a note-upon-installation telling
> the sys admin that that's going to start happening, and point to where
> it can be turned off or tweaked.

I'm afraid that users who automatically update (most of them in 16.04
since it's default) will never see such a message.

[Michael Heuberger (michael.heuberger)]

While you do this for v16.04, could you also publish certbot v0.9.x for v16.10 at the same time? Or need a new ticket for that?

[Harlan Lieberman-Berg (hlieberman)]

Hello everyone.

I'm not sure exactly what form you wanted the SRU in, but... I've included a tarball of the source changes files and dsc's, signed, along with the files included in them.

This upload has two changes from the version in sid. First, I tweaked the versions in python-certbot to reduce the dependency on pyopenssl. The version requirement was to work around an undeclared Breaks relationship between a version of pyopenssl and python-cryptography in a later version (16.0.0) than is in Xenial.

Second, I added a new dummy package to assist the upgrade of python-letsencrypt-apache => python-certbot-apache. The dummy package was never used in Debian, as letsencrypt didn't make it to stable before the name-change occurred.

Please let me know if you have any questions, or if you would like the SRU in some other manner.

[Robie Basak (racb)]

The format sounds fine, thanks. We need one uploader and SRU team member to review. I suggest we figure out who is doing what and both people review before uploading to avoid confusion.

Since you mentioned that RAOF could sponsor, I subscribed him to the bug. Since both of us are on the SRU team, we should figure out who will wear which hat. RAOF, any comments?

[Chris Halse Rogers (raof)]

I'm happy to do either.

[Peter Eckersley (pde-lists)]

>>> import random
>>> x = ["RAOF", "rbasak"]
>>> random.shuffle(x)
>>> print "{0} should upload and {1} should review".format(x)
RAOF should upload and rbasak should review

[Peter Eckersley (pde-lists)]

.format(*x), sorry :)

[Peter Thomassen (mail-peter-thomassen)]

Will it be reasonable to wait for certbot 0.10.0 (due on Jan 4, according to https://github.com/certbot/certbot/milestones), and go ahead then?

[Peter Eckersley (pde-lists)]

I don't think so. We would want 0.10.0 to have at least several weeks of field testing before stable distributions include it as an update, and any regressions in that release would mean we do 0.10.x bugfix releases that would stretch the timeline further.

[Peter Thomassen (mail-peter-thomassen)]

Those are good arguments, thanks for explaining!

[Chris Halse Rogers (raof)]

Hm. It was unclear to me that this requires updating python-acme, python library expected to be used by 3rd parties.

Have any python-acme APIs changed? What are the chances that the update will break downstream users of python-acme?

[Peter Eckersley (pde-lists)]

The changes in python-acme were comparatively minor. A few bugs were fixed; one new feature was added to the API (support for DNS-01 challenges); some protocol messages were removed (because they were believed to have security problems, were never used by Let's Encrypt, and were removed from the IETF ACME draft).

None of those changes should negatively impact non-Certbot ACME clients that use python-acme. It also does not appear that any of those ACME clients are packaged in Debian unstable:

apt-cache rdepends python-acme
python-acme
Reverse Depends:
  python-certbot
  python-certbot-nginx
  python-certbot-apache

[Chris Halse Rogers (raof)]

It's the clients that are *unpackaged* that are the most concerning. Packaged clients we can update if necessary; random ad-hoc clients not so much 😀

[Peter Eckersley (pde-lists)]

* nod. We haven't changed any of the preexisting API calls or semantics, so even unpackaged clients should be fine.

[Chris Halse Rogers (raof)]

Packages uploaded to the unapproved queue, awaiting review.

I changed the version numbers (so that the version in Xenial < Yakkety < Zesty), and added bug links.

[Robie Basak (racb)]

I'm confused on a couple of points. Here's my understanding:

AFAICT, there are three source packages in play here: src:python-acme, src:python-letsencrypt, and src:python-certbot.

src:python-acme appears relatively straightforward. A version bump is needed for new features used by this proposed update, but there is no other package renaming going on.

src:python-certbot supersedes src:python-letsencrypt, though of course apt doesn't see that.

src:python-letsencrypt created binaries letsencrypt, python-letsencrypt and python-letsencrypt-doc in Xenial. It doesn't exist in Yakkety or Zesty.

src:python-certbot creates binaries letsencrypt (transitional to certbot), certbot, python-certbot and python-certbot-doc from Yakkety onwards, and doesn't currently exist in Xenial.

The proposed backport of src:python-certbot to Xenial makes some backport-related dependency changes (so it seems) but I don't see anything that changes the structure of package names or transitional packages from what is in sid (and Yakkety, etc).

So to my questions:

1) What happens to a Xenial user who is using the library provided by python-letsencrypt after this update? Is it intentional that src:python-letsencrypt in Xenial will continue to provide it?

2) This is more of a question for an archive admin. Is it acceptable in Xenial for src:python-certbot to take over the letsencrypt binary name, but for us to retain src:python-letsencrypt that also produces it?

3) What happens if a future update (say a security update) is needed against src:python-letsencrypt for users still using the python-letsencrypt binary package in Xenial (from question 1)? This update would still produce the letsencrypt binary, but we'd have a higher version from src:python-certbot in the archive. So would the subsequent binary upload fail?

Does this mean that we also need to SRU src:python-letsencrypt at the same time to stop producing the letsencrypt binary package at the same time?

This is for Xenial only. I haven't considered Yakkety yet, nor have I completed my review since I want to understand and fix this point before considering any other further interactions (which may change if we change how we're doing this here).

[Robie Basak (racb)]

Based on https://irclogs.ubuntu.com/2017/01/12/%23ubuntu-release.html#t13:47, can you upload a python-letsencrypt for Xenial that drops the letsencrypt package please? We'll need to accept these all together.

Please remember to run the update-maintainer field when preparing these source packages. See https://wiki.ubuntu.com/DebianMaintainerField for the background. I think this needs doing on some of the existing uploads in the queue also, but let's wait to see if anything else needs fixing.

[Peter Eckersley (pde-lists)]

I think it's my job to gently nudge whoever it is that's making the slightly tweaked packages here, but I'm not sure whether it should be Chris or Harlan... if all else is equal perhaps it should be Chris, since we also have a new 0.10.1 release that needs to be packaged for sid and that would be in Harlan's wheelhouse.

[Robie Basak (racb)]

Any news? I feel that changing python-letsencrypt as well exceeds the bounds of the SRU hat I'm wearing - it should have review from a different uploader as well.

[Chris Halse Rogers (raof)]

Sorry; LCA and then email backlog. I'll upload new packages tomorrow.

[Peter Eckersley (pde-lists)]

That's great Chris!

We had some discussions yesterday about whether this has taken long enough that 0.10.x should be considered for the SRU instead. 0.10.1 has been in the field for 13 days; people found a couple of bugs in it, but they only affected new 0.10.x functionality rather than being regressions.

0.10.2 was released yesterday to fix those things and Harlan is working to get it into Debian in time to make the stretch freeze 11 days from now.

Our conclusion was the best thing to do is probably make the 0.9.3 SRU now, and start work on a 0.10.2+ SRU in around 3 weeks, once the 0.10.2 debian packages have received some testing and we've had time to sweep for any final bugs.

[Robie Basak (racb)]

Can I suggest that you maintain these packages in a git tree somewhere (if you aren't doing so already), create branches for the Ubuntu stable releases you're targeting, and then get reviews straight from there? I'll be happy to do an SRU review straight from a git branch. Then review changes could get committed there presumably by more people, so we could move faster. Chris will just need to review new commits that land since his last review, and upload when everyone is happy. And I can allow through any upload after just verifying that it is identical to the thing I already reviewed and accepted in git.

It'll save confusion with multiple versions in the queue this way, too.

[Peter Eckersley (pde-lists)]

Harlan and the other Debian developers have git trees here:

https://alioth.debian.org/plugins/scmgit/cgi-bin/gitweb.cgi?a=project_list;pf=letsencrypt

Maybe Chris should also get access to those repos?

[Chris Halse Rogers (raof)]

python-letsencrypt dropping the letsencrypt binary uploaded.

If you'd like me to have access to the git repos you'll need to add raof-guest to the letsencrypt group (or, alternatively, make the git repository accessible to collab-maint).

[Robie Basak (racb)]

python-letsencrypt and python-letsencrypt-apache don't exist in Yakkety or Zesty, so let's mark those Invalid as there is no action to take for these packages neither in Yakkety nor Zesty (only Xenial).

[Robie Basak (racb)]

Setting all tasks for which I see an upload in the queue as "In Progress" so I can more easily see what's what. Here's my current view of the queues:

python-acme
queue/xenial/unapproved/cb94bae
queue/yakkety/unapproved/15d7282

python-certbot
queue/xenial/new/e67c69a
queue/yakkety/unapproved/c6e7c0b

python-certbot-apache
queue/xenial/new/a42f913
queue/yakkety/unapproved/b496663

python-certbot-nginx
queue/xenial/new/b15997a
queue/yakkety/new/480f5d3

python-letsencrypt
queue/xenial/unapproved/21d1a2f

python-letsencrypt-apache
[empty]

[Robie Basak (racb)]

No changes needed to python-letsencrypt-apache: this will be left as-is in Xenial, will use python-letsencrypt which is only being changed to drop the letsencrypt package for the transition (libraries stay all the same).

[Robie Basak (racb)]

I've verified all the versions of the queue assuming we are backporting in SRUs <= 0.10.2-1 of python-{acme,certbot{,-apache,-nginx}} only and leaving behind python-letsencrypt and python-letsencrypt-apache. So let's retitle for exactly this.

[Robie Basak (racb)]

I spent some time on reviewing this on Friday, but I got interrupted. There was also some discussion on IRC (https://irclogs.ubuntu.com/2017/01/27/%23ubuntu-release.html#t12:31)

My notes:

Can we put a note that there will be a new cronjob active now in debian/NEWS and debian/changelog as suggested on IRC?

What happens if user has installed letsencrypt outside of packaging (for example, to get a newer version while leaving the system one installed)? Will adding a cronjob in the SRU cause any issues?

Can we nominate a person or a team to handle bugs as a consequence of this SRU, and subscribe to package bugs to make sure?

update-maintainer needs running (but I can do this from the queue if no other changes are needed).

[Peter Eckersley (pde-lists)]

This has been stuck for a while, I suspect because it hasn't been clearly on anyone's plate :(.

Let's fix that:

* Brad Warren on the Certbot team is going to construct a retrospective changelog.txt, and post a link here.
* RAOF should probably revise the packages to include that and the news file.
* Anything else?

Separately, this has taken so long that it might be advisable to switch to an 0.10.2 SRU, since that version is now well-tested, and is also the one that's frozen into Debian Stretch. Thoughts for and against? What extra steps would we need for an 0.10.2 SRU?

[Robie Basak (racb)]

I'd like to suggest again that everyone involved does all packaging changes in git, and gets all approvals (from a suitable Ubuntu uploader and the Ubuntu SRU team) in git, before uploading. I really think this will speed things up.

I'm not sure I see everything in https://alioth.debian.org/plugins/scmgit/cgi-bin/gitweb.cgi?pf=letsencrypt, though that may be because repositories have not been renamed following source package renames.

I'm not too bothered about who can push where. Since git is nicely distributed, as long as we can all get to the "current" branch, it shouldn't matter as long as we all agree on where that "current" branch is (as it changes).

RAOF, if you're struggling for time, let me know and I'll see if a colleague on the server team is available to help (since I can only wear the one hat).

[Brad Warren (bradmwarren)]

Here's a link to the changelog:

https://github.com/certbot/certbot/blob/master/CHANGELOG.md

[Chris Halse Rogers (raof)]

OK. So, my understanding of the current state is that:
a) I shall update the SRU packages (in git and upload to the queue):
 *) Run update-maintainer
 *) Add a NEWS entry for the new cron job
 *) Possibly update to 0.10.2
b) Robie will then do the SRU-review for them.

That leaves the question: “What happens if a user has installed letsencrypt outside of packaging? Will the new cron job cause any problems?”. Someone want to answer that? :)

[Chris Halse Rogers (raof)]

Aaargh. Time!

Robie, if someone on the server team has some spare cycles to prepare the packages, that'd be great. Sorry!

There's still the question - “What happens if a user has installed letsencrypt outside of packaging? Will the new cron job cause any problems?” to be resolved, too.

[Peter Eckersley (pde-lists)]

Hi Chris!

I think your todo list looks accurate.

On the question of cron jobs, here are the answers as we understand them upstream:

What happens if the user runs two multiple cron jobs?

Answer 0: probably nothing. "certbot renew" is designed to be run as often as you like, and is normally a no-op.

Answer 1: with some small probability, the user might have two "certbot renew" commands that are executed at the same time. In that case, it would be fairly common for one or both of those to fail with an error, that would produce cron email. The baseline probability of this is collision is about once per 5,000 cert renewals if the hour in the user's cron job is uniform-random, and one in 200 cert renewals if they picked the same two hours (noon and midnight) that are baked into Debian's cron job.

Answer 2: with some much smaller probability, two overlapping "certbot renew" commands could experience a race condition in writing cert lineage files in /etc/letsencrypt/archive, or symlinks in /etc/letsencrypt/live. These would cause a configuration problem (certs and privkeys don't match) about 75 - 80% of the time. I just measured this race condition window on a AWS tiny instance, and estimate that a cert writing race might happen about once every 36,000 cert renewals if the cron job hours line up, or once every 864,000 renewals if users have cron jobs at uniform-random hours.

It is however possible that there are other race conditions in some of our plugins (apache, nginx) that are more likely to occur.

We have a few mitigation options:

Mitigation 0: write a patch to add locking to Certbot 0.10.2 / 0.10.3. This would add a new dependency on python-filelock, and we'd have to make a choice about how much field testing we want for this patch before SRUing it.

Mitigation 1: change the cron job, which picks random times in the hours after noon and midnight, to a systemd timer that runs at two uniform-random hours, or a cron job that has two hours that are less likely to be chosen by sys admins. We can probably use LE serverside data to pick the two least common hours.

Mitigation 2: study the plugin code to ensure that problematic race conditions are really as rare as we think. We could probably tolerate a temporary risk of failure that's one in a million cert renewals on the subset of systems which have two cron processes and where the admin ignored the notice about it -- hard disks fail faster than that.

I think the upstream team favours mitigation 0 :)

[Brad Warren (bradmwarren)]

Small update: If we go with mitigation 0, we won't be using python-filelock. We'll either use an existing Python file locking module packaged in Ubuntu or add our own code that implements lockfiles to Certbot.

[Nish Aravamudan (nacc)]

From a fear of breaking existing deployments in an update -- I would also prefer #0.

Robie is out this week, so maybe RAOF, you can you provide your input here?

[Chris Halse Rogers (raof)]

Mitigation 0 gets a +1 from me. That seems the sensible path.

[Peter Eckersley (pde-lists)]

We have the mitigation in our git master tree (https://github.com/certbot/certbot/pull/4394/files) and are shipping it in an 0.12.1 release today to get field testing. Once that patch has been used to issue ~100K certs I'd be okay with it going into an SRU.

[Peter Eckersley (pde-lists)]

Here's a slightly better link: https://github.com/certbot/certbot/pull/4369

[Peter Eckersley (pde-lists)]

Apologies for the delay here :(

The Certbot locking patch turned out to be more subtle to implement correctly than we had expected, but we finalised and version and shipped it in Certbot 0.14.0 last week. The patch is here: https://github.com/certbot/certbot/pull/4449#issuecomment-299802507

Since that release, there have around 200,000 certificates issued with Certbot 0.14.0. We have had one user report that the locking patch caused a problem for them; that user was intentionally running multiple Certbot instances in parallel for performance reasons. There are probably no great solutions for such users, since their current practices are subject to race conditions that might eventually cause corruption of cert files or even webserver configs.

My instinct is that we should apply the locking patch (perhaps augmenting the error message to explain that users who want to run multiple Certbots safely in parallel should supply --config-dir, --work-dir and --log-dir arguments to each instance), and ship Certbot 0.10.2 to Xenial users ASAP.

[Peter Eckersley (pde-lists)]

The Certbot team has had a couple of calls with rbasak to sort out progress on this.

Lask week we met and concluded that we should SRU 0.14.1 because it's well-tested and has additional bugfixes, as well as creating a streamlined process for the Certbot team to get well-tested releases SRU'd quickly in the future. Rbasak took the task of preparing packages; he would work with Nish to get them reviewed and uploaded; Peter will write a draft/template SRU document for Certbot updates; and Brad will see if he can recruit a member of the upstream Certbot team to be more involved in Ubuntu packaging in the future.

This week, we met again. Rbasak had packages for us to test: https://launchpad.net/~racb/+archive/ubuntu/experimental/+packages ; he's going to ping Harlan and offer those to Debian experimental, as well as updating them to include a notice about the renewal cron job.

We'll meet again in a week to coordinate the actual SRU.

[Peter Eckersley (pde-lists)]

The updated SRU request documentation is here: https://docs.google.com/document/d/1mhmBt6umfdWEqvvnc17ILl5xbW4xSLlFhEG85ZAUYhg/edit

[Peter Eckersley (pde-lists)]

Also, a more generic template for any future SRUs: https://docs.google.com/document/d/1oQOV_kw-gs0ZeNFLyd-nIzpsaoMW5zMLlooaQkoKnF0/edit

[Robie Basak (racb)]

I've prepared backport branches for 0.14.2 for Xenial, Yakkety and Zesty against python-acme, python-certbot, python-certbot-apache and python-certbot-nginx. That's 12 branches. There are an additional 2 branches that handle the source package renames in Xenial. I've pushed all of these to Alioth git prefixed "rbasak-guest/".

Next step: review.

Managing the branches by hand was a nightmare, so I ended up writing a script that creates all the branches. This allowed me to iterate. It's probably easier to review my generator script than review each branch individually. I've pushed the generator scripts (and support files) to a branch called "rbasak-guest/backport-tools" in the Alioth python-certbot repository: https://anonscm.debian.org/cgit/letsencrypt/python-certbot.git/log/?h=rbasak-guest/backport-tools

You can find the repositories for the other source packages on Alioth at: https://anonscm.debian.org/cgit/letsencrypt/

Nish, could you review these please?

Separately, I'll look at Peter's SRU documentation.

[Nish Aravamudan (nacc)]

On Fri, Jul 7, 2017 at 7:06 AM, Robie Basak <email address hidden> wrote:
> I've prepared backport branches for 0.14.2 for Xenial, Yakkety and Zesty
> against python-acme, python-certbot, python-certbot-apache and python-
> certbot-nginx. That's 12 branches. There are an additional 2 branches
> that handle the source package renames in Xenial. I've pushed all of
> these to Alioth git prefixed "rbasak-guest/".
>
> Next step: review.
>
> Managing the branches by hand was a nightmare, so I ended up writing a
> script that creates all the branches. This allowed me to iterate. It's
> probably easier to review my generator script than review each branch
> individually. I've pushed the generator scripts (and support files) to a
> branch called "rbasak-guest/backport-tools" in the Alioth python-certbot
> repository: https://anonscm.debian.org/cgit/letsencrypt/python-
> certbot.git/log/?h=rbasak-guest/backport-tools

The logic of the scripts seems sound to me. Do you want me to do the
individual reviews of each branch as well in this case?

[Robie Basak (racb)]

On Wed, Jul 12, 2017 at 11:30:14PM -0000, Nish Aravamudan wrote:
> The logic of the scripts seems sound to me. Do you want me to do the
> individual reviews of each branch as well in this case?

Could you review what I put in the branches, please? That is: that the
backport contents are suitable in your opinion, and that I haven't
missed anything that might make it inappropriate for SRU - IOW, would
you be happy to sponsor these?

I don't think it's necessary for you to review that the branches were in
fact produced by my script, so I think it'd be sufficient for you to
examine just my scripts.

Note though that there are two branches not produced by my scripts
xenial-transitional for python-certbot and python-certbot-apache
(python-letsencrypt-apache repo). Please check these look good to you
too.

Thanks!

[Nish Aravamudan (nacc)]

On Wed, Jul 12, 2017 at 5:43 PM, Robie Basak <email address hidden> wrote:
> On Wed, Jul 12, 2017 at 11:30:14PM -0000, Nish Aravamudan wrote:
>> The logic of the scripts seems sound to me. Do you want me to do the
>> individual reviews of each branch as well in this case?
>
> Could you review what I put in the branches, please? That is: that the
> backport contents are suitable in your opinion, and that I haven't
> missed anything that might make it inappropriate for SRU - IOW, would
> you be happy to sponsor these?

Reviewed:

- python-acme/rbasak-guest/xenial
- python-acme/rbasak-guest/yakkety
- python-acme/rbasak-guest/zesty
- python-certbot/rbasak-guest/xenial
- python-certbot/rbasak-guest/yakkety
- python-certbot/rbasak-guest/zesty
- python-certbot/rbasak-guest/xenial-transitional
- python-letsencrypt-apache/rbasak-guest/xenial
- python-letsencrypt-apache/rbasak-guest/yakkety
- python-letsencrypt-apache/rbasak-guest/zesty
- python-letsencrypt-apache/rbasak-guest/xenial-transitional
- python-letsencrypt-nginx/rbasak-guest/xenial
- python-letsencrypt-nginx/rbasak-guest/yakkety
- python-letsencrypt-nginx/rbasak-guest/zesty

These all look good to me. Excellent work on a very complicated series
of changes that all need to land together and I think they are
suitable for uploading.

[Brad Warren (bradmwarren)]

Below is a draft of the instructions for how to run integration tests on the proposed Certbot packages. Two comments about this script:

1. I'm not sure if there's a better way to get the Ubuntu codename or how reliable the method I'm using is.
2. I don't know if the way I'm installing the proposed packages will work because I don't know what will happen when apt tries to resolve python-certbot-apache's dependencies.

Please feel free to make/suggest edits if anyone knows how to make these areas work (more robustly).

== Running integration tests ==

Integration tests should be run on the proposed updates for each modified Ubuntu release. You can do this using the script below in a virtual machine for the release of Ubuntu you would like to test.

```
if test "$(id -u)" -ne "0"; then
    echo "Must run as root to install packages and use Docker." >&2
    exit 1
fi

apt update
apt install docker.io git lsb-release net-tools wget -y

wget https://github.com/docker/compose/releases/download/1.15.0-rc1/docker-compose-Linux-x86_64 -O /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

codename=$(lsb_release -a 2>/dev/null | grep Codename | cut -f2)
echo "Package: *" > /etc/apt/preferences.d/proposed-updates
echo "Pin: release a=$codename-proposed" >> /etc/apt/preferences.d/proposed-updates
echo "Pin-Priority: 400" >> /etc/apt/preferences.d/proposed-updates

apt update
apt install python-certbot-apache/$codename-proposed -y
service apache2 stop
apt install python-certbot-nginx/$codename-proposed -y
service nginx stop
certbot_version=$(certbot --version 2>&1 | grep "^certbot" | cut -d " " -f 2)

cd ~
git clone https://github.com/certbot/certbot --branch v$certbot_version --depth 1
cd certbot
tests/boulder-fetch.sh
until curl http://localhost:4000/directory 2>/dev/null; do
  echo waiting for boulder
  sleep 1
done
tests/boulder-integration.sh
echo "Success!"
```

[Brad Warren (bradmwarren)]

Sorry for the 2nd message.

I just noticed I missed the shebang line when I copied my script and having `set -e` is important. I was using `#!/bin/bash -xe`.

[Robie Basak (racb)]

Thanks Brad!

Based on this, and our conversations on IRC, I've prepared documentation for an SRU policy exception at https://wiki.ubuntu.com/StableReleaseUpdates/Certbot

Rather than mandate use of your script, I tried to write it so that it's the outcomes that are mandated, and linked to the script as one potential way of achieving that.

I'd like to seek an opinion from another SRU team member that what I am doing here is in principle reasonable, and then I think we're ready to upload into the proposed pockets.

[Robie Basak (racb)]

I also welcome anyone to comment on this plan or my proposed exception documentation at https://wiki.ubuntu.com/StableReleaseUpdates/Certbot.

[Brian Murray (brian-murray)]

@brad - lsb_release -c returns just the codename so if you use that you could remove the pipe to grep.

[Peter Eckersley (pde-lists)]

@racb the exception document looks good to me.

[Brian Murray (brian-murray)]

Hello Peter, or anyone else affected,

Accepted python-acme into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-acme/0.14.2-0ubuntu0.17.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Peter, or anyone else affected,

Accepted python-certbot into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-certbot/0.14.2-0ubuntu0.17.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Peter, or anyone else affected,

Accepted python-certbot-nginx into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-certbot-nginx/0.14.2-0ubuntu0.17.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Peter, or anyone else affected,

Accepted python-certbot-apache into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-certbot-apache/0.14.2-0ubuntu0.17.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Peter, or anyone else affected,

Accepted python-acme into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-acme/0.14.2-0ubuntu0.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

I don't see an upload of the python-certbot package in the -proposed queue for Xenial, am I missing something?

+python-letsencrypt (0.4.1-1ubuntu0.16.04.1) xenial; urgency=medium
+
+ * Drop letsencrypt binary package; this is now produced by the
+ python-certbot package (LP: #1640978)

[Brad Warren (bradmwarren)]

I just successfully ran the test script I provided earlier on the proposed Ubuntu Zesty packages. I had to make a couple modifications to properly install the packages from the proposed archive. An updated version of the script is attached.

[Peter Eckersley (pde-lists)]

What are next steps here? Is the Zesty SRU ready to go? Does Robie (or someone else) need to make a python-certbot package for Xenial?

[Nish Aravamudan (nacc)]

Hi Peter,

Robie is on leave the next few days, but here's what I know:

1) As far as I can tell all the packages we expect to see in Zesty are in zesty-proposed now. They bake for at least a week there. Someone needs to go through the testing of the packages from proposed and update the metadata in this bug (see any of the messages from Brian Murray on August 10).

2) https://launchpad.net/ubuntu/xenial/+queue indicates the packages we expect to see in Xenial (as noted by Brian in c#69) are in the NEW queue. I am not sure if the SRU team can accept those themselves, or if an AA is needed.

@brian-murray, if you need me to ask an AA to help here, please let me know.

[Steve Langasek (vorlon)]

Hello Peter, or anyone else affected,

Accepted python-certbot-apache into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-certbot-apache/0.14.2-0ubuntu0.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Steve Langasek (vorlon)]

Hello Peter, or anyone else affected,

Accepted python-certbot into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-certbot/0.14.2-0ubuntu0.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Steve Langasek (vorlon)]

Hello Peter, or anyone else affected,

Accepted python-certbot-nginx into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-certbot-nginx/0.14.2-0ubuntu0.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Robie Basak (racb)]

Hello Peter, or anyone else affected,

Accepted python-letsencrypt into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-letsencrypt/0.4.1-1ubuntu0.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brad Warren (bradmwarren)]

Unfortunately the test script at https://wiki.ubuntu.com/StableReleaseUpdates/Certbot/TestScript cannot be used to test the Xenial packages yet because the python-certbot-nginx binary still needs to be accepted. See https://launchpad.net/ubuntu/xenial/+queue.

Is someone able to approve this binary?

[Brad Warren (bradmwarren)]

Thanks for approving the binary.

Tests passed for the proposed Xenial packages using https://wiki.ubuntu.com/StableReleaseUpdates/Certbot/TestScript.

[Brad Warren (bradmwarren)]

While I didn't look any more into the build failure for the letsencrypt package (let me know if you'd like me to), I added a check to https://wiki.ubuntu.com/StableReleaseUpdates/Certbot/TestScript testing that a symlink is properly created in the user's PATH. The test passes on the letsencrypt package in xenial-proposed.

Also after talking to racb in IRC last week, I think we should include a python-letsencrypt package in Xenial similar to the one we have at https://pypi.python.org/pypi/letsencrypt. This allows other Python code to continue to import Certbot through the name letsencrypt.

While the majority of letsencrypt/certbot's internals do not have stable interfaces for other packages to use, there is a plugin interface, base classes, and utilities that are used by Certbot's third party plugins listed at https://certbot.eff.org/docs/using.html#third-party-plugins.

Doing this for python-letsencrypt-apache might also be a good idea so users can continue to install Certbot and our Apache plugin through this package name. This could also allow Python code to import the plugin through the name "letsencrypt_apache", however, the plugin is not expected to be used this way by third parties.

If we do this for the Apache plugin as well, our Python wrapper is available at https://pypi.python.org/pypi/letsencrypt-apache.

[Jeremy Bícha (jbicha)]

Should this update just be let into Xenial now?

Someone needs to start yet another backport of certbot for LP: #1745227.

[Jeremy Bícha (jbicha)]

Oh, I see the Xenial update was blocked by build failures.

[Łukasz Zemczak (sil2100)]

Please someone look into the build failures in the nearest days. If there will be no movement on these packages I will be removing them from xenial-proposed and marking the bug as Won't Fix (as per SRU policy).

[Brad Warren (bradmwarren)]

As a developer of letsencrypt/certbot, it'll be pretty disappointing to us if we can't some kind of update here. In addition to the many bugs listed in the original description that would be fixed by this SRU, the python-letsencrypt-apache package has been unable to obtain new certificates since January due to https://bugs.launchpad.net/ubuntu/+source/python-certbot/+bug/1745227.

[Łukasz Zemczak (sil2100)]

I agree it would be a shame, but we need someone to sort out the build failures and make sure the packages are in proper shape. Robie, since you were the original backporter, would you be able to drive this forward? The point release is nearing and I'd prefer not to have stale packages in -proposed that I don't intend to release for the point-release.

[Kraut.Hosting (kraut.hosting)]

This is still a valid issue for everyone want to do a proper QA chain in software delivery. Both the recent Snap package from Robie as the certbot PPA by our Debian heroes have the problem on different install times you have different versions installed. On a big bunch of boxes this gives us the pain ensuring QA of new versions before roll out to production. Doing cheap cherry picking from the certbot PPA in the past i just gave the packages from proposed a rebuilt try.

Could at first not reproduce the build failure in my pbuilder chain of crap with the rebuilt order of python-letsencrypt, python-acme, python-certbot. But when rebuilding python-letsencrypt then with python-acme_0.14.2-0ubuntu0.16.04.1 as an Build-Depends (without an updated version require) i hit it as well:

   dh_auto_test -O--buildsystem=pybuild
I: pybuild base:184: python2.7 setup.py test
running test
Searching for acme==0.4.1

Note: Bypassing https://pypi.python.org/simple/acme/ (disallowed host; see http://bit.ly/1dg9ijs for details).

Couldn't find index page for 'acme' (maybe misspelled?)
Scanning index of all packages (this may take a while)

Note: Bypassing https://pypi.python.org/simple/ (disallowed host; see http://bit.ly/1dg9ijs for details).

No local packages or download links found for acme==0.4.1
error: Could not find suitable distribution for Requirement.parse('acme==0.4.1')
E: pybuild pybuild:274: test: plugin distutils failed with: exit code=1: python2.7 setup.py test
dh_auto_test: pybuild --test -i python{version} -p 2.7 --dir . returned exit code 13
debian/rules:8: recipe for target 'build' failed
make: *** [build] Error 25

Just like in: https://launchpadlibrarian.net/335455860/buildlog_ubuntu-xenial-amd64.python-letsencrypt_0.4.1-1ubuntu0.16.04.1_BUILDING.txt.gz

Unfortunately i am not too deep into Python here for fixing the fix easily. But as this build failure is the blocker for an SRU i hope someone of the involved rock stars here might figured out easy how to fix it.

[Kraut.Hosting (kraut.hosting)]

FYI pushed some backporting from bionic to xenial without touching existing Python dependencies:

https://launchpad.net/~kraut.hosting/+archive/ubuntu/certbot/+packages

Would recommend to rebase this SRU effort on the packages we have in bionic now anyway.

So far my push does only cover Certbot it self but not all the plugins and integrations.

Feedback on this effort is welcome and i am open for some more polishing ;)

[Brad Warren (bradmwarren)]

Unless we want this package to suddenly break for approximately 10,000 users in February, I think we probably want to do an SRU to Certbot version 0.21.1 or higher. I wrote more about the problem in the relevant issue at https://bugs.launchpad.net/ubuntu/+source/python-letsencrypt/+bug/1745126.

While we may not want to do regular SRUs for Certbot due to the amount of work involved, I think it probably makes sense to do one now due to breaking changes server side.

Please let me know if there's anything I can do on my end to help get this resolved.

[Michael Casadevall (mcasadevall)]

Did some major cleanup of this bug. Debdiffs attached for review.

In general, the primary change for the backport was to modify the package to build against Python 2 instead of Python 3; one code change was needed in certbot to disable API documentation as the necessary sphinx module is not available. We're still sorting out the letsencrypt compat shims but I wanted to get these debdiffs reviewed as they are final or very close to.

[Michael Casadevall (mcasadevall)]

These patches handle straight backports; and are debdiffs from the bionic versions currently being shipped. For the letsencrypt shims, these should be handled as NEW packages and not as a debdiff as they won't exist in bionic and only exist for compat reasons. We're resolving some final issues but this should let Robbie to start looking at it.

[Michael Casadevall (mcasadevall)]

Breakdown of josepy for SRU:

josepy was split from python-acme, and is a NEW package going into Trusty. This is a straight backport of the bionic packaging.

--- python-josepy-1.1.0/debian/compat 2018-04-15 20:45:24.000000000 +0000
+++ python-josepy-1.1.0/debian/compat 2018-11-23 19:55:13.000000000 +0000
@@ -1 +1 @@
-11
+9
diff -Nru python-josepy-1.1.0/debian/control python-josepy-1.1.0/debian/control
--- python-josepy-1.1.0/debian/control 2018-04-15 20:45:24.000000000 +0000
+++ python-josepy-1.1.0/debian/control 2018-11-23 19:55:13.000000000 +0000
@@ -1,9 +1,10 @@
 Source: python-josepy
 Section: python
 Priority: optional
-Maintainer: Debian Let's Encrypt <email address hidden>
+Maintainer: Ubuntu Developers <email address hidden>
+XSBC-Original-Maintainer: Debian Let's Encrypt <email address hidden>
 Uploaders: Harlan Lieberman-Berg <email address hidden>
-Build-Depends: debhelper (>= 11~),
+Build-Depends: debhelper (>= 9~),
         dh-python,
         python (>= 2.7),
         python-cryptography (>= 0.8),
diff -Nru python-josepy-1.1.0/debian/rules python-josepy-1.1.0/debian/rules

Explanation: Debhelper 9 was the shipped version in trusty so changes to dependencies to work with it were required.

--- python-josepy-1.1.0/debian/rules 2018-04-15 20:45:24.000000000 +0000
+++ python-josepy-1.1.0/debian/rules 2018-11-23 19:55:13.000000000 +0000
@@ -17,6 +17,6 @@
  find $(CURDIR)/debian/ -type d -name testdata -print0 | xargs -0 rm -rf '{}' \;
  rm -f $(CURDIR)/debian/python*-josepy/usr/bin/jws

-override_dh_installdocs:
- dh_installdocs --doc-main-package=python-josepy-doc -p python-josepy-doc
- dh_installdocs -p python-josepy -p python3-josepy
+#override_dh_installdocs:
+# dh_installdocs --doc-main-package=python-josepy-doc -p python-josepy-doc
+# dh_installdocs -p python-josepy -p python3-josepy

Explanation: installdocs doesn't support -p on Trusty which is used to create symlinks. This behavior mimics the way the docs packages were installed on Xenial for the existing packages; documentation still ends up properly in the -doc package (checked manually).

[Michael Casadevall (mcasadevall)]

Breakdown of changes for python ACME; I caught a mistake in the previous debdiff so reuploading a revised one with this comment:

diff -Nru python-acme-0.22.2/debian/compat python-acme-0.22.2/debian/compat
--- python-acme-0.22.2/debian/compat 2018-03-17 15:24:35.000000000 +0000
+++ python-acme-0.22.2/debian/compat 2018-11-25 15:09:41.000000000 +0000
@@ -1 +1 @@
-11
+9

Explaination: debhelper version change

diff -Nru python-acme-0.22.2/debian/control python-acme-0.22.2/debian/control
--- python-acme-0.22.2/debian/control 2018-03-17 15:24:35.000000000 +0000
+++ python-acme-0.22.2/debian/control 2018-11-25 15:09:41.000000000 +0000
@@ -1,13 +1,14 @@
 Source: python-acme
 Section: python
 Priority: optional
-Maintainer: Debian Let's Encrypt <email address hidden>
+Maintainer: Ubuntu Developers <email address hidden>
+XSBC-Original-Maintainer: Debian Let's Encrypt <email address hidden>
 Uploaders: Harlan Lieberman-Berg <email address hidden>,
            Francois Marier <email address hidden>
-Build-Depends: debhelper (>= 11~),
+Build-Depends: debhelper (>= 9~),
                dh-python,
                python-all (>= 2.7),
- python-cryptography (>= 1.3.4),
+ python-cryptography,
                python-docutils,
                python-josepy,
                python-mock,

Explanation: This was an incorrect dependency in bionic; acme works proeprly against cryotpgraphy 1.2 which was shipped in Xenial.

@@ -19,7 +20,7 @@
                python-six (>= 1.9),
                python-tz,
                python3 (>= 3.4),
- python3-cryptography (>= 1.3.4),
+ python3-cryptography,
                python3-docutils,
                python3-josepy,
                python3-mock,

Explanation: See above.

diff -Nru python-acme-0.22.2/debian/rules python-acme-0.22.2/debian/rules
--- python-acme-0.22.2/debian/rules 2018-03-17 15:24:35.000000000 +0000
+++ python-acme-0.22.2/debian/rules 2018-11-25 15:09:30.000000000 +0000
@@ -7,11 +7,10 @@

 override_dh_auto_build:
  dh_auto_build
- PYTHONPATH=. \
- http_proxy='127.0.0.1:9' \
- https_proxy='127.0.0.1:9' \
- sphinx-build -N -bhtml docs/ build/html
+ PYTHONPATH=. http_proxy='127.0.0.1:9' sphinx-build -N -bhtml docs/ build/html

 override_dh_auto_install:
  dh_auto_install
  find $(CURDIR)/debian/ -type d -name testdata -print0 | xargs -0 rm -rf '{}' \;

Explaination: I reused the rules from the existing python-acme version in Trusty, dropping https_proxy. This may not be necessary TBH.

[Michael Casadevall (mcasadevall)]

Breakdown of python-certbot changes; this is a NEW package going into trusty backported from bionic following the letsencrypt->certbot name change. A revised debdiff is attached to this bug.

diff -Nru python-certbot-0.23.0/debian/compat python-certbot-0.23.0/debian/compat
--- python-certbot-0.23.0/debian/compat 2018-03-23 23:00:13.000000000 +0000
+++ python-certbot-0.23.0/debian/compat 2018-11-25 16:20:44.000000000 +0000
@@ -1 +1 @@
-11
+9

Explanation: Debhelper compat change

diff -Nru python-certbot-0.23.0/debian/control python-certbot-0.23.0/debian/control
--- python-certbot-0.23.0/debian/control 2018-04-07 03:21:05.000000000 +0000
+++ python-certbot-0.23.0/debian/control 2018-11-25 16:20:44.000000000 +0000
@@ -1,47 +1,41 @@
 Source: python-certbot
 Section: python
 Priority: optional
-Maintainer: Debian Let's Encrypt <email address hidden>
+Maintainer: Ubuntu Developers <email address hidden>
+XSBC-Original-Maintainer: Debian Let's Encrypt <email address hidden>
 Uploaders: Harlan Lieberman-Berg <email address hidden>,
            Francois Marier <email address hidden>
-Build-Depends: debhelper (>= 11~),
+Build-Depends: debhelper (>= 9~),
                dh-python,
- python3,
- python3-acme (>= 0.22.0~),
- python3-configargparse (>= 0.10.0),
- python3-configobj,
- python3-cryptography (>= 1.2),
- python3-distutils,
- python3-josepy,
- python3-mock,
- python3-parsedatetime (>= 1.3),
- python3-repoze.sphinx.autointerface,
- python3-requests (>= 2.4.3),
- python3-rfc3339,
- python3-setuptools (>= 1.0),
- python3-sphinx,
- python3-sphinx-rtd-theme,
- python3-tz,
- python3-zope.component,
- python3-zope.interface
+ python,
+ python-acme (>= 0.22.0~),
+ python-configargparse (>= 0.10.0),
+ python-configobj,
+ python-cryptography (>= 1.2),
+ python-josepy,
+ python-mock,
+ python-parsedatetime (>= 1.3),
+ python-requests (>= 2.4.3),
+ python-rfc3339,
+ python-setuptools (>= 1.0),
+ python-sphinx,
+ python-sphinx-rtd-theme,
+ python-tz,
+ python-zope.component,
+ python-zope.interface
 Standards-Version: 4.1.4
 Homepage: https://certbot.eff.org/
 Vcs-Git: https://salsa.debian.org/letsencrypt-team/certbot/certbot.git
 Vcs-Browser: https://salsa.debian.org/letsencrypt-team/certbot/certbot
 Testsuite: autopkgtest-pkg-python

Explanation: Some of the dependencies for certbot do not have Python3 versions in Xenial; this was resolved by having certbot use Python2 and becoming a python2 package; it could also have been resolved by SRUing python-parsedtimedate as it supports Python 3 but doesn't have the necessary deb. An SRU for the bionic package may be needed to remove this with the Python 3 version.

-Package: python3-certbot
+Package: python-...

[Michael Casadevall (mcasadevall)]

After testing and further work, a revised python-acme debdiff is attached. This includes one additional backport that allows josepy to be imported through the acme package; this change has already been integrated upstream, and is required to allow packages that assume acme.jose still existed as part of the python-acme module.

[Michael Casadevall (mcasadevall)]

This is the backported patch to python-acme, as stated before, this is a direct backport from upstream to handle compat issues, and extends the unittests to cover the new backworks compatibility area.

mcasadevall@lighthouse:~/src/le/sru/python-acme-0.22.2/debian/patches$ cat fix-jose-import
Description: Allow josepy to be imported via acme.jose
 This is a backwards compatibility fixed taken from upstream
 from the following commits:
 https://github.com/certbot/certbot/commit/e3cb782e5992ba306de59ba96dfb6f125720cd06.patch
 https://github.com/certbot/certbot/commit/ec297ccf72e95961586ec2382c3e3225ce578aa4.patch
 .
 python-acme (0.22.2-1ubuntu0.16.04.1~ppa4) xenial; urgency=medium
 .
   * Backport to Xenial for LE change (LP: #1640978)
Author: Michael Casadevall <email address hidden>
Bug-Ubuntu: https://bugs.launchpad.net/bugs/1640978

---
The information above should follow the Patch Tagging Guidelines, please
checkout http://dep.debian.net/deps/dep3/ to learn about the format. Here
are templates for supplementary fields that you might want to add:

Origin: upstream
Bug: https://github.com/certbot/certbot/pull/6592
Forwarded: not-needed
Reviewed-By: Michael Casadevall <email address hidden>
Last-Update: 2019-01-11

--- python-acme-0.22.2.orig/acme/__init__.py
+++ python-acme-0.22.2/acme/__init__.py
@@ -10,3 +10,18 @@ supported version: `draft-ietf-acme-01`_
   https://github.com/ietf-wg-acme/acme/tree/draft-ietf-acme-acme-01

 """
+import sys
+
+# This code exists to keep backwards compatibility with people using acme.jose
+# before it became the standalone josepy package.
+#
+# It is based on
+# https://github.com/requests/requests/blob/1278ecdf71a312dc2268f3bfc0aabfab3c006dcf/requests/packages.py
+
+import josepy as jose
+
+for mod in list(sys.modules):
+ # This traversal is apparently necessary such that the identities are
+ # preserved (acme.jose.* is josepy.*)
+ if mod == 'josepy' or mod.startswith('josepy.'):
+ sys.modules['acme.' + mod.replace('josepy', 'jose', 1)] = sys.modules[mod]
--- /dev/null
+++ python-acme-0.22.2/acme/jose_test.py
@@ -0,0 +1,53 @@
+"""Tests for acme.jose shim."""
+import importlib
+import unittest
+
+class JoseTest(unittest.TestCase):
+ """Tests for acme.jose shim."""
+
+ def _test_it(self, submodule, attribute):
+ if submodule:
+ acme_jose_path = 'acme.jose.' + submodule
+ josepy_path = 'josepy.' + submodule
+ else:
+ acme_jose_path = 'acme.jose'
+ josepy_path = 'josepy'
+ acme_jose_mod = importlib.import_module(acme_jose_path)
+ josepy_mod = importlib.import_module(josepy_path)
+
+ self.assertIs(acme_jose_mod, josepy_mod)
+ self.assertIs(getattr(acme_jose_mod, attribute), getattr(josepy_mod, attribute))
+
+ # We use the imports below with eval, but pylint doesn't
+ # understand that.
+ # pylint: disable=eval-used,unused-variable
+ import acme
+ import josepy
+ acme_jose_mod = eval(acme_jose_path)
+ josepy_mod = eval(josepy_path)
+ self.assertIs(acme_jose_mod, josepy_mod)
+ self.assertIs(getattr(acme_jose_mod, attribute), getattr(josepy_...

[Fabien COMBERNOUS (fc.)]

when will new package arrive on the repositories ?

This update is important since the TLS-SNI-01 validation option is soon turned off by LE.

TLS-SNI-01 validation is reaching end-of-life. It will stop working
temporarily on February 13th, 2019, and permanently on March 13th, 2019.

[Robie Basak (racb)]

I'm reviewing this now.

Michael, did you miss the fix from Bionic uploaded in 0.22.2-1ubuntu0.1?

  * Add ready status type to be compatible with the new Let's Encrypt ACMEv2
    endpoint (LP: #1777205).

From the perspective of current Bionic it looks like this is getting dropped in the backport, which I think is by accident?

Though I'd ask now, before I've finished my review (this is taking a while!), to speed things up if we find a change is needed.

[Robie Basak (racb)]

This is on python-acme, sorry.

[Robie Basak (racb)]

python-certbot backport review:

The changes you've described look good in general, but I have some specific questions for pieces for which I couldn't find any explanation:

python-certbot no longer Recommends certbot, but python3-certbot did. Is this intentional?

In debian/rules, what's the reason for including pkg-info.mk into the backport as a delta? I don't see the variables it defines being used anywhere.

What's the reason for the new clean target in debian/rules?

Why is certbot.timer's installation being removed? Has auto renewal been tested? If so, what's the mechanism it uses now?

Why remove the installation of the cli.ini that disables certbot-internal log rotation?

Some notes to self explaining further bits of the delta I did manage to understand:

python3-distutils build-dep drop not explained but distutils is integrated in Python 2 in Xenial and not in its own packaging so dropping the build dependency is presumably intentional.

The dropping of the python3-repoze.sphinx.autointerface build dependency is explained later with the disable-autointerface-docs patch.

[Robie Basak (racb)]

python-letsencrypt backport review:

I think debian/README.source is now redundant and incorrect and should be removed?

I see that pkg-info.mk was included from Debian previously. Is this still necessary?

debian/rules arranges debian/letsencrypt/usr/bin/* but no longer ships a binary package called letsencrypt.

python-letsencrypt-apache backport review:

I think debian/README.source is now redundant and incorrect and should be removed?

[Robie Basak (racb)]

Test plan:

Could we agree a specific test plan for this SRU please, in addition to the usual testing agreed in https://wiki.ubuntu.com/StableReleaseUpdates/Certbot? Specifically there are a few things that are special about this one. I'd like to make sure that somebody makes sure that all the gotchas of which we've already become aware and fixed are still fixed in the SRU verification step, before the update is pushed out for automatic update by users.

This includes:

1. letsencrypt rename compatibility, including the CLI, letsencrypt and letsencrypt-apache Python modules.

2. Log rotation behaviour

3. Automatic renewal behaviour

4. Bug 1777205 and its patch

Once agreed we can update the Test Plan section of the bug description.

[Brad Warren (bradmwarren)]

I updated and added additional checks to the test script at https://wiki.ubuntu.com/StableReleaseUpdates/Certbot/TestScript including tests for all four of the areas Robie flagged in his most recent post.

[Robie Basak (racb)]

Thanks Brad!

As Michael seems to be unavailable at the moment I'll try to fix my review comments up myself. I've asked my colleague Christian to peer-review my work from this point to fulfil Ubuntu SRU review requirements.

What I don't know the answer to is some of my questions from my reviews above. These are in comments 99, 101 and 102.

I assume that the patch from 0.22.2-1ubuntu0.1 needs to be included.

I'm going to drop the bits I think are packaging code skeletons.

Two questions I'm really uncertain about right now:

Why is certbot.timer's installation being removed? Has auto renewal been tested? If so, what's the mechanism it uses now?

Why remove the installation of the cli.ini that disables certbot-internal log rotation?

I'd appreciate any insight into these please. What were the mechanisms for auto renewal and log rotation in 0.4, what are they in 0.22, and what should packaging be doing in 0.22?

I've asked Harlan for upload access to the ~certbot PPA on IRC. I hope to have updated packages uploaded there soon for further testing.

[Brad Warren (bradmwarren)]

> I assume that the patch from 0.22.2-1ubuntu0.1 needs to be included.

Yes, I think this patch should be included. The packages in 18.04 included support Let's Encrypt's newer endpoint, however, this feature is broken without this patch.

Thanks for catching this. The updated tests now test against the new API as well.

The other questions here were largely answered in IRC, but to keep this thread here up-to-date and maybe add a little more context...

> Why is certbot.timer's installation being removed?

I think removing this was a mistake. The current Ubuntu 16.04 package does not configure automatic renewal, but I think it should be added as it benefits users and keeps the package closer to the one found in Bionic.

There's actually a lot of discussion about the addition of automatic renewal earlier in this thread from a couple years ago.

> Has auto renewal been tested?

I suspect not. At the very least, I hadn't done so.

Previously my test script was almost exclusively running our upstream tests on the installed packages and did not do much to test things specifically found in the .deb packages.

For testing automatic renewal, however, the updates to the script I made last week check:

1. certbot.timer exists and is enabled.
2. /etc/cron.d/certbot exists.

> Why remove the installation of the cli.ini that disables certbot-internal log rotation?

I think we probably should keep this file unless there is a good reason not to do so. Why make unnecessary modifications to the packages?

In the current Xenial packages, Certbot is using its own log rotation provided by the Python standard library. Each run of Certbot creates a new log file in /var/log/letsencrypt and up to 10 log files are created after which old log files are deleted.

The cli.ini file in the Bionic package turns off this log rotation causing Certbot to always append to the same file at /var/log/letsencrypt/letsencrypt.log with the thinking that this will better play with programs like logrotate.

[Robie Basak (racb)]

I've pulled Michael's current work into git trees available at: https://code.launchpad.net/~certbot/+git (each has a branch called lp1640978). Please maintain these branches as fast-forwarding only.

I have updated python-acme, python-certbot, python-letsencrypt and python-letsencrypt-apache based on my review comments above, bumped the ppa suffix versions, tweaked the version strings as needed, pushed to the git branches, and uploaded to the xenial-sru PPA.

Next, I need to check the PPA versions work as expected. Then I'll get a colleague's review. If all is good, we will be able to upload and accept into xenial-proposed.

[Brad Warren (bradmwarren)]

I tried manually installing packages from the PPA and running https://wiki.ubuntu.com/StableReleaseUpdates/Certbot/TestScript but the script failed because the Certbot systemd timer wasn't found.

[Robie Basak (racb)]

On Wed, Feb 13, 2019 at 05:26:38PM -0000, Brad Warren wrote:
> I tried manually installing packages from the PPA and running
> https://wiki.ubuntu.com/StableReleaseUpdates/Certbot/TestScript but the
> script failed because the Certbot systemd timer wasn't found.

Thanks. I'll follow up.

I also found that I missed review of python-certbot-apache, so I'll get
on to that also.

[Robie Basak (racb)]

I've fixed that, updated the git repositories under ~certbot, and re-uploaded to the PPA. I've included some other fixes as well. Full changes are available to view in the git repositories.

Christian, please could you review all changes in the range review/2019-02-08..lp1640978 for the packages python-josepy, python-acme, python-letsencrypt, python-letsencrypt-apache, python-certbot and all commits but the root commit in python-certbot-apache? That should give us full review coverage then.

[Brad Warren (bradmwarren)]

Test script passes on the new packages in the PPA.

[Christian Ehrhardt  (paelzer)]

TL;DR
- LGTM a few questions for clarification, but no nacks

----

I took logs for everything non trivial that I did (wonder about) on review.
+ = LGTM
? = ok, but I'd have a question
x = nack, this should be different

python-acme:
? rebase for 0.22.2-1ubuntu0.1
  - The bug #1777205 should get a xenial task before upload
  - and then the bug be verified on the SRU
  - please remember to buildpkg with -v0.4.1-1
+ changelog tweaks
+ backport version string
  Double ubuntu in the version seems odd, but it seems correct.
  It is not part of [2] and an even more special case of [1]
x Add SRU explanation to changelog
  The version is 0.22.2 the entry calls it "version 0.23 in 16.04"
  Seems wrong, either fix or explain why it is that way.
  I see that all others e.g. python-certbot is at 0.23 for real
  will that mismatch be a problem?

python-certbot:
+ recommends certbot
? dh_clean ok but why, would it hurt?
? pkg-info.mk ok but why, would it hurt?
? cli.ini
  That is the one to keep the logrotations as is (ok)
  I thought from our talk Xenial did that as well, but it didn't
  There is nothing about it in pkg/ubuntu/xenial-devel:debian/rules
  nor in pkg/import/0.14.2-1:debian/rules
  I guess the old default was different, which is ok.
  But do we now want/need any conffile cleanup in >=Bionic?
  This would be independent to this SRU, but I wanted to ask.
+ build rules
+ install init no-op
? Restore original timer service installation
  The "original" used the older dh_systemd_enable/dh_systemd_start
  I think your's work only with newer debhelper, it is not "the original"
  I just found you clean that up later - would have been nice to squash those two :-/
+ backport version string (that one I like more)
? use dh_systemd
  The old package used install -D --mode=644 for the timer
  any reason you have chosen mkdir + cp instead ?
  That commit also adds whitespace damage above "# Install certbot.timer"
+ doc paths
+ version/changelog changes
? Add SRU explanation to changelog
  you might mention changes to d/rules and d/compat to make the backport working here

python-letsencrypt:
+ Remove redundant incorrect d/README.source
? further changes to changelog ( I have a general statement about those below)
? It is not clear to me how that "shimming" takes place, extending the changelog on that would be useful as well.

python-josepy:
? changelog/version bumps in general ok
  You mentioned the new certbot.timer and such in python-letsencrypt (which doesn't hold the timers)
  Please choose:
  - Why did you mention it there?
  - Why didn't you mention it here?
  I assume it is because of the package takeover letsencrypt->certbot.
  But really, that could use some words (more about that below already)

python-letsencrypt-apache
+ cleanup
+ changelog
+ version bumps

python-certbot-apache:
+ empty dir
+ VCS
+ cleanup (again ok but why)
+ doc building
+ version string changes
+ doc-base paths
+ SRU explanation (nothing more than what was said already)

I just realized that the summary changelogs all say: "Log rotation is switched to logrotate via /etc/logrotate.d/certbot"
This whole section could need more why it was done and where the old/new tunable...

[Robie Basak (racb)]

Thank you for the review! I'll look in detail later, but in case it speeds up an iteration, note that I've started from the perspective that this is a backport from Bionic - so the cleanups are on the principle of reducing the diff against Bionic. Where I saw a diff hunk that didn't seem necessary, that's what I removed in my cleanup commits.

[Robie Basak (racb)]

Thank you for the review! I've addressed review comments inline. With
Christian sitting next to me we've been through my fixes and he's happy
with them.

I've updated git branches and updated the PPA for python-certbot ~ppa7
only, as that's the only one that contains a functional change from
review feedback; the others are changelog clarifications only. I will
test this ~ppa7.

Assuming all is good, what's in the PPA, except for changelog text
adjustments, will then go into the Xenial proposed pocket ready for
wider testing.

> TL;DR
> - LGTM a few questions for clarification, but no nacks
>
> ----
>
> I took logs for everything non trivial that I did (wonder about) on review.
> + = LGTM
> ? = ok, but I'd have a question
> x = nack, this should be different
>
> python-acme:
> ? rebase for 0.22.2-1ubuntu0.1
> - The bug #1777205 should get a xenial task before upload
> - and then the bug be verified on the SRU
> - please remember to buildpkg with -v0.4.1-1

Upstream confirmed that 1777205 never affected Xenial (except perhaps in
proposed) so a task there would not be accurate. However I agree that
it's prudent to verify that it doesn't regress in this SRU during SRU
verification, so it's part of the test plan I proposed in comment 103
and I will copy that up to the bug description.

> + changelog tweaks
> + backport version string
> Double ubuntu in the version seems odd, but it seems correct.
> It is not part of [2] and an even more special case of [1]

I decided to stick to backportpackage(1)'s behaviour as a de facto
standard for these SRUs, as ugly as the result is in this case :-/

> x Add SRU explanation to changelog
> The version is 0.22.2 the entry calls it "version 0.23 in 16.04"
> Seems wrong, either fix or explain why it is that way.
> I see that all others e.g. python-certbot is at 0.23 for real
> will that mismatch be a problem?

Changelog updated with an explanation in f4de847.

> python-certbot:
> + recommends certbot
> ? dh_clean ok but why, would it hurt?
> ? pkg-info.mk ok but why, would it hurt?

These two changes minimise the diff against the backport source.

> ? cli.ini
> That is the one to keep the logrotations as is (ok)
> I thought from our talk Xenial did that as well, but it didn't
> There is nothing about it in pkg/ubuntu/xenial-devel:debian/rules
> nor in pkg/import/0.14.2-1:debian/rules
> I guess the old default was different, which is ok.
> But do we now want/need any conffile cleanup in >=Bionic?
> This would be independent to this SRU, but I wanted to ask.

cli.ini is a new conffile that was introduced at some point in the
certbot binary package only. AFAICT it never existed in the letsencrypt
binary package, or at least not in the one in Xenial. We ship the
conffile to turn off certbot's internal log rotation to avoid collision
with logrotate behaviour.

Since we're backporting from Bionic, this conffile will be introduced
into Xenial as part of this SRU.

The conffile continues to exist in the current development release, so I
don't think any further or special conffile handling is needed, since
we're only introducing it and not doing anything else with it.

As discussed I'v...

[Robie Basak (racb)]

FTR, I spotted a couple of things that turned out to be non-issues:

1) The letsencrypt binary has become a symlink and moved to the certbot binary package. This needs a Breaks/Replaces, which does exist but doesn't cover the version of letsencrypt currently in xenial-proposed. However this isn't a problem because the movement had happened prior to the version in xenial-proposed.

2) We're still shipping /etc/cron.d/certbot as it is in the Bionic backport source. However it is inert, as intended, since Ubuntu >= Xenial always has systemd (unless you're upgrading from Trusty but haven't rebooted yet). Even if the cron job did run, it wouldn't be a problem.

[Adam Conrad (adconrad)]

Hello Peter, or anyone else affected,

Accepted python-josepy into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-josepy/1.1.0-1~ubuntu16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Robie Basak (racb)]

Hello Peter, or anyone else affected,

Accepted python-acme into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-acme/0.22.2-1ubuntu0.1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Robie Basak (racb)]

Hello Peter, or anyone else affected,

Accepted python-certbot into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-certbot/0.23.0-1~ubuntu16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Robie Basak (racb)]

Hello Peter, or anyone else affected,

Accepted python-certbot-apache into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-certbot-apache/0.23.0-1~ubuntu16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Robie Basak (racb)]

Hello Peter, or anyone else affected,

Accepted python-letsencrypt into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-letsencrypt/0.7.0-0ubuntu0.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Robie Basak (racb)]

Hello Peter, or anyone else affected,

Accepted python-letsencrypt-apache into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-letsencrypt-apache/0.7.0-0ubuntu0.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Robie Basak (racb)]

Since the dependency tree is a few packages deep, this set of proposed updates will take some time to become available for testing in xenial-proposed. I will coax things through manually as dependencies are built and published, and comment again with a call for testing when ready.

[Brad Warren (bradmwarren)]

I successfully ran the integration tests described at https://wiki.ubuntu.com/StableReleaseUpdates/Certbot.

The packages that were used for testing were:

certbot 0.23.0-1~ubuntu16.04.1
letsencrypt 0.23.0-1~ubuntu16.04.1
python-acme 0.22.2-1ubuntu0.1~16.04.1
python-certbot 0.23.0-1~ubuntu16.04.1
python-certbot-apache 0.23.0-1~ubuntu16.04.1
python-josepy 1.1.0-1~ubuntu16.04.1
python-letsencrypt 0.7.0-0ubuntu0.16.04.1
python-letsencrypt-apache 0.7.0-0ubuntu0.16.04.1

[Robie Basak (racb)]

The Let's Encrypt packages in Ubuntu 16.04 will stop working on 13 March 2019. Updates to fix this problem are now ready for testing. Please help us test!

Certbot (formerly called Let's Encrypt) will stop working properly on 13 March 2019 when TLS-SNI-01 validation is turned off by the primary Let's Encrypt CA. This will make the Ubuntu 16.04 letsencrypt package effectively useless for many users. Newer Ubuntu releases package versions of Certbot that implement alternate validation methods and are not affected.

In Ubuntu 16.04 we are addressing this by backporting the existing Certbot packaging from Ubuntu 18.04. Proposed updates for 16.04 are now available and will be released to the usual update channels when they have received sufficient testing and validation.

Please help us test these packages on 16.04. Reports of both success and failure are appreciated. Please include details of how you tested, including the package versions used and any specifics of your environment to https://launchpad.net/bugs/1640978

To test:

To avoid upgrading wholesale to the proposed pocket (doing so may break your system), ensure that apt is pinned from doing so by making sure that a file exists in `/etc/apt/preferences.d/`, for example `/etc/apt/preferences.d/proposed-updates`, as follows:

    Package: *
    Pin: release a=xenial-proposed
    Pin-Priority: 400

Allow apt to upgrade all the packages provided in this Certbot update by creating `/etc/apt/preferences.d/certbot-proposed` as follows:

    Package: python-acme-doc python-acme python3-acme certbot letsencrypt python-certbot-doc python-certbot python-certbot-apache python-certbot-apache-doc python-josepy-doc python-josepy python3-josepy python-letsencrypt python-letsencrypt-apache
    Pin: release a=xenial-proposed
    Pin-Priority: 500

Make sure the following line exists in your `/etc/apt/sources.list` (or in a file in `/etc/apt/sources.list.d/`):

    deb http://archive.ubuntu.com/ubuntu/ xenial-proposed main universe

(other components such as `restricted` and `multiverse` may also be present).

Now you may simply `apt update` and `apt upgrade` as usual and apt will upgrade to this proposed update ready for testing.

Reports of both success and failure are appreciated. Please include details of how you tested, including the package versions used and any specifics of your environment. You can generate the status and versions of all the Certbot-related packages with the following command:

    dpkg-query -W -f='${db:Status-Abbrev} ${binary:Package} ${version}\n' python-acme-doc python-acme python3-acme certbot letsencrypt python-certbot-doc python-certbot python-certbot-apache python-certbot-apache-doc python-josepy-doc python-josepy python3-josepy python-letsencrypt python-letsencrypt-apache

Please add testing reports to the Launchpad tracking bug at: https://launchpad.net/bugs/1640978

[Mark (nulleuro)]

Successfully updated:
    letsencrypt (0.4.1-1 => 0.23.0-1~ubuntu16.04.1)
    python-acme (0.4.1-1 => 0.22.2-1ubuntu0.1~16.04.1)
    python-letsencrypt (0.4.1-1 => 0.7.0-0ubuntu0.16.04.1)
    python-letsencrypt-apache (0.4.1-1 => 0.7.0-0ubuntu0.16.04.1)

(though I actually had to apt-get install letencrypt python-letsencrypt-apache instead of upgrade, because the new packages from the proposed repo were held back by apt)

    $ dpkg-query
    ii certbot 0.23.0-1~ubuntu16.04.1
    ii letsencrypt 0.23.0-1~ubuntu16.04.1
    ii python-acme 0.22.2-1ubuntu0.1~16.04.1
    un python-acme-doc
    ii python-certbot 0.23.0-1~ubuntu16.04.1
    ii python-certbot-apache 0.23.0-1~ubuntu16.04.1
    un python-certbot-apache-doc
    un python-certbot-doc
    ii python-josepy 1.1.0-1~ubuntu16.04.1
    ii python-letsencrypt 0.4.1-1
    ii python-letsencrypt-apache 0.7.0-0ubuntu0.16.04.1

Ran certbot renew --force-renewal which successfully replaced all existing certs.

[Paride Legovini (paride)]

Successfully upgraded:

ii letsencrypt 0.4.1-1
ii python-acme 0.4.1-1
ii python-letsencrypt 0.4.1-1
ii python-letsencrypt-apache 0.4.1-1

to:

ii certbot 0.23.0-1~ubuntu16.04.1
ii letsencrypt 0.23.0-1~ubuntu16.04.1
ii python-acme 0.22.2-1ubuntu0.1~16.04.1
un python-acme-doc
ii python-certbot 0.23.0-1~ubuntu16.04.1
ii python-certbot-apache 0.23.0-1~ubuntu16.04.1
un python-certbot-apache-doc
un python-certbot-doc
ii python-josepy 1.1.0-1~ubuntu16.04.1
ii python-letsencrypt 0.7.0-0ubuntu0.16.04.1
ii python-letsencrypt-apache 0.7.0-0ubuntu0.16.04.1

The upgrade went smoothly (no held back packages), but I didn't setup any pinning (testing in a dedicated container). I'm able to generate/renew certificates with the upgraded packages.

The python-certbot-nginx plugin is not functional, but from the discussion we had on IRC I know it won't be made part of the SRU, as it wasn't part of Xenial in the first place.

Everything LGTM.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-josepy - 1.1.0-1~ubuntu16.04.1

---------------
python-josepy (1.1.0-1~ubuntu16.04.1) xenial; urgency=medium

  [ Robie Basak ]
  * This update is part of the set of major updates moving Let's
    Encrypt/Certbot to version 0.23 in 16.04 in order to allow it to
    continue working following the general shutdown of TLS-SNI-01
    validation (LP: #1640978).

  [ Michael Casadevall ]
  * Backport SRU dependency for certbot backport

 -- Robie Basak <email address hidden> Fri, 22 Feb 2019 12:38:36 +0000

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for python-josepy has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-letsencrypt - 0.7.0-0ubuntu0.16.04.1

---------------
python-letsencrypt (0.7.0-0ubuntu0.16.04.1) xenial; urgency=medium

  [ Robie Basak ]
  * This update is part of the set of major updates moving Let's
    Encrypt/Certbot to version 0.23 in 16.04 in order to allow it to
    continue working following the general shutdown of TLS-SNI-01
    validation (LP: #1640978).
  * The new source package python-certbot takes over this package's
    functions, with binary packages certbot, python-certbot and
    python-certbot-doc taking over respectively.
  * The following two functional changes are additionally made:
    - Log rotation is switched to logrotate via
      /etc/logrotate.d/certbot, and /etc/letsencrypt/cli.ini is
      introduced to disable internal log rotation to avoid collision.
    - Automatic renewal is enabled via the certbot.timer and
      certbot.service systemd units.

  [ Michael Casadevall ]
  * Replace python-letsencrypt with a compatibility shim

 -- Robie Basak <email address hidden> Fri, 22 Feb 2019 12:42:44 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-acme - 0.22.2-1ubuntu0.1~16.04.1

---------------
python-acme (0.22.2-1ubuntu0.1~16.04.1) xenial; urgency=medium

  [ Robie Basak ]
  * This update is part of the set of major updates moving Let's
    Encrypt/Certbot to version 0.23 in 16.04 in order to allow it to
    continue working following the general shutdown of TLS-SNI-01
    validation (LP: #1640978).
  * This package is a dependency of Certbot and using the 0.22 version
    in Bionic is the most straightforward backport choice. Version 0.22
    of this package is sufficient to fulfil the dependency requirement
    for Certbot 0.23.

  [ Michael Casadevall ]
  * Backport to Xenial for LE change

 -- Robie Basak <email address hidden> Fri, 22 Feb 2019 12:41:07 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-letsencrypt-apache - 0.7.0-0ubuntu0.16.04.1

---------------
python-letsencrypt-apache (0.7.0-0ubuntu0.16.04.1) xenial; urgency=medium

  [ Robie Basak ]
  * This update is part of the set of major updates moving Let's
    Encrypt/Certbot to version 0.23 in 16.04 in order to allow it to
    continue working following the general shutdown of TLS-SNI-01
    validation (LP: #1640978).
  * The new source package python-certbot-apache takes over this
    package's functions.

  [ Michael Casadevall ]
  * Replace python-letsencrypt-apache with a compatibility shim

 -- Robie Basak <email address hidden> Fri, 22 Feb 2019 12:43:04 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-certbot - 0.23.0-1~ubuntu16.04.1

---------------
python-certbot (0.23.0-1~ubuntu16.04.1) xenial; urgency=medium

  [ Robie Basak ]
  * This update is part of the set of major updates moving Let's
    Encrypt/Certbot to version 0.23 in 16.04 in order to allow it to
    continue working following the general shutdown of TLS-SNI-01
    validation (LP: #1640978).
  * This new source package takes over the function of
    the previous source package python-letsencrypt, with binary packages
    certbot, python-certbot and python-certbot-doc taking over
    respectively.
  * The following two functional changes are additionally made:
    - Log rotation is switched to logrotate via
      /etc/logrotate.d/certbot, and /etc/letsencrypt/cli.ini is
      introduced to disable internal log rotation to avoid collision.
    - Automatic renewal is enabled via the certbot.timer and
      certbot.service systemd units.

  [ Michael Casadevall ]
  * Backport to Xenial

 -- Robie Basak <email address hidden> Fri, 22 Feb 2019 12:41:51 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-certbot-apache - 0.23.0-1~ubuntu16.04.1

---------------
python-certbot-apache (0.23.0-1~ubuntu16.04.1) xenial; urgency=medium

  [ Robie Basak ]
  * This update is part of the set of major updates moving Let's
    Encrypt/Certbot to version 0.23 in 16.04 in order to allow it to
    continue working following the general shutdown of TLS-SNI-01
    validation (LP: #1640978).
  * This new source package takes over the function of the previous
    source package python-letsencrypt-apache.

  [ Michael Casadevall ]
  * Backport to Xenial for LE change

 -- Robie Basak <email address hidden> Fri, 22 Feb 2019 12:42:16 +0000

[Robie Basak (racb)]

python-certbot-nginx (nor any python-letsencrypt-nginx) was never released on 16.04 so cannot regress. Michael didn't include it in his SRU work, so it didn't make this update and isn't expected to do so in the future either.