[SRU] Backport letsencrypt from bionic

[Peter Eckersley]

[Impact]

Certbot (formerly called Let's Encrypt, as released in Xenial) will stop working on 13 March 2019 when TLS-SNI-01 validation is turned off by the primary Let's Encrypt CA. This will make the package effectively useless for just about all users.

[Development Fix]

Newer validation options are present in the packages in Bionic onwards, including Disco.

[Stable Fix]

For Xenial, we are backporting the version of Certbot in Bionic.

Note that this update includes two important functional changes:

1) Automatic renewal is being enabled.

2) Log rotation is switching to being handled by logrotate.

See the discussion in this bug for details.

Since the upstream project has been renamed from "Let's Encrypt" to "Certbot" to better differentiate between the tooling and the CA, the /usr/bin/certbot command will become available. However, a compatibility symlink is provided under the old name /usr/bin/letsencrypt.

[Test Case]

Upstream have an extensive test suite and are participating in this SRU to help us validate and land it.

[Test Plan]

See https://wiki.ubuntu.com/StableReleaseUpdates/Certbot#SRU_Verification_Process and https://wiki.ubuntu.com/StableReleaseUpdates/Certbot/TestScript

In addition, we will test the upgrade path from the Xenial release pocket to proposed explicitly.

[Regression Potential]

The Certbot team has viewed breakage of existing workflows (especially ones that may be automated) as a serious issue, has strived to avoid them, and has treated workflow changes as regressions where it has occurred.

We have the following test suites in place for Certbot:

* Nosetest unit tests with coverage for each module between 97% and 100%; *test.py in the relevant tree.
* Integration tests that run Certbot against the current copy of Let's Encrypt's serverside boulder codebase. These require docker and are a little more involved to run. See tests/boulder_integration.sh for instructions.
* "Compatibility tests" that run the Apache and Nginx plugins against corpora of configuration files for those webservers; these live in certbot-compatibility-test/
* Test farm tests, which we use to check that our releases run correctly on a wide range of platforms. These spin up Amazon EC2 instances for numerous OSes and run various tests on them. They live in tests/letstest

We recommend that Ubuntu run the first of these test suites during build (but we believe the Debian packages already do that).

All of these tests mitigate the risk of regressions in our releases; nonetheless, some regressions do slip past. Because many of our users auto-update, these tend to be reported and fixed quickly in point releases. For instance, regressions in 0.9.0 were fixed in 0.9.1, 0.9.2 and 0.9.3. Certbot 0.9.3 has been used to issue hundreds of thousands of Certs in the field, so we are fairly confident that no further significant regressions exist in it, and that release is likely to be safe as a Xenial SRU.

At least two changes in functionality between 0.4.1 and 0.9.3 do bear specific consideration for Xenial though:

Debian has added a "certbot renew" twice-daily cron job to their packages between 0.4.1 and 0.9.3; we believe this is low regression risk (having secondary renewal mechanisms in place is a NOOP) but Xenial packages may want to increase the debconf verbosity to get consent for this from Xenial users who are upgrading?

We had a custom log rotation scheme (rotate logs after every run), we now act like a more typical daemon, so packages need to be rotating our logs:
https://github.com/certbot/certbot/issues/3382

[Peter Eckersley (pde-lists)]

Also fixes: Launchpad bug #1608214

[Peter Eckersley (pde-lists)]

On IRC, rbasak asked:

<rbasak> From an SRU review perspective, what I most want spelled out the exact list of any behaviour changes you expect Xenial users to receive, and your confirmation that you don't believe that there are any other changes.
<rbasak> Your word as upstream carries a great deal of weight on this.
<rbasak> What you've written is great and sound like it covers most of this already - it's just a clear summary of what the final upload to Xenial will do is what I feel is missing - though of course I know you haven't finalised that yet.

Debian's certbot 0.9.3 packages are a close approximation to what we expect users to get. They Provide: and Replace: letsencrypt, and include aliases for all previous "letsencrypt" commands.

With the exception of the two (an auto-renewal cron job and slightly different log rotation) points noted above, we believe that the functionality and workflows supported by certbot 0.9.3 should be an but less buggy, super-set of the functionality and workflows supported by letsencrypt 0.4.1.

[Peter Eckersley (pde-lists)]

* "a compatible but less buggy"

I should also note that most of our users have been running the certbot-auto script which automatically upgrades them to the latest release of Certbot as soon as it's public, so the odds of us getting bug reports about workflow changes is fairly high.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Harlan Lieberman-Berg (hlieberman)]

Hello everyone -- I'm (the primary, but not only) Debian maintainer of the certbot suite of packages. pde asked me to weigh in here to help answer some of rbasak's questions in a definitive manner.

Since the 0.4.1 release, there have been a handful of user-facing changes, but only a few are workflow changing ones.

The largest is the change in package name from letsencrypt to certbot. This was done for trademark reasons, however, we continue to ship both a transitional package and a symlink for the binary that should preserve users' workflows unmodified.

The second largest change is the addition of an automatic renewal of any certificates, through both a cronjob installed into cron.d and a systemd timer. The certbot binary will only act to renew the certificate, however, when it is "near" to expiring (<15 days). Users who have manually setup a cron already shouldn't experience any failures.

Other than that, there is new functionality available in upstream, but pde has already mentioned all of those. I'm happy to answer any questions or concerns the stable release managers have.

[Peter Eckersley (pde-lists)]

@hlieberman My understanding is that the package name change shouldn't affect any workflows, because "sudo apt-get install letsencrypt" will still work and the "letsencrypt" command will still work. Anything I might have missed?

[Robie Basak (racb)]

Hi Harlan,

Thank you for taking the time to help us out!

On Sun, Nov 20, 2016 at 02:47:14AM -0000, Harlan Lieberman-Berg wrote:
> The second largest change is the addition of an automatic renewal of any
> certificates, through both a cronjob installed into cron.d and a systemd
> timer.

Does this mean that users who currently won't get auto-renewed will
start getting auto-renewed after this proposed update? If so, will that
also include users who currently have expired (languishing) old
certificates?

All,

Is there any progress on the preparation of the update itself? I hope
you're not waiting on my approval. To be clear, my opinion is that
we should update letsencrypt/certbot in 16.04, and I appreciate all the
interest in helping us do this; it's just a matter of the details
(documenting, understanding and mitigating any behavioural changes as
appropriate, and agreeing on the QA plan).

Robie

[Peter Eckersley (pde-lists)]

> Does this mean that users who currently won't get auto-renewed will
> start getting auto-renewed after this proposed update? If so, will that
> also include users who currently have expired (languishing) old
> certificates?

Yes. Each certificate (or more precisely, each lineage of certificates, where a lineage is a series of certificates that replace each other with new validity dates and possibly new domains added) gets a renewal configuration file in /etc/letsencrypt/renewal/ ; the "certbot renew" command walks through those and tries to renew any that are within 30 days of expiry. The Debian packages run that task twice a day out of the box.

I think we've concluded that we'll add a note-upon-installation telling the sys admin that that's going to start happening, and point to where it can be turned off or tweaked.

[Robie Basak (racb)]

On Mon, Nov 28, 2016 at 09:22:28PM -0000, Peter Eckersley wrote:
> I think we've concluded that we'll add a note-upon-installation telling
> the sys admin that that's going to start happening, and point to where
> it can be turned off or tweaked.

I'm afraid that users who automatically update (most of them in 16.04
since it's default) will never see such a message.

[Michael Heuberger (michael.heuberger)]

While you do this for v16.04, could you also publish certbot v0.9.x for v16.10 at the same time? Or need a new ticket for that?

[Harlan Lieberman-Berg (hlieberman)]

Hello everyone.

I'm not sure exactly what form you wanted the SRU in, but... I've included a tarball of the source changes files and dsc's, signed, along with the files included in them.

This upload has two changes from the version in sid. First, I tweaked the versions in python-certbot to reduce the dependency on pyopenssl. The version requirement was to work around an undeclared Breaks relationship between a version of pyopenssl and python-cryptography in a later version (16.0.0) than is in Xenial.

Second, I added a new dummy package to assist the upgrade of python-letsencrypt-apache => python-certbot-apache. The dummy package was never used in Debian, as letsencrypt didn't make it to stable before the name-change occurred.

Please let me know if you have any questions, or if you would like the SRU in some other manner.

[Robie Basak (racb)]

The format sounds fine, thanks. We need one uploader and SRU team member to review. I suggest we figure out who is doing what and both people review before uploading to avoid confusion.

Since you mentioned that RAOF could sponsor, I subscribed him to the bug. Since both of us are on the SRU team, we should figure out who will wear which hat. RAOF, any comments?

[Chris Halse Rogers (raof)]

I'm happy to do either.

[Peter Eckersley (pde-lists)]

>>> import random
>>> x = ["RAOF", "rbasak"]
>>> random.shuffle(x)
>>> print "{0} should upload and {1} should review".format(x)
RAOF should upload and rbasak should review

[Peter Eckersley (pde-lists)]

.format(*x), sorry :)

[Peter Thomassen (mail-peter-thomassen)]

Will it be reasonable to wait for certbot 0.10.0 (due on Jan 4, according to https://github.com/certbot/certbot/milestones), and go ahead then?

[Peter Eckersley (pde-lists)]

I don't think so. We would want 0.10.0 to have at least several weeks of field testing before stable distributions include it as an update, and any regressions in that release would mean we do 0.10.x bugfix releases that would stretch the timeline further.

[Peter Thomassen (mail-peter-thomassen)]

Those are good arguments, thanks for explaining!

[Chris Halse Rogers (raof)]

Hm. It was unclear to me that this requires updating python-acme, python library expected to be used by 3rd parties.

Have any python-acme APIs changed? What are the chances that the update will break downstream users of python-acme?

[Peter Eckersley (pde-lists)]

The changes in python-acme were comparatively minor. A few bugs were fixed; one new feature was added to the API (support for DNS-01 challenges); some protocol messages were removed (because they were believed to have security problems, were never used by Let's Encrypt, and were removed from the IETF ACME draft).

None of those changes should negatively impact non-Certbot ACME clients that use python-acme. It also does not appear that any of those ACME clients are packaged in Debian unstable:

apt-cache rdepends python-acme
python-acme
Reverse Depends:
  python-certbot
  python-certbot-nginx
  python-certbot-apache

[Chris Halse Rogers (raof)]

It's the clients that are *unpackaged* that are the most concerning. Packaged clients we can update if necessary; random ad-hoc clients not so much 😀

[Peter Eckersley (pde-lists)]

* nod. We haven't changed any of the preexisting API calls or semantics, so even unpackaged clients should be fine.

[Chris Halse Rogers (raof)]

Packages uploaded to the unapproved queue, awaiting review.

I changed the version numbers (so that the version in Xenial < Yakkety < Zesty), and added bug links.

[Robie Basak (racb)]

I'm confused on a couple of points. Here's my understanding:

AFAICT, there are three source packages in play here: src:python-acme, src:python-letsencrypt, and src:python-certbot.

src:python-acme appears relatively straightforward. A version bump is needed for new features used by this proposed update, but there is no other package renaming going on.

src:python-certbot supersedes src:python-letsencrypt, though of course apt doesn't see that.

src:python-letsencrypt created binaries letsencrypt, python-letsencrypt and python-letsencrypt-doc in Xenial. It doesn't exist in Yakkety or Zesty.

src:python-certbot creates binaries letsencrypt (transitional to certbot), certbot, python-certbot and python-certbot-doc from Yakkety onwards, and doesn't currently exist in Xenial.

The proposed backport of src:python-certbot to Xenial makes some backport-related dependency changes (so it seems) but I don't see anything that changes the structure of package names or transitional packages from what is in sid (and Yakkety, etc).

So to my questions:

1) What happens to a Xenial user who is using the library provided by python-letsencrypt after this update? Is it intentional that src:python-letsencrypt in Xenial will continue to provide it?

2) This is more of a question for an archive admin. Is it acceptable in Xenial for src:python-certbot to take over the letsencrypt binary name, but for us to retain src:python-letsencrypt that also produces it?

3) What happens if a future update (say a security update) is needed against src:python-letsencrypt for users still using the python-letsencrypt binary package in Xenial (from question 1)? This update would still produce the letsencrypt binary, but we'd have a higher version from src:python-certbot in the archive. So would the subsequent binary upload fail?

Does this mean that we also need to SRU src:python-letsencrypt at the same time to stop producing the letsencrypt binary package at the same time?

This is for Xenial only. I haven't considered Yakkety yet, nor have I completed my review since I want to understand and fix this point before considering any other further interactions (which may change if we change how we're doing this here).

[Robie Basak (racb)]

Based on https://irclogs.ubuntu.com/2017/01/12/%23ubuntu-release.html#t13:47, can you upload a python-letsencrypt for Xenial that drops the letsencrypt package please? We'll need to accept these all together.

Please remember to run the update-maintainer field when preparing these source packages. See https://wiki.ubuntu.com/DebianMaintainerField for the background. I think this needs doing on some of the existing uploads in the queue also, but let's wait to see if anything else needs fixing.

[Peter Eckersley (pde-lists)]

I think it's my job to gently nudge whoever it is that's making the slightly tweaked packages here, but I'm not sure whether it should be Chris or Harlan... if all else is equal perhaps it should be Chris, since we also have a new 0.10.1 release that needs to be packaged for sid and that would be in Harlan's wheelhouse.

[Robie Basak (racb)]

Any news? I feel that changing python-letsencrypt as well exceeds the bounds of the SRU hat I'm wearing - it should have review from a different uploader as well.

[Chris Halse Rogers (raof)]

Sorry; LCA and then email backlog. I'll upload new packages tomorrow.

[Peter Eckersley (pde-lists)]

That's great Chris!

We had some discussions yesterday about whether this has taken long enough that 0.10.x should be considered for the SRU instead. 0.10.1 has been in the field for 13 days; people found a couple of bugs in it, but they only affected new 0.10.x functionality rather than being regressions.

0.10.2 was released yesterday to fix those things and Harlan is working to get it into Debian in time to make the stretch freeze 11 days from now.

Our conclusion was the best thing to do is probably make the 0.9.3 SRU now, and start work on a 0.10.2+ SRU in around 3 weeks, once the 0.10.2 debian packages have received some testing and we've had time to sweep for any final bugs.

[Robie Basak (racb)]

Can I suggest that you maintain these packages in a git tree somewhere (if you aren't doing so already), create branches for the Ubuntu stable releases you're targeting, and then get reviews straight from there? I'll be happy to do an SRU review straight from a git branch. Then review changes could get committed there presumably by more people, so we could move faster. Chris will just need to review new commits that land since his last review, and upload when everyone is happy. And I can allow through any upload after just verifying that it is identical to the thing I already reviewed and accepted in git.

It'll save confusion with multiple versions in the queue this way, too.

[Peter Eckersley (pde-lists)]

Harlan and the other Debian developers have git trees here:

https://alioth.debian.org/plugins/scmgit/cgi-bin/gitweb.cgi?a=project_list;pf=letsencrypt

Maybe Chris should also get access to those repos?

[Chris Halse Rogers (raof)]

python-letsencrypt dropping the letsencrypt binary uploaded.

If you'd like me to have access to the git repos you'll need to add raof-guest to the letsencrypt group (or, alternatively, make the git repository accessible to collab-maint).

[Robie Basak (racb)]

python-letsencrypt and python-letsencrypt-apache don't exist in Yakkety or Zesty, so let's mark those Invalid as there is no action to take for these packages neither in Yakkety nor Zesty (only Xenial).

[Robie Basak (racb)]

Setting all tasks for which I see an upload in the queue as "In Progress" so I can more easily see what's what. Here's my current view of the queues:

python-acme
queue/xenial/unapproved/cb94bae
queue/yakkety/unapproved/15d7282

python-certbot
queue/xenial/new/e67c69a
queue/yakkety/unapproved/c6e7c0b

python-certbot-apache
queue/xenial/new/a42f913
queue/yakkety/unapproved/b496663

python-certbot-nginx
queue/xenial/new/b15997a
queue/yakkety/new/480f5d3

python-letsencrypt
queue/xenial/unapproved/21d1a2f

python-letsencrypt-apache
[empty]

[Robie Basak (racb)]

No changes needed to python-letsencrypt-apache: this will be left as-is in Xenial, will use python-letsencrypt which is only being changed to drop the letsencrypt package for the transition (libraries stay all the same).

[Robie Basak (racb)]

I've verified all the versions of the queue assuming we are backporting in SRUs <= 0.10.2-1 of python-{acme,certbot{,-apache,-nginx}} only and leaving behind python-letsencrypt and python-letsencrypt-apache. So let's retitle for exactly this.

[Robie Basak (racb)]

I spent some time on reviewing this on Friday, but I got interrupted. There was also some discussion on IRC (https://irclogs.ubuntu.com/2017/01/27/%23ubuntu-release.html#t12:31)

My notes:

Can we put a note that there will be a new cronjob active now in debian/NEWS and debian/changelog as suggested on IRC?

What happens if user has installed letsencrypt outside of packaging (for example, to get a newer version while leaving the system one installed)? Will adding a cronjob in the SRU cause any issues?

Can we nominate a person or a team to handle bugs as a consequence of this SRU, and subscribe to package bugs to make sure?

update-maintainer needs running (but I can do this from the queue if no other changes are needed).

[Peter Eckersley (pde-lists)]

This has been stuck for a while, I suspect because it hasn't been clearly on anyone's plate :(.

Let's fix that:

* Brad Warren on the Certbot team is going to construct a retrospective changelog.txt, and post a link here.
* RAOF should probably revise the packages to include that and the news file.
* Anything else?

Separately, this has taken so long that it might be advisable to switch to an 0.10.2 SRU, since that version is now well-tested, and is also the one that's frozen into Debian Stretch. Thoughts for and against? What extra steps would we need for an 0.10.2 SRU?

[Robie Basak (racb)]

I'd like to suggest again that everyone involved does all packaging changes in git, and gets all approvals (from a suitable Ubuntu uploader and the Ubuntu SRU team) in git, before uploading. I really think this will speed things up.

I'm not sure I see everything in https://alioth.debian.org/plugins/scmgit/cgi-bin/gitweb.cgi?pf=letsencrypt, though that may be because repositories have not been renamed following source package renames.

I'm not too bothered about who can push where. Since git is nicely distributed, as long as we can all get to the "current" branch, it shouldn't matter as long as we all agree on where that "current" branch is (as it changes).

RAOF, if you're struggling for time, let me know and I'll see if a colleague on the server team is available to help (since I can only wear the one hat).

[Brad Warren (bradmwarren)]

Here's a link to the changelog:

https://github.com/certbot/certbot/blob/master/CHANGELOG.md

[Michael Casadevall (mcasadevall)]

Breakdown of python-certbot changes; this is a NEW package going into trusty backported from bionic following the letsencrypt->certbot name change. A revised debdiff is attached to this bug.

diff -Nru python-certbot-0.23.0/debian/compat python-certbot-0.23.0/debian/compat
--- python-certbot-0.23.0/debian/compat 2018-03-23 23:00:13.000000000 +0000
+++ python-certbot-0.23.0/debian/compat 2018-11-25 16:20:44.000000000 +0000
@@ -1 +1 @@
-11
+9

Explanation: Debhelper compat change

diff -Nru python-certbot-0.23.0/debian/control python-certbot-0.23.0/debian/control
--- python-certbot-0.23.0/debian/control 2018-04-07 03:21:05.000000000 +0000
+++ python-certbot-0.23.0/debian/control 2018-11-25 16:20:44.000000000 +0000
@@ -1,47 +1,41 @@
 Source: python-certbot
 Section: python
 Priority: optional
-Maintainer: Debian Let's Encrypt <email address hidden>
+Maintainer: Ubuntu Developers <email address hidden>
+XSBC-Original-Maintainer: Debian Let's Encrypt <email address hidden>
 Uploaders: Harlan Lieberman-Berg <email address hidden>,
            Francois Marier <email address hidden>
-Build-Depends: debhelper (>= 11~),
+Build-Depends: debhelper (>= 9~),
                dh-python,
- python3,
- python3-acme (>= 0.22.0~),
- python3-configargparse (>= 0.10.0),
- python3-configobj,
- python3-cryptography (>= 1.2),
- python3-distutils,
- python3-josepy,
- python3-mock,
- python3-parsedatetime (>= 1.3),
- python3-repoze.sphinx.autointerface,
- python3-requests (>= 2.4.3),
- python3-rfc3339,
- python3-setuptools (>= 1.0),
- python3-sphinx,
- python3-sphinx-rtd-theme,
- python3-tz,
- python3-zope.component,
- python3-zope.interface
+ python,
+ python-acme (>= 0.22.0~),
+ python-configargparse (>= 0.10.0),
+ python-configobj,
+ python-cryptography (>= 1.2),
+ python-josepy,
+ python-mock,
+ python-parsedatetime (>= 1.3),
+ python-requests (>= 2.4.3),
+ python-rfc3339,
+ python-setuptools (>= 1.0),
+ python-sphinx,
+ python-sphinx-rtd-theme,
+ python-tz,
+ python-zope.component,
+ python-zope.interface
 Standards-Version: 4.1.4
 Homepage: https://certbot.eff.org/
 Vcs-Git: https://salsa.debian.org/letsencrypt-team/certbot/certbot.git
 Vcs-Browser: https://salsa.debian.org/letsencrypt-team/certbot/certbot
 Testsuite: autopkgtest-pkg-python

Explanation: Some of the dependencies for certbot do not have Python3 versions in Xenial; this was resolved by having certbot use Python2 and becoming a python2 package; it could also have been resolved by SRUing python-parsedtimedate as it supports Python 3 but doesn't have the necessary deb. An SRU for the bionic package may be needed to remove this with the Python 3 version.

-Package: python3-certbot
+Package: python-...

[Michael Casadevall (mcasadevall)]

After testing and further work, a revised python-acme debdiff is attached. This includes one additional backport that allows josepy to be imported through the acme package; this change has already been integrated upstream, and is required to allow packages that assume acme.jose still existed as part of the python-acme module.

[Michael Casadevall (mcasadevall)]

This is the backported patch to python-acme, as stated before, this is a direct backport from upstream to handle compat issues, and extends the unittests to cover the new backworks compatibility area.

mcasadevall@lighthouse:~/src/le/sru/python-acme-0.22.2/debian/patches$ cat fix-jose-import
Description: Allow josepy to be imported via acme.jose
 This is a backwards compatibility fixed taken from upstream
 from the following commits:
 https://github.com/certbot/certbot/commit/e3cb782e5992ba306de59ba96dfb6f125720cd06.patch
 https://github.com/certbot/certbot/commit/ec297ccf72e95961586ec2382c3e3225ce578aa4.patch
 .
 python-acme (0.22.2-1ubuntu0.16.04.1~ppa4) xenial; urgency=medium
 .
   * Backport to Xenial for LE change (LP: #1640978)
Author: Michael Casadevall <email address hidden>
Bug-Ubuntu: https://bugs.launchpad.net/bugs/1640978

---
The information above should follow the Patch Tagging Guidelines, please
checkout http://dep.debian.net/deps/dep3/ to learn about the format. Here
are templates for supplementary fields that you might want to add:

Origin: upstream
Bug: https://github.com/certbot/certbot/pull/6592
Forwarded: not-needed
Reviewed-By: Michael Casadevall <email address hidden>
Last-Update: 2019-01-11

--- python-acme-0.22.2.orig/acme/__init__.py
+++ python-acme-0.22.2/acme/__init__.py
@@ -10,3 +10,18 @@ supported version: `draft-ietf-acme-01`_
   https://github.com/ietf-wg-acme/acme/tree/draft-ietf-acme-acme-01

 """
+import sys
+
+# This code exists to keep backwards compatibility with people using acme.jose
+# before it became the standalone josepy package.
+#
+# It is based on
+# https://github.com/requests/requests/blob/1278ecdf71a312dc2268f3bfc0aabfab3c006dcf/requests/packages.py
+
+import josepy as jose
+
+for mod in list(sys.modules):
+ # This traversal is apparently necessary such that the identities are
+ # preserved (acme.jose.* is josepy.*)
+ if mod == 'josepy' or mod.startswith('josepy.'):
+ sys.modules['acme.' + mod.replace('josepy', 'jose', 1)] = sys.modules[mod]
--- /dev/null
+++ python-acme-0.22.2/acme/jose_test.py
@@ -0,0 +1,53 @@
+"""Tests for acme.jose shim."""
+import importlib
+import unittest
+
+class JoseTest(unittest.TestCase):
+ """Tests for acme.jose shim."""
+
+ def _test_it(self, submodule, attribute):
+ if submodule:
+ acme_jose_path = 'acme.jose.' + submodule
+ josepy_path = 'josepy.' + submodule
+ else:
+ acme_jose_path = 'acme.jose'
+ josepy_path = 'josepy'
+ acme_jose_mod = importlib.import_module(acme_jose_path)
+ josepy_mod = importlib.import_module(josepy_path)
+
+ self.assertIs(acme_jose_mod, josepy_mod)
+ self.assertIs(getattr(acme_jose_mod, attribute), getattr(josepy_mod, attribute))
+
+ # We use the imports below with eval, but pylint doesn't
+ # understand that.
+ # pylint: disable=eval-used,unused-variable
+ import acme
+ import josepy
+ acme_jose_mod = eval(acme_jose_path)
+ josepy_mod = eval(josepy_path)
+ self.assertIs(acme_jose_mod, josepy_mod)
+ self.assertIs(getattr(acme_jose_mod, attribute), getattr(josepy_...

[Fabien COMBERNOUS (fc.)]

when will new package arrive on the repositories ?

This update is important since the TLS-SNI-01 validation option is soon turned off by LE.

TLS-SNI-01 validation is reaching end-of-life. It will stop working
temporarily on February 13th, 2019, and permanently on March 13th, 2019.

[Robie Basak (racb)]

I'm reviewing this now.

Michael, did you miss the fix from Bionic uploaded in 0.22.2-1ubuntu0.1?

  * Add ready status type to be compatible with the new Let's Encrypt ACMEv2
    endpoint (LP: #1777205).

From the perspective of current Bionic it looks like this is getting dropped in the backport, which I think is by accident?

Though I'd ask now, before I've finished my review (this is taking a while!), to speed things up if we find a change is needed.

[Robie Basak (racb)]

This is on python-acme, sorry.

[Robie Basak (racb)]

python-certbot backport review:

The changes you've described look good in general, but I have some specific questions for pieces for which I couldn't find any explanation:

python-certbot no longer Recommends certbot, but python3-certbot did. Is this intentional?

In debian/rules, what's the reason for including pkg-info.mk into the backport as a delta? I don't see the variables it defines being used anywhere.

What's the reason for the new clean target in debian/rules?

Why is certbot.timer's installation being removed? Has auto renewal been tested? If so, what's the mechanism it uses now?

Why remove the installation of the cli.ini that disables certbot-internal log rotation?

Some notes to self explaining further bits of the delta I did manage to understand:

python3-distutils build-dep drop not explained but distutils is integrated in Python 2 in Xenial and not in its own packaging so dropping the build dependency is presumably intentional.

The dropping of the python3-repoze.sphinx.autointerface build dependency is explained later with the disable-autointerface-docs patch.

[Robie Basak (racb)]

python-letsencrypt backport review:

I think debian/README.source is now redundant and incorrect and should be removed?

I see that pkg-info.mk was included from Debian previously. Is this still necessary?

debian/rules arranges debian/letsencrypt/usr/bin/* but no longer ships a binary package called letsencrypt.

python-letsencrypt-apache backport review:

I think debian/README.source is now redundant and incorrect and should be removed?

[Robie Basak (racb)]

Test plan:

Could we agree a specific test plan for this SRU please, in addition to the usual testing agreed in https://wiki.ubuntu.com/StableReleaseUpdates/Certbot? Specifically there are a few things that are special about this one. I'd like to make sure that somebody makes sure that all the gotchas of which we've already become aware and fixed are still fixed in the SRU verification step, before the update is pushed out for automatic update by users.

This includes:

1. letsencrypt rename compatibility, including the CLI, letsencrypt and letsencrypt-apache Python modules.

2. Log rotation behaviour

3. Automatic renewal behaviour

4. Bug 1777205 and its patch

Once agreed we can update the Test Plan section of the bug description.

[Brad Warren (bradmwarren)]

I updated and added additional checks to the test script at https://wiki.ubuntu.com/StableReleaseUpdates/Certbot/TestScript including tests for all four of the areas Robie flagged in his most recent post.

[Robie Basak (racb)]

Thanks Brad!

As Michael seems to be unavailable at the moment I'll try to fix my review comments up myself. I've asked my colleague Christian to peer-review my work from this point to fulfil Ubuntu SRU review requirements.

What I don't know the answer to is some of my questions from my reviews above. These are in comments 99, 101 and 102.

I assume that the patch from 0.22.2-1ubuntu0.1 needs to be included.

I'm going to drop the bits I think are packaging code skeletons.

Two questions I'm really uncertain about right now:

Why is certbot.timer's installation being removed? Has auto renewal been tested? If so, what's the mechanism it uses now?

Why remove the installation of the cli.ini that disables certbot-internal log rotation?

I'd appreciate any insight into these please. What were the mechanisms for auto renewal and log rotation in 0.4, what are they in 0.22, and what should packaging be doing in 0.22?

I've asked Harlan for upload access to the ~certbot PPA on IRC. I hope to have updated packages uploaded there soon for further testing.

[Brad Warren (bradmwarren)]

> I assume that the patch from 0.22.2-1ubuntu0.1 needs to be included.

Yes, I think this patch should be included. The packages in 18.04 included support Let's Encrypt's newer endpoint, however, this feature is broken without this patch.

Thanks for catching this. The updated tests now test against the new API as well.

The other questions here were largely answered in IRC, but to keep this thread here up-to-date and maybe add a little more context...

> Why is certbot.timer's installation being removed?

I think removing this was a mistake. The current Ubuntu 16.04 package does not configure automatic renewal, but I think it should be added as it benefits users and keeps the package closer to the one found in Bionic.

There's actually a lot of discussion about the addition of automatic renewal earlier in this thread from a couple years ago.

> Has auto renewal been tested?

I suspect not. At the very least, I hadn't done so.

Previously my test script was almost exclusively running our upstream tests on the installed packages and did not do much to test things specifically found in the .deb packages.

For testing automatic renewal, however, the updates to the script I made last week check:

1. certbot.timer exists and is enabled.
2. /etc/cron.d/certbot exists.

> Why remove the installation of the cli.ini that disables certbot-internal log rotation?

I think we probably should keep this file unless there is a good reason not to do so. Why make unnecessary modifications to the packages?

In the current Xenial packages, Certbot is using its own log rotation provided by the Python standard library. Each run of Certbot creates a new log file in /var/log/letsencrypt and up to 10 log files are created after which old log files are deleted.

The cli.ini file in the Bionic package turns off this log rotation causing Certbot to always append to the same file at /var/log/letsencrypt/letsencrypt.log with the thinking that this will better play with programs like logrotate.

[Robie Basak (racb)]

I've pulled Michael's current work into git trees available at: https://code.launchpad.net/~certbot/+git (each has a branch called lp1640978). Please maintain these branches as fast-forwarding only.

I have updated python-acme, python-certbot, python-letsencrypt and python-letsencrypt-apache based on my review comments above, bumped the ppa suffix versions, tweaked the version strings as needed, pushed to the git branches, and uploaded to the xenial-sru PPA.

Next, I need to check the PPA versions work as expected. Then I'll get a colleague's review. If all is good, we will be able to upload and accept into xenial-proposed.

[Brad Warren (bradmwarren)]

I tried manually installing packages from the PPA and running https://wiki.ubuntu.com/StableReleaseUpdates/Certbot/TestScript but the script failed because the Certbot systemd timer wasn't found.

[Robie Basak (racb)]

On Wed, Feb 13, 2019 at 05:26:38PM -0000, Brad Warren wrote:
> I tried manually installing packages from the PPA and running
> https://wiki.ubuntu.com/StableReleaseUpdates/Certbot/TestScript but the
> script failed because the Certbot systemd timer wasn't found.

Thanks. I'll follow up.

I also found that I missed review of python-certbot-apache, so I'll get
on to that also.

[Robie Basak (racb)]

I've fixed that, updated the git repositories under ~certbot, and re-uploaded to the PPA. I've included some other fixes as well. Full changes are available to view in the git repositories.

Christian, please could you review all changes in the range review/2019-02-08..lp1640978 for the packages python-josepy, python-acme, python-letsencrypt, python-letsencrypt-apache, python-certbot and all commits but the root commit in python-certbot-apache? That should give us full review coverage then.

[Brad Warren (bradmwarren)]

Test script passes on the new packages in the PPA.

[Christian Ehrhardt  (paelzer)]

TL;DR
- LGTM a few questions for clarification, but no nacks

----

I took logs for everything non trivial that I did (wonder about) on review.
+ = LGTM
? = ok, but I'd have a question
x = nack, this should be different

python-acme:
? rebase for 0.22.2-1ubuntu0.1
  - The bug #1777205 should get a xenial task before upload
  - and then the bug be verified on the SRU
  - please remember to buildpkg with -v0.4.1-1
+ changelog tweaks
+ backport version string
  Double ubuntu in the version seems odd, but it seems correct.
  It is not part of [2] and an even more special case of [1]
x Add SRU explanation to changelog
  The version is 0.22.2 the entry calls it "version 0.23 in 16.04"
  Seems wrong, either fix or explain why it is that way.
  I see that all others e.g. python-certbot is at 0.23 for real
  will that mismatch be a problem?

python-certbot:
+ recommends certbot
? dh_clean ok but why, would it hurt?
? pkg-info.mk ok but why, would it hurt?
? cli.ini
  That is the one to keep the logrotations as is (ok)
  I thought from our talk Xenial did that as well, but it didn't
  There is nothing about it in pkg/ubuntu/xenial-devel:debian/rules
  nor in pkg/import/0.14.2-1:debian/rules
  I guess the old default was different, which is ok.
  But do we now want/need any conffile cleanup in >=Bionic?
  This would be independent to this SRU, but I wanted to ask.
+ build rules
+ install init no-op
? Restore original timer service installation
  The "original" used the older dh_systemd_enable/dh_systemd_start
  I think your's work only with newer debhelper, it is not "the original"
  I just found you clean that up later - would have been nice to squash those two :-/
+ backport version string (that one I like more)
? use dh_systemd
  The old package used install -D --mode=644 for the timer
  any reason you have chosen mkdir + cp instead ?
  That commit also adds whitespace damage above "# Install certbot.timer"
+ doc paths
+ version/changelog changes
? Add SRU explanation to changelog
  you might mention changes to d/rules and d/compat to make the backport working here

python-letsencrypt:
+ Remove redundant incorrect d/README.source
? further changes to changelog ( I have a general statement about those below)
? It is not clear to me how that "shimming" takes place, extending the changelog on that would be useful as well.

python-josepy:
? changelog/version bumps in general ok
  You mentioned the new certbot.timer and such in python-letsencrypt (which doesn't hold the timers)
  Please choose:
  - Why did you mention it there?
  - Why didn't you mention it here?
  I assume it is because of the package takeover letsencrypt->certbot.
  But really, that could use some words (more about that below already)

python-letsencrypt-apache
+ cleanup
+ changelog
+ version bumps

python-certbot-apache:
+ empty dir
+ VCS
+ cleanup (again ok but why)
+ doc building
+ version string changes
+ doc-base paths
+ SRU explanation (nothing more than what was said already)

I just realized that the summary changelogs all say: "Log rotation is switched to logrotate via /etc/logrotate.d/certbot"
This whole section could need more why it was done and where the old/new tunable...

[Robie Basak (racb)]

Thank you for the review! I'll look in detail later, but in case it speeds up an iteration, note that I've started from the perspective that this is a backport from Bionic - so the cleanups are on the principle of reducing the diff against Bionic. Where I saw a diff hunk that didn't seem necessary, that's what I removed in my cleanup commits.

[Robie Basak (racb)]

Thank you for the review! I've addressed review comments inline. With
Christian sitting next to me we've been through my fixes and he's happy
with them.

I've updated git branches and updated the PPA for python-certbot ~ppa7
only, as that's the only one that contains a functional change from
review feedback; the others are changelog clarifications only. I will
test this ~ppa7.

Assuming all is good, what's in the PPA, except for changelog text
adjustments, will then go into the Xenial proposed pocket ready for
wider testing.

> TL;DR
> - LGTM a few questions for clarification, but no nacks
>
> ----
>
> I took logs for everything non trivial that I did (wonder about) on review.
> + = LGTM
> ? = ok, but I'd have a question
> x = nack, this should be different
>
> python-acme:
> ? rebase for 0.22.2-1ubuntu0.1
> - The bug #1777205 should get a xenial task before upload
> - and then the bug be verified on the SRU
> - please remember to buildpkg with -v0.4.1-1

Upstream confirmed that 1777205 never affected Xenial (except perhaps in
proposed) so a task there would not be accurate. However I agree that
it's prudent to verify that it doesn't regress in this SRU during SRU
verification, so it's part of the test plan I proposed in comment 103
and I will copy that up to the bug description.

> + changelog tweaks
> + backport version string
> Double ubuntu in the version seems odd, but it seems correct.
> It is not part of [2] and an even more special case of [1]

I decided to stick to backportpackage(1)'s behaviour as a de facto
standard for these SRUs, as ugly as the result is in this case :-/

> x Add SRU explanation to changelog
> The version is 0.22.2 the entry calls it "version 0.23 in 16.04"
> Seems wrong, either fix or explain why it is that way.
> I see that all others e.g. python-certbot is at 0.23 for real
> will that mismatch be a problem?

Changelog updated with an explanation in f4de847.

> python-certbot:
> + recommends certbot
> ? dh_clean ok but why, would it hurt?
> ? pkg-info.mk ok but why, would it hurt?

These two changes minimise the diff against the backport source.

> ? cli.ini
> That is the one to keep the logrotations as is (ok)
> I thought from our talk Xenial did that as well, but it didn't
> There is nothing about it in pkg/ubuntu/xenial-devel:debian/rules
> nor in pkg/import/0.14.2-1:debian/rules
> I guess the old default was different, which is ok.
> But do we now want/need any conffile cleanup in >=Bionic?
> This would be independent to this SRU, but I wanted to ask.

cli.ini is a new conffile that was introduced at some point in the
certbot binary package only. AFAICT it never existed in the letsencrypt
binary package, or at least not in the one in Xenial. We ship the
conffile to turn off certbot's internal log rotation to avoid collision
with logrotate behaviour.

Since we're backporting from Bionic, this conffile will be introduced
into Xenial as part of this SRU.

The conffile continues to exist in the current development release, so I
don't think any further or special conffile handling is needed, since
we're only introducing it and not doing anything else with it.

As discussed I'v...

[Robie Basak (racb)]

FTR, I spotted a couple of things that turned out to be non-issues:

1) The letsencrypt binary has become a symlink and moved to the certbot binary package. This needs a Breaks/Replaces, which does exist but doesn't cover the version of letsencrypt currently in xenial-proposed. However this isn't a problem because the movement had happened prior to the version in xenial-proposed.

2) We're still shipping /etc/cron.d/certbot as it is in the Bionic backport source. However it is inert, as intended, since Ubuntu >= Xenial always has systemd (unless you're upgrading from Trusty but haven't rebooted yet). Even if the cron job did run, it wouldn't be a problem.

[Adam Conrad (adconrad)]

Hello Peter, or anyone else affected,

Accepted python-josepy into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-josepy/1.1.0-1~ubuntu16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Robie Basak (racb)]

Hello Peter, or anyone else affected,

Accepted python-acme into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-acme/0.22.2-1ubuntu0.1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Robie Basak (racb)]

Hello Peter, or anyone else affected,

Accepted python-certbot into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-certbot/0.23.0-1~ubuntu16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Robie Basak (racb)]

Hello Peter, or anyone else affected,

Accepted python-certbot-apache into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-certbot-apache/0.23.0-1~ubuntu16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Robie Basak (racb)]

Hello Peter, or anyone else affected,

Accepted python-letsencrypt into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-letsencrypt/0.7.0-0ubuntu0.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Robie Basak (racb)]

Hello Peter, or anyone else affected,

Accepted python-letsencrypt-apache into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-letsencrypt-apache/0.7.0-0ubuntu0.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Robie Basak (racb)]

Since the dependency tree is a few packages deep, this set of proposed updates will take some time to become available for testing in xenial-proposed. I will coax things through manually as dependencies are built and published, and comment again with a call for testing when ready.

[Brad Warren (bradmwarren)]

I successfully ran the integration tests described at https://wiki.ubuntu.com/StableReleaseUpdates/Certbot.

The packages that were used for testing were:

certbot 0.23.0-1~ubuntu16.04.1
letsencrypt 0.23.0-1~ubuntu16.04.1
python-acme 0.22.2-1ubuntu0.1~16.04.1
python-certbot 0.23.0-1~ubuntu16.04.1
python-certbot-apache 0.23.0-1~ubuntu16.04.1
python-josepy 1.1.0-1~ubuntu16.04.1
python-letsencrypt 0.7.0-0ubuntu0.16.04.1
python-letsencrypt-apache 0.7.0-0ubuntu0.16.04.1

[Robie Basak (racb)]

The Let's Encrypt packages in Ubuntu 16.04 will stop working on 13 March 2019. Updates to fix this problem are now ready for testing. Please help us test!

Certbot (formerly called Let's Encrypt) will stop working properly on 13 March 2019 when TLS-SNI-01 validation is turned off by the primary Let's Encrypt CA. This will make the Ubuntu 16.04 letsencrypt package effectively useless for many users. Newer Ubuntu releases package versions of Certbot that implement alternate validation methods and are not affected.

In Ubuntu 16.04 we are addressing this by backporting the existing Certbot packaging from Ubuntu 18.04. Proposed updates for 16.04 are now available and will be released to the usual update channels when they have received sufficient testing and validation.

Please help us test these packages on 16.04. Reports of both success and failure are appreciated. Please include details of how you tested, including the package versions used and any specifics of your environment to https://launchpad.net/bugs/1640978

To test:

To avoid upgrading wholesale to the proposed pocket (doing so may break your system), ensure that apt is pinned from doing so by making sure that a file exists in `/etc/apt/preferences.d/`, for example `/etc/apt/preferences.d/proposed-updates`, as follows:

    Package: *
    Pin: release a=xenial-proposed
    Pin-Priority: 400

Allow apt to upgrade all the packages provided in this Certbot update by creating `/etc/apt/preferences.d/certbot-proposed` as follows:

    Package: python-acme-doc python-acme python3-acme certbot letsencrypt python-certbot-doc python-certbot python-certbot-apache python-certbot-apache-doc python-josepy-doc python-josepy python3-josepy python-letsencrypt python-letsencrypt-apache
    Pin: release a=xenial-proposed
    Pin-Priority: 500

Make sure the following line exists in your `/etc/apt/sources.list` (or in a file in `/etc/apt/sources.list.d/`):

    deb http://archive.ubuntu.com/ubuntu/ xenial-proposed main universe

(other components such as `restricted` and `multiverse` may also be present).

Now you may simply `apt update` and `apt upgrade` as usual and apt will upgrade to this proposed update ready for testing.

Reports of both success and failure are appreciated. Please include details of how you tested, including the package versions used and any specifics of your environment. You can generate the status and versions of all the Certbot-related packages with the following command:

    dpkg-query -W -f='${db:Status-Abbrev} ${binary:Package} ${version}\n' python-acme-doc python-acme python3-acme certbot letsencrypt python-certbot-doc python-certbot python-certbot-apache python-certbot-apache-doc python-josepy-doc python-josepy python3-josepy python-letsencrypt python-letsencrypt-apache

Please add testing reports to the Launchpad tracking bug at: https://launchpad.net/bugs/1640978

[Mark (nulleuro)]

Successfully updated:
    letsencrypt (0.4.1-1 => 0.23.0-1~ubuntu16.04.1)
    python-acme (0.4.1-1 => 0.22.2-1ubuntu0.1~16.04.1)
    python-letsencrypt (0.4.1-1 => 0.7.0-0ubuntu0.16.04.1)
    python-letsencrypt-apache (0.4.1-1 => 0.7.0-0ubuntu0.16.04.1)

(though I actually had to apt-get install letencrypt python-letsencrypt-apache instead of upgrade, because the new packages from the proposed repo were held back by apt)

    $ dpkg-query
    ii certbot 0.23.0-1~ubuntu16.04.1
    ii letsencrypt 0.23.0-1~ubuntu16.04.1
    ii python-acme 0.22.2-1ubuntu0.1~16.04.1
    un python-acme-doc
    ii python-certbot 0.23.0-1~ubuntu16.04.1
    ii python-certbot-apache 0.23.0-1~ubuntu16.04.1
    un python-certbot-apache-doc
    un python-certbot-doc
    ii python-josepy 1.1.0-1~ubuntu16.04.1
    ii python-letsencrypt 0.4.1-1
    ii python-letsencrypt-apache 0.7.0-0ubuntu0.16.04.1

Ran certbot renew --force-renewal which successfully replaced all existing certs.

[Paride Legovini (paride)]

Successfully upgraded:

ii letsencrypt 0.4.1-1
ii python-acme 0.4.1-1
ii python-letsencrypt 0.4.1-1
ii python-letsencrypt-apache 0.4.1-1

to:

ii certbot 0.23.0-1~ubuntu16.04.1
ii letsencrypt 0.23.0-1~ubuntu16.04.1
ii python-acme 0.22.2-1ubuntu0.1~16.04.1
un python-acme-doc
ii python-certbot 0.23.0-1~ubuntu16.04.1
ii python-certbot-apache 0.23.0-1~ubuntu16.04.1
un python-certbot-apache-doc
un python-certbot-doc
ii python-josepy 1.1.0-1~ubuntu16.04.1
ii python-letsencrypt 0.7.0-0ubuntu0.16.04.1
ii python-letsencrypt-apache 0.7.0-0ubuntu0.16.04.1

The upgrade went smoothly (no held back packages), but I didn't setup any pinning (testing in a dedicated container). I'm able to generate/renew certificates with the upgraded packages.

The python-certbot-nginx plugin is not functional, but from the discussion we had on IRC I know it won't be made part of the SRU, as it wasn't part of Xenial in the first place.

Everything LGTM.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-josepy - 1.1.0-1~ubuntu16.04.1

---------------
python-josepy (1.1.0-1~ubuntu16.04.1) xenial; urgency=medium

  [ Robie Basak ]
  * This update is part of the set of major updates moving Let's
    Encrypt/Certbot to version 0.23 in 16.04 in order to allow it to
    continue working following the general shutdown of TLS-SNI-01
    validation (LP: #1640978).

  [ Michael Casadevall ]
  * Backport SRU dependency for certbot backport

 -- Robie Basak <email address hidden> Fri, 22 Feb 2019 12:38:36 +0000

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for python-josepy has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-letsencrypt - 0.7.0-0ubuntu0.16.04.1

---------------
python-letsencrypt (0.7.0-0ubuntu0.16.04.1) xenial; urgency=medium

  [ Robie Basak ]
  * This update is part of the set of major updates moving Let's
    Encrypt/Certbot to version 0.23 in 16.04 in order to allow it to
    continue working following the general shutdown of TLS-SNI-01
    validation (LP: #1640978).
  * The new source package python-certbot takes over this package's
    functions, with binary packages certbot, python-certbot and
    python-certbot-doc taking over respectively.
  * The following two functional changes are additionally made:
    - Log rotation is switched to logrotate via
      /etc/logrotate.d/certbot, and /etc/letsencrypt/cli.ini is
      introduced to disable internal log rotation to avoid collision.
    - Automatic renewal is enabled via the certbot.timer and
      certbot.service systemd units.

  [ Michael Casadevall ]
  * Replace python-letsencrypt with a compatibility shim

 -- Robie Basak <email address hidden> Fri, 22 Feb 2019 12:42:44 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-acme - 0.22.2-1ubuntu0.1~16.04.1

---------------
python-acme (0.22.2-1ubuntu0.1~16.04.1) xenial; urgency=medium

  [ Robie Basak ]
  * This update is part of the set of major updates moving Let's
    Encrypt/Certbot to version 0.23 in 16.04 in order to allow it to
    continue working following the general shutdown of TLS-SNI-01
    validation (LP: #1640978).
  * This package is a dependency of Certbot and using the 0.22 version
    in Bionic is the most straightforward backport choice. Version 0.22
    of this package is sufficient to fulfil the dependency requirement
    for Certbot 0.23.

  [ Michael Casadevall ]
  * Backport to Xenial for LE change

 -- Robie Basak <email address hidden> Fri, 22 Feb 2019 12:41:07 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-letsencrypt-apache - 0.7.0-0ubuntu0.16.04.1

---------------
python-letsencrypt-apache (0.7.0-0ubuntu0.16.04.1) xenial; urgency=medium

  [ Robie Basak ]
  * This update is part of the set of major updates moving Let's
    Encrypt/Certbot to version 0.23 in 16.04 in order to allow it to
    continue working following the general shutdown of TLS-SNI-01
    validation (LP: #1640978).
  * The new source package python-certbot-apache takes over this
    package's functions.

  [ Michael Casadevall ]
  * Replace python-letsencrypt-apache with a compatibility shim

 -- Robie Basak <email address hidden> Fri, 22 Feb 2019 12:43:04 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-certbot - 0.23.0-1~ubuntu16.04.1

---------------
python-certbot (0.23.0-1~ubuntu16.04.1) xenial; urgency=medium

  [ Robie Basak ]
  * This update is part of the set of major updates moving Let's
    Encrypt/Certbot to version 0.23 in 16.04 in order to allow it to
    continue working following the general shutdown of TLS-SNI-01
    validation (LP: #1640978).
  * This new source package takes over the function of
    the previous source package python-letsencrypt, with binary packages
    certbot, python-certbot and python-certbot-doc taking over
    respectively.
  * The following two functional changes are additionally made:
    - Log rotation is switched to logrotate via
      /etc/logrotate.d/certbot, and /etc/letsencrypt/cli.ini is
      introduced to disable internal log rotation to avoid collision.
    - Automatic renewal is enabled via the certbot.timer and
      certbot.service systemd units.

  [ Michael Casadevall ]
  * Backport to Xenial

 -- Robie Basak <email address hidden> Fri, 22 Feb 2019 12:41:51 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-certbot-apache - 0.23.0-1~ubuntu16.04.1

---------------
python-certbot-apache (0.23.0-1~ubuntu16.04.1) xenial; urgency=medium

  [ Robie Basak ]
  * This update is part of the set of major updates moving Let's
    Encrypt/Certbot to version 0.23 in 16.04 in order to allow it to
    continue working following the general shutdown of TLS-SNI-01
    validation (LP: #1640978).
  * This new source package takes over the function of the previous
    source package python-letsencrypt-apache.

  [ Michael Casadevall ]
  * Backport to Xenial for LE change

 -- Robie Basak <email address hidden> Fri, 22 Feb 2019 12:42:16 +0000

[Robie Basak (racb)]

python-certbot-nginx (nor any python-letsencrypt-nginx) was never released on 16.04 so cannot regress. Michael didn't include it in his SRU work, so it didn't make this update and isn't expected to do so in the future either.