[SRU] Fix invalid CSR version in python-acme

[Harlan Lieberman-Berg]

[ Impact ]

This bug causes certbot to generate CSRs which are invalid. These CSRs are then sent to ACME servers or otherwise parsed. Some software validate CSR validity more aggressively, whichmeans it will reject these CSRs.

The principle motivation for backporting this fix is to stop certbot from generating CSRs. This will both alleviate bugs experienced by users, as well as reduce pressure on CSR parsers to accept _invalid_ CSRs.

[ Test plan ]

See https://bugs.launchpad.net/ubuntu/+source/python-acme/+bug/2018252/comments/11 for complete test plan including links to assets.

[ Where problems could occur ]

For a problem to occur, it would require software that not only accepted, but in fact _required_, an invalid CSR, and which also did not process CSRs from recent versions of certbot or versions from Debian or RHEL containing the backport.

The worst-case scenario for such software would be something that copied the version value from a CSR into a certificate it was issuing (CSRs have only a single valid version, v1. X.509 certificates can be either v1 or v3, however in practice v3 is the only version in use.). Such software would end up producing different (and less correct/compatible) certificates. I am not aware of any software with this behavior.

A more likely (though still improbable) bug would be software which merely asserts that the CSR's version is something incorrect.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Stefano Rivera (stefanor)]

Sponsored the uploads (reversioned as XXXubuntu0.1). They're now pending SRU team approval.

[Steve Langasek (vorlon)]

This is software whose primary function is to talk to remote services. Adding a test case that confirms that your code change changed the code is an insufficient test plan for this SRU. The test plan should demonstrate how the software misbehaved previously, and demonstrate that the new behavior is correct.

[Alex Gaynor (alex-gaynor)]

Hi Steve,

Can you explain a bit more on what you're looking for in terms of a test plan?

As the SRU states, it's been in upstream for nearly a year, and backported to Debian and RHEL (EPEL) already as well (for something like 6 months). Therefore it's received widespread real-world testing.

I've also manually verified that without this patch, CSRs generated by the package raise an exception on parsing with pyca/cryptography 40, but with the patch they are parsed successfully. pyca/cryptography enforces validity of CSR version.

[Robie Basak (racb)]

I think it might help to elaborate on the user story that is currently broken and would be fixed by this change. Right now the description explains things from the point of view of the code, not the point of view for users. What is the problem that users are experiencing today that needs fixing, from the point of view of users?

Once that is described clearly, then an appropriate Test Plan should become clear - to verify that the change fixes the problem, we should replay the user story that is described as being broken. We should also test the common case of use of python-acme, which could straightforwardly be to ensure that certbot works against the proposed update of python-acme to configure (say) nginx in the common case, and that users then don't get a certificate warning on browsing that server.

[Alex Gaynor (alex-gaynor)]

You're right, I think the user story will make things clearer:

Users on impacted versions of Ubuntu use certbot to generate CSRs (which are invalid), which are then processed by an ACME server that enforces version validity, and therefore rejects them.

While there are no public CAs that do this enforcement, we (the pyca/cryptography maintainers) have received several reports of private CAs with internal customers running impacted versions of certbot.

[Steve Langasek (vorlon)]

Thanks, can you provide a test plan that demonstrates (from scratch) how to reproduce such a rejection of a CSR?

[Mathias Ertl (mathiasertl)]

Hi,

I attached a docker-compose.yml file and a Dockerfile packed as a tarball that allow you to reproduce this issue.

The docker-compose.yml file starts pebble (the ACMEv2 demo server provided by Lets Encrypt) and is copied from [1] and extended to also add a Dockerfile that can start certbot. It requires that you need to have Docker Compose installed[2].

To reproduce, first extract the tarball and cd to ubuntu-2018252/:

  $ tar xf ubuntu-2018252.tar.gz
  $ cd ubuntu-2018252/

Then, start the setup with Ubuntu Jammy. A certificate will be retrieved automatically after five seconds in csr/:

  $ DIST=jammy docker compose build
  $ docker compose up

Quit the container and try the same with lunar:

  $ DIST=lunar docker compose build
  $ docker compose up

Now see the (ordered) CSRs retrieved:

  $ ls csr/
  0000_csr-certbot.pem 0001_csr-certbot.pem

... and see that the CSR version created differs from the first (Jammy) and second (Lunar):

  $ openssl req -in csr/0000_csr-certbot.pem -noout -text | head -n 3
  Certificate Request:
      Data:
          Version: Unknown (2)

  $ openssl req -in csr/0001_csr-certbot.pem -noout -text | head -n 3
  Certificate Request:
      Data:
          Version: 1 (0x0)

You can then also start the container manually and run certbot (e.g. after applying a patch):

  $ docker compose run certbot /bin/bash
  # certbot certonly --standalone -n -d example.com -m <email address hidden> --agree-tos --server https://pebble:14000/dir
  ...

I hope this helps and we can finally get this trivial patch into affected distributions.

Thanks, Mat

[1] https://github.com/letsencrypt/pebble
[2] https://docs.docker.com/compose/install/

[Steve Langasek (vorlon)]

Hello Harlan, or anyone else affected,

Accepted python-acme into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-acme/1.21.0-1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Harlan, or anyone else affected,

Accepted python-acme into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-acme/1.1.0-1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Mathias Ertl (mathiasertl)]

Hello Steve,

I was able to verify the fix in both Focal and Jammy:

* For both distributions, I verified the test-plan as above.
* For Focal, I tested 1.1.0-1 vs. 1.1.0-1ubuntu0.1.
* For Jammy, I tested 1.21.0-1 vs. 1.21.0-1ubuntu0.1.

Additionally, for both distributions, I tested certbot in a Docker image to validate that the old version indeed triggers the bug, and updating to the new version in proposed indeed fixes the issue. I used the procedure described in my own cryptography-based ACMEv2 implementation[1] (with a minor modification to use the appropriate distribution) to run the full certificate validation and issuance processes using certbot. In both cases, this looked like this (note that certbot is configured to use the local ACMEv2 server via configuration file, output trimmed for brievety):

  root@certbot:~# dpkg-query --showformat='${Version}\n' --show python3-acme
  1.1.0-1
  root@certbot:~# certbot register
  ...
  root@certbot:~# django-ca-test-validation.sh http http-01.example.com
  + certbot certonly --standalone --preferred-challenges http -d http-01.example.com
  ...
  An unexpected error occurred:
  The CSR is unacceptable (e.g., due to a short key) :: Invalid CSR version.

  # add -proposed as described in the docs

  root@certbot:~# dpkg-query --showformat='${Version}\n' --show python3-acme
  1.1.0-1ubuntu0.1

  root@certbot:~# django-ca-test-validation.sh http http-01.example.com
  + certbot certonly --standalone --preferred-challenges http -d http-01.example.com
  ...
  Waiting for verification...
  Cleaning up challenges

  IMPORTANT NOTES:
   - Congratulations! Your certificate and chain have been saved at:
  ...

I hope this properly confirms the fix and this is speedily released.

kr, Mat

[1] https://django-ca.readthedocs.io/en/latest/dev/acme.html

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-acme - 1.21.0-1ubuntu0.1

---------------
python-acme (1.21.0-1ubuntu0.1) jammy; urgency=medium

  * Fix incorrect CSR version (LP: #2018252)

 -- Harlan Lieberman-Berg <email address hidden> Mon, 01 May 2023 16:17:31 -0400

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for python-acme has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-acme - 1.1.0-1ubuntu0.1

---------------
python-acme (1.1.0-1ubuntu0.1) focal; urgency=medium

  * Fix CSR version (LP: #2018252)
  * Allow higher version of python3-idna (LP: #1876929)

 -- Harlan Lieberman-Berg <email address hidden> Mon, 01 May 2023 17:10:09 -0400