dehydrated: Missing ID field for new registrations

[Frédéric Bourqui]

[ Inpact ]

https://bugs.debian.org/934039
https://github.com/lukas2511/dehydrated/issues/647

Changes in the Let's Encrypt API caused several issues due to a non-RFC compliant handling of the account ID.

Furthermore, I want to take this occasion to also introduce a few fixes for some upcoming changes in November.
The simplest (and imho more safe) way to fix all of these is to take the version in Debian stable 0.6.2-2+deb10u1.

[ Test Case ]

I can't quite generate a trivial test case, as afaik this requires actually running the program to get a certificate.
If you do, you get a "400 Bad Request" with 0.6.1-2 up to 0.6.5-1.

[ Regression Potential ]

This update has been widely tested on several production setup. The very same package is not used on Debian stable as well, just with a different version.
Compared to the current version 0.6.1-2, there are not many non-bugfix change (the only relevant one being a new hook deploy_ocsp), and those should not affect any production environment.

[Mattia Rizzolo (mapreri)]

both uploaded.

[Frédéric Bourqui (fbourqui)]

When you ran your"Test Case" did you first run ?
/usr/bin/dehydrated --register --accept-terms

[Frédéric Bourqui (fbourqui)]

This is a real BUG fixed in 0.6.2-2.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=934039

It impact any fresh install of 18.04, trying to run dehydrated as well:
Same error as Debian Bug report logs - #934039

On Ubuntu 18.04 dehydrated is stuck in version 0.6.1-2
Let's encrypt production API was changed in August 2019 (before the issue was only on the Staging Environment API).

Here is the version used in Debian:
Package dehydrated

    stretch (oldstable) (misc): ACME client implemented in Bash
    0.3.1-3+deb9u2: all
    stretch-backports (misc): ACME client implemented in Bash
    0.6.2-2~bpo9+1: all
    buster (stable) (misc): ACME client implemented in Bash
    0.6.2-2+deb10u1: all
    buster-backports (misc): ACME client implemented in Bash
    0.6.5-1~bpo10+1: all
    bullseye (testing) (misc): ACME client implemented in Bash
    0.6.5-1: all
    sid (unstable) (misc): ACME client implemented in Bash
    0.6.5-1: all

[Mattia Rizzolo (mapreri)]

Yes… isn't that what I said? I uploaded 0.6.2-2+deb10u1 to ubuntu 18.04 (pending release managers approval) exactly to fix everything you mentioned.

[Frédéric Bourqui (fbourqui)]

OK, I missed the fact that a release manager needed to approve, before a new package would be generated/published.

Thanks

[Mattia Rizzolo (mapreri)]

FTR, I've been using the proposed packages on my xenial servers for a few weeks without noticing anything (and certs keep getting renewed)

[Brian Murray (brian-murray)]

Hello Frédéric, or anyone else affected,

Accepted dehydrated into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dehydrated/0.6.2-2ubuntu0.19.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Frédéric, or anyone else affected,

Accepted dehydrated into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dehydrated/0.6.2-2ubuntu0.18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Frédéric Bourqui (fbourqui)]

------------------------
Freshly installed bionic
------------------------

root@test:~# uname -a
Linux test 4.15.0-64-generic #73-Ubuntu SMP Thu Sep 12 13:16:13 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
root@test:~# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"

root@test:~# /usr/bin/dehydrated --version
# INFO: Using main config file /etc/dehydrated/config
# INFO: Using additional config file /etc/dehydrated/conf.d/config.sh
Dehydrated by Lukas Schauer
https://dehydrated.de

Dehydrated version: 0.6.1
GIT-Revision: unknown

OS: Ubuntu 18.04.3 LTS
Used software:
 bash: 4.4.20(1)-release
 curl: curl 7.58.0
 awk: GNU Awk 4.1.4, API: 1.1 (GNU MPFR 4.0.1, GNU MP 6.1.2)
 sed: sed (GNU sed) 4.4
 mktemp: mktemp (GNU coreutils) 8.28
 grep: grep (GNU grep) 3.1
 diff: diff (GNU diffutils) 3.6
 openssl: OpenSSL 1.1.1 11 Sep 2018

root@test:~# /usr/bin/dehydrated --register --accept-terms
# INFO: Using main config file /etc/dehydrated/config
# INFO: Using additional config file /etc/dehydrated/conf.d/config.sh
+ Generating account key...
+ Registering account key with ACME server...
  + ERROR: An error occurred while sending post-request to https://acme-v02.api.letsencrypt.org/acme/new-acct (Status 400)

Details:
HTTP/2 400
server: nginx
date: Wed, 25 Sep 2019 10:56:42 GMT
content-type: application/problem+json
content-length: 112
cache-control: public, max-age=0, no-cache
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
replay-nonce: 0102rpm6-W8ukECwM63wgKX3pPpfR5719i6zjf_xkpAo1W0

{
  "type": "urn:ietf:params:acme:error:badNonce",
  "detail": "JWS has no anti-replay nonce",
  "status": 400
}

Error registering account key. See message above for more information.

-------------------------
install proposed version:
-------------------------

root@test:~# /usr/bin/dehydrated --version
# INFO: Using main config file /etc/dehydrated/config
# INFO: Using additional config file /etc/dehydrated/conf.d/config.sh
Dehydrated by Lukas Schauer
https://dehydrated.io

Dehydrated version: 0.6.2
GIT-Revision: unknown

OS: Ubuntu 18.04.3 LTS
Used software:
 bash: 4.4.20(1)-release
 curl: curl 7.58.0
 awk: mawk 1.3.3 Nov 1996, Copyright (C) Michael D. Brennan
 sed: sed (GNU sed) 4.4
 mktemp: mktemp (GNU coreutils) 8.28
 grep: grep (GNU grep) 3.1
 diff: diff (GNU diffutils) 3.6
 openssl: OpenSSL 1.1.1 11 Sep 2018

root@test:~# /usr/bin/dehydrated --register --accept-terms
# INFO: Using main config file /etc/dehydrated/config
# INFO: Using additional config file /etc/dehydrated/conf.d/config.sh
+ Generating account key...
+ Registering account key with ACME server...
+ Fetching account ID...
+ Done!

Bug is fixed with proposed backage

[Frédéric Bourqui (fbourqui)]

-------------------------
fresh disco install:
-------------------------
root@test2:~# uname -a
Linux test2 4.15.0-64-generic #73-Ubuntu SMP Thu Sep 12 13:16:13 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
root@test2:~# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=19.04
DISTRIB_CODENAME=disco
DISTRIB_DESCRIPTION="Ubuntu 19.04"
root@test2:~# /usr/bin/dehydrated --version
# INFO: Using main config file /etc/dehydrated/config
# INFO: Using additional config file /etc/dehydrated/conf.d/config.sh
Dehydrated by Lukas Schauer
https://dehydrated.io

Dehydrated version: 0.6.2
GIT-Revision: unknown

OS: Ubuntu 19.04
Used software:
 bash: 5.0.3(1)-release
 curl: curl 7.64.0
 awk: mawk 1.3.3 Nov 1996, Copyright (C) Michael D. Brennan
 sed: sed (GNU sed) 4.7
 mktemp: mktemp (GNU coreutils) 8.30
 grep: grep (GNU grep) 3.3
 diff: diff (GNU diffutils) 3.7
 openssl: OpenSSL 1.1.1b 26 Feb 2019

root@test2:~# /usr/bin/dehydrated --register --accept-terms
# INFO: Using main config file /etc/dehydrated/config
# INFO: Using additional config file /etc/dehydrated/conf.d/config.sh
+ Account already registered!

root@test2:~# /usr/bin/dehydrated -c
# INFO: Using main config file /etc/dehydrated/config
# INFO: Using additional config file /etc/dehydrated/conf.d/config.sh
Processing test2.exemple.com
 + Signing domains...
 + Generating private key...
 + Generating signing request...
 + Requesting new certificate order from CA...
  + ERROR: An error occurred while sending head-request to https://acme-v02.api.letsencrypt.org/acme/new-nonce (Status 000)

Details:

  + ERROR: An error occurred while sending post-request to https://acme-v02.api.letsencrypt.org/acme/new-order (Status 400)

Details:
HTTP/2 400
server: nginx
date: Wed, 25 Sep 2019 12:05:29 GMT
content-type: application/problem+json
content-length: 178
cache-control: public, max-age=0, no-cache
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
replay-nonce: 0101Ja-Em1z22OiKhp4h8QoCTUBU-qoKeGS-J-8nzMCcAzo

{
  "type": "urn:ietf:params:acme:error:malformed",
  "detail": "Malformed account ID in KeyID header URL: \"https://acme-v02.api.letsencrypt.org/acme/acct/\"",
  "status": 400
}

-------------------------
install proposed version:
-------------------------

Unpacking dehydrated (0.6.2-2ubuntu0.19.04.1) over (0.6.2-2) ...
Setting up dehydrated (0.6.2-2ubuntu0.19.04.1) ...
Press Return to continue, 'q' followed by Return to quit.
q
root@test2:~# /usr/bin/dehydrated --version
# INFO: Using main config file /etc/dehydrated/config
# INFO: Using additional config file /etc/dehydrated/conf.d/config.sh
Dehydrated by Lukas Schauer
https://dehydrated.io

Dehydrated version: 0.6.2
GIT-Revision: unknown

OS: Ubuntu 19.04
Used software:
 bash: 5.0.3(1)-release
 curl: curl 7.64.0
 awk: mawk 1.3.3 Nov 1996, Copyright (C) Michael D. Brennan
 sed: sed (GNU sed) 4.7
 mktemp: mktemp (GNU coreutils) 8.30
 grep: grep (GNU grep) 3.3
 diff: diff (GNU diffutils) 3.7
 openssl: OpenSSL 1.1.1b 26 Feb 2019

root@test2:~# /usr/bin/dehydrated -c
# INFO: Using main config file /etc/dehydrated/config
# INFO: Using additional config file /etc/dehydrated/conf.d/config.sh
Processing test2.exemple.com
 + Signing domain...

[Frédéric Bourqui (fbourqui)]

a fully working certificate, on Disco with proposed package, dns domain changed to exemple.com

root@test2:~# /usr/bin/dehydrated -c -t dns-01 -k /etc/dehydrated/remote_dns_hook.sh
# INFO: Using main config file /etc/dehydrated/config
# INFO: Using additional config file /etc/dehydrated/conf.d/config.sh
Processing test2.exemple.com
 + Signing domains...
 + Generating private key...
 + Generating signing request...
 + Requesting new certificate order from CA...
 + Received 1 authorizations URLs from the CA
 + Handling authorization for test2.exemple.com
 + 1 pending challenge(s)
 + Deploying challenge tokens...
 + OVH hook executing: deploy_challenge
+ TXT record created, ID: 1624286182
+ Zone refreshed on OVH side
+ SOA SERIAL of zone: 2019092503
Testing DNS record against ...
 + Record not available yet. Checking again in 10s...
Got: / Expecting: wlAsgTTTdsyaR1lNSCNFBq0g4UOyqvZBTTgxTHSAmNI
Testing DNS record against ...
 + Responding to challenge for test2.exemple.com authorization...
 + Challenge is valid!
 + Cleaning challenge tokens...
 + OVH hook executing: clean_challenge
 + Deleting TXT record name: _acme-challenge.test2
+ Zone refreshed on OVH side
+ SOA SERIAL of zone: 2019092504
 + Requesting certificate...
 + Checking certificate...
 + Done!
 + Creating fullchain.pem...

[Launchpad Janitor (janitor)]

This bug was fixed in the package dehydrated - 0.6.2-2ubuntu0.19.04.1

---------------
dehydrated (0.6.2-2ubuntu0.19.04.1) disco; urgency=medium

  * Add three more patches from upstream.
    Fixing the following bug:
     + Fixed fetching of account information. LP: #1841619
     + Followup fixes for account ID handling, and APIv1 compatibility.

 -- Mattia Rizzolo <email address hidden> Sun, 08 Sep 2019 18:54:59 +0200

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for dehydrated has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package dehydrated - 0.6.2-2ubuntu0.18.04.1

---------------
dehydrated (0.6.2-2ubuntu0.18.04.1) bionic; urgency=medium

  * Add three more patches from upstream.
    Fixing the following bug:
     + Fixed fetching of account information. LP: #1841619
     + Followup fixes for account ID handling, and APIv1 compatibility.

dehydrated (0.6.2-2) unstable; urgency=medium

  * Add a number of patches from upstream.
    Fixing the following bugs:
     + HTTP/2 support, where header names are lowercase
     + Avoid over matching, checking for the Replay-Nonce header only at BOL
     + A bug causing deletion of domains.txt when incorrect parameters are used
     + Document the DOMAINS_D config option
     + Impoent POST-as-GET, for the upcoming change in LE's API
     + Document PRIVATE_KEY_ROLLOVER per-cert config option
  * d/control: bump Standards-Version to 4.3.0, no changes needed.

dehydrated (0.6.2-1) unstable; urgency=medium

  * New upstream release 0.6.2.
  * Remove all patches - applied upstream.
  * d/control: update Homepage field.

 -- Mattia Rizzolo <email address hidden> Sun, 08 Sep 2019 19:00:15 +0200