Ubuntu 22.04.2, nsupdate stopped recognizing HMAC-MD5 key after update from 1:9.18.1-1ubuntu1.3 to 1:9.18.12-0ubuntu0.22.04.1

[Wladimir Mutel]

[Impact]

Bind9 upstream accidentally introduced a regression that made old HMAC-MD5 key pair files unreadable in version 9.18.8.

This capability was fixed with the release of 9.18.17 through https://gitlab.isc.org/isc-projects/bind9/-/merge_requests/8069. This means the issue will be fixed alongside the MRE release of 9.18.18 in Lunar and Jammy.

The issue is fixed by restoring the missing files and pointing to them correctly.

[Test Plan]

To test that this fix specifically is successful, you can run:

# lxc launch ubuntu:{lunar, jammy} test-bind9
# lxc exec test-bind9 bash

# apt update && apt dist-upgrade -y
# apt install bind9

Create example key files since HMAC-MD5 is deprecated and creation of them was removed from focal onward
# cat <<EOF >Kexample.com.+157+15178.key
example.com. IN KEY 512 3 157 SItPKKvb7T9QEBRl9Mmrng==
EOF

# cat <<EOF >Kexample.com.+157+15178.private
Private-key-format: v1.3
Algorithm: 157 (HMAC_MD5)
Key: SItPKKvb7T9QEBRl9Mmrng==
Bits: AAA=
Created: 20230920212628
Publish: 20230920212628
Activate: 20230920212628
EOF

# nsupdate -k Kexample.com.+157+15178.private

Prior to the fix, this results in something like:
20-Sep-2023 21:41:40.730 Kexample.com.+157+15178.private:1: unknown option 'Private-key-format:'
20-Sep-2023 21:41:40.730 Kexample.com.+157+15178.private:8: unexpected token near end of file
could not read key from Kexample.com.+157+15178.{private,key}: unexpected token

After the fix, the command should succeed with a possible deprecation warning:
20-Sep-2023 21:36:24.723 Kexample.com.+157+15178.private: Use of K* file pairs for HMAC is deprecated

[Where problems could occur]

Problems with this release would most likely occour outside the scope of this issue, as the MRE release includes many other fixes and updates alongside this. However, issues related directly to this change would likely revolve around other key files breaking or the HMAC-MD5 files not being restored properly to match their original state.

[Original Description]

I have a key with the following contents (key material replaced with ...) :

+ cat /etc/bind/Khost.+157+35878.key
host. IN KEY 0 3 157 YSp... ...QsQ==

+ cat /etc/bind/Khost.+157+35878.private
Private-key-format: v1.3
Algorithm: 157 (HMAC_MD5)
Key: YSp......QsQ==
Bits: AAA=
Created: 20180616045813
Publish: 20180616045813
Activate: 20180616045813

it worked for long time, up till package version 1:9.18.1-1ubuntu1.3
but since upgrading to 1:9.18.12-0ubuntu0.22.04.1 , it stopped working with nsupdate giving out the following :

Creating key...
could not read key from /etc/bind/Khost.+157+35878.{private,key}: file not found

in strace printout, I see that nsupdate successfully opens and reads .private key file but then checks existence of the same file name without suffix (as specified after nsupdate -k) and fails.

were there any changes in key parsing from 9.18.1 to 9.18.13 ?
reverting bind9-utils, bind9-dnsutils and bind9-libs back to 1:9.18.1-1ubuntu1.3 restored the desired behavior.
please advise if I should fix the key format after the upgrade, or if this is a regression to be fixed from your side.

[Sergio Durigan Junior (sergiodj)]

Thank you for taking the time to report this bug.

As you have noticed, bind9 has been updated to its latest version in the 9.18.x series. This was quite a leap, which brought many important bugfixes to Ubuntu users across many release. We do our best to guarantee that these bigger-than-normal updates go smooth, but unfortunately there are scenarios which are too complex or uncommon to cover. It seems that you've hit one of those bumps.

I was able to successfully reproduce the problem you reported after I generated an HMAC-MD5 key on Bionic (which is the only LTS distro that's still able to generate keys using this algorithm), by doing:

$ dnssec-keygen -a HMAC-MD5 -b 128 -n HOST bind9.lxd
$ cat Kbind9.lxd.+157+25064.key
bind9.lxd. IN KEY 512 3 157 0OaUhv7uRii9yrq6kdSnbA==
$ cat Kbind9.lxd.+157+25064.private
Private-key-format: v1.3
Algorithm: 157 (HMAC_MD5)
Key: 0OaUhv7uRii9yrq6kdSnbA==
Bits: AAA=
Created: 20230405201738
Publish: 20230405201738
Activate: 20230405201738

This effectively created a pair of keys, which I transferred to a Jammy container and then tried invoking "nsupdate -k":

# nsupdate -k /etc/bind/Kbind9.lxd.+157+25064
could not read key from /etc/bind/Kbind9.lxd.+157+25064.{private,key}: file not found

I spent some time debugging the problem, and was able to track down the issue to the fact that bind9 has chosen different numbers to represent key algorithms internally. These numbers happen to be hardcoded inside the keys, which means that, when nsupdate checks if the specified algorithm is supported (and HMAC-MD5 is still supported), it can't find the entry associated with the old number and bails out.

In order to make things work again, a quick workaround is to edit your key files and replace "157" with "160". For example, here are the two files listed above with the modifications in place:

# cat Kbind9.lxd.+157+25064.key
bind9.lxd. IN KEY 512 3 160 0OaUhv7uRii9yrq6kdSnbA==
# cat Kbind9.lxd.+157+25064.private
Private-key-format: v1.3
Algorithm: 160 (HMAC_MD5)
Key: 0OaUhv7uRii9yrq6kdSnbA==
Bits: AAA=
Created: 20230405201738
Publish: 20230405201738
Activate: 20230405201738

There is also a newer format for key files which you can use. For the example I'm using here, it looks like this:

# cat Kbind9.lxd.+157+25064
key "bind9.lxd" {
        algorithm hmac-md5;
        secret "0OaUhv7uRii9yrq6kdSnbA==";
};

Note that this format doesn't contain any internal references to how bind9 numbers the algorithms, which is more future-proof. An even better solution would be to move away from HMAC-MD5, but I understand that that's not always possible.

For reference, the upstream commit that broke things was:

https://gitlab.isc.org/isc-projects/bind9/-/commit/09f7e0607a34d90eae53f862954e98c31b5ae532

There's an upstream bug about this problem, which I will link in a moment.

[Sergio Durigan Junior (sergiodj)]

Upstream bug: https://gitlab.isc.org/isc-projects/bind9/-/issues/3668

[Robie Basak (racb)]

I wonder if there are any plans to fix this, or are we going to have to leave it now that the damage is done?

[Lena Voytek (lvoytek)]

For now I would like to avoid a second regression for users of Ubuntu stable releases, so I will avoid updates unless there is enough demand for them. If anyone else has experienced this issue too and comes across this, please let me know. As for mantic, I'll wait and see if upstream comes up with a solution in 9.19.x and port it in accordingly for future consistency. Thanks!

[Lena Voytek (lvoytek)]

Still waiting on upstream to make a decision, removing server-todo for now

[Launchpad Janitor (janitor)]

This bug was fixed in the package bind9 - 1:9.18.18-0ubuntu1

---------------
bind9 (1:9.18.18-0ubuntu1) mantic; urgency=medium

  * New upstream release 9.18.18 (LP: #2034367)
    - Updates:
      + Mark a primary server as temporarily unreachable when a TCP connection
        response to an SOA query times out, matching behavior of a refused TCP
        connection.
      + Mark dialup and heartbeat-interval options as deprecated.
      + Retry DNS queries without an EDNS COOKIE when the first response is
        FORMERR with the EDNS COOKIE that was sent originally.
      + Use NS records for the relaxed QNAME minimization mode to reduce the
        number of queries from named.
    - Bug Fixes:
      + Fix assertion failure from processing already-queued queries while
        server is being reconfigured or cache is being flushed.
      + Fix failure to load zones containing resource records with a TTL value
        larger than 86400 seconds when dnssec-policy is set to insecure.
      + Fix the ability to read HMAC-MD5 key files (LP: #2015176).
      + Fix stability issues with the catalog zone implementation.
    - See https://bind9.readthedocs.io/en/v9.18.18/notes.html for additional
      information.

 -- Lena Voytek <email address hidden> Tue, 05 Sep 2023 13:20:06 -0700

[Wladimir Mutel (mwg)]

great that it is fixed for mantic, how about fixing it for jammy as well ?

[Steve Langasek (vorlon)]

Hello Wladimir, or anyone else affected,

Accepted bind9 into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/bind9/1:9.18.18-0ubuntu0.22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Wladimir, or anyone else affected,

Accepted bind9 into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/bind9/1:9.18.18-0ubuntu0.23.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Lena Voytek (lvoytek)]

Verified for Lunar and Jammy

# lxc launch ubuntu:jammy test-bind9-jammy
# lxc exec test-bind9-jammy bash

cat <<EOF >/etc/apt/sources.list.d/ubuntu-$(lsb_release -cs)-proposed.list
# Enable Ubuntu proposed archive
deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs)-proposed restricted main multiverse universe
EOF

# apt update && apt dist-upgrade -y
# apt install bind9 -y

# cat <<EOF >Kexample.com.+157+15178.key
example.com. IN KEY 512 3 157 SItPKKvb7T9QEBRl9Mmrng==
EOF

# cat <<EOF >Kexample.com.+157+15178.private
Private-key-format: v1.3
Algorithm: 157 (HMAC_MD5)
Key: SItPKKvb7T9QEBRl9Mmrng==
Bits: AAA=
Created: 20230920212628
Publish: 20230920212628
Activate: 20230920212628
EOF

# nsupdate -k Kexample.com.+157+15178.private

29-Sep-2023 22:23:44.897 Kexample.com.+157+15178.private: Use of K* file pairs for HMAC is deprecated

# lxc launch ubuntu:lunar test-bind9-lunar
# lxc exec test-bind9-lunar bash

cat <<EOF >/etc/apt/sources.list.d/ubuntu-$(lsb_release -cs)-proposed.list
# Enable Ubuntu proposed archive
deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs)-proposed restricted main multiverse universe
EOF

# apt update && apt dist-upgrade -y
# apt install bind9 -y

# cat <<EOF >Kexample.com.+157+15178.key
example.com. IN KEY 512 3 157 SItPKKvb7T9QEBRl9Mmrng==
EOF

# cat <<EOF >Kexample.com.+157+15178.private
Private-key-format: v1.3
Algorithm: 157 (HMAC_MD5)
Key: SItPKKvb7T9QEBRl9Mmrng==
Bits: AAA=
Created: 20230920212628
Publish: 20230920212628
Activate: 20230920212628
EOF

# nsupdate -k Kexample.com.+157+15178.private

29-Sep-2023 22:51:23.253 Kexample.com.+157+15178.private: Use of K* file pairs for HMAC is deprecated

[Launchpad Janitor (janitor)]

This bug was fixed in the package bind9 - 1:9.18.18-0ubuntu0.23.04.1

---------------
bind9 (1:9.18.18-0ubuntu0.23.04.1) lunar; urgency=medium

  * New upstream release 9.18.18 (LP: #2028413)
    - Updates:
      + Mark a primary server as temporarily unreachable when a TCP connection
        response to an SOA query times out, matching behavior of a refused TCP
        connection.
      + Mark dialup and heartbeat-interval options as deprecated.
      + Retry DNS queries without an EDNS COOKIE when the first response is
        FORMERR with the EDNS COOKIE that was sent originally.
      + Use NS records for the relaxed QNAME minimization mode to reduce the
        number of queries from named.
      + Mark TKEY mode 2 as deprecated.
      + Mark delegation-only and root-delegation-only as deprecated.
      + Run RPZ and catalog zone updates on specialized offload threads to
        reduce blocked query processing time.
    - Bug Fixes:
      + Fix assertion failure from processing already-queued queries while
        server is being reconfigured or cache is being flushed.
      + Fix failure to load zones containing resource records with a TTL value
        larger than 86400 seconds when dnssec-policy is set to insecure.
      + Fix the ability to read HMAC-MD5 key files (LP: #2015176).
      + Fix stability issues with the catalog zone implementation.
      + Fix bind9 getting stuck when listen-on statement for HTTP is removed
        from configuration.
      + Do not return delegation from cache after stale-answer-client-timeout.
      + Fix failure to auto-tune clients-per-query limit in some situations.
      + Fix proper timeouts when using max-transfer-time-in and
        max-transfer-idle-in statements.
      + Bring rndc read timeout back to 60 seconds from 30.
      + Treat libuv returning ISC_R_INVALIDPROTO as a network error.
      + Clean up empty-non-terminal NSEC3 records.
      + Fix log file rotation cleanup for absolute file path destinations.
      + Fix various catalog zone processing crashes.
      + Fix transfer hang when downloading large zones over TLS.
      + Fix named crash when adding a new zone into the configuration file for
        a name which was already configured as member zone for a catalog zone.
      + Delay DNSSEC key queries until all zones have finished loading.
    - See https://bind9.readthedocs.io/en/v9.18.18/notes.html for additional
      information.
  * d/p/CVE-2023-2828.patch, CVE-2023-2911.patch: Remove - fixed upstream in
    9.18.16.
  * d/p/CVE-2023-3341.patch: Refresh, matching upstream, to apply in 9.18.18.
  * d/t/control, d/t/dyndb-ldap: add DEP8 test (LP: #2032650)

 -- Lena Voytek <email address hidden> Wed, 20 Sep 2023 14:52:27 -0700

[Robie Basak (racb)]

The verification of the Stable Release Update for bind9 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package bind9 - 1:9.18.18-0ubuntu0.22.04.1

---------------
bind9 (1:9.18.18-0ubuntu0.22.04.1) jammy; urgency=medium

  * New upstream release 9.18.18 (LP: #2028413)
    - Updates:
      + Mark a primary server as temporarily unreachable when a TCP connection
        response to an SOA query times out, matching behavior of a refused TCP
        connection.
      + Mark dialup and heartbeat-interval options as deprecated.
      + Retry DNS queries without an EDNS COOKIE when the first response is
        FORMERR with the EDNS COOKIE that was sent originally.
      + Use NS records for the relaxed QNAME minimization mode to reduce the
        number of queries from named.
      + Mark TKEY mode 2 as deprecated.
      + Mark delegation-only and root-delegation-only as deprecated.
      + Run RPZ and catalog zone updates on specialized offload threads to
        reduce blocked query processing time.
    - Bug Fixes:
      + Fix assertion failure from processing already-queued queries while
        server is being reconfigured or cache is being flushed.
      + Fix failure to load zones containing resource records with a TTL value
        larger than 86400 seconds when dnssec-policy is set to insecure.
      + Fix the ability to read HMAC-MD5 key files (LP: #2015176).
      + Fix stability issues with the catalog zone implementation.
      + Fix bind9 getting stuck when listen-on statement for HTTP is removed
        from configuration.
      + Do not return delegation from cache after stale-answer-client-timeout.
      + Fix failure to auto-tune clients-per-query limit in some situations.
      + Fix proper timeouts when using max-transfer-time-in and
        max-transfer-idle-in statements.
      + Bring rndc read timeout back to 60 seconds from 30.
      + Treat libuv returning ISC_R_INVALIDPROTO as a network error.
      + Clean up empty-non-terminal NSEC3 records.
      + Fix log file rotation cleanup for absolute file path destinations.
      + Fix various catalog zone processing crashes.
      + Fix transfer hang when downloading large zones over TLS.
      + Fix named crash when adding a new zone into the configuration file for
        a name which was already configured as member zone for a catalog zone.
      + Delay DNSSEC key queries until all zones have finished loading.
    - See https://bind9.readthedocs.io/en/v9.18.18/notes.html for additional
      information.
  * d/p/CVE-2023-2828.patch, CVE-2023-2911.patch: Remove - fixed upstream in
    9.18.16.
  * d/p/CVE-2023-3341.patch: Refresh, matching upstream, to apply in 9.18.18.
  * d/t/control, d/t/dyndb-ldap: add DEP8 test (LP: #2032650)

 -- Lena Voytek <email address hidden> Wed, 20 Sep 2023 15:15:41 -0700