memcached looks for SASL configuration at wrong path /etc/sasl2/memcached.conf/memcached.conf (18.04→20.04 regression)

Bug #1878721 reported by Anders Kaseorg on 2020-05-15
20
This bug affects 2 people
Affects Status Importance Assigned to Milestone
memcached (Ubuntu)
High
Sergio Durigan Junior
Eoan
High
Unassigned
Focal
High
Unassigned

Bug Description

[Impact]

memcached 1.5.22 in focal has a bug where it looks for its SASL configuration at /etc/sasl2/memcached.conf/memcached.conf instead of /etc/sasl2/memcached.conf. This causes a memcached setup with authentication that was working in bionic to fail in focal.

The bug was introduced upstream in 1.5.7~3:

https://github.com/memcached/memcached/commit/39151c870c5e598f039714bdb790bd46f614856e
https://github.com/memcached/memcached/commit/80dd99d831535ddeec73d55a0adcaeaac8cb7298
https://github.com/memcached/memcached/pull/366

and fixed upstream in 1.6.0~15:

https://github.com/memcached/memcached/commit/6207330c2705fdb5f02de13b99a0d994f7c4f14a

The reason this bug happens is because sasl works with paths (i.e., directories) when determining which configuration files it should load, whereas, after the two commits mentioned above (version 1.5.7~3), memcached started to pass a full pathname (including the filename) of the configuration file.

So, while in a "normal" setup memcached's configuration file would live at /etc/sasl2/memcached.conf, meaning that memcached would tell sasl that the config path is "/etc/sasl2/", with the buggy memcached sasl would think that the configuration directory is "/etc/sasl2/memcached.conf/", and would try to look for a "memcached.conf" file inside it.

Users could workaroud this bug by creating a directory named "/etc/sasl2/memcached.conf/". For this reason, the backported patch also takes this workaround into account. The patch also accepts the "/etc/sasl/" directory.

[Test Case]

To test the fix, one can do:

$ lxc launch ubuntu-daily:focal memcached-bug1878721
$ lxc shell memcached-bug1878721
# apt update && apt upgrade -y
# apt install -y memcached libmemcached-tools libsasl2-modules sasl2-bin
# mkdir -p /etc/sasl2
# cat > /etc/sasl2/memcached.conf << __EOF__
mech_list: plain
sasldb_path: /etc/sasl2/memcached-sasldb2
__EOF__
# echo bar | saslpasswd2 -p -f /etc/sasl2/memcached-sasldb2 -a memcached foo
# chown memcache: /etc/sasl2/memcached-sasldb2
# echo '-S' >> /etc/memcached.conf
# systemctl restart memcached.service
# memcping --servers=127.0.0.1 --binary --username=foo --password=bar

With the last command, you should see an error like:

  Failed to ping 127.0.0.1:11211 WRITE FAILURE

or:

  Failed to ping 127.0.0.1:11211 READ FAILURE

You can also check its exit status:

# echo $?
1

It is possible to test the workaround workaround mentioned in the previous section by doing:

# mv /etc/sasl2/memcached.conf /tmp/
# mkdir /etc/sasl2/memcached.conf
# mv /tmp/memcached.conf /etc/sasl2/memcached.conf/
# systemctl restart memcached.service
# memcping --servers=127.0.0.1 --binary --username=foo --password=bar
# echo $?
0

Using the fix provided, one can verify that both tests above will work.

Here are all four locations that will now work by default:

• /etc/sasl/memcached.conf/memcached.conf
• /etc/sasl/memcached.conf
• /etc/sasl2/memcached.conf/memcached.conf
• /etc/sasl2/memcached.conf

[Regression Potential]

Low risk. The upstream patch is targeted and applies cleanly to 1.5.22. It looks for the SASL configuration at both the incorrect and correct paths, so even in the (unlikely) event that someone worked around this bug by manually creating a configuration file at the incorrect path /etc/sasl2/memcached.conf/memcached.conf, that will continue to be respected.

If there were to be a regression, it would likely manifest as an authentication failure, which clients may display as a read or write failure, like the failure mode of the regression being fixed here.

Related branches

Anders Kaseorg (andersk) wrote :

Here’s a debdiff with the upstream patch. I built this in https://launchpad.net/~andersk/+archive/ubuntu/ppa and verified it against the test case.

tags: added: patch
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in memcached (Ubuntu):
status: New → Confirmed

Thanks for reporting this bug and also the provided patch.

tags: added: server-next
Changed in memcached (Ubuntu):
importance: Undecided → High
Changed in memcached (Ubuntu Focal):
status: New → Confirmed
importance: Undecided → High
Changed in memcached (Ubuntu):
status: Confirmed → Fix Released
Changed in memcached (Ubuntu Eoan):
status: New → Triaged
tags: added: block-proposed-eoan block-proposed-focal block-proposed-groovy
Changed in memcached (Ubuntu Focal):
assignee: nobody → Sergio Durigan Junior (sergiodj)
Changed in memcached (Ubuntu):
assignee: nobody → Sergio Durigan Junior (sergiodj)
Changed in memcached (Ubuntu Focal):
assignee: Sergio Durigan Junior (sergiodj) → nobody

It seems that bionic is also affected:

(c)rafaeldtinoco@groovy:~$ rmadison memcached
...
 memcached | 1.5.6-0ubuntu1 | bionic |
 memcached | 1.5.6-0ubuntu1.2 | bionic-security |
 memcached | 1.5.6-0ubuntu1.2 | bionic-updates |
 memcached | 1.5.10-0ubuntu1 | disco |
 memcached | 1.5.10-0ubuntu1.19.04.2 | disco-security |
 memcached | 1.5.10-0ubuntu1.19.04.2 | disco-updates |
 memcached | 1.5.10-0ubuntu3 | eoan |
 memcached | 1.5.22-2 | focal |
 memcached | 1.6.6-1 | groovy |

subscribed @sergiodj to check everything out.

thx!

Changed in memcached (Ubuntu Bionic):
status: New → Triaged
Changed in memcached (Ubuntu Eoan):
importance: Undecided → High
Changed in memcached (Ubuntu Bionic):
importance: Undecided → High
Anders Kaseorg (andersk) wrote :

Rafael: I said that bionic is *not* affected, both because bionic has 1.5.6 while the bug was introduced in 1.5.7, and because the test case passes on bionic. What makes you think otherwise?

@andersk, you're totally right. I was using this case as a reference to explain our triage process and since I was talking and focused in some other stuff I failed to check versions (you will see that I have even pasted rmadison output).

I assigned @sergiodj for any leftovers on the SRU (keep your credits, of course) and subscribed ubuntu-sponsors IF they arrive first. So please ignore those weird movements in this case and your fix will be kept as-is and credits maintained.

Sorry for the noise!

Changed in memcached (Ubuntu Bionic):
status: Triaged → Fix Released
Mathew Hodson (mhodson) on 2020-05-23
no longer affects: memcached (Ubuntu Bionic)
Anders Kaseorg (andersk) on 2020-06-02
description: updated
Anders Kaseorg (andersk) on 2020-06-02
description: updated
Anders Kaseorg (andersk) on 2020-06-02
description: updated
description: updated
description: updated

Hello Anders, or anyone else affected,

Accepted memcached into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/memcached/1.5.22-2ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in memcached (Ubuntu Focal):
status: Confirmed → Fix Committed
tags: added: verification-needed verification-needed-focal
Steve Langasek (vorlon) wrote :

why is this bug marked block-proposed?

Anders Kaseorg (andersk) wrote :

Verified that 1.5.22-2ubuntu0.1 works on focal with the config file in all four locations.

tags: added: verification-done-focal
removed: verification-needed-focal
Anders Kaseorg (andersk) on 2020-06-13
tags: removed: block-proposed-eoan block-proposed-focal block-proposed-groovy
Anders Kaseorg (andersk) wrote :

I assume Rafael added block-proposed mistakenly; there is no staged upload involved.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package memcached - 1.5.22-2ubuntu0.1

---------------
memcached (1.5.22-2ubuntu0.1) focal; urgency=medium

  * d/p/fix-bug-where-sasl-will-load-config-the-wrong-path.patch:
    Fix the path from which SASL configuration is loaded. (LP: #1878721)
    The bug happened because sasl expects memcached to provide a
    path (i.e., a directory, not a filename) where the sasl
    configuration file(s) is (are). However, memcached was passing
    the filename (/etc/sasl2/memcached.conf) to sasl, which was
    interpreting it as a directory, and looking for a configuration
    file inside it (i.e., /etc/sasl2/memcached.conf/memcached.conf).
    Users could workaround this bug by creating a directory named
    /etc/sasl2/memcached.conf/, and putting the configuration file
    inside it. This patch not only fixes this bug (by passing the
    right directory, /etc/sasl2/, to sasl) but also supports the
    workaround described above.

 -- Anders Kaseorg <email address hidden> Thu, 14 May 2020 17:13:17 -0700

Changed in memcached (Ubuntu Focal):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for memcached has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Hello Anders, or anyone else affected,

Accepted memcached into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/memcached/1.5.10-0ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in memcached (Ubuntu Eoan):
status: Triaged → Fix Committed
tags: added: verification-needed-eoan
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers