apparmor profile denied for kerberos client keytab and credential cache files
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
openldap (Ubuntu) |
Fix Released
|
Medium
|
Andreas Hasenack | ||
Trusty |
Fix Released
|
Medium
|
Andreas Hasenack | ||
Xenial |
Fix Released
|
Medium
|
Andreas Hasenack | ||
Bionic |
Fix Released
|
Medium
|
Andreas Hasenack | ||
Cosmic |
Fix Released
|
Medium
|
Andreas Hasenack |
Bug Description
[Impact]
When using syncrepl replication with openldap, the consumer needs to authenticate to the provider in order to perform the searches and fetch the data. When this authentication is a simple bind, a simple username/password pair is used and that can be easily supplied in a configuration file.
When one wants to use a stronger authentication mechanism, like gssapi (kerberos), the authentication is based on keytab files and tickets. The consumer needs to obtain a ticket from the KDC, and use that ticket to authenticate itself with the provider.
For users, it's a simple matter of running kinit(1) and providing a password. For services, the solution is to extract the key from the KDC and store it in a local keytab file, which is then used by the service to obtain the TGT.
Problem is this TGT expires, and needs to be renewed periodically. Solutions have popped up for that issue, the most famous one being kstart (https:/
Via the default_
The default value of that setting is /etc/krb5/
Turns out the openldap slapd apparmor profile doesn't account for that, and blocks attempts to read that file. It also blocks reading the TGT that is obtained and stored in /tmp/krb5cc_<uid>, resulting in such DENIED errors in the logs:
apparmor="DENIED" operation="open" profile=
apparmor="DENIED" operation=
Since the slapd apparmor is enabled by default, this blocks the usage of this very helpful feature. Also considering that kerberos/gssapi errors are usually hard to debug, it might take a while for an admin to figure out what is going on.
The fix is to update the apparmor profile and allow access to those files. Unfortunately there are no rich globbing rules for paths in an apparmor profile, nothing like @{uid} of the current process yet, or a regexp rule like [0-9]+, so the rules have to be a bit accomodating. For this bug, I came up with these two new lines:
/etc/
owner /tmp/krb5cc_* rwk,
One could relax the first one perhaps into something like /etc/krb5/**, but the above works with the default settings.
[Test Case]
Setting up openldap replication via gssapi can be complicated, so I wrote some scripts to help.
- setup-kdc.sh: sets up the KDC server
- setup-provider.sh: sets up the openldap provider
- setup-consumer.sh: sets up the openldap consumer
The scripts expect LXD containers to be used, that have a working DNS setup for a ".lxd" domain. In other words, if you do:
lxc launch ubuntu-daily:bionic bionic-provider
The script expects the container to be reachable via "bionic-
So here we go. Here are the steps for a cosmic test, just replace "cosmic" with the name of the ubuntu release you are testing.
= KDC =
* launch the KDC container and copy the setup-kdc.sh script into it:
lxc launch ubuntu-daily:cosmic cosmic-kdc
lxc file push ./setup-kdc.sh cosmic-kdc/root/
* Enter the container and run the setup-kdc.sh script:
lxc exec cosmic-kdc bash
bash ./setup-kdc.sh
* The script will show two prompts: one from debconf, with just an "ok" option, so hit ENTER there. The second prompt will be for a password. Use any one you like, it won't be needed again.
* The KDC is setup, you can exit the container.
= PROVIDER =
* launch the provider and copy the setup-provider.sh script into it:
lxc launch ubuntu-daily:cosmic cosmic-provider
lxc file push ./setup-provider.sh cosmic-
* Enter the container and run the setup-provider.sh script:
lxc exec cosmic-provider bash
bash ./setup-provider.sh
* Leave a tail on the slapd logs, we will come back to this later:
tail -f /var/log/
= CONSUMER =
* launch the consumer and copy the setup-consumer.sh script into it:
lxc launch ubuntu-daily:cosmic cosmic-consumer
lxc file push ./setup-consumer.sh cosmic-
* Enter the container and run the setup-consumer.sh script:
lxc exec cosmic-consumer bash
bash ./setup-consumer.sh
* On the provider's log tail, you should soon see a connection and immediate disconnection like this:
Oct 23 17:37:18 cosmic-provider slapd[2396]: conn=1004 fd=12 ACCEPT from IP=10.0.
Oct 23 17:37:18 cosmic-provider slapd[2396]: conn=1004 op=0 UNBIND
Oct 23 17:37:18 cosmic-provider slapd[2396]: conn=1004 fd=12 closed
* On this consumer log, you should see an error like this:
tail /var/log/syslog |grep slapd
...
Oct 23 17:37:18 cosmic-consumer slapd[2408]: do_syncrepl: rid=001 rc -1 retrying
* On the host's log, dmesg will show an apparmor DENIED message like this:
[104159.081883] audit: type=1400 audit(154031623
This confirms the bug.
The replication request from the consumer to the provider will be repeated every 60s, so these messages will continue until the fixed package is installed.
To verify the fix, install the updated openldap packages on the consumer. Right after, the provider should log what is now a successful replication:
Oct 23 17:41:34 cosmic-provider slapd[2396]: conn=1009 fd=12 ACCEPT from IP=10.0.
Oct 23 17:41:34 cosmic-provider slapd[2396]: conn=1009 op=0 BIND dn="" method=163
Oct 23 17:41:39 cosmic-provider slapd[2396]: conn=1009 op=0 RESULT tag=97 err=14 text=SASL(0): successful result:
Oct 23 17:41:39 cosmic-provider slapd[2396]: conn=1009 op=1 BIND dn="" method=163
Oct 23 17:41:39 cosmic-provider slapd[2396]: conn=1009 op=1 RESULT tag=97 err=14 text=SASL(0): successful result:
Oct 23 17:41:39 cosmic-provider slapd[2396]: conn=1009 op=2 BIND dn="" method=163
Oct 23 17:41:39 cosmic-provider slapd[2396]: conn=1009 op=2 BIND authcid="consumer" authzid="consumer"
Oct 23 17:41:39 cosmic-provider slapd[2396]: conn=1009 op=2 BIND dn="uid=
Oct 23 17:41:39 cosmic-provider slapd[2396]: conn=1009 op=2 RESULT tag=97 err=0 text=
Oct 23 17:41:39 cosmic-provider slapd[2396]: conn=1009 op=3 SRCH base="dc=lxd" scope=2 deref=0 filter=
Oct 23 17:41:39 cosmic-provider slapd[2396]: conn=1009 op=3 SRCH attr=* +
The consumer should also have a TGT file in /tmp, with the uid number of the slapd process:
root@cosmic-
-rw------- 1 openldap openldap 1903 Oct 23 17:41 /tmp/krb5cc_110
root@cosmic-
110
[Regression Potential]
The fix opens up reading of a file in a subdirectory of /etc/krb5. To prevent slapd from reading other files in that directory, or other directories, the sysadmin has to use normal filesystem permissions.
In /tmp we open up writing of TGT files with a globbing pattern (krb5cc_*). The somewhat generous globbing is counteracted by the "owner" restriction we use in that rule, so that should be fine.
[Other Info]
Setting the test scenario up manually is complicated even for admins familiar with the technologies involved, so a set of scripts was used. To prevent the scripts from becoming super complicated, a specific test scenario with LXD is targeted, and error conditions are not thoroughly checked. I welcome feedback if I missed some obvious case, but the scripts won't be changed into generic tools to setup slapd gssapi replication. That is a task for a blog post or manual :)
This will be submitted to Debian as well, as reviews happen here first.
[Original Description]
Can we get /etc/krb5/** and /tmp/krb5cc_* added with the appropriate permissions to the slapd apparmor profile? I'm getting the following kinds of errors:
apparmor="DENIED" operation="open" profile=
apparmor="DENIED" operation=
Related branches
- Robie Basak: Approve
- Canonical Server: Pending requested
-
Diff: 29 lines (+10/-0)2 files modifieddebian/apparmor-profile (+2/-0)
debian/changelog (+8/-0)
- Christian Ehrhardt (community): Approve
- Canonical Server: Pending requested
-
Diff: 29 lines (+10/-0)2 files modifieddebian/apparmor-profile (+2/-0)
debian/changelog (+8/-0)
- Christian Ehrhardt (community): Approve
- Canonical Server: Pending requested
-
Diff: 29 lines (+10/-0)2 files modifieddebian/apparmor-profile (+2/-0)
debian/changelog (+8/-0)
- Christian Ehrhardt (community): Approve
- Canonical Server: Pending requested
-
Diff: 29 lines (+10/-0)2 files modifieddebian/apparmor-profile (+2/-0)
debian/changelog (+8/-0)
- Christian Ehrhardt (community): Approve
- Seth Arnold (community): Approve
- Canonical Server: Pending requested
-
Diff: 29 lines (+10/-0)2 files modifieddebian/apparmor-profile (+2/-0)
debian/changelog (+8/-0)
"/etc/krb5/ user/389/ client. keytab" feels like a local modification you made, to store keytab files somewhere under /etc/krb5. I suggest you add an apparmor exception in /etc/apparmor. d/local/ usr.sbin. slapd.
Unless I'm wrong and that directory is being used as a standard location by some package. Please let me know which is the case.
As to the /tpm/krb5cc_389 file, can you elaborate on the scenario that led to this behavior? Why is slapd trying to read that ticket cache file? Maybe because it failed to read the keytab file?