rpc.gssd does not handle missing machine credential cache

[Alec Warner]

We use NFSv3 with kerberos authentication. The filer is a netapp. The client is:

Description: Ubuntu 10.04 LTS
Release: 10.04

We patch /etc/init/gssd.conf to add extra credential caches:

exec rpc.gssd -d /var/run/ccache:/var/spool/tickets

We enabled extra rpc.gssd logging and received the following:

Oct 15 01:31:40 sh12.redacted rpc.gssd[320]: Full hostname for 'filer.redacted' is 'filer.redacted'
Oct 15 01:31:40 sh12.redacted rpc.gssd[320]: Full hostname for 'sh12.redacted' is 'sh12.redacted'
Oct 15 01:31:40 sh12.redacted rpc.gssd[320]: Key table entry not found while getting keytab entry for 'root/sh12.redacted@REALM'
Oct 15 01:31:40 sh12.redacted rpc.gssd[320]: Success getting keytab entry for 'nfs/sh12.redacted@REALM'
Oct 15 01:31:40 sh12.redacted rpc.gssd[320]: ERROR: Credentials cache file '/var/run/ccache/krb5cc_machine_REALM' not found while initializing credential cache 'FILE:/var/run/ccache/krb5cc_machine_REALM'
Oct 15 01:31:40 sh12.redacted rpc.gssd[320]: INFO: Credentials in CC 'FILE:/var/run/ccache/krb5cc_machine_REALM' are good until 1287210700
Oct 15 01:31:40 sh12.redacted rpc.gssd[320]: using FILE:/var/run/ccache/krb5cc_machine_REALM as credentials cache for machine creds
Oct 15 01:31:40 sh12.redacted rpc.gssd[320]: using environment variable to select krb5 ccache FILE:/var/run/ccache/krb5cc_machine_REALM
Oct 15 01:31:40 sh12.redacted rpc.gssd[320]: creating context using fsuid 0 (save_uid 0)
Oct 15 01:31:40 sh12.redacted rpc.gssd[320]: ERROR: GSS-API: error in gss_acquire_cred(): Unspecified GSS failure. Minor code may provide more information - Credentials cache file '/var/run/ccache/krb5cc_machine_REALM' not found
Oct 15 01:31:40 sh12.redacted rpc.gssd[320]: WARNING: Failed while limiting krb5 encryption types for user with uid 0
Oct 15 01:31:40 sh12.redacted rpc.gssd[320]: WARNING: Failed to create krb5 context for user with uid 0 with credentials cache FILE:/var/run/ccache/krb5cc_machine_REALM for server filer.redacted
Oct 15 01:31:40 sh12.redacted rpc.gssd[320]: WARNING: Failed to create krb5 context for user with uid 0 with any credentials cache for server filer.redacted
Oct 15 01:31:40 sh12.redacted rpc.gssd[320]: doing error downcall

Steps to Reproduce:

Install lucid (with sec=krb5 mounts and rpc.gssd enabled...)
mount a sec=krb5 volume (rpc.gssd will generate a krb5cc_machine_REALM credential cache properly.)
Delete the credential cache
try to mount another sec=krb5 volume -> fails.

This logline:
Oct 15 01:31:40 sh12.redacted rpc.gssd[320]: INFO: Credentials in CC 'FILE:/var/run/ccache/krb5cc_machine_REALM' are good until 1287210700

Seems to indicate that rpc.gssd is keeping some kind of in-process state that the credential cache is not expired (and thus good) even if the credential cache is deleted from under it.

I thought this was fixed upstream in:
http://git.linux-nfs.org/?p=cel/nfs-utils.git;a=commit;h=bd947185cfc7978c562fddf2f14f602c44a5cac9

However when I back-ported the nfs-utils-1.2.2 package (from maverick) that has this patch applied; however the issue is still repeatable when running that version.

There is a nagging issue as well as to what exactly is deleting the credentials cache on my affected machines (this is not normal behavior and only a small number of machines are affected.) I hope to get a better idea of that problem shortly.

[Etienne Goyer (etienne-goyer-outlands)]

nfs-utils 1.2.3 shows the same behavior.

[Alec Warner (antarus)]

We figured out why this was happening (a race condition on our custom initscript that creates /var/run/ccache) and we have fixed it internally. There is still the case that rpc.gssd does not renew its credentials; however it is not a important issue now that we have fixed the race.

-A