rpc.gssd crashes in gss_pipe_downcall

[Carlos Aguado]

Source: nfs-utils

# lsb_release -rd
Description: Ubuntu 12.04.1 LTS
Release: 12.04

# apt-cache policy nfs-common
nfs-common:
  Installed: 1:1.2.5-3ubuntu3
  Candidate: 1:1.2.5-3ubuntu3
  Version table:
 *** 1:1.2.5-3ubuntu3 0
        500 http://archive.ubuntu.com/ubuntu/ precise/main amd64 Packages
        100 /var/lib/dpkg/status

Expected:
To be able to mount a krb5-protected NFSv4 share.
# cat /etc/fstab
san.example.com:/data/volume /nfs4/san.example.com/volume nfs4 _netdev,sec=krb5 0 0

What actually happens:
The kerberos service ticket is successfully requested as shown below [1], but the mount of the share never happens. Instead, /var/log/syslog sees the trace of what is reported as a kernel bug [2].

[1]
# klist -c /tmp/krb5cc_machine_EXAMPLE.COM
Ticket cache: FILE:/tmp/krb5cc_machine_EXAMPLE.COM
Default principal: <email address hidden>

Valid starting Expires Service principal
06/09/2012 11:44 06/09/2012 21:44 <email address hidden>
 renew until 11/09/2012 16:44
06/09/2012 11:44 06/09/2012 21:44 <email address hidden>
 renew until 11/09/2012 16:44

[2]
gss_pipe_downcall: bad return from gss_fill_context: -4
Sep 6 11:44:20 client kernel: [ 23.655189] ------------[ cut here ]------------
Sep 6 11:44:20 client kernel: [ 23.655190] kernel BUG at /build/buildd/linux-3.2.0/net/sunrpc/auth_gss/auth_gss.c:671!
Sep 6 11:44:20 client kernel: [ 23.655192] invalid opcode: 0000 [#1] SMP
Sep 6 11:44:20 client kernel: [ 23.655193] CPU 2
Sep 6 11:44:20 client kernel: [ 23.655194] Modules linked in: bnep rfcomm bluetooth parport_pc ppdev vesafb rpcsec_gss_krb5 nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc snd_hda
_codec_hdmi snd_hda_codec_realtek nvidia(P) snd_hda_intel snd_hda_codec snd_hwdep snd_pcm joydev snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device snd mac_hid mei(
C) soundcore snd_page_alloc video tpm_tis lp parport firewire_ohci firewire_core usbhid crc_itu_t hid e1000e
Sep 6 11:44:20 client kernel: [ 23.655211]
Sep 6 11:44:20 client kernel: [ 23.655212] Pid: 820, comm: rpc.gssd Tainted: P C O 3.2.0-29-generic #46-Ubuntu /DQ77MK
Sep 6 11:44:20 client kernel: [ 23.655215] RIP: 0010:[<ffffffffa0d79b9a>] [<ffffffffa0d79b9a>] gss_pipe_downcall+0x27a/0x2b0 [auth_rpcgss]
Sep 6 11:44:20 client kernel: [ 23.655219] RSP: 0018:ffff8803fb2d7e58 EFLAGS: 00010296
Sep 6 11:44:20 client kernel: [ 23.655220] RAX: 000000000000004d RBX: 0000000000000044 RCX: 000000000000001e
Sep 6 11:44:20 client kernel: [ 23.655221] RDX: 0000000000000000 RSI: 0000000000000086 RDI: 0000000000000246
Sep 6 11:44:20 client kernel: [ 23.655222] RBP: ffff8803fb2d7eb8 R08: 000000000000000a R09: 0000000000000000
Sep 6 11:44:20 client kernel: [ 23.655223] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8803ff32c7e0
Sep 6 11:44:20 client kernel: [ 23.655224] R13: ffff8803ff32cc00 R14: ffff8803fb085600 R15: fffffffffffffffc
Sep 6 11:44:20 client kernel: [ 23.655226] FS: 00007f846eabd740(0000) GS:ffff88041dc80000(0000) knlGS:0000000000000000
Sep 6 11:44:20 client kernel: [ 23.655227] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 6 11:44:20 client kernel: [ 23.655228] CR2: 000000000061ac88 CR3: 00000003fe8b8000 CR4: 00000000001406e0
Sep 6 11:44:20 client kernel: [ 23.655229] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Sep 6 11:44:20 client kernel: [ 23.655230] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Sep 6 11:44:20 client kernel: [ 23.655231] Process rpc.gssd (pid: 820, threadinfo ffff8803fb2d6000, task ffff880400cc4500)
Sep 6 11:44:20 client kernel: [ 23.655232] Stack:
Sep 6 11:44:20 client kernel: [ 23.655233] ffff8804009a1e00 0000000000000010 ffff8803ff32c824 ffff880402398700
Sep 6 11:44:20 client kernel: [ 23.655235] ffff8803fb2d7ec8 0000000081178fb3 ffff8803fb2d7e98 ffff880402398720
Sep 6 11:44:20 client kernel: [ 23.655237] ffff8803fe143c00 00000000013da320 ffffffffffffffe0 0000000000000044
Sep 6 11:44:20 client kernel: [ 23.655239] Call Trace:
Sep 6 11:44:20 client kernel: [ 23.655248] [<ffffffffa0d546a0>] rpc_pipe_write+0x60/0x90 [sunrpc]
Sep 6 11:44:20 client kernel: [ 23.655251] [<ffffffff81177b43>] vfs_write+0xb3/0x180
Sep 6 11:44:20 client kernel: [ 23.655253] [<ffffffff81177e6a>] sys_write+0x4a/0x90
Sep 6 11:44:20 client kernel: [ 23.655255] [<ffffffff81661ec2>] system_call_fastpath+0x16/0x1b
Sep 6 11:44:20 client kernel: [ 23.655256] Code: 89 c7 74 3b 7f 27 48 83 f8 81 74 40 48 83 f8 da 90 74 2c 48 89 c2 48 c7 c6 50 e3 d7 a0 48 c7 c7 f8 e5 d7 a0 31 c0 e8 41 74 8c e0 <0f> 0b 48 83 f8 f3 74 19 48 83 f8 f4 74 06 48 83 f8 f2 75 d4 41
Sep 6 11:44:20 client kernel: [ 23.655273] RIP [<ffffffffa0d79b9a>] gss_pipe_downcall+0x27a/0x2b0 [auth_rpcgss]
Sep 6 11:44:20 client kernel: [ 23.655276] RSP <ffff8803fb2d7e58>
Sep 6 11:44:20 client kernel: [ 23.655277] ---[ end trace 66b6cbe2d5cfccd8 ]---

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: nfs-common 1:1.2.5-3ubuntu3
ProcVersionSignature: Ubuntu 3.2.0-29.46-generic 3.2.24
Uname: Linux 3.2.0-29-generic x86_64
NonfreeKernelModules: nvidia
ApportVersion: 2.0.1-0ubuntu12
Architecture: amd64
Date: Thu Sep 6 11:52:36 2012
ProcEnviron:
 TERM=xterm
 LANG=en_US
 SHELL=/bin/bash
SourcePackage: nfs-utils
UpgradeStatus: No upgrade log present (probably fresh install)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Robert Huitl (freedesktop-huitl)]

I observe the same crash on a 3.2.0-31-generic kernel running as a Dom-U on XEN. On another machine running 3.2.0-30-generic (without Xen) it works just fine. nfs-common version is the same as yours on both machines.

[Robert Huitl (freedesktop-huitl)]

It seems that this crash happens only at boot time, because if I comment out the nfs4 mount in /etc/fstab and mount manually, there's no crash and the filesystem can be accessed. Looks like some sort of race.

Carlos, can you confirm that this happens only during boot?

[Carlos Aguado (caguadosa)]

Yes, sorry for the delay in testing. I can confirm that such a workaround does not trigger the crash and we are using it from now on.

For sake of full reference, we currently mount those filesystems from /etc/rc.local.