regression: "df: `/sys/kernel/debug': Function not implemented" with 3.2.0-85.122

[Jamie Strandboge]

$ cat /proc/version_signature
Ubuntu 3.2.0-84.121-generic 3.2.68

$ df | head -3
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/vda1 7481832 4262872 2838904 61% /
udev 371920 4 371916 1% /dev

$ cat /proc/version_signature
Ubuntu 3.2.0-85.122-generic 3.2.69

$ df | head -3
df: `/sys/kernel/debug': Function not implemented
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/vda1 7481832 4263012 2838764 61% /
udev 371920 4 371916 1% /dev

This is confirmed in an amd64 VM and on an amd64 system. It doesn't make a difference if I use sudo.

[Jamie Strandboge (jdstrand)]

Marking as confirmed since I was able to reproduce on two different systems.

[Joseph Salisbury (jsalisbury)]

This bug may be due to one of these commits:

7b0f287 kdb: fix incorrect counts in KDB summary command output
df438af debugfs: leave freeing a symlink body until inode eviction

I'll build a test kernel with these two commits reverted before starting the bisect process. I'll post a link to the test kernel shortly.

[Joseph Salisbury (jsalisbury)]

I built a Precise test kernel with a revert of 7b0f287 and df438af. Can you test this kernel and see if it still exhibits the bug?

It can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1465322/

[Jamie Strandboge (jdstrand)]

This works as advertised in an amd64 VM.

$ cat /proc/version_signature
Ubuntu 3.2.0-85.122~lp1465322-generic 3.2.69

$ df | head -3
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/vda1 7481832 4264172 2837604 61% /
udev 371924 4 371920 1% /dev

[Joseph Salisbury (jsalisbury)]

Great, thanks, Jamie. I'll reproduce the bug and figure out which of these commits is the cause.

[Slavisa Djokic (djokic)]

Hi Joseph,

thanks for taking care. Looks like this bug might be related: https://bugs.launchpad.net/ubuntu/+source/coreutils/+bug/1465180

We have here tivoli-storage-manager-client (tsm) which reports error since upgrade: "ANS0361I DIAG: TransErrno: Unexpected error from GetFSInfo:statfs, errno = 38". The backup finishes successfully though.

[Joseph Salisbury (jsalisbury)]

I built two more test kernels. One has commit df438af reverted and the other has df438af reverted. Can you test each of these kernels and see which one resolves the bug? The kernels can be downloaded from:

http://kernel.ubuntu.com/~jsalisbury/lp1465322/df438af-reverted
and
http://kernel.ubuntu.com/~jsalisbury/lp1465322/7b0f287-reverted

Thanks in advance!

[Luis Henriques (henrix)]

I can confirm that reverting df438afe2d93 ("debugfs: leave freeing a symlink body until inode eviction") fixes the issue on an i386 VM. But it would be great to have the original bug reporter to test that too.

[Luis Henriques (henrix)]

Here's some more detail:

running 'strace df' shows the following:

  statfs64("/sys/kernel/debug", 84, 0xbfddc6bc) = -1 ENOSYS (Function not implemented)

A quick test shows that adding '.statfs = simple_statfs' in the debugfs_super_operations struct fixes the problem, but I'm not sure
that's the right thing to do.

(This has been reported upstream: http://thread.gmane.org/gmane.linux.kernel.stable/134708/focus=134588)

[Jamie Strandboge (jdstrand)]

Reverting df438af seems to have fixed it.

$ cat /proc/version_signature
Ubuntu 3.2.0-86.123~lp1465322Commitdf438afReverted-generic 3.2.69

$ df | head -3
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/vda1 7481832 4735296 2366480 67% /
udev 371924 4 371920 1% /dev

[Agesp Anonymous (agespam)]

Please note, does this bug affect any other software but just df command?
(definitely kernel related, after update from 3.2.0-84 to 3.2.0-86)

My setup is also affected by this bug:

$ df -h
df: ”/sys/kernel/debug”: Funktion toteutus puuttuu
Tiedostojärjestelmä Koko Käyt Vapaa Käy% Liitospiste
/dev/sda1 22G 4,2G 16G 21% /
...

$ LANG=C df -h
df: `/sys/kernel/debug': Function not implemented
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 22G 4.2G 16G 21% /
...

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04.5 LTS"

$ uname -a
Linux machine 3.2.0-86-generic-pae #123-Ubuntu SMP Sun Jun 14 18:30:54 UTC 2015 i686 athlon i386 GNU/Linux

[Andy Zeigler (k-andy-s)]

Agesp: In response to your question about whether or not this impacts other software, I can tell you that it affects our Ruby on Rails app in production. I'm not sure what is triggering it -- it may be IRB, rails, or a gem that we have, but there is a constant stream of these messages on stderr when the rails console ('rails c') is loaded:

deploy@ip-10-0-0-41:/var/www/current$ rails c
Loading production environment (Rails 4.1.10)
20150622025515(production) > df: `/sys/kernel/debug': Function not implemented
df: `/sys/kernel/debug': Function not implemented
df: `/sys/kernel/debug': Function not implemented
df: `/sys/kernel/debug': Function not implemented
...

I get a message every few seconds without any user interaction.

Obviously there is a simple workaround here -- 'rails c 2>/dev/null`. But it would be nice to have stderr back at some point. :)

Thanks,

Andy

[mig5 (mig5)]

After upgrading 12.04 machines to

 linux-image-3.2.0-86-generic 3.2.0-86.124

from

 linux-image-3.2.0-86-generic 3.2.0-86.123

.. I still continue to get the same 'function not implemented', including after reboot.

For 12.04 machines running the Trusty HWE kernel 3.13.0-55-generic #94~precise1-Ubuntu SMP, and Trusty machines themselves, the issue appears resolved..

I wonder if it still affects machines on 3.2.0-86 *if* they had previously upgraded to the regression kernel last week? I haven't tested on a machine on 85->newest 86 yet.

[Luis Henriques (henrix)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-precise' to 'verification-done-precise'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[mikelococo (mikelococo)]

Luis: The meaning of your last comment isn't clear to me (probably because I'm insufficiently familiar with the standard process between fix-committed and fix-released).

Are you calling for community testers to validate this bug in order to prevent the fix from being thrown away 5 days from now?

Or does the addition of the verification-needed-precise merely signal to the assignee that some release-process step is ready to happen?

[Luis Henriques (henrix)]

mikelococo: this is a request for testing the Precise kernel that is currently in -proposed (kernel version 3.2.0-87.125). This kernel version should contain the fix for this bug, and I am asking the original bug reporter (or anyone else affected by this bug) to verify that it is indeed fixed.

If the bug is fixed by this kernel, the bug tag 'verification-needed-precise' should be replaced by 'verification-needed-precise'.

Anyway, since I was able to reproduce this bug, I will eventually verify it too but as I've worked on the fix it would be good to have someone else to double-check.

[mikelococo (mikelococo)]

I'm not the original reporter, but I am affected and the fix looks good to me. Details below (forgive any typos or abbreviations, copy/paste not setup for my test-vm)...

### First boot
$ uname -r
3.2.0-86-generic
$uname -v
#124-Ubuntu SMP Wed Jun 17 <... snip build time ...>
$ df
df: '/sys/kernel/debug': Function not implemented # Error reproduced with build-124
<... snip ...>
$ echo 'deb http://archive.ubuntu.com/ubuntu/ trusty-proposed restricted main multiverse universe' > /etc/apt/sources.list
$ sudo aptitude update
<... snip ...>
$ sudo aptitude upgrade # installs build-125
<... snip ...>
$ sudo reboot

### Second boot
$ uname -r
3.2.0-86-generic
$ uname -v
#125-Ubuntu SMP Wed Jun 19 <... snip build time ...>
$ df
<... snip ...> # No error with build-125

[mikelococo (mikelococo)]

verification-done-precise tag added
verification-needed-precise tag removed

[Jamie Strandboge (jdstrand)]

The kernel in precise-proposed fixes this for me.

$ cat /proc/version_signature
Ubuntu 3.2.0-87.125-generic 3.2.69

$ df | head -3
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/vda1 7481832 5146416 1955360 73% /
udev 371920 4 371916 1% /dev

[mig5 (mig5)]

Confirming 3.2.0-87-generic from precise-proposed fixes the issue for me also. Thanks

[pwolanin (pwolanin)]

I tested 3.2.0-87-virtual from precise-proposed on AWS and it fixes the df issue which we saw in our automated testing

[mikelococo (mikelococo)]

Any word on release schedule?

[Jamie Strandboge (jdstrand)]

RiCh, 3.2.0-87.125 has not been published to precise-updates/precise-security yet, so it should not be marked Fix Released.

[mikelococo (mikelococo)]

Periodic reminder that some folks (minimally myself) are looking forward to the general availability of this fix with great anticipation. Lots of third-party scripts depend on the exit status of df.

[Jamie Strandboge (jdstrand)]

3.2.0-87.125 was published to precise-updates/precise-security on 2015-07-06, I don't know why it didn't get autoclosed.

$ apt-cache policy linux-image-3.2.0-87-generic
$ apt-cache policy linux-image-3.2.0-87-generic
linux-image-3.2.0-87-generic:
  Installed: 3.2.0-87.125
  Candidate: 3.2.0-87.125
  Version table:
 *** 3.2.0-87.125 0
        500 http://us.archive.ubuntu.com/ubuntu/ precise-updates/main amd64 Packages
        500 http://us.archive.ubuntu.com/ubuntu/ precise-security/main amd64 Packages
        100 /var/lib/dpkg/status

$ cat /proc/version_signature
Ubuntu 3.2.0-87.125-generic 3.2.69

$ df | head -3
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/vda1 7481832 5147004 1954772 73% /
udev 371920 4 371916 1% /dev

Marking as Fix Released.