overlayfs alters /proc/self/exe link(s), making result a dead link.

Bug #1007089 reported by Mike Mestnik on 2012-05-31
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
linux-lowlatency (Ubuntu)
Medium
Unassigned

Bug Description

I've had this issue with other Ubuntu kernels.

Problem A:
This is not so much a bug in PulseAudio, but it is effected by this bug.
arcadia:~# invoke-rc.d pulseaudio start * Starting system PulseAudio Daemon W: [pulseaudio] main.c: /proc/self/exe does not point to /usr/bin/pulseaudio, cannot self execute. Are you playing games?
invoke-rc.d: initscript pulseaudio, action "start" failed.

Details:
Looking at other process, like ntpd, it's clear something outside of PA is not functional.

arcadia:~# ps $(pgrep ntpd)
  PID TTY STAT TIME COMMAND
 4595 ? Ss 0:00 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 105:108
arcadia:~# ls -l /proc/4595/exe
lrwxrwxrwx 1 root root 0 May 31 13:53 /proc/4595/exe -> /sbin/ntpd <--- wow, that's wrong!
arcadia:~# ls /sbin/ntpd /usr/sbin/ntpd
ls: cannot access /sbin/ntpd: No such file or directory
/usr/sbin/ntpd

System settings:
arcadia:~# grep usr /etc/fstab
/dev/mapper/fsUsr.ro /.usr-ro squashfs defaults,ro 0 1
/dev/mapper/fsUsr.rw /.usr-rw ext4 defaults,discard,errors=remount-ro 0 1
none /usr overlayfs defaults,lowerdir=/.usr-ro/local/..,upperdir=/.usr-rw/lost+found/.. 0 2
UUID=e7f6ec6c-631d-11e1-9522-9b80600299c1 /usr/local ext4 defaults,discard 0 2

I guess the contents of /.usr-rw and /.usr-ro may matter.
arcadia:~# ls -l /.usr-r?/*bin/{ntpd,pulseaudio}
-rw-r--r-- 1 root root 79260 Mar 8 21:36 /.usr-ro/bin/pulseaudio
-rwxr-xr-x 1 root root 601028 Mar 6 09:03 /.usr-ro/sbin/ntpd
-rwxr-xr-x 1 root root 79260 Apr 12 00:23 /.usr-rw/bin/pulseaudio

...I attempted to 'disable' PA by removing it's execute bits. I'm now more fully making use of PA and life is better now.

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: linux-image-3.2.0-23-lowlatency-pae 3.2.0-23.31
ProcVersionSignature: Ubuntu 3.2.0-23.31-lowlatency-pae 3.2.14
Uname: Linux 3.2.0-23-lowlatency-pae i686
ApportVersion: 2.0.1-0ubuntu5
Architecture: i386
Date: Thu May 31 14:14:18 2012
SourcePackage: linux-lowlatency
UpgradeStatus: Upgraded to precise on 2012-01-03 (149 days ago)
---
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.24.
AplayDevices:
 Error: command ['aplay', '-l'] failed with exit code -6: ALSA lib conf.c:1690:(snd_config_load1) :67:1:Unexpected }
 ALSA lib conf.c:3406:(config_file_open) /etc/asound.conf may be old or corrupted: consider to remove or fix it
 **** List of PLAYBACK Hardware Devices ****
ApportVersion: 2.0.1-0ubuntu5
Architecture: i386
ArecordDevices:
 Error: command ['arecord', '-l'] failed with exit code -6: ALSA lib conf.c:1690:(snd_config_load1) :67:1:Unexpected }
 ALSA lib conf.c:3406:(config_file_open) /etc/asound.conf may be old or corrupted: consider to remove or fix it
 **** List of CAPTURE Hardware Devices ****
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path', '/dev/snd/controlC1', '/dev/snd/hwC1D0', '/dev/snd/pcmC1D3p', '/dev/snd/controlC0', '/dev/snd/hwC0D0', '/dev/snd/pcmC0D0c', '/dev/snd/pcmC0D0p', '/dev/snd/pcmC0D1p', '/dev/snd/pcmC0D2p', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
Card0.Amixer.info:
 Error: command ['amixer', '-c', '0', 'info'] failed with exit code -6: ALSA lib conf.c:1690:(snd_config_load1) :67:1:Unexpected }
 ALSA lib conf.c:3406:(config_file_open) /etc/asound.conf may be old or corrupted: consider to remove or fix it
Card0.Amixer.values:
 Error: command ['amixer', '-c', '0'] failed with exit code -6: ALSA lib conf.c:1690:(snd_config_load1) :67:1:Unexpected }
 ALSA lib conf.c:3406:(config_file_open) /etc/asound.conf may be old or corrupted: consider to remove or fix it
Card1.Amixer.info:
 Error: command ['amixer', '-c', '1', 'info'] failed with exit code -6: ALSA lib conf.c:1690:(snd_config_load1) :67:1:Unexpected }
 ALSA lib conf.c:3406:(config_file_open) /etc/asound.conf may be old or corrupted: consider to remove or fix it
Card1.Amixer.values:
 Error: command ['amixer', '-c', '1'] failed with exit code -6: ALSA lib conf.c:1690:(snd_config_load1) :67:1:Unexpected }
 ALSA lib conf.c:3406:(config_file_open) /etc/asound.conf may be old or corrupted: consider to remove or fix it
DistroRelease: Ubuntu 12.04
IwConfig: Error: [Errno 2] No such file or directory
MachineType: System manufacturer System Product Name
Package: linux (not installed)
ProcFB: 0 radeondrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.2.0-23-lowlatency-pae root=/dev/mapper/fsRoot ro rootfstype=ext4 crashkernel=384M-2G:64M,2G-:128M elevator=deadline
ProcVersionSignature: Ubuntu 3.2.0-23.31-lowlatency-pae 3.2.14
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
 linux-restricted-modules-3.2.0-23-lowlatency-pae N/A
 linux-backports-modules-3.2.0-23-lowlatency-pae N/A
 linux-firmware 1.80
RfKill: Error: [Errno 2] No such file or directory
SourcePackage: linux
StagingDrivers: zram
Tags: precise staging precise staging
UdevDb: Error: [Errno 2] No such file or directory
Uname: Linux 3.2.0-23-lowlatency-pae i686
UpgradeStatus: Upgraded to precise on 2012-01-03 (149 days ago)
UserGroups: adm admin audio cdrom dialout dip disk floppy fuse games kvm libvirtd plugdev pulse-access video
WpaSupplicantLog:

dmi.bios.date: 02/17/2011
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 1902
dmi.board.asset.tag: To Be Filled By O.E.M.
dmi.board.name: Crosshair IV Formula
dmi.board.vendor: ASUSTeK Computer INC.
dmi.board.version: Rev 1.xx
dmi.chassis.asset.tag: Asset-1234567890
dmi.chassis.type: 3
dmi.chassis.vendor: Chassis Manufacture
dmi.chassis.version: Chassis Version
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1902:bd02/17/2011:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnCrosshairIVFormula:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion:
dmi.product.name: System Product Name
dmi.product.version: System Version
dmi.sys.vendor: System manufacturer
---
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.24.
ApportVersion: 2.0.1-0ubuntu5
Architecture: i386
ArecordDevices:
 **** List of CAPTURE Hardware Devices ****
 card 0: SB [HDA ATI SB], device 0: VT2020 Analog [VT2020 Analog]
   Subdevices: 2/2
   Subdevice #0: subdevice #0
   Subdevice #1: subdevice #1
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path', '/dev/snd/controlC1', '/dev/snd/hwC1D0', '/dev/snd/pcmC1D3p', '/dev/snd/controlC0', '/dev/snd/hwC0D0', '/dev/snd/pcmC0D0c', '/dev/snd/pcmC0D0p', '/dev/snd/pcmC0D1p', '/dev/snd/pcmC0D2p', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
Card0.Amixer.info:
 Card hw:0 'SB'/'HDA ATI SB at 0xfe3f8000 irq 16'
   Mixer name : 'VIA VT2020'
   Components : 'HDA:11060441,104383e4,00100100'
   Controls : 45
   Simple ctrls : 23
Card1.Amixer.info:
 Card hw:1 'HDMI'/'HDA ATI HDMI at 0xfe9ec000 irq 88'
   Mixer name : 'ATI R6xx HDMI'
   Components : 'HDA:1002aa01,00aa0100,00100000'
   Controls : 6
   Simple ctrls : 1
Card1.Amixer.values:
 Simple mixer control 'IEC958',0
   Capabilities: pswitch pswitch-joined penum
   Playback channels: Mono
   Mono: Playback [on]
DistroRelease: Ubuntu 12.04
IwConfig: Error: [Errno 2] No such file or directory
MachineType: System manufacturer System Product Name
Package: linux (not installed)
ProcFB: 0 radeondrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.2.0-23-lowlatency-pae root=/dev/mapper/fsRoot ro rootfstype=ext4 crashkernel=384M-2G:64M,2G-:128M elevator=deadline
ProcVersionSignature: Ubuntu 3.2.0-23.31-lowlatency-pae 3.2.14
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
 linux-restricted-modules-3.2.0-23-lowlatency-pae N/A
 linux-backports-modules-3.2.0-23-lowlatency-pae N/A
 linux-firmware 1.80
RfKill: Error: [Errno 2] No such file or directory
SourcePackage: linux
StagingDrivers: zram
Tags: precise staging precise staging
UdevDb: Error: [Errno 2] No such file or directory
Uname: Linux 3.2.0-23-lowlatency-pae i686
UpgradeStatus: Upgraded to precise on 2012-01-03 (149 days ago)
UserGroups: adm admin audio cdrom dialout dip disk floppy fuse games kvm libvirtd plugdev pulse-access video
WpaSupplicantLog:

dmi.bios.date: 02/17/2011
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 1902
dmi.board.asset.tag: To Be Filled By O.E.M.
dmi.board.name: Crosshair IV Formula
dmi.board.vendor: ASUSTeK Computer INC.
dmi.board.version: Rev 1.xx
dmi.chassis.asset.tag: Asset-1234567890
dmi.chassis.type: 3
dmi.chassis.vendor: Chassis Manufacture
dmi.chassis.version: Chassis Version
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1902:bd02/17/2011:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnCrosshairIVFormula:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion:
dmi.product.name: System Product Name
dmi.product.version: System Version
dmi.sys.vendor: System manufacturer

Mike Mestnik (cheako) wrote :

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1007089

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete

apport information

tags: added: apport-collected staging
description: updated

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

Mike Mestnik (cheako) wrote :

Ohh, my. Yeah, it looks like PA is not starting for another reason. Though this still looks like a bug in overlayfs that's causing a warning from PA.

I have too many warnings on boot, that much is clear.

apport information

description: updated

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

Mike Mestnik (cheako) wrote :

I've made changes to /etc/asound.conf and now PulseAudio is starting and working. However I still get this warning:

arcadia:~# invoke-rc.d pulseaudio restart
 * Stopping system PulseAudio Daemon [ OK ]
 * Starting system PulseAudio Daemon W: [pulseaudio] main.c: /proc/self/exe does not point to /usr/bin/pulseaudio, cannot self execute. Are you playing games?
                                                                        [ OK ]

Joseph Salisbury (jsalisbury) wrote :

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v3.5kernel[0] (Not a kernel in the daily directory) and install both the linux-image and linux-image-extra .deb packages.

Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag(Only that one tag, please leave the other tags). This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text.

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.5-rc1-quantal/

Changed in linux (Ubuntu):
importance: Undecided → Medium
tags: added: needs-upstream-testing
Mike Mestnik (cheako) wrote :

Joseph,
  Any one running any kernel can test for this bug. Let me write a short shell script that will do just that. I've tested this script on my box and it revealed the bug.

(
set -e
mkdir -p /mnt/test /tmp/usr
mount none /mnt/test -t overlayfs -o lowerdir=/usr,upperdir=/tmp/usr
/mnt/test/bin/tail -f /dev/null&
ls -lart /proc/$!/exe
kill $!
umount /mnt/test
rmdir /mnt/test /tmp/usr
)

# If a red broken link to/bin/tail is printed then the bug exists. If the link points to /mnt/test/bin/tail then there is no bug in the currently running kernel.

Mike Mestnik (cheako) on 2012-06-06
Changed in linux (Ubuntu):
status: Incomplete → Confirmed

Came here while looking for a reason for an unlink failing in https://bugzilla.mozilla.org/show_bug.cgi?id=763192
If so, its rather more important than medium, autoupdating software potentially has the power to break itself

Mike Mestnik (cheako) wrote :

John,
  If so then the bug 763192 would be that it uses /proc/self/exe. Usage of the exe link should be limited to checking to see if this PID is a running instance of a daemon. I'm not sure I've ever seen another good use for the exe link, but I can tell at a glace that 763192 isn't it.

In that bug report ask them if they use this link and also ask them if they can stop and/or what there reasoning is. I highly doubt the value of this link is used for that application at all.

Also, can you test my little script? Your kernel may or may not even be effected by this bug.

Thank you.

Yeah, its red for me,

lrwxrwxrwx 1 root root 0 Jun 11 10:27 /proc/21357/exe -> /bin/tail

john@joran ~ > uname -a
Linux joran.nextraweb.com 3.2.0-25-generic #40-Ubuntu SMP Wed May 23 20:30:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

Mike, we are using /proc/self/exe to substitute the executable with a newer version. We can stop using it, but it was the most secure way to get the path for the executable (see https://mxr.mozilla.org/mozilla-central/source/xpcom/build/BinaryPath.h#83 for the other opportunity).

Mike Mestnik (cheako) wrote :

That's interesting. For self updating applications, well first of all applications should not have access to modify there own source. Leaving that aside as it's obviously working for you and your users are presumably aware of the security risks.

Then I have to point out that the code you linked to is correct, though I may add a replacement for 3 and that's to try and use /proc/self/exe. It should be rare for the other two attempts not to function.

I'm sad to hear that this bug that got into the kernel is effecting your project also. As a work around, you can try adding the path to the mount point. I'll just write some pseudo code in shell.

lnk=$(readlink /proc/$(pgrep firefox)/exe); for ech in $(cut -f2 -d\ /proc/mounts); do [ -f "$ech$lnk" ] && echo "$ech$lnk"; done
/.usr-ro/lib/firefox/firefox
/.usr-rw/lib/firefox/firefox
/usr/lib/firefox/firefox

I guess it's not the most sane solution, you could spice it up a bit.
lnk=$(readlink /proc/$(pgrep firefox)/exe); for ech in $(cut -f2 -d\ /proc/mounts | grep -e ^/var -e ^/usr -e ^/lib -e ^/home); do [ -f "$ech$lnk" ] && echo "$ech$lnk"; done
/usr/lib/firefox/firefox

Mike Mestnik (cheako) wrote :

I forgot ^/opt and the like.

Mike Mestnik (cheako) wrote :

Ohh, one added point is that you are only looking for overlayfs mounts:
lnk=$(readlink /proc/$(pgrep firefox)/exe); for ech in $(cut -f2,3 -d\ /proc/mounts | grep ' overlayfs$' | cut -f1 -d\ ); do [ -f "$ech$lnk" ] && echo "$ech$lnk"; done
/usr/lib/firefox/firefox

Mike Mestnik (cheako) wrote :

What is the status of this bug? Has the OverlayFS team been made aware of the issue? Why can't the Importance be higher?

This seams to effect a number of packages, albeit these packages are broken in there use of sefl/exe.

Is any one even reading these?

Mike Mestnik (cheako) wrote :

This issue is responsible for:
Bug number 1027510 /usr/lib/jvm VS /lib/jvm under overlayfs.
Bug number 1030519 perlvar: EXECUTABLE_NAME or ^X broken under overlayfs.
Bug number 1030520 logwatch: Makes use of $^X, broken under overlayfs.
Bug number 1030530 start-stop-daemon: --exec option broken under overlayfs.

Mike Mestnik (cheako) wrote :

I removed the bug watch on https://bugzilla.mozilla.org/show_bug.cgi?id=763192 because this originally was a pulseaudio bug. It would be prudent to open another bug for Mozilla to track the fix for that package.

I'm also contemplating opening a new bug for the kernel aspect of this bug, though this bug seams to be it.

Serge Hallyn (serge-hallyn) wrote :

Seems to go along with (but perhaps not be a dup of) bug 959352.

Mike Mestnik (cheako) wrote :

Serge,
  Applications should not be making use of /proc, so any bug related to an application reading /proc and implicitly trusting it's contents is a separate bug. However correcting inaccuracy in /proc is worthy of a bug report.

/sys is the file system for applications to be using as a reliable API, though I don't know if I'd personally trust it's contents either.

Applications must check for failures when returning from system calls and /proc is no different, the validity of information exposed must be validated to catch failures.

Mike Mestnik (cheako) wrote :

This is not an issue in Wheezy with aufs.

purgatory:~# find /proc/self/exe -ls
897320 0 lrwxrwxrwx 1 root root 0 Sep 18 16:57 /proc/self/exe -> /usr/bin/find
purgatory:~# grep usr /proc/mounts
/dev/mapper/nyso-usrext /run/shm/26975/usr ext4 rw,noatime,errors=remount-ro,user_xattr,acl,barrier=1,data=ordered 0 0
/dev/mapper/nyso-usrext /run/shm/26975/usr ext4 rw,noatime,errors=remount-ro,user_xattr,acl,barrier=1,data=ordered 0 0
/dev/mapper/nyso-nuusr26975.rw /.usr.rw ext4 rw,relatime,user_xattr,barrier=1,data=writeback 0 0
/dev/mapper/nyso-nuusr26975.ro /.usr.ro squashfs ro,relatime 0 0
none /usr aufs rw,relatime,si=558584e1 0 0
purgatory:~# uname -a
Linux purgatory.mikemestnik.net 3.2.0-3-686-pae #1 SMP Mon Jul 23 03:50:34 UTC 2012 i686 GNU/Linux
purgatory:~# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux testing (wheezy)
Release: testing
Codename: wheezy

Mike Mestnik (cheako) wrote :

1079350 warzone2100

Scott Moser (smoser) on 2013-06-04
tags: added: overlayfs
affects: linux (Ubuntu) → linux-lowlatency (Ubuntu)
Geist (norman-geist) wrote :

This OverlayFS bug is more importance that all of you noticed. It does also break the RPATH library location mechanism and so many software products like: LibreOffice and Java (jre-openjdk)

Chris Friesen (cbf123) wrote :

To follow up on Geist's comments, I've seen it break RPATH as well.

To post a comment you must log in.