powernap 2.15-0ubuntu2 causes unclean rootfs unmount

[heckheck]

I have determined that powernap is causing my system to be unable to unmount root (/) on shutdown or reboot. I noticed that my system was doing an fsck check of root on every boot following installation of powernap. I found that the system was failing to unmount '/' due to it being busy. I instrumented the /etc/init.d/umountroot script by adding 'fuser -vm / > /etc/init.d/umountroot.err 2>&1; sync' after the failing mount command that attempts to unmount '/'. This gave me a list of processes that had open file descriptors. I found the following line in that list

                    root 1414 Frce. powernapd

So it seems that powernapd has an open file handle that is causing root to be busy and so it won't unmount.

I then proceeded to patch the /etc/init/powernap.conf upstart file to add a stop directive so powernapd will get unloaded on shutdown

jheck@paragon:/etc/init$ diff powernap.conf.orig powernap.conf
6a7
> stop on runlevel [!2345]

This fixes the problem. Now the root mount point properly unmounts on shutdown and reboot, and the fsck is no longer happening.

More info:
jheck@paragon:/etc/init$ lsb_release -rd
Description: Ubuntu 11.04
Release: 11.04

jheck@paragon:/etc/init$ apt-cache policy powernap
powernap:
  Installed: 2.15-0ubuntu2
  Candidate: 2.15-0ubuntu2
  Version table:
 *** 2.15-0ubuntu2 0
        100 /var/lib/dpkg/status
     2.6-0ubuntu2.1 0
        500 http://us.archive.ubuntu.com/ubuntu/ natty-updates/main amd64 Packages
     2.6-0ubuntu1 0
        500 http://us.archive.ubuntu.com/ubuntu/ natty/main amd64 Packages

Please note that I built this package from source using dpkg-buildpackage so I could get a later version to go with Natty.

Attached please find my /etc/powernap/config file. Note that I do not have debugging enabled (DEBUG=0), but I noticed that log and err files are still being generated (albeit very small files with nothing in them)

jheck@paragon:/etc/powernap$ ls -la /var/log/powernap.*
-rw-r--r-- 1 root root 731 2011-10-10 23:33 /var/log/powernap.err
-rw-r--r-- 1 root root 20 2011-10-09 18:59 /var/log/powernap.err.1.gz
-rw-r--r-- 1 root root 20 2011-10-06 07:34 /var/log/powernap.err.2.gz
-rw-r--r-- 1 root root 20 2011-10-03 14:06 /var/log/powernap.err.3.gz
-rw-r--r-- 1 root root 20 2011-09-30 15:09 /var/log/powernap.err.4.gz
-rw-r--r-- 1 root root 20 2011-09-28 08:23 /var/log/powernap.err.5.gz
-rw-r--r-- 1 root root 0 2011-10-10 08:19 /var/log/powernap.log
-rw-r--r-- 1 root root 20 2011-10-09 18:59 /var/log/powernap.log.1.gz
-rw-r--r-- 1 root root 20 2011-10-06 07:34 /var/log/powernap.log.2.gz
-rw-r--r-- 1 root root 20 2011-10-03 14:06 /var/log/powernap.log.3.gz
-rw-r--r-- 1 root root 20 2011-09-30 15:09 /var/log/powernap.log.4.gz
-rw-r--r-- 1 root root 20 2011-09-28 08:23 /var/log/powernap.log.5.gz

[Andres Rodriguez (andreserl)]

Hi Jim,

Could you please provide a sample step by step to be able to reproduce this bug report exactly. Also, do you have any keyboard/mouse connected through USB?

Cheers,

[heckheck (jinfo)]

Steps:

1. Install powernap and configure (in my case see above config file)
2. Enable powernap debugging by setting
DEBUG=1
in /etc/powernap/config
3. Reboot system (using 'shutdown -r now')
4. Disable powernap debugging by setting
DEBUG=0
in /etc/powernap/config
5. Reboot system (using 'shutdown -r now')
6. On reboot you should see a warning indicating that the root drive was unmounted uncleanly and see a fsck performed on it

Notes:

I am not sure steps 2-4 are absolutely necessary, but that was part of my process to produce the bug as I was trying to debug powernap power down behavior. I mention it, since I noticed that the debug logs are always present after they have been enabled once, even if you set DEBUG=0 again, and they may need to be present for the bug to manifest (even though they are not currently being used).

As for my system configuration, I do have a USB mouse, and a PS2 keyboard. Also, I have seen this manifest with the root device as a normal hard disk, but I want to note that my nominal system configuration is that I am booting from a USB device that contains the same root filesystem as the hard disk had (the hard disk is disabled in this scenario).

Finally, I want to point out that as I stated in the original bug report, the daemon upstart script does not contain a line

 stop on runlevel [!2345]

This means that the daemon does not get called to stop during shutdown (e.g transition to runlevel 6), and this is almost certainly the cause of the open file handle.

Adding that line to the upstart script eliminates my problem, and makes the script consistent with other daemons such as ssh (ssh.conf)

-Jim

[Andres Rodriguez (andreserl)]

Hi Jim,

Thank you for the info. I think I have an Idea of what might be going wrong... I'll investigate throughout the week!

Thank you again!

[Josep Pujadas-Jubany (jpujades)]

Hello!

I'm having similar problem with 12.04 LTS 32 bit.

I use powernap to poweroff the machines with action script if there is inactivity during 15 minutes.

I have about 50 computers and some of them are experiencing this problem.

Please, could you say me how to test it?

Thanks,

Josep Pujadas-Jubany

[Josep Pujadas-Jubany (jpujades)]

I think I will try the mentioned work-a-round. In fact, many services do the same:

# grep "stop on runlevel \[!2345\]" /etc/init/*
acpid.conf:stop on runlevel [!2345]
anacron.conf:stop on runlevel [!2345]
apport.conf:stop on runlevel [!2345]
atd.conf:stop on runlevel [!2345]
console.conf:stop on runlevel [!2345]
cron.conf:stop on runlevel [!2345]
irqbalance.conf:stop on runlevel [!2345]
nmbd.conf:stop on runlevel [!2345]
smbd.conf:stop on runlevel [!2345]
ssh.conf:stop on runlevel [!2345]
tty1.conf:stop on runlevel [!2345]
upstart-socket-bridge.conf:stop on runlevel [!2345]
whoopsie.conf:stop on runlevel [!2345]

[Josep Pujadas-Jubany (jpujades)]

I confirm that the problem occurs in computers with:

* PS/2 keyboard & USB mouse
* USB keyboard & USB mouse

No found the problem in computers with:

* PS/2 keyboard & PS/2 mouse

At job, I'm migrating all desktops from WinXP to Ubuntu 12.04 LTS. Up to now I don't have computers with:

* USB keyboard
* PS/2 mouse

I think this bug is [High] because crashes the computers affected.