umount of /var partition fails during shutdown, due to lingering dhclient

[Graham Waugh]

The 'Unmounting local filesystems...' stage of shutdown fails, and reports that the /var partition is busy.
This occurs where /var is on its own separate partition.

I am using 64 bit Kubuntu 12.10 (quantal) but I was also able to reproduce this problem on a clean install of
xubuntu 12.10 i386 in virtualbox.

I added debugging messages to the script: /etc/init.d/umountfs
'fuser -m /var' shows that the only process with a file open on /var is dhclient, which has a dhcp lease file in /var/lib/dhcp/
Using 'fuser -k -m /var' in the umountfs script kills the offending process, allowing for a clean shutdown,
but that's not the right fix.

I'm not sure how to proceed from here.
dhclient is a child process of NetworkManager - should NetworkManager clean it up?
the /etc/init.d/sendsigs script, which kills other processes during shutdown, is specifically blocked from killing dhclient
by an entry in /run/sendsigs.omit.d/ so it is not clear to me what should be happening.

Apologies if I filed this in the wrong place.

Description: Ubuntu 12.10
Release: 12.10

upstart:
  Installed: 1.5-0ubuntu9
  Candidate: 1.5-0ubuntu9
  Version table:
 *** 1.5-0ubuntu9 0
        500 http://jp.archive.ubuntu.com/ubuntu/ quantal/main amd64 Packages
        100 /var/lib/dpkg/status

[Steve Langasek (vorlon)]

> dhclient is a child process of NetworkManager - should
> NetworkManager clean it up?

Yes, it should.

[Clint Byrum (clint-fewbar)]

Actually I don't think this is network-manager's fault. By all rights, it should be *dead* by this point.

umountnfs.sh emits unmounted-remote-filesystems, which stops networking (stop on umounted-remote-filesystems), which emits deconfiguring-networking, which causes dbus to stop (stop on deconfiguring-networking), which causes network-manager to stop (stop on stopping dbus). So umountnfs.sh should not exit until network-manager (the upstart job) is considered *stopped*. This means that NetworkManager should be completely *dead*, even SIGKILL'd if it didn't die within 5 seconds.

/etc/init/networking.conf emits deconfiguring-networking in post-stop. If I'm reading the upstart state machine properly, job_finished() is not called until a job's state gets to JOB_WAITING, which would have to be after the post-stop. However, its possible post-stop had an error, which would still finish the stopping event for networking, but would not emit deconfiguring-networking. So, verbose logging would confirm this by showing that the networking post-stop failed.

If that shows true, I believe this would be resolved by changing dbus to

stop on stopping networking

[Stéphane Graber (stgraber)]

Can you attach a tarball of /var/log/upstart/ ?

[Graham Waugh (gureamu)]

> Can you attach a tarball of /var/log/upstart/ ?

Ok. These are from the xubuntu virtualbox install where I reproduced the bug.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.