Disk check on every boot

[P Jones]

Binary package hint: e2fsprogs

On a fully updated (as of 07/10/10) Lucid Lynx, disk check runs on every boot/reboot. No errors are reported.

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: e2fsprogs 1.41.11-1ubuntu2
ProcVersionSignature: Ubuntu 2.6.32-23.37-generic 2.6.32.15+drm33.5
Uname: Linux 2.6.32-23-generic i686
Architecture: i386
Date: Sat Jul 10 07:51:40 2010
InstallationMedia: Ubuntu 10.04 LTS "Lucid Lynx" - Release i386 (20100427.1)
ProcEnviron:
 PATH=(custom, user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: e2fsprogs
---
Architecture: i386
DistroRelease: Ubuntu 10.04
InstallationMedia: Ubuntu 10.04 LTS "Lucid Lynx" - Release i386 (20100427.1)
Package: e2fsprogs 1.41.11-1ubuntu2
PackageArchitecture: i386
ProcEnviron:
 PATH=(custom, user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.32-24.39-generic 2.6.32.15+drm33.5
Tags: lucid
Uname: Linux 2.6.32-24-generic i686
UserGroups: adm admin audio cdrom fuse lpadmin netdev plugdev sambashare vboxusers video

[P Jones (deerfieldtech)]

I should add that this issue is occurring on a laptop with full disk encryption installed with the alternate CD, no other OS installed.

[Surbhi Palande (csurbhi)]

@P Jones, Can you attach the output of:
1) apport-collect
2) sudo tune2fs -l <device corresponding to />
3) less /etc/fstab

Thanks!

[P Jones (deerfieldtech)]

apport information

[P Jones (deerfieldtech)]

tune2fs for /dev/sda1 attached. Since this is an encrypted volume, /dev/sda1 is /boot .

[P Jones (deerfieldtech)]

fstab attached.

[Theodore Ts'o (tytso)]

The file system is not using a journal (i.e., it is an ext2 file system instead of an ext3 or ext4 file system). Was this deliberate?

At a guess it looks like it's not getting unmounted for some reason when you shutdown your system, which is why it is getting checked after each shutdown/reboot. You are doing an explicit shutdown as opposed to just powering off your system, right?

[P Jones (deerfieldtech)]

@ Theodore Ts'o: No, it was not my intention to format the /boot partition as ext2. I used the automated option to use the entire disk and encrypt it using the alternate CD. I don't recall being prompted to choose file systems or do the partition layout.

Also, I always do a proper shutdown using the menu option.

[Surbhi Palande (csurbhi)]

Reassigning this bug to mountall -
When a device is not ready by the time a mount is about to be attempted, a wait is performed till udev catches up and discovers the device completely. If such wait is performed then a fsck is always called. Devices under device mapper are susceptible to this wait and hence a per boot fsck is occurring in this case. This needs fixing in mountall. So resigning the bug to mountall rather than e2fsprogs.

[Johan Kiviniemi (ion)]

Please add ‘--debug >/dev/mountall.log 2>&1’ to the ‘exec mountall…’ line in /etc/init/mountall.conf and reboot the system, reproducing the undesired disk check. Then attach /dev/mountall.log to this bug report. Thanks.

[Surbhi Palande (csurbhi)]

ok, so mountall calls fsck for all filesystems, but its upto fsck to do the actual fs check (by looking at the max mount count and interval check). So reassigning the bug to e2fsprogs.

[Surbhi Palande (csurbhi)]

@ P Jones, can you do the following:
1) Before rebooting: sudo -s
    a) umount /boot
    b) tune2fs -l /dev/sda1 2>&1 > /<some dir you want>/tune2fs-sda1. Attach this output here.
    c) tune2fs -l /dev/mapper/dragon-root 2>&1 > /<some dir you want>/tune2fs-root. Attach this output here.
    d) As Johan Kiviniemi has suggested, add the --debug switch to the mountall daemon line in /etc/init/mountall.conf. Attach the output here.
   e) reboot; Attach the /dev/mountall.log here.

I have installed Lucid in qemu and encrypted the disk. However I am not able to reproduce this bug here. So, the output from your machine will be helpful in debugging this. Thanks a lot!

[Theodore Ts'o (tytso)]

Surbhi,

The bug reporter said that he did not intentionally chose ext2 as his file system for his file system. I wonder if there is also an installer bug hiding here?

[P Jones (deerfieldtech)]

Output of tune2fs -l /dev/sda1 2>&1 > /tmp/tune2fs-sda1

[P Jones (deerfieldtech)]

Output of tune2fs -l /dev/mapper/dragon-root 2>&1 > /tmp/tune2fs-root

[P Jones (deerfieldtech)]

/dev/mountall.log attached.

[Surbhi Palande (csurbhi)]

P Jones, looks like this time, your /boot or /dev/sda2 was not fscked. Line from your mountall.log:
"/dev/sda1: clean, 224/124496 files, 99974/248832 blocks" indicates that fsck was invoked but did not go through the entire filesystem, instead only read through the super block the filesystem is clean. Also your root filesystem is not getting fscked :)

So it does seem that if your file system is getting fscked at boot then something wrong is possibly happening at poweroff/reboot. To verify this, can you please do the following once again:

1) Before rebooting: sudo -s
    a) umount /boot
    b) tune2fs -l /dev/sda1 2>&1 > /<some dir you want>/tune2fs-sda1.before-fsck. Attach this here.
    c) This time run a fsck on it : fsck -V -t ext2 /dev/sda1 2>&1 > /<some dir you want>/fsck-sda1.out. Attach this here
    d) tune2fs -l /dev/sda1 2>&1 > /<some dir you want>/tune2fs-sda1.after-fsck. Attach this output here.
    e) Add the --debug switch to the mountall daemon line in /etc/init/mountall.conf. Reboot; Attach the /dev/mountall.log here as mountall-reboot.log
    f) tune2fs -l /dev/sda1 2>&1 > /<some dir you want>/tune2fs-sda1-after-reboot. Attach this here

As expected fsck should not have any effect in the above path. Your output should say that.

2) Now simply reboot without un-mounting /boot and then attach the new /dev/moutall.log as mountall-fresh.log here. We expect fsck to touch your /boot in this case. Also attach the output of tune2fs -l /dev/sda2 2>&1 > /<some dir you want>/tune2fs-sda1.fresh

Thanks a lot for your help.

[P Jones (deerfieldtech)]

tune2fs -l /dev/sda1 2>&1 > /root/tune2fs-sda1.before-fsck attached.

[P Jones (deerfieldtech)]

fsck -V -t ext2 /dev/sda1 2>&1 > /root/fsck-sda1.out attached.

[P Jones (deerfieldtech)]

tune2fs -l /dev/sda1 2>&1 > /root/tune2fs-sda1.after-fsck attached.

[P Jones (deerfieldtech)]

/dev/mountall.log renamed to mountall-reboot.log attached.

[P Jones (deerfieldtech)]

tune2fs -l /dev/sda1 2>&1 > /root/tune2fs-sda1-after-reboot attached.

[P Jones (deerfieldtech)]

/dev/mountall.log renamed to mountall-fresh.log attached.

[P Jones (deerfieldtech)]

I assume the last instruction is supposed to be tune2fs -l /dev/sda1 2>&1 > /<some dir you want>/tune2fs-sda1.fresh, NOT tune2fs -l /dev/sda2 2>&1 > /<some dir you want>/tune2fs-sda1.fresh (notice the /dev/sda2 after the -l switch). Output of tune2fs -l /dev/sda1 2>&1 > /tmp/tune2fs-sda1.fresh is attached.

[Surbhi Palande (csurbhi)]

@P Jones, Thanks for the output files.
/dev/sda1 is not getting unmounted at reboot or halt time, which is why it is getting checked at every boot. Can you please copy paste the output of:

1) ls -lh /etc/rc0.d/
2) ls -lh /etc/rc6.d/

And then: open your /etc/init.d/umountfs and then add the following:
1) at the top of the file after the comments add VERBOSE="yes"
2) and in the #Unmount local filesystems section add "2>1 | tee /umount-local.out" to the line:
fstab-decode umount -f -r -d $REG_MTPTS

so that it looks like:
fstab-decode umount -f -r -d $REG_MTPTS 2>1 | tee /umount-local.out

Then reboot; and after that attach /umount-local.out here.

Thanks!

[Surbhi Palande (csurbhi)]

@Ted Tso, Thanks a lot for your help with this bug resolution. I brought up the issue of /boot being formatted as ext2 by default. It seems that, currently, a user can opt to format /boot with another fs but by default as of now /boot is formatted with ext2. This conscious decision tends to arise from the fact that both the size of /boot and the number of changes made to it are quiet small. In case of a crash, the correction of the filesystem would be quick owing to the small size of the partition. Thus the space and time overhead of a journal based ext fs seems to be avoidable in this case.

[Colin Watson (cjwatson)]

And just to clarify further, the need for a separate /boot is rare nowadays; encrypted / is one of the few remaining cases. Obviously we don't use ext2 by default for larger filesystems.

[Theodore Ts'o (tytso)]

Out of curiosity, was this a change (formatting small /boot partitions using ext2) something that started with Lucid?

I wonder if the issue of not correctly umounting /boot has been around for a while, but was masked by the fact that the journal was getting replayed automatically by fsck and/or the first time /boot is mounted. Just a theory, of course. We still don't know why /boot isn't getting unmounted cleanly on shutdown for this user.

[P Jones (deerfieldtech)]

pjones@dragon:~$ ls -lh /etc/rc0.d/
total 4.0K
lrwxrwxrwx 1 root root 24 2010-07-16 09:44 K20openvas-server -> ../init.d/openvas-server
lrwxrwxrwx 1 root root 17 2010-06-12 16:00 K20vboxdrv -> ../init.d/vboxdrv
lrwxrwxrwx 1 root root 20 2010-06-12 14:12 K31atieventsd -> ../init.d/atieventsd
lrwxrwxrwx 1 root root 19 2010-06-12 09:55 K74bluetooth -> ../init.d/bluetooth
lrwxrwxrwx 1 root root 17 2010-06-12 15:48 K80openvpn -> ../init.d/openvpn
-rw-r--r-- 1 root root 353 2009-09-07 14:58 README
lrwxrwxrwx 1 root root 29 2010-06-12 09:55 S10unattended-upgrades -> ../init.d/unattended-upgrades
lrwxrwxrwx 1 root root 22 2010-06-12 09:57 S15wpa-ifupdown -> ../init.d/wpa-ifupdown
lrwxrwxrwx 1 root root 18 2010-06-12 08:46 S20sendsigs -> ../init.d/sendsigs
lrwxrwxrwx 1 root root 17 2010-06-12 08:46 S30urandom -> ../init.d/urandom
lrwxrwxrwx 1 root root 22 2010-06-12 08:46 S31umountnfs.sh -> ../init.d/umountnfs.sh
lrwxrwxrwx 1 root root 20 2010-06-12 08:46 S35networking -> ../init.d/networking
lrwxrwxrwx 1 root root 18 2010-06-12 08:46 S40umountfs -> ../init.d/umountfs
lrwxrwxrwx 1 root root 20 2010-06-12 08:52 S48cryptdisks -> ../init.d/cryptdisks
lrwxrwxrwx 1 root root 26 2010-06-12 08:52 S59cryptdisks-early -> ../init.d/cryptdisks-early
lrwxrwxrwx 1 root root 20 2010-06-12 08:46 S60umountroot -> ../init.d/umountroot
lrwxrwxrwx 1 root root 14 2010-06-12 08:46 S90halt -> ../init.d/halt

[P Jones (deerfieldtech)]

pjones@dragon:~$ ls -lh /etc/rc6.d/
total 4.0K
lrwxrwxrwx 1 root root 24 2010-07-16 09:44 K20openvas-server -> ../init.d/openvas-server
lrwxrwxrwx 1 root root 17 2010-06-12 16:00 K20vboxdrv -> ../init.d/vboxdrv
lrwxrwxrwx 1 root root 20 2010-06-12 14:12 K31atieventsd -> ../init.d/atieventsd
lrwxrwxrwx 1 root root 19 2010-06-12 09:55 K74bluetooth -> ../init.d/bluetooth
lrwxrwxrwx 1 root root 17 2010-06-12 15:48 K80openvpn -> ../init.d/openvpn
-rw-r--r-- 1 root root 351 2009-09-07 14:58 README
lrwxrwxrwx 1 root root 29 2010-06-12 09:55 S10unattended-upgrades -> ../init.d/unattended-upgrades
lrwxrwxrwx 1 root root 22 2010-06-12 09:57 S15wpa-ifupdown -> ../init.d/wpa-ifupdown
lrwxrwxrwx 1 root root 18 2010-06-12 08:46 S20sendsigs -> ../init.d/sendsigs
lrwxrwxrwx 1 root root 17 2010-06-12 08:46 S30urandom -> ../init.d/urandom
lrwxrwxrwx 1 root root 22 2010-06-12 08:46 S31umountnfs.sh -> ../init.d/umountnfs.sh
lrwxrwxrwx 1 root root 20 2010-06-12 08:46 S35networking -> ../init.d/networking
lrwxrwxrwx 1 root root 18 2010-06-12 08:46 S40umountfs -> ../init.d/umountfs
lrwxrwxrwx 1 root root 20 2010-06-12 08:52 S48cryptdisks -> ../init.d/cryptdisks
lrwxrwxrwx 1 root root 26 2010-06-12 08:52 S59cryptdisks-early -> ../init.d/cryptdisks-early
lrwxrwxrwx 1 root root 20 2010-06-12 08:46 S60umountroot -> ../init.d/umountroot
lrwxrwxrwx 1 root root 16 2010-06-12 08:46 S90reboot -> ../init.d/reboot

[P Jones (deerfieldtech)]

umount-local.out is a zero length file. My edited /etc/init.d/umountfs is pasted below for your review.

#! /bin/sh
### BEGIN INIT INFO
# Provides: umountfs
# Required-Start:
# Required-Stop: umountroot
# Default-Start:
# Default-Stop: 0 6
# Short-Description: Turn off swap and unmount all local file systems.
# Description:
### END INIT INFO

VERBOSE="yes"

PATH=/sbin:/usr/sbin:/bin:/usr/bin
. /lib/init/vars.sh

. /lib/lsb/init-functions

umask 022

do_stop () {
 exec 9<&0 </proc/mounts

 PROTECTED_MOUNTS="$(sed -n '0,/^\/[^ ]* \/ /p' /proc/mounts)"
 WEAK_MTPTS="" # be gentle, don't use force
 REG_MTPTS=""
 TMPFS_MTPTS=""
 while read -r DEV MTPT FSTYPE REST
 do
  echo "$PROTECTED_MOUNTS" | grep -qs "^$DEV $MTPT " && continue
  case "$MTPT" in
    /|/proc|/dev|/.dev|/dev/pts|/dev/shm|/dev/.static/dev|/proc/*|/sys|/lib/init/rw)
   continue
   ;;
    /var/run)
   continue
   ;;
    /var/lock)
   continue
   ;;
  esac
  case "$FSTYPE" in
    proc|procfs|linprocfs|devfs|sysfs|usbfs|usbdevfs|devpts)
   continue
   ;;
    tmpfs)
   TMPFS_MTPTS="$MTPT $TMPFS_MTPTS"
   ;;
    *)
   if echo "$PROTECTED_MOUNTS" | grep -qs "^$DEV "; then
    WEAK_MTPTS="$MTPT $WEAK_MTPTS"
   else
    REG_MTPTS="$MTPT $REG_MTPTS"
   fi
   ;;
  esac
 done

 exec 0<&9 9<&-

 #
 # Make sure tmpfs file systems are umounted before turning off
 # swap, to avoid running out of memory if the tmpfs filesystems
 # use a lot of space.
 #
 if [ "$TMPFS_MTPTS" ]
 then
  if [ "$VERBOSE" = no ]
  then
   log_action_begin_msg "Unmounting temporary filesystems"
   fstab-decode umount $TMPFS_MTPTS
   log_action_end_msg $?
  else
   log_daemon_msg "Will now unmount temporary filesystems"
   fstab-decode umount -v $TMPFS_MTPTS
   log_end_msg $?
  fi
 fi

 #
 # Deactivate swap
 #
 if [ "$VERBOSE" = no ]
 then
  log_action_begin_msg "Deactivating swap"
  swapoff -a >/dev/null
  log_action_end_msg $?
 else
  log_daemon_msg "Will now deactivate swap"
  swapoff -a -v
  log_end_msg $?
 fi

 #
 # Unmount local filesystems
 #
 if [ "$WEAK_MTPTS" ]; then
  # Do not use -f umount option for WEAK_MTPTS
  if [ "$VERBOSE" = no ]
  then
   log_action_begin_msg "Unmounting weak filesystems"
   fstab-decode umount -r -d $WEAK_MTPTS
   log_action_end_msg $?
  else
   log_daemon_msg "Will now unmount weak filesystems"
   fstab-decode umount -v -r -d $WEAK_MTPTS
   log_end_msg $?
  fi
 fi
 if [ "$REG_MTPTS" ]
 then
  if [ "$VERBOSE" = no ]
  then
   log_action_begin_msg "Unmounting local filesystems"
   fstab-decode umount -f -r -d $REG_MTPTS 2>1 | tee /umount-local.out
   log_action_end_msg $?
  else
   log_daemon_msg "Will now unmount local filesystems"
   fstab-decode umount -f -v -r -d $REG_MTPTS
   log_end_msg $?
  fi
 fi
}

case "$1" in
  start)
 # No-op
 ;;
  restart|reload|force-reload)
 echo "Error: argument '$1' not supported" >&2
 exit 3
 ;;
  stop)
 do_stop
 ;;
  *)
 echo "Usage: $0 start|stop" >&2
 exit 3
 ;;
esac

:

[Surbhi Palande (csurbhi)]

@P Jones,
Please add the 2>1 | tee /umount-local.out to the else part of the local filesystem. I have pasted the snippet here.

then
  if [ "$VERBOSE" = no ]
  then
   log_action_begin_msg "Unmounting local filesystems"
   fstab-decode umount -f -r -d $REG_MTPTS
   log_action_end_msg $?
  else
   log_daemon_msg "Will now unmount local filesystems"
   fstab-decode umount -f -v -r -d $REG_MTPTS 2>1 | tee /umount-local.out
   log_end_msg $?
  fi
 fi

Thanks!

[P Jones (deerfieldtech)]

@Surbhi

Ok, now in / I have an empty (again) umount-local.out, but I also have a file named "1" that has this as its contents:

umount2: Device or resource busy
umount: /smb busy - remounted read-only
Segmentation fault

[P Jones (deerfieldtech)]

I commented out the "/smb <--> /etc/auto.smb" line in my /etc/auto.master and rebooted twice. On the second (and now third) reboots, no disk check. And now I have an empty /1 file and an empty /umount-local.out .

[P Jones (deerfieldtech)]

After powering off my laptop and powering it back on a few hours later, the disk checks are back...

[Launchpad Janitor (janitor)]

[Expired for e2fsprogs (Ubuntu) because there has been no activity for 60 days.]