fsck in iscsi mounted filesystems is done at boot time

[Xavi Serrano]

Binary package hint: open-iscsi

Closely related to bug #192158 (iscsi checkfs and mount)

Affected package: open-iscsi

   # apt-cache policy open-iscsi
   open-iscsi:
     Installed: 2.0.871-0ubuntu4
     Candidate: 2.0.871-0ubuntu4
     Version table:
    *** 2.0.871-0ubuntu4 0
           500 http://es.archive.ubuntu.com/ubuntu/ lucid/main Packages
           100 /var/lib/dpkg/status

Fresh install of Ubuntu Server 10.04 (official release, no RC).
root filesystem in iSCSI.
/boot in a separate filesystem.
swap in a dedicated partition (in iSCSI as well).

Tested on both real and virtual machines.

The problem is: if "_netdev" option is given in /etc/fstab for filesystems imported through iSCSI at boot time (all filesystems in my test cases), fsck is done over already mounted filesystems. A warning like this is displayed during boot process on console 1 (tty1):

   WARNING!!! The filesystem is mounted. If you continue you ***WILL***
   cause ***SEVERE*** filesystem damage.

   Do you really want to continue (y/n)?

After that, the message dissappears from tty1 (stays on console 7), and the boot process seem to continue normally. The login prompt appears on tty1:

   Ubuntu 10.04 LTS itest5 tty1

   itest5 login:

But tty1 is actually awaiting response for the previous confirmation for the fsck command. So typing "y" may cause a great disaster to your filesystem:

   itest5 login: y ----> expands to "yes"

And these messages are displayed:

              itest5-root: recovering journal
   itest5-root has been mounted 23 times without being checked, check forced.
   itest5-root: ***** REBOOT LINUX *****
   itest5-root: 46370/411264 files (0.1% non-contiguous), 228075/1644800 blocks

Please note the problem disappears if no "_netdev" option is given in /etc/fstab in the explained test cases. The boot process though should be smart enough to issue a warning and not to execute fsck over such filesystems (which ultimately will lead to severe data corruption).

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: open-iscsi 2.0.871-0ubuntu4
ProcVersionSignature: Ubuntu 2.6.32-21.32-server 2.6.32.11+drm33.2
Uname: Linux 2.6.32-21-server x86_64
Architecture: amd64
Date: Wed May 5 12:16:46 2010
InstallationMedia: Ubuntu-Server 10.04 LTS "Lucid Lynx" - Release amd64 (20100427)
ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: open-iscsi

[Xavi Serrano (xserrano)]

Very related to this bug report, to the point I think the problem is the same.

Assuming we use no "_netdev" option in /etc/fstab.

I intentionally caused data corruption to one of my filesystems (ext4) mounted through iSCSI.
The filesystem is mounted under /users at boot time, and does not contain any system-critical data.

Data corruption was caused with something similar to:
   # dd if=/dev/zero of=/dev/itest4/users bs=512 count=3 skip=NN

where NN is a couple of randomly picked blocks (e.g: 29000 and 90), and /dev/itest4/users is the logical volume (LV) where the "/users" filesystem lies.

Setting the /users filesystem mount count to the maximum with "tune2fs -C" will force a fsck in next boot.

So we reboot the machine (it is a virtual machine under KVM).
And boot process freezes when arriving to the fsck part... and nothing can be done. No <Control+C>, no other tty's reponding.

Rebooting in recovery mode from GRUB leads us to the same point: nothing can be done.

So it is required to boot the system with an installation CDROM and pick "Repair a broken system". After all the required steps (connecting to iSCSI target, mounting root and executing shell) we can manually repair the corrupted /users filesystem with a fsck command.

In my opinion, this (booting with an installation CDROM) shouldn't be necessary for non-critical filesystems such us "/users" (I am not talking about root or "/var"). For iSCSI imported filesystems during boot process, integrity must be checked in some way, and if errors are encountered, proper mechanisms should be provided to allow the administrator to fix the problem: entering single-user mode, etc.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Paweł Szlubecki (pawel-szlubecki)]

Looks like bug in mountall. I made small test on Ubuntu 12.4LTS(without latest updates):

I removed file "/etc/init/mountall-net.conf" and modified one line in "/etc/init/mountall.conf" by adding extra option "--dev-wait-time=200" to invocation of mountall (after that it looks like "exec mountall --daemon $force_fsck $fsck_fix --dev-wait-time=200"). After that problem never happened again.

Package "mountall" was in version 2.36.4(so its probably latest version).

I don't know if its important , but my rootfs and /etc are on NFS mounted with option "nolock" ("/etc" is mouted by initrd, its my own extension also rootfs is read only ).