10.04 beta 1 mountall fails to mount iSCSI volume at boot

[ptashek]

Binary package hint: mountall

Description: Ubuntu lucid (development branch)
Release: 10.04
Relevant Packages:
- mountall 2.8
- mount 2.17.2-0ubuntu1
- open-iscsi 2.0.871-0ubuntu4
- open-iscsi-utils 2.0.871-0ubuntu4

After installing open-iscsi I can discover and login to a target. This creates relevant entries in /etc/iscsi/nodes. In iscsid.conf "node.startup" is set to "automatic". My /etc/fstab contains an entry to mount the iSCSI volume at boot:

UUID=<disk_uid> /var/lib/eucalyptus/instances ext3 rw,relatime,_netdev 0 0

UUID is determined by using "fdisk -l" and then checking the relevant drive entry under /dev/disk/by-uuid. Manual mount works, so it is expected that it should work on boot.

Expected behavior:
The iSCSI volume is mounted according to its fstab entry at boot

Actual behavior:
Boot fails and hangs with the following message:

mount: special device UUID=<disk_uid_from_fstab> does not exist
mountall: mount /var/lib/eucalyptus/instances [900] terminated with status 32
mountall: Filesystem could not be mounted: /var/lib/eucalyptus/instances
mountall:mountall.c:2726: Not reached assertion failed in boredom_timeout
General error mounting filesystems.
A maintenance shell will now be started.

After dropping to the maintenance shell and issuing "mount -a -O _netdev" the iSCSI volume mounts fine. Also, dmesg shows that iSCSI is started at boot and the volume is detected and attached and "fdisk -l" confirms this.

[ 17.762576] eth0: Broadcom NetXtreme II BCM5708 1000Base-SX (B2) PCI-X 64-bit 133MHz found at mem da000000, IRQ 17, node addr 00:1a:64:xx:yy:aa
[ 17.762613] alloc irq_desc for 19 on node -1
[ 17.762615] alloc kstat_irqs on node -1
[ 17.762622] bnx2 0000:06:00.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19
[ 17.762758] bnx2 0000:06:00.0: firmware: requesting bnx2/bnx2-mips-06-5.0.0.j3.fw
[ 17.764349] bnx2 0000:06:00.0: firmware: requesting bnx2/bnx2-rv2p-06-5.0.0.j3.fw
[ 17.766241] eth1: Broadcom NetXtreme II BCM5708 1000Base-SX (B2) PCI-X 64-bit 133MHz found at mem d8000000, IRQ 19, node addr 00:1a:64:xx:yy:zz
[ 17.874168] Bridge firewalling registered
[ 17.891842] device eth0 entered promiscuous mode
[ 17.892682] alloc irq_desc for 58 on node -1
[ 17.892685] alloc kstat_irqs on node -1
[ 17.892698] bnx2 0000:04:00.0: irq 58 for MSI/MSI-X
[ 18.001471] bnx2: eth0: using MSI
[ 18.003037] ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 18.040402] bnx2: eth0 NIC SerDes Link is Up, 1000 Mbps full duplex
[ 18.041606] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 18.041635] br0: port 1(eth0) entering learning state
[ 18.047339] iscsi: registered transport (iser)
[ 27.030008] br0: port 1(eth0) entering forwarding state
[ 28.381880] br0: no IPv6 routers present
[ 28.440007] eth0: no IPv6 routers present
[ 49.427827] scsi4 : iSCSI Initiator over TCP/IP
[ 49.456335] connection1:0: detected conn error (1020)
[ 50.218722] scsi 4:0:0:0: RAID HP HSV210 6100 PQ: 0 ANSI: 5
[ 50.218881] scsi 4:0:0:0: Attached scsi generic sg2 type 12
[ 50.220581] scsi 4:0:0:1: Direct-Access HP HSV210 6100 PQ: 0 ANSI: 5
[ 50.220758] sd 4:0:0:1: Attached scsi generic sg3 type 0
[ 50.221991] sd 4:0:0:1: [sdb] 4292870144 512-byte logical blocks: (2.19 TB/1.99 TiB)
[ 50.223156] sd 4:0:0:1: [sdb] Write Protect is off
[ 50.223159] sd 4:0:0:1: [sdb] Mode Sense: 97 00 10 08
[ 50.224469] sd 4:0:0:1: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA
[ 50.229737] sdb: sdb1
[ 50.301949] sd 4:0:0:1: [sdb] Attached SCSI disk

[Scott James Remnant (Canonical) (canonical-scott)]

I have reassigned this bug to open-iscsi

From the report, it appears that mountall has actually run "mount" for this device - and it's the mount that fails.

This implies an error with the event being sent too early?

[Scott James Remnant (Canonical) (canonical-scott)]

(the assertion error has been fixed)

[Francesco Pretto (ceztko)]

Who is responsibile to honour _netdev option?

In my /etc/fstab I have the line:
/dev/sda /storage ext4 defaults,auto,_netdev 0 0

(yes, for ease administration, I created a partition in the whole iscsi "disk", that is a logical volume in the iscsitarget)

Still I get many:

mount: special device /dev/sda does not exist
mountall: mount /storage [xxx] terminated with status 32

as the reporter. At last the fs is mounted **anyway**, so mountall is repeating mount attempts, but boot is soiled with many warnings, with a last:

mountall: Disconnected from plymouth

in front of the first tty.

Not clear if this is the expected behavior or the boot is supposed to be more silent.

[Serge van Ginderachter (svg)]

I have a similar problem on Ubuntu precise after I upgraded to 2.0.873-3ubuntu5~ubuntu12.04.1 from backports. Before it worked well with 2.0.871-0ubuntu9.12.04.2 and 2.0.871-0ubuntu9

The error message at boot time is not exact the same, given a newer distro, but the problem is the same: the iscsi device is not connected yet, One is then given the choice to (s)kip mounting or (M)anual recovery.

Pressing S resumes booting. The first message that briefly appears afther that is the startup of the open-iscsi daemon....
When logging in and issuing mount -a the device gets mounted without further problem.

For sake of completeness, our fstab does have the _netdev option.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[P (p92)]

This bug is affecting me on 15.04 -
iscsi does not start at boot
boot time is very long due to timeouts
there are error messages in dmesg that explain why (see attachement).
After boot I can start open-iscsi mannually and mount iscsi devices.

[ 4.581291] systemd-sysv-generator[238]: Overwriting existing symlink /run/systemd/generator.late/umountiscsi.service with real service
[ 4.652179] systemd[1]: Found dependency on open-iscsi.service/start
[ 4.652196] systemd[1]: Breaking ordering cycle by deleting job open-iscsi.service/start
[ 4.652200] systemd[1]: Job open-iscsi.service/start deleted to break ordering cycle starting with sysinit.target/start

[P (p92)]

maybe not an iscsi problem but a systemd problem now

[Chris Johnson (ctjctj)]

The issue is that the mountall command processes all entries in /etc/fstab, including -netdev. If the only requirement is for the network to be up, such as NFS or CIFS then it works correctly. But for iscsi the device is "remote" but requires that open-iscsi init.d script be run first. Thus the mountall fails because the hard device doesn't exist yet.

The work around that I'm currently using is to set all the iscsi devices in /etc/fstab to "noauto". Then the open-iscsi start script is modified so the "mount -a -O _netdev" command is replaced with "mount /mount_point" We can't use the -a option because the noauto causes -a to fail. We can't use _netdev because the mountall fails.