| 2013-01-22 16:02:04 |
Joris S'heeren |
description |
Ubuntu Server 12.04.1 LTS (GNU/Linux 3.2.0-29-virtual x86_64)
glusterfs 3.3.1 (from semiosis PPA)
mountall 2.36.3
We are running an OpenStack cloud (essex). We use Gluster as a network filesystem for storage of data.
When we updated mountall ono our client VMs we experienced a failure of the automatic mounting of the gluster-shares at boot-time. The gluster-shares can be mounted manually when the VMs are booted and online.
When we downgraded the mountall package from new version 2.36.3 to old version 2.36, the gluster-shares mounted again during boot.
boot.log (added --verbose to /etc/init/mountall.conf)
=======
Connected to Plymouth
/ is local
/proc is virtual
/sys is virtual
/sys/fs/fuse/connections is virtual
/sys/kernel/debug is virtual
/sys/kernel/security is virtual
/dev is virtual
/dev/pts is virtual
/tmp is local
/run is virtual
/run/lock is virtual
/run/shm is virtual
/home is nowait
/scratch is nowait
/opt is nowait
/mnt is nowait
mounting event sent for /sys/fs/fuse/connections
mounting event sent for /sys/kernel/debug
mounting event sent for /sys/kernel/security
mounting event sent for /run/lock
mounting event sent for /run/shm
mounting event sent for /
mounted event handled for /sys
swap finished
local 0/2 remote 0/0 virtual 1/10 swap 0/0
mounted event handled for /dev/pts
local 0/2 remote 0/0 virtual 2/10 swap 0/0
mounting event handled for /sys/fs/fuse/connections
mounting /sys/fs/fuse/connections
mount /sys/fs/fuse/connections [282] exited normally
mounting event handled for /sys/kernel/debug
mounting /sys/kernel/debug
mount /sys/kernel/debug [296] exited normally
mounting event handled for /sys/kernel/security
mounting /sys/kernel/security
mount /sys/kernel/security [299] exited normally
mounting event handled for /run/lock
mounting /run/lock
mount /run/lock [301] exited normally
mounting event handled for /run/shm
mounting /run/shm
mount /run/shm [304] exited normally
mounted event handled for /proc
local 0/2 remote 0/0 virtual 3/10 swap 0/0
mounting event handled for /
remounting /
mount / [310] exited normally
mounted event handled for /dev
mount / [321] exited normally
mount /proc [322] exited normally
mount /sys [323] exited normally
mount /sys/fs/fuse/connections [324] exited normally
mount /sys/kernel/debug [325] exited normally
mount /sys/kernel/security [326] exited normally
mount /dev [327] exited normally
mount /dev/pts [328] exited normally
mount /run [329] exited normally
mount /run/lock [330] exited normally
mount /run/shm [331] exited normally
local 0/2 remote 0/0 virtual 4/10 swap 0/0
mounted event handled for /sys/fs/fuse/connections
local 0/2 remote 0/0 virtual 5/10 swap 0/0
mounted event handled for /sys/kernel/security
local 0/2 remote 0/0 virtual 6/10 swap 0/0
mounting event sent for /tmp
mounting event sent for /home
mounting event sent for /scratch
mounting event sent for /opt
mounting event sent for /mnt
mounted event handled for /sys/kernel/debug
local 0/2 remote 0/0 virtual 7/10 swap 0/0
mounted event handled for /run/lock
local 0/2 remote 0/0 virtual 8/10 swap 0/0
mounted event handled for /run/shm
local 0/2 remote 0/0 virtual 9/10 swap 0/0
mounting event handled for /tmp
mounting event handled for /home
mounting /home
Mount failed. Please check the log file for more details.
mountall: mount /home [333] terminated with status 1
mountall: Filesystem could not be mounted: /home
mounting event handled for /scratch
mounting /scratch
Mount failed. Please check the log file for more details.
mountall: mount /scratch [389] terminated with status 1
mountall: Filesystem could not be mounted: /scratch
mounting event handled for /opt
mounting /opt
Mount failed. Please check the log file for more details.
mountall: mount /opt [424] terminated with status 1
mountall: Filesystem could not be mounted: /opt
mounting event handled for /mnt
mounting /mnt
mount /mnt [459] exited normally
mounting event sent for /home
mounting event sent for /scratch
mounting event sent for /opt
mounted event handled for /tmp
local 1/2 remote 0/0 virtual 9/10 swap 0/0
mounting event handled for /home
mounting event handled for /scratch
mounting event handled for /opt
mounted event handled for /mnt
local 1/2 remote 0/0 virtual 9/10 swap 0/0
mounting event sent for /home
mounting event sent for /scratch
mounting event sent for /opt
mounting event handled for /home
mounting event handled for /scratch
mounting event handled for /opt
cloud-init start-local running: Tue, 22 Jan 2013 15:55:55 +0000. up 3.83 seconds
mounted event handled for /run
virtual finished
remote finished
local 1/2 remote 0/0 virtual 10/10 swap 0/0
mounting event sent for /home
mounting event sent for /scratch
mounting event sent for /opt
mounting event handled for /home
mounting event handled for /scratch
mounting event handled for /opt
no instance data found in start-local
mounting event sent for /mnt
mounting event handled for /mnt
Received SIGUSR1 (network device up)
mounting event sent for /home
mounting event sent for /scratch
mounting event sent for /opt
mounting event handled for /home
mounting event handled for /scratch
mounting event handled for /opt
Received SIGUSR1 (network device up)
mounting event sent for /home
mounting event sent for /scratch
mounting event sent for /opt
mounting event handled for /home
mounting event handled for /scratch
mounting event handled for /opt
ci-info: lo : 1 127.0.0.1 255.0.0.0 .
ci-info: eth0 : 1 192.168.0.12 255.255.255.0 fa:16:3e:0a:0b:4c
ci-info: route-0: 0.0.0.0 192.168.0.11 0.0.0.0 eth0 UG
ci-info: route-1: 192.168.0.0 0.0.0.0 255.255.255.0 eth0 U
cloud-init start running: Tue, 22 Jan 2013 15:55:57 +0000. up 5.78 seconds
found data source: DataSourceEc2
mounted event handled for /
local finished
All filesystems mounted
local 2/2 remote 0/0 virtual 10/10 swap 0/0
mounting event sent for /home
mounting event sent for /scratch
mounting event sent for /opt
mounting event handled for /home
mounting event handled for /scratch
mounting event handled for /opt
Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd
Skipping profile in /etc/apparmor.d/disable: usr.bin.firefox
* Starting AppArmor profiles [ OK ]
Starting memcached: * Stopping System V initialisation compatibility [ OK ]
* Starting System V runlevel compatibility [ OK ]
* Starting automatic crash report generation [ OK ]
* Starting deferred execution scheduler [ OK ]
* Starting regular background program processing daemon [ OK ]
* Starting ACPI daemon [ OK ]
* Starting save kernel messages [ OK ]
* Starting CPU interrupts balancing daemon [ OK ]
* Stopping save kernel messages [ OK ]
* Starting crash report submission daemon [ OK ]
* Stopping Handle applying cloud-config [ OK ]
memcached.
* Starting Name Service Cache Daemon nscd [ OK ]
* Starting Postfix Mail Transport Agent postfix [ OK ]
* Starting HTTP accelerator varnishd [ OK ]
* Starting NTP server ntpd [ OK ]
landscape-client is not configured, please run landscape-config.
Warning: DocumentRoot [/home/www] does not exist
* Starting web server apache2 [ OK ]
* Stopping System V runlevel compatibility [ OK ]
* Starting execute cloud user/final scripts [ OK ]
/etc/fstab
========
LABEL=cloudimg-rootfs / ext4 defaults 0 0
10.0.0.100:/home /home glusterfs defaults,nobootwait,comment=cloudconfig 0 0
10.0.0.100:/scratch /scratch glusterfs defaults,nobootwait,comment=cloudconfig 0 0
10.0.0.100:/ubuntu_11_10_opt /opt glusterfs defaults,nobootwait,comment=cloudconfig 0 0
/dev/vdb /mnt auto defaults,nobootwait,comment=cloudconfig 0 2
gluster log of home volume
======================
[2013-01-22 13:17:58.788134] I [glusterfsd.c:1666:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.1
[2013-01-22 13:17:58.862333] E [common-utils.c:125:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not known)
[2013-01-22 13:17:58.862395] E [name.c:243:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host 10.0.0.100
[2013-01-22 13:17:58.862514] E [glusterfsd-mgmt.c:1787:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: Success
[2013-01-22 13:17:58.862546] I [glusterfsd-mgmt.c:1790:mgmt_rpc_notify] 0-glusterfsd-mgmt: -1 connect attempts left
[2013-01-22 13:17:58.863087] W [glusterfsd.c:831:cleanup_and_exit] (-->/usr/sbin/glusterfs(glusterfs_mgmt_init+0x1ff) [0x7f5d23e811af] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_start+0x12) [0x7f5d237cd922] (-->/usr/sbin/glusterfs(+0xd486) [0x7f5d23e81486]))) 0-: received signum (1), shutting down
[2013-01-22 13:17:58.863134] I [fuse-bridge.c:4648:fini] 0-fuse: Unmounting '/home'.
Please let me know if you need any more info. |
Ubuntu Server 12.04.1 LTS (GNU/Linux 3.2.0-29-virtual x86_64)
glusterfs 3.3.1 (from semiosis PPA)
mountall 2.36.3
We are running an OpenStack cloud (essex). We use Gluster as a network filesystem for storage of data.
When we updated mountall on our client VMs we experienced a failure of the automatic mounting of the gluster-shares at boot-time. The gluster-shares can be mounted manually when the VMs are booted and online.
When we downgraded the mountall package from new version 2.36.3 to old version 2.36, the gluster-shares mounted again during boot.
boot.log (added --verbose to /etc/init/mountall.conf)
=======
Connected to Plymouth
/ is local
/proc is virtual
/sys is virtual
/sys/fs/fuse/connections is virtual
/sys/kernel/debug is virtual
/sys/kernel/security is virtual
/dev is virtual
/dev/pts is virtual
/tmp is local
/run is virtual
/run/lock is virtual
/run/shm is virtual
/home is nowait
/scratch is nowait
/opt is nowait
/mnt is nowait
mounting event sent for /sys/fs/fuse/connections
mounting event sent for /sys/kernel/debug
mounting event sent for /sys/kernel/security
mounting event sent for /run/lock
mounting event sent for /run/shm
mounting event sent for /
mounted event handled for /sys
swap finished
local 0/2 remote 0/0 virtual 1/10 swap 0/0
mounted event handled for /dev/pts
local 0/2 remote 0/0 virtual 2/10 swap 0/0
mounting event handled for /sys/fs/fuse/connections
mounting /sys/fs/fuse/connections
mount /sys/fs/fuse/connections [282] exited normally
mounting event handled for /sys/kernel/debug
mounting /sys/kernel/debug
mount /sys/kernel/debug [296] exited normally
mounting event handled for /sys/kernel/security
mounting /sys/kernel/security
mount /sys/kernel/security [299] exited normally
mounting event handled for /run/lock
mounting /run/lock
mount /run/lock [301] exited normally
mounting event handled for /run/shm
mounting /run/shm
mount /run/shm [304] exited normally
mounted event handled for /proc
local 0/2 remote 0/0 virtual 3/10 swap 0/0
mounting event handled for /
remounting /
mount / [310] exited normally
mounted event handled for /dev
mount / [321] exited normally
mount /proc [322] exited normally
mount /sys [323] exited normally
mount /sys/fs/fuse/connections [324] exited normally
mount /sys/kernel/debug [325] exited normally
mount /sys/kernel/security [326] exited normally
mount /dev [327] exited normally
mount /dev/pts [328] exited normally
mount /run [329] exited normally
mount /run/lock [330] exited normally
mount /run/shm [331] exited normally
local 0/2 remote 0/0 virtual 4/10 swap 0/0
mounted event handled for /sys/fs/fuse/connections
local 0/2 remote 0/0 virtual 5/10 swap 0/0
mounted event handled for /sys/kernel/security
local 0/2 remote 0/0 virtual 6/10 swap 0/0
mounting event sent for /tmp
mounting event sent for /home
mounting event sent for /scratch
mounting event sent for /opt
mounting event sent for /mnt
mounted event handled for /sys/kernel/debug
local 0/2 remote 0/0 virtual 7/10 swap 0/0
mounted event handled for /run/lock
local 0/2 remote 0/0 virtual 8/10 swap 0/0
mounted event handled for /run/shm
local 0/2 remote 0/0 virtual 9/10 swap 0/0
mounting event handled for /tmp
mounting event handled for /home
mounting /home
Mount failed. Please check the log file for more details.
mountall: mount /home [333] terminated with status 1
mountall: Filesystem could not be mounted: /home
mounting event handled for /scratch
mounting /scratch
Mount failed. Please check the log file for more details.
mountall: mount /scratch [389] terminated with status 1
mountall: Filesystem could not be mounted: /scratch
mounting event handled for /opt
mounting /opt
Mount failed. Please check the log file for more details.
mountall: mount /opt [424] terminated with status 1
mountall: Filesystem could not be mounted: /opt
mounting event handled for /mnt
mounting /mnt
mount /mnt [459] exited normally
mounting event sent for /home
mounting event sent for /scratch
mounting event sent for /opt
mounted event handled for /tmp
local 1/2 remote 0/0 virtual 9/10 swap 0/0
mounting event handled for /home
mounting event handled for /scratch
mounting event handled for /opt
mounted event handled for /mnt
local 1/2 remote 0/0 virtual 9/10 swap 0/0
mounting event sent for /home
mounting event sent for /scratch
mounting event sent for /opt
mounting event handled for /home
mounting event handled for /scratch
mounting event handled for /opt
cloud-init start-local running: Tue, 22 Jan 2013 15:55:55 +0000. up 3.83 seconds
mounted event handled for /run
virtual finished
remote finished
local 1/2 remote 0/0 virtual 10/10 swap 0/0
mounting event sent for /home
mounting event sent for /scratch
mounting event sent for /opt
mounting event handled for /home
mounting event handled for /scratch
mounting event handled for /opt
no instance data found in start-local
mounting event sent for /mnt
mounting event handled for /mnt
Received SIGUSR1 (network device up)
mounting event sent for /home
mounting event sent for /scratch
mounting event sent for /opt
mounting event handled for /home
mounting event handled for /scratch
mounting event handled for /opt
Received SIGUSR1 (network device up)
mounting event sent for /home
mounting event sent for /scratch
mounting event sent for /opt
mounting event handled for /home
mounting event handled for /scratch
mounting event handled for /opt
ci-info: lo : 1 127.0.0.1 255.0.0.0 .
ci-info: eth0 : 1 192.168.0.12 255.255.255.0 fa:16:3e:0a:0b:4c
ci-info: route-0: 0.0.0.0 192.168.0.11 0.0.0.0 eth0 UG
ci-info: route-1: 192.168.0.0 0.0.0.0 255.255.255.0 eth0 U
cloud-init start running: Tue, 22 Jan 2013 15:55:57 +0000. up 5.78 seconds
found data source: DataSourceEc2
mounted event handled for /
local finished
All filesystems mounted
local 2/2 remote 0/0 virtual 10/10 swap 0/0
mounting event sent for /home
mounting event sent for /scratch
mounting event sent for /opt
mounting event handled for /home
mounting event handled for /scratch
mounting event handled for /opt
Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd
Skipping profile in /etc/apparmor.d/disable: usr.bin.firefox
* Starting AppArmor profiles [ OK ]
Starting memcached: * Stopping System V initialisation compatibility [ OK ]
* Starting System V runlevel compatibility [ OK ]
* Starting automatic crash report generation [ OK ]
* Starting deferred execution scheduler [ OK ]
* Starting regular background program processing daemon [ OK ]
* Starting ACPI daemon [ OK ]
* Starting save kernel messages [ OK ]
* Starting CPU interrupts balancing daemon [ OK ]
* Stopping save kernel messages [ OK ]
* Starting crash report submission daemon [ OK ]
* Stopping Handle applying cloud-config [ OK ]
memcached.
* Starting Name Service Cache Daemon nscd [ OK ]
* Starting Postfix Mail Transport Agent postfix [ OK ]
* Starting HTTP accelerator varnishd [ OK ]
* Starting NTP server ntpd [ OK ]
landscape-client is not configured, please run landscape-config.
Warning: DocumentRoot [/home/www] does not exist
* Starting web server apache2 [ OK ]
* Stopping System V runlevel compatibility [ OK ]
* Starting execute cloud user/final scripts [ OK ]
/etc/fstab
========
LABEL=cloudimg-rootfs / ext4 defaults 0 0
10.0.0.100:/home /home glusterfs defaults,nobootwait,comment=cloudconfig 0 0
10.0.0.100:/scratch /scratch glusterfs defaults,nobootwait,comment=cloudconfig 0 0
10.0.0.100:/ubuntu_11_10_opt /opt glusterfs defaults,nobootwait,comment=cloudconfig 0 0
/dev/vdb /mnt auto defaults,nobootwait,comment=cloudconfig 0 2
gluster log of home volume
======================
[2013-01-22 13:17:58.788134] I [glusterfsd.c:1666:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.1
[2013-01-22 13:17:58.862333] E [common-utils.c:125:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not known)
[2013-01-22 13:17:58.862395] E [name.c:243:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host 10.0.0.100
[2013-01-22 13:17:58.862514] E [glusterfsd-mgmt.c:1787:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: Success
[2013-01-22 13:17:58.862546] I [glusterfsd-mgmt.c:1790:mgmt_rpc_notify] 0-glusterfsd-mgmt: -1 connect attempts left
[2013-01-22 13:17:58.863087] W [glusterfsd.c:831:cleanup_and_exit] (-->/usr/sbin/glusterfs(glusterfs_mgmt_init+0x1ff) [0x7f5d23e811af] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_start+0x12) [0x7f5d237cd922] (-->/usr/sbin/glusterfs(+0xd486) [0x7f5d23e81486]))) 0-: received signum (1), shutting down
[2013-01-22 13:17:58.863134] I [fuse-bridge.c:4648:fini] 0-fuse: Unmounting '/home'.
Please let me know if you need any more info. |
|
