================================================================================ Version : glusterfs 3.0.5 built on Oct 16 2010 09:48:24 git: v3.0.5 Starting Time: 2011-04-13 16:40:48 Command line : /usr/sbin/glusterfs --log-level=NORMAL --volfile=/etc/glusterfs/glusterfs.gluster.vol /gluster/gluster PID : 1199 System name : Linux Nodename : ttt Kernel Release : 2.6.38-8-generic Hardware Identifier: i686 Given volfile: +------------------------------------------------------------------------------+ 1: # Config for gluster 2: # Generated by Puppet 3: 4: volume vol-0 5: type protocol/client 6: option transport-type tcp 7: option remote-host test 8: option transport.socket.nodelay on 9: option remote-port 6996 10: option remote-subvolume iothreads 11: option username username 12: option password password 13: end-volume 14: volume vol-1 15: type protocol/client 16: option transport-type tcp 17: option remote-host ttt 18: option transport.socket.nodelay on 19: option remote-port 6996 20: option remote-subvolume iothreads 21: option username username 22: option password password 23: end-volume 24: 25: volume mirror-0 26: type cluster/replicate 27: subvolumes vol-0 vol-1 28: option read-subvolume vol-1 29: end-volume 30: 31: volume writebehind 32: type performance/write-behind 33: option cache-size 4MB 34: # option flush-behind on # olecam: increasing the performance of handling lots of small files 35: subvolumes mirror-0 36: end-volume 37: 38: volume iothreads 39: type performance/io-threads 40: option thread-count 16 # default is 16 41: subvolumes writebehind 42: end-volume 43: 44: volume iocache 45: type performance/io-cache 46: option cache-size `grep 'MemTotal' /proc/meminfo | awk '{print $2 * 0.2 / 1024}' | cut -f1 -d.`MB 47: option cache-timeout 30 48: subvolumes iothreads 49: end-volume 50: 51: volume statprefetch 52: type performance/stat-prefetch 53: subvolumes iocache 54: end-volume +------------------------------------------------------------------------------+ [2011-04-13 16:40:48] N [glusterfsd.c:1409:main] glusterfs: Successfully started [2011-04-13 16:40:49] E [socket.c:802:socket_connect_finish] vol-0: connection to failed (Connection refused) [2011-04-13 16:40:49] N [fuse-bridge.c:2953:fuse_init] glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.16 [2011-04-13 16:40:49] E [socket.c:802:socket_connect_finish] vol-0: connection to failed (Connection refused) [2011-04-13 16:40:49] N [client-protocol.c:6288:client_setvolume_cbk] vol-1: Connected to 127.0.1.1:6996, attached to remote volume 'iothreads'. [2011-04-13 16:40:49] N [afr.c:2648:notify] mirror-0: Subvolume 'vol-1' came back up; going online. [2011-04-13 16:40:49] N [client-protocol.c:6288:client_setvolume_cbk] vol-1: Connected to 127.0.1.1:6996, attached to remote volume 'iothreads'. [2011-04-13 16:40:49] N [afr.c:2648:notify] mirror-0: Subvolume 'vol-1' came back up; going online. [2011-04-13 16:51:14] E [socket.c:802:socket_connect_finish] vol-1: connection to 127.0.1.1:6996 failed (Connection refused) [2011-04-13 16:51:14] N [client-protocol.c:7036:notify] vol-1: disconnected [2011-04-13 16:51:14] E [afr.c:2678:notify] mirror-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2011-04-13 16:51:14] E [socket.c:802:socket_connect_finish] vol-1: connection to 127.0.1.1:6996 failed (Connection refused) [2011-04-13 16:51:25] N [client-protocol.c:6288:client_setvolume_cbk] vol-1: Connected to 127.0.1.1:6996, attached to remote volume 'iothreads'. [2011-04-13 16:51:25] N [afr.c:2648:notify] mirror-0: Subvolume 'vol-1' came back up; going online. [2011-04-13 16:51:25] N [client-protocol.c:6288:client_setvolume_cbk] vol-1: Connected to 127.0.1.1:6996, attached to remote volume 'iothreads'. [2011-04-13 16:51:25] N [afr.c:2648:notify] mirror-0: Subvolume 'vol-1' came back up; going online. pending frames: frame : type(1) op(STAT) patchset: v3.0.5 signal received: 6 time of crash: 2011-04-13 16:51:34 configuration details: argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.0.5 [0x14e400] /lib/i386-linux-gnu/libc.so.6(abort+0x17e)[0xa0334e] /lib/i386-linux-gnu/libc.so.6(+0x61577)[0xa36577] /lib/i386-linux-gnu/libc.so.6(__fortify_fail+0x50)[0xabadf0] /lib/i386-linux-gnu/libc.so.6(+0xe4cca)[0xab9cca] /lib/i386-linux-gnu/libc.so.6(__strcpy_chk+0x3f)[0xab905f] /usr/lib/glusterfs/3.0.5/xlator/protocol/client.so(client_lookup+0x215)[0x4b0d75] /usr/lib/glusterfs/3.0.5/xlator/cluster/replicate.so(afr_lookup+0x4e5)[0x52f875] /usr/lib/libglusterfs.so.0(default_lookup+0xb5)[0x313dc5] /usr/lib/glusterfs/3.0.5/xlator/performance/io-threads.so(iot_lookup_wrapper+0xb5)[0x297375] /usr/lib/libglusterfs.so.0(call_resume+0x402)[0x324c42] /usr/lib/glusterfs/3.0.5/xlator/performance/io-threads.so(iot_worker_unordered+0x20)[0x297ae0] /lib/i386-linux-gnu/libpthread.so.0(+0x5e99)[0x253e99] /lib/i386-linux-gnu/libc.so.6(clone+0x5e)[0xaa573e] --------- ================================================================================ Version : glusterfs 3.0.4 built on May 10 2010 09:49:35 git: v3.0.4 Starting Time: 2011-04-13 16:51:58 Command line : /usr/sbin/glusterfs --log-level=NORMAL --volfile=/etc/glusterfs/glusterfs.gluster.vol /gluster/gluster PID : 556 System name : Linux Nodename : ttt Kernel Release : 2.6.38-8-generic Hardware Identifier: i686 Given volfile: +------------------------------------------------------------------------------+ 1: # Config for gluster 2: # Generated by Puppet 3: 4: volume vol-0 5: type protocol/client 6: option transport-type tcp 7: option remote-host test 8: option transport.socket.nodelay on 9: option remote-port 6996 10: option remote-subvolume iothreads 11: option username username 12: option password password 13: end-volume 14: volume vol-1 15: type protocol/client 16: option transport-type tcp 17: option remote-host ttt 18: option transport.socket.nodelay on 19: option remote-port 6996 20: option remote-subvolume iothreads 21: option username username 22: option password password 23: end-volume 24: 25: volume mirror-0 26: type cluster/replicate 27: subvolumes vol-0 vol-1 28: option read-subvolume vol-1 29: end-volume 30: 31: volume writebehind 32: type performance/write-behind 33: option cache-size 4MB 34: # option flush-behind on # olecam: increasing the performance of handling lots of small files 35: subvolumes mirror-0 36: end-volume 37: 38: volume iothreads 39: type performance/io-threads 40: option thread-count 16 # default is 16 41: subvolumes writebehind 42: end-volume 43: 44: volume iocache 45: type performance/io-cache 46: option cache-size `grep 'MemTotal' /proc/meminfo | awk '{print $2 * 0.2 / 1024}' | cut -f1 -d.`MB 47: option cache-timeout 30 48: subvolumes iothreads 49: end-volume 50: 51: volume statprefetch 52: type performance/stat-prefetch 53: subvolumes iocache 54: end-volume +------------------------------------------------------------------------------+ [2011-04-13 16:51:58] N [glusterfsd.c:1408:main] glusterfs: Successfully started [2011-04-13 16:51:58] E [socket.c:762:socket_connect_finish] vol-0: connection to failed (Connection refused) [2011-04-13 16:51:58] N [fuse-bridge.c:2950:fuse_init] glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.16 [2011-04-13 16:51:58] E [socket.c:762:socket_connect_finish] vol-0: connection to failed (Connection refused) [2011-04-13 16:51:58] E [socket.c:762:socket_connect_finish] vol-1: connection to failed (Connection refused) [2011-04-13 16:51:58] E [socket.c:762:socket_connect_finish] vol-1: connection to failed (Connection refused) [2011-04-13 16:52:01] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 2: LOOKUP() / => -1 (Transport endpoint is not connected) [2011-04-13 16:52:09] N [client-protocol.c:6246:client_setvolume_cbk] vol-1: Connected to 127.0.1.1:6996, attached to remote volume 'iothreads'. [2011-04-13 16:52:09] N [afr.c:2632:notify] mirror-0: Subvolume 'vol-1' came back up; going online. [2011-04-13 16:52:09] N [client-protocol.c:6246:client_setvolume_cbk] vol-1: Connected to 127.0.1.1:6996, attached to remote volume 'iothreads'. [2011-04-13 16:52:09] N [afr.c:2632:notify] mirror-0: Subvolume 'vol-1' came back up; going online.