Bug #1541946 “Deployment of ceph node fails: ceph-disk activate-...” : Bugs : Fuel for OpenStack

Revision history for this message

Artem Panchenko (apanchenko-8) wrote on 2016-02-04:

#1

diagnostic_logs.tgz Edit (36.6 MiB, application/x-tar)

Revision history for this message

Vladimir Kuklin (vkuklin) wrote on 2016-02-04:

#2

This is a really strange error. It looks like disks just left the node during the test. Look at the blkid output, it looks like there is no disks that could have been used as ceph partitions. For me it seems like buggy hardware or some other libvirt issues on the node that was used. I would also consider the issue to be due to some purge happenning on the host node, e.g. that the disks were purged or ejected from the VM.

Revision history for this message

Vladimir Kuklin (vkuklin) wrote on 2016-02-04:

#3

Download full text (29.3 KiB)

Here is the HW discovery call form node-5 which shows that there were 2 disks: vdb and vdc, in the beginning of it all, but we cannot find them in blkid output which is rather strange.
{
"mac": "64:48:F8:DA:75:9C",
"ip": "10.109.45.6",
"os_platform": "ubuntu",
"manufacturer": "QEMU",
"platform_name": "Standard PC (i440FX + PIIX, 1996)",
"meta": {
  "system": {
   "manufacturer": "QEMU",
   "uuid": "78E8657E-75C6-470A-96B8-106B98E3700F",
   "product": "Standard PC (i440FX + PIIX, 1996)",
   "version": "pc-i440fx-2.4",
   "fqdn": "node-5.test.domain.local"
  },
  "interfaces": [{
    "name": "enp0s3",
    "state": "up",
    "mac": "64:48:f8:da:75:9c",
    "pxe": false,
    "driver": "e1000",
    "bus_info": "0000:00:03.0",
    "max_speed": 1000,
    "current_speed": 1000,
    "ip": "10.109.45.6",
    "netmask": "255.255.255.0",
    "offloading_modes": [{
     "name": "rx-all",
     "state": null,
     "sub": []
    }, {
     "name": "rx-fcs",
     "state": null,
     "sub": []
    }, {
     "name": "tx-nocache-copy",
     "state": null,
     "sub": []
    }, {
     "name": "rx-vlan-offload",
     "state": null,
     "sub": []
    }, {
     "name": "generic-receive-offload",
     "state": null,
     "sub": []
    }, {
     "name": "generic-segmentation-offload",
     "state": null,
     "sub": []
    }, {
     "name": "tcp-segmentation-offload",
     "state": null,
     "sub": [{
      "name": "tx-tcp-segmentation",
      "state": null,
      "sub": []
     }]
    }, {
     "name": "scatter-gather",
     "state": null,
     "sub": [{
      "name": "tx-scatter-gather",
      "state": null,
      "sub": []
     }]
    }, {
     "name": "tx-checksumming",
     "state": null,
     "sub": [{
      "name": "tx-checksum-ip-generic",
      "state": null,
      "sub": []
     }]
    }, {
     "name": "rx-checksumming",
     "state": null,
     "sub": []
    }]
   }, {
    "name": "enp0s4",
    "state": "down",
    "mac": "64:98:86:b8:3f:ed",
    "pxe": false,
    "driver": "e1000",
    "bus_info": "0000:00:04.0",
    "max_speed": 1000,
    "current_speed": 1000,
    "offloading_modes": [{
     "name": "rx-all",
     "state": null,
     "sub": []
    }, {
     "name": "rx-fcs",
     "state": null,
     "sub": []
    }, {
     "name": "tx-nocache-copy",
     "state": null,
     "sub": []
    }, {
     "name": "rx-vlan-offload",
     "state": null,
     "sub": []
    }, {
     "name": "generic-receive-offload",
     "state": null,
     "sub": []
    }, {
     "name": "generic-segmentation-offload",
     "state": null,
     "sub": []
    }, {
     "name": "tcp-segmentation-offload",
     "state": null,
     "sub": [{
      "name": "tx-tcp-segmentation",
      "state": null,
      "sub": []
     }]
    }, {
     "name": "scatter-gather",
     "state": null,
     "sub": [{
      "name": "tx-scatter-gather",
      "state": null,
      "sub": []
     }]
    }, {
     "name": "tx-checksumming",
     "state": null,
     "sub": [{
      "name": "tx-checksum-ip-generic",
      "state": null,
      "sub": []
     }]
    }, {
     "name": "rx-checksumming",
     "state": null,
     "sub": []
    }]
   }, {
    "name": "enp0s5",
    "state": "down",
    "mac": "6...

Here is the HW discovery call form node-5 which shows that there were 2 disks: vdb and vdc, in the beginning of it all, but we cannot find them in blkid output which is rather strange.
{
	"mac": "64:48:F8:DA:75:9C",
	"ip": "10.109.45.6",
	"os_platform": "ubuntu",
	"manufacturer": "QEMU",
	"platform_name": "Standard PC (i440FX + PIIX, 1996)",
	"meta": {
		"system": {
			"manufacturer": "QEMU",
			"uuid": "78E8657E-75C6-470A-96B8-106B98E3700F",
			"product": "Standard PC (i440FX + PIIX, 1996)",
			"version": "pc-i440fx-2.4",
			"fqdn": "node-5.test.domain.local"
		},
		"interfaces": [{
				"name": "enp0s3",
				"state": "up",
				"mac": "64:48:f8:da:75:9c",
				"pxe": false,
				"driver": "e1000",
				"bus_info": "0000:00:03.0",
				"max_speed": 1000,
				"current_speed": 1000,
				"ip": "10.109.45.6",
				"netmask": "255.255.255.0",
				"offloading_modes": [{
					"name": "rx-all",
					"state": null,
					"sub": []
				}, {
					"name": "rx-fcs",
					"state": null,
					"sub": []
				}, {
					"name": "tx-nocache-copy",
					"state": null,
					"sub": []
				}, {
					"name": "rx-vlan-offload",
					"state": null,
					"sub": []
				}, {
					"name": "generic-receive-offload",
					"state": null,
					"sub": []
				}, {
					"name": "generic-segmentation-offload",
					"state": null,
					"sub": []
				}, {
					"name": "tcp-segmentation-offload",
					"state": null,
					"sub": [{
						"name": "tx-tcp-segmentation",
						"state": null,
						"sub": []
					}]
				}, {
					"name": "scatter-gather",
					"state": null,
					"sub": [{
						"name": "tx-scatter-gather",
						"state": null,
						"sub": []
					}]
				}, {
					"name": "tx-checksumming",
					"state": null,
					"sub": [{
						"name": "tx-checksum-ip-generic",
						"state": null,
						"sub": []
					}]
				}, {
					"name": "rx-checksumming",
					"state": null,
					"sub": []
				}]
			}, {
				"name": "enp0s4",
				"state": "down",
				"mac": "64:98:86:b8:3f:ed",
				"pxe": false,
				"driver": "e1000",
				"bus_info": "0000:00:04.0",
				"max_speed": 1000,
				"current_speed": 1000,
				"offloading_modes": [{
					"name": "rx-all",
					"state": null,
					"sub": []
				}, {
					"name": "rx-fcs",
					"state": null,
					"sub": []
				}, {
					"name": "tx-nocache-copy",
					"state": null,
					"sub": []
				}, {
					"name": "rx-vlan-offload",
					"state": null,
					"sub": []
				}, {
					"name": "generic-receive-offload",
					"state": null,
					"sub": []
				}, {
					"name": "generic-segmentation-offload",
					"state": null,
					"sub": []
				}, {
					"name": "tcp-segmentation-offload",
					"state": null,
					"sub": [{
						"name": "tx-tcp-segmentation",
						"state": null,
						"sub": []
					}]
				}, {
					"name": "scatter-gather",
					"state": null,
					"sub": [{
						"name": "tx-scatter-gather",
						"state": null,
						"sub": []
					}]
				}, {
					"name": "tx-checksumming",
					"state": null,
					"sub": [{
						"name": "tx-checksum-ip-generic",
						"state": null,
						"sub": []
					}]
				}, {
					"name": "rx-checksumming",
					"state": null,
					"sub": []
				}]
			}, {
				"name": "enp0s5",
				"state": "down",
				"mac": "64:e6:6d:b8:1b:7c",
				"pxe": false,
				"driver": "e1000",
				"bus_info": "0000:00:05.0",
				"max_speed": 1000,
				"current_speed": 1000,
				"offloading_modes": [{
					"name": "rx-all",
					"state": null,
					"sub": []
				}, {
					"name": "rx-fcs",
					"state": null,
					"sub": []
				}, {
					"name": "tx-nocache-copy",
					"state": null,
					"sub": []
				}, {
					"name": "rx-vlan-offload",
					"state": null,
					"sub": []
				}, {
					"name": "generic-receive-offload",
					"state": null,
					"sub": []
				}, {
					"name": "generic-segmentation-offload",
					"state": null,
					"sub": []
				}, {
					"name": "tcp-segmentation-offload",
					"state": null,
					"sub": [{
						"name": "tx-tcp-segmentation",
						"state": null,
						"sub": []
					}]
				}, {
					"name": "scatter-gather",
					"state": null,
					"sub": [{
						"name": "tx-scatter-gather",
						"state": null,
						"sub": []
					}]
				}, {
					"name": "tx-checksumming",
					"state": null,
					"sub": [{
						"name": "tx-checksum-ip-generic",
						"state": null,
						"sub": []
					}]
				}, {
					"name": "rx-checksumming",
					"state": null,
					"sub": []
				}]
			}, {
				"name": "enp0s6",
				"state": "down",
				"mac": "64:b5:88:38:97:0c",
				"pxe": false,
				"driver": "e1000",
				"bus_info": "0000:00:06.0",
				"max_speed": 1000,
				"current_speed": 1000,
				"offloading_modes": [{
					"name": "rx-all",
					"state": null,
					"sub": []
				}, {
					"name": "rx-fcs",
					"state": null,
					"sub": []
				}, {
					"name": "tx-nocache-copy",
					"state": null,
					"sub": []
				}, {
					"name": "rx-vlan-offload",
					"state": null,
					"sub": []
				}, {
					"name": "generic-receive-offload",
					"state": null,
					"sub": []
				}, {
					"name": "generic-segmentation-offload",
					"state": null,
					"sub": []
				}, {
					"name": "tcp-segmentation-offload",
					"state": null,
					"sub": [{
						"name": "tx-tcp-segmentation",
						"state": null,
						"sub": []
					}]
				}, {
					"name": "scatter-gather",
					"state": null,
					"sub": [{
						"name": "tx-scatter-gather",
						"state": null,
						"sub": []
					}]
				}, {
					"name": "tx-checksumming",
					"state": null,
					"sub": [{
						"name": "tx-checksum-ip-generic",
						"state": null,
						"sub": []
					}]
				}, {
					"name": "rx-checksumming",
					"state": null,
					"sub": []
				}]
			}, {
				"name": "enp0s7",
				"state": "down",
				"mac": "64:e8:0a:cf:1b:c4",
				"pxe": false,
				"driver": "e1000",
				"bus_info": "0000:00:07.0",
				"max_speed": 1000,
				"current_speed": 1000,
				"offloading_modes": [{
						"name": "rx-all",
						"state": null,
						"sub": []
					}, {
						"name": "rx-fcs",
						"state": null,
						"sub": []
					}, {
						"name": "tx-nocache-
						copy ","
						state ":null,"
						sub ":[]},{"
						name ":"
						rx - vlan - offload ","
						state ":null,"
						sub ":[]},{"
						name ":"
						generic - receive - offload ","
						state ":null,"
						sub ":[]},{"
						name ":"
						generic - segmentation - offload ","
						state ":null,"
						sub ":[]},{"
						name ":"
						tcp - segmentation - offload ","
						state ":null,"
						sub ":[{"
						name ":"
						tx - tcp - segmentation ","
						state ":null,"
						sub ":[]}]},{"
						name ":"
						scatter - gather ","
						state ":null,"
						sub ":[{"
						name ":"
						tx - scatter - gather ","
						state ":null,"
						sub ":[]}]},{"
						name ":"
						tx - checksumming ","
						state ":null,"
						sub ":[{"
						name ":"
						tx - checksum - ip - generic ","
						state ":null,"
						sub ":[]}]},{"
						name ":"
						rx - checksumming ","
						state ":null,"
						sub ":[]}]}],"
						cpu ":{"
						total ":2,"
						real ":2,"
						spec ":[{"
						frequency ":3300,"
						model ":"
						Intel(R) Xeon(R) CPU E3 - 1230 v3 @ 3.30 GHz "},{"
						frequency ":3300,"
						model ":"
						Intel(R) Xeon(R) CPU E3 - 1230 v3 @ 3.30 GHz "}]},"
						disks ":[{"
						name ":"
						vda ","
						model ":null,"
						size ":53687091200,"
						disk ":"
						vda ","
						extra ":["
						disk / by - id / virtio - b932d70d3b1242409cee "],"
						removable ":"
						0 "},{"
						name ":"
						vdb ","
						model ":null,"
						size ":53687091200,"
						disk ":"
						vdb ","
						extra ":["
						disk / by - id / virtio - da73ca057acb4f43a9f4 "],"
						removable ":"
						0 "},{"
						name ":"
						vdc ","
						model ":null,"
						size ":53687091200,"
						disk ":"
						vdc ","
						extra ":["
						disk / by - id / virtio - 3682 cb59423e4598bb2c "],"
						removable ":"
						0 "}],"
						memory ":{"
						devices ":[{"
						size ":3174039552,"
						type ":"
						RAM "}],"
						total ":3174039552,"
						maximum_capacity ":3174039552,"
						slots ":1},"
						pci_devices ":{"
						id ":"
						node - 5. test.domain.local ","
						class ":"
						system ","
						claimed ":true,"
						handle ":"
						DMI: 0100 ","
						description ":"
						Computer ","
						product ":"
						Standard PC(i440FX + PIIX, 1996)()
						","
						vendor ":"
						QEMU ","
						version ":"
						pc - i440fx - 2.4 ","
						width ":64,"
						configuration ":{"
						boot ":"
						normal ","
						uuid ":"
						7E65 E878 - C675 - 0 A47 - 96 B8 - 106 B98E3700F "},"
						capabilities ":{"
						smbios - 2.8 ":"
						SMBIOS version 2.8 ","
						dmi - 2.8 ":"
						DMI version 2.8 ","
						vsyscall32 ":"
						32 - bit processes "},"
						children ":[{"
						id ":"
						core ","
						class ":"
						bus ","
						claimed ":true,"
						description ":"
						Motherboard ","
						physid ":"
						0 ","
						children ":[{"
						id ":"
						firmware ","
						class ":"
						memory ","
						claimed ":true,"
						description ":"
						BIOS ","
						vendor ":"
						SeaBIOS ","
						physid ":"
						0 ","
						version ":"
						1.7 .5 - 20151012 _155810 - obs - 1 ","
						date ":"
						04 / 01 / 2014 ","
						units ":"
						bytes ","
						size ":98304},{"
						id ":"
						cpu: 0 ","
						class ":"
						processor ","
						claimed ":true,"
						handle ":"
						DMI: 0400 ","
						description ":"
						CPU ","
						product ":"
						Intel(R) Xeon(R) CPU E3 - 1230 v3 @ 3.30 GHz ","
						vendor ":"
						Intel Corp.
						","
						physid ":"
						400 ","
						businfo ":"
						cpu @0 ","
						version ":"
						pc - i440fx - 2.4 ","
						slot ":"
						CPU 0 ","
						units ":"
						Hz ","
						size ":2000000000,"
						capacity ":2000000000,"
						width ":64,"
						configuration ":{"
						cores ":"
						1 ","
						enabledcores ":"
						1 ","
						threads ":"
						1 "},"
						capabilities ":{"
						fpu ":"
						mathematical co - processor ","
						fpu_exception ":"
						FPU exceptions reporting ","
						wp ":true,"
						vme ":"
						virtual mode extensions ","
						de ":"
						debugging extensions ","
						pse ":"
						page size extensions ","
						tsc ":"
						time stamp counter ","
						msr ":"
						model - specific registers ","
						pae ":"
						4 GB + memory addressing(Physical Address Extension)
						","
						mce ":"
						machine check exceptions ","
						cx8 ":"
						compare and exchange 8 - byte ","
						apic ":"
						on - chip advanced programmable interrupt controller(APIC)
						","
						sep ":"
						fast system calls ","
						mtrr ":"
						memory type range registers ","
						pge ":"
						page global enable ","
						mca ":"
						machine check architecture ","
						cmov ":"
						conditional move instruction ","
						pat ":"
						page attribute table ","
						pse36 ":"
						36 - bit page size extensions ","
						clflush ":true,"
						mmx ":"
						multimedia extensions(MMX)
						","
						fxsr ":"
						fast floating point save / restore ","
						sse ":"
						streaming SIMD extensions(SSE)
						","
						sse2 ":"
						streaming SIMD extensions(SSE2)
						","
						ss ":"
						self - snoop ","
						syscall ":"
						fast system calls ","
						nx ":"
						no - execute bit(NX)
						","
						pdpe1gb ":true,"
						rdtscp ":true,"
						x86 - 64 ":"
						64 bits extensions(x86 - 64)
						","
						constant_tsc ":true,"
						arch_perfmon ":true,"
						rep_good ":true,"
						nopl ":true,"
						eagerfpu ":true,"
						pni ":true,"
						pclmulqdq ":true,"
						vmx ":true,"
						ssse3 ":true,"
						fma ":true,"
						cx16 ":true,"
						pcid ":true,"
						sse4_1 ":true,"
						sse4_2 ":true,"
						x2apic ":true,"
						movbe ":true,"
						popcnt ":true,"
						tsc_deadline_timer ":true,"
						aes ":true,"
						xsave ":true,"
						avx ":true,"
						f16c ":true,"
						rdrand ":true,"
						hypervisor ":true,"
						lahf_lm ":true,"
						abm ":true,"
						arat ":true,"
						xsaveopt ":true,"
						tpr_shadow ":true,"
						vnmi ":true,"
						flexpriority ":true,"
						ept ":true,"
						fsgsbase ":true,"
						tsc_adjust ":true,"
						bmi1 ":true,"
						hle ":true,"
						avx2 ":true,"
						smep ":true,"
						bmi2 ":true,"
						erms ":true,"
						invpcid ":true,"
						rtm ":true}},{"
						id ":"
						cpu: 1 ","
						class ":"
						processor ","
						claimed ":true,"
						handle ":"
						DMI: 0401 ","
						description ":"
						CPU ","
						product ":"
						Intel(R) Xeon(R) CPU E3 - 1230 v3 @ 3.30 GHz ","
						vendor ":"
						Intel Corp.
						","
						physid ":"
						401 ",
						"businfo": "cpu@1",
						"version": "pc-i440fx-2.4",
						"slot": "CPU 1",
						"units": "Hz",
						"size": 2000000000,
						"capacity": 2000000000,
						"width": 64,
						"configuration": {
							"cores": "1",
							"enabledcores": "1",
							"threads": "1"
						},
						"capabilities": {
							"fpu": "mathematical co-processor",
							"fpu_exception": "FPU exceptions reporting",
							"wp": true,
							"vme": "virtual mode extensions",
							"de": "debugging extensions",
							"pse": "page size extensions",
							"tsc": "time stamp counter",
							"msr": "model-specific registers",
							"pae": "4GB+ memory addressing (Physical Address Extension)",
							"mce": "machine check exceptions",
							"cx8": "compare and exchange 8-byte",
							"apic": "on-chip advanced programmable interrupt controller (APIC)",
							"sep": "fast system calls",
							"mtrr": "memory type range registers",
							"pge": "page global enable",
							"mca": "machine check architecture",
							"cmov": "conditional move instruction",
							"pat": "page attribute table",
							"pse36": "36-bit page size extensions",
							"clflush": true,
							"mmx": "multimedia extensions (MMX)",
							"fxsr": "fast floating point save/restore",
							"sse": "streaming SIMD extensions (SSE)",
							"sse2": "streaming SIMD extensions (SSE2)",
							"ss": "self-snoop",
							"syscall": "fast system calls",
							"nx": "no-execute bit (NX)",
							"pdpe1gb": true,
							"rdtscp": true,
							"x86-64": "64bits extensions (x86-64)",
							"constant_tsc": true,
							"arch_perfmon": true,
							"rep_good": true,
							"nopl": true,
							"eagerfpu": true,
							"pni": true,
							"pclmulqdq": true,
							"vmx": true,
							"ssse3": true,
							"fma": true,
							"cx16": true,
							"pcid": true,
							"sse4_1": true,
							"sse4_2": true,
							"x2apic": true,
							"movbe": true,
							"popcnt": true,
							"tsc_deadline_timer": true,
							"aes": true,
							"xsave": true,
							"avx": true,
							"f16c": true,
							"rdrand": true,
							"hypervisor": true,
							"lahf_lm": true,
							"abm": true,
							"arat": true,
							"xsaveopt": true,
							"tpr_shadow": true,
							"vnmi": true,
							"flexpriority": true,
							"ept": true,
							"fsgsbase": true,
							"tsc_adjust": true,
							"bmi1": true,
							"hle": true,
							"avx2": true,
							"smep": true,
							"bmi2": true,
							"erms": true,
							"invpcid": true,
							"rtm": true
						}
					}, {
						"id": "memory",
						"class": "memory",
						"claimed": true,
						"handle": "DMI:1000",
						"description": "System Memory",
						"physid": "1000",
						"units": "bytes",
						"size": 3174039552,
						"capacity": 3174039552,
						"children": [{
							"id": "bank",
							"class": "memory",
							"claimed": true,
							"handle": "DMI:1100",
							"description": "DIMM RAM",
							"vendor": "QEMU",
							"physid": "0",
							"slot": "DIMM 0",
							"units": "bytes",
							"size": 3174039552
						}]
					}, {
						"id": "pci",
						"class": "bridge",
						"claimed": true,
						"handle": "PCIBUS:0000:00",
						"description": "Host bridge",
						"product": "440FX - 82441FX PMC [Natoma]",
						"vendor": "Intel Corporation",
						"physid": "100",
						"businfo": "pci@0000:00:00.0",
						"version": "02",
						"width": 32,
						"clock": 33000000,
						"children": [{
								"id": "isa",
								"class": "bridge",
								"claimed": true,
								"handle": "PCI:0000:00:01.0",
								"description": "ISA bridge",
								"product": "82371SB PIIX3 ISA [Natoma/Triton II]",
								"vendor": "Intel Corporation",
								"physid": "1",
								"businfo": "pci@0000:00:01.0",
								"version": "00",
								"width": 32,
								"clock": 33000000,
								"configuration": {
									"latency": "0"
								},
								"capabilities": {
									"isa": true
								}
							}, {
								"id": "ide",
								"class": "storage",
								"claimed": true,
								"handle": "PCI:0000:00:01.1",
								"description": "IDE interface",
								"product": "82371SB PIIX3 IDE [Natoma/Triton II]",
								"vendor": "Intel Corporation",
								"physid": "1.1",
								"businfo": "pci@0000:00:01.1",
								"version": "00",
								"width": 32,
								"clock": 33000000,
								"configuration": {
									"driver": "ata_piix",
									"latency": "0"
								},
								"capabilities": {
									"ide": true,
									"bus_master": "bus mastering"
								}
							}, {
								"id": "display",
								"class": "display",
								"handle": "PCI:0000:00:02.0",
								"description": "VGA compatible controller",
								"physid": "2",
								"businfo": "pci@0000:00:02.0",
								"version": "02",
								"width": 32,
								"clock": 33000000,
								"configuration": {
									"latency": "0"
								},
								"capabilities": {
									"vga_controller": true,
									"bus_master": "bus mastering"
								}
							}, {
								"id": "network:0",
								"class": "network",
								"claimed": true,
								"handle": "PCI:0000:00:03.0",
								"description": "Ethernet interface",
								"product": "82540EM Gigabit Ethernet Controller",
								"vendor": "Intel Corporation",
								"physid": "3",
								"businfo": "pci@0000:00:03.0",
								"logicalname": "enp0s3",
								"version": "03",
								"serial": "64:48:f8:da:75:9c",
								"units": "bit/s",
								"size": 1000000000,
								"capacity": 1000000000,
								"width": 32,
								"clock": 33000000,
								"configuration": {
									"autonegotiation": "on",
									"broadcast": "yes",
									"driver": "e1000",
									"driverversion": "7.3.21-k8-NAPI",
									"duplex": "full",
									"ip": "10.109.45.6",
									"latency": "0",
									"link": "yes",
									"multicast": "yes",
									"port": "twisted pair",
									"speed": "1Gbit/s"
								},
								"capabilities": {
									"bus_master": "bus mastering",
									"rom": "extension ROM",
									"ethernet": true,
									"physical": "
									Physical interface ","
									tp ":"
									twisted pair ","
									10 bt ":"
									10 Mbit / s ","
									10 bt - fd ":"
									10 Mbit / s(full duplex)
									","
									100 bt ":"
									100 Mbit / s ","
									100 bt - fd ":"
									100 Mbit / s(full duplex)
									","
									1000 bt - fd ":"
									1 Gbit / s(full duplex)
									","
									autonegotiation ":"
									Auto - negotiation "}},{"
									id ":"
									network: 1 ","
									class ":"
									network ","
									disabled ":true,"
									claimed ":true,"
									handle ":"
									PCI: 0000: 00: 04.0 ","
									description ":"
									Ethernet interface ","
									product ":"
									82540E M Gigabit Ethernet Controller ","
									vendor ":"
									Intel Corporation ","
									physid ":"
									4 ","
									businfo ":"
									pci @0000: 00: 04.0 ","
									logicalname ":"
									enp0s4 ","
									version ":"
									03 ","
									serial ":"
									64: 98: 86: b8: 3 f: ed ","
									units ":"
									bit / s ","
									size ":1000000000,"
									capacity ":1000000000,"
									width ":32,"
									clock ":33000000,"
									configuration ":{"
									autonegotiation ":"
									on ","
									broadcast ":"
									yes ","
									driver ":"
									e1000 ","
									driverversion ":"
									7.3 .21 - k8 - NAPI ","
									duplex ":"
									full ","
									latency ":"
									0 ","
									link ":"
									no ","
									multicast ":"
									yes ","
									port ":"
									twisted pair ","
									speed ":"
									1 Gbit / s "},"
									capabilities ":{"
									bus_master ":"
									bus mastering ","
									rom ":"
									extension ROM ","
									ethernet ":true,"
									physical ":"
									Physical interface ","
									tp ":"
									twisted pair ","
									10 bt ":"
									10 Mbit / s ","
									10 bt - fd ":"
									10 Mbit / s(full duplex)
									","
									100 bt ":"
									100 Mbit / s ","
									100 bt - fd ":"
									100 Mbit / s(full duplex)
									","
									1000 bt - fd ":"
									1 Gbit / s(full duplex)
									","
									autonegotiation ":"
									Auto - negotiation "}},{"
									id ":"
									network: 2 ","
									class ":"
									network ","
									disabled ":true,"
									claimed ":true,"
									handle ":"
									PCI: 0000: 00: 05.0 ","
									description ":"
									Ethernet interface ","
									product ":"
									82540E M Gigabit Ethernet Controller ","
									vendor ":"
									Intel Corporation ","
									physid ":"
									5 ","
									businfo ":"
									pci @0000: 00: 05.0 ","
									logicalname ":"
									enp0s5 ","
									version ":"
									03 ","
									serial ":"
									64: e6: 6 d: b8: 1 b: 7 c ","
									units ":"
									bit / s ","
									size ":1000000000,"
									capacity ":1000000000,"
									width ":32,"
									clock ":33000000,"
									configuration ":{"
									autonegotiation ":"
									on ","
									broadcast ":"
									yes ","
									driver ":"
									e1000 ","
									driverversion ":"
									7.3 .21 - k8 - NAPI ","
									duplex ":"
									full ","
									latency ":"
									0 ","
									link ":"
									no ","
									multicast ":"
									yes ","
									port ":"
									twisted pair ","
									speed ":"
									1 Gbit / s "},"
									capabilities ":{"
									bus_master ":"
									bus mastering ","
									rom ":"
									extension ROM ","
									ethernet ":true,"
									physical ":"
									Physical interface ","
									tp ":"
									twisted pair ","
									10 bt ":"
									10 Mbit / s ","
									10 bt - fd ":"
									10 Mbit / s(full duplex)
									","
									100 bt ":"
									100 Mbit / s ","
									100 bt - fd ":"
									100 Mbit / s(full duplex)
									","
									1000 bt - fd ":"
									1 Gbit / s(full duplex)
									","
									autonegotiation ":"
									Auto - negotiation "}},{"
									id ":"
									network: 3 ","
									class ":"
									network ","
									disabled ":true,"
									claimed ":true,"
									handle ":"
									PCI: 0000: 00: 06.0 ","
									description ":"
									Ethernet interface ","
									product ":"
									82540E M Gigabit Ethernet Controller ","
									vendor ":"
									Intel Corporation ","
									physid ":"
									6 ","
									businfo ":"
									pci @0000: 00: 06.0 ","
									logicalname ":"
									enp0s6 ","
									version ":"
									03 ","
									serial ":"
									64: b5: 88: 38: 97: 0 c ","
									units ":"
									bit / s ","
									size ":1000000000,"
									capacity ":1000000000,"
									width ":32,"
									clock ":33000000,"
									configuration ":{"
									autonegotiation ":"
									on ","
									broadcast ":"
									yes ","
									driver ":"
									e1000 ","
									driverversion ":"
									7.3 .21 - k8 - NAPI ","
									duplex ":"
									full ","
									latency ":"
									0 ","
									link ":"
									no ","
									multicast ":"
									yes ","
									port ":"
									twisted pair ","
									speed ":"
									1 Gbit / s "},"
									capabilities ":{"
									bus_master ":"
									bus mastering ","
									rom ":"
									extension ROM ","
									ethernet ":true,"
									physical ":"
									Physical interface ","
									tp ":"
									twisted pair ","
									10 bt ":"
									10 Mbit / s ","
									10 bt - fd ":"
									10 Mbit / s(full duplex)
									","
									100 bt ":"
									100 Mbit / s ","
									100 bt - fd ":"
									100 Mbit / s(full duplex)
									","
									1000 bt - fd ":"
									1 Gbit / s(full duplex)
									","
									autonegotiation ":"
									Auto - negotiation "}},{"
									id ":"
									network: 4 ","
									class ":"
									network ","
									disabled ":true,"
									claimed ":true,"
									handle ":"
									PCI: 0000: 00: 07.0 ","
									description ":"
									Ethernet interface ","
									product ":"
									82540E M Gigabit Ethernet Controller ","
									vendor ":"
									Intel Corporation ","
									physid ":"
									7 ","
									businfo ":"
									pci @0000: 00: 07.0 ","
									logicalname ":"
									enp0s7 ","
									version ":"
									03 ","
									serial ":"
									64: e8: 0 a: cf: 1 b: c4 ","
									units ":"
									bit / s ","
									size ":1000000000,"
									capacity ":1000000000,"
									width ":32,"
									clock ":33000000,"
									configuration ":{"
									autonegotiation ":"
									on ","
									broadcast ":"
									yes ","
									driver ":"
									e1000 ","
									driverversion ":"
									7.3 .21 - k8 - NAPI ","
									duplex ":"
									full ","
									latency ":"
									0 ","
									link ":"
									no ","
									multicast ":"
									yes ","
									port ":"
									twisted pair ","
									speed ":"
									1 Gbit / s "},"
									capabilities ":{"
									bus_master ":"
									bus mastering ","
									rom ":"
									extension ROM ","
									ethernet ":true,"
									physical ":"
									Physical interface ","
									tp ":"
									twisted pair ","
									10 bt ":"
									10 Mbit / s ","
									10 bt - fd ":"
									10 Mbit / s(full duplex)
									","
									100 bt ":"
									100 Mbit / s ","
									100 bt - fd ":"
									100 Mbit / s(full duplex)
									","
									1000 bt - fd ":"
									1 Gbit / s(full duplex)
									","
									autonegotiation ":"
									Auto - negotiation "}},{"
									id ":"
									usb ","
									class ":"
									bus ","
									claimed ":true,"
									handle ":"
									PCI: 0000: 00: 08.0 ","
									description ":"
									USB controller ","
									product ":"
									uPD720200 USB 3.0 Host Controller ","
									vendor ":
									"NEC Corporation",
									"physid": "8",
									"businfo": "pci@0000:00:08.0",
									"version": "03",
									"width": 64,
									"clock": 33000000,
									"configuration": {
										"driver": "xhci_hcd",
										"latency": "0"
									},
									"capabilities": {
										"msix": "MSI-X",
										"msi": "Message Signalled Interrupts",
										"xhci": true,
										"bus_master": "bus mastering",
										"cap_list": "PCI capabilities listing"
									}
								},
								{
									"id": "scsi:0",
									"class": "storage",
									"claimed": true,
									"handle": "PCI:0000:00:09.0",
									"description": "SCSI storage controller",
									"product": "Virtio block device",
									"vendor": "Red Hat, Inc",
									"physid": "9",
									"businfo": "pci@0000:00:09.0",
									"version": "00",
									"width": 32,
									"clock": 33000000,
									"configuration": {
										"driver": "virtio-pci",
										"latency": "0"
									},
									"capabilities": {
										"scsi": true,
										"msix": "MSI-X",
										"bus_master": "bus mastering",
										"cap_list": "PCI capabilities listing"
									}
								},
								{
									"id": "scsi:1",
									"class": "storage",
									"claimed": true,
									"handle": "PCI:0000:00:0a.0",
									"description": "SCSI storage controller",
									"product": "Virtio block device",
									"vendor": "Red Hat, Inc",
									"physid": "a",
									"businfo": "pci@0000:00:0a.0",
									"version": "00",
									"width": 32,
									"clock": 33000000,
									"configuration": {
										"driver": "virtio-pci",
										"latency": "0"
									},
									"capabilities": {
										"scsi": true,
										"msix": "MSI-X",
										"bus_master": "bus mastering",
										"cap_list": "PCI capabilities listing"
									}
								},
								{
									"id": "scsi:2",
									"class": "storage",
									"claimed": true,
									"handle": "PCI:0000:00:0b.0",
									"description": "SCSI storage controller",
									"product": "Virtio block device",
									"vendor": "Red Hat, Inc",
									"physid": "b",
									"businfo": "pci@0000:00:0b.0",
									"version": "00",
									"width": 32,
									"clock": 33000000,
									"configuration": {
										"driver": "virtio-pci",
										"latency": "0"
									},
									"capabilities": {
										"scsi": true,
										"msix": "MSI-X",
										"bus_master": "bus mastering",
										"cap_list": "PCI capabilities listing"
									}
								},
								{
									"id": "generic",
									"class": "generic",
									"claimed": true,
									"handle": "PCI:0000:00:0c.0",
									"description": "Unclassified device",
									"product": "Virtio memory balloon",
									"vendor": "Red Hat, Inc",
									"physid": "c",
									"businfo": "pci@0000:00:0c.0",
									"version": "00",
									"width": 32,
									"clock": 33000000,
									"configuration": {
										"driver": "virtio-pci",
										"latency": "0"
									},
									"capabilities": {
										"bus_master": "bus mastering"
									}
								}]
						}]
				}]
		}
	},
	"is_agent": true,
	"agent_checksum": "52581c700c46e748905bd89883f64a7188f848da"
}

Revision history for this message

Vladimir Kuklin (vkuklin) wrote on 2016-02-04:

#4

I checked for the disks on the env - it seems that partitions are in place, but it seems the deployment was lacking prepare step for partitions /dev/vdb3 and /dev/vdc3 - file systems were not created there. So we need to ensure that we did not miss imporant relationships in puppet manifests

Changed in fuel:
status:	New → Confirmed

OpenStack Infra (hudson-openstack) on 2016-02-04

Changed in fuel:
assignee:	Fuel Library Team (fuel-library) → Andrew Woodward (xarses)
status:	Confirmed → In Progress

Revision history for this message

Andrew Woodward (xarses) wrote on 2016-02-04:

#5

https://review.openstack.org/#/c/276409/

Revision history for this message

Andrew Woodward (xarses) wrote on 2016-02-04:

#6

`ceph-disk activate-all` is used in conjuction with `udevadm trigger`
to prevent allready prepapred/provisioned disks from being re-formatted.

The error expressed in bug: 1541946 is that activate-all has errors
because there is no filesystem on the devices. This is accurate on nodes
where the OSD was not previously sucessful. To combat this we should
allow for the command to return errors

Revision history for this message

Bogdan Dobrelya (bogdando) wrote on 2016-02-05:

#7

I believe the issue affects 8.0 as well

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-02-05: Fix merged to fuel-library (master)

#8

Reviewed: https://review.openstack.org/276409
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=d01f994d0e2ac8a0fdafe8834f835b8a49afdc2d
Submitter: Jenkins
Branch: master

commit d01f994d0e2ac8a0fdafe8834f835b8a49afdc2d
Author: Andrew Woodward <email address hidden>
Date: Thu Feb 4 10:47:11 2016 -0800

Allow ceph-disk activate-all to return errors

`ceph-disk activate-all` is used in conjuction with `udevadm trigger`
to prevent allready prepapred/provisioned disks from being re-formatted.

    The error expressed in bug: 1541946 is that activate-all has errors
    because there is no filesystem on the devices. This is accurate on nodes
    where the OSD was not previously sucessful. To combat this we should
    allow for the command to return errors

Change-Id: I5cf29fcba7b0443832489c90518a9197c065ac8c
Closes-bug: 1541946

Changed in fuel:
status:	In Progress → Fix Committed

Revision history for this message

Eugene Bogdanov (ebogdanov) wrote on 2016-02-05:

#9

Downgraded to high - no data corruption and there is a workaround. So, doesn't match Critical definition.

Changed in fuel:
importance:	Critical → High

Mike Scherbakov (mihgen) on 2016-02-06

tags:

added: area-mos
removed: area-library

Revision history for this message

Alexei Sheplyakov (asheplyakov) wrote on 2016-02-08:

#10

I don't think the patch [1] is correct. ceph-disk exits via raising the SystemExit exception [2].
The exit code 1 means that the error code hasn't been explicitly specified:
"If the associated value is a plain integer, it specifies the system exit status (passed to C’s exit() function);
if it is None, the exit status is zero; if it has another type (such as a string), the object’s value is printed
and the exit status is one." [3] Thus the only possible error code is 1, and ignoring it is not safe at all.

[1] https://review.openstack.org/276409
[2] https://github.com/ceph/ceph/blob/hammer/src/ceph-disk#L2974-L2990
[3] https://docs.python.org/2/library/exceptions.html#exceptions.SystemExit

Revision history for this message

Roman Podoliaka (rpodolyaka) wrote on 2016-02-09:

#11

Per discussion with QA team:

This has never been seen on 8.0 and given the fact it's a part of BVT (run 4 times a day), we would hit it for sure if 8.0 was affected. We believe this regression was introduced in 9.0. Thus marking as Invalid.

Revision history for this message

Andrey Sledzinskiy (asledzinskiy) wrote on 2016-02-29:

#12

Issue reproduced again on next cluster configuration:
neutron vlan, ceph for volumes, images, Rados, Ironic selected
1 controller+ceph, 2 controllers+ceph+ironic, 1 compute, 1 ironic

Deployment failed on node-1

Revision history for this message

Andrey Sledzinskiy (asledzinskiy) wrote on 2016-02-29:

#13

fail_error_ironic_deploy_ceph-fuel-snapshot-2016-02-29_00-09-16.tar.xz Edit (27.3 MiB, application/octet-stream)

Revision history for this message

Andrey Sledzinskiy (asledzinskiy) wrote on 2016-02-29:

#14

fuel version - http://paste.openstack.org/show/488532/

Changed in fuel:
status:	Fix Committed → Confirmed

Revision history for this message

Alexei Sheplyakov (asheplyakov) wrote on 2016-02-29:

#15

Andrey,

> Issue reproduced again

Nope, it's a completely different problem:

2016-02-29 00:08:55 +0000 /Stage[main]/Ceph::Osds/Ceph::Osds::Osd[/dev/vdc3]/Exec[ceph-deploy osd prepare node-1:/dev/vdc3]/returns (notice): [ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
2016-02-29 00:08:55 +0000 /Stage[main]/Ceph::Osds/Ceph::Osds::Osd[/dev/vdc3]/Exec[ceph-deploy osd prepare node-1:/dev/vdc3]/returns (notice): [ceph_deploy.cli][INFO ] Invoked (1.5.20): /usr/bin/ceph-deploy osd prepare node-1:/dev/vdc3
2016-02-29 00:08:55 +0000 /Stage[main]/Ceph::Osds/Ceph::Osds::Osd[/dev/vdc3]/Exec[ceph-deploy osd prepare node-1:/dev/vdc3]/returns (notice): [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks node-1:/dev/vdc3:
2016-02-29 00:08:55 +0000 /Stage[main]/Ceph::Osds/Ceph::Osds::Osd[/dev/vdc3]/Exec[ceph-deploy osd prepare node-1:/dev/vdc3]/returns (notice): [ceph_deploy][ERROR ] RuntimeError: bootstrap-osd keyring not found; run 'gatherkeys'
2016-02-29 00:08:55 +0000 Puppet (err): ceph-deploy osd prepare node-1:/dev/vdc3 returned 1 instead of one of [0]

ceph-deploy osd prepare node-1:/dev/vdc3 fails because bootstrap-osd keyring hasn't been created properly (there's no file node-1/root/ceph.bootstrap-osd.keyring). Also there's no partition named /dev/vdc3

[10.109.6.4] out: /dev/vda3 ext2 /boot ddf014a5-f8a4-4e6e-9b45-efbb457543cc
[10.109.6.4] out: /dev/vda4 LVM2_member (in use) TVkdrb-IPO8-nZuR-0esd-IA0f-DaoH-fFr7Y8
[10.109.6.4] out: /dev/vda5 LVM2_member (in use) oraQcF-IteM-xfFc-PylP-2X1Z-1HhU-HK5H8H
[10.109.6.4] out: /dev/vda6 LVM2_member (in use) djVcBD-VDUa-wlVg-1pUh-RRcQ-HYZO-XVmlx5
[10.109.6.4] out: /dev/vda7 LVM2_member (in use) Ng1If3-PgZG-5KNT-ZN0Z-mvCu-8tX3-ZTxoXu
[10.109.6.4] out: /dev/vda8 iso9660 cidata (not mounted)
[10.109.6.4] out: /dev/vdb3 LVM2_member (in use) t67Vcj-YptD-y0WP-fiQF-vSTs-l3GC-Q0E9hf

Please file a new bug (with a proper description) and attach the relevant data.

Andrey,

> Issue reproduced again

Nope, it's a completely different problem:

2016-02-29 00:08:55 +0000 /Stage[main]/Ceph::Osds/Ceph::Osds::Osd[/dev/vdc3]/Exec[ceph-deploy osd prepare node-1:/dev/vdc3]/returns (notice): [ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
2016-02-29 00:08:55 +0000 /Stage[main]/Ceph::Osds/Ceph::Osds::Osd[/dev/vdc3]/Exec[ceph-deploy osd prepare node-1:/dev/vdc3]/returns (notice): [ceph_deploy.cli][INFO  ] Invoked (1.5.20): /usr/bin/ceph-deploy osd prepare node-1:/dev/vdc3
2016-02-29 00:08:55 +0000 /Stage[main]/Ceph::Osds/Ceph::Osds::Osd[/dev/vdc3]/Exec[ceph-deploy osd prepare node-1:/dev/vdc3]/returns (notice): [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks node-1:/dev/vdc3:
2016-02-29 00:08:55 +0000 /Stage[main]/Ceph::Osds/Ceph::Osds::Osd[/dev/vdc3]/Exec[ceph-deploy osd prepare node-1:/dev/vdc3]/returns (notice): [ceph_deploy][ERROR ] RuntimeError: bootstrap-osd keyring not found; run 'gatherkeys'
2016-02-29 00:08:55 +0000 Puppet (err): ceph-deploy osd prepare node-1:/dev/vdc3 returned 1 instead of one of [0]

ceph-deploy osd prepare node-1:/dev/vdc3 fails because bootstrap-osd keyring hasn't been created properly (there's no file node-1/root/ceph.bootstrap-osd.keyring). Also there's no partition named /dev/vdc3

[10.109.6.4] out: /dev/vda3  ext2             /boot          ddf014a5-f8a4-4e6e-9b45-efbb457543cc
[10.109.6.4] out: /dev/vda4  LVM2_member      (in use)       TVkdrb-IPO8-nZuR-0esd-IA0f-DaoH-fFr7Y8
[10.109.6.4] out: /dev/vda5  LVM2_member      (in use)       oraQcF-IteM-xfFc-PylP-2X1Z-1HhU-HK5H8H
[10.109.6.4] out: /dev/vda6  LVM2_member      (in use)       djVcBD-VDUa-wlVg-1pUh-RRcQ-HYZO-XVmlx5
[10.109.6.4] out: /dev/vda7  LVM2_member      (in use)       Ng1If3-PgZG-5KNT-ZN0Z-mvCu-8tX3-ZTxoXu
[10.109.6.4] out: /dev/vda8  iso9660 cidata   (not mounted)  
[10.109.6.4] out: /dev/vdb3  LVM2_member      (in use)       t67Vcj-YptD-y0WP-fiQF-vSTs-l3GC-Q0E9hf

Please file a new bug (with a proper description) and attach the relevant data.

Changed in fuel:
status:	Confirmed → Fix Committed

Revision history for this message

Ksenia Svechnikova (kdemina) wrote on 2016-05-24:

#16

Verify on fuel-9.0-mos-393-2016-05-24

Changed in fuel:
status:	Fix Committed → Fix Released

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	Fix Released	High	Andrew Woodward	Fuel for OpenStack 9.0
	8.0.x	Invalid	High	MOS Ceph	Fuel for OpenStack 8.0

Fuel for OpenStack

Deployment of ceph node fails: ceph-disk activate-all returned 1 instead of one of [0]

Bug Description

Other bug subscribers

Bug attachments

Remote bug watches