Failing to creating a vol from an image 3par FC driver
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
os-brick |
In Progress
|
Undecided
|
Unassigned |
Bug Description
Created attachment 1522157 [details]
Cinder.conf plus cinder logs
Description of problem: A simple deployment 1 controller + 2 computes, creating an empty Cinder volume works. However creating a volume from an image fails with error described below.
Unsure if config issue or a possible driver bug?
Version-Release number of selected component (if applicable):
RHEL 7.6
puppet-
openstack-
python2-
python-
python2-
python-
openstack-
puppet-
python2-
openstack-
python-
3par - HPE_3PAR 8200
HPE 3PAR OS version - 3.3.1.410 (MU2)+P32,
Cisco FC MDS switch 9148 - version 5.0(1a)
How reproducible:
Hit same issue on two deployments (reused same HW).
Then again it might be my cloned config issue.
Steps to Reproduce:
1. Configure Openstack 14 with 3par FC storage and Cinder back end.
2. Creating an empty volumes works fine
#cinder create 1 --volume-type 3parfc --name 3parEmptyVol7
Volume is created/avaliable, cinder list ->
| 569d57ae-
Basic Cinder/3par access works fine
3. Creating a volume from an image (cirros) fails
#cinder create 1 --volume-type 3parfc --name 3parVolFromImage1 --image cirros
+------
| Property | Value |
+------
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2019-01-
| description | None |
| encrypted | False |
| id | 0fafa271-
..
| status | creating
But after a while we see it failed to create,
#cinder list return ->
| 0fafa271-
On c-vol log I noticed and os-brick error ->
2019-01-21 12:32:13.400 70 ERROR os_brick.
2019-01-21 12:32:13.401 70 ERROR oslo.service.
2019-01-21 12:32:13.401 70 ERROR oslo.service.
2019-01-21 12:32:13.401 70 ERROR oslo.service.
2019-01-21 12:32:13.401 70 ERROR oslo.service.
2019-01-21 12:32:13.401 70 ERROR oslo.service.
2019-01-21 12:32:13.401 70 ERROR oslo.service.
2019-01-21 12:32:13.401 70 ERROR oslo.service.
on contoller's BM host, installed
#yum install sysfsutils
#systool -c fc_host -v Same output below when I run systool inside c-vol docker.
[root@controller-0 cinder]# systool -c fc_host -v
Class = "fc_host"
Class Device = "host6"
Class Device path = "/sys/devices/
dev_loss_tmo = "16"
fabric_name = "0x2002000573a5
issue_lip = <store method only>
max_npiv_vports = "254"
node_name = "0x50014380186a
npiv_
port_id = "0x6b1000"
port_name = "0x50014380186a
port_state = "Online"
port_type = "NPort (fabric via point-to-point)"
speed = "8 Gbit"
supported_
supported_
symbolic_name = "HPAJ764A FW:v8.07.00 DVR:v10.
system_hostname = ""
tgtid_bind_type = "wwpn (World Wide Port Name)"
uevent =
vport_create = <store method only>
vport_delete = <store method only>
Device = "host6"
Device path = "/sys/devices/
fw_dump =
issue_logo = <store method only>
nvram = "ISP "
optrom_ctl = <store method only>
optrom =
reset = <store method only>
sfp = ""
uevent = "DEVTYPE=scsi_host"
vpd = "�$"
Class Device = "host7"
Class Device path = "/sys/devices/
dev_loss_tmo = "16"
fabric_name = "0x2002000573a5
issue_lip = <store method only>
max_npiv_vports = "254"
node_name = "0x50014380186a
npiv_
port_id = "0x6b0a00"
port_name = "0x50014380186a
port_state = "Online"
port_type = "NPort (fabric via point-to-point)"
speed = "8 Gbit"
supported_
supported_
symbolic_name = "HPAJ764A FW:v8.07.00 DVR:v10.
system_hostname = ""
tgtid_bind_type = "wwpn (World Wide Port Name)"
uevent =
vport_create = <store method only>
vport_delete = <store method only>
Device = "host7"
Device path = "/sys/devices/
fw_dump =
issue_logo = <store method only>
nvram = "ISP "
optrom_ctl = <store method only>
optrom =
reset = <store method only>
sfp = ""
uevent = "DEVTYPE=scsi_host"
vpd = "�$"
4. Attaching an empty volume to an instance works.
Attaching volume failed on my previous system unsure why.
But it's working now so a good sign/progress.
Nova instance booted/running ->
| d38e10e4-
Attach empty volume created on step .1 to instance:
#nova volume-attach d38e10e4-
+------
| Property | Value |
+------
| device | /dev/vdb |
| id | 569d57ae-
| serverId | d38e10e4-
| volumeId | 569d57ae-
+------
Volume is attached, Cinder list ->
569d57ae-
Actual results:
Failing to create a 3par FC vol from image.
Additional info:
All controller/compute nodes as well as 3par's 4 FC links reside in the same FC zone.
Prior to installing Openstack, I'd successfully attached an FC volume to one of the hosts. So I gather FC zoning is fine.
All hosts belong to same rhos-fc host set on 3par.
The FC switch is a Cisco NX-OS MDS version 5.0(1a)
Not sure if while all ports belong to same FC zone do or don't I need to configure Cinder's zone manager?
I noticed this -> Cinder fc zone manger requirement -> Cisco MDS NX-OS Release 6.2(9) or later, later then my current switch version.
Just in case here is the zone info
zone name hp_3par_
member pwwn 21:00:00:
member pwwn 21:01:00:
member pwwn 51:40:2e:
member pwwn 51:40:2e:
member pwwn 21:01:00:
member pwwn 50:01:43:
member pwwn 51:40:2e:
member pwwn 21:00:00:
member pwwn 21:00:00:
member pwwn 21:00:00:
member pwwn 20:01:00:
member pwwn 20:02:00:
member pwwn 21:01:00:
member pwwn 21:02:00:
The last 4 one *6b are 3Par's 4 FC ports.
All the other wwn are dual port FC HBAs attached to my controllers/
Changed in cinder: | |
assignee: | nobody → Sneha Rai (sneharai4) |
assignee: | Sneha Rai (sneharai4) → nobody |
Changed in cinder: | |
status: | New → In Progress |
Changed in cinder: | |
assignee: | nobody → Raghavendra Tilay (raghavendrat) |
Changed in cinder: | |
assignee: | Raghavendra Tilay (raghavendrat) → nobody |
affects: | cinder → os-brick |
We found the issue, it turns out OS brick and HP's AJ76A HBA don't play nicely together.
I had this same HP HBA in my controller and one of my compute nodes. olumeDeviceFoun d: Unable to find a Fibre Channel volume device.
Both of them I had hit the same error
NoFibreChannelV
Create volume from image failed, attach volume failed, back of a volume failed.
However on a second compute attach volume worked, it just so happens that compute used another type of HBA. Working on a hunch I had swapped my controller's HBA to Qlogic, create volume from image works.
The odd thing is the HP AJ764A HBA is based on same Qlogic QLE2526 chip on my "working" HBA.
So same chip set Qlogic branded HBA works while HP clone doesn't.