Liberty devstack failed to launch instance w/ NetApp eSeries.

[Hong]

1. Exact version of Nova/OpenStack you are running:

Liberty Devstack

commit f4485bae9c719ee6b0c243cf5a69a6461df0bf23
Merge: ace1e8f e5a6f82
Author: Jenkins <email address hidden>
Date: Thu Oct 1 07:14:41 2015 +0000

    Merge "Cleanup nova v2.1 API testing options"

2. Relevant log files: n-cpu.log file is in the attachment.

3. Reproduce steps:
- Setup is running with Liberty devstack version on Ubuntu 14.04.
- Connected to a NetApp eSeries (iSCSI) for storage. (Using multipath to manage the array)
- Launch an instance from Horizon, by selecting "launch instance", input an Instance Name, m1.small, Instance count: 1, select "Boot from image (creates a new volume)", select "cirros..." image, default size(20G for small), then click on "Launch"

- The launch instance failed with the following error:

Error: Failed to perform requested operation on instance "testvm", the instance has an error status: Please try again later [Error: Build of instance 1304643b-f8f2-4894-89d8-26c1b8d3e438 aborted: Block Device Mapping is Invalid.].

It seems the host failed to get the new disk from the eSeries storage.

Did some more tests with the following observation:

When I create a new (1st) volume with certain image (cirros), the host created a host on the array, started the iSCSI sessions, and mapped the volume. But then the iSCSI sessions disconnected and the host failed to discover the volume, “sudo multipath –ll” did not list any volume.

Then I tried to create a 2nd instance, the host restarted the iSCSI sessions, created and mapped a new (2nd) volume. This time the host discovered the first volume, but not the newly created (2nd) volume. Also, the iSCSI sessions stay this time, they didn’t get disconnected.

It seem like there might be a problem with when the newly added volume is being discover is not in proper order, the discover/rescan command is being use too early.

Also, tried the same with the Kilo Devstack version, and this version is working fine.

[John Garbutt (johngarbutt)]

it looks like the volume create failed in cinder?

VolumeNotCreated: Volume 6a35bbdd-e6d6-4e79-84c9-12a84b08f7e2 did not finish being created even after we waited 24 seconds or 8 attempts. And its status is error.

It looks like when Nova tried to create the volume in cinder, there was an error in cinder. Did you check what the errors were on the cinder side?

[Hong (hong-chung)]

Took a look at the c-vol.log, the volume is created and successfully mapped, but it errored out on copy image, but what would cause this? Full log of c-vol.log is in a new attachment.

2015-10-01 15:22:37.970 DEBUG cinder.volume.drivers.netapp.eseries.library [req-4d137a57-2305-414e-b04b-12da785774e1 None None] Mapped volume 6a35bbdd-e6d6-4e79-84c9-12a84b08f7e2 to the initiator iqn.1993-08.org.debian:01:d32516909649. from (pid=23972) initialize_connection_iscsi /opt/stack/cinder/cinder/volume/drivers/netapp/eseries/library.py:770

2015-10-01 15:22:38.008 DEBUG cinder.volume.drivers.netapp.eseries.library [req-4d137a57-2305-414e-b04b-12da785774e1 None None] Successfully fetched target details for volume 6a35bbdd-e6d6-4e79-84c9-12a84b08f7e2 and initiator iqn.1993-08.org.debian:01:d32516909649. from (pid=23972) initialize_connection_iscsi /opt/stack/cinder/cinder/volume/drivers/netapp/eseries/library.py:776

2015-10-01 15:22:56.177 ERROR cinder.volume.flows.manager.create_volume [req-4d137a57-2305-414e-b04b-12da785774e1 None None] Failed to copy image 5ab1b848-4b74-4a41-b653-75a9e21acd0d to volume: 6a35bbdd-e6d6-4e79-84c9-12a84b08f7e2

2015-10-01 15:22:56.220 DEBUG cinder.volume.flows.common [req-4d137a57-2305-414e-b04b-12da785774e1 None None] Updating volume: 6a35bbdd-e6d6-4e79-84c9-12a84b08f7e2 with {'status': 'error'} due to: ??? from (pid=23972) _update_object /opt/stack/cinder/cinder/volume/flows/common.py:87

2015-10-01 15:22:56.380 ERROR cinder.volume.flows.manager.create_volume [req-4d137a57-2305-414e-b04b-12da785774e1 None None] Volume 6a35bbdd-e6d6-4e79-84c9-12a84b08f7e2: create failed

[Alex Meade (alex-meade)]

"Unable to find target portal 192.168.131.101:3260"

Can you confirm that the iscsi port is configured correctly and pingable? It's possible that you have a disconnected E-Series port that Cinder is trying to use.

[Alex Meade (alex-meade)]

I can note that the change to and changes in os-brick during the liberty release is likely why it works in Kilo.

[Alex Meade (alex-meade)]

After working with Hong directly, we have determined this was an environment issue.