NetApp iSCSI fails after extending a volume multiple times

Bug #1281279 reported by Andrew Kerr on 2014-02-17
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Undecided
Andrew Kerr
Havana
Undecided
Dirk Mueller

Bug Description

This is true for HAVANA and ICEHOUSE:

When extending a single volume over iSCSI connection to a cmode FAS, the following error is thrown after a pseudo-random number of extends:

2014-02-17 15:26:20.223 INFO cinder.volume.manager [req-93752b7f-544c-473c-8095-76a00f5276ff 989ee68034604d298503b25635615ab7 0b1d292e252e4d47822d8000e7e36449] volume 5ec14ba8-2da0-4e19-8594-9bee9a6ab641: extending
2014-02-17 15:26:20.244 INFO cinder.volume.drivers.netapp.iscsi [req-93752b7f-544c-473c-8095-76a00f5276ff 989ee68034604d298503b25635615ab7 0b1d292e252e4d47822d8000e7e36449] Resizing lun volume-5ec14ba8-2da0-4e19-8594-9bee9a6ab641 using sub clone to new size.
2014-02-17 15:26:20.308 DEBUG cinder.volume.drivers.netapp.iscsi [req-93752b7f-544c-473c-8095-76a00f5276ff 989ee68034604d298503b25635615ab7 0b1d292e252e4d47822d8000e7e36449] Getting lun block count. from (pid=28590) _get_lun_block_count /opt/stack/cinder/cinder/volume/drivers/netapp/iscsi.py:748
2014-02-17 15:26:20.435 DEBUG cinder.volume.drivers.netapp.iscsi [req-93752b7f-544c-473c-8095-76a00f5276ff 989ee68034604d298503b25635615ab7 0b1d292e252e4d47822d8000e7e36449] Destroyed LUN new-volume-5ec14ba8-2da0-4e19-8594-9bee9a6ab641 from (pid=28590) _destroy_lun /opt/stack/cinder/cinder/volume/drivers/netapp/iscsi.py:203
2014-02-17 15:26:20.436 ERROR cinder.volume.manager [req-93752b7f-544c-473c-8095-76a00f5276ff 989ee68034604d298503b25635615ab7 0b1d292e252e4d47822d8000e7e36449] volume 5ec14ba8-2da0-4e19-8594-9bee9a6ab641: Error trying to extend volume
2014-02-17 15:26:20.436 TRACE cinder.volume.manager Traceback (most recent call last):
2014-02-17 15:26:20.436 TRACE cinder.volume.manager File "/opt/stack/cinder/cinder/volume/manager.py", line 940, in extend_volume
2014-02-17 15:26:20.436 TRACE cinder.volume.manager self.driver.extend_volume(volume, new_size)
2014-02-17 15:26:20.436 TRACE cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/netapp/iscsi.py", line 609, in extend_volume
2014-02-17 15:26:20.436 TRACE cinder.volume.manager self._do_sub_clone_resize(path, new_size_bytes)
2014-02-17 15:26:20.436 TRACE cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/netapp/iscsi.py", line 713, in _do_sub_clone_resize
2014-02-17 15:26:20.436 TRACE cinder.volume.manager self._destroy_lun(new_path)
2014-02-17 15:26:20.436 TRACE cinder.volume.manager File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2014-02-17 15:26:20.436 TRACE cinder.volume.manager self.gen.next()
2014-02-17 15:26:20.436 TRACE cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/netapp/iscsi.py", line 708, in _do_sub_clone_resize
2014-02-17 15:26:20.436 TRACE cinder.volume.manager self._clone_lun(name, new_lun, block_count=block_count)
2014-02-17 15:26:20.436 TRACE cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/netapp/iscsi.py", line 988, in _clone_lun
2014-02-17 15:26:20.436 TRACE cinder.volume.manager self.client.invoke_successfully(clone_create, True)
2014-02-17 15:26:20.436 TRACE cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/netapp/api.py", line 222, in invoke_successfully
2014-02-17 15:26:20.436 TRACE cinder.volume.manager raise NaApiError(code, msg)
2014-02-17 15:26:20.436 TRACE cinder.volume.manager NaApiError: NetApp api failed. Reason - 13115:Integer value 41943040 out of range [1..2^24] for block-count
2014-02-17 15:26:20.436 TRACE cinder.volume.manager

It is not clear how many times it will take to cause the error to occur, as original volume size and extend increment size appear to have some influence, but the following sequence appears to always reproduce: create 4, extend 8, extend 20, extend 30.

akerr@openstack4:~/devstack$ cinder create 4
+---------------------+--------------------------------------+
| Property | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| created_at | 2014-02-17T20:25:44.961265 |
| display_description | None |
| display_name | None |
| id | 5ec14ba8-2da0-4e19-8594-9bee9a6ab641 |
| metadata | {} |
| size | 4 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| volume_type | None |
+---------------------+--------------------------------------+
akerr@openstack4:~/devstack$ cinder extend 5ec14ba8-2da0-4e19-8594-9bee9a6ab641 8
akerr@openstack4:~/devstack$ cinder extend 5ec14ba8-2da0-4e19-8594-9bee9a6ab641 20
akerr@openstack4:~/devstack$ cinder extend 5ec14ba8-2da0-4e19-8594-9bee9a6ab641 30
akerr@openstack4:~/devstack$ cinder list
+--------------------------------------+-----------------+--------------+------+-------------+----------+-------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------------+--------------+------+-------------+----------+-------------+
| 5ec14ba8-2da0-4e19-8594-9bee9a6ab641 | error_extending | None | 20 | None | false | |
+--------------------------------------+-----------------+--------------+------+-------------+----------+-------------+

This was run using Ubuntu 12.04.4LTS with devstack Havana and Icehouse, FAS is running cDOT 8.2P3 (real hardware, not virtual). The flexvol used has 1TB capacity and is otherwise empty.

summary: - NetApp iSCSI fails after multiple extending a volume multiple times
+ NetApp iSCSI fails after extending a volume multiple times
Navneet (singn) wrote :

Andrew can you also run a round of test pointing to ONTAP 8.1.x. I suspect the behavior is changed between the two releases but want to confirm the same before opening the issue.

Changed in cinder:
status: New → Incomplete
assignee: nobody → Navneet (singn)
Andrew Kerr (andrew-kerr) wrote :

Not sure why infra isn't linking this, but review submitted: https://review.openstack.org/74524

Changed in cinder:
status: Incomplete → In Progress
assignee: Navneet (singn) → Andrew Kerr (andrew-kerr)
tags: added: grizzly-backport-potential havana-backport-potential
tags: removed: grizzly-backport-potential

Reviewed: https://review.openstack.org/74524
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=a1d78df1b7c95dbb5024023662932cd94313dd58
Submitter: Jenkins
Branch: master

commit a1d78df1b7c95dbb5024023662932cd94313dd58
Author: Andrew Kerr <email address hidden>
Date: Tue Feb 18 17:38:12 2014 -0500

    Ensures NetApp iSCSI driver correctly compares int values for size

    The NetApp iSCSI driver needs to verify that the new size requested
    during an extend operation does not exceed the max_resize of the
    lun. However, the driver was doing a >= compare against str values
    of the sizes rather than the int values. This caused the incorrect
    code path to be used in certain situations.

    Change-Id: I8bc66c71db6b469c7adf00ce8b5091513fccb740
    Closes-Bug: 1281279

Changed in cinder:
status: In Progress → Fix Committed
Navneet (singn) wrote :

I guess the problem is bigger than just type as int. We need to test sub lun clone against 8.1.1.

Reviewed: https://review.openstack.org/75354
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=78d7e7ace77ac185637dc93baeccbce0f298217c
Submitter: Jenkins
Branch: stable/havana

commit 78d7e7ace77ac185637dc93baeccbce0f298217c
Author: Andrew Kerr <email address hidden>
Date: Tue Feb 18 17:38:12 2014 -0500

    Ensures NetApp iSCSI driver correctly compares int values for size

    The NetApp iSCSI driver needs to verify that the new size requested
    during an extend operation does not exceed the max_resize of the
    lun. However, the driver was doing a >= compare against str values
    of the sizes rather than the int values. This caused the incorrect
    code path to be used in certain situations.

    Change-Id: I8bc66c71db6b469c7adf00ce8b5091513fccb740
    Closes-Bug: 1281279
    (cherry picked from commit a1d78df1b7c95dbb5024023662932cd94313dd58)

tags: added: in-stable-havana
Thierry Carrez (ttx) on 2014-03-05
Changed in cinder:
milestone: none → icehouse-3
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2014-04-17
Changed in cinder:
milestone: icehouse-3 → 2014.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers