"Invalid volume: Volume is not local to this node" in neutron multinode job (pike)

[Matt Riedemann]

http://logs.openstack.org/12/491012/11/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/c6b4271/console.html#_2017-08-09_22_26_58_860887

2017-08-09 22:26:58.860887 | tempest.api.volume.admin.test_groups.GroupsTest.test_group_update[id-4a8a6fd2-8b3b-4641-8f54-6a6f99320006]
2017-08-09 22:26:58.860931 | ----------------------------------------------------------------------------------------------------------
2017-08-09 22:26:58.860943 |
2017-08-09 22:26:58.860961 | Captured traceback:
2017-08-09 22:26:58.860980 | ~~~~~~~~~~~~~~~~~~~
2017-08-09 22:26:58.861011 | Traceback (most recent call last):
2017-08-09 22:26:58.861080 | File "tempest/api/volume/admin/test_groups.py", line 286, in test_group_update
2017-08-09 22:26:58.861127 | self.groups_client.update_group(grp['id'], **grp_params)
2017-08-09 22:26:58.861166 | File "tempest/lib/services/volume/v3/groups_client.py", line 108, in update_group
2017-08-09 22:26:58.861202 | resp, body = self.put('groups/%s' % group_id, put_body)
2017-08-09 22:26:58.861254 | File "tempest/lib/common/rest_client.py", line 334, in put
2017-08-09 22:26:58.861306 | return self.request('PUT', url, extra_headers, headers, body, chunked)
2017-08-09 22:26:58.861366 | File "tempest/lib/services/volume/base_client.py", line 38, in request
2017-08-09 22:26:58.861422 | method, url, extra_headers, headers, body, chunked)
2017-08-09 22:26:58.861479 | File "tempest/lib/common/rest_client.py", line 659, in request
2017-08-09 22:26:58.861525 | self._error_checker(resp, resp_body)
2017-08-09 22:26:58.861583 | File "tempest/lib/common/rest_client.py", line 770, in _error_checker
2017-08-09 22:26:58.861634 | raise exceptions.BadRequest(resp_body, resp=resp)
2017-08-09 22:26:58.861681 | tempest.lib.exceptions.BadRequest: Bad request
2017-08-09 22:26:58.861745 | Details: {u'message': u'Invalid volume: Volume is not local to this node.', u'code': 400}

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Details%3A%20%7Bu'message'%3A%20u'Invalid%20volume%3A%20Volume%20is%20not%20local%20to%20this%20node.'%2C%20u'code'%3A%20400%7D%5C%22&from=7d

8 hits in 7 days, check queue only right now but multiple changes, and this failed on a non-cinder change.

[Matt Riedemann (mriedem)]

The failures are hitting since this change in Tempest:

https://github.com/openstack/tempest/commit/9ee986075935eaa10c3102af87e669618b634ee1

[Sean McGinnis (sean-mcginnis)]

Xing, any thoughts on this?

[Ghanshyam Mann (ghanshyammann)]

Its failing on neutron side also as neutron also run this job - gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv

http://logs.openstack.org/33/485333/20/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/15586f4/logs/testr_results.html.gz

Actually there is no other job than this which run API tests on multinode. And this job is run in check pipeline in nova and neutron only. Tempest has it running as experimental (which we missed to check in Tempest patch where this tests got merged).

I feel it is worth to move this job in check pipeline of Tempest to detect these kind of issues. I will push that.

As this job is nv, we can wait cinder to investigate this issues and do the Tempest change accordingly.

[Ghanshyam Mann (ghanshyammann)]

to move neutron multindoe job to Tempest check pipeline in - https://review.openstack.org/#/c/492797/1

[Xing Yang (xing-yang)]

This is not a bug in Cinder API. It should be fixed in the tempest update_group test. I'll work on a patch. There are two ways to fix this:

1. When the volume is created outside of the group, we should check whether the volume is created on the same backend as the group. If yes, add volume to the group; otherwise don't add it.

2. Create two volumes in the group, first test remove a volume from the group, then test add the volume back to the group. This involves two update_group API calls and makes the test longer. During the review, I was recommended to make one update_group API call to speed up the test.

I'll submit a patch using approach #1.

[Xing Yang (xing-yang)]

Tried approach #1, but found out I can't use #1 because 'host' is an internal field for volume and group, not accessible by using volume or group APIs. I'll have to go with #2.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/492969

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/492969
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=5805d32137d6b930244b06a7f8373c167c35cb26
Submitter: Jenkins
Branch: master

commit 5805d32137d6b930244b06a7f8373c167c35cb26
Author: xing-yang <email address hidden>
Date: Fri Aug 11 04:31:27 2017 -0700

    Volume and group on same backend in update_group

    In a multiple backend environment, a volume can be created
    on a different backend from a group, add volume to group
    will fail in this case. The 'host' field of a volume or a
    group is an internal field that cannot be retrieved by
    using Cinder volume APIs. The only way to make sure a volume
    is created on the same backend as the host is to create the
    volume with the group_id parameter. After that remove the
    volume from the group, and then add the volume back to the
    group. This way the volume to be added to the group will be
    on the same backend as the group.

    This will make the test longer than desired, but it is
    the only way to make sure the added volume is on the same
    backend as the group.

    Change-Id: I376f0188984d706e83740f67bd6f301f680914f8
    Closes-Bug: #1709739

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tempest 17.0.0 release.