DHSS=False generic driver CI job fails all the time

Bug #1644523 reported by Valeriy Ponomaryov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Shared File Systems Service (Manila)
Fix Released
High
Valeriy Ponomaryov

Bug Description

Our CI job that runs generic driver in DHSS=False mode fails all the time not being able to connect to service VM with following logs:

http://logs.openstack.org/58/400758/5/check/gate-manila-tempest-dsvm-generic-no-share-servers-ubuntu-xenial-nv/8187cc0/console.html#_2016-11-24_11_32_14_887984

Raw:

2016-11-24 11:31:53.874844 | 2016-11-24 11:31:53.874 | ++ /opt/stack/new/manila/contrib/ci/common.sh:manila_check_service_vm_availability:23 : (( wait_timeout-=10 ))
2016-11-24 11:31:53.876085 | 2016-11-24 11:31:53.875 | ++ /opt/stack/new/manila/contrib/ci/common.sh:manila_check_service_vm_availability:24 : sleep 10
2016-11-24 11:32:03.874777 | 2016-11-24 11:32:03.874 | ++ /opt/stack/new/manila/contrib/ci/common.sh:manila_check_service_vm_availability:18 : (( wait_timeout > 0 ))
2016-11-24 11:32:03.875988 | 2016-11-24 11:32:03.875 | ++ /opt/stack/new/manila/contrib/ci/common.sh:manila_check_service_vm_availability:19 : ping -w 1 10.0.0.10
2016-11-24 11:32:04.878342 | 2016-11-24 11:32:04.877 | PING 10.0.0.10 (10.0.0.10) 56(84) bytes of data.
2016-11-24 11:32:04.879794 | 2016-11-24 11:32:04.879 |
2016-11-24 11:32:04.881124 | 2016-11-24 11:32:04.880 | --- 10.0.0.10 ping statistics ---
2016-11-24 11:32:04.882415 | 2016-11-24 11:32:04.882 | 1 packets transmitted, 0 received, 100% packet loss, time 0ms
2016-11-24 11:32:04.883760 | 2016-11-24 11:32:04.883 |
2016-11-24 11:32:04.885126 | 2016-11-24 11:32:04.884 | ++ /opt/stack/new/manila/contrib/ci/common.sh:manila_check_service_vm_availability:23 : (( wait_timeout-=10 ))
2016-11-24 11:32:04.886370 | 2016-11-24 11:32:04.886 | ++ /opt/stack/new/manila/contrib/ci/common.sh:manila_check_service_vm_availability:24 : sleep 10
2016-11-24 11:32:14.884707 | 2016-11-24 11:32:14.884 | ++ /opt/stack/new/manila/contrib/ci/common.sh:manila_check_service_vm_availability:18 : (( wait_timeout > 0 ))
2016-11-24 11:32:14.886203 | 2016-11-24 11:32:14.886 | ++ /opt/stack/new/manila/contrib/ci/common.sh:manila_check_service_vm_availability:27 : [[ false == \t\r\u\e ]]
2016-11-24 11:32:14.887984 | 2016-11-24 11:32:14.887 | ++ /opt/stack/new/manila/contrib/ci/common.sh:manila_check_service_vm_availability:30 : echo 'FAILURE! Service VM 10.0.0.10 is not available.'
2016-11-24 11:32:14.889451 | 2016-11-24 11:32:14.889 | FAILURE! Service VM 10.0.0.10 is not available.
2016-11-24 11:32:14.890614 | 2016-11-24 11:32:14.890 | ++ /opt/stack/new/manila/contrib/ci/common.sh:manila_check_service_vm_availability:31 : exit 1

The root cause for it is following devstack's change:
https://github.com/openstack-dev/devstack/commit/df5e69114fa2af663c50e734b4a842c727e6a7e9
That removed route we were depending on.

Changed in manila:
importance: Undecided → High
milestone: none → ocata-2
assignee: nobody → Valeriy Ponomaryov (vponomaryov)
description: updated
Changed in manila:
status: New → In Progress
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (master)

Reviewed: https://review.openstack.org/403651
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=8c5ed29ba6e293a47112772e58bbed1ce4f0fe94
Submitter: Jenkins
Branch: master

commit 8c5ed29ba6e293a47112772e58bbed1ce4f0fe94
Author: Valeriy Ponomaryov <email address hidden>
Date: Thu Dec 1 21:49:15 2016 +0300

    [Devstack] Fix DHSS=False setup for Generic driver

    Recent change to devstack [1] broke our DHSS=False CI job running
    generic driver that was depending on ip route from host to
    private network. So, to avoid this error use floating ip address
    for connection to service Nova VMs from host machine.
    Also, fix generation of second export location that should be dedicated
    for service needs such as mounting share doing host-assisted
    migration.

    [1] If45e3fc15c050cfbac11b57c1eaf137dd7ed816f

    Change-Id: Ieea992293ae02898741c939da15f0dbb4609d8b0
    Closes-Bug: #1644523
    Closes-Bug: #1646097

Changed in manila:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/manila 4.0.0.0b2

This issue was fixed in the openstack/manila 4.0.0.0b2 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.