Comment 10 for bug 1751923

Revision history for this message
sean mooney (sean-k-mooney) wrote : Re: _heal_instance_info_cache periodic task bases on port list from nova db, not from neutron server

Hi i have a customer that filed a downstream bug for this on newton.

i have been able to reproduce this without any db surgery however i did have to use
curl to send a raw command to neutron api.

i could try and create a function test for this if that helped.

currently the only workaround i have found is to delete teh neutorn ports
and create new ones. when the instance is in this broken state you cannot use
add/remove port to fix it.

paste bin log of script execution
http://paste.openstack.org/show/735818/

repro scipt below.
------------------------------------------------------------------------
#!/bin/bash

set -x

IMAGE="cirros-0.3.5-x86_64-disk"
NETWORK="private"
FLAVOR="m1.nano"
TEMP_TOKEN=$(openstack token issue -c id -f value)
SERVER=$(openstack server create --image ${IMAGE} --flavor ${FLAVOR} --network ${NETWORK} -c id -f value --wait repro-bug)
openstack server show ${SERVER}
PORT_ID=$(openstack port list --device-id ${SERVER} -f value -c id)
openstack port show ${PORT_ID}

#wait for it to be fully up
sleep 10

NEUTRON_ENDPOINT=$(openstack endpoint list --service network -f value -c URL)
curl -X PUT -H "X-Auth-Token:${TEMP_TOKEN}" -d '{ "port":{"device_id":"","device_owner":"", "binding:host_id":"" }}' "${NEUTRON_ENDPOINT}v2.0/ports/${PORT_ID}" | python -mjson.tool
# after this curl command nova and neutron will diagree as to the state of the port.

openstack server reboot --hard --wait ${SERVER}
# after the vm is rebooted the vm will not have an interface attached
openstack server show ${SERVER}
openstack port show ${PORT_ID}

#try to fix the issue by attaching the port again
openstack server add port ${SERVER} ${PORT_ID}
# note this will result in fixing the port in neutron but it will be broken on the nova side
# as a result the vm will still not have an interface attach but nuetron will say it is.
openstack server show ${SERVER}
openstack port show ${PORT_ID}

# wait for nova to have time to try and attach the interface
sleep 30
openstack server reboot --hard --wait ${SERVER}

set +x