tripleo

  1. Bug #1774653
  2. Comment #1

Comment 1 for bug 1774653

Revision history for this message
Alan Bishop (alan-bishop) wrote :

The failure seems to be due to a rabbitmq glitch at the time when Nova first
requests Cinder delete the attachment. Cinder appears to properly handle the
request at [1], but note the glitch at [2] before it response 200 OK at [3].

[1] https://thirdparty.logs.rdoproject.org/jenkins-oooq-master-rdo_trunk-bmu-haa16-lab-float_nic_with_vlans-170/overcloud-controller-2/var/log/containers/cinder/cinder-api.log.txt.gz#_2018-05-31_08_18_27_525
[2] https://thirdparty.logs.rdoproject.org/jenkins-oooq-master-rdo_trunk-bmu-haa16-lab-float_nic_with_vlans-170/overcloud-controller-2/var/log/containers/cinder/cinder-api.log.txt.gz#_2018-05-31_08_18_27_574
[3] https://thirdparty.logs.rdoproject.org/jenkins-oooq-master-rdo_trunk-bmu-haa16-lab-float_nic_with_vlans-170/overcloud-controller-2/var/log/containers/cinder/cinder-api.log.txt.gz#_2018-05-31_08_18_31_025

So, from Cinder's perspective the attachment has been deleted. This explains
why Cinder returns 404 for all subsequent requests to delete the attachment.
Note the [4],[5],[6] timestamps follow the successful deletion at [3].

[4] https://thirdparty.logs.rdoproject.org/jenkins-oooq-master-rdo_trunk-bmu-haa16-lab-float_nic_with_vlans-170/overcloud-controller-2/var/log/containers/cinder/cinder-api.log.txt.gz#_2018-05-31_08_18_34_053
[5] https://thirdparty.logs.rdoproject.org/jenkins-oooq-master-rdo_trunk-bmu-haa16-lab-float_nic_with_vlans-170/overcloud-controller-2/var/log/containers/cinder/cinder-api.log.txt.gz#_2018-05-31_08_18_36_893
[6] https://thirdparty.logs.rdoproject.org/jenkins-oooq-master-rdo_trunk-bmu-haa16-lab-float_nic_with_vlans-170/overcloud-controller-0/var/log/containers/cinder/cinder-api.log.txt.gz#_2018-05-31_08_18_37_816

Lastly, Nova also reports a rabbitmq issue with controller-2 (which handled
the initial request to delete the attachment). Nova's log is [7].

[7] https://thirdparty.logs.rdoproject.org/jenkins-oooq-master-rdo_trunk-bmu-haa16-lab-float_nic_with_vlans-170/overcloud-controller-1/var/log/containers/nova/nova-api.log.txt.gz#_2018-05-31_08_18_25_174