In case of HA with different openstack/config nodes, cinder services are not starting

[wenqing liang]

2017-08-14 19:27:19.691 3126 CRITICAL cinder [req-e8c33bec-dcf7-497c-8ce9-ee6890268dea - - - - -] AttributeError: ‘NoneType’ object has no attribute ‘shutdown’
2017-08-14 19:27:19.691 3126 ERROR cinder Traceback (most recent call last):
2017-08-14 19:27:19.691 3126 ERROR cinder File “/usr/bin/cinder-scheduler”, line 10, in <module>
2017-08-14 19:27:19.691 3126 ERROR cinder sys.exit(main())
2017-08-14 19:27:19.691 3126 ERROR cinder File “/usr/lib/python2.7/dist-packages/cinder/cmd/scheduler.py”, line 56, in main
2017-08-14 19:27:19.691 3126 ERROR cinder service.wait()
2017-08-14 19:27:19.691 3126 ERROR cinder File “/usr/lib/python2.7/dist-packages/cinder/service.py”, line 487, in wait
2017-08-14 19:27:19.691 3126 ERROR cinder _launcher.wait()
2017-08-14 19:27:19.691 3126 ERROR cinder File “/usr/lib/python2.7/dist-packages/oslo_service/service.py”, line 314, in wait
2017-08-14 19:27:19.691 3126 ERROR cinder super(ServiceLauncher, self).wait()
2017-08-14 19:27:19.691 3126 ERROR cinder File “/usr/lib/python2.7/dist-packages/oslo_service/service.py”, line 223, in wait
2017-08-14 19:27:19.691 3126 ERROR cinder self.services.wait()
2017-08-14 19:27:19.691 3126 ERROR cinder File “/usr/lib/python2.7/dist-packages/oslo_service/service.py”, line 659, in wait
2017-08-14 19:27:19.691 3126 ERROR cinder service.wait()
2017-08-14 19:27:19.691 3126 ERROR cinder File “/usr/lib/python2.7/dist-packages/cinder/service.py”, line 297, in wait
2017-08-14 19:27:19.691 3126 ERROR cinder self.rpcserver.wait()
2017-08-14 19:27:19.691 3126 ERROR cinder File “/usr/lib/python2.7/dist-packages/oslo_messaging/server.py”, line 267, in wrapper
2017-08-14 19:27:19.691 3126 ERROR cinder log_after, timeout_timer)
2017-08-14 19:27:19.691 3126 ERROR cinder File “/usr/lib/python2.7/dist-packages/oslo_messaging/server.py”, line 187, in run_once
2017-08-14 19:27:19.691 3126 ERROR cinder post_fn = fn()
2017-08-14 19:27:19.691 3126 ERROR cinder File “/usr/lib/python2.7/dist-packages/oslo_messaging/server.py”, line 266, in <lambda>
2017-08-14 19:27:19.691 3126 ERROR cinder states[state].run_once(lambda: fn(self, args, *kwargs),
2017-08-14 19:27:19.691 3126 ERROR cinder File “/usr/lib/python2.7/dist-packages/oslo_messaging/server.py”, line 445, in wait
2017-08-14 19:27:19.691 3126 ERROR cinder self._poll_executor.shutdown(wait=True)
2017-08-14 19:27:19.691 3126 ERROR cinder AttributeError: ‘NoneType’ object has no attribute ‘shutdown’
2017-08-14 19:27:19.691 3126 ERROR cinder
^C
root@5b9s1-vm1:~#

root@5b9s1-vm2 (10.87.67.94):~/fabric-utils/fabfile/testbeds# grep vip testbed.py
    'external_vip' : '10.87.67.137',
    'contrail_external_vip' : '10.87.67.138',
env.ha['internal_vip'] = '192.16.20.28'
env.ha['contrail_internal_vip'] = '192.16.20.27'
root@5b9s1-vm2:~/fabric-utils/fabfile/testbeds#

rabbit host was pointing to 192.16.20.28

By default all services point to rabbit running in the config node(contrail_internal_vip)

If we want to point the openstack services to rabbit running in the openstack node,
We need to make use one of this params

https://github.com/Juniper/contrail-fabric-utils/blob/R3.2.3.x/fabfile/testbeds/testbed_multibox_example.py#L370
https://github.com/Juniper/contrail-fabric-utils/blob/R3.2.3.x/fabfile/testbeds/testbed_multibox_example.py#L371

[Jeya ganesh babu J (jjeya)]

Not related to 4.0. Issue is seen only in Fab based provision in 3.2

[Jeya ganesh babu J (jjeya)]

Workaround: change rabbit_host in the /etc/cinder/cinder.conf to contrail_internal_vip (same as in /etc/nova/nova.conf) and restart cinder-scheduler, cinder-volume and cinder-api services.

[Jeya ganesh babu J (jjeya)]

reprioritizing storage bugs.