tripleo

[nova placement] no version was discoverable

Bug #1838633 reported by Rafael Folco
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Unassigned
tripleo train-3

Bug Description

NotSupported: The placement service for 192.168.24.3:regionOne exists but no version was discoverable.

http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/46a8ae7/logs/undercloud/var/log/extra/errors.txt.txt.gz

2019-07-31 20:15:06.381 ERROR /var/log/containers/nova/nova-conductor.log: 9 ERROR nova Traceback (most recent call last):
2019-07-31 20:15:06.381 ERROR /var/log/containers/nova/nova-conductor.log: 9 ERROR nova File "/usr/bin/nova-conductor", line 10, in <module>
2019-07-31 20:15:06.381 ERROR /var/log/containers/nova/nova-conductor.log: 9 ERROR nova sys.exit(main())
2019-07-31 20:15:06.381 ERROR /var/log/containers/nova/nova-conductor.log: 9 ERROR nova File "/usr/lib/python2.7/site-packages/nova/cmd/conductor.py", line 44, in main
2019-07-31 20:15:06.381 ERROR /var/log/containers/nova/nova-conductor.log: 9 ERROR nova topic=rpcapi.RPC_TOPIC)
2019-07-31 20:15:06.381 ERROR /var/log/containers/nova/nova-conductor.log: 9 ERROR nova File "/usr/lib/python2.7/site-packages/nova/service.py", line 257, in create
2019-07-31 20:15:06.381 ERROR /var/log/containers/nova/nova-conductor.log: 9 ERROR nova periodic_interval_max=periodic_interval_max)
2019-07-31 20:15:06.381 ERROR /var/log/containers/nova/nova-conductor.log: 9 ERROR nova File "/usr/lib/python2.7/site-packages/nova/service.py", line 129, in __init__
2019-07-31 20:15:06.381 ERROR /var/log/containers/nova/nova-conductor.log: 9 ERROR nova self.manager = manager_class(host=self.host, *args, **kwargs)
2019-07-31 20:15:06.381 ERROR /var/log/containers/nova/nova-conductor.log: 9 ERROR nova File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 117, in __init__
2019-07-31 20:15:06.381 ERROR /var/log/containers/nova/nova-conductor.log: 9 ERROR nova self.compute_task_mgr = ComputeTaskManager()
2019-07-31 20:15:06.381 ERROR /var/log/containers/nova/nova-conductor.log: 9 ERROR nova File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 243, in __init__
2019-07-31 20:15:06.381 ERROR /var/log/containers/nova/nova-conductor.log: 9 ERROR nova self.report_client = report.SchedulerReportClient()
2019-07-31 20:15:06.381 ERROR /var/log/containers/nova/nova-conductor.log: 9 ERROR nova File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 200, in __init__
2019-07-31 20:15:06.381 ERROR /var/log/containers/nova/nova-conductor.log: 9 ERROR nova self._client = self._create_client()
2019-07-31 20:15:06.381 ERROR /var/log/containers/nova/nova-conductor.log: 9 ERROR nova File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 243, in _create_client
2019-07-31 20:15:06.381 ERROR /var/log/containers/nova/nova-conductor.log: 9 ERROR nova client = self._adapter or utils.get_sdk_adapter('placement')
2019-07-31 20:15:06.381 ERROR /var/log/containers/nova/nova-conductor.log: 9 ERROR nova File "/usr/lib/python2.7/site-packages/nova/utils.py", line 1253, in get_sdk_adapter
2019-07-31 20:15:06.381 ERROR /var/log/containers/nova/nova-conductor.log: 9 ERROR nova return getattr(conn, service_type)
2019-07-31 20:15:06.381 ERROR /var/log/containers/nova/nova-conductor.log: 9 ERROR nova File "/usr/lib/python2.7/site-packages/openstack/service_description.py", line 86, in __get__
2019-07-31 20:15:06.381 ERROR /var/log/containers/nova/nova-conductor.log: 9 ERROR nova instance._proxies[self.service_type] = self._make_proxy(instance)
2019-07-31 20:15:06.381 ERROR /var/log/containers/nova/nova-conductor.log: 9 ERROR nova File "/usr/lib/python2.7/site-packages/openstack/service_description.py", line 212, in _make_proxy
2019-07-31 20:15:06.381 ERROR /var/log/containers/nova/nova-conductor.log: 9 ERROR nova region_name=region_name))
2019-07-31 20:15:06.381 ERROR /var/log/containers/nova/nova-conductor.log: 9 ERROR nova NotSupported: The placement service for 192.168.24.3:regionOne exists but no version was discoverable.

Error happens in undercloud install:
2019-07-31 20:25:35 | "Error running ['podman', 'run', '--name', 'nova_wait_for_compute_service', '--label', 'config_id=tripleo_step4', '--label', 'container_name=nova_wait_for_compute_service', '--label', 'managed_by=paunch', '--label', 'config_data={\"start_order\": 101, \"command\": \"/container-config-scripts/pyshim.sh /container-config-scripts/nova_wait_for_compute_service.py\", \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/var/lib/config-data/nova/etc/my.cnf.d/:/etc/my.cnf.d/:ro\", \"/var/lib/config-data/nova/etc/nova/:/etc/nova/:ro\", \"/var/log/containers/nova:/var/log/nova\", \"/var/lib/container-config-scripts/:/container-config-scripts/\"], \"image\": \"192.168.24.1:8787/tripleomaster/centos-binary-nova-compute-ironic:348374f5d244990b3db6825c7d388339a3e0294f_23df8773-updated-20190731195007\", \"detach\": false, \"net\": \"host\"}', '--conmon-pidfile=/var/run/nova_wait_for_compute_service.pid', '--log-driver', 'k8s-file', '--log-opt', 'path=/var/log/containers/stdouts/nova_wait_for_compute_service.log', '--net=host', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/var/lib/config-data/nova/etc/my.cnf.d/:/etc/my.cnf.d/:ro', '--volume=/var/lib/config-data/nova/etc/nova/:/etc/nova/:ro', '--volume=/var/log/containers/nova:/var/log/nova', '--volume=/var/lib/container-config-scripts/:/container-config-scripts/', '192.168.24.1:8787/tripleomaster/centos-binary-nova-compute-ironic:348374f5d244990b3db6825c7d388339a3e0294f_23df8773-updated-20190731195007', '/container-config-scripts/pyshim.sh', '/container-config-scripts/nova_wait_for_compute_service.py']. [1]",
2019-07-31 20:25:35 | "stdout: INFO:nova_wait_for_compute_service:Waiting for nova-compute service to register",
2019-07-31 20:25:35 | "INFO:nova_wait_for_compute_service:Waiting for nova-compute service to register",
2019-07-31 20:25:35 | "stderr: + command -v python3",
2019-07-31 20:25:35 | "+ command -v python2",
2019-07-31 20:25:35 | "+ python2 /container-config-scripts/nova_wait_for_compute_service.py"
2019-07-31 20:25:35 | ]
2019-07-31 20:25:35 | }
2019-07-31 20:25:35 | ]
2019-07-31 20:25:35 | ]
2019-07-31 20:25:35 | Not cleaning working directory /home/zuul/tripleo-heat-installer-templates
2019-07-31 20:25:35 | Not cleaning ansible directory /home/zuul/undercloud-ansible-ut7Mwb
2019-07-31 20:25:35 | Install artifact is located at /home/zuul/undercloud-install-20190731202534.tar.bzip2
2019-07-31 20:25:35 |
2019-07-31 20:25:35 | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2019-07-31 20:25:35 |
2019-07-31 20:25:35 | Deployment Failed!

http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/46a8ae7/logs/undercloud/home/zuul/undercloud_install.log.txt.gz

Rafael Folco (rafaelfolco)
tags: added: promotion-blocker
removed: promoti
Changed in tripleo:
importance: Undecided → Critical
Revision history for this message
yatin (yatinkarel) wrote :

This issue is happening after nova switched to openstacksdk with [1], prior to this it seems '/' was appended to url by keystone adapter. Further checking placement wsgi returns unauthorized for GET /placement, and returns version for GET /placement/, i think placement should handle both '/placement' and '/placement/', will propose a patch and get feedback from placement folks. Also need to check why it's happening after switching to openstacksdk(like it being handled with keystone adapter).

[1] https://review.opendev.org/#/c/656023/

Revision history for this message
Marios Andreou (marios-b) wrote :

was about to file a new bug but ykarel pointed at this :D so filing acomment instead this happened on this mornings periodics too some more examples at http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/95d47c4/logs/undercloud/home/zuul/install-undercloud.log.txt.gz
http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/e656039/logs/undercloud/home/zuul/install-undercloud.log.txt.gz

Revision history for this message
yatin (yatinkarel) wrote :

Proposed https://review.opendev.org/#/c/674543/1 in placement and tested with a test patch.

Revision history for this message
wes hayutin (weshayutin) wrote :

looks fixed https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master

Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.