devstack

nova-compute in singleconductor mode (grenade jobs) trying (and failing) to reach scheduler

Bug #1813215 reported by Matt Riedemann on 2019-01-24

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	devstack	Invalid	Undecided	Unassigned

Bug Description

Seen here:

http://logs.openstack.org/04/632904/2/check/legacy-grenade-dsvm-neutron-multinode-live-migration/8f886af/logs/subnode-2/screen-n-cpu.txt.gz?level=TRACE#_Jan_24_20_22_40_090198

Jan 24 20:22:40.090198 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo.messaging._drivers.impl_rabbit [None req-a87b4bb9-936f-4f56-add7-d95e69ca5173 None None] Unable to connect to AMQP server on 158.69.71.175:5672 after None tries: Basic.publish: (404) NOT_FOUND - no exchange 'scheduler_fanout' in vhost '/': NotFound: Basic.publish: (404) NOT_FOUND - no exchange 'scheduler_fanout' in vhost '/'
Jan 24 20:22:40.092698 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo_service.periodic_task [None req-a87b4bb9-936f-4f56-add7-d95e69ca5173 None None] Error during ComputeManager._sync_scheduler_instance_info: MessageDeliveryFailure: Unable to connect to AMQP server on 158.69.71.175:5672 after None tries: Basic.publish: (404) NOT_FOUND - no exchange 'scheduler_fanout' in vhost '/'
Jan 24 20:22:40.092914 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo_service.periodic_task Traceback (most recent call last):
Jan 24 20:22:40.093125 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo_service.periodic_task File "/usr/local/lib/python2.7/dist-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
Jan 24 20:22:40.093317 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo_service.periodic_task task(self, context)
Jan 24 20:22:40.093517 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo_service.periodic_task File "/opt/stack/old/nova/nova/compute/manager.py", line 1746, in _sync_scheduler_instance_info
Jan 24 20:22:40.093755 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo_service.periodic_task self.scheduler_client.sync_instance_info(context, self.host, uuids)
Jan 24 20:22:40.094092 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo_service.periodic_task File "/opt/stack/old/nova/nova/scheduler/client/__init__.py", line 80, in sync_instance_info
Jan 24 20:22:40.094386 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo_service.periodic_task self.queryclient.sync_instance_info(context, host_name, instance_uuids)
Jan 24 20:22:40.094536 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo_service.periodic_task File "/opt/stack/old/nova/nova/scheduler/client/__init__.py", line 35, in __run_method
Jan 24 20:22:40.094672 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo_service.periodic_task return getattr(self.instance, __name)(*args, **kwargs)
Jan 24 20:22:40.094811 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo_service.periodic_task File "/opt/stack/old/nova/nova/scheduler/client/query.py", line 95, in sync_instance_info
Jan 24 20:22:40.094964 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo_service.periodic_task instance_uuids)
Jan 24 20:22:40.095163 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo_service.periodic_task File "/opt/stack/old/nova/nova/scheduler/rpcapi.py", line 185, in sync_instance_info
Jan 24 20:22:40.095313 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo_service.periodic_task instance_uuids=instance_uuids)
Jan 24 20:22:40.095450 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo_service.periodic_task File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 154, in cast
Jan 24 20:22:40.095597 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo_service.periodic_task self.transport._send(self.target, msg_ctxt, msg, retry=self.retry)
Jan 24 20:22:40.095739 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo_service.periodic_task File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 133, in _send
Jan 24 20:22:40.095850 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo_service.periodic_task retry=retry)
Jan 24 20:22:40.095984 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo_service.periodic_task File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
Jan 24 20:22:40.096094 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo_service.periodic_task call_monitor_timeout, retry=retry)
Jan 24 20:22:40.096207 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo_service.periodic_task File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 557, in _send
Jan 24 20:22:40.096358 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo_service.periodic_task conn.fanout_send(target.topic, msg, retry=retry)
Jan 24 20:22:40.096495 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo_service.periodic_task File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1274, in fanout_send
Jan 24 20:22:40.096616 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo_service.periodic_task self._ensure_publishing(self._publish, exchange, msg, retry=retry)
Jan 24 20:22:40.096736 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo_service.periodic_task File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1141, in _ensure_publishing
Jan 24 20:22:40.096844 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo_service.periodic_task self.ensure(method, retry=retry, error_callback=_error_callback)
Jan 24 20:22:40.096967 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo_service.periodic_task File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 811, in ensure
Jan 24 20:22:40.097088 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo_service.periodic_task raise exceptions.MessageDeliveryFailure(msg)
Jan 24 20:22:40.097226 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo_service.periodic_task MessageDeliveryFailure: Unable to connect to AMQP server on 158.69.71.175:5672 after None tries: Basic.publish: (404) NOT_FOUND - no exchange 'scheduler_fanout' in vhost '/'
Jan 24 20:22:40.097396 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo_service.periodic_task
Jan 24 20:22:40.097557 ubuntu-xenial-ovh-bhs1-0002107027 nova-compute[27166]: ERROR oslo.messaging._drivers.impl_rabbit [-] [fc28e0b5-9a3f-4a6c-8aba-bfbc9dd7a7cf] AMQP server on 158.69.71.175:5672 is unreachable: connection already closed. Trying again in 1 seconds.: RecoverableConnectionError: connection already closed

This could be a side effect of https://review.openstack.org/#/c/631811/ because nova-compute is maybe using the wrong transport_url?

Although the subnode is the only one hitting that; nova-compute on the primary node is not:

http://logs.openstack.org/04/632904/2/check/legacy-grenade-dsvm-neutron-multinode-live-migration/8f886af/logs/screen-n-cpu.txt.gz?level=TRACE

Tags:

Revision history for this message

Matt Riedemann (mriedem) wrote on 2019-01-24:

Looks like this is in all multinode grenade jobs and has been happening since before Jan 21:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22after%20None%20tries%3A%20Basic.publish%3A%20(404)%20NOT_FOUND%20-%20no%20exchange%20'scheduler_fanout'%20in%20vhost%20'%2F'%3A%20NotFound%3A%20Basic.publish%3A%20(404)%20NOT_FOUND%20-%20no%20exchange%20'scheduler_fanout'%20in%20vhost%20'%2F'%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=10d

tags:	added: nova
summary:	- nova-compute in singleconductor mode (grenade jobs) trying to reach - scheduler + nova-compute in singleconductor mode (grenade jobs) trying (and failing) + to reach scheduler
Changed in devstack:
status:	New → Confirmed

Revision history for this message

Dr. Jens Harbott (j-harbott) wrote on 2022-10-11:

It seems that this issue is about old jobs that are no longer being used. Feel free to reopen with fresh logs if this is still current.

Changed in devstack:
status:	Confirmed → Invalid

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.