Description of problem:
VM with SRIOV port cannot boot on setup with OVN
It looks like the SRIOV NIC Agent RPC Daemon Started but there is an error that Agent out of sync with plugin!:
/var/log/neutron/sriov-nic-agent.log
2017-10-31 15:51:56.281 17158 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [req-119c03f8-f722-4b7a-a69b-3aa3064fd407 - - - - -] Agent initialized successfully, now running...
2017-10-31 15:51:56.282 17158 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [req-119c03f8-f722-4b7a-a69b-3aa3064fd407 - - - - -] SRIOV NIC Agent RPC Daemon Started!
2017-10-31 15:51:56.283 17158 INFO neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [req-119c03f8-f722-4b7a-a69b-3aa3064fd407 - - - - -] Agent out of sync with plugin!
2017-10-31 15:51:57.106 17158 INFO oslo_rootwrap.client [req-119c03f8-f722-4b7a-a69b-3aa3064fd407 - - - - -] Spawned new rootwrap daemon process with pid=18224
2017-10-31 18:57:41.765 17158 ERROR oslo.messaging._drivers.impl_rabbit [-] [163a9ea7-017c-4278-a0db-054e201e3985] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 110] Connection timed out. Trying again in 1 seconds. Client port: None: error: [Errno 110] Connection timed out
2017-10-31 18:57:48.895 17158 ERROR oslo.messaging._drivers.impl_rabbit [-] [6ab7209f-dbc9-4d36-9f28-6281ddd174d1] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 110] Connection timed out. Trying again in 1 seconds. Client port: None: error: [Errno 110] Connection timed out
2017-10-31 18:57:54.694 17158 ERROR oslo.messaging._drivers.impl_rabbit [-] [2274bf5e-bec3-41c3-b34d-514b8427aadd] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out
When trying to boot VM with SRIOV port (direct) it starts with error:
/var/log/containers/nova/nova-conductor.log
017-10-31 15:53:59.834 20 WARNING oslo_config.cfg [req-a76e925d-b0d9-48ac-9e67-0fcfd5d2ad13 - - - - -] Option "rabbit_password" from group "oslo_messaging_rabbit" is deprecated for removal (Replaced by [DEFAULT]/transport_url). Its value may be silently ignored in the future.
2017-11-01 07:26:52.210 21 ERROR nova.conductor.manager [req-98048079-d574-4e0a-b1b7-70ae4663d56a 9daa58f5aaa949d8be28ac894ee65340 6f210d906fee4f1b9a7a6c26bcc96b46 - default default] Failed to schedule instances: NoValidHost_Remote: No valid host was found. There are not enough hosts available.
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 232, in inner
return func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 149, in select_destinations
alloc_reqs_by_rp_uuid, provider_summaries)
File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 109, in select_destinations
raise exception.NoValidHost(reason=reason)
NoValidHost: No valid host was found. There are not enough hosts available.
2017-11-01 07:26:52.210 21 ERROR nova.conductor.manager Traceback (most recent call last):
2017-11-01 07:26:52.210 21 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 1027, in schedule_and_build_instances
2017-11-01 07:26:52.210 21 ERROR nova.conductor.manager instance_uuids)
2017-11-01 07:26:52.210 21 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 626, in _schedule_instances
2017-11-01 07:26:52.210 21 ERROR nova.conductor.manager request_spec, instance_uuids)
2017-11-01 07:26:52.210 21 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/utils.py", line 586, in wrapped
2017-11-01 07:26:52.210 21 ERROR nova.conductor.manager return func(*args, **kwargs)
2017-11-01 07:26:52.210 21 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 52, in select_destinations
2017-11-01 07:26:52.210 21 ERROR nova.conductor.manager instance_uuids)
2017-11-01 07:26:52.210 21 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2017-11-01 07:26:52.210 21 ERROR nova.conductor.manager return getattr(self.instance, __name)(*args, **kwargs)
2017-11-01 07:26:52.210 21 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 33, in select_destinations
2017-11-01 07:26:52.210 21 ERROR nova.conductor.manager instance_uuids)
2017-11-01 07:26:52.210 21 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 137, in select_destinations
2017-11-01 07:26:52.210 21 ERROR nova.conductor.manager return cctxt.call(ctxt, 'select_destinations', **msg_args)
2017-11-01 07:26:52.210 21 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2017-11-01 07:26:52.210 21 ERROR nova.conductor.manager retry=self.retry)
2017-11-01 07:26:52.210 21 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 123, in _send
2017-11-01 07:26:52.210 21 ERROR nova.conductor.manager timeout=timeout, retry=retry)
2017-11-01 07:26:52.210 21 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 578, in send
2017-11-01 07:26:52.210 21 ERROR nova.conductor.manager retry=retry)
2017-11-01 07:26:52.210 21 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 569, in _send
2017-11-01 07:26:52.210 21 ERROR nova.conductor.manager raise result
2017-11-01 07:26:52.210 21 ERROR nova.conductor.manager NoValidHost_Remote: No valid host was found. There are not enough hosts available.
2017-11-01 07:26:52.210 21 ERROR nova.conductor.manager Traceback (most recent call last):
SOS-report attached
Version-Release number of selected component (if applicable):
Pike
How reproducible:
always
Steps to Reproduce:
1. deploy sriov setup with OVN
2. create environment with SRiov instance
3. check the logs
Actual results:
Error
Expected results:
SRIOV instance should boot with no errors
Additional info:
SOS-report attached
networking-odl is for OpenDaylight. I guess it's typo.