kolla

Nova VM does not reach ACTIVE state in CentOS binary gate

Bug #1568927 reported by Vikram Hosakote on 2016-04-11

This bug affects 2 people

	Status	Importance	Assigned to	Milestone
kolla	Invalid	Low	Unassigned	kolla newton-rc2 "rc2"
Liberty	Fix Released	Critical	Unassigned	kolla 1.1.3 "liberty"
Mitaka	Fix Released	Critical	Unassigned	kolla 2.0.3 "mitaka"

Bug Description

Nova VM does not reach ACTIVE state in CentOS binary gate.

http://logs.openstack.org/13/245013/6/check/gate-kolla-dsvm-deploy-centos-binary/497b18e/console.html#_2016-04-11_08_57_15_935

Looks like a nova / neutron issue.

nova-api.log:
==========

http://logs.openstack.org/13/245013/6/check/gate-kolla-dsvm-deploy-centos-binary/497b18e/logs/kolla/nova/nova-api.log

Errors in the above nova-api.log:

2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions [req-ec671e79-2c9c-4d47-8544-45664916a4ba e1bc63258580423891d7141b17f4bf31 4a2971b47f5c446d83095126dfda5089 - - -] Unexpected exception in API method
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 630, in create
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions **create_kwargs)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 154, in inner
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions rv = f(*args, **kwargs)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1556, in create
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions check_server_group_quota=check_server_group_quota)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1139, in _create_instance
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions reservation_id, max_count)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 834, in _validate_and_build_base_options
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions requested_networks, max_count)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 448, in _check_requested_networks
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions max_count)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 49, in wrapped
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions return func(self, context, *args, **kwargs)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 399, in validate_networks
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions requested_networks)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/network/rpcapi.py", line 212, in validate_networks
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions return self.client.call(ctxt, 'validate_networks', networks=networks)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 413, in call
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions return self.prepare().call(ctxt, method, **kwargs)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions retry=self.retry)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions timeout=timeout, retry=retry)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions retry=retry)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in _send
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions result = self._waiter.wait(msg_id, timeout)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 342, in wait
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions message = self.waiters.get(msg_id, timeout=timeout)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in get
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions 'to message ID %s' % msg_id)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions MessagingTimeout: Timed out waiting for a reply to message ID 0a95d829d05c492c9a11c7854e83c72c
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions
2016-04-11 08:57:13.588 27 INFO nova.api.openstack.wsgi [req-ec671e79-2c9c-4d47-8544-45664916a4ba e1bc63258580423891d7141b17f4bf31 4a2971b47f5c446d83095126dfda5089 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.exceptions.MessagingTimeout'>
2016-04-11 08:57:13.594 27 INFO nova.osapi_compute.wsgi.server [req-ec671e79-2c9c-4d47-8544-45664916a4ba e1bc63258580423891d7141b17f4bf31 4a2971b47f5c446d83095126dfda5089 - - -] Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 483, in handle_one_response
    write(b''.join(towrite))
  File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 426, in write
    _writelines(towrite)
  File "/usr/lib64/python2.7/socket.py", line 334, in writelines
    self.flush()
  File "/usr/lib64/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 377, in sendall
    tail = self.send(data, flags)
  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 359, in send
    total_sent += fd.send(data[total_sent:], flags)
error: [Errno 104] Connection reset by peer

nova-compute.log:
===============

http://logs.openstack.org/13/245013/6/check/gate-kolla-dsvm-deploy-centos-binary/497b18e/logs/kolla/nova/nova-compute.log

Errors in the above nova-compute.log:

2016-04-11 08:56:00.824 1 ERROR nova.compute.manager [req-015ab636-e753-479d-a699-4809608beb8d - - - - -] No compute node record for host node1

neutron-dhcp-agent.log:
====================

http://logs.openstack.org/13/245013/6/check/gate-kolla-dsvm-deploy-centos-binary/497b18e/logs/kolla/neutron/

Errors in the above neutron-dhcp-agent.log:

2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent [req-df0efeb6-9915-4d73-b10e-a0b657e0f3c1 - - - - -] Failed reporting state!
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py", line 581, in _report_state
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent ctx, self.agent_state, True)
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 86, in report_state
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent return method(context, 'report_state', **kwargs)
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent retry=self.retry)
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent timeout=timeout, retry=retry)
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent retry=retry)
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in _send
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent result = self._waiter.wait(msg_id, timeout)
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 342, in wait
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent message = self.waiters.get(msg_id, timeout=timeout)
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in get
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent 'to message ID %s' % msg_id)
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent MessagingTimeout: Timed out waiting for a reply to message ID 7dfbc0addf3b4cbc8aa78ac4569422a9
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent
2016-04-11 08:55:58.616 1 WARNING oslo.service.loopingcall [req-df0efeb6-9915-4d73-b10e-a0b657e0f3c1 - - - - -] Function 'neutron.agent.dhcp.agent.DhcpAgentWithStateReport._report_state' run outlasted interval by 30.09 sec
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent [-] Unable to sync network state.
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py", line 154, in sync_state
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent active_networks = self.plugin_rpc.get_active_networks_info()
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py", line 438, in get_active_networks_info
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent host=self.host)
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent retry=self.retry)
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent timeout=timeout, retry=retry)
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent retry=retry)
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in _send
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent result = self._waiter.wait(msg_id, timeout)
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 342, in wait
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent message = self.waiters.get(msg_id, timeout=timeout)
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in get
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent 'to message ID %s' % msg_id)
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent MessagingTimeout: Timed out waiting for a reply to message ID 14a548c2b7104cac82922aef04acd141
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent

Tags:

Steven Dake (sdake) on 2016-04-17

Changed in kolla:
status:	New → Triaged
importance:	Undecided → Critical
milestone:	none → newton-1

Revision history for this message

Swapnil Kulkarni (coolsvap-deactivatedaccount) wrote on 2016-06-30:

does not seem to exist currently.[1]

[1] http://logs.openstack.org/46/327146/11/check/gate-kolla-dsvm-deploy-centos-binary/c56cfd1/console.html#_2016-06-30_02_42_03_063873

tags:	added: retriage
Changed in kolla:
milestone:	newton-1 → newton-2

Swapnil Kulkarni (coolsvap-deactivatedaccount) on 2016-07-20

Changed in kolla:
milestone:	newton-2 → newton-3

Swapnil Kulkarni (coolsvap-deactivatedaccount) on 2016-08-31

Changed in kolla:
milestone:	newton-3 → newton-rc1

zhubingbing (zhubingbing) on 2016-09-18

Changed in kolla:
milestone:	newton-rc1 → newton-rc2

Revision history for this message

zhubingbing (zhubingbing) wrote on 2016-09-19:

I tested the master branch, gate now looks good, this bug I think it should be closed

Changed in kolla:
status:	Triaged → Won't Fix
importance:	Critical → Low

Revision history for this message

zhubingbing (zhubingbing) wrote on 2016-09-19:

Now can't reproduce this bug

Steven Dake (sdake) on 2016-10-07

Changed in kolla:
status:	Won't Fix → Invalid

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.