maximum attempts reached neutronclient

[Vasilios Tzanoudakis]

3 Controllers HA + GRE
CEPH
Ubuntu

horizon after login error : Unable to retrieve network quota information.

Error: http://paste.openstack.org/show/132927/

node1

nova logs:
http://paste.openstack.org/show/132928/

neutron logs:
http://paste.openstack.org/show/132929/

/var/log/neutron/dhcp-agent.log
http://paste.openstack.org/show/132931/

root@node-1:~/scripts# crm status
Last updated: Thu Nov 13 22:16:23 2014
Last change: Thu Nov 13 22:16:17 2014 via crm_attribute on node-3
Stack: classic openais (with plugin)
Current DC: node-3 - partition with quorum
Version: 1.1.10-42f2063
3 Nodes configured, 3 expected votes
22 Resources configured

Online: [ node-1 node-2 node-3 ]

 vip__management_old (ocf::mirantis:ns_IPaddr2): Started node-1
 vip__public_old (ocf::mirantis:ns_IPaddr2): Started node-3
 p_ceilometer-alarm-evaluator (ocf::mirantis:ceilometer-alarm-evaluator): Started node-2
 p_ceilometer-agent-central (ocf::mirantis:ceilometer-agent-central): Started node-3
 Master/Slave Set: master_p_rabbitmq-server [p_rabbitmq-server]
     Masters: [ node-1 ]
     Slaves: [ node-2 node-3 ]
 Clone Set: clone_p_mysql [p_mysql]
     Started: [ node-1 node-2 node-3 ]
 Clone Set: clone_p_haproxy [p_haproxy]
     Started: [ node-1 node-2 node-3 ]
 p_heat-engine (ocf::mirantis:heat-engine): Started node-1
 Clone Set: clone_p_neutron-plugin-openvswitch-agent [p_neutron-plugin-openvswitch-agent]
     Started: [ node-1 node-2 node-3 ]
 Clone Set: clone_p_neutron-metadata-agent [p_neutron-metadata-agent]
     Started: [ node-1 node-2 node-3 ]
 p_neutron-dhcp-agent (ocf::mirantis:neutron-agent-dhcp): Started node-1
 p_neutron-l3-agent (ocf::mirantis:neutron-agent-l3): Started node-3

[Vasilios Tzanoudakis (vtzanoudakis)]

5.1.1 nightly ISO 28_10_2014

[Vasilios Tzanoudakis (vtzanoudakis)]

node2:

/var/log/nova/nova-api.log
http://paste.openstack.org/show/132933/

neutron logs
http://paste.openstack.org/show/132934/

please advice any possible solution

[Vasilios Tzanoudakis (vtzanoudakis)]

root@node-1# nova --debug list
http://paste.openstack.org/show/132946/

[Vasilios Tzanoudakis (vtzanoudakis)]

/etc/init.d/keystone restart on node1 fixed the issue

/var/log/keystone-all.log doesn't show any bad things but i am pasting it just FYI.

http://paste.openstack.org/show/132988/

thanks

[Matthew Mosesohn (raytrac3r)]

Can you provide a diagnostic snapshot?

[Mike Scherbakov (mihgen)]

Vasilios, thanks for the bug report.
There is an exception in neutronclient, according to the logs, so assigning mos-neutron team to take a look and possibly to reproduce the issue.

2014-11-13 22:18:57.787 118067 ERROR nova.api.openstack.wsgi [req-45ca22b4-2660-49c1-aeca-71e4b40e292e None] Exception handling resource: unsupported operand type(s) for +: 'NoneType' and 'str'
2014-11-13 22:18:57.787 118067 TRACE nova.api.openstack.wsgi Traceback (most recent call last):
2014-11-13 22:18:57.787 118067 TRACE nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 985, in _process_stack
2014-11-13 22:18:57.787 118067 TRACE nova.api.openstack.wsgi action_result = self.dispatch(meth, request, action_args)
2014-11-13 22:18:57.787 118067 TRACE nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1072, in dispatch
2014-11-13 22:18:57.787 118067 TRACE nova.api.openstack.wsgi return method(req=request, **action_args)
2014-11-13 22:18:57.787 118067 TRACE nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/floating_ips.py", line 142, in index
2014-11-13 22:18:57.787 118067 TRACE nova.api.openstack.wsgi floating_ips = self.network_api.get_floating_ips_by_project(context)
2014-11-13 22:18:57.787 118067 TRACE nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 887, in get_floating_ips_by_project
2014-11-13 22:18:57.787 118067 TRACE nova.api.openstack.wsgi fips = client.list_floatingips(tenant_id=project_id)['floatingips']
2014-11-13 22:18:57.787 118067 TRACE nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 111, in with_params
2014-11-13 22:18:57.787 118067 TRACE nova.api.openstack.wsgi ret = self.function(instance, *args, **kwargs)
2014-11-13 22:18:57.787 118067 TRACE nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 437, in list_floatingips
2014-11-13 22:18:57.787 118067 TRACE nova.api.openstack.wsgi **_params)
2014-11-13 22:18:57.787 118067 TRACE nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1250, in list
2014-11-13 22:18:57.787 118067 TRACE nova.api.openstack.wsgi for r in self._pagination(collection, path, **params):
2014-11-13 22:18:57.787 118067 TRACE nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1263, in _pagination
2014-11-13 22:18:57.787 118067 TRACE nova.api.openstack.wsgi res = self.get(path, params=params)
2014-11-13 22:18:57.787 118067 TRACE nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1236, in get
2014-11-13 22:18:57.787 118067 TRACE nova.api.openstack.wsgi headers=headers, params=params)
2014-11-13 22:18:57.787 118067 TRACE nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1221, in retry_request
2014-11-13 22:18:57.787 118067 TRACE nova.api.openstack.wsgi headers=headers, params=params)
2014-11-13 22:18:57.7...

[Vasilios Tzanoudakis (vtzanoudakis)]

Ok let me know if you need any further information.

[Vladimir Kuklin (vkuklin)]

Not confirmed for 6.0/Juno release. Marking as incomplete for 6.0

[Vasilios Tzanoudakis (vtzanoudakis)]

Here is the diagnostic snapshot guys.

https://www.dropbox.com/s/sqb17b2dg239mll/fuel-snapshot-2014-11-18_08-32-19.tgz?dl=0

I had to restart keystone service on master node some minutes ago once again.

Regards

[Ilya Shakhat (shakhat)]

According to the attached log from Nova the error occurs on initialization of neutron client. This happens if it gets empty parameter "auth_url", that in turn is set in https://github.com/openstack/nova/blob/stable/icehouse/nova/network/neutronv2/__init__.py#L58 from "neutron_admin_auth_url" defined in /etc/nova/nova.conf
On the latest 5.1.1 build (#17, 2014-11-16_21-00-23) both configuration file and code are correct and the issue is not reproducible.

[Ilya Shakhat (shakhat)]

The issue is not reproducible on 5.1.1 build (#17, 2014-11-16_21-00-23). Moving into Incomplete state

[Dmitry Mescheryakov (dmitrymex)]

The issue is hanging in incomplete for a month, hence moving it to invalid