ConnectionFailed: Connection to neutron failed: Maximum attempts reached while trying to create neutron network in kilo

[Larry Michel]

Seeing issue below when using nsx-transport-node with kilo and trying to create a neutron network.

This is not recreatable with Juno.

+ oil-ci do prepare --yaml_out artifacts/resources.yaml --resources image,network --env_yaml artifacts/envrc.yaml --environment ci-oil-slave10 --run_config artifacts/run_config
2016-03-06 07:43:59,089 [INFO] oil_ci.openstack.image: Publishing local image from /mnt/images/lxc/trusty-server-cloudimg-amd64.img to glance as "trusty-server-cloudimg-amd64.img"
2016-03-06 07:43:59,097 [INFO] urllib3.connectionpool: Starting new HTTP connection (1): 10.245.39.108
2016-03-06 07:57:20,149 [INFO] oil_ci.openstack.image: Published image trusty-server-cloudimg-amd64.img /w ID 3901422c-8ef3-4607-8868-5bee04819aff
2016-03-06 07:57:20,156 [INFO] urllib3.connectionpool: Starting new HTTP connection (1): 10.245.39.108
2016-03-06 07:57:20,984 [ERROR] oil_ci.prepare.oil_prepper: Failed to prepare deployed cloud
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/oil_ci/prepare/oil_prepper.py", line 134, in do_prep
    net = self.prepare_network()
  File "/usr/lib/python2.7/dist-packages/oil_ci/prepare/oil_prepper.py", line 86, in prepare_network
    self._env)
  File "/usr/lib/python2.7/dist-packages/oil_ci/prepare/network.py", line 306, in prepare_neutron_network
    tenant_net = neutron.create_network(name=params['fixed_net_name'])
  File "/usr/lib/python2.7/dist-packages/oil_ci/prepare/network.py", line 76, in create_network
    net = self.client.list_networks(name=name).get('networks')
  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 111, in with_params
    ret = self.function(instance, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 333, in list_networks
    **_params)
  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1250, in list
    for r in self._pagination(collection, path, **params):
  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1263, in _pagination
    res = self.get(path, params=params)
  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1236, in get
    headers=headers, params=params)
  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1228, in retry_request
    raise exceptions.ConnectionFailed(reason=_("Maximum attempts reached"))
ConnectionFailed: Connection to neutron failed: Maximum attempts reached

Here is a bundle to recreate:

https://pastebin.canonical.com/151319/

[Jason Hobbs (jason-hobbs)]

Can you attach the logs from the neutron API server? Since the client couldn't connect to it, that's the first place to look.

[Larry Michel (lmic)]

I couldn't find any neutron-api logs. Only neutron log I found on neutron-gateway, neutron-api and nova-compute were these:

$ find . -name "neutron*.log"
./neutron-gateway_0-nsx-transport-node_0-hayward-29.oil/var/log/neutron/neutron-metadata-agent.log
./neutron-gateway_0-nsx-transport-node_0-hayward-29.oil/var/log/upstart/neutron-metadata-agent.log
./neutron-gateway_0-nsx-transport-node_0-hayward-29.oil/var/log/upstart/neutron-dhcp-agent.log
./nova-compute_0-neutron-openvswitch_0-ceilometer-agent_0-swift-storage_0-mesprit.oil/var/log/upstart/neutron-plugin-openvswitch-agent.log
./nova-compute_0-neutron-openvswitch_0-ceilometer-agent_0-swift-storage_0-mesprit.oil/var/log/upstart/neutron-ovs-cleanup.log

I am attaching all logs from those nodes.