[master] metadata is not working on multi-node setup

[Prashant Shetty]

Setup:
1. One controller
2. KVM and ESX nova compute
3. Two Network nodes running q-dhcp and q-meta

Nodes: Ubuntu 14.04.3 amd64 Trusty

On above setup, tried to run few metadata queries we see 500 internal server error.

$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 1.1.107.1 0.0.0.0 UG 0 0 0 eth0
1.1.107.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
169.254.169.254 1.1.107.1 255.255.255.255 UGH 0 0 0 eth0
$ curl http://169.254.169.254
500 Internal Server Error

Remote metadata server experienced an internal server error.

$

NOTE: Moved all services to single node(controller) and metadata queries seems to work fine.

vmware@cntr11:~$ neutron agent-list
+--------------------------------------+----------------+-----------+-------------------+-------+----------------+------------------------+
| id | agent_type | host | availability_zone | alive | admin_state_up | binary |
+--------------------------------------+----------------+-----------+-------------------+-------+----------------+------------------------+
| 0168206e-6c13-40df-a0b8-7772220ab9cd | DHCP agent | network-1 | nova | :-) | True | neutron-dhcp-agent |
| 08c8bb6e-0c7f-41e3-b134-81e2d5334aea | Metadata agent | network-1 | | :-) | True | neutron-metadata-agent |
| 25e9fa6d-6acc-4b8e-bb0b-f2d3ac20981d | Metadata agent | network-2 | | :-) | True | neutron-metadata-agent |
| 534d349c-8830-4648-814b-611a30f59287 | DHCP agent | network-2 | nova | :-) | True | neutron-dhcp-agent |
+--------------------------------------+----------------+-----------+-------------------+-------+----------------+------------------------+
vmware@cntr11:~$
vmware@cntr11:~$ nova service-list
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| 7 | nova-conductor | cntr11 | internal | enabled | up | 2016-09-16T10:00:22.000000 | - |
| 9 | nova-compute | esx-ubuntu-01 | nova | enabled | up | 2016-09-16T10:00:14.000000 | - |
| 10 | nova-compute | kvm-3 | nova | enabled | up | 2016-09-16T10:00:23.000000 | - |
| 11 | nova-compute | kvm-2 | nova | enabled | up | 2016-09-16T10:00:19.000000 | - |
| 12 | nova-compute | kvm-1 | nova | enabled | up | 2016-09-16T10:00:19.000000 | - |
| 13 | nova-scheduler | cntr11 | internal | enabled | up | 2016-09-16T10:00:15.000000 | - |
| 14 | nova-consoleauth | cntr11 | internal | enabled | up | 2016-09-16T10:00:20.000000 | - |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
vmware@cntr11:~$

Logs:
2016-09-13 13:31:50.713 14309 DEBUG eventlet.wsgi.server [-] (14309) accepted '' server /usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py:868
2016-09-13 13:31:50.715 14309 DEBUG neutron.agent.metadata.agent [-] Request: GET / HTTP/1.0^M
Accept-Encoding: gzip, deflate^M
Content-Length: 0^M
Content-Type: text/plain^M
Host: 169.254.169.254^M
User-Agent: Python-httplib2/0.9.2 (gzip)^M
X-Forwarded-For: 1.1.107.3^M
X-Neutron-Router-Id: bbe453a5-db77-4cd9-af02-31232a222f16 __call__ /opt/stack/neutron/neutron/agent/metadata/agent.py:86
2016-09-13 13:31:50.716 14309 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: 50b283bdeab945cb93872300abd5b47c exchange 'neutron' topic 'q-plugin' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448
2016-09-13 13:31:50.807 14309 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 50b283bdeab945cb93872300abd5b47c __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:296
2016-09-13 13:31:50.808 14309 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: 52646ead6313405fb688b80cbb4bfc73 exchange 'neutron' topic 'q-plugin' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448
2016-09-13 13:31:50.846 14309 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 52646ead6313405fb688b80cbb4bfc73 __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:296
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent [-] Unexpected error.
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent Traceback (most recent call last):
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent File "/opt/stack/neutron/neutron/agent/metadata/agent.py", line 90, in __call__
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent return self._proxy_request(instance_id, tenant_id, req)
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent File "/opt/stack/neutron/neutron/agent/metadata/agent.py", line 193, in _proxy_request
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent body=req.body)
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1609, in request
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1351, in _request
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent (response, content) = self._conn_request(conn, request_uri, method, body, headers)
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1307, in _conn_request
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent response = conn.getresponse()
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/httplib.py", line 1129, in getresponse
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent response = self.response_class(*args, **kwds)
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/httplib.py", line 390, in __init__
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent self.fp = sock.makefile('rb', 0)
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 298, in makefile
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent dupped = self.dup()
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 289, in dup
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent newsock = type(self)(sock, set_nonblocking=False)
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent TypeError: __init__() got an unexpected keyword argument 'set_nonblocking'
2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent
2016-09-13 13:31:50.849 14309 INFO eventlet.wsgi.server [-] 1.1.107.3,<local> - - [13/Sep/2016 13:31:50] "GET / HTTP/1.1" 500 229 0.134582
2016-09-13 13:31:51.854 13766 DEBUG oslo_messaging._drivers.amqpdriver [-] CAST unique_id: 757d1802bce344a18d2ae1df8221058f exchange 'neutron' topic 'q-reports-plugin' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448