kolla-ansible

TLS misconfigured for neutron metadata agent

Bug #1864615 reported by Radosław Piliszek on 2020-02-25

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
kolla-ansible	Fix Released	High	Radosław Piliszek	kolla-ansible 10.0.0 "ussuri"
Train	Fix Released	High	Mark Goddard	kolla-ansible 9.1.0 "Train"
Ussuri	Fix Released	High	Radosław Piliszek	kolla-ansible 10.0.0 "ussuri"

Bug Description

Neutron metadata agent trying to contact with nova api metadata:

neutron-metadata-agent:
2020-02-24 21:09:03.871 724 DEBUG neutron.agent.metadata.agent [-] Gotten ports for remote_address 10.0.0.28, network_id 67c62046-6a3b-4cea-97e1-ac98600b4341, router_id None are: [{'id': '78199450-a67f-42eb-ad25-9515ba948049', 'name': '', 'network_id': '67c62046-6a3b-4cea-97e1-ac98600b4341', 'tenant_id': '9949b95aa9f7434ea594e2ea4073635b', 'mac_address': 'fa:16:3e:08:c1:f6', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '4862f37a-7164-4dab-86dc-9c7c34f35130', 'device_owner': 'compute:nova', 'fixed_ips': [{'subnet_id': 'd60989d4-4657-485c-bd2a-e98e489d82dd', 'ip_address': '10.0.0.28'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': ['fd94e201-78cb-4593-a744-618c99cdd38d'], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'primary', 'binding:vif_type': 'ovs', 'binding:vif_details': {'connectivity': 'l2', 'port_filter': True, 'ovs_hybrid_plug': True, 'datapath_type': 'system', 'bridge_name': 'br-int'}, 'port_security_enabled': True, 'tags': [], 'created_at': '2020-02-24T21:07:53Z', 'updated_at': '2020-02-24T21:08:02Z', 'revision_number': 4, 'project_id': '9949b95aa9f7434ea594e2ea4073635b'}] _get_instance_and_tenant_id /var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/metadata/agent.py:168
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent [-] Unexpected error.: requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent Traceback (most recent call last):
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent chunked=chunked,
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 421, in _make_request
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent six.raise_from(e, None)
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent File "<string>", line 3, in raise_from
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 416, in _make_request
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent httplib_response = conn.getresponse()
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent File "/usr/lib/python3.6/http/client.py", line 1346, in getresponse
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent response.begin()
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent File "/usr/lib/python3.6/http/client.py", line 307, in begin
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent version, status, reason = self._read_status()
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent File "/usr/lib/python3.6/http/client.py", line 276, in _read_status
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent raise RemoteDisconnected("Remote end closed connection without"
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent http.client.RemoteDisconnected: Remote end closed connection without response
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent During handling of the above exception, another exception occurred:
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent Traceback (most recent call last):
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent timeout=timeout
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/urllib3/util/retry.py", line 400, in increment
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent raise six.reraise(type(error), error, _stacktrace)
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/urllib3/packages/six.py", line 734, in reraise
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent raise value.with_traceback(tb)
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent chunked=chunked,
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 421, in _make_request
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent six.raise_from(e, None)
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent File "<string>", line 3, in raise_from
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 416, in _make_request
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent httplib_response = conn.getresponse()
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent File "/usr/lib/python3.6/http/client.py", line 1346, in getresponse
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent response.begin()
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent File "/usr/lib/python3.6/http/client.py", line 307, in begin
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent version, status, reason = self._read_status()
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent File "/usr/lib/python3.6/http/client.py", line 276, in _read_status
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent raise RemoteDisconnected("Remote end closed connection without"
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent During handling of the above exception, another exception occurred:
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent Traceback (most recent call last):
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/metadata/agent.py", line 90, in __call__
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent return self._proxy_request(instance_id, tenant_id, req)
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/metadata/agent.py", line 208, in _proxy_request
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent verify=verify_cert)
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/requests/api.py", line 61, in request
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent return session.request(method=method, url=url, **kwargs)
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent resp = self.send(prep, **send_kwargs)
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent r = adapter.send(request, **kwargs)
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent raise ConnectionError(err, request=request)
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
2020-02-24 21:09:03.874 724 ERROR neutron.agent.metadata.agent
2020-02-24 21:09:03.876 724 INFO eventlet.wsgi.server [-] 10.0.0.28,<local> "GET /2009-04-04/meta-data/instance-id HTTP/1.1" status: 500 len: 362 time: 0.0907757

haproxy:
{"Payload":"Feb 24 14:50:33 haproxy[29]: 192.0.2.10:35240 [24/Feb/2020:14:50:33.182] nova_metadata/1: SSL handshake failure","log_level":"info","Hostname":"primary","programname":"haproxy"}

Tags:

Revision history for this message

Radosław Piliszek (yoctozepto) wrote on 2020-02-25:

testing fix in https://review.opendev.org/709369

Revision history for this message

Radosław Piliszek (yoctozepto) wrote on 2020-02-25:

basic fix needs to go to Train as we support the global switch since then; CA is Ussuri-only so far

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-02-25: Fix proposed to kolla-ansible (master)

Fix proposed to branch: master
Review: https://review.opendev.org/709829

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-02-27: Fix merged to kolla-ansible (master)

Reviewed: https://review.opendev.org/709829
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=7c200db70ee071bdfc1c77216ece434bc8816328
Submitter: Zuul
Branch: master

commit 7c200db70ee071bdfc1c77216ece434bc8816328
Author: Radosław Piliszek <email address hidden>
Date: Tue Feb 25 21:30:56 2020 +0100

Fix client TLS in neutron-metadata-agent

Backport to: Train.

Change-Id: Ide96ea43739d47e623026f0aecd4163f3a2abe7f
Closes-bug: #1864615

Changed in kolla-ansible:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-02-28: Fix proposed to kolla-ansible (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/710417

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-02-28: Fix merged to kolla-ansible (stable/train)

Reviewed: https://review.opendev.org/710417
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=4e0106f5480935f6486de5cc4e39e07c711b63ab
Submitter: Zuul
Branch: stable/train

commit 4e0106f5480935f6486de5cc4e39e07c711b63ab
Author: Radosław Piliszek <email address hidden>
Date: Tue Feb 25 21:30:56 2020 +0100

Fix client TLS in neutron-metadata-agent

Backport to: Train.

    Change-Id: Ide96ea43739d47e623026f0aecd4163f3a2abe7f
    Closes-bug: #1864615
    (cherry picked from commit 7c200db70ee071bdfc1c77216ece434bc8816328)

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.