Juniper Openstack

Contrail :: 16.04 build 4 Ocata :: metadata ssl fails.

Series trunk
Bug #1734110

Bug #1734110 reported by Ritam Gangopadhyay on 2017-11-23

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Juniper Openstack	Status tracked in Trunk
R4.1	Fix Committed	Critical	Ramprakash R	Juniper Openstack r4.1.1.0
Trunk	Fix Committed	Critical	Ramprakash R	Juniper Openstack r5.0.0

Bug Description

Setup:- R4.1 build 3 Ocata multi node setup.

nodei19 10.204.217.131 openstack
nodec28 10.204.217.13 controller, analytics, analyticsdb
nodec10 10.204.217.176 controller, analytics, analyticsdb
nodec33 10.204.217.168 controller, analytics, analyticsdb
nodeg37 10.204.217.77 lb
nodei17 10.204.217.129 compute
nodei20 10.204.217.132 compute

***************************************************************************************

metadata ssl fails and seeing a traceback in metadata service logs

Created VM on nodei17 and sent request to the service ip

ubuntu@ctest-vm-in-vn2-metadata-26806778:~$ curl http://169.254.169.254
<html>
<head>
<title>502 Bad Gateway</title>
</head>
</html>
ubuntu@ctest-vm-in-vn2-metadata-26806778:~$ curl https://169.254.169.254
curl: (7) couldn't connect to host
ubuntu@ctest-vm-in-vn2-metadata-26806778:~$

***************************************************************************************

configuration on nova and vrouter agent side looks fine:-

root@nodei17:~# grep -rn metadata /etc/contrail/contrail-vrouter-agent.conf
206:metadata_proxy_secret = 2C157798B383
207:metadata_client_cert = /etc/contrail/ssl/certs/server.pem
208:metadata_use_ssl = True
210:metadata_client_key = /etc/contrail/ssl/private/server-privkey.pem

root@nodei19:~# docker exec -it nova_api cat /etc/nova/nova.conf | grep ssl
enabled_ssl_apis = metadata
ssl_cert_file = /etc/nova/ssl/certs/nova.pem
ssl_key_file = /etc/nova/ssl/private/novakey.pem
ssl_ca_file = /etc/nova/ssl/certs/ca.pem
root@nodei19:~#

***************************************************************************************

Traceback seen in neutron metadata agent

2017-11-23 16:35:38.702 7 ERROR neutron.agent.metadata.agent [-] Failed reporting state!
2017-11-23 16:35:38.702 7 ERROR neutron.agent.metadata.agent Traceback (most recent call last):
2017-11-23 16:35:38.702 7 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 262, in _report_state
2017-11-23 16:35:38.702 7 ERROR neutron.agent.metadata.agent use_call=self.agent_state.get('start_flag'))
2017-11-23 16:35:38.702 7 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/rpc.py", line 87, in report_state
2017-11-23 16:35:38.702 7 ERROR neutron.agent.metadata.agent return method(context, 'report_state', **kwargs)
2017-11-23 16:35:38.702 7 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
2017-11-23 16:35:38.702 7 ERROR neutron.agent.metadata.agent retry=self.retry)
2017-11-23 16:35:38.702 7 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 97, in _send
2017-11-23 16:35:38.702 7 ERROR neutron.agent.metadata.agent timeout=timeout, retry=retry)
2017-11-23 16:35:38.702 7 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 458, in send
2017-11-23 16:35:38.702 7 ERROR neutron.agent.metadata.agent retry=retry)
2017-11-23 16:35:38.702 7 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 447, in _send
2017-11-23 16:35:38.702 7 ERROR neutron.agent.metadata.agent result = self._waiter.wait(msg_id, timeout)
2017-11-23 16:35:38.702 7 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 339, in wait
2017-11-23 16:35:38.702 7 ERROR neutron.agent.metadata.agent message = self.waiters.get(msg_id, timeout=timeout)
2017-11-23 16:35:38.702 7 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 238, in get
2017-11-23 16:35:38.702 7 ERROR neutron.agent.metadata.agent 'to message ID %s' % msg_id)
2017-11-23 16:35:38.702 7 ERROR neutron.agent.metadata.agent MessagingTimeout: Timed out waiting for a reply to message ID 48b4e5c2576b43b9a6940e948271f03b
2017-11-23 16:35:38.702 7 ERROR neutron.agent.metadata.agent
2017-11-23 16:35:38.703 7 WARNING oslo.service.loopingcall [-] Function 'neutron.agent.metadata.agent.UnixDomainMetadataProxy._report_state' run outlasted interval by 30.00 sec

***************************************************************************************

I see nova listening on 8775 for metadata service connections:-

root@nodei19:~# netstat -anp | grep 8775
tcp 0 0 192.168.100.15:8775 0.0.0.0:* LISTEN 11356/python
tcp 0 0 10.204.217.184:8775 0.0.0.0:* LISTEN 28932/haproxy
tcp 0 0 192.168.100.20:8775 0.0.0.0:* LISTEN 28932/haproxy
root@nodei19:~#

Tags:

Ritam Gangopadhyay (ritam) on 2017-11-23

tags:

added: nova vrouter
removed: vro

Ritam Gangopadhyay (ritam) on 2017-11-23

Changed in juniperopenstack:
assignee:	nobody → Kumar Harsh (hkumar)

Revision history for this message

Hari Prasad Killi (haripk) wrote on 2017-11-24:

2017-11-24 10:04:13.954 MetadataTrace: GET request for VM : 141.131.186.3 URL : controller/src/vnsw/agent/services/metadata_proxy.cc 185
2017-11-24 10:04:13.957 MetadataTrace: Metadata for VM : 141.131.186.3 Error : SSL connect error controller/src/vnsw/agent/services/metadata_proxy.cc 268

Sudheendra Rao (sudheendra-k) on 2017-11-27

Changed in juniperopenstack:
importance:	High → Critical

Hari Prasad Killi (haripk) on 2017-11-28

Changed in juniperopenstack:
importance:	Critical → High

Revision history for this message

Hari Prasad Killi (haripk) wrote on 2017-11-28:

Summary : Metadata proxy using SSL doesnt work for Ocata

tags:

added: releasenote

Kumar Harsh (hkumar) on 2017-11-28

Changed in juniperopenstack:
assignee:	Kumar Harsh (hkumar) → Ramprakash R (ramprakash)

Revision history for this message

Kumar Harsh (hkumar) wrote on 2017-11-28:

For ocata there is a misconfiguration in nova.conf:

nova_metadata_insecure = True

Correct should be :

nova_metadata_insecure = False

Despite this metadata ssl is not working in ocata ; Response is not received from nova-api to agent .

This works for Mitaka and newton and nothing is different from vrouter-agent side.

Revision history for this message

Jeba Paulaiyan (jebap) wrote on 2017-12-01:

Releasenotes:

Provisioning SSL for metadata fetch does not work if the Openstack SKU is Ocata.

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2018-01-04: [Review update] master

Review in progress for https://review.opencontrail.org/38665
Submitter: Ramprakash R (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2018-01-04: [Review update] R4.1

Review in progress for https://review.opencontrail.org/38666
Submitter: Ramprakash R (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2018-01-04: A change has been merged

Reviewed: https://review.opencontrail.org/38665
Committed: http://github.com/Juniper/contrail-ansible/commit/f47f0a865451ef1c40b7e57d092a99b579fb7430
Submitter: Zuul (<email address hidden>)
Branch: master

commit f47f0a865451ef1c40b7e57d092a99b579fb7430
Author: Ramprakash Ram Mohan <email address hidden>
Date: Wed Jan 3 16:39:36 2018 -0800

haproxy config changes to support metadata_ssl_enable

haproxy.cfg needs to be configured with "mode tcp" to support metadata SSL.
This bug will be applicable only when virtual IP is being used in the openstack
node. For the certificates to be generated for all the virtual IPs, this review
is also required for complete fix: https://review.opencontrail.org/#/c/38655/1

Change-Id: Ifcad1c2ba6352a750524c2ecaef1b644bfadbc29
Closes-bug: #1734110

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2018-01-04:

#10

Reviewed: https://review.opencontrail.org/38666
Committed: http://github.com/Juniper/contrail-ansible/commit/71a75f76b8c6ac0ab8328aec590ea34effc9f387
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit 71a75f76b8c6ac0ab8328aec590ea34effc9f387
Author: Ramprakash Ram Mohan <email address hidden>
Date: Wed Jan 3 16:39:36 2018 -0800

haproxy config changes to support metadata_ssl_enable

Change-Id: Ifcad1c2ba6352a750524c2ecaef1b644bfadbc29
Closes-bug: #1734110

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.