R4.0.0.0-3034 - Api server fails to come up during provisioning with SSL enabled

Bug #1661084 reported by Jeba Paulaiyan
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
Trunk
Fix Committed
Critical
Sundaresan Rajangam

Bug Description

4.0.0.0-3034-Kilo

Provisioning log:

2017-01-31 09:34:28:382779: Requested: setup-vnc-compute --self_ip 192.168.192.2 --cfgm_ip 192.168.192.251 --cfgm_user root --cfgm_passwd c0ntrail123 --ncontrols 3 --amqp_server_ip 192.168.192.251 --service_token 760cca0c0f3a1bce7459 --orchestrator openstack --hypervisor libvirt --non_mgmt_ip 192.168.192.2 --non_mgmt_gw 192.168.192.254 --keystone_ip 192.168.192.251 --keystone_version v2.0 --openstack_mgmt_ip 10.204.216.64 --keystone_auth_protocol https --keystone_auth_port 35357 --quantum_service_protocol https --keystone_admin_user admin --keystone_admin_password contrail123 --nova_password contrail123 --neutron_password contrail123 --service_tenant_name service --region_name RegionOne --internal_vip 192.168.192.251 --contrail_internal_vip 192.168.192.251 --mgmt_self_ip 10.204.217.114
2017-01-31 09:34:28:382779: Executed: sudo -S -p 'sudo password:' /bin/bash -l -c "cd /opt/contrail/bin && setup-vnc-compute --self_ip 192.168.192.2 --cfgm_ip 192.168.192.251 --cfgm_user root --cfgm_passwd c0ntrail123 --ncontrols 3 --amqp_server_ip 192.168.192.251 --service_token 760cca0c0f3a1bce7459 --orchestrator openstack --hypervisor libvirt --non_mgmt_ip 192.168.192.2 --non_mgmt_gw 192.168.192.254 --keystone_ip 192.168.192.251 --keystone_version v2.0 --openstack_mgmt_ip 10.204.216.64 --keystone_auth_protocol https --keystone_auth_port 35357 --quantum_service_protocol https --keystone_admin_user admin --keystone_admin_password contrail123 --nova_password contrail123 --neutron_password contrail123 --service_tenant_name service --region_name RegionOne --internal_vip 192.168.192.251 --contrail_internal_vip 192.168.192.251 --mgmt_self_ip 10.204.217.114"
2017-01-31 09:34:28:382779:
2017-01-31 09:34:28:382839: Aborting.
2017-01-31 09:34:28:382839: 2017-01-31 09:34:28:382622: [root@10.204.217.113] out: Traceback (most recent call last):
2017-01-31 09:34:28:581403: [root@10.204.217.113] out: File "/opt/contrail/utils/provision_vrouter.py", line 188, in <module>
2017-01-31 09:34:28:581470: [root@10.204.217.113] out: main()
2017-01-31 09:34:28:581535: [root@10.204.217.113] out: File "/opt/contrail/utils/provision_vrouter.py", line 184, in main
2017-01-31 09:34:28:581588: [root@10.204.217.113] out: VrouterProvisioner(args_str)
2017-01-31 09:34:28:581634: [root@10.204.217.113] out: File "/opt/contrail/utils/provision_vrouter.py", line 43, in __init__
2017-01-31 09:34:28:581688: [root@10.204.217.113] out: fq_name=['default-global-system-config'])
2017-01-31 09:34:28:581736: [root@10.204.217.113] out: File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 40, in wrapper
2017-01-31 09:34:28:581780: [root@10.204.217.113] out: retry_on_error=False)
2017-01-31 09:34:28:581849: [root@10.204.217.113] out: File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 842, in _request
2017-01-31 09:34:28:581894: [root@10.204.217.113] out: raise ServiceUnavailableError('Service Unavailable Timeout %d' % status)
2017-01-31 09:34:28:581937: [root@10.204.217.113] out: cfgm_common.exceptions.ServiceUnavailableError: Service unavailable time out due to: Service Unavailable Timeout 503
2017-01-31 09:34:28:582021: [root@10.204.217.113] out:
2017-01-31 09:34:28:583828: [root@10.204.217.113] out: Fatal error: local() encountered an error (return code 1) while executing 'python /opt/contrail/utils/provision_vrouter.py --host_name nodei1 --host_ip 192.168.192.1 --api_server_ip 192.168.192.251 --oper add --admin_user admin --admin_password contrail123 --admin_tenant_name admin --openstack_ip 192.168.192.251 --api_server_use_ssl True'
2017-01-31 09:34:28:583884: [root@10.204.217.113] out:
2017-01-31 09:34:28:583941: [root@10.204.217.113] out: Aborting.
2017-01-31 09:34:28:584003: [root@10.204.217.113] out:
2017-01-31 09:34:28:599565:

2017-01-31 09:34:28:610751: Fatal error: sudo() received nonzero return code 1 while executing!

Apiserver log:

global os = <module 'os' from '/usr/lib/python2.7/os.pyc'>
os.stat = <built-in function stat>
filename = '/etc/contrail/ssl/certs/server.pem'
).st_mtime undefined
<type 'exceptions.OSError'>: [Errno 2] No such file or directory: '/etc/contrail/ssl/certs/server.pem'
    __class__ = <type 'exceptions.OSError'>
    __delattr__ = <method-wrapper '__delattr__' of exceptions.OSError object>
    __dict__ = {}
    __doc__ = 'OS system call failed.'
    __format__ = <built-in method __format__ of exceptions.OSError object>
    __getattribute__ = <method-wrapper '__getattribute__' of exceptions.OSError object>
    __getitem__ = <method-wrapper '__getitem__' of exceptions.OSError object>
    __getslice__ = <method-wrapper '__getslice__' of exceptions.OSError object>
    __hash__ = <method-wrapper '__hash__' of exceptions.OSError object>
    __init__ = <method-wrapper '__init__' of exceptions.OSError object>
    __new__ = <built-in method __new__ of type object>
    __reduce__ = <built-in method __reduce__ of exceptions.OSError object>
    __reduce_ex__ = <built-in method __reduce_ex__ of exceptions.OSError object>
    __repr__ = <method-wrapper '__repr__' of exceptions.OSError object>
    __setattr__ = <method-wrapper '__setattr__' of exceptions.OSError object>
    __setstate__ = <built-in method __setstate__ of exceptions.OSError object>
    __sizeof__ = <built-in method __sizeof__ of exceptions.OSError object>
    __str__ = <method-wrapper '__str__' of exceptions.OSError object>
    __subclasshook__ = <built-in method __subclasshook__ of type object>
    __unicode__ = <built-in method __unicode__ of exceptions.OSError object>
    args = (2, 'No such file or directory')
    errno = 2
    filename = '/etc/contrail/ssl/certs/server.pem'
    message = ''
    strerror = 'No such file or directory'

The above is a description of an error in a Python program. Here is
the original traceback:

Traceback (most recent call last):
  File "/usr/bin/contrail-api", line 9, in <module>
    load_entry_point('vnc-cfg-api-server==0.1dev', 'console_scripts', 'contrail-api')()
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 3700, in server_main
    main(args_str, VncApiServer(args_str))
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 1545, in __init__
    auth_svc = vnc_auth_keystone.AuthServiceKeystone(self, self._args)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_auth_keystone.py", line 146, in __init__
    _kscertbundle=cfgmutils.getCertKeyCaBundle(_DEFAULT_KS_CERT_BUNDLE,certs)
  File "/usr/lib/python2.7/dist-packages/cfgm_common/utils.py", line 146, in getCertKeyCaBundle
    if os.path.getmtime(cert) > bundle_mod_time:
  File "/usr/lib/python2.7/genericpath.py", line 54, in getmtime
    return os.stat(filename).st_mtime
OSError: [Errno 2] No such file or directory: '/etc/contrail/ssl/certs/server.pem'

Jeba Paulaiyan (jebap)
tags: added: blocker provisioning sanity
Revision history for this message
Ignatious Johnson Christopher (ijohnson-x) wrote :

ksopts already has certfile, keyfile and cafile options,

https://github.com/Juniper/contrail-controller/commit/c0b6ba8b2befd9c549672620096d56a9b2c4e14a#diff-21de1389f387230b32a2130c4a8dccbdR786

Recently added sandeshopts is also using the same option name and ends up
overwrriting the ksopts

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/28376
Submitter: Sundaresan Rajangam (<email address hidden>)

information type: Proprietary → Public
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/28376
Committed: http://github.org/Juniper/contrail-controller/commit/1da9155cdc9ccf482f24346e2ed356db4d9a7637
Submitter: Zuul (<email address hidden>)
Branch: master

commit 1da9155cdc9ccf482f24346e2ed356db4d9a7637
Author: Sundaresan Rajangam <email address hidden>
Date: Thu Feb 2 11:10:24 2017 -0800

Rename sandesh ssl options

Naming convention for ssl options (keyfile, certfile, ca_cert) under [SANDESH]
conflicts with [SECURITY] options in contrail-api. Hence, prepended the
ssl options under [SANDESH] with sandesh_

Change-Id: I23f71d103270209eae5951ef044832bacff6e667
Closes-Bug: #1661084

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.