kolla-kubernetes: nova-api fails to start at boot time due to inappropriate validation of rfc3986

Bug #1629729 reported by Takashi Sogabe on 2016-10-03
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla
Undecided
Unassigned

Bug Description

In kolla-kubernetes environments, nova-api fails to start at boot time due to ConfigFileValueError as follows.

2016-09-26 09:06:54.912 6 CRITICAL nova [req-a3b39f84-355b-409f-b83b-4a10573d8322 - - - - -] ConfigFileValueError: Value for option url is not valid: invalid URI: 'http://neutron-server:9696'
2016-09-26 09:06:54.912 6 ERROR nova Traceback (most recent call last):
2016-09-26 09:06:54.912 6 ERROR nova File "/usr/bin/nova-api", line 10, in <module>
2016-09-26 09:06:54.912 6 ERROR nova sys.exit(main())
2016-09-26 09:06:54.912 6 ERROR nova File "/usr/lib/python2.7/site-packages/nova/cmd/api.py", line 73, in main
2016-09-26 09:06:54.912 6 ERROR nova launcher.wait()
2016-09-26 09:06:54.912 6 ERROR nova File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 568, in wait
2016-09-26 09:06:54.912 6 ERROR nova self.conf.log_opt_values(LOG, logging.DEBUG)
2016-09-26 09:06:54.912 6 ERROR nova File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2630, in log_opt_values
2016-09-26 09:06:54.912 6 ERROR nova _sanitize(opt, getattr(group_attr, opt_name)))
2016-09-26 09:06:54.912 6 ERROR nova File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 3061, in __getattr__
2016-09-26 09:06:54.912 6 ERROR nova return self._conf._get(name, self._group)
2016-09-26 09:06:54.912 6 ERROR nova File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2672, in _get
2016-09-26 09:06:54.912 6 ERROR nova value = self._do_get(name, group, namespace)
2016-09-26 09:06:54.912 6 ERROR nova File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2715, in _do_get
2016-09-26 09:06:54.912 6 ERROR nova % (opt.name, str(ve)))
2016-09-26 09:06:54.912 6 ERROR nova ConfigFileValueError: Value for option url is not valid: invalid URI: 'http://neutron-server:9696'
2016-09-26 09:06:54.912 6 ERROR nova

I dug into causes of the error above, and found that python-rfc3986 could not handle URI which has '-' characters in case of older versions of python-rfc3986 than 0.3.0.

https://pypi.python.org/pypi/rfc3986

Current version of RDO uses v0.2.0 of python-rfc3986.
I worked around by tweaking of docker/nova/nova-base/Dockerfile.j2 as follows:

----
{% if install_type == 'binary' and base_distro in ['centos', 'rhel'] %}
RUN pip install rfc3986 --upgrade
{% endif %}
----

Are there appropriate ways to correct the issue?

Ryan Hallisey (rthall14) wrote :

If there isn't a problem with 0.2.0 and RDO ships 0.2.0 we shouldn't have a problem. How did you end up with an older version? Where you using kolla tagged images like 2.0.0?

A solution for this could be to stop using '-' for DNS names and use '_' instead.

Changed in kolla:
status: New → Triaged
Takashi Sogabe (sogabe) wrote :

Currently, I use default configuration of koalla repos (3.0.0).

kolla/common/config.py:
DELOREAN = ("http://buildlogs.centos.org/centos/7/cloud/x86_64/"
            "rdo-trunk-master-tested/delorean.repo")

It seems that both newton and ocata releases of RDO use python-rfc3986-0.2.0-1.el7.noarch.rpm.
- http://buildlogs.centos.org/centos/7/cloud/x86_64/openstack-newton/common/
- http://buildlogs.centos.org/centos/7/cloud/x86_64/openstack-ocata/common/

Ryan Hallisey (rthall14) wrote :

You mentioned this occurs in older versions than python-rfc3986-0.2.0-1, but does this also occur in 0.2.0-1? I haven't hit this issue and been using the same repos. If the issue is only in old versions we should be ok, but can make note of it in the python dependencies.

Takashi Sogabe (sogabe) wrote :

Sorry for wrong information about version number of python-rfc3986.

python-rfc3986 could not handle URI which has '-' characters in case of older versions of python-rfc3986 than 0.3.0.

In release notes of python-rfc3986(https://pypi.python.org/pypi/rfc3986):
> 0.2.2 – 2015-05-27
>
> Update the regular name regular expression to accept all of the characters allowed in the RFC. Closes bug #11 (Thanks > Viktor Haag). Previously URIs similar to “http://http-bin.org” would be considered invalid.

I tested with yesterday's version of kolla-kubernetes, and nova-api did not installed in nova-api.

$ kubectl exec nova-api-3662253284-8fl4q -- rpm -qa | grep rfc3986
(n/a)

I will re-check with today's version of kolla-kubernetes.

description: updated
Takashi Sogabe (sogabe) wrote :

Amend the previous comment as follows:

I checked with yesterday's version of kolla-kubernetes, and 2.0.0 of python-rfc3986 was installed in nova-api.

[vagrant@localhost ~]$ kubectl exec -c main nova-api-3662253284-8fl4q -- rpm -qa | grep rfc
python-rfc3986-0.2.0-1.el7.noarch

[vagrant@localhost ~]$ kubectl exec -c main nova-api-3662253284-8fl4q cat /etc/nova/nova.conf | grep neutron
use_neutron = True
[neutron]
url = http://neutron-server:9696
username = neutron

Takashi Sogabe (sogabe) wrote :

I tested with latest version of kolla-kubernetes. The result is that the issue is resolved.

It seems rfc3986 lib is updated to non-issue version(0.3.1).

[vagrant@localhost ~]$ kubectl exec -c main nova-api-992251114-6jwuj -- rpm -qa | grep rfc
python2-rfc3986-0.3.1-1.el7.noarch

Changed in kolla:
milestone: none → ocata-3
Christian Berendt (berendt) wrote :

Closing as invalid, issue resolved.

Changed in kolla:
status: Triaged → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers