Adding TLS Connection String To Database Connection Causes malformed URL in cell_mappings DB

[Kyle Dean]

Description
===========

Fresh Install

Ubuntu 16.04

I followed Ubuntu install guide up to Install Nova on Controller, I configured Nova.conf and specified api_database and database connection string with SSL/TLS Parameters, then ran the DB commands. The following are successful.

su -s /bin/sh -c "nova-manage api_db sync" nova
su -s /bin/sh -c "nova-manage cell_v2 map_cell0" nova
su -s /bin/sh -c "nova-manage cell_v2 create_cell --name=cell1 --verbose" nova

The following command fails with error

su -s /bin/sh -c "nova-manage db sync" nova

ERROR: could not access cell mapping database - has api db been created?

My connection string is as follows

connection = mysql+pymysql://nova:my_password@my_IP/nova_api?charset=utf8&ssl_ca=/etc/nova/tls/mysql/ca-cert.pem&ssl_cert=/etc/nova/tls/mysql/server-cert.pem&ssl_key=/etc/nova/tls/mysql/server-key.pem

The following connection string is what appears in the database table

mysql+pymysql://nova:4f35d884c9960df0ac80@10.30.0.2/nova?charset=utf8&ssl_ca=/etc/nova/tls/mysql/ca-cert.pem&ssl_cert=/etc/nova/tls/mysql/server-cert.pem&ssl_key=/etc/nova/tls/mysql/server-key.pemnova_cell0

<mriedem> on #openstack-nova suggests that this is a string substitution bug.

Steps to reproduce
===============

1. Install Ubuntu 16.04
2. Install Keystone
3. Configure Database connection string with TLS/SSL Parameters
4. Install Glance
5. Configure Database connection string with TLS/SSL Parameters
6. Install Nova
7. Configure Database connection string with TLS/SSL Parameters
8. Run the DB sync commands in order from the Docs
Failure at > su -s /bin/sh -c "nova-manage db sync" nova

Expected result
===============
Nova database Populates without error

Actual result
=============
Nova database does not populate and following error thrown

ERROR: could not access cell mapping database - has api db been created?

Environment
===========
Ocata

Ubuntu cloud Archive

nova-api 2:15.0.5-0ubuntu1~cloud0 all OpenStack Compute - API frontend
nova-common 2:15.0.5-0ubuntu1~cloud0 all OpenStack Compute - common files
nova-conductor 2:15.0.5-0ubuntu1~cloud0 all OpenStack Compute - conductor service
nova-consoleauth 2:15.0.5-0ubuntu1~cloud0 all OpenStack Compute - Console Authenticator
nova-novncproxy 2:15.0.5-0ubuntu1~cloud0 all OpenStack Compute - NoVNC proxy
nova-placement-api 2:15.0.5-0ubuntu1~cloud0 all OpenStack Compute - placement API frontend
nova-scheduler 2:15.0.5-0ubuntu1~cloud0 all OpenStack Compute - virtual machine scheduler
python-nova 2:15.0.5-0ubuntu1~cloud0 all OpenStack Compute Python libraries
python-novaclient 2:7.1.0-0ubuntu1~cloud0 all client library for OpenStack Compute API - Python 2.7

OpenVSwitch

KVM

Logs & Configs
==============
[DEFAULT]
#debug = true
#####################################
#dhcpbridge_flagfile=/etc/nova/nova.conf
#dhcpbridge=/usr/bin/nova-dhcpbridge
#force_dhcp_release=true
#####################################
my_ip = my_ip
state_path = /var/lib/nova
####
enabled_apis = osapi_compute,metadata
enabled_ssl_apis = osapi_compute,metadata
####
osapi_compute_listen = my_ip
osapi_compute_listen_port = 8774
####
metadata_host = $my_ip
metadata_listen = my_ip
metadata_listen_port = 8775
metadata_port = 8775
#####
rootwrap_config = /etc/nova/rootwrap.conf
api_paste_config = /etc/nova/api-paste.ini
log_dir = /var/log/nova
use_neutron = True
firewall_driver = nova.virt.firewall.NoopFirewallDriver
######
cert = /etc/nova/tls/server-cert.pem
key = /etc/nova/tls/server-key.pem
ssl_only = True

transport_url = rabbit://controller01:my_password@my_ip

compute_api_class=nova.compute.cells_api.ComputeCellsAPI

[api]
auth_strategy = keystone

[api_database]
connection = mysql+pymysql://nova:my_password@my_ip/nova_api?charset=utf8&ssl_ca=/etc/nova/tls/mysql/ca-cert.pem&ssl_cert=/etc/nova/tls/mysql/server-cert.pem&ssl_key=/etc/nova/tls/mysql/server-key.pem

#connection_debug = 1

[barbican]
[cache]
[cells]
enable=True
cell_type = api
[cinder]
[cloudpipe]
[conductor]
[console]
[consoleauth]

[cors]
[cors.subdomain]
[crypto]
##
cert_file = /etc/nova/tls/server-cert.pem
key_file = /etc/nova/tls/server-key.pem
ca_file = /etc/nova/tls/ca-cert.pem
##
[database]
connection = mysql+pymysql://nova:my_password@my_ip/nova?charset=utf8&ssl_ca=/etc/nova/tls/mysql/ca-cert.pem&ssl_cert=/etc/nova/tls/mysql/server-cert.pem&ssl_key=/etc/nova/tls/mysql/server-key.pem

#connection_debug = 1

[ephemeral_storage_encryption]
[filter_scheduler]
[glance]
api_servers = https://my_ip:9292
api_insecure = False
[guestfs]
[healthcheck]
[hyperv]
[image_file_url]
[ironic]
[key_manager]
[keystone_authtoken]
auth_uri = https://my_ip:5000
auth_url = https://my_ip:35357
memcached_servers = my_ip:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = nova
password = my_password
###
certfile = /etc/nova/tls/server-cert.pem
keyfile = /etc/nova/tls/server-key.pem
cafile = /etc/nova/tls/ca-cert.pem
service_token_roles_required = True
###
[libvirt]
[matchmaker_redis]
[metrics]
[mks]
[neutron]
[notifications]
[osapi_v21]
[oslo_concurrency]
lock_path=/var/lib/nova/tmp
[oslo_messaging_amqp]
[oslo_messaging_kafka]
[oslo_messaging_notifications]
[oslo_messaging_rabbit]
rabbit_use_ssl = True
kombu_ssl_keyfile = /etc/nova/tls/server-key.pem
kombu_ssl_certfile = /etc/nova/tls/server-cert.pem
kombu_ssl_ca_certs = /etc/nova/tls/ca-cert.pem
kombu_ssl_version = TLSv1_2
rpc_reply_retry_attempts = 10
rpc_retry_delay = 10.0
socket_timeout = 0.25
tcp_user_timeout = 10.0
[oslo_messaging_zmq]
[oslo_middleware]
[oslo_policy]
[pci]
[placement]
os_region_name = RegionOne
project_domain_name = Default
project_name = service
user_domain_name = Default
username = placement
password = my_password
auth_url = https://my_IP:35357/v3
auth_type = password
certfile = /etc/nova/tls/server-cert.pem
keyfile = /etc/nova/tls/server-key.pem
cafile = /etc/nova/tls/ca-cert.pem
insecure = False
[quota]
[rdp]
[remote_debug]
[scheduler]
[serial_console]
[service_user]
[spice]
[ssl]
cert_file = /etc/nova/tls/server-cert.pem
key_file = /etc/nova/tls/server-key.pem
ca_file = /etc/nova/tls/ca-cert.pem
ciphers = AES256-GCM-SHA384
version = TLSv1_2
[trusted_computing]
[upgrade_levels]
[vendordata_dynamic_auth]
[vmware]
[vnc]
enabled = True
vncserver_listen = 0.0.0.0
vncserver_proxyclient_address = $my_ip
novncproxy_base_url = https://my_IP:6080/vnc_auto.html

[workarounds]
[wsgi]
api_paste_config=/etc/nova/api-paste.ini
secure_proxy_ssl_header = https
ssl_key_file = /etc/nova/tls/server-key.pem
ssl_cert_file = /etc/nova/tls/server-cert.pem
ssl_ca_file = /etc/nova/tls/ca-cert.pem

[xenserver]
[xvp]

[Matt Riedemann (mriedem)]

As noted in IRC, it sounds like another variant of bug 1673613 which was fixed in Ocata 15.0.5:

https://docs.openstack.org/releasenotes/nova/ocata.html#id1

Which is what's being used here, so probably another case we need to handle in this code:

https://github.com/openstack/nova/blob/321f426c30f8f4f6e3dcafbbc7e5d87560b58c82/nova/cmd/manage.py#L1190-L1204

And that's parsed from the nova (cell1) database connection string, which from above is this:

"mysql+pymysql://nova:my_password@my_ip/nova?charset=utf8&ssl_ca=/etc/nova/tls/mysql/ca-cert.pem&ssl_cert=/etc/nova/tls/mysql/server-cert.pem&ssl_key=/etc/nova/tls/mysql/server-key.pem"

[Matt Riedemann (mriedem)]

The workaround for now is specify the --database_connection option with the 'nova-manage cell_v2 map_cell0' command to avoid the command messing up the URL stored for the cell0 mapping in the nova_api.cell_mappings table.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/486660
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=05f3d9d39b7b95fac343ba431855482a5d96584a
Submitter: Jenkins
Branch: master

commit 05f3d9d39b7b95fac343ba431855482a5d96584a
Author: Matt Riedemann <email address hidden>
Date: Mon Jul 24 11:30:35 2017 -0400

    Provide hints when nova-manage db sync fails to sync cell0

    Lots of people get tripped up on the error message when
    syncing cell0 fails and the question asked is confusing and
    possibly misleading, so this change includes several questions
    for troubleshooting and also dumps the actual error message.

    Related-Bug: #1706118

    Change-Id: I865f76705f10493152af50b9842c6fedc563fea4

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/501745

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/501746

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/501745
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=423c7bbdb99e14fd49681dd58edd348060f25005
Submitter: Jenkins
Branch: stable/pike

commit 423c7bbdb99e14fd49681dd58edd348060f25005
Author: Matt Riedemann <email address hidden>
Date: Mon Jul 24 11:30:35 2017 -0400

    Provide hints when nova-manage db sync fails to sync cell0

    Lots of people get tripped up on the error message when
    syncing cell0 fails and the question asked is confusing and
    possibly misleading, so this change includes several questions
    for troubleshooting and also dumps the actual error message.

    Related-Bug: #1706118

    Change-Id: I865f76705f10493152af50b9842c6fedc563fea4
    (cherry picked from commit 05f3d9d39b7b95fac343ba431855482a5d96584a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/501746
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b55ca354463a387d79e265701dd78cf3a51e457f
Submitter: Jenkins
Branch: stable/ocata

commit b55ca354463a387d79e265701dd78cf3a51e457f
Author: Matt Riedemann <email address hidden>
Date: Mon Jul 24 11:30:35 2017 -0400

    Provide hints when nova-manage db sync fails to sync cell0

    Lots of people get tripped up on the error message when
    syncing cell0 fails and the question asked is confusing and
    possibly misleading, so this change includes several questions
    for troubleshooting and also dumps the actual error message.

    Related-Bug: #1706118

    Change-Id: I865f76705f10493152af50b9842c6fedc563fea4
    (cherry picked from commit 05f3d9d39b7b95fac343ba431855482a5d96584a)