Using composable network or custom named network for MysqlNetwork results in deployment failure

Bug #1736254 reported by bugzy
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
tripleo
Incomplete
Medium
Unassigned

Bug Description

Description
===========
Setting MysqlNetwork: internal_api2 or any other name that does not match the default networks (internal_api, storage, storage_mgmt, tenant, management, ctrlplane, and external) cause deployment to fail at overcloud.AllNodesDeploySteps.ControllerDeployment_Step1

How reproducible
================
Always; this problem occurs not only with composable/custom networks but also when a network is renamed. Since reproducing the problem is much simpler with renamed networks, the steps to reproduce below use renamed networks instead of composable/custom networks.

Steps to reproduce
==================
In network-environment.yaml
~~~
parameter_defaults:
(...)
  InternalApiNetName: 'internal_api2'
  InternalApiSubnetName: 'internal_api2_subnet'
  InternalApiNetValueSpecs:
    'provider:physical_network': 'internal_api2'
    'provider:network_type': 'flat'

  ServiceNetMap:
    MysqlNetwork: internal_api2
(...)
~~~

Expected result
===============
Successful deployment

Actual result:
==============
openstack stack failures list overcloud2 --long
(...)
"Warning: ModuleLoader: module 'rabbitmq' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
            "Warning: ModuleLoader: module 'redis' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
            "Warning: ModuleLoader: module 'ssh' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
            "Warning: ModuleLoader: module 'timezone' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
            "Warning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.",
            "Error: Execution of '/usr/bin/mysql_install_db --defaults-extra-file=/etc/my.cnf.d/galera.cnf --basedir=/usr --datadir=/var/lib/mysql --user=mysql' returned 1: Installing MariaDB/MySQL system tables in '/var/lib/mysql' ...",
            "2017-12-04 7:33:32 140627219650752 [ERROR] /usr/libexec/mysqld: option '--bind-address' requires an argument",
            "2017-12-04 7:33:32 140627219650752 [ERROR] Aborting",
            "",
            "Installation of system tables failed! Examine the logs in",
            "/var/lib/mysql for more information.",
            "The problem could be conflicting information in an external",
            "my.cnf files. You can ignore these by doing:",
            " shell> /usr/scripts/scripts/mysql_install_db --defaults-file=~/.my.cnf",
            "You can also try to start the mysqld daemon with:",
            " shell> /usr/libexec/mysqld --skip-grant --general-log &",
            "and use the command line tool /usr/bin/mysql",
            "to connect to the mysql database and look at the grant tables:",
            " shell> /usr/bin/mysql -u root mysql",
            " mysql> show tables;",
            "Try 'mysqld --help' if you have problems with paths. Using",
            "--general-log gives you a log in /var/lib/mysql that may be helpful.",
            "The latest information about mysql_install_db is available at",
            "https://mariadb.com/kb/en/installing-system-tables-mysql_install_db",
            "MariaDB is hosted on launchpad; You can find the latest source and",
            "email lists at http://launchpad.net/maria",
            "Please check all of the above before submitting a bug report",
            "at http://mariadb.org/jira",
            "Error: /Stage[main]/Mysql::Server::Installdb/Mysql_datadir[/var/lib/mysql]/ensure: change from absent to present failed: Execution of '/usr/bin/mysql_install_db --defaults-extra-file=/etc/my.cnf.d/galera.cnf --basedir=/usr --datad
ir=/var/lib/mysql --user=mysql' returned 1: Installing MariaDB/MySQL system tables in '/var/lib/mysql' ...",
            "Warning: /Stage[main]/Mysql::Server::Installdb/File[/var/log/mariadb/mariadb.log]: Skipping because of failed dependencies",
            "Warning: /Stage[main]/Mysql::Server::Root_password/Exec[remove install pass]: Skipping because of failed dependencies",
            "Warning: /Stage[main]/Mysql::Server/Anchor[mysql::server::end]: Skipping because of failed dependencies",

(...)

Additional info
===============
1. This is a variant of Bug #1697722; however, the fix released for that bug only resolves the problem for specifically named networks (i.e. internal_api, storage, storage_mgmt, tenant, management, ctrlplane, and external). Any networks (even the same networks renamed) that do not have a name that matches any of the names of the default networks will trigger this bug.

2. Description of the resolution of Bug #1697722 says:

commit cd7b49795daa36f95ccf966a134af6dd696ab67b
Author: Alex Schultz <email address hidden>
Date: Tue Jun 13 09:39:11 2017 -0600

    Add fqdn_external

    In newton, we used to construct the fqdn_$NETWORK in puppet-tripleo for
    external, internal_api, storage, storage_mgmt, tenant, management, and
    ctrlplane. When this was moved into THT, we accidently dropped external
    which leads to deployment failures if a service is moved to the external
    network and the configuration consumes the fqdn_external hiera key.
    Specifically this is reproduced if the MysqlNetwork is switch to to
    exernal, then the deployment fails because the bind address which is set
    to use fqdn_external is blank.

    Change-Id: I01ad0c14cb3dc38aad7528345c928b86628433c1
    Closes-Bug: #1697722
    (cherry picked from commit 426de202880c890360bd446907aca44ca1e73a03)

Changed in tripleo:
milestone: none → queens-3
importance: Undecided → Medium
Changed in tripleo:
status: New → Triaged
Revision history for this message
bugzy (bugzylittle) wrote :

It seems that this issue may go a bit beyond the problem in Bug #1697722. A workaround to Bug #1697722 (https://bugzilla.redhat.com/show_bug.cgi?id=1460460#c9) and the solution in the patch set at https://review.openstack.org/#/c/474196/ is to add "fqdn_<lowercase_network_name>: {get_attr: [NetHostMap, value, <lowercase_network_name>, fqdn]}" to the controller-role.yaml in /usr/share/openstack-tripleo-heat-templates/puppet/.

In this case, that meant adding "fqdn_internal_api2: {get_attr: [NetHostMap, value, internal_api2, fqdn]}" to relevant location in controller-role.yaml template file. However, that solution did not work here.

Changed in tripleo:
milestone: queens-3 → queens-rc1
Changed in tripleo:
milestone: queens-rc1 → rocky-1
Changed in tripleo:
milestone: rocky-1 → rocky-2
Changed in tripleo:
milestone: rocky-2 → rocky-3
Changed in tripleo:
milestone: rocky-3 → rocky-rc1
Changed in tripleo:
milestone: rocky-rc1 → stein-1
Changed in tripleo:
milestone: stein-1 → stein-2
Changed in tripleo:
milestone: stein-2 → stein-3
Revision history for this message
Juan Antonio Osorio Robles (juan-osorio-robles) wrote :

Is this still an issue?

Changed in tripleo:
milestone: stein-3 → train-1
Changed in tripleo:
milestone: train-1 → train-2
Changed in tripleo:
milestone: train-2 → train-3
Changed in tripleo:
milestone: train-3 → ussuri-1
Changed in tripleo:
milestone: ussuri-1 → ussuri-2
wes hayutin (weshayutin)
Changed in tripleo:
milestone: ussuri-2 → ussuri-3
wes hayutin (weshayutin)
Changed in tripleo:
milestone: ussuri-3 → ussuri-rc3
wes hayutin (weshayutin)
Changed in tripleo:
status: Triaged → Incomplete
wes hayutin (weshayutin)
Changed in tripleo:
milestone: ussuri-rc3 → victoria-1
Changed in tripleo:
milestone: victoria-1 → victoria-3
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.