Ephemeral heat/rabbitmq fail to start on IPv6 undercloud

Bug #1983121 reported by Harald Jensås
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Harald Jensås

Bug Description

2022-07-29 15:30:04.031 113897 INFO tripleoclient.heat_launcher [-] Checking that message bus (rabbitmq) is upESC[00m
ERROR: invalid literal for int() with base 10: 'dead:beef:4::3'

Looks like a IPv6 bracket wrapping issue, since the actual address is "2620:dead:beef:4::3".

Testing patch:

diff --git a/tripleoclient/heat_launcher.py b/tripleoclient/heat_launcher.py
index 6e7d2c8c..453c45d3 100644
--- a/tripleoclient/heat_launcher.py
+++ b/tripleoclient/heat_launcher.py
@@ -43,6 +43,7 @@ from tripleoclient.constants import (DEFAULT_HEAT_CONTAINER,
                                      DEFAULT_TEMPLATES_DIR,
                                      EPHEMERAL_HEAT_POD_NAME)
 from tripleoclient.exceptions import HeatPodMessageQueueException
+from tripleoclient import utils as oooutils

 log = logging.getLogger(__name__)

@@ -706,7 +707,7 @@ class HeatPodLauncher(HeatContainerLauncher):
         return ('mysql+pymysql://'
                 'heat:heat@{}/heat?read_default_file='
                 '/etc/my.cnf.d/tripleo.cnf&read_default_group=tripleo'.format(
- self._get_ctlplane_vip()))
+ oooutils.bracket_ipv6(self._get_ctlplane_vip())))

     def _get_ctlplane_vip(self):
         return self._decode(subprocess.check_output(

Result is slightly different:
022-07-29 16:20:23.207 172061 INFO tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Using ephemeral heat for stack operation
2022-07-29 16:20:24.165 172061 INFO tripleoclient.heat_launcher [-] Skipping container image pull.
2022-07-29 16:20:24.173 172061 INFO tripleoclient.heat_launcher [-] Checking that database is up
2022-07-29 16:20:24.577 172061 INFO tripleoclient.heat_launcher [-] Checking that message bus (rabbitmq) is up
2022-07-29 16:20:28.131 2 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 10 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '2620:dead:beef:4::3'
 ([Errno 101] Network is unreachable)")
2022-07-29 16:20:38.145 2 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 9 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '2620:dead:beef:4::3'
([Errno 101] Network is unreachable)")
2022-07-29 16:20:48.148 2 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 8 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '2620:dead:beef:4::3'
([Errno 101] Network is unreachable)")
2022-07-29 16:20:58.159 2 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 7 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '2620:dead:beef:4::3'
([Errno 101] Network is unreachable)")
2022-07-29 16:21:08.171 2 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 6 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '2620:dead:beef:4::3'
([Errno 101] Network is unreachable)")
2022-07-29 16:21:18.182 2 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 5 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '2620:dead:beef:4::3'
([Errno 101] Network is unreachable)")
2022-07-29 16:21:28.187 2 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 4 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '2620:dead:beef:4::3'
([Errno 101] Network is unreachable)")
2022-07-29 16:21:38.199 2 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 3 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '2620:dead:beef:4::3'
([Errno 101] Network is unreachable)")
2022-07-29 16:21:48.202 2 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 2 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '2620:dead:beef:4::3'
([Errno 101] Network is unreachable)")
2022-07-29 16:21:58.213 2 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 1 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '2620:dead:beef:4::3'
([Errno 101] Network is unreachable)")
ERROR: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '2620:dead:beef:4::3' ([Errno 101] Network is unreachable)")
(Background on this error at: http://sqlalche.me/e/13/e3q8)
2022-07-29 16:22:08.616 172061 WARNING tripleoclient.utils.safe_write [-] The output file /home/stack/overcloud-deploy/overcloud/overcloud-deployment_status.yaml will be overriden: subprocess.CalledProcessError: Command '['sudo', 'podma$', 'run', '--rm', '--user', 'heat', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher/heat.conf:/etc/heat/heat.conf:z', '--volume', '/home/stack/overcloud-deploy/overcloud/heat-launcher:/home/stack/overcloud-deploy/overc$oud/heat-launcher:z', 'localhost/tripleo/openstack-heat-api:ephemeral', 'heat-manage', 'db_sync']' returned non-zero exit status 1.

description: updated
Revision history for this message
Harald Jensås (harald-jensas) wrote :

We probably need to enable IPv6 in podman networking.

[root@undercloud ~]# podman network ls
NETWORK ID NAME DRIVER
2f259bab93aa podman bridge
[root@undercloud ~]# podman network inspect 2f259bab93aa
[
     {
          "name": "podman",
          "id": "2f259bab93aaaaa2542ba43ef33eb990d0999ee1b9924b557b7be53c0b7a1bb9",
          "driver": "bridge",
          "network_interface": "podman0",
          "created": "2022-07-29T16:37:09.084636812Z",
          "subnets": [
               {
                    "subnet": "10.88.0.0/16",
                    "gateway": "10.88.0.1"
               }
          ],
          "ipv6_enabled": false,
          "internal": false,
          "dns_enabled": false,
          "ipam_options": {
               "driver": "host-local"
          }
     }
]

Revision history for this message
Harald Jensås (harald-jensas) wrote :

With this network configuration ephemeral heat pods start - and overcloud heat stack create runs:

[root@undercloud ~]# podman network inspect podman
[
     {
          "name": "podman",
          "id": "2f259bab93aaaaa2542ba43ef33eb990d0999ee1b9924b557b7be53c0b7a1bb9",
          "driver": "bridge",
          "network_interface": "podman0",
          "created": "2022-07-29T16:42:29.309477427Z",
          "subnets": [
               {
                    "subnet": "10.88.0.0/16",
                    "gateway": "10.88.0.1"
               },
               {
                    "subnet": "fd11:2222:3333::/64",
                    "gateway": "fd11:2222:3333::1"
               }
          ],
          "ipv6_enabled": true,
          "internal": false,
          "dns_enabled": false,
          "ipam_options": {
               "driver": "host-local"
          }
     }
]

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to python-tripleoclient (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/python-tripleoclient/+/851597

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-ansible (master)
Changed in tripleo:
status: Triaged → In Progress
Changed in tripleo:
assignee: nobody → Harald Jensås (harald-jensas)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to python-tripleoclient (master)

Reviewed: https://review.opendev.org/c/openstack/python-tripleoclient/+/851597
Committed: https://opendev.org/openstack/python-tripleoclient/commit/ec8e13423449841ec701086d50e7414afc7792bf
Submitter: "Zuul (22348)"
Branch: master

commit ec8e13423449841ec701086d50e7414afc7792bf
Author: Harald Jensås <email address hidden>
Date: Fri Jul 29 18:44:41 2022 +0200

    Heat Launcher - bracket IPv6 in db connection URI

    In the database connection URI ensure IPv6 address is wrapped in
    brackets. Without this a port number is assumed and error is raised:
      ERROR: invalid literal for int() with base 10: 'dead:beef:4::3'

    Related-Bug: #1983121
    Change-Id: I37fda2aa161120369d0e3734b9485c27cb59846c

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to python-tripleoclient (stable/wallaby)

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/python-tripleoclient/+/851715

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to python-tripleoclient (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/python-tripleoclient/+/851715
Committed: https://opendev.org/openstack/python-tripleoclient/commit/edaeefac655ae87c7cee3bcdad490200b7151e68
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit edaeefac655ae87c7cee3bcdad490200b7151e68
Author: Harald Jensås <email address hidden>
Date: Fri Jul 29 18:44:41 2022 +0200

    Heat Launcher - bracket IPv6 in db connection URI

    In the database connection URI ensure IPv6 address is wrapped in
    brackets. Without this a port number is assumed and error is raised:
      ERROR: invalid literal for int() with base 10: 'dead:beef:4::3'

    Related-Bug: #1983121
    Change-Id: I37fda2aa161120369d0e3734b9485c27cb59846c
    (cherry picked from commit ec8e13423449841ec701086d50e7414afc7792bf)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-ansible (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/851608
Committed: https://opendev.org/openstack/tripleo-ansible/commit/1a87dad727c2e9db4620cf7987762971ea176929
Submitter: "Zuul (22348)"
Branch: master

commit 1a87dad727c2e9db4620cf7987762971ea176929
Author: Harald Jensås <email address hidden>
Date: Fri Jul 29 20:23:03 2022 +0200

    Configure podman netavark network - enable IPv6

    In RHEL9 the default container networking used is "netavark".
    With this change the default "podman" network is configured to
    enable IPv6 networking for containers.

    The default IPv4 IP address used is updated the match the ones used
    in tripleo with "cni" networking.

    Also, when "netavark" is the backend skip the tasks that do cni related
    configuration.

    Closes-Bug: #1983121
    Depends-On: https://review.opendev.org/c/openstack/python-tripleoclient/+/851597
    Change-Id: I90efb1639d7ffd899ab73446620354403c1ca536

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-ansible (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/851877

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-ansible (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/851877
Committed: https://opendev.org/openstack/tripleo-ansible/commit/93e8f8631ca4ed3b35814a9805578566813ac12f
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 93e8f8631ca4ed3b35814a9805578566813ac12f
Author: Harald Jensås <email address hidden>
Date: Fri Jul 29 20:23:03 2022 +0200

    Configure podman netavark network - enable IPv6

    In RHEL9 the default container networking used is "netavark".
    With this change the default "podman" network is configured to
    enable IPv6 networking for containers.

    The default IPv4 IP address used is updated the match the ones used
    in tripleo with "cni" networking.

    Also, when "netavark" is the backend skip the tasks that do cni related
    configuration.

    Closes-Bug: #1983121
    Depends-On: https://review.opendev.org/c/openstack/python-tripleoclient/+/851597
    Change-Id: I90efb1639d7ffd899ab73446620354403c1ca536
    (cherry picked from commit 1a87dad727c2e9db4620cf7987762971ea176929)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-ansible 5.0.0

This issue was fixed in the openstack/tripleo-ansible 5.0.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.