tripleo-ansible-inventory error's with - Error generating inventory for overcloud: 'ctlplane_cidr'

Bug #1928926 reported by Harald Jensås
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Critical
Harald Jensås

Bug Description

2021-05-19 08:17:35.093420 | fa163e74-25e0-478b-41a0-00000000000b | FATAL | Generate ansible inventory | localhost | error={"changed": false, "error": "'ctlplane_cidr'", "msg": "Error generating inventory for overcloud: 'ctlplane_cidr'", "success": false}

This is likely introduced with [1]
  https://review.opendev.org/c/openstack/tripleo-common/+/790527

It will try to use neutron as a source for the inventory when overcloud is deployed without 'network-v2'. The ctlplane port's will not have the required tags, and simply continuing in that case is wrong.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (master)
Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-common/+/792095
Committed: https://opendev.org/openstack/tripleo-common/commit/d3286377132ee6b0689a39e52858c07954711d13
Submitter: "Zuul (22348)"
Branch: master

commit d3286377132ee6b0689a39e52858c07954711d13
Author: Harald Jensås <email address hidden>
Date: Wed May 19 11:25:51 2021 +0200

    Partial-Revert:Skip port if missing mandatory tag

    We need to be more specific when to allow ports without
    mandatory tags. We can ignore OVN mac ports, but we do
    care about the overcloud node ctlplane ports.

    Add the raise MissingMandatoryNeutronResourceTag back
    in _ports_by_role_and_host.

    Change-Id: Ie7bd84f40f3a0bf5155c0bd455815fa3ecacdef4
    Closes-Bug: #1928926

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
Sandeep Yadav (sandeepyadav93) wrote :

Hello Harald,

We are hitting this in the master periodic integration line. Is this completely resolved?

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/8e4fa01/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
~~~
2021-05-26 04:19:55 | 2021-05-26 04:19:55.126272 | fa163eec-71d4-0bac-7e59-00000000000b | FATAL | Generate ansible inventory | localhost | error={"changed": false, "error": "'ctlplane_cidr'", "msg": "Error generating inventory for overcloud: 'ctlplane_cidr'", "success": false}
2021-05-26 04:19:55 | 2021-05-26 04:19:55.128095 | fa163eec-71d4-0bac-7e59-00000000000b | TIMING | Generate ansible inventory | localhost | 0:00:21.802745 | 6.31s
2021-05-26 04:19:55 |
~~~

The version of tripleo-common installed on undercloud is (86ba5af) latest and is greater than https://review.opendev.org/c/openstack/tripleo-common/+/792095 (d32863) as per https://github.com/openstack/tripleo-common/commits/master

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/8e4fa01/logs/undercloud/var/log/extra/package-list-installed.txt.gz

~~~
openstack-tripleo-common.noarch 16.1.0-0.20210525151910.86ba5af.el8 @delorean-current
openstack-tripleo-common-containers.noarch 16.1.0-0.20210525151910.86ba5af.el8 @delorean-current
~~~

Changed in tripleo:
status: Fix Released → Triaged
Revision history for this message
Sandeep Yadav (sandeepyadav93) wrote :

fs035 also failing with same error:-

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master/781f996/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
~~~
2021-05-26 02:58:30 | 2021-05-26 02:58:30.584767 | fa163e9a-6d05-57ae-89d3-00000000000b | FATAL | Generate ansible inventory | localhost | error={"changed": false, "error": "'ctlplane_cidr'", "msg": "Error generating inventory for overcloud: 'ctlplane_cidr'", "success": false}
2021-05-26 02:58:30 | 2021-05-26 02:58:30.586672 | fa163e9a-6d05-57ae-89d3-00000000000b | TIMING | Generate ansible inventory | localhost | 0:00:27.233258 | 8.42s
~~~

This is blocking master promotion, adding promotion-blocker flag.

Changed in tripleo:
importance: High → Critical
tags: added: ci promotion-blocker
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (master)
Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-common/+/793118
Committed: https://opendev.org/openstack/tripleo-common/commit/5d6649fc9d816537ec653eac4d5e29a49b996746
Submitter: "Zuul (22348)"
Branch: master

commit 5d6649fc9d816537ec653eac4d5e29a49b996746
Author: Harald Jensås <email address hidden>
Date: Wed May 26 10:39:07 2021 +0200

    Disable neutron inventory src if no ctlplane ports

    ctlplane ports created by THT Heat Server
    resources, or nova less without --network-ports/--network-config
    enabled, does not have the 'tripleo_stack_name' tag. We
    shouldn't use neutron as a source if no ctlplane ports are
    tagged with the 'tripleo_stack_name'.

    This patch add's a check for to ensure atleast 1 port is in the
    ctlplane network. If no such port is found _get_neutron_data
    returns None, and the inventory generation will continue to use
    the heat stack as the source.

    Also filter "tripleo_service_vip" and "tripleo_vip_net" ports
    when getting the ports so that the test doesn't match on a
    control plane virtual IP.

    Closes-Bug: #1928926
    Change-Id: Ida1cc17ae6b9adf4275e0616706158bfc3b93204

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-common/+/793548

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/tripleo-common/+/793548
Committed: https://opendev.org/openstack/tripleo-common/commit/983377bae2ad649874ca915ad5b2ff58dbfa4877
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 983377bae2ad649874ca915ad5b2ff58dbfa4877
Author: Harald Jensås <email address hidden>
Date: Wed May 26 10:39:07 2021 +0200

    Disable neutron inventory src if no ctlplane ports

    ctlplane ports created by THT Heat Server
    resources, or nova less without --network-ports/--network-config
    enabled, does not have the 'tripleo_stack_name' tag. We
    shouldn't use neutron as a source if no ctlplane ports are
    tagged with the 'tripleo_stack_name'.

    This patch add's a check for to ensure atleast 1 port is in the
    ctlplane network. If no such port is found _get_neutron_data
    returns None, and the inventory generation will continue to use
    the heat stack as the source.

    Also filter "tripleo_service_vip" and "tripleo_vip_net" ports
    when getting the ports so that the test doesn't match on a
    control plane virtual IP.

    Closes-Bug: #1928926
    Change-Id: Ida1cc17ae6b9adf4275e0616706158bfc3b93204
    (cherry picked from commit 5d6649fc9d816537ec653eac4d5e29a49b996746)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-common 15.2.1

This issue was fixed in the openstack/tripleo-common 15.2.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-common 16.1.0

This issue was fixed in the openstack/tripleo-common 16.1.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers