Deployment with default roles data broken: container-image-prepare service not passing resource_registry to tripleo-container-image-prepare

Bug #1794103 reported by Jiří Stránský
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Emilien Macchi

Bug Description

When i'm using default roles data, my deployment fails on:

ERROR: 25814 -- Failed to pull image: 192.168.24.1:8787/tripleomaster/centos-binary-neutron-server:current-tripleo

The reason this happens is that the tripleo-container-image-prepare command has downloaded centos-binary-neutron-server-opendaylight image instead.

I investigated a bit, it seems tripleo-container-image-prepare has the logic to handle "OS::Heat::None" resource_registry values. However, we don't pass resource_registry to that command when calling it from the container-image-prepare.j2.yaml composable service:

https://github.com/openstack/tripleo-heat-templates/blob/300c366701ab09cbef9b58f82bfd615e0d901dea/puppet/services/container-image-prepare.j2.yaml#L89-L95

So the tripleo-container-image-prepare command is receiving the roles data with OpenDaylight service entries, but it's not receiving that the OpenDaylight services are mapped to None, so it thinks i'm using OpenDaylight.

tags: added: containers
description: updated
Changed in tripleo:
assignee: nobody → Steve Baker (steve-stevebaker)
Revision history for this message
Steve Baker (steve-stevebaker) wrote :

FWIW it has always worried me that the neutron driver is implicitly derived from what services the role has deployed, instead of being indicated by an actual heat parameter. I'm not seeing any alternative to adding that parameter now, let me come up with something.

tags: added: rocky-backport-potential
Revision history for this message
Steve Baker (steve-stevebaker) wrote :

At first look, the existing parameter NeutronMechanismDrivers may be able to accurately determine what to set neutron_driver to.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.openstack.org/604952

Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
Steve Baker (steve-stevebaker) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/604952
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=6450d206c67d44ea866f2d2d44064e6f4d4eb6d2
Submitter: Zuul
Branch: master

commit 6450d206c67d44ea866f2d2d44064e6f4d4eb6d2
Author: Steve Baker <email address hidden>
Date: Tue Sep 25 13:52:45 2018 +1200

    Pass NeutronMechanismDrivers parameter to prepare

    This change is part of the proposal to use the value of
    NeutronMechanismDrivers to determine what neutron images to prepare
    (specifically whether the Opendaylight or OVN images should be
    prepared).

    Change-Id: I1fd78fd1eeb071788fc23ba343915ce7d6d14f02
    Partial-Bug: #1794103

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/605867

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/605870

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-heat-templates (stable/rocky)

Change abandoned by Alex Schultz (<email address hidden>) on branch: stable/rocky
Review: https://review.openstack.org/605867
Reason: http://lists.openstack.org/pipermail/openstack-dev/2018-September/135224.html

Changed in tripleo:
assignee: Steve Baker (steve-stevebaker) → Emilien Macchi (emilienm)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/rocky)

Reviewed: https://review.openstack.org/605867
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=4aa7f8cca41a294dfafc57fee4df404981e16835
Submitter: Zuul
Branch: stable/rocky

commit 4aa7f8cca41a294dfafc57fee4df404981e16835
Author: Steve Baker <email address hidden>
Date: Tue Sep 25 13:52:45 2018 +1200

    Pass NeutronMechanismDrivers parameter to prepare

    This change is part of the proposal to use the value of
    NeutronMechanismDrivers to determine what neutron images to prepare
    (specifically whether the Opendaylight or OVN images should be
    prepared).

    Change-Id: I1fd78fd1eeb071788fc23ba343915ce7d6d14f02
    Partial-Bug: #1794103
    (cherry picked from commit 6450d206c67d44ea866f2d2d44064e6f4d4eb6d2)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (stable/rocky)

Reviewed: https://review.openstack.org/605870
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=f8fe46d7cff02321d8e6d07c36ea1dda00a5dc24
Submitter: Zuul
Branch: stable/rocky

commit f8fe46d7cff02321d8e6d07c36ea1dda00a5dc24
Author: Steve Baker <email address hidden>
Date: Tue Sep 25 13:56:05 2018 +1200

    Set prepare neutron_driver from NeutronMechanismDrivers

    Since the fix for bug #1727347 landed, this interacts poorly with the
    logic which decides whether Opendaylight or OVN are deployed. Since
    tripleo-heat-templates driven image prepare doesn't have access to the
    actual resource registry, it doesn't know if a service in the roles
    data is mapped to a real implementation in the resource registry.

    This change switches to determining the neutron driver by using the
    value of the NeutronMechanismDrivers parameter.

    Change-Id: Ia51a825d11bd9b94d0110f13cdf2a6bbcedf6194
    Depends-On: I1fd78fd1eeb071788fc23ba343915ce7d6d14f02
    Closes-Bug: #1794103
    (cherry picked from commit ddda1549f29c806ff952dd3e7f21a08e64394778)

Changed in tripleo:
assignee: Emilien Macchi (emilienm) → wes hayutin (weshayutin)
Changed in tripleo:
assignee: wes hayutin (weshayutin) → Emilien Macchi (emilienm)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (master)

Reviewed: https://review.openstack.org/604953
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=62e86139b989b764e8a1905f40ff07e791e6a832
Submitter: Zuul
Branch: master

commit 62e86139b989b764e8a1905f40ff07e791e6a832
Author: Steve Baker <email address hidden>
Date: Tue Sep 25 13:56:05 2018 +1200

    Set prepare neutron_driver from NeutronMechanismDrivers

    Since the fix for bug #1727347 landed, this interacts poorly with the
    logic which decides whether Opendaylight or OVN are deployed. Since
    tripleo-heat-templates driven image prepare doesn't have access to the
    actual resource registry, it doesn't know if a service in the roles
    data is mapped to a real implementation in the resource registry.

    This change switches to determining the neutron driver by using the
    value of the NeutronMechanismDrivers parameter.

    Change-Id: Ia51a825d11bd9b94d0110f13cdf2a6bbcedf6194
    Depends-On: I1fd78fd1eeb071788fc23ba343915ce7d6d14f02
    Closes-Bug: #1794103

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
John Fulton (jfulton-org) wrote :

I have the fix [0] but I'm still hitting the issue [1] and needing to use a workaround [2]. I used oooq with RELEASE=master-tripleo-ci on Oct 16, 2018.

[0]
[root@undercloud image]# grep -n set_neutron_driver kolla_builder.py -A 15 | head -15
113:def set_neutron_driver(pd, mapping_args):
114- """Set the neutron_driver images variable based on parameters
115-
116- :param pd: Parameter defaults from the environment
117- :param mapping_args: Dict to set neutron_driver value on
118- """
119- if not pd or 'NeutronMechanismDrivers' not in pd:
120- return
121- nmd = pd['NeutronMechanismDrivers']
122- if 'opendaylight_v2' in nmd:
123- mapping_args['neutron_driver'] = 'odl'
124- elif 'ovn' in nmd:
125- mapping_args['neutron_driver'] = 'ovn'
126-
127-
[root@undercloud image]# pwd
/usr/lib/python2.7/site-packages/tripleo_common/image
[root@undercloud image]#

[1]
[root@undercloud overcloud]# tail -250 ansible.log | grep "Unable to find image"
        "2018-10-17 12:34:33,567 ERROR: 25932 -- Unable to find image '192.168.24.1:8787/tripleomaster/centos-binary-neutron-server:current-tripleo' locally",
[root@undercloud overcloud]# curl -s http://192.168.24.1:8787/v2/_catalog | jq "." | grep neutron
    "tripleomaster/centos-binary-ironic-neutron-agent",
    "tripleomaster/centos-binary-neutron-dhcp-agent",
    "tripleomaster/centos-binary-neutron-l3-agent",
    "tripleomaster/centos-binary-neutron-metadata-agent",
    "tripleomaster/centos-binary-neutron-openvswitch-agent",
    "tripleomaster/centos-binary-neutron-server-opendaylight",
[root@undercloud overcloud]#

[2]

echo "resource_registry:" > ~/no-odl.yaml
echo " OS::TripleO::Services::OpenDaylightApi: OS::Heat::None" >> ~/no-odl.yaml
echo " OS::TripleO::Services::OpenDaylightOvs: OS::Heat::None" >> ~/no-odl.yaml
echo "resource_registry:" > ~/no-ovn.yaml
echo " OS::TripleO::Services::OVNDBs: OS::Heat::None" >> ~/no-ovn.yaml
echo " OS::TripleO::Services::OVNController: OS::Heat::None" >> ~/no-ovn.yaml
echo " OS::TripleO::Services::OVNMetadataAgent: OS::Heat::None" >> ~/no-ovn.yaml

sudo openstack tripleo container image prepare default \
       --local-push-destination \
       --output-env-file containers-prepare-parameter.yaml

sudo openstack tripleo container image prepare \
  -e ~/containers-prepare-parameter.yaml \
  -e ~/no-odl.yaml \
  -e ~/no-ovn.yaml \
  --output-env-file ~/containers-default-parameters.yaml

Revision history for this message
Jiří Stránský (jistr) wrote :

We need to check the content in containers too. I hit the same problem yesterday, but it seems to be due to Quickstart not updating my containers despite setting:

update_containers: true

I have the set_neutron_driver function present on baremetal but not in the mistral_executor container, which i assume is the cause for still hitting the issue.

Revision history for this message
Jiří Stránský (jistr) wrote :

Something might be wrong with the "modify image" feature *for undercloud specifically*. The -updated* images exist for overcloud and the overcloud uses them, but undercloud doesn't.

Revision history for this message
Jiří Stránský (jistr) wrote :

John, this is likely due to and issue with Quickstart defaults, see https://review.openstack.org/#/c/611601/ and the attached bug. I hope that could help, i'm testing now.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-common 10.1.0

This issue was fixed in the openstack/tripleo-common 10.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-common 9.5.0

This issue was fixed in the openstack/tripleo-common 9.5.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.