CI: containers overcloud can't start vm when deployed with multinic network isolation

Bug #1689556 reported by Sagi (Sergey) Shnaidman on 2017-05-09
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Critical
Steve Baker

Bug Description

When running containers job with multinic isolation (with new templates, see the patch below[1]) it deploys overcloud, but pingtest fails because "Message: No valid host was found. There are not enough hosts available., Code: 500"[2]

[1] https://review.openstack.org/#/c/422448/
[2] https://review.openstack.org/#/c/454185/
    logs: http://logs.openstack.org/85/454185/3/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq-nv/41e54d1/
    validate log: http://logs.openstack.org/85/454185/3/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq-nv/41e54d1/logs/oooq/undercloud/home/jenkins/overcloud_validate.log.txt.gz#_2017-05-08_20_02_47

Changed in tripleo:
status: New → Triaged
importance: Undecided → High
Changed in tripleo:
milestone: none → pike-2
Changed in tripleo:
milestone: pike-2 → pike-3
Changed in tripleo:
status: Triaged → In Progress
Changed in tripleo:
milestone: pike-3 → pike-rc1

The problem still exists:

neutron can't bind the port to vm because an error "Bridge br-ex for physical network datacentre does not exist":

https://thirdparty.logs.rdoproject.org/jenkins-tq-gate-devmode-master-ovb-rdocloud-public-bond-173/overcloud-novacompute-0/var/log/containers/neutron/neutron-openvswitch-agent.log.gz

2017-08-17 01:43:45.075 22140 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-efac378b-c7b8-4fd7-8535-7cfe4a2a7431 - - - - -] Bridge br-ex for physical network datacentre does not exist. Agent terminated!

Changed in tripleo:
status: In Progress → Triaged
Changed in tripleo:
importance: High → Critical
Ronelle Landy (rlandy) wrote :

We would like to add containers jobs running on OVB with network isolation in the new RDO Cloud pieplines. This bug is blocking that as the ping test fails.

Changed in tripleo:
milestone: pike-rc1 → pike-rc2
wes hayutin (weshayutin) on 2017-08-25
tags: added: alert

@Ihar, yes, but actually 1691403 is duplicate of this bug as this one was opened before. But it's fine to close it if all activities are in the other bug.

Changed in tripleo:
assignee: nobody → Steve Baker (steve-stevebaker)
status: Triaged → In Progress
Steve Baker (steve-stevebaker) wrote :

I'm proposing this as a fix https://review.openstack.org/#/c/493695/

Change abandoned by Emilien Macchi (<email address hidden>) on branch: master
Review: https://review.openstack.org/493695
Reason: I need to purge the gate because TripleO CI gate has critical issues right now, I'll make this patch goes to the gate.

Reviewed: https://review.openstack.org/493695
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=76f130d6e8f7434433b2602af9794f1e9c742e1f
Submitter: Jenkins
Branch: master

commit 76f130d6e8f7434433b2602af9794f1e9c742e1f
Author: Steve Baker <email address hidden>
Date: Tue Aug 15 10:26:24 2017 +1200

    container ovs-agent, ensure br-ex exists

    Currently the container neutron-ovs-agent is stuck in a restart loop
    in many environments because the bridge br-ex is missing.

    This bridge is created by running the puppet class
    neutron::agents::ml2::ovs but limiting that run to tag
    neutron::plugins::ovs::bridge.

    The hiera neutron::agents::ml2::ovs::bridge_mappings should already
    exists to create the bridge with the required settings.

    This change should ensure br-ex exists after step 3.

    Since br-ex is created regardless of the chosen network config,
    environments/docker-network.yaml is not longer required. It can be
    deleted once there are no more references to it in CI and
    documentation.

    Change-Id: Ie425148b0ad0f38e149c5fa0a97d98ec35d0a5bb
    Closes-Bug: #1699261
    Closes-Bug: #1691403
    Closes-Bug: #1689556

Changed in tripleo:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/499137
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=0f2af352d15cb9c486414619a8342e727a2e207c
Submitter: Jenkins
Branch: stable/pike

commit 0f2af352d15cb9c486414619a8342e727a2e207c
Author: Steve Baker <email address hidden>
Date: Tue Aug 15 10:26:24 2017 +1200

    container ovs-agent, ensure br-ex exists

    Currently the container neutron-ovs-agent is stuck in a restart loop
    in many environments because the bridge br-ex is missing.

    This bridge is created by running the puppet class
    neutron::agents::ml2::ovs but limiting that run to tag
    neutron::plugins::ovs::bridge.

    The hiera neutron::agents::ml2::ovs::bridge_mappings should already
    exists to create the bridge with the required settings.

    This change should ensure br-ex exists after step 3.

    Since br-ex is created regardless of the chosen network config,
    environments/docker-network.yaml is not longer required. It can be
    deleted once there are no more references to it in CI and
    documentation.

    Change-Id: Ie425148b0ad0f38e149c5fa0a97d98ec35d0a5bb
    Closes-Bug: #1699261
    Closes-Bug: #1691403
    Closes-Bug: #1689556
    (cherry picked from commit 76f130d6e8f7434433b2602af9794f1e9c742e1f)

tags: added: in-stable-pike

This issue was fixed in the openstack/tripleo-heat-templates 7.0.0.0rc2 release candidate.

This issue was fixed in the openstack/tripleo-heat-templates 8.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers