[master][fs035][ipv6] Deployments failing with 503 Service Unavailable(neutron)

[yatin]

Both check[1] and promotion[2] master fs035 ipv6 jobs is broken most likely after https://review.opendev.org/#/c/668902.

The TASK [os_tempest : Ensure private network exists] fails with below error:-

HttpException: 503: Server Error for url: https://[2001:db8:fd00:1000::5]:13696/v2.0/networks.json?tenant_id=e1986c6ac7ab4e1aacb5f15ef31b06ad, No server is available to handle this request.: 503 Service Unavailable

[Log] https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/195f727/job-output.txt

From haproxy stats, neutron is DOWN on all three controllers
neutron,overcloud-controller-2.internalapi.localdomain,0,0,0,0,,0,0,0,,0,,0,0,0,0,DOWN,1,1,0,1,1,974,974,,1,11,1,,0,,2,0,,0,L7TOUT,,10002,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0,
neutron,overcloud-controller-1.internalapi.localdomain,0,0,0,0,,0,0,0,,0,,0,0,0,0,DOWN,1,1,0,1,1,974,974,,1,11,2,,0,,2,0,,0,L7TOUT,,10002,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0,
neutron,overcloud-controller-0.internalapi.localdomain,0,0,0,0,,0,0,0,,0,,0,0,0,0,DOWN,1,1,0,1,1,974,974,,1,11,3,,0,,2,0,,0,L7TOUT,,10000,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0,

[Log] https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/195f727/logs/overcloud-controller-0/var/log/extra/haproxy-stats.txt.gz

From neutron sever log, neutron is not able to contact south db
Unable to open stream to tcp:[fd00:fd00:fd00:2000::f7]:6641 to retrieve schema: Connection refused: Exception: Could not retrieve schema from tcp:[fd00:fd00:fd00:2000::f7]:6641

[Log] https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/195f727/logs/overcloud-controller-0/var/log/containers/neutron/server.log.txt.gz

[1] https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035&branch=master
[2] https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master&job_name=periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/709404

[Marios Andreou (marios-b)]

    upstream fix https://review.opendev.org/#/c/709404/ blocked https://bugs.launchpad.net/tripleo/+bug/1864399

    periodic: 11:06 < ykarel> marios|ruck, i ran fs035 master periodic with the fix https://review.rdoproject.org/r/#/c/25254/ 11:06 < ykarel> it reported to dlrnapi and now master is promoting

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/709404
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=c0e37c4a34ae01837a8bd83a6eb270d0a424b3a3
Submitter: Zuul
Branch: master

commit c0e37c4a34ae01837a8bd83a6eb270d0a424b3a3
Author: yatinkarel <email address hidden>
Date: Mon Feb 24 09:36:00 2020 +0530

    Fix listen_on_master_ip_only

    https://review.opendev.org/#/c/668902/ had
    wrongly set the value of listen_on_master_ip_only,
    it should to be 'no' when external lb VIP is used else
    'yes'.

    Depends-On: https://review.opendev.org/#/c/709389/
    Closes-Bug: #1864409
    Change-Id: I25b9366f8a097bda89734cc27fe83b94de24507e

[Marios Andreou (marios-b)]

this should be fixed after the patch merged https://review.opendev.org/709404 and the test at https://review.rdoproject.org/r/#/c/25254/ showed the fix is good

unfortunately last 2 runs of the periodic failed for unrelated reason so we haven't yet had a good periodic run to confirm

     * https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master

---

      * https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/6601ed7/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
            * 2020-02-25 03:45:01 | ResourceInError: resources.Controller: Went to status ERROR due to "Message: No valid host was found. , Code: 500"

        * https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/f4e342a/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
            * 2020-02-24 20:50:41 | ResourceInError: resources.Controller: Went to status ERROR due to "Message: No valid host was found. , Code: 500"

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 12.2.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/727771

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/727771
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=663f103da76760df1f2b44da73781ba5a799d9a3
Submitter: Zuul
Branch: stable/train

commit 663f103da76760df1f2b44da73781ba5a799d9a3
Author: yatinkarel <email address hidden>
Date: Mon Feb 24 09:36:00 2020 +0530

    Fix listen_on_master_ip_only

    https://review.opendev.org/#/c/668902/ had
    wrongly set the value of listen_on_master_ip_only,
    it should to be 'no' when external lb VIP is used else
    'yes'.

    Depends-On: https://review.opendev.org/#/c/709389/
    Closes-Bug: #1864409
    Change-Id: I25b9366f8a097bda89734cc27fe83b94de24507e
    (cherry picked from commit c0e37c4a34ae01837a8bd83a6eb270d0a424b3a3)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/727946

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/727792
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=50c5e1e6a52798ecebd35c3a1892573ec454061b
Submitter: Zuul
Branch: master

commit 50c5e1e6a52798ecebd35c3a1892573ec454061b
Author: Michele Baldessari <email address hidden>
Date: Thu May 14 09:01:26 2020 +0200

    Only run the VIP creation code when enable_load_balancer is set to true

    When we use an external loadbalancer we do not want neither haproxy nor
    VIPs on our control plane, since they will be managed externally.

    So a user will usually include something like the following template:
    /usr/share/openstack-tripleo-heat-templates/environments/external-loadbalancer-vip.yaml
    or implement a similar set of parameters/resource mapping.
    The main needed params are: EnableLoadBalancer set to false, the
    noop-ing of the haproxy service via 'OS::TripleO::Services::HAproxy: OS::Heat::None'
    and the hard-coding of the external VIPs via the *VirtualFixedIPs
    parameters.

    To make sure we do not create the OVNDbs VIP we will use
    the listen_on_master_ip_only as the variable to
    determine if an external loadbalancer is used:
    - no -> external LB used
    - yes -> pcmk+haproxy i.e. our own loadbalancer

    We cannot use the enable_load_balancer hiera key that is normally
    set by haproxy, because the haproxy service will be nooped.

    Related-Bug: #1864409

    Change-Id: Ie313c0410ed684661aeea77158572035ddfcfcd5

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/727946
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=8cf3793626f90c4a982cdf884f83722cd1563337
Submitter: Zuul
Branch: stable/train

commit 8cf3793626f90c4a982cdf884f83722cd1563337
Author: Michele Baldessari <email address hidden>
Date: Thu May 14 09:06:22 2020 +0200

    Only run the VIP creation code when enable_load_balancer is set to true

    When we use an external loadbalancer we do not want neither haproxy nor
    VIPs on our control plane, since they will be managed externally.

    So a user will usually include something like the following template:
    /usr/share/openstack-tripleo-heat-templates/environments/external-loadbalancer-vip.yaml
    or implement a similar set of parameters/resource mapping.
    The main needed params are: EnableLoadBalancer set to false, the
    noop-ing of the haproxy service via 'OS::TripleO::Services::HAproxy: OS::Heat::None'
    and the hard-coding of the external VIPs via the *VirtualFixedIPs
    parameters.

    To make sure we do not create the OVNDbs VIP we will use
    the listen_on_master_ip_only as the variable to
    determine if an external loadbalancer is used:
    - no -> external LB used
    - yes -> pcmk+haproxy i.e. our own loadbalancer

    We cannot use the enable_load_balancer hiera key that is normally
    set by haproxy, because the haproxy service will be nooped.

    NB: cherry-pick is not clean due to context

    Related-Bug: #1864409
    (cherry picked from commit 50c5e1e6a52798ecebd35c3a1892573ec454061b)

    Change-Id: Ie313c0410ed684661aeea77158572035ddfcfcd5

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/731062

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/731062
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=7c9cd6e6209ba8647007740f49916d08f81f18ee
Submitter: Zuul
Branch: stable/ussuri

commit 7c9cd6e6209ba8647007740f49916d08f81f18ee
Author: Michele Baldessari <email address hidden>
Date: Thu May 14 09:01:26 2020 +0200

    Only run the VIP creation code when enable_load_balancer is set to true

    When we use an external loadbalancer we do not want neither haproxy nor
    VIPs on our control plane, since they will be managed externally.

    So a user will usually include something like the following template:
    /usr/share/openstack-tripleo-heat-templates/environments/external-loadbalancer-vip.yaml
    or implement a similar set of parameters/resource mapping.
    The main needed params are: EnableLoadBalancer set to false, the
    noop-ing of the haproxy service via 'OS::TripleO::Services::HAproxy: OS::Heat::None'
    and the hard-coding of the external VIPs via the *VirtualFixedIPs
    parameters.

    To make sure we do not create the OVNDbs VIP we will use
    the listen_on_master_ip_only as the variable to
    determine if an external loadbalancer is used:
    - no -> external LB used
    - yes -> pcmk+haproxy i.e. our own loadbalancer

    We cannot use the enable_load_balancer hiera key that is normally
    set by haproxy, because the haproxy service will be nooped.

    Related-Bug: #1864409

    Change-Id: Ie313c0410ed684661aeea77158572035ddfcfcd5
    (cherry picked from commit 50c5e1e6a52798ecebd35c3a1892573ec454061b)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 11.4.0 release.