System config failed by Server died: listen tcp :44134: bind: address already in use

[Peng Peng]

Brief Description
-----------------
Using ansible-playbook to bring up regular system failed. kubectl log showed that tiller-deploy- pod "Server died: listen tcp :44134: bind: address already in use"

Severity
--------
Major

Steps to Reproduce
------------------
ansible-playbook lab-install-playbook.yaml -e "@local-install-overrides.yaml

TC-name:

Expected Behavior
------------------

Actual Behavior
----------------

Reproducibility
---------------
Unknown - first time this is seen in sanity, will monitor

System Configuration
--------------------
Multi-node system

Lab-name: WCP_3-6

Branch/Pull Time/Commit
-----------------------
2019-12-11_21-20-16

Last Pass
---------
2019-12-10_20-00-00

Timestamp/Logs
--------------
controller-0:~$ kubectl logs tiller-deploy-d6b59fcb-jwm9s -n kube-system
[main] 2019/12/12 16:33:32 Server died: listen tcp :44134: bind: address already in use

Test Activity
-------------
installation

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/698864

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/698864
Committed: https://git.openstack.org/cgit/starlingx/stx-puppet/commit/?id=34f3c3ff416090cfee8a65c1f8e0efa079e778de
Submitter: Zuul
Branch: master

commit 34f3c3ff416090cfee8a65c1f8e0efa079e778de
Author: Robert Church <email address hidden>
Date: Thu Dec 12 19:51:23 2019 -0500

    Reserve tiller ports as part of controller bootstrap

    Ephemeral port reservation was taking place only after the initial
    controller unlock. This still allowed for kubernetes based services to
    claim the tiller ports in the ephemeral range after initial bootstrap.

    Include tiller port reservation as part of the bootstrap manifests and
    add an explicit dependency with the platform::kubernetes::master class.

    Change-Id: I7540cdfdd13ded63d154ee8ea0b1c8c326491d63
    Closes-Bug: #1856209
    Signed-off-by: Robert Church <email address hidden>

[Ghada Khalil (gkhalil)]

Marking for stx.3.0 as per Bob Church's recommendation.
The fix will be cherry-picked for the first maintenance release as the issue results in failures for initial controller configuration

[Ghada Khalil (gkhalil)]

@Bob Church, please cherrypick the changes to the r/stx.3.0 branch for inclusion in the first maintenance release.

[Peng Peng (ppeng)]

Not seeing this issue recently

[Bill Zvonar (billzvonar)]

Bob - reminder: please cherrypick the changes to the r/stx.3.0 branch for inclusion in the first maintenance release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: r/stx.3.0
Review: https://review.opendev.org/748520

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/748520
Committed: https://git.openstack.org/cgit/starlingx/stx-puppet/commit/?id=e7bd0d2c3e5205a701e17afad5fd526ff41be2c8
Submitter: Zuul
Branch: r/stx.3.0

commit e7bd0d2c3e5205a701e17afad5fd526ff41be2c8
Author: Robert Church <email address hidden>
Date: Thu Dec 12 19:51:23 2019 -0500

    Reserve tiller ports as part of controller bootstrap

    Ephemeral port reservation was taking place only after the initial
    controller unlock. This still allowed for kubernetes based services to
    claim the tiller ports in the ephemeral range after initial bootstrap.

    Include tiller port reservation as part of the bootstrap manifests and
    add an explicit dependency with the platform::kubernetes::master class.

    Change-Id: I7540cdfdd13ded63d154ee8ea0b1c8c326491d63
    Closes-Bug: #1856209
    Signed-off-by: Robert Church <email address hidden>
    (cherry picked from commit 34f3c3ff416090cfee8a65c1f8e0efa079e778de)