Bug #1864810 “haproxy restarted multiple times during deploy” : Bugs : kolla-ansible

Revision history for this message

Mark Goddard (mgoddard) wrote on 2020-03-02:

#1

It should only be restarted once, since the role is executed only from one play.

Revision history for this message

Radosław Piliszek (yoctozepto) wrote on 2020-03-02:

#2

"should be" != "is" ;-)

Grab any random CI log to discover.

Revision history for this message

Radosław Piliszek (yoctozepto) wrote on 2020-04-07:

#3

Xing Zhang, in the duplicate report, linked https://review.opendev.org/601648 as a possible cause for this. This is quite likely as it was observed since Stein for sure.

Revision history for this message

Xing Zhang (xingzhang) wrote on 2020-04-07:

#4

Got it, thanks.

OpenStack Infra (hudson-openstack) on 2020-04-21

Changed in kolla-ansible:
assignee:	nobody → Mark Goddard (mgoddard)
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-04-26: Fix merged to kolla-ansible (master)

#5

Reviewed: https://review.opendev.org/708385
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=1efe0fe333ff869ea0a04ff87a11ff27a251560b
Submitter: Zuul
Branch: master

commit 1efe0fe333ff869ea0a04ff87a11ff27a251560b
Author: Christian Berendt <email address hidden>
Date: Tue Feb 18 13:02:28 2020 +0100

Avoid multiple haproxy restarts after reconfiguration

    The haproxy role and the site.yml file calls the
    haproxy-config role to provide configuration for individual
    services.

If the configuration within a service changes, the haproxy
container is restarted.

    If the configuration in n services changes, there will be n
    restarts. This is not necessary, a restart at the end is
    sufficient.

    By removing the handler from the haproxy-config role and
    using the listen parameter in the handler of the haproxy role,
    the handler is executed only once.

Change-Id: I535fe67579fb748093bb4b30a6bd31b81e021a1b
Closes-Bug: #1864810

Changed in kolla-ansible:
status:	In Progress → Fix Released

Revision history for this message

Radosław Piliszek (yoctozepto) wrote on 2020-04-26:

#6

amendment: https://review.opendev.org/723245

There is no need to duplicate handler name in listen.
The issue was because we had two handlers with the same
name in the same environment.
This causes Ansible not to mark handler as already run.

We should probably lint this situation, and also report to upstream about issue.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-04-26: Related fix proposed to kolla-ansible (master)

#7

Related fix proposed to branch: master
Review: https://review.opendev.org/723258

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-04-26: Fix proposed to kolla-ansible (stable/train)

#8

Fix proposed to branch: stable/train
Review: https://review.opendev.org/723260

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-04-26: Fix proposed to kolla-ansible (stable/stein)

#9

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/723261

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-04-27: Related fix merged to kolla-ansible (master)

#10

Reviewed: https://review.opendev.org/723245
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=98e86ea62b64eb1c5a7dfaca6c366527f584afc0
Submitter: Zuul
Branch: master

commit 98e86ea62b64eb1c5a7dfaca6c366527f584afc0
Author: Radosław Piliszek <email address hidden>
Date: Sun Apr 26 19:59:26 2020 +0200

Remove redundant listen on haproxy handler

Follow-up on [1] "Avoid multiple haproxy restarts after
reconfiguration".

    There is no need to duplicate handler name in listen.
    The issue was because we had two handlers with the same
    name in the same environment.
    This causes Ansible not to mark handler as already run.

[1] https://review.opendev.org/708385

Change-Id: I5425a8037b6860ef71bce59becff8dfe5b601d4c
Related-Bug: #1864810

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-04-27:

#11

Reviewed: https://review.opendev.org/723258
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=04effaa9032b052343e3cf82a65054f0d35834ea
Submitter: Zuul
Branch: master

commit 04effaa9032b052343e3cf82a65054f0d35834ea
Author: Radosław Piliszek <email address hidden>
Date: Sun Apr 26 21:51:20 2020 +0200

Fix haproxy restarting twice per Ansible run

    Since haproxy is orchestrated via site.yml in a single play,
    it does not need flushing handlers as handlers run will
    happen at the end of this play.

    Change-Id: Ia3743575da707325be93c39b4a2bcae9211cacb2
    Related-Bug: #1864810
    Closes-Bug: #1875228

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-04-27: Related fix proposed to kolla-ansible (stable/train)

#12

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/723553

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-04-27: Related fix proposed to kolla-ansible (stable/stein)

#13

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/723611

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-04-27: Related fix merged to kolla-ansible (stable/train)

#14

Reviewed: https://review.opendev.org/723553
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=e2fd048430e9833c6c3eaf6e560001adc775c08d
Submitter: Zuul
Branch: stable/train

commit e2fd048430e9833c6c3eaf6e560001adc775c08d
Author: Radosław Piliszek <email address hidden>
Date: Sun Apr 26 21:51:20 2020 +0200

Fix haproxy restarting twice per Ansible run

    Since haproxy is orchestrated via site.yml in a single play,
    it does not need flushing handlers as handlers run will
    happen at the end of this play.

    Change-Id: Ia3743575da707325be93c39b4a2bcae9211cacb2
    Related-Bug: #1864810
    Closes-Bug: #1875228
    (cherry picked from commit 04effaa9032b052343e3cf82a65054f0d35834ea)

tags:

added: in-stable-train

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-04-27: Related fix merged to kolla-ansible (stable/stein)

#15

Reviewed: https://review.opendev.org/723611
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=5f563ed490aa63fd2e4551ca558ce6fe860d2179
Submitter: Zuul
Branch: stable/stein

commit 5f563ed490aa63fd2e4551ca558ce6fe860d2179
Author: Radosław Piliszek <email address hidden>
Date: Sun Apr 26 21:51:20 2020 +0200

Fix haproxy restarting twice per Ansible run

    Since haproxy is orchestrated via site.yml in a single play,
    it does not need flushing handlers as handlers run will
    happen at the end of this play.

    Change-Id: Ia3743575da707325be93c39b4a2bcae9211cacb2
    Related-Bug: #1864810
    Closes-Bug: #1875228
    (cherry picked from commit 04effaa9032b052343e3cf82a65054f0d35834ea)

tags:

added: in-stable-stein

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-04-27: Fix merged to kolla-ansible (stable/train)

#16

Reviewed: https://review.opendev.org/723260
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=e2a3834942e4dfc64d5ec79b9a2bc76b7cfde453
Submitter: Zuul
Branch: stable/train

commit e2a3834942e4dfc64d5ec79b9a2bc76b7cfde453
Author: Christian Berendt <email address hidden>
Date: Tue Feb 18 13:02:28 2020 +0100

Avoid multiple haproxy restarts after reconfiguration

    The haproxy role and the site.yml file calls the
    haproxy-config role to provide configuration for individual
    services.

If the configuration within a service changes, the haproxy
container is restarted.

    If the configuration in n services changes, there will be n
    restarts. This is not necessary, a restart at the end is
    sufficient.

    By removing the handler from the haproxy-config role and
    using the listen parameter in the handler of the haproxy role,
    the handler is executed only once.

    Change-Id: I535fe67579fb748093bb4b30a6bd31b81e021a1b
    Closes-Bug: #1864810
    (cherry picked from commit 1efe0fe333ff869ea0a04ff87a11ff27a251560b)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-04-28: Fix merged to kolla-ansible (stable/stein)

#17

Reviewed: https://review.opendev.org/723261
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=ca39a9c2747e07480363c64ea675ea71b4cfae56
Submitter: Zuul
Branch: stable/stein

commit ca39a9c2747e07480363c64ea675ea71b4cfae56
Author: Christian Berendt <email address hidden>
Date: Tue Feb 18 13:02:28 2020 +0100

Avoid multiple haproxy restarts after reconfiguration

    The haproxy role and the site.yml file calls the
    haproxy-config role to provide configuration for individual
    services.

If the configuration within a service changes, the haproxy
container is restarted.

    If the configuration in n services changes, there will be n
    restarts. This is not necessary, a restart at the end is
    sufficient.

    By removing the handler from the haproxy-config role and
    using the listen parameter in the handler of the haproxy role,
    the handler is executed only once.

    Change-Id: I535fe67579fb748093bb4b30a6bd31b81e021a1b
    Closes-Bug: #1864810
    (cherry picked from commit 1efe0fe333ff869ea0a04ff87a11ff27a251560b)

Revision history for this message

Radosław Piliszek (yoctozepto) wrote on 2020-04-28:

#18

I was really curious what the root cause really was and so I set on a journey to discover it.
The result is: https://github.com/yoctozepto/ansible-experiments/commit/e2dc35c3c55a61906aa6f95761d5e7d68ae556bd

TL;DR including/importing role with handlers along with setting vars will create
separate handler instances if there was any include along the path to role
with handler.

So the 'duplicate name' also was a false reason. Good to know I think.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-04-28: Related fix proposed to kolla-ansible (stable/train)

#19

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/723877

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-04-28: Related fix proposed to kolla-ansible (stable/stein)

#20

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/723878

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-04-28: Related fix merged to kolla-ansible (stable/train)

#21

Reviewed: https://review.opendev.org/723877
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=b3b31525833c619efd32d7818688325bbf8881ad
Submitter: Zuul
Branch: stable/train

commit b3b31525833c619efd32d7818688325bbf8881ad
Author: Radosław Piliszek <email address hidden>
Date: Sun Apr 26 19:59:26 2020 +0200

Remove redundant listen on haproxy handler

Follow-up on [1] "Avoid multiple haproxy restarts after
reconfiguration".

    There is no need to duplicate handler name in listen.
    The issue was because we had two handlers with the same
    name in the same environment.
    This causes Ansible not to mark handler as already run.

[1] https://review.opendev.org/708385

    Change-Id: I5425a8037b6860ef71bce59becff8dfe5b601d4c
    Related-Bug: #1864810
    (cherry picked from commit 98e86ea62b64eb1c5a7dfaca6c366527f584afc0)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-04-30: Related fix merged to kolla-ansible (stable/stein)

#22

Reviewed: https://review.opendev.org/723878
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=72cd552d794a77666387749cc94e5a4f50661d47
Submitter: Zuul
Branch: stable/stein

commit 72cd552d794a77666387749cc94e5a4f50661d47
Author: Radosław Piliszek <email address hidden>
Date: Sun Apr 26 19:59:26 2020 +0200

Remove redundant listen on haproxy handler

Follow-up on [1] "Avoid multiple haproxy restarts after
reconfiguration".

    There is no need to duplicate handler name in listen.
    The issue was because we had two handlers with the same
    name in the same environment.
    This causes Ansible not to mark handler as already run.

[1] https://review.opendev.org/708385

    Change-Id: I5425a8037b6860ef71bce59becff8dfe5b601d4c
    Related-Bug: #1864810
    (cherry picked from commit 98e86ea62b64eb1c5a7dfaca6c366527f584afc0)

	Status	Importance	Assigned to	Milestone
kolla-ansible	Fix Released	Low	Mark Goddard	kolla-ansible 10.0.0 "ussuri"
Stein	Fix Released	Low	Mark Goddard	kolla-ansible 8.2.0 "stein"
Train	Fix Released	Low	Mark Goddard	kolla-ansible 9.2.0 "Train"
Ussuri	Fix Released	Low	Mark Goddard	kolla-ansible 10.0.0 "ussuri"

kolla-ansible

haproxy restarted multiple times during deploy

Bug Description

Duplicates of this bug

Other bug subscribers

Remote bug watches