delegate_to on deploying host uses dns?

[Ebbex]

In /opt/openstack-ansible/playbooks/common-tasks/haproxy-endpoint-manage.yml there is a delegate_to;

- name: Set haproxy service state
  ...
  delegate_to: "{{ delegate_host }}"
  with_items: "{{ groups['haproxy'] }}"
  loop_control:
    loop_var: delegate_host

For some reason, this does a dns lookup for the 'delegate_host' (instead of using the inventory?). Which (because of my sheer dumb luck) resolves to a different IP than what the deploying host has access to.

e.g. for clarification;
'deploy host' resides on 172.16.1.0/24 (maas) and 172.16.2.0/24 (mgmt)
'delegate_host' resides on 172.16.1.0/24(maas), 172.16.2.0/24(mgmt) AND 172.16.3.0/24 (storage)

And my maas-dns decides to return '172.16.3.123' as the ip address for 'delegate_host'.

Now, all the hosts deployed to get a big fat /etc/hosts file. Perhaps the deploying host should get one aswell?

[Mohammed Naser (mnaser)]

Hi

Can you please let us know what release are you running?

Thanks,
Mohammed

[Logan V (loganv)]

Discussion from the bug triage meeting today. I am seeing it on Pike, logs included:

[16:39:10] <mnaser> #link https://bugs.launchpad.net/openstack-ansible/+bug/1790779
[16:39:11] <openstack> Launchpad bug 1790779 in openstack-ansible "delegate_to on deploying host uses dns?" [Undecided,New]
[16:39:58] <mnaser> hwoarang: or logan- might chime on this?
[16:40:17] <hwoarang> hmm
[16:40:23] <mnaser> afaik this should use the physical_mapping thing no?
[16:40:43] <logan-> yes one sec
[16:41:00] <logan-> i hit this on pike last week
[16:41:02] <evrardjp> interesting, is it the case of delegation to something not in the inventory that triggers?
[16:41:03] <logan-> it is broken as of https://github.com/openstack/openstack-ansible/commit/a8a809839484105d9cd27463defc19a8a617c64b#diff-db999e390dd84f2a8c2a48b19aa9533f
[16:41:31] <logan-> you can see it on a prod deploy here: http://logs.openstack.org/e9/e957a7dc2fd07f86bc2ab60dd25e00504c705eca/post/limestone-ci-deploy/705562f/
[16:41:41] <evrardjp> oh god it's loop_var that causing the issue?
[16:41:43] <hwoarang> reporter does not say what version it is
[16:41:48] <logan-> http://logs.openstack.org/e9/e957a7dc2fd07f86bc2ab60dd25e00504c705eca/post/limestone-ci-deploy/705562f/logs/ara-report/result/45ed4626-65ca-4b65-a4c4-d39cf32eaadf/
[16:41:57] <jamesdenton> mnaser pong
[16:42:02] <hwoarang> i thought the interpolation of loop_var was fixed by the recent commit in the plugins repo
[16:42:23] <logan-> hwoarang: i was hoping that backporting your change to the other branches would solve this, but it did not in pike
[16:42:30] <openstackgerrit> Jacob Wagner proposed openstack/openstack-ansible-ops master: Add ability to deploy Ceph into a Multinode AIO https://review.openstack.org/585003
[16:42:48] <openstackgerrit> Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_keystone master: Implement LDAP domains using last keystone host https://review.openstack.org/603440
[16:43:18] <hwoarang> we should clarify on the bug what is the OSA version that he is using
[16:43:20] <evrardjp> debug of https://github.com/openstack/openstack-ansible-plugins/blob/master/strategy/linear.py#L138-L147 would help I guess
[16:43:20] <logan-> that build used dc4abed00960c0123d45e9e67e666a5e9a1bb51f from plugins http://logs.openstack.org/e9/e957a7dc2fd07f86bc2ab60dd25e00504c705eca/post/limestone-ci-deploy/705562f/job-output.txt.gz#_2018-09-17_19_32_59_217416
[16:43:25] <hwoarang> because master and rocky should be ok
[16:43:37] <logan-> which is your fix: https://github.com/openstack/openstack-ansible-plugins/commits/stable/pike
[16:43:40] *** Quits: Darcidride (~Darcidrid@2a01:e35:8b4a:ca10::719e:e3e1) (Remote host closed the connection)
[16:43:49] <evrardjp> hwoarang: do we have the loop_var in testing?
[16:43:56] <hwoarang> prob not
[16:44:03] <odyssey4me> we do not, because there is only one cluster member
[16:44:14] <hwoarang> but in opnfv we use rocky with multinode and it's ok (so we are using the loop_var thing)
[16:44:15] <odyssey4me> we should add one to the plugins repo I guess
[16:44:16] *** Joins: Darcidride (~Darcidrid@2a01:e35:8b4a:ca10::719e:e3e1)
[16:44:22] <mnaser> jamesdenton: we can discuss...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/603450

[Logan V (loganv)]

The patch above fixed the bug in my pike environment ^

[Ebbex (eb4x)]

Thanks for looking into this!

Yes apologies, this is stable/pike :)

[Logan V (loganv)]

Thanks, if you have a moment please try the patch from #3 out and see if it fixes the problem for you.

I assume you are also running your haproxy in containers? That was one of the factors for triggering the bug in my env.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/603450
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=63f5d0e4fa1f745cbf5ee644f8dee88a63de26bb
Submitter: Zuul
Branch: master

commit 63f5d0e4fa1f745cbf5ee644f8dee88a63de26bb
Author: Logan V <email address hidden>
Date: Tue Sep 18 12:07:57 2018 -0500

    Avoid using loop_var in endpoint manage

    When the OSA connection plugin is used on older ansible versions,
    and when haproxy runs in a container, a bug exists where the loop_var
    fails to properly delegate the task to the haproxy container.

    We can avoid using loop_var in the delegation of the task by simply
    moving the loop_var up one level so that the endpoint-manage task
    file can use {{ item }}.

    This was introduced in the commit:
    https://github.com/openstack/openstack-ansible/commit/a8a809839484105d9cd27463defc19a8a617c64b#diff-db999e390dd84f2a8c2a48b19aa9533f

    Fixes-Bug: #1790779
    Change-Id: I17dafb283e41ce05083ae4adb3a325aaca0253dd

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/603547

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/603548

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/603550

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/603548
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=e3a688ed6efdf0704917575af8a0780c2dff08d9
Submitter: Zuul
Branch: stable/queens

commit e3a688ed6efdf0704917575af8a0780c2dff08d9
Author: Logan V <email address hidden>
Date: Tue Sep 18 12:07:57 2018 -0500

    Avoid using loop_var in endpoint manage

    When the OSA connection plugin is used on older ansible versions,
    and when haproxy runs in a container, a bug exists where the loop_var
    fails to properly delegate the task to the haproxy container.

    We can avoid using loop_var in the delegation of the task by simply
    moving the loop_var up one level so that the endpoint-manage task
    file can use {{ item }}.

    This was introduced in the commit:
    https://github.com/openstack/openstack-ansible/commit/a8a809839484105d9cd27463defc19a8a617c64b#diff-db999e390dd84f2a8c2a48b19aa9533f

    Fixes-Bug: #1790779
    Change-Id: I17dafb283e41ce05083ae4adb3a325aaca0253dd
    (cherry picked from commit 63f5d0e4fa1f745cbf5ee644f8dee88a63de26bb)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/603550
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=83f0bdaeeb579666affeb46552a59c8cd4bd4434
Submitter: Zuul
Branch: stable/pike

commit 83f0bdaeeb579666affeb46552a59c8cd4bd4434
Author: Logan V <email address hidden>
Date: Tue Sep 18 12:07:57 2018 -0500

    Avoid using loop_var in endpoint manage

    When the OSA connection plugin is used on older ansible versions,
    and when haproxy runs in a container, a bug exists where the loop_var
    fails to properly delegate the task to the haproxy container.

    We can avoid using loop_var in the delegation of the task by simply
    moving the loop_var up one level so that the endpoint-manage task
    file can use {{ item }}.

    This was introduced in the commit:
    https://github.com/openstack/openstack-ansible/commit/a8a809839484105d9cd27463defc19a8a617c64b#diff-db999e390dd84f2a8c2a48b19aa9533f

    Fixes-Bug: #1790779
    Change-Id: I17dafb283e41ce05083ae4adb3a325aaca0253dd
    (cherry picked from commit 63f5d0e4fa1f745cbf5ee644f8dee88a63de26bb)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/603547
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=6745b3727f70d609895e18ce15caa367e481265b
Submitter: Zuul
Branch: stable/rocky

commit 6745b3727f70d609895e18ce15caa367e481265b
Author: Logan V <email address hidden>
Date: Tue Sep 18 12:07:57 2018 -0500

    Avoid using loop_var in endpoint manage

    When the OSA connection plugin is used on older ansible versions,
    and when haproxy runs in a container, a bug exists where the loop_var
    fails to properly delegate the task to the haproxy container.

    We can avoid using loop_var in the delegation of the task by simply
    moving the loop_var up one level so that the endpoint-manage task
    file can use {{ item }}.

    This was introduced in the commit:
    https://github.com/openstack/openstack-ansible/commit/a8a809839484105d9cd27463defc19a8a617c64b#diff-db999e390dd84f2a8c2a48b19aa9533f

    Fixes-Bug: #1790779
    Change-Id: I17dafb283e41ce05083ae4adb3a325aaca0253dd
    (cherry picked from commit 63f5d0e4fa1f745cbf5ee644f8dee88a63de26bb)