tripleo-ci-centos-7-standalone-upgrade-stein is broken by container registry login task

[Sagi (Sergey) Shnaidman]

tripleo-ci-centos-7-standalone-upgrade-stein job fails:

http://logs.openstack.org/57/669057/10/check/tripleo-ci-centos-7-standalone-upgrade-stein/98479d5/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz#_2019-07-05_15_25_39

after commits:
https://review.opendev.org/#/c/669222/
https://review.opendev.org/#/c/666646/

TASK [Perform container registry login(s)] *************************************
2019-07-05 15:25:39 | 2019-07-05 15:25:39.729 140874 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] fatal: [standalone]: FAILED! => {"msg": "An unhandled exception occurred while running the lookup plugin 'dict'. Error was a <class 'ansible.errors.AnsibleError'>, original message: with_dict expects a dict"}
2019-07-05 15:25:39 | 2019-07-05 15:25:39.731 140874 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] 
2019-07-05 15:25:39 | 2019-07-05 15:25:39.732 140874 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] NO MORE HOSTS LEFT *************************************************************
2019-07-05 15:25:39 | 2019-07-05 15:25:39.734 140874 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] 
2019-07-05 15:25:39 | 2019-07-05 15:25:39.735 140874 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] PLAY RECAP *********************************************************************
2019-07-05 15:25:39 | 2019-07-05 15:25:39.736 140874 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] standalone : ok=13 changed=3 unreachable=0 failed=1 skipped=20 rescued=0 ignored=0
2019-07-05 15:25:39 | 2019-07-05 15:25:39.736 140874 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] 
2019-07-05 15:25:39 | 2019-07-05 15:25:39.857 140874 ERROR tripleoclient.v1.tripleo_upgrade.Upgrade [-] Exception: Upgrade failed: DeploymentError: Upgrade failed
2019-07-05 15:25:39 | Traceback (most recent call last):
2019-07-05 15:25:39 | File "/usr/lib/python2.7/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1253, in _standalone_deploy
2019-07-05 15:25:39 | raise exceptions.DeploymentError('Upgrade failed')
2019-07-05 15:25:39 | DeploymentError: Upgrade failed
2019-07-05 15:25:39 | 2019-07-05 15:25:39.861 140874 ERROR tripleoclient.v1.tripleo_upgrade.Upgrade [-] None: DeploymentError: Upgrade failed
2019-07-05 15:25:41 | 2019-07-05 15:25:41.575 140874 ERROR tripleoclient.v1.tripleo_upgrade.Upgrade [-] ** Found ansible errors for standalone deployment! **: DeploymentError: Upgrade failed
2019-07-05 15:25:41 | 2019-07-05 15:25:41.577 140874 ERROR tripleoclient.v1.tripleo_upgrade.Upgrade [-] [
2019-07-05 15:25:41 | [
2019-07-05 15:25:41 | "Perform container registry login(s)",
2019-07-05 15:25:41 | {
2019-07-05 15:25:41 | "msg": "An unhandled exception occurred while running the lookup plugin 'dict'. Error was a <class 'ansible.errors.AnsibleError'>, original message: with_dict expects a dict",
2019-07-05 15:25:41 | "_ansible_no_log": null
2019-07-05 15:25:41 | }
2019-07-05 15:25:41 | ]
2019-07-05 15:25:41 | ]: DeploymentError: Upgrade failed

[Emilien Macchi (emilienm)]

I couldn't reproduce it locally yet, I'm reverting it to unblock CI.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/669575

[Emilien Macchi (emilienm)]

- block:
  - set_fact:
      container_registry_insecure_registries:
      - 192.168.24.1:8787
      container_registry_login: false
      container_registry_logins: {}
  - name: ensure podman and deps are installed
    package:
      name: podman
      state: latest
  - copy:
      content: ''
      dest: /etc/cni/net.d/87-podman-bridge.conflist
      force: true
    ignore_errors: true
    name: Remove default cni config for cni0 if exists
  - command: ip link delete cni0
    ignore_errors: true
    name: Delete cni0 interface if exists
  - ini_file:
      option: registries
      path: /etc/containers/registries.conf
      section: registries.insecure
      value: '{{ container_registry_insecure_registries }}'
    name: configure insecure registries /etc/containers/registries.conf
    when: container_registry_insecure_registries | length > 0
  - environment:
      REGISTRY: '{{ item.key }}'
      REGISTRY_PASSWORD: '{{ lookup(''dict'', item.value).value }}'
      REGISTRY_USERNAME: '{{ lookup(''dict'', item.value).key }}'
    loop: '{{ lookup(''dict'', container_registry_logins) }}'
    name: Perform container registry login(s)
    shell: podman login --username=$REGISTRY_USERNAME --password=$REGISTRY_PASSWORD
      $REGISTRY
    when:
    - container_registry_login | bool
    - container_registry_logins | length > 0
  name: Install and configure Podman

container_registry_logins is an empty dict, I'm not sure why it's failing yet.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/669576

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/669577

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/669575
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=ac5145c28d02e54453ab69d32237ca2b81e2568c
Submitter: Zuul
Branch: stable/stein

commit ac5145c28d02e54453ab69d32237ca2b81e2568c
Author: Emilien Macchi <email address hidden>
Date: Sun Jul 7 16:05:07 2019 +0000

    Revert "Add container engine authentication support"

    Related-Bug: #1835657
    This reverts commit 169f4ac83710916f801db278385a750306f9c8ea.

    Change-Id: I8b77cce5bb6b40eb2cb27c0bdd2d95e9c1a0d227

[Attila Fazekas (afazekas)]

               loop: "{{ lookup('dict', container_registry_logins is none | ternary({}, container_registry_logins | default({}))) }}"

This might work.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/670082

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/670083

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/670174

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/670174
Committed: https://git.openstack.org/cgit/openstack/ansible-role-container-registry/commit/?id=b295cc9aefc5b6635312067b4b399f20664d4412
Submitter: Zuul
Branch: master

commit b295cc9aefc5b6635312067b4b399f20664d4412
Author: Kevin Carter <email address hidden>
Date: Wed Jul 10 14:52:18 2019 -0500

    Covert lookup to query

    This change updates our loop so that it will expect a list.

    More on the query lookup can be seen here[0]

    [0] https://docs.ansible.com/ansible/2.6/plugins/lookup.html#invoking-lookup-plugins-with-query

    Change-Id: Id8bfea751a7239fd9be6e9dbbb5a0a700e29b64e
    Closes-Bug: #1835657
    Related-Bug: #1833584
    Signed-off-by: Kevin Carter <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/670082
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=d6bd20d5b46d7ede598e022f0d34814c8dbe8a66
Submitter: Zuul
Branch: stable/stein

commit d6bd20d5b46d7ede598e022f0d34814c8dbe8a66
Author: Emilien Macchi <email address hidden>
Date: Wed Jul 10 10:46:25 2019 -0400

    Stein: Re-enable container auth support

    Squash of the revert of the revert + the fix

    1) Revert "Revert "Add container engine authentication support""

    This reverts commit ac5145c28d02e54453ab69d32237ca2b81e2568c.

    2) Convert the heat json format to a py dict

    This change converts a heat json format option to a py dict within
    a jinja expresion.

    Closes-Bug: #1835657
    Related-Bug: #1833584
    Change-Id: I4b44214cd7007dc31ad5f4e0a0d7a3a531a9f20e
    Signed-off-by: Kevin Carter <email address hidden>
    (cherry picked from commit 6e07f2a7675b2dd8ff340a24823eb3808d2b07b3)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/670077
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=6e07f2a7675b2dd8ff340a24823eb3808d2b07b3
Submitter: Zuul
Branch: master

commit 6e07f2a7675b2dd8ff340a24823eb3808d2b07b3
Author: Kevin Carter <email address hidden>
Date: Wed Jul 10 09:28:07 2019 -0500

    Convert the heat json format to a py dict

    This change converts a heat json format option to a py dict within
    a jinja expresion.

    Closes-Bug: #1835657
    Related-Bug: #1833584
    Change-Id: I4b44214cd7007dc31ad5f4e0a0d7a3a531a9f20e
    Signed-off-by: Kevin Carter <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/670205
Committed: https://git.openstack.org/cgit/openstack/ansible-role-container-registry/commit/?id=1217799b1bef82dcd2a6c69eb795c398cbac0d1b
Submitter: Zuul
Branch: master

commit 1217799b1bef82dcd2a6c69eb795c398cbac0d1b
Author: Kevin Carter <email address hidden>
Date: Wed Jul 10 15:33:31 2019 -0500

    Add molecule testing

    This change adds molecule testing using a simple base job and pre|run playbooks.
    The test will be executed via a native zuul job and will ensure we're exercising
    all of the available code path's as provide by this role.

    Two molecule scenarios will be executed whenever any change is made to this role

    * default - runs through the typical main code path
    * login - tests a secure docker registry ensuring our login capabilities are
              never broken.

    Documentation in the readme has been added to show how local testing can be run.

    A bindep.txt file has been added to ensure zuul knows how to install our
    required base packages.

    Closes-Bug: #1835657
    Related-Bug: #1833584
    Change-Id: I48f74b69c5d29dce4a576fa96e79563a4b484469
    Signed-off-by: Kevin Carter <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/670349

[OpenStack Infra (hudson-openstack)]

Change abandoned by Emilien Macchi (<email address hidden>) on branch: master
Review: https://review.opendev.org/670083

[OpenStack Infra (hudson-openstack)]

Change abandoned by Emilien Macchi (<email address hidden>) on branch: stable/stein
Review: https://review.opendev.org/669577

[OpenStack Infra (hudson-openstack)]

Change abandoned by Emilien Macchi (<email address hidden>) on branch: master
Review: https://review.opendev.org/669576

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/670349
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=59e4b8140a22fadf5e7c8c2256cfee5c865755e1
Submitter: Zuul
Branch: stable/rocky

commit 59e4b8140a22fadf5e7c8c2256cfee5c865755e1
Author: Emilien Macchi <email address hidden>
Date: Wed Jul 10 10:46:25 2019 -0400

    Rocky: enable container auth support

    Squash of adding container enginene support and the subsequent fix

    1) Add container engine authentication support

    See I98a527f363056767fea45ab4828ae61c01de20ca. This is only the docker
    support as podman was added in Stein

    2) Convert the heat json format to a py dict

    This change converts a heat json format option to a py dict within
    a jinja expresion.

    Closes-Bug: #1835657
    Related-Bug: #1833584
    Change-Id: I4b44214cd7007dc31ad5f4e0a0d7a3a531a9f20e
    Signed-off-by: Kevin Carter <email address hidden>
    (cherry picked from commit 6e07f2a7675b2dd8ff340a24823eb3808d2b07b3)
    (cherry picked from commit d6bd20d5b46d7ede598e022f0d34814c8dbe8a66)

[Attila Fazekas (afazekas)]

Does not seams to work.

I am still getting the same issue.

This ad-hoc try allowed me to move forward (hit a completely different issue).

--- /tmp/orig 2019-07-12 10:54:02.349663562 +0000
+++ /usr/share/openstack-tripleo-heat-templates/deployment/podman/podman-baremetal-ansible.yaml 2019-07-12 10:54:35.774901254 +0000
@@ -110,7 +110,7 @@
               REGISTRY_USERNAME: "{{ lookup('dict', item.value).key }}"
               REGISTRY_PASSWORD: "{{ lookup('dict', item.value).value }}"
               REGISTRY: "{{ item.key }}"
- loop: "{{ query('dict', container_registry_logins) }}"
+ loop: "{{ lookup('dict', container_registry_logins is none | ternary({}, container_registry_logins | default({}))) }}"
             when:
               - container_registry_login | bool
               - container_registry_logins

[yatin (yatinkarel)]

stein Upgrade jobs are still failing:- http://logs.openstack.org/25/654925/9/check/tripleo-ci-centos-7-standalone-upgrade-stein/44390d5/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz#_2019-07-11_10_11_17

http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-standalone-upgrade-stein

Can u reiterate on the fixes done?

Also, The test patch https://review.opendev.org/#/c/670083/ where the fixes were tested didn't picked up the fixes via Depends-On(mixture of stable/stein and master, release: stein).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/670627

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/670663

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/670665

[OpenStack Infra (hudson-openstack)]

Change abandoned by Kevin Carter (cloudnull) (<email address hidden>) on branch: master
Review: https://review.opendev.org/670627

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/670663
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=a6ff00885ce0540015b0ed1fc99c431a87348077
Submitter: Zuul
Branch: master

commit a6ff00885ce0540015b0ed1fc99c431a87348077
Author: Alex Schultz <email address hidden>
Date: Fri Jul 12 14:55:35 2019 -0600

    Specify a default for container_registry_logins

    When we run this, we'll run the controller host_prep_tasks but the
    compute nodes are included. Ansible will skip the set_fact on the
    computes but still try to evaluate the lookup. We need to use |default
    to handle the unset case.

    Change-Id: I1ff41befbd66325d90c3ea18640a298549d1b659
    Closes-Bug: #1835657

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/670686

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/670686
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=2871ce0fa9ed658fb01179cda3c6522bcaa1121f
Submitter: Zuul
Branch: stable/stein

commit 2871ce0fa9ed658fb01179cda3c6522bcaa1121f
Author: Alex Schultz <email address hidden>
Date: Fri Jul 12 14:55:35 2019 -0600

    Specify a default for container_registry_logins

    When we run this, we'll run the controller host_prep_tasks but the
    compute nodes are included. Ansible will skip the set_fact on the
    computes but still try to evaluate the lookup. We need to use |default
    to handle the unset case.

    Change-Id: I1ff41befbd66325d90c3ea18640a298549d1b659
    Closes-Bug: #1835657
    (cherry picked from commit a6ff00885ce0540015b0ed1fc99c431a87348077)

[Attila Fazekas (afazekas)]

The undefined and set to null/none/None is two different thing . default typically helps in case of undefined.
I hope this change was tested before.
Before the other changes the `null` was one of the problematic case.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/670665
Committed: https://git.openstack.org/cgit/openstack/ansible-role-container-registry/commit/?id=884a8f6f1b45166a79a4eafbec2eb5156eff9eb4
Submitter: Zuul
Branch: master

commit 884a8f6f1b45166a79a4eafbec2eb5156eff9eb4
Author: Alex Schultz <email address hidden>
Date: Fri Jul 12 15:01:20 2019 -0600

    Specify a default for container_registry_logins

    When we run this, if we try to evaulate this without the variable being
    set it errors. We assume empty if the variable is unset.

    Change-Id: Ic6eea050cd627d9d09745bf194c2dd36b015e6ff
    Closes-Bug: #1835657

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 11.1.0 release.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.opendev.org/674955

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 10.6.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 9.4.1 release.