fatal: [openstack002] => One or more undefined variables: 'dict object' has no attribute 'stdout'

[Chandra]

Hi,

i am getting error while deploying multi node . branch is liberty.

** getting error when my config is as below **

[control]
openstack002

[network]
openstack002

[compute]
openstack003

here are logs:

http://paste.openstack.org/show/480181/

http://paste.openstack.org/show/480252/

http://paste.openstack.org/show/480253/

*** its works when my config file is as below **

[control]
openstack002
openstack003

[network]
openstack002
openstack003

[compute]
openstack002
openstack003

if i use single node for each one it fails. but when i use 2 nodes for each then it works.

[Chandra (reddydodda)]

Update :

i am using ubuntu and source

[Chandra (reddydodda)]

Here are new log files :

global.yml
http://paste.openstack.org/show/480299/

interface
http://paste.openstack.org/show/480301/

site.yml
http://paste.openstack.org/show/480303/

deploy logs:
http://paste.openstack.org/show/480304/
http://paste.openstack.org/show/480305/

[yuanying (yuanying)]

Below link is a useful reference.

* https://review.openstack.org/#/c/242777/7/ansible/roles/neutron/tasks/main.yml

I give +1 to Hui's comment.

[Liyi Meng (liyi-meng)]

I got the same error on master, which has no change 242777 included.

[weiyu (weiyu)]

This is ansible bug, when use run_once, delegate_to is not available, only run task on first host.
https://github.com/ansible/ansible/issues/9784 This is ansible issue,but it not fixed.

my debug.yam

- hosts: all

  tasks:

  - name: debug for run_once and delegate_to
    debug: msg=success
    run_once: True
    delegate_to: "{{ groups['neutron-server'][0] }}"
    when: inventory_hostname in groups['neutron-server']

# ansible-playbook -i multinode debug.yaml

PLAY [all] ********************************************************************

GATHERING FACTS ***************************************************************
weiyu_debug: delegate_to hosts is None
ok: [control01]
ok: [control02]
ok: [control03]
ok: [network01]
ok: [storage01]
ok: [compute01]

TASK: [debug for run_once and delegate_to] ************************************
skipping: [compute01]

PLAY RECAP ********************************************************************
compute01 : ok=1 changed=0 unreachable=0 failed=0
control01 : ok=1 changed=0 unreachable=0 failed=0
control02 : ok=1 changed=0 unreachable=0 failed=0
control03 : ok=1 changed=0 unreachable=0 failed=0
network01 : ok=1 changed=0 unreachable=0 failed=0
storage01 : ok=1 changed=0 unreachable=0 failed=0

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/271148

[weiyu (weiyu)]

This bug of ansible will fixed has been fixed in devel and will be included in 2.0.1. It will most likely not be back-ported to 1.9.
https://github.com/ansible/ansible/issues/9784

[Liyi Meng (liyi-meng)]

But Kolla dose not support ansible 2.0, right. I try it and fail miserably.

[Ross Krumbeck (rkrum)]

I have the same problem with multinode deployment. I was able to get this working
by updating the neutron bootstrap.yml to use 'when' to control which host the commands are run on rather than 'run_once' with 'delegate_to'.

---
- name: Creating Neutron database
  command: docker exec -t kolla_ansible /usr/bin/ansible localhost
    -m mysql_db
    -a "login_host='{{ database_address }}'
        login_user='{{ database_user }}'
        login_password='{{ database_password }}'
        name='{{ neutron_database_name }}'"
  register: database
  changed_when: "{{ database.stdout.find('localhost | SUCCESS => ') != -1 and
                    (database.stdout.split('localhost | SUCCESS => ')[1]|from_json).changed }}"
  failed_when: database.stdout.split()[2] != 'SUCCESS'
  delegate_to: "{{ groups['neutron-server'][0] }}"

- name: Reading json from variable
  set_fact:
    database_created: "{{ (database.stdout.split('localhost | SUCCESS => ')[1]|from_json).changed }}"

- name: Creating Neutron database user and setting permissions
  command: docker exec -t kolla_ansible /usr/bin/ansible localhost
    -m mysql_user
    -a "login_host='{{ database_address }}'
        login_user='{{ database_user }}'
        login_password='{{ database_password }}'
        name='{{ neutron_database_name }}'
        password='{{ neutron_database_password }}'
        host='%'
        priv='{{ neutron_database_name }}.*:ALL'
        append_privs='yes'"
  register: database_user_create
  changed_when: "{{ database_user_create.stdout.find('localhost | SUCCESS => ') != -1 and
                    (database_user_create.stdout.split('localhost | SUCCESS => ')[1]|from_json).changed }}"
  failed_when: database_user_create.stdout.split()[2] != 'SUCCESS'
  when: database_created | bool and inventory_hostname == groups['neutron-server'][0]

- name: Starting Neutron bootstrap container
  docker:
    tty: True
    detach: False
    net: host
    pull: "{{ docker_pull_policy }}"
    restart_policy: "no"
    state: reloaded
    registry: "{{ docker_registry }}"
    username: "{{ docker_registry_username }}"
    password: "{{ docker_registry_password }}"
    insecure_registry: "{{ docker_insecure_registry }}"
    name: bootstrap_neutron
    image: "{{ neutron_server_image_full }}"
    volumes: "{{ node_config_directory }}/neutron-server/:{{ container_config_directory }}/:ro"
    env:
      KOLLA_BOOTSTRAP:
      KOLLA_CONFIG_STRATEGY: "{{ config_strategy }}"
  when: database_created | bool and inventory_hostname == groups['neutron-server'][0]

# https://github.com/ansible/ansible-modules-core/pull/1031
- name: Waiting for bootstrap container to exit
  command: docker wait bootstrap_neutron
  register: bootstrap_result
  failed_when: bootstrap_result.stdout != "0"
  when: database_created | bool and inventory_hostname == groups['neutron-server'][0]

- name: Cleaning up Neutron bootstrap container
  docker:
    name: bootstrap_neutron
    image: "{{ neutron_server_image_full }}"
    state: absent
  delegate_to: "{{ groups['neutron-server'][0] }}"
  when: database_created | bool and inventory_hostname == groups['neutron-server'][0]

[Ross Krumbeck (rkrum)]

Sorry about the formatting you can get it from here:

http://pastebin.com/Q5XqXpLF

[Ross Krumbeck (rkrum)]

I've updated the fix, note I should have mentioned this and the previous ones are for kolla liberty:

http://paste.openstack.org/show/485355/

[Hui Kang (huikang27)]

This issue has been dangling for a while. For new users, a quick workaround is to comment out this two lines

https://github.com/openstack/kolla/blob/master/ansible/roles/neutron/tasks/deploy.yml#L6

https://github.com/openstack/kolla/blob/master/ansible/roles/neutron/tasks/deploy.yml#L19

[Liyi Meng (liyi-meng)]

Hi Hui,

Dose this workaround work for multiple controller nodes? I deploy with tree controller nodes after apply your workaround, only one of the controller node get full deployed, other two seem beening skipped in some way. Is this expected?

Any reply is appreciated!

/Liyi

[Hui Kang (huikang27)]

you can add ignore error to avoid skipping the error. Since neutron has already registered, it should not be harmful. I know it causes too much hacking, but you can use it as a temporary solution.

> On Jan 29, 2016, at 8:02 AM, Liyi Meng <email address hidden> wrote:
>
> Hi Hui,
>
> Dose this workaround work for multiple controller nodes? I deploy with
> tree controller nodes after apply your workaround, only one of the
> controller node get full deployed, other two seem beening skipped in
> some way. Is this expected?
>
> Any reply is appreciated!
>
> /Liyi
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1520728
>
> Title:
> fatal: [openstack002] => One or more undefined variables: 'dict
> object' has no attribute 'stdout'
>
> Status in kolla:
> In Progress
>
> Bug description:
> Hi,
>
> i am getting error while deploying multi node . branch is liberty.
>
> ** getting error when my config is as below **
>
> [control]
> openstack002
>
> [network]
> openstack002
>
> [control]
> openstack003
>
>
> here are logs:
>
> http://paste.openstack.org/show/480181/
>
> http://paste.openstack.org/show/480252/
>
> http://paste.openstack.org/show/480253/
>
>
> *** its works when my config file is as below **
>
> [control]
> openstack002
> openstack003
>
> [network]
> openstack002
> openstack003
>
> [compute]
> openstack002
> openstack003
>
>
> if i use single node for each one it fails. but when i use 2 nodes for each then it works.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/kolla/+bug/1520728/+subscriptions

[Steven Dake (sdake)]

moving this to rc1. Seems like some easy ansible changes can workaround this bug.

[Carlos Cesario (ccesario)]

Only to report:

  - With ansible 1.9.5-rc1 this problem still happen.

[Sam Yaple (s8m)]

This will be handled only in the switch to Ansible 2.0.

It will not be fixed in Liberty or Mitaka unfortunately since it is an Ansible bug

[OpenStack Infra (hudson-openstack)]

Change abandoned by weiyu (<email address hidden>) on branch: master
Review: https://review.openstack.org/271148

[Steven Dake (sdake)]

I believe this is a duplicate of https://bugs.launchpad.net/kolla/+bug/1546789 which is fixed in master and stable/mitaka. Could folks chime in if they still see this problem with either of those two branches?

Thanks
-steve

[Steven Dake (sdake)]

Note if this still reproducible on master with the neutron reorder, we absolutely must find a workaround, otherwise 50% of the field will be unable to deploy Kolla.

Hui,
If you can get me a list of bug ids to backport to ansible which actually fix this problem, I can get them backported in 1.9.5-rc1.

Regards
-steve

[Steven Dake (sdake)]

Moving back to triaged - have not heard back if this is still reproducible with stable/mitaka or master.

[Swapnil Kulkarni (coolsvap-deactivatedaccount)]

Is the bug still valid? Please confirm

[km (kmlau777)]

Just curious, the inventory from op is looking little suspicious:

Why are there two control groups?

<quote>
** getting error when my config is as below **

[control]
openstack002

[network]
openstack002

[control]
openstack003
</quote>

[Vladislav Belogrudov (vlad-belogrudov)]

Ansible does not support such inventory syntax, it is not Kolla issue.

[Guillaume HARVEY (guillaume-harvey)]

I can see that their is 2 [control] sections and no [compute] section. Is this a typo or is it on purpose? If it's on purpose, why?

[Chandra (reddydodda)]

Hi ,

That was typo. please ignore it.

Thanks,
Chandra.

On Wed, Oct 26, 2016 at 5:25 PM, Guillaume HARVEY <email address hidden>
wrote:

> I can see that their is 2 [control] sections and no [compute] section.
> Is this a typo or is it on purpose? If it's on purpose, why?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1520728
>
> Title:
> fatal: [openstack002] => One or more undefined variables: 'dict
> object' has no attribute 'stdout'
>
> Status in kolla:
> Triaged
> Status in kolla liberty series:
> Triaged
> Status in kolla mitaka series:
> Triaged
>
> Bug description:
> Hi,
>
> i am getting error while deploying multi node . branch is liberty.
>
> ** getting error when my config is as below **
>
> [control]
> openstack002
>
> [network]
> openstack002
>
> [control]
> openstack003
>
>
> here are logs:
>
> http://paste.openstack.org/show/480181/
>
> http://paste.openstack.org/show/480252/
>
> http://paste.openstack.org/show/480253/
>
>
> *** its works when my config file is as below **
>
> [control]
> openstack002
> openstack003
>
> [network]
> openstack002
> openstack003
>
> [compute]
> openstack002
> openstack003
>
>
> if i use single node for each one it fails. but when i use 2 nodes for
> each then it works.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/kolla/+bug/1520728/+subscriptions
>

--
chandu

[Jeffrey Zhang (jeffrey4l)]

This is not reported recently. mark it as invalid, and feel free to re-open it.