R5.0.1 - micro-services provisioning - contrail-ansible-deployer fails.

[Ritam Gangopadhyay]

Seeing provisioning error while running the configure instances play:-

ansible-playbook -i inventory/ -e orchestrator=openstack playbooks/configure_instances.yml

ERROR:-
*****************************************
*****************************************

TASK [configure_instances : Build hosts file for rabbitmq with domain suffix when provided] ********************************************************************************************************
skipping: [10.204.216.99] => (item=[u'10.204.216.103', u'10.204.216.99'])
skipping: [10.204.216.103] => (item=[u'10.204.216.103', u'10.204.216.99'])
skipping: [10.204.216.96] => (item=[u'10.204.216.103', u'10.204.216.99'])
skipping: [10.204.216.98] => (item=[u'10.204.216.103', u'10.204.216.99'])
skipping: [10.204.216.97] => (item=[u'10.204.216.103', u'10.204.216.99'])
ok: [10.204.216.103] => (item=[u'10.204.216.103', u'10.204.216.103'])
skipping: [10.204.216.103] => (item=[u'10.204.216.103', u'10.204.216.96'])
ok: [10.204.216.99] => (item=[u'10.204.216.103', u'10.204.216.103'])
fatal: [10.204.216.103]: FAILED! => {"msg": "The conditional check 'item.0 in hostvars[item.1].ansible_all_ipv4_addresses' failed. The error was: error while evaluating conditional (item.0 in hostvars[item.1].ansible_all_ipv4_addresses): Unable to look up a name or access an attribute in template string ({% if item.0 in hostvars[item.1].ansible_all_ipv4_addresses %} True {% else %} False {% endif %}).\nMake sure your variable name does not contain invalid characters like '-': argument of type 'StrictUndefined' is not iterable\n\nThe error appears to have been in '/tmp/ansible.5BU7Dz_contrail/contrail-ansible-deployer/playbooks/roles/configure_instances/tasks/install_software_Linux.yml': line 63, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: \"Build hosts file for rabbitmq with domain suffix when provided\"\n ^ here\n"}
skipping: [10.204.216.99] => (item=[u'10.204.216.103', u'10.204.216.96'])
fatal: [10.204.216.99]: FAILED! => {"msg": "The conditional check 'item.0 in hostvars[item.1].ansible_all_ipv4_addresses' failed. The error was: error while evaluating conditional (item.0 in hostvars[item.1].ansible_all_ipv4_addresses): Unable to look up a name or access an attribute in template string ({% if item.0 in hostvars[item.1].ansible_all_ipv4_addresses %} True {% else %} False {% endif %}).\nMake sure your variable name does not contain invalid characters like '-': argument of type 'StrictUndefined' is not iterable\n\nThe error appears to have been in '/tmp/ansible.5BU7Dz_contrail/contrail-ansible-deployer/playbooks/roles/configure_instances/tasks/install_software_Linux.yml': line 63, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: \"Build hosts file for rabbitmq with domain suffix when provided\"\n ^ here\n"}
ok: [10.204.216.96] => (item=[u'10.204.216.103', u'10.204.216.103'])
ok: [10.204.216.98] => (item=[u'10.204.216.103', u'10.204.216.103'])
skipping: [10.204.216.96] => (item=[u'10.204.216.103', u'10.204.216.96'])
skipping: [10.204.216.98] => (item=[u'10.204.216.103', u'10.204.216.96'])
ok: [10.204.216.97] => (item=[u'10.204.216.103', u'10.204.216.103'])
fatal: [10.204.216.96]: FAILED! => {"msg": "The conditional check 'item.0 in hostvars[item.1].ansible_all_ipv4_addresses' failed. The error was: error while evaluating conditional (item.0 in hostvars[item.1].ansible_all_ipv4_addresses): Unable to look up a name or access an attribute in template string ({% if item.0 in hostvars[item.1].ansible_all_ipv4_addresses %} True {% else %} False {% endif %}).\nMake sure your variable name does not contain invalid characters like '-': argument of type 'StrictUndefined' is not iterable\n\nThe error appears to have been in '/tmp/ansible.5BU7Dz_contrail/contrail-ansible-deployer/playbooks/roles/configure_instances/tasks/install_software_Linux.yml': line 63, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: \"Build hosts file for rabbitmq with domain suffix when provided\"\n ^ here\n"}
fatal: [10.204.216.98]: FAILED! => {"msg": "The conditional check 'item.0 in hostvars[item.1].ansible_all_ipv4_addresses' failed. The error was: error while evaluating conditional (item.0 in hostvars[item.1].ansible_all_ipv4_addresses): Unable to look up a name or access an attribute in template string ({% if item.0 in hostvars[item.1].ansible_all_ipv4_addresses %} True {% else %} False {% endif %}).\nMake sure your variable name does not contain invalid characters like '-': argument of type 'StrictUndefined' is not iterable\n\nThe error appears to have been in '/tmp/ansible.5BU7Dz_contrail/contrail-ansible-deployer/playbooks/roles/configure_instances/tasks/install_software_Linux.yml': line 63, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: \"Build hosts file for rabbitmq with domain suffix when provided\"\n ^ here\n"}
skipping: [10.204.216.97] => (item=[u'10.204.216.103', u'10.204.216.96'])
fatal: [10.204.216.97]: FAILED! => {"msg": "The conditional check 'item.0 in hostvars[item.1].ansible_all_ipv4_addresses' failed. The error was: error while evaluating conditional (item.0 in hostvars[item.1].ansible_all_ipv4_addresses): Unable to look up a name or access an attribute in template string ({% if item.0 in hostvars[item.1].ansible_all_ipv4_addresses %} True {% else %} False {% endif %}).\nMake sure your variable name does not contain invalid characters like '-': argument of type 'StrictUndefined' is not iterable\n\nThe error appears to have been in '/tmp/ansible.5BU7Dz_contrail/contrail-ansible-deployer/playbooks/roles/configure_instances/tasks/install_software_Linux.yml': line 63, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: \"Build hosts file for rabbitmq with domain suffix when provided\"\n ^ here\n"}
        to retry, use: --limit @/tmp/ansible.5BU7Dz_contrail/contrail-ansible-deployer/playbooks/configure_instances.retry

*****************************************
*****************************************

Instances.yaml
*****************************************
*****************************************

global_configuration:
   REGISTRY_PRIVATE_INSECURE: True
   CONTAINER_REGISTRY: 10.204.217.152:5000
provider_config:
  bms:
    domainsuffix: englab.juniper.net
    ntpserver: 10.204.217.158
    ssh_pwd: c0ntrail123
    ssh_user: root

instances:
  nodem10:
      ip: 10.204.216.99
      provider: bms
      roles:
          openstack_compute: null
          vrouter:
              PHYSICAL_INTERFACE: ens2f1
  nodem14:
      ip: 10.204.216.103
      provider: bms
      roles:
          analytics: null
          analytics_database: null
          config: null
          config_database: null
          control: null
          openstack: null
          webui: null
  nodem6:
      ip: 10.204.216.95
      provider: bms
      roles:
          analytics: null
          analytics_database: null
          config: null
          config_database: null
          control: null
          openstack: null
          webui: null
  nodem7:
      ip: 10.204.216.96
      provider: bms
      roles:
          analytics: null
          analytics_database: null
          config: null
          config_database: null
          control: null
          openstack: null
          webui: null
  nodem8:
      ip: 10.204.216.97
      provider: bms
      roles:
          openstack_compute: null
          vrouter:
              PHYSICAL_INTERFACE: bond0
  nodem9:
      ip: 10.204.216.98
      provider: bms
      roles:
          openstack_compute: null
          vrouter:
              PHYSICAL_INTERFACE: ens2f1

contrail_configuration:
  CONTRAIL_VERSION: ocata-5.0-198
  CLOUD_ORCHESTRATOR: openstack
  METADATA_PROXY_SECRET: c0ntrail123
  OPENSTACK_NODES: 10.204.216.103,10.204.216.95,10.204.216.96
  CONTROLLER_NODES: 10.204.216.103,10.204.216.95,10.204.216.96
  AUTH_MODE: keystone
  KEYSTONE_AUTH_ADMIN_PASSWORD: c0ntrail123
  KEYSTONE_AUTH_URL_VERSION: "/v3"
  RABBITMQ_NODE_PORT: 5673
  KEYSTONE_AUTH_HOST: 10.204.216.140
  IPFABRIC_SERVICE_HOST: 10.204.216.140
  AAA_MODE: rbac
  CLOUD_ORCHESTRATOR: openstack
  CONFIG_API_VIP: 10.204.216.103
  CONTAINER_REGISTRY: 10.204.217.152:5000
  CONTRAIL_VERSION: ocata-5.0-198
  CONTROLLER_NODES: 10.204.216.103,10.204.216.95,10.204.216.96
  CONTROL_NODES: 10.10.10.14,10.10.10.6,10.10.10.7
  ENCAP_PRIORITY: VXLAN,MPLSoUDP,MPLSoGRE
  IPFABRIC_SERVICE_HOST: 10.204.216.140
  KEYSTONE_AUTH_ADMIN_PASSWORD: c0ntrail123
  KEYSTONE_AUTH_HOST: 10.204.216.140
  KEYSTONE_AUTH_URL_VERSION: /v3
  OPENSTACK_NODES: 10.204.216.103,10.204.216.95,10.204.216.96
  RABBITMQ_NODE_PORT: 5673
  REGISTRY_PRIVATE_INSECURE: true
  VROUTER_GATEWAY: 10.10.10.101
  XMPP_SSL_ENABLE: true

kolla_config:
  kolla_globals:
    kolla_internal_vip_address: 10.204.216.140
    kolla_external_vip_address: 10.204.216.140
    contrail_api_interface_address: 10.204.216.103
    keepalived_virtual_router_id: 213
    enable_haproxy: yes
    docker_registry: docker.io
    docker_namespace: kolla
    enable_ironic: no
    enable_swift: "no"
  kolla_passwords:
    metadata_secret: c0ntrail123
    keystone_admin_password: c0ntrail123

[Ramprakash R (ramprakash)]

Please give access to the deployer node to debug this issue. We have observed this happens when there is an error for one of the hosts earlier in the play for some task. That is the root cause of the issue.

[Ritam Gangopadhyay (ritam)]

Deployer Node :: nodem14 - 10.204.216.103 - root/c0ntrail123

[root@nodem14 ~]# ls -la /tmp/ansible.5BU7Dz_contrail/contrail-ansible-deployer/
total 88
drwxr-xr-x 7 root root 4096 Aug 14 00:07 .
drwx------ 4 root root 4096 Aug 13 13:24 ..
-rw-r--r-- 1 root root 129 Aug 14 00:07 ansible.cfg
drwxr-xr-x 2 root root 4096 Aug 14 00:01 config
-rw-r--r-- 1 root root 10640 Aug 13 13:24 contrail_configuration.md
drwxr-xr-x 3 root root 4096 Aug 13 13:24 examples
drwxr-xr-x 8 root root 4096 Aug 13 13:24 .git
-rw-r--r-- 1 root root 52 Aug 13 13:24 .gitignore
-rw-r--r-- 1 root root 441 Aug 13 13:24 .gitreview
drwxr-xr-x 2 root root 4096 Aug 13 13:24 inventory
-rw-r--r-- 1 root root 11357 Aug 13 13:24 LICENSE
drwxr-xr-x 3 root root 4096 Aug 13 13:32 playbooks
-rw-r--r-- 1 root root 14126 Aug 13 13:24 README.md
-rw-r--r-- 1 root root 6274 Aug 13 13:24 README_vcenter.md
[root@nodem14 ~]#

[Ritam Gangopadhyay (ritam)]

As pointed out in comment #1, I see one of the node becomes unreachable -- nodem6.

[ritam@nodem3 ~]$ ping -c 3 nodem6
PING nodem6.englab.juniper.net (10.204.216.95) 56(84) bytes of data.
From nodem3.englab.juniper.net.204.10.in-addr.arpa (10.204.216.92) icmp_seq=1 Destination Host Unreachable
From nodem3.englab.juniper.net.204.10.in-addr.arpa (10.204.216.92) icmp_seq=2 Destination Host Unreachable
From nodem3.englab.juniper.net.204.10.in-addr.arpa (10.204.216.92) icmp_seq=3 Destination Host Unreachable

--- nodem6.englab.juniper.net ping statistics ---
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2047ms
pipe 3
[ritam@nodem3 ~]$

[Ramprakash R (ramprakash)]

This error should go away when the reachability for 10.204.216.95 is restored. The task that fails is trying to create /etc/hosts entries for all the hosts in the cluster but it is not able to gather facts for one of the hosts and thus fails.

%<------------------------
2018-08-14 00:12:03,277 p=33984 u=root | PLAY RECAP *****************************************************************************************************************************************************************************************
2018-08-14 00:12:03,278 p=33984 u=root | 10.204.216.103 : ok=24 changed=6 unreachable=0 failed=1
2018-08-14 00:12:03,278 p=33984 u=root | 10.204.216.95 : ok=0 changed=0 unreachable=1 failed=0
2018-08-14 00:12:03,278 p=33984 u=root | 10.204.216.96 : ok=24 changed=5 unreachable=0 failed=1
2018-08-14 00:12:03,278 p=33984 u=root | 10.204.216.97 : ok=26 changed=5 unreachable=0 failed=1
2018-08-14 00:12:03,278 p=33984 u=root | 10.204.216.98 : ok=26 changed=5 unreachable=0 failed=1
2018-08-14 00:12:03,278 p=33984 u=root | 10.204.216.99 : ok=26 changed=5 unreachable=0 failed=1
2018-08-14 00:12:03,279 p=33984 u=root | localhost : ok=19 changed=2 unreachable=0 failed=0
------------------------->%

[Ritam Gangopadhyay (ritam)]

Marking the bug invalid because the issue was seen due to unreachability of nodem6. Once nodem6 became reachable provisioning worked.