ceph-ansible - msg: 'AnsibleUndefinedVariable: ''dict object'' has no attribute ''split'' on TASK ==> generate ceph osd docker run script

[Cagri Ersen]

Description
===========

I have an AnsibleUndefinedVariable issue while trying to deploy, containarized ceph on a openstack (HCI) installation through tripleO.

I just passed these two environments to in order to deploty ceph via ceph-ansible:

-e ~/custom-tripleo-heat-templates-generated/environments/ceph-ansible/ceph-ansible.yaml
-e ~/custom-tripleo-heat-templates-generated/environments/ceph-ansible/ceph-rgw.yaml \

However, I got the errors for some of my nodes (not all) below,

```
2020-01-21 23:01:03,543 p=890833 u=root | TASK [ceph-osd : generate ceph osd docker run script] **************************
2020-01-21 23:01:03,543 p=890833 u=root | Tuesday 21 January 2020 23:01:03 +0300 (0:00:01.415) 0:24:14.961 *******
2020-01-21 23:01:03,749 p=890833 u=root | fatal: [computehci-6]: FAILED! => changed=false
  msg: 'AnsibleUndefinedVariable: ''dict object'' has no attribute ''split'''
2020-01-21 23:01:03,840 p=890833 u=root | fatal: [computehci-7]: FAILED! => changed=false
  msg: 'AnsibleUndefinedVariable: ''dict object'' has no attribute ''split'''
2020-01-21 23:01:03,936 p=890833 u=root | fatal: [computehci-4]: FAILED! => changed=false
  msg: 'AnsibleUndefinedVariable: ''dict object'' has no attribute ''split'''
2020-01-21 23:01:04,039 p=890833 u=root | fatal: [computehci-5]: FAILED! => changed=false
  msg: 'AnsibleUndefinedVariable: ''dict object'' has no attribute ''split'''
2020-01-21 23:01:04,131 p=890833 u=root | fatal: [computehci-2]: FAILED! => changed=false
  msg: 'AnsibleUndefinedVariable: ''dict object'' has no attribute ''split'''
2020-01-21 23:01:04,232 p=890833 u=root | fatal: [computehci-3]: FAILED! => changed=false
  msg: 'AnsibleUndefinedVariable: ''dict object'' has no attribute ''split'''
2020-01-21 23:01:04,238 p=890833 u=root | fatal: [computehci-0]: FAILED! => changed=false
  msg: 'AnsibleUndefinedVariable: ''dict object'' has no attribute ''split'''
2020-01-21 23:01:04,340 p=890833 u=root | fatal: [computehci-1]: FAILED! => changed=false
  msg: 'AnsibleUndefinedVariable: ''dict object'' has no attribute ''split'''
```

Steps to reproduce
==================
Install undercloud (stein)
Prepare environment files (ceph custom config:)

```
parameter_defaults:
  CephConfigOverrides:
    mon_max_pg_per_osd: 3072
    journal_size: 5120
    osd_pool_default_size: 3
    osd_pool_default_min_size: 2
    osd_pool_default_pg_num: 128
    osd_pool_default_pgp_num: 128
  CephAnsibleDisksConfig:
    osd_scenario: lvm
    osd_objectstore: bluestore
    devices:
      - /dev/disk/by-path/pci-0000:01:00.0-sas-0x4433221100000000-lun-0
      - /dev/disk/by-path/pci-0000:01:00.0-sas-0x4433221101000000-lun-0
      - /dev/disk/by-path/pci-0000:01:00.0-sas-0x4433221102000000-lun-0
      - /dev/disk/by-path/pci-0000:01:00.0-sas-0x4433221103000000-lun-0
      - /dev/disk/by-path/pci-0000:01:00.0-sas-0x4433221104000000-lun-0
      - /dev/disk/by-path/pci-0000:01:00.0-sas-0x4433221105000000-lun-0

  CephPools:
    - {"name": .rgw.root, "pg_num": 16, "pgp_num": 16, "application": rados}
    - {"name": default.rgw.control, "pg_num": 16, "pgp_num": 16, "application": rados}
    - {"name": default.rgw.meta, "pg_num": 16, "pgp_num": 16, "application": rados}
    - {"name": default.rgw.log, "pg_num": 16, "pgp_num": 16, "application": rados}
    - {"name": images, "pg_num": 128, "pgp_num": 128, "application": rbd}
    - {"name": metrics, "pg_num": 16, "pgp_num": 16, "application":openstack_gnocchi}
    - {"name": backups, "pg_num": 512, "pgp_num": 512, "application": rbd}
    - {"name": vms, "pg_num": 256, "pgp_num": 256, "application": rbd}
    - {"name": volumes, "pg_num": 1024, "pgp_num": 1024, "application": rbd}
  CephPoolDefaultPgNum: 128
```

and deploy overcloud via:

```
openstack overcloud deploy \
    --timeout 120 \
    --templates \
    -r ~/templates/roles_data.yaml \
    -n ~/templates/network_data.yaml \
    -e ~/custom-tripleo-heat-templates-generated/environments/docker-ha.yaml \
    -e ~/custom-tripleo-heat-templates-generated/environments/network-isolation.yaml \
    -e ~/custom-tripleo-heat-templates-generated/environments/network-environment.yaml \
    -e ~/custom-tripleo-heat-templates-generated/environments/services/neutron-ovn-dvr-ha.yaml \
    -e ~/templates/nic-mapping.yaml \
    -e ~/templates/network.yaml \
    -e ~/templates/node-info.yaml \
    -e ~/templates/scheduler_hints_env.yaml \
    -e ~/templates/ips-from-pool-all.yaml \
    -e ~/templates/fixed-ip-vips.yaml \
    -e ~/templates/ceph-custom-config.yaml \
    -e ~/custom-tripleo-heat-templates-generated/environments/ceph-ansible/ceph-ansible.yaml \
    -e ~/custom-tripleo-heat-templates-generated/environments/ceph-ansible/ceph-rgw.yaml \
    -e ~/custom-tripleo-heat-templates-generated/environments/disable-telemetry.yaml \
    -e ~/custom-tripleo-heat-templates-generated/environments/services/mistral.yaml \
    -e ~/templates/misc-settings.yaml \
    -e ~/templates/timezone.yaml
```

Expected result
===============

ceph-ansible should deploy the ceph cluster

Actual result
=============

Fails with the error above

Environment
===========
1. Stein

2 ceph-ansible version: ceph-ansible-4.0.10-1.el7.noarch
  Ceph version: v4.0.8-stable-4.0-nautilus-centos-7-x86_64
  Ansible version: 2.8.6

3. OVN

[Cagri Ersen (cagri-ersen)]

** UPDATE **

I have also opened a issue on ceph-ansible project's github page and I got a reasonable answer. It seems this related to TripleO. If you make a HCI deployment then, this issue will occure.

Please check the issue[1] to get the point.
[1] https://github.com/ceph/ceph-ansible/issues/4970#issuecomment-577334186

[wes hayutin (weshayutin)]

@John, can you please help me direct this bug to right place. Thank you!

[John Fulton (jfulton-org)]

HI Cagri,

- TripleO CI standalone scenari001/004 use HCI
- This issue doesn't impact HCI when using podman
- In Stein and newer we dropped support for docker in place of podman

So I recommend you simply use podman. If it still doesn't work with podman then please let us know.

I'm going to close this as it's an old issue at this point.

[John Fulton (jfulton-org)]

I'm setting this wont fix because tripleo stein hci deployments are not going to support docker though the support podman instead.