Deploying Train fails on nova-consoleauth

[Viorel-Cosmin Miron]

We running Stein while upgrading to Train, and using a fresh precheck and bootstrap with deploy we receive continually this issue.

Moving back to Stein this works out of the box, without altering anything on our side.

TASK [nova : Copying over nova.conf] *****************************************************************************************************************
skipping: [ss] => (item={'key': 'nova-libvirt', 'value': {'container_name': 'nova_libvirt', 'group': 'compute', 'enabled': True, 'image': 'kolla/ubuntu-source-nova-libvirt:train', 'pid_mode': 'host', 'privileged': True, 'volumes': ['/etc/kolla//nova-libvirt/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/lib/modules:/lib/modules:ro', '/run/:/run/:shared', '/dev:/dev', '/sys/fs/cgroup:/sys/fs/cgroup', 'kolla_logs:/var/log/kolla/', 'libvirtd:/var/lib/libvirt', 'nova_compute:/var/lib/nova/', '', 'nova_libvirt_qemu:/etc/libvirt/qemu', ''], 'dimensions': {}}})
skipping: [ss] => (item={'key': 'nova-ssh', 'value': {'container_name': 'nova_ssh', 'group': 'compute', 'image': 'kolla/ubuntu-source-nova-ssh:train', 'enabled': True, 'volumes': ['/etc/kolla//nova-ssh/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', 'kolla_logs:/var/log/kolla', 'nova_compute:/var/lib/nova', '', ''], 'dimensions': {}}})
changed: [ss] => (item={'key': 'nova-api', 'value': {'container_name': 'nova_api', 'group': 'nova-api', 'image': 'kolla/ubuntu-source-nova-api:train', 'enabled': True, 'privileged': True, 'volumes': ['/etc/kolla//nova-api/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/lib/modules:/lib/modules:ro', 'kolla_logs:/var/log/kolla/', ''], 'dimensions': {}, 'haproxy': {'nova_api': {'enabled': True, 'mode': 'http', 'external': False, 'port': '8774', 'listen_port': '8774'}, 'nova_api_external': {'enabled': True, 'mode': 'http', 'external': True, 'port': '8774', 'listen_port': '8774'}, 'nova_metadata': {'enabled': True, 'mode': 'http', 'external': False, 'port': '8775', 'listen_port': '8775'}, 'nova_metadata_external': {'enabled': True, 'mode': 'http', 'external': True, 'port': '8775', 'listen_port': '8775'}, 'nova_rdp': {'enabled': False, 'mode': 'http', 'external': False, 'port': '8001', 'host_group': 'hyperv'}}}})
changed: [ss] => (item={'key': 'nova-consoleauth', 'value': {'container_name': 'nova_consoleauth', 'group': 'nova-consoleauth', 'image': 'kolla/ubuntu-source-nova-consoleauth:train', 'enabled': True, 'volumes': ['/etc/kolla//nova-consoleauth/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', 'kolla_logs:/var/log/kolla/', ''], 'dimensions': {}}})
changed: [ss] => (item={'key': 'nova-novncproxy', 'value': {'container_name': 'nova_novncproxy', 'group': 'nova-novncproxy', 'image': 'kolla/ubuntu-source-nova-novncproxy:train', 'enabled': True, 'volumes': ['/etc/kolla//nova-novncproxy/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', 'kolla_logs:/var/log/kolla/', ''], 'dimensions': {}, 'haproxy': {'nova_novncproxy': {'enabled': True, 'mode': 'http', 'external': False, 'port': '6080', 'listen_port': '6080', 'backend_http_extra': ['timeout tunnel 1h']}, 'nova_novncproxy_external': {'enabled': True, 'mode': 'http', 'external': True, 'port': '6080', 'listen_port': '6080'}}}})
changed: [ss] => (item={'key': 'nova-scheduler', 'value': {'container_name': 'nova_scheduler', 'group': 'nova-scheduler', 'image': 'kolla/ubuntu-source-nova-scheduler:train', 'enabled': True, 'volumes': ['/etc/kolla//nova-scheduler/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', 'kolla_logs:/var/log/kolla/', ''], 'dimensions': {}}})
skipping: [ss] => (item={'key': 'nova-spicehtml5proxy', 'value': {'container_name': 'nova_spicehtml5proxy', 'group': 'nova-spicehtml5proxy', 'image': 'kolla/ubuntu-source-nova-spicehtml5proxy:train', 'enabled': False, 'volumes': ['/etc/kolla//nova-spicehtml5proxy/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', 'kolla_logs:/var/log/kolla/', ''], 'dimensions': {}, 'haproxy': {'nova_spicehtml5proxy': {'enabled': False, 'mode': 'http', 'external': False, 'port': '6082', 'listen_port': '6082'}, 'nova_spicehtml5proxy_external': {'enabled': False, 'mode': 'http', 'external': True, 'port': '6082', 'listen_port': '6082'}}}})
skipping: [ss] => (item={'key': 'nova-serialproxy', 'value': {'container_name': 'nova_serialproxy', 'group': 'nova-serialproxy', 'image': 'kolla/ubuntu-source-nova-serialproxy:train', 'enabled': False, 'volumes': ['/etc/kolla//nova-serialproxy/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', 'kolla_logs:/var/log/kolla/', ''], 'dimensions': {}, 'haproxy': {'nova_serialconsole_proxy': {'enabled': False, 'mode': 'http', 'external': False, 'port': '6083', 'listen_port': '6083', 'backend_http_extra': ['timeout tunnel 10m']}, 'nova_serialconsole_proxy_external': {'enabled': False, 'mode': 'http', 'external': True, 'port': '6083', 'listen_port': '6083', 'backend_http_extra': ['timeout tunnel 10m']}}}})
changed: [ss] => (item={'key': 'nova-conductor', 'value': {'container_name': 'nova_conductor', 'group': 'nova-conductor', 'enabled': True, 'image': 'kolla/ubuntu-source-nova-conductor:train', 'volumes': ['/etc/kolla//nova-conductor/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', 'kolla_logs:/var/log/kolla/', ''], 'dimensions': {}}})
changed: [ss] => (item={'key': 'nova-compute', 'value': {'container_name': 'nova_compute', 'group': 'compute', 'image': 'kolla/ubuntu-source-nova-compute:train', 'environment': {'LIBGUESTFS_BACKEND': 'direct'}, 'privileged': True, 'enabled': True, 'ipc_mode': 'host', 'volumes': ['/etc/kolla//nova-compute/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/lib/modules:/lib/modules:ro', '/run:/run:shared', '/dev:/dev', 'kolla_logs:/var/log/kolla/', 'iscsi_info:/etc/iscsi', 'libvirtd:/var/lib/libvirt', 'nova_compute:/var/lib/nova/', '', ''], 'dimensions': {}}})
skipping: [ss] => (item={'key': 'nova-compute-ironic', 'value': {'container_name': 'nova_compute_ironic', 'group': 'nova-compute-ironic', 'image': 'kolla/ubuntu-source-nova-compute-ironic:train', 'enabled': False, 'volumes': ['/etc/kolla//nova-compute-ironic/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', 'kolla_logs:/var/log/kolla/', ''], 'dimensions': {}}})

TASK [nova : Copying over libvirt configuration] *****************************************************************************************************
changed: [ss] => (item={'src': 'qemu.conf.j2', 'dest': 'qemu.conf'})
changed: [ss] => (item={'src': 'libvirtd.conf.j2', 'dest': 'libvirtd.conf'})

TASK [nova : Copying files for nova-ssh] *************************************************************************************************************
changed: [ss] => (item={'src': 'sshd_config.j2', 'dest': 'sshd_config'})
changed: [ss] => (item={'src': 'id_rsa', 'dest': 'id_rsa'})
changed: [ss] => (item={'src': 'id_rsa.pub', 'dest': 'id_rsa.pub'})
changed: [ss] => (item={'src': 'ssh_config.j2', 'dest': 'ssh_config'})

TASK [nova : Copying VMware vCenter CA file] *********************************************************************************************************
skipping: [ss]

TASK [nova : Copying 'release' file for nova_compute] ************************************************************************************************

TASK [nova : Copying over existing policy file] ******************************************************************************************************
skipping: [ss] => (item={'key': 'nova-libvirt', 'value': {'container_name': 'nova_libvirt', 'group': 'compute', 'enabled': True, 'image': 'kolla/ubuntu-source-nova-libvirt:train', 'pid_mode': 'host', 'privileged': True, 'volumes': ['/etc/kolla//nova-libvirt/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/lib/modules:/lib/modules:ro', '/run/:/run/:shared', '/dev:/dev', '/sys/fs/cgroup:/sys/fs/cgroup', 'kolla_logs:/var/log/kolla/', 'libvirtd:/var/lib/libvirt', 'nova_compute:/var/lib/nova/', '', 'nova_libvirt_qemu:/etc/libvirt/qemu', ''], 'dimensions': {}}})
skipping: [ss] => (item={'key': 'nova-ssh', 'value': {'container_name': 'nova_ssh', 'group': 'compute', 'image': 'kolla/ubuntu-source-nova-ssh:train', 'enabled': True, 'volumes': ['/etc/kolla//nova-ssh/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', 'kolla_logs:/var/log/kolla', 'nova_compute:/var/lib/nova', '', ''], 'dimensions': {}}})
skipping: [ss] => (item={'key': 'nova-api', 'value': {'container_name': 'nova_api', 'group': 'nova-api', 'image': 'kolla/ubuntu-source-nova-api:train', 'enabled': True, 'privileged': True, 'volumes': ['/etc/kolla//nova-api/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/lib/modules:/lib/modules:ro', 'kolla_logs:/var/log/kolla/', ''], 'dimensions': {}, 'haproxy': {'nova_api': {'enabled': True, 'mode': 'http', 'external': False, 'port': '8774', 'listen_port': '8774'}, 'nova_api_external': {'enabled': True, 'mode': 'http', 'external': True, 'port': '8774', 'listen_port': '8774'}, 'nova_metadata': {'enabled': True, 'mode': 'http', 'external': False, 'port': '8775', 'listen_port': '8775'}, 'nova_metadata_external': {'enabled': True, 'mode': 'http', 'external': True, 'port': '8775', 'listen_port': '8775'}, 'nova_rdp': {'enabled': False, 'mode': 'http', 'external': False, 'port': '8001', 'host_group': 'hyperv'}}}})
skipping: [ss] => (item={'key': 'nova-consoleauth', 'value': {'container_name': 'nova_consoleauth', 'group': 'nova-consoleauth', 'image': 'kolla/ubuntu-source-nova-consoleauth:train', 'enabled': True, 'volumes': ['/etc/kolla//nova-consoleauth/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', 'kolla_logs:/var/log/kolla/', ''], 'dimensions': {}}})
skipping: [ss] => (item={'key': 'nova-novncproxy', 'value': {'container_name': 'nova_novncproxy', 'group': 'nova-novncproxy', 'image': 'kolla/ubuntu-source-nova-novncproxy:train', 'enabled': True, 'volumes': ['/etc/kolla//nova-novncproxy/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', 'kolla_logs:/var/log/kolla/', ''], 'dimensions': {}, 'haproxy': {'nova_novncproxy': {'enabled': True, 'mode': 'http', 'external': False, 'port': '6080', 'listen_port': '6080', 'backend_http_extra': ['timeout tunnel 1h']}, 'nova_novncproxy_external': {'enabled': True, 'mode': 'http', 'external': True, 'port': '6080', 'listen_port': '6080'}}}})
skipping: [ss] => (item={'key': 'nova-scheduler', 'value': {'container_name': 'nova_scheduler', 'group': 'nova-scheduler', 'image': 'kolla/ubuntu-source-nova-scheduler:train', 'enabled': True, 'volumes': ['/etc/kolla//nova-scheduler/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', 'kolla_logs:/var/log/kolla/', ''], 'dimensions': {}}})
skipping: [ss] => (item={'key': 'nova-spicehtml5proxy', 'value': {'container_name': 'nova_spicehtml5proxy', 'group': 'nova-spicehtml5proxy', 'image': 'kolla/ubuntu-source-nova-spicehtml5proxy:train', 'enabled': False, 'volumes': ['/etc/kolla//nova-spicehtml5proxy/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', 'kolla_logs:/var/log/kolla/', ''], 'dimensions': {}, 'haproxy': {'nova_spicehtml5proxy': {'enabled': False, 'mode': 'http', 'external': False, 'port': '6082', 'listen_port': '6082'}, 'nova_spicehtml5proxy_external': {'enabled': False, 'mode': 'http', 'external': True, 'port': '6082', 'listen_port': '6082'}}}})
skipping: [ss] => (item={'key': 'nova-serialproxy', 'value': {'container_name': 'nova_serialproxy', 'group': 'nova-serialproxy', 'image': 'kolla/ubuntu-source-nova-serialproxy:train', 'enabled': False, 'volumes': ['/etc/kolla//nova-serialproxy/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', 'kolla_logs:/var/log/kolla/', ''], 'dimensions': {}, 'haproxy': {'nova_serialconsole_proxy': {'enabled': False, 'mode': 'http', 'external': False, 'port': '6083', 'listen_port': '6083', 'backend_http_extra': ['timeout tunnel 10m']}, 'nova_serialconsole_proxy_external': {'enabled': False, 'mode': 'http', 'external': True, 'port': '6083', 'listen_port': '6083', 'backend_http_extra': ['timeout tunnel 10m']}}}})
skipping: [ss] => (item={'key': 'nova-conductor', 'value': {'container_name': 'nova_conductor', 'group': 'nova-conductor', 'enabled': True, 'image': 'kolla/ubuntu-source-nova-conductor:train', 'volumes': ['/etc/kolla//nova-conductor/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', 'kolla_logs:/var/log/kolla/', ''], 'dimensions': {}}})
skipping: [ss] => (item={'key': 'nova-compute', 'value': {'container_name': 'nova_compute', 'group': 'compute', 'image': 'kolla/ubuntu-source-nova-compute:train', 'environment': {'LIBGUESTFS_BACKEND': 'direct'}, 'privileged': True, 'enabled': True, 'ipc_mode': 'host', 'volumes': ['/etc/kolla//nova-compute/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/lib/modules:/lib/modules:ro', '/run:/run:shared', '/dev:/dev', 'kolla_logs:/var/log/kolla/', 'iscsi_info:/etc/iscsi', 'libvirtd:/var/lib/libvirt', 'nova_compute:/var/lib/nova/', '', ''], 'dimensions': {}}})
skipping: [ss] => (item={'key': 'nova-compute-ironic', 'value': {'container_name': 'nova_compute_ironic', 'group': 'nova-compute-ironic', 'image': 'kolla/ubuntu-source-nova-compute-ironic:train', 'enabled': False, 'volumes': ['/etc/kolla//nova-compute-ironic/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', 'kolla_logs:/var/log/kolla/', ''], 'dimensions': {}}})

TASK [nova : Check nova containers] ******************************************************************************************************************
changed: [ss] => (item={'key': 'nova-libvirt', 'value': {'container_name': 'nova_libvirt', 'group': 'compute', 'enabled': True, 'image': 'kolla/ubuntu-source-nova-libvirt:train', 'pid_mode': 'host', 'privileged': True, 'volumes': ['/etc/kolla//nova-libvirt/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/lib/modules:/lib/modules:ro', '/run/:/run/:shared', '/dev:/dev', '/sys/fs/cgroup:/sys/fs/cgroup', 'kolla_logs:/var/log/kolla/', 'libvirtd:/var/lib/libvirt', 'nova_compute:/var/lib/nova/', '', 'nova_libvirt_qemu:/etc/libvirt/qemu', ''], 'dimensions': {}}})
changed: [ss] => (item={'key': 'nova-ssh', 'value': {'container_name': 'nova_ssh', 'group': 'compute', 'image': 'kolla/ubuntu-source-nova-ssh:train', 'enabled': True, 'volumes': ['/etc/kolla//nova-ssh/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', 'kolla_logs:/var/log/kolla', 'nova_compute:/var/lib/nova', '', ''], 'dimensions': {}}})
changed: [ss] => (item={'key': 'nova-api', 'value': {'container_name': 'nova_api', 'group': 'nova-api', 'image': 'kolla/ubuntu-source-nova-api:train', 'enabled': True, 'privileged': True, 'volumes': ['/etc/kolla//nova-api/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/lib/modules:/lib/modules:ro', 'kolla_logs:/var/log/kolla/', ''], 'dimensions': {}, 'haproxy': {'nova_api': {'enabled': True, 'mode': 'http', 'external': False, 'port': '8774', 'listen_port': '8774'}, 'nova_api_external': {'enabled': True, 'mode': 'http', 'external': True, 'port': '8774', 'listen_port': '8774'}, 'nova_metadata': {'enabled': True, 'mode': 'http', 'external': False, 'port': '8775', 'listen_port': '8775'}, 'nova_metadata_external': {'enabled': True, 'mode': 'http', 'external': True, 'port': '8775', 'listen_port': '8775'}, 'nova_rdp': {'enabled': False, 'mode': 'http', 'external': False, 'port': '8001', 'host_group': 'hyperv'}}}})
changed: [ss] => (item={'key': 'nova-consoleauth', 'value': {'container_name': 'nova_consoleauth', 'group': 'nova-consoleauth', 'image': 'kolla/ubuntu-source-nova-consoleauth:train', 'enabled': True, 'volumes': ['/etc/kolla//nova-consoleauth/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', 'kolla_logs:/var/log/kolla/', ''], 'dimensions': {}}})
changed: [ss] => (item={'key': 'nova-novncproxy', 'value': {'container_name': 'nova_novncproxy', 'group': 'nova-novncproxy', 'image': 'kolla/ubuntu-source-nova-novncproxy:train', 'enabled': True, 'volumes': ['/etc/kolla//nova-novncproxy/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', 'kolla_logs:/var/log/kolla/', ''], 'dimensions': {}, 'haproxy': {'nova_novncproxy': {'enabled': True, 'mode': 'http', 'external': False, 'port': '6080', 'listen_port': '6080', 'backend_http_extra': ['timeout tunnel 1h']}, 'nova_novncproxy_external': {'enabled': True, 'mode': 'http', 'external': True, 'port': '6080', 'listen_port': '6080'}}}})
changed: [ss] => (item={'key': 'nova-scheduler', 'value': {'container_name': 'nova_scheduler', 'group': 'nova-scheduler', 'image': 'kolla/ubuntu-source-nova-scheduler:train', 'enabled': True, 'volumes': ['/etc/kolla//nova-scheduler/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', 'kolla_logs:/var/log/kolla/', ''], 'dimensions': {}}})
skipping: [ss] => (item={'key': 'nova-spicehtml5proxy', 'value': {'container_name': 'nova_spicehtml5proxy', 'group': 'nova-spicehtml5proxy', 'image': 'kolla/ubuntu-source-nova-spicehtml5proxy:train', 'enabled': False, 'volumes': ['/etc/kolla//nova-spicehtml5proxy/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', 'kolla_logs:/var/log/kolla/', ''], 'dimensions': {}, 'haproxy': {'nova_spicehtml5proxy': {'enabled': False, 'mode': 'http', 'external': False, 'port': '6082', 'listen_port': '6082'}, 'nova_spicehtml5proxy_external': {'enabled': False, 'mode': 'http', 'external': True, 'port': '6082', 'listen_port': '6082'}}}})
skipping: [ss] => (item={'key': 'nova-serialproxy', 'value': {'container_name': 'nova_serialproxy', 'group': 'nova-serialproxy', 'image': 'kolla/ubuntu-source-nova-serialproxy:train', 'enabled': False, 'volumes': ['/etc/kolla//nova-serialproxy/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', 'kolla_logs:/var/log/kolla/', ''], 'dimensions': {}, 'haproxy': {'nova_serialconsole_proxy': {'enabled': False, 'mode': 'http', 'external': False, 'port': '6083', 'listen_port': '6083', 'backend_http_extra': ['timeout tunnel 10m']}, 'nova_serialconsole_proxy_external': {'enabled': False, 'mode': 'http', 'external': True, 'port': '6083', 'listen_port': '6083', 'backend_http_extra': ['timeout tunnel 10m']}}}})
changed: [ss] => (item={'key': 'nova-conductor', 'value': {'container_name': 'nova_conductor', 'group': 'nova-conductor', 'enabled': True, 'image': 'kolla/ubuntu-source-nova-conductor:train', 'volumes': ['/etc/kolla//nova-conductor/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', 'kolla_logs:/var/log/kolla/', ''], 'dimensions': {}}})
changed: [ss] => (item={'key': 'nova-compute', 'value': {'container_name': 'nova_compute', 'group': 'compute', 'image': 'kolla/ubuntu-source-nova-compute:train', 'environment': {'LIBGUESTFS_BACKEND': 'direct'}, 'privileged': True, 'enabled': True, 'ipc_mode': 'host', 'volumes': ['/etc/kolla//nova-compute/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/lib/modules:/lib/modules:ro', '/run:/run:shared', '/dev:/dev', 'kolla_logs:/var/log/kolla/', 'iscsi_info:/etc/iscsi', 'libvirtd:/var/lib/libvirt', 'nova_compute:/var/lib/nova/', '', ''], 'dimensions': {}}})
skipping: [ss] => (item={'key': 'nova-compute-ironic', 'value': {'container_name': 'nova_compute_ironic', 'group': 'nova-compute-ironic', 'image': 'kolla/ubuntu-source-nova-compute-ironic:train', 'enabled': False, 'volumes': ['/etc/kolla//nova-compute-ironic/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', 'kolla_logs:/var/log/kolla/', ''], 'dimensions': {}}})

TASK [nova : include_tasks] **************************************************************************************************************************
skipping: [ss]

TASK [nova : include_tasks] **************************************************************************************************************************
included: /root/venv3/share/kolla-ansible/ansible/roles/nova/tasks/bootstrap.yml for ss

TASK [nova : Creating Nova databases] ****************************************************************************************************************
changed: [ss -> ss] => (item=nova)
changed: [ss -> ss] => (item=nova_cell0)
changed: [ss -> ss] => (item=nova_api)

TASK [nova : Creating Nova databases user and setting permissions] ***********************************************************************************
changed: [ss -> ss] => (item=None)
changed: [ss -> ss] => (item=None)
changed: [ss -> ss] => (item=None)
changed: [ss]

TASK [nova : include_tasks] **************************************************************************************************************************
included: /root/venv3/share/kolla-ansible/ansible/roles/nova/tasks/bootstrap_service.yml for ss

TASK [nova : Running Nova bootstrap container] *******************************************************************************************************
changed: [ss -> ss]

TASK [nova : include_tasks] **************************************************************************************************************************
included: /root/venv3/share/kolla-ansible/ansible/roles/nova/tasks/create_cells.yml for ss

TASK [nova : Create cell0 mappings] ******************************************************************************************************************
changed: [ss -> ss]

TASK [nova : include_tasks] **************************************************************************************************************************
included: /root/venv3/share/kolla-ansible/ansible/roles/nova/tasks/bootstrap_service.yml for ss

TASK [nova : Running Nova bootstrap container] *******************************************************************************************************
changed: [ss -> ss]

TASK [nova : Get list of existing cells] *************************************************************************************************************
ok: [ss -> ss]

TASK [nova : Check if a base cell already exists] ****************************************************************************************************
ok: [ss -> ss]

TASK [nova : Create base cell for legacy instances] **************************************************************************************************
changed: [ss -> ss]

TASK [nova : Update base cell for legacy instances] **************************************************************************************************
skipping: [ss]

TASK [nova : Print warning if a duplicate cell is detected] ******************************************************************************************
skipping: [ss]

RUNNING HANDLER [nova : Restart nova-conductor container] ********************************************************************************************
changed: [ss]

RUNNING HANDLER [nova : Restart nova-ssh container] **************************************************************************************************
changed: [ss]

RUNNING HANDLER [nova : Restart nova-libvirt container] **********************************************************************************************
changed: [ss]

RUNNING HANDLER [nova : Restart nova-scheduler container] ********************************************************************************************
changed: [ss]

RUNNING HANDLER [nova : Restart nova-consoleauth container] ******************************************************************************************
fatal: [ss]: FAILED! => {"changed": true, "msg": "'Traceback (most recent call last):\\n File \"/tmp/ansible_kolla_docker_payload_1uh4Ea/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 983, in main\\n File \"/tmp/ansible_kolla_docker_payload_1uh4Ea/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 731, in recreate_or_restart_container\\n File \"/tmp/ansible_kolla_docker_payload_1uh4Ea/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 744, in start_container\\n File \"/tmp/ansible_kolla_docker_payload_1uh4Ea/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 556, in pull_image\\n File \"/usr/local/lib/python2.7/dist-packages/docker/api/image.py\", line 415, in pull\\n self._raise_for_status(response)\\n File \"/usr/local/lib/python2.7/dist-packages/docker/api/client.py\", line 263, in _raise_for_status\\n raise create_api_error_from_http_exception(e)\\n File \"/usr/local/lib/python2.7/dist-packages/docker/errors.py\", line 31, in create_api_error_from_http_exception\\n raise cls(e, response=response, explanation=explanation)\\nNotFound: 404 Client Error: Not Found (\"manifest for kolla/ubuntu-source-nova-consoleauth:train not found: manifest unknown: manifest unknown\")\\n'"}

RUNNING HANDLER [nova : Restart nova-novncproxy container] *******************************************************************************************

RUNNING HANDLER [nova : Restart nova-api container] **************************************************************************************************

RUNNING HANDLER [nova : Restart nova-compute container] **********************************************************************************************

RUNNING HANDLER [nova : Wait for nova-compute services to update service versions] *******************************************************************

RUNNING HANDLER [nova : Restart nova services to remove RPC version cap] *****************************************************************************

NO MORE HOSTS LEFT ***********************************************************************************************************************************

PLAY RECAP *******************************************************************************************************************************************
localhost : ok=5 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
ss : ok=297 changed=193 unreachable=0 failed=1 skipped=161 rescued=0 ignored=0

[Radosław Piliszek (yoctozepto)]

Are you sure you are running kolla-ansible for Train?

[Mark Goddard (mgoddard)]

You are running the Stein version of kolla-ansible. In train, the nova role was split into nova and nova-cell, which we don't see in the above logs. The failure is due to the removal of the nova-consoleauth image.

Please use the stable/train branch of kolla-ansible. (No

[Mark Goddard (mgoddard)]

(note that kolla-ansible train is not yet released)

[Viorel-Cosmin Miron (uhl-hosting)]

We had tried several ways to make this work, inclusive the train kolla releases were several times upgraded to latest version for train. Nothing helped.

We get this on pulling images. Will get same if I run a fresh deployment.

TASK [nova : include_tasks] **************************************************************************************************************************
task path: /root/venv3/share/kolla-ansible/ansible/roles/nova/tasks/main.yml:2
included: /root/venv3/share/kolla-ansible/ansible/roles/nova/tasks/pull.yml for ss

TASK [nova : Pulling nova images] ********************************************************************************************************************
task path: /root/venv3/share/kolla-ansible/ansible/roles/nova/tasks/pull.yml:2
Using module file /root/venv3/share/kolla-ansible/ansible/library/kolla_docker.py
Pipelining is enabled.
<ss> ESTABLISH SSH CONNECTION FOR USER: None
<ss> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/bcd6204063 ss '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-lqjfxaqqozdzldrfjtuhmvubcnjurhav ; /usr/bin/python'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded
<ss> (0, b'\n{"invocation": {"module_args": {"tty": false, "tls_key": null, "image": "kolla/ubuntu-source-nova-api:train", "labels": {}, "tls_verify": false, "tls_cacert": null, "auth_password": null, "graceful_timeout": 10, "environment": {"KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS"}, "auth_registry": null, "state": "running", "volumes_from": null, "security_opt": [], "privileged": false, "api_version": "auto", "remove_on_exit": true, "restart_retries": 10, "detach": true, "auth_username": null, "dimensions": {}, "name": null, "tls_cert": null, "cap_add": [], "restart_policy": "unless-stopped", "auth_email": null, "command": null, "volumes": null, "action": "pull_image"}}, "changed": true, "result": false}\n', b'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n 7 Dec 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: /etc/ssh/ssh_config line 52: Deprecated option "useroaming"\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 10289\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
changed: [ss] => (item={'key': 'nova-api', 'value': {'container_name': 'nova_api', 'group': 'nova-api', 'image': 'kolla/ubuntu-source-nova-api:train', 'enabled': True, 'privileged': True, 'volumes': ['/etc/kolla/n...

[Viorel-Cosmin Miron (uhl-hosting)]

TASK [nova : Ensuring config directories exist] ******************************************************************************************************
changed: [ss] => (item={'key': 'nova-api', 'value': {'container_name': 'nova_api', 'group': 'nova-api', 'image': 'kolla/ubuntu-source-nova-api:train', 'enabled': True, 'privileged': True, 'volumes': ['/etc/kolla/nova-api/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/lib/modules:/lib/modules:ro', 'kolla_logs:/var/log/kolla/', ''], 'dimensions': {}, 'haproxy': {'nova_api': {'enabled': True, 'mode': 'http', 'external': False, 'port': '8774', 'listen_port': '8774'}, 'nova_api_external': {'enabled': True, 'mode': 'http', 'external': True, 'port': '8774', 'listen_port': '8774'}, 'nova_metadata': {'enabled': True, 'mode': 'http', 'external': False, 'port': '8775', 'listen_port': '8775'}, 'nova_metadata_external': {'enabled': True, 'mode': 'http', 'external': True, 'port': '8775', 'listen_port': '8775'}, 'nova_rdp': {'enabled': False, 'mode': 'http', 'external': False, 'port': '8001', 'host_group': 'hyperv'}}}})
changed: [ss] => (item={'key': 'nova-scheduler', 'value': {'container_name': 'nova_scheduler', 'group': 'nova-scheduler', 'image': 'kolla/ubuntu-source-nova-scheduler:train', 'enabled': True, 'volumes': ['/etc/kolla/nova-scheduler/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', 'kolla_logs:/var/log/kolla/', ''], 'dimensions': {}}})
fatal: [ss]: FAILED! => {"msg": "The conditional check 'inventory_hostname in groups[item.value.group]' failed. The error was: error while evaluating conditional (inventory_hostname in groups[item.value.group]): Unable to look up a name or access an attribute in template string ({% if inventory_hostname in groups[item.value.group] %} True {% else %} False {% endif %}).\nMake sure your variable name does not contain invalid characters like '-': argument of type 'AnsibleUndefined' is not iterable\n\nThe error appears to be in '/root/venv3/share/kolla-ansible/ansible/roles/nova/tasks/config.yml': line 2, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n---\n- name: Ensuring config directories exist\n ^ here\n"}

PLAY RECAP *******************************************************************************************************************************************
localhost : ok=5 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
ss : ok=332 changed=202 unreachable=0 failed=1 skipped=166 rescued=0 ignored=0

[Viorel-Cosmin Miron (uhl-hosting)]

Now after attempt to revert to Stein, we will end up with:

TASK [baremetal : Start docker] **********************************************************************************************************************
fatal: [ss]: FAILED! => {"changed": false, "msg": "Unable to start service docker: Job for docker.service failed because the control process exited with error code.\nSee \"systemctl status docker.service\" and \"journalctl -xe\" for details.\n"}

Dec 16 10:55:59 ss systemd[1]: docker.socket: Failed with result 'service-start-limit-hit'.

[Viorel-Cosmin Miron (uhl-hosting)]

(venv3) root@ss:~/kolla-ansible_setup# journalctl -fu docker
-- Logs begin at Tue 2019-11-26 23:06:42 UTC. --
Dec 16 10:55:57 ss dockerd[5615]: unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: log-opts: (from flag: map[max-file:5 max-size:50m], from file: map[max-file:5 max-size:50m])
Dec 16 10:55:57 ss systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Dec 16 10:55:57 ss systemd[1]: docker.service: Failed with result 'exit-code'.
Dec 16 10:55:57 ss systemd[1]: Failed to start Docker Application Container Engine.
Dec 16 10:55:59 ss systemd[1]: docker.service: Service hold-off time over, scheduling restart.
Dec 16 10:55:59 ss systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Dec 16 10:55:59 ss systemd[1]: Stopped Docker Application Container Engine.
Dec 16 10:55:59 ss systemd[1]: docker.service: Start request repeated too quickly.
Dec 16 10:55:59 ss systemd[1]: docker.service: Failed with result 'exit-code'.
Dec 16 10:55:59 ss systemd[1]: Failed to start Docker Application Container Engine.

[Viorel-Cosmin Miron (uhl-hosting)]

Seems removing /etc/docker/daemon.json helps with docker.

[Viorel-Cosmin Miron (uhl-hosting)]

The above errors are still valid.

[Mark Goddard (mgoddard)]

In your last comment about nova, it shows that your inventory has not been updated for Train as it does not include the nova-super-conductor group. Please ensure that the inventory is updated to match the groups in the multinode inventory for train (ansible/inventory/multinode).

[Viorel-Cosmin Miron (uhl-hosting)]

@mgoddard thanks for the hint, it seems changing globals and multinode helped and we bypassed all errors previously meet.