Description
===========
I got an error while trying to deploy HCI based overcloud.
The error message for the failed step says
"The following containers failed validations and were not started: mistral_executor, mistral_event_engine, mistral_engine"
And this error threw only for controller nodes.
Please see the full log:
```
2020-01-23 20:49:09,596 p=534993 u=mistral | fatal: [controller-0]: FAILED! => {
"failed_when_result": true,
"outputs.stdout_lines | default([]) | union(outputs.stderr_lines | default([]))": [
"docker runtime is deprecated in Stein and will be removed in Train.",
"image_exist isn't supported by docker",
"stdout: Trying to pull repository docker.io/tripleostein/centos-binary-mistral-engine ... ",
"current-tripleo: Pulling from docker.io/tripleostein/centos-binary-mistral-engine",
"ab5ef0e58194: Already exists",
"8b3196430879: Already exists",
"8776eabc5790: Already exists",
"e700300c12a7: Already exists",
"747b4b719859: Already exists",
"ae319dba9fb3: Already exists",
"040c4d3d3d9e: Already exists",
"f9e6a83d7b76: Already exists",
"b9626d47033a: Already exists",
"4eaf732b6ca4: Already exists",
"3c3d2653b903: Already exists",
"885630aa9eba: Already exists",
"879366603a45: Already exists",
"52281ca2cdd8: Already exists",
"68c07276d744: Already exists",
"5ac037880e2e: Already exists",
"13adb54e4024: Already exists",
"06050385d2f3: Already exists",
"d8e7772317cf: Already exists",
"164d1fec1f47: Already exists",
"06e585854e49: Already exists",
"f42968eba810: Already exists",
"db8f12b8af4b: Already exists",
"1c1d5c8034ea: Already exists",
"8930fcda96a6: Pulling fs layer",
"653942f85a51: Pulling fs layer",
"653942f85a51: Verifying Checksum",
"653942f85a51: Download complete",
"8930fcda96a6: Verifying Checksum",
"8930fcda96a6: Download complete",
"8930fcda96a6: Pull complete",
"653942f85a51: Pull complete",
"Digest: sha256:49de7b8d818ba1cf0259f2530285b84e943aee7c7be903755604774442405723",
"Status: Downloaded newer image for docker.io/tripleostein/centos-binary-mistral-engine:current-tripleo",
"",
"stderr: ",
"stdout: Trying to pull repository docker.io/tripleostein/centos-binary-mistral-event-engine ... ",
"current-tripleo: Pulling from docker.io/tripleostein/centos-binary-mistral-event-engine",
"fad6fccbcd52: Pulling fs layer",
"0324215892b9: Pulling fs layer",
"0324215892b9: Verifying Checksum",
"0324215892b9: Download complete",
"fad6fccbcd52: Verifying Checksum",
"fad6fccbcd52: Download complete",
"fad6fccbcd52: Pull complete",
"0324215892b9: Pull complete",
"Digest: sha256:d452f943b89c89fc407cea0a5e579608fe1131d6eb5ad26e645d545792e3425a",
"Status: Downloaded newer image for docker.io/tripleostein/centos-binary-mistral-event-engine:current-tripleo",
"stdout: Trying to pull repository docker.io/tripleostein/centos-binary-mistral-executor ... ",
"current-tripleo: Pulling from docker.io/tripleostein/centos-binary-mistral-executor",
"90ec330ef491: Pulling fs layer",
"7960c78577fa: Pulling fs layer",
"83083a8d83fa: Pulling fs layer",
"83083a8d83fa: Verifying Checksum",
"83083a8d83fa: Download complete",
"7960c78577fa: Verifying Checksum",
"7960c78577fa: Download complete",
"90ec330ef491: Verifying Checksum",
"90ec330ef491: Download complete",
"90ec330ef491: Pull complete",
"7960c78577fa: Pull complete",
"83083a8d83fa: Pull complete",
"Digest: sha256:88a3b7a6d58f0188b080088a208e786b422b810c63a869e077e978fc92ad1607",
"Status: Downloaded newer image for docker.io/tripleostein/centos-binary-mistral-executor:current-tripleo",
"stdout: Trying to pull repository docker.io/tripleostein/centos-binary-nova-consoleauth ... ",
"current-tripleo: Pulling from docker.io/tripleostein/centos-binary-nova-consoleauth",
"24c7ad93c148: Already exists",
"5a34f4432db3: Already exists",
"5f9f99b62ac6: Already exists",
"6a05da57acdb: Already exists",
"7fd91b95ea2a: Already exists",
"9ee5d285ce89: Pulling fs layer",
"d901ce04648e: Pulling fs layer",
"d901ce04648e: Verifying Checksum",
"d901ce04648e: Download complete",
"9ee5d285ce89: Verifying Checksum",
"9ee5d285ce89: Download complete",
"9ee5d285ce89: Pull complete",
"d901ce04648e: Pull complete",
"Digest: sha256:d24dac3bb305f3a8a387adf05112b7f90bcd88f1724fe416b6331fc4a237b02d",
"Status: Downloaded newer image for docker.io/tripleostein/centos-binary-nova-consoleauth:current-tripleo",
"stdout: Trying to pull repository docker.io/tripleostein/centos-binary-nova-novncproxy ... ",
"current-tripleo: Pulling from docker.io/tripleostein/centos-binary-nova-novncproxy",
"89c94cd05dfd: Pulling fs layer",
"b4009c14f927: Pulling fs layer",
"b4009c14f927: Verifying Checksum",
"b4009c14f927: Download complete",
"89c94cd05dfd: Verifying Checksum",
"89c94cd05dfd: Download complete",
"89c94cd05dfd: Pull complete",
"b4009c14f927: Pull complete",
"Digest: sha256:9a73c952424aa44b8b8787f47981973776e916326c376edcf679fb5b5f13c4a8",
"Status: Downloaded newer image for docker.io/tripleostein/centos-binary-nova-novncproxy:current-tripleo",
"stdout: Trying to pull repository docker.io/tripleostein/centos-binary-nova-scheduler ... ",
"current-tripleo: Pulling from docker.io/tripleostein/centos-binary-nova-scheduler",
"9b0d8abffb15: Pulling fs layer",
"619c3df6b32f: Pulling fs layer",
"619c3df6b32f: Download complete",
"9b0d8abffb15: Verifying Checksum",
"9b0d8abffb15: Download complete",
"9b0d8abffb15: Pull complete",
"619c3df6b32f: Pull complete",
"Digest: sha256:8ed9255e3a5cc82591f24e2aa6532d56884d633c8e7a2d27bb17e9fbb0858809",
"Status: Downloaded newer image for docker.io/tripleostein/centos-binary-nova-scheduler:current-tripleo",
"/usr/share/openstack-tripleo-validations is not a valid volume source",
"/usr/share/openstack-tripleo-heat-templates is not a valid volume source",
"stdout: 80472bcb40882f67c59154c6772e9221552662b4fa2a5653dd9f48a894ed4c2f",
"stdout: 199b7e11ca1b6f0a65898dcb31efdc2929fcf75688ba420ebb22283885a19f72",
"stdout: 705991ef2b536515268321fe9c46ca0f2ebdb919327c8c02001b8781efb3552a",
"stdout: ce0b4a8d4d563a05695b7d7988a58e1751a68cdc4854d013c0a8ff66d1fa861f",
"stdout: ",
"stderr: net_mlx5: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory",
"net_mlx5: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx5)",
"PMD: net_mlx4: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory",
"PMD: net_mlx4: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx4)",
"stdout: 245ded51aa943724a82724a8c606830b7a8a36f4b73d6010259b55df1779cd77",
"stdout: 1a7ace755fce9307e0244a8b32675e232acd0b5e419c445211d4f71f6681941d",
"stdout: 5081bf2380aa0e0522255fafa8a3b33ab609b77ce61f32012e429271fbcd8354",
"stdout: 3d073ce20dca0c4eaa1ff5a6ca9dd13f3d6c36bdc16cbd469522513364e40454",
"stdout: 777ef3a8e379ca4f2eb5d7bf02e65a6c4e340b6291dd9073dc53ca9072a67e40",
"stdout: eb211a3de2b8e27e1d8f602ff03d1e0da0b8aefc6c2c4634a6d99347f8414472",
"stdout: 774eb00fa6fa6d5af8b529d996c9052629627234686ff377d2690cee87545571",
"stdout: b5a93c9c8200d53b72435ad7a0dc7c443c95351a53925552c875de39c13c8732",
"stdout: 1207ee9dac2fc26472638aee04ca8a46fa1501862c8d534fbd9e7704ef25c94c",
"stdout: b51f294c4c6bd22453924fd206dd4955a3738655b6193e490ef77cd190e05f03",
"Did not find container with \"['docker', 'ps', '-a', '--filter', 'label=container_name=keystone', '--filter', 'label=config_id=tripleo_step4', '--format', '{{.Names}}']\" - retrying without config_id",
"container_running isn't supported by docker",
"stdout: 230dbbd73ed475624932be619b5c3d0fe0e6b6b8c2148e0bb0b8cbe089eb8807",
"stdout: 2e474d5e639dd50dab839b68d92ffe0b2c2bf48d58b1ad4ba8c63bc7ec9d6928",
"stdout: 5104186f543c3114b986871cf6232b581d8ca97325a3fa5d394d8b833a74d205",
"stdout: INFO:nova_wait_for_placement_service:Placement service up! - {\"versions\": [{\"status\": \"CURRENT\", \"min_version\": \"1.0\", \"max_version\": \"1.30\", \"id\": \"v1.0\", \"links\": [{\"href\": \"\", \"rel\": \"self\"}]}]}",
"stderr: + command -v python3",
"+ command -v python2",
"+ python2 /container-config-scripts/placement_wait_for_service.py",
"stdout: 072ee4fe1293681a9c6e0d5dc3dccf39562bf2a6c962bf7bfef62bbc0c721e08",
"stdout: INFO:nova_wait_for_api_service:Nova-api service active",
"+ python2 /container-config-scripts/nova_wait_for_api_service.py",
"stdout: 3d3968395c3f6b29fd6ae40e35e1ee2785a947cc8d68660bed4fc91a223a950c",
"stdout: c6d34fae6ec513119ae13ca18ff1163087465e452256eb02e5e298299ccea141",
"The following containers failed validations and were not started: mistral_executor, mistral_event_engine, mistral_engine"
]
}
```
Steps to reproduce
==================
1. Install undercloud
2. Deploy overcloud with:
openstack overcloud deploy \
--timeout 300 \
--templates \
-r ~/templates/roles_data.yaml \
-n ~/templates/network_data.yaml \
-e ~/custom-tripleo-heat-templates-generated/environments/docker-ha.yaml \
-e ~/custom-tripleo-heat-templates-generated/environments/network-isolation.yaml \
-e ~/custom-tripleo-heat-templates-generated/environments/network-environment.yaml \
-e ~/custom-tripleo-heat-templates-generated/environments/services/neutron-ovn-dvr-ha.yaml \
-e ~/templates/nic-mapping.yaml \
-e ~/templates/network.yaml \
-e ~/templates/node-info.yaml \
-e ~/templates/scheduler_hints_env.yaml \
-e ~/templates/ips-from-pool-all.yaml \
-e ~/templates/fixed-ip-vips.yaml \
-e ~/templates/ceph-custom-config.yaml \
-e ~/custom-tripleo-heat-templates-generated/environments/ceph-ansible/ceph-ansible.yaml \
-e ~/custom-tripleo-heat-templates-generated/environments/ceph-ansible/ceph-rgw.yaml \
-e ~/custom-tripleo-heat-templates-generated/environments/disable-telemetry.yaml \
-e ~/custom-tripleo-heat-templates-generated/environments/services/mistral.yaml \
-e ~/templates/misc-settings.yaml \
-e ~/templates/timezone.yaml
Environment
===========
1. Stein
2. 17 baremetal node (3 controller + 8 ComputeHCI + 6 Compute node)
Could you provide /var/log/paunch.log from controller-0?