Validation error while deploying overcloud

Bug #1860712 reported by Cagri Ersen
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Expired
Undecided
Unassigned

Bug Description

Description
===========

I got an error while trying to deploy HCI based overcloud.

The error message for the failed step says

"The following containers failed validations and were not started: mistral_executor, mistral_event_engine, mistral_engine"

And this error threw only for controller nodes.

Please see the full log:

```
2020-01-23 20:49:09,596 p=534993 u=mistral | fatal: [controller-0]: FAILED! => {
    "failed_when_result": true,
    "outputs.stdout_lines | default([]) | union(outputs.stderr_lines | default([]))": [
        "docker runtime is deprecated in Stein and will be removed in Train.",
        "image_exist isn't supported by docker",
        "stdout: Trying to pull repository docker.io/tripleostein/centos-binary-mistral-engine ... ",
        "current-tripleo: Pulling from docker.io/tripleostein/centos-binary-mistral-engine",
        "ab5ef0e58194: Already exists",
        "8b3196430879: Already exists",
        "8776eabc5790: Already exists",
        "e700300c12a7: Already exists",
        "747b4b719859: Already exists",
        "ae319dba9fb3: Already exists",
        "040c4d3d3d9e: Already exists",
        "f9e6a83d7b76: Already exists",
        "b9626d47033a: Already exists",
        "4eaf732b6ca4: Already exists",
        "3c3d2653b903: Already exists",
        "885630aa9eba: Already exists",
        "879366603a45: Already exists",
        "52281ca2cdd8: Already exists",
        "68c07276d744: Already exists",
        "5ac037880e2e: Already exists",
        "13adb54e4024: Already exists",
        "06050385d2f3: Already exists",
        "d8e7772317cf: Already exists",
        "164d1fec1f47: Already exists",
        "06e585854e49: Already exists",
        "f42968eba810: Already exists",
        "db8f12b8af4b: Already exists",
        "1c1d5c8034ea: Already exists",
        "8930fcda96a6: Pulling fs layer",
        "653942f85a51: Pulling fs layer",
        "653942f85a51: Verifying Checksum",
        "653942f85a51: Download complete",
        "8930fcda96a6: Verifying Checksum",
        "8930fcda96a6: Download complete",
        "8930fcda96a6: Pull complete",
        "653942f85a51: Pull complete",
        "Digest: sha256:49de7b8d818ba1cf0259f2530285b84e943aee7c7be903755604774442405723",
        "Status: Downloaded newer image for docker.io/tripleostein/centos-binary-mistral-engine:current-tripleo",
        "",
        "stderr: ",
        "stdout: Trying to pull repository docker.io/tripleostein/centos-binary-mistral-event-engine ... ",
        "current-tripleo: Pulling from docker.io/tripleostein/centos-binary-mistral-event-engine",
        "fad6fccbcd52: Pulling fs layer",
        "0324215892b9: Pulling fs layer",
        "0324215892b9: Verifying Checksum",
        "0324215892b9: Download complete",
        "fad6fccbcd52: Verifying Checksum",
        "fad6fccbcd52: Download complete",
        "fad6fccbcd52: Pull complete",
        "0324215892b9: Pull complete",
        "Digest: sha256:d452f943b89c89fc407cea0a5e579608fe1131d6eb5ad26e645d545792e3425a",
        "Status: Downloaded newer image for docker.io/tripleostein/centos-binary-mistral-event-engine:current-tripleo",
        "stdout: Trying to pull repository docker.io/tripleostein/centos-binary-mistral-executor ... ",
        "current-tripleo: Pulling from docker.io/tripleostein/centos-binary-mistral-executor",
        "90ec330ef491: Pulling fs layer",
        "7960c78577fa: Pulling fs layer",
        "83083a8d83fa: Pulling fs layer",
        "83083a8d83fa: Verifying Checksum",
        "83083a8d83fa: Download complete",
        "7960c78577fa: Verifying Checksum",
        "7960c78577fa: Download complete",
        "90ec330ef491: Verifying Checksum",
        "90ec330ef491: Download complete",
        "90ec330ef491: Pull complete",
        "7960c78577fa: Pull complete",
        "83083a8d83fa: Pull complete",
        "Digest: sha256:88a3b7a6d58f0188b080088a208e786b422b810c63a869e077e978fc92ad1607",
        "Status: Downloaded newer image for docker.io/tripleostein/centos-binary-mistral-executor:current-tripleo",
        "stdout: Trying to pull repository docker.io/tripleostein/centos-binary-nova-consoleauth ... ",
        "current-tripleo: Pulling from docker.io/tripleostein/centos-binary-nova-consoleauth",
        "24c7ad93c148: Already exists",
        "5a34f4432db3: Already exists",
        "5f9f99b62ac6: Already exists",
        "6a05da57acdb: Already exists",
        "7fd91b95ea2a: Already exists",
        "9ee5d285ce89: Pulling fs layer",
        "d901ce04648e: Pulling fs layer",
        "d901ce04648e: Verifying Checksum",
        "d901ce04648e: Download complete",
        "9ee5d285ce89: Verifying Checksum",
        "9ee5d285ce89: Download complete",
        "9ee5d285ce89: Pull complete",
        "d901ce04648e: Pull complete",
        "Digest: sha256:d24dac3bb305f3a8a387adf05112b7f90bcd88f1724fe416b6331fc4a237b02d",
        "Status: Downloaded newer image for docker.io/tripleostein/centos-binary-nova-consoleauth:current-tripleo",
        "stdout: Trying to pull repository docker.io/tripleostein/centos-binary-nova-novncproxy ... ",
        "current-tripleo: Pulling from docker.io/tripleostein/centos-binary-nova-novncproxy",
        "89c94cd05dfd: Pulling fs layer",
        "b4009c14f927: Pulling fs layer",
        "b4009c14f927: Verifying Checksum",
        "b4009c14f927: Download complete",
        "89c94cd05dfd: Verifying Checksum",
        "89c94cd05dfd: Download complete",
        "89c94cd05dfd: Pull complete",
        "b4009c14f927: Pull complete",
        "Digest: sha256:9a73c952424aa44b8b8787f47981973776e916326c376edcf679fb5b5f13c4a8",
        "Status: Downloaded newer image for docker.io/tripleostein/centos-binary-nova-novncproxy:current-tripleo",
        "stdout: Trying to pull repository docker.io/tripleostein/centos-binary-nova-scheduler ... ",
        "current-tripleo: Pulling from docker.io/tripleostein/centos-binary-nova-scheduler",
        "9b0d8abffb15: Pulling fs layer",
        "619c3df6b32f: Pulling fs layer",
        "619c3df6b32f: Download complete",
        "9b0d8abffb15: Verifying Checksum",
        "9b0d8abffb15: Download complete",
        "9b0d8abffb15: Pull complete",
        "619c3df6b32f: Pull complete",
        "Digest: sha256:8ed9255e3a5cc82591f24e2aa6532d56884d633c8e7a2d27bb17e9fbb0858809",
        "Status: Downloaded newer image for docker.io/tripleostein/centos-binary-nova-scheduler:current-tripleo",
        "/usr/share/openstack-tripleo-validations is not a valid volume source",
        "/usr/share/openstack-tripleo-heat-templates is not a valid volume source",
        "stdout: 80472bcb40882f67c59154c6772e9221552662b4fa2a5653dd9f48a894ed4c2f",
        "stdout: 199b7e11ca1b6f0a65898dcb31efdc2929fcf75688ba420ebb22283885a19f72",
        "stdout: 705991ef2b536515268321fe9c46ca0f2ebdb919327c8c02001b8781efb3552a",
        "stdout: ce0b4a8d4d563a05695b7d7988a58e1751a68cdc4854d013c0a8ff66d1fa861f",
        "stdout: ",
        "stderr: net_mlx5: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory",
        "net_mlx5: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx5)",
        "PMD: net_mlx4: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory",
        "PMD: net_mlx4: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx4)",
        "stdout: 245ded51aa943724a82724a8c606830b7a8a36f4b73d6010259b55df1779cd77",
        "stdout: 1a7ace755fce9307e0244a8b32675e232acd0b5e419c445211d4f71f6681941d",
        "stdout: 5081bf2380aa0e0522255fafa8a3b33ab609b77ce61f32012e429271fbcd8354",
        "stdout: 3d073ce20dca0c4eaa1ff5a6ca9dd13f3d6c36bdc16cbd469522513364e40454",
        "stdout: 777ef3a8e379ca4f2eb5d7bf02e65a6c4e340b6291dd9073dc53ca9072a67e40",
        "stdout: eb211a3de2b8e27e1d8f602ff03d1e0da0b8aefc6c2c4634a6d99347f8414472",
        "stdout: 774eb00fa6fa6d5af8b529d996c9052629627234686ff377d2690cee87545571",
        "stdout: b5a93c9c8200d53b72435ad7a0dc7c443c95351a53925552c875de39c13c8732",
        "stdout: 1207ee9dac2fc26472638aee04ca8a46fa1501862c8d534fbd9e7704ef25c94c",
        "stdout: b51f294c4c6bd22453924fd206dd4955a3738655b6193e490ef77cd190e05f03",
        "Did not find container with \"['docker', 'ps', '-a', '--filter', 'label=container_name=keystone', '--filter', 'label=config_id=tripleo_step4', '--format', '{{.Names}}']\" - retrying without config_id",
        "container_running isn't supported by docker",
        "stdout: 230dbbd73ed475624932be619b5c3d0fe0e6b6b8c2148e0bb0b8cbe089eb8807",
        "stdout: 2e474d5e639dd50dab839b68d92ffe0b2c2bf48d58b1ad4ba8c63bc7ec9d6928",
        "stdout: 5104186f543c3114b986871cf6232b581d8ca97325a3fa5d394d8b833a74d205",
        "stdout: INFO:nova_wait_for_placement_service:Placement service up! - {\"versions\": [{\"status\": \"CURRENT\", \"min_version\": \"1.0\", \"max_version\": \"1.30\", \"id\": \"v1.0\", \"links\": [{\"href\": \"\", \"rel\": \"self\"}]}]}",
        "stderr: + command -v python3",
        "+ command -v python2",
        "+ python2 /container-config-scripts/placement_wait_for_service.py",
        "stdout: 072ee4fe1293681a9c6e0d5dc3dccf39562bf2a6c962bf7bfef62bbc0c721e08",
        "stdout: INFO:nova_wait_for_api_service:Nova-api service active",
        "+ python2 /container-config-scripts/nova_wait_for_api_service.py",
        "stdout: 3d3968395c3f6b29fd6ae40e35e1ee2785a947cc8d68660bed4fc91a223a950c",
        "stdout: c6d34fae6ec513119ae13ca18ff1163087465e452256eb02e5e298299ccea141",
        "The following containers failed validations and were not started: mistral_executor, mistral_event_engine, mistral_engine"
    ]
}
```

Steps to reproduce
==================
1. Install undercloud
2. Deploy overcloud with:
openstack overcloud deploy \
    --timeout 300 \
    --templates \
    -r ~/templates/roles_data.yaml \
    -n ~/templates/network_data.yaml \
    -e ~/custom-tripleo-heat-templates-generated/environments/docker-ha.yaml \
    -e ~/custom-tripleo-heat-templates-generated/environments/network-isolation.yaml \
    -e ~/custom-tripleo-heat-templates-generated/environments/network-environment.yaml \
    -e ~/custom-tripleo-heat-templates-generated/environments/services/neutron-ovn-dvr-ha.yaml \
    -e ~/templates/nic-mapping.yaml \
    -e ~/templates/network.yaml \
    -e ~/templates/node-info.yaml \
    -e ~/templates/scheduler_hints_env.yaml \
    -e ~/templates/ips-from-pool-all.yaml \
    -e ~/templates/fixed-ip-vips.yaml \
    -e ~/templates/ceph-custom-config.yaml \
    -e ~/custom-tripleo-heat-templates-generated/environments/ceph-ansible/ceph-ansible.yaml \
    -e ~/custom-tripleo-heat-templates-generated/environments/ceph-ansible/ceph-rgw.yaml \
    -e ~/custom-tripleo-heat-templates-generated/environments/disable-telemetry.yaml \
    -e ~/custom-tripleo-heat-templates-generated/environments/services/mistral.yaml \
    -e ~/templates/misc-settings.yaml \
    -e ~/templates/timezone.yaml

Environment
===========
1. Stein

2. 17 baremetal node (3 controller + 8 ComputeHCI + 6 Compute node)

tags: added: tripleo-heat-templates
Changed in tripleo:
milestone: none → ussuri-2
Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

Could you provide /var/log/paunch.log from controller-0?

Changed in tripleo:
status: New → Incomplete
Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

Also it would be nice to see the /var/log/messages contents for the docker runtime details

Revision history for this message
Cagri Ersen (cagri-ersen) wrote :

As a wordaround, I have just copied /usr/share/openstack-tripleo-validations and /usr/share/openstack-tripleo-heat-templates directories from undercloud to controller nodes and re-run the deployment It worked and deployment finished successfully.

While, I re-deploy the cluster, I don't have paunch logs. But obviously it was related with these two folder, which wasn't placed on the controllers (openstack-tripleo packege wasn't installed on the controllers.).

"/usr/share/openstack-tripleo-validations is not a valid volume source",
"/usr/share/openstack-tripleo-heat-templates is not a valid volume source",

That folders was required because we passed mistral.yaml to our deployment command.

"-e ~/custom-tripleo-heat-templates-generated/environments/services/mistral.yaml"

wes hayutin (weshayutin)
Changed in tripleo:
milestone: ussuri-2 → ussuri-3
wes hayutin (weshayutin)
Changed in tripleo:
milestone: ussuri-3 → ussuri-rc3
wes hayutin (weshayutin)
Changed in tripleo:
milestone: ussuri-rc3 → victoria-1
Changed in tripleo:
milestone: victoria-1 → victoria-3
Changed in tripleo:
milestone: victoria-3 → wallaby-1
Changed in tripleo:
milestone: wallaby-1 → wallaby-2
Changed in tripleo:
milestone: wallaby-2 → wallaby-3
Revision history for this message
Marios Andreou (marios-b) wrote :

Bug status has been set to 'Incomplete' and target milestone has been removed due to inactivity. If you disagree please re-set these values and reach out to us on freenode #tripleo

Changed in tripleo:
milestone: wallaby-3 → none
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for tripleo because there has been no activity for 60 days.]

Changed in tripleo:
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.