R3-2RHOSP_DPDK VM went in Error state

[shajuvk]

Overcloud DPDK VM went to Error state with dpdk-Bond configuration. Could you please check my setup?
Undercloud 10.87.67.42 – root/c0ntrail123

I have customized the overcloud-dpdk image using below command:

virt-customize -a overcloud-full-dpdk.qcow2 --sm-credentials aranjan.redhat:password:H3Ub9pth3x --sm-register --sm-attach auto --run-command 'subscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-extras-rpms --enable=rhel-7-server-rh-common-rpms --enable=rhel-ha-for-rhel-7-server-rpms --enable=rhel-7-server-openstack-10-rpms --enable=rhel-7-server-openstack-10-devtools-rpms' --copy-in /etc/yum.repos.d/contrail.repo:/etc/yum.repos.d --run-command 'yum install -y contrail-vrouter-utils contrail-vrouter-dpdk contrail-vrouter-dpdk-init supervisor contrail-vrouter-agent contrail-nodemgr contrail-setup contrail-tripleo-puppet puppet-contrail python-contrail lshw' --run-command 'rm -rf /etc/yum.repos.d/contrail.repo' --run-command 'subscription-manager unregister' --selinux-relabel

[stack@undercloud-new ~]$ nova list
+--------------------------------------+---------------------------------------+--------+------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+---------------------------------------+--------+------------+-------------+---------------------+
| 5524976f-ed0a-4214-89ea-57c360b3d324 | overcloud-contrailanalytics-0 | ACTIVE | - | Running | ctlplane=192.0.2.15 |
| aff498ac-b211-4e9d-af4c-3389a215cb77 | overcloud-contrailanalytics-1 | ACTIVE | - | Running | ctlplane=192.0.2.11 |
| 65c87b5d-f265-46a7-831c-6d589dc46b1a | overcloud-contrailanalytics-2 | ACTIVE | - | Running | ctlplane=192.0.2.14 |
| 4688d617-6572-4cee-bcf7-9fc80835abe9 | overcloud-contrailanalyticsdatabase-0 | ACTIVE | - | Running | ctlplane=192.0.2.12 |
| 44413966-f18a-46f9-92b0-156704d0066b | overcloud-contrailanalyticsdatabase-1 | ACTIVE | - | Running | ctlplane=192.0.2.19 |
| 0075926a-436e-4fb9-9338-1e60a488e866 | overcloud-contrailanalyticsdatabase-2 | ACTIVE | - | Running | ctlplane=192.0.2.16 |
| 57d47f35-b004-4726-ba3b-5e72b5097bac | overcloud-contrailcontroller-0 | ACTIVE | - | Running | ctlplane=192.0.2.13 |
| 43f1533c-d5db-4df3-9311-58ed8bfa7785 | overcloud-contrailcontroller-1 | ACTIVE | - | Running | ctlplane=192.0.2.8 |
| d0e58243-eb36-46de-8b8b-bdc035b6c7b0 | overcloud-contrailcontroller-2 | ACTIVE | - | Running | ctlplane=192.0.2.6 |
| e0884686-5a51-4089-9faa-8a43287438b2 | overcloud-controller-0 | ACTIVE | - | Running | ctlplane=192.0.2.24 |
| 81408da2-ece6-4ca7-b739-0d191c6a988d | overcloud-controller-1 | ACTIVE | - | Running | ctlplane=192.0.2.20 |
| 5d5012bf-fe59-4ad5-b9f0-e9b12c9e2245 | overcloud-controller-2 | ACTIVE | - | Running | ctlplane=192.0.2.22 |
| a81af4b2-0800-405e-a3d8-a47390271af5 | overcloud-novacompute-0 | ERROR | - | NOSTATE | |
+--------------------------------------+---------------------------------------+--------+------------+-------------+---------------------+
[stack@undercloud-new ~]$

Deploy log:
=========

2017-09-28 21:48:22Z [overcloud.Compute.0]: CREATE_FAILED ResourceInError: resources[0].resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
2017-09-28 21:48:22Z [overcloud.Compute]: CREATE_FAILED Resource CREATE failed: ResourceInError: resources[0].resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
2017-09-28 21:48:25Z [overcloud.ContrailAnalyticsDatabase.1.StoragePort]: CREATE_COMPLETE state changed
2017-09-28 21:48:25Z [overcloud.Compute]: CREATE_FAILED ResourceInError: resources.Compute.resources[0].resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
2017-09-28 21:48:26Z [overcloud.ContrailController.1.UpdateDeployment]: SIGNAL_IN_PROGRESS Signal: deployment cd8fd3e2-5ac9-4354-9921-c958910385eb succeeded
2017-09-28 21:48:26Z [overcloud.ContrailAnalytics]: CREATE_FAILED CREATE aborted
2017-09-28 21:48:27Z [overcloud.ContrailController.1.ExternalPort]: CREATE_COMPLETE state changed
2017-09-28 21:48:27Z [overcloud.ContrailController.0.TenantPort]: CREATE_COMPLETE state changed
2017-09-28 21:48:29Z [overcloud.ContrailAnalytics.1.StorageMgmtPort]: CREATE_COMPLETE state changed
2017-09-28 21:48:30Z [overcloud.ContrailAnalytics.1]: CREATE_FAILED CREATE aborted
2017-09-28 21:48:30Z [overcloud.ContrailController.2.StorageMgmtPort]: CREATE_COMPLETE state changed
2017-09-28 21:48:30Z [overcloud.ContrailController]: CREATE_FAILED CREATE aborted
2017-09-28 21:48:31Z [overcloud.ContrailController.1]: CREATE_FAILED CREATE aborted
2017-09-28 21:48:31Z [overcloud.ContrailAnalyticsDatabase]: CREATE_FAILED CREATE aborted
2017-09-28 21:48:31Z [overcloud.ContrailController.2.UpdateDeployment]: SIGNAL_IN_PROGRESS Signal: deployment deac8d4a-54a7-40e9-986e-0719e0ff06a1 succeeded
2017-09-28 21:48:31Z [overcloud.ContrailController.0]: CREATE_FAILED CREATE aborted
2017-09-28 21:48:31Z [overcloud.ContrailAnalytics.0]: CREATE_FAILED CREATE aborted
2017-09-28 21:48:31Z [overcloud.ContrailAnalyticsDatabase.1.InternalApiPort]: CREATE_COMPLETE state changed
2017-09-28 21:48:32Z [overcloud.ContrailAnalyticsDatabase.1]: CREATE_FAILED CREATE aborted
2017-09-28 21:48:32Z [overcloud.ContrailController.2]: CREATE_FAILED CREATE aborted
2017-09-28 21:48:32Z [overcloud.Controller]: CREATE_FAILED CREATE aborted
2017-09-28 21:48:32Z [overcloud.ContrailController]: CREATE_FAILED Resource CREATE failed: Operation cancelled
2017-09-28 21:48:32Z [overcloud.Controller.0.TenantPort]: CREATE_IN_PROGRESS state changed
2017-09-28 21:48:32Z [overcloud.ContrailController.0.ExternalPort]: CREATE_COMPLETE state changed
2017-09-28 21:48:33Z [overcloud.ContrailAnalytics.2]: CREATE_FAILED CREATE aborted
2017-09-28 21:48:33Z [overcloud.ContrailAnalytics.0.TenantPort]: CREATE_IN_PROGRESS state changed
2017-09-28 21:48:34Z [overcloud.ContrailAnalyticsDatabase.0]: CREATE_FAILED CREATE aborted
2017-09-28 21:48:34Z [overcloud]: CREATE_FAILED Resource CREATE failed: ResourceInError: resources.Compute.resources[0].resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
2017-09-28 21:48:34Z [overcloud.ContrailAnalytics]: CREATE_FAILED Resource CREATE failed: Operation cancelled
2017-09-28 21:48:34Z [overcloud.ContrailAnalytics.1.ManagementPort]: CREATE_COMPLETE state changed
2017-09-28 21:48:34Z [overcloud.Controller.1]: CREATE_FAILED CREATE aborted
2017-09-28 21:48:35Z [overcloud.ContrailAnalyticsDatabase.0.UpdateDeployment]: CREATE_IN_PROGRESS state changed
2017-09-28 21:48:35Z [overcloud.ContrailAnalytics.2.ManagementPort]: CREATE_IN_PROGRESS state changed
2017-09-28 21:48:36Z [overcloud.ContrailAnalyticsDatabase.2]: CREATE_FAILED CREATE aborted
2017-09-28 21:48:37Z [overcloud.ContrailAnalyticsDatabase]: CREATE_FAILED Resource CREATE failed: Operation cancelled
2017-09-28 21:48:37Z [overcloud.ContrailController.2.StoragePort]: CREATE_COMPLETE state changed
2017-09-28 21:48:37Z [overcloud.Controller.0]: CREATE_FAILED CREATE aborted
2017-09-28 21:48:37Z [overcloud.ContrailAnalyticsDatabase.2.InternalApiPort]: CREATE_IN_PROGRESS state changed

[Michael Henkel (mhenkel-3)]

did you profile ironic nodes with compute-dpdk role?

[shajuvk (shajuvk)]

Now the profile with contrail-dpdk. Will change it to compute-dpdk

[stack@undercloud-new ~]$ ironic node-show compute2
+------------------------+--------------------------------------------------------------------------+
| Property | Value |
+------------------------+--------------------------------------------------------------------------+
| chassis_uuid | |
| clean_step | {} |
| console_enabled | False |
| created_at | 2017-09-25T08:09:52+00:00 |
| driver | pxe_ipmitool |
| driver_info | {u'deploy_kernel': u'386cb51f-42e8-4596-91c6-f729016542d1', |
| | u'ipmi_address': u'10.87.122.154', u'deploy_ramdisk': u'cacc5a3a- |
| | c5f7-4640-9684-fae780241982', u'ipmi_password': u'******', |
| | u'ipmi_username': u'admin'} |
| driver_internal_info | {u'agent_url': u'http://192.0.2.16:9999', u'root_uuid_or_disk_id': |
| | u'e6e645f3-3cb6-4dc2-95f9-d3312d908ce4', u'is_whole_disk_image': False, |
| | u'agent_last_heartbeat': 1506536205} |
| extra | {u'hardware_swift_object': u'extra_hardware-2b6c2ae0-48db-483c-801f- |
| | b531e947ed75'} |
| inspection_finished_at | None |
| inspection_started_at | None |
| instance_info | {} |
| instance_uuid | None |
| last_error | None |
| maintenance | False |
| maintenance_reason | None |
| name | compute2 |
| network_interface | |
| power_state | power off |
| properties | {u'memory_mb': u'262144', u'cpu_arch': u'x86_64', u'local_gb': u'110', |
| | u'cpus': u'24', u'capabilities': u'profile:contrail-dpdk,cpu_hugepages:t |
| | rue,cpu_txt:true,boot_option:local,cpu_aes:true,cpu_vt:true,cpu_hugepage |
| ...

[shajuvk (shajuvk)]

[stack@undercloud-new ~]$ openstack flavor list
+--------------------------------------+-----------------------------+------+------+-----------+-------+-----------+
| ID | Name | RAM | Disk | Ephemeral | VCPUs | Is Public |
+--------------------------------------+-----------------------------+------+------+-----------+-------+-----------+
| 009217c2-6911-448c-a334-5b8196ab886b | baremetal | 4096 | 40 | 0 | 1 | True |
| 276f7f3b-f183-4c37-b358-fc2d7bbbfb64 | ceph-storage | 4096 | 40 | 0 | 1 | True |
| 281a326e-3396-41bd-851b-e43ddcb236e2 | contrail-databse | 4096 | 40 | 0 | 2 | True |
| 930029eb-b366-4c83-b784-b66f08caf002 | contrail-analytics-database | 4096 | 40 | 0 | 2 | True |
| 9a17df6d-fb3f-4be1-8d6c-01adc29c6478 | compute | 4096 | 40 | 0 | 1 | True |
| 9eaf563d-2398-4a7f-af41-c57183079c3b | contrail-dpdk | 4096 | 40 | 0 | 1 | True |
| b9e2b778-e8e6-48aa-adea-b32a444a2cad | contrail-analytics | 4096 | 40 | 0 | 2 | True |
| ca1b5af5-333f-495d-9ddd-d09277f85b7d | swift-storage | 4096 | 40 | 0 | 1 | True |
| cfa46be2-834d-4af9-8805-1f0a8eb9888a | contrail-database | 4096 | 40 | 0 | 1 | True |
| ed5cc339-c6a6-4fec-8c49-4e43a1d5dba1 | compute-dpdk | 4096 | 40 | 0 | 1 | True |
| f94b36b2-6baa-42d9-8050-49ea405609bf | control | 4096 | 40 | 0 | 1 | True |
| ff3481f9-333b-43c1-89f1-65d2e20c1f37 | contrail-controller | 4096 | 40 | 0 | 2 | True |
| ff839a9a-7519-4fd2-9c30-94457819e4ab | block-storage | 4096 | 40 | 0 | 1 | True |
+--------------------------------------+-----------------------------+------+------+-----------+-------+-----------+

[stack@undercloud-new ~]$ ironic node-show ff839a9a-7519-4fd2-9c30-94457819e4ab
Node ff839a9a-7519-4fd2-9c30-94457819e4ab could not be found. (HTTP 404)
[stack@undercloud-new ~]$ ironic node-show compute2
+------------------------+--------------------------------------------------------------------------+
| Property | Value |
+------------------------+--------------------------------------------------------------------------+
| chassis_uuid | |
| clean_step | {} |
| console_enabled | False |
| created_at | 2017-09-25T08:09:52+00:00 |
| driver | pxe_ipmitool |
| driver_info | {u'deploy_kernel': u'386cb51f-42e8-4596-91c6-f729016542d1', |
| | u'ipmi_addre...

[shajuvk (shajuvk)]

[stack@undercloud-new ~]$ cd tripleo-heat-templates/environments/contrail/
[stack@undercloud-new contrail]$ grep -r "compute-dpdk" *
contrail-net-dpdk.yaml: OS::TripleO::ContrailDpdk::Net::SoftwareConfig: contrail-nic-config-compute-dpdk.yaml
contrail-services-storage-mgmt.yaml: OvercloudContrailDpdkFlavor: compute-dpdk
contrail-services.yaml: OvercloudContrailDpdkFlavor: compute-dpdk
[stack@undercloud-new contrail]$

[shajuvk (shajuvk)]

[stack@undercloud-new contrail]$ nova list
+--------------------------------------+---------------------------------------+--------+------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+---------------------------------------+--------+------------+-------------+---------------------+
| 53f895ca-c83d-495a-a15f-768a5a2773a0 | overcloud-contrailanalytics-0 | ACTIVE | - | Running | ctlplane=192.0.2.10 |
| d04b6cb8-425c-4b92-b774-53837636c8bb | overcloud-contrailanalytics-1 | ACTIVE | - | Running | ctlplane=192.0.2.7 |
| c2f4b508-8a61-4734-8f1b-15483cc21623 | overcloud-contrailanalytics-2 | ACTIVE | - | Running | ctlplane=192.0.2.18 |
| 809dcc73-ebf8-4652-b14c-5da77652f84f | overcloud-contrailanalyticsdatabase-0 | ACTIVE | - | Running | ctlplane=192.0.2.21 |
| cbd67fef-12d9-4e45-9c0a-69078b11b0a2 | overcloud-contrailanalyticsdatabase-1 | ACTIVE | - | Running | ctlplane=192.0.2.15 |
| 5d4a4dd7-aefa-4835-819a-616e1147be51 | overcloud-contrailanalyticsdatabase-2 | ACTIVE | - | Running | ctlplane=192.0.2.23 |
| fb366c73-2426-4162-a6c7-c8ac1e1e79b9 | overcloud-contrailcontroller-0 | ACTIVE | - | Running | ctlplane=192.0.2.19 |
| 2e7ededb-08d7-42f8-ad4f-6d62e0033f87 | overcloud-contrailcontroller-1 | ACTIVE | - | Running | ctlplane=192.0.2.11 |
| 79dbc690-5411-4236-bc52-11ebbab52683 | overcloud-contrailcontroller-2 | ACTIVE | - | Running | ctlplane=192.0.2.16 |
| caf9356f-6aff-41a9-9139-5a8f5df6f73a | overcloud-controller-0 | ACTIVE | - | Running | ctlplane=192.0.2.6 |
| 6ca06321-1d3f-4702-a254-693af287e6d0 | overcloud-controller-1 | ACTIVE | - | Running | ctlplane=192.0.2.12 |
| 6f2443af-17cc-49a7-9930-36205f8ba4c6 | overcloud-controller-2 | ACTIVE | - | Running | ctlplane=192.0.2.9 |
| 4912b29a-f132-470a-a8e7-49f32ef1dfa1 | overcloud-novacompute-0 | ERROR | - | NOSTATE | |
+--------------------------------------+---------------------------------------+--------+------------+-------------+---------------------+
[stack@undercloud-new contrail]$

[alexey-mr (alexey-morlang)]

The reason of the last fail:

Failed instance:
90b16548-fedd-4f34-887e-3c01bd6d9791 | overcloud-contraildpdk-0 | ERROR | - | NOSTATE

There is an error in nova logs:
…
/var/log/nova/nova-compute.log-20170929.gz:2017-09-29 00:24:07.705 2786 ERROR oslo.service.loopingcall InstanceDeployFailure: Failed to provision instance 90b16548-fedd-4f34-887e-3c01bd6d9791: Failed to deploy. Error: Disk volume where '/var/lib/ironic/master_images/tmp9TPRWj' is located doesn't have enough disk space. Required 122880 MiB, only 95714 MiB available space present.
/var/log/nova/nova-compute.log-20170929.gz:2017-09-29 00:24:07.707 2786 ERROR nova.virt.ironic.driver [req-78399dd9-3c10-4c3e-9a6e-0b1e12278578 8c4034c9482e451abaea447e727d1d2f cb2b772f30f64d6aaa9cece572d8e9db - - -] Error deploying instance 90b16548-fedd-4f34-887e-3c01bd6d9791 on baremetal node 2b6c2ae0-48db-483c-801f-b531e947ed75.
...
 [instance: 90b16548-fedd-4f34-887e-3c01bd6d9791] Build of instance 90b16548-fedd-4f34-887e-3c01bd6d9791 was re-scheduled: Failed to provision instance 90b16548-fedd-4f34-887e-3c01bd6d9791: Failed to deploy. Error: Disk volume where '/var/lib/ironic/master_images/tmp9TPRWj' is located doesn't have enough disk space. Required 122880 MiB, only 95714 MiB available space present. _do_build_and_run_instance /usr/lib/python2.7/site-packages/nova/compute/manager.py:1802

It looks that you have to increase size of disk on your compute node, or reduce size of dpdk image somehow..

PS: Dont forget to set ContrailVersion: 3

[shajuvk (shajuvk)]

Thanks Alexey, deployment proceeded further after recreating new overcloud-dpdk image.
Moving bug to invalid