stx-openstack deploy fail stx3.0

[Macks]

Brief Description
-----------------
stx3.0 dedicated storage bare metal fails to apply stx-openstack

Severity
--------
Critical

Steps to Reproduce
------------------
i am running stx 3.0 dedicated storage bare metal. i followed the instructions under https://docs.starlingx.io/deploy_install_guides/r3_release/bare_metal/dedicated_storage.html with success. i am using ovs vswitch (**NOT DPDK**)

next i tried https://docs.starlingx.io/deploy_install_guides/r3_release/kubernetes_access.html. deployment was ok but i cannot access https://<oam-floating-ip-address>:30000.

then i tried https://docs.starlingx.io/deploy_install_guides/r3_release/openstack/install.html. i downloaded http://mirror.starlingx.cengn.ca/mirror/starlingx/release/3.0.0/centos/outputs/helm-charts/stx-openstack-1.0-19-centos-stable-latest.tgz and tried to apply it `system application-apply stx-openstack` but this failed at tiller "Name resolution failure". Logs attached:

Timestamp/Logs
--------------
0-03-22 04:35:19.887 414 DEBUG armada.handlers.document [-] Resolving reference /manifests/stx-openstack/1.0-19-centos-stable-latest/stx-openstack-stx-openstack.yaml. resolve_reference /usr/local/lib/python3.6/dist-packages/armada/handlers/document.py:49
2020-03-22 04:35:20.570 414 DEBUG armada.handlers.tiller [-] Using Tiller host IP: tiller-deploy.kube-system.svc.cluster.local _get_tiller_ip /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:165
2020-03-22 04:35:20.571 414 DEBUG armada.handlers.tiller [-] Using Tiller host port: 44134 _get_tiller_port /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:174
2020-03-22 04:35:20.571 414 DEBUG armada.handlers.tiller [-] Tiller getting gRPC insecure channel at tiller-deploy.kube-system.svc.cluster.local:44134 with options: [grpc.max_send_message_length=429496729, grpc.max_receive_message_length=429496729] get_channel /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:127
2020-03-22 04:35:20.575 414 DEBUG armada.handlers.tiller [-] Armada is using Tiller at: tiller-deploy.kube-system.svc.cluster.local:44134, namespace=kube-system, timeout=300 __init__ /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:107
2020-03-22 04:35:20.587 414 INFO armada.handlers.lock [-] Acquiring lock
2020-03-22 04:35:20.908 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-telemetry validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.909 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-nova-api-proxy validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.910 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-rabbitmq validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.911 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-rabbitmq validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.911 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-fm-rest-api validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.911 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] kube-system-ingress validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.911 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-ceilometer validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.912 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-heat validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.913 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] kube-system-ingress validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.913 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-garbd validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.914 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-fm-rest-api validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.915 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-libvirt validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.915 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-dcdbsync validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.916 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-placement validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.916 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-glance validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.917 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-barbican validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.917 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-keystone validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.918 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-mariadb validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.918 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-keystone validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.919 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-nova validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.919 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-panko validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.920 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-ceph-rgw validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.921 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-barbican validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.921 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-ingress validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.922 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-dcdbsync validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.923 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-mariadb validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.924 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-ingress validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.924 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-horizon validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.924 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-memcached validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.925 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-aodh validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.926 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-memcached validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.926 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-ceph-rgw validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.926 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-cinder validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.927 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-glance validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.927 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-cinder validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.928 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-gnocchi validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.929 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-neutron validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.930 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-nginx-ports-control validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.931 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] helm-toolkit validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.931 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-compute-kit validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.931 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-ironic validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.932 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-keystone-api-proxy validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.933 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-horizon validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.933 414 DEBUG armada.utils.validate [-] Validating document [armada/Manifest/v1] armada-manifest validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.934 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-keystone-api-proxy validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.934 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-heat validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.935 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-openvswitch validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.946 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-telemetry validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.947 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-nova-api-proxy validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.948 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-rabbitmq validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.949 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-rabbitmq validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.949 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-fm-rest-api validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.949 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] kube-system-ingress validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.949 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-ceilometer validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.950 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-heat validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.950 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] kube-system-ingress validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.951 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-garbd validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.952 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-fm-rest-api validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.953 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-libvirt validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.953 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-dcdbsync validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.953 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-placement validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.954 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-glance validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.954 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-barbican validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.955 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-keystone validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.955 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-mariadb validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.956 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-keystone validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.956 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-nova validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.957 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-panko validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.958 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-ceph-rgw validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.959 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-barbican validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.959 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-ingress validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.959 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-dcdbsync validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.960 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-mariadb validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.961 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-ingress validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.961 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-horizon validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.962 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-memcached validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.962 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-aodh validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.963 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-memcached validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.963 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-ceph-rgw validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.964 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-cinder validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.964 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-glance validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.965 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-cinder validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.965 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-gnocchi validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.966 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-neutron validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.967 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-nginx-ports-control validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.967 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] helm-toolkit validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.968 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-compute-kit validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.968 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-ironic validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.969 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-keystone-api-proxy validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.970 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-horizon validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.970 414 DEBUG armada.utils.validate [-] Validating document [armada/Manifest/v1] armada-manifest validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.971 414 DEBUG armada.utils.validate [-] Validating document [armada/ChartGroup/v1] openstack-keystone-api-proxy validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.971 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-heat validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.972 414 DEBUG armada.utils.validate [-] Validating document [armada/Chart/v1] openstack-openvswitch validate_armada_document /usr/local/lib/python3.6/dist-packages/armada/utils/validate.py:108
2020-03-22 04:35:20.993 414 INFO armada.handlers.armada [-] Performing pre-flight operations.
2020-03-22 04:35:20.993 414 DEBUG armada.handlers.tiller [-] Using Tiller host IP: tiller-deploy.kube-system.svc.cluster.local _get_tiller_ip /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:165
2020-03-22 04:35:20.994 414 DEBUG armada.handlers.tiller [-] Getting Tiller Status: Tiller exists tiller_status /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:186
2020-03-22 04:35:20.994 414 INFO armada.handlers.armada [-] Downloading tarball from: http://controller:8080/helm_charts/starlingx/ingress-0.1.0.tgz
2020-03-22 04:35:20.994 414 WARNING armada.handlers.armada [-] Disabling server validation certs to extract charts
2020-03-22 04:35:21.024 414 INFO armada.handlers.armada [-] Downloading tarball from: http://controller:8080/helm_charts/starlingx/helm-toolkit-0.1.0.tgz
2020-03-22 04:35:21.024 414 WARNING armada.handlers.armada [-] Disabling server validation certs to extract charts
2020-03-22 04:35:21.046 414 INFO armada.handlers.armada [-] Downloading tarball from: http://controller:8080/helm_charts/starlingx/nginx-ports-control-0.1.0.tgz
2020-03-22 04:35:21.046 414 WARNING armada.handlers.armada [-] Disabling server validation certs to extract charts
2020-03-22 04:35:21.050 414 INFO armada.handlers.armada [-] Downloading tarball from: http://controller:8080/helm_charts/starlingx/mariadb-0.1.0.tgz
2020-03-22 04:35:21.050 414 WARNING armada.handlers.armada [-] Disabling server validation certs to extract charts
2020-03-22 04:35:21.082 414 INFO armada.handlers.armada [-] Downloading tarball from: http://controller:8080/helm_charts/starlingx/garbd-0.1.0.tgz
2020-03-22 04:35:21.083 414 WARNING armada.handlers.armada [-] Disabling server validation certs to extract charts
2020-03-22 04:35:21.107 414 INFO armada.handlers.armada [-] Downloading tarball from: http://controller:8080/helm_charts/starlingx/memcached-0.1.0.tgz
2020-03-22 04:35:21.107 414 WARNING armada.handlers.armada [-] Disabling server validation certs to extract charts
2020-03-22 04:35:21.132 414 INFO armada.handlers.armada [-] Downloading tarball from: http://controller:8080/helm_charts/starlingx/rabbitmq-0.1.0.tgz
2020-03-22 04:35:21.133 414 WARNING armada.handlers.armada [-] Disabling server validation certs to extract charts
2020-03-22 04:35:21.161 414 INFO armada.handlers.armada [-] Downloading tarball from: http://controller:8080/helm_charts/starlingx/keystone-0.1.0.tgz
2020-03-22 04:35:21.161 414 WARNING armada.handlers.armada [-] Disabling server validation certs to extract charts
2020-03-22 04:35:21.193 414 INFO armada.handlers.armada [-] Downloading tarball from: http://controller:8080/helm_charts/starlingx/glance-0.1.0.tgz
2020-03-22 04:35:21.193 414 WARNING armada.handlers.armada [-] Disabling server validation certs to extract charts
2020-03-22 04:35:21.224 414 INFO armada.handlers.armada [-] Downloading tarball from: http://controller:8080/helm_charts/starlingx/cinder-0.1.0.tgz
2020-03-22 04:35:21.224 414 WARNING armada.handlers.armada [-] Disabling server validation certs to extract charts
2020-03-22 04:35:21.259 414 INFO armada.handlers.armada [-] Downloading tarball from: http://controller:8080/helm_charts/starlingx/libvirt-0.1.0.tgz
2020-03-22 04:35:21.259 414 WARNING armada.handlers.armada [-] Disabling server validation certs to extract charts
2020-03-22 04:35:21.286 414 INFO armada.handlers.armada [-] Downloading tarball from: http://controller:8080/helm_charts/starlingx/openvswitch-0.1.0.tgz
2020-03-22 04:35:21.286 414 WARNING armada.handlers.armada [-] Disabling server validation certs to extract charts
2020-03-22 04:35:21.311 414 INFO armada.handlers.armada [-] Downloading tarball from: http://controller:8080/helm_charts/starlingx/nova-0.1.0.tgz
2020-03-22 04:35:21.311 414 WARNING armada.handlers.armada [-] Disabling server validation certs to extract charts
2020-03-22 04:35:21.351 414 INFO armada.handlers.armada [-] Downloading tarball from: http://controller:8080/helm_charts/starlingx/nova-api-proxy-0.1.0.tgz
2020-03-22 04:35:21.351 414 WARNING armada.handlers.armada [-] Disabling server validation certs to extract charts
2020-03-22 04:35:21.376 414 INFO armada.handlers.armada [-] Downloading tarball from: http://controller:8080/helm_charts/starlingx/neutron-0.1.0.tgz
2020-03-22 04:35:21.376 414 WARNING armada.handlers.armada [-] Disabling server validation certs to extract charts
2020-03-22 04:35:21.410 414 INFO armada.handlers.armada [-] Downloading tarball from: http://controller:8080/helm_charts/starlingx/placement-0.1.0.tgz
2020-03-22 04:35:21.410 414 WARNING armada.handlers.armada [-] Disabling server validation certs to extract charts
2020-03-22 04:35:21.437 414 INFO armada.handlers.armada [-] Downloading tarball from: http://controller:8080/helm_charts/starlingx/heat-0.1.0.tgz
2020-03-22 04:35:21.438 414 WARNING armada.handlers.armada [-] Disabling server validation certs to extract charts
2020-03-22 04:35:21.471 414 INFO armada.handlers.armada [-] Downloading tarball from: http://controller:8080/helm_charts/starlingx/fm-rest-api-0.1.0.tgz
2020-03-22 04:35:21.471 414 WARNING armada.handlers.armada [-] Disabling server validation certs to extract charts
2020-03-22 04:35:21.497 414 INFO armada.handlers.armada [-] Downloading tarball from: http://controller:8080/helm_charts/starlingx/horizon-0.1.0.tgz
2020-03-22 04:35:21.497 414 WARNING armada.handlers.armada [-] Disabling server validation certs to extract charts
2020-03-22 04:35:21.526 414 DEBUG armada.handlers.tiller [-] Tiller ListReleases() with timeout=300, request=limit: 32
status_codes: UNKNOWN
status_codes: DEPLOYED
status_codes: DELETED
status_codes: DELETING
status_codes: FAILED
status_codes: PENDING_INSTALL
status_codes: PENDING_UPGRADE
status_codes: PENDING_ROLLBACK
 get_results /usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py:215
2020-03-22 04:35:31.614 414 INFO armada.handlers.lock [-] Releasing lock
2020-03-22 04:35:31.619 414 ERROR armada.cli [-] Caught unexpected exception: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "Name resolution failure"
        debug_error_string = "{"created":"@1584851731.537473572","description":"Failed to create subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":2721,"referenced_errors":[{"created":"@1584851731.537469194","description":"Name resolution failure","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3026,"grpc_status":14}]}"
>
2020-03-22 04:35:31.619 414 ERROR armada.cli Traceback (most recent call last):
2020-03-22 04:35:31.619 414 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/__init__.py", line 38, in safe_invoke
2020-03-22 04:35:31.619 414 ERROR armada.cli self.invoke()
2020-03-22 04:35:31.619 414 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 213, in invoke
2020-03-22 04:35:31.619 414 ERROR armada.cli resp = self.handle(documents, tiller)
2020-03-22 04:35:31.619 414 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/lock.py", line 81, in func_wrapper
2020-03-22 04:35:31.619 414 ERROR armada.cli return future.result()
2020-03-22 04:35:31.619 414 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
2020-03-22 04:35:31.619 414 ERROR armada.cli return self.__get_result()
2020-03-22 04:35:31.619 414 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2020-03-22 04:35:31.619 414 ERROR armada.cli raise self._exception
2020-03-22 04:35:31.619 414 ERROR armada.cli File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
2020-03-22 04:35:31.619 414 ERROR armada.cli result = self.fn(*self.args, **self.kwargs)
2020-03-22 04:35:31.619 414 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/cli/apply.py", line 256, in handle
2020-03-22 04:35:31.619 414 ERROR armada.cli return armada.sync()
2020-03-22 04:35:31.619 414 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/armada.py", line 189, in sync
2020-03-22 04:35:31.619 414 ERROR armada.cli known_releases = self.tiller.list_releases()
2020-03-22 04:35:31.619 414 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 252, in list_releases
2020-03-22 04:35:31.619 414 ERROR armada.cli releases = get_results()
2020-03-22 04:35:31.619 414 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 220, in get_results
2020-03-22 04:35:31.619 414 ERROR armada.cli for message in response:
2020-03-22 04:35:31.619 414 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 364, in __next__
2020-03-22 04:35:31.619 414 ERROR armada.cli return self._next()
2020-03-22 04:35:31.619 414 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 358, in _next
2020-03-22 04:35:31.619 414 ERROR armada.cli raise self
2020-03-22 04:35:31.619 414 ERROR armada.cli grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
2020-03-22 04:35:31.619 414 ERROR armada.cli status = StatusCode.UNAVAILABLE
2020-03-22 04:35:31.619 414 ERROR armada.cli details = "Name resolution failure"
2020-03-22 04:35:31.619 414 ERROR armada.cli debug_error_string = "{"created":"@1584851731.537473572","description":"Failed to create subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":2721,"referenced_errors":[{"created":"@1584851731.537469194","description":"Name resolution failure","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3026,"grpc_status":14}]}"
2020-03-22 04:35:31.619 414 ERROR armada.cli >
2020-03-22 04:35:31.619 414 ERROR armada.cli

[Frank Miller (sensfan22)]

One possible item to check is if your tiller was initialized properly. From https://bugs.launchpad.net/starlingx/+bug/1856078 there is a known intermittent issue with tiller. The comments in that LP give instructions to see if this is the issue you are seeing and how to work around it:

If helm commands fail with the following error then a restart of the tiller pod will be required:
controller-0:~$ helm ls -a
Error: the server could not find the requested resource (get configmaps)
To restart the tiller pod use the following:
controller-0:~$ kubectl get pods --all-namespaces -o wide|grep tiller-deploy
kube-system tiller-deploy-d6b59fcb-g676l 1/1 Running 1 18h fd01:1::3 controller-0 <none> <none>

controller-0:~$ kubectl delete pods -n kube-system tiller-deploy-d6b59fcb-g676l

Confirm tiller was restarted:
controller-0:~$ kubectl get pods --all-namespaces -o wide|grep tiller-deploy

Please determine if the helm ls -a command works in your system or not and add a comment to communicate if your system has this helm issue.

[Macks (macksme)]

`helm ls -a` doesn't show any issue. There are a couple of weird issues I see after deploying controller-0. I followed this doc:

https://docs.starlingx.io/deploy_install_guides/r3_release/bare_metal/dedicated_storage_install_kubernetes.html

all the way up to `system host-unlock controller-0`.

After controller-0 reboots:

- if i run `nslookup tiller-deploy.kube-system.svc.cluster.local` on controller-0, dns resolution fails every now and then because 8.8.8.8 is resolving it instead of `192.168.204.1`

- still running on controller-0, `ping tiller-deploy.kube-system.svc.cluster.local` fails all the time. i can cfm that the request is going out of the oam interface.

[Macks (macksme)]

just ran another fresh install. looks like `nslookup tiller-deploy.kube-system.svc.cluster.local` doesn't work reliably inside armada_service too. the problem goes deeper than what i originally thought: dnsmasq on the controller is having connection timeouts to 10.96.0.10:53 once every so often.

i am using this iso image btw:

http://mirror.starlingx.cengn.ca/mirror/starlingx/release/3.0.0/centos/outputs/iso/bootimage.iso

has this been resolved in a later release or am i the only one running into this issue?

[Macks (macksme)]

coming back to this. i realized that the problem is unique on a dell server. Could be the Mallenox network cards inside too...

[Ghada Khalil (gkhalil)]

Adding the release that this issue was reported in

[Macks (macksme)]

hi i just want to update anyone tracking this: i have isolated the problem to dell perc raid card. mine was h330 mini, but not sure if it's problematic with other cards as well. sysinv is stuck on profiling hardware...it is unable to discover any disks, which is probably what caused all the mayhem.

[yong hu (yhu6)]

@Macks, according to your analysis, the issue might be relevant to the particular HW, which we don't have in lab, so I have to mark this LP as "low". What we can do proactively is to observe and collect info accumulatively.

[Jim Somerville (jsomervi)]

@Macks can you get the exact pci id numbers for your raid controller, ie just copy and paste the output of lspci -nn

[Macks (macksme)]

$ lspci -nn | grep RAID
c1:00.0 RAID bus controller [0104]: LSI Logic / Sybios MegaRAID SAS-3 3008 [Fury] [1000:005f] (rev 02)

[Macks (macksme)]

sorry it should be Symbios. Could be https://pci-ids.ucw.cz/read/PC/1000/005f

[Ghada Khalil (gkhalil)]

Closing as stx.3.0 is EOL as of Dec 2020