5.0.1-0.214-queens - VM fails to boot due to failing metadata

[Charlie Kim]

Initiated cirros VM, and if fails as below. It boots up without failure if config-drive is flagged. However, the VM cannot get metadata once it boots up.

Console log:

[ 1.370756] Write protecting the kernel read-only data: 14336k
[ 1.373588] Freeing unused kernel memory: 1860K (ffff88000182f000 - ffff880001a00000)
[ 1.377615] Freeing unused kernel memory: 168K (ffff880001dd6000 - ffff880001e00000)

info: initramfs: up at 0.98
modprobe: module virtio_pci not found in modules.dep
modprobe: module virtio_blk not found in modules.dep
modprobe: module virtio_net not found in modules.dep
modprobe: module vfat not found in modules.dep
modprobe: module nls_cp437 not found in modules.dep
info: copying initramfs to /dev/vda1
info: initramfs loading root from /dev/vda1
info: /etc/init.d/rc.sysinit: up at 2.52
info: container: none
Starting logging: OK
modprobe: module virtio_pci not found in modules.dep
modprobe: module virtio_blk not found in modules.dep
modprobe: module virtio_net not found in modules.dep
modprobe: module vfat not found in modules.dep
modprobe: module nls_cp437 not found in modules.dep
WARN: /etc/rc3.d/S10-load-modules failed
Initializing random number generator... [ 3.006022] random: dd urandom read with 8 bits of entropy available
done.
Starting acpid: OK
Starting network...
udhcpc (v1.23.2) started
Sending discover...
Sending select for 10.0.2.4...
Lease of 10.0.2.4 obtained, lease time 268435455
checking http://169.254.169.254/2009-04-04/instance-id
failed 1/20: up 2.77. request failed
failed 2/20: up 4.79. request failed
failed 3/20: up 6.81. request failed
failed 4/20: up 8.83. request failed

Instance.yaml:

provider_config:
  bms:
    ssh_user: contrail
    ssh_public_key: /root/.ssh/id_rsa.pub
    ssh_private_key: /root/.ssh/id_rsa
    ntpserver: 10.85.130.130
    domainsuffix: pslab.net
instances:
  bms1:
    provider: bms
    ip: 10.85.192.5
    roles:
      config_database:
      config:
      control:
      analytics_database:
      analytics:
      webui:
      openstack:
      vrouter:
      openstack_compute:
global_configuration:
  CONTAINER_REGISTRY: hub.juniper.net/contrail
  CONTAINER_REGISTRY_USERNAME: XXXX
  CONTAINER_REGISTRY_PASSWORD: YYYY
contrail_configuration:
  OPENSTACK_VERSION: queens
  CONTRAIL_CONTAINER_TAG: 5.0.1-0.214-queens
  CLOUD_ORCHESTRATOR: openstack
  RABBITMQ_NODE_PORT: 5673
  AUTH_MODE: keystone
  KEYSTONE_AUTH_URL_VERSION: /v3
  KEYSTONE_AUTH_ADMIN_PASSWORD: contrail123
  UPGRADE_KERNEL: true
kolla_config:
  kolla_globals:
    network_interface: "eno1"
    kolla_external_vip_interface: "eno1"
    enable_haproxy: "no"
    enable_ironic: "no"
    enable_swift: "no"
  kolla_passwords:
    keystone_admin_password: contrail123

[contrail@server7 ~]$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
kolla/centos-binary-nova-compute queens 5cf24be93c30 3 days ago 1.39GB
kolla/centos-binary-nova-conductor queens 6d529a092af8 3 days ago 877MB
kolla/centos-binary-nova-ssh queens 6de41c45f900 3 days ago 897MB
kolla/centos-binary-nova-placement-api queens 88dc3d17bc5e 3 days ago 930MB
kolla/centos-binary-nova-api queens 7ec3c1ce4cf2 3 days ago 930MB
kolla/centos-binary-nova-scheduler queens f1bb200e05ce 3 days ago 877MB
kolla/centos-binary-nova-consoleauth queens d3b977b25679 3 days ago 877MB
kolla/centos-binary-glance-api queens e70fcbc001a3 3 days ago 892MB
kolla/centos-binary-nova-novncproxy queens 12397e993dcb 3 days ago 878MB
kolla/centos-binary-glance-registry queens 6453449ce2a4 3 days ago 830MB
kolla/centos-binary-heat-api queens fafdb57919e4 3 days ago 776MB
kolla/centos-binary-heat-engine queens 2d9b4dc737ba 3 days ago 776MB
kolla/centos-binary-heat-api-cfn queens db63b5d15035 3 days ago 776MB
kolla/centos-binary-barbican-keystone-listener queens 80b40e554eda 3 days ago 718MB
kolla/centos-binary-barbican-worker queens ff92b2244e90 3 days ago 718MB
kolla/centos-binary-neutron-server queens 59614d39710c 3 days ago 828MB
kolla/centos-binary-keystone queens 83bfa2c694da 3 days ago 758MB
kolla/centos-binary-horizon queens 76d179253d56 3 days ago 971MB
kolla/centos-binary-kolla-toolbox queens 8cf7979ef6d0 3 days ago 783MB
kolla/centos-binary-fluentd queens 377f9b10f551 3 days ago 451MB
kolla/centos-binary-mariadb queens 354ddef3fdf6 3 days ago 626MB
kolla/centos-binary-nova-libvirt queens 895497fd4603 3 days ago 997MB
kolla/centos-binary-cron queens 568c3a124e68 3 days ago 434MB
kolla/centos-binary-memcached queens 352f3edcc473 3 days ago 435MB
kolla/centos-binary-rabbitmq queens 4a9be2bc180c 3 days ago 495MB
kolla/centos-binary-barbican-api queens 1c5c8df24ab8 11 days ago 772MB
hub.juniper.net/contrail/contrail-vrouter-kernel-init 5.0.1-0.214-queens 05c5cc43af6c 5 weeks ago 951MB
hub.juniper.net/contrail/contrail-vrouter-agent 5.0.1-0.214-queens 1eacd10a3370 5 weeks ago 1.1GB
hub.juniper.net/contrail/contrail-status 5.0.1-0.214-queens cb1b09cd3f81 5 weeks ago 753MB
hub.juniper.net/contrail/contrail-openstack-neutron-init 5.0.1-0.214-queens 272e78bb455d 5 weeks ago 869MB
hub.juniper.net/contrail/contrail-openstack-heat-init 5.0.1-0.214-queens 383baf53d57f 5 weeks ago 753MB
hub.juniper.net/contrail/contrail-openstack-compute-init 5.0.1-0.214-queens 7dab8226331b 5 weeks ago 753MB
hub.juniper.net/contrail/contrail-nodemgr 5.0.1-0.214-queens d647350e5af5 5 weeks ago 771MB
hub.juniper.net/contrail/contrail-node-init 5.0.1-0.214-queens b8f351c40c85 5 weeks ago 753MB
hub.juniper.net/contrail/contrail-external-zookeeper 5.0.1-0.214-queens 793c758afdf6 5 weeks ago 144MB
hub.juniper.net/contrail/contrail-external-redis 5.0.1-0.214-queens e29015ed986d 5 weeks ago 107MB
hub.juniper.net/contrail/contrail-external-rabbitmq 5.0.1-0.214-queens 11abca144426 5 weeks ago 124MB
hub.juniper.net/contrail/contrail-external-kafka 5.0.1-0.214-queens 306379c4e4a5 5 weeks ago 735MB
hub.juniper.net/contrail/contrail-external-cassandra 5.0.1-0.214-queens e626907cf4f9 5 weeks ago 323MB
hub.juniper.net/contrail/contrail-controller-webui-web 5.0.1-0.214-queens aa3b019c5d98 5 weeks ago 976MB
hub.juniper.net/contrail/contrail-controller-webui-job 5.0.1-0.214-queens 96ef4a2effee 5 weeks ago 976MB
hub.juniper.net/contrail/contrail-controller-control-named 5.0.1-0.214-queens 3d6ac891a89d 5 weeks ago 821MB
hub.juniper.net/contrail/contrail-controller-control-dns 5.0.1-0.214-queens 44e9f87193e5 5 weeks ago 821MB
hub.juniper.net/contrail/contrail-controller-control-control 5.0.1-0.214-queens 40a660d74a87 5 weeks ago 821MB
hub.juniper.net/contrail/contrail-controller-config-svcmonitor 5.0.1-0.214-queens 718e0afaba79 5 weeks ago 1.04GB
hub.juniper.net/contrail/contrail-controller-config-schema 5.0.1-0.214-queens 4831cee9e526 5 weeks ago 1.04GB
hub.juniper.net/contrail/contrail-controller-config-devicemgr 5.0.1-0.214-queens 89c954ffa4fa 5 weeks ago 1.04GB
hub.juniper.net/contrail/contrail-controller-config-api 5.0.1-0.214-queens 42f51fce7915 5 weeks ago 1.09GB
hub.juniper.net/contrail/contrail-analytics-topology 5.0.1-0.214-queens 08093588facb 5 weeks ago 837MB
hub.juniper.net/contrail/contrail-analytics-snmp-collector 5.0.1-0.214-queens 6fa784ff1248 5 weeks ago 837MB
hub.juniper.net/contrail/contrail-analytics-query-engine 5.0.1-0.214-queens 9a49600c2129 5 weeks ago 837MB
hub.juniper.net/contrail/contrail-analytics-collector 5.0.1-0.214-queens fa954c2ae007 5 weeks ago 837MB
hub.juniper.net/contrail/contrail-analytics-api 5.0.1-0.214-queens 5e8117736023 5 weeks ago 837MB
hub.juniper.net/contrail/contrail-analytics-alarm-gen 5.0.1-0.214-queens cd71419757a6 5 weeks ago 837MB
[contrail@server7 ~]$

[Nagendra Prasath (npchandran)]

Could you retrieve below info and post here please...

from the config_api_1 container:
netstat -nalp |grep 8082|grep LISTEN | awk '{print $4}' <= should print IP address of the api server

with that run below command:
curl -uadmin:<keystone-passwd> http://<api-server-ip>:8082/global-vrouter-configs |python -m json.tool

in above o/p, for the url specified at href seen in the section under global-vrouter-configs, run
curl -uadmin:<keystone-passwd> <url> | python -m json.tool

Also:
from vrouter_vrouter-agent_1 docker container:
grep metadata /etc/contrail/contrail-vrouter-agent.conf

from nova_api docker container:
grep metadata /etc/nova/nova.conf

[Nagendra Prasath (npchandran)]

This doesnt seems to be a release specific issue. Moving it to R5.0.3, will wait for response from bug reporter.