nova metadata broken in containerized environment

[Emilien Macchi]

I'm trying to run Tempest against a simple deployment where the overcloud is containerized.

It fails on trying to ssh a VM (using Tempest):
http://logs.openstack.org/84/494284/2/check/gate-tripleo-ci-centos-7-containers-multinode/c7c5f63/logs/undercloud/home/jenkins/tempest_output.log.txt.gz#_2017-08-16_22_58_51

Looking at Neutron Metadata agent logs, it looks like it fails to talk to the metadata API service.
Setting to critical since it's a huge regression in TripleO right now.

[Ihar Hrachyshka (ihar-hrachyshka)]

Metadata agent log: http://logs.openstack.org/84/494284/2/check/gate-tripleo-ci-centos-7-containers-multinode/c7c5f63/logs/subnode-2/var/log/containers/neutron/neutron-metadata-agent.log.txt.gz

[Daniel Alvarez (dalvarezs)]

Maybe it's got to do with this:
https://docs.openstack.org/nova/latest/user/wsgi.html

[Emilien Macchi (emilienm)]

It sounds like nova api metadata isn't running: http://logs.openstack.org/84/494284/2/check/gate-tripleo-ci-centos-7-containers-multinode/c7c5f63/logs/subnode-2/var/log/containers/nova/

if that's the case, it's a big problem and this is why we can't ping the vm.

[Emilien Macchi (emilienm)]

I confirm: http://logs.openstack.org/84/494284/2/check/gate-tripleo-ci-centos-7-containers-multinode/c7c5f63/logs/subnode-2/var/log/extra/docker/containers/

We need to run the container for nova-api-metadata.

[Emilien Macchi (emilienm)]

docker/services/nova-metadata.yaml isn't finished at all and can't work.

[Emilien Macchi (emilienm)]

Workaround in CI: https://review.openstack.org/494671 until this bug is solved.

[Dan Prince (dan-prince)]

Emilien: per your comment nova-metadata *was* finished. It used to run via the nova-api container when we had that under eventlet. I think what happened was we recently broke this in:

commit f24d5d4c0237d2703cf2744aa6db65865401e94e
Author: Juan Antonio Osorio Robles <email address hidden>
Date: Mon Jun 19 15:21:15 2017 +0300

    Make containerized nova-api run with httpd

    The non-containerized version will run over httpd [1], and for the
    containerized TLS work, it is needed in the container version as well.

-----

So your comment might be a bit of an overstatement. The quick fix could be to revert that commit which landed only 3 days ago.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/494723

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/494727

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/494750

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/494750
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=09b357677e1465014704389e69a85cb53dd31c81
Submitter: Jenkins
Branch: master

commit 09b357677e1465014704389e69a85cb53dd31c81
Author: Oliver Walsh <email address hidden>
Date: Fri Aug 18 00:54:36 2017 +0100

    Add NovaMetadata to nova-api image params

    Pre-requisite to splitting the nova-metadata container out of nova-api.yaml.

    Change-Id: Id3407e151566d16c6ae1e1ea8c1b021dac22e727
    Related-bug: #1711425

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/494727
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=2760f53274ab0c842f584af5b9041456337ea06f
Submitter: Jenkins
Branch: master

commit 2760f53274ab0c842f584af5b9041456337ea06f
Author: Oliver Walsh <email address hidden>
Date: Thu Aug 17 22:47:21 2017 +0100

    Restore and split nova metadata docker service out of nova-api.

    I2c39a2957fd95dd261b5b8c4df5e66e00a68d2f7 changed nova api to http from
    eventlet, however we need to continue running the eventlet service as
    it is required for the nova metadata api.

    However this should be tied to the OS::TripleO::Services::NovaMetadata
    service, so duplicate the required config in nova-metadata.yaml.

    Change-Id: I398575d565d5527bcaa1c8b33b9de2e1e0f2f6fd
    Depends-On: Id3407e151566d16c6ae1e1ea8c1b021dac22e727
    Closes-bug: #1711425

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dan Prince (<email address hidden>) on branch: master
Review: https://review.openstack.org/494723

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 7.0.0.0rc1 release candidate.