Comment 12 for bug 1821912

Today I spent some time to investigate one of the problems which I found.

It was http://logs.openstack.org/47/653947/3/gate/tempest-slow-py3/629abee/testr_results.html.gz and as can be checked there, test failed due to paramiko.ssh_exception.AuthenticationException and that means that problem was no with connection to instance but with authentication to guest vm as cirros user.

So I looked into instance's console log and I found something like:

Sending discover...
Sending select for 10.1.0.10...
Lease of 10.1.0.10 obtained, lease time 86400
route: SIOCADDRT: File exists
WARN: failed: route add -net "0.0.0.0/0" gw "10.1.0.1"
checking http://169.254.169.254/2009-04-04/instance-id
successful after 1/20 tries: up 19.66. iid=i-00000010
failed to get http://169.254.169.254/2009-04-04/meta-data/public-keys
warning: no ec2 metadata for public-keys
failed to get http://169.254.169.254/2009-04-04/user-data
warning: no ec2 metadata for user-data

So it looks that reason of the problem was that public-keys wasn't provided by metadata service to the instance.
What is also interesting is fact that connectivity to metadata in overall worked as instance-id was fetched from there properly.

So lets check now neutron-metadata-agent logs. I found there that request for public-keys was there:
http://logs.openstack.org/47/653947/3/gate/tempest-slow-py3/629abee/controller/logs/screen-q-meta.txt.gz#_Apr_25_14_56_39_349166
And neutron metadata agent properly passed it to nova metadata server.

Lets then check nova metadata api logs: http://logs.openstack.org/47/653947/3/gate/tempest-slow-py3/629abee/controller/logs/screen-n-api-meta.txt.gz#_Apr_25_14_56_39_902289
Request comes there from neutron-metadata agent but I don't see any response for it.
So maybe in this case it is some issue on nova's side? I think we should ask someone from nova to take a look into this.