Comment 15 for bug 1836252

Seen this issue multiple times in past week.
In today's sanity, it started to happen on a 2+2 system after a compute lock/unlock test, and went away after another test that involves lock/unlock of both computes.

# lock/unlock compute-1
[2019-08-06 09:28:39,579] 301 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-lock compute-1'

[2019-08-06 09:29:12,349] 301 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-unlock compute-1'

# launch a vm and ping failed.
[2019-08-06 09:38:42,027] 301 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne volume create --size=2 --image=fdfe03a5-6e99-4f9d-80a2-fb84aa3e0505 --bootable vol-tis-centos-guest-3'

[2019-08-06 09:42:30,383] 301 DEBUG MainThread ssh.send :: Send 'ping -c 3 192.168.100.174'

# VM console log shows no IP is assigned on eth0 and metadata server is unreachable from VM
[ 69.114638] cloud-init[1146]: Cloud-init v. 0.7.9 running 'init' at Tue, 06 Aug 2019 09:40:12 +0000. Up 69.08 seconds.
[ 69.143033] cloud-init[1146]: ci-info: +++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++
[ 69.146783] cloud-init[1146]: ci-info: +--------+------+-----------+-----------+-------+-------------------+
[ 69.151250] cloud-init[1146]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
[ 69.156456] cloud-init[1146]: ci-info: +--------+------+-----------+-----------+-------+-------------------+
[ 69.164276] cloud-init[1146]: ci-info: | lo: | True | 127.0.0.1 | 255.0.0.0 | . | . |
[ 69.173020] cloud-init[1146]: ci-info: | lo: | True | . | . | d | . |
[ 69.178813] cloud-init[1146]: ci-info: | eth0: | True | . | . | . | fa:16:3e:06:6d:c0 |
[ 69.185948] cloud-init[1146]: ci-info: | eth0: | True | . | . | d | fa:16:3e:06:6d:c0 |
[ 69.190294] cloud-init[1146]: ci-info: +--------+------+-----------+-----------+-------+-------------------+

[ 70.231039] cloud-init[1146]: 2019-08-06 05:40:13,541 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [1/30s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x372abd0>: Failed to establish a new connection: [Errno 101] Network is unreachable',))]

# After lock/unlock of both computes, the issue went away.
[2019-08-06 11:46:52,269] 301 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-unlock compute-1'

[2019-08-06 11:57:54,934] 301 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-unlock compute-0'

# Launch a vm and it is reachable from NatBox
-os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server list --a'
[2019-08-06 12:10:37,523] 423 DEBUG MainThread ssh.expect :: Output:
+--------------------------------------+------------------------------+--------+-------------------------------------------------------------+-------+---------------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+------------------------------+--------+-------------------------------------------------------------+-------+---------------+
| d09f462e-5603-4a4a-8734-fec9ac95984c | tenant1-mempool_configured-7 | ACTIVE | tenant1-mgmt-net=192.168.100.154; tenant2-net0=172.18.0.241 | | flavor-mem10g |
+--------------------------------------+------------------------------+--------+-------------------------------------------------------------+-------+---------------+
[sysadmin@controller-1 ~(keystone_admin)]$

[2019-08-06 12:10:37,627] 301 DEBUG MainThread ssh.send :: Send 'ping -c 3 192.168.100.154'
PING 192.168.100.154 (192.168.100.154) 56(84) bytes of data.
64 bytes from 192.168.100.154: icmp_seq=1 ttl=62 time=0.571 ms
64 bytes from 192.168.100.154: icmp_seq=2 ttl=62 time=0.351 ms
64 bytes from 192.168.100.154: icmp_seq=3 ttl=62 time=0.461 ms

--- 192.168.100.154 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.351/0.461/0.571/0.089 ms
svc-cgcsauto@tis-lab-nat-box:~$

New logs are attached.