Seen this issue multiple times in past week. In today's sanity, it started to happen on a 2+2 system after a compute lock/unlock test, and went away after another test that involves lock/unlock of both computes. # lock/unlock compute-1 [2019-08-06 09:28:39,579] 301 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-lock compute-1' [2019-08-06 09:29:12,349] 301 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-unlock compute-1' # launch a vm and ping failed. [2019-08-06 09:38:42,027] 301 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne volume create --size=2 --image=fdfe03a5-6e99-4f9d-80a2-fb84aa3e0505 --bootable vol-tis-centos-guest-3' [2019-08-06 09:42:30,383] 301 DEBUG MainThread ssh.send :: Send 'ping -c 3 192.168.100.174' # VM console log shows no IP is assigned on eth0 and metadata server is unreachable from VM [ 69.114638] cloud-init[1146]: Cloud-init v. 0.7.9 running 'init' at Tue, 06 Aug 2019 09:40:12 +0000. Up 69.08 seconds. [ 69.143033] cloud-init[1146]: ci-info: +++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++ [ 69.146783] cloud-init[1146]: ci-info: +--------+------+-----------+-----------+-------+-------------------+ [ 69.151250] cloud-init[1146]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address | [ 69.156456] cloud-init[1146]: ci-info: +--------+------+-----------+-----------+-------+-------------------+ [ 69.164276] cloud-init[1146]: ci-info: | lo: | True | 127.0.0.1 | 255.0.0.0 | . | . | [ 69.173020] cloud-init[1146]: ci-info: | lo: | True | . | . | d | . | [ 69.178813] cloud-init[1146]: ci-info: | eth0: | True | . | . | . | fa:16:3e:06:6d:c0 | [ 69.185948] cloud-init[1146]: ci-info: | eth0: | True | . | . | d | fa:16:3e:06:6d:c0 | [ 69.190294] cloud-init[1146]: ci-info: +--------+------+-----------+-----------+-------+-------------------+ [ 70.231039] cloud-init[1146]: 2019-08-06 05:40:13,541 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [1/30s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable',))] # After lock/unlock of both computes, the issue went away. [2019-08-06 11:46:52,269] 301 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-unlock compute-1' [2019-08-06 11:57:54,934] 301 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-unlock compute-0' # Launch a vm and it is reachable from NatBox -os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server list --a' [2019-08-06 12:10:37,523] 423 DEBUG MainThread ssh.expect :: Output: +--------------------------------------+------------------------------+--------+-------------------------------------------------------------+-------+---------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+------------------------------+--------+-------------------------------------------------------------+-------+---------------+ | d09f462e-5603-4a4a-8734-fec9ac95984c | tenant1-mempool_configured-7 | ACTIVE | tenant1-mgmt-net=192.168.100.154; tenant2-net0=172.18.0.241 | | flavor-mem10g | +--------------------------------------+------------------------------+--------+-------------------------------------------------------------+-------+---------------+ [sysadmin@controller-1 ~(keystone_admin)]$ [2019-08-06 12:10:37,627] 301 DEBUG MainThread ssh.send :: Send 'ping -c 3 192.168.100.154' PING 192.168.100.154 (192.168.100.154) 56(84) bytes of data. 64 bytes from 192.168.100.154: icmp_seq=1 ttl=62 time=0.571 ms 64 bytes from 192.168.100.154: icmp_seq=2 ttl=62 time=0.351 ms 64 bytes from 192.168.100.154: icmp_seq=3 ttl=62 time=0.461 ms --- 192.168.100.154 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2000ms rtt min/avg/max/mdev = 0.351/0.461/0.571/0.089 ms svc-cgcsauto@tis-lab-nat-box:~$ New logs are attached.