Seen this issue multiple times in past week.
In today's sanity, it started to happen on a 2+2 system after a compute lock/unlock test, and went away after another test that involves lock/unlock of both computes.
# VM console log shows no IP is assigned on eth0 and metadata server is unreachable from VM
[ 69.114638] cloud-init[1146]: Cloud-init v. 0.7.9 running 'init' at Tue, 06 Aug 2019 09:40:12 +0000. Up 69.08 seconds.
[ 69.143033] cloud-init[1146]: ci-info: +++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++
[ 69.146783] cloud-init[1146]: ci-info: +--------+------+-----------+-----------+-------+-------------------+
[ 69.151250] cloud-init[1146]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
[ 69.156456] cloud-init[1146]: ci-info: +--------+------+-----------+-----------+-------+-------------------+
[ 69.164276] cloud-init[1146]: ci-info: | lo: | True | 127.0.0.1 | 255.0.0.0 | . | . |
[ 69.173020] cloud-init[1146]: ci-info: | lo: | True | . | . | d | . |
[ 69.178813] cloud-init[1146]: ci-info: | eth0: | True | . | . | . | fa:16:3e:06:6d:c0 |
[ 69.185948] cloud-init[1146]: ci-info: | eth0: | True | . | . | d | fa:16:3e:06:6d:c0 |
[ 69.190294] cloud-init[1146]: ci-info: +--------+------+-----------+-----------+-------+-------------------+
[ 70.231039] cloud-init[1146]: 2019-08-06 05:40:13,541 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [1/30s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x372abd0>: Failed to establish a new connection: [Errno 101] Network is unreachable',))]
# After lock/unlock of both computes, the issue went away.
[2019-08-06 11:46:52,269] 301 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-unlock compute-1'
# Launch a vm and it is reachable from NatBox
-os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server list --a'
[2019-08-06 12:10:37,523] 423 DEBUG MainThread ssh.expect :: Output:
+--------------------------------------+------------------------------+--------+-------------------------------------------------------------+-------+---------------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+------------------------------+--------+-------------------------------------------------------------+-------+---------------+
| d09f462e-5603-4a4a-8734-fec9ac95984c | tenant1-mempool_configured-7 | ACTIVE | tenant1-mgmt-net=192.168.100.154; tenant2-net0=172.18.0.241 | | flavor-mem10g |
+--------------------------------------+------------------------------+--------+-------------------------------------------------------------+-------+---------------+
[sysadmin@controller-1 ~(keystone_admin)]$
[2019-08-06 12:10:37,627] 301 DEBUG MainThread ssh.send :: Send 'ping -c 3 192.168.100.154'
PING 192.168.100.154 (192.168.100.154) 56(84) bytes of data.
64 bytes from 192.168.100.154: icmp_seq=1 ttl=62 time=0.571 ms
64 bytes from 192.168.100.154: icmp_seq=2 ttl=62 time=0.351 ms
64 bytes from 192.168.100.154: icmp_seq=3 ttl=62 time=0.461 ms
Seen this issue multiple times in past week.
In today's sanity, it started to happen on a 2+2 system after a compute lock/unlock test, and went away after another test that involves lock/unlock of both computes.
# lock/unlock compute-1 192.168. 204.2:5000/ v3 --os-user- domain- name Default --os-project- domain- name Default --os-endpoint-type internalURL --os-region-name RegionOne host-lock compute-1'
[2019-08-06 09:28:39,579] 301 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://
[2019-08-06 09:29:12,349] 301 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http:// 192.168. 204.2:5000/ v3 --os-user- domain- name Default --os-project- domain- name Default --os-endpoint-type internalURL --os-region-name RegionOne host-unlock compute-1'
# launch a vm and ping failed. keystone. openstack. svc.cluster. local/v3 --os-user- domain- name Default --os-project- domain- name Default --os-identity- api-version 3 --os-interface internal --os-region-name RegionOne volume create --size=2 --image= fdfe03a5- 6e99-4f9d- 80a2-fb84aa3e05 05 --bootable vol-tis- centos- guest-3'
[2019-08-06 09:38:42,027] 301 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://
[2019-08-06 09:42:30,383] 301 DEBUG MainThread ssh.send :: Send 'ping -c 3 192.168.100.174'
# VM console log shows no IP is assigned on eth0 and metadata server is unreachable from VM +++++++ +++++++ ++++++Net device info+++ +++++++ +++++++ +++++++ +++ --+---- --+---- ------- +------ -----+- ------+ ------- ------- -----+ --+---- --+---- ------- +------ -----+- ------+ ------- ------- -----+ --+---- --+---- ------- +------ -----+- ------+ ------- ------- -----+
[ 69.114638] cloud-init[1146]: Cloud-init v. 0.7.9 running 'init' at Tue, 06 Aug 2019 09:40:12 +0000. Up 69.08 seconds.
[ 69.143033] cloud-init[1146]: ci-info: +++++++
[ 69.146783] cloud-init[1146]: ci-info: +------
[ 69.151250] cloud-init[1146]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
[ 69.156456] cloud-init[1146]: ci-info: +------
[ 69.164276] cloud-init[1146]: ci-info: | lo: | True | 127.0.0.1 | 255.0.0.0 | . | . |
[ 69.173020] cloud-init[1146]: ci-info: | lo: | True | . | . | d | . |
[ 69.178813] cloud-init[1146]: ci-info: | eth0: | True | . | . | . | fa:16:3e:06:6d:c0 |
[ 69.185948] cloud-init[1146]: ci-info: | eth0: | True | . | . | d | fa:16:3e:06:6d:c0 |
[ 69.190294] cloud-init[1146]: ci-info: +------
[ 70.231039] cloud-init[1146]: 2019-08-06 05:40:13,541 - url_helper. py[WARNING] : Calling 'http:// 169.254. 169.254/ 2009-04- 04/meta- data/instance- id' failed [1/30s]: request error [HTTPConnection Pool(host= '169.254. 169.254' , port=80): Max retries exceeded with url: /2009-04- 04/meta- data/instance- id (Caused by NewConnectionEr ror('<requests. packages. urllib3. connection. HTTPConnection object at 0x372abd0>: Failed to establish a new connection: [Errno 101] Network is unreachable',))]
# After lock/unlock of both computes, the issue went away. 192.168. 204.2:5000/ v3 --os-user- domain- name Default --os-project- domain- name Default --os-endpoint-type internalURL --os-region-name RegionOne host-unlock compute-1'
[2019-08-06 11:46:52,269] 301 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://
[2019-08-06 11:57:54,934] 301 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http:// 192.168. 204.2:5000/ v3 --os-user- domain- name Default --os-project- domain- name Default --os-endpoint-type internalURL --os-region-name RegionOne host-unlock compute-0'
# Launch a vm and it is reachable from NatBox api-version 3 --os-interface internal --os-region-name RegionOne server list --a' ------- ------- ------- ------- ----+-- ------- ------- ------- ------- +------ --+---- ------- ------- ------- ------- ------- ------- ------- ------- -+----- --+---- ------- ----+ ------- ------- ------- ------- ----+-- ------- ------- ------- ------- +------ --+---- ------- ------- ------- ------- ------- ------- ------- ------- -+----- --+---- ------- ----+ 5603-4a4a- 8734-fec9ac9598 4c | tenant1- mempool_ configured- 7 | ACTIVE | tenant1- mgmt-net= 192.168. 100.154; tenant2- net0=172. 18.0.241 | | flavor-mem10g | ------- ------- ------- ------- ----+-- ------- ------- ------- ------- +------ --+---- ------- ------- ------- ------- ------- ------- ------- ------- -+----- --+---- ------- ----+ controller- 1 ~(keystone_admin)]$
-os-identity-
[2019-08-06 12:10:37,523] 423 DEBUG MainThread ssh.expect :: Output:
+------
| ID | Name | Status | Networks | Image | Flavor |
+------
| d09f462e-
+------
[sysadmin@
[2019-08-06 12:10:37,627] 301 DEBUG MainThread ssh.send :: Send 'ping -c 3 192.168.100.154'
PING 192.168.100.154 (192.168.100.154) 56(84) bytes of data.
64 bytes from 192.168.100.154: icmp_seq=1 ttl=62 time=0.571 ms
64 bytes from 192.168.100.154: icmp_seq=2 ttl=62 time=0.351 ms
64 bytes from 192.168.100.154: icmp_seq=3 ttl=62 time=0.461 ms
--- 192.168.100.154 ping statistics --- 461/0.571/ 0.089 ms tis-lab- nat-box: ~$
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.351/0.
svc-cgcsauto@
New logs are attached.