Comment 8 for bug 1740241

Revision history for this message
Jacolex (jacolex) wrote :

Hi
I have the same issue. The only way to fix I found is run openstack port delete and then attach new port and update fixed ip for it (detaching interface is not working, because conmpute node throws error that port is not attached).

Example of broken network on instance.
# openstack server show 248d1672-88e2-43c2-93f3-379884109f23
+-------------------------------------+----------------------------------------------------------+
| Field | Value |
+-------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | ----- |
| OS-EXT-SRV-ATTR:host | lab-cn-h115 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | lab-cn-h115 |
| OS-EXT-SRV-ATTR:instance_name | instance-0000f3c7 |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2017-08-30T13:55:34.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | |
| config_drive | |
| created | 2017-08-30T13:54:47Z |
| flavor | s.16VCPU_32GB (s16-32) |
| hostId | 5915f20ff98db4c228b7415339917b2b9d5673e43f25a5a7b205523b |
| id | 248d1672-88e2-43c2-93f3-379884109f23 |
| image | |
| key_name | ... |
| name | instancename |
| progress | 0 |
| project_id | 8cb1b772ccbf4d3d82d6a72f31053564 |
| properties | |
| security_groups | name='ssh' |
| | name='default' |
| status | ACTIVE |
| updated | 2017-08-30T13:55:35Z |
| user_id | 09ea15ab0c2e406fba6abfb9f63a3f61 |
| volumes_attached | id='de54a17a-673c-4397-b1e3-b16ebf2eea15' |
+-------------------------------------+----------------------------------------------------------+

# openstack port show 773ea191-642b-479a-aa78-0cb0dc29e0e7
+-----------------------+----------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+----------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| allowed_address_pairs | |
| binding_host_id | lab-cn-h115 |
| binding_profile | |
| binding_vif_details | ovs_hybrid_plug='True', port_filter='True' |
| binding_vif_type | ovs |
| binding_vnic_type | normal |
| created_at | 2017-08-30T13:54:50Z |
| description | |
| device_id | 248d1672-88e2-43c2-93f3-379884109f23 |
| device_owner | compute:- |
| dns_assignment | fqdn='instancename.blabla', hostname='instancename', ip_address='192.168.0.12' |
| dns_name | instancename |
| extra_dhcp_opts | |
| fixed_ips | ip_address='192.168.0.12', subnet_id='8a6ef96a-fdb2-4396-86c8-a52afdf3ebd0' |
| id | 773ea191-642b-479a-aa78-0cb0dc29e0e7 |
| ip_address | None |
| mac_address | fa:16:3e:6c:9a:23 |
| name | |
| network_id | 49194a0d-a9b6-441e-a425-881a6f8a9802 |
| option_name | None |
| option_value | None |
| port_security_enabled | False |
| project_id | 8cb1b772ccbf4d3d82d6a72f31053564 |
| qos_policy_id | None |
| revision_number | 460 |
| security_groups | c3ca1eec-1dde-4982-a710-53c987deee9e, c86ac4c1-7ffc-4ce7-9064-ed9bc6e1ad55 |
| status | ACTIVE |
| subnet_id | None |
| updated_at | 2017-12-20T13:18:46Z |
+-----------------------+----------------------------------------------------------------------------------------------+

No info_cache in database:
mysql> select network_info from instance_info_caches where instance_uuid='248d1672-88e2-43c2-93f3-379884109f23';
+--------------+
| network_info |
+--------------+
| [] |
+--------------+
1 row in set (0.00 sec)

Instance starts without network ports.

The compute node is not refreshing info cache, because it is already broken:

2018-03-22 11:25:01.300 54731 DEBUG nova.network.base_api [req-c4fafbdd-7ab8-4327-8fa9-2ac887599066 6e603c5e51d64db1afb0b36ce597a6f3 8cb1b772ccbf4d3d82d6a72f31053564 - - -] [instance: 248d1672-88e2-43c2-93f3-379884109f23] Updating instance_info_cache with network_info:
[] update_instance_cache_with_nw_info /usr/lib/python2.7/dist-packages/nova/network/base_api.py:48

I can't reproduce the problem as Gleb Zimin, but I experienced it from time to time on several instances. The bug is critical for me.