Get error NetworkAlreadyExists if network is unsync between docker and Zun DB

[hongbin]

Steps to reproduce:

$ source /opt/stack/devstack/openrc demo demo
$ openstack network create test
$ NET_ID=$(openstack network show test -c id -f value)
$ openstack subnet create --subnet-range 10.10.10.0/24 --network $NET_ID testsubnet
$ zun run --name testcontainer --net network=$NET_ID nginx
$ zun delete --stop testcontainer
$ docker network rm $NET_ID
$ zun run --name testcontainer --net network=$NET_ID nginx # this container will fail

Above create a network in neutron, run a container on that network. Then, use docker command to remove the network. This make the Zun DB inconsistent with docker. Then, run another container on that network will fail.

The following error on the zun_compute's log:

Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: Traceback (most recent call last):
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 457, in fire_timers
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: timer()
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 58, in __call__
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: cb(*args, **kw)
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: File "/opt/stack/zun/zun/common/utils.py", line 199, in context_wrapper
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: func(*args, **kwargs)
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 274, in inner
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: return f(*args, **kwargs)
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: File "/opt/stack/zun/zun/compute/manager.py", line 270, in do_container_create
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: pci_requests, limits)
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: File "/opt/stack/zun/zun/common/utils.py", line 659, in decorated_function
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: return function(self, context, *args, **kwargs)
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: File "/opt/stack/zun/zun/compute/manager.py", line 369, in _do_container_create
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: sandbox, limits)
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: File "/opt/stack/zun/zun/compute/manager.py", line 349, in _do_container_create_base
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: unset_host=True)
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: self.force_reraise()
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: six.reraise(self.type_, self.value, self.tb)
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: File "/opt/stack/zun/zun/compute/manager.py", line 333, in _do_container_create_base
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: requested_volumes)
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: File "/opt/stack/zun/zun/container/docker/driver.py", line 254, in create
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: network_api = zun_network.api(context=context, docker_api=docker)
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: File "/opt/stack/zun/zun/container/docker/driver.py", line 394, in _provision_network
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]:
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: File "/opt/stack/zun/zun/container/docker/driver.py", line 1060, in _get_or_create_docker_network
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: neutron_net_id)
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: File "/opt/stack/zun/zun/network/kuryr_network.py", line 121, in create_network
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: network.create(self.context)
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: return fn(self, *args, **kwargs)
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: File "/opt/stack/zun/zun/objects/network.py", line 75, in create
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: db_network = dbapi.create_network(context, values)
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 159, in wrapper
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: result = f(*args, **kwargs)
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: File "/opt/stack/zun/zun/db/api.py", line 1008, in create_network
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: return _get_dbdriver_instance().create_network(context, values)
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: File "/opt/stack/zun/zun/db/sqlalchemy/api.py", line 1198, in create_network
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: field='neutron_net_id', value=values['neutron_net_id'])
Oct 10 02:59:39 ip-172-31-1-38 zun-compute[16487]: NetworkAlreadyExists: A network with neutron_net_id 3ff32202-3a88-4d0b-82d2-7b5029429c9f already exists.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/609204

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/609204
Committed: https://git.openstack.org/cgit/openstack/zun/commit/?id=a156577f1b2c154770ac4d586753a98403ce220b
Submitter: Zuul
Branch: master

commit a156577f1b2c154770ac4d586753a98403ce220b
Author: Hongbin Lu <email address hidden>
Date: Wed Oct 10 03:22:31 2018 +0000

    Handle network is already created in DB

    Change-Id: Iea68870add0bcf80e58ac7d7a93e3e87fcaefb03
    Closes-Bug: #1797022

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/zun 3.0.0.0rc1 release candidate.