Attach an already-attached network stuck

[hongbin]

Create a container. Attach the container to an already attached network

  $ openstack appcontainer run --name testcontainer --net network=private --wait nginx
  $ openstack appcontainer network attach --network private testcontainer

The container 'task_state' stuck at 'network_attaching' forever.

  $ openstack appcontainer list
+--------------------------------------+---------------+-------+---------+-------------------+-------------------------------------------------+-------+
| uuid | name | image | status | task_state | addresses | ports |
+--------------------------------------+---------------+-------+---------+-------------------+-------------------------------------------------+-------+
| 4aec78e3-0b19-48ed-b551-a2187eb829ac | testcontainer | nginx | Running | network_attaching | 10.0.0.16, fda5:9aad:d2d0:0:f816:3eff:fe39:3f0c | [80] |
+--------------------------------------+---------------+-------+---------+-------------------+-------------------------------------------------+-------+

The following error in the server log:

Jan 21 02:58:18 ip-172-31-7-192 zun-compute[20907]: ERROR zun.common.utils [None req-5edaa73a-d40a-4317-9af2-5b0323f18b3a demo demo] Unexpected exception: Container 4aec78e3-0b19-48ed-b551-a2187eb829ac has alreay connected to the network eb459960-3c94-48a5-a4c8-072dadd665b8.: ZunException: Container 4aec78e3-0b19-48ed-b551-a2187eb829ac has alreay connected to the network eb459960-3c94-48a5-a4c8-072dadd665b8.
Jan 21 02:58:18 ip-172-31-7-192 zun-compute[20907]: ERROR zun.common.utils Traceback (most recent call last):
Jan 21 02:58:18 ip-172-31-7-192 zun-compute[20907]: ERROR zun.common.utils File "/opt/stack/zun/zun/common/utils.py", line 675, in decorated_function
Jan 21 02:58:18 ip-172-31-7-192 zun-compute[20907]: ERROR zun.common.utils return function(self, context, container, *args, **kwargs)
Jan 21 02:58:18 ip-172-31-7-192 zun-compute[20907]: ERROR zun.common.utils File "/opt/stack/zun/zun/common/utils.py", line 664, in decorated_function
Jan 21 02:58:18 ip-172-31-7-192 zun-compute[20907]: ERROR zun.common.utils return function(self, context, *args, **kwargs)
Jan 21 02:58:18 ip-172-31-7-192 zun-compute[20907]: ERROR zun.common.utils File "/opt/stack/zun/zun/compute/manager.py", line 1294, in _do_network_attach
Jan 21 02:58:18 ip-172-31-7-192 zun-compute[20907]: ERROR zun.common.utils self.driver.network_attach(context, container, requested_network)
Jan 21 02:58:18 ip-172-31-7-192 zun-compute[20907]: ERROR zun.common.utils File "/opt/stack/zun/zun/container/docker/driver.py", line 1231, in network_attach
Jan 21 02:58:18 ip-172-31-7-192 zun-compute[20907]: ERROR zun.common.utils 'network': network})
Jan 21 02:58:18 ip-172-31-7-192 zun-compute[20907]: ERROR zun.common.utils ZunException: Container 4aec78e3-0b19-48ed-b551-a2187eb829ac has alreay connected to the network eb459960-3c94-48a5-a4c8-072dadd665b8.
Jan 21 02:58:18 ip-172-31-7-192 zun-compute[20907]: ERROR zun.common.utils
Jan 21 02:58:18 ip-172-31-7-192 zun-compute[20907]: DEBUG oslo_concurrency.lockutils [None req-5edaa73a-d40a-4317-9af2-5b0323f18b3a demo demo] Lock "4aec78e3-0b19-48ed-b551-a2187eb829ac" released by "zun.compute.manager.do_network_attach" :: held 0.062s {{(pid=20907) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:339}}

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/632012

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/632016

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/632012
Committed: https://git.openstack.org/cgit/openstack/zun/commit/?id=58218c1d0baf0ad34f62d47748f9a1c6353d8a24
Submitter: Zuul
Branch: master

commit 58218c1d0baf0ad34f62d47748f9a1c6353d8a24
Author: Hongbin Lu <email address hidden>
Date: Mon Jan 21 03:47:28 2019 +0000

    Always reset task state to None

    Upon the completion of attaching/detaching network, always set the
    task_state to none. This ensures the task_state is cleanup on
    exception.

    Change-Id: Ia664df05b13090b28be7dcfc44649954bb4c0f9d
    Closes-Bug: #1812602

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/632016
Committed: https://git.openstack.org/cgit/openstack/zun/commit/?id=ced5e9e3d82c68037cc90442f555adcc8c2a5d76
Submitter: Zuul
Branch: master

commit ced5e9e3d82c68037cc90442f555adcc8c2a5d76
Author: Hongbin Lu <email address hidden>
Date: Mon Jan 21 05:11:59 2019 +0000

    Add validation on network attach/detach

    Change-Id: I9e4987aec424bfaa66838662eccd69799351ec37
    Related-Bug: #1812602

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/634097

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/634097
Committed: https://git.openstack.org/cgit/openstack/zun/commit/?id=d2459b08ba34009c0a7f7679ed3d38536d3381be
Submitter: Zuul
Branch: stable/rocky

commit d2459b08ba34009c0a7f7679ed3d38536d3381be
Author: Hongbin Lu <email address hidden>
Date: Mon Jan 21 05:11:59 2019 +0000

    Add validation on network attach/detach

    Change-Id: I9e4987aec424bfaa66838662eccd69799351ec37
    Related-Bug: #1812602
    (cherry picked from commit ced5e9e3d82c68037cc90442f555adcc8c2a5d76)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/zun 3.0.0.0rc1 release candidate.