Openstack cannot start the container properly after disabling the network gateway

[hezhiqiang]

[root@controller-0-10-10-100-2 ~]# cat /var/log/zun/zun-compute.log | grep -i error
2020-06-03 21:37:36.879 8150 ERROR zun.compute.manager [req-6fce7657-c03b-43b1-8913-75450951e8bc 460f829c337d4a818a1c074c561fccbf d03454bd2a844c138c514c97775f1854 default - -] Error occurred while calling Docker create API: Docker internal error: 500 Server Error: Internal Server Error ("failed to allocate gateway (): IpamDriver.RequestAddress: Requested gateway does not match with gateway None in existed network.").: DockerError: Docker internal error: 500 Server Error: Internal Server Error ("failed to allocate gateway (): IpamDriver.RequestAddress: Requested gateway does not match with gateway None in existed network.").

[hongbin (hongbin034)]

Hi @hezhiqiang,

Several questions:

* How did you install Zun (i.e. manual install, kolla, devstack)
* Which version you installed (i.e. master, stable/train, ...)
* What are the exact steps to reproduce the problem?
* Could you attach the log of kuryr-libnetwork?

[hezhiqiang (hezhiqiang)]

I disabled the gateway to the network
kuryr-lib 2.0.0
kuryr-libnetwork 5.1.0.dev10

[hezhiqiang (hezhiqiang)]

manual install

[hezhiqiang (hezhiqiang)]

[root@controller-0-10-10-100-2 ~]# cat /var/log/kuryr.log
2020-06-05 16:10:49.841 6076 INFO werkzeug [-] * Running on http://127.0.0.1:23750/ (Press CTRL+C to quit)
2020-06-05 16:11:45.145 6076 INFO kuryr_libnetwork.controllers [-] Creating subnetpool with the given pool CIDR
2020-06-05 16:11:45.657 6076 INFO werkzeug [-] 127.0.0.1 - - [05/Jun/2020 16:11:45] "POST /IpamDriver.RequestPool HTTP/1.1" 200 -
2020-06-05 16:11:45.907 6076 ERROR kuryr_libnetwork.utils [-] Unexpected error happened: Requested gateway does not match with gateway None in existed network.: GatewayConflictFailure: Requested gateway does not match with gateway None in existed network.
2020-06-05 16:11:45.909 6076 INFO werkzeug [-] 127.0.0.1 - - [05/Jun/2020 16:11:45] "POST /IpamDriver.RequestAddress HTTP/1.1" 500 -
2020-06-05 16:11:46.262 6076 INFO werkzeug [-] 127.0.0.1 - - [05/Jun/2020 16:11:46] "POST /IpamDriver.ReleasePool HTTP/1.1" 200 -

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/733992

[hezhiqiang (hezhiqiang)]

thank you

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/733992
Committed: https://git.openstack.org/cgit/openstack/kuryr-libnetwork/commit/?id=960993652e1ab526c1ce8cd3cd04d28f13facc86
Submitter: Zuul
Branch: master

commit 960993652e1ab526c1ce8cd3cd04d28f13facc86
Author: Hongbin Lu <email address hidden>
Date: Sun Jun 7 20:43:22 2020 +0000

    Handle subnet without gateway

    * When libnetwork requests the IP address of the gateway
      (via /IpamDriver.RequestAddress) and the neutron subnet has gateway
      as None, return '0.0.0.0/0' as a placeholder.
    * Disable default docker gateway via the 'DisableGatewayService' flag
      (see https://github.com/moby/libnetwork/pull/778).

    Change-Id: I3033d28eb268a01de8cf038b1ed20110ca9a31ea
    Closes-Bug: #1881910

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/738293

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/738294

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/738295

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/738296

[hezhiqiang (hezhiqiang)]

ok

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/738294
Committed: https://git.openstack.org/cgit/openstack/kuryr-libnetwork/commit/?id=862c7e73edb44a70d37ae3247c0ba549054b8c60
Submitter: Zuul
Branch: stable/train

commit 862c7e73edb44a70d37ae3247c0ba549054b8c60
Author: Hongbin Lu <email address hidden>
Date: Sun Jun 7 20:43:22 2020 +0000

    Handle subnet without gateway

    * When libnetwork requests the IP address of the gateway
      (via /IpamDriver.RequestAddress) and the neutron subnet has gateway
      as None, return '0.0.0.0/0' as a placeholder.
    * Disable default docker gateway via the 'DisableGatewayService' flag
      (see https://github.com/moby/libnetwork/pull/778).

    Change-Id: I3033d28eb268a01de8cf038b1ed20110ca9a31ea
    Closes-Bug: #1881910
    (cherry picked from commit 960993652e1ab526c1ce8cd3cd04d28f13facc86)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/738293
Committed: https://git.openstack.org/cgit/openstack/kuryr-libnetwork/commit/?id=9b164c51929792d7ea0a1e349858bc04cc67860c
Submitter: Zuul
Branch: stable/ussuri

commit 9b164c51929792d7ea0a1e349858bc04cc67860c
Author: Hongbin Lu <email address hidden>
Date: Sun Jun 7 20:43:22 2020 +0000

    Handle subnet without gateway

    * When libnetwork requests the IP address of the gateway
      (via /IpamDriver.RequestAddress) and the neutron subnet has gateway
      as None, return '0.0.0.0/0' as a placeholder.
    * Disable default docker gateway via the 'DisableGatewayService' flag
      (see https://github.com/moby/libnetwork/pull/778).

    Change-Id: I3033d28eb268a01de8cf038b1ed20110ca9a31ea
    Closes-Bug: #1881910
    (cherry picked from commit 960993652e1ab526c1ce8cd3cd04d28f13facc86)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/738295
Committed: https://git.openstack.org/cgit/openstack/kuryr-libnetwork/commit/?id=c8af0405a950e15dd675212b0a7372a547be4b96
Submitter: Zuul
Branch: stable/stein

commit c8af0405a950e15dd675212b0a7372a547be4b96
Author: Hongbin Lu <email address hidden>
Date: Sun Jun 7 20:43:22 2020 +0000

    Handle subnet without gateway

    * When libnetwork requests the IP address of the gateway
      (via /IpamDriver.RequestAddress) and the neutron subnet has gateway
      as None, return '0.0.0.0/0' as a placeholder.
    * Disable default docker gateway via the 'DisableGatewayService' flag
      (see https://github.com/moby/libnetwork/pull/778).

    Change-Id: I3033d28eb268a01de8cf038b1ed20110ca9a31ea
    Closes-Bug: #1881910
    (cherry picked from commit 960993652e1ab526c1ce8cd3cd04d28f13facc86)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/738296
Committed: https://git.openstack.org/cgit/openstack/kuryr-libnetwork/commit/?id=2d5158767121922a38577a0c8abdf8c383bf7bcd
Submitter: Zuul
Branch: stable/rocky

commit 2d5158767121922a38577a0c8abdf8c383bf7bcd
Author: Hongbin Lu <email address hidden>
Date: Sun Jun 7 20:43:22 2020 +0000

    Handle subnet without gateway

    * When libnetwork requests the IP address of the gateway
      (via /IpamDriver.RequestAddress) and the neutron subnet has gateway
      as None, return '0.0.0.0/0' as a placeholder.
    * Disable default docker gateway via the 'DisableGatewayService' flag
      (see https://github.com/moby/libnetwork/pull/778).

    Change-Id: I3033d28eb268a01de8cf038b1ed20110ca9a31ea
    Closes-Bug: #1881910
    (cherry picked from commit 960993652e1ab526c1ce8cd3cd04d28f13facc86)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/733996
Committed: https://git.openstack.org/cgit/openstack/zun-tempest-plugin/commit/?id=1e6bcd4759a8c0ab20b8d51c3dec3a96b0a892f1
Submitter: Zuul
Branch: master

commit 1e6bcd4759a8c0ab20b8d51c3dec3a96b0a892f1
Author: Hongbin Lu <email address hidden>
Date: Sun Jun 7 23:36:56 2020 +0000

    Test container in subnet without gateway_ip

    Related-Bug: #1881910
    Change-Id: I89a25883b6341357d6d98b60445eba9d0fe69623

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kuryr-libnetwork 4.0.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kuryr-libnetwork 5.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kuryr-libnetwork rocky-eol release.