kuryr-libnetwork IPAM gateway address creation fails

[Matt McEuen]

When I follow the kuryr-libnetwork readme instructions, and try to create a test network per the guide:
    docker network create --driver kuryr --ipam-driver kuryr --subnet 10.10.0.0/16 test_net

I get the error:
    Error response from daemon: failed to allocate gateway (): No address returned

I'm running Kuryr master against a Kolla-deployed Newton multinode cluster, and I get the same behavior running the prebuilt container on baremetal (veth), custom-built Kuryr on baremetal (veth), and custom-built in a VM (macvlan).

I get these logs [1], which show:
  /IpamDriver.RequestPool getting called (successfully creates a subnet pool)
  /IpamDriver.RequestAddress getting called (to allocate a gateway address)
  /IpamDriver.ReleasePool getting called (rolling back)

I see the subnet pools getting created in Neutron before getting rolled back [2].

Stepping through the code, I see that Docker is rolling back (ReleasePool) because RequestAddress is returning an allocated_address of '' (hence "No address returned"). The blocks where allocated_address would get set get skipped over:
  Here [3], because subnet doesn't get set - the subnet hasn't been created in Neutron yet
  Here [4], because a blank req_address is passed in from Docker.

Here is the JSON sent from Docker:
{u'Address': u'', u'Options': {u'RequestAddressType': u'com.docker.network.gateway'}, u'PoolID': u'2d50bdc6-4fd8-4e11-930b-844ea3336bcf'}

Please let me know if there's anything I could have misconfigured on my side that would cause something like this. I'd be happy to help fix this if someone could point me in the right direction.

[1]: http://paste.openstack.org/show/593632/
[2]: http://paste.openstack.org/show/593633/
[3]: https://github.com/openstack/kuryr-libnetwork/blob/master/kuryr_libnetwork/controllers.py#L1345
[4]: https://github.com/openstack/kuryr-libnetwork/blob/master/kuryr_libnetwork/controllers.py#L1416

[Dongcan Ye (hellochosen)]

Hello, Matt McEuen. Do you specify "--gateway" when you creating network?

[Matt McEuen (mm9745)]

Hi Dongcan -- no, I didn't supply the "--gateway" parameter.

The libnetwork IPAM documentation [1] says "On network creation, libnetwork will... request the network gateway address if specified. Otherwise [it will] request any address from the pool to be used as network gateway." If I'm reading this right, that means RequestAddress will be called to acquire a gateway address in any case.

[1]: https://github.com/docker/libnetwork/blob/master/docs/ipam.md

[Dongcan Ye (hellochosen)]

Hi, Matt. I'm not sure this is correct in kuryr-libnetwork, following the instructions of kuryr, we need to pass "--gateway" when we creating network.

[Matt McEuen (mm9745)]

I think you may be right, Dongcan. When I pass the "--gateway" argument, network creation works fine!

The next question is: is this just a kuryr-libnetwork documentation issue ("--gateway" should always be passed); or alternately, if "--gateway" is omitted, should a default gateway be provisioned?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/420493
Committed: https://git.openstack.org/cgit/openstack/kuryr-libnetwork/commit/?id=9cdb9a3f309c03fe20d0ff2f1f240eb4b07b90f5
Submitter: Jenkins
Branch: master

commit 9cdb9a3f309c03fe20d0ff2f1f240eb4b07b90f5
Author: mattmceuen <email address hidden>
Date: Sun Jan 15 21:56:31 2017 -0600

    Add --gateway to README network create examples

    The --gateway parameter needs to be passed to docker network create
    when using kuryr-libnetwork. This changes updates the README to
    consistently use --gateway in its examples.

    Change-Id: I9dd02f5df6d9886fe5e6c1365843bac5ad62e0c7
    Closes-Bug: #1653178