BindingNotSupportedFailure: vif_type(binding_failed) is not supported

[Edvinas Sulzickis]

Hello, I used kuryr-libnetwork successfully for about a month, but unfortunately after full openstack reinstallation using juju I got error when trying to run docker container using kuryr-libnetwork directly on compute node. My juju bundle is the same like it used to be when it ran smoothly. Journalctl output is attached below

docker version 19.03.2

In /usr/local/libexec directory I have:

bridge iovisor midonet ovs tap unbound

* Installed Zun manually (source: https://docs.openstack.org/zun/stein/install/compute-install.html)
* stable/stein version of Zun and Kuryr-libnetwork
* Installed Kuryr-libnetwork manually (source: https://docs.openstack.org/kuryr-libnetwork/stein/install/compute-install-ubuntu.html)
* OS Ubuntu 18.04.3

[Edvinas Sulzickis (madness31)]

kuryr-libnetwork config file

[hongbin (hongbin034)]

It looks the port binding failed on neutron side, which is why vif_type is set to 'binding_failed'. Could you check neutron logs to see if there is any error message?

[Edvinas Sulzickis (madness31)]

I attached some logs from neutron, although I don't think the error I get is because of neutron. The thing is that when I ssh to compute node and run directly command "docker run --net test_net cirros ifconfig" on compute host (not via Openstack) I get this error. So I think neutron should't be the case

[Edvinas Sulzickis (madness31)]

In addition to, I can assure that I am able to create nova instances on nodes without errors

[Edvinas Sulzickis (madness31)]

Okay. So I can now explain how to fully reproduce this error. First I am adding unit with nova-compute command and completely reinstall OS on compute node

1) juju add-unit nova-compute --to computei7-2.maas node

During step above nova-compute, ntp, neutron-openvswitch is installed on a completely fresh node

2) Installing docker from repo:

https://docs.docker.com/install/linux/docker-ce/ubuntu/

3) Because I have already installed kuryr on controller (https://docs.openstack.org/kuryr-libnetwork/stein/install/controller-install.html) I only install kuryr on compute using this guide:

https://docs.openstack.org/kuryr-libnetwork/stein/install/compute-install-ubuntu.html

4) When creating docker network (https://docs.openstack.org/kuryr-libnetwork/stein/install/verify.html)

docker network create --driver kuryr --ipam-driver kuryr \
      --subnet 10.10.0.0/16 --gateway=10.10.0.1 test_net

Error response from daemon: IpamDriver.RequestPool: Another pool with same cidr exist. ipam and network options not used to pass pool name

Although I don't have the same cidr on a compute node because I have inspected all of my networks

NETWORK ID NAME DRIVER SCOPE
4bdc0dd4bef0 bridge bridge local
71bbf6f23bdd host host local
94b75f0199ff none null local

After I change it for example

docker network create --driver kuryr --ipam-driver kuryr \
      --subnet 10.11.0.0/16 --gateway=10.11.0.1 test_net

I get:

b70cff92465b03f52cfba47f2333519fcd775a66fbc545d2d4956c70df4b9b64

5) Finally, when I enter command "sudo docker run --net test_net cirros ifconfig"

I get:

docker: Error response from daemon: failed to create endpoint cocky_franklin on network test_net: NetworkDriver.CreateEndpoint: vif_type(binding_failed) is not supported. A binding script for this type can't be found.
ERRO[0012] error waiting for container: context canceled

So, like I said even if I install kuryr-libnetwork on a clean compute node I get the same error

[Edvinas Sulzickis (madness31)]

What I noticed is that when I try to create docker container within test_net network in a neutron-openvswitch logs I see this message:

INFO neutron.agent.common.ovs_lib [req-6bb7efbc-1cc7-4b0b-8819-a61f3ae1c175 - - - - -] Port a56f0e7f-e564-42ad-a752-0b32db37f919 not present in bridge br-int

Then, I check openstack port list using cli and get and do not see port with ID a56f0e7f-e564-42ad-a752-0b32db37f919

Howerver, when creating nova instance in neutron-openvswitch logs I see that it is port is being created and openstack port list command returns the same port ID as it was mentioned in neutron-openvswitch. Therefore, VM is created successfully

[Edvinas Sulzickis (madness31)]

So in neutron-openvswitch logs (on compute node) I see mentioned port a56f0e7f-e564-42ad-a752-0b32db37f919 from message:

INFO neutron.agent.common.ovs_lib [req-6bb7efbc-1cc7-4b0b-8819-a61f3ae1c175 - - - - -] Port a56f0e7f-e564-42ad-a752-0b32db37f919 not present in bridge br-int

It's ID differs from port ID's listed in openstack port list. In openstack port list I see that kuryr network port was created, but it's own ID is different (cbec64ec-70ab-4d66-ae5e-4c3d3b271999) from neutron-openvswitch. In other words, Neutron tries to find port with a56f0e7f-e564-42ad-a752-0b32db37f919 ID instead of using cbec64ec-70ab-4d66-ae5e-4c3d3b271999

[Edvinas Sulzickis (madness31)]

From this link http://eavesdrop.openstack.org/irclogs/%23openstack-kuryr/%23openstack-kuryr.2017-02-20.log.html I see that problem could be that neutron was not able to identify how to bind the port (vif type), therefore local kuryr fails to find the proper script. But it is very strange if problem will be because of neutron .conf files. I did not change anything there before it was working for some time. Kuryr-libnetwork worked before without any neutron files configuration

[Edvinas Sulzickis (madness31)]

Any help?

[Edvinas Sulzickis (madness31)]

The only error I found in neutron logs is this:

ERROR neutron.plugins.ml2.managers [req-d5bb3e97-40b3-4261-aaea-dadad88537bf b3e7880917484f6899d79bdfa9f91cfc 0043ed77cd8b453d897d499c457d2247 - 3a3cd562d047408fbf743d22f2f22eb8 3a3cd562d047408fbf743d22f2f22eb8] Failed to bind port 2661688f-90b2-4c72-bd32-58acaf1c9a2c on host computei7-2 for vnic_type normal using segments [{'id': '9f27f7c7-414c-4012-bed8-efe802ef1d31', 'network_type': 'gre', 'physical_network': None, 'segmentation_id': 43, 'network_id': 'cf5e6f43-2a0b-4015-8b8f-5fe8be335056'}]

[Edvinas Sulzickis (madness31)]

And does this error from neutron-server says something for you? This error
is the only one I get from all logs

ERROR neutron.plugins.ml2.managers
[req-d5bb3e97-40b3-4261-aaea-dadad88537bf b3e7880917484f6899d79bdfa9f91cfc
0043ed77cd8b453d897d499c457d2247 - 3a3cd562d047408fbf743d22f2f22eb8
3a3cd562d047408fbf743d22f2f22eb8] Failed to bind port
2661688f-90b2-4c72-bd32-58acaf1c9a2c on host computei7-2 for vnic_type
normal using segments [{'id': '9f27f7c7-414c-4012-bed8-efe802ef1d31',
'network_type': 'gre', 'physical_network': None, 'segmentation_id': 43,
'network_id': 'cf5e6f43-2a0b-4015-8b8f-5fe8be335056'}]

On Sat, 30 Nov 2019 at 18:25, hongbin <email address hidden> wrote:

> It looks the port binding failed on neutron side, which is why vif_type
> is set to 'binding_failed'. Could you check neutron logs to see if there
> is any error message?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1854461
>
> Title:
> BindingNotSupportedFailure: vif_type(binding_failed) is not supported
>
> Status in kuryr-libnetwork:
> New
>
> Bug description:
> Hello, I used kuryr-libnetwork successfully for about a month, but
> unfortunately after full openstack reinstallation using juju I got
> error when trying to run docker container using kuryr-libnetwork
> directly on compute node. My juju bundle is the same like it used to
> be when it ran smoothly. Journalctl output is attached below
>
> docker version 19.03.2
>
> In /usr/local/libexec directory I have:
>
> bridge iovisor midonet ovs tap unbound
>
> * Installed Zun manually (source:
> https://docs.openstack.org/zun/stein/install/compute-install.html)
> * stable/stein version of Zun and Kuryr-libnetwork
> * Installed Kuryr-libnetwork manually (source:
> https://docs.openstack.org/kuryr-libnetwork/stein/install/compute-install-ubuntu.html
> )
> * OS Ubuntu 18.04.3
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/kuryr-libnetwork/+bug/1854461/+subscriptions
>

[Edvinas Sulzickis (madness31)]

After all this time I finaly found out difference between error message shown node name and node name in "openstack compute service list" command. In error above neutron tries to bind port on host computei7-2 although when I type openstack compute service list I see only computei7-2.maas

What I did was:

1) Changed /etc/nova/nova.conf file host line in compute node instead of computei7-2.maas to computei7

2) Changed /etc/nova/neutron.conf file host line in compute node instead of computei7-2.maas to computei7

3) Reboot compute node

4) In openstack controller removed compute service with a name ending .maas:

openstack compute service delete $ID

5) Rebooted a node

After all these procedures I am able to launch docker container on compute node with command "docker run --net test_net cirros ifconfig"