wsgi and rpc-server has issues with dhcp when creating subnet

[Ebbex]

I'm not sure how to trace this down further, but with neutron-api running as a wsgi script in a container, and neutron-rpc-server running in a separate container. When creating a subnet in the openstack-cli, e.g. "subnet create --network network1 --subnet-range 10.0.0.0/24 subnet1", the rpc-server logs and dhcp-agent logs fill up with these errors;

ValueError: Unrecognized attribute(s) 'binding:host_id'

I'm attaching some logs with a stack-trace.

[Ebbex (eb4x)]

Could this be related to any extensions? Should the rpc-server possibly load an ExtensionManager?

[Kevin Benton (kevinbenton)]

Is the RPC server actually loading the right core plugin?

[Ebbex (eb4x)]

Here's a snippet form the initialization, it does say "Loading core plugin: ml2", I've noticed that the neutron-api has "Initializing extension manager" which has among other things "Loaded extension: binding"

INFO neutron.common.config [-] Logging enabled!
INFO neutron.common.config [-] /opt/openstack/neutron/venv/bin/neutron-rpc-server version 10.0.3.dev92
INFO neutron.server.rpc_eventlet [-] Eventlet based AMQP RPC server starting...
INFO neutron.manager [-] Loading core plugin: ml2
INFO neutron.plugins.ml2.managers [-] Configured type driver names: ['flat', 'vlan', 'vxlan', 'local']
INFO neutron.plugins.ml2.drivers.type_flat [-] Arbitrary flat physical_network names allowed
INFO neutron.plugins.ml2.drivers.type_vlan [-] Network VLAN ranges: {'public': [(500, 1000)]}
INFO neutron.plugins.ml2.drivers.type_local [-] ML2 LocalTypeDriver initialization complete
INFO neutron.plugins.ml2.managers [-] Loaded type driver names: ['flat', 'vlan', 'local', 'vxlan']
INFO neutron.plugins.ml2.managers [-] Registered types: ['flat', 'vlan', 'local', 'vxlan']
INFO neutron.plugins.ml2.managers [-] Tenant network_types: ['vxlan']
INFO neutron.plugins.ml2.managers [-] Configured extension driver names: ['port_security', 'qos']
INFO neutron.plugins.ml2.managers [-] Loaded extension driver names: ['port_security', 'qos']
INFO neutron.plugins.ml2.managers [-] Registered extension drivers: ['port_security', 'qos']
INFO neutron.plugins.ml2.managers [-] Configured mechanism driver names: ['openvswitch', 'l2population']
INFO neutron.plugins.ml2.managers [-] Loaded mechanism driver names: ['openvswitch', 'l2population']
INFO neutron.plugins.ml2.managers [-] Registered mechanism drivers: ['openvswitch', 'l2population']
INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'flat'
INFO neutron.plugins.ml2.drivers.type_flat [-] ML2 FlatTypeDriver initialization complete
INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'vlan'
INFO neutron.plugins.ml2.drivers.type_vlan [req-aa1d0bd0-ab86-4368-a8a5-542602ba2ff7 - - - - -] VlanTypeDriver initialization complete
INFO neutron.plugins.ml2.managers [req-aa1d0bd0-ab86-4368-a8a5-542602ba2ff7 - - - - -] Initializing driver for type 'local'
INFO neutron.plugins.ml2.managers [req-aa1d0bd0-ab86-4368-a8a5-542602ba2ff7 - - - - -] Initializing driver for type 'vxlan'
INFO neutron.plugins.ml2.drivers.type_tunnel [req-aa1d0bd0-ab86-4368-a8a5-542602ba2ff7 - - - - -] vxlan ID ranges: [(1, 1000)]
INFO neutron.plugins.ml2.managers [req-aa1d0bd0-ab86-4368-a8a5-542602ba2ff7 - - - - -] Initializing extension driver 'port_security'
INFO neutron.plugins.ml2.extensions.port_security [req-aa1d0bd0-ab86-4368-a8a5-542602ba2ff7 - - - - -] PortSecurityExtensionDriver initialization complete
INFO neutron.plugins.ml2.managers [req-aa1d0bd0-ab86-4368-a8a5-542602ba2ff7 - - - - -] Initializing extension driver 'qos'
INFO neutron.plugins.ml2.managers [req-aa1d0bd0-ab86-4368-a8a5-542602ba2ff7 - - - - -] Initializing mechanism driver 'openvswitch'
INFO neutron.plugins.ml2.managers [req-aa1d0bd0-ab86-4368-a8a5-542602ba2ff7 - - - - -] Initializing mechanism driver 'l2population'
INFO neutron.plugins.ml2.plugin [req-87816419-d07f-4...

[Boden R (boden)]

Is the port 'binding' extension [1] running in your neutron server?

IIUC this is the extension that provides the 'bindings:*' attributes for ports, including the 'binding:host_id'. If this extension is not running, then the API won't know how to deal with the bindings related attributes in an API request.

[1] https://github.com/openstack/neutron/blob/master/neutron/extensions/portbindings.py

[Ebbex (eb4x)]

It's running under the neutron-api under wsgi, but not under neutron-rpc-server.

So I tried extending the rpc-server code by adding ext_mgr after manager.init()

manager.init()
ext_mgr = extensions.PluginAwareExtensionManager.get_instance()

Which leads to a lot more output, including "Loaded extension: binding", but it's not triggering get_extended_resources in Portbindings.

It should be noted that I'm using the latest code from the ocata branch.

[Sam Yaple (s8m)]

Was this ever resolved? Running into the same issue

[Sam Yaple (s8m)]

Added info:

running neutron-server, everything works
running neutron-api wsgi (uwsgi/nginx/apache/etc) and neutron-rpc-server, everything works for provider networks, but i get the above stack traces when setting up a tenant network.

I can properly launch instances on a provider network with dhcp enabled in both configurations. It seems to only be tenant networks causing this issue

[Ebbex (eb4x)]

Nope, it's not a viable option to run wsgi/rpc.

We loaded the whole paste app to get past this, but ran into other problems later. https://bugs.launchpad.net/bugs/1687896

Like security groups not being applied at runtime. They will only apply if you restart the neutron-openvswitch-agent. Not sure how the RPC is supposed to work here. It seems that the wsgi part doesn't notify correctly, while the rpc-server is happy to hand out the proper security group configuration on a restart of the ovs-agent.

I should probably write a bug-report on that too.

[Sam Yaple (s8m)]

Thanks for the promt response! I suppose for now, ill roll back to neutron-server

If you wouldn't mind getting me added on any bugs related to this, I am also in a position to iterate and test any patches purposed as a fix.

Also, and again this is just strange, ive seen everything work with wsgi + rpc-server when using provider networks. Something to keep in mind. My testing is not exhaustive though.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/555608
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2f98f56cb5fc0e5400bdb4173c752e0e92c702f5
Submitter: Zuul
Branch: master

commit 2f98f56cb5fc0e5400bdb4173c752e0e92c702f5
Author: Nguyen Phuong An <email address hidden>
Date: Wed Mar 21 09:47:23 2018 +0700

    Allow neutron-api load config from WSGI process

    neutron-api needs to load config files when deployed under
    WSGI process. This patch will do this. neutron-rpc-server
    also need to be aware of plugin extension.

    uWSGI seems not happy with eventlet.GreenPool. So this patch
    will switch to futurist.ThreadPoolExcutor for ML2 OVO push
    notification.

    Partially-implements: blueprint run-in-wsgi-server

    Change-Id: Ia2659c9c8f1deeceefc1c75f1149a1e47c91e7a9
    Closes-Bug: #1708389

[Slawek Kaplonski (slaweq)]

Fix for this is part of blueprint which was completed in Rocky. I don't think we should backport it to stable releases.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 13.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/592931

[OpenStack Infra (hudson-openstack)]

Change abandoned by Iago Santos (<email address hidden>) on branch: stable/queens
Review: https://review.openstack.org/592931