mitaka b2: neutron->nova external event notification "Service catalog is empty"

[James Page]

With the same configuration that is confirmed to work with mitaka b1, I'm seeing the following error with mitaka b2 - neutron is unable to send external event notifications back to nova:

2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova [-] Failed to notify nova on events: [{'status': 'completed', 'tag': u'8cbc4861-97cb-484e-bd10-fd6a226c332a', 'name': 'network-vif-plugged', 'server_uuid': u'2831a5ed-26f9-42ed-bb9d-363b1117307f'}]
2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova Traceback (most recent call last):
2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/neutron/notifiers/nova.py", line 209, in send_events
2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova batched_events)
2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/novaclient/v2/contrib/server_external_events.py", line 39, in create
2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova return_raw=True)
2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 169, in _create
2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova _resp, body = self.api.client.post(url, body=body)
2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 176, in post
2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova return self.request(url, 'POST', **kwargs)
2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 91, in request
2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova **kwargs)
2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 206, in request
2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 95, in request
2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova return self.session.request(url, method, **kwargs)
2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/keystoneauth1/_utils.py", line 180, in inner
2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova return func(*args, **kwargs)
2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 345, in request
2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova base_url = self.get_endpoint(auth, **endpoint_filter)
2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 615, in get_endpoint
2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova return auth.get_endpoint(self, **kwargs)
2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 213, in get_endpoint
2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova service_name=service_name)
2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/keystoneauth1/_utils.py", line 180, in inner
2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova return func(*args, **kwargs)
2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/keystoneauth1/access/service_catalog.py", line 190, in url_for
2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova raise exceptions.EmptyCatalog('The service catalog is empty.')
2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova EmptyCatalog: The service catalog is empty.
2016-02-24 10:36:45.844 26152 ERROR neutron.notifiers.nova

neutron.conf:

[DEFAULT]
verbose = False
debug = False
use_syslog = False
state_path = /var/lib/neutron
bind_host = 0.0.0.0
auth_strategy = keystone
notification_driver = neutron.openstack.common.notifier.rpc_notifier
api_workers = 2
rpc_workers = 2

router_distributed = False

l3_ha = False
bind_port = 9686
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin
service_plugins = router,firewall,lbaas,vpnaas,metering
allow_overlapping_ips = True
neutron_firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
notify_nova_on_port_status_changes = True
notify_nova_on_port_data_changes = True

[nova]
auth_plugin = password
region_name = RegionOne
auth_url = http://10.5.3.86:35357/v2.0
username = neutron
password = XXX
tenant_id = 8679f8352c4b4be0a7edda3b41099731

[quotas]
quota_driver = neutron.db.quota_db.DbQuotaDriver
quota_items = network,subnet,port,security_group,security_group_rule
quota_security_group = 10
quota_security_group_rule = 100
quota_network = 10
quota_subnet = 10
quota_port = 50
quota_vip = 10
quota_pool = 10
quota_member = -1
quota_health_monitors = -1
quota_router = 10
quota_floatingip = 50

[agent]
root_helper = sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf

[keystone_authtoken]
identity_uri = http://10.5.3.86:35357/
auth_uri = http://10.5.3.86:5000/
admin_tenant_name = services
admin_user = neutron
admin_password = XXXX
signing_dir = /var/cache/neutron

[database]
connection = mysql://neutron:XXXX@10.5.3.88/neutron
max_pool_size = 8

[oslo_messaging_rabbit]
rabbit_userid = neutron
rabbit_virtual_host = openstack
rabbit_password = XXX
rabbit_host = 10.5.3.96

[oslo_concurrency]
lock_path = $state_path/lock

[James Page (james-page)]

switching nova.tenant_id to nova.project_id resolved this problem, but I'm not sure that it should have done.

[Gary Kotton (garyk)]

I think that this something that we need to document

[Brian Haley (brian-haley)]

Everything is now project_id in the samples and this is so old I'm going to close it.