Neutron loadbalancer VIP port fails to create

[Turbo Fredriksson]

When trying to create a Loadbalancer (v1) VIP with the command:

    neutron lb-vip-create --address 10.97.0.254 --name vip-97 \
        --protocol-port 22 --protocol TCP --subnet-id subnet-97 hapool-97

Where subnet-97 is a subnet to tenant-97, which have 'dns_domain' set to an existing domain. The domain works - creating an instance + floating IP on that will register the set dns_name in the domain.

However, the lb-vip-create will fail with

    Request Failed: internal server error while processing your request.
    Neutron server returns request_ids: ['req-ee6a68f1-ed8a-4f22-9dea-646fb97ff795']

and the log will say:

    ==> /var/log/neutron/neutron-server.log <==
    2016-07-21 18:08:54.940 7926 INFO neutron.wsgi [req-cc53af04-89fc-482c-8a4f-0a3f5cc2e614 4b0e25c70d2b4ad6ba4c50250f2f0b0b 04ee0e71babe4fd7aa16c3f64a8fca89 - - -] 10.0.4.1 - - [21/Jul/2016 18:08:54] "GET /v2.0/lb/pools.json?fields=id&name=hapool-97 HTTP/1.1" 200 257 0.070421
    2016-07-21 18:08:55.027 7926 INFO neutron.wsgi [req-e95bbb13-c38e-4cdf-afc5-9bba3351b8ff 4b0e25c70d2b4ad6ba4c50250f2f0b0b 04ee0e71babe4fd7aa16c3f64a8fca89 - - -] 10.0.4.1 - - [21/Jul/2016 18:08:55] "GET /v2.0/subnets.json?fields=id&name=subnet-97 HTTP/1.1" 200 259 0.081731
    2016-07-21 18:08:55.037 7926 INFO neutron.quota [req-ee6a68f1-ed8a-4f22-9dea-646fb97ff795 4b0e25c70d2b4ad6ba4c50250f2f0b0b 04ee0e71babe4fd7aa16c3f64a8fca89 - - -] Loaded quota_driver: <neutron.db.quota.driver.DbQuotaDriver object at 0x7f37d17c4f90>.
    2016-07-21 18:08:55.494 7926 INFO neutron.plugins.ml2.managers [req-ee6a68f1-ed8a-4f22-9dea-646fb97ff795 4b0e25c70d2b4ad6ba4c50250f2f0b0b 04ee0e71babe4fd7aa16c3f64a8fca89 - - -] Extension driver 'dns' failed in process_create_port
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource [req-ee6a68f1-ed8a-4f22-9dea-646fb97ff795 4b0e25c70d2b4ad6ba4c50250f2f0b0b 04ee0e71babe4fd7aa16c3f64a8fca89 - - -] create failed
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource Traceback (most recent call last):
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/resource.py", line 84, in resource
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource result = method(request=request, **args)
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 410, in create
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource return self._create(request, body, **kwargs)
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 148, in wrapper
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource self.force_reraise()
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource return f(*args, **kwargs)
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 521, in _create
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource obj = do_create(body)
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 503, in do_create
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource request.context, reservation.reservation_id)
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource self.force_reraise()
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 496, in do_create
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource return obj_creator(request.context, **kwargs)
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_lbaas/services/loadbalancer/plugin.py", line 147, in create_vip
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource v = super(LoadBalancerPlugin, self).create_vip(context, vip)
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_lbaas/db/loadbalancer/loadbalancer_db.py", line 404, in create_vip
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource context.session.flush()
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource self.force_reraise()
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_lbaas/db/loadbalancer/loadbalancer_db.py", line 399, in create_vip
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource context, vip_db, v['subnet_id'], v.get('address'))
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_lbaas/db/loadbalancer/loadbalancer_db.py", line 346, in _create_port_for_vip
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource port = self._core_plugin.create_port(context, {'port': port_data})
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/plugin.py", line 1096, in create_port
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource result, mech_context = self._create_port_db(context, port)
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/plugin.py", line 1068, in _create_port_db
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource self.extension_manager.process_create_port(context, attrs, result)
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/managers.py", line 858, in process_create_port
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource data, result)
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/managers.py", line 833, in _call_on_ext_drivers
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource {'name': driver.name, 'method': method_name})
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource self.force_reraise()
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/managers.py", line 828, in _call_on_ext_drivers
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource getattr(driver.obj, method_name)(plugin_context, data, result)
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/extensions/dns_integration.py", line 78, in process_create_port
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource if not request_data[dns.DNSNAME]:
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource KeyError: 'dns_name'
    2016-07-21 18:08:55.719 7926 ERROR neutron.api.v2.resource
    2016-07-21 18:08:55.728 7926 INFO neutron.wsgi [req-ee6a68f1-ed8a-4f22-9dea-646fb97ff795 4b0e25c70d2b4ad6ba4c50250f2f0b0b 04ee0e71babe4fd7aa16c3f64a8fca89 - - -] 10.0.4.1 - - [21/Jul/2016 18:08:55] "POST /v2.0/lb/vips.json HTTP/1.1" 500 368 0.695096

I'm somewhat unsure if this a problem with Designate or Neutron (the log DO say Neutron), but on the other hand, this worked just fine before I added Designate to the mix.. Feel free to reassign to Neutron.

This is Neutron v4.2.0 and Designate v2.1.0 on Debian GNU/Linux Sid/Unstable - Mitaka.

[Turbo Fredriksson (turbo-bayour)]

This also affects (?) Trove in a

    trove create testdb m2.2small --size=2 --volume_type lvm \
        --availability_zone nova --nic "net-id=${netid}" \
        --databases=test1 --users test1:TEST1 \
        --datastore mysql --datastore_version 5.6

Same error message..

[Kiall Mac Innes (kiall)]

The stack trace indicates this is a Neutron issue, likely cause by enabling Designate/DNS integration, but still within Neutron / before Designate is called. I've added an affects Neutron.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/346221

[Miguel Lavalle (minsel)]

When LBaaS _create_port_for_vip creates the the request body for the port here, it doesn't include the 'dns_name' attribute: https://github.com/openstack/neutron-lbaas/blob/master/neutron_lbaas/db/loadbalancer/loadbalancer_db.py#L334

If we got the port creation request from the Neutron ReST API and the DNS integration extension was enabled, the API would assign a default value to the 'dns_name' attribute if the user didn't specify it. But in this case, LBaaS is by-passing the ReST API, so when the DNS extension processes the request, we get a KeyError exception when looking for 'dns_name' in the request.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/346961

[OpenStack Infra (hudson-openstack)]

Change abandoned by Graham Hayes (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/346961

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/346221
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=625fdb423e289ee98dbfbd4c81edf64b598cb352
Submitter: Jenkins
Branch: master

commit 625fdb423e289ee98dbfbd4c81edf64b598cb352
Author: Graham Hayes <email address hidden>
Date: Fri Jul 22 20:55:44 2016 +0100

    Avoid KeyError when accessing "dns_name" as it may not exist

    Neutron LBaaS does not pass a full copy of the request_data
    into this function, and causes the port create to fail
    with a KeyError

    Change-Id: Ib81cbbaf24a4ffaa983e1b05146aea0dc74e29bb
    Fixes-Bug: #1605336

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/346961
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b0778fe801331284b5b43d1e7c79ac6a9b0c1cce
Submitter: Jenkins
Branch: stable/mitaka

commit b0778fe801331284b5b43d1e7c79ac6a9b0c1cce
Author: Graham Hayes <email address hidden>
Date: Fri Jul 22 20:55:44 2016 +0100

    Avoid KeyError when accessing "dns_name" as it may not exist

    Neutron LBaaS does not pass a full copy of the request_data
    into this function, and causes the port create to fail
    with a KeyError

    Change-Id: Ib81cbbaf24a4ffaa983e1b05146aea0dc74e29bb
    Fixes-Bug: #1605336
    (cherry picked from commit 625fdb423e289ee98dbfbd4c81edf64b598cb352)