MidoNet Neutron Plugin upgrade from kilo stable 2015.1.0 to kilo unstable 2015.1.1.2.0-1~rc0 (MNv5.0) not supported

[Lucas Eznarriaga]

New supported features in last unstable version of the kilo plugin 2015.1.1.2.0-1~rc0 such as port_security cause backwards incompatibility with stable version of kilo plugin 2015.1.0.

E.g. neutron-server logs:

2015-10-19 11:23:23.722 29190 ERROR neutron.api.v2.resource [req-007bd588-78a5-4cdd-a893-7522c1820edc ] index failed
2015-10-19 11:23:23.722 29190 TRACE neutron.api.v2.resource Traceback (most recent call last):
2015-10-19 11:23:23.722 29190 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/resource.py", line 83, in resource
2015-10-19 11:23:23.722 29190 TRACE neutron.api.v2.resource result = method(request=request, **args)
2015-10-19 11:23:23.722 29190 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 319, in index
2015-10-19 11:23:23.722 29190 TRACE neutron.api.v2.resource return self._items(request, True, parent_id)
2015-10-19 11:23:23.722 29190 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 249, in _items
2015-10-19 11:23:23.722 29190 TRACE neutron.api.v2.resource obj_list = obj_getter(request.context, **kwargs)
2015-10-19 11:23:23.722 29190 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/db/db_base_plugin_v2.py", line 1970, in get_ports
2015-10-19 11:23:23.722 29190 TRACE neutron.api.v2.resource items = [self._make_port_dict(c, fields) for c in query]
2015-10-19 11:23:23.722 29190 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/db/db_base_plugin_v2.py", line 936, in _make_port
_dict
2015-10-19 11:23:23.722 29190 TRACE neutron.api.v2.resource attributes.PORTS, res, port)
2015-10-19 11:23:23.722 29190 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/db/common_db_mixin.py", line 162, in _apply_dict_
extend_functions
2015-10-19 11:23:23.722 29190 TRACE neutron.api.v2.resource func(*args)
2015-10-19 11:23:23.722 29190 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/db/portsecurity_db.py", line 31, in _extend_port_
security_dict
2015-10-19 11:23:23.722 29190 TRACE neutron.api.v2.resource psec_value = db_data['port_security'][psec.PORTSECURITY]
2015-10-19 11:23:23.722 29190 TRACE neutron.api.v2.resource TypeError: 'NoneType' object has no attribute '__getitem__'
2015-10-19 11:23:23.722 29190 TRACE neutron.api.v2.resource
2015-10-19 11:23:24.283 29190 ERROR oslo_messaging.rpc.dispatcher [req-21c014b0-c418-4ebe-822f-3789fc680af6 ] Exception during message handling: 'NoneType' ob
ject has no attribute '__getitem__'
2015-10-19 11:23:24.283 29190 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-10-19 11:23:24.283 29190 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _di
spatch_and_reply
2015-10-19 11:23:24.283 29190 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-10-19 11:23:24.283 29190 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _di
spatch
2015-10-19 11:23:24.283 29190 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2015-10-19 11:23:24.283 29190 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do
_dispatch
2015-10-19 11:23:24.283 29190 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-10-19 11:23:24.283 29190 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 120, in
 get_active_networks_info
2015-10-19 11:23:24.283 29190 TRACE oslo_messaging.rpc.dispatcher networks = self._get_active_networks(context, **kwargs)
2015-10-19 11:23:24.283 29190 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 63, in
_get_active_networks
2015-10-19 11:23:24.283 29190 TRACE oslo_messaging.rpc.dispatcher context, host)
2015-10-19 11:23:24.283 29190 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/neutron/db/agentschedulers_db.py", line 420, in lis
t_active_networks_on_active_dhcp_agent
2015-10-19 11:23:24.283 29190 TRACE oslo_messaging.rpc.dispatcher filters={'id': net_ids, 'admin_state_up': [True]}
2015-10-19 11:23:24.283 29190 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/midonet/neutron/plugin_v2.py", line 126, in get_net
works
2015-10-19 11:23:24.283 29190 TRACE oslo_messaging.rpc.dispatcher limit, marker, page_reverse)
2015-10-19 11:23:24.283 29190 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/neutron/db/db_base_plugin_v2.py", line 1035, in get_networks
2015-10-19 11:23:24.283 29190 TRACE oslo_messaging.rpc.dispatcher page_reverse=page_reverse)
2015-10-19 11:23:24.283 29190 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/neutron/db/common_db_mixin.py", line 184, in _get_collection
2015-10-19 11:23:24.283 29190 TRACE oslo_messaging.rpc.dispatcher items = [dict_func(c, fields) for c in query]
2015-10-19 11:23:24.283 29190 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/neutron/db/db_base_plugin_v2.py", line 873, in _make_network_dict
2015-10-19 11:23:24.283 29190 TRACE oslo_messaging.rpc.dispatcher attributes.NETWORKS, res, network)
2015-10-19 11:23:24.283 29190 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/neutron/db/common_db_mixin.py", line 162, in _apply_dict_extend_functions
2015-10-19 11:23:24.283 29190 TRACE oslo_messaging.rpc.dispatcher func(*args)
2015-10-19 11:23:24.283 29190 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/neutron/db/portsecurity_db.py", line 31, in _extend_port_security_dict
2015-10-19 11:23:24.283 29190 TRACE oslo_messaging.rpc.dispatcher psec_value = db_data['port_security'][psec.PORTSECURITY]
2015-10-19 11:23:24.283 29190 TRACE oslo_messaging.rpc.dispatcher TypeError: 'NoneType' object has no attribute '__getitem__'
2015-10-19 11:23:24.283 29190 TRACE oslo_messaging.rpc.dispatcher
2015-10-19 11:23:24.316 29190 ERROR oslo_messaging._drivers.common [req-21c014b0-c418-4ebe-822f-3789fc680af6 ] Returning exception 'NoneType' object has no attribute '__getitem__' to caller
2015-10-19 11:23:24.317 29190 ERROR oslo_messaging._drivers.common [req-21c014b0-c418-4ebe-822f-3789fc680af6 ] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n executor_callback))\n', ' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n executor_callback)\n', ' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python2.7/dist-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 120, in get_active_networks_info\n networks = self._get_active_networks(context, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 63, in _get_active_networks\n context, host)\n', ' File "/usr/lib/python2.7/dist-packages/neutron/db/agentschedulers_db.py", line 420, in list_active_networks_on_active_dhcp_agent\n filters={\'id\': net_ids, \'admin_state_up\': [True]}\n', ' File "/usr/lib/python2.7/dist-packages/midonet/neutron/plugin_v2.py", line 126, in get_networks\n limit, marker, page_reverse)\n', ' File "/usr/lib/python2.7/dist-packages/neutron/db/db_base_plugin_v2.py", line 1035, in get_networks\n page_reverse=page_reverse)\n', ' File "/usr/lib/python2.7/dist-packages/neutron/db/common_db_mixin.py", line 184, in _get_collection\n items = [dict_func(c, fields) for c in query]\n', ' File "/usr/lib/python2.7/dist-packages/neutron/db/db_base_plugin_v2.py", line 873, in _make_network_dict\n attributes.NETWORKS, res, network)\n', ' File "/usr/lib/python2.7/dist-packages/neutron/db/common_db_mixin.py", line 162, in _apply_dict_extend_functions\n func(*args)\n', ' File "/usr/lib/python2.7/dist-packages/neutron/db/portsecurity_db.py", line 31, in _extend_port_security_dict\n psec_value = db_data[\'port_security\'][psec.PORTSECURITY]\n', "TypeError: 'NoneType' object has no attribute '__getitem__'\n"]
2015-10-19 11:23:24.467 29190 ERROR oslo_messaging.rpc.dispatcher [req-21c014b0-c418-4ebe-822f-3789fc680af6 ] Exception during message handling: 'NoneType' object has no attribute '__getitem__'

[Ryu Ishimoto (ryu-midokura)]

What did you do get this error? You got this when you upgraded the package without doing any operation?

[Lucas Eznarriaga (lezbar)]

Right, we just replaced the package without any kind of db migration.

[YAMAMOTO Takashi (yamamoto)]

relevant neutron fix: https://review.openstack.org/#/c/24024/

[YAMAMOTO Takashi (yamamoto)]

Lucus,

how did you update the backend (midonet)?

[Lucas Eznarriaga (lezbar)]

We only updated the python-neutron-plugin-midonet package by changing the artifact repo entry from stable to unstable.
Then apt-get update,
apt-cache policy python-neutron-plugin-midonet
python-neutron-plugin-midonet:
  Installed: 1:2015.1.0
  Candidate: 1:2015.1.1.2.0-1~rc0

and finally apt-get install python-neutron-plugin-midonet
The rest of MidoNet packages remained on the same version: 5.0.0~rc2

[YAMAMOTO Takashi (yamamoto)]

Lucas,
do you mean http://repo.midonet.org/openstack-kilo/unstable/ ?

[YAMAMOTO Takashi (yamamoto)]

there seems to be at least a few problems:

1. lucas didn't run migration
2. ml2's migration script might not be executed depending from/to versions
3. we backported port-sec to kilo and it effectively created migration branches

[YAMAMOTO Takashi (yamamoto)]

suggested solutions:

a. runtime fixup ("treat no row as the default value")
b. custom script to clean up data

[YAMAMOTO Takashi (yamamoto)]

https://review.openstack.org/#/c/238381/

[Lucas Eznarriaga (lezbar)]

Yamamoto,

We were using the internal artifactory unstable repositories for the rc updates.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/238381
Committed: https://git.openstack.org/cgit/openstack/networking-midonet/commit/?id=832aade785c685a87a46bad9ca49044e7e0114c6
Submitter: Jenkins
Branch: master

commit 832aade785c685a87a46bad9ca49044e7e0114c6
Author: YAMAMOTO Takashi <email address hidden>
Date: Thu Oct 22 15:00:21 2015 +0900

    plugin_v2: Deal with the lack of portsecurity DB rows

    For get methods, consider the lack of the binding row as
    the API default value of network portsecurity.
    (It's similar to ML2. The reason it was necessary
    for ML2 is a bit different from midonet, though.)

    For update methods, create the binding row for the resource
    if it isn't found.

    Similar ML2 change: I73abc98d83372f6259f17680806e6541458e2077
    Closes-Bug: #1507651
    Change-Id: I97b128f2801b7bc5bf1ff028b7927bda068643d8

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/245657

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/245657
Committed: https://git.openstack.org/cgit/openstack/networking-midonet/commit/?id=0013f782d033258a8a6727a1d1d876a7059e837a
Submitter: Jenkins
Branch: stable/kilo

commit 0013f782d033258a8a6727a1d1d876a7059e837a
Author: YAMAMOTO Takashi <email address hidden>
Date: Thu Oct 22 15:00:21 2015 +0900

    plugin_v2: Deal with the lack of portsecurity DB rows

    For get methods, consider the lack of the binding row as
    the API default value of network portsecurity.
    (It's similar to ML2. The reason it was necessary
    for ML2 is a bit different from midonet, though.)

    For update methods, create the binding row for the resource
    if it isn't found.

    Similar ML2 change: I73abc98d83372f6259f17680806e6541458e2077
    Closes-Bug: #1507651
    Change-Id: I97b128f2801b7bc5bf1ff028b7927bda068643d8
    (cherry picked from commit 832aade785c685a87a46bad9ca49044e7e0114c6)
    Conflicts:
            midonet/neutron/tests/unit/test_midonet_plugin_v2.py

[YAMAMOTO Takashi (yamamoto)]

i manually tested kilo -> kilo upgrade.
from 8934f3cb1129f25fa5b0f2a5aea466360c50e1ae
to 69c7c61e1cf9934bfdaac6774e2f02a2aa5d5472

1. create a net and a port
2. upgrade only the plugin
3. show the net and the port
4. update portsec attribute of them

[YAMAMOTO Takashi (yamamoto)]

i added neutron as an affected project because i think it's better to fix this in neutron.
see https://review.openstack.org/#/c/245657/ for discussion.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/297212

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/297212
Committed: https://git.openstack.org/cgit/openstack/networking-midonet/commit/?id=18b1178d167b2714df9cea8704103597b790df24
Submitter: Jenkins
Branch: master

commit 18b1178d167b2714df9cea8704103597b790df24
Author: YAMAMOTO Takashi <email address hidden>
Date: Fri Mar 25 00:21:23 2016 +0900

    Remove port-security binding hack

    The recent Neutron change [1] made this unnecessary.

    Keep the corresponding unit tests because they are still useful
    to examine the above mentioned Neutron code.

    [1] I8607cdecdc16c5f94635c94e2f02700c732806eb

    Related-Bug: #1507651
    Related-Bug: #1509312
    Change-Id: Ic4d0d6a642ba14bdc1583cc9ec2c15e880e84599

[Slawek Kaplonski (slaweq)]

I'm marking this bug as incomplete for neutron as Kilo version is EOL for very long time.