unable to use tenant network after kilo to liberty update due to port security extension

[Mihai Gheorghe]

After updating to liberty from kilo all networks created in kilo release are useless in liberty.

If i try to spawn a new isntance with a port on a network created in kilo i get the following error in nova-compute.log :

BadRequest: Port does not have port security binding.

I guess this has to do with the new extension in ml2 plugin port_security.

Using neutron DVR on Ubuntu 14.04.3!

This is my first bug report so sry in advance for any mistakes.

[Matthew Thode (prometheanfire)]

I was using LB/vxlan with both tenant and flat networks.
I enabled port_security after the upgrade.
net-show showed flat network had port_security enabled.
flat network didn't get the port_security flag in the 'networksecuritybindings' table.
entered the entry into the 'networksecuritybindings' table manually as follows (postgres syntax).

    insert into networksecuritybindings values ('NETWORK_UUID_HERE', 't');

[Matthew Thode (prometheanfire)]

to be clear, I enabled port security after the db upgrade/sync and do not use DVR.

[Mihai Gheorghe (m-capsali)]

I figured that out too! It works with DVR too! But this could have been done auto on db upgrade/sync!

[Armando Migliaccio (armando-migliaccio)]

port security was added in kilo so the db upgrade should not be required to triage the issue. Can you elaborate on what you figured out? Is it possible that you didn't stamp your DB correctly?

[Mihai Gheorghe (m-capsali)]

I manually added entries for every network in networksecuritybindings in neutron db. When issuing neutron net-show on any network it showed port-security enabled even though it was missing in the db.

By manually adding them in networksecuritybindings table i could attach ports to instances.

This is only for networks created on kilo release. Any network created after upgrade to liberty will work just fine and have entries in networksecuritybindings table!

[Armando Migliaccio (armando-migliaccio)]

The DB migration [1] would have done that for you. Something went bad during the upgrade.

[1] neutron/db/migration/alembic_migrations/versions/35a0f3365720_add_port_security_in_ml2.py

[Armando Migliaccio (armando-migliaccio)]

I am no longer sure the title of this bug report is accurate. You might want to try to reproduce why the DB migration didn't succeed, otherwise this is clearly an invalid/incomplete issue

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]

[Matthew Thode (prometheanfire)]

I suggest we un-expire this

The migration occurred before the kilo branch was tagged, you can see that in the history below. Also, during kilo that table was not populated. Any ports/networks created during the time kilo was deployed needs to be re-migrated when the liberty upgrade is done as those tables were not populated, once liberty is deployed those tables are populated properly.

1. kilo had the migration already run
2. kilo did not populate the tables when new ports/networks were created
3. liberty does populate the tables when new ports/networks are created
4. liberty needs to re-migrate those ports/networks to the tables

9859ac9c136 -> 34af2b5c5a59 (expand) (head), Add dns_name to Port
48153cb5f051 -> 9859ac9c136 (expand), quota_reservations
1b4c6e320f79 -> 48153cb5f051 (expand), qos db changes
1c844d1677f7 -> 1b4c6e320f79 (expand), address scope support in subnetpool
26c371498592 -> 1c844d1677f7 (expand), add order to dnsnameservers
45f955889773 -> 26c371498592 (expand), subnetpool hash
8675309a5c4f -> 45f955889773 (expand), quota_usage
11926bcfe72d -> 4af11ca47297 (contract) (head), Drop cisco monolithic tables
2e5352a0ad4d -> 11926bcfe72d (contract), add geneve ml2 type driver
2a16083502f3 -> 2e5352a0ad4d (contract), Add missing foreign keys
5498d17be016 -> 2a16083502f3 (contract), Metaplugin removal
4ffceebfada -> 5498d17be016 (contract), Drop legacy OVS and LB plugin tables
30018084ec99 (8675309a5c4f) -> 4ffceebfada (contract), network_rbac
kilo -> 30018084ec99 (contract), Initial no-op Liberty contract rule.
313373c0ffee -> 8675309a5c4f (expand), network_rbac
52c5312f6baf -> 313373c0ffee (expand), Flavor framework
599c6a226151 -> 52c5312f6baf (expand), Initial operations in support of address scopes
354db87e3225 -> 599c6a226151 (expand), neutrodb_ipam
kilo -> 354db87e3225 (expand), nsxv_vdr_metadata.py
20c469a5f920 -> kilo (branchpoint), kilo <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
28a09af858a8 -> 20c469a5f920, add index for port
268fb5e99aa2 -> 28a09af858a8, Initial operations to support basic quotas on prefix space in a subnet pool
034883111f -> 268fb5e99aa2, Initial operations in support of subnet allocation from a pool
20b99fd19d4f -> 034883111f, Remove allow_overlap from subnetpools
589f9237ca0e -> 20b99fd19d4f, Cisco UCS Manager Mechanism Driver
51c54792158e -> 589f9237ca0e, Cisco N1kv ML2 driver tables
1955efc66455 -> 51c54792158e, Initial operations for subnetpools
35a0f3365720 -> 1955efc66455, weight_scheduler
341ee8a4ccb5 -> 35a0f3365720, add port-security in ml2 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
f15b1fb526dd -> 341ee8a4ccb5, sync with cisco repo
57dd745253a6 -> f15b1fb526dd, Cascade Floating IP Floating Port deletion
2b801560a332 -> 57dd745253a6, nuage_kilo_migrate
2d2a8a565438 -> 2b801560a332, Remove Hyper-V Neutron Plugin
4119216b7365 -> 2d2a8a565438, ML2 hierarchical binding
bebba223288 -> 4119216b7365, Add index on tenant_id column
43763a9618fd -> bebba223288, Add vlan transparent property to network
16cdf118d31d -> 43763a9618fd, add mtu attributes to network
14be42f3d0a5 -> 16cdf118d31d, extra_dhcp_options IPv6 support
26b54cf9024d ->...

[David Wilde (dave-wilde)]

+1 to un-expiring this

[Nolan Brubaker (nolan-brubaker)]

This is affecting openstack-ansible now; I'm not sure if that's because we're doing the upgrade out of order, but it appears that the migration script mentioned runs on Kilo install, then doesn't get run again for Liberty upgrades.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/294132

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: liberty
Review: https://review.openstack.org/294152

[Nolan Brubaker (nolan-brubaker)]

Copying Ihar's email here: "Yeah. To trigger the bug, you don’t need to upgrade. Just create a network/port without the extension enabled; then enable the extension; then try to start an instance using the network/port."

This is the exact behavior we're seeing in our upgrade - openstack-ansible Kilo did not configure the extension, but Liberty did. So it seems this isn't necessarily a version/branch issue, but a matter of enabling/disabling the extension.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Nolan Brubaker (<email address hidden>) on branch: liberty
Review: https://review.openstack.org/294152
Reason: Abandoning this bug as the work around is more of a hack than I think is worth maintaining. Instead, we should disable the port_security extension.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/294231

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: liberty
Review: https://review.openstack.org/294237

[OpenStack Infra (hudson-openstack)]

Change abandoned by Nolan Brubaker (<email address hidden>) on branch: liberty
Review: https://review.openstack.org/294237
Reason: I had forgotten this is superceded by the config_template syntax. This change is unnecessary.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Nolan Brubaker (<email address hidden>) on branch: master
Review: https://review.openstack.org/294231

[Armando Migliaccio (armando-migliaccio)]

This is not an RC critical issue.

[Armando Migliaccio (armando-migliaccio)]

Btw, no-one ever responded to my comment #7. The DB migration was meant to create the bindings irrespective of the extension being available. So I suspect this is a DB migration still.

https://github.com/openstack/neutron/blob/stable/kilo/neutron/db/migration/alembic_migrations/versions/35a0f3365720_add_port_security_in_ml2.py

[Jesse Pretorius (jesse-pretorius)]

For OpenStack-Ansible, we've implemented a workaround for the upgrade process to ensure that if port_security_bindings are not enabled in Kilo, they will remain disabled in the upgrade process. We'll keep this in place and shift to an alternative once one is available in Neutron itself. ref: https://review.openstack.org/293911

[Ihar Hrachyshka (ihar-hrachyshka)]

Replying to comment #22:

This is because the extension was enabled after some resources were created in Kilo or later without the extension enabled.

[Ihar Hrachyshka (ihar-hrachyshka)]

Ideally, we would not have the migration script adding binding models at all and handle everything dynamically in db code.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/294132
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b0519cf0ada3b3d9b76f84948f9ad3c142fc50be
Submitter: Jenkins
Branch: master

commit b0519cf0ada3b3d9b76f84948f9ad3c142fc50be
Author: Ihar Hrachyshka <email address hidden>
Date: Thu Mar 17 16:20:52 2016 +0100

    port security: gracefully handle resources with no bindings

    Resources could be created before the extension was enabled in the
    setup. In that case, no bindings are created for them. In that case, we
    should gracefully return default (True) value when extracting the value
    using the mixin; and we should also create binding model on update
    request, if there is no existing binding model for the resource.

    While at it, introduced a constant to store the default value for port
    security (True) and changed several tests to use the constant instead of
    extracting it from extension resource map.

    Change-Id: I8607cdecdc16c5f94635c94e2f02700c732806eb
    Closes-Bug: #1509312

[Matthew Thode (prometheanfire)]

this gonna be backported to liberty?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/298993

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/299204

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/liberty
Review: https://review.openstack.org/299212

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/299217

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/kilo
Review: https://review.openstack.org/299218

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: stable/liberty
Review: https://review.openstack.org/299212

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/299218

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/298993
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=90b9cd334b1b33df933bf1b61b38c6e087c431af
Submitter: Jenkins
Branch: stable/mitaka

commit 90b9cd334b1b33df933bf1b61b38c6e087c431af
Author: Ihar Hrachyshka <email address hidden>
Date: Thu Mar 17 16:20:52 2016 +0100

    port security: gracefully handle resources with no bindings

    Resources could be created before the extension was enabled in the
    setup. In that case, no bindings are created for them. In that case, we
    should gracefully return default (True) value when extracting the value
    using the mixin; and we should also create binding model on update
    request, if there is no existing binding model for the resource.

    While at it, introduced a constant to store the default value for port
    security (True) and changed several tests to use the constant instead of
    extracting it from extension resource map.

    Change-Id: I8607cdecdc16c5f94635c94e2f02700c732806eb
    Closes-Bug: #1509312
    (cherry picked from commit b0519cf0ada3b3d9b76f84948f9ad3c142fc50be)

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dave Walker (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/299217
Reason: Kilo is now approaching EOL. We are in freeze pending the final release and no freeze exception has been raised. Therefore I am abandoning this change, if it is required in the release - please restore and raise a request. Thanks

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/neutron 8.1.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/314250

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/299204
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d1618c91075e32b1d3f58869b02065f4e0709c49
Submitter: Jenkins
Branch: stable/liberty

commit d1618c91075e32b1d3f58869b02065f4e0709c49
Author: Ihar Hrachyshka <email address hidden>
Date: Thu Mar 17 16:20:52 2016 +0100

    port security: gracefully handle resources with no bindings

    Resources could be created before the extension was enabled in the
    setup. In that case, no bindings are created for them. In that case, we
    should gracefully return default (True) value when extracting the value
    using the mixin; and we should also create binding model on update
    request, if there is no existing binding model for the resource.

    While at it, introduced a constant to store the default value for port
    security (True) and changed several tests to use the constant instead of
    extracting it from extension resource map.

    Change-Id: I8607cdecdc16c5f94635c94e2f02700c732806eb
    Closes-Bug: #1509312
    (cherry picked from commit b0519cf0ada3b3d9b76f84948f9ad3c142fc50be)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/314250
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3bf73801df169de40d365e6240e045266392ca63
Submitter: Jenkins
Branch: master

commit a323769143001d67fd1b3b4ba294e59accd09e0e
Author: Ryan Moats <email address hidden>
Date: Tue Oct 20 15:51:37 2015 +0000

    Revert "Improve performance of ensure_namespace"

    This reverts commit 81823e86328e62850a89aef9f0b609bfc0a6dacd.

    Unneeded optimization: this commit only improves execution
    time on the order of milliseconds, which is less than 1% of
    the total router update execution time at the network node.

    This also

    Closes-bug: #1574881

    Change-Id: Icbcdf4725ba7d2e743bb6761c9799ae436bd953b

commit 7fcf0253246832300f13b0aa4cea397215700572
Author: OpenStack Proposal Bot <email address hidden>
Date: Thu Apr 21 07:05:16 2016 +0000

    Imported Translations from Zanata

    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure

    Change-Id: I9e930750dde85a9beb0b6f85eeea8a0962d3e020

commit 643b4431606421b09d05eb0ccde130adbf88df64
Author: OpenStack Proposal Bot <email address hidden>
Date: Tue Apr 19 06:52:48 2016 +0000

    Imported Translations from Zanata

    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure

    Change-Id: I52d7460b3265b5460b9089e1cc58624640dc7230

commit 1ffea42ccdc14b7a6162c1895bd8f2aae48d5dae
Author: OpenStack Proposal Bot <email address hidden>
Date: Mon Apr 18 15:03:30 2016 +0000

    Updated from global requirements

    Change-Id: Icb27945b3f222af1d9ab2b62bf2169d82b6ae26c

commit b970ed5bdac60c0fa227f2fddaa9b842ba4f51a7
Author: Kevin Benton <email address hidden>
Date: Fri Apr 8 17:52:14 2016 -0700

    Clear DVR MAC on last agent deletion from host

    Once all agents are deleted from a host, the DVR MAC generated
    for that host should be deleted as well to prevent a buildup of
    pointless flows generated in the OVS agent for hosts that don't
    exist.

    Closes-Bug: #1568206
    Change-Id: I51e736aa0431980a595ecf810f148ca62d990d20
    (cherry picked from commit 92527c2de2afaf4862fddc101143e4d02858924d)

commit eee9e58ed258a48c69effef121f55fdaa5b68bd6
Author: Mike Bayer <email address hidden>
Date: Tue Feb 9 13:10:57 2016 -0500

    Add an option for WSGI pool size

    Neutron currently hardcodes the number of
    greenlets used to process requests in a process to 1000.
    As detailed in
    http://lists.openstack.org/pipermail/openstack-dev/2015-December/082717.html

    this can cause requests to wait within one process
    for available database connection while other processes
    remain available.

    By adding a wsgi_default_pool_size option functionally
    identical to that of Nova, we can lower the number of
    greenlets per process to be more in line with a typical
    max database connection pool size.

    DocImpact: a previously unused configuration value
               wsgi_default_pool_size is now used to a...

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/neutron 7.1.0 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/neutron 9.0.0.0b1 development milestone.