Mechanism to register and run all external neutron alembic migrations automatically

Bug #1470625 reported by Henry Gessau
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
devstack
Fix Released
Undecided
Henry Gessau
networking-cisco
Fix Committed
High
Henry Gessau
networking-l2gw
Fix Released
Medium
Henry Gessau
neutron
Fix Released
High
Henry Gessau

Bug Description

For alembic migration branches that are out-of-tree, we need a mechanism whereby the external code can register its branches when it is installed, and then neutron will provide automation of running all installed external migration branches when neutron-db-manage is used for upgrading.

Henry Gessau (gessau)
Changed in python-neutronclient:
status: New → Triaged
importance: Undecided → High
assignee: nobody → Henry Gessau (gessau)
no longer affects: python-neutronclient
Changed in neutron:
status: New → Triaged
importance: Undecided → High
assignee: nobody → Henry Gessau (gessau)
tags: added: db
Changed in neutron:
status: Triaged → In Progress
Revision history for this message
Henry Gessau (gessau) wrote :

This is a mini-RFE since we are adding a new mechanism using entry-points to register external alembic branches at install time.

tags: added: rfe
Changed in networking-cisco:
assignee: nobody → Henry Gessau (gessau)
Changed in networking-l2gw:
assignee: nobody → Henry Gessau (gessau)
Revision history for this message
Henry Gessau (gessau) wrote :

I need to make a decision about how the neutron-db-manage script will behave when sub-projects are installed that register their migration scripts.

Previously, neutron-db-manage (henceforth abbreviated to 'ndbm') would run the command against only one project:

ndbm upgrade head # upgraded core neutron branch only
ndbm --service lbaas upgrade head # upgraded neutron-lbaas branch only

Now we need to support installed services and sub-projects that register their migration script locations, so ndbm can discover them and run all branches by default. Let's say lbaas and networking-l2gw are installed:

ndbm upgrade head # upgrade neutron, neutron-lbaas and networking-l2gw branches
ndbm --service lbaas upgrade head # upgrade neutron-lbaas branch only
ndbm --subproject networking-l2gw upgrade head # upgrade networking-l2gw branch only
ndbm --neutron_only upgrade head # upgrade core neutron branch only

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to networking-l2gw (master)

Fix proposed to branch: master
Review: https://review.openstack.org/205866

Changed in networking-l2gw:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron-lbaas (master)

Reviewed: https://review.openstack.org/198522
Committed: https://git.openstack.org/cgit/openstack/neutron-lbaas/commit/?id=1a779cf8afc4a548c07bbc8e507baf7afa3c69cc
Submitter: Jenkins
Branch: master

commit 1a779cf8afc4a548c07bbc8e507baf7afa3c69cc
Author: Henry Gessau <email address hidden>
Date: Sun Jul 5 04:43:24 2015 -0400

    Register alembic_migrations at install time

    This allows neutron-db-manage to find the alembic migrations
    automatically if this project is installed.

    The config file alembic.ini is not needed in this project since it's
    tables are in neutron's DB and so it depends on neutron's config.

    Partial-Bug: #1470625

    Change-Id: If4cff446ec7922224461cb76770ff9681bc27caa

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron-vpnaas (master)

Reviewed: https://review.openstack.org/198523
Committed: https://git.openstack.org/cgit/openstack/neutron-vpnaas/commit/?id=3a18ab4a4e0e54f858b9eebd3d3362e7ed042b82
Submitter: Jenkins
Branch: master

commit 3a18ab4a4e0e54f858b9eebd3d3362e7ed042b82
Author: Henry Gessau <email address hidden>
Date: Sun Jul 5 04:20:38 2015 -0400

    Register alembic_migrations at install time

    This allows neutron-db-manage to find the alembic migrations
    automatically if this project is installed.

    The config file alembic.ini is not needed in this project since it's
    tables are in neutron's DB and so it depends on neutron's config.

    Partial-Bug: #1470625

    Change-Id: I90a4a10d10769e48571327b98cbc67b68260e909

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron-fwaas (master)

Reviewed: https://review.openstack.org/198521
Committed: https://git.openstack.org/cgit/openstack/neutron-fwaas/commit/?id=85325e888b902541b6234b0f94a899b1c2d40c8c
Submitter: Jenkins
Branch: master

commit 85325e888b902541b6234b0f94a899b1c2d40c8c
Author: Henry Gessau <email address hidden>
Date: Sun Jul 5 04:36:38 2015 -0400

    Register alembic_migrations at install time

    This allows neutron-db-manage to find the alembic migrations
    automatically if this project is installed.

    The config file alembic.ini is not needed in this project since it's
    tables are in neutron's DB and so it depends on neutron's config.

    Partial-Bug: #1470625

    Change-Id: I6bf9fe272dedc019f032d7be9476a1c58977cbe0

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/198524
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=603c0d03aed5fdeefb471086c0aef879938f9641
Submitter: Jenkins
Branch: master

commit 603c0d03aed5fdeefb471086c0aef879938f9641
Author: Henry Gessau <email address hidden>
Date: Sun Jul 5 03:29:38 2015 -0400

    Support for independent alembic branches in sub-projects

    Sub-projects shall now register their independent alembic migrations
    via entrypoints in setup.cfg, and neutron-db-manage will discover them
    and run them automatically.

    If a service or sub-project is specified explicitly, then
    neutron-db-manage will run on only that service or sub-project.

    The advanced services project are just special cases of sub-projects.
    For example, specifying the CLI option '--service lbaas' is the same
    as specifying '--subproject neutron-lbaas'.

    Specifying no service or sub-project will cause neutron-db-manage to
    run the command on neutron and all installed sub-projects.

    Added and consolidated documentation into devref for alembic migrations.

    Partial-Bug: #1471333
    Partial-Bug: #1470625

    Change-Id: I9a06de64ce35675af28adf819de6f22dc832390d

Henry Gessau (gessau)
summary: - Mechanism to register and run all external alembic migrations
+ Mechanism to register and run all external neutron alembic migrations
automatically
Changed in devstack:
assignee: nobody → Henry Gessau (gessau)
Changed in networking-cisco:
importance: Undecided → High
Changed in neutron:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to networking-l2gw (master)

Reviewed: https://review.openstack.org/205866
Committed: https://git.openstack.org/cgit/openstack/networking-l2gw/commit/?id=cc0b186a9c211cf0a5007ac6d9fee2896c56f0ab
Submitter: Jenkins
Branch: master

commit cc0b186a9c211cf0a5007ac6d9fee2896c56f0ab
Author: Henry Gessau <email address hidden>
Date: Fri Jul 24 17:19:15 2015 -0400

    Register alembic_migrations at install time

    This allows neutron-db-manage to find the alembic migrations
    automatically if this project is installed.

    The config file alembic.ini is not needed in this project since it's
    tables are in neutron's DB and so it depends on neutron's config.

    Remove the l2gw-db-manage script since we will use
    neutron-db-manage.

    Update the alembic env to know about all neutron core tables when
    generating diffs for --autogenerate.

    Partial-Bug: #1470625

    Depends-On: I9a06de64ce35675af28adf819de6f22dc832390d

    Change-Id: I581932d333e378d91d5b581760018ec274330681

Henry Gessau (gessau)
Changed in devstack:
status: New → In Progress
Changed in networking-l2gw:
status: In Progress → Fix Committed
Changed in networking-cisco:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to networking-cisco (master)

Fix proposed to branch: master
Review: https://review.openstack.org/214403

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (feature/pecan)

Fix proposed to branch: feature/pecan
Review: https://review.openstack.org/218710

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (feature/pecan)
Download full text (155.6 KiB)

Reviewed: https://review.openstack.org/218710
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2c5f44e1b3bd4ed8a0b7232fd293b576cc8c1c87
Submitter: Jenkins
Branch: feature/pecan

commit f35d1c5c50dccbef1a2e079f967b82f0df0e22e9
Author: Adelina Tuvenie <email address hidden>
Date: Thu Aug 27 02:27:28 2015 -0700

    Fixes wrong neutron Hyper-V Agent name in constants

    Change Id03fb147e11541be309c1cd22ce27e70fadc28b5 moved the
    AGENT_TYPE_HYPERV constant from common.constants to
    plugins.ml2.drivers.hyperv.constants but change the value of the
    constant from 'HyperV agent' to 'hyperv'. This patch changes
    the name back to 'HyperV agent'

    Change-Id: If74b4b2a84811e266c8b12e70bf6bfe74ed4ea21
    Partial-Bug: #1487598

commit de604de334854e2eb6b4312ff57920564cbd4459
Author: OpenStack Proposal Bot <email address hidden>
Date: Sun Aug 30 01:39:06 2015 +0000

    Updated from global requirements

    Change-Id: Ie52aa3b59784722806726e4046bd07f4a4d97328

commit f0415ac20eaf5ab4abb9bd4839bf6d04ceee85d0
Author: armando-migliaccio <email address hidden>
Date: Fri Aug 28 13:53:04 2015 -0700

    Revert "Add support for unaddressed port"

    This implementation may expose a vulnerability where a malicious
    user can sieze the opportunity of a time window where a port
    may land unaddressed on a shared network, thus allowing him/her
    to suck up all the tenant traffic he/she wants....oh the shivers.

    This reverts commit d4c52b7f5a36a103a92bf9dcda7f371959112292.

    Change-Id: I7ebdaa8d3defa80eab90e460fde541a5bdd8864c

commit 013fdcd2a6d45dbe4de5d6e7077e5e9b60985ef9
Author: Assaf Muller <email address hidden>
Date: Fri Aug 28 16:41:07 2015 -0400

    Improve logging upon failure in iptables functional tests

    This will help us nail down a more accurate and efficient logstash
    query.

    Change-Id: Iee4238e358f7b056e373c7be8d6aa3202117a680
    Related-Bug: #1478847

commit 622dea818d851224a43d5276a81d5ce8a6eebb76
Author: Ivar Lazzaro <email address hidden>
Date: Mon Aug 17 17:17:42 2015 -0700

    handle gw_info outside of the db transaction on router creation

    Move the gateway interface creation outside the DB transaction
    to avoid lock timeout.

    Change-Id: I5a78d7f32e8ca912016978105221d5f34618af19
    Closes-bug: 1485809

commit 5b27d290a0a95f6247fc5a0fe6da1e7d905e6b2d
Author: Assaf Muller <email address hidden>
Date: Wed Aug 26 10:07:03 2015 -0400

    Remove ml2 resource extension success logging

    This is the cause of a tremendous amount of logs, for no
    perceivable gain. A normal dvr run in the gate shows this debug
    message around 120K times, which is way too much.

    Closes-Bug: #1489952

    Change-Id: I26fca8515d866a7cc1638d07fa33bc04479ae221

commit 8d3faf549cba2f58c872ef4121b2481e73464010
Author: huangpengtao <email address hidden>
Date: Fri Aug 28 23:20:46 2015 +0800

    Replace "prt" variable by "port"

    the local variable prt is meaningless,
    and port is used popular.

    Change-Id: I20849102cf5b4d84433c46791b4b1e2a22dc4739

commit ee374e7a5f4dea538fcd942f5...

tags: added: in-feature-pecan
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to devstack (master)

Reviewed: https://review.openstack.org/198542
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=091d1ff39d47bf9bebf564b11bbbe5edc984340b
Submitter: Jenkins
Branch: master

commit 091d1ff39d47bf9bebf564b11bbbe5edc984340b
Author: Henry Gessau <email address hidden>
Date: Sun Jul 5 08:55:18 2015 -0400

    Neutron auto-discovers installed alembic_migrations

    Neutron and its sub-projects have been made more intelligent about the
    alembic migrations of installed sub-projects. Neutron will now
    discover the installed migrations and run them automatically.

    Partial-Bug: #1470625

    Change-Id: Iec8993b02400ae306abf520e6e70d86bba042c8d

Henry Gessau (gessau)
Changed in devstack:
status: In Progress → Fix Committed
Changed in neutron:
milestone: none → liberty-3
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/219456

Changed in networking-l2gw:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/219456
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8ba57a2bf1ce3693db47de4ff8dd5a7a9b5347d7
Submitter: Jenkins
Branch: master

commit 8ba57a2bf1ce3693db47de4ff8dd5a7a9b5347d7
Author: Henry Gessau <email address hidden>
Date: Tue Sep 1 17:17:01 2015 -0400

    Deprecate --service option for neutron-db-manage

    Now that https://review.openstack.org/198542 has merged we can
    deprecate the --service option. From now on instead of
      --service fwaas
    we should instead use
      --subproject neutron-fwaas

    This puts the *aas subprojects on equal footing with the other
    projects in the Neutron Stadium for neutron-db-manage.

    In the Liberty release the --service option will be marked as
    deprecated. It will be removed in Mitaka.

    Related-Bug: #1470625

    Change-Id: Iecc678efafd798b62bb83e6e85333c64760f16b5

Thierry Carrez (ttx)
Changed in neutron:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (feature/pecan)

Related fix proposed to branch: feature/pecan
Review: https://review.openstack.org/224334

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: feature/pecan
Review: https://review.openstack.org/224357

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (feature/pecan)
Download full text (73.6 KiB)

Reviewed: https://review.openstack.org/224357
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fdc3431ccd219accf6a795079d9b67b8656eed8e
Submitter: Jenkins
Branch: feature/pecan

commit fe236bdaadb949661a0bfb9b62ddbe432b4cf5f1
Author: Miguel Angel Ajo <email address hidden>
Date: Thu Sep 3 15:40:12 2015 +0200

    No network devices on network attached qos policies

    Network devices, like internal router legs, or dhcp ports
    should not be affected by bandwidth limiting rules.

    This patch disables application of network attached policies
    to network/neutron owned ports.

    Closes-bug: #1486039
    DocImpact

    Change-Id: I75d80227f1e6c4b3f5fa7762b8dc3b0c0f1abd46

commit db4a06f7caa20a4c7879b58b20e95b223ed8eeaf
Author: Ken'ichi Ohmichi <email address hidden>
Date: Wed Sep 16 10:04:32 2015 +0000

    Use tempest-lib's token_client

    Now tempest-lib provides token_client modules as library and the
    interface is stable. So neutron repogitory doesn't need to contain
    these modules.
    This patch makes neutron use tempest-lib's token_client and removes
    the own modules for the maintenance.

    Change-Id: Ieff7eb003f6e8257d83368dbc80e332aa66a156c

commit 78aed58edbe6eb8a71339c7add491fe9de9a0546
Author: Jakub Libosvar <email address hidden>
Date: Thu Aug 13 09:08:20 2015 +0000

    Fix establishing UDP connection

    Previously, in establish_connection() for UDP protocol data were sent
    but never read on peer socket. That lead to successful read on peer side
    if this connection was filtered. Having constant testing string masked
    this issue as we can't distinguish to which test of connectivity data
    belong.

    This patch makes unique data string per test_connectivity() and
    also makes establish_connection() to create an ASSURED entry in
    conntrack table. Finally, in last test after firewall filter was
    removed, connection is re-established in order to avoid troubles with
    terminated processes or TCP continuing sending packets which weren't
    successfully delivered.

    Closes-Bug: 1478847
    Change-Id: I2920d587d8df8d96dc1c752c28f48ba495f3cf0f

commit e6292fcdd6262434a7b713ad8802db6bc8a6d3dc
Author: YAMAMOTO Takashi <email address hidden>
Date: Wed Sep 16 13:20:51 2015 +0900

    ovsdb: Fix a few docstring

    Change-Id: I53e1e21655b28fe5da60e58aeeb7cbbd103ae014

commit c22949a4449d96a67caa616290cf76b67b182917
Author: fumihiko kakuma <email address hidden>
Date: Wed Sep 16 11:52:59 2015 +0900

    Remove requirements.txt for the ofagent mechanism driver

    It is no longer used.

    Related-Blueprint: core-vendor-decomposition
    https://blueprints.launchpad.net/neutron/+spec/core-vendor-decomposition

    Change-Id: Ib31fb3febf8968e50d86dd66e1e6e1ea2313f8ac

commit d1d4de19d85f961d388c91e70f31b3bafec418c5
Author: Kevin Benton <email address hidden>
Date: Thu Sep 3 20:25:57 2015 -0700

    Always return iterables in L3 get_candidates

    The caller of this function expects iterables.

    Closes-Bug: #1494996
    Change-Id: I3d103e63f4e127a77268502415c0ddb0d804b54a

commit 1ad6ac448067306...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (feature/pecan)

Change abandoned by Doug Wiegley (<email address hidden>) on branch: feature/pecan
Review: https://review.openstack.org/224334

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to networking-cisco (master)

Reviewed: https://review.openstack.org/214403
Committed: https://git.openstack.org/cgit/openstack/networking-cisco/commit/?id=87b4b41ece051f4b390d738abd721011788d9f48
Submitter: Jenkins
Branch: master

commit 87b4b41ece051f4b390d738abd721011788d9f48
Author: Henry Gessau <email address hidden>
Date: Mon Aug 17 21:48:03 2015 -0400

    Register alembic_migrations at install time

    This allows neutron-db-manage to find the alembic migrations
    automatically if this project is installed.

    Partial-Bug: #1470625

    Change-Id: I56380dac41931861ffdbcce2b071d5d9bb9f8058

Thierry Carrez (ttx)
Changed in neutron:
milestone: liberty-3 → 7.0.0
Henry Gessau (gessau)
Changed in networking-cisco:
status: In Progress → Fix Committed
milestone: none → 2.0.0
Sean Dague (sdague)
Changed in devstack:
status: Fix Committed → Fix Released
Changed in networking-l2gw:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers