Adopt oslo.versionedobjects for core resources (ports, networks, subnets, ...)

[Ihar Hrachyshka]

Starting Mitaka, we started adoption of oslo.versionedobjects library for managing database access to models for core resources (ports, networks, subnets, ...) This bug will serve as a catch all bug for patches related to the effort.

[Armando Migliaccio (armando-migliaccio)]

Are we sure this is not a duplicate? I recall seeing something along these lines already. I recall suggesting a devref to outline plan of action etc.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/283711

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/287756

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/275790
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4b227c3771eba1cbaa27c6c33829108981cd9b69
Submitter: Jenkins
Branch: master

commit 4b227c3771eba1cbaa27c6c33829108981cd9b69
Author: Artur Korzeniewski <email address hidden>
Date: Tue Mar 1 12:07:15 2016 +0100

    Objects DB api: added composite key to handle multiple primary key

    Moving CRUD DB operation for objects from db/api.py to objects/db/api.py
    Renaming object get_by_id(id) to get_object(**kwargs)

    Many models in Neutron DB have complex primary keys, concatenated from
    a few properties. This patch adds ability to define multiple primary keys
    in NeutronDbObject, which are automatically evaluated into DB query
    when performing operations.

    Partial-Bug: #1541928
    Change-Id: I0f63a62418db76415ddd40c30c778ff7541b93dc

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/281850
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=412012de59191d1340e3a03ab87af0ae704e4675
Submitter: Jenkins
Branch: master

commit 412012de59191d1340e3a03ab87af0ae704e4675
Author: Artur Korzeniewski <email address hidden>
Date: Thu Mar 10 14:05:12 2016 +0100

    Create a hook in base object to modify the fields before DB operations

    Added modify_fields_to_db(), modify_fields_from_db() methods and
    fields_need_translation dict to define the map what name in object
    implementation should be changed to database naming.
    It will prepare dicts before writing to DB and also modify the DB output
    to match the OVO layout.

    Partial-Bug: #1541928
    Change-Id: I923b58870584c5e5756b307760f9d502c53c18b1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/292829

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/277558
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8785edc0e7c51844966487dce73fb80902b8e634
Submitter: Jenkins
Branch: master

commit 8785edc0e7c51844966487dce73fb80902b8e634
Author: Saisrikiran Mudigonda <email address hidden>
Date: Mon Feb 8 12:53:26 2016 -0800

    Add custom SQLAlchemy type for IP addresses

    Co-Authored-By: Ihar Hrachyshka <email address hidden>
    Co-Authored-By: Martin Hickey <email address hidden>
    Change-Id: I4709548b385cf19156bc0589058a31cb301a6b5f
    Partial-Bug: #1541928

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/292488
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9cf09ef680ed837ae26bc818e26cee8b586d72ed
Submitter: Jenkins
Branch: master

commit 9cf09ef680ed837ae26bc818e26cee8b586d72ed
Author: Martin Hickey <email address hidden>
Date: Mon Mar 14 14:45:23 2016 +0000

    Prevent all primary keys in Neutron OVOs from being updated

    Primary keys in Neutron Oslo Versioned Objects should be prevented
    from being updated. In the future we may reconsider the limitation.

    Current thinking is that you would delete/add instead of update.

    Change-Id: Ice0a43fdd8889f9ce34a7d83d2355d905555b788
    Partial-Bug: #1541928

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/293453

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/293477

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/294018

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/283711
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8bccf9c2a5821be606cded7294efa00078bd3e4b
Submitter: Jenkins
Branch: master

commit 8bccf9c2a5821be606cded7294efa00078bd3e4b
Author: rossella <email address hidden>
Date: Mon Feb 22 13:07:04 2016 +0100

    Handle synthetic fields in NeutronDbObject

    Many objects in neutron have synthetic fields. Synthetic fields
    are filled using the data from fields of a different table.
    This patch makes it possible to handle synthetic fields directly
    in NeutronDbObject so that objects can share this implementation.

    Change-Id: Ia4695b1b10c0370c77b66f31588a56de332f462e
    Partial-bug: #1541928

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/294018
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d503c830fbd5da040a32fa15fdb843b068ccb365
Submitter: Jenkins
Branch: master

commit d503c830fbd5da040a32fa15fdb843b068ccb365
Author: Ihar Hrachyshka <email address hidden>
Date: Thu Mar 17 15:01:03 2016 +0100

    objects: fixed base to_dict implementation

    The implementation was using wrong approach to detect field type, and it
    never triggered the actual code. Adopting the base class implementation
    for QoS policy object revealed the problem.

    Now, we fix the base class to_dict implementation for synthetic fields,
    and are able to adopt it for the policy object.

    Note that the bug in the base class was never exposed in production
    because no objects are currently relying on it.

    Change-Id: I3b8727fd3837b51cc1a778dc73d8a1e06f2de5a9
    Partial-Bug: #1541928

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/273072
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ecde8a9ddbbb226683d3718ee6cd13004c33c3a7
Submitter: Jenkins
Branch: master

commit ecde8a9ddbbb226683d3718ee6cd13004c33c3a7
Author: Martin Hickey <email address hidden>
Date: Wed Jan 27 14:25:44 2016 +0000

    objects: introduce object for extra dhcp options

    This patch introduces a new versioned object for extra dhcp options. It
    will be adopted in Neutron in Newton.

    Change-Id: I45b54b80d994f7e7f06128f73256b94675db97b2
    Partial-Bug: #1541928

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/292178
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=99df8d54bf7d2f88705bd19bd0efaeddc1e4dd42
Submitter: Jenkins
Branch: master

commit 99df8d54bf7d2f88705bd19bd0efaeddc1e4dd42
Author: enriquetaso <email address hidden>
Date: Sun Mar 13 23:51:26 2016 +0000

    Port Security added to OVO

    This patch portes Port Security to Oslo Versioned Objects.

    Co-Authored-By: Victor Morales <email address hidden>
    Change-Id: I5f72b730aaa0439c50845454c0a5f6438b53605f
    Partial-bug: #1541928

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/295275
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1f33d6b43d0f06bbd75a775bc8fa2791f8595faf
Submitter: Jenkins
Branch: master

commit 1f33d6b43d0f06bbd75a775bc8fa2791f8595faf
Author: sayalilunkad <email address hidden>
Date: Mon Mar 21 14:18:19 2016 +0100

    Adds Enum fields for different types

    This patch adds enumfields of type FlowDirection, EtherType and
    IpProtocol which is required for object fields in security group rules.

    Partial-Bug: #1541928

    Change-Id: Ia9bc4f3994789ff04091e43c6e024854fbc527a7

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/293453
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f86e46e50b1542efcdb87e53d79a9b80d84be078
Submitter: Jenkins
Branch: master

commit f86e46e50b1542efcdb87e53d79a9b80d84be078
Author: Ihar Hrachyshka <email address hidden>
Date: Wed Mar 16 14:12:47 2016 +0100

    objects: minor cleanup in base class

    Some comments from Ia4695b1b10c0370c77b66f31588a56de332f462e were not
    addressed in the original patch, but are still worth an effort.

    Change-Id: Ibef7adac940b36cc4a4acdf4a80683237e6ec9ae
    Partial-bug: #1541928

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/275789
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=408173360fd5aef16acd0354be4c35b2a7270c7a
Submitter: Jenkins
Branch: master

commit 408173360fd5aef16acd0354be4c35b2a7270c7a
Author: Victor Morales <email address hidden>
Date: Fri Mar 4 08:31:48 2016 -0600

    objects: SubnetPool, SubnetPoolPrefix

    This patch adds SubnetPool and SubnetPoolPrefix objects. Note: objects
    are not integrated into database plugin code. This is left for a
    follow-up patch.

    Other changes:
    - made modify_fields_to_db object instance method since some objects
      may need to access all fields set on the object;
    - provided a new RangeConstrainedInteger type that validates that value
      is in the range;
    - modified several base test cases to work with objects that modify
      other tables that are different from the object db model.

    Co-Authored-By: Ihar Hrachyshka <email address hidden>
    Related-Bug: #1541928
    Change-Id: I7cbc694ab00c05d0a020fffe4f73141c6ceff7e9

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/301837

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/293477
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f28679c7b06dcbf54dfcd2c94e6f3a3c96f5d20b
Submitter: Jenkins
Branch: master

commit f28679c7b06dcbf54dfcd2c94e6f3a3c96f5d20b
Author: Jakub Libosvar <email address hidden>
Date: Tue Mar 22 16:23:11 2016 +0000

    ovo: Make return values from objects db api consistent

    Previously in db api, we returned dictionaries from create() and update()
    methods while get() methods returned db models. This patch makes the
    return values consistent and thus we always get db model from db api
    operations.

    Related-Bug: 1541928
    Change-Id: I7e9ec4efa20e3d09e1d5a0ca04799095b6ea42b7

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/301837
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=34a328fe12950c339b8259451262470c627f2f00
Submitter: Jenkins
Branch: master

commit 34a328fe12950c339b8259451262470c627f2f00
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Apr 5 19:00:21 2016 +0200

    standard attributes: expose created_at/updated_at on models

    Instead of enforcing all model users to know that those fields belong to
    a separate model, consistently use association_proxy for all standard
    attributes similar to what we already do with description.

    This change should reduce some complexity in db handling code, for
    example for versioned objects.

    Change-Id: I2df5c3514e1f868baa835adff2d95c1156ec7384
    Related-Bug: #1541928

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/302267
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a8b7b275b0687ce967f8bc8c7595d59c2732267e
Submitter: Jenkins
Branch: master

commit a8b7b275b0687ce967f8bc8c7595d59c2732267e
Author: Jakub Libosvar <email address hidden>
Date: Wed Apr 6 13:49:00 2016 +0000

    ovo: Update object with data from db on update()

    The former behavior caused that on calling update() load_from_db()
    object was populated with data from object itself instead of data
    returned from database.

    Partial-Bug: 1541928
    Change-Id: If91934ffca43732457b7d3e414124d998e77cfd8

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/302704

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/302863

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/293479
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b8cd05273f39d010c840c6a0aa89989e03233f95
Submitter: Jenkins
Branch: master

commit b8cd05273f39d010c840c6a0aa89989e03233f95
Author: Martin Hickey <email address hidden>
Date: Wed Mar 16 14:52:25 2016 +0000

    Add custom OVO field type for MAC address

    Partial-Bug: #1541928
    Co-Authored-By: Ihar Hrachyshka <email address hidden>
    Change-Id: If791b25aa673dde32b41c799fad9dc36efc87d04

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/302704
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c2eb7289cbc15acc9a4432103672a385e0a0d0e7
Submitter: Jenkins
Branch: master

commit c2eb7289cbc15acc9a4432103672a385e0a0d0e7
Author: Ihar Hrachyshka <email address hidden>
Date: Thu Apr 7 14:05:50 2016 +0200

    objects: avoid validation in stringify for custom field types

    stringify() is used only in base __repr__ method for objects, where the
    value passed into the method is getattr(self, field_name). Since
    validation was already executed when the attribute was set on the object
    (in 'coerce'), there is no need to repeat the validation in stringify.

    Removed test_stringify_invalid test case since it does not reflect how
    the method is actually used in real life.

    Change-Id: I0846b2005c00afc1e15bb976ba626293d79a27e7
    Partial-Bug: #1541928

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/303406

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/292829
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=633d8acd3383995ea52daf0d32057ed169cd7f38
Submitter: Jenkins
Branch: master

commit 633d8acd3383995ea52daf0d32057ed169cd7f38
Author: Jakub Libosvar <email address hidden>
Date: Tue Mar 15 10:50:28 2016 +0000

    ovo: Introduce standard attributes to objects

    Patch adds hook to NeutronDbObject that checks if db model has standard
    attributes. If so, it extends object fields with standard attributes.

    Partial-Bug: 1541928
    Change-Id: Ib5ef2cc23a48f092ebebe9a7d5ab753c9b9a33df

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/285397
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7ed3cb2ca12ca7a0449c96643dcc0e3f7939bb35
Submitter: Jenkins
Branch: master

commit 7ed3cb2ca12ca7a0449c96643dcc0e3f7939bb35
Author: Martin Hickey <email address hidden>
Date: Fri Feb 26 16:02:22 2016 +0000

    Integrate the Extra Dhcp Opt VersionedObject in Neutron

    This patch is dependent on commit I45b54b80d994f7e7f06128f73256b94675db97b2
    which introduces the Extra Dhcp Opt OVO.
    This patch integrates the VersionedObject with the existing code.

    Change-Id: I0e01831b2b869fbd1ef90bd72a5b63173f554e66
    Partial-Bug: #1541928

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/304322

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/306380

[OpenStack Infra (hudson-openstack)]

Change abandoned by Artur Korzeniewski (<email address hidden>) on branch: master
Review: https://review.openstack.org/306380
Reason: By mistake the patch was pushed with wrong change-id and was not merged with original topic:
https://review.openstack.org/#/c/269658

[Ihar Hrachyshka (ihar-hrachyshka)]

Corresponding blueprint: https://blueprints.launchpad.net/neutron/+spec/adopt-oslo-versioned-objects-for-db

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/287756
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8ecb28dd09aedc0df6b91af4712aa8469a90e850
Submitter: Jenkins
Branch: master

commit 8ecb28dd09aedc0df6b91af4712aa8469a90e850
Author: Martin Hickey <email address hidden>
Date: Thu Mar 3 12:20:27 2016 +0000

    Integrate the port allowed address pairs VersionedObject in Neutron

    This patch is dependent on commit I8d03528f8f45f5f50fa467b39245a513a37c5d89.
    It integrates the VersionedObject with the existing code.

    Integration revealed that using IPAddress is not correct for allowed
    address pairs, because the address can also represent a subnet.

    Another issue revealed by the integration is that we must retain the
    original string format passed by users through API for MAC addresses.

    Neither we can use IPNetworkField from oslo.versionedobjects for
    ip_address field because it will then always append prefix length to
    base network address, even if prefix length is maximum for the type of
    IP network (meaning, the address actually represents a single host),
    which is contradictory to how API currently behaves (returning mask-less
    addresses for /32 - for ipv4 - and /128 - for ipv6 - prefix lengths).

    To solve those issues, 'authentic' flavors for netaddr.EUI and
    netaddr.IPNetwork types are introduced. Those 'authentic' flavors
    attempt to retain the original string representation, as passed by the
    caller.

    Since base IPNetworkField recreates network object on coerce(), and
    hence looses information about the original string representation, we
    introduce our custom flavor of the field type that reuses the network
    object passed by the caller.

    The change for the type of ip_address field triggers hash change.
    Anyway, we are safe to change it without considering backwards
    compatibility, because the object is not used anywhere yet.

    Co-Authored-By: Ihar Hrachyshka <email address hidden>
    Change-Id: I3c937267ce789ed510373616713b3fa9517c18ac
    Partial-Bug: #1541928

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jakub Libosvar (<email address hidden>) on branch: master
Review: https://review.openstack.org/302276
Reason: This patch is not needed anymore for standard attributes as declarative proxies were introduced to timestamps. The API of modify_() methods remain inconsistent so we can get back to it later if needed.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/300055
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5decc850f126d74b89ec79593afcc68b9e8988e6
Submitter: Jenkins
Branch: master

commit 5decc850f126d74b89ec79593afcc68b9e8988e6
Author: Ihar Hrachyshka <email address hidden>
Date: Wed May 18 19:19:48 2016 +0200

    objects: support advanced criteria for get_objects

    Those are needed to accommodate to API request needs without handling
    sorting or pagination in Python.

    Instead of adding four new arguments to get_objects interface, they are
    consolidated using a single Pager object that is passed through the
    _pager argument. The name uses underscore to avoid breaking objects that
    want to pass 'pager' filter into SQLAlchemy. Hopefully, no objects will
    ever have '_pager' attribute in their models.

    Related-Bug: #1541928
    Change-Id: I7dafc4dbd80f0ac35dbc2c2f30e56e441f5b1fc0

[Slawek Kaplonski (slaweq)]

I added tags liberty-backport-potential and mitaka-backport-potential because patch https://review.openstack.org/#/c/300055/ fixes bug https://bugs.launchpad.net/neutron/+bug/1589969

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/269658
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/302863
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/349018

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ankur (<email address hidden>) on branch: master
Review: https://review.openstack.org/349018
Reason: Original rebased

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/356223

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/356660

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/356766

[asingh (aradhana1-singh)]

Implementation for standard_attr_id in Tag model when compared to Port, SecurityGroup etc model is different.
https://github.com/openstack/neutron/blob/master/neutron/db/tag_db.py#L23
https://github.com/openstack/neutron/blob/master/neutron/db/models_v2.py#L117
https://github.com/openstack/neutron/blob/master/neutron/db/models/securitygroup.py#L23
Why is standard_attr_id attribute in Tag is being treated differently? Does the model for tag need to be modified before ovo object is created for it?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/356825

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/357488

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/357516

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/358845

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/359379

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/360908

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/361443

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/363196

[Ihar Hrachyshka (ihar-hrachyshka)]

Ongoing work, should be moved to Ocata when the release line is ready.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Nakul Dahiwade (<email address hidden>) on branch: master
Review: https://review.openstack.org/357516
Reason: The code is supposed to go out of the tree.

[Artur Korzeniewski (artur-korzeniewski)]

This effort is tracked by other BP:
https://blueprints.launchpad.net/neutron/+spec/adopt-oslo-versioned-objects-for-db

I have some patches for Subnet:
https://review.openstack.org/351740
https://review.openstack.org/321001

But they are tracked under blueprint adopt-oslo-versioned-objects-for-db