instance_get_all_by_filters can perform unnecessary joins

[Andrew Garner]

When listing server details, instance_get_all_by_filters() can be invoked with duplicates in the columns_to_join list. This can result in both a join and a separate query against the potential "manual_join" tables.

This appears to have been introduced with the behavior in this commit:

https://github.com/openstack/nova/commit/2e68b2298e94a15d1282c0fb46804b9efa6c8b3a

Where the list of expected_attr here:

https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/views/servers.py#L55
https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/views/servers.py#L110

is further extended with the expected_attr list in the compute_api here:

https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2092

Resulting in a columns_to_join list resembling:

['flavor', 'info_cache', 'metadata', 'metadata', 'system_metadata', 'info_cache', 'security_groups']

In nova.db.sqlalchemy.api:_manual_join_columns(), only the first 'metadata' entry gets removed resulting in both a sqlalchemy joinedload() hint (joining against instance_metadata) and separately querying instance_metadata via _instances_fill_metadata(). The 'metadata' join in particular can be rather inefficient. In some cases this results in about 10x the data being pulled from the database compared to just the "manual join" - this problem is particularly amplified for projects with a large number of associated instances.

[John Garbutt (johngarbutt)]

Hmm, does look like _get_instances_by_filters adds too many joins.

We could just filter out duplicates in the object layer, but maybe best to just check for duplicates and raise an exception, and fix up what we find.

I think we should look why its adding those extra joins only when you add some joins. Its a bit odd, probably need to look at why the callers are doing all that. Its very possible the object layer adds everything that is needed extra now anyways.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/342299

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/342299
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=dc7c21f3cfbd5f68e9bc6b88f75e160826fadb82
Submitter: Jenkins
Branch: master

commit dc7c21f3cfbd5f68e9bc6b88f75e160826fadb82
Author: Pushkar Umaranikar <email address hidden>
Date: Thu Jul 14 17:32:02 2016 +0000

    Removing duplicates from columns_to_join list

    columns_to_join list contains duplicate values as we keep
    extending expected_attrs list in api level.
    This commit involves removing duplicates from columns_to_join
    attribute of instance model to avoid unnecessary joins in
    instance_get_all_by_filters method.
    Sorting resultant set based on list index is required to maintain
    the sequence.

    Change-Id: I2c93cef0babad23fa456bf10ba2ccaec7844b4cd
    Closes-Bug: #1599256

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.0.0.0b1 development milestone.