Metadefs have no 'name'; tricky for sorting; problematic for search

[Steve McLellan]

Metadefs have no 'name' field (because they use namespace), which matches their data, but since name will likely be a common sort field it will potentially cause problems with sort. Several options:
* add one
* add 'unmapped_type' to sort fields
* not worry about it

[Travis Tripp (travis-tripp)]

When doing cross resource searching on name (no type specified), the current queries result in zero results being returned if 1) OS::Glance::Metadef are specified first or 2) no types are specified (the server automatically adds all the types to the query to elasticsearch.

For example, the below will get Nova results (assume server with name demo):

{
    "type": ["OS::Nova::Server", "OS::Glance::Metadef"],
    "query": {
        "bool": {
            "must": [
                {
                    "query_string": {
                        "fuzzy_prefix_length": 2,
                        "fields": [
                            "name"
                        ],
                        "query": "demo~"
                    }
                }
            ]
        }
    }
}

The below will not:

{
    "type": ["OS::Glance::Metadef", "OS::Nova::Server"],
    "query": {
        "bool": {
            "must": [
                {
                    "query_string": {
                        "fuzzy_prefix_length": 2,
                        "fields": [
                            "name"
                        ],
                        "query": "demo~"
                    }
                }
            ]
        }
    }
}

And neither will the below:

{
    "query": {
        "bool": {
            "must": [
                {
                    "query_string": {
                        "fuzzy_prefix_length": 2,
                        "fields": [
                            "name"
                        ],
                        "query": "demo~"
                    }
                }
            ]
        }
    }
}

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/234182

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/234182
Committed: https://git.openstack.org/cgit/openstack/searchlight/commit/?id=eb02e89bed5bb702677aedf29fd35be4cf85ecd2
Submitter: Jenkins
Branch: master

commit eb02e89bed5bb702677aedf29fd35be4cf85ecd2
Author: Lakshmi N Sampath <email address hidden>
Date: Tue Oct 13 17:59:31 2015 +0530

    Fix for no name field in metadef namespace

    Metadefs have no 'name' field (because they use namespace),
    which matches their data, but since name will likely be a
    common sort field it will potentially cause problems with sort.

    When doing cross resource searching on name (no type specified),
    the current queries result in zero results being returned if
    1) OS::Glance::Metadef are specified first or
    2) no types are specified (the server automatically adds all the
       types to the query to elasticsearch.

    Change-Id: I05b26c951ea6d0df890878d40d5113d87af08754
    Closes-Bug: #1501123

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/234432

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/234432
Committed: https://git.openstack.org/cgit/openstack/searchlight/commit/?id=75334b7bdb10f0100f2eac41a73367d50a7a06a9
Submitter: Jenkins
Branch: stable/liberty

commit 75334b7bdb10f0100f2eac41a73367d50a7a06a9
Author: Lakshmi N Sampath <email address hidden>
Date: Tue Oct 13 17:59:31 2015 +0530

    Fix for no name field in metadef namespace

    Metadefs have no 'name' field (because they use namespace),
    which matches their data, but since name will likely be a
    common sort field it will potentially cause problems with sort.

    When doing cross resource searching on name (no type specified),
    the current queries result in zero results being returned if
    1) OS::Glance::Metadef are specified first or
    2) no types are specified (the server automatically adds all the
       types to the query to elasticsearch.

    Change-Id: I05b26c951ea6d0df890878d40d5113d87af08754
    Closes-Bug: #1501123

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/237337

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/237337
Committed: https://git.openstack.org/cgit/openstack/searchlight/commit/?id=f833723a0d2580132e980c62165dcabb13da9751
Submitter: Jenkins
Branch: master

commit 75334b7bdb10f0100f2eac41a73367d50a7a06a9
Author: Lakshmi N Sampath <email address hidden>
Date: Tue Oct 13 17:59:31 2015 +0530

    Fix for no name field in metadef namespace

    Metadefs have no 'name' field (because they use namespace),
    which matches their data, but since name will likely be a
    common sort field it will potentially cause problems with sort.

    When doing cross resource searching on name (no type specified),
    the current queries result in zero results being returned if
    1) OS::Glance::Metadef are specified first or
    2) no types are specified (the server automatically adds all the
       types to the query to elasticsearch.

    Change-Id: I05b26c951ea6d0df890878d40d5113d87af08754
    Closes-Bug: #1501123

commit 31ed77274649cdd3db240cef97c7621f6ff9d612
Author: Steve McLellan <email address hidden>
Date: Mon Oct 12 14:44:23 2015 -0500

    Remove 'not_analyzed' from server.name

    Remove 'not_analyzed' from server name. In general,
    not_analyzed should only be used for id fields and those where it
    doesn't make sense to tokenize values.

    Change-Id: I35bff3a422d1a809b461e3f1fdf162e2fb1c5f24
    Closes-Bug: #1505358

commit 1d0310010d6341fb69e0e45d084ce7f0b34d9423
Author: Travis Tripp <email address hidden>
Date: Mon Oct 12 11:42:34 2015 -0600

    More Doc improvements for Liberty Release

    General organization and wording improvements
    for the documentation in the liberty release.

    Also fixes bad designate configuration option
    documentation.

    Change-Id: I6d3ef1cd3970a7b39e0ce13f1cde5d4226b4def1
    Partial-Bug: #1500647
    (cherry picked from commit b5e5d9181ca9849b3f376c4a178026f4aeae7bc2)

commit 83ed8e723de91bd095aaa505fea41b49fd7532c1
Author: Travis Tripp <email address hidden>
Date: Fri Oct 9 13:45:56 2015 -0600

    Document unsearchable_fields configuration option

    Although RBAC prevents protected fields appearing in search results, it
    cannot prevent searching those fields directly. A long term solution is
    a separate admin-only index but for now we add a resource plugin option
    'unsearchable_fields' which is comma separated and accepts wildcards as
    elasticsearch's mapping_template allows. These fields are turned into
    index: no, include_in_all: False mapping templates. The value of this
    (plus anything coded into a plugin) is also used to redact fields in
    _source for non-administrative users.

    This patch provides the documentation update for:
     - https://review.openstack.org/#/c/233225/

    Change-Id: I94f47515391cf81d72a6a0d09e7a554259a1f8fb
    Partial-Bug: #1504399
    (cherry picked from commit 0f1518e3d636045350df4a5e8dd8807530691359)

commit 9ea77ec031e785335796120d4cf369f2c6e28710
Author: Travis Tripp <email address hidden>
Date: Mon Sep 28 06:44:04 2015 -0600

    Improve documentation for Liberty release

    The documentation cross linking from the devstac...

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/searchlight 0.2.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/searchlight 0.1.0 release.