neutron provider parses the wrong json output, breaking all idempotency (prefetch) in a lot of providers.

Bug #1530163 reported by Sofer Athlan-Guyot on 2015-12-30
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
puppet-neutron
Critical
Sofer Athlan-Guyot

Bug Description

Hi,

The neutron network provider is parsing the wrong json format leading
to an error when you're unlucky. Affected providers[1]:
 - neutron_port
 - neutron_subnet
 - neutron_router
 - neutron_network

The context:

while trying puppet-openstack-integration on centos7 I had a failure:

Error: Could not prefetch neutron_network provider
       'neutron': No resource and no name in property hash in neutron instance
Error: /Stage[main]/Main/Neutron_subnet[public-subnet]/ensure: change
       from absent to present failed: did not get expected message on subnet
       creation, got

It was on the second run, the idempotency check.

Basically the prefetch step was failing for both neutron_network and
neutron_subnet with different outcomes.

For neutron_network, a second network was silently created, for
subnet, the API refused the creation of the subnet and the neutron end
up failing as show above.

The reason:

You have the complete explication there [2] but I recap here for
convenience.

Basically neutron provider parses the json of the cliff-tablib's
output while the official one is the cliff' output.

neutron (and openstack) client use stevedore for dynamically loading
plugin. No configuration is available to the user, it's just loaded.
It appears that on most installation this command:

   neutron --os-username admin --os-password a_big_secret \
     --os-project-name openstack --os-user-domain-name Default \
     --os-project-domain-name Default --os-auth-url \
     http://127.0.0.1:5000/v3 \
       net-show 485ab85e-e183-47ba-ac08-6dbaa058cc4e --format json

would returns

      [{"Field": "admin_state_up", "Value": true}, {"Field": "id",
      "Value": "485ab85e-e183-47ba-ac08-6dbaa058cc4e"}, {"Field":
      "mtu", "Value": 0}, {"Field": "name", "Value": "public"},
      {"Field": "provider:network_type", "Value": "vxlan"}, {"Field":
      "provider:physical_network", "Value": ""}, {"Field":
      "provider:segmentation_id", "Value": 89}, {"Field":
      "router:external", "Value": true}, {"Field": "shared", "Value":
      false}, {"Field": "status", "Value": "ACTIVE"}, {"Field":
      "subnets", "Value": "4528f536-c2ea-4cb2-860e-65200a6d8651"},
      {"Field": "tenant_id", "Value":
      "3db46df94f1045fab93950758b794a52"}]

But on some others it would return:

    {
      "status": "ACTIVE",
      "router:external": true,
      "subnets": "4528f536-c2ea-4cb2-860e-65200a6d8651",
      "name": "public",
      "provider:physical_network": "",
      "admin_state_up": true,
      "tenant_id": "3db46df94f1045fab93950758b794a52",
      "provider:network_type": "vxlan",
      "shared": false,
      "mtu": 0,
      "id": "485ab85e-e183-47ba-ac08-6dbaa058cc4e",
      "provider:segmentation_id": 89
    }

It all depends on how stevedore is loading the pluging (inode sort
order ?)

You can see the loading using the "--debug" option to the neutron
command:

    DEBUG: stevedore.extension found extension EntryPoint.parse('yaml = clifftablib.formatters:YamlFormatter')
    DEBUG: stevedore.extension found extension EntryPoint.parse('json = clifftablib.formatters:JsonFormatter')
    DEBUG: stevedore.extension found extension EntryPoint.parse('html = clifftablib.formatters:HtmlFormatter')
    DEBUG: stevedore.extension found extension EntryPoint.parse('table = cliff.formatters.table:TableFormatter')
    DEBUG: stevedore.extension found extension EntryPoint.parse('json = cliff.formatters.json_format:JSONFormatter')
    DEBUG: stevedore.extension found extension EntryPoint.parse('shell = cliff.formatters.shell:ShellFormatter')
    DEBUG: stevedore.extension found extension EntryPoint.parse('value = cliff.formatters.value:ValueFormatter')
    DEBUG: stevedore.extension found extension EntryPoint.parse('yaml = cliff.formatters.yaml_format:YAMLFormatter')

Oki that's the recap. See [2] for more.

What it means for the puppet neutron module ?

Well it will fail randomly on different install if you use the code
that include[3]. It will completely fail if the openstack cli (or
something) is not pulling the clifftablib as the json parser is
expecting the clifftablib output[4].

This is unfortunate as the supported version is cliff output[2]. It
only works because openstack cli is requiring clifftablib. This
requirement will disappear in Mitaka[2].

You can reproduce the problem by editing [5] or [6] depending on the
output you want.

[1] they all consume get_neutron_resource_attrs
[2] https://bugs.launchpad.net/python-neutronclient/+bug/1529914
[3] https://review.openstack.org/#/c/238156/
[4] https://github.com/openstack/puppet-neutron/blob/master/lib/puppet/provider/neutron.rb#L187-L200
[5] /usr/lib/python2.7/site-packages/cliff-1.15.0-py2.7.egg-info/entry_points.txt
[6] /usr/lib/python2.7/site-packages/cliff_tablib-1.1-py2.7.egg-info/entry_points.txt

Changed in puppet-neutron:
importance: Undecided → Critical
status: New → Confirmed

Fix proposed to branch: master
Review: https://review.openstack.org/262809

Changed in puppet-neutron:
assignee: nobody → Sofer Athlan-Guyot (sofer-athlan-guyot)
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/262809
Committed: https://git.openstack.org/cgit/openstack/puppet-neutron/commit/?id=273ef1b228fbcb73edf021fabbf77ff5054e7dd7
Submitter: Jenkins
Branch: master

commit 273ef1b228fbcb73edf021fabbf77ff5054e7dd7
Author: Sofer Athlan-Guyot <email address hidden>
Date: Thu Dec 31 16:36:05 2015 +0100

    Neutron parses the wrong json revert to shell/csv.

    The neutron network provider is parsing the wrong json format leading
    to an error when you're unlucky. Affected providers:
     - neutron_port
     - neutron_subnet
     - neutron_router
     - neutron_network

    The neutron provider parses the json of the cliff-tablib's output while
    the official one is the cliff' output[1].

    As discussed in the ML[2] this patch implements a simple revert of the
    json change.

    Revert "Use json output instead plain-text"

    This reverts commit 79f0e636c07b80b2c6699d934b42641bfef4352b[3]
     and part of commit 791a0f146b28544ec3d70c0ab2a950c4d5ca9f98[4]

    [1]: https://bugs.launchpad.net/python-neutronclient/+bug/1529914
    [2]: http://lists.openstack.org/pipermail/openstack-dev/2015-December/083172.html
    [3]: https://review.openstack.org/#/c/238156/
    [4]: https://review.openstack.org/#/c/261418/

    Change-Id: I93263256eb9d83544910ca4055bb3e9a463645e3
    Closes-bug: 1530163

Changed in puppet-neutron:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/275282
Committed: https://git.openstack.org/cgit/openstack/puppet-neutron/commit/?id=e95d00d7b6d5f25b21d84928cee6cff5f2f6ea3e
Submitter: Jenkins
Branch: stable/liberty

commit e95d00d7b6d5f25b21d84928cee6cff5f2f6ea3e
Author: Sofer Athlan-Guyot <email address hidden>
Date: Thu Dec 31 16:36:05 2015 +0100

    Neutron parses the wrong json revert to shell/csv.

    The neutron network provider is parsing the wrong json format leading
    to an error when you're unlucky. Affected providers:
     - neutron_port
     - neutron_subnet
     - neutron_router
     - neutron_network

    The neutron provider parses the json of the cliff-tablib's output while
    the official one is the cliff' output[1].

    As discussed in the ML[2] this patch implements a simple revert of the
    json change.

    Revert "Use json output instead plain-text"

    This reverts commit 79f0e636c07b80b2c6699d934b42641bfef4352b[3]
     and part of commit 791a0f146b28544ec3d70c0ab2a950c4d5ca9f98[4]

    [1]: https://bugs.launchpad.net/python-neutronclient/+bug/1529914
    [2]: http://lists.openstack.org/pipermail/openstack-dev/2015-December/083172.html
    [3]: https://review.openstack.org/#/c/238156/
    [4]: https://review.openstack.org/#/c/261418/

    Change-Id: I93263256eb9d83544910ca4055bb3e9a463645e3
    Closes-bug: 1530163
    (cherry picked from commit 273ef1b228fbcb73edf021fabbf77ff5054e7dd7)

tags: added: in-stable-liberty

This issue was fixed in the openstack/puppet-neutron 7.1.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers