jsonutils should use simplejson on python 2.6 if available

[Ihar Hrachyshka]

Python 2.6 ships 'json' module that is very slow because it's written in pure Python. Python 2.7 updated [1] its 'json' module from simplejson PyPI repo with a version that is based on C extension (and quick). Quoting: "Updated module: The json module was upgraded to version 2.0.9 of the simplejson package, which includes a C extension that makes encoding and decoding faster. (Contributed by Bob Ippolito; issue 4136.)"

We should strive to use simplejson library when running on Python 2.6.

[1]: https://docs.python.org/dev/whatsnew/2.7.html

[Ihar Hrachyshka (ihar-hrachyshka)]

Added all projects that sync jsonutils from oslo-incubator to the list of affected projects.

[Ihar Hrachyshka (ihar-hrachyshka)]

For some reason, Gerrit didn't post the comment with the link to review here, so here it is: https://review.openstack.org/#/c/90968/

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/90968
Committed: https://git.openstack.org/cgit/openstack/oslo-incubator/commit/?id=a6b2aecf3a75e4bdef493b645112a7944bced1e1
Submitter: Jenkins
Branch: master

commit a6b2aecf3a75e4bdef493b645112a7944bced1e1
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Apr 29 12:03:30 2014 +0200

    On Python <= 2.6, use simplejson if available

    Until Python 2.7, stdlib json module was not boosted with a C extension,
    meaning bad performance. Try to use simplejson module if available.

    RHEL 6.5 is one of the platforms that still ship Python 2.6.

    The original idea of the patch belongs to Miguel Angel Ajo Pelayo @
    <email address hidden>

    Change-Id: Ib3dc0b713ed90396919feba018772243b3b9c90f
    Closes-Bug: 1314129

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: master
Review: https://review.openstack.org/91049

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: master
Review: https://review.openstack.org/91053

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: master
Review: https://review.openstack.org/91054

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: master
Review: https://review.openstack.org/91055

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: master
Review: https://review.openstack.org/91057

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: master
Review: https://review.openstack.org/91080

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/91070
Committed: https://git.openstack.org/cgit/openstack/marconi/commit/?id=a4f1b642fbb683d0534e32af4929c483ce503c2a
Submitter: Jenkins
Branch: master

commit a4f1b642fbb683d0534e32af4929c483ce503c2a
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Apr 29 17:21:29 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that drastically enhances performance on Python
    2.6 with fresh simplejson library installed.

    The latest commit in oslo-incubator:
    - 732bdb6297eb9de81667f7713ebcb1ccc2ee45a7

    Change-Id: Ib3dc0b713ed90396919feba018772243b3b9c90f
    Closes-Bug: 1314129

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/91077
Committed: https://git.openstack.org/cgit/openstack/tuskar/commit/?id=5cba8de9a06e2adf25f4b11c62a813b389eb1fc2
Submitter: Jenkins
Branch: master

commit 5cba8de9a06e2adf25f4b11c62a813b389eb1fc2
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Apr 29 17:21:52 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that drastically enhances performance on Python
    2.6 with fresh simplejson library installed.

    The latest commit in oslo-incubator:
    - 732bdb6297eb9de81667f7713ebcb1ccc2ee45a7

    Change-Id: Ib3dc0b713ed90396919feba018772243b3b9c90f
    Closes-Bug: 1314129

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/91055
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=adb4cac2fba84a4cb4c81bf1dd1403c73d090122
Submitter: Jenkins
Branch: master

commit adb4cac2fba84a4cb4c81bf1dd1403c73d090122
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Apr 29 16:37:34 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that drastically enhances performance on Python
    2.6 with fresh simplejson library installed.

    The latest commit in oslo-incubator:
    - 732bdb6297eb9de81667f7713ebcb1ccc2ee45a7

    Change-Id: Ib3dc0b713ed90396919feba018772243b3b9c90f
    Closes-Bug: 1314129

[Openstack Gerrit (openstack-gerrit)]

Related fix proposed to branch: master
Review: https://review.openstack.org/91344

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/91041
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=88f9bbb6e643472f9218ea88e19afc57ffab025b
Submitter: Jenkins
Branch: master

commit 88f9bbb6e643472f9218ea88e19afc57ffab025b
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Apr 29 16:23:46 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that drastically enhances performance on Python
    2.6 with fresh simplejson library installed.

    The latest commit in oslo-incubator:
    - 732bdb6297eb9de81667f7713ebcb1ccc2ee45a7

    Change-Id: Ib3dc0b713ed90396919feba018772243b3b9c90f
    Closes-Bug: 1314129

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/91454

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/91457

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/91075
Committed: https://git.openstack.org/cgit/openstack/python-novaclient/commit/?id=8e0e25f6ceac9525cf0fd6fc36e6295874081d5a
Submitter: Jenkins
Branch: master

commit 8e0e25f6ceac9525cf0fd6fc36e6295874081d5a
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Apr 29 17:26:19 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that drastically enhances performance on Python
    2.6 with fresh simplejson library installed.

    The latest commit in oslo-incubator:
    - 732bdb6297eb9de81667f7713ebcb1ccc2ee45a7

    Change-Id: Ib3dc0b713ed90396919feba018772243b3b9c90f
    Closes-Bug: 1314129

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/91049
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=9e60bcdb6b2955fe1681e3a28fa0ad7d12348c5d
Submitter: Jenkins
Branch: master

commit 9e60bcdb6b2955fe1681e3a28fa0ad7d12348c5d
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Apr 29 16:32:15 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that drastically enhances performance on Python
    2.6 with fresh simplejson library installed.

    The latest commit in oslo-incubator:
    - 732bdb6297eb9de81667f7713ebcb1ccc2ee45a7

    Change-Id: Ib3dc0b713ed90396919feba018772243b3b9c90f
    Closes-Bug: 1314129

[Ihar Hrachyshka (ihar-hrachyshka)]

Several unicode related issues were revealed by actual syncs from oslo-incubator to affected projects.

https://review.openstack.org/91044: nova expects json.loads() to always return unicode strings.
https://review.openstack.org/91068: trove expects json.loads() to always return unicode strings.
https://review.openstack.org/91047: glance expected json.load() to always return unicode strings.

Simplejson library applies some optimisations that does not guarantee unicode strings if bytes ASCII-only input is parsed. That's why we need to pass unicode strings and file objects to underlying json implementation. For this, another oslo-incubator patch was sent to make sure json.load() and json.loads() always return unicode strings inside json dictionaries: https://review.openstack.org/91344

After discussion with Doug in IRC and Gerrit, we've come to the conclusion that we can't just apply codecs.getreader('utf-8') to fp argument of jsonutils.load() since this may fail for other ASCII-based encodings other than UTF-8. As per official json module description at https://docs.python.org/2/library/json.html#json.load

"If the contents of fp are encoded with an ASCII based encoding other than UTF-8 (e.g. latin-1), then an appropriate encoding name must be specified. Encodings that are not ASCII based (such as UCS-2) are not allowed, and should be wrapped with codecs.getreader(encoding)(fp), or simply decoded to a unicode object and passed to loads()."

Also as per JSONDecoder.__init__() in official json.decoder module,

"``encoding`` determines the encoding used to interpret any ``str`` objects decoded by this instance (utf-8 by default). It has no effect when decoding ``unicode`` objects."

So to make sure we're on par with stdlib json implementation, we need to support encoding argument to allow non UTF-8 encodings, and assume we're passed a 'utf-8' file object that is safe to codecs.getreader('utf-8')(fp) otherwise.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/91344
Committed: https://git.openstack.org/cgit/openstack/oslo-incubator/commit/?id=18f2bc1bf080b41a22c70842e7c127da21c63b8b
Submitter: Jenkins
Branch: master

commit 18f2bc1bf080b41a22c70842e7c127da21c63b8b
Author: Ihar Hrachyshka <email address hidden>
Date: Wed Apr 30 14:53:15 2014 +0200

    Enforce unicode json output for jsonutils.load[s]()

    simplejson module applies some optimizations on ASCII-only unicode
    strings which result in non-unicode json output. See details at [1]
    and [2]. To make sure we always return consistent json output no
    matter which json implementation is used, we should explicitly convert
    the input for json.load[s]() to unicode.

    If user wants to pass a file object of non UTF-8 encoding to
    json.load[s](), she must also specify this encoding as an argument. To
    support this scenario too, we've added 'encoding' argument to
    jsonutils.load[s]() implementation.

    Made all present JSON tests to run on both supported json library
    implementations.

    Added explicit dependency for simplejson to be able to test different
    implementations in unit tests. Distributors still running Python 2.6
    are recommended but not required to install simplejson.

    Related-Bug: 1314129

    [1]: https://code.djangoproject.com/ticket/11742
    [2]: https://code.google.com/p/simplejson/issues/detail?id=40

    Change-Id: Ic815ca3df94c33edec9104172048b2cd94b92e3f

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/95690

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/95691

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/95692

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/95693

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/95694

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/95695

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/95696

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/95697

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/95698

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/95699

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/95700

[Zhi Yan Liu (lzy-dev)]

https://review.openstack.org/#/c/91047/ change addressed Glance part.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/95749

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/havana
Review: https://review.openstack.org/95753

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/95695
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2dcdd3bdb74b808afccb8278b0cca52c76f99d1a
Submitter: Jenkins
Branch: master

commit 2dcdd3bdb74b808afccb8278b0cca52c76f99d1a
Author: Ihar Hrachyshka <email address hidden>
Date: Tue May 27 10:22:49 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that makes sure we get unicode-only dicts from
    jsonutils no matter which json module implementation is selected.

    The latest commit in oslo-incubator:
    - 0f4586c0076183c6356eec682c8a593648125abd

    The sync adds a new 'strutils' module that is now used in jsonutils.

    Change-Id: Ic815ca3df94c33edec9104172048b2cd94b92e3f
    Closes-Bug: 1314129

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/91057
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=628470a3aca301db6c4a54adf3ce862381b9adf1
Submitter: Jenkins
Branch: master

commit 628470a3aca301db6c4a54adf3ce862381b9adf1
Author: Ihar Hrachyshka <email address hidden>
Date: Tue May 27 10:24:07 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that drastically enhances performance on Python
    2.6 with fresh simplejson library installed.

    The latest commit in oslo-incubator:
    - 0f4586c0076183c6356eec682c8a593648125abd

    The sync adds a new 'strutils' module that is now used in jsonutils.

    Change-Id: Ib3dc0b713ed90396919feba018772243b3b9c90f
    Closes-Bug: 1314129

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/92226
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=185e4562df47a101cf41d1e66d75de2644c78022
Submitter: Jenkins
Branch: master

commit 185e4562df47a101cf41d1e66d75de2644c78022
Author: pran1990 <email address hidden>
Date: Mon May 5 15:31:30 2014 -0700

    Sync oslo network utils

    A new function was added to oslo-incubator for setting tcp keepalive
    settings, and dependent projects can use it.

    Ran update.py in oslo-incubator to copy changed modules into nova.
    Removing synced changes for jsonutils because of issues with how
    strings are encoded with simplejson that cause some of our unittests
    to fail.

    Changes are:
    network_utils.py (which contains the function we need, others are deps)
    edd73c9 Merge "Improve help strings"
    fe3389e Improve help strings
    8a0f567 Remove str() from LOG.* and exceptions
    f61a4e7 Fix python26 compatibility for RFCSysLogHandler
    da6d713 Revert setting oslo-incubator logs to INFO
    0d18381 Set default log levels for oslo.messaging and oslo-incubator

    gettextutils.py
    9912e5d Add API for creating translation functions
    6cc96d0 Fix test_gettextutils on Python 3

    log.py
    edd73c9 Merge "Improve help strings"
    fe3389e Improve help strings
    8a0f567 Remove str() from LOG.* and exceptions
    f61a4e7 Fix python26 compatibility for RFCSysLogHandler
    da6d713 Revert setting oslo-incubator logs to INFO
    0d18381 Set default log levels for oslo.messaging and oslo-incubator

    Current HEAD in OSLO:
    ----------------------------------------------
    Merge: 2640847 e433899
    Date: Mon May 5 19:41:28 2014 +0000
    Merge "Enable configuring tcp keepalive"
    ----------------------------------------------

    Implements: blueprint configure-tcp-keepalive
    Related-Bug: 1314129
    Change-Id: Idd04ad4ad9a06ef76941dc5c1be41ac3ac8877a7

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/95691
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=e5dd3b6ab06789774b58c3b7508ed3ad1219249b
Submitter: Jenkins
Branch: master

commit e5dd3b6ab06789774b58c3b7508ed3ad1219249b
Author: Ihar Hrachyshka <email address hidden>
Date: Tue May 27 10:22:28 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that makes sure we get unicode-only dicts from
    jsonutils no matter which json module implementation is selected.

    The latest commit in oslo-incubator:
    - 0f4586c0076183c6356eec682c8a593648125abd

    Change-Id: Ic815ca3df94c33edec9104172048b2cd94b92e3f
    Closes-Bug: 1314129

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/95845
Committed: https://git.openstack.org/cgit/openstack/python-keystoneclient/commit/?id=54cc999a4def0bcf6faef4da9bc2f7caf131044f
Submitter: Jenkins
Branch: master

commit 54cc999a4def0bcf6faef4da9bc2f7caf131044f
Author: Steve Martinelli <email address hidden>
Date: Tue May 27 13:53:58 2014 -0400

    Sync with oslo-incubator caed79d

    This syncs python-keystoneclient with oslo-incubator commit hash
    caed79d8239679cb74476bb0d9e5011b4fcc39da

    First, remove the existing code to cleanup:
     $ rm -r keystoneclient/openstack/*

    Then, sync from oslo-incubator:
     $ python update.py ../python-keystoneclient

    Commits since last sync (2640847):
    ----------------------------------
    1173e46 Remove ValueError when accessing sys.modules
    90ae24b Remove redundant default=None for config options
    18f2bc1 Enforce unicode json output for jsonutils.load[s]()
    4a777e5 Fix warnings in doc build for apiclient

    Related-Bug: #1315523
    Closes-Bug: #1314129
    Change-Id: Id53ea2d4828e81bc6270f186849f391e7bc73101

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/95694
Committed: https://git.openstack.org/cgit/openstack/marconi/commit/?id=7c1de0e4d8c9e5b8d5134cf27868a2c0f8c1bc04
Submitter: Jenkins
Branch: master

commit 7c1de0e4d8c9e5b8d5134cf27868a2c0f8c1bc04
Author: Ihar Hrachyshka <email address hidden>
Date: Tue May 27 10:22:49 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that makes sure we get unicode-only dicts from
    jsonutils no matter which json module implementation is selected.

    The latest commit in oslo-incubator:
    - 0f4586c0076183c6356eec682c8a593648125abd

    The sync adds a new 'strutils' module that is now used in jsonutils.

    Change-Id: Ic815ca3df94c33edec9104172048b2cd94b92e3f
    Closes-Bug: 1314129

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: master
Review: https://review.openstack.org/91054
Reason: Superseded by full sync.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: master
Review: https://review.openstack.org/95697
Reason: Another sync already merged.

[Dolph Mathews (dolph)]

Removed python-keystoneclient from this bug due to launchpad issues - fix released in python-keystoneclient 0.9.0.

[Dirk Mueller (dmllr)]

fwiw this causes bug 1325235 for me.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/95699
Committed: https://git.openstack.org/cgit/openstack/taskflow/commit/?id=e60ce2db7ee8225534aa15c6470cde284011366d
Submitter: Jenkins
Branch: master

commit e60ce2db7ee8225534aa15c6470cde284011366d
Author: Ihar Hrachyshka <email address hidden>
Date: Tue May 27 10:22:49 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that makes sure we get unicode-only dicts from
    jsonutils no matter which json module implementation is selected.

    The latest commit in oslo-incubator:
    - 0f4586c0076183c6356eec682c8a593648125abd

    The sync adds a new 'strutils' module that is now used in jsonutils.

    Change-Id: Ic815ca3df94c33edec9104172048b2cd94b92e3f
    Closes-Bug: 1314129

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/95698
Committed: https://git.openstack.org/cgit/openstack/python-novaclient/commit/?id=1e40b41a9e3bdbfbf485ddbace55d47e78e66a1b
Submitter: Jenkins
Branch: master

commit 1e40b41a9e3bdbfbf485ddbace55d47e78e66a1b
Author: Ihar Hrachyshka <email address hidden>
Date: Tue May 27 10:22:49 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that makes sure we get unicode-only dicts from
    jsonutils no matter which json module implementation is selected.

    The latest commit in oslo-incubator:
    - 0f4586c0076183c6356eec682c8a593648125abd

    Change-Id: Ic815ca3df94c33edec9104172048b2cd94b92e3f
    Closes-Bug: 1314129

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: stable/havana
Review: https://review.openstack.org/95753
Reason: Havana is for security fixes only now, so abandoning the patch.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: stable/havana
Review: https://review.openstack.org/91457
Reason: Havana is for security fixes only now, so abandoning the patch.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/95690
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=fe3e0f171895ad4b4344d6cd24aa624f0e911621
Submitter: Jenkins
Branch: master

commit fe3e0f171895ad4b4344d6cd24aa624f0e911621
Author: Ihar Hrachyshka <email address hidden>
Date: Tue May 27 10:22:28 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that makes sure we get unicode-only dicts from
    jsonutils no matter which json module implementation is selected.

    The latest commit in oslo-incubator:
    - 0f4586c0076183c6356eec682c8a593648125abd

    Change-Id: Ic815ca3df94c33edec9104172048b2cd94b92e3f
    Closes-Bug: 1314129

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/95700
Committed: https://git.openstack.org/cgit/openstack/tuskar/commit/?id=41d6350e5c92f42ed7fc3cc137fd6267f9d6c761
Submitter: Jenkins
Branch: master

commit 41d6350e5c92f42ed7fc3cc137fd6267f9d6c761
Author: Ihar Hrachyshka <email address hidden>
Date: Tue May 27 10:22:49 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that makes sure we get unicode-only dicts from
    jsonutils no matter which json module implementation is selected.

    The latest commit in oslo-incubator:
    - 0f4586c0076183c6356eec682c8a593648125abd

    The sync adds a new 'strutils' module that is now used in jsonutils.

    Change-Id: Ic815ca3df94c33edec9104172048b2cd94b92e3f
    Closes-Bug: 1314129

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/91074
Committed: https://git.openstack.org/cgit/openstack/python-neutronclient/commit/?id=44d45203e828b4812638da970dc9acb9ff4065d9
Submitter: Jenkins
Branch: master

commit 44d45203e828b4812638da970dc9acb9ff4065d9
Author: Ihar Hrachyshka <email address hidden>
Date: Tue May 27 10:22:49 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that drastically enhances performance on Python
    2.6 with fresh simplejson library installed.

    The latest commit in oslo-incubator:
    - 0f4586c0076183c6356eec682c8a593648125abd

    Change-Id: Ib3dc0b713ed90396919feba018772243b3b9c90f
    Closes-Bug: 1314129

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/91454
Committed: https://git.openstack.org/cgit/openstack/oslo-incubator/commit/?id=d14f56f02a289a9d1f6ac9a9c43adbc32fd569d6
Submitter: Jenkins
Branch: stable/icehouse

commit d14f56f02a289a9d1f6ac9a9c43adbc32fd569d6
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Apr 29 12:03:30 2014 +0200

    On Python <= 2.6, use simplejson if available

    Until Python 2.7, stdlib json module was not boosted with a C extension,
    meaning bad performance. Try to use simplejson module if available.

    RHEL 6.5 is one of the platforms that still ship Python 2.6.

    The original idea of the patch belongs to Miguel Angel Ajo Pelayo @
    <email address hidden>

    Change-Id: Ib3dc0b713ed90396919feba018772243b3b9c90f
    Closes-Bug: 1314129
    (cherry picked from commit a6b2aecf3a75e4bdef493b645112a7944bced1e1)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/95749
Committed: https://git.openstack.org/cgit/openstack/oslo-incubator/commit/?id=1b9018859fa3b420d01ed191b916476db6eb9182
Submitter: Jenkins
Branch: stable/icehouse

commit 1b9018859fa3b420d01ed191b916476db6eb9182
Author: Ihar Hrachyshka <email address hidden>
Date: Wed Apr 30 14:53:15 2014 +0200

    Enforce unicode json output for jsonutils.load[s]()

    simplejson module applies some optimizations on ASCII-only unicode
    strings which result in non-unicode json output. See details at [1]
    and [2]. To make sure we always return consistent json output no
    matter which json implementation is used, we should explicitly convert
    the input for json.load[s]() to unicode.

    If user wants to pass a file object of non UTF-8 encoding to
    json.load[s](), she must also specify this encoding as an argument. To
    support this scenario too, we've added 'encoding' argument to
    jsonutils.load[s]() implementation.

    Made all present JSON tests to run on both supported json library
    implementations.

    Added explicit dependency for simplejson to be able to test different
    implementations in unit tests. Distributors still running Python 2.6
    are recommended but not required to install simplejson.

    Related-Bug: 1314129

    [1]: https://code.djangoproject.com/ticket/11742
    [2]: https://code.google.com/p/simplejson/issues/detail?id=40

    Conflicts:
     tests/unit/test_jsonutils.py

    Change-Id: Ic815ca3df94c33edec9104172048b2cd94b92e3f
    (cherry picked from commit 18f2bc1bf080b41a22c70842e7c127da21c63b8b)

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steve Baker (<email address hidden>) on branch: master
Review: https://review.openstack.org/95693
Reason: should be fixed by full oslo merge

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/102036

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/102036
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8131a2eca838d1cb584646615e48fc11d8a90571
Submitter: Jenkins
Branch: stable/icehouse

commit 8131a2eca838d1cb584646615e48fc11d8a90571
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Jun 24 00:13:17 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that drastically enhances performance on Python
    2.6 with fresh simplejson library installed.

    Included commits:
    * jsonutils.py:
      - 1b901885: Enforce unicode json output for jsonutils.load[s]()
      - cd909d83: jsonutils.load() accepts file pointer, not string
      - d14f56f0: On Python <= 2.6, use simplejson if available

    This change also introduces new incubator module 'strutils' that is used
    by jsonutils now.

    Change-Id: Ief94101488214ddf93c3547a9c50134826b9b165
    Closes-Bug: 1314129

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/103130

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: stable/havana
Review: https://review.openstack.org/103130

[Mark Washenberger (markwash)]

It appears this was fixed in glance in https://review.openstack.org/91047

[Marios Andreou (marios-b)]

WRT Tuskar, afaics this was included in v0.4.5 (commit 41d6350e5c92f42ed7fc3cc137fd6267f9d6c761) so setting fix released there (I am here because I ran the tripleo release last week, 0.4.7 for Tuskar, and closing bugs as part of that process)

thanks

[Marios Andreou (marios-b)]

Am trying to set this as fix-released for Tuskar ... WRT Tuskar, afaics this was included in v0.4.5 (commit 41d6350e5c92f42ed7fc3cc137fd6267f9d6c761) so setting fix released there (I am here because I ran the tripleo release last week, 0.4.7 for Tuskar, and closing bugs as part of that process)

I can't change to fix-released though, keeps timing out, not sure if this is a permissions issue (do I need to assign to myself first?)

[Carl Baldwin (carl-baldwin)]

Here is one example where json and simplejson and regular old json are not the same. How many types were tested for equivalence?

#!/usr/bin/env python

import collections
import simplejson
import json

PortInfo = collections.namedtuple("PortInfo", "mac_address ip_address")

pi = PortInfo('ca:fe:de:ad:be:ef', '192.168.1.1')

print json.dumps(pi)
print simplejson.dumps(pi)

This prints:
["ca:fe:de:ad:be:ef", "192.168.1.1"]
{"ip_address": "192.168.1.1", "mac_address": "ca:fe:de:ad:be:ef"}

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/95753
Committed: https://git.openstack.org/cgit/openstack/oslo-incubator/commit/?id=6a290f5dedcc00538c12e7b1e3a2374ea48919c0
Submitter: Jenkins
Branch: stable/havana

commit 6a290f5dedcc00538c12e7b1e3a2374ea48919c0
Author: Ihar Hrachyshka <email address hidden>
Date: Wed Apr 30 14:53:15 2014 +0200

    Enforce unicode json output for jsonutils.load[s]()

    simplejson module applies some optimizations on ASCII-only unicode
    strings which result in non-unicode json output. See details at [1]
    and [2]. To make sure we always return consistent json output no
    matter which json implementation is used, we should explicitly convert
    the input for json.load[s]() to unicode.

    If user wants to pass a file object of non UTF-8 encoding to
    json.load[s](), she must also specify this encoding as an argument. To
    support this scenario too, we've added 'encoding' argument to
    jsonutils.load[s]() implementation.

    Made all present JSON tests to run on both supported json library
    implementations.

    Added explicit dependency for simplejson to be able to test different
    implementations in unit tests. Distributors still running Python 2.6
    are recommended but not required to install simplejson.

    Related-Bug: 1314129

    [1]: https://code.djangoproject.com/ticket/11742
    [2]: https://code.google.com/p/simplejson/issues/detail?id=40

    Conflicts:
     test-requirements-py3.txt
     tests/unit/test_jsonutils.py

    Change-Id: Ic815ca3df94c33edec9104172048b2cd94b92e3f
    (cherry picked from commit 18f2bc1bf080b41a22c70842e7c127da21c63b8b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/91457
Committed: https://git.openstack.org/cgit/openstack/oslo-incubator/commit/?id=7e3c27fb50d99c23d35f7e909086ee06416adc1d
Submitter: Jenkins
Branch: stable/havana

commit 7e3c27fb50d99c23d35f7e909086ee06416adc1d
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Apr 29 12:03:30 2014 +0200

    On Python <= 2.6, use simplejson if available

    Until Python 2.7, stdlib json module was not boosted with a C extension,
    meaning bad performance. Try to use simplejson module if available.

    RHEL 6.5 is one of the platforms that still ship Python 2.6.

    The original idea of the patch belongs to Miguel Angel Ajo Pelayo @
    <email address hidden>

    Conflicts:
     openstack/common/jsonutils.py

    Change-Id: Ib3dc0b713ed90396919feba018772243b3b9c90f
    Closes-Bug: 1314129
    (cherry picked from commit a6b2aecf3a75e4bdef493b645112a7944bced1e1)

[Ihar Hrachyshka (ihar-hrachyshka)]

@Carl, yes, latest simplejson gets new features that should eventually arrive into stdlib json module once another sync is done. We should handle any significant differences that results in unexpected discrepancies as we do when supporting multiple Python versions. For this specific case, if we really need to retain the old behaviour of stdlib json module, and we target simplejson > 2.2, we could enforce it via namedtuple_as_object set to True. Though I would make such modifications only if real issues arise.

Answering to your question ('How many types were tested for equivalence?'), none of them specifically. Tempest runs and tests in the lab were used to ensure no regressions are introduced.

[Carl Baldwin (carl-baldwin)]

The issue came up in a patch that I had in flight here: https://review.openstack.org/#/c/112178. I wrote a unit test to check my assumption and it failed on python2.6 in Jenkins but not locally in my own python2.6 virtualenv. I was perplexed until I updated oslo.messaging and then installed simplejson to match the Jenkins run. Without my new unit test, I never would have caught the issue because there is not enough testing with python2.6 in the gate to catch this.

What I'm reading is that it was assumed that simplejson would be a suitable drop-in replacement for for json without really looking very deep and you'd rather wait for issues to arise. Given the luck involved in catching this issue and reproducing it locally, I'm not sure that is wise. At this point, I'm not going to hold up my work to wait for a fix in oslo messaging and an update to Neutron to consume the updated dependency. I'm going to have to just change direction and find another way.

It should be noted that namedtuples should not be used in any RPC message with the current oslo.messaging as the serialized JSON representation will be platform and environment dependent.

[Ihar Hrachyshka (ihar-hrachyshka)]

@Carl, the issue is fixed in oslo.serialization: https://review.openstack.org/113973 Do you think we need to backport this to oslo-incubator for neutron to consume, or we can wait till it migrates to the new library?

[Carl Baldwin (carl-baldwin)]

Ihar, I'm okay either way. I have a work-around in place.

[Davanum Srinivas (DIMS) (dims-v)]

For Nova, this will be fixed by https://review.openstack.org/#/c/114686/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: stable/havana
Review: https://review.openstack.org/103130
Reason: Not the change that is worth freeze exception.

[Angus Salkeld (asalkeld)]

no longer effects heat as we now have oslo.serialization
https://github.com/openstack/heat/commit/c5bc3c21b9fbdec71ffef344b3a5222c752a99c1

[Ihar Hrachyshka (ihar-hrachyshka)]

oslo.messaging is now using oslo.serialization that includes all the needed patches.