jsonutils should use simplejson on python 2.6 if available

Bug #1314129 reported by Ihar Hrachyshka on 2014-04-29
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
oslo.messaging
Undecided
Ihar Hrachyshka
taskflow
Undecided
Ihar Hrachyshka
tuskar
Low
Ihar Hrachyshka

Bug Description

Python 2.6 ships 'json' module that is very slow because it's written in pure Python. Python 2.7 updated [1] its 'json' module from simplejson PyPI repo with a version that is based on C extension (and quick). Quoting: "Updated module: The json module was upgraded to version 2.0.9 of the simplejson package, which includes a C extension that makes encoding and decoding faster. (Contributed by Bob Ippolito; issue 4136.)"

We should strive to use simplejson library when running on Python 2.6.

[1]: https://docs.python.org/dev/whatsnew/2.7.html

Added all projects that sync jsonutils from oslo-incubator to the list of affected projects.

For some reason, Gerrit didn't post the comment with the link to review here, so here it is: https://review.openstack.org/#/c/90968/

Dolph Mathews (dolph) on 2014-04-29
Changed in keystone:
importance: Undecided → Low
Changed in python-keystoneclient:
importance: Undecided → Low
Changed in oslo:
assignee: nobody → Ihar Hrachyshka (ihar-hrachyshka)
Changed in oslo:
importance: Undecided → Low
status: New → Triaged

Reviewed: https://review.openstack.org/90968
Committed: https://git.openstack.org/cgit/openstack/oslo-incubator/commit/?id=a6b2aecf3a75e4bdef493b645112a7944bced1e1
Submitter: Jenkins
Branch: master

commit a6b2aecf3a75e4bdef493b645112a7944bced1e1
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Apr 29 12:03:30 2014 +0200

    On Python <= 2.6, use simplejson if available

    Until Python 2.7, stdlib json module was not boosted with a C extension,
    meaning bad performance. Try to use simplejson module if available.

    RHEL 6.5 is one of the platforms that still ship Python 2.6.

    The original idea of the patch belongs to Miguel Angel Ajo Pelayo @
    <email address hidden>

    Change-Id: Ib3dc0b713ed90396919feba018772243b3b9c90f
    Closes-Bug: 1314129

Changed in oslo:
status: Triaged → Fix Committed
Changed in neutron:
assignee: nobody → Ihar Hrachyshka (ihar-hrachyshka)
Changed in nova:
assignee: nobody → Ihar Hrachyshka (ihar-hrachyshka)
Changed in glance:
assignee: nobody → Ihar Hrachyshka (ihar-hrachyshka)
Changed in ceilometer:
assignee: nobody → Ihar Hrachyshka (ihar-hrachyshka)

Fix proposed to branch: master
Review: https://review.openstack.org/91049

Changed in ceilometer:
status: New → In Progress
Changed in cinder:
assignee: nobody → Ihar Hrachyshka (ihar-hrachyshka)

Fix proposed to branch: master
Review: https://review.openstack.org/91053

Changed in cinder:
status: New → In Progress
Changed in keystone:
assignee: nobody → Ihar Hrachyshka (ihar-hrachyshka)

Fix proposed to branch: master
Review: https://review.openstack.org/91054

Changed in keystone:
status: New → In Progress
Changed in ironic:
assignee: nobody → Ihar Hrachyshka (ihar-hrachyshka)

Fix proposed to branch: master
Review: https://review.openstack.org/91055

Changed in ironic:
status: New → In Progress
Changed in horizon:
assignee: nobody → Ihar Hrachyshka (ihar-hrachyshka)

Fix proposed to branch: master
Review: https://review.openstack.org/91057

Changed in horizon:
status: New → In Progress
Changed in sahara:
assignee: nobody → Ihar Hrachyshka (ihar-hrachyshka)
Changed in trove:
assignee: nobody → Ihar Hrachyshka (ihar-hrachyshka)
Changed in heat:
assignee: nobody → Ihar Hrachyshka (ihar-hrachyshka)
Changed in marconi:
assignee: nobody → Ihar Hrachyshka (ihar-hrachyshka)
Changed in oslo.messaging:
assignee: nobody → Ihar Hrachyshka (ihar-hrachyshka)
Changed in python-keystoneclient:
assignee: nobody → Ihar Hrachyshka (ihar-hrachyshka)
Changed in python-neutronclient:
assignee: nobody → Ihar Hrachyshka (ihar-hrachyshka)
Changed in python-novaclient:
assignee: nobody → Ihar Hrachyshka (ihar-hrachyshka)
Changed in taskflow:
assignee: nobody → Ihar Hrachyshka (ihar-hrachyshka)

Fix proposed to branch: master
Review: https://review.openstack.org/91080

Changed in python-keystoneclient:
status: New → In Progress
Changed in tuskar:
assignee: nobody → Ihar Hrachyshka (ihar-hrachyshka)
Changed in ceilometer:
importance: Undecided → Low
milestone: none → juno-1

Reviewed: https://review.openstack.org/91070
Committed: https://git.openstack.org/cgit/openstack/marconi/commit/?id=a4f1b642fbb683d0534e32af4929c483ce503c2a
Submitter: Jenkins
Branch: master

commit a4f1b642fbb683d0534e32af4929c483ce503c2a
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Apr 29 17:21:29 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that drastically enhances performance on Python
    2.6 with fresh simplejson library installed.

    The latest commit in oslo-incubator:
    - 732bdb6297eb9de81667f7713ebcb1ccc2ee45a7

    Change-Id: Ib3dc0b713ed90396919feba018772243b3b9c90f
    Closes-Bug: 1314129

Changed in marconi:
status: New → Fix Committed
Kurt Griffiths (kgriffs) on 2014-04-29
Changed in marconi:
milestone: none → juno-1

Reviewed: https://review.openstack.org/91077
Committed: https://git.openstack.org/cgit/openstack/tuskar/commit/?id=5cba8de9a06e2adf25f4b11c62a813b389eb1fc2
Submitter: Jenkins
Branch: master

commit 5cba8de9a06e2adf25f4b11c62a813b389eb1fc2
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Apr 29 17:21:52 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that drastically enhances performance on Python
    2.6 with fresh simplejson library installed.

    The latest commit in oslo-incubator:
    - 732bdb6297eb9de81667f7713ebcb1ccc2ee45a7

    Change-Id: Ib3dc0b713ed90396919feba018772243b3b9c90f
    Closes-Bug: 1314129

Changed in tuskar:
status: New → Fix Committed

Reviewed: https://review.openstack.org/91055
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=adb4cac2fba84a4cb4c81bf1dd1403c73d090122
Submitter: Jenkins
Branch: master

commit adb4cac2fba84a4cb4c81bf1dd1403c73d090122
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Apr 29 16:37:34 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that drastically enhances performance on Python
    2.6 with fresh simplejson library installed.

    The latest commit in oslo-incubator:
    - 732bdb6297eb9de81667f7713ebcb1ccc2ee45a7

    Change-Id: Ib3dc0b713ed90396919feba018772243b3b9c90f
    Closes-Bug: 1314129

Changed in ironic:
status: In Progress → Fix Committed
Petr Blaho (pblaho) on 2014-04-30
Changed in tuskar:
importance: Undecided → Low

Reviewed: https://review.openstack.org/91041
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=88f9bbb6e643472f9218ea88e19afc57ffab025b
Submitter: Jenkins
Branch: master

commit 88f9bbb6e643472f9218ea88e19afc57ffab025b
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Apr 29 16:23:46 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that drastically enhances performance on Python
    2.6 with fresh simplejson library installed.

    The latest commit in oslo-incubator:
    - 732bdb6297eb9de81667f7713ebcb1ccc2ee45a7

    Change-Id: Ib3dc0b713ed90396919feba018772243b3b9c90f
    Closes-Bug: 1314129

Changed in neutron:
status: New → Fix Committed
Tracy Jones (tjones-i) on 2014-04-30
tags: added: oslo

Reviewed: https://review.openstack.org/91075
Committed: https://git.openstack.org/cgit/openstack/python-novaclient/commit/?id=8e0e25f6ceac9525cf0fd6fc36e6295874081d5a
Submitter: Jenkins
Branch: master

commit 8e0e25f6ceac9525cf0fd6fc36e6295874081d5a
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Apr 29 17:26:19 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that drastically enhances performance on Python
    2.6 with fresh simplejson library installed.

    The latest commit in oslo-incubator:
    - 732bdb6297eb9de81667f7713ebcb1ccc2ee45a7

    Change-Id: Ib3dc0b713ed90396919feba018772243b3b9c90f
    Closes-Bug: 1314129

Changed in python-novaclient:
status: New → Fix Committed
Kyle Mestery (mestery) on 2014-05-02
Changed in neutron:
importance: Undecided → Low
milestone: none → juno-1
Joshua Harlow (harlowja) on 2014-05-03
Changed in taskflow:
status: New → Fix Committed

Reviewed: https://review.openstack.org/91049
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=9e60bcdb6b2955fe1681e3a28fa0ad7d12348c5d
Submitter: Jenkins
Branch: master

commit 9e60bcdb6b2955fe1681e3a28fa0ad7d12348c5d
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Apr 29 16:32:15 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that drastically enhances performance on Python
    2.6 with fresh simplejson library installed.

    The latest commit in oslo-incubator:
    - 732bdb6297eb9de81667f7713ebcb1ccc2ee45a7

    Change-Id: Ib3dc0b713ed90396919feba018772243b3b9c90f
    Closes-Bug: 1314129

Changed in ceilometer:
status: In Progress → Fix Committed

Several unicode related issues were revealed by actual syncs from oslo-incubator to affected projects.

https://review.openstack.org/91044: nova expects json.loads() to always return unicode strings.
https://review.openstack.org/91068: trove expects json.loads() to always return unicode strings.
https://review.openstack.org/91047: glance expected json.load() to always return unicode strings.

Simplejson library applies some optimisations that does not guarantee unicode strings if bytes ASCII-only input is parsed. That's why we need to pass unicode strings and file objects to underlying json implementation. For this, another oslo-incubator patch was sent to make sure json.load() and json.loads() always return unicode strings inside json dictionaries: https://review.openstack.org/91344

After discussion with Doug in IRC and Gerrit, we've come to the conclusion that we can't just apply codecs.getreader('utf-8') to fp argument of jsonutils.load() since this may fail for other ASCII-based encodings other than UTF-8. As per official json module description at https://docs.python.org/2/library/json.html#json.load

"If the contents of fp are encoded with an ASCII based encoding other than UTF-8 (e.g. latin-1), then an appropriate encoding name must be specified. Encodings that are not ASCII based (such as UCS-2) are not allowed, and should be wrapped with codecs.getreader(encoding)(fp), or simply decoded to a unicode object and passed to loads()."

Also as per JSONDecoder.__init__() in official json.decoder module,

"``encoding`` determines the encoding used to interpret any ``str`` objects decoded by this instance (utf-8 by default). It has no effect when decoding ``unicode`` objects."

So to make sure we're on par with stdlib json implementation, we need to support encoding argument to allow non UTF-8 encodings, and assume we're passed a 'utf-8' file object that is safe to codecs.getreader('utf-8')(fp) otherwise.

Changed in trove:
importance: Undecided → Low
importance: Low → Wishlist
status: New → Triaged

Reviewed: https://review.openstack.org/91344
Committed: https://git.openstack.org/cgit/openstack/oslo-incubator/commit/?id=18f2bc1bf080b41a22c70842e7c127da21c63b8b
Submitter: Jenkins
Branch: master

commit 18f2bc1bf080b41a22c70842e7c127da21c63b8b
Author: Ihar Hrachyshka <email address hidden>
Date: Wed Apr 30 14:53:15 2014 +0200

    Enforce unicode json output for jsonutils.load[s]()

    simplejson module applies some optimizations on ASCII-only unicode
    strings which result in non-unicode json output. See details at [1]
    and [2]. To make sure we always return consistent json output no
    matter which json implementation is used, we should explicitly convert
    the input for json.load[s]() to unicode.

    If user wants to pass a file object of non UTF-8 encoding to
    json.load[s](), she must also specify this encoding as an argument. To
    support this scenario too, we've added 'encoding' argument to
    jsonutils.load[s]() implementation.

    Made all present JSON tests to run on both supported json library
    implementations.

    Added explicit dependency for simplejson to be able to test different
    implementations in unit tests. Distributors still running Python 2.6
    are recommended but not required to install simplejson.

    Related-Bug: 1314129

    [1]: https://code.djangoproject.com/ticket/11742
    [2]: https://code.google.com/p/simplejson/issues/detail?id=40

    Change-Id: Ic815ca3df94c33edec9104172048b2cd94b92e3f

Fix proposed to branch: master
Review: https://review.openstack.org/95691

Changed in nova:
status: New → In Progress
Changed in sahara:
status: New → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/95692

Changed in trove:
status: Triaged → In Progress
Changed in heat:
status: New → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/95695

Changed in oslo.messaging:
status: New → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/95697

Changed in python-neutronclient:
status: New → In Progress
Changed in sahara:
importance: Undecided → Medium
milestone: none → juno-1
Zhi Yan Liu (lzy-dev) wrote :

https://review.openstack.org/#/c/91047/ change addressed Glance part.

Changed in glance:
importance: Undecided → Low
status: New → In Progress
milestone: none → juno-1

Reviewed: https://review.openstack.org/95695
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2dcdd3bdb74b808afccb8278b0cca52c76f99d1a
Submitter: Jenkins
Branch: master

commit 2dcdd3bdb74b808afccb8278b0cca52c76f99d1a
Author: Ihar Hrachyshka <email address hidden>
Date: Tue May 27 10:22:49 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that makes sure we get unicode-only dicts from
    jsonutils no matter which json module implementation is selected.

    The latest commit in oslo-incubator:
    - 0f4586c0076183c6356eec682c8a593648125abd

    The sync adds a new 'strutils' module that is now used in jsonutils.

    Change-Id: Ic815ca3df94c33edec9104172048b2cd94b92e3f
    Closes-Bug: 1314129

Akihiro Motoki (amotoki) on 2014-05-28
Changed in horizon:
milestone: none → juno-1
importance: Undecided → Low

Reviewed: https://review.openstack.org/91057
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=628470a3aca301db6c4a54adf3ce862381b9adf1
Submitter: Jenkins
Branch: master

commit 628470a3aca301db6c4a54adf3ce862381b9adf1
Author: Ihar Hrachyshka <email address hidden>
Date: Tue May 27 10:24:07 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that drastically enhances performance on Python
    2.6 with fresh simplejson library installed.

    The latest commit in oslo-incubator:
    - 0f4586c0076183c6356eec682c8a593648125abd

    The sync adds a new 'strutils' module that is now used in jsonutils.

    Change-Id: Ib3dc0b713ed90396919feba018772243b3b9c90f
    Closes-Bug: 1314129

Changed in horizon:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/92226
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=185e4562df47a101cf41d1e66d75de2644c78022
Submitter: Jenkins
Branch: master

commit 185e4562df47a101cf41d1e66d75de2644c78022
Author: pran1990 <email address hidden>
Date: Mon May 5 15:31:30 2014 -0700

    Sync oslo network utils

    A new function was added to oslo-incubator for setting tcp keepalive
    settings, and dependent projects can use it.

    Ran update.py in oslo-incubator to copy changed modules into nova.
    Removing synced changes for jsonutils because of issues with how
    strings are encoded with simplejson that cause some of our unittests
    to fail.

    Changes are:
    network_utils.py (which contains the function we need, others are deps)
    edd73c9 Merge "Improve help strings"
    fe3389e Improve help strings
    8a0f567 Remove str() from LOG.* and exceptions
    f61a4e7 Fix python26 compatibility for RFCSysLogHandler
    da6d713 Revert setting oslo-incubator logs to INFO
    0d18381 Set default log levels for oslo.messaging and oslo-incubator

    gettextutils.py
    9912e5d Add API for creating translation functions
    6cc96d0 Fix test_gettextutils on Python 3

    log.py
    edd73c9 Merge "Improve help strings"
    fe3389e Improve help strings
    8a0f567 Remove str() from LOG.* and exceptions
    f61a4e7 Fix python26 compatibility for RFCSysLogHandler
    da6d713 Revert setting oslo-incubator logs to INFO
    0d18381 Set default log levels for oslo.messaging and oslo-incubator

    Current HEAD in OSLO:
    ----------------------------------------------
    Merge: 2640847 e433899
    Date: Mon May 5 19:41:28 2014 +0000
    Merge "Enable configuring tcp keepalive"
    ----------------------------------------------

    Implements: blueprint configure-tcp-keepalive
    Related-Bug: 1314129
    Change-Id: Idd04ad4ad9a06ef76941dc5c1be41ac3ac8877a7

Reviewed: https://review.openstack.org/95691
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=e5dd3b6ab06789774b58c3b7508ed3ad1219249b
Submitter: Jenkins
Branch: master

commit e5dd3b6ab06789774b58c3b7508ed3ad1219249b
Author: Ihar Hrachyshka <email address hidden>
Date: Tue May 27 10:22:28 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that makes sure we get unicode-only dicts from
    jsonutils no matter which json module implementation is selected.

    The latest commit in oslo-incubator:
    - 0f4586c0076183c6356eec682c8a593648125abd

    Change-Id: Ic815ca3df94c33edec9104172048b2cd94b92e3f
    Closes-Bug: 1314129

Reviewed: https://review.openstack.org/95845
Committed: https://git.openstack.org/cgit/openstack/python-keystoneclient/commit/?id=54cc999a4def0bcf6faef4da9bc2f7caf131044f
Submitter: Jenkins
Branch: master

commit 54cc999a4def0bcf6faef4da9bc2f7caf131044f
Author: Steve Martinelli <email address hidden>
Date: Tue May 27 13:53:58 2014 -0400

    Sync with oslo-incubator caed79d

    This syncs python-keystoneclient with oslo-incubator commit hash
    caed79d8239679cb74476bb0d9e5011b4fcc39da

    First, remove the existing code to cleanup:
     $ rm -r keystoneclient/openstack/*

    Then, sync from oslo-incubator:
     $ python update.py ../python-keystoneclient

    Commits since last sync (2640847):
    ----------------------------------
    1173e46 Remove ValueError when accessing sys.modules
    90ae24b Remove redundant default=None for config options
    18f2bc1 Enforce unicode json output for jsonutils.load[s]()
    4a777e5 Fix warnings in doc build for apiclient

    Related-Bug: #1315523
    Closes-Bug: #1314129
    Change-Id: Id53ea2d4828e81bc6270f186849f391e7bc73101

Changed in python-keystoneclient:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/95694
Committed: https://git.openstack.org/cgit/openstack/marconi/commit/?id=7c1de0e4d8c9e5b8d5134cf27868a2c0f8c1bc04
Submitter: Jenkins
Branch: master

commit 7c1de0e4d8c9e5b8d5134cf27868a2c0f8c1bc04
Author: Ihar Hrachyshka <email address hidden>
Date: Tue May 27 10:22:49 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that makes sure we get unicode-only dicts from
    jsonutils no matter which json module implementation is selected.

    The latest commit in oslo-incubator:
    - 0f4586c0076183c6356eec682c8a593648125abd

    The sync adds a new 'strutils' module that is now used in jsonutils.

    Change-Id: Ic815ca3df94c33edec9104172048b2cd94b92e3f
    Closes-Bug: 1314129

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: master
Review: https://review.openstack.org/91054
Reason: Superseded by full sync.

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: master
Review: https://review.openstack.org/95697
Reason: Another sync already merged.

Changed in keystone:
status: In Progress → Fix Committed
status: Fix Committed → In Progress
Dolph Mathews (dolph) on 2014-05-29
Changed in python-keystoneclient:
milestone: none → 0.9.0
Dolph Mathews (dolph) wrote :

Removed python-keystoneclient from this bug due to launchpad issues - fix released in python-keystoneclient 0.9.0.

no longer affects: python-keystoneclient
Changed in keystone:
status: In Progress → Fix Committed
Changed in sahara:
status: In Progress → Fix Committed
Dirk Mueller (dmllr) wrote :

fwiw this causes bug 1325235 for me.

Reviewed: https://review.openstack.org/95699
Committed: https://git.openstack.org/cgit/openstack/taskflow/commit/?id=e60ce2db7ee8225534aa15c6470cde284011366d
Submitter: Jenkins
Branch: master

commit e60ce2db7ee8225534aa15c6470cde284011366d
Author: Ihar Hrachyshka <email address hidden>
Date: Tue May 27 10:22:49 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that makes sure we get unicode-only dicts from
    jsonutils no matter which json module implementation is selected.

    The latest commit in oslo-incubator:
    - 0f4586c0076183c6356eec682c8a593648125abd

    The sync adds a new 'strutils' module that is now used in jsonutils.

    Change-Id: Ic815ca3df94c33edec9104172048b2cd94b92e3f
    Closes-Bug: 1314129

Reviewed: https://review.openstack.org/95698
Committed: https://git.openstack.org/cgit/openstack/python-novaclient/commit/?id=1e40b41a9e3bdbfbf485ddbace55d47e78e66a1b
Submitter: Jenkins
Branch: master

commit 1e40b41a9e3bdbfbf485ddbace55d47e78e66a1b
Author: Ihar Hrachyshka <email address hidden>
Date: Tue May 27 10:22:49 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that makes sure we get unicode-only dicts from
    jsonutils no matter which json module implementation is selected.

    The latest commit in oslo-incubator:
    - 0f4586c0076183c6356eec682c8a593648125abd

    Change-Id: Ic815ca3df94c33edec9104172048b2cd94b92e3f
    Closes-Bug: 1314129

Changed in glance:
milestone: juno-1 → juno-2
Thierry Carrez (ttx) on 2014-06-11
Changed in sahara:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2014-06-11
Changed in horizon:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2014-06-11
Changed in keystone:
milestone: none → juno-1
status: Fix Committed → Fix Released

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: stable/havana
Review: https://review.openstack.org/95753
Reason: Havana is for security fixes only now, so abandoning the patch.

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: stable/havana
Review: https://review.openstack.org/91457
Reason: Havana is for security fixes only now, so abandoning the patch.

Thierry Carrez (ttx) on 2014-06-12
Changed in ceilometer:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2014-06-12
Changed in neutron:
status: Fix Committed → Fix Released
Changed in oslo:
milestone: none → juno-1
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2014-06-12
Changed in marconi:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2014-06-12
Changed in ironic:
status: Fix Committed → Fix Released
milestone: none → juno-1

Reviewed: https://review.openstack.org/95690
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=fe3e0f171895ad4b4344d6cd24aa624f0e911621
Submitter: Jenkins
Branch: master

commit fe3e0f171895ad4b4344d6cd24aa624f0e911621
Author: Ihar Hrachyshka <email address hidden>
Date: Tue May 27 10:22:28 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that makes sure we get unicode-only dicts from
    jsonutils no matter which json module implementation is selected.

    The latest commit in oslo-incubator:
    - 0f4586c0076183c6356eec682c8a593648125abd

    Change-Id: Ic815ca3df94c33edec9104172048b2cd94b92e3f
    Closes-Bug: 1314129

Reviewed: https://review.openstack.org/95700
Committed: https://git.openstack.org/cgit/openstack/tuskar/commit/?id=41d6350e5c92f42ed7fc3cc137fd6267f9d6c761
Submitter: Jenkins
Branch: master

commit 41d6350e5c92f42ed7fc3cc137fd6267f9d6c761
Author: Ihar Hrachyshka <email address hidden>
Date: Tue May 27 10:22:49 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that makes sure we get unicode-only dicts from
    jsonutils no matter which json module implementation is selected.

    The latest commit in oslo-incubator:
    - 0f4586c0076183c6356eec682c8a593648125abd

    The sync adds a new 'strutils' module that is now used in jsonutils.

    Change-Id: Ic815ca3df94c33edec9104172048b2cd94b92e3f
    Closes-Bug: 1314129

Reviewed: https://review.openstack.org/91074
Committed: https://git.openstack.org/cgit/openstack/python-neutronclient/commit/?id=44d45203e828b4812638da970dc9acb9ff4065d9
Submitter: Jenkins
Branch: master

commit 44d45203e828b4812638da970dc9acb9ff4065d9
Author: Ihar Hrachyshka <email address hidden>
Date: Tue May 27 10:22:49 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that drastically enhances performance on Python
    2.6 with fresh simplejson library installed.

    The latest commit in oslo-incubator:
    - 0f4586c0076183c6356eec682c8a593648125abd

    Change-Id: Ib3dc0b713ed90396919feba018772243b3b9c90f
    Closes-Bug: 1314129

Changed in python-neutronclient:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/91454
Committed: https://git.openstack.org/cgit/openstack/oslo-incubator/commit/?id=d14f56f02a289a9d1f6ac9a9c43adbc32fd569d6
Submitter: Jenkins
Branch: stable/icehouse

commit d14f56f02a289a9d1f6ac9a9c43adbc32fd569d6
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Apr 29 12:03:30 2014 +0200

    On Python <= 2.6, use simplejson if available

    Until Python 2.7, stdlib json module was not boosted with a C extension,
    meaning bad performance. Try to use simplejson module if available.

    RHEL 6.5 is one of the platforms that still ship Python 2.6.

    The original idea of the patch belongs to Miguel Angel Ajo Pelayo @
    <email address hidden>

    Change-Id: Ib3dc0b713ed90396919feba018772243b3b9c90f
    Closes-Bug: 1314129
    (cherry picked from commit a6b2aecf3a75e4bdef493b645112a7944bced1e1)

tags: added: in-stable-icehouse

Reviewed: https://review.openstack.org/95749
Committed: https://git.openstack.org/cgit/openstack/oslo-incubator/commit/?id=1b9018859fa3b420d01ed191b916476db6eb9182
Submitter: Jenkins
Branch: stable/icehouse

commit 1b9018859fa3b420d01ed191b916476db6eb9182
Author: Ihar Hrachyshka <email address hidden>
Date: Wed Apr 30 14:53:15 2014 +0200

    Enforce unicode json output for jsonutils.load[s]()

    simplejson module applies some optimizations on ASCII-only unicode
    strings which result in non-unicode json output. See details at [1]
    and [2]. To make sure we always return consistent json output no
    matter which json implementation is used, we should explicitly convert
    the input for json.load[s]() to unicode.

    If user wants to pass a file object of non UTF-8 encoding to
    json.load[s](), she must also specify this encoding as an argument. To
    support this scenario too, we've added 'encoding' argument to
    jsonutils.load[s]() implementation.

    Made all present JSON tests to run on both supported json library
    implementations.

    Added explicit dependency for simplejson to be able to test different
    implementations in unit tests. Distributors still running Python 2.6
    are recommended but not required to install simplejson.

    Related-Bug: 1314129

    [1]: https://code.djangoproject.com/ticket/11742
    [2]: https://code.google.com/p/simplejson/issues/detail?id=40

    Conflicts:
     tests/unit/test_jsonutils.py

    Change-Id: Ic815ca3df94c33edec9104172048b2cd94b92e3f
    (cherry picked from commit 18f2bc1bf080b41a22c70842e7c127da21c63b8b)

Change abandoned by Steve Baker (<email address hidden>) on branch: master
Review: https://review.openstack.org/95693
Reason: should be fixed by full oslo merge

Reviewed: https://review.openstack.org/102036
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8131a2eca838d1cb584646615e48fc11d8a90571
Submitter: Jenkins
Branch: stable/icehouse

commit 8131a2eca838d1cb584646615e48fc11d8a90571
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Jun 24 00:13:17 2014 +0200

    Synced jsonutils from oslo-incubator

    The sync includes change that drastically enhances performance on Python
    2.6 with fresh simplejson library installed.

    Included commits:
    * jsonutils.py:
      - 1b901885: Enforce unicode json output for jsonutils.load[s]()
      - cd909d83: jsonutils.load() accepts file pointer, not string
      - d14f56f0: On Python <= 2.6, use simplejson if available

    This change also introduces new incubator module 'strutils' that is used
    by jsonutils now.

    Change-Id: Ief94101488214ddf93c3547a9c50134826b9b165
    Closes-Bug: 1314129

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: stable/havana
Review: https://review.openstack.org/103130

Michael Still (mikal) on 2014-07-10
Changed in python-novaclient:
milestone: none → 2.18.0
Mark Washenberger (markwash) wrote :

It appears this was fixed in glance in https://review.openstack.org/91047

Changed in glance:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2014-07-29
Changed in glance:
milestone: juno-2 → none
status: Fix Committed → Fix Released
milestone: none → juno-2
Kyle Mestery (mestery) on 2014-07-29
Changed in python-neutronclient:
importance: Undecided → Low
milestone: none → 2.3.5
Marios Andreou (marios-b) wrote :

WRT Tuskar, afaics this was included in v0.4.5 (commit 41d6350e5c92f42ed7fc3cc137fd6267f9d6c761) so setting fix released there (I am here because I ran the tripleo release last week, 0.4.7 for Tuskar, and closing bugs as part of that process)

thanks

Marios Andreou (marios-b) wrote :

Am trying to set this as fix-released for Tuskar ... WRT Tuskar, afaics this was included in v0.4.5 (commit 41d6350e5c92f42ed7fc3cc137fd6267f9d6c761) so setting fix released there (I am here because I ran the tripleo release last week, 0.4.7 for Tuskar, and closing bugs as part of that process)

I can't change to fix-released though, keeps timing out, not sure if this is a permissions issue (do I need to assign to myself first?)

Carl Baldwin (carl-baldwin) wrote :

Here is one example where json and simplejson and regular old json are not the same. How many types were tested for equivalence?

#!/usr/bin/env python

import collections
import simplejson
import json

PortInfo = collections.namedtuple("PortInfo", "mac_address ip_address")

pi = PortInfo('ca:fe:de:ad:be:ef', '192.168.1.1')

print json.dumps(pi)
print simplejson.dumps(pi)

This prints:
["ca:fe:de:ad:be:ef", "192.168.1.1"]
{"ip_address": "192.168.1.1", "mac_address": "ca:fe:de:ad:be:ef"}

Reviewed: https://review.openstack.org/95753
Committed: https://git.openstack.org/cgit/openstack/oslo-incubator/commit/?id=6a290f5dedcc00538c12e7b1e3a2374ea48919c0
Submitter: Jenkins
Branch: stable/havana

commit 6a290f5dedcc00538c12e7b1e3a2374ea48919c0
Author: Ihar Hrachyshka <email address hidden>
Date: Wed Apr 30 14:53:15 2014 +0200

    Enforce unicode json output for jsonutils.load[s]()

    simplejson module applies some optimizations on ASCII-only unicode
    strings which result in non-unicode json output. See details at [1]
    and [2]. To make sure we always return consistent json output no
    matter which json implementation is used, we should explicitly convert
    the input for json.load[s]() to unicode.

    If user wants to pass a file object of non UTF-8 encoding to
    json.load[s](), she must also specify this encoding as an argument. To
    support this scenario too, we've added 'encoding' argument to
    jsonutils.load[s]() implementation.

    Made all present JSON tests to run on both supported json library
    implementations.

    Added explicit dependency for simplejson to be able to test different
    implementations in unit tests. Distributors still running Python 2.6
    are recommended but not required to install simplejson.

    Related-Bug: 1314129

    [1]: https://code.djangoproject.com/ticket/11742
    [2]: https://code.google.com/p/simplejson/issues/detail?id=40

    Conflicts:
     test-requirements-py3.txt
     tests/unit/test_jsonutils.py

    Change-Id: Ic815ca3df94c33edec9104172048b2cd94b92e3f
    (cherry picked from commit 18f2bc1bf080b41a22c70842e7c127da21c63b8b)

tags: added: in-stable-havana

Reviewed: https://review.openstack.org/91457
Committed: https://git.openstack.org/cgit/openstack/oslo-incubator/commit/?id=7e3c27fb50d99c23d35f7e909086ee06416adc1d
Submitter: Jenkins
Branch: stable/havana

commit 7e3c27fb50d99c23d35f7e909086ee06416adc1d
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Apr 29 12:03:30 2014 +0200

    On Python <= 2.6, use simplejson if available

    Until Python 2.7, stdlib json module was not boosted with a C extension,
    meaning bad performance. Try to use simplejson module if available.

    RHEL 6.5 is one of the platforms that still ship Python 2.6.

    The original idea of the patch belongs to Miguel Angel Ajo Pelayo @
    <email address hidden>

    Conflicts:
     openstack/common/jsonutils.py

    Change-Id: Ib3dc0b713ed90396919feba018772243b3b9c90f
    Closes-Bug: 1314129
    (cherry picked from commit a6b2aecf3a75e4bdef493b645112a7944bced1e1)

@Carl, yes, latest simplejson gets new features that should eventually arrive into stdlib json module once another sync is done. We should handle any significant differences that results in unexpected discrepancies as we do when supporting multiple Python versions. For this specific case, if we really need to retain the old behaviour of stdlib json module, and we target simplejson > 2.2, we could enforce it via namedtuple_as_object set to True. Though I would make such modifications only if real issues arise.

Answering to your question ('How many types were tested for equivalence?'), none of them specifically. Tempest runs and tests in the lab were used to ensure no regressions are introduced.

Carl Baldwin (carl-baldwin) wrote :

The issue came up in a patch that I had in flight here: https://review.openstack.org/#/c/112178. I wrote a unit test to check my assumption and it failed on python2.6 in Jenkins but not locally in my own python2.6 virtualenv. I was perplexed until I updated oslo.messaging and then installed simplejson to match the Jenkins run. Without my new unit test, I never would have caught the issue because there is not enough testing with python2.6 in the gate to catch this.

What I'm reading is that it was assumed that simplejson would be a suitable drop-in replacement for for json without really looking very deep and you'd rather wait for issues to arise. Given the luck involved in catching this issue and reproducing it locally, I'm not sure that is wise. At this point, I'm not going to hold up my work to wait for a fix in oslo messaging and an update to Neutron to consume the updated dependency. I'm going to have to just change direction and find another way.

It should be noted that namedtuples should not be used in any RPC message with the current oslo.messaging as the serialized JSON representation will be platform and environment dependent.

@Carl, the issue is fixed in oslo.serialization: https://review.openstack.org/113973 Do you think we need to backport this to oslo-incubator for neutron to consume, or we can wait till it migrates to the new library?

Carl Baldwin (carl-baldwin) wrote :

Ihar, I'm okay either way. I have a work-around in place.

For Nova, this will be fixed by https://review.openstack.org/#/c/114686/

Changed in nova:
importance: Undecided → Low

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: stable/havana
Review: https://review.openstack.org/103130
Reason: Not the change that is worth freeze exception.

Sean Dague (sdague) on 2014-09-18
no longer affects: zaqar
no longer affects: python-neutronclient
no longer affects: trove
no longer affects: python-novaclient
no longer affects: horizon
no longer affects: keystone
no longer affects: glance
no longer affects: ceilometer
no longer affects: cinder
no longer affects: ironic
no longer affects: sahara
no longer affects: oslo-incubator
no longer affects: neutron
no longer affects: nova
Angus Salkeld (asalkeld) wrote :

no longer effects heat as we now have oslo.serialization
https://github.com/openstack/heat/commit/c5bc3c21b9fbdec71ffef344b3a5222c752a99c1

no longer affects: heat
Joshua Harlow (harlowja) on 2014-10-06
Changed in taskflow:
status: Fix Committed → Fix Released

oslo.messaging is now using oslo.serialization that includes all the needed patches.

Changed in oslo.messaging:
status: In Progress → Fix Committed
Mehdi Abaakouk (sileht) on 2014-12-02
Changed in oslo.messaging:
milestone: none → 1.5.0
status: Fix Committed → Fix Released
Jay Dobies (jdob) on 2015-04-10
Changed in tuskar:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.