Change in iso8601 0.1.12 date format breaks parsing with py35

Bug #1744160 reported by Sean McGinnis on 2018-01-18
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Undecided
Sean McGinnis
Glance
Undecided
Deepak Mourya
Manila
Undecided
junboli
OpenStack Compute (nova)
Undecided
Deepak Mourya
OpenStack Identity (keystone)
Undecided
ChangBo Guo(gcb)
oslo.utils
Medium
John L. Villalovos
oslo.versionedobjects
Medium
Sean McGinnis

Bug Description

New package of iso8601 returns string in the format:

'2012-02-14T20:53:07UTC+00:00'

instead of:

'2012-02-14T20:53:07Z'

This is resulting in date string comparison failures and timeutils.parse_isotime errors with:

ValueError: Unable to parse date string '2014-08-08T00:00:00UTC+00:00'

summary: - Change in iso8601 1.12.0 date format breaks parsing
+ Change in iso8601 1.12.0 date format breaks parsing with py35

Most errors appear to be coming from oslo.utils with the exception of Glance. Glance has its own implementation of timeutils, so it may be the same solution, just directly in the project. (or by getting rid of local duplication and moving to oslo.utils)

Sean McGinnis (sean-mcginnis) wrote :

Traced this through, and seems to be coming from the fact that iso8601 switched from using their own internal TZ info, to using Python3's TZ info. The difference in these objects end up being that the custom iso8601 one stringifies to 'UTC', while the python one stringifies to 'UTC+00:00'.

This causes problems in oslo.versionedobjects to_primative call here:

https://github.com/openstack/oslo.versionedobjects/blob/master/oslo_versionedobjects/_utils.py#L28

Could be simple enough as change "tz == 'UTC'" to something like "'UTC' in tz". I will try that out locally and see how it goes.

Changed in oslo.versionedobjects:
status: New → Confirmed
Sean McGinnis (sean-mcginnis) wrote :

For reference, here is the iso8601 commit that introduced this:

https://bitbucket.org/micktwomey/pyiso8601/commits/a1f771bc36754edb2fd649ab2fbe6b788b6c6149

Fix proposed to branch: master
Review: https://review.openstack.org/535520

Changed in oslo.versionedobjects:
assignee: nobody → Sean McGinnis (sean-mcginnis)
status: Confirmed → In Progress
Changed in cinder:
assignee: nobody → Deepak Mourya (mourya007)
Changed in glance:
assignee: nobody → Deepak Mourya (mourya007)

Fix proposed to branch: master
Review: https://review.openstack.org/535608

Changed in glance:
status: New → In Progress
Changed in cinder:
assignee: Deepak Mourya (mourya007) → nobody
junboli (junboli) on 2018-01-19
Changed in cinder:
assignee: nobody → junboli (junboli)
liuyamin (liuyamin) on 2018-01-19
Changed in oslo.utils:
assignee: nobody → liuyamin (liuyamin)
junboli (junboli) on 2018-01-19
Changed in cinder:
assignee: junboli (junboli) → nobody
Changed in manila:
assignee: nobody → junboli (junboli)
Changed in manila:
status: New → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/535672

Changed in oslo.utils:
status: New → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/535700

Changed in nova:
assignee: nobody → Deepak Mourya (mourya007)
status: New → In Progress

Some notes on testing and checking for this in projects.

You can see if a given project does its own iso formatting copied from oslo by doing:

    grep -r \'UTC\' ./* --include="*.py"

To test if a project is affected by the change, it needs to use the 1.12.0 version that is currently above our cap. To get that, run:

    . .tox/py35/bin/activate
    pip install --upgrade --force iso8601
    deactive
    tox -e py35

Until the oslo.versionedobjects fix is available, some projects will see a lot of unit test failures coming out of there. Simplest way to check if there are other issues until then is to edit the locally installed copy in the venv:

    vi .tox/py35/lib/python3.5/site-packages/oslo_versionedobjects/_utils.py

junboli (junboli) on 2018-01-22
Changed in cinder:
assignee: nobody → junboli (junboli)

Fix proposed to branch: master
Review: https://review.openstack.org/536182

Changed in cinder:
status: New → In Progress
Changed in keystone:
assignee: nobody → Deepak Mourya (mourya007)
Changed in glance:
milestone: none → queens-3
summary: - Change in iso8601 1.12.0 date format breaks parsing with py35
+ Change in iso8601 0.1.12 date format breaks parsing with py35
Changed in oslo.versionedobjects:
importance: Undecided → Medium
Changed in oslo.utils:
importance: Undecided → Medium
ChangBo Guo(gcb) (glongwave) wrote :

we don't bump the constraint in https://github.com/openstack/requirements/blob/master/upper-constraints.txt#L327. so I won't break our gate, right?

Sean McGinnis (sean-mcginnis) wrote :

Correct. These should all be backwards compatible changes to get these projects ready for when we actually do raise the upper-constraint with patch https://review.openstack.org/#/c/503691/

Colleen Murphy (krinkle) on 2018-01-24
Changed in keystone:
milestone: none → queens-rc1
Colleen Murphy (krinkle) wrote :

Not seeing unit test failures for keystone with the upgraded library, but we do do our own isoformatting: http://git.openstack.org/cgit/openstack/keystone/tree/keystone/common/utils.py#n501 so it's possible we're affected and don't have sufficient test coverage for it.

Reviewed: https://review.openstack.org/535520
Committed: https://git.openstack.org/cgit/openstack/oslo.versionedobjects/commit/?id=9c4aefb8ea88fd5505602c95f4762fdeb3aea183
Submitter: Zuul
Branch: master

commit 9c4aefb8ea88fd5505602c95f4762fdeb3aea183
Author: Sean McGinnis <email address hidden>
Date: Thu Jan 18 16:52:03 2018 -0600

    Handle TZ change in iso8601 >=0.1.12

    The iso8601 lib introduced a change such that if running on python
    3.2 or later it internally uses the python timezone information
    instead of its own implementation. This does not change direct
    date handling, but when converting this value there is a slight
    difference where now python 2.x will show UTC times as "UTC", but
    on python 3 they will end up with "UTC+00:00".

    The to_primitive call for DateTime fields was doing an exact match
    on "UTC" to determine whether to include "Z" in the resulting string.
    This updates that handling to recognize either of the new values.

    Change-Id: I71b58e8fd8fee8a57ee275ff3e0b77f165eca836
    Closes-bug: #1744160

Changed in oslo.versionedobjects:
status: In Progress → Fix Released

Change abandoned by Matthew Thode (<email address hidden>) on branch: master
Review: https://review.openstack.org/538035

Fix proposed to branch: master
Review: https://review.openstack.org/538263

Changed in keystone:
assignee: Deepak Mourya (mourya007) → ChangBo Guo(gcb) (glongwave)
status: New → In Progress

Reviewed: https://review.openstack.org/538036
Committed: https://git.openstack.org/cgit/openstack/oslo.versionedobjects/commit/?id=c95f0c876840e36f37acb14d5eec5238d85e7dce
Submitter: Zuul
Branch: stable/queens

commit c95f0c876840e36f37acb14d5eec5238d85e7dce
Author: Sean McGinnis <email address hidden>
Date: Thu Jan 18 16:52:03 2018 -0600

    Handle TZ change in iso8601 >=0.1.12

    The iso8601 lib introduced a change such that if running on python
    3.2 or later it internally uses the python timezone information
    instead of its own implementation. This does not change direct
    date handling, but when converting this value there is a slight
    difference where now python 2.x will show UTC times as "UTC", but
    on python 3 they will end up with "UTC+00:00".

    The to_primitive call for DateTime fields was doing an exact match
    on "UTC" to determine whether to include "Z" in the resulting string.
    This updates that handling to recognize either of the new values.

    Change-Id: Iff2e5a5b056605fae59f2489cc7baa1fc2e3352f
    Closes-bug: #1744160
    (cherry picked from commit 9c4aefb8ea88fd5505602c95f4762fdeb3aea183)
    Signed-off-by: Matthew Thode <email address hidden>

tags: added: in-stable-queens

This issue was fixed in the openstack/oslo.versionedobjects 1.31.2 release.

Changed in cinder:
assignee: junboli (junboli) → Sean McGinnis (sean-mcginnis)

Reviewed: https://review.openstack.org/535700
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=11222bbf777a9418b9262bf56a61e5f3aa5f6501
Submitter: Zuul
Branch: master

commit 11222bbf777a9418b9262bf56a61e5f3aa5f6501
Author: deepak_mourya <email address hidden>
Date: Fri Jan 19 15:36:18 2018 +0530

    Handle TZ change in iso8601 >=0.1.12

    The iso8601 lib introduced a change such that if running on python
    3.2 or later it internally uses the python timezone information
    instead of its own implementation. This does not change direct
    date handling, but when converting this value there is a slight
    difference where now python 2.x will show UTC times as "UTC",
    but on python 3 they will end up with "UTC+00:00".

    The to_primitive call for DateTime fields was doing an exact match
    on "UTC" to determine whether to include "Z" in the resulting string.
    This updates that handling to recognize either of the new values

    Change-Id: I426cf42ddcf6e8aa2d43f286eb76908670cc8d16
    Closes-bug: #1744160

Changed in nova:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/536182
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=d24963f7e84b519864ece4fce99f9b0da0204da4
Submitter: Zuul
Branch: master

commit d24963f7e84b519864ece4fce99f9b0da0204da4
Author: junboli <email address hidden>
Date: Mon Jan 22 09:28:42 2018 +0800

    Handle TZ change in iso8601 >=0.1.12

    The iso8601 lib introduced a change such that if running on python
    3.2 or later it internally uses the python timezone information
    instead of its own implementation. This does not change direct
    date handling, but when converting this value there is a slight
    difference where now python 2.x will show UTC times as "UTC", but
    on python 3 they will end up with "UTC+00:00".

    The to_primitive call for DateTime fields was doing an exact match
    on "UTC" to determine whether to include "Z" in the resulting string.
    This updates that handling to recognize either of the new values.

    Change-Id: Idfefd41e45727a375a5ea296a3348716c43f17b5
    Closes-bug: #1744160

Changed in cinder:
status: In Progress → Fix Released
Changed in glance:
milestone: queens-3 → queens-rc1

Reviewed: https://review.openstack.org/538263
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=ace2e1088cda4ae4dfde93a8ffdafde2d33262cb
Submitter: Zuul
Branch: master

commit ace2e1088cda4ae4dfde93a8ffdafde2d33262cb
Author: ChangBo Guo(gcb) <email address hidden>
Date: Fri Jan 26 23:03:39 2018 +0800

    Handle TZ change in iso8601 >=0.1.12

    The iso8601 lib introduced a change such that if running on python
    3.2 or later it internally uses the python timezone information
    instead of its own implementation. This does not change direct
    date handling, but when converting this value there is a slight
    difference where now python 2.x will show UTC times as "UTC",
    but on python 3 they will end up with "UTC+00:00".

    The to_primitive call for DateTime fields was doing an exact match
    on "UTC" to determine whether to include "Z" in the resulting string.
    This updates that handling to recognize either of the new values

    Closes-bug: #1744160

    Change-Id: I505434facc7adc4a479f67eeedb31cf7b4bf7caf

Changed in keystone:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/535608
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=daa3c88d8b824207577cfd5a17ee54edc786bba2
Submitter: Zuul
Branch: master

commit daa3c88d8b824207577cfd5a17ee54edc786bba2
Author: deepak_mourya <email address hidden>
Date: Fri Jan 19 11:14:32 2018 +0530

    Handle TZ change in iso8601 >=0.1.12

    The iso8601 lib introduced a change such that if running on python
    3.2 or later it internally uses the python timezone information
    instead of its own implementation. This does not change direct
    date handling, but when converting this value there is a slight
    difference where now python 2.x will show UTC times as "UTC",
    but on python 3 they will end up with "UTC+00:00".

    The to_primitive call for DateTime fields was doing an exact match
    on "UTC" to determine whether to include "Z" in the resulting string.
    This updates that handling to recognize either of the new values

    Change-Id: Icb29d71472932f3ddfe485298d1a5fdd08be6f12
    Closes-bug: #1744160

Changed in glance:
status: In Progress → Fix Released
Changed in manila:
milestone: none → queens-rc1

Reviewed: https://review.openstack.org/535611
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=e10c4b2f5df78cbca02d34dd5cff0058142c37e6
Submitter: Zuul
Branch: master

commit e10c4b2f5df78cbca02d34dd5cff0058142c37e6
Author: junboli <email address hidden>
Date: Fri Jan 19 18:53:21 2018 +0800

    Handle TZ change in iso8601 >=0.1.12

    The iso8601 lib introduced a change such that if running on
    python 3.2 or later it internally uses the python timezone
    information instead of its own implementation.
    This does not change direct date handling, but when converting
    this value there is a slight difference where now python 2.x
    will show UTC times as "UTC", but on python 3 they will end up
    with "UTC+00:00".
    The to_primitive call for DateTime fields was doing an exact match
    on "UTC" to determine whether to include "Z" in the resulting string.
    This updates that handling to recognize either of the new values.

    Closes-bug: #1744160
    Change-Id: I9fcc55b36178ff88795e1d8e14da349e338a7392

Changed in manila:
status: In Progress → Fix Released
Changed in oslo.utils:
assignee: liuyamin (liuyamin) → John L. Villalovos (happycamp)

Change abandoned by liuyamin (<email address hidden>) on branch: master
Review: https://review.openstack.org/535672

Reviewed: https://review.openstack.org/541142
Committed: https://git.openstack.org/cgit/openstack/oslo.utils/commit/?id=010fe3b1023871740b57dbc450f80e6c0c0f6e43
Submitter: Zuul
Branch: master

commit 010fe3b1023871740b57dbc450f80e6c0c0f6e43
Author: John L. Villalovos <email address hidden>
Date: Mon Feb 5 22:29:38 2018 -0800

    Fix breaking unit tests due to iso8601 changes

    The move from iso8601===0.1.11 to iso8601===0.1.12 broke unit
    tests in oslo.utils.

    iso8601 used to do:
        from datetime import datetime

    But now they call datetime.datetime():
        import datetime
        datetime.datetime()

    Unfortunately the unit tests that mocked datetime.datetime() are now
    mocking the one in iso8601. This causes a failure in the unit tests.

    Fix this by using the 'wraps' argument to mock. So that the calls will
    get passed through to datetime.datetime. Also changed to using the
    decorator style of mock.

    In addition Python 3 unit tests were broken due to changing how the
    UTC time zone is represented from 'UTC' to 'UTC+00:00'.

    Closes-Bug: #1747575
    Closes-Bug: #1744160
    Change-Id: Ia80ffb5e571cc5366bef2bc1a32c457a3c16843d

Changed in oslo.utils:
status: In Progress → Fix Released

This issue was fixed in the openstack/cinder 12.0.0.0rc1 release candidate.

This issue was fixed in the openstack/glance 16.0.0.0rc1 release candidate.

This issue was fixed in the openstack/manila 6.0.0.0rc1 release candidate.

This issue was fixed in the openstack/nova 17.0.0.0rc1 release candidate.

Reviewed: https://review.openstack.org/541543
Committed: https://git.openstack.org/cgit/openstack/oslo.utils/commit/?id=1e0570b2a598b43a85faa28e07ab97d56bf5ba5e
Submitter: Zuul
Branch: stable/queens

commit 1e0570b2a598b43a85faa28e07ab97d56bf5ba5e
Author: John L. Villalovos <email address hidden>
Date: Mon Feb 5 22:29:38 2018 -0800

    Fix breaking unit tests due to iso8601 changes

    The move from iso8601===0.1.11 to iso8601===0.1.12 broke unit
    tests in oslo.utils.

    iso8601 used to do:
        from datetime import datetime

    But now they call datetime.datetime():
        import datetime
        datetime.datetime()

    Unfortunately the unit tests that mocked datetime.datetime() are now
    mocking the one in iso8601. This causes a failure in the unit tests.

    Fix this by using the 'wraps' argument to mock. So that the calls will
    get passed through to datetime.datetime. Also changed to using the
    decorator style of mock.

    In addition Python 3 unit tests were broken due to changing how the
    UTC time zone is represented from 'UTC' to 'UTC+00:00'.

    Closes-Bug: #1747575
    Closes-Bug: #1744160
    Change-Id: Ia80ffb5e571cc5366bef2bc1a32c457a3c16843d
    (cherry picked from commit 010fe3b1023871740b57dbc450f80e6c0c0f6e43)

This issue was fixed in the openstack/keystone 13.0.0.0rc1 release candidate.

This issue was fixed in the openstack/oslo.utils 3.36.0 release.

This issue was fixed in the openstack/oslo.versionedobjects 1.32.0 release.

This issue was fixed in the openstack/oslo.utils 3.35.1 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers