Change in iso8601 0.1.12 date format breaks parsing with py35

[Sean McGinnis]

New package of iso8601 returns string in the format:

'2012-02-14T20:53:07UTC+00:00'

instead of:

'2012-02-14T20:53:07Z'

This is resulting in date string comparison failures and timeutils.parse_isotime errors with:

ValueError: Unable to parse date string '2014-08-08T00:00:00UTC+00:00'

[Sean McGinnis (sean-mcginnis)]

Most errors appear to be coming from oslo.utils with the exception of Glance. Glance has its own implementation of timeutils, so it may be the same solution, just directly in the project. (or by getting rid of local duplication and moving to oslo.utils)

[Sean McGinnis (sean-mcginnis)]

Traced this through, and seems to be coming from the fact that iso8601 switched from using their own internal TZ info, to using Python3's TZ info. The difference in these objects end up being that the custom iso8601 one stringifies to 'UTC', while the python one stringifies to 'UTC+00:00'.

This causes problems in oslo.versionedobjects to_primative call here:

https://github.com/openstack/oslo.versionedobjects/blob/master/oslo_versionedobjects/_utils.py#L28

Could be simple enough as change "tz == 'UTC'" to something like "'UTC' in tz". I will try that out locally and see how it goes.

[Sean McGinnis (sean-mcginnis)]

For reference, here is the iso8601 commit that introduced this:

https://bitbucket.org/micktwomey/pyiso8601/commits/a1f771bc36754edb2fd649ab2fbe6b788b6c6149

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/535520

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/535608

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/535672

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/535700

[Sean McGinnis (sean-mcginnis)]

Some notes on testing and checking for this in projects.

You can see if a given project does its own iso formatting copied from oslo by doing:

    grep -r \'UTC\' ./* --include="*.py"

To test if a project is affected by the change, it needs to use the 1.12.0 version that is currently above our cap. To get that, run:

    . .tox/py35/bin/activate
    pip install --upgrade --force iso8601
    deactive
    tox -e py35

Until the oslo.versionedobjects fix is available, some projects will see a lot of unit test failures coming out of there. Simplest way to check if there are other issues until then is to edit the locally installed copy in the venv:

    vi .tox/py35/lib/python3.5/site-packages/oslo_versionedobjects/_utils.py

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/536182

[ChangBo Guo(gcb) (glongwave)]

we don't bump the constraint in https://github.com/openstack/requirements/blob/master/upper-constraints.txt#L327. so I won't break our gate, right?

[Sean McGinnis (sean-mcginnis)]

Correct. These should all be backwards compatible changes to get these projects ready for when we actually do raise the upper-constraint with patch https://review.openstack.org/#/c/503691/

[Colleen Murphy (krinkle)]

Not seeing unit test failures for keystone with the upgraded library, but we do do our own isoformatting: http://git.openstack.org/cgit/openstack/keystone/tree/keystone/common/utils.py#n501 so it's possible we're affected and don't have sufficient test coverage for it.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/535520
Committed: https://git.openstack.org/cgit/openstack/oslo.versionedobjects/commit/?id=9c4aefb8ea88fd5505602c95f4762fdeb3aea183
Submitter: Zuul
Branch: master

commit 9c4aefb8ea88fd5505602c95f4762fdeb3aea183
Author: Sean McGinnis <email address hidden>
Date: Thu Jan 18 16:52:03 2018 -0600

    Handle TZ change in iso8601 >=0.1.12

    The iso8601 lib introduced a change such that if running on python
    3.2 or later it internally uses the python timezone information
    instead of its own implementation. This does not change direct
    date handling, but when converting this value there is a slight
    difference where now python 2.x will show UTC times as "UTC", but
    on python 3 they will end up with "UTC+00:00".

    The to_primitive call for DateTime fields was doing an exact match
    on "UTC" to determine whether to include "Z" in the resulting string.
    This updates that handling to recognize either of the new values.

    Change-Id: I71b58e8fd8fee8a57ee275ff3e0b77f165eca836
    Closes-bug: #1744160

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/538035

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matthew Thode (<email address hidden>) on branch: master
Review: https://review.openstack.org/538035

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/538036

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/538263

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/538036
Committed: https://git.openstack.org/cgit/openstack/oslo.versionedobjects/commit/?id=c95f0c876840e36f37acb14d5eec5238d85e7dce
Submitter: Zuul
Branch: stable/queens

commit c95f0c876840e36f37acb14d5eec5238d85e7dce
Author: Sean McGinnis <email address hidden>
Date: Thu Jan 18 16:52:03 2018 -0600

    Handle TZ change in iso8601 >=0.1.12

    The iso8601 lib introduced a change such that if running on python
    3.2 or later it internally uses the python timezone information
    instead of its own implementation. This does not change direct
    date handling, but when converting this value there is a slight
    difference where now python 2.x will show UTC times as "UTC", but
    on python 3 they will end up with "UTC+00:00".

    The to_primitive call for DateTime fields was doing an exact match
    on "UTC" to determine whether to include "Z" in the resulting string.
    This updates that handling to recognize either of the new values.

    Change-Id: Iff2e5a5b056605fae59f2489cc7baa1fc2e3352f
    Closes-bug: #1744160
    (cherry picked from commit 9c4aefb8ea88fd5505602c95f4762fdeb3aea183)
    Signed-off-by: Matthew Thode <email address hidden>

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/oslo.versionedobjects 1.31.2 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/535700
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=11222bbf777a9418b9262bf56a61e5f3aa5f6501
Submitter: Zuul
Branch: master

commit 11222bbf777a9418b9262bf56a61e5f3aa5f6501
Author: deepak_mourya <email address hidden>
Date: Fri Jan 19 15:36:18 2018 +0530

    Handle TZ change in iso8601 >=0.1.12

    The iso8601 lib introduced a change such that if running on python
    3.2 or later it internally uses the python timezone information
    instead of its own implementation. This does not change direct
    date handling, but when converting this value there is a slight
    difference where now python 2.x will show UTC times as "UTC",
    but on python 3 they will end up with "UTC+00:00".

    The to_primitive call for DateTime fields was doing an exact match
    on "UTC" to determine whether to include "Z" in the resulting string.
    This updates that handling to recognize either of the new values

    Change-Id: I426cf42ddcf6e8aa2d43f286eb76908670cc8d16
    Closes-bug: #1744160

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/536182
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=d24963f7e84b519864ece4fce99f9b0da0204da4
Submitter: Zuul
Branch: master

commit d24963f7e84b519864ece4fce99f9b0da0204da4
Author: junboli <email address hidden>
Date: Mon Jan 22 09:28:42 2018 +0800

    Handle TZ change in iso8601 >=0.1.12

    The iso8601 lib introduced a change such that if running on python
    3.2 or later it internally uses the python timezone information
    instead of its own implementation. This does not change direct
    date handling, but when converting this value there is a slight
    difference where now python 2.x will show UTC times as "UTC", but
    on python 3 they will end up with "UTC+00:00".

    The to_primitive call for DateTime fields was doing an exact match
    on "UTC" to determine whether to include "Z" in the resulting string.
    This updates that handling to recognize either of the new values.

    Change-Id: Idfefd41e45727a375a5ea296a3348716c43f17b5
    Closes-bug: #1744160

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/538263
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=ace2e1088cda4ae4dfde93a8ffdafde2d33262cb
Submitter: Zuul
Branch: master

commit ace2e1088cda4ae4dfde93a8ffdafde2d33262cb
Author: ChangBo Guo(gcb) <email address hidden>
Date: Fri Jan 26 23:03:39 2018 +0800

    Handle TZ change in iso8601 >=0.1.12

    The iso8601 lib introduced a change such that if running on python
    3.2 or later it internally uses the python timezone information
    instead of its own implementation. This does not change direct
    date handling, but when converting this value there is a slight
    difference where now python 2.x will show UTC times as "UTC",
    but on python 3 they will end up with "UTC+00:00".

    The to_primitive call for DateTime fields was doing an exact match
    on "UTC" to determine whether to include "Z" in the resulting string.
    This updates that handling to recognize either of the new values

    Closes-bug: #1744160

    Change-Id: I505434facc7adc4a479f67eeedb31cf7b4bf7caf

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/535608
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=daa3c88d8b824207577cfd5a17ee54edc786bba2
Submitter: Zuul
Branch: master

commit daa3c88d8b824207577cfd5a17ee54edc786bba2
Author: deepak_mourya <email address hidden>
Date: Fri Jan 19 11:14:32 2018 +0530

    Handle TZ change in iso8601 >=0.1.12

    The iso8601 lib introduced a change such that if running on python
    3.2 or later it internally uses the python timezone information
    instead of its own implementation. This does not change direct
    date handling, but when converting this value there is a slight
    difference where now python 2.x will show UTC times as "UTC",
    but on python 3 they will end up with "UTC+00:00".

    The to_primitive call for DateTime fields was doing an exact match
    on "UTC" to determine whether to include "Z" in the resulting string.
    This updates that handling to recognize either of the new values

    Change-Id: Icb29d71472932f3ddfe485298d1a5fdd08be6f12
    Closes-bug: #1744160

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/535611
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=e10c4b2f5df78cbca02d34dd5cff0058142c37e6
Submitter: Zuul
Branch: master

commit e10c4b2f5df78cbca02d34dd5cff0058142c37e6
Author: junboli <email address hidden>
Date: Fri Jan 19 18:53:21 2018 +0800

    Handle TZ change in iso8601 >=0.1.12

    The iso8601 lib introduced a change such that if running on
    python 3.2 or later it internally uses the python timezone
    information instead of its own implementation.
    This does not change direct date handling, but when converting
    this value there is a slight difference where now python 2.x
    will show UTC times as "UTC", but on python 3 they will end up
    with "UTC+00:00".
    The to_primitive call for DateTime fields was doing an exact match
    on "UTC" to determine whether to include "Z" in the resulting string.
    This updates that handling to recognize either of the new values.

    Closes-bug: #1744160
    Change-Id: I9fcc55b36178ff88795e1d8e14da349e338a7392

[OpenStack Infra (hudson-openstack)]

Change abandoned by liuyamin (<email address hidden>) on branch: master
Review: https://review.openstack.org/535672

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/541142
Committed: https://git.openstack.org/cgit/openstack/oslo.utils/commit/?id=010fe3b1023871740b57dbc450f80e6c0c0f6e43
Submitter: Zuul
Branch: master

commit 010fe3b1023871740b57dbc450f80e6c0c0f6e43
Author: John L. Villalovos <email address hidden>
Date: Mon Feb 5 22:29:38 2018 -0800

    Fix breaking unit tests due to iso8601 changes

    The move from iso8601===0.1.11 to iso8601===0.1.12 broke unit
    tests in oslo.utils.

    iso8601 used to do:
        from datetime import datetime

    But now they call datetime.datetime():
        import datetime
        datetime.datetime()

    Unfortunately the unit tests that mocked datetime.datetime() are now
    mocking the one in iso8601. This causes a failure in the unit tests.

    Fix this by using the 'wraps' argument to mock. So that the calls will
    get passed through to datetime.datetime. Also changed to using the
    decorator style of mock.

    In addition Python 3 unit tests were broken due to changing how the
    UTC time zone is represented from 'UTC' to 'UTC+00:00'.

    Closes-Bug: #1747575
    Closes-Bug: #1744160
    Change-Id: Ia80ffb5e571cc5366bef2bc1a32c457a3c16843d

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/541543

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 12.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance 16.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 6.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/541543
Committed: https://git.openstack.org/cgit/openstack/oslo.utils/commit/?id=1e0570b2a598b43a85faa28e07ab97d56bf5ba5e
Submitter: Zuul
Branch: stable/queens

commit 1e0570b2a598b43a85faa28e07ab97d56bf5ba5e
Author: John L. Villalovos <email address hidden>
Date: Mon Feb 5 22:29:38 2018 -0800

    Fix breaking unit tests due to iso8601 changes

    The move from iso8601===0.1.11 to iso8601===0.1.12 broke unit
    tests in oslo.utils.

    iso8601 used to do:
        from datetime import datetime

    But now they call datetime.datetime():
        import datetime
        datetime.datetime()

    Unfortunately the unit tests that mocked datetime.datetime() are now
    mocking the one in iso8601. This causes a failure in the unit tests.

    Fix this by using the 'wraps' argument to mock. So that the calls will
    get passed through to datetime.datetime. Also changed to using the
    decorator style of mock.

    In addition Python 3 unit tests were broken due to changing how the
    UTC time zone is represented from 'UTC' to 'UTC+00:00'.

    Closes-Bug: #1747575
    Closes-Bug: #1744160
    Change-Id: Ia80ffb5e571cc5366bef2bc1a32c457a3c16843d
    (cherry picked from commit 010fe3b1023871740b57dbc450f80e6c0c0f6e43)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystone 13.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/oslo.utils 3.36.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/oslo.versionedobjects 1.32.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/oslo.utils 3.35.1 release.