Zope 2.11.1 DateTime - strftime uses current date/time

[ChrisW]

I recently upgraded a Zope instance from 2.9.3 to 2.11.1.

Some odd behaviour occurs when calling the strftime method on DateTime instances created in 2.9.3 or earlier:
The date that is formatted is always the current date and time, not that stored in the DateTime object.

For me, this has been on Date properties and saving the properties via the manage_propertiesForm seems to fix the issue. But, there's obviously a bug lurking here which could be much more serious for other people...

=============

This problem is caused by the changes in DateTime between Zope 2.10 and 2.11. The DateTime object now calculates and stores a date in micro-seconds and stores that on the DateTime object using an attribute _micros.

The ZODB uses the pickler to load DateTime objects.

First the pickler creates a new DateTime object using a single parameter None, which creates a new DateTime and initialises it to the currrent time. The DateTime dictionary is constructed, including the _micros attribute.

The the pickler loads the object dictionary using the values stored in the ZODB. The stored values do not include a _micros attribute, so the _micros attribute is not overridden (thus it includes the current time).

If you retrieve the DateTime objects directly or use the str() method, the correct values are returned. However, if you use a method which uses the underlying _micros value such as comparison functions or asdatetime(), you get an incorrect value.

[Andreas Jung (ajung)]

How to reproduce in detail?

[ChrisW (chris-simplistix)]

- create a Date property on, say, a file object in Zope 2.9 or earlier with a date in the past (say, 2001/01/01)
- move the resulting ZODB to Zope 2.11.1
- try the following:

your_file.your_date.strftime('%Y')

You'll get 2008 instead on 2001.

[Andreas Jung (ajung)]

With a date property (value 2001/10/10) the DateTime instance has this:

>>> from pprint import pprint
>>> pprint(app.xx.datum.__dict__)
{'_aday': 'Wed',
 '_amon': 'Oct',
 '_d': 36806.916666666664,
 '_day': 10,
 '_dayoffset': 3,
 '_fday': 'Wednesday',
 '_fmon': 'October',
 '_hour': 0,
 '_millis': 1002664800000L,
 '_minute': 0,
 '_month': 10,
 '_nearsec': 0.0,
 '_pday': 'Wed.',
 '_pm': 'am',
 '_pmhour': 12,
 '_pmon': 'Oct.',
 '_second': 0.0,
 '_t': 1002664800.0,
 '_tz': 'GMT+2',
 '_year': 2001,
 'time': 0.91666666666424135}
>>>

After moving the Data.fs to Zope 2.11 I can reproduce the issue

>>> app.xx.datum
DateTime('2001/10/10')

>>> app.xx.datum.strftime('%Y')
'2008'

>>> app.xx.datum.__dict__
{'_amon': 'Oct', '_aday': 'Wed', '_pmon': 'Oct.', '_hour': 0, '_fmon': 'October', '_pday': 'Wed.', '_fday': 'Wednesday', '_pm': 'am', '_t': 1002664800.0, '_minute': 0, '_micros': 1225201190428991L, '_millis': 1002664800000L, '_d': 36806.916666666664, '_second': 0.0, '_tz': 'GMT+2', '_month': 10, '_timezone_naive': False, '_day': 10, '_year': 2001, '_nearsec': 0.0, '_pmhour': 12, '_dayoffset': 3, 'time': 0.91666666666424135}
>>> from pprint import pprint
>>> pprint(app.xx.datum.__dict__)
{'_aday': 'Wed',
 '_amon': 'Oct',
 '_d': 36806.916666666664,
 '_day': 10,
 '_dayoffset': 3,
 '_fday': 'Wednesday',
 '_fmon': 'October',
 '_hour': 0,
 '_micros': 1225201190428991L,
 '_millis': 1002664800000L,
 '_minute': 0,
 '_month': 10,
 '_nearsec': 0.0,
 '_pday': 'Wed.',
 '_pm': 'am',
 '_pmhour': 12,
 '_pmon': 'Oct.',
 '_second': 0.0,
 '_t': 1002664800.0,
 '_timezone_naive': False,
 '_tz': 'GMT+2',
 '_year': 2001,
 'time': 0.91666666666424135}
>>>

[Tres Seaver (tseaver)]

I can't reproduce with this test. Note that I verified that dt.__dict__
matches the one in the 2.9 'datum' above.

    def test_strftime_old_instance(self):
        # https://bugs.launchpad.net/zope2/+bug/290254
        # Ensure that dates without _micros attribute (e.g., from old
        # pickles) still render correctly in strftime.
        ISO = '2001-10-10T00:00:00+02:00'
        dt = DateTime(ISO)
        dt._millis = dt._micros / 1000
        del dt._micros
        self.assertEqual(dt.strftime('%Y'), '2001')

[Tres Seaver (tseaver)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tres Seaver wrote:
> I can't reproduce with this test. Note that I verified that dt.__dict__
> matches the one in the 2.9 'datum' above.
>
> def test_strftime_old_instance(self):
> # https://bugs.launchpad.net/zope2/+bug/290254
> # Ensure that dates without _micros attribute (e.g., from old
> # pickles) still render correctly in strftime.
> ISO = '2001-10-10T00:00:00+02:00'
> dt = DateTime(ISO)
> dt._millis = dt._micros / 1000
> del dt._micros
> self.assertEqual(dt.strftime('%Y'), '2001')

I have a version of the test which actually emulates the ZODB load. The
bad news is that it passes as well::

    def test_strftime_old_instance(self):
        # https://bugs.launchpad.net/zope2/+bug/290254
        # Ensure that dates without _micros attribute (e.g., from old
        # pickles) still render correctly in strftime.
        ISO = '2001-10-10T00:00:00+02:00'
        dt = DateTime(ISO)
        dt._millis = dt._micros / 1000
        del dt._micros
        self.assertEqual(dt.strftime('%Y'), '2001')

        # Now, create one via pickling / unpickling.
        from cStringIO import StringIO
        buf = StringIO()

        from cPickle import Pickler
        pickler = Pickler(buf, 1)
        state = pickler.dump(dt)
        buf.seek(0)

        from cPickle import Unpickler
        unpickler = Unpickler(buf)
        dt1 = unpickler.load()

        self.assertEqual(dt.strftime('%Y'), '2001')

- --
===================================================================
Tres Seaver [phone] <email address hidden>
Palladion Software "Excellence by Design" http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFJrqjY+gerLs4ltQ4RAsr9AJ93uWSco00gxfczMQ2oG16pGrlx/QCdH7Gp
1PdjtFrlLNIKRlJHXzcAg9c=
=5cEt
-----END PGP SIGNATURE-----

[Tres Seaver (tseaver)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tres Seaver wrote:
> Tres Seaver wrote:
>> I can't reproduce with this test. Note that I verified that dt.__dict__
>> matches the one in the 2.9 'datum' above.
>
>> def test_strftime_old_instance(self):
>> # https://bugs.launchpad.net/zope2/+bug/290254
>> # Ensure that dates without _micros attribute (e.g., from old
>> # pickles) still render correctly in strftime.
>> ISO = '2001-10-10T00:00:00+02:00'
>> dt = DateTime(ISO)
>> dt._millis = dt._micros / 1000
>> del dt._micros
>> self.assertEqual(dt.strftime('%Y'), '2001')
>
> I have a version of the test which actually emulates the ZODB load. The
> bad news is that it passes as well::
>
>
> def test_strftime_old_instance(self):
> # https://bugs.launchpad.net/zope2/+bug/290254
> # Ensure that dates without _micros attribute (e.g., from old
> # pickles) still render correctly in strftime.
> ISO = '2001-10-10T00:00:00+02:00'
> dt = DateTime(ISO)
> dt._millis = dt._micros / 1000
> del dt._micros
> self.assertEqual(dt.strftime('%Y'), '2001')
>
> # Now, create one via pickling / unpickling.
> from cStringIO import StringIO
> buf = StringIO()
>
> from cPickle import Pickler
> pickler = Pickler(buf, 1)
> state = pickler.dump(dt)
> buf.seek(0)
>
> from cPickle import Unpickler
> unpickler = Unpickler(buf)
> dt1 = unpickler.load()
>
> self.assertEqual(dt.strftime('%Y'), '2001')

OK, a typo on the last line (should be testing 'dt1', not 'dt'). Now
the test fails. I think this is actually a bug in Python's pickling
support: it doesn't clear the instance dict before upddating from the
state dict.

I have added a '__setstate__' which clears the instance dict before
updating from the unpickled state. Committed on the 2.11 branch:

- - http://svn.zope.org/Zope/branches/2.11/?rev=97471&view=rev

- - http://svn.zope.org/Zope/branches/2.11/?rev=97472&view=rev

and the trunk of the DateTime package, released as DateTime 2.12.0:

- - http://pypi.python.org/pypi/DateTime/2.12.0

status fixreleased
assignee tseaver

Tres.
- --
===================================================================
Tres Seaver [phone] <email address hidden>
Palladion Software "Excellence by Design" http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFJrrbC+gerLs4ltQ4RAqPRAKCH+B0gzM+0YK3MMA283KSOHfyEUACeNn3F
xD77s2AEA803IBy9BDS6I88=
=gkmN
-----END PGP SIGNATURE-----

[ChrisW (chris-simplistix)]

Yay, thanks, you rock :-)

Chris

Tres Seaver wrote:
> ** Changed in: zope2
> Assignee: (unassigned) => Tres Seaver (tseaver)
> Status: New => Fix Committed
>

--
Simplistix - Content Management, Zope & Python Consulting
            - http://www.simplistix.co.uk