Autopilot lacks support for large timestamps

Bug #1328600 reported by Nicholas Skaggs on 2014-06-10
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Autopilot
High
Nicholas Skaggs
Ubuntu Calendar App
High
Unassigned
Ubuntu Clock App
Undecided
Unassigned
Ubuntu UI Toolkit
Fix Released
Critical
Nicholas Skaggs
autopilot (Ubuntu)
Undecided
Unassigned
Vivid
Undecided
Unassigned

Bug Description

On the device, test_new_event fails with:

OverflowError: timestamp out of range for platform time_t

applicable log is below:

file:///opt/click.ubuntu.com/com.ubuntu.calendar/0.4.315/NewEvent.qml:64: ReferenceError: startTime is not defined
}}}

test-log: {{{
15:16:28.314 INFO globals:59 - ************************************************************
15:16:28.314 INFO globals:60 - Starting test calendar_app.tests.test_calendar.TestMainView.test_new_event (with touch)
15:16:28.316 WARNING testcase:112 - No tracing available - install the python-autopilot-trace package!
15:16:28.343 WARNING testcase:153 - Process manager backend unavailable, application snapshot support disabled.
15:16:28.377 INFO logging:45 - TestMainView: launch_test_click. Arguments (). Keyword arguments: {}.
15:16:28.379 INFO _launcher:276 - Attempting to launch click application '(default)' from click package 'com.ubuntu.calendar' and URIs ''
15:16:30.318 INFO _launcher:116 - Attempting to launch application 'com.ubuntu.calendar_calendar_0.4.315' with URIs '' via upstart-app-launch
15:16:38.053 INFO logging:45 - MainView: Open a tab. Arguments ('dayTab',). Keyword arguments: {}.
15:16:38.131 DEBUG dbus:356 - Selecting objects of type Tab with attributes: {}
15:16:38.328 INFO logging:45 - MainView: Open a tab. Arguments (3,). Keyword arguments: {}.
15:16:38.417 DEBUG dbus:356 - Selecting objects of type Tab with attributes: {}
15:16:39.689 INFO logging:45 - Header: Open a tab. This only supports the new tabs in the header. Arguments (3,). Keyword arguments: {}.
15:16:39.817 DEBUG _common:45 - Moving to object's globalRect coordinates.
15:16:39.818 DEBUG _uinput:475 - Tapping at: 45,125
15:16:41.153 DEBUG _common:45 - Moving to object's globalRect coordinates.
15:16:41.154 DEBUG _uinput:475 - Tapping at: 396,487
15:16:41.776 DEBUG dbus:356 - Selecting objects of type Tab with attributes: {}
15:16:42.816 DEBUG _common:45 - Moving to object's globalRect coordinates.
15:16:42.817 DEBUG _uinput:475 - Tapping at: 1065,125
15:16:42.961 DEBUG dbus:356 - Selecting objects of type EventBubble with attributes: {}
15:16:43.989 DEBUG _common:45 - Moving to object's globalRect coordinates.
15:16:43.991 DEBUG _uinput:475 - Tapping at: 1155,125
15:16:44.760 DEBUG _common:45 - Moving to object's globalRect coordinates.
15:16:44.761 DEBUG _uinput:475 - Tapping at: 804,263
15:16:48.149 DEBUG _common:45 - Moving to object's globalRect coordinates.
15:16:48.150 DEBUG _uinput:475 - Tapping at: 600,593
15:16:50.371 DEBUG _common:45 - Moving to object's globalRect coordinates.
15:16:50.372 DEBUG _uinput:475 - Tapping at: 318,301
}}}

Traceback (most recent call last):
File "/home/phablet/autopilot/calendar_app/tests/test_calendar.py", line 53, in test_new_event
yesterday)
File "/home/phablet/autopilot/calendar_app/emulators.py", line 47, in set_picker
pickers.DatePicker, mode=mode_value, visible=True)
File "/usr/lib/python3/dist-packages/autopilot/introspection/dbus.py", line 299, in wait_select_single
return self.select_single(type_name, **kwargs)
File "/usr/lib/python3/dist-packages/autopilot/introspection/dbus.py", line 244, in select_single
instances = self._execute_query(new_query)
File "/usr/lib/python3/dist-packages/autopilot/introspection/dbus.py", line 94, in _execute_query
type(self)
File "/usr/lib/python3/dist-packages/autopilot/introspection/backends.py", line 245, in execute_query_get_proxy_instances
for t in data
File "/usr/lib/python3/dist-packages/autopilot/introspection/backends.py", line 245, in <listcomp>
for t in data
File "/usr/lib/python3/dist-packages/autopilot/introspection/backends.py", line 295, in make_introspection_object
return class_object(state, path, backend)
File "/home/phablet/autopilot/ubuntuuitoolkit/_custom_proxy_objects/_common.py", line 76, in __init__
super(UbuntuUIToolkitCustomProxyObjectBase, self).__init__(*args)
File "/usr/lib/python3/dist-packages/autopilot/introspection/dbus.py", line 80, in __init__
self._set_properties(state_dict)
File "/usr/lib/python3/dist-packages/autopilot/introspection/dbus.py", line 119, in _set_properties
self.__state[key] = create_value_instance(value, self, key)
File "/usr/lib/python3/dist-packages/autopilot/introspection/types.py", line 106, in create_value_instance
return type_class(*value, parent=parent, name=name)
File "/usr/lib/python3/dist-packages/autopilot/introspection/types.py", line 607, in __init__
self._cached_dt = datetime.fromtimestamp(self[0])
OverflowError: timestamp out of range for platform time_t

Ran 18 tests in 379.462s
FAILED (failures=1)

Related branches

Nicholas Skaggs (nskaggs) wrote :

Autopilot needs to allow support for larger timestamps. The error above is a 32bit / c limitation that could be removed by using other datetime functions that support large timestamps (beyond 2038). I got it working by tweaking autopilot to use timedelta instead of fromtimestamp for instance . . .

Christopher Lee (veebers) wrote :

Interesting about the use of timedelta, I need to look into it further as I notice this inconsistency (note the hour difference, also this is CnP from SO):

>>> datetime.datetime.fromtimestamp(2047570047)
datetime.datetime(2034, 11, 19, 17, 27, 27)

>>> datetime.datetime.fromtimestamp(0) + datetime.timedelta(seconds=2047570047)
datetime.datetime(2034, 11, 19, 18, 27, 27)

I'll be looking into this further.

Changed in autopilot:
status: New → Confirmed
importance: Undecided → High
Nicholas Skaggs (nskaggs) wrote :

For calendar, this prevents using the time/date selection during the new event tests.

summary: - test_new_event autopilot test fails on device (r315)
+ Autopilot lacks support for large timestamps
Changed in ubuntu-calendar-app:
status: New → Confirmed
Leo Arias (elopio) wrote :

This has caused a lot of failures on the ubuntu ui toolklit on image #136.
http://ci.ubuntu.com/smokeng/utopic/touch/mako/136:20140717.1:20140717.1/9111/ubuntuuitoolkit/1385035/

Nothing seems to have changed, so I guess we just have a bigger timestamp now.

Tim Peeters (tpeeters) wrote :

This (Leo's comment) means that we cannot land any changes for UITK because the autopilot tests fail

Tim Peeters (tpeeters) on 2014-07-18
Changed in ubuntu-ui-toolkit:
importance: Undecided → Critical
tags: added: lt-category-noimpact lt-date-20140717 lt-prio-high
tags: added: lt-blocker
Leo Arias (elopio) wrote :

Veebers, the differences you are seeing are due to timezone.

>>> datetime.datetime.utcfromtimestamp(2047570047)
datetime.datetime(2034, 11, 19, 17, 27, 27)
>>> datetime.datetime.utcfromtimestamp(0) + datetime.timedelta(seconds=2047570047)
datetime.datetime(2034, 11, 19, 17, 27, 27)

There seems to be something weird on your machine, because without the utc calls you should get the same date on both calls, that's the part I don't understand.

>>> datetime.datetime.fromtimestamp(2047570047)
datetime.datetime(2034, 11, 19, 11, 27, 27)
>>> datetime.datetime.fromtimestamp(0) + datetime.timedelta(seconds=2047570047)
datetime.datetime(2034, 11, 19, 11, 27, 27)

My tz is -6, so I correctly get 11 as the hour.

Nicholas Skaggs (nskaggs) wrote :

Still investigating this to see where the easiest fix lies, along with the real source of the issues. Looking at the toolkit, you could adjust the maximum property for the datepicker objects to avoid this issue. Adjusting the qml for the affected tests to set a lower maximum property does work, and the display is updated to reflect the new max year properly.

However, the default maxyear (currently 2064) object still exists. It's seemingly being created by AP? It's unclear.

Nicholas Skaggs (nskaggs) wrote :

I can confirm the maxyear is coming from the toolkit; the timepicker object also has these values. Setting the maximum property inside the tests will workaround the 32bit limitation.

Although autopilot should still add support, we can workaround the issue inside the testcases themselves by modifying these properties to ensure we don't hit the overrun error.

So, I've spent quite some time looking at this problem today.

There are actually several, related problems here:

First, on 32 bit platforms, you cannot create a timedelta object with large values either. This fails:

python3 -c "import datetime; datetime.timedelta(2983579200)"

so the proposed fix for autopilot doesn't, in fact, fix anything.

Second, adding a timedelta to a datetime object takes into account the local timezone, including the DST. So, for example:

>>> def do_test(num_seconds):
... dt1 = datetime.fromtimestamp(num_seconds)
... dt2 = datetime.fromtimestamp(0) + timedelta(seconds=num_seconds)
... print("Timestamps are %s the same: %s %s" % ("" if dt1 == dt2 else "not", dt1, dt2))
...
>>> do_test(2222121600)
Timestamps are the same: 2040-06-01 12:00:00 2040-06-01 12:00:00
>>> do_test(2208988800)
Timestamps are not the same: 2040-01-01 13:00:00 2040-01-01 12:00:00

This has nothing to do with the year of the timestamp, but rather because in New Zealand (which is my current locale), Jan has DST applied, but June does not (or the other way around, whatever).

So, sadly, both approaches have problems:

* Using datetime.fromtimestamp (current implementation) breaks because datetime uses C libraries that, on a 32 bit system, won't work with timestamps that overflow an int (which will be 32 bits).

* Using datetime.timedelta for some reason applies DST adjustments, which is frankly dumb. I'd expect this to hold true all the time:

datetime.fromtimestamp(N) == datetime.fromtimestamp(0) + timedelta(seconds=N)

but it only evaluates to true in local winter months it seems.

So this needs more investigation, is my final verdict :)

Alexander Sack (asac) wrote :

more investigation? we are not able to figure out how to do timestamp calcs properly in python? sigh...

who can fix this?

Barry Warsaw (barry) wrote :

In the example:

python3 -c "import datetime; datetime.timedelta(2983579200)"

realize that the first argument is days, not seconds, so you're trying to create a date more than 8 million years in the future. In the abstract, I hope Ubuntu - and the human race - is around that long :). Try this:

python3 -c "import datetime; datetime.timedelta(seconds=2983579200)"

That should work even on 32 bit machines (it works in my i386 utopic chroot).

The documentation does describe that datetime.{utc,}fromtimestamp() is limited by the argument size to libc's gtime() so you'd have the same problems with a C program. How would you solve 64 bit timestamps on 32 bit platforms in C or C++? I'm not sure there is a solution for 64 bit datetimes on 32 bit platforms. E.g. numpy has a datetime64, but substituting the above I get a NaT (i.e. not-a-time). Maybe I'm using it wrong though. There's also egenix-datetime, but that's Python 2 only in the archive (and maybe upstream). It's probably worth investigating available options on PyPI.

On the second point, remember that Python datetimes have two "modes", a naive mode where timezones are ignored but local time is usually assumed, and timezone aware datetimes. IMHO, it's *always* a good idea to do your internal time calculations in UTC time, and create the appropriate timezone objects for conversion/display to local time at the edges. I think of it the same way as Unicode: convert from whatever encoding your bytes are in to Unicode at the edges, manipulate internally as unicode, and then encode to bytes on the way out. So by analogy, convert to UTC time on the way in and local time on the way out. E.g. datetime.utcfromteimstamp() is probably better for your use case.

Tim Peeters (tpeeters) on 2014-07-22
Changed in ubuntu-ui-toolkit:
status: New → Confirmed
Nicholas Skaggs (nskaggs) wrote :

Updating my MP after discussing with Barry and thomi. We should be able to deal with the timezone issue thomi is describing and continue to use timedelta to provide larger timestamp support.

Changed in autopilot:
assignee: nobody → Nicholas Skaggs (nskaggs)
Changed in ubuntu-ui-toolkit:
assignee: nobody → Nicholas Skaggs (nskaggs)
status: Confirmed → Fix Released
Changed in autopilot:
status: Confirmed → In Progress
Changed in ubuntu-calendar-app:
importance: Undecided → Critical
milestone: none → rtm14
Nicholas Skaggs (nskaggs) wrote :

Chris, Thomi and I discussed the need for a set of functional tests to assert what should be occuring with datetimestamps and timezones that go beyond what exists in autopilot trunk.

Namely, we'd like a suite of tests that can assert the same date + time in python as is present in a Qml. We'd like scenarios then to cover all 4 seasons (one every quarter to deal with daylight savings and other oddities), as well as several timezones around the globe.

The current MP does this on a unit testing level, but there is concern about how things correspond at a functional testing level. Chris and I agreed it would be useful to propose these functional level test enhancements as a separate MP against autopilot trunk to assert and understand what is occurring in autopilot today (filing bugs if found). These tests can then be used to verify the changes made in the MP and land them safely.

David Planella (dpm) wrote :

Is this bug still affecting Calendar? I understand there's nothing in the app that needs to be fixed, and the fixes would rather be on the test suite. Do we need a bug task for Calendar at all?

Changed in ubuntu-calendar-app:
status: Confirmed → Incomplete
importance: Critical → High
PS Jenkins bot (ps-jenkins) wrote :

Fix committed into lp:autopilot at revision None, scheduled for release in autopilot, milestone 1.4

Changed in autopilot:
status: In Progress → Fix Committed
Nicholas Skaggs (nskaggs) wrote :

Both calendar and clock tests can now check for this. For calendar, this means the tests for specific dates and using the datefields should be re-added.

Changed in autopilot:
status: Fix Committed → Fix Released
Changed in ubuntu-calendar-app:
status: Incomplete → Opinion
status: Opinion → Triaged
Changed in ubuntu-clock-app:
status: New → Confirmed
Changed in ubuntu-calendar-app:
status: Triaged → Confirmed
tags: added: needs-autopilot-test
Nicholas Skaggs (nskaggs) wrote :

Tracking fixes for calendar and clock in individual bugs. The issue no longer affects them directly.

Changed in ubuntu-calendar-app:
status: Confirmed → Invalid
Changed in ubuntu-clock-app:
status: Confirmed → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers