Support files greater than 2.2Gb in size of 32 bit systems

[Matthew Holder]

The offset of a file being uploaded or downloaded is defined by an integer, on 32 bit systems this is represented by 32 bits.

This number is approximately 2.2Gb.

I have tested this using the API, uploading to a 1.3 (type) release and when I hit the limit the offset returned by the server is the negative limit of a 32bit integer.

The database stores filesize as a BIGINT, so this will not need to be changed

It looks to me that this bug will affect:

1) The way the server reports offsets
2) The way clients and any other apps deal with offsets

I can see two ways of proceeding

1) Change the server reporting to the long data type and then change how the clients deal
2) Change the server reporting to sending two offset values, offset and offset_high. The clients and applications can then put together the two numbers with bit shifting of the offset_high to create the long value for the offset.

Option two would mean that clients can remain backwards compatible for files less than 2.2Gb.

I am not sure how much of a consideration backwards compatibility is, especially if offsets are reported as long from a new major revision.

Matt

[Alex Harrington (alexharrington)]

Python Client / Offline Download Client Implications:
 * Need to ensure offsets are converted to long types
 * Need to ensure all places dealing with file size are converted to long types

API Testsuite Implications:
 * Need to ensure offsets are converted to long types
 * Need to ensure that filesizes are represented as longs

[Dan Garner (dangarner)]

Server / API implications
 * None as PHP will treat larger than INT values as a FLOAT automatically

.NET client implications
 * Will need to change the ints to doubles in the FileCollector

[Dan Garner (dangarner)]

I've untagged this because it is fundamentally not supported in PHP at the moment - the majority of file based functions will return incorrect data for files > 2gb.