Files over a certain size get truncated when restored from Trash

[Paul Everitt]

Tom at Oxfam wrote, regarding the staging site at gocept:

Files over a certain size (something like 1Mb I think) get corrupted somewhere between being deleted and being restored from the Trash. In case it helps, I have attached a copy of a file that was corrupted in this way, before and after:

Assigning to Jim to try and recreate.

[JimPGlenn (jpglenn09)]

I recreated this with a 4.3mb file. When I restored it it was only 4.1mb.

[Paul Everitt (paul-agendaless)]

Jim said this is true on Oxfam staging but not on OSF production.

[Shane Hathaway (shane-hathawaymix)]

Due to a schema bug (caused by a misunderstanding of the SQLAlchemy docs), repozitory was sometimes retrieving the chunks of a blob out of order. This is fixed in repozitory 1.0.

[Paul Everitt (paul-agendaless)]

Is it safe to say we have some working copies with corrupt files?

--Paul

On Nov 17, 2011, at 9:03 PM, Shane Hathaway wrote:

> Due to a schema bug (caused by a misunderstanding of the SQLAlchemy
> docs), repozitory was sometimes retrieving the chunks of a blob out of
> order. This is fixed in repozitory 1.0.
>
> ** Changed in: karl3
> Assignee: Chris Rossi (chris-archimedeanco) => Shane Hathaway (shane-hathawaymix)
>
> ** Changed in: karl3
> Status: New => Fix Committed
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/891277
>
> Title:
> Files over a certain size get truncated when restored from Trash
>
> Status in KARL3:
> Fix Committed
>
> Bug description:
> Tom at Oxfam wrote, regarding the staging site at gocept:
>
> Files over a certain size (something like 1Mb I think) get corrupted
> somewhere between being deleted and being restored from the Trash. In
> case it helps, I have attached a copy of a file that was corrupted in
> this way, before and after:
>
> Assigning to Jim to try and recreate.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/karl3/+bug/891277/+subscriptions

[Shane Hathaway (shane-hathawaymix)]

On 11/17/2011 08:14 PM, Paul Everitt wrote:
> Is it safe to say we have some working copies with corrupt files?

Only if people have deleted large files and then undeleted them. This
bug did not break the contents of the archive, it only broke reading
large files from the archive. We can check for affected files as follows:

- Make a list of all file objects with contents larger than 1 MB.

- Compare the length of each of those files with the length of the
archived version. If the length is different, then this bug did not
affect the file. (The bug did *not* truncate files. I think the 4.1 vs.
4.3 MB comment is due to mebibytes vs. megabytes.)

- Compare the md5 sum with the archive. If it's different, flag the file
as possibly broken.

We can do all those steps on staging. If we find any broken files,
restore them from the archive using repozitory 1.0.

I'm guessing we'll find a small number of broken files, if any.

Shane

[Paul Everitt (paul-agendaless)]

Should we make a new task for this, keeping the other one closed?

--Paul

On Nov 17, 2011, at 11:25 PM, Shane Hathaway wrote:

> On 11/17/2011 08:14 PM, Paul Everitt wrote:
>> Is it safe to say we have some working copies with corrupt files?
>
> Only if people have deleted large files and then undeleted them. This
> bug did not break the contents of the archive, it only broke reading
> large files from the archive. We can check for affected files as follows:
>
> - Make a list of all file objects with contents larger than 1 MB.
>
> - Compare the length of each of those files with the length of the
> archived version. If the length is different, then this bug did not
> affect the file. (The bug did *not* truncate files. I think the 4.1 vs.
> 4.3 MB comment is due to mebibytes vs. megabytes.)
>
> - Compare the md5 sum with the archive. If it's different, flag the file
> as possibly broken.
>
> We can do all those steps on staging. If we find any broken files,
> restore them from the archive using repozitory 1.0.
>
> I'm guessing we'll find a small number of broken files, if any.
>
> Shane
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/891277
>
> Title:
> Files over a certain size get truncated when restored from Trash
>
> Status in KARL3:
> Fix Committed
>
> Bug description:
> Tom at Oxfam wrote, regarding the staging site at gocept:
>
> Files over a certain size (something like 1Mb I think) get corrupted
> somewhere between being deleted and being restored from the Trash. In
> case it helps, I have attached a copy of a file that was corrupted in
> this way, before and after:
>
> Assigning to Jim to try and recreate.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/karl3/+bug/891277/+subscriptions

[Shane Hathaway (shane-hathawaymix)]

Let's add another task for the cleanup.

[Paul Everitt (paul-agendaless)]

Jim, mark this with the release tag for today's release. Test it on branch1 then test it after the production update.

To test, re-do what you did on comment #1 and see if it is fixed.

[JimPGlenn (jpglenn09)]

fixed