Cleanup file objects corrupted by restoring over 1 Mb

[Paul Everitt]

From lp:891277 comment 6, a cleanup task:

"""
Only if people have deleted large files and then undeleted them. This
bug did not break the contents of the archive, it only broke reading
large files from the archive. We can check for affected files as follows:

- Make a list of all file objects with contents larger than 1 MB.

- Compare the length of each of those files with the length of the
archived version. If the length is different, then this bug did not
affect the file. (The bug did *not* truncate files. I think the 4.1 vs.
4.3 MB comment is due to mebibytes vs. megabytes.)

- Compare the md5 sum with the archive. If it's different, flag the file
as possibly broken.

We can do all those steps on staging. If we find any broken files,
restore them from the archive using repozitory 1.0.

I'm guessing we'll find a small number of broken files, if any.
"""

[Paul Everitt (paul-agendaless)]

For Chris, for later on in the month.

[Shane Hathaway (shane-hathawaymix)]

I can take this if you're busy, Chris.

[Chris Rossi (chris-archimedeanco)]

Sure, thanks!

On Tue, Dec 20, 2011 at 6:33 PM, Shane Hathaway <email address hidden> wrote:
> I can take this if you're busy, Chris.
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/898383
>
> Title:
>  Cleanup file objects corrupted by restoring over 1 Mb
>
> Status in KARL3:
>  New
>
> Bug description:
>  From lp:891277 comment 6, a cleanup task:
>
>  """
>  Only if people have deleted large files and then undeleted them. This
>  bug did not break the contents of the archive, it only broke reading
>  large files from the archive. We can check for affected files as follows:
>
>  - Make a list of all file objects with contents larger than 1 MB.
>
>  - Compare the length of each of those files with the length of the
>  archived version. If the length is different, then this bug did not
>  affect the file. (The bug did *not* truncate files. I think the 4.1 vs.
>  4.3 MB comment is due to mebibytes vs. megabytes.)
>
>  - Compare the md5 sum with the archive. If it's different, flag the file
>  as possibly broken.
>
>  We can do all those steps on staging. If we find any broken files,
>  restore them from the archive using repozitory 1.0.
>
>  I'm guessing we'll find a small number of broken files, if any.
>  """
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/karl3/+bug/898383/+subscriptions

[Paul Everitt (paul-agendaless)]

Moving out of the way to let us do a chameleon2 release.

[Shane Hathaway (shane-hathawaymix)]

I wrote a script to check all uploaded files against the bug. I found no instances of corrupted files. I also manually checked files for which multiple versions have been uploaded, but found no corruption there, either. It looks like we dodged the bullet on this one.

I've attached the script I used. It might come in handy for something else.

[Paul Everitt (paul-agendaless)]

Thanks Shane.