sha256sum mismatch in cloud images

[Donald Eugene Kutach]

6/20 Build

http://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-root.tar.gz

We have an automated process which downloads images. First the sha256sum check passed but we saw unexpected EOF errors when trying to untar the file. Later we were able to download and untar the file but the sha256sum does not match.

From the sha256sums file.

7584d5214285c249a9ae80f98b71d7cc216fd915be2ca40d82a1e4603b7ae257 *xenial-server-cloudimg-amd64-root.tar.gz
Output in terminal

sha256sum xenial-server-cloudimg-amd64-root.tar.gz
04f9bd06a12636ac1e747856de2797a8a17c8ebcfca4714c080912bff90eeca1 xenial-server-cloudimg-amd64-root.tar.gz

I suspect the file may have been replaced but the sha256sum value not updated?

This fails on all of our build systems and the automated process has been in place for months.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Ganpat Agarwal (gans-developer)]

I am also getting error for my build using openstack diskbuilder-tool

sha256sum --check -xenial-server-cloudimg-amd64-root.tar.gz: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match

[Dimitri John Ledkov (xnox)]

Moving the bug to the "cloud-images" project - the correct bug tracker for issues with image publication and images themselves.

[Adam Stokes (adam-stokes)]

conjure-up localhost no longer works because of this.

[David Britton (dpb)]

Steps to repro for me on 16.04:

lxc image list
lxc image delete <xenial_image>
lxc launch ubuntu-daily:xenial

It will retry a number of times, then:

error: Hash mismatch for https://cloud-images.ubuntu.com/daily/server/xenial/20170620/xenial-server-cloudimg-amd64.squashfs: 1ca106f30241e7d4aba003923292116c229e8e811800cfeaa4a771e9e932f894 != 23271027b75765a7d46d44a7c8f9e7cf81526165856526b75c6007ea78346886

[Patricia Gaughen (gaughen)]

Thank you for reporting the issue, we are digging into this.

[Christian Ehrhardt  (paelzer)]

Dupped my bug onto this.
Carrying some info from there:

1. this is essentially a re-occurrence of bug 1668876

2. another way to repro:
$ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=xenial
[...]
    raise checksum_util.invalid_checksum_for_reader(self)
simplestreams.checksum_util.InvalidChecksum: Invalid sha256 Checksum at http://cloud-images.ubuntu.com/daily/server/xenial/20170620/xenial-server-cloudimg-amd64-disk1.img. Found 9fc24ce8113fc635567a3944ccb518d3c09bd9270315f3d654811db7221391d1. Expected bb5e963ece5c1f6071436470dbc09955a9227ce51d47636ffcf43dab60605ddf. read 287506432 bytes expected 146210816 bytes. (size 287506432 expected 146210816)

3. in the old bug we had a test to check all images published, that executed lists all xenial arch/types as broken (see file attached to the dup bug 1699454)

Other than fixing the images, you should consider running a checker (maybe local to the publishing machine for speed) of images vs checksums file - almost on a 5 minute schedule or such and auto-bug you if it fails. Would that be reasonable?

[Patricia Gaughen (gaughen)]

Quick update - We've identified the issue. The checksums are now correct, streams are in the process of being addressed. This only impacts the Xenial 20170620 daily image.

[Michael Johnson (johnsom)]

Also a forums post about it: https://ubuntuforums.org/showthread.php?t=2364292

[Francis Ginther (fginther)]

The streams data for the Xenial 20170620 daily image has now been updated to contain the correct checksums and it is now usable again.

We are still working to correct the underlying issue so that this doesn't happen again.

[Robert C Jennings (rcj)]

No issue with the images themselves. The indexing that calculates checksums ran before the files were completely written due to a failure to catch a file transfer issue. Later the complete files were synced over so that's how we have files that boot but SHAs that don't match (check .qindex.json in the same path as the daily image and you'll see that the size of the file is too small and a dd of just those bytes to sha256sum matches the computed incorrect sum).

The fix on our side is to catch the transfer failure and halt publication.

[Christian Ehrhardt  (paelzer)]

Thanks a lot for the extra explanation Robert!

[Robert C Jennings (rcj)]

For anyone's education and amusement
(details @ http://mywiki.wooledge.org/BashFAQ/105)

####################################################################################
# ISSUE: Piped commands evaluate to the return of the last process in the pipe
# To test any of these consider 'true' or 'false' in place of 'cmd'
# Example:

#!/bin/sh -e
cmd | tee file # The 'false' does not stop execution
echo $? # The return is 0

####################################################################################
# Solution #1: Capture the output, then echo to tee (we need the output in a file)
# Limitation: The output of the command is not displayed until after it completes
# and only if the exit is non-zero, we lose debug information here

#!/bin/sh -e
foo=$(cmd) # This will cause us to exit the shell as we intend
echo $foo | tee file # If the command was successful we'd have output on
                       # the console and in file

####################################################################################
# Solution #2: Subshells and 'kill 0'

#!/bin/sh -e
(cmd || kill 0) | tee file # 'kill 0' kills the process group of the caller

####################################################################################
# Solution #3: Edit 'cmd' to eliminate the need to save output for later processing.

******************************
All jobs have been updated to use the pattern from solution #2 and we now have a
task to move to solution #3.

[Van Hung Pham (hungphv)]

I'm seeing this bug again when pulling image from http://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-root.tar.gz. Any update?