Yakkety daily image (1st March) sync broken

[Christian Ehrhardt ]

$uvt-simplestreams-libvirt sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=yakkety
Traceback (most recent call last):
  File "/usr/bin/uvt-simplestreams-libvirt", line 27, in <module>
    uvtool.libvirt.simplestreams.main()
  File "/usr/lib/python2.7/dist-packages/uvtool/libvirt/simplestreams.py", line 326, in main
    args.func(args)
  File "/usr/lib/python2.7/dist-packages/uvtool/libvirt/simplestreams.py", line 269, in main_sync
    tmirror.sync(smirror, initial_path)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 91, in sync
    return self.sync_index(reader, path, data, content)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 254, in sync_index
    self.sync(reader, path=epath)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 89, in sync
    return self.sync_products(reader, path, data, content)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 341, in sync_products
    self.insert_item(item, src, target, pgree, ipath_cs)
  File "/usr/lib/python2.7/dist-packages/uvtool/libvirt/simplestreams.py", line 236, in insert_item
    pool_name=LIBVIRT_POOL_NAME
  File "/usr/lib/python2.7/dist-packages/uvtool/libvirt/__init__.py", line 55, in create_volume_from_fobj
    shutil.copyfileobj(fobj, compressed_fobj)
  File "/usr/lib/python2.7/shutil.py", line 49, in copyfileobj
    buf = fsrc.read(length)
  File "/usr/lib/python2.7/dist-packages/simplestreams/contentsource.py", line 286, in read
    raise checksum_util.invalid_checksum_for_reader(self)
simplestreams.checksum_util.InvalidChecksum: Invalid sha256 Checksum at http://cloud-images.ubuntu.com/daily/server/yakkety/20170222.1/yakkety-server-cloudimg-amd64.img. Found c66b28f55a8d89269055077924fa53cab0478b93c76b3243eca43efe831e4dba. Expected ea089b13a75cc6ad92aa6533796f912adf3fd8f860b0134752bade9b6644ccc8. read 342622208 bytes expected 342622208 bytes. (size 342622208 expected 342622208)

Size is ok, but checksum is not.
Not sure if that is a simplestreams issue - it might "just" be the image that is broken, but starting to file here.

[Christian Ehrhardt  (paelzer)]

Repeatable from Trusty and Xenial.

A direct wget of http://cloud-images.ubuntu.com/daily/server/yakkety/20170222.1/yakkety-server-cloudimg-amd64.img gives different sizes and checksums.
Need to read the code ...

[Christian Ehrhardt  (paelzer)]

Simplestreams data on the file matches report:

Handle item: disk1.img/{u'size': 342622208, u'path': u'server/yakkety/20170222.1/yakkety-server-cloudimg-amd64.img', u'ftype': u'disk1.img', u'sha256': u'ea089b13a75cc6ad92aa6533796f912adf3fd8f860b0134752bade9b6644ccc8', u'md5': u'2501c517c203638e5c398c02ab72a793'}

/usr/lib/python2.7/dist-packages/simplestreams/contentsource.py function "read"
This is called iteratively until all is read.
After it hits expected size it compares checksums.

Since a full wget fetches a bigger size 342818816 but even on that the checksum is not as expected I'm not sure where the error might be.

[Christian Ehrhardt  (paelzer)]

In
http://cloud-images.ubuntu.com/daily/server/yakkety/20170222.1/SHA256SUMS
I have
ea089b13a75cc6ad92aa6533796f912adf3fd8f860b0134752bade9b6644ccc8 *yakkety-server-cloudimg-amd64.img
in MD5SUMS
2501c517c203638e5c398c02ab72a793 *yakkety-server-cloudimg-amd64.img

That matches the simplestreams data on sha256 and md5

The file referenced there is
http://cloud-images.ubuntu.com/daily/server/yakkety/20170222.1/yakkety-server-cloudimg-amd64.img

But when fetching "the real file" I get:
Size 342818816
sha256sum 9f250be2b0657c11ea76a46cadd546a592418a6eb45861b38dcd5df7f6b9090c
md5sum b18dceb1e8864e90c94da1d90a1711db

I checked a few more, it seems none of them matches the checksums.
Uploaded yesterday (28th Feb)
Publishing issue?

[Christian Ehrhardt  (paelzer)]

As sanity check I verified that e.g. Xenial files match what I find in
http://cloud-images.ubuntu.com/daily/server/xenial/current/SHA256SUMS

So the wget+sha256 approach is valid.

I expect stale metadata.

[Christian Ehrhardt  (paelzer)]

But Xenial is also older from 24th Feb.

Zesty from 28th Feb as well seem to be ok.

[Christian Ehrhardt  (paelzer)]

Subscribing Scott and Daniel for their expertise on image publishing.

[Scott Moser (smoser)]

I had opened https://bugs.launchpad.net/cloud-images/+bug/1668710
yesterday, rcj fixed it, but apparently other files were affected.

[Scott Moser (smoser)]

To pro-actively see what else was broken I did this. The errors below are just 'arch=amd64 release=yakkety'. I'll go ahead and run a full pass.

$ bzr branch lp:simplestreams simplestreams
$ cd simplestreams

# I then modified sstream-mirror to only warn on errors
# and log to file /tmp/errors.log
$ bzr diff
=== modified file 'simplestreams/mirrors/__init__.py'
--- simplestreams/mirrors/__init__.py 2016-06-10 11:20:18 +0000
+++ simplestreams/mirrors/__init__.py 2017-03-01 13:41:33 +0000
@@ -473,9 +473,16 @@
         if not self.config.get('item_download', True):
             return
         LOG.debug("inserting %s to %s", contentsource.url, data['path'])
- self.store.insert(data['path'], contentsource,
- checksums=checksum_util.item_checksums(data),
- mutable=False, size=data.get('size'))
+ try:
+ self.store.insert(data['path'], contentsource,
+ checksums=checksum_util.item_checksums(data),
+ mutable=False, size=data.get('size'))
+ except checksum_util.InvalidChecksum as e:
+ with open("/tmp/errors.log", "a") as fp:
+ fp.write("%s\n" % e)
+ import sys
+ sys.stderr.write("%s\n" % e)
+
         self._inc_rc(data['path'], src, pedigree)

     def insert_index_entry(self, data, src, pedigree, contentsource):

$ url=http://cloud-images.ubuntu.com/daily/streams/v1/index.json
$ rm -Rf out.d
$ ./tools/tenv sstream-mirror --max=1 $url --progress out.d release=yakkety arch=amd64
...
cat /tmp/errors.log
Invalid sha256 Checksum at http://cloud-images.ubuntu.com/daily/server/yakkety/20170222.1/yakkety-server-cloudimg-amd64-lxd.tar.xz. Found fd9f97481a3d21f6cf017b5e8b199c7b5a9fbd3fe0896fa697edd44267b93036. Expected 522ea193e4414cc41baa578e82f4eba7b6c5e516049656ba8cdae84a4d10b0cb. read 848 bytes expected 840 bytes. (size 848 expected 840)
Invalid sha256 Checksum at http://cloud-images.ubuntu.com/daily/server/yakkety/20170222.1/yakkety-server-cloudimg-amd64.manifest. Found a7018e21f0b9119231d7c25cacb360b753d4d40e11bf6fdbda9398ea8e00fc60. Expected d6f2d40732308b2f0a95d2b36fec0d2899976f459536abedfeb5e2af48474326. read 14255 bytes expected 14255 bytes. (size 14255 expected 14255)
Invalid sha256 Checksum at http://cloud-images.ubuntu.com/daily/server/yakkety/20170222.1/yakkety-server-cloudimg-amd64.squashfs.manifest. Found 0bda9b9a517cb85dd818136d4cef13b998607fc555bc2f6e0ce0637950dddd15. Expected 955f530579667721260241d9c82df219ecb66817ea6aeba70515a5d5033ada92. read 13736 bytes expected 13736 bytes. (size 13736 expected 13736)
Invalid sha256 Checksum at http://cloud-images.ubuntu.com/daily/server/yakkety/20170222.1/yakkety-server-cloudimg-amd64.img. Found 9a209e74abceb899c73e678758401eed18832e90347ad81f56975525cd718224. Expected ea089b13a75cc6ad92aa6533796f912adf3fd8f860b0134752bade9b6644ccc8. read 342630400 bytes expected 342622208 bytes. (size 342630400 expected 342622208)
Invalid sha256 Checksum at http://cloud-images.ubuntu.com/daily/server/yakkety/20170222.1/yakkety-server-cloudimg-amd64.tar.gz. Found ebaecba0a4aba918637e7d5ab4861290fd513ec3113b7f2c6be19d2c82382aba....

[Scott Moser (smoser)]

Attaching a full log. It seems that yakkety build 20170222.1 is the extent of the errors.

[Christian Ehrhardt  (paelzer)]

That seems to become more and more often, today I see:

+ lxc launch ubuntu-daily:zesty/ppc64el testkvm-zesty-from --profile default --profile kvm
Creating testkvm-zesty-from
error: Hash mismatch for server/zesty/20170228.1/zesty-server-cloudimg-ppc64el-lxd.tar.xz: e4a25922af6ef169b32e18b527b7595008d4ceb77551fa0d21b6b761cc1a964d != 7f0a3a7c0e4566f54c58774dd32ef739b3ff52f5f2fb6606d40b6034eb60a493

Listed as:
7f0a3a7c0e4566f54c58774dd32ef739b3ff52f5f2fb6606d40b6034eb60a493 *zesty-server-cloudimg-ppc64el-lxd.tar.xz

e4a25922... is the correct hash of the file downloaded from various places.
Is that error systematic and growing?

Automated testing is getting hard these days ...

[Jose L. VG (josvaz)]

20170222.1 is what rcj fixed yesterday, but seems something is still not right there. That was a manual rerun by mistake I think.

20170228.1 I am not sure yet, might have been another mistake rerun.

Going to dig in now...

[Jose L. VG (josvaz)]

I can confirm that both issues flagged here were to mistaken manual runs. Last one happening yesterday and affecting 20170228.1 zesty

We are also running a check on the side in Jerff to make sure there are no other undetected inconsistencies.

[Jose L. VG (josvaz)]

20170221.1 daily needs to go away, as it contains some images build from more recent state in the archive but still labelled as 20170228.1

That can happen once we have a newer daily for zesty.

[Jose L. VG (josvaz)]

Meant to say that 20170228.1 needs to go away.

[Robert C Jennings (rcj)]

A newer zesty build (20170302) has been publish and zesty daily with the serial 20170228.1 will be removed.

(Reposted to clarify this is for zesty given the title of this bug)

[Robert C Jennings (rcj)]

Please verify with the latest daily for zesty.

[Joshua Powers (powersj)]

LGTM got tests running again on ppc64el and s390x

[Jose L. VG (josvaz)]

Thanks to Rob this is now fixed.

All occurrences were due to mistaken manual re-runs. We will implement safeguards to avoid this mistakes to have such impact again.

The script run to check integrity in Jerff was sucking too much IO on an already heavily loaded machine, so we could not finish the validation.

I will revisit this bug in a few days and mark it fix released if no other issues come up.

[Christian Ehrhardt  (paelzer)]

Works for me as well now.
Feel free to set to Fix Released if no update shows up in a few days.

[Jose L. VG (josvaz)]

No new reports, so will mark this solved now.