`meph2-util insert` failing

[Dan Watkins]

When running on jerff, we get the following traceback:

+ python3 /var/lib/jenkins/jobs/MAAS_v2-Merge_from_Swift/workspace/ARCH/amd64/maasv2_branch/bin/meph2-util insert /var/lib/jenkins/jobs/MAAS_v2-Merge_from_Swift/workspace/ARCH/amd64/maasv2/amd64/tmp/maas_final/streams/v1/index.json /srv/maas-images/ephemeral-v2/daily --keyring /srv/jenkins/.gnupg/keyrings/ubuntu-cloudimage-keyring.gpg
content_id=com.ubuntu.maas:daily:v2:download path=streams/v1/com.ubuntu.maas:daily:v2:download.json
inserting item com.ubuntu.maas.daily:v2:boot:16.10:amd64:hwe-y/20160803.1/di-initrd
inserting item com.ubuntu.maas.daily:v2:boot:16.10:amd64:hwe-y/20160803.1/root-image.gz
inserting item com.ubuntu.maas.daily:v2:boot:16.10:amd64:hwe-y/20160803.1/boot-initrd
inserting item com.ubuntu.maas.daily:v2:boot:16.10:amd64:hwe-y/20160803.1/manifest
Traceback (most recent call last):
  File "/var/lib/jenkins/jobs/MAAS_v2-Merge_from_Swift/workspace/ARCH/amd64/sstreams/simplestreams/contentsource.py", line 244, in __init__
    csummer = checksum_util.SafeCheckSummer(checksums)
  File "/var/lib/jenkins/jobs/MAAS_v2-Merge_from_Swift/workspace/ARCH/amd64/sstreams/simplestreams/checksum_util.py", line 79, in __init__
    (checksums, allowed))
ValueError: provided checksums ({}) did not include any allowed (('md5', 'sha256', 'sha512'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/jenkins/jobs/MAAS_v2-Merge_from_Swift/workspace/ARCH/amd64/maasv2_branch/bin/meph2-util", line 25, in <module>
    call_entry_point("meph2.commands.meph2_util.main")
  File "/var/lib/jenkins/jobs/MAAS_v2-Merge_from_Swift/workspace/ARCH/amd64/maasv2_branch/bin/meph2-util", line 22, in call_entry_point
    sys.exit(getattr(sys.modules[istr], ent)())
  File "/var/lib/jenkins/jobs/MAAS_v2-Merge_from_Swift/workspace/ARCH/amd64/maasv2_branch/meph2/commands/meph2_util.py", line 847, in main
    return args.action(args)
  File "/var/lib/jenkins/jobs/MAAS_v2-Merge_from_Swift/workspace/ARCH/amd64/maasv2_branch/meph2/commands/meph2_util.py", line 459, in main_insert
    tmirror.sync(smirror, src_path)
  File "/var/lib/jenkins/jobs/MAAS_v2-Merge_from_Swift/workspace/ARCH/amd64/sstreams/simplestreams/mirrors/__init__.py", line 91, in sync
    return self.sync_index(reader, path, data, content)
  File "/var/lib/jenkins/jobs/MAAS_v2-Merge_from_Swift/workspace/ARCH/amd64/sstreams/simplestreams/mirrors/__init__.py", line 254, in sync_index
    self.sync(reader, path=epath)
  File "/var/lib/jenkins/jobs/MAAS_v2-Merge_from_Swift/workspace/ARCH/amd64/sstreams/simplestreams/mirrors/__init__.py", line 89, in sync
    return self.sync_products(reader, path, data, content)
  File "/var/lib/jenkins/jobs/MAAS_v2-Merge_from_Swift/workspace/ARCH/amd64/sstreams/simplestreams/mirrors/__init__.py", line 337, in sync_products
    checksums=checksum_util.item_checksums(flat))
  File "/var/lib/jenkins/jobs/MAAS_v2-Merge_from_Swift/workspace/ARCH/amd64/sstreams/simplestreams/contentsource.py", line 246, in __init__
    raise checksum_util.invalid_checksum_for_reader(self, msg=str(e))
simplestreams.checksum_util.InvalidChecksum: provided checksums ({}) did not include any allowed (('md5', 'sha256', 'sha512'))

[Dan Watkins (oddbloke)]

< smoser> Odd_Bloke, there is no checksum for the manifest file in the metadata
< smoser> this is probably regression of revno 316

[Lee Trager (ltrager)]

Are the items in /var/lib/jenkins/jobs/MAAS_v2-Merge_from_Swift/workspace/ARCH/amd64/maasv2/amd64/tmp/maas_final/streams/v1/index.json just newer versions than what is in /srv/maas-images/ephemeral-v2/daily? If so could you try the merge command instead of insert?

If that doesn't work is there a way I could get access to the streams you are trying to merge, or at least the metadata?

[Scott Moser (smoser)]

Lee,

'insert' is right. The new data is all new, it was created from a build, so we want to insert a specific bit of it into the existing stream.

Dan can probably point you at the code that actually invokes it.

It runs in 2 stages as the README explains. The 'sync' process only runs any more for builds < xenial.

a.) during the build process on launchpad it does something similar to
  == build a single release/arch combination ===
  something like
meph2-build amd64 xenial 20160805 xenial-server-cloudimg-amd64.squashfs xenial-output.d/
it does seem from the above that they run with '--enable-di'.

b.) after the data from maas is collected, the stream data from the new build is inserted into existing data (that you can see on images.maas.io.
  That runs something like As described in 'streams management' and 'insert data'.

   meph2-util insert xenial-output.d/streams/v1/index.json my-new.d

I did recreate this with the above, but didn't have any time to debug.

[Scott Moser (smoser)]

attaching the output of
  meph2-build amd64 xenial 20160805 xenial-server-cloudimg-amd64.squashfs xenial-output.d/

A couple things.
a.) the root of the problem
      "squashfs": {
       "ftype": "squashfs",
       "path": "xenial/amd64/20160805/xenial-server-cloudimg-amd64.squashfs",
       "sha256": null,
       "size": null
      }

b.) i don't like the filename for the squashfs image. It *is* just a filename, but it is inconsistent with the others. root.squashfs would be good.

[Lee Trager (ltrager)]

I've upload a new revision to lp:maas-images which should fix the issue. It appears I was being a little over zealous with the cleanup and removed too many files.

I kept the SquashFS filename the same as the source(http://cloud-images.ubuntu.com/) to show lineage but I could rename the filename to be the filetype to be consistent with other files.

[Scott Moser (smoser)]

Hi,
We're still seeing failures on this, and it means our latest maas image for yakkety is 20160731.

[Scott Moser (smoser)]

This script you can run on a fresh cloud instance and it will reproduce the failure.

[Lee Trager (ltrager)]

Thanks for the script. What happened was I did all development for the SquashFS images using the meph2-cloudimg-sync command. The provided script uses meph2-build. I was taking the SHA256 and size in cloudimg_sync.py when I should of been doing it in stream.py, which is used by both commands.

I noticed the output of meph2-build contains both the squashfs and root-image.gz. For images which we can get sqaushfs images from cloud-images.ubuntu.com we want to provide only the root-image.gz in the V2 stream and only the squashfs in the V3 stream. When generating the images with meph2-cloudimg-sync and pointing at the different meph2-v2.yaml and mpeh2-v3.yaml files this is the case.

Is meph2-build used in the image publishing process? Do we need a way to choose which filetypes to copy during an insert?