meph2-util insert attempts to insert all items

[Scott Moser]

One of the maas image build failed like this (full log attached):
 $ meph2-util insert MAAS_i386_BUILD_OUTPUT DAILY_STREAM --keyring $KEYRING
  content_id=com.ubuntu.maas:daily:v2:download path=streams/v1/com.ubuntu.maas:daily:v2:download.json
inserting item com.ubuntu.maas.daily:v2:boot:15.04:i386:hwe-v/20150307.4/di-kernel
inserting item com.ubuntu.maas.daily:v2:boot:15.04:i386:hwe-v/20150307.4/boot-kernel
Traceback (most recent call last):
  File "/var/lib/jenkins/jobs/MAAS_v2-15.04/workspace/maasv2_branch/bin/meph2-util", line 25, in <module>
    call_entry_point("meph2.commands.meph2_util.main")
  File "/var/lib/jenkins/jobs/MAAS_v2-15.04/workspace/maasv2_branch/bin/meph2-util", line 22, in call_entry_point
    sys.exit(getattr(sys.modules[istr], ent)())
  File "/var/lib/jenkins/jobs/MAAS_v2-15.04/workspace/maasv2_branch/meph2/commands/meph2_util.py", line 474, in main
    return args.action(args)
  File "/var/lib/jenkins/jobs/MAAS_v2-15.04/workspace/maasv2_branch/meph2/commands/meph2_util.py", line 303, in main_insert
    tmirror.sync(smirror, src_path)
  File "/var/lib/jenkins/jobs/MAAS_v2-15.04/workspace/sstreams/simplestreams/mirrors/__init__.py", line 88, in sync
    return self.sync_index(reader, path, data, content)
  File "/var/lib/jenkins/jobs/MAAS_v2-15.04/workspace/sstreams/simplestreams/mirrors/__init__.py", line 236, in sync_index
    self.sync(reader, path=epath)
  File "/var/lib/jenkins/jobs/MAAS_v2-15.04/workspace/sstreams/simplestreams/mirrors/__init__.py", line 86, in sync
    return self.sync_products(reader, path, data, content)
  File "/var/lib/jenkins/jobs/MAAS_v2-15.04/workspace/sstreams/simplestreams/mirrors/__init__.py", line 315, in sync_products
    self.insert_item(item, src, target, pgree, ipath_cs)
  File "/var/lib/jenkins/jobs/MAAS_v2-15.04/workspace/maasv2_branch/meph2/commands/meph2_util.py", line 137, in insert_item
    data, src, target, pedigree, contentsource)
  File "/var/lib/jenkins/jobs/MAAS_v2-15.04/workspace/sstreams/simplestreams/mirrors/__init__.py", line 450, in insert_item
    size=data.get('size'))
  File "/var/lib/jenkins/jobs/MAAS_v2-15.04/workspace/sstreams/simplestreams/objectstores/__init__.py", line 133, in insert
    buf = reader.read(self.read_size)
  File "/var/lib/jenkins/jobs/MAAS_v2-15.04/workspace/sstreams/simplestreams/contentsource.py", line 140, in read
    self.open()
  File "/var/lib/jenkins/jobs/MAAS_v2-15.04/workspace/sstreams/simplestreams/contentsource.py", line 136, in open
    self.fd = self._open()
  File "/var/lib/jenkins/jobs/MAAS_v2-15.04/workspace/sstreams/simplestreams/contentsource.py", line 132, in _open
    raise myerr
IOError: Unable to open /var/lib/jenkins/jobs/MAAS_v2-15.04/workspace/maasv2/amd64/tmp/maas_final/vivid/i386/20150307.4/vivid/generic/boot-kernel. mirrors=[]
+ fail 'failed to insert stream for amd64'

My belief on what happened there was that the build built a serial numbered 20150314.1 (or something like that). But during the image build process, and collection of the data back to the maas build system, a 'clean' cleaned out the 20150307.4.

Thus, the reference to build 20150307.4 lived in the stream output from the build, but not in the target. And as a result, the sync tried copy from the stream output to the target. However, the stream output is a sparse output (it only contains what it built).

So, the fix here would be possibly to have it operate in a way that only inserted the most recent things.

[Jack Lloyd-Walters (lloydwaltersj)]

Closing as we haven't had anyone complain about this for a few years.