image-builder might be running too soon so built packages aren't yet available on the PPA index

[Ursula Junque]

We've experienced image-builder failures like that in the imagebuild_worker.log:

CalledProcessError: Command '['chroot', '/tmp/tmpM1dDqL/mountpoint', '/usr/bin/apt-get', 'install', '-y', 'software-properties']' returned non-zero exit status 100", u'exit': True, u'result': u'FAILED'

Image builder can't find the package in the PPA, even after branch_source_builder job verified the package was published in Launchpad, and only returned success after that.

<Ursinha> maybe what's going on with that ppa error is that launchpad says the package is published but it's not in the actual ppa index yet
<Ursinha> there's a small gap between packages being marked as published in launchpad ui (database) and they being published to the archive index
<Ursinha> I know this happens to the main ubuntu archive, it's probably true for ppas as well
<fginther> ohh, so the image builder may be running too soon?
<Ursinha> yes

We need to be sure the package is available on the PPA index and is downloadable (or add a delay) before triggering image_builder.

[Para Siva (psivaa)]

I'm seeing this too on imagebuilder step:
CalledProcessError: Command '['chroot', '/tmp/tmpVNABPa/mountpoint', '/usr/bin/apt-get', 'update']' returned non-zero exit status 100

I added a 5 min sleep on the lander_image_builder jenkins job, 5 min on lander_master before "/srv/lander_jenkins_sub/lander/bin/lander_merge_parameters.py --result-file params.json --service image_builder --output-file all.json --prior-file all.json" and another 5 min in cloud_image.py itself (imagebuilder) but still seeing the same error.

Image builder starting too soon could be an explanation for apt-get install to fail with 100, but I don't understand how apt-get update could fail with 100.

[Para Siva (psivaa)]

Ok, the reason for the above apt-get update failing (comment #1) is that bsb failing to build pkg (due to the changes being unsigned) but the relevant lander jenkins is reporting successful. Bug 1283186 deals with it.

[Andy Doan (doanac)]

there's a theory that this might have been caused by using unsigned packages and that

 https://bugs.launchpad.net/ubuntu-ci-services-itself/+bug/1290174

might permantely fix this. I'm going to mark this "medium" for now

[Ursula Junque (ursinha)]

Hm, IIRC all times I was able to reproduce this problem I was using signed packages.

[Paul Larson (pwlars)]

I asked Ursula about this on IRC and she said she hasn't been able to reproduce this for a while now. I've never been able to reproduce it, so I'm guessing it was fixed by some other patch at some point. If someone is able to reproduce it, please do reopen this, but I think we can close this one now.