I uploaded a source package to the ticket system. It built in the PPA fine, but failed at the image builder. There was no feedback beyond "Image Building Failed" in the UI, which is a bit terse. Ideally this would link off to some build logs as artifacts in Swift.
Also, there's a bug in here in looking for trusty packages when I uploaded a saucy package.
From the webui:
Workflow Step: Failed Status: Image Building Failed
From the lander:
Started by upstream project "lander_master" build number 1
originally caused by:
Started by remote host 10.55.61.77
Building in workspace /var/lib/jenkins/jobs/lander_image_builder/workspace
[workspace] $ /bin/bash /tmp/hudson425732495885170659.sh
+ '[' -d results ']'
+ mkdir results
+ echo '{"bsbuilder":' '{"status":' '{"building":' '"set([])",' '"failed":' '"set([])",' '"pending":' '"set([])"},' '"source_packages":' '["https://swift.canonistack.canonical.com/v1/AUTH_c0136db8dc454f2d8523d9326bca2d78/ticket-system.1//tmp/whoopsie_0.2.24.4~ev2_source.changes",' '"https://swift.canonistack.canonical.com/v1/AUTH_c0136db8dc454f2d8523d9326bca2d78/ticket-system.1//tmp/whoopsie_0.2.24.4~ev2.tar.gz",' '"https://swift.canonistack.canonical.com/v1/AUTH_c0136db8dc454f2d8523d9326bca2d78/ticket-system.1//tmp/whoopsie_0.2.24.4~ev2.dsc"],' '"artifacts":' '[],' '"ppa":' '"ppa:ev/ci-pool-002",' '"state":' '"COMPLETED",' '"exit":' true, '"result":' '"PASSED"},' '"master":' '{"base_image":' '{"image_type":' '"cloud",' '"series":' '"trusty",' '"url_list":' '["http://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img"]},' '"ppa_assigner_url":' '"http://10.55.61.78:8080",' '"progress_trigger":' '"lander_master-1",' '"imgbuilder_url":' '"http://10.55.60.223:8080",' '"tr_url":' '"http://10.55.61.87:8080",' '"package_list":' '["phablet-tools"],' '"bsb_url":' '"http://10.55.60.133:8080",' '"request_id":' '"1",' '"ts_url":' '"http://10.55.61.88:8080",' '"request_parameters":' '{"status":' '"Not' 'started",' '"added_binaries":' null, '"updated":' '"Tue,' 25 Feb 2014 14:08:07 '+0000",' '"current_workflow_step":' '"Queued",' '"description":' '"A' ticket to 'test.",' '"created":' '"Tue,' 25 Feb 2014 14:08:01 '+0000",' '"series":' '"SAUCY",' '"subticket":' '[{"status":' '"New",' '"current_workflow_step":' '"New",' '"artifact":' '[{"id":' 3, '"type":' '"SPU",' '"name":' '"whoopsie_0.2.24.4~ev2_source.changes",' '"reference":' '"https://swift.canonistack.canonical.com/v1/AUTH_c0136db8dc454f2d8523d9326bca2d78/ticket-system.1//tmp/whoopsie_0.2.24.4~ev2_source.changes",' '"resource_uri":' '"/api/v1/fullsubticketartifact/3/"},' '{"id":' 2, '"type":' '"SPU",' '"name":' '"whoopsie_0.2.24.4~ev2.tar.gz",' '"reference":' '"https://swift.canonistack.canonical.com/v1/AUTH_c0136db8dc454f2d8523d9326bca2d78/ticket-system.1//tmp/whoopsie_0.2.24.4~ev2.tar.gz",' '"resource_uri":' '"/api/v1/fullsubticketartifact/2/"},' '{"id":' 1, '"type":' '"SPU",' '"name":' '"whoopsie_0.2.24.4~ev2.dsc",' '"reference":' '"https://swift.canonistack.canonical.com/v1/AUTH_c0136db8dc454f2d8523d9326bca2d78/ticket-system.1//tmp/whoopsie_0.2.24.4~ev2.dsc",' '"resource_uri":' '"/api/v1/fullsubticketartifact/1/"}],' '"assignee":' '"ev",' '"source_package_upload":' '{"version":' '"0.2.24.4~ev2",' '"sourcepackage":' '{"id":' 1, '"name":' '"whoopsie",' '"resource_uri":' '"/api/v1/sourcepackage/1/"},' '"id":' 1, '"resource_uri":' '"/api/v1/spu/1/"},' '"id":' 1, '"resource_uri":' '"/api/v1/fullsubticket/1/"}],' '"bug_id":' null, '"master_ppa":' '"ppa:ci-engineering-airline/ci-archive",' '"owner":' '"ev",' '"title":' '"Testing' 'ticket",' '"base_image":' '"http://cloud-images.ubuntu.com/releases/13.10/release-20131015/ubuntu-13.10-server-cloudimg-amd64-disk1.img",' '"removed_binaries":' null, '"id":' 1, '"resource_uri":' '"/api/v1/next/1/"}},' '"ppa_assigner":' '{"ppa":' '"ppa:ev/ci-pool-002",' '"result":' '"PASSED",' '"location":' '"http://10.55.61.78:8080/api/v1/ppa/1/"}}'
+ /srv/lander_jenkins_sub/lander/bin/lander_service_wrapper.py --input-file params.json --output-file results/params.json --service image_builder
14:28:44 DEBUG lander_service_wrapper: Executing service wrapper: image_builder
14:28:44 INFO lander_service_wrapper: starting progress handler...
14:28:44 DEBUG amqplib: Start from server, version: 8.0, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2011 VMware, Inc.', u'capabilities': {}, u'platform': u'Erlang/OTP', u'version': u'2.7.1'}, mechanisms: [u'PLAIN', u'AMQPLAIN'], locales: [u'en_US']
14:28:44 DEBUG amqplib: Open OK! known_hosts []
14:28:44 DEBUG amqplib: using channel_id: 1
14:28:44 DEBUG amqplib: Channel open
14:28:44 INFO root: Waiting for messages. ^C to exit.
14:28:44 INFO PROGRESS_TRIGGER: WAITING: {u'progress_trigger': u'lander_master-1-imgbuilder', u'series': u'trusty', u'image': u'http://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img', u'ppa_list': [u'ppa:ev/ci-pool-002'], u'package_list': [u'phablet-tools']}
14:28:44 INFO PROGRESS_TRIGGER: STATUS: {u'progress_trigger': u'lander_master-1-imgbuilder', u'series': u'trusty', u'image': u'http://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img', u'ppa_list': [u'ppa:ev/ci-pool-002'], u'package_list': [u'phablet-tools']}
14:28:44 INFO PROGRESS_TRIGGER: STATUS: {u'message': u'loading nbd...'}
14:28:44 INFO PROGRESS_TRIGGER: STATUS: {u'message': u'downloading http://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img...'}
14:28:55 INFO PROGRESS_TRIGGER: STATUS: {u'message': u'updating image http://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img...'}
14:28:56 INFO PROGRESS_TRIGGER: STATUS: {u'message': u'Adding requested packages to the image...'}
14:30:11 INFO PROGRESS_TRIGGER: COMPLETED: {u'message': u'', u'exit': True, u'result': u'FAILED'}
14:30:11 DEBUG amqplib: Closed channel #1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Triggering a new build of lander_archiver #2
Finished: FAILURE
From imagebuild-worker:
Traceback (most recent call last):
File "./image-builder/run_worker", line 69, in <module>
amqp_utils.process_queue(config, 'imagebuilder', on_message)
File "./image-builder/../ci-utils/ci_utils/amqp_utils.py", line 101, in process_queue
conn = connection(config)
File "./image-builder/../ci-utils/ci_utils/amqp_utils.py", line 48, in connection
password=rabbit_config.AMQP_PASSWORD
File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/connection.py", line 144, in __init__
(10, 30), # tune
File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 95, in wait
self.channel_id, allowed_methods)
File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/connection.py", line 202, in _wait_method
self.method_reader.read_method()
File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 221, in read_method
raise m
IOError: Socket closed
INFO:root:Waiting for messages. ^C to exit.
INFO:__main__:on_message: {"progress_trigger": "lander_master-1-imgbuilder", "series": "trusty", "image": "http://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img", "ppa_list": ["ppa:ev/ci-pool-002"], "package_list": ["phablet-tools"], "state": "WAITING"}
W: Failed to fetch http://ppa.launchpad.net/ev/ci-pool-002/ubuntu/dists/trusty/main/source/Sources 404 Not Found
E: Some index files failed to download. They have been ignored, or old ones used instead.
/dev/nbd0 disconnected
ERROR:__main__:Image build failed:
Traceback (most recent call last):
File "./image-builder/run_worker", line 49, in on_message
image_id = modify_image(image, repos, series, packages, status_cb)
File "/srv/imagebuild_worker/image-builder/imagebuilder/cloud_image.py", line 183, in modify_image
_run_chroot_cmds(mountpoint, ppalist, packages)
File "/srv/imagebuild_worker/image-builder/imagebuilder/cloud_image.py", line 78, in _run_chroot_cmds
_chrooted(chroot, "/usr/bin/apt-get update")
File "/srv/imagebuild_worker/image-builder/imagebuilder/cloud_image.py", line 67, in _chrooted
subprocess.check_output(['chroot', chroot] + cmd.split(" "))
File "/usr/lib/python2.7/subprocess.py", line 544, in check_output
raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command '['chroot', '/tmp/tmpnnqRhb/mountpoint', '/usr/bin/apt-get', 'update']' returned non-zero exit status 100
The linked branch contains what I consider a partial/simple fix to this. The branch will make it log the command that was attempted when the failure happened in the jenkins log looking something like this:
04:38:59 INFO PROGRESS_TRIGGER: STATUS: {u'message': u'Adding requested packages to the image...'}
04:39:22 INFO PROGRESS_TRIGGER: COMPLETED: {u'message': u"CalledProcess
I'm testing out another branch right now too, which will also create an artifact with the full output of everything that happened during the imagebuilder process, and store it in swift. This can be linked to for more complete information. I wanted to get this one in there quickly though, as I think it's useful by itself, as well as in addition to the other patch.