| 2014-03-11 07:31:22 |
Evan |
bug |
|
|
added bug |
| 2014-03-11 07:57:09 |
Evan |
description |
I fed a ticket all the way through CI Core last night. However, this took quite a while to complete:
Created: Mon, 10 Mar 2014 22:38:44 +0000
Updated: Tue, 11 Mar 2014 03:10:18 +0000 |
I fed a ticket all the way through CI Core last night. However, this took quite a while to complete:
Created: Mon, 10 Mar 2014 22:38:44 +0000
Updated: Tue, 11 Mar 2014 03:10:18 +0000
About 40 minutes of that was waiting on the PPA. That we'll just have to bump build scores for (and longer term get PPAs deployed as microservices).
The real problem seems to be in the test runner.
22:41:48 INFO PROGRESS_TRIGGER: WAITING: {u'source_packages': [u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2_source.changes', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.debian.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0.orig.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.dsc'], u'progress_trigger': u'lander_master-1-bsbuilder', u'series': u'saucy', u'ppa': u'ppa:ev/ci-pool-002', u'ticket_id': u'2', u'archive': u'ppa:ci-engineering-airline/ci-archive'}
*snip*
23:03:42 INFO PROGRESS_TRIGGER: STATUS: {u'artifacts': [], u'status': {u'building': u'set([python-scipy - 0.12.0-2ubuntu2~ev2])', u'failed': u'set([])', u'pending': u'set([])'}, u'source_packages': [u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2_source.changes', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.debian.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0.orig.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.dsc'], u'ppa': u'ppa:ev/ci-pool-002'}
*snip*
23:23:55 INFO PROGRESS_TRIGGER: COMPLETED: {u'status': {u'building': u'set([])', u'failed': u'set([])', u'pending': u'set([])'}, u'source_packages': [u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2_source.changes', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.debian.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0.orig.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.dsc'], u'artifacts': [], u'ppa': u'ppa:ev/ci-pool-002', u'exit': True, u'result': u'PASSED'}
From ticket-2/lander_test_runner.log:
23:33:54 INFO root: Waiting for messages. ^C to exit.
23:33:54 INFO PROGRESS_TRIGGER: WAITING: {u'progress_trigger': u'lander_master-1-testrunner', u'image_id': u'5fb841da-a8ac-11e3-a788-fa163e39aa80.img', u'ticket_id': u'2', u'package_list': [u'python-scipy']}
23:33:54 INFO PROGRESS_TRIGGER: STATUS: {u'progress_trigger': u'lander_master-1-testrunner', u'image_id': u'5fb841da-a8ac-11e3-a788-fa163e39aa80.img', u'ticket_id': u'2', u'package_list': [u'python-scipy']}
23:36:37 INFO PROGRESS_TRIGGER: STATUS: {u'message': u'testing python-scipy'}
03:10:07 INFO PROGRESS_TRIGGER: COMPLETED: {u'artifacts': [{u'type': u'RESULTS', u'name': u'test_runner.python-scipy-subunit-stream', u'reference': u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/test_runner.python-scipy-subunit-stream'}], u'exit': True, u'result': u'PASSED'}
03:10:07 DEBUG amqplib: Closed channel #1 |
|
| 2014-03-11 08:08:33 |
Evan |
description |
I fed a ticket all the way through CI Core last night. However, this took quite a while to complete:
Created: Mon, 10 Mar 2014 22:38:44 +0000
Updated: Tue, 11 Mar 2014 03:10:18 +0000
About 40 minutes of that was waiting on the PPA. That we'll just have to bump build scores for (and longer term get PPAs deployed as microservices).
The real problem seems to be in the test runner.
22:41:48 INFO PROGRESS_TRIGGER: WAITING: {u'source_packages': [u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2_source.changes', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.debian.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0.orig.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.dsc'], u'progress_trigger': u'lander_master-1-bsbuilder', u'series': u'saucy', u'ppa': u'ppa:ev/ci-pool-002', u'ticket_id': u'2', u'archive': u'ppa:ci-engineering-airline/ci-archive'}
*snip*
23:03:42 INFO PROGRESS_TRIGGER: STATUS: {u'artifacts': [], u'status': {u'building': u'set([python-scipy - 0.12.0-2ubuntu2~ev2])', u'failed': u'set([])', u'pending': u'set([])'}, u'source_packages': [u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2_source.changes', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.debian.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0.orig.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.dsc'], u'ppa': u'ppa:ev/ci-pool-002'}
*snip*
23:23:55 INFO PROGRESS_TRIGGER: COMPLETED: {u'status': {u'building': u'set([])', u'failed': u'set([])', u'pending': u'set([])'}, u'source_packages': [u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2_source.changes', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.debian.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0.orig.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.dsc'], u'artifacts': [], u'ppa': u'ppa:ev/ci-pool-002', u'exit': True, u'result': u'PASSED'}
From ticket-2/lander_test_runner.log:
23:33:54 INFO root: Waiting for messages. ^C to exit.
23:33:54 INFO PROGRESS_TRIGGER: WAITING: {u'progress_trigger': u'lander_master-1-testrunner', u'image_id': u'5fb841da-a8ac-11e3-a788-fa163e39aa80.img', u'ticket_id': u'2', u'package_list': [u'python-scipy']}
23:33:54 INFO PROGRESS_TRIGGER: STATUS: {u'progress_trigger': u'lander_master-1-testrunner', u'image_id': u'5fb841da-a8ac-11e3-a788-fa163e39aa80.img', u'ticket_id': u'2', u'package_list': [u'python-scipy']}
23:36:37 INFO PROGRESS_TRIGGER: STATUS: {u'message': u'testing python-scipy'}
03:10:07 INFO PROGRESS_TRIGGER: COMPLETED: {u'artifacts': [{u'type': u'RESULTS', u'name': u'test_runner.python-scipy-subunit-stream', u'reference': u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/test_runner.python-scipy-subunit-stream'}], u'exit': True, u'result': u'PASSED'}
03:10:07 DEBUG amqplib: Closed channel #1 |
I fed a ticket all the way through CI Core last night. However, this took quite a while to complete:
Created: Mon, 10 Mar 2014 22:38:44 +0000
Updated: Tue, 11 Mar 2014 03:10:18 +0000
About 40 minutes of that was waiting on the PPA. That we'll just have to bump build scores for (and longer term get PPAs deployed as microservices):
22:41:48 INFO PROGRESS_TRIGGER: WAITING: {u'source_packages': [u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2_source.changes', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.debian.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0.orig.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.dsc'], u'progress_trigger': u'lander_master-1-bsbuilder', u'series': u'saucy', u'ppa': u'ppa:ev/ci-pool-002', u'ticket_id': u'2', u'archive': u'ppa:ci-engineering-airline/ci-archive'}
*snip*
23:03:42 INFO PROGRESS_TRIGGER: STATUS: {u'artifacts': [], u'status': {u'building': u'set([python-scipy - 0.12.0-2ubuntu2~ev2])', u'failed': u'set([])', u'pending': u'set([])'}, u'source_packages': [u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2_source.changes', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.debian.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0.orig.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.dsc'], u'ppa': u'ppa:ev/ci-pool-002'}
*snip*
23:23:55 INFO PROGRESS_TRIGGER: COMPLETED: {u'status': {u'building': u'set([])', u'failed': u'set([])', u'pending': u'set([])'}, u'source_packages': [u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2_source.changes', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.debian.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0.orig.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.dsc'], u'artifacts': [], u'ppa': u'ppa:ev/ci-pool-002', u'exit': True, u'result': u'PASSED'}
The real problem seems to be in the test runner.
Part of this may just be that the scipy tests take ages to run on an m1.small instance. If that's the case, we should make the default instance size for the test runner larger.
http://paste.ubuntu.com/7072218/
It looks like it took 94.5846 minutes to complete the tests:
sudo cat /var/log/upstart/tr_rabbit_worker.log | grep "tests in " | sed 's/.* //' | sed 's/s//' | awk '{ sum += $1 } END { print sum / 60.0 }'
From ticket-2/lander_test_runner.log:
23:33:54 INFO root: Waiting for messages. ^C to exit.
23:33:54 INFO PROGRESS_TRIGGER: WAITING: {u'progress_trigger': u'lander_master-1-testrunner', u'image_id': u'5fb841da-a8ac-11e3-a788-fa163e39aa80.img', u'ticket_id': u'2', u'package_list': [u'python-scipy']}
23:33:54 INFO PROGRESS_TRIGGER: STATUS: {u'progress_trigger': u'lander_master-1-testrunner', u'image_id': u'5fb841da-a8ac-11e3-a788-fa163e39aa80.img', u'ticket_id': u'2', u'package_list': [u'python-scipy']}
23:36:37 INFO PROGRESS_TRIGGER: STATUS: {u'message': u'testing python-scipy'}
03:10:07 INFO PROGRESS_TRIGGER: COMPLETED: {u'artifacts': [{u'type': u'RESULTS', u'name': u'test_runner.python-scipy-subunit-stream', u'reference': u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/test_runner.python-scipy-subunit-stream'}], u'exit': True, u'result': u'PASSED'}
03:10:07 DEBUG amqplib: Closed channel #1 |
|
| 2014-03-11 08:09:26 |
Evan |
description |
I fed a ticket all the way through CI Core last night. However, this took quite a while to complete:
Created: Mon, 10 Mar 2014 22:38:44 +0000
Updated: Tue, 11 Mar 2014 03:10:18 +0000
About 40 minutes of that was waiting on the PPA. That we'll just have to bump build scores for (and longer term get PPAs deployed as microservices):
22:41:48 INFO PROGRESS_TRIGGER: WAITING: {u'source_packages': [u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2_source.changes', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.debian.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0.orig.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.dsc'], u'progress_trigger': u'lander_master-1-bsbuilder', u'series': u'saucy', u'ppa': u'ppa:ev/ci-pool-002', u'ticket_id': u'2', u'archive': u'ppa:ci-engineering-airline/ci-archive'}
*snip*
23:03:42 INFO PROGRESS_TRIGGER: STATUS: {u'artifacts': [], u'status': {u'building': u'set([python-scipy - 0.12.0-2ubuntu2~ev2])', u'failed': u'set([])', u'pending': u'set([])'}, u'source_packages': [u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2_source.changes', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.debian.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0.orig.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.dsc'], u'ppa': u'ppa:ev/ci-pool-002'}
*snip*
23:23:55 INFO PROGRESS_TRIGGER: COMPLETED: {u'status': {u'building': u'set([])', u'failed': u'set([])', u'pending': u'set([])'}, u'source_packages': [u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2_source.changes', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.debian.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0.orig.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.dsc'], u'artifacts': [], u'ppa': u'ppa:ev/ci-pool-002', u'exit': True, u'result': u'PASSED'}
The real problem seems to be in the test runner.
Part of this may just be that the scipy tests take ages to run on an m1.small instance. If that's the case, we should make the default instance size for the test runner larger.
http://paste.ubuntu.com/7072218/
It looks like it took 94.5846 minutes to complete the tests:
sudo cat /var/log/upstart/tr_rabbit_worker.log | grep "tests in " | sed 's/.* //' | sed 's/s//' | awk '{ sum += $1 } END { print sum / 60.0 }'
From ticket-2/lander_test_runner.log:
23:33:54 INFO root: Waiting for messages. ^C to exit.
23:33:54 INFO PROGRESS_TRIGGER: WAITING: {u'progress_trigger': u'lander_master-1-testrunner', u'image_id': u'5fb841da-a8ac-11e3-a788-fa163e39aa80.img', u'ticket_id': u'2', u'package_list': [u'python-scipy']}
23:33:54 INFO PROGRESS_TRIGGER: STATUS: {u'progress_trigger': u'lander_master-1-testrunner', u'image_id': u'5fb841da-a8ac-11e3-a788-fa163e39aa80.img', u'ticket_id': u'2', u'package_list': [u'python-scipy']}
23:36:37 INFO PROGRESS_TRIGGER: STATUS: {u'message': u'testing python-scipy'}
03:10:07 INFO PROGRESS_TRIGGER: COMPLETED: {u'artifacts': [{u'type': u'RESULTS', u'name': u'test_runner.python-scipy-subunit-stream', u'reference': u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/test_runner.python-scipy-subunit-stream'}], u'exit': True, u'result': u'PASSED'}
03:10:07 DEBUG amqplib: Closed channel #1 |
I fed a ticket all the way through CI Core last night. However, this took quite a while to complete:
Created: Mon, 10 Mar 2014 22:38:44 +0000
Updated: Tue, 11 Mar 2014 03:10:18 +0000
About 40 minutes of that was waiting on the PPA. That we'll just have to bump build scores for (and longer term get PPAs deployed as microservices):
22:41:48 INFO PROGRESS_TRIGGER: WAITING: {u'source_packages': [u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2_source.changes', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.debian.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0.orig.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.dsc'], u'progress_trigger': u'lander_master-1-bsbuilder', u'series': u'saucy', u'ppa': u'ppa:ev/ci-pool-002', u'ticket_id': u'2', u'archive': u'ppa:ci-engineering-airline/ci-archive'}
*snip*
23:03:42 INFO PROGRESS_TRIGGER: STATUS: {u'artifacts': [], u'status': {u'building': u'set([python-scipy - 0.12.0-2ubuntu2~ev2])', u'failed': u'set([])', u'pending': u'set([])'}, u'source_packages': [u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2_source.changes', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.debian.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0.orig.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.dsc'], u'ppa': u'ppa:ev/ci-pool-002'}
*snip*
23:23:55 INFO PROGRESS_TRIGGER: COMPLETED: {u'status': {u'building': u'set([])', u'failed': u'set([])', u'pending': u'set([])'}, u'source_packages': [u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2_source.changes', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.debian.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0.orig.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.dsc'], u'artifacts': [], u'ppa': u'ppa:ev/ci-pool-002', u'exit': True, u'result': u'PASSED'}
The real problem seems to be in the test runner.
Part of this may just be that the scipy tests take ages to run on an m1.small instance. If that's the case, we should give the default instance for the test runner multiple cores.
http://paste.ubuntu.com/7072218/
It looks like it took 94.5846 minutes to complete the tests:
sudo cat /var/log/upstart/tr_rabbit_worker.log | grep "tests in " | sed 's/.* //' | sed 's/s//' | awk '{ sum += $1 } END { print sum / 60.0 }'
From ticket-2/lander_test_runner.log:
23:33:54 INFO root: Waiting for messages. ^C to exit.
23:33:54 INFO PROGRESS_TRIGGER: WAITING: {u'progress_trigger': u'lander_master-1-testrunner', u'image_id': u'5fb841da-a8ac-11e3-a788-fa163e39aa80.img', u'ticket_id': u'2', u'package_list': [u'python-scipy']}
23:33:54 INFO PROGRESS_TRIGGER: STATUS: {u'progress_trigger': u'lander_master-1-testrunner', u'image_id': u'5fb841da-a8ac-11e3-a788-fa163e39aa80.img', u'ticket_id': u'2', u'package_list': [u'python-scipy']}
23:36:37 INFO PROGRESS_TRIGGER: STATUS: {u'message': u'testing python-scipy'}
03:10:07 INFO PROGRESS_TRIGGER: COMPLETED: {u'artifacts': [{u'type': u'RESULTS', u'name': u'test_runner.python-scipy-subunit-stream', u'reference': u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/test_runner.python-scipy-subunit-stream'}], u'exit': True, u'result': u'PASSED'}
03:10:07 DEBUG amqplib: Closed channel #1 |
|
| 2014-03-11 08:32:08 |
Evan |
summary |
Tickets take a very long time to successfully complete |
Tickets take 4.5 hours to successfully complete |
|
| 2014-03-11 08:43:34 |
Evan |
ubuntu-ci-services-itself: importance |
Undecided |
Medium |
|
| 2014-03-11 08:51:03 |
Evan |
ubuntu-ci-services-itself: assignee |
|
Parameswaran Sivatharman (psivaa) |
|
| 2014-03-11 10:27:35 |
Evan |
description |
I fed a ticket all the way through CI Core last night. However, this took quite a while to complete:
Created: Mon, 10 Mar 2014 22:38:44 +0000
Updated: Tue, 11 Mar 2014 03:10:18 +0000
About 40 minutes of that was waiting on the PPA. That we'll just have to bump build scores for (and longer term get PPAs deployed as microservices):
22:41:48 INFO PROGRESS_TRIGGER: WAITING: {u'source_packages': [u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2_source.changes', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.debian.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0.orig.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.dsc'], u'progress_trigger': u'lander_master-1-bsbuilder', u'series': u'saucy', u'ppa': u'ppa:ev/ci-pool-002', u'ticket_id': u'2', u'archive': u'ppa:ci-engineering-airline/ci-archive'}
*snip*
23:03:42 INFO PROGRESS_TRIGGER: STATUS: {u'artifacts': [], u'status': {u'building': u'set([python-scipy - 0.12.0-2ubuntu2~ev2])', u'failed': u'set([])', u'pending': u'set([])'}, u'source_packages': [u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2_source.changes', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.debian.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0.orig.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.dsc'], u'ppa': u'ppa:ev/ci-pool-002'}
*snip*
23:23:55 INFO PROGRESS_TRIGGER: COMPLETED: {u'status': {u'building': u'set([])', u'failed': u'set([])', u'pending': u'set([])'}, u'source_packages': [u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2_source.changes', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.debian.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0.orig.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.dsc'], u'artifacts': [], u'ppa': u'ppa:ev/ci-pool-002', u'exit': True, u'result': u'PASSED'}
The real problem seems to be in the test runner.
Part of this may just be that the scipy tests take ages to run on an m1.small instance. If that's the case, we should give the default instance for the test runner multiple cores.
http://paste.ubuntu.com/7072218/
It looks like it took 94.5846 minutes to complete the tests:
sudo cat /var/log/upstart/tr_rabbit_worker.log | grep "tests in " | sed 's/.* //' | sed 's/s//' | awk '{ sum += $1 } END { print sum / 60.0 }'
From ticket-2/lander_test_runner.log:
23:33:54 INFO root: Waiting for messages. ^C to exit.
23:33:54 INFO PROGRESS_TRIGGER: WAITING: {u'progress_trigger': u'lander_master-1-testrunner', u'image_id': u'5fb841da-a8ac-11e3-a788-fa163e39aa80.img', u'ticket_id': u'2', u'package_list': [u'python-scipy']}
23:33:54 INFO PROGRESS_TRIGGER: STATUS: {u'progress_trigger': u'lander_master-1-testrunner', u'image_id': u'5fb841da-a8ac-11e3-a788-fa163e39aa80.img', u'ticket_id': u'2', u'package_list': [u'python-scipy']}
23:36:37 INFO PROGRESS_TRIGGER: STATUS: {u'message': u'testing python-scipy'}
03:10:07 INFO PROGRESS_TRIGGER: COMPLETED: {u'artifacts': [{u'type': u'RESULTS', u'name': u'test_runner.python-scipy-subunit-stream', u'reference': u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/test_runner.python-scipy-subunit-stream'}], u'exit': True, u'result': u'PASSED'}
03:10:07 DEBUG amqplib: Closed channel #1 |
tl;dr we need to chip away at all the places where we're moving slowly because the instances are too small (make them bigger), are polling too infrequently (going from 1 min down to 15 seconds, etc), are blocked on builds (get PPA scores increased), etc.
--
I fed a ticket all the way through CI Core last night. However, this took quite a while to complete:
Created: Mon, 10 Mar 2014 22:38:44 +0000
Updated: Tue, 11 Mar 2014 03:10:18 +0000
About 40 minutes of that was waiting on the PPA. That we'll just have to bump build scores for (and longer term get PPAs deployed as microservices):
22:41:48 INFO PROGRESS_TRIGGER: WAITING: {u'source_packages': [u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2_source.changes', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.debian.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0.orig.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.dsc'], u'progress_trigger': u'lander_master-1-bsbuilder', u'series': u'saucy', u'ppa': u'ppa:ev/ci-pool-002', u'ticket_id': u'2', u'archive': u'ppa:ci-engineering-airline/ci-archive'}
*snip*
23:03:42 INFO PROGRESS_TRIGGER: STATUS: {u'artifacts': [], u'status': {u'building': u'set([python-scipy - 0.12.0-2ubuntu2~ev2])', u'failed': u'set([])', u'pending': u'set([])'}, u'source_packages': [u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2_source.changes', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.debian.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0.orig.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.dsc'], u'ppa': u'ppa:ev/ci-pool-002'}
*snip*
23:23:55 INFO PROGRESS_TRIGGER: COMPLETED: {u'status': {u'building': u'set([])', u'failed': u'set([])', u'pending': u'set([])'}, u'source_packages': [u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2_source.changes', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.debian.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0.orig.tar.gz', u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/python-scipy_0.12.0-2ubuntu2~ev2.dsc'], u'artifacts': [], u'ppa': u'ppa:ev/ci-pool-002', u'exit': True, u'result': u'PASSED'}
The real problem seems to be in the test runner.
Part of this may just be that the scipy tests take ages to run on an m1.small instance. If that's the case, we should give the default instance for the test runner multiple cores.
http://paste.ubuntu.com/7072218/
It looks like it took 94.5846 minutes to complete the tests:
sudo cat /var/log/upstart/tr_rabbit_worker.log | grep "tests in " | sed 's/.* //' | sed 's/s//' | awk '{ sum += $1 } END { print sum / 60.0 }'
From ticket-2/lander_test_runner.log:
23:33:54 INFO root: Waiting for messages. ^C to exit.
23:33:54 INFO PROGRESS_TRIGGER: WAITING: {u'progress_trigger': u'lander_master-1-testrunner', u'image_id': u'5fb841da-a8ac-11e3-a788-fa163e39aa80.img', u'ticket_id': u'2', u'package_list': [u'python-scipy']}
23:33:54 INFO PROGRESS_TRIGGER: STATUS: {u'progress_trigger': u'lander_master-1-testrunner', u'image_id': u'5fb841da-a8ac-11e3-a788-fa163e39aa80.img', u'ticket_id': u'2', u'package_list': [u'python-scipy']}
23:36:37 INFO PROGRESS_TRIGGER: STATUS: {u'message': u'testing python-scipy'}
03:10:07 INFO PROGRESS_TRIGGER: COMPLETED: {u'artifacts': [{u'type': u'RESULTS', u'name': u'test_runner.python-scipy-subunit-stream', u'reference': u'https://region-a.geo-1.objects.hpcloudsvc.com/v1/11597203075020/ticket-2/test_runner.python-scipy-subunit-stream'}], u'exit': True, u'result': u'PASSED'}
03:10:07 DEBUG amqplib: Closed channel #1 |
|
| 2014-03-13 09:11:38 |
Evan |
ubuntu-ci-services-itself: status |
New |
Confirmed |
|
| 2014-03-13 13:06:51 |
Para Siva |
branch linked |
|
lp:~psivaa/ubuntu-ci-services-itself/speedup-imagebuild-tr |
|
| 2014-03-13 15:32:23 |
Andy Doan |
ubuntu-ci-services-itself: status |
Confirmed |
In Progress |
|
| 2014-03-13 15:33:38 |
Para Siva |
branch linked |
|
lp:~fginther/ubuntu-ci-services-itself/reduce-polling-times |
|
| 2014-03-21 20:41:41 |
Evan |
ubuntu-ci-services-itself: milestone |
phase-0 |
phase-0.5 |
|
| 2014-03-26 18:25:19 |
Ursula Junque |
uci-engine: importance |
Undecided |
Medium |
|
| 2014-03-26 18:25:19 |
Ursula Junque |
uci-engine: status |
New |
In Progress |
|
| 2014-03-26 18:25:19 |
Ursula Junque |
uci-engine: milestone |
|
phase-0.5 |
|
| 2014-03-26 18:25:19 |
Ursula Junque |
uci-engine: assignee |
|
Parameswaran Sivatharman (psivaa) |
|
| 2014-03-31 14:04:26 |
Evan |
bug task deleted |
ubuntu-ci-services-itself |
|
|
| 2016-01-04 12:43:28 |
Para Siva |
uci-engine: assignee |
Parameswaran Sivatharman (psivaa) |
|
|
