juju-core

ensure-availability takes more than 20 minutes to complete

Bug #1384275 reported by Aaron Bentley
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
juju-core
Triaged
Medium
Unassigned

Bug Description

Most runs of http://juju-ci.vapour.ws:8080/job/functional-ha-recovery succeed, but recently we had one that took more than 20 minutes. (our timeout is 20 minutes).

Started by user admin
Building on master
[workspace] $ /bin/bash /tmp/hudson5573133040716020553.sh
++ rm /var/lib/jenkins/jobs/functional-ha-recovery/workspace/artifacts /var/lib/jenkins/jobs/functional-ha-recovery/workspace/buildvars.bash /var/lib/jenkins/jobs/functional-ha-recovery/workspace/extracted-bin /var/lib/jenkins/jobs/functional-ha-recovery/workspace/juju-core_1.21-alpha2-0ubuntu1~12.04.1~juju1_amd64.deb /var/lib/jenkins/jobs/functional-ha-recovery/workspace/juju-local_1.21-alpha2-0ubuntu1~12.04.1~juju1_all.deb -rf
++ mkdir -p /var/lib/jenkins/jobs/functional-ha-recovery/workspace/artifacts
++ touch /var/lib/jenkins/jobs/functional-ha-recovery/workspace/artifacts/empty
++ afact=lastSuccessfulBuild/artifact
+++ lsb_release -sr
++ RELEASE=12.04
+++ dpkg --print-architecture
++ ARCH=amd64
++ [[ -n 1948 ]]
++ wget -q http://juju-ci.vapour.ws:8080/job/build-revision/lastSuccessfulBuild/artifact/buildvars.bash
++ source buildvars.bash
+++ export BRANCH=gitbranch:master:github.com/juju/juju
+++ BRANCH=gitbranch:master:github.com/juju/juju
+++ export REVISION_ID=b4a1e09b75c4def4f8326197f1a46e062dd52a84
+++ REVISION_ID=b4a1e09b75c4def4f8326197f1a46e062dd52a84
+++ export VERSION=1.21-alpha2
+++ VERSION=1.21-alpha2
++ PACKAGES_JOB=publish-revision
++ JUJU_LOCAL_DEB=juju-local_1.21-alpha2-0ubuntu1~12.04.1~juju1_all.deb
++ JUJU_CORE_DEB=juju-core_1.21-alpha2-0ubuntu1~12.04.1~juju1_amd64.deb
+++ head -c8
+++ echo b4a1e09b75c4def4f8326197f1a46e062dd52a84
++ rev=b4a1e09b
++ echo 'Testing gitbranch:master:github.com/juju/juju b4a1e09b on test-function-hp-2'
Testing gitbranch:master:github.com/juju/juju b4a1e09b on test-function-hp-2
+++ sed 's,[+],%2B,'
+++ echo juju-local_1.21-alpha2-0ubuntu1~12.04.1~juju1_all.deb
++ ENCODED_LOCAL_DEB=juju-local_1.21-alpha2-0ubuntu1~12.04.1~juju1_all.deb
+++ sed 's,[+],%2B,'
+++ echo juju-core_1.21-alpha2-0ubuntu1~12.04.1~juju1_amd64.deb
++ ENCODED_CORE_DEB=juju-core_1.21-alpha2-0ubuntu1~12.04.1~juju1_amd64.deb
++ wget -q http://juju-ci.vapour.ws:8080/job/publish-revision/lastSuccessfulBuild/artifact/juju-local_1.21-alpha2-0ubuntu1~12.04.1~juju1_all.deb
++ wget -q http://juju-ci.vapour.ws:8080/job/publish-revision/lastSuccessfulBuild/artifact/juju-core_1.21-alpha2-0ubuntu1~12.04.1~juju1_amd64.deb
++ dpkg-deb -x /var/lib/jenkins/jobs/functional-ha-recovery/workspace/juju-core_1.21-alpha2-0ubuntu1~12.04.1~juju1_amd64.deb extracted-bin
+++++ find extracted-bin -name juju
++++ dirname extracted-bin/usr/lib/juju-1.21-alpha2/bin/juju
+++ readlink -f extracted-bin/usr/lib/juju-1.21-alpha2/bin
++ export NEW_JUJU_BIN=/mnt/jenkinshome/jobs/functional-ha-recovery/workspace/extracted-bin/usr/lib/juju-1.21-alpha2/bin
++ NEW_JUJU_BIN=/mnt/jenkinshome/jobs/functional-ha-recovery/workspace/extracted-bin/usr/lib/juju-1.21-alpha2/bin
++ '[' test-function-hp-2 == manual ']'
++ jenv=/var/lib/jenkins/cloud-city/environments/test-function-hp-2.jenv
++ [[ -e /var/lib/jenkins/cloud-city/environments/test-function-hp-2.jenv ]]
++ jenv=/var/lib/jenkins/cloud-city/environments/functional-ha-recovery.jenv
++ [[ -e /var/lib/jenkins/cloud-city/environments/functional-ha-recovery.jenv ]]
+++ find /var/lib/jenkins/cloud-city/environments -name '$JOB_NAME*.jenv'
++ azure_jenvs=
+ timeout -s INT 90m assess_recovery.py --ha --charm-prefix=local:precise/ /mnt/jenkinshome/jobs/functional-ha-recovery/workspace/extracted-bin/usr/lib/juju-1.21-alpha2/bin test-function-hp-2
juju --show-log bootstrap -e test-function-hp-2 --constraints mem=2G
2014-10-22 03:41:51 INFO juju.cmd supercommand.go:37 running juju [1.21-alpha2-precise-amd64 gc]
2014-10-22 03:41:52 INFO juju.provider.openstack provider.go:248 opening environment "test-function-hp-2"
2014-10-22 03:41:53 INFO juju.network network.go:102 setting prefer-ipv6 to false
2014-10-22 03:41:53 INFO juju.utils http.go:59 hostname SSL verification enabled
2014-10-22 03:41:53 INFO juju.utils http.go:59 hostname SSL verification enabled
2014-10-22 03:41:53 INFO juju.utils http.go:59 hostname SSL verification enabled
Bootstrapping environment "test-function-hp-2"
2014-10-22 03:41:54 INFO juju.environs.tools tools.go:86 reading tools with major.minor version 1.21
2014-10-22 03:41:54 INFO juju.utils http.go:59 hostname SSL verification enabled
2014-10-22 03:41:55 INFO juju.utils http.go:59 hostname SSL verification enabled
2014-10-22 03:41:55 INFO juju.utils http.go:59 hostname SSL verification enabled
2014-10-22 03:41:55 INFO juju.utils http.go:59 hostname SSL verification enabled
Starting new instance for initial state server
Launching instance
2014-10-22 03:42:01 INFO juju.utils http.go:59 hostname SSL verification enabled
2014-10-22 03:42:01 INFO juju.utils http.go:59 hostname SSL verification enabled
2014-10-22 03:42:01 INFO juju.utils http.go:59 hostname SSL verification enabled
2014-10-22 03:42:03 INFO juju.provider.openstack provider.go:1037 allocated public IP 15.125.102.151
2014-10-22 03:42:13 INFO juju.provider.openstack provider.go:1085 started instance "d06add77-1a7a-49e1-a441-5edf49953cf1"
2014-10-22 03:42:26 INFO juju.provider.openstack provider.go:1095 assigned public IP 15.125.102.151 to "d06add77-1a7a-49e1-a441-5edf49953cf1"
 - d06add77-1a7a-49e1-a441-5edf49953cf1
2014-10-22 03:42:29 INFO juju.environs.bootstrap bootstrap.go:178 newest version: 1.21-alpha2
2014-10-22 03:42:29 INFO juju.environs.bootstrap bootstrap.go:206 picked bootstrap tools version: 1.21-alpha2
Installing Juju agent on bootstrap instance
Waiting for address
2014-10-22 03:42:29 INFO juju.provider.openstack provider.go:453 instance d06add77-1a7a-49e1-a441-5edf49953cf1 has floating IP address: 15.125.102.151
Attempting to connect to 15.125.102.151:22
Attempting to connect to 10.0.0.78:22
2014-10-22 03:42:40 INFO juju.provider.openstack provider.go:453 instance d06add77-1a7a-49e1-a441-5edf49953cf1 has floating IP address: 15.125.102.151
2014-10-22 03:42:50 INFO juju.provider.openstack provider.go:453 instance d06add77-1a7a-49e1-a441-5edf49953cf1 has floating IP address: 15.125.102.151
2014-10-22 03:43:00 INFO juju.provider.openstack provider.go:453 instance d06add77-1a7a-49e1-a441-5edf49953cf1 has floating IP address: 15.125.102.151
2014-10-22 03:43:09 INFO juju.provider.openstack provider.go:453 instance d06add77-1a7a-49e1-a441-5edf49953cf1 has floating IP address: 15.125.102.151
2014-10-22 03:43:19 INFO juju.provider.openstack provider.go:453 instance d06add77-1a7a-49e1-a441-5edf49953cf1 has floating IP address: 15.125.102.151
2014-10-22 03:43:29 INFO juju.provider.openstack provider.go:453 instance d06add77-1a7a-49e1-a441-5edf49953cf1 has floating IP address: 15.125.102.151
2014-10-22 03:43:39 INFO juju.provider.openstack provider.go:453 instance d06add77-1a7a-49e1-a441-5edf49953cf1 has floating IP address: 15.125.102.151
Warning: Permanently added '15.125.102.151' (ECDSA) to the list of known hosts.
sudo: unable to resolve host juju-test-function-hp-2-machine-0
Logging to /var/log/cloud-init-output.log on remote host
Running apt-get update
Running apt-get upgrade
Installing package: curl
Installing package: cpu-checker
Installing package: bridge-utils
Installing package: rsyslog-gnutls
Fetching tools: curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --retry 10 -o $bin/tools.tar.gz <[https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools/releases/juju-1.21-alpha2-trusty-amd64.tgz]>
Bootstrapping Juju machine agent
Starting Juju machine agent (jujud-machine-0)
Bootstrap complete
2014-10-22 03:50:15 INFO juju.provider.openstack provider.go:453 instance d06add77-1a7a-49e1-a441-5edf49953cf1 has floating IP address: 15.125.102.151
2014-10-22 03:50:15 INFO juju.cmd supercommand.go:329 command finished
juju --show-log deploy -e test-function-hp-2 local:precise/ubuntu
2014-10-22 03:50:23 INFO juju.cmd supercommand.go:37 running juju [1.21-alpha2-precise-amd64 gc]
2014-10-22 03:50:23 INFO juju.api api.go:251 connecting to API addresses: [15.125.102.151:17070 10.0.0.78:17070]
2014-10-22 03:50:23 INFO juju.api apiclient.go:252 dialing "wss://15.125.102.151:17070/environment/58eb8daf-59ad-4eb0-8426-e12270164dcc/api"
2014-10-22 03:50:23 INFO juju.api apiclient.go:252 dialing "wss://10.0.0.78:17070/environment/58eb8daf-59ad-4eb0-8426-e12270164dcc/api"
2014-10-22 03:50:24 INFO juju.api apiclient.go:175 connection established to "wss://15.125.102.151:17070/environment/58eb8daf-59ad-4eb0-8426-e12270164dcc/api"
2014-10-22 03:50:24 INFO juju.utils http.go:66 hostname SSL verification disabled
Added charm "local:precise/ubuntu-1" to the environment.
2014-10-22 03:50:31 INFO juju.cmd supercommand.go:329 command finished
...
test-function-hp-2 is ready to testing
juju --show-log ensure-availability -e test-function-hp-2 -n 3
2014-10-22 03:53:47 INFO juju.cmd supercommand.go:37 running juju [1.21-alpha2-precise-amd64 gc]
2014-10-22 03:53:47 INFO juju.api api.go:251 connecting to API addresses: [15.125.102.151:17070 10.0.0.78:17070]
2014-10-22 03:53:47 INFO juju.api apiclient.go:252 dialing "wss://15.125.102.151:17070/environment/58eb8daf-59ad-4eb0-8426-e12270164dcc/api"
2014-10-22 03:53:48 INFO juju.api apiclient.go:252 dialing "wss://10.0.0.78:17070/environment/58eb8daf-59ad-4eb0-8426-e12270164dcc/api"
2014-10-22 03:53:48 INFO juju.api apiclient.go:175 connection established to "wss://15.125.102.151:17070/environment/58eb8daf-59ad-4eb0-8426-e12270164dcc/api"
maintaining machines: 0
adding machines: 2, 3

2014-10-22 03:53:52 INFO juju.cmd supercommand.go:329 command finished
adding-vote: 2, 3
...
adding-vote: 2, 3
Warning: Permanently added '15.125.100.154' (ECDSA) to the list of known hosts.
sudo: unable to resolve host juju-test-function-hp-2-machine-1
Warning: Permanently added '15.125.100.154' (ECDSA) to the list of known hosts.
Warning: Permanently added '15.125.102.151' (ECDSA) to the list of known hosts.
sudo: unable to resolve host juju-test-function-hp-2-machine-0
Warning: Permanently added '15.125.102.151' (ECDSA) to the list of known hosts.
Warning: Permanently added '15.125.102.160' (ECDSA) to the list of known hosts.
sudo: unable to resolve host juju-test-function-hp-2-machine-3
Warning: Permanently added '15.125.102.160' (ECDSA) to the list of known hosts.
Warning: Permanently added '15.125.100.3' (ECDSA) to the list of known hosts.
sudo: unable to resolve host juju-test-function-hp-2-machine-2
Warning: Permanently added '15.125.100.3' (ECDSA) to the list of known hosts.
juju --show-log destroy-environment test-function-hp-2 --force -y
2014-10-22 04:14:20 INFO juju.cmd supercommand.go:37 running juju [1.21-alpha2-precise-amd64 gc]
2014-10-22 04:14:20 INFO juju.provider.openstack provider.go:248 opening environment "test-function-hp-2"
2014-10-22 04:14:20 INFO juju.provider.common destroy.go:15 destroying environment "test-function-hp-2"
2014-10-22 04:14:27 WARNING juju.provider.openstack provider.go:1587 cannot delete security group "juju-test-function-hp-2-0". Used by another environment?
2014-10-22 04:14:33 INFO juju.cmd supercommand.go:329 command finished

EXCEPTION CAUGHT:

Timed out waiting for voting to be enabled.

FAIL
Build step 'Execute shell' marked build as failure
Archiving artifacts
Description set: gitbranch:master:github.com/juju/juju b4a1e09b
[BFA] Scanning build for known causes...

[BFA] Done. 0s
Finished: FAILURE

See original description
Curtis Hovey (sinzui)
tags: added: ha
description: updated
Changed in juju-core:
importance: High → Medium
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Related blueprints

Remote bug watches

Bug watches keep track of this bug in other bug trackers.