I'm taking a look now.


On Thu, Dec 19, 2013 at 5:03 AM, Curtis Hovey <email address hidden> wrote:

> Now that the bootstrap private ip fix is in trunk. CI now sees this bug
> when deploying wordpress and mysql using the the 1.17.0 candidate on both
> the client and server.  CI sees a hook failure. 1.16.5 does not encounter
> this problem. This issue could relate to HP's speed or internals. The mysql
> charm itself might be brittle. I have played out this scenario 3 times
> using my one command line to allow for human slowness
>
> http://bazaar.launchpad.net/~juju-qa/juju-core/ci-cd-scripts2/view/head:/deploy_stack.py
>
> We can see from the logs that the right juju version is always found so
> the script does not attempt to force the version.
> I see config-changed hook failures of the mysql agent is stuck in pending
> for 40 minutes.
>
> I am attaching a log from CI. I have confirmed using debug log that my
> own tests see this, but I think it is orthogonal:
>
> machine-1: 2013-12-18 17:40:59 ERROR juju.worker.authenticationworker
> worker.go:68 adding current Juju keys to ssh authorised keys: generating
> key fingerprint: exit status 1
> machine-1: 2013-12-18 17:40:59 ERROR juju runner.go:220 worker: exited
> "authenticationworker": adding current Juju keys to ssh authorised keys:
> generating key fingerprint: exit status 1
> machine-1: 2013-12-18 17:40:59 INFO juju runner.go:254 worker: restarting
> "authenticationworker" in 3s
> machine-1: 2013-12-18 17:41:02 INFO juju runner.go:262 worker: start
> "authenticationworker"
> machine-1: 2013-12-18 17:41:02 WARNING juju.ssh authorisedkeys.go:244
> invalid Juju ssh key : generating key fingerprint: exit status 1
>
> The hook failures seen in 2 of my 3 manual tests report
> ubuntu0.12.04.1) ...
> 2013-12-18 17:55:51 INFO juju.worker.uniter context.go:323 HOOK 131218
> 17:55:51 [Warning] Using unique option prefix key_buffer instead of
> key_buffer_size is deprecated and will be removed in a future release.
> Please use the full name instead.
> 2013-12-18 17:56:00 INFO juju.worker.uniter context.go:323 HOOK mysql
> start/running, process 20579
> 2013-12-18 17:56:00 INFO juju.worker.uniter context.go:323 HOOK Setting up
> libhtml-template-perl (2.10-1) ...
> 2013-12-18 17:56:00 INFO juju.worker.uniter context.go:323 HOOK Setting up
> mysql-server (5.5.34-0ubuntu0.12.04.1) ...
> 2013-12-18 17:56:01 INFO worker.uniter.jujuc server.go:102 running hook
> tool "juju-log" ["-l" "INFO" "dataset size in bytes: 822083584"]
> 2013-12-18 17:56:01 INFO juju juju-log.go:66 mysql/0: dataset size in
> bytes: 822083584
> 2013-12-18 17:56:03 INFO juju.worker.uniter context.go:323 HOOK mysql
> stop/waiting
> 2013-12-18 17:56:05 INFO juju.worker.uniter context.go:323 HOOK start: Job
> failed to start
> 2013-12-18 17:56:05 INFO juju.worker.uniter context.go:323 HOOK Traceback
> (most recent call last):
> 2013-12-18 17:56:05 INFO juju.worker.uniter context.go:323 HOOK   File
> "/var/lib/juju/agents/unit-mysql-0/charm/hooks/config-changed", line 356,
> in <module>
> 2013-12-18 17:56:05 INFO juju.worker.uniter context.go:323 HOOK
> check_call(['service','mysql','start'])
> 2013-12-18 17:56:05 INFO juju.worker.uniter context.go:323 HOOK   File
> "/usr/lib/python2.7/subprocess.py", line 511, in check_call
> 2013-12-18 17:56:05 INFO juju.worker.uniter context.go:323 HOOK     raise
> CalledProcessError(retcode, cmd)
> 2013-12-18 17:56:05 INFO juju.worker.uniter context.go:323 HOOK
> subprocess.CalledProcessError: Command '['service', 'mysql', 'start']'
> returned non-zero exit status 1
>
>
> ** Attachment added: "HP deployment log lp:juju-core r2158"
>
> https://bugs.launchpad.net/juju-core/+bug/1255242/+attachment/3932040/+files/all-machines-test-release-hp.log.gz
>
> --
> You received this bug notification because you are subscribed to juju-
> core.
> Matching subscriptions: MOAR JUJU SPAM!
> https://bugs.launchpad.net/bugs/1255242
>
> Title:
>   upgrade-juju and bootstrap on HP cloud broken in devel
>
> Status in juju-core:
>   Triaged
>
> Bug description:
>   I am sorry for this late bug report. CI consistently shows that
>   upgrade-juju consistently fails on hp-cloud. Revno 2072 appears to be
>   the culprit. Revno 2071 was the last pass, and we verified 2071 was
>   good in two repeats runs of CI. Revno 2072 always fails and fails the
>   same way...the mysql charm has a hook error.
>
>   The specific test that fails does this: A 1.16.3 env is bootstrapped
>   on hp cloud. Wordpress + mysql is deployed and verified to be up. The
>   test workspace installed the juju-core 1.17.0. package created from
>   revno 2072. The new tools and metadata were published to all the
>   clouds uder the testing/ prefix.  Next juju-upgrade was called on HP
>   cloud and we know 1.17.0 was found.
>
>   We see this error after the second effort to get the tools
>       ERROR juju runner.go:220 worker: exited "environ-provisioner": no
> state server machines with addresses found
>
>   Much latter we see a hook failure
>       juju-test-release-hp-machine-2:2013-11-26 15:10:05 INFO
> juju.worker.uniter context.go:263
>       HOOK subprocess.CalledProcessError: Command '['service', 'mysql',
> 'start']' returned non-zero exit status 1
>
>   The mysql charm did not fail on aws, canonistack, lxc, or azure. Just
>   Hp.
>
>   I am attaching the juju and CI log.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1255242/+subscriptions
>