Comment 10 for bug 1255242

Oh dear, things are quite broken in the cloud images today.

Preparing to replace landscape-client 12.05-0ubuntu1.12.04 (using
.../landscape-client_13.07.3-0ubuntu0.12.04_amd64.deb) ...
 * Stopping landscape-client daemon
   ...fail!

On Thu, Dec 19, 2013 at 11:49 AM, David Cheney
<email address hidden>wrote:

> I'm taking a look now.
>
>
> On Thu, Dec 19, 2013 at 5:03 AM, Curtis Hovey <email address hidden>wrote:
>
>> Now that the bootstrap private ip fix is in trunk. CI now sees this bug
>> when deploying wordpress and mysql using the the 1.17.0 candidate on both
>> the client and server. CI sees a hook failure. 1.16.5 does not encounter
>> this problem. This issue could relate to HP's speed or internals. The mysql
>> charm itself might be brittle. I have played out this scenario 3 times
>> using my one command line to allow for human slowness
>>
>> http://bazaar.launchpad.net/~juju-qa/juju-core/ci-cd-scripts2/view/head:/deploy_stack.py
>>
>> We can see from the logs that the right juju version is always found so
>> the script does not attempt to force the version.
>> I see config-changed hook failures of the mysql agent is stuck in pending
>> for 40 minutes.
>>
>> I am attaching a log from CI. I have confirmed using debug log that my
>> own tests see this, but I think it is orthogonal:
>>
>> machine-1: 2013-12-18 17:40:59 ERROR juju.worker.authenticationworker
>> worker.go:68 adding current Juju keys to ssh authorised keys: generating
>> key fingerprint: exit status 1
>> machine-1: 2013-12-18 17:40:59 ERROR juju runner.go:220 worker: exited
>> "authenticationworker": adding current Juju keys to ssh authorised keys:
>> generating key fingerprint: exit status 1
>> machine-1: 2013-12-18 17:40:59 INFO juju runner.go:254 worker: restarting
>> "authenticationworker" in 3s
>> machine-1: 2013-12-18 17:41:02 INFO juju runner.go:262 worker: start
>> "authenticationworker"
>> machine-1: 2013-12-18 17:41:02 WARNING juju.ssh authorisedkeys.go:244
>> invalid Juju ssh key : generating key fingerprint: exit status 1
>>
>> The hook failures seen in 2 of my 3 manual tests report
>> ubuntu0.12.04.1) ...
>> 2013-12-18 17:55:51 INFO juju.worker.uniter context.go:323 HOOK 131218
>> 17:55:51 [Warning] Using unique option prefix key_buffer instead of
>> key_buffer_size is deprecated and will be removed in a future release.
>> Please use the full name instead.
>> 2013-12-18 17:56:00 INFO juju.worker.uniter context.go:323 HOOK mysql
>> start/running, process 20579
>> 2013-12-18 17:56:00 INFO juju.worker.uniter context.go:323 HOOK Setting
>> up libhtml-template-perl (2.10-1) ...
>> 2013-12-18 17:56:00 INFO juju.worker.uniter context.go:323 HOOK Setting
>> up mysql-server (5.5.34-0ubuntu0.12.04.1) ...
>> 2013-12-18 17:56:01 INFO worker.uniter.jujuc server.go:102 running hook
>> tool "juju-log" ["-l" "INFO" "dataset size in bytes: 822083584"]
>> 2013-12-18 17:56:01 INFO juju juju-log.go:66 mysql/0: dataset size in
>> bytes: 822083584
>> 2013-12-18 17:56:03 INFO juju.worker.uniter context.go:323 HOOK mysql
>> stop/waiting
>> 2013-12-18 17:56:05 INFO juju.worker.uniter context.go:323 HOOK start:
>> Job failed to start
>> 2013-12-18 17:56:05 INFO juju.worker.uniter context.go:323 HOOK Traceback
>> (most recent call last):
>> 2013-12-18 17:56:05 INFO juju.worker.uniter context.go:323 HOOK File
>> "/var/lib/juju/agents/unit-mysql-0/charm/hooks/config-changed", line 356,
>> in <module>
>> 2013-12-18 17:56:05 INFO juju.worker.uniter context.go:323 HOOK
>> check_call(['service','mysql','start'])
>> 2013-12-18 17:56:05 INFO juju.worker.uniter context.go:323 HOOK File
>> "/usr/lib/python2.7/subprocess.py", line 511, in check_call
>> 2013-12-18 17:56:05 INFO juju.worker.uniter context.go:323 HOOK raise
>> CalledProcessError(retcode, cmd)
>> 2013-12-18 17:56:05 INFO juju.worker.uniter context.go:323 HOOK
>> subprocess.CalledProcessError: Command '['service', 'mysql', 'start']'
>> returned non-zero exit status 1
>>
>>
>> ** Attachment added: "HP deployment log lp:juju-core r2158"
>>
>> https://bugs.launchpad.net/juju-core/+bug/1255242/+attachment/3932040/+files/all-machines-test-release-hp.log.gz
>>
>> --
>> You received this bug notification because you are subscribed to juju-
>> core.
>> Matching subscriptions: MOAR JUJU SPAM!
>> https://bugs.launchpad.net/bugs/1255242
>>
>> Title:
>> upgrade-juju and bootstrap on HP cloud broken in devel
>>
>> Status in juju-core:
>> Triaged
>>
>> Bug description:
>> I am sorry for this late bug report. CI consistently shows that
>> upgrade-juju consistently fails on hp-cloud. Revno 2072 appears to be
>> the culprit. Revno 2071 was the last pass, and we verified 2071 was
>> good in two repeats runs of CI. Revno 2072 always fails and fails the
>> same way...the mysql charm has a hook error.
>>
>> The specific test that fails does this: A 1.16.3 env is bootstrapped
>> on hp cloud. Wordpress + mysql is deployed and verified to be up. The
>> test workspace installed the juju-core 1.17.0. package created from
>> revno 2072. The new tools and metadata were published to all the
>> clouds uder the testing/ prefix. Next juju-upgrade was called on HP
>> cloud and we know 1.17.0 was found.
>>
>> We see this error after the second effort to get the tools
>> ERROR juju runner.go:220 worker: exited "environ-provisioner": no
>> state server machines with addresses found
>>
>> Much latter we see a hook failure
>> juju-test-release-hp-machine-2:2013-11-26 15:10:05 INFO
>> juju.worker.uniter context.go:263
>> HOOK subprocess.CalledProcessError: Command '['service', 'mysql',
>> 'start']' returned non-zero exit status 1
>>
>> The mysql charm did not fail on aws, canonistack, lxc, or azure. Just
>> Hp.
>>
>> I am attaching the juju and CI log.
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/juju-core/+bug/1255242/+subscriptions
>>
>
>