HAProxy charm broken by recent commit

[Curtis Hovey]

As seen in
    http://reports.vapour.ws/releases/3084/job/aws-quickstart-bundle/attempt/1050
    http://reports.vapour.ws/releases/3084/job/joyent-quickstart-bundle/attempt/1398
    http://reports.vapour.ws/releases/3084/job/maas17-quickstart-bundle/attempt/1118
    http://reports.vapour.ws/releases/3084/job/hp-quickstart-bundle/attempt/1196

The HAProxy charm consistently fails in several substrates. This error was not seen in the last tested revision (9283134). The suspects are:
    https://github.com/juju/juju/commit/0006aa6ee202382d022ff1b5906a156139df2885
    https://github.com/juju/juju/commit/f06d94d457188b431b9f1bfe4a68ff713491641b
    https://github.com/juju/juju/commit/23419ca7b50b439e2ed9fa4c489e46647b4ac02a
    https://github.com/juju/juju/commit/b18afee0d2873e09f0bc8329208eb2d959559807
    https://github.com/juju/juju/commit/747dd366b4f659d084f029170caefcca76d91d43
    https://github.com/juju/juju/commit/3936d2994d8a9ca8fe130e84cc3a96e8f7e6b626
    https://github.com/juju/juju/commit/d518001c93a142241d6c6e1456dad144fbbe2743
    https://github.com/juju/juju/commit/c66316aa513ec8872f9b7137688da0ea5bd6b31f
    https://github.com/juju/juju/commit/21541c589aab95796cd2c91a040db3164ab812fd
    https://github.com/juju/juju/commit/c060aaf9d129133b86eec781524258e296966d6a
    https://github.com/juju/juju/commit/b3e137bf0110392a555d9c38500a20b7ce751e87
    https://github.com/juju/juju/commit/29b7209a509a86daac7bab758d49ef061670b9df
    https://github.com/juju/juju/commit/5ad5c7d351b62b6357950329663e359757cc637a

This OpenStack Deployment may have also failed for the same reason, or might be a different bug.
    http://reports.vapour.ws/releases/3084/job/OS-deployer/attempt/307

[Katherine Cox-Buday (cox-katherine-e)]

Wayne is unable to reproduce locally, and there's not enough information in the logs to properly diagnose. Curtis is going to do two things:
1. Give Wayne the exact command line that it's utilizing to help reproduce.
2. Re-run with a debug flag on quickstart enabled in the hopes that we'll get more helpful logs.

[Curtis Hovey (sinzui)]

We added --debug to the four quickstart jobs that deploy the landsscape scalable bundle. All four jobs failed again like the original run. revision-build 3034 is still being tested
    http://reports.vapour.ws/releases/3090
but at this hour we can see these failure will be the cause of the curse
    http://reports.vapour.ws/releases/3090/job/aws-quickstart-bundle/attempt/1055
    http://reports.vapour.ws/releases/3090/job/hp-quickstart-bundle/attempt/1202
    http://reports.vapour.ws/releases/3090/job/maas17-quickstart-bundle/attempt/1122
    http://reports.vapour.ws/releases/3090/job/joyent-quickstart-bundle/attempt/1404

Per the output in the console logs quickstart requires use to export the path to the Juju bin under test. eg:
     export JUJU=./extracted-bin/usr/lib/juju-1.25-beta1/bin/juju
     juju --debug quickstart -e aws-quickstart-bundle --constraints mem=2G --no-browser /var/lib/jenkins/repository/landscape-scalable.yaml
then poll with status until all units are started or one is in error.

[Wayne Witzel III (wwitzel3)]

Thanks for the update. I will continue to investigate.
On Sep 22, 2015 23:20, "Curtis Hovey" <email address hidden> wrote:

> We added --debug to the four quickstart jobs that deploy the landsscape
> scalable bundle. All four jobs failed again like the original run.
> revision-build 3034 is still being tested
> http://reports.vapour.ws/releases/3090
> but at this hour we can see these failure will be the cause of the curse
>
> http://reports.vapour.ws/releases/3090/job/aws-quickstart-bundle/attempt/1055
>
> http://reports.vapour.ws/releases/3090/job/hp-quickstart-bundle/attempt/1202
>
> http://reports.vapour.ws/releases/3090/job/maas17-quickstart-bundle/attempt/1122
>
> http://reports.vapour.ws/releases/3090/job/joyent-quickstart-bundle/attempt/1404
>
> Per the output in the console logs quickstart requires use to export the
> path to the Juju bin under test. eg:
> export JUJU=./extracted-bin/usr/lib/juju-1.25-beta1/bin/juju
> juju --debug quickstart -e aws-quickstart-bundle --constraints mem=2G
> --no-browser /var/lib/jenkins/repository/landscape-scalable.yaml
> then poll with status until all units are started or one is in error.
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1498481
>
> Title:
> HAProxy charm broken by recent commit
>
> Status in juju-core:
> Incomplete
> Status in juju-core 1.25 series:
> Triaged
>
> Bug description:
> As seen in
>
> http://reports.vapour.ws/releases/3084/job/aws-quickstart-bundle/attempt/1050
>
> http://reports.vapour.ws/releases/3084/job/joyent-quickstart-bundle/attempt/1398
>
> http://reports.vapour.ws/releases/3084/job/maas17-quickstart-bundle/attempt/1118
>
> http://reports.vapour.ws/releases/3084/job/hp-quickstart-bundle/attempt/1196
>
> The HAProxy charm consistently fails in several substrates. This error
> was not seen in the last tested revision (9283134). The suspects are:
>
> https://github.com/juju/juju/commit/0006aa6ee202382d022ff1b5906a156139df2885
>
> https://github.com/juju/juju/commit/f06d94d457188b431b9f1bfe4a68ff713491641b
>
> https://github.com/juju/juju/commit/23419ca7b50b439e2ed9fa4c489e46647b4ac02a
>
> https://github.com/juju/juju/commit/b18afee0d2873e09f0bc8329208eb2d959559807
>
> https://github.com/juju/juju/commit/747dd366b4f659d084f029170caefcca76d91d43
>
> https://github.com/juju/juju/commit/3936d2994d8a9ca8fe130e84cc3a96e8f7e6b626
>
> https://github.com/juju/juju/commit/d518001c93a142241d6c6e1456dad144fbbe2743
>
> https://github.com/juju/juju/commit/c66316aa513ec8872f9b7137688da0ea5bd6b31f
>
> https://github.com/juju/juju/commit/21541c589aab95796cd2c91a040db3164ab812fd
>
> https://github.com/juju/juju/commit/c060aaf9d129133b86eec781524258e296966d6a
>
> https://github.com/juju/juju/commit/b3e137bf0110392a555d9c38500a20b7ce751e87
>
> https://github.com/juju/juju/commit/29b7209a509a86daac7bab758d49ef061670b9df
>
> https://github.com/juju/juju/commit/5ad5c7d351b62b6357950329663e359757cc637a
>
> This OpenStack Deployment may have also failed for the same reason, or
> might be a different bug.
> http://reports.vapour.ws/releases/3084/job/OS-deployer/attempt/307
>
> To manage notifications...

[Wayne Witzel III (wwitzel3)]

So using the bundle from here - http://bazaar.launchpad.net/~juju-qa/juju-ci-tools/repository/
And ensuring the the JUJU=`which juju` env variable my local juju build of 1.25 - https://github.com/juju/juju/tree/1.25

I am unable to reproduce the error from CI against AWS or MAAS using the following command:

    juju --show-log quickstart -e amazon --constraints mem=2G --no-browser ./repository/bundles.yaml --debug

[Curtis Hovey (sinzui)]

The quickstart failures were using the genuine landscape scalable bundle
    juju --show-log quickstart -e aws-quickstart-bundle --constraints mem=2G --no-browser /var/lib/jenkins/repository/landscape-scalable.yaml
Maybe the issue is the combination of charms, not just the haproxy charm

[Andrew Wilkins (axwalk)]

I can reproduce this failure. Wayne, the issue is that landscape-scalable.yaml has 'services: ""' in its config. This smells like it's related to the recent change to how we initialise service settings.

[Andrew Wilkins (axwalk)]

Apparently neither https://github.com/juju/juju/commit/eaaea0ddf60d43825d35c1e4f2685f17b3fa658c, nor Nate's follow-up http://reviews.vapour.ws/r/2680/diff/#, made their way to 1.25. Testing now, but this is probably the issue.

[Canonical Juju QA Bot (juju-qa-bot)]

Juju-CI verified that this issue is Fix Released in juju-core 1.25:
    http://reports.vapour.ws/releases/3099