open-stack provider breaks swift with standard config

[Curtis Hovey]

Canonistack swift access has spontaneously combusted. First juju-qa lost
access. Now orangesquad has lost access. Juju cannot get permission to
use swift. juju-qa and orangesquad can not use swift-client after juju's
configs were changed to match the setup recommended by

    juju init

Both accounts lost access when control-bucket and admin-secret were
removed from the juju configs. In both cases, there were no jenv files
left behind nor where there any running environments when the change
was made. An example config looks like this:

  test-cloud-canonistack:
    # juju --show-log bootstrap -e test-release-canonistack
    type: openstack
    test-mode: true
    use-floating-ip: false
    auth-url: https://keystone.canonistack.canonical.com:443/v2.0/
    authorized-keys: |
      ssh-rsa <SECRET>
    username: juju-qa
    password: <SECRET>
    tenant-name: <SECRET>
    region: lcy02
    default-series: precise
    tools-metadata-url: https://swift.canonistack.canonical.com/v1/AUTH_526ad877f3e3464589dc1145dfeaac60/juju-dist/tools

The output of bootstrap showed this:

    juju --show-log bootstrap -e test-cloud-canonistack --constraints mem=2G
    2014-04-23 14:45:35 INFO juju.cmd supercommand.go:297 running juju-1.18.0-precise-amd64 [gc]
    2014-04-23 14:45:35 INFO juju.provider.openstack provider.go:202 opening environment "test-cloud-canonistack"
    2014-04-23 14:45:36 WARNING juju.environs open.go:289 failed to write bootstrap-verify file: cannot make Swift control container: failed to create container: 17338a9c21264d6b8e5b34b9e443c049
    caused by: Resource at https://swift.canonistack.canonical.com/v1/AUTH_bfbb283705034a7c8eeefb918b2e8df8/17338a9c21264d6b8e5b34b9e443c049 not found
    caused by: request (https://swift.canonistack.canonical.com/v1/AUTH_bfbb283705034a7c8eeefb918b2e8df8/17338a9c21264d6b8e5b34b9e443c049) returned unexpected status: 404; error info: 404 Not Found

    The resource could not be found.

After juju tried to create a container, swift shows this (after the lib was hacked to survive the report a a unicode error):

    swift list
    Account GET failed:
    https://swift.canonistack.canonical.com/v1/AUTH_bbda63d64a9e4dfda1072e7743704f1e?
    format=json 403 Forbidden 
    JMN+THII-IM
b

CI is not testing canonistack since juju and its accounts are crippled.

WORK AROUND:
Always include "admin-secret", do not let Juju create one for you.
If you are affected by this, swift will not be usable until the token expires, about 2 weeks.

[John A Meinel (jameinel)]

It sound like our change to set readable on the control-bucket is failing
on Canonistack.

Maybe this is an incompatibility with Icehouse? (Did canonistack get
upgraded?)

John
=:->

On Thu, Apr 24, 2014 at 6:44 PM, Curtis Hovey <email address hidden> wrote:

> Public bug reported:
>
> Canonistack swift access has spontaneously combusted. First juju-qa lost
> access. Now orangesquad has lost access. Juju cannot get permission to
> use swift. juju-qa and orangesquad can not use swift-client after juju's
> configs were changed to match the setup recommended by
>
> juju init
>
> Both accounts lost access when control-bucket and admin-secret were
> removed from the juju configs. In both cases, there were no jenv files
> left behind nor where there any running environments when the change
> was made. An example config looks like this:
>
> test-cloud-canonistack:
> # juju --show-log bootstrap -e test-release-canonistack
> type: openstack
> test-mode: true
> use-floating-ip: false
> auth-url: https://keystone.canonistack.canonical.com:443/v2.0/
> authorized-keys: |
> ssh-rsa <SECRET>
> username: juju-qa
> password: <SECRET>
> tenant-name: <SECRET>
> region: lcy02
> default-series: precise
> tools-metadata-url:
> https://swift.canonistack.canonical.com/v1/AUTH_526ad877f3e3464589dc1145dfeaac60/juju-dist/tools
>
> The output of bootstrap showed this:
>
> juju --show-log bootstrap -e test-cloud-canonistack --constraints
> mem=2G
> 2014-04-23 14:45:35 INFO juju.cmd supercommand.go:297 running
> juju-1.18.0-precise-amd64 [gc]
> 2014-04-23 14:45:35 INFO juju.provider.openstack provider.go:202
> opening environment "test-cloud-canonistack"
> 2014-04-23 14:45:36 WARNING juju.environs open.go:289 failed to write
> bootstrap-verify file: cannot make Swift control container: failed to
> create container: 17338a9c21264d6b8e5b34b9e443c049
> caused by: Resource at
> https://swift.canonistack.canonical.com/v1/AUTH_bfbb283705034a7c8eeefb918b2e8df8/17338a9c21264d6b8e5b34b9e443c049not found
> caused by: request (
> https://swift.canonistack.canonical.com/v1/AUTH_bfbb283705034a7c8eeefb918b2e8df8/17338a9c21264d6b8e5b34b9e443c049)
> returned unexpected status: 404; error info: 404 Not Found
>
> The resource could not be found.
>
> After juju tried to create a container, swift shows this (after the lib
> was hacked to survive the report a a unicode error):
>
> swift list
> Account GET failed:
>
> https://swift.canonistack.canonical.com/v1/AUTH_bbda63d64a9e4dfda1072e7743704f1e
> ?
> format=json 403 Forbidden
> JMN+THII-IM b
>
> CI is not testing canonistack since juju and its accounts are crippled.
>
> ** Affects: juju-core
> Importance: High
> Status: Triaged
>
>
> ** Tags: openstack-provider
>
> --
> You received this bug notification because you are subscribed to juju-
> core.
> https://bugs.launchpad.net/bugs/1312217
>
> Title:
> open-stack provider breaks swift with standard config
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1312217/+subscriptions
>

[David Lawson (deej)]

It's not Icehouse, the Swift machines appear to be Grizzly, which I believe matches lcy02 which Curtis is using.

[Curtis Hovey (sinzui)]

I can again list swift for the juju-qa and orangesquad accounts. I suspect the bad token or something expired. Using just swift I can create containers and upload file.

I then restore the control-bucket and access-secret key-values to the canonistack envs. I could bootstrap. I entered new values and I could bootstrap.

I have requested a canary account to test juju without 1 or both keys. I don't want to loose the juju-qa team account again.

[Curtis Hovey (sinzui)]

I cannot reproduce this bug after testing many combinations.

This issue may have been resolved by other changes elsewhere. After
assuring myself that the first and second combination work on my
own computer with the formerly broken account. I switched to a new
account to find the combination that breaks. I found None. I am
running CI in canonistack with 1.18.2 admin-secret + control-bucket
because I don't trust Juju without the options.

Juju version and config options localhost canonistack-slave
1.18.2 admin-secret + control-bucket O O
1.18.2 admin-secret O O
1.18.2 none O O
1.19.1 admin-secret + control-bucket O O
1.19.1 admin-secret O O
1.19.1 none O O

The difference between now and a few weeks ago is the juju version.
We experienced this bug with juju 1.18.0 and early version of 1.19.0.
If this was a bug in juju it was fixed in the 1.18.1 or 1.18.2 release.

I have setup Juju ci to run hourly health checks with 1.18.2 in
canonistack. If there are no errors, I will enable 1.19.3 deploy and
upgrade tests.

[Jorge Niedbalski (niedbalski)]

Hello guys,

I am experiencing this same issue with 1.18.1 on trusty using the Openstack provider on Canonistack.

ubuntu@juju-1:~/.juju$ juju bootstrap -v --constraints mem=1G
WARNING failed to write bootstrap-verify file: cannot make Swift control container: failed to create container: 5a7b26e50ad2481e8aa83ee6ed96fc05
caused by: Resource at https://swift.canonistack.canonical.com/v1/AUTH_33320b62f71f40d9a20caf51d30f9f28/5a7b26e50ad2481e8aa83ee6ed96fc05 not found
caused by: request (https://swift.canonistack.canonical.com/v1/AUTH_33320b62f71f40d9a20caf51d30f9f28/5a7b26e50ad2481e8aa83ee6ed96fc05) returned unexpected status: 404; error info: 404 Not Found

The resource could not be found.

Bootstrap failed, destroying environment
ERROR provider storage is not writable

[Launchpad Janitor (janitor)]

[Expired for juju-core because there has been no activity for 60 days.]

[Chris Johnston (cjohnston)]

Can we re-open this bug as https://github.com/juju/juju/pull/689 is supposed to be a fix for it. Thanks