ERROR environment destruction failed: destroying storage: listing volumes: Get https://x.x.x.x:8776/v2/<UUID>/volumes/detail: local error: record overflow

[Ryan Beisner]

With juju 1.25, we're seeing failed enviro destroys. Take note that no instances in the deployment had a cinder volume involved at all.

Once this occurs, the .jenv file remains on disk, while the bootstrap node has been destroyed. That makes `juju stat` and `juju bootstrap` unusable unless one manually removes the jenv files, even with a --force destroy.

ubuntu@beisner-bastion:~/bzr/openstack-charm-testing$ juju switch
beis1
ubuntu@beisner-bastion:~/bzr/openstack-charm-testing$ juju destroy-environment beis1
WARNING! this command will destroy the "beis1" environment (type: openstack)
This includes all machines, services, data and other resources.

Continue [y/N]? y
ERROR environment destruction failed: destroying storage: listing volumes: Get https://10.245.161.160:8776/v2/e5965159218d4836950b2e5f27d1c9b2/volumes/detail: local error: record overflow
ubuntu@beisner-bastion:~/bzr/openstack-charm-testing$ echo $?
1

...

...

ubuntu@beisner-bastion:~/bzr/openstack-charm-testing$ apt-cache policy juju
juju:
  Installed: 1.25.0-0ubuntu1~14.04.1~juju1
  Candidate: 1.25.0-0ubuntu1~14.04.1~juju1
  Version table:
 *** 1.25.0-0ubuntu1~14.04.1~juju1 0
        500 http://ppa.launchpad.net/juju/stable/ubuntu/ trusty/main amd64 Packages
        100 /var/lib/dpkg/status
     1.22.8-0ubuntu1~14.04.1 0
        500 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty-updates/universe amd64 Packages
     1.18.1-0ubuntu1 0
        500 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty/universe amd64 Packages
ubuntu@beisner-bastion:~/bzr/openstack-charm-testing$

[Cheryl Jennings (cherylj)]

Can you include the results of running destroy-environment with --debug?

[Ryan Beisner (1chb1n)]

FWIW, this also impacts Amulet testing, when a destroy is done between test targets:

http://paste.ubuntu.com/13084642/

Yes indeed, I'll grab --debug output from a bootstrap/destroy loop and paste here shortly.

[Ryan Beisner (1chb1n)]

@cherylj - here is the debug output:

$ juju bootstrap --debug
$ juju destroy-environment --debug beis1 -y
$ juju stat --debug

http://paste.ubuntu.com/13084821/

[Ryan Beisner (1chb1n)]

Just to dig a bit more: The cinder service, api and volume are all in a good state. There is one unrelated cinder volume in existence.
$ cinder list
$ cinder show 5adaaf95-b9eb-4efc-ab0c-2963c2a1b0fd
http://paste.ubuntu.com/13084833/

[Ryan Beisner (1chb1n)]

This has become a prevalent issue in OpenStack charm testing. Our gating CI is on hold until we work around or consume a fix-committed binary. A workaround for us would involved hot-patching juju-deployer, amulet, mojo and bundletester with some sort of jenv file nuker.

[Cheryl Jennings (cherylj)]

I haven't been able to recreate the error with a simple deployment.

Some searching shows that this may be a TLS error, but at this point, I don't have any other information. I'll see if I can pass this off to the bug squad team to take a look at when they come online.

[Ryan Beisner (1chb1n)]

@cherylj - What release of OpenStack is in the cloud that where you're trying to reproduce? Ours is Trusty + Kilo.

[Cheryl Jennings (cherylj)]

The Openstack env I was using was Havana.

I did some more searching on this today, and haven't been able to come up with anything new, except to ask if, when you get this error, can you actually get Get https://x.x.x.x:8776/v2/<UUID>/volumes/detail using curl (or something) without an error?

[Ryan Beisner (1chb1n)]

@cherylj -

No, the https://x.x.x.x:8776/v2/<UUID>/volumes/detail address is not successful, and not expected to be, as the endpoints are not https (they are http). So, I'm not sure where juju is getting that URL, but it seems to be in err.

The keystone endpoint-list for the Kilo cloud we are using shows cinder's endpoint data to be:
http://paste.ubuntu.com/13101534/

Also: Havana is EOL and no longer tested or recommended. Currently, the supported Ubuntu OpenStack releases are: Icehouse, Juno, Kilo, Liberty. For reference: https://wiki.ubuntu.com/ServerTeam/CloudArchive

[Cheryl Jennings (cherylj)]

Ah, that helps. I'll look more to see why we're using "https"

[Martin Packman (gz)]

In goose, cinder/autogenerated_client.go templates with "https://volume.example.com" and then SetEndpointFn in cinder/client.go overrides the Host. However, the scheme is not overriden.

[Martin Packman (gz)]

Proposed a fix for goose: <https://github.com/go-goose/goose/pull/16>

Juju should be able to just switch to that rev when it lands, but probably also needs to gain some code that checks it's got the v2 api.

[Martin Packman (gz)]

Update to master proposed: <https://github.com/juju/juju/pull/3762>

[Martin Packman (gz)]

And the same change proposed for 1.25: https://github.com/juju/juju/pull/3764

[Ryan Beisner (1chb1n)]

So far, all iterative bootstrap/destroy loops with 1.25.1 from ppa:juju/proposed have been successful. \o/

Attached 1 loop with debug output for reference.

juju:
  Installed: 1.25.1-0ubuntu1~14.04.1~juju1
  Candidate: 1.25.1-0ubuntu1~14.04.1~juju1
  Version table:
 *** 1.25.1-0ubuntu1~14.04.1~juju1 0
        500 http://ppa.launchpad.net/juju/proposed/ubuntu/ trusty/main amd64 Packages
        100 /var/lib/dpkg/status
     1.25.0-0ubuntu1~14.04.1~juju1 0
        500 http://ppa.launchpad.net/juju/stable/ubuntu/ trusty/main amd64 Packages
     1.24.7-0ubuntu1~14.04.1 0
        500 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty-updates/universe amd64 Packages
     1.18.1-0ubuntu1 0
        500 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty/universe amd64 Packages

[Cheryl Jennings (cherylj)]

Thanks for the update!