error destroying models on OpenStack cloud with no block storage services

Bug #1509635 reported by James Page
26
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Canonical Juju
Fix Released
Critical
Tim Penhey
juju-core
Won't Fix
Undecided
Unassigned
1.25
Won't Fix
Undecided
Unassigned

Bug Description

$ juju destroy-environment --force overcloud
ERROR destroying storage: getting volume source: volume endpoint not found for "RegionOne" endpoint

An OpenStack cloud can be many different things, including not having block storage services; so I don't think that this should be throwing an error on destruction.

juju version: 1.25.0

Changed in juju-core:
status: New → Triaged
importance: Undecided → High
tags: added: storage
tags: added: sts
tags: added: sts-needs-review
Changed in juju-core:
milestone: none → 2.0-rc1
tags: added: oil oil-2.0 v-pil
summary: - error destroying environments on OpenStack cloud with no block storage
+ error destroying models on OpenStack cloud with no block storage
services
Revision history for this message
George Kraft (cynerva) wrote :

We are seeing this with vpil, where we add and destroy models frequently. Over time we end up with many models stuck in a "destroying" state:

MODEL OWNER STATUS LAST CONNECTION
controller admin@local available 1 minute ago
default* admin@local available 12 minutes ago
scenario-86323099 admin@local destroying 2016-07-11
scenario-40025070 admin@local destroying 2016-07-11
scenario-910106729 admin@local destroying 2016-07-11
...

From the controller logs, we see the "volume endpoint not found" message gradually increasing in frequency, in proportion to the number of destroying models.

These seem to be consuming resources in neutron, which eventually becomes unresponsive; as a result, we can no longer deploy instances.

Revision history for this message
George Kraft (cynerva) wrote :

Attached controller log where we see the `endpoint "volume" in region "RegionLXD" not found` spam, and eventual deployment failures.

Changed in juju-core:
milestone: 2.0-rc1 → 2.0.0
tags: removed: sts sts-needs-review
affects: juju-core → juju
Changed in juju:
milestone: 2.0.0 → none
milestone: none → 2.0.0
Changed in juju-core:
status: New → Won't Fix
Revision history for this message
George Kraft (cynerva) wrote :

As of beta 15, we're seeing this message not just when destroying models, but also during controller bootstrap:

ERROR cmd supercommand.go:458 failed to initialize state: getting storage provider "cinder": volumes not supported: endpoint "volume" in region "RegionLXD" not found

Full output of `juju bootstrap` failure is attached.

Had to downgrade to beta 14, for now, to work around it.

Revision history for this message
George Kraft (cynerva) wrote :

Same error occurs during bootstrap on beta 17.

Marco Ceppi (marcoceppi)
Changed in juju:
importance: High → Critical
Revision history for this message
George Kraft (cynerva) wrote :

Attached output of failed bootstrap on beta 17.

Changed in juju:
milestone: 2.0.0 → 2.0-beta18
assignee: nobody → Alexis Bruemmer (alexis-bruemmer)
Tim Penhey (thumper)
Changed in juju:
assignee: Alexis Bruemmer (alexis-bruemmer) → Tim Penhey (thumper)
status: Triaged → In Progress
Andrew Wilkins (axwalk)
Changed in juju:
status: In Progress → Fix Committed
Revision history for this message
Anastasia (anastasia-macmood) wrote :
Curtis Hovey (sinzui)
Changed in juju:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.