ec2: destroy-controller blows the rate limit trying to delete security group - can leave instances around

Bug #1537620 reported by Dave Cheney
44
This bug affects 6 people
Affects Status Importance Assigned to Milestone
Canonical Juju
Fix Released
High
Anastasia

Bug Description

Repro:

1. Deploy largish environment, I had the one from LP #1511659
2. In repo'ing 1511659, stop a machine that has been deployed by sshing to the machine and sudo shutdown
2a. check in AWS console that that the machine is in STOPPED state (not TERMINATED)
3. juju kill-controller ap-southeast-2

WARNING! This command will destroy the "ap-southeast-2" controller.
This includes all machines, services, data and other resources.

Continue [y/N]? y
Destroying controller "ap-southeast-2"
Waiting for resources to be reclaimed
All hosted environments reclaimed, cleaning up controller machines
WARNING could not delete security group "juju-349b8bc2-bcc8-4967-8961-3c4dbdbdd861-7": There are active instances using security group 'juju-349b8bc2-bcc8-4967-8961-3c4dbdbdd861-7' (InvalidGroup.InUse)
WARNING could not delete security group "juju-349b8bc2-bcc8-4967-8961-3c4dbdbdd861-7": There are active instances using security group 'juju-349b8bc2-bcc8-4967-8961-3c4dbdbdd861-7' (InvalidGroup.InUse)

(repeated)

WARNING could not delete security group "juju-349b8bc2-bcc8-4967-8961-3c4dbdbdd861-5": Request limit exceeded. (RequestLimitExceeded)

Helpful suggestion from a duplicate bug:

  This warning isn't helpful and the user can't act upon it. It's working according to plan and the output should be more clear. I'd suggest:

  % juju destroy-controller b3aws
  WARNING! This command will destroy the "local.b3aws" controller.
  This includes all machines, services, data and other resources.

  Continue [y/N]? y
  Destroying controller "local.b3aws"
  Cleaning up services and machines.......(and to do a . for each of these warnings as an attempt at a progress-like meter)

summary: - environs/ec2: a stopped machine will cause kill-machine to fail and blow
- the rate limit
+ environs/ec2: a stopped machine will cause kill-controller to fail and
+ blow the rate limit
Aaron Bentley (abentley)
Changed in juju-core:
importance: Undecided → High
status: New → Triaged
milestone: none → 2.0-beta1
Revision history for this message
Cheryl Jennings (cherylj) wrote : Re: environs/ec2: a stopped machine will cause kill-controller to fail and blow the rate limit

As per bug #1542992, this doesn't require a stopped machine to hit this issue.

Curtis Hovey (sinzui)
Changed in juju-core:
milestone: 2.0-beta1 → 2.0-beta2
Curtis Hovey (sinzui)
Changed in juju-core:
milestone: 2.0-beta2 → 2.0-beta3
summary: - environs/ec2: a stopped machine will cause kill-controller to fail and
- blow the rate limit
+ environs/ec2: kill-controller / destroy-controller blow the rate limit
+ trying to delete security group
Revision history for this message
Cheryl Jennings (cherylj) wrote : Re: environs/ec2: kill-controller / destroy-controller blow the rate limit trying to delete security group

This seems to have gotten worse with the new admin / default model behavior (coming in beta3)

Changed in juju-core:
milestone: 2.0-beta3 → 2.0-beta4
tags: added: 2.0-count
description: updated
summary: - environs/ec2: kill-controller / destroy-controller blow the rate limit
- trying to delete security group
+ ec2: destroy-controller blow the rate limit trying to delete security
+ group - can leave instances around
Revision history for this message
Cheryl Jennings (cherylj) wrote : Re: ec2: destroy-controller blow the rate limit trying to delete security group - can leave instances around

I've seen that once the rate limit is hit, the destroy / kill controller command will complete and remove the controller information from the cache.yaml / controller info and can leave resources behind.

Martin Packman (gz)
tags: added: ci jujuqa
summary: - ec2: destroy-controller blow the rate limit trying to delete security
+ ec2: destroy-controller blows the rate limit trying to delete security
group - can leave instances around
Changed in juju-core:
status: Triaged → In Progress
assignee: nobody → Anastasia (anastasia-macmood)
Revision history for this message
Anastasia (anastasia-macmood) wrote :
Changed in juju-core:
status: In Progress → Fix Committed
Curtis Hovey (sinzui)
Changed in juju-core:
status: Fix Committed → Fix Released
affects: juju-core → juju
Changed in juju:
milestone: 2.0-beta4 → none
milestone: none → 2.0-beta4
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.