cannot bootstrap on openstack provider: Multiple security_group matches found for name 'XYZ', use an ID to be more specific.

[Vincent Ladeuil]

Now observed with Ubuntu OpenStack Kilo as the undercloud, with juju 1.25.0 and juju 1.25.1(proposed) and the juju openstack provider.

This blocks deploy/test automation where iterative bootstrap/deploy/destroy loops occur, such as bundletester, amulet, juju test, and perhaps other CI.

2015-11-24 14:26:47 ERROR juju.cmd supercommand.go:429 failed to bootstrap environment: cannot start bootstrap instance: cannot run instance: failed to run a server with nova.RunServerOpts{ ...

caused by: request (http://10.245.161.158:8774/v2/d2be765ac7a7490a899995bdad501cc6/servers) returned unexpected status: 409; error info: {"conflictingRequest": {"message": "Multiple security_group matches found for name 'juju-osci-sv07-0', use an ID to be more specific.", "code": 409}}

# juju bootstrap --debug output:
http://paste.ubuntu.com/13492802/

--- original description ---

On HP cloud, I encounter a case where I end up with the juju-hp-global security group defined twice.

I use:
    firewall-mode: global
    use-floating-ip: true

in my environments.yaml file.

This is not reproducible at will but more likely to happen when a deployment is already present and I'm running more juju deployer commands to complete a deployment.

juju status also gives some hard to parse hints:

  "12":
    agent-state-info: '(error: cannot run instance: failed to run a server with nova.RunServerOpts{Name:"juju-hp-machine-12",
      FlavorId:"100", ImageId:"b35fe7b8-78bb-44a8-a6bd-7bc5ebdfc633", UserData:[]uint8{0x1f,

< ~500 lines of: 0x99, 0xb6, 0x7f, 0x14, 0x4d, 0x1b, 0xc7, 0x72, 0xa9, 0x84, 0x2e, 0x98, 0x72, stuff>

      SecurityGroupNames:[]nova.SecurityGroupName{nova.SecurityGroupName{Name:"juju-hp"},
      nova.SecurityGroupName{Name:"juju-hp-global"}}, Networks:[]nova.ServerNetworks{}}

      caused by: request (https://region-a.geo-1.compute.hpcloudsvc.com/v2/11206487910601/servers)
      returned unexpected status: 409; error info: {"conflictingRequest": {"message":
      "Multiple security_group matches found for name ''juju-hp-global'', use an ID
      to be more specific.", "code": 409}})'
    instance-id: pending
    series: precise

I would guess that checking the existence of juju-hp-global fails for some unexpected cause (rate limitation, nova transient error) and juju believes the group doesn't exist and create a *new* one.

At that point there are two existing juju-hp-global secgroups and juju can't recover:

 $ nova secgroup-list
+--------------------------------------+----------------+-------------+
| Id | Name | Description |
+--------------------------------------+----------------+-------------+
| f9b0e939-1a37-4f61-92b4-28f88b125d74 | default | default |
| c37f45d9-0787-4272-8558-dc4db9310802 | juju-hp | juju group |
| 32fb3fef-e1c1-41df-9227-5eda15e1cfba | juju-hp-global | juju group |
| d9cf6d0a-747a-474a-920d-90e7381eb1f5 | juju-hp-global | juju group |
+--------------------------------------+----------------+-------------+

[Curtis Hovey (sinzui)]

Juju CI sees this too. We added a script to purge the env's secgroup after tests to ensure the next use the the env doesn't see the phantom secgroup.

[Vincent Ladeuil (vila)]

$ nova secgroup-delete 32fb3fef-e1c1-41df-9227-5eda15e1cfba
ERROR: 409-{u'QuantumError': u'Security Group 32fb3fef-e1c1-41df-9227-5eda15e1cfba in use.'} (HTTP 400) (Request-ID: req-07a2a2ca-646d-4192-9ff7-248cfd7ba174)

:-/

$ nova secgroup-delete d9cf6d0a-747a-474a-920d-90e7381eb1f5
+--------------------------------------+----------------+-------------+
| Id | Name | Description |
+--------------------------------------+----------------+-------------+
| d9cf6d0a-747a-474a-920d-90e7381eb1f5 | juju-hp-global | juju group |
+--------------------------------------+----------------+-------------+

At least.

$ juju destroy-environment hp --force
$ nova secgroup-delete 32fb3fef-e1c1-41df-9227-5eda15e1cfba
+--------------------------------------+----------------+-------------+
| Id | Name | Description |
+--------------------------------------+----------------+-------------+
| 32fb3fef-e1c1-41df-9227-5eda15e1cfba | juju-hp-global | juju group |
+--------------------------------------+----------------+-------------+

Not ideal for us, the node 0 needs to be tear down.

[Ryan Beisner (1chb1n)]

I am observing this with juju 1.25.0 and 1.25.1(proposed) on the openstack provider (Kilo).

The work-around for secgroup cleanup isn't viable for deployment automation where the operator does not control the moments between destroy and deploy.

When the situation arises, it causes all subsequent deployments to fail due to bootstrap errors. This requires human interaction to resolve, short of programmatically doing secgroup housekeeping, and patching that housekeeping into juju test, amulet and/or bundletester.

In OpenStack charm testing, we have 20+ jenkins slaves, each with unique juju environments, all operating against a private cloud via the juju openstack provider. Last night, all slave enviros entered this "Multiple security_group matches" state, and are unusable. I plan to inspect and remove duplicate secgroups to dig more, but I seek a more clever provider solution.

[Ryan Beisner (1chb1n)]

Perhaps the best way to address this would be by addressing how secgroups are identified and referenced by juju. ie. Use the guaranteed-unique object ID, instead of the human-readable name string.

See https://bugs.launchpad.net/juju-core/+bug/1461957

[Andreas Hasenack (ahasenack)]

This is one of the "leaks" discussed during the cdo-qa sprint. It should be brought up to Mark Ramm's attention, per his request.

[Ryan Beisner (1chb1n)]

Here is another example of this impacting test automation:
http://paste.ubuntu.com/13580777/

[Antonio Rosales (arosales)]

I also saw this in HP Cloud and had to request to have my security group limit increased. This may have some impacts on the GOOSE provider implementation.

-thanks,
Antonio

[Anastasia (anastasia-macmood)]

We have done a lot of work in the area. Could we please confirm if this is still failing.

[Aaron Bentley (abentley)]

CI hasn't shown this since we last tested HP.

[Ryan Beisner (1chb1n)]

This is still happening on 1.25.6 as of Nov 1 2016.

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_amulet_full/openstack/charm-ceilometer/390202/2/332/consoleText.test_charm_amulet_full_1128.txt

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_amulet_full/openstack/charm-ceilometer/390202/2/332/index.html

[Ryan Beisner (1chb1n)]

The message is real. Juju should not use secgroup names to track secgroups on OpenStack. It should use unique secgroup IDs. Even with vanilla nova client, this can be expected to fail like so:

ubuntu@osci-bastion:⟫ for i in $(nova secgroup-list | awk '{ print $4 }'); do nova secgroup-delete $i; done
ERROR (NoUniqueMatch): Multiple security group matches found for name 'juju-osci-sv06', use an ID to be more specific.
ERROR (NoUniqueMatch): Multiple security group matches found for name 'juju-osci-sv06', use an ID to be more specific.

[Anastasia (anastasia-macmood)]

@Ryan

When you get a chance, please let us know if you are seeing the same behavior with Juju 2. Thank you very much for the update!

[Ryan Beisner (1chb1n)]

Just to reiterate: there is no guarantee from openstack nova that secgroup names will be unique. Juju appears to assume they will be unique. If juju were to use the unique ID of the secgroup object instead of the name, problem solved.

[Ryan Beisner (1chb1n)]

We've now upgraded to 1.25.9 in all of the OSCI Juju 1.x slaves. But before we'll know how well those fixes work toward this sec group cleanup issue, we'll have to turn off our own post-destroy vacuum cleaners. I'll plan to do that some day soon and provide feedback either way. Thanks, all!

[Anastasia (anastasia-macmood)]

Marking as Incomplete for 1.25 while awaiting Ryan's feedback.

[Anastasia (anastasia-macmood)]

We have addressed issue of deleting security groups as part of bug # 1625624.

On the openstack deployment with Juju 2.1 bootstrapped, I can see security groups identified uniquely with appropriate juju- tag as Name: http://pastebin.ubuntu.com/24031915/

I am marking this as Fix Committed.

[Anastasia (anastasia-macmood)]

As per above, the fix went in into earlier releases of Juju. However, I can only assign currently active milestones. I have put this against 2.1.0.

[Mario Splivalo (mariosplivalo)]

Hi.

From what I can see, this is also fixed in 1.25.10.

[Anastasia (anastasia-macmood)]

@Mario Splivalo (mariosplivalo),

Thank you for confirmation! I'll mark as 'Fix Committed' for 1.25 as well :D

[Anastasia (anastasia-macmood)]

Well, 'Fix Released' as I cannot target 1.25.10 milestone since it's already out.

[Corey Bryant (corey.bryant)]

Ryan, can we close this out now for OpenStack Charm Test Infra?

[Tom Haddon (mthaddon)]

We're seeing this again on 1.25.12 I believe.

We have a Jenkins instance that runs about 80 jobs each weekend, a mix of Juju 1.25.12 and 2.2.2. This is running against the OpenStack provider, and each job spins up a Juju environment and then runs tests. For the last few weekends we've had a large number of failures in jobs, all with the same error as per the original description.

Looking at one example this morning I saw the following message:

"Multiple security_group matches found for name 'juju-mojo-ols-logging-trusty', use an ID to be more specific."

I confirmed manually that there were no security groups matching that name, and reran the job. It was able to bootstrap without problems. The first part of the job does:

juju destroy-environment --force -y $JUJU_ENV

It then runs juju status in a loop until the output matches:

'Please check your credentials or use .* to create a new environment.'

It then runs juju bootstrap.