[2.9-RC12] Bootstrap fails, security group 'juju-XXXXX' does not exist in default VPC

[Michael Skalka]

Seen during the juju 2.9-rc12 release test, we fail to bootstrap due to a juju-created security group not being found:

2021-04-23 18:17:12.253857 Launching controller instance(s) on aws/us-east-1...
2021-04-23 18:17:18.654122 - Verifying availability zone
 - Making user data
 - Setting up groups
 - Trying to start instance in availability zone "us-east-1a"
 - Start instance attempt 1
 - Verifying availability zone
 - Making user data
 - Setting up groups
ERROR failed to bootstrap model: cannot start bootstrap instance: cannot set up groups: fetching security group "juju-8dd56ca7-27b1-44b9-8c86-f3bbfa79d21a-0": The security group 'juju-8dd56ca7-27b1-44b9-8c86-f3bbfa79d21a-0' does not exist in default VPC 'vpc-c0451dba' (InvalidGroup.NotFound)

Full details of the run can be found here: https://solutions.qa.canonical.com/testruns/testRun/24ac9264-2b83-4c77-926f-e8f97d157895

Unfortunately we did not capture any logs from the local juju client snap.

[Ian Booth (wallyworld)]

The code basically does

name = "<controllerUUID>-0" #0 is machine 0, the bootstrap machine

createGroup(name)
if group already exists {
  readGroup(name) <--- failing here
  setGroupPermissions(name)
} else {
  setGroupTags(tags)
}

At bootstrap, the group should not previously exist so it should just set the tags.
However, it's failing because the create API call appears to get back a "InvalidGroup.Duplicate" error, so Juju tries to read the group. It's understandable that there could be an eventual consistency error but not in this circumstance as the group named after the new controller UUID should not already exist so the code path in question should never get executed.

Does this issue occur frequently?

[Alexander Balderson (asbalderson)]

Ian,

it doenst look like it happens too frequently; but we're marking all the occurrences with this bug; you can view the occurrences here as they happen.

https://solutions.qa.canonical.com/bugs/bugs/bug/1926169

[Joshua Genet (genet022)]

The most recent run from that list has crashdumps this time. :)

Test run:
https://solutions.qa.canonical.com/testruns/testRun/7dd62462-2c82-4662-b65f-356940185098

Artifacts/Logs/Bundles:
https://oil-jenkins.canonical.com/artifacts/7dd62462-2c82-4662-b65f-356940185098/index.html

[Alexander Balderson (asbalderson)]

we also seen this 2.9.5

[John A Meinel (jameinel)]

Following the Solutions QA bug link: https://solutions.qa.canonical.com/bugs/bugs/bug/1926169

This hasn't been seen since October (and then only a single time). There may be a timing related issue w/ AWS (we create the security group, but because of eventual consistency, when we query for it later it doesn't exist yet).
However, since it happens very infrequently, I don't think we'll prioritize this until we have an easy way to reproduce, so that we know that we have fixed it.

[Launchpad Janitor (janitor)]

[Expired for juju because there has been no activity for 60 days.]