vsphere machine provisioning error, "The name 'juju-template-...' already exists."

[George Kraft]

Juju 2.9.3

My Juju controller on vSphere is failing to provision new machines:

$ juju status
Model Controller Cloud/Region Version SLA Timestamp
test gkk-vsphere vsphere/Boston 2.9.3 unsupported 15:56:58-05:00

Machine State DNS Inst id Series AZ Message
0 down pending focal The name 'juju-template-51345475823e73b240844546fc8a361a35d6464f68b13b76b26aab229945a067' already exists.
1 down pending focal The name 'juju-template-51345475823e73b240844546fc8a361a35d6464f68b13b76b26aab229945a067' already exists.

The same behavior occurs in newly created models as well.

This looks similar to https://bugs.launchpad.net/juju/+bug/1880720 but I think slightly different - in my case, it appears to be failing to create a template VM, as opposed to the linked bug where it's failing to clone it.

[George Kraft (cynerva)]

This controller was originally deployed with Juju 2.8.10, then upgraded via `juju upgrade-juju` to 2.9.3.

[George Kraft (cynerva)]

I'm consistently able to reproduce this by upgrading Juju controllers on vSphere from 2.8.10 to 2.9.3. If I bootstrap with 2.9.3 then the problem does not occur.

Reproduction steps:

sudo snap refresh juju --channel 2.8/stable
juju bootstrap vsphere
sudo snap refresh juju --channel latest/stable
juju upgrade-juju -m controller
juju add-machine

[Heather Lanigan (hmlanigan)]

This occurs because the VM templates created by juju pre-2.9, do not have arch tags. A fix is in the works.

A workaround is to log into the vSphere and delete the templates located in a folder under the controller folder. Juju will then recreate the template when provisioning a machine.

[Heather Lanigan (hmlanigan)]

https://github.com/juju/juju/pull/13079