`add-model` for existing k8s namespace returns strange error message

[Jordan Barrett]

I already have a namespace called "m" in my cluster. When I try to add a model with the same name:
--------------------------------------------------------
$ juju add-model m
ERROR failed to open kubernetes client: annotations map[controller.juju.is/id:af908c9c-4d24-4466-882e-c145bf0c754e model.juju.is/id:780625aa-10d3-4f86-84a7-69058a61adb1] for namespace "m" not valid must include map[controller.juju.is/id:9c8831d9-5ee0-46d6-81e6-6ac917959214 model.juju.is/id:d35fae5f-d5ee-4652-81fa-eaae843b4bcf]
--------------------------------------------------------

I would expect a nicer error message such as
--------------------------------------------------------
$ juju add-model m
ERROR cannot create model "m": there already exists a namespace called "m" on this cluster
Please choose a different model name.
--------------------------------------------------------

This is observed on Juju 3.0.1-dc95aea. Info about this namespace:
--------------------------------------------------------
$ k get namespace m -o yaml
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    controller.juju.is/id: af908c9c-4d24-4466-882e-c145bf0c754e
    model.juju.is/id: 780625aa-10d3-4f86-84a7-69058a61adb1
  creationTimestamp: "2022-10-25T05:09:46Z"
  labels:
    app.kubernetes.io/managed-by: juju
    kubernetes.io/metadata.name: m
    model.juju.is/name: m
  name: m
  resourceVersion: "1920352"
  uid: df343047-fee3-47b4-9c40-ec39b12530e6
spec:
  finalizers:
  - kubernetes
status:
  phase: Active
--------------------------------------------------------

[Jordan Barrett (barrettj12)]

Repro steps:

Create a file ns.yaml with the following contents:
----------------------------------------------------------------------
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    controller.juju.is/id: af908c9c-4d24-4466-882e-c145bf0c754e
    model.juju.is/id: 780625aa-10d3-4f86-84a7-69058a61adb1
  labels:
    app.kubernetes.io/managed-by: juju
    kubernetes.io/metadata.name: m
    model.juju.is/name: m
  name: m
  resourceVersion: "1920352"
spec:
  finalizers:
    - kubernetes
status:
  phase: Active
----------------------------------------------------------------------

In terminal, run
----------------------------------------------------------------------
$ k create -f ns.yaml
namespace/m created
----------------------------------------------------------------------

Switch to a k8s controller (or bootstrap a new one), and run
----------------------------------------------------------------------
$ juju add-model m
ERROR failed to open kubernetes client: annotations map[controller.juju.is/id:af908c9c-4d24-4466-882e-c145bf0c754e model.juju.is/id:780625aa-10d3-4f86-84a7-69058a61adb1] for namespace "m" not valid must include map[controller.juju.is/id:0c9d505c-6d4d-4a23-8b67-ec5dc999db7a model.juju.is/id:8e59657d-103a-4c43-8e70-fa16049c488e]
----------------------------------------------------------------------

[Jordan Barrett (barrettj12)]

Successfully reproduced on 2.9-latest so I'll target it there instead.

$ juju version --all
version: 2.9.40-ubuntu-amd64
git-commit: 78deea80f0832063711d9f5508517348c6eaa218

[Jordan Barrett (barrettj12)]

The error is created in caas/kubernetes/provider/k8s.go, line 265. There is a comment saying:
// This should never happen unless we changed annotations for a new juju version.
// But in this case, we should have already managed to fix it in upgrade steps.

[Jordan Barrett (barrettj12)]

It can happen, for example, if Juju#1 creates a new model (which is not removed properly), then Juju#2 tries to create a model of the same name. This might even happen if Juju#1 and Juju#2 are the same version.

[Jordan Barrett (barrettj12)]

https://github.com/juju/juju/pull/15552