fail to upgrade juju controller on top of charmed k8s

Bug #2073301 reported by Jeffrey Chang
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Canonical Juju
Fix Committed
Critical
Ian Booth

Bug Description

SolQA deployed Charmed Kubernetes 1.30 and bootstrap a juju controller on top of it, with juju 3.4.4, latest 3.4/stable. And there's no other models on this controller.

with "juju --debug upgrade-controller" I got below error:
# juju --debug upgrade-controller -c foundations-kubernetes
20:30:08 INFO juju.cmd supercommand.go:56 running juju [3.4.4 5abe6dd51d9e7f8c997fb7bc0454454f430c48ea gc go1.20.14]
20:30:08 DEBUG juju.cmd supercommand.go:57 args: []string{"/snap/juju/27554/bin/juju", "--debug", "upgrade-controller", "-c", "foundations-kubernetes"}
20:30:08 INFO juju.juju api.go:86 connecting to API addresses: [10.152.183.180:17070]
20:30:08 INFO juju.kubernetes.klog klog.go:113 Use tokens from the TokenRequest API or manually created secret-based tokens instead of auto-generated secret-based tokens.%!(EXTRA []interface {}=[])
20:30:08 DEBUG juju.api apiclient.go:645 starting proxier for connection
20:30:08 DEBUG juju.api apiclient.go:649 tunnel proxy in use at localhost on port 42159
20:30:08 DEBUG juju.api apiclient.go:825 looked up localhost -> [127.0.0.1]
20:30:08 DEBUG juju.api apiclient.go:1172 successfully dialed "wss://localhost:42159/api"
20:30:08 INFO juju.api apiclient.go:707 connection established to "wss://localhost:42159/api"
20:30:08 INFO juju.juju api.go:86 connecting to API addresses: [10.152.183.180:17070]
20:30:08 DEBUG juju.api apiclient.go:645 starting proxier for connection
20:30:08 DEBUG juju.api apiclient.go:649 tunnel proxy in use at localhost on port 40341
20:30:08 DEBUG juju.api apiclient.go:1172 successfully dialed "wss://localhost:40341/api"
20:30:08 INFO juju.api apiclient.go:1074 cannot resolve "localhost": lookup localhost: operation was canceled
20:30:08 INFO juju.api apiclient.go:707 connection established to "wss://localhost:40341/api"
20:30:08 INFO juju.juju api.go:86 connecting to API addresses: [10.152.183.180:17070]
20:30:08 DEBUG juju.api apiclient.go:645 starting proxier for connection
20:30:08 DEBUG juju.api apiclient.go:649 tunnel proxy in use at localhost on port 36031
20:30:08 DEBUG juju.api apiclient.go:1172 successfully dialed "wss://localhost:36031/model/cff75055-3dad-4c7d-8762-8fcb83b0e766/api"
20:30:08 INFO juju.api apiclient.go:1074 cannot resolve "localhost": lookup localhost: operation was canceled
20:30:08 INFO juju.api apiclient.go:707 connection established to "wss://localhost:36031/model/cff75055-3dad-4c7d-8762-8fcb83b0e766/api"
20:30:10 DEBUG juju.api monitor.go:35 RPC connection died
20:30:10 DEBUG juju.api monitor.go:35 RPC connection died
20:30:10 DEBUG juju.api monitor.go:35 RPC connection died
20:30:10 DEBUG juju.cmd.juju.commands upgrademodel.go:356 upgradeModel failed cannot get architecture for jujud-operator:3.4.3: can not get manifests for jujud-operator:3.4.3: manifest response version "application/vnd.docker.distribution.manifest.list.v2+json" not supported (not supported)
ERROR cannot get architecture for jujud-operator:3.4.3: can not get manifests for jujud-operator:3.4.3: manifest response version "application/vnd.docker.distribution.manifest.list.v2+json" not supported (not supported)
20:30:10 INFO cmd supercommand.go:556 command finished

# juju show-controller foundations-kubernetes
foundations-kubernetes:
  details:
    controller-uuid: d2e94bdb-4eca-42f7-8852-73328b9fc1aa
    api-endpoints: ['10.152.183.180:17070']
    cloud: kubernetes_cloud
    agent-version: 3.4.4
    agent-git-commit: 5abe6dd51d9e7f8c997fb7bc0454454f430c48ea
    controller-model-version: 3.4.4
    mongo-version: 4.4.18
    ca-fingerprint: 25:67:DD:2E:4F:B0:22:DC:BE:9A:13:A6:1B:20:6F:F2:AE:FB:49:F2:4C:4E:F4:38:1B:96:3C:42:2C:1F:80:5B
    ca-cert: |
      -----BEGIN CERTIFICATE-----
      MIIEEzCCAnugAwIBAgIVAMA7LxsSBGxcEg6MOue9nUPGRPbMMA0GCSqGSIb3DQEB
      CwUAMCExDTALBgNVBAoTBEp1anUxEDAOBgNVBAMTB2p1anUtY2EwHhcNMjQwNzE2
      MTkxMzU0WhcNMzQwNzE2MTkxODU0WjAhMQ0wCwYDVQQKEwRKdWp1MRAwDgYDVQQD
      EwdqdWp1LWNhMIIBojANBgkqhkiG9w0BAQEFAAOCAY8AMIIBigKCAYEAvG5/q1H2
      cCXVo1NT8V8KKlMzFg9DbYFoFWeS3vjIHu5enfiaG4g+hqv06vMXLpXycq15kziJ
      fFzgClDrXF97ybrACaZA13Jk6ygw2TekVGw9l2eGQJ7pZ9jdqfPFi6NeMPeaCOy1
      KABPDzMgH+VpUJpz/C/QAyd2I0T4YxYUxiY0EFQkCDJlW+pwUntZWv1S9T9OSodI
      A531G8Znxaelq4qjmOjS/8xN8Y3SszkfrqOgt7RJS/Cz5gMCvLPJ6YPevxNBPocN
      LOXPMDLb6u83effSNlWF0nN42CIeL5cZ14u2cRZwz0V1XctzVcVlfjiiiCSlBxEa
      w4/Z3nWTbC9FtLWwR0XbKVVV5IIZiEYPvY6Oq7ki0dWYETH8XdSNqJ9VJ630fwiz
      wpPPZXzQnT3LKJrPzKygMG8sUuMnqXNZofKlhk3PSFTfMjEXRvS61NicShOk0UDZ
      RuSNby8IVPahQnwWjS96DKFfpBu2wzF6c9mufrgvZFka30zjt2KLsWHlAgMBAAGj
      QjBAMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRo
      Jm3fPHdtO/pbUlbDN2pK7ARPIDANBgkqhkiG9w0BAQsFAAOCAYEAuaY0/P084igl
      K/tEPrn/GCqAvdAOARUFKNtO6X5fPel3sSaKQ3L5PCtV0Xwv+iFoy9Z5DhX8KTQA
      MvjxoyenX94p2cyKwoutVobUM1KO1j8E1Z3hPwtUJifh9e6MyJnoxBuW6t/4dwWg
      ZPyzKn0f2iAvkxDxk4ifUKA9D8m6gY2kG+866/zsrk75onWXq1OI4K/lVeyakMj7
      9RLBUt3Y2stLYRCahEYYXyPMCcXkG6MfH+o+zSBaniwAN+CPa4OfUXFfJPN995C0
      I2oGEmiXz4PvLiYSwjc0DlVdgWAUf0cIoNsiRYbgEM+/iSm2o1gNGmOAmP3tsfRG
      r4nAdfuWBzKM762dPYtJoh7d8/BU0zC9snxJ9xAShrlsBHhY3mj1Iiv7cZhJ5Mi1
      1rfkwL5vg3BqQmPkj6MeIFn3AThmRp2HlOebBusl6DwCz9Nnzq6rgh74B/kI4C2Y
      e5HDhcTJxarNKtSELPaBNIEv/YR9iTgrxf8HBHl4ayaXKODztE+m
      -----END CERTIFICATE-----
  models:
    controller:
      model-uuid: cff75055-3dad-4c7d-8762-8fcb83b0e766
      unit-count: 1
  current-model: admin/controller
  account:
    user: admin
    access: superuser

description: updated
Revision history for this message
Ian Booth (wallyworld) wrote :

I tested with the 3.4 snap and confirmed an upgrade from 3.4.3 breaks. It seems the docker client we use to publish the manifests may have changed, since we don't support the fat manifest header and that's what we're now getting.

$ juju status
Model Controller Cloud/Region Version SLA Timestamp
controller test microk8s/localhost 3.4.3 unsupported 10:33:57+10:00

App Version Status Scale Charm Channel Rev Address Exposed Message
controller active 1 juju-controller 3.4/stable 101 no

Unit Workload Agent Address Ports Message
controller/0* active idle 10.1.78.73 37017/TCP

$ juju upgrade-controller
ERROR cannot get architecture for jujud-operator:3.4.3: can not get manifests for jujud-operator:3.4.3: manifest response version "application/vnd.docker.distribution.manifest.list.v2+json" not supported (not supported)

Changed in juju:
importance: Undecided → Critical
milestone: none → 3.4.5
status: New → Triaged
Revision history for this message
Jeffrey Chang (modern911) wrote :

I see same issue on juju 3.3.5 as well.

ERROR cannot get architecture for jujud-operator:3.3.5: can not get manifests for jujud-operator:3.3.5: manifest response version "application/vnd.docker.distribution.manifest.list.v2+json" not supported (not supported)

Ian Booth (wallyworld)
Changed in juju:
assignee: nobody → Ian Booth (wallyworld)
status: Triaged → In Progress
Revision history for this message
Ian Booth (wallyworld) wrote :

A fix will land in 3.4 and later.

https://github.com/juju/juju/pull/17762

Ian Booth (wallyworld)
Changed in juju:
status: In Progress → Fix Committed
Changed in juju:
milestone: 3.4.5 → 3.4.6
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.