Upgrading a HA controller cluster, the jujud processes are failing to restart causing complete loss of the controller. This is the machine log on one of the controllers at the time of the upgrade;
ERROR must restart: an agent upgrade is available
2020-09-16 12:54:43 INFO juju.cmd supercommand.go:54 running jujud [2.8.2 0 a44e6eb38430da695737f5e9f37819478b9587c3 gc go1.14.9]
2020-09-16 12:54:43 DEBUG juju.cmd supercommand.go:55 args: []string{"/var/lib/juju/tools/machine-4/jujud", "machine", "--data-dir", "/var/lib/juju", "--machine-id", "4", "--debug"}
2020-09-16 12:54:43 DEBUG juju.utils gomaxprocs.go:24 setting GOMAXPROCS to 2
2020-09-16 12:54:43 DEBUG juju.agent agent.go:583 read agent config, format "2.0"
2020-09-16 12:54:43 INFO juju.cmd.jujud agent.go:138 setting logging config to "<root>=WARNING;unit=DEBUG"
2020-09-16 12:54:44 ERROR juju.worker.dependency engine.go:671 "api-caller" manifold worker returned unexpected error: [b5fb8e] "machine-4" cannot open api: unable to connect to API: dial tcp 127.0.0.1:17070: connect: connection refused
2020-09-16 12:54:48 ERROR juju.worker.dependency engine.go:671 "api-caller" manifold worker returned unexpected error: [b5fb8e] "machine-4" cannot open api: unable to connect to API: dial tcp 127.0.0.1:17070: connect: connection refused
2020-09-16 12:54:50 WARNING juju.state txns.go:89 Running no-op transaction - called by /workspace/_build/src/github.com/juju/juju/state/database.go:399 /workspace/_build/src/github.com/juju/juju/state/upgrades.go:2994 /workspace/_build/src/github.com/juju/juju/state/upgrades.go:59 /workspace/_build/src/github.com/juju/juju/state/upgrades.go:2961 /workspace/_build/src/github.com/juju/juju/upgrades/backend.go:354 /workspace/_build/src/github.com/juju/juju/upgrades/steps_282.go:21 /workspace/_build/src/github.com/juju/juju/upgrades/upgrade.go:184 /workspace/_build/src/github.com/juju/juju/upgrades/upgrade.go:137 /workspace/_build/src/github.com/juju/juju/upgrades/upgrade.go:113 /workspace/_build/src/github.com/juju/juju/worker/upgradedatabase/manifold.go:80 /workspace/_build/src/github.com/juju/juju/worker/upgradedatabase/worker.go:240 /workspace/_build/src/github.com/juju/juju/cmd/jujud/agent/agent.go:112 /workspace/_build/src/github.com/juju/juju/cmd/jujud/agent/machine.go:659 /workspace/_build/src/github.com/juju/juju/worker/upgradedatabase/worker.go:219 /workspace/_build/src/github.com/juju/juju/worker/upgradedatabase/worker.go:188 /workspace/_build/src/github.com/juju/juju/vendor/gopkg.in/tomb.v2/tomb.go:163 /workspace/_build/src/github.com/juju/juju/vendor/gopkg.in/tomb.v2/tomb.go:159
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x1d031ca]
goroutine 460 [running]:
github.com/juju/juju/state.ResetDefaultRelationLimitInCharmMetadata.func1(0xc00072c7e0, 0x0, 0x0)
/workspace/_build/src/github.com/juju/juju/state/upgrades.go:2973 +0x64a
github.com/juju/juju/state.runForAllModelStates(0xc0008989c0, 0x4a01118, 0x0, 0x0)
/workspace/_build/src/github.com/juju/juju/state/upgrades.go:59 +0x308
github.com/juju/juju/state.ResetDefaultRelationLimitInCharmMetadata(0xc0008989c0, 0xc00008ca80, 0x7ff9c67037d0)
/workspace/_build/src/github.com/juju/juju/state/upgrades.go:2961 +0x37
github.com/juju/juju/upgrades.stateBackend.ResetDefaultRelationLimitInCharmMetadata(0xc0008989c0, 0x5266160, 0xc0008989c0)
/workspace/_build/src/github.com/juju/juju/upgrades/backend.go:354 +0x2b
github.com/juju/juju/upgrades.stateStepsFor282.func1(0x51cfd60, 0xc0000baf90, 0xc00097f901, 0xc00097f9b0)
/workspace/_build/src/github.com/juju/juju/upgrades/steps_282.go:21 +0x48
github.com/juju/juju/upgrades.(*upgradeStep).Run(0xc0000baf60, 0x51cfd60, 0xc0000baf90, 0x489c4c6, 0x18)
/workspace/_build/src/github.com/juju/juju/upgrades/upgrade.go:184 +0x3e
github.com/juju/juju/upgrades.runUpgradeSteps(0xc000aa1e00, 0xc00097f520, 0x1, 0x1, 0x51cfd60, 0xc0000baf90, 0xc000aa1e00, 0xc00012f200)
/workspace/_build/src/github.com/juju/juju/upgrades/upgrade.go:137 +0x21d
github.com/juju/juju/upgrades.PerformStateUpgrade(0x2, 0x8, 0x0, 0x0, 0x1, 0x0, 0xc00097f520, 0x1, 0x1, 0x51cfd60, ...)
/workspace/_build/src/github.com/juju/juju/upgrades/upgrade.go:113 +0xb0
github.com/juju/juju/worker/upgradedatabase.Manifold.func1.2(0x2, 0x8, 0x0, 0x0, 0x1, 0x0, 0xc00097f520, 0x1, 0x1, 0xc0006a8320, ...)
/workspace/_build/src/github.com/juju/juju/worker/upgradedatabase/manifold.go:80 +0x90
github.com/juju/juju/worker/upgradedatabase.(*upgradeDB).runUpgradeSteps(0xc0001b0a00, 0x7ff99fa85e50, 0xc00012f200, 0x0, 0x7ff99fa85e50)
/workspace/_build/src/github.com/juju/juju/worker/upgradedatabase/worker.go:240 +0x1d8
github.com/juju/juju/cmd/jujud/agent.(*agentConf).ChangeConfig(0xc0004fbcb0, 0xc00097f510, 0x0, 0x0)
/workspace/_build/src/github.com/juju/juju/cmd/jujud/agent/agent.go:112 +0xab
github.com/juju/juju/cmd/jujud/agent.(*MachineAgent).ChangeConfig(0xc0005ffe60, 0xc00097f510, 0xc0006c2701, 0xc00097f510)
/workspace/_build/src/github.com/juju/juju/cmd/jujud/agent/machine.go:659 +0x41
github.com/juju/juju/worker/upgradedatabase.(*upgradeDB).runUpgrade(0xc0001b0a00)
/workspace/_build/src/github.com/juju/juju/worker/upgradedatabase/worker.go:219 +0x11d
github.com/juju/juju/worker/upgradedatabase.(*upgradeDB).run(0xc0001b0a00, 0x0, 0x0)
/workspace/_build/src/github.com/juju/juju/worker/upgradedatabase/worker.go:188 +0x237
gopkg.in/tomb%2ev2.(*Tomb).run(0xc0001b0a00, 0xc00083de70)
/workspace/_build/src/github.com/juju/juju/vendor/gopkg.in/tomb.v2/tomb.go:163 +0x38
created by gopkg.in/tomb%2ev2.(*Tomb).Go
/workspace/_build/src/github.com/juju/juju/vendor/gopkg.in/tomb.v2/tomb.go:159 +0xba
I did just do an upgrade using HA 2.8.1 on LXD running a single application to 2.8.2 using the official published agents.
So simple upgrades are not affected by this.
However, this is a very critical issue so we'll keep digging to make sure it is fixed.