Attempting to login to a juju controller as a local user causes the controller to issue a login challenge in the form of a macaroon discharge request. The client is then directed to https://controller.url/auth/discharge which causes it to then perform an interaction that uses the users password.
Because in a HA configuration each controller machine has a different public-key configured if the discharge request goes to the wrong controller then it fails as it doesn't understand the incoming request.
In the JAAS configuration we use a HA controller cluster, but front the controller machines with a redundant haproxy application. This allows us to give the controller a DNS name signed by a CA and other benefits. In such a configuration there is no guarantee that two requests from the same client will go to the same controller machine, as a general rule we try to load-balance across the controllers. As a result in this configuration we are seeing login failures trying to authenticate with the DNS names of the controller.
One can easily see the different configuration by repeatedly requesting the public key of a controller e.g:
$ curl https://aws-eu-west-1.mhilton.dev:443/auth/publickey
{"PublicKey":"PhyUBdgbP5TEm/H5ZCVgYjYEcgoOKoOLCfW4JE05kV0="}
$ curl https://aws-eu-west-1.mhilton.dev:443/auth/publickey
{"PublicKey":"HeToyo6sGserEgN2rro0d6aTmxHHS2Oi+IZrs1oR/lg="}
$ curl https://aws-eu-west-1.mhilton.dev:443/auth/publickey
{"PublicKey":"3noVPedEYTfvEwVStMFmtle+O3vaw+oHiiA/8zsVanY="}
I have attached a full trace of a login attempt in case it helps.
An interesting side note is that this does not seem to have been a problem before juju 2.7 however older controllers do exhibit the same behaviour w.r.t having a single public key per controller machine. It seems that something else has come along to cause that to become a problem.
While the Juju API servers should probably share a macaroon bakery PublicKey in HA mode, this is presumably a feature request rather than a regression. There do not appear to be any changes in this area between Juju 2.6 and 2.7. Also, as already noted, if you query /auth/publickey directly on the controllers (port 17070), the same behaviour is observed (a different PublicKey per server) in both versions.
In the case of JAAS, you're hitting haproxy and the controller that you then reach is determined by the haproxy configuration - this appears to be using least connection balancing, so if this is working it is probably only by chance rather than design (there is probability that repeated connections will hit the same API server, since it has the least connections). To avoid this the haproxy configuration could use 'balance source' (although that's not without its own set of potential gotchas). This may also be currently working if the /auth requests are occurring over the same TCP connection (e.g. a persistent HTTP session).