api cannot connect, fills deployed machine log with spam

[William Reade]

Apparently trying to connect as the correct entity, but repeatedly rejected. Seen on trunk as upgraded from 1.10; not investigated further.

[John A Meinel (jameinel)]

Looking at the log, it is spinning trying to login with "oldpassword". When I look at a freshly started 1.11 trunk, oldpassword is set to "". So it seems to try a different password, and then calls SetPasswords.

[John A Meinel (jameinel)]

Found the problem.

Specifically, 1.10 doesn't set entity.PasswordHash when you set the password for an agent. It just sets the Mongo password.

Which means that after an upgrade, an agent can connect to the DB and do stuff, but cannot connect over the API because it has no (valid) password.

So we need to change the agent runners so that after being upgraded, if they have a State connection, they go check if their entity has a valid password hash, and if not, give it one.

[John A Meinel (jameinel)]

I just realized things are probably worse than we expected.

Specifically, on machine-0 being unable to connect the API is bad, but it doesn't stop any of the actual workers.

However, the code in MachineAgent is:

if a.MachineId == "0" {
  ...
  ensureStateWorker()
}
a.runner.StartWorker("api", func() (worker.Worker, error) {
  return a.APIWorker(ensureStateWorker)
}

And the APIWorker call actually starts the state worker *when it sees we need a state connection after reading the API*.

Which means that upgrading juju 1.10 => 1.11 will spam the log file for machine-0, but fail to start any actual worker tasks for all the other machine agents.