"Invalid entity name or password" error with valid credentials.
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Canonical Juju |
Fix Released
|
Critical
|
Anastasia | ||
juju-core |
Fix Released
|
Critical
|
Anastasia | ||
1.25 |
Fix Released
|
Critical
|
Anastasia |
Bug Description
(agent uninstall addressed in lp:1580233 and lp:1580221)
[Environment]
Juju-core 1.24.5 - 1.25.5 (Still reproducible with 1.24.7)
Trusty 14.04
[Description]
We noticed this behavior on multiple machines,
First for some unknown reason , the agent fails to authenticate on the state servers.
184176-2015-10-22 09:21:50 INFO juju.api apiclient.go:263 connection established to "wss://
184177-2015-10-22 09:21:50 ERROR juju.cmd.jujud agent.go:298 agent terminating due to error returned during API open: invalid entity name or password
Then because of this code, the workers are stopped, and the uninstall of the agent occurs.
184168-2015-10-22 09:06:02 INFO juju.worker runner.go:275 stopped "api", err: try again
184169-2015-10-22 09:06:02 DEBUG juju.worker runner.go:203 "api" done: try again
184170-2015-10-22 09:06:02 ERROR juju.worker runner.go:223 exited "api": try again
184171-2015-10-22 09:06:02 INFO juju.worker runner.go:261 restarting "api" in 3s
184172-2015-10-22 09:06:05 INFO juju.worker runner.go:269 start "api"
184173-2015-10-22 09:06:05 INFO juju.api apiclient.go:331 dialing "wss://
184174-2015-10-22 09:06:05 INFO juju.api apiclient.go:263 connection established to "wss://
184175-2015-10-22 09:21:49 INFO juju.api apiclient.go:331 dialing "wss://
184176-2015-10-22 09:21:50 INFO juju.api apiclient.go:263 connection established to "wss://
184177-2015-10-22 09:21:50 ERROR juju.cmd.jujud agent.go:298 agent terminating due to error returned during API open: invalid entity name or password
184178:2015-10-22 09:21:50 INFO juju.worker runner.go:275 stopped "api", err: agent should be terminated
184179:2015-10-22 09:21:50 DEBUG juju.worker runner.go:203 "api" done: agent should be terminated
184180:2015-10-22 09:21:50 ERROR juju.worker runner.go:212 fatal "api": agent should be terminated
184181-2015-10-22 09:21:50 DEBUG juju.worker runner.go:248 killing "statestarter"
184182-2015-10-22 09:21:50 DEBUG juju.worker runner.go:248 killing "termination"
184183-2015-10-22 09:21:50 INFO juju.worker runner.go:275 stopped "statestarter", err: <nil>
184184-2015-10-22 09:21:50 INFO juju.worker runner.go:275 stopped "termination", err: <nil>
184185-2015-10-22 09:21:50 DEBUG juju.worker runner.go:203 "statestarter" done: <nil>
184186-2015-10-22 09:21:50 DEBUG juju.worker runner.go:227 no restart, removing "statestarter" from known workers
184187-2015-10-22 09:21:50 DEBUG juju.worker runner.go:203 "termination" done: <nil>
184188-2015-10-22 09:21:50 DEBUG juju.worker runner.go:227 no restart, removing "termination" from known workers
184189-2015-10-22 09:21:50 DEBUG juju.service discovery.go:115 discovered init system "upstart" from local host
184190-2015-10-22 09:21:51 DEBUG juju.service discovery.go:115 discovered init system "upstart" from local host
184191-2015-10-22 09:21:51 INFO juju.cmd supercommand.go:436 command finished
After this point, you can observe messages such as:
/proc/self/fd/9: 9: exec: /var/lib/
/proc/self/fd/9: 9: exec: /var/lib/
[Other ways to reproduce]
1) Modify the apipassword on the /var/lib/
2) Restart jujud-machine-8
The following error is printed:
2015-11-10 14:14:28 ERROR juju.worker.
2015-11-10 14:14:28 INFO juju.worker runner.go:275 stopped "api", err: agent should be terminated
2015-11-10 14:14:28 DEBUG juju.worker runner.go:203 "api" done: agent should be terminated
2015-11-10 14:14:28 ERROR juju.worker runner.go:212 fatal "api": agent should be terminated
3) /var/lib/juju dissapears
ls -lh /var/lib/jujuju
ls: cannot access /var/lib/juju: No such file or directory
Note:
Please note that we still don't know exactly what caused the authentication to fail, but, even if you are manually modifying the agent.conf , I don't think that a correct reaction is to wipe the juju agent from the machine.
description: | updated |
tags: | added: sts-needs-review |
tags: | removed: sts sts-needs-review |
description: | updated |
Changed in juju-core: | |
assignee: | nobody → Eric Snow (ericsnowcurrently) |
Changed in juju-core: | |
status: | Triaged → In Progress |
summary: |
- Invalid entity name or password error, causes Juju to uninstall + "Invalid entity name or password" error with valid credentials. |
description: | updated |
Changed in juju-core: | |
milestone: | none → 2.0-beta7 |
Changed in juju-core: | |
milestone: | 2.0-beta7 → 2.0-beta8 |
tags: | added: blocker |
Changed in juju-core: | |
assignee: | nobody → Anastasia (anastasia-macmood) |
status: | Triaged → In Progress |
tags: | removed: blocker |
Changed in juju-core: | |
status: | In Progress → Fix Committed |
tags: | removed: blocker |
Changed in juju-core: | |
status: | Fix Committed → In Progress |
importance: | Critical → High |
tags: | added: blocker |
Changed in juju-core: | |
status: | Fix Committed → Fix Released |
tags: | removed: blocker |
affects: | juju-core → juju |
Changed in juju: | |
milestone: | 2.0-beta8 → none |
milestone: | none → 2.0-beta8 |
Changed in juju-core: | |
assignee: | nobody → Anastasia (anastasia-macmood) |
importance: | Undecided → Critical |
status: | New → Fix Released |
I agree that it is a bit of an overreaction to wipe the juju agent from the machine.
I am concerned about the initial authentication error, though. Could you upload logs from the state servers and the machine which failed to authenticate?