2.8.7 juju migrate fails: agents failed phase "validating"

Bug #1910595 reported by Heather Lanigan on 2021-01-07
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
juju
High
Joseph Phillips
2.8
High
Joseph Phillips

Bug Description

I've tried 4 times to migrate a model running charmed-kubernetes from a 2.8.7 controller to a 2.9-rc3 controller unsuccessfully. First 3 times failure due to:

2021-01-07 18:58:20 ERROR juju.worker.migrationmaster.9d7c87 worker.go:758 agents failed phase "validating" (units: flannel/4)
2021-01-07 18:58:20 ERROR juju.worker.migrationmaster.9d7c87 worker.go:292 validating, some agents reported failure
2021-01-07 19:08:41 ERROR juju.worker.migrationmaster.9d7c87 worker.go:758 agents failed phase "validating" (units: etcd/2)
2021-01-07 19:08:41 ERROR juju.worker.migrationmaster.9d7c87 worker.go:292 validating, some agents reported failure
2021-01-07 19:11:37 ERROR juju.worker.migrationmaster.9d7c87 worker.go:758 agents failed phase "validating" (machines: 3)
2021-01-07 19:11:37 ERROR juju.worker.migrationmaster.9d7c87 worker.go:292 validating, some agents reported failure

Agent errors:
juju.worker.migrationminion reporting back for phase VALIDATION: false
juju.worker.migrationminion reporting back for phase VALIDATION: true
juju.worker.dependency "migration-minion" manifold worker returned unexpected error: failed to report phase progress: conflicting reports received for 0c3ae1d9-d660-4841-8330-423dd59d7c87:0/VALIDATION/unit-flannel-4

Pastebins of the errors reported by these agents.
https://pastebin.canonical.com/p/d8q2DYgZPj/
https://pastebin.canonical.com/p/JGBGbQbSxs/
https://pastebin.canonical.com/p/9my9mPRTKK/

Heather Lanigan (hmlanigan) wrote :
Download full text (3.3 KiB)

Another attempt produced on half of the machines in the model:

machine-8: 21:48:10 INFO juju.worker.migrationminion migration phase is now: VALIDATION
machine-8: 21:48:10 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:10 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:10 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:11 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:11 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:11 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:11 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:12 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:12 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:13 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:13 ERROR juju.worker.migrationminion validation failed: failed to open API to target controller: try again (try again)
machine-8: 21:48:13 INFO juju.worker.migrationminion migration phase is now: VALIDATION
machine-8: 21:48:13 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:14 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:14 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:14 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:14 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:14 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:15 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:15 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:16 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:17 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:17 ERROR juju.worker.migrationminion validation failed: failed to open API to target controller: invalid e...

Read more...

Changed in juju:
milestone: none → 2.9-rc4
John A Meinel (jameinel) wrote :

The pastebins look like we are successfully connecting to the new controller, but it is rejecting our login information.

Changed in juju:
status: New → In Progress
importance: Undecided → High
assignee: nobody → Joseph Phillips (manadart)
Changed in juju:
status: In Progress → Fix Committed
Changed in juju:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers