Canonical Juju

2.8.7 juju migrate fails: agents failed phase "validating"

Bug #1910595 reported by Heather Lanigan
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Canonical Juju
Fix Released
High
Joseph Phillips
Canonical Juju 2.9-rc4
2.8
Fix Released
High
Joseph Phillips
Canonical Juju 2.8.8

Bug Description

I've tried 4 times to migrate a model running charmed-kubernetes from a 2.8.7 controller to a 2.9-rc3 controller unsuccessfully. First 3 times failure due to:

2021-01-07 18:58:20 ERROR juju.worker.migrationmaster.9d7c87 worker.go:758 agents failed phase "validating" (units: flannel/4)
2021-01-07 18:58:20 ERROR juju.worker.migrationmaster.9d7c87 worker.go:292 validating, some agents reported failure
2021-01-07 19:08:41 ERROR juju.worker.migrationmaster.9d7c87 worker.go:758 agents failed phase "validating" (units: etcd/2)
2021-01-07 19:08:41 ERROR juju.worker.migrationmaster.9d7c87 worker.go:292 validating, some agents reported failure
2021-01-07 19:11:37 ERROR juju.worker.migrationmaster.9d7c87 worker.go:758 agents failed phase "validating" (machines: 3)
2021-01-07 19:11:37 ERROR juju.worker.migrationmaster.9d7c87 worker.go:292 validating, some agents reported failure

Agent errors:
juju.worker.migrationminion reporting back for phase VALIDATION: false
juju.worker.migrationminion reporting back for phase VALIDATION: true
juju.worker.dependency "migration-minion" manifold worker returned unexpected error: failed to report phase progress: conflicting reports received for 0c3ae1d9-d660-4841-8330-423dd59d7c87:0/VALIDATION/unit-flannel-4

Pastebins of the errors reported by these agents.
https://pastebin.canonical.com/p/d8q2DYgZPj/
https://pastebin.canonical.com/p/JGBGbQbSxs/
https://pastebin.canonical.com/p/9my9mPRTKK/

Revision history for this message
Heather Lanigan (hmlanigan) wrote :
Download full text (3.3 KiB)

Another attempt produced on half of the machines in the model:

machine-8: 21:48:10 INFO juju.worker.migrationminion migration phase is now: VALIDATION
machine-8: 21:48:10 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:10 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:10 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:11 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:11 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:11 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:11 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:12 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:12 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:13 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:13 ERROR juju.worker.migrationminion validation failed: failed to open API to target controller: try again (try again)
machine-8: 21:48:13 INFO juju.worker.migrationminion migration phase is now: VALIDATION
machine-8: 21:48:13 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:14 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:14 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:14 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:14 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:14 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:15 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:15 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:16 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:17 INFO juju.api connection established to "wss://10.74.147.148:17070/model/0c3ae1d9-d660-4841-8330-423dd59d7c87/api"
machine-8: 21:48:17 ERROR juju.worker.migrationminion validation failed: failed to open API to target controller: invalid e...

Read more...

Simon Richardson (simonrichardson)
Changed in juju:
milestone: none → 2.9-rc4
Revision history for this message
John A Meinel (jameinel) wrote :

The pastebins look like we are successfully connecting to the new controller, but it is rejecting our login information.

Joseph Phillips (manadart)
Changed in juju:
status: New → In Progress
importance: Undecided → High
assignee: nobody → Joseph Phillips (manadart)
Revision history for this message
Joseph Phillips (manadart) wrote :

https://github.com/juju/juju/pull/12503

Joseph Phillips (manadart)
Changed in juju:
status: In Progress → Fix Committed
Joseph Phillips (manadart)
Changed in juju:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.