Removing a machine should remove any series-upgrade side-effects

Bug #1879663 reported by Joseph Phillips
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Canonical Juju
Fix Released
High
Unassigned
2.7
Fix Released
High
Unassigned
2.8
Fix Released
High
Unassigned

Bug Description

The following sequence of events has been observed:

- A series upgrade is commenced.
- The upgrade cannot progress, probably due to https://bugs.launchpad.net/juju/+bug/1855013.
- The machine is removed.
- There remains a lock document for the the machine in machineUpgradeSeriesLocks, and;
- Leadership that was pinned for applications with units on the machine remains pinned.

If the pinned leader for an application was one of the units on the removed machine, this can mean an application with no leaders, and no units able to successfully claim leadership.

When the leadership pinning facade was conceived during the implementation of series upgrades, it was anticipated that there might be future client exposure of the functionality. This might be something to consider, but at very least a removed machine should properly clean up after itself.

summary: - Removing a machine should remove any series-upgrade locks
+ Removing a machine should remove any series-upgrade side-effects
Changed in juju:
status: New → Triaged
Revision history for this message
Joseph Phillips (manadart) wrote :

This is one possible cause for the symptoms observed here:
https://bugs.launchpad.net/juju/+bug/1858693

Revision history for this message
Joseph Phillips (manadart) wrote :

This patch should ensure that we can't get into a situation of leadership pinned to a removed unit:
https://github.com/juju/juju/pull/11593

Proper clean-up of machine upgrade-series artifacts is outstanding and in-progress.

Changed in juju:
importance: Undecided → High
assignee: nobody → Joseph Phillips (manadart)
milestone: none → 2.9-beta1
status: Triaged → In Progress
Changed in juju:
assignee: Joseph Phillips (manadart) → Simon Richardson (simonrichardson)
Revision history for this message
Joseph Phillips (manadart) wrote :

Upgrade locks prevent machine removal without force and using force ensures lock removal under:
https://github.com/juju/juju/pull/11620

Revision history for this message
Joseph Phillips (manadart) wrote :

Still waiting on one more patch to handle the machine-level application leader un-pinning.

Revision history for this message
Joseph Phillips (manadart) wrote :

Patch for unpinning for 2.7:
https://github.com/juju/juju/pull/11639

Changed in juju:
milestone: 2.9-beta1 → 2.9-rc1
Pen Gale (pengale)
Changed in juju:
milestone: 2.9-rc1 → 3.0.0
Harry Pidcock (hpidcock)
Changed in juju:
assignee: Simon Richardson (simonrichardson) → nobody
Revision history for this message
Joseph Phillips (manadart) wrote :

This one is actually all done I believe.

Changed in juju:
status: In Progress → Fix Committed
John A Meinel (jameinel)
Changed in juju:
milestone: 3.0.0 → 3.0-beta1
Ian Booth (wallyworld)
Changed in juju:
status: Fix Committed → Fix Released
Ian Booth (wallyworld)
Changed in juju:
status: Fix Released → Fix Committed
Changed in juju:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.