Removing a machine should remove any series-upgrade side-effects

Bug #1879663 reported by Joseph Phillips on 2020-05-20
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
juju
High
Simon Richardson
2.7
High
Simon Richardson
2.8
High
Simon Richardson

Bug Description

The following sequence of events has been observed:

- A series upgrade is commenced.
- The upgrade cannot progress, probably due to https://bugs.launchpad.net/juju/+bug/1855013.
- The machine is removed.
- There remains a lock document for the the machine in machineUpgradeSeriesLocks, and;
- Leadership that was pinned for applications with units on the machine remains pinned.

If the pinned leader for an application was one of the units on the removed machine, this can mean an application with no leaders, and no units able to successfully claim leadership.

When the leadership pinning facade was conceived during the implementation of series upgrades, it was anticipated that there might be future client exposure of the functionality. This might be something to consider, but at very least a removed machine should properly clean up after itself.

summary: - Removing a machine should remove any series-upgrade locks
+ Removing a machine should remove any series-upgrade side-effects
Changed in juju:
status: New → Triaged
Joseph Phillips (manadart) wrote :

This is one possible cause for the symptoms observed here:
https://bugs.launchpad.net/juju/+bug/1858693

Joseph Phillips (manadart) wrote :

This patch should ensure that we can't get into a situation of leadership pinned to a removed unit:
https://github.com/juju/juju/pull/11593

Proper clean-up of machine upgrade-series artifacts is outstanding and in-progress.

Changed in juju:
importance: Undecided → High
assignee: nobody → Joseph Phillips (manadart)
milestone: none → 2.9-beta1
status: Triaged → In Progress
Changed in juju:
assignee: Joseph Phillips (manadart) → Simon Richardson (simonrichardson)
Joseph Phillips (manadart) wrote :

Upgrade locks prevent machine removal without force and using force ensures lock removal under:
https://github.com/juju/juju/pull/11620

Joseph Phillips (manadart) wrote :

Still waiting on one more patch to handle the machine-level application leader un-pinning.

Joseph Phillips (manadart) wrote :

Patch for unpinning for 2.7:
https://github.com/juju/juju/pull/11639

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers