juju upgrade-series prepare $MACHINE bionic will run and hang if already on Bionic

[Ryan Farrell]

While performing series upgrades we accidentally ran juju upgrade-series prepare $machine bionic to a host which had already been upgraded to Bionic.

Juju started running the preparation hooks, but ultimately hung.

I can see from logs that it's trying and failing to do "lstat /var/lib/juju/tools/2.6.10-bionic-amd64"

The call to upgrade-series prepare should first verify what series the current OS is on and error out without performing any actions if its the same as the specified series.

Snip from /var/log/juju/machine-10.log:

2020-04-01 16:06:03 INFO juju.worker.upgradeseries worker.go:172 machine series upgrade status is "prepare started"
2020-04-01 16:06:05 INFO juju.worker.upgradeseries worker.go:370 unpin leader for application "hw-health"
2020-04-01 16:06:05 INFO juju.worker.upgradeseries worker.go:370 unpin leader for application "neutron-openvswitch"
2020-04-01 16:06:05 INFO juju.worker.upgradeseries worker.go:370 unpin leader for application "telegraf"
2020-04-01 16:06:05 INFO juju.worker.upgradeseries worker.go:370 unpin leader for application "filebeat"
2020-04-01 16:06:05 INFO juju.worker.upgradeseries worker.go:370 unpin leader for application "ceilometer-agent"
2020-04-01 16:06:05 INFO juju.worker.upgradeseries worker.go:370 unpin leader for application "nrpe-host"
2020-04-01 16:06:05 INFO juju.worker.upgradeseries worker.go:370 unpin leader for application "ntp"
2020-04-01 16:06:05 INFO juju.worker.upgradeseries worker.go:370 unpin leader for application "sysconfig-compute"
2020-04-01 16:06:05 INFO juju.worker.upgradeseries worker.go:370 unpin leader for application "bcache-tuning"
2020-04-01 16:06:05 INFO juju.worker.upgradeseries worker.go:370 unpin leader for application "clamav"
2020-04-01 16:06:05 INFO juju.worker.upgradeseries worker.go:370 unpin leader for application "trilio-data-mover"
2020-04-01 16:06:05 INFO juju.worker.upgradeseries worker.go:370 unpin leader for application "ceph-osd"
2020-04-01 16:06:05 INFO juju.worker.upgradeseries worker.go:370 unpin leader for application "lldpd"
2020-04-01 16:06:05 INFO juju.worker.upgradeseries worker.go:370 unpin leader for application "nova-compute-kvm"
2020-04-01 16:06:05 INFO juju.worker.upgradeseries worker.go:370 unpin leader for application "rsyslog-forwarder-ha"
2020-04-01 16:06:05 INFO juju.worker.upgradeseries worker.go:370 unpin leader for application "landscape-client"
2020-04-01 16:06:05 DEBUG juju.service discovery.go:64 discovered init system "systemd" from series "bionic"
2020-04-01 16:06:05 INFO juju.worker.upgradeseries worker.go:218 preparing service units for series upgrade
2020-04-01 16:06:05 DEBUG juju.service discovery.go:64 discovered init system "systemd" from series "bionic"
2020-04-01 16:06:05 DEBUG juju.service discovery.go:64 discovered init system "systemd" from series "bionic"
2020-04-01 16:06:05 DEBUG juju.worker.dependency engine.go:599 "upgrade-series" manifold worker stopped: failed to copy tools: lstat /var/lib/juju/tools/2.6.10-bionic-amd64: no such file or directory
2020-04-01 16:06:05 ERROR juju.worker.dependency engine.go:663 "upgrade-series" manifold worker returned unexpected error: failed to copy tools: lstat /var/lib/juju/tools/2.6.10-bionic-amd64: no such file or dir
ectory
2020-04-01 16:06:05 DEBUG juju.worker.dependency engine.go:674 stack trace:
lstat /var/lib/juju/tools/2.6.10-bionic-amd64: no such file or directory
/workspace/_build/src/github.com/juju/juju/service/agentconf.go:294: failed to copy tools
/workspace/_build/src/github.com/juju/juju/worker/upgradeseries/upgrader.go:129:
/workspace/_build/src/github.com/juju/juju/worker/upgradeseries/upgrader.go:84:
/workspace/_build/src/github.com/juju/juju/worker/upgradeseries/worker.go:228:
/workspace/_build/src/github.com/juju/juju/worker/upgradeseries/worker.go:211:
/workspace/_build/src/github.com/juju/juju/worker/upgradeseries/worker.go:182:
/workspace/_build/src/github.com/juju/juju/worker/upgradeseries/worker.go:145:

[Peter Matulis (petermatulis)]

I tried this on 2.7.6 and I do not observe this.

$ juju upgrade-series --yes 7 prepare bionic
ERROR machine-7 is already running series bionic

[Tim Penhey (thumper)]

Can you please recheck with 2.7?

[James Troup (elmo)]

We say this on 2.7 with an upgrade Ian was helping me with.

[Tim Penhey (thumper)]

I think this is more a situation where Juju thinks the machine is 'xenial', but the machine is actually 'bionic' because it was upgraded out of band.

[Joseph Phillips (manadart)]

@thumper is correct. I've reproduced.

[Joseph Phillips (manadart)]

https://github.com/juju/juju/pull/11764