kubernetes-master stuck at Waiting to retry addon deployment

[Илия Анастасов]

I have CDK1.19 installation on manual cloud/lxd (with ceph bundle as CMR) - for a couple of days now, and today I found that the current master unit was waiting with "Waiting to retry addon deployment" message on juju status screen. The only thing that i was able to find was that today there was some snap activities and cdk-addons version was refreshed to 1.19.4 rev.4723, while the rest was sitting on 1.19.3. I was able to resolved that by reinstalling cdk-addons package with 1.19.3 version.

Here are some entries from juju debug-log:

unit-kubernetes-master-0: 09:45:24 INFO unit.kubernetes-master/0.juju-log Reactive main running for hook update-status
unit-kubernetes-master-0: 09:45:46 INFO unit.kubernetes-master/0.juju-log Initializing Snap Layer
unit-kubernetes-master-0: 09:45:48 WARNING unit.kubernetes-master/0.update-status All snaps up to date.
unit-kubernetes-master-0: 09:45:48 INFO unit.kubernetes-master/0.juju-log Initializing Apt Layer
unit-kubernetes-master-0: 09:45:48 INFO unit.kubernetes-master/0.juju-log Initializing Leadership Layer (is leader)
unit-kubernetes-master-0: 09:45:50 INFO unit.kubernetes-master/0.juju-log Coordinator: Initializing coordinator layer
unit-kubernetes-master-0: 09:45:51 INFO unit.kubernetes-master/0.juju-log Invoking reactive handler: reactive/kubernetes_master_worker_base.py:18:process_snapd_timer
unit-kubernetes-master-0: 09:45:51 INFO unit.kubernetes-master/0.juju-log Get config refresh.timer for snap core
unit-kubernetes-master-0: 09:45:51 INFO unit.kubernetes-master/0.juju-log Invoking reactive handler: reactive/apt.py:50:ensure_package_status
unit-kubernetes-master-0: 09:45:51 INFO unit.kubernetes-master/0.juju-log Invoking reactive handler: reactive/tls_client.py:18:store_ca
unit-kubernetes-master-0: 09:45:51 INFO unit.kubernetes-master/0.juju-log Invoking reactive handler: reactive/tls_client.py:44:store_server
unit-kubernetes-master-0: 09:45:51 INFO unit.kubernetes-master/0.juju-log Invoking reactive handler: reactive/tls_client.py:71:store_client
unit-kubernetes-master-0: 09:45:51 INFO unit.kubernetes-master/0.juju-log Invoking reactive handler: reactive/kubernetes_master.py:422:check_cohort_updates
unit-kubernetes-master-0: 09:45:51 INFO unit.kubernetes-master/0.juju-log Invoking reactive handler: reactive/kubernetes_master.py:431:safely_join_cohort
unit-kubernetes-master-0: 09:45:51 INFO unit.kubernetes-master/0.juju-log Invoking reactive handler: reactive/kubernetes_master.py:675:set_app_version
unit-kubernetes-master-0: 09:45:53 INFO unit.kubernetes-master/0.juju-log Invoking reactive handler: ../.venv/lib/python3.8/site-packages/charmhelpers/core/host.py:721:wrapped_f
unit-kubernetes-master-0: 09:45:54 INFO unit.kubernetes-master/0.juju-log Invoking reactive handler: reactive/kubernetes_master.py:1162:etcd_data_change
unit-kubernetes-master-0: 09:45:54 INFO unit.kubernetes-master/0.juju-log Invoking reactive handler: reactive/kubernetes_master.py:1268:push_service_data
unit-kubernetes-master-0: 09:45:54 INFO unit.kubernetes-master/0.juju-log Invoking reactive handler: reactive/kubernetes_master.py:1285:send_data
unit-kubernetes-master-0: 09:45:54 INFO unit.kubernetes-master/0.juju-log Invoking reactive handler: reactive/kubernetes_master.py:1359:configure_cdk_addons
unit-kubernetes-master-0: 09:45:55 INFO unit.kubernetes-master/0.juju-log Installing ['ceph-common'] with options: ['--option=Dpkg::Options::=--force-confold']
unit-kubernetes-master-0: 09:45:59 INFO unit.kubernetes-master/0.juju-log Installing ['ceph-common'] with options: ['--option=Dpkg::Options::=--force-confold']
unit-kubernetes-master-0: 09:46:06 WARNING unit.kubernetes-master/0.update-status Traceback (most recent call last):
unit-kubernetes-master-0: 09:46:06 WARNING unit.kubernetes-master/0.update-status File "/snap/cdk-addons/4723/apply", line 398, in <module>
unit-kubernetes-master-0: 09:46:06 WARNING unit.kubernetes-master/0.update-status main()
unit-kubernetes-master-0: 09:46:06 WARNING unit.kubernetes-master/0.update-status File "/snap/cdk-addons/4723/apply", line 24, in main
unit-kubernetes-master-0: 09:46:06 WARNING unit.kubernetes-master/0.update-status if render_templates():
unit-kubernetes-master-0: 09:46:06 WARNING unit.kubernetes-master/0.update-status File "/snap/cdk-addons/4723/apply", line 136, in render_templates
unit-kubernetes-master-0: 09:46:06 WARNING unit.kubernetes-master/0.update-status cephfs_context['mounter'] = get_snap_config("cephfs-mounter") or "default"
unit-kubernetes-master-0: 09:46:06 WARNING unit.kubernetes-master/0.update-status File "/snap/cdk-addons/4723/apply", line 353, in get_snap_config
unit-kubernetes-master-0: 09:46:06 WARNING unit.kubernetes-master/0.update-status raise MissingSnapConfig("%s is required" % name)
unit-kubernetes-master-0: 09:46:06 WARNING unit.kubernetes-master/0.update-status __main__.MissingSnapConfig: cephfs-mounter is required
unit-kubernetes-master-0: 09:46:06 INFO unit.kubernetes-master/0.juju-log Addons are not ready yet.

And that particular change from snap.
snap change 891
Status Spawn Ready Summary
Done today at 09:06 UTC today at 09:06 UTC Ensure prerequisites for "cdk-addons" are available
Done today at 09:06 UTC today at 09:06 UTC Download snap "cdk-addons" (4723) from channel "1.19/stable"
Done today at 09:06 UTC today at 09:06 UTC Fetch and check assertions for snap "cdk-addons" (4723)
Done today at 09:06 UTC today at 09:07 UTC Mount snap "cdk-addons" (4723)
Done today at 09:06 UTC today at 09:07 UTC Run pre-refresh hook of "cdk-addons" snap if present
Done today at 09:06 UTC today at 09:07 UTC Stop snap "cdk-addons" services
Done today at 09:06 UTC today at 09:07 UTC Remove aliases for snap "cdk-addons"
Done today at 09:06 UTC today at 09:07 UTC Make current revision for snap "cdk-addons" unavailable
Done today at 09:06 UTC today at 09:07 UTC Copy snap "cdk-addons" data
Done today at 09:06 UTC today at 09:07 UTC Setup snap "cdk-addons" (4723) security profiles
Done today at 09:06 UTC today at 09:07 UTC Make snap "cdk-addons" (4723) available to the system
Done today at 09:06 UTC today at 09:07 UTC Automatically connect eligible plugs and slots of snap "cdk-addons"
Done today at 09:06 UTC today at 09:07 UTC Set automatic aliases for snap "cdk-addons"
Done today at 09:06 UTC today at 09:07 UTC Setup snap "cdk-addons" aliases
Done today at 09:06 UTC today at 09:07 UTC Run post-refresh hook of "cdk-addons" snap if present
Done today at 09:06 UTC today at 09:07 UTC Start snap "cdk-addons" (4723) services
Done today at 09:06 UTC today at 09:07 UTC Remove data for snap "cdk-addons" (4695)
Done today at 09:06 UTC today at 09:07 UTC Remove snap "cdk-addons" (4695) from the system
Done today at 09:06 UTC today at 09:07 UTC Clean up "cdk-addons" (4723) install
Done today at 09:06 UTC today at 09:07 UTC Run configure hook of "cdk-addons" snap if present
Done today at 09:06 UTC today at 09:07 UTC Run health check of "cdk-addons" snap
Done today at 09:06 UTC today at 09:07 UTC Consider re-refresh of "cdk-addons"

......................................................................
Consider re-refresh of "cdk-addons"

[George Kraft (cynerva)]

The relevant error is this bit:

__main__.MissingSnapConfig: cephfs-mounter is required

That error was caused by a broken release of cdk-addons 1.19.4 that we accidentally put out to stable on 2020-11-12. We released a fix later the same day, and fixed our release process to ensure it doesn't happen again. It should be safe for you to run the latest cdk-addons 1.19 now.

Details here: https://bugs.launchpad.net/cdk-addons/+bug/1904063