Incorrect handling of cached_hostset

[Andrey Pavlov]

During lifecycle of nova-cloud-controller I have next stacktrace in its log

2021-05-07 03:46:17 WARNING unit.nova-cloud-controller/0.cloud-compute-relation-changed logger.go:60 Traceback (most recent call last):
2021-05-07 03:46:17 WARNING unit.nova-cloud-controller/0.cloud-compute-relation-changed logger.go:60 File "/var/lib/juju/agents/unit-nova-cloud-controller-0/charm/hooks/cloud-compute-relation-changed", line 1402, in <module>
2021-05-07 03:46:17 WARNING unit.nova-cloud-controller/0.cloud-compute-relation-changed logger.go:60 main()
2021-05-07 03:46:17 WARNING unit.nova-cloud-controller/0.cloud-compute-relation-changed logger.go:60 File "/var/lib/juju/agents/unit-nova-cloud-controller-0/charm/hooks/cloud-compute-relation-changed", line 1394, in main
2021-05-07 03:46:17 WARNING unit.nova-cloud-controller/0.cloud-compute-relation-changed logger.go:60 hooks.execute(sys.argv)
2021-05-07 03:46:17 WARNING unit.nova-cloud-controller/0.cloud-compute-relation-changed logger.go:60 File "/var/lib/juju/agents/unit-nova-cloud-controller-0/charm/charmhelpers/core/hookenv.py", line 945, in execute
2021-05-07 03:46:17 WARNING unit.nova-cloud-controller/0.cloud-compute-relation-changed logger.go:60 self._hooks[hook_name]()
2021-05-07 03:46:17 WARNING unit.nova-cloud-controller/0.cloud-compute-relation-changed logger.go:60 File "/var/lib/juju/agents/unit-nova-cloud-controller-0/charm/hooks/cloud-compute-relation-changed", line 695, in cloud_compute_relation_changed
2021-05-07 03:46:17 WARNING unit.nova-cloud-controller/0.cloud-compute-relation-changed logger.go:60 update_ssh_keys_and_notify_compute_units(rid=None, unit=None)
2021-05-07 03:46:17 WARNING unit.nova-cloud-controller/0.cloud-compute-relation-changed logger.go:60 File "/var/lib/juju/agents/unit-nova-cloud-controller-0/charm/hooks/cloud-compute-relation-changed", line 740, in update_ssh_keys_and_notify_compute_units
2021-05-07 03:46:17 WARNING unit.nova-cloud-controller/0.cloud-compute-relation-changed logger.go:60 update_ssh_key(rid=rid, unit=unit)
2021-05-07 03:46:17 WARNING unit.nova-cloud-controller/0.cloud-compute-relation-changed logger.go:60 File "/var/lib/juju/agents/unit-nova-cloud-controller-0/charm/hooks/cloud-compute-relation-changed", line 853, in update_ssh_key
2021-05-07 03:46:17 WARNING unit.nova-cloud-controller/0.cloud-compute-relation-changed logger.go:60 resolved_hosts = ncc_utils.resolve_hosts_for(private_address, hostname)
2021-05-07 03:46:17 WARNING unit.nova-cloud-controller/0.cloud-compute-relation-changed logger.go:60 File "/var/lib/juju/agents/unit-nova-cloud-controller-0/charm/hooks/nova_cc_utils.py", line 1350, in resolve_hosts_for
2021-05-07 03:46:17 WARNING unit.nova-cloud-controller/0.cloud-compute-relation-changed logger.go:60 return cached_hostset + hostname
2021-05-07 03:46:17 WARNING unit.nova-cloud-controller/0.cloud-compute-relation-changed logger.go:60 TypeError: can only concatenate list (not "str") to list

[Alex Kavanagh (ajkavanagh)]

In order to help to diagnose the issue please could you provide:

1. a Juju bundle for the system + logs from the relevant services.

OR

2. Juju version
3. Ubuntu version
4. charms versions

Optionally a juju crashdump would be very useful. Thanks.

[Andrey Pavlov (apavlov-e)]

Please let me know what is juju crash dump and where it can be found.
Please let me know if something else is required.

Just a note - we started to catch this issue a week ago even if we have pinned charms in the bundle. And only in HA setups - All-In-One works well.
Attached bundle is from hybrid setup - OpenStack + K8s. but we also have seen same behaviour in pure OpenStack setup.

[Alex Kavanagh (ajkavanagh)]

Hi Andrey

Thanks for the additional info. I've now seen the bug in the code and will work on a fix.

juju crashdump is a useful tool that captures all the units/logs/etc. for a running juju model. It's in the snap store here: https://snapcraft.io/juju-crashdump

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/790221

[Andrey Pavlov (apavlov-e)]

Thank you Alex!

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/790221
Committed: https://opendev.org/openstack/charm-nova-cloud-controller/commit/e63c68d505ee2eb88a0edef83c4a63a8d827f9f6
Submitter: "Zuul (22348)"
Branch: master

commit e63c68d505ee2eb88a0edef83c4a63a8d827f9f6
Author: Alex Kavanagh <email address hidden>
Date: Fri May 7 12:13:29 2021 +0100

    Fix bug where str is added to cached_hostset list

    The original code was appending a str to a list rather than either
    appending it or adding it as a list of one element. The code avoids
    append to avoid unintentional side-effects.

    Change-Id: I1466981f1d68f8dea3bbe32fdde6c4825056c0d0
    Closes-Bug: #1927698

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/859920

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/859921

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/859922

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/859923

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/859924

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/859920
Committed: https://opendev.org/openstack/charm-nova-cloud-controller/commit/91d721f8b2b7e04c29999be7215c9fea2c06af0f
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 91d721f8b2b7e04c29999be7215c9fea2c06af0f
Author: Alex Kavanagh <email address hidden>
Date: Fri May 7 12:13:29 2021 +0100

    Fix bug where str is added to cached_hostset list

    The original code was appending a str to a list rather than either
    appending it or adding it as a list of one element. The code avoids
    append to avoid unintentional side-effects.

    Change-Id: I1466981f1d68f8dea3bbe32fdde6c4825056c0d0
    Closes-Bug: #1927698
    (cherry picked from commit e63c68d505ee2eb88a0edef83c4a63a8d827f9f6)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/859921
Committed: https://opendev.org/openstack/charm-nova-cloud-controller/commit/2988a7f4a32e77abe85af30d7f04747da3461f9d
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 2988a7f4a32e77abe85af30d7f04747da3461f9d
Author: Alex Kavanagh <email address hidden>
Date: Fri May 7 12:13:29 2021 +0100

    Fix bug where str is added to cached_hostset list

    The original code was appending a str to a list rather than either
    appending it or adding it as a list of one element. The code avoids
    append to avoid unintentional side-effects.

    Change-Id: I1466981f1d68f8dea3bbe32fdde6c4825056c0d0
    Closes-Bug: #1927698
    (cherry picked from commit e63c68d505ee2eb88a0edef83c4a63a8d827f9f6)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/859922
Committed: https://opendev.org/openstack/charm-nova-cloud-controller/commit/c2f00d81549e879b17d171b6c7dc98623f45e4c8
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit c2f00d81549e879b17d171b6c7dc98623f45e4c8
Author: Alex Kavanagh <email address hidden>
Date: Fri May 7 12:13:29 2021 +0100

    Fix bug where str is added to cached_hostset list

    The original code was appending a str to a list rather than either
    appending it or adding it as a list of one element. The code avoids
    append to avoid unintentional side-effects.

    Change-Id: I1466981f1d68f8dea3bbe32fdde6c4825056c0d0
    Closes-Bug: #1927698
    (cherry picked from commit e63c68d505ee2eb88a0edef83c4a63a8d827f9f6)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/859923
Committed: https://opendev.org/openstack/charm-nova-cloud-controller/commit/2687b874fe08b4eab6b4a9e39d112c8d37f763a0
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 2687b874fe08b4eab6b4a9e39d112c8d37f763a0
Author: Alex Kavanagh <email address hidden>
Date: Fri May 7 12:13:29 2021 +0100

    Fix bug where str is added to cached_hostset list

    The original code was appending a str to a list rather than either
    appending it or adding it as a list of one element. The code avoids
    append to avoid unintentional side-effects.

    Change-Id: I1466981f1d68f8dea3bbe32fdde6c4825056c0d0
    Closes-Bug: #1927698
    (cherry picked from commit e63c68d505ee2eb88a0edef83c4a63a8d827f9f6)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/859924
Committed: https://opendev.org/openstack/charm-nova-cloud-controller/commit/dc6b82845d428fc0b5c94099d19b826bcdaa8099
Submitter: "Zuul (22348)"
Branch: stable/train

commit dc6b82845d428fc0b5c94099d19b826bcdaa8099
Author: Alex Kavanagh <email address hidden>
Date: Fri May 7 12:13:29 2021 +0100

    Fix bug where str is added to cached_hostset list

    The original code was appending a str to a list rather than either
    appending it or adding it as a list of one element. The code avoids
    append to avoid unintentional side-effects.

    Change-Id: I1466981f1d68f8dea3bbe32fdde6c4825056c0d0
    Closes-Bug: #1927698
    (cherry picked from commit e63c68d505ee2eb88a0edef83c4a63a8d827f9f6)