KeyError: 'storage.zfs_pool_name'

[Chris MacNaughton]

On a new install of nova-lxd with charm-lxd, nova-compute has placement errors in its logs:

2018-07-18 09:52:24.821 28967 ERROR nova.compute.manager [req-024fb04d-ea2d-444a-a8a6-6473b3853354 - - - - -] No compute node record for host juju-570928-auto-osci-sv15-5: ComputeHostNotFound_Remote: Compute host juju-570928-auto-osci-sv15-5 could not be found.
2018-07-18 09:52:24.995 28967 ERROR nova.compute.manager [req-024fb04d-ea2d-444a-a8a6-6473b3853354 - - - - -] Error updating resources for node juju-570928-auto-osci-sv15-5.: KeyError: 'storage.zfs_pool_name'
2018-07-18 09:52:24.995 28967 ERROR nova.compute.manager Traceback (most recent call last):
2018-07-18 09:52:24.995 28967 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 7284, in update_available_resource_for_node
2018-07-18 09:52:24.995 28967 ERROR nova.compute.manager rt.update_available_resource(context, nodename)
2018-07-18 09:52:24.995 28967 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 664, in update_available_resource
2018-07-18 09:52:24.995 28967 ERROR nova.compute.manager resources = self.driver.get_available_resource(nodename)
2018-07-18 09:52:24.995 28967 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/virt/lxd/driver.py", line 1021, in get_available_resource
2018-07-18 09:52:24.995 28967 ERROR nova.compute.manager lxd_config['config']['storage.zfs_pool_name']
2018-07-18 09:52:24.995 28967 ERROR nova.compute.manager KeyError: 'storage.zfs_pool_name'

[Ryan Beisner (1chb1n)]

More logs and artifacts:

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline/openstack/charm-lxd/583499/2/5193/index.html

[Alex Kavanagh (ajkavanagh)]

On a xenial with lxd 2.x:

In [5]: c.host_info['config']
Out[5]:
{u'core.https_address': u'[::]',
 u'core.proxy_ignore_hosts': u'127.0.0.1,::1,localhost',
 u'core.trust_password': True,
 u'storage.zfs_pool_name': u'lxd'}

On bionic with lxd 3.0.1:

In [8]: c.host_info['config']
Out[8]:
{u'core.https_address': u'[::]',
 u'core.proxy_ignore_hosts': u'127.0.0.1,::1,localhost',
 u'core.trust_password': True}

Looks like the deprecated key has been removed from the lxd 3.x.

To fix it will need to patch in some code to just grab the first storage pool if the key doesn't exist.

Testing in an ipython shell, shows that this should do the trick:

    c = pylxd.Client()
    try:
        name = lxd_config['config']['storage.zfs_pool_name']
    except KeyError:
        name = c.storage_pools.all()[0].name

I'll do some testing to see if it works.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/583969

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/583969
Committed: https://git.openstack.org/cgit/openstack/nova-lxd/commit/?id=f1bbc03b65ab8af25be5fa982e3ac24ba591bee9
Submitter: Zuul
Branch: master

commit f1bbc03b65ab8af25be5fa982e3ac24ba591bee9
Author: Alex Kavanagh <email address hidden>
Date: Thu Jul 19 15:04:30 2018 +0100

    Fix problem with 'storage.zfs_pool_name' being removed from lxd 3

    LXD 3.x onwards removes the key 'storage.zfs_pool_name' from the config.
    This means the storage_pool API needs to be used to get the name of the
    pool that juju is using for ZFS. This is a temporary fix until storage
    pools can be threaded into nova-lxd properly. This occurs on bionic due
    to LXD 3 being shipped as standard.

    Change-Id: I6527640438331b86f2140cf0a772f7d207a6fd40
    Closes-Bug: #1782329

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/584900

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/584900
Committed: https://git.openstack.org/cgit/openstack/nova-lxd/commit/?id=84a8c59be69cb0c2c473eb3307985264b4fba98f
Submitter: Zuul
Branch: stable/queens

commit 84a8c59be69cb0c2c473eb3307985264b4fba98f
Author: Alex Kavanagh <email address hidden>
Date: Thu Jul 19 15:04:30 2018 +0100

    Fix problem with 'storage.zfs_pool_name' being removed from lxd 3

    Cherry-pick from master: f1bbc03b65ab8af25be5fa982e3ac24ba591bee9

    LXD 3.x onwards removes the key 'storage.zfs_pool_name' from the config.
    This means the storage_pool API needs to be used to get the name of the
    pool that juju is using for ZFS. This is a temporary fix until storage
    pools can be threaded into nova-lxd properly. This occurs on bionic due
    to LXD 3 being shipped as standard.

    Change-Id: Ic80ad942759718785b62e4fc887ded0a345bb260
    Related-Bug: #1782329

[Sahid Orentino (sahid-ferdjaoui)]

I would say the fix is committed for charm-lxd, part of commit:

commit 4d7b3e924008af878b0c369c1bd3e92238d51a93
Author: Chris MacNaughton <email address hidden>
Date: Wed Jul 18 10:21:03 2018 +0200

    Remove old nova-compute / mysql relation and patch for nova-lxd bug

    This review is modified to include a patch to resolve bug #1793137
    which is due to a bug in nova-lxd where the wrong pool name (lxd) is
    used with zfs if lxd is new enough that the storage.zfs_pool_name key
    is missing from the config. This is the case on bionic and snap
    installed lxd.

    Closes-Bug: #1713807
    Related-Bug: #1793137
    Change-Id: I1123d9882bdb00e9183424b363fc3d1ff25f2346

We should probably update status of this bug for "OpenStack LXD Charm"