upgrade charm from 19.01 to 19.04 fails on initalize storage pool

[Narinder Gupta]

Deploy using 19.02 charm then upgrade to 19.04 charm. config-change hook error.

unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed subprocess.CalledProcessError: Command '['lxc', 'storage', 'create', 'lxd', 'btrfs', 'source=/dev/sdh']' returned non-zero exit status 1
unit-lxd-1: 16:21:16 ERROR juju.worker.uniter.operation hook "config-changed" failed: exit status 1
unit-lxd-0: 16:21:14 INFO unit.lxd/0.juju-log Configuring LXD container storage
unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed +------+-------------+--------+--------------------------------------+---------+
unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed | NAME | DESCRIPTION | DRIVER | SOURCE | USED BY |
unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed +------+-------------+--------+--------------------------------------+---------+
unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed | lxd | | btrfs | 4e0bbaa6-6fad-4428-8ffd-81828cdf5746 | 0 |
unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed +------+-------------+--------+--------------------------------------+---------+
unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed Error: The storage pool already exists
unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed Traceback (most recent call last):
unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed File "/var/lib/juju/agents/unit-lxd-0/charm/hooks/config-changed", line 173, in <module>
unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed main()
unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed File "/var/lib/juju/agents/unit-lxd-0/charm/hooks/config-changed", line 166, in main
unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed hooks.execute(sys.argv)
unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed File "/var/lib/juju/agents/unit-lxd-0/charm/hooks/charmhelpers/core/hookenv.py", line 914, in execute
unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed self._hooks[hook_name]()
unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed File "/var/lib/juju/agents/unit-lxd-0/charm/hooks/config-changed", line 86, in config_changed
unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed configure_lxd_block()
unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed File "/var/lib/juju/agents/unit-lxd-0/charm/hooks/lxd_utils.py", line 233, in configure_lxd_block
unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed config_btrfs(dev)
unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed File "/var/lib/juju/agents/unit-lxd-0/charm/hooks/lxd_utils.py", line 246, in config_btrfs
unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed check_call(cmd)
unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed raise CalledProcessError(retcode, cmd)
unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed subprocess.CalledProcessError: Command '['lxc', 'storage', 'create', 'lxd', 'btrfs', 'source=/dev/sdh']' returned non-zero exit status 1
unit-lxd-0: 16:21:14 ERROR juju.worker.uniter.operation hook "config-changed" failed: exit status 1
unit-lxd-1: 16:21:16 INFO unit.lxd/1.juju-log Configuring LXD container storage
unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed +------+-------------+--------+--------------------------------------+---------+
unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed | NAME | DESCRIPTION | DRIVER | SOURCE | USED BY |
unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed +------+-------------+--------+--------------------------------------+---------+
unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed | lxd | | btrfs | d33d8b1b-8096-48a4-8a4c-447168e6d049 | 0 |
unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed +------+-------------+--------+--------------------------------------+---------+
unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed Error: The storage pool already exists
unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed Traceback (most recent call last):
unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed File "/var/lib/juju/agents/unit-lxd-1/charm/hooks/config-changed", line 173, in <module>
unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed main()
unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed File "/var/lib/juju/agents/unit-lxd-1/charm/hooks/config-changed", line 166, in main
unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed hooks.execute(sys.argv)
unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed File "/var/lib/juju/agents/unit-lxd-1/charm/hooks/charmhelpers/core/hookenv.py", line 914, in execute
unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed self._hooks[hook_name]()

[Narinder Gupta (narindergupta)]

inital deployment with config

2019-04-16 16:31:25 DEBUG install Processing triggers for libc-bin (2.27-3ubuntu1) ...
2019-04-16 16:34:15 INFO juju-log Configuring LXD container storage
2019-04-16 16:34:17 DEBUG config-changed If this is your first time running LXD on this machine, you should also run: lxd init
2019-04-16 16:34:17 DEBUG config-changed To start your first container, try: lxc launch ubuntu:18.04
2019-04-16 16:34:17 DEBUG config-changed
2019-04-16 16:34:17 DEBUG config-changed +------+-------------+--------+--------+---------+
2019-04-16 16:34:17 DEBUG config-changed | NAME | DESCRIPTION | DRIVER | SOURCE | USED BY |
2019-04-16 16:34:17 DEBUG config-changed +------+-------------+--------+--------+---------+
2019-04-16 16:34:17 DEBUG config-changed Storage pool lxd created
2019-04-16 16:34:17 INFO juju-log >= Wily deployment - configuring LXD trust password and address
2019-04-16 16:34:17 DEBUG config-changed none
2019-04-16 16:34:17 INFO juju-log Loading kernel module netlink_diag
2019-04-16 16:34:18 DEBUG config-changed 81343
2019-04-16 16:34:29 INFO juju-log Unknown hook start - skipping.
2019-04-16 16:34:30 DEBUG start 81343
2019-04-16 16:34:55 DEBUG lxd-relation-joined +------+-------------+--------+--------------------------------------+---------+
2019-04-16 16:34:55 DEBUG lxd-relation-joined | NAME | DESCRIPTION | DRIVER | SOURCE | USED BY |
2019-04-16 16:34:55 DEBUG lxd-relation-joined +------+-------------+--------+--------------------------------------+---------+
2019-04-16 16:34:55 DEBUG lxd-relation-joined | lxd | | btrfs | 4e0bbaa6-6fad-4428-8ffd-81828cdf5746 | 0 |
2019-04-16 16:34:55 DEBUG lxd-relation-joined +------+-------------+--------+--------------------------------------+---------+
2019-04-16 16:34:56 DEBUG lxd-relation-joined 81343
2019-04-16 16:35:15 DEBUG lxd-migration-relation-joined +------+-------------+--------+--------------------------------------+---------+
2019-04-16 16:35:15 DEBUG lxd-migration-relation-joined | NAME | DESCRIPTION | DRIVER | SOURCE | USED BY |
2019-04-16 16:35:15 DEBUG lxd-migration-relation-joined +------+-------------+--------+--------------------------------------+---------+
2019-04-16 16:35:15 DEBUG lxd-migration-relation-joined | lxd | | btrfs | 4e0bbaa6-6fad-4428-8ffd-81828cdf5746 | 0 |
2019-04-16 16:35:15 DEBUG lxd-migration-relation-joined +------+-------------+--------+--------------------------------------+---------+
2019-04-16 16:35:16 DEBUG lxd-migration-relation-joined 81343
2019-04-16 16:35:32 DEBUG lxd-relation-changed 81343
2019-04-16 16:36:10 INFO juju-log lxd-migration:35: Adding new remote wa1okosl013:10.20.174.89
2019-04-16 16:36:10 DEBUG lxd-migration-relation-changed Generating a client certificate. This may take a minute...
2019-04-16 16:36:12 DEBUG lxd-migration-relation-changed Client certificate stored at server: wa1okosl013
2019-04-16 16:36:13 DEBUG lxd-migration-relation-changed 81343
2019-04-16 16:36:52 INFO juju-log lxd-migration:35: Updating remote wa1okosl013:10.20.174.89
2019-04-16 16:36:53 DEBUG lxd-migration-relation-changed 81343
2019-04-16 16:3...

[Alex Kavanagh (ajkavanagh)]

Hi Narinder

Could you please provide:

* Ubuntu version
* LXD version (unless it's just distro)
* juju config of the lxd charm
* juju logs from the broken unit in full (if possible, including the original deploy)
* bundle used to deploy the openstack
* /etc/nova* config files

Also, what's a 19.01 charm?? Is it the last release, which was 18.11? Do you have the charm store revision number for the originally deployed charm?

Thanks

[Narinder Gupta (narindergupta)]

Alex I can provide few info not all as we are deploying the environment now.

* Ubuntu version

Bionic 18.04 stable
* LXD version (unless it's just distro)

distro only.
* juju config of the lxd charm

  lxd:

    charm: cs:lxd

    options:

      block-devices: '/dev/sdh'

  nova-compute-lxd:
    charm: cs:nova-compute
    num_units: 2
    bindings:
      "": *oam-space
      internal: *internal-space
    options:
      openstack-origin: *openstack-origin
      # this is still a beta feature
      # CRIU is very workload dependent
      enable-live-migration: False
      # maybe in the future
      # enable-resize: True
      # RBD support for nova-lxd will be in Pike
      use-internal-endpoints: True
      restrict-ceph-pools: False
      aa-profile-mode: complain
      virt-type: lxd
      customize-failure-domain: *customize-failure-domain
      reserved-host-memory: *reserved-host-memory
    to:
    - 1012
    - 1013

* juju logs from the broken unit in full (if possible, including the
original deploy)
not available as it is being redeployed

cannot provide as redeployed

* bundle used to deploy the openstack

It is customer bundle with Bionic Rocky.

* /etc/nova* config files

cannot provide as redeployed the environment

Also, what's a 19.01 charm?? Is it the last release, which was 18.11? Do
you have the charm store revision number for the originally deployed
charm?

NG> yes last release where inital charm revison was 22 and after
upgrade it uses 24.

Thanks and Regards,
Narinder Gupta
Canonical, Ltd.
+1.281.736.5150

Ubuntu- Linux for human beings | www.ubuntu.com | www.canonical.com

On Thu, Apr 18, 2019 at 12:50 PM Alex Kavanagh <email address hidden>
wrote:

> Hi Narinder
>
> Could you please provide:
>
> * Ubuntu version
> * LXD version (unless it's just distro)
> * juju config of the lxd charm
> * juju logs from the broken unit in full (if possible, including the
> original deploy)
> * bundle used to deploy the openstack
> * /etc/nova* config files
>
> Also, what's a 19.01 charm?? Is it the last release, which was 18.11? Do
> you have the charm store revision number for the originally deployed
> charm?
>
> Thanks
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1825393
>
> Title:
> upgrade charm from 19.01 to 19.04 fails on initalize storage pool
>
> Status in OpenStack LXD Charm:
> New
>
> Bug description:
> Deploy using 19.02 charm then upgrade to 19.04 charm. config-change
> hook error.
>
> unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed
> subprocess.CalledProcessError: Command '['lxc', 'storage', 'create', 'lxd',
> 'btrfs', 'source=/dev/sdh']' returned non-zero exit status 1
> unit-lxd-1: 16:21:16 ERROR juju.worker.uniter.operation hook
> "config-changed" failed: exit status 1
> unit-lxd-0: 16:21:14 INFO unit.lxd/0.juju-log Configuring LXD container
> storage
> unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed
> +------+-------------+--------+--------------------------------------+---------+
> unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed | NAME |
> DESCRIPTION | DRIVER | SOURCE ...

[Alex Kavanagh (ajkavanagh)]

Thanks Narinda

I think I've worked out what the bug is, although I'm not quite sure the best way to fix it.

-- Notes for fix:

The main issue is in hooks/lxd_utils.py at line 192:

    if filesystem_mounted('/var/lib/lxd'):
        log('/var/lib/lxd already configured, skipping')
        return

With LXD 3.x.x and greater, lxc storage is used, and so there is no filesystem mounted at /var/lib/lxd. Therefore, whenever the config-changed hook fires, it will try to re-setup the block device as the charm hasn't detected that it's already set up.

There are a couple of ways to fix the bug:

1. Add an 'has_storage()' test and then see if the LXD_POOL storage is set up; if it is then exit.
2. Use the the fact that the /var/lib/lxd directory exists as evidence that the storage device has been set-up; it's more efficient than 1. but maybe not as dependable (e.g. if the previous config-change failed due to the block device not being mounted, but the directory was made).

I'm leaning towards 1. as an appropriate fix.

[Narinder Gupta (narindergupta)]

I agreed Alex and appropriate to have fix no 1.

1. Add an 'has_storage()' test and then see if the LXD_POOL storage is set
up; if it is then exit.

Thanks and Regards,
Narinder Gupta
Canonical, Ltd.
+1.281.736.5150

Ubuntu- Linux for human beings | www.ubuntu.com | www.canonical.com

On Thu, Apr 18, 2019 at 2:45 PM Alex Kavanagh <email address hidden>
wrote:

> Thanks Narinda
>
> I think I've worked out what the bug is, although I'm not quite sure the
> best way to fix it.
>
> -- Notes for fix:
>
> The main issue is in hooks/lxd_utils.py at line 192:
>
> if filesystem_mounted('/var/lib/lxd'):
> log('/var/lib/lxd already configured, skipping')
> return
>
> With LXD 3.x.x and greater, lxc storage is used, and so there is no
> filesystem mounted at /var/lib/lxd. Therefore, whenever the config-
> changed hook fires, it will try to re-setup the block device as the
> charm hasn't detected that it's already set up.
>
> There are a couple of ways to fix the bug:
>
> 1. Add an 'has_storage()' test and then see if the LXD_POOL storage is set
> up; if it is then exit.
> 2. Use the the fact that the /var/lib/lxd directory exists as evidence
> that the storage device has been set-up; it's more efficient than 1. but
> maybe not as dependable (e.g. if the previous config-change failed due to
> the block device not being mounted, but the directory was made).
>
> I'm leaning towards 1. as an appropriate fix.
>
> ** Changed in: charm-lxd
> Status: New => Confirmed
>
> ** Changed in: charm-lxd
> Importance: Undecided => High
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1825393
>
> Title:
> upgrade charm from 19.01 to 19.04 fails on initalize storage pool
>
> Status in OpenStack LXD Charm:
> Confirmed
>
> Bug description:
> Deploy using 19.02 charm then upgrade to 19.04 charm. config-change
> hook error.
>
> unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed
> subprocess.CalledProcessError: Command '['lxc', 'storage', 'create', 'lxd',
> 'btrfs', 'source=/dev/sdh']' returned non-zero exit status 1
> unit-lxd-1: 16:21:16 ERROR juju.worker.uniter.operation hook
> "config-changed" failed: exit status 1
> unit-lxd-0: 16:21:14 INFO unit.lxd/0.juju-log Configuring LXD container
> storage
> unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed
> +------+-------------+--------+--------------------------------------+---------+
> unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed | NAME |
> DESCRIPTION | DRIVER | SOURCE | USED BY |
> unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed
> +------+-------------+--------+--------------------------------------+---------+
> unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed | lxd |
> | btrfs | 4e0bbaa6-6fad-4428-8ffd-81828cdf5746 | 0 |
> unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed
> +------+-------------+--------+--------------------------------------+---------+
> unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed Error: The storage
> pool already exists
> unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed Traceback (most
> r...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/655168

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/655168
Committed: https://git.openstack.org/cgit/openstack/charm-lxd/commit/?id=4bf72620c3cf9cef5905a07132510df2cec5de41
Submitter: Zuul
Branch: master

commit 4bf72620c3cf9cef5905a07132510df2cec5de41
Author: Alex Kavanagh <email address hidden>
Date: Tue Apr 23 15:11:11 2019 +0100

    Fix detection of already initialised lxc storage pool

    The related bug was caused because the charm did not detect when
    the lxc storage had already been set up using the `lxc storage`
    command which is used in the 3.x series, rather than the old way
    with the 2.x series.

    Also fixes issue on xenial where linux-image-extra-* has been renamed to
    linux-modules-extra-* from from -4.4.0-145-generic onwards. Otherwise,
    the install fails on xenial.

    Change-Id: If8bdad6f5641ee21fc7860dceeb4c3facbaecc76
    Closes-Bug: #1825393

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/19.04
Review: https://review.opendev.org/655470

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/655470
Committed: https://git.openstack.org/cgit/openstack/charm-lxd/commit/?id=6c61ad42b56866aa1dd5d031fda31b88d92d7c7b
Submitter: Zuul
Branch: stable/19.04

commit 6c61ad42b56866aa1dd5d031fda31b88d92d7c7b
Author: Alex Kavanagh <email address hidden>
Date: Tue Apr 23 15:11:11 2019 +0100

    Fix detection of already initialised lxc storage pool

    The related bug was caused because the charm did not detect when
    the lxc storage had already been set up using the `lxc storage`
    command which is used in the 3.x series, rather than the old way
    with the 2.x series.

    Also fixes issue on xenial where linux-image-extra-* has been renamed to
    linux-modules-extra-* from from -4.4.0-145-generic onwards. Otherwise,
    the install fails on xenial.

    Closes-Bug: #1825393
    (cherry picked from commit 4bf72620c3cf9cef5905a07132510df2cec5de41)

    Change-Id: Ieb8b5c3cfd13b2a601d79fa7a99d29229acc287e