Failed to create subvolume /var/lib/machines/ when run 'openstack-ansible setup-hosts.yml' multiple times

[Gökhan]

I tried to install openstack ansible 16.0.4 on ubuntu xenial.
on running 'openstack-ansible setup-hosts.yml' step, I got error about lxc-cache-prep timeout value. for this error I run "openstack-ansible setup-hosts.yml" 3 times. after that I searched about problem and find it.I applied this commit https://github.com/openstack/openstack-ansible-lxc_hosts/commit/f6afac77870f7c2d386e0522c084c68e9ad342d0 and override lxc_cache_prep_timeout value to 1200. After this commit I run again "openstack-ansible setup-hosts.yml" but I got an error which is Disk quota exceeded. I override lxc_host_machine_volume_size and again got same error.this is error log: http://paste.openstack.org/show/653386/

My kernel version is: 4.4.0-112-generic

some command results are below:

'mount': http://paste.openstack.org/show/653393/

'sudo btrfs fi show /var/lib/machines' : http://paste.openstack.org/show/653401/

'sudo btrfs fi df /var/lib/machines' : http://paste.openstack.org/show/653402/

'btrfs check /var/lib/machines': http://paste.openstack.org/show/653403/

[Gökhan (skylightcoder)]

I temproraly solved this problem with 3 steps:
1. rm -rf /var/lib/machines.raw
2. reboot
3. rm -rf /var/lib/machines

After these steps, I again run "openstack-ansible setup-hosts.yml" and didn't see any error about this problem.

[Kevin Carter (kevin-carter)]

Hi Gökhan, the issue is within the quota system which should be disabled by default. I have a PR up here; https://review.openstack.org/#/c/527592/ which should allow for this. In the meantime you can run

  * btrfs quota disable /var/lib/machines

using ansible across the infra hosts you could do something like so:

  * ansible -m shell -a 'btrfs quota disable /var/lib/machines' infra_hosts

I will update the PR and hopefully get something backported to stable pike soon.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/527592
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-lxc_hosts/commit/?id=f179f21a6695269259091f4ef1b37630527185e9
Submitter: Zuul
Branch: master

commit f179f21a6695269259091f4ef1b37630527185e9
Author: Kevin Carter <email address hidden>
Date: Wed Dec 13 00:12:56 2017 -0600

    Clean-up old systemd prep and allow machinctl to grow

    The machinectl cache is currently set image to 16G by default. If
    multiple container images are imported into the cache this may be too
    small by default. This change sets the cache to "64G" by default allowing
    the cache more room to grow by.

    This change also disables the quota system once the limit has been set
    The option `lxc_host_machine_quota_disabled` has been added to disable or
    enable the quota system as needed. This is done after the default limit has
    been set so an adequately sized sparce file can be created should it not
    already exist.

    > More documentation can be seen here [0] with regard to the set-limit
      option.

    Because we support both modern and older systemd, the cache prep tasks
    for old systemd have been updated so that deployers using earlier
    versions of systemd can benefit from the ability to grow an existing
    cache via playbook run.

    [0] https://www.freedesktop.org/software/systemd/man/machinectl.html#set-limit%20%5BNAME%5D%20BYTES

    Closes-Bug: #1745361
    Change-Id: I85fefc6ce186bb6808ac37a9ea79a50e29671115
    Signed-off-by: Kevin Carter <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/545850

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/545851

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/545851
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-lxc_hosts/commit/?id=49edbd39b14e648ce2fe2c5afa9e01adf684c80d
Submitter: Zuul
Branch: stable/pike

commit 49edbd39b14e648ce2fe2c5afa9e01adf684c80d
Author: Kevin Carter <email address hidden>
Date: Wed Dec 13 00:12:56 2017 -0600

    Clean-up old systemd prep and allow machinctl to grow

    This is a backport of:
    - https://review.openstack.org/527592
    - https://review.openstack.org/545801

    The machinectl cache is currently set image to 16G by default. If
    multiple container images are imported into the cache this may be too
    small by default. This change sets the cache to "64G" by default allowing
    the cache more room to grow by.

    This change also disables the quota system once the limit has been set
    The option `lxc_host_machine_quota_disabled` has been added to disable or
    enable the quota system as needed. This is done after the default limit has
    been set so an adequately sized sparce file can be created should it not
    already exist.

    > More documentation can be seen here [0] with regard to the set-limit
      option.

    Because we support both modern and older systemd, the cache prep tasks
    for old systemd have been updated so that deployers using earlier
    versions of systemd can benefit from the ability to grow an existing
    cache via playbook run.

    [0] https://www.freedesktop.org/software/systemd/man/machinectl.html#set-limit%20%5BNAME%5D%20BYTES

    Closes-Bug: #1745361
    Change-Id: I85fefc6ce186bb6808ac37a9ea79a50e29671115
    Signed-off-by: Kevin Carter <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/545850
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-lxc_hosts/commit/?id=c4a076d2c0508cf30f242462decb342d58f88058
Submitter: Zuul
Branch: stable/queens

commit c4a076d2c0508cf30f242462decb342d58f88058
Author: Kevin Carter <email address hidden>
Date: Wed Dec 13 00:12:56 2017 -0600

    Clean-up old systemd prep and allow machinctl to grow

    This is a backport of:
    - https://review.openstack.org/527592
    - https://review.openstack.org/545801

    The machinectl cache is currently set image to 16G by default. If
    multiple container images are imported into the cache this may be too
    small by default. This change sets the cache to "64G" by default allowing
    the cache more room to grow by.

    This change also disables the quota system once the limit has been set
    The option `lxc_host_machine_quota_disabled` has been added to disable or
    enable the quota system as needed. This is done after the default limit has
    been set so an adequately sized sparce file can be created should it not
    already exist.

    > More documentation can be seen here [0] with regard to the set-limit
      option.

    Because we support both modern and older systemd, the cache prep tasks
    for old systemd have been updated so that deployers using earlier
    versions of systemd can benefit from the ability to grow an existing
    cache via playbook run.

    [0] https://www.freedesktop.org/software/systemd/man/machinectl.html#set-limit%20%5BNAME%5D%20BYTES

    Closes-Bug: #1745361
    Change-Id: I85fefc6ce186bb6808ac37a9ea79a50e29671115
    Signed-off-by: Kevin Carter <email address hidden>

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-lxc_hosts 17.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-lxc_hosts 16.0.10 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-lxc_hosts 18.0.0.0b1 development milestone.

[Satish Patel (satish-txt)]

I got same error today i am install Queens 17.0.8

TASK [lxc_hosts : Disable|Enable the machinectl quota system] ***********************************************************************************************
Friday 03 August 2018 23:24:55 -0400 (0:00:01.469) 0:03:29.673 *********
fatal: [ostack-infra-01]: FAILED! => {"changed": false, "cmd": ["btrfs", "quota", "enable", "/var/lib/machines"], "delta": "0:00:00.010204", "end": "2018-08-03 23:24:56.396433", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2018-08-03 23:24:56.386229", "stderr": "ERROR: not a btrfs filesystem: /var/lib/machines", "stderr_lines": ["ERROR: not a btrfs filesystem: /var/lib/machines"], "stdout": "", "stdout_lines": []}
fatal: [ostack-infra-03]: FAILED! => {"changed": false, "cmd": ["btrfs", "quota", "enable", "/var/lib/machines"], "delta": "0:00:00.014247", "end": "2018-08-03 23:24:56.471986", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2018-08-03 23:24:56.457739", "stderr": "ERROR: not a btrfs filesystem: /var/lib/machines", "stderr_lines": ["ERROR: not a btrfs filesystem: /var/lib/machines"], "stdout": "", "stdout_lines": []}
fatal: [ostack-infra-02]: FAILED! => {"changed": false, "cmd": ["btrfs", "quota", "enable", "/var/lib/machines"], "delta": "0:00:00.013839", "end": "2018-08-03 23:24:56.589439", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2018-08-03 23:24:56.575600", "stderr": "ERROR: not a btrfs filesystem: /var/lib/machines", "stderr_lines": ["ERROR: not a btrfs filesystem: /var/lib/machines"], "stdout": "", "stdout_lines": []}