Provisioning issue - LV for /var/log causes incorrect /var/log content

[Dmitry Sutyagin]

Issue summary:

Customizing provisioning via CLI in the form of adding additional LVs and mountpoints for existing directories causes issues with original directory content, which in turn may lead to unexpected issues.

Verified for: Fuel 6.1

How to reproduce:
1. Create ubuntu env
2. Assign 1 controller to deploy
3. Increase base system space via Web UI to account for extra LV space (for ex. add 10GB to base, take from glance)
3. `fuel provisioning default --env X`
4. edit the node yaml - make root LV smaller by 10G, add an extra LV 'log' with 10G size
5. `fuel provisioning upload --env X`
6. start deployment

Expected results: /var/log will be on a separate LV, no impact to the environment
Observed results: /var/log is on a separate LV but does not contain the original data from /var/log folder, for example no "upstart" folder, which causes upstart to accumulate logs in memory and freeze at some point.

[Alexander Gordeev (a-gordeev)]

looks like a duplicate of https://bugs.launchpad.net/fuel/+bug/1494684

[Alexey Stupnikov (astupnikov)]

It is a known issue described in bug #1494684. There was a WA provided for MOS 7.0 and next releases; though it doesn't fix this issue completely, upstart logs will not be an issue anymore. Since we have the same problem with the following dirs, I will re-write the patch to work well and will check this issue against other releases.

fsck
ntpstats
upstart
unattended-upgrades

[Alexey Stupnikov (astupnikov)]

7.0 review https://review.openstack.org/#/c/224460/

[Alexey Stupnikov (astupnikov)]

6.1 review https://review.openstack.org/#/c/305714/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/310717

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/310717
Committed: https://git.openstack.org/cgit/openstack/fuel-agent/commit/?id=5809a3fea96b76386ddc4ccd367c21c00aa8337d
Submitter: Jenkins
Branch: master

commit 5809a3fea96b76386ddc4ccd367c21c00aa8337d
Author: Vladimir Kozhukalov <email address hidden>
Date: Thu Sep 17 02:24:07 2015 +0300

    Move files from mount points to actual file systems

    The thing is that sometimes we have file system images and mount
    point hierachies which are not aligned. Let's say, we have root
    file system image, while partition scheme says that two file
    systems should be created on the node: / and /var. In this case
    root image has /var directory with a set of files. Obviously, we
    need to move all these files from /var directory on the root
    file system to /var file system because /var directory will be
    used as mount point.
    In order to achieve this we mount all existent file systems into
    a flat set of temporary directories. We then try to find
    specific paths which correspond to mount points and move all
    files from these paths to corresponding file systems.

    Co-Authored-By: Alexey Stupnikov <email address hidden>

    Change-Id: I1aa7523055ac4bcf6f8a93e9740ccf652ed35cc1
    Closes-Bug: #1537699

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/318237

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/318467

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/318237
Committed: https://git.openstack.org/cgit/openstack/fuel-agent/commit/?id=dacf3d99873fb298570d65dbfc8e31a7a68786dd
Submitter: Jenkins
Branch: stable/mitaka

commit dacf3d99873fb298570d65dbfc8e31a7a68786dd
Author: Vladimir Kozhukalov <email address hidden>
Date: Thu Sep 17 02:24:07 2015 +0300

    Move files from mount points to actual file systems

    The thing is that sometimes we have file system images and mount
    point hierachies which are not aligned. Let's say, we have root
    file system image, while partition scheme says that two file
    systems should be created on the node: / and /var. In this case
    root image has /var directory with a set of files. Obviously, we
    need to move all these files from /var directory on the root
    file system to /var file system because /var directory will be
    used as mount point.
    In order to achieve this we mount all existent file systems into
    a flat set of temporary directories. We then try to find
    specific paths which correspond to mount points and move all
    files from these paths to corresponding file systems.

    Co-Authored-By: Alexey Stupnikov <email address hidden>

    Change-Id: I1aa7523055ac4bcf6f8a93e9740ccf652ed35cc1
    Closes-Bug: #1537699
    (cherry picked from commit 5809a3fea96b76386ddc4ccd367c21c00aa8337d)

[Alexey Stupnikov (astupnikov)]

All patches are ready to be merged. So we are looking for their inclusion on MU scope.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/305714
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=37631934faeb2e430126fc57da381c8036c43029
Submitter: Jenkins
Branch: stable/6.1

commit 37631934faeb2e430126fc57da381c8036c43029
Author: Vladimir Kozhukalov <email address hidden>
Date: Thu Sep 17 02:24:07 2015 +0300

    Move files from mount points to actual file systems

    The thing is that sometimes we have file system images and mount
    point hierachies which are not aligned. Let's say, we have root
    file system image, while partition scheme says that two file
    systems should be created on the node: / and /var. In this case
    root image has /var directory with a set of files. Obviously, we
    need to move all these files from /var directory on the root
    file system to /var file system because /var directory will be
    used as mount point.
    In order to achieve this we mount all existent file systems into
    a flat set of temporary directories. We then try to find
    specific paths which correspond to mount points and move all
    files from these paths to corresponding file systems.

    Co-Authored-By: Alexey Stupnikov <email address hidden>

    Change-Id: I1aa7523055ac4bcf6f8a93e9740ccf652ed35cc0
    Closes-Bug: #1537699
    (cherry picked from commit 5809a3fea96b76386ddc4ccd367c21c00aa8337d)

[Alexey Stupnikov (astupnikov)]

Verification note. Example patch to apply to YAML configuration (i.e node-1.yaml):

    - _allocate_size: min
      id: os
      label: Base System
      min_size: 19456
      type: vg
      volumes:
      - file_system: ext4
        mount: /
        name: root
- size: 15360
+ size: 11260
        type: lv
+ - file_system: ext4
+ mount: /var/log/upstart
+ name: root
+ size: 4096
+ type: lv
      - file_system: swap
        mount: swap
        name: swap
        size: 4096
        type: lv

[Alexey Stupnikov (astupnikov)]

I have made a mistake in previous message: you have to rename added partition (it shouldn't have name 'root').

[Alexander Petrov (apetrov-n)]

Verified on MOS 9.0, ISO 465

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/341334

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alexey Stupnikov (<email address hidden>) on branch: stable/7.0
Review: https://review.openstack.org/341334
Reason: Incorrect change ID.

[Dmitry Belyaninov (dbelyaninov)]

Verified on 6.1 MU7

Described scenario + #12 comment works fine (full logs list inside the upstart directory).

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/224460
Committed: https://git.openstack.org/cgit/openstack/fuel-agent/commit/?id=cb5739b2c84b7a75a5e0e34bac78a9a0f87258a9
Submitter: Jenkins
Branch: stable/7.0

commit cb5739b2c84b7a75a5e0e34bac78a9a0f87258a9
Author: Vladimir Kozhukalov <email address hidden>
Date: Thu Sep 17 02:24:07 2015 +0300

    Move files from mount points to actual file systems

    The thing is that sometimes we have file system images and mount
    point hierachies which are not aligned. Let's say, we have root
    file system image, while partition scheme says that two file
    systems should be created on the node: / and /var. In this case
    root image has /var directory with a set of files. Obviously, we
    need to move all these files from /var directory on the root
    file system to /var file system because /var directory will be
    used as mount point.
    In order to achieve this we mount all existent file systems into
    a flat set of temporary directories. We then try to find
    specific paths which correspond to mount points and move all
    files from these paths to corresponding file systems.

    Co-Authored-By: Alexey Stupnikov <email address hidden>

    Change-Id: I1aa7523055ac4bcf6f8a93e9740ccf652ed35cc0
    Closes-Bug: #1537699
    (cherry picked from commit 5809a3fea96b76386ddc4ccd367c21c00aa8337d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/318467
Committed: https://git.openstack.org/cgit/openstack/fuel-agent/commit/?id=21048016a7081c1167e71b52324be8524f295877
Submitter: Jenkins
Branch: stable/8.0

commit 21048016a7081c1167e71b52324be8524f295877
Author: Vladimir Kozhukalov <email address hidden>
Date: Thu Sep 17 02:24:07 2015 +0300

    Move files from mount points to actual file systems

    The thing is that sometimes we have file system images and mount
    point hierachies which are not aligned. Let's say, we have root
    file system image, while partition scheme says that two file
    systems should be created on the node: / and /var. In this case
    root image has /var directory with a set of files. Obviously, we
    need to move all these files from /var directory on the root
    file system to /var file system because /var directory will be
    used as mount point.
    In order to achieve this we mount all existent file systems into
    a flat set of temporary directories. We then try to find
    specific paths which correspond to mount points and move all
    files from these paths to corresponding file systems.

    Co-Authored-By: Alexey Stupnikov <email address hidden>

    Change-Id: I1aa7523055ac4bcf6f8a93e9740ccf652ed35cc1
    Closes-Bug: #1537699
    (cherry picked from commit 5809a3fea96b76386ddc4ccd367c21c00aa8337d)

[Ekaterina Shutova (eshutova)]

Verified on MOS 7.0 + MU6 updates.

Described scenario + #12 comment works fine(full logs list inside the upstart directory).
root@node-1:~# ls /var/log/upstart
cinder-scheduler.log network-interface-br-aux.log network-interface-eth0.log resolvconf.log ureadahead-other.log
console-setup.log network-interface-br-ex.log network-interface-lo.log rsyslog.log
glance-registry.log network-interface-br-fw-admin.log nova-scheduler.log swift-container-sync.log
heat-engine.log network-interface-br-mgmt.log procps-static-network-up.log swift-proxy.log
lost+found network-interface-br-storage.log procps-virtual-filesystems.log systemd-logind.log

[Ekaterina Shutova (eshutova)]

Verified on MOS 8.0 + mu4 updates.

Described scenario + #12 comment works fine(full logs list inside the upstart directory).
root@node-1:~# ls /var/log/upstart
acpid.log lost+found nova-api.log resolvconf.log
cinder-api.log network-interface-br-aux.log nova-cert.log rsyslog.log
cinder-scheduler.log network-interface-br-ex.log nova-conductor.log swift-container-sync.log
console-setup.log network-interface-br-floating.log nova-consoleauth.log swift-proxy.log
glance-api.log network-interface-br-fw-admin.log nova-novncproxy.log systemd-logind.log
glance-registry.log network-interface-br-mgmt.log nova-objectstore.log ureadahead-other.log
heat-api-cfn.log network-interface-br-prv.log nova-scheduler.log
heat-api-cloudwatch.log network-interface-br-storage.log procps-static-network-up.log
heat-api.log neutron-server.log procps-virtual-filesystems.log

[Dmitry Belyaninov (dbelyaninov)]

Verified on Newton #1556 iso.
Moved to FixReleased