Pacemaker fails whole deploy because of insufficient disk space on mongo partition

[Vyacheslav Struk]

Fuel 8.0 has no minimal disk space checks for mongo(at least) logical volume (mounted to /var/lib/mongo)

Deploy fails on pacemaker which dismissing the controller node during deploy because of low disk space.
Default node-health-strategy is 'migrate-on-red' so the process is triggering by SysInfo health check resource.

How to reproduce:

1) Make an environment with a single controller, add mongo role to the controller, set mongo partition to 11 Gb.
2) Start deploy and wail till it fails.
Check it on the controller node:

# df -h
  Filesystem Size Used Avail Use% Mounted on
  ...
  /dev/mapper/mongo-mongodb 11G 11G 34M 100% /var/lib/mongo

# cibadmin -Q | grep health_disk
          <nvpair id="status-1-#health_disk" name="#health_disk" value="red"/>

# crm resource list
   Clone Set: clone_p_vrouter [p_vrouter]
       Stopped: [ node-1.domain.tld ]
   vip__management (ocf::fuel:ns_IPaddr2): Stopped
   vip__baremetal (ocf::fuel:ns_IPaddr2): Stopped
   vip__vrouter_pub (ocf::fuel:ns_IPaddr2): Stopped
   vip__vrouter (ocf::fuel:ns_IPaddr2): Stopped
   vip__public (ocf::fuel:ns_IPaddr2): Stopped
   Master/Slave Set: master_p_conntrackd [p_conntrackd]
       Stopped: [ node-1.domain.tld ]
   sysinfo_node-1.domain.tld (ocf::pacemaker:SysInfo): Stopped
   Clone Set: clone_p_haproxy [p_haproxy]
       Stopped: [ node-1.domain.tld ]

============================

# cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "570"
  build_id: "570"
  fuel-nailgun_sha: "558ca91a854cf29e395940c232911ffb851899c1"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "c2a335b5b725f1b994f78d4c78723d29fa44685a"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "78ffc73065a9674b707c081d128cb7eea611474f"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "d605bcbabf315382d56d0ce8143458be67c53434"

[Matthew Mosesohn (raytrac3r)]

The minimum size is determined in nailgun logic. It is 10gb
https://github.com/openstack/fuel-web/blob/stable/8.0/nailgun/nailgun/extensions/volume_manager/manager.py#L830

Perhaps the minimum size should be 20gb. Moving to fuel-python

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/309351

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/310599

[OpenStack Infra (hudson-openstack)]

Change abandoned by Maksim Malchuk (<email address hidden>) on branch: master
Review: https://review.openstack.org/309351
Reason: correct fix: https://review.openstack.org/310599

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/310599
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=1bfdc6d08377f471f4e975f250d77179f8997bcd
Submitter: Jenkins
Branch: master

commit 1bfdc6d08377f471f4e975f250d77179f8997bcd
Author: Maksim Malchuk <email address hidden>
Date: Thu Apr 28 00:53:29 2016 +0300

    Use default size for the MongoDB replication operation log

    We should use default size for the replication operation log if the
    value not defined in Hiera. On 64-bit systems, the 'oplog' is
    typically 5% of available disk space.

    Change-Id: Iafd5bc518f59e018b0e37410927e0e6f635ff0db
    Closes-Bug: #1572957
    Signed-off-by: Maksim Malchuk <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/310709

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/310719

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/310709
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=51ddf5f9d924686cf807f81b0eaae9e8b3090549
Submitter: Jenkins
Branch: stable/mitaka

commit 51ddf5f9d924686cf807f81b0eaae9e8b3090549
Author: Maksim Malchuk <email address hidden>
Date: Thu Apr 28 00:53:29 2016 +0300

    Use default size for the MongoDB replication operation log

    We should use default size for the replication operation log if the
    value not defined in Hiera. On 64-bit systems, the 'oplog' is
    typically 5% of available disk space.

    Change-Id: Iafd5bc518f59e018b0e37410927e0e6f635ff0db
    Closes-Bug: #1572957
    Signed-off-by: Maksim Malchuk <email address hidden>
    (cherry picked from commit 1bfdc6d08377f471f4e975f250d77179f8997bcd)

[Dmitriy Kruglov (dkruglov)]

Verified on MOS 9.0, ISO build 418.
The issue is fixed.

ISO details:
root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 418
cat /etc/fuel_build_number:
 418
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6347.noarch
 fuel-bootstrap-cli-9.0.0-1.mos284.noarch
 fuel-migrate-9.0.0-1.mos8398.noarch
 rubygem-astute-9.0.0-1.mos746.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8709.noarch
 network-checker-9.0.0-1.mos72.x86_64
 fuel-mirror-9.0.0-1.mos137.noarch
 fuel-openstack-metadata-9.0.0-1.mos8709.noarch
 fuel-notify-9.0.0-1.mos8398.noarch
 nailgun-mcagents-9.0.0-1.mos746.noarch
 python-fuelclient-9.0.0-1.mos316.noarch
 fuelmenu-9.0.0-1.mos270.noarch
 fuel-9.0.0-1.mos6347.noarch
 fuel-utils-9.0.0-1.mos8398.noarch
 fuel-setup-9.0.0-1.mos6347.noarch
 fuel-library9.0-9.0.0-1.mos8398.noarch
 shotgun-9.0.0-1.mos90.noarch
 fuel-agent-9.0.0-1.mos284.noarch
 fuel-ui-9.0.0-1.mos2706.noarch
 fuel-ostf-9.0.0-1.mos934.noarch
 fuel-misc-9.0.0-1.mos8398.noarch
 python-packetary-9.0.0-1.mos137.noarch
 fuel-nailgun-9.0.0-1.mos8709.noarch

[OpenStack Infra (hudson-openstack)]

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: stable/8.0
Review: https://review.opendev.org/310719
Reason: This repo is retired now, no further work will get merged.