default partitioning doing bad on small HDD/SSD/Other drives

Bug #1394864 reported by Aleksandr Shaposhnikov
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Fedor Zhadaev
6.0.x
Won't Fix
High
Fuel Python (Deprecated)
6.1.x
Won't Fix
High
Fuel Python (Deprecated)

Bug Description

Just found that script for disks partitionings in Fuel have no minimal requirement to the base/root partition and will scale it as any other using percentage. IMHO base/root partition should have minimal size and if there is other roles assigned to node it should try not to go with size less that minimal for root partition. Here is the example how default partitioner worked for case of 80Gb SSD + 1Tb HDD:

80Gb(31Gb - System(15Gb swap, 15Gb root), 42.9 - Mongo)
1Tb(Mongo)

This default partitioning completely unacceptable in case of writing logs on root partition. In case of future upgrades it also would be a problem. I suggest using of partitions sizes: at least 50Gb for root, at least 30Gb for logs, half-ram size for swap (as currently done) as separate partitions. All remaining space could be partitioned less strict.

Revision history for this message
Sergey Galkin (sgalkin) wrote :
Changed in fuel:
assignee: nobody → Fuel Library Team (fuel-library)
importance: Undecided → Medium
Changed in fuel:
milestone: none → 6.1
status: New → Confirmed
Revision history for this message
Matthew Mosesohn (raytrac3r) wrote :

Partitioning schema is located inside nailgun. Moving to fuel-python.

I filed this bug with fuel-python team already: https://bugs.launchpad.net/fuel/+bug/1393369

I've also partially mitigated this bug by reducing the verbosity of MongoDB logging[1] and rotating more frequently when running in Debug logging mode[2]
https://review.openstack.org/#/c/134930/
https://review.openstack.org/#/c/134973/

That having been said, it's quite possible to run out of disk space with only 16GB root partition. It only takes a few administrator commands to fill the disk up to 90% and then heavy user activity to push it to the limit.

Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Fuel Python Team (fuel-python)
tags: added: release-notes scale
Revision history for this message
Aleksandr Shaposhnikov (alashai8) wrote :

100 nodes, moderate load, 4 hours -> 7-8 Gb of logs on each node.

Andrew Woodward (xarses)
tags: added: customer-found
Changed in fuel:
importance: Medium → High
Revision history for this message
Aleksandr Shaposhnikov (alashai8) wrote :

The impact of this bug is that after a short period of time under moderate load cloud becomes unoperational and cannot be easily repaired without a lot of work and great deal of luck. The core problem is that this couldn't be solved after cloud was provisioned and the only way to repartition everything is to redeploy the cloud. This means that there is no easy workaround to already deployed cloud with incorrect partitioning.

Łukasz Oleś (loles)
Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Łukasz Oleś (loles)
Łukasz Oleś (loles)
Changed in fuel:
importance: High → Medium
Revision history for this message
Łukasz Oleś (loles) wrote :

Ok, so fuel has minimal requirement for base system and it's 15GB and it always will be 15 GB. Such behavior is not optimal. We need to calculate default value according to cluster size or add new partition for /var/log and calculate its size according to cluster size.

Changing importance to medium as user can always change it manually so it's not a blocker. In lab we created script which changes it.

Changed in fuel:
status: Confirmed → Triaged
assignee: Łukasz Oleś (loles) → Fuel Python Team (fuel-python)
Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

Unless a user is aware that they should do that they will run out of space before they discover this bug on LaunchPad. We can't downgrade it to Medium until we have added this bug to release notes and documented the workaround you're talking about.

Changed in fuel:
importance: Medium → High
Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

Related problem on the master node is bug #1383741

Revision history for this message
Dmitry Pyzhov (dpyzhov) wrote :

Unfortunately we can not fix this bug without this refactoring: https://blueprints.launchpad.net/fuel/+spec/volume-manager-refactoring
So I've moved it to 7.0

Changed in fuel:
milestone: 6.1 → 7.0
status: Triaged → Confirmed
Changed in fuel:
milestone: 7.0 → 6.1
status: Confirmed → New
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
milestone: 6.1 → 7.0
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
status: New → Confirmed
Dmitry Pyzhov (dpyzhov)
tags: added: volumes
Dmitry Pyzhov (dpyzhov)
tags: added: module-volumes
removed: volumes
Revision history for this message
Mike Scherbakov (mihgen) wrote :

Dmitry P., why we can't fix it without refactoring of volume manager? Can we have some more details? Can't we just change the logic a bit to allocate more space for root partition by default?

tags: added: qa-agree-7.0
no longer affects: fuel/7.0.x
tags: added: release-notes-done
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-docs (stable/6.1)

Related fix proposed to branch: stable/6.1
Review: https://review.openstack.org/194961

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-docs (stable/6.1)
Download full text (45.4 KiB)

Reviewed: https://review.openstack.org/194961
Committed: https://git.openstack.org/cgit/stackforge/fuel-docs/commit/?id=0e26e7d7cc153d179ec34985645dd23cdd239ddb
Submitter: Jenkins
Branch: stable/6.1

commit 5cc5f0c643aebecaf3bf4580535a3ea7c3334a6c
Author: Mike Scherbakov <email address hidden>
Date: Tue Jun 23 13:43:35 2015 -0700

    Removed streamlined patching backend pieces

    Change-Id: I955e76ccdbd12a9145f4e9b689f80bdf9fcaf929

commit 563c4b5c78ebfcb1f4f91047c2919f6270f9a1d4
Author: Mike Scherbakov <email address hidden>
Date: Tue Jun 23 13:30:30 2015 -0700

    Removed outdated patching guide

    Change-Id: I76180c277789ade9c5ebedd19fe2092847c0b7d9

commit 8d120c14bec1ab41d448683ad146a3053a57c4ee
Author: Irina Povolotskaya <email address hidden>
Date: Tue Jun 23 19:59:11 2015 +0300

    Add dual hypervisor ref arch into 6.1 docs

    Change-Id: I900c24c9de878eafadbfc995aa879b7f55737fac

commit feebd1592d3305b64bbdfd0bc5fe108190aef120
Author: OlgaGusarenko <email address hidden>
Date: Tue Jun 23 18:38:17 2015 +0300

    [OPs guide] Running Ceilometer section edits

    1. conf file extract is updated
    2. note is updated

    Closes-bug: 1467817
    Change-Id: I0217e164108e0ba6c1397045a5e57d13ff429223

commit 44a93f9dead7511a3461ec35248dbb689c81eafd
Author: OlgaGusarenko <email address hidden>
Date: Tue Jun 23 18:04:40 2015 +0300

    [RN6_1] Final changes

    1. capitalization
    2. 2014.2 to 2014.2.2
    3. general improvements

    Change-Id: I45057e90c90550559f66bc67ccdf97a559fd9000

commit bb41389cae58084285688853281516b659686422
Author: evkonstantinov <email address hidden>
Date: Tue Jun 23 16:45:35 2015 +0300

    Update patching decription

    Update patching description with
    the standard Linux commands.

    Change-Id: Ia1a8346639c468fdfce15a11d2430bf3a4731244

commit bf3018fae3f2e564413d33aba6cdebf8868f0b4e
Author: OlgaGusarenko <email address hidden>
Date: Tue Jun 23 15:55:49 2015 +0300

    [RN6_1] Clean up

    1. Rearranges sections
    2. Improves RST
    3. Changes titles order

    Change-Id: I6110bf515667d3d6ba08ad35ff5d593dbc96641e

commit 1c7e4457808e8f2d6c56fdf31252170972e444b9
Author: Maria Zlatkova <email address hidden>
Date: Tue Jun 23 15:26:28 2015 +0300

    Replaces VBOX screenshots

    This patch:
    - replaces VBOX screenshots
    - changes the link for Download Mirantis VirtualBox scripts
     to https://docs.mirantis.com/openstack/fuel/fuel-master/#downloads

    Change-Id: I58dede960c5c3355d39b07ff44b757403f6af02c
    Closes-Bug: #1467872

commit 0a568bf53fc0e25d1d692d5d74b4a7b4d983bbcc
Author: evkonstantinov <email address hidden>
Date: Tue Jun 23 14:01:55 2015 +0300

    6.1 --separate repos

    change wording and add links to the
    separate repos feature.

    Change-Id: Ib5d0778a0d8f1534f79ed2f553574cb69a3150b0

commit 95a188b21cbdd064d92696b7920e6a0105fe0c56
Author: Maria Zlatkova <email address hidden>
Date: Tue Jun 23 12:07:28 2015 +0300

    Corrects the output 'pcs status'

    Changes the example outputs to appropriate ones.

    Change-Id: Ib6d83...

tags: added: feature
Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Alexander Kislitsky (akislitsky)
Changed in fuel:
assignee: Alexander Kislitsky (akislitsky) → Fuel Python Team (fuel-python)
Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Sebastian Kalinowski (prmtl)
Changed in fuel:
assignee: Sebastian Kalinowski (prmtl) → Fuel Python Team (fuel-python)
Dima Shulyak (dshulyak)
Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Dima Shulyak (dshulyak)
Revision history for this message
Dima Shulyak (dshulyak) wrote :

After analyzing this thread it looks like the root cause of this bug is that we dont have separate partition for /var/log, and previously it wasnt easy to add because we install os on a plain partition in case of ubuntu.

I would also add API and CLi to work with volumes, but it would be big chunk of code, and i dont know if thats ok

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (master)

Fix proposed to branch: master
Review: https://review.openstack.org/211960

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-web (master)

Change abandoned by Dmitry Shulyak (<email address hidden>) on branch: master
Review: https://review.openstack.org/211960

Revision history for this message
Dima Shulyak (dshulyak) wrote :
Changed in fuel:
assignee: Dima Shulyak (dshulyak) → Fuel Python Team (fuel-python)
status: In Progress → Confirmed
Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Fedor Zhadaev (fzhadaev)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (master)

Reviewed: https://review.openstack.org/204091
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=b7ab2e992f90ced87c013ab529a8fa62f224d8dd
Submitter: Jenkins
Branch: master

commit b7ab2e992f90ced87c013ab529a8fa62f224d8dd
Author: Alexander Kislitsky <email address hidden>
Date: Tue Jul 21 16:50:38 2015 +0300

    Mount point /var/log added for controller

    As workaround before volume manager refactoring we add
    /var/log mount point for role controller.

    Change-Id: I65c1c2f627a0124ed1f2d09d82df397012fe87f3
    Closes-Bug: #1278964
    Closes-Bug: #1394864

Changed in fuel:
status: In Progress → Fix Committed
Igor (ipukha)
tags: added: on verification
Igor (ipukha)
Changed in fuel:
status: Fix Committed → Fix Released
Igor (ipukha)
tags: removed: on verification
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-docs (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/223467

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-docs (master)

Reviewed: https://review.openstack.org/223467
Committed: https://git.openstack.org/cgit/stackforge/fuel-docs/commit/?id=2c33bcea9cbf009efbe79bbc470479307319cb06
Submitter: Jenkins
Branch: master

commit 2c33bcea9cbf009efbe79bbc470479307319cb06
Author: evkonstantinov <email address hidden>
Date: Tue Sep 15 10:53:42 2015 +0300

    Add partition for /var/log/ resolved issue to relnotes

    Change-Id: I05875381a3264f3dc2fcb8ac93a122780343108e
    Related-Bug:#1394864

tags: added: wontfix-risky
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.