[2.2.0~rc3] maas reports an incorrect amount of available storage space for a virsh pod

Bug #1687305 reported by Dmitrii Shcherbakov on 2017-04-30
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
MAAS
High
Данило Шеган
2.2
High
Данило Шеган

Bug Description

Created a virsh pod:

qemu+ssh://<username>@10.0.0.1/system

2 things:

1) storage is reported as GiB (powers of 2, usually used with RAM) in a pod resources tab while the node storage is reported in GB (powers of 10 like the storage quantities are usually reported).

2) The available storage is negative and its absolute value is equal to the storage used (see the screenshot)

On the actual system there's a lot of storage:

root@maas-master:/home/dmitriis/ha-bundle# df -h /var/lib/libvirt/images/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/Ubuntu-root 2.1T 34G 1.9T 2% /

This is rc3 originally upgraded from maas 2.1 to 2.2rc2 and then to rc3.

root@maas-master:/home/dmitriis/ha-bundle# dpkg -l 'maas*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=========================================-=========================-=========================-========================================================================================
ii maas 2.2.0~rc3+bzr6025-0ubuntu all "Metal as a Service" is a physical cloud and IPAM
ii maas-cli 2.2.0~rc3+bzr6025-0ubuntu all MAAS client and command-line interface
un maas-cluster-controller <none> <none> (no description available)
ii maas-common 2.2.0~rc3+bzr6025-0ubuntu all MAAS server common files
ii maas-dhcp 2.2.0~rc3+bzr6025-0ubuntu all MAAS DHCP server
ii maas-dns 2.2.0~rc3+bzr6025-0ubuntu all MAAS DNS server
ii maas-proxy 2.2.0~rc3+bzr6025-0ubuntu all MAAS Caching Proxy
ii maas-rack-controller 2.2.0~rc3+bzr6025-0ubuntu all Rack Controller for MAAS
ii maas-region-api 2.2.0~rc3+bzr6025-0ubuntu all Region controller API service for MAAS
ii maas-region-controller 2.2.0~rc3+bzr6025-0ubuntu all Region Controller for MAAS
un maas-region-controller-min <none> <none> (no description available)

Related branches

Dmitrii Shcherbakov (dmitriis) wrote :
description: updated
tags: added: cpe
tags: added: pod
Changed in maas:
milestone: none → 2.2.0rc4
importance: Undecided → High
Changed in maas:
assignee: nobody → Newell Jensen (newell-jensen)
Changed in maas:
milestone: 2.2.0rc4 → 2.2.1
Blake Rouse (blake-rouse) wrote :

Can you provide the output of:

virsh pool-list

Repeat the following for every pool.
virsh pool-info {name-of-pool}

Changed in maas:
status: New → Incomplete
assignee: Newell Jensen (newell-jensen) → nobody
Dmitrii Shcherbakov (dmitriis) wrote :

dmitriis@maas-master:~$ virsh pool-list
 Name State Autostart
-------------------------------------------
 default active yes

dmitriis@maas-master:~$ virsh pool-info default
Name: default
UUID: 0b36ec7d-4bda-4a1d-a35a-e06b8443d7df
State: running
Persistent: yes
Autostart: yes
Capacity: 2.02 TiB
Allocation: 35.68 GiB
Available: 1.99 TiB

Nobuto Murata (nobuto) wrote :

$ virsh pool-list
 Name State Autostart
-------------------------------------------
 default active yes
 uvtool active yes

$ virsh pool-info default
Name: default
UUID: 14fe8b2e-6a3f-475c-be84-4731cf08253a
State: running
Persistent: yes
Autostart: yes
Capacity: 117.99 GiB
Allocation: 35.02 GiB
Available: 82.97 GiB

$ virsh pool-info uvtool
Name: uvtool
UUID: 98f8636d-7c51-4363-9490-c5c64f003b75
State: running
Persistent: yes
Autostart: yes
Capacity: 117.99 GiB
Allocation: 35.02 GiB
Available: 82.97 GiB

$ maas ubuntu pods read
Success.
Machine-readable output follows:
[
    {
        "resource_uri": "/MAAS/api/2.0/pods/1/",
        "capabilities": [
            "composable",
            "dynamic_local_storage",
            "over_commit"
        ],
        "available": {
            "local_storage": 227381595626,
            "cores": -3,
            "memory": 10243
        },
        "architectures": [
            "amd64/generic"
        ],
        "used": {
            "local_storage": 26000000000,
            "cores": 7,
            "memory": 9728
        },
        "name": "pod-localhost",
        "total": {
            "local_storage": 253381595626,
            "cores": 4,
            "memory": 19971
        },
        "id": 1,
        "type": "virsh"
    }
]

Dmitrii Shcherbakov (dmitriis) wrote :

So Nobuto's stats for available disk space seem to be inconsistent as well if you convert to GiB:

echo $((227381595626 / 2**30)) GiB
211 GiB

It looks like his available space is a sum of available space in all pools

Nobuto Murata (nobuto) wrote :

Looks like the code only handles GiB, no consideration of TiB of Dmitrii's case.

====
    def get_pod_pool_size_map(self, key):
        """Return the mapping for a size calulation based on key."""
        pools = {}
        for pool in self.list_pools():
            output = self.run(['pool-info', pool]).strip()
            if output is None:
                # Skip if cannot get more information.
                continue
            capacity = float(
                self.get_key_value(output, key))
            # Convert GiB to bytes.
            pools[pool] = int(capacity * 2**30)
        return pools

Changed in maas:
status: Incomplete → New
Changed in maas:
status: New → Triaged
summary: - maas reports an incorrect amount of available storage space for a virsh
- pod
+ [2.2.0~rc3] maas reports an incorrect amount of available storage space
+ for a virsh pod
Changed in maas:
assignee: nobody → Данило Шеган (danilo)

There's a few problems in all the listings—I'll file separate bugs for the following issues:

 1. cores count can go to subzero if you compose more cores on your VMs than available on the pod (since it's customary to do that, we should probably instead display core utilisation, maybe even in percentages when lower than 1, and possibly like Nx [eg. "2.2x utilisation"] when greater than 1)

 2. local storage will be added up even though it might be using the same backing devices — however, this one can be really hard to solve properly, even if libvirtd gives us enough information (path and local storage), because some of it might still be only known to the system (eg. /var/lib/libvirt/pool1 might be a different mount point than /var/lib/libvirt/pool2). I think this one is not really worth fixing, but I'll file it anyway for reference.

Filed #1698766 for the core utilisation bug, bug #1698767 for the local storage size multiplication bug.

I'd appreciate it if you can test the change up at https://code.launchpad.net/~danilo/maas/virsh-storage-units/+merge/325911 with more than 1TiB storage pod. If not, I'll do it myself, but only later.

Changed in maas:
status: Triaged → In Progress
Dmitrii Shcherbakov (dmitriis) wrote :

On-site right now but I will try to check in the evening.

Thanks, Dmitrii: that one was against the wrong branch: try https://code.launchpad.net/~danilo/maas/virsh-storage-units/+merge/325913 instead.

You can also apply the following patch using "patch -p1" while inside /usr/lib/python3/dist-packages/:

  http://paste.ubuntu.com/24898590/

and then restart rackd. (Revert it after with "patch -p1 -R")

Dmitrii Shcherbakov (dmitriis) wrote :

Здраво Данило,

Works for me (see the screenshot).

Thanks, Дмитрий :-)

no longer affects: maas/trunk
Changed in maas:
milestone: 2.3.0 → 2.3.0alpha1
Changed in maas:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers