MAAS is not listing the storage disks

Bug #1604393 reported by Praveen
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
MAAS
Expired
Undecided
Unassigned
1.9
Fix Released
Undecided
Unassigned
2.0
Expired
Undecided
Unassigned

Bug Description

The MAAS server is not listing the storage disks after the COMMISSION.
Though the commission output shows some block devices are getting detected.
I have attached commission output for reference.

Revision history for this message
Praveen (panniger) wrote :
Revision history for this message
Praveen (panniger) wrote :
Revision history for this message
Praveen (panniger) wrote :
Revision history for this message
Andres Rodriguez (andreserl) wrote :

Can you please provide the version of MAAS you are using, as well as what you are using for commissioning?

Changed in maas:
status: New → Incomplete
Revision history for this message
Praveen (panniger) wrote :

I am using the MAAS of the version 1.9.3+bzr4577-0ubuntu1 (trusty1)
And I am trying to commission a CISCO UCS Server.

Revision history for this message
Andres Rodriguez (andreserl) wrote :

Can you please upgrade to MAAS 1.9.4 and retry ? What store are you using ?

Revision history for this message
Praveen (panniger) wrote :

Hi Andre,

I tried to update the maas-cert-server but it is saying that i am already running the latest version.

Thanks,
Praveen

Revision history for this message
Praveen (panniger) wrote :

Hi Andre,

We have tried with the latest version. Please let us know if you have any work around to processed further.
Our cert run is blocked.

I really appreciate any help you provide.

Thanks,
Praveen

Revision history for this message
Virginie Dotta (vdotta) wrote :

Similar bug on the same release. Any ideas ?

Revision history for this message
Praveen (panniger) wrote :

We have not seen similar issue. Please let us know if you have any work around.

Thanks,
Praveen

Changed in maas:
status: Incomplete → New
Revision history for this message
Andres Rodriguez (andreserl) wrote :

Logs needed:

1. cloud-init.log and cloud-init-output.log from the commissioning environment (There's an option to enable SSH. You can use that to connect to the environment, wait till everything is done, and grab the logs).

2. On the same commissioning environment, grab the output of:

lsblk --exclude 1,2,7 -d -P -o NAME,RO,RM,MODEL,ROTA
udevadm info -q all -n <block-device-name>

3. On the same commissioning environment, cloud-init version (dpkg -l | grep cloud-init)

4. MAAS logs at the *exact* moment this particular machine is commissioning.

5. The commissioning output (see in the UI, the device block section. NOT lshw).

Changed in maas:
status: New → Incomplete
Revision history for this message
Virginie Dotta (vdotta) wrote :

Find logs below,

1. cloud-init.log and cloud-init-output.log - attached in the rar file

2.lsblk --exclude 1,2,7 -d -P -o NAME,RO,RM,MODEL,ROTA
udevadm info -q all -n <block-device-name>

ubuntu@Server-MAAS:/var/log$ lsblk --exclude 1,2,7 -d -P -o NAME,RO,RM,MODEL,ROTA
NAME="vda" RO="0" RM="0" MODEL="" ROTA="1"
NAME="vdb" RO="0" RM="0" MODEL="" ROTA="1"
ubuntu@Server-MAAS:/var/log$ udevadm info -q all -n vda
P: /devices/pci0000:00/0000:00:06.0/virtio3/block/vda
N: vda
E: DEVNAME=/dev/vda
E: DEVPATH=/devices/pci0000:00/0000:00:06.0/virtio3/block/vda
E: DEVTYPE=disk
E: ID_PART_TABLE_TYPE=dos
E: MAJOR=253
E: MINOR=0
E: SUBSYSTEM=block
E: USEC_INITIALIZED=4807

ubuntu@Server-MAAS:/var/log$ udevadm info -q all -n vdb
P: /devices/pci0000:00/0000:00:07.0/virtio4/block/vdb
N: vdb
S: disk/by-label/cidata
E: DEVLINKS=/dev/disk/by-label/cidata
E: DEVNAME=/dev/vdb
E: DEVPATH=/devices/pci0000:00/0000:00:07.0/virtio4/block/vdb
E: DEVTYPE=disk
E: ID_FS_LABEL=cidata
E: ID_FS_LABEL_ENC=cidata
E: ID_FS_TYPE=iso9660
E: ID_FS_USAGE=filesystem
E: ID_FS_VERSION=Joliet Extension
E: MAJOR=253
E: MINOR=16
E: SUBSYSTEM=block
E: USEC_INITIALIZED=4930

3. dpkg -l | grep cloud-init

ubuntu@Server-MAAS:/var/log$ dpkg -l | grep cloud-init
ii cloud-init 0.7.5-0ubuntu1.19 all Init scripts for cloud instances

4. MaaS Logs

The one taken from the maas.log from the MaaS Server.

Aug 1 14:38:46 Server-MAAS maas.api: [INFO] Request from user ubuntu to acquire a node with constraints <QueryDict: {u'agent_name': [u'158f0820-6427-418d-85b8-334f05cd3901'], u'name': [u'bootstrap.maas']}>
Aug 1 14:38:46 Server-MAAS maas.node: [INFO] bootstrap: Status transition from READY to ALLOCATED
Aug 1 14:38:46 Server-MAAS maas.node: [INFO] bootstrap: allocated to user ubuntu
Aug 1 14:38:52 Server-MAAS maas.node: [INFO] bootstrap: Releasing node
Aug 1 14:38:52 Server-MAAS maas.node: [INFO] bootstrap: Status transition from ALLOCATED to RELEASING
Aug 1 14:38:52 Server-MAAS maas.node: [INFO] bootstrap: Status transition from RELEASING to READY
Aug 1 14:38:52 Server-MAAS maas.dns: [INFO] Generating new DNS zone file for maas
Aug 1 14:38:52 Server-MAAS maas.dns: [INFO] Generating new DNS zone file for 4.18.172.in-addr.arpa

The output of the command sudo ./deploy.sh -o juno -s opencontrail -t nonha IBM-TME -f none
is the Logs.rar attached

5. The commissioning output (see in the UI, the device block section. NOT lshw)

[
 {
  "BLOCK_SIZE": "4096",
  "NAME": "sda",
  "ID_PATH": "/dev/disk/by-id/wwn-0x3000000600000001",
  "PATH": "/dev/sda",
  "ROTA": "1",
  "RM": "0",
  "MODEL": "VIRTUAL-DISK",
  "RO": "1",
  "SERIAL": "3000000600000001",
  "SIZE": "1468006400"
 },
 {
  "BLOCK_SIZE": "4096",
  "NAME": "vda",
  "PATH": "/dev/vda",
  "ROTA": "1",
  "RM": "0",
  "MODEL": "",
  "RO": "0",
  "SIZE": "128849018880"
 }
]

Revision history for this message
Virginie Dotta (vdotta) wrote :
Revision history for this message
sivarama prasad Ravipati (sivar-y-deactivatedaccount) wrote :
Download full text (27.5 KiB)

Hi. I am Using MAAS 1.9.3 on Ubuntu 14.04[trusty]. I am using DELL servers. Power type is IPMI. Power driver as LAN_2_0[IPMI 2.0].

I followed http://www.ubuntu.com/download/cloud/install-openstack-with-autopilot. I want to deploy OpenStack using Autopilot on top of Ubuntu 14.04.

Ubuntu MAAS not recognizing Nodes storage devices. Without recognizing nodes it was moving to "Ready" state and the power state is "on".

And When I tried to deploy Nodes it was showing "The deploy action for 1 node failed with error: {"network": ["Node must be configured to use a network"]}"

cloud-init.log
==============

Aug 1 12:24:51 rich-oil [CLOUDINIT] helpers.py[DEBUG]: Running config-scripts-per-instance using lock (<FileLock using file '/var/lib/cloud/instances/node-8d1bc364-57e0-11e6-bcc1-080027893615/sem/config_scripts_per_instance'>)
Aug 1 12:24:51 rich-oil [CLOUDINIT] util.py[DEBUG]: Writing to /var/lib/cloud/instances/node-8d1bc364-57e0-11e6-bcc1-080027893615/sem/config_scripts_user - wb: [420] 20 bytes
Aug 1 12:24:51 rich-oil [CLOUDINIT] helpers.py[DEBUG]: Running config-scripts-user using lock (<FileLock using file '/var/lib/cloud/instances/node-8d1bc364-57e0-11e6-bcc1-080027893615/sem/config_scripts_user'>)
Aug 1 12:24:51 rich-oil [CLOUDINIT] util.py[DEBUG]: Running command ['/var/lib/cloud/instance/scripts/user_data.sh'] with allowed return codes [0] (shell=False, capture=False)
Aug 1 12:26:31 rich-oil [CLOUDINIT] util.py[DEBUG]: Writing to /var/lib/cloud/instances/node-8d1bc364-57e0-11e6-bcc1-080027893615/sem/config_ssh_authkey_fingerprints - wb: [420] 20 bytes
Aug 1 12:26:31 rich-oil [CLOUDINIT] helpers.py[DEBUG]: Running config-ssh-authkey-fingerprints using lock (<FileLock using file '/var/lib/cloud/instances/node-8d1bc364-57e0-11e6-bcc1-080027893615/sem/config_ssh_authkey_fingerprints'>)
Aug 1 12:26:31 rich-oil [CLOUDINIT] importer.py[DEBUG]: Looking for modules ['cloudinit.mergers.m_list'] that have attributes ['Merger']
Aug 1 12:26:31 rich-oil [CLOUDINIT] importer.py[DEBUG]: Found m_list with attributes ['Merger'] in ['cloudinit.mergers.m_list']
Aug 1 12:26:31 rich-oil [CLOUDINIT] importer.py[DEBUG]: Looking for modules ['cloudinit.mergers.m_dict'] that have attributes ['Merger']
Aug 1 12:26:31 rich-oil [CLOUDINIT] importer.py[DEBUG]: Found m_dict with attributes ['Merger'] in ['cloudinit.mergers.m_dict']
Aug 1 12:26:31 rich-oil [CLOUDINIT] importer.py[DEBUG]: Looking for modules ['cloudinit.mergers.m_str'] that have attributes ['Merger']
Aug 1 12:26:31 rich-oil [CLOUDINIT] importer.py[DEBUG]: Found m_str with attributes ['Merger'] in ['cloudinit.mergers.m_str']
Aug 1 12:26:31 rich-oil [CLOUDINIT] __init__.py[DEBUG]: Merging 'dict' into 'dict' using method '_handle_unknown' of 'LookupMerger: (3)'
Aug 1 12:26:31 rich-oil [CLOUDINIT] __init__.py[DEBUG]: Merging using located merger 'DictMerger: (method=no_replace,recurse_str=False,recurse_dict=True,recurse_array=False,allow_delete=False)' since it had method '_on_dict'
Aug 1 12:26:31 rich-oil [CLOUDINIT] importer.py[DEBUG]: Looking for modules ['cloudinit.mergers.m_list'] that have attributes ['Merger']
Aug 1 12:26:31 rich-oil [CLOUDINIT] importer.py[DEBUG]: Found m_lis...

Revision history for this message
Virginie Dotta (vdotta) wrote :

I forgot to say that I upgraded to MaaS 1.9.4 and same issue !!

Revision history for this message
sivarama prasad Ravipati (sivar-y-deactivatedaccount) wrote :
Download full text (12.2 KiB)

1. cloud-init.log and cloud-init-output.log from the commissioning environment
ubuntu@inconsequential-dock:/var/log$ sudo tail -f cloud-init.log
sudo: unable to resolve host inconsequential-dock
Aug 1 15:39:17 inconsequential-dock [CLOUDINIT] util.py[DEBUG]: Cloud-init v. 0.7.5 finished at Mon, 01 Aug 2016 15:39:17 +0000. Datasource DataSourceMAAS [http://192.168.10.15/MAAS/metadata/]. Up 138.65 seconds
Aug 1 15:39:17 inconsequential-dock [CLOUDINIT] util.py[DEBUG]: Writing to /var/lib/cloud/instance/boot-finished - wb: [420] 52 bytes
Aug 1 15:39:17 inconsequential-dock [CLOUDINIT] util.py[DEBUG]: Writing to /var/lib/cloud/instances/node-5202c09a-57fc-11e6-9daf-080027893615/sem/config_power_state_change - wb: [420] 20 bytes
Aug 1 15:39:17 inconsequential-dock [CLOUDINIT] helpers.py[DEBUG]: Running config-power-state-change using lock (<FileLock using file '/var/lib/cloud/instances/node-5202c09a-57fc-11e6-9daf-080027893615/sem/config_power_state_change'>)
Aug 1 15:39:17 inconsequential-dock [CLOUDINIT] cc_power_state_change.py[DEBUG]: no power_state provided. doing nothing
Aug 1 15:39:17 inconsequential-dock [CLOUDINIT] cloud-init[DEBUG]: Ran 11 modules with 0 failures
Aug 1 15:39:17 inconsequential-dock [CLOUDINIT] util.py[DEBUG]: Creating symbolic link from '/run/cloud-init/result.json' => '../../var/lib/cloud/data/result.json'
Aug 1 15:39:17 inconsequential-dock [CLOUDINIT] util.py[DEBUG]: Reading from /proc/uptime (quiet=False)
Aug 1 15:39:17 inconsequential-dock [CLOUDINIT] util.py[DEBUG]: Read 15 bytes from /proc/uptime
Aug 1 15:39:17 inconsequential-dock [CLOUDINIT] util.py[DEBUG]: cloud-init mode 'modules' took 95.530 seconds (95.53)
^Cubuntu@inconsequential-dock:/var/log$ sudo tail -f cloud-init-output.log
sudo: unable to resolve host inconsequential-dock
Success
Success
Success
Success
Success
Success
Success
Success
Success
Cloud-init v. 0.7.5 finished at Mon, 01 Aug 2016 15:39:17 +0000. Datasource DataSourceMAAS [http://192.168.10.15/MAAS/metadata/]. Up 138.65 seconds

2. On the same commissioning environment, grab the output of:
lsblk --exclude 1,2,7 -d -P -o NAME,RO,RM,MODEL,ROTA
udevadm info -q all -n <block-device-name>

ubuntu@inconsequential-dock:/var/log$ lsblk --exclude 1,2,7 -d -P -o NAME,RO,RM,MODEL,ROTA
NAME="sda" RO="0" RM="0" MODEL="ST1000NM0033-9ZM" ROTA="1"
NAME="sdb" RO="1" RM="0" MODEL="VIRTUAL-DISK " ROTA="1"

ubuntu@inconsequential-dock:/var/log$ udevadm info -q all -n sda
P: /devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:0/1:0:0:0/block/sda
N: sda
S: disk/by-id/ata-ST1000NM0033-9ZM173_Z1W2TF35
S: disk/by-id/wwn-0x5000c500794c8b10
E: DEVLINKS=/dev/disk/by-id/ata-ST1000NM0033-9ZM173_Z1W2TF35 /dev/disk/by-id/wwn-0x5000c500794c8b10
E: DEVNAME=/dev/sda
E: DEVPATH=/devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:0/1:0:0:0/block/sda
E: DEVTYPE=disk
E: ID_ATA=1
E: ID_ATA_DOWNLOAD_MICROCODE=1
E: ID_ATA_FEATURE_SET_HPA=1
E: ID_ATA_FEATURE_SET_HPA_ENABLED=1
E: ID_ATA_FEATURE_SET_PM=1
E: ID_ATA_FEATURE_SET_PM_ENABLED=1
E: ID_ATA_FEATURE_SET_SECURITY=1
E: ID_ATA_FEATURE_SET_SECURITY_ENABLED=0
E: ID_ATA_FEATURE_SET_SECURITY_ENHANCED_ERASE_UNIT_MIN=116
E: ID_ATA_FEATURE_SET_SECURITY_ERASE_UNIT_MIN=1...

Revision history for this message
Virginie Dotta (vdotta) wrote :

Hi there,

I managed to list nodes and see all disks.

To make this possible,
1/ first, install the MaaS Server
2/ Boot a node you want to enlist, using an ubuntu ISO
3/ Choose to connect it to an existing MaaS Server. The node will shutdown
4/ Go to the MaaS Server into the Nodes tab and click on the Add Hardware button on the top right corner. Choose Machine
5/ Fill in all information regarding your node. When you will click on the Save button, the node will automatically boot and commission itself
6/ Just wait some minutes, the node will turns off and will be set to Ready in your MaaS GUI. You will see also disk information

:)

Revision history for this message
Andres Rodriguez (andreserl) wrote :

We investigated a similar issue in a different bug report and we found out that this was a problem when using the trusty kernel rather than the xenial kernel. In order to verify this, please do the following:

1. Change from images from 'releases' to 'daily' (Under the Settings Page, Boot Images section).
2. Use hwe-x Kernel for commissioning instead. (Under the Settings page 'Commissioning' section).
3. Re-attempt commissioning.

In the meantime, I'll mark this bug as 'Incomplete'. If you believe this bug is still present, please mark it again as 'New'.

Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for MAAS trunk because there has been no activity for 60 days.]

Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for MAAS 1.9 because there has been no activity for 60 days.]

Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for MAAS 2.0 because there has been no activity for 60 days.]

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.