maas 2.1.3 nodes do not deploy, Juju reports 'started but not deployed'

[John George]

Fresh install of maas 2.1.4+bzr5591-0ubuntu1~16.04.1
Juju 2.1.1-xenial-amd64

Juju bootstrap does not complete when selected MAAS node is KVM.

[Andres Rodriguez (andreserl)]

Hi Jon,

The following is that I see in the logs. This leads me to believe that the issue here is not MAAS bu rather Juju. Also, please attach the machines node event log (look at the machine's details page in the UI, you can view the full event log).

That said, can you try the following:

1. Deploy kvm-node-1 manually. Does it successfully deploy? If so, then this seems to definitely be a juju issue.
2. If the above succeeds, try to deploy manually.

  6 20:45:55 localhost maas.node: [info] kvm-node-1: Status transition from READY to ALLOCATED
[...]
Mar 6 20:45:56 localhost maas.node: [info] kvm-node-1: Status transition from ALLOCATED to DEPLOYING
[...]
Mar 6 21:05:48 localhost maas.node: [info] kvm-node-1: Releasing node
Mar 6 21:05:48 localhost maas.node: [info] kvm-node-1: Status transition from DEPLOYING to RELEASING
Mar 6 21:05:48 localhost maas.power: [info] Changing power state (off) of node: kvm-node-1 (4k8kmm)
Mar 6 21:05:53 localhost maas.node: [info] kvm-node-1: Status transition from RELEASING to READ

[John George (jog)]

Tried to move past this issue by removing the KVM node and deploy to real hardware. Juju was able to bootstrap it's controller but while trying to deploy the first charm, hit what looks to be the same failure. That failing machine name is 'duision'. I'll attach the maas server logs and event log below.

[John George (jog)]

I tried to deploy duision using the MAAS UI. Again it did not deploy. Here is a screen shot of the console.?field.comment=I tried to deploy duision using the MAAS UI. Again it did not deploy. Here is a screen shot of the console.

[Andres Rodriguez (andreserl)]

This sounds like your bug is this:

https://launchpad.net/bugs/1671605

Could you please confirm ?

[Craig Bender (craig-bender)]

I ran into this on our OrangeBoxes. After I eliminated every possible
option/combination (controllers, graphics, nic types, etc) , I found that
it was dual disk machines that cause the bootstrap failure.

If you run the following function, test-vm-00 will bootstrap, test-vm-01
will not (note the maas profile name and username and change accordingly)

make-test-vms(){
mkdir -p ~/TESTVMs
for i in {0..1}; do
local vmname="test-vm-0${i}"
virsh destroy $vmname &>/dev/null
virsh undefine $vmname &>/dev/null
if [[ $i = 0 ]];then
virt-install --noautoconsole --print-xml --name=$vmname --controller=scsi
--ram=4096 --disk=path= ~/TESTVM/${vmname}-1.img,bus=scsi,size=20 --vcpus=2
--network=bridge=br0 --boot=network --video=cirrus --graphics=vnc >
 ~/TESTVM/${vmname}.xml
else
virt-install --noautoconsole --print-xml --name=$vmname --controller=scsi
--ram 4096 --disk=path= ~/TESTVM/${vmname}-1.img,bus=scsi,size=20
--disk=path= ~/TESTVM/${vmname}-2.img,bus=scsi,size=5 --vcpus=2
--network=bridge=br0 --boot=network --video=cirrus --graphics=vnc >
 ~/TESTVM/${vmname}.xml
fi
virsh define ~/TESTVM/${vmname}.xml
done
unset i
printf "\nAdding virsh power chassis to MaaS...\n"
maas admin machines add-chassis hostname=qemu+ssh://ubuntu@localhost/system
chassis_type=virsh prefix_filter='test-vm-'
sleep 3
printf "\nCommissioning virtual nodes...\n"
maas admin machines accept-all
}

On Fri, Mar 10, 2017 at 7:50 AM, Chris Gregan <email address hidden>
wrote:

> ** Also affects: juju-core
> Importance: Undecided
> Status: New
>
> --
> You received this bug notification because you are subscribed to MAAS.
> Matching subscriptions: maas-bugs
> https://bugs.launchpad.net/bugs/1670499
>
> Title:
> maas 2.1.3 kvm nodes do not deploy, Juju reports 'started but not
> deployed'
>
> Status in juju-core:
> New
> Status in MAAS:
> New
>
> Bug description:
> Fresh install of maas 2.1.4+bzr5591-0ubuntu1~16.04.1
> Juju 2.1.1-xenial-amd64
>
> Juju bootstrap does not complete when selected MAAS node is KVM.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1670499/+subscriptions
>

--
*Craig Bender*
Solutions Architect
*Canonical*
Direct Dial: [phone]
Ubuntu - Linux for Human Beings
www.canonical.com - www.ubuntu.com

[Anastasia (anastasia-macmood)]

Re-targeting to "juju" project but marking as Incomplete as it seems that the issue is with MAAS 2.1.3 and has been addressed in MAAS 2.2.0. Feel free to re-open if the issue is still there for you.

[Chris Gregan (cgregan)]

Confirmed to still be present in 2.2

[Anastasia (anastasia-macmood)]

I am marking this as Invalid for Juju.
According to our CI, http://reports.vapour.ws/releases/5015, juju kvm tests on trusty/amd64 against 2.2-alpha1 pass.

[Andres Rodriguez (andreserl)]

Hi Chris, Jon,

Can you please attach the output of the following commissioning script:

00-maas-07-block-devices.out

Also, please attach a screenshot of the storage section of the said machine.

[Chris Gregan (cgregan)]

Andres,
As requested

[Chris Gregan (cgregan)]

sudo cat /tmp/user_data.sh.2XkJf6/out/00-maas-07-block-devices.out
sudo: unable to resolve host kvm-node-1
[
 {
  "ROTA": "1",
  "SIZE": "42949672960",
  "BLOCK_SIZE": "4096",
  "NAME": "vdb",
  "RO": "0",
  "PATH": "/dev/vdb",
  "RM": "0",
  "MODEL": ""
 },
 {
  "SERIAL": "60000000000000000e00000000030001",
  "ROTA": "1",
  "SIZE": "188751872",
  "BLOCK_SIZE": "1024",
  "NAME": "sda",
  "ID_PATH": "/dev/disk/by-id/wwn-0x60000000000000000e00000000030001",
  "RO": "1",
  "PATH": "/dev/sda",
  "RM": "0",
  "MODEL": "VIRTUAL-DISK"
 },
 {
  "ROTA": "1",
  "SIZE": "42949672960",
  "BLOCK_SIZE": "4096",
  "NAME": "vda",
  "RO": "0",
  "PATH": "/dev/vda",
  "RM": "0",
  "MODEL": ""
 }
]