Commissioning fails with cloud-init meta-data unavailable

[Vance Morris]

1. I have attached the contents of /var/log/maas/*
2. $ dpkg -l '*maas*'|cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===============================-=====================================-============-=================================================
ii maas 2.1.0~alpha2+bzr5321-0ubuntu1~16.04.2 all "Metal as a Service" is a physical cloud and IPAM
ii maas-cli 2.1.0~alpha2+bzr5321-0ubuntu1~16.04.2 all MAAS client and command-line interface
un maas-cluster-controller <none> <none> (no description available)
ii maas-common 2.1.0~alpha2+bzr5321-0ubuntu1~16.04.2 all MAAS server common files
ii maas-dhcp 2.1.0~alpha2+bzr5321-0ubuntu1~16.04.2 all MAAS DHCP server
ii maas-dns 2.1.0~alpha2+bzr5321-0ubuntu1~16.04.2 all MAAS DNS server
ii maas-proxy 2.1.0~alpha2+bzr5321-0ubuntu1~16.04.2 all MAAS Caching Proxy
ii maas-rack-controller 2.1.0~alpha2+bzr5321-0ubuntu1~16.04.2 all Rack Controller for MAAS
ii maas-region-api 2.1.0~alpha2+bzr5321-0ubuntu1~16.04.2 all Region controller API service for MAAS
ii maas-region-controller 2.1.0~alpha2+bzr5321-0ubuntu1~16.04.2 all Region Controller for MAAS
un maas-region-controller-min <none> <none> (no description available)
un python-django-maas <none> <none> (no description available)
un python-maas-client <none> <none> (no description available)
un python-maas-provisioningserver <none> <none> (no description available)
ii python3-django-maas 2.1.0~alpha2+bzr5321-0ubuntu1~16.04.2 all MAAS server Django web framework (Python 3)
ii python3-maas-client 2.1.0~alpha2+bzr5321-0ubuntu1~16.04.2 all MAAS python API client (Python 3)
ii python3-maas-provisioningserver 2.1.0~alpha2+bzr5321-0ubuntu1~16.04.2 all MAAS server provisioning libraries (Python 3)

3. Steps to repro and expected result:
I have created 2 KVM machines, and on one I have installed Ubuntu 16.04.1 and MAAS 2.1.0 alpha2.
I have created an OVS bridge for both machines and connected them to my physical LAN. After enabling DHCP on the MAAS controller, and PXE booting the 2nd virtual machine, it gets an IP address and loads the ephemeral image. Unfortunately, it fails during this commissioning stage, as both 169.254.169.254 and the default gateway are not responding with the cloud-init meta-data:
http://imgur.com/a/V5unw
http://imgur.com/a/VRsyQ

I have enabled the backdoor as described here:
https://maas.ubuntu.com/docs/troubleshooting.html#debugging-ephemeral-image

$ cat /proc/cmdline
BOOT_IMAGE=ubuntu/amd64/generic/xenial/daily/boot-kernel nomodeset iscsi_target_name=iqn.2004-05.com.ubuntu:maas:ephemeral-ubuntu-amd64-generic-xenial-daily iscsi_target_ip=192.168.2.101 iscsi_target_port=3260 iscsi_initiator=cool-sheep ip=::::cool-sheep:BOOTIF ro root=/dev/disk/by-path/ip-192.168.2.101:3260-iscsi-iqn.2004-05.com.ubuntu:maas:ephemeral-ubuntu-amd64-generic-xenial-daily-lun-1 overlayroot=tmpfs cloud-config-url=http://192.168.2.73/MAAS/metadata/latest/by-id/nbf4se/?op=get_preseed log_host=192.168.2.73 log_port=514 initrd=ubuntu/amd64/generic/xenial/daily/boot-initrd BOOTIF=01-52-54-00-8e-fe-6c

[Vance Morris (vmorris)]

I cannot determine where that 192.168.2.73 address in the command line came from.. perhaps it was the IP address that was assigned during the initial DHCP PXE boot, then the system picked up a different address after cloud-init failed to find any meta-data?

I'm attaching the /boot/cloud-init.log and /boot/cloud-init-output.log files.

[Andres Rodriguez (andreserl)]

When MAAS installs, it tries to determine which is the interface / IP combination with the default gateway, so that it sets it in the configuration. That IP address is then placed in /etc/maas/regiond.conf.

Looking at your log, cloud-init is trying to contact 192.168.2.73, which seems to be coming from /etc/maas/regiond.conf. If you no longer have that IP on your system. The reason why that may be is because /etc/maas/rackd.conf is pointing to localhost for the MAAS URL. So, to fix this, do the following:

1. update /etc/maas/rackd.conf maas_url with the IP address the machines can communicate to. This is the Region IP address (if Rack and Region are in the same machine, I believe this would be 192.168.2.101 based on your logs). So:

maas_url = http://192.168.2.101:5240/MAAS

2. update your /etc/maas/regiond.conf to not use 192.168.2.73. The IP address you chose here doesn't have to be 192.168.2.101, but it can be any IP machines can reach to. Note that this IP is only used if MAAS cannot automaticlaly determine what region IP address the machine should use, and this only happens if rackd.conf is pointing to 'localhost'