Wily Deployements are failing in Maas

[Sean Feole]

MAAS Version: 1.9.2 , 1.9.3, 2.0 Beta 8

Problem Description: MAAS Wily Deployments have been failing, Even though End of Life is upon us, it still should at least be addressed, to those who are running regression testing across the distributions.

This has failed for me on arm64/amd64 so should be easily reproducible and I have rebuilt MAAS 2.0 recently and still encountering this issue.

Maas log:

Jun 30 13:39:23 maas-devel maas.node: [INFO] ms10-39-mcdivittb0: Status transition from READY to ALLOCATED
Jun 30 13:39:23 maas-devel maas.node: [INFO] ms10-39-mcdivittb0: allocated to user ubuntu
Jun 30 13:39:23 maas-devel maas.node: [INFO] ms10-39-mcdivittb0: Status transition from ALLOCATED to DEPLOYING
Jun 30 13:39:23 maas-devel maas.power: [INFO] Changing power state (on) of node: ms10-39-mcdivittb0 (4y3h7p)
Jun 30 13:39:35 maas-devel maas.power: [INFO] Changed power state (on) of node: ms10-39-mcdivittb0 (4y3h7p)
Jun 30 13:42:02 maas-devel maas.node: [INFO] ms10-39-mcdivittb0: Status transition from DEPLOYING to FAILED_DEPLOYMENT
Jun 30 13:42:02 maas-devel maas.node: [ERROR] ms10-39-mcdivittb0: Marking node failed: Installation failed (refer to the installation log for more information).

At the time Maas Switches the deployment attempt to failed we can see on the Host Console:

[ OK ] Started Initial cloud-init job (pre-networking).
         Starting Initial cloud-init job (metadata service crawler)...
[ 61.547288] cloud-init[998]: Cloud-init v. 0.7.7 running 'init' at Thu, 30 Jun 2016 10:02:08 +0000. Up 61.35 seconds.
[ 61.547966] cloud-init[998]: ci-info: ++++++++++++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++++++++++++
[ 61.548500] cloud-init[998]: ci-info: +----------+-------+------------------------------+---------------+-------+-------------------+
[ 61.549001] cloud-init[998]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
[ 61.549483] cloud-init[998]: ci-info: +----------+-------+------------------------------+---------------+-------+-------------------+
[ 61.550051] cloud-init[998]: ci-info: | enp1s0 | True | 10.229.65.139 | 255.255.0.0 | . | fc:15:b4:21:00:c2 |
[ 61.550562] cloud-init[998]: ci-info: | enp1s0 | True | fe80::fe15:b4ff:fe21:c2/64 | . | link | fc:15:b4:21:00:c2 |
[ 61.551064] cloud-init[998]: ci-info: | enp1s0d1 | False | . | . | . | fc:15:b4:21:00:c3 |
[ 61.551582] cloud-init[998]: ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | . | . |
[ 61.552094] cloud-init[998]: ci-info: | lo | True | ::1/128 | . | host | . |
[ 61.552588] cloud-init[998]: ci-info: | lxcbr0 | True | 10.0.3.1 | 255.255.255.0 | . | 96:55:5e:f3:e8:4d |
[ 61.553092] cloud-init[998]: ci-info: | lxcbr0 | True | fe80::9455:5eff:fef3:e84d/64 | . | link | 96:55:5e:f3:e8:4d |
[ 61.553568] cloud-init[998]: ci-info: +----------+-------+------------------------------+---------------+-------+-------------------+
[ 61.553971] cloud-init[998]: ci-info: ++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++
[ 61.554368] cloud-init[998]: ci-info: +-------+-------------+------------+---------------+-----------+-------+
[ 61.554762] cloud-init[998]: ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
[ 61.555161] cloud-init[998]: ci-info: +-------+-------------+------------+---------------+-----------+-------+
[ 61.555561] cloud-init[998]: ci-info: | 0 | 0.0.0.0 | 10.229.0.1 | 0.0.0.0 | enp1s0 | UG |
[ 61.555956] cloud-init[998]: ci-info: | 1 | 10.0.3.0 | 0.0.0.0 | 255.255.255.0 | lxcbr0 | U |
[ 61.556349] cloud-init[998]: ci-info: | 2 | 10.229.0.0 | 0.0.0.0 | 255.255.0.0 | enp1s0 | U |
[ 61.556741] cloud-init[998]: ci-info: +-------+-------------+------------+---------------+-----------+-------+
[ 61.559020] cloud-init[998]: 2016-06-30 10:02:09,115 - url_helper.py[WARNING]: Setting oauth clockskew for 10.229.32.22 to 14849
[ 61.559436] cloud-init[998]: 2016-06-30 10:02:09,115 - handlers.py[WARNING]: failed posting event: start: init-network/check-cache: attempting to read from cache
[FAILED] Failed to start Seed the pseudo random number generator on first boot.
See 'systemctl status pollinate.service' for details.
[ 70.916481] cloud-init[998]: Jun 30 10:02:18 ubuntu <13>Jun 30 10:02:18 pollinate[1087]: ERROR: Network communication failed [7]\n % Total % Received % Xferd Average Speed Time Time Time Current
[ 70.916974] cloud-init[998]: Dload Upload Total Spent Left Speed
[ 70.917434] cloud-init[998]: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 010:02:15.311301 * Trying 91.189.94.10...
[ 70.917866] cloud-init[998]: 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 010:02:17.513933 * After 1498ms connect time, move on!
[ 70.918287] cloud-init[998]: 10:02:17.513995 * connect to 91.189.94.10 port 443 failed: Connection timed out
[ 70.918706] cloud-init[998]: 10:02:17.514045 * Trying 91.189.94.24...
[ 70.919120] cloud-init[998]: 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 010:02:17.910749 * After 396ms connect time, move on!
[ 70.919533] cloud-init[998]: 10:02:17.910810 * connect to 91.189.94.24 port 443 failed: Connection timed out
[ 70.919944] cloud-init[998]: 10:02:17.910864 * Failed to connect to entropy.ubuntu.com port 443: Connection timed out
[ 70.920403] cloud-init[998]: 10:02:17.910906 * Closing connection 0
[ 70.920813] cloud-init[998]: curl: (7) Failed to connect to entropy.ubuntu.com port 443: Connection timed out
[ 71.011395] cloud-init[998]: 2016-06-30 10:02:18,579 - util.py[WARNING]: Running module seed_random (<module 'cloudinit.config.cc_seed_random' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_seed_random.py'>) failed
[ 71.764332] cloud-init[998]: 2016-06-30 10:02:19,332 - cc_resizefs.py[WARNING]: Device 'overlayroot' did not exist. cannot resize: dev=overlayroot mnt_point=/ path=/
         Stopping System Logging Service...
[ OK ] Stopped System Logging Service.

However in a Xenial Deployment on the same host, we do not run into networking issues of any sort.

[ 46.251591] cloud-init[957]: Cloud-init v. 0.7.7 running 'init' at Thu, 30 Jun 2016 14:15:24 +0000. Up 46.06 seconds.
[ 46.380403] cloud-init[957]: ci-info: ++++++++++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++++++++++
[ 46.524409] cloud-init[957]: ci-info: +----------+-------+----------------------------+-------------+-------+-------------------+
[ 46.668520] cloud-init[957]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
[ 46.812374] cloud-init[957]: ci-info: +----------+-------+----------------------------+-------------+-------+-------------------+
[ 46.960500] cloud-init[957]: ci-info: | enp1s0d1 | False | . | . | . | fc:15:b4:21:00:c3 |
[ 47.104407] cloud-init[957]: ci-info: | enp1s0 | True | 10.229.65.139 | 255.255.0.0 | . | fc:15:b4:21:00:c2 |
[ 47.248445] cloud-init[957]: ci-info: | enp1s0 | True | fe80::fe15:b4ff:fe21:c2/64 | . | link | fc:15:b4:21:00:c2 |
[ 47.392371] cloud-init[957]: ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | . | . |
[ 47.540464] cloud-init[957]: ci-info: | lo | True | ::1/128 | . | host | . |
[ 47.684364] cloud-init[957]: ci-info: +----------+-------+----------------------------+-------------+-------+-------------------+
[ 47.828339] cloud-init[957]: ci-info: +++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++
[ 47.948360] cloud-init[957]: ci-info: +-------+-------------+------------+-------------+-----------+-------+
[ 48.068359] cloud-init[957]: ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
[ 48.188454] cloud-init[957]: ci-info: +-------+-------------+------------+-------------+-----------+-------+
[ 48.312353] cloud-init[957]: ci-info: | 0 | 0.0.0.0 | 10.229.0.1 | 0.0.0.0 | enp1s0 | UG |
[ 48.432341] cloud-init[957]: ci-info: | 1 | 10.229.0.0 | 0.0.0.0 | 255.255.0.0 | enp1s0 | U |
[ 48.552334] cloud-init[957]: ci-info: +-------+-------------+------------+-------------+-----------+-------+
[ 48.672321] cloud-init[957]: 2016-06-30 14:15:24,599 - url_helper.py[WARNING]: Setting oauth clockskew for 10.229.32.22 to 371
[ 48.816374] cloud-init[957]: 2016-06-30 14:15:24,599 - handlers.py[WARNING]: failed posting event: start: init-network/check-cache: attempting to read from cache [trust]
[ 51.255210] cloud-init[957]: 2016-06-30 14:15:29,602 - cc_resizefs.py[WARNING]: Device 'overlayroot' did not exist. cannot resize: dev=overlayroot mnt_point=/ path=/
         Stopping System Logging Service...
[ OK ] Stopped System Logging Service.
         Starting System Logging Service...

[Sean Feole (sfeole)]

As mentioned I have encountered this in both 1.9 and 2.0 of Maas, Both Managed and Unmanaged Maas Environments

[Blake Rouse (blake-rouse)]

This is an issue with cloud-init not MAAS.

[Scott Moser (smoser)]

marked as fix-released.
as bug states, it is fixed in xenial.
wily is not supported any more, so its wont-fix there.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2686