MAAS 1.8.0+bzr4001-0ubuntu2 won't import boot resource images

[Scott Croft]

MAAS 1.8.0 will import the Trusty and Precise images without issue, but executing the maas admin boot-resources import fails. The only errors generated are:

In /var/log/maas/maas.log
Jul 15 11:42:01 OrangeBox56 maas.import-images: [INFO] Started importing boot images.
Jul 15 11:42:04 OrangeBox56 maas.import-images: [WARNING] I/O error while syncing boot images. If this problem persists, verify network connectivity and disk usage.
Jul 15 11:42:04 OrangeBox56 maas.import-images: [WARNING] Finished importing boot images, the region does not have any boot images available.
Jul 15 11:42:04 OrangeBox56 maas.tftp: [WARNING] No boot images have been imported from the region.

ubuntu@OrangeBox56:~$ dpkg -l '*maas*'|cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=====================================================-===================================================-============-===============================================================================
ii maas 1.8.0+bzr4001-0ubuntu2~trusty1 all MAAS server all-in-one metapackage
ii maas-cli 1.8.0+bzr4001-0ubuntu2~trusty1 all MAAS command line API tool
ii maas-cluster-controller 1.8.0+bzr4001-0ubuntu2~trusty1 all MAAS server cluster controller
ii maas-common 1.8.0+bzr4001-0ubuntu2~trusty1 all MAAS server common files
ii maas-dhcp 1.8.0+bzr4001-0ubuntu2~trusty1 all MAAS DHCP server
ii maas-dns 1.8.0+bzr4001-0ubuntu2~trusty1 all MAAS DNS server
ii maas-proxy 1.8.0+bzr4001-0ubuntu2~trusty1 all MAAS Caching Proxy
ii maas-region-controller 1.8.0+bzr4001-0ubuntu2~trusty1 all MAAS server complete region controller
ii maas-region-controller-min 1.8.0+bzr4001-0ubuntu2~trusty1 all MAAS Server minimum region controller
ii python-django-maas 1.8.0+bzr4001-0ubuntu2~trusty1 all MAAS server Django web framework
ii python-maas-client 1.8.0+bzr4001-0ubuntu2~trusty1 all MAAS python API client
ii python-maas-provisioningserver 1.8.0+bzr4001-0ubuntu2~trusty1 all MAAS server provisioning libraries

[Blake Rouse (blake-rouse)]

Looking at the logs it looks the region was able to download the images, but the cluster is having an IOError when it is trying to communicate to the region. You need to verify that the MAAS_URL in your configuration is an ipaddress that your cluster can communicate to.

The cluster will connect to the region at images-stream/streams/v1/index.json, make sure that you can access that address. Also make sure that the cluster can download that file, if it cannot access that endpoint then it will not be able to sync to the region.

Other questions that can help with this issue are: Is the cluster on the same machine as the region? Is any iptables rules in place that could be blocking the traffic?

[Andres Rodriguez (andreserl)]

Hi Scott,

I see this in the logs:

Jul 15 11:42:04 OrangeBox56 maas.import-images: [WARNING] I/O error while syncing boot images. If this problem persists, verify network connectivity and disk usage.

Are you sure you have network connectivity? Is there a firewall or proxy that might be preventing from importing the images? Or, is your disk full?

[Andres Rodriguez (andreserl)]

Looking at the logs that seems to be issues with your environment rather than the import of the images. I'll mark this invalid in the meantime. If you confirm that MAAS can access the internet, there's no proxy or anything that's preventing MAAS from downloading the images, you can re-open this bug.

[Scott Croft (secroft)]

Andres,
     I've verified that I can get to the internet from the machine. This is
on an orangebox. We're testing out a new package install and I've finally
gotten it to work, but had to do some workarounds, which we're trying to
avoid. I'll let you know.

Thanks

Scott

On Wed, Jul 15, 2015 at 2:04 PM, Andres Rodriguez <email address hidden>
wrote:

> Looking at the logs that seems to be issues with your environment rather
> than the import of the images. I'll mark this invalid in the meantime.
> If you confirm that MAAS can access the internet, there's no proxy or
> anything that's preventing MAAS from downloading the images, you can re-
> open this bug.
>
> ** Changed in: maas
> Status: Incomplete => Invalid
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1474967
>
> Title:
> MAAS 1.8.0+bzr4001-0ubuntu2 won't import boot resource images
>
> Status in MAAS:
> Invalid
>
> Bug description:
> MAAS 1.8.0 will import the Trusty and Precise images without issue,
> but executing the maas admin boot-resources import fails. The only
> errors generated are:
>
> In /var/log/maas/maas.log
> Jul 15 11:42:01 OrangeBox56 maas.import-images: [INFO] Started importing
> boot images.
> Jul 15 11:42:04 OrangeBox56 maas.import-images: [WARNING] I/O error
> while syncing boot images. If this problem persists, verify network
> connectivity and disk usage.
> Jul 15 11:42:04 OrangeBox56 maas.import-images: [WARNING] Finished
> importing boot images, the region does not have any boot images available.
> Jul 15 11:42:04 OrangeBox56 maas.tftp: [WARNING] No boot images have
> been imported from the region.
>
>
> ubuntu@OrangeBox56:~$ dpkg -l '*maas*'|cat
> Desired=Unknown/Install/Remove/Purge/Hold
> |
> Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
> |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
> ||/ Name Version
> Architecture Description
>
> +++-=====================================================-===================================================-============-===============================================================================
> ii maas
> 1.8.0+bzr4001-0ubuntu2~trusty1 all MAAS
> server all-in-one metapackage
> ii maas-cli
> 1.8.0+bzr4001-0ubuntu2~trusty1 all MAAS
> command line API tool
> ii maas-cluster-controller
> 1.8.0+bzr4001-0ubuntu2~trusty1 all MAAS
> server cluster controller
> ii maas-common
> 1.8.0+bzr4001-0ubuntu2~trusty1 all MAAS
> server common files
> ii maas-dhcp
> 1.8.0+bzr4001-0ubuntu2~trusty1 all MAAS DHCP
> server
> ii maas-dns
> 1.8.0+bzr4001-0ubuntu2~trusty1 all MAAS DNS
> server
> ii maas-proxy
> 1.8.0+bzr4001-0ubuntu2~trusty1 all MAAS
> Caching Proxy
> ii maas-region-controller
> 1.8.0+bzr4001-0ubuntu2~trusty1 all MAAS
> s...

[Dmitri (dmitry-a-moudraninets)]

Same issue here.

Sep 10 13:57:20 ubuntu-maas-test3 maas.rpc.clusters: [INFO] Cluster registered: Cluster master (fbe658cf-c003-4aeb-b110-0d30296f8945)
Sep 10 13:57:20 ubuntu-maas-test3 maas.import-images: [INFO] Started importing boot images.
Sep 10 13:57:20 ubuntu-maas-test3 maas.import-images: [WARNING] I/O error while syncing boot images. If this problem persists, verify network connectivity and disk usage.
Sep 10 13:57:20 ubuntu-maas-test3 maas.import-images: [WARNING] Finished importing boot images, the region does not have any boot images available.
Sep 10 13:57:20 ubuntu-maas-test3 maas.rpc.clusters: [INFO] Cluster registered: Cluster master (fbe658cf-c003-4aeb-b110-0d30296f8945)
Sep 10 13:57:22 ubuntu-maas-test3 maas.lease_upload_service: [INFO] Uploading 0 DHCP leases to region controller.
Sep 10 13:57:22 ubuntu-maas-test3 maas.bootsources: [INFO] Updated boot sources cache.
Sep 10 13:57:22 ubuntu-maas-test3 maas.bootresources: [INFO] Started importing of boot images from 1 source(s).
Sep 10 13:57:23 ubuntu-maas-test3 maas.bootresources: [INFO] Importing images from source: http://maas.ubuntu.com/images/ephemeral-v2/releases/
Sep 10 13:57:24 ubuntu-maas-test3 maas.bootsources: [INFO] Updated boot sources cache.
Sep 10 13:57:24 ubuntu-maas-test3 maas.bootresources: [INFO] Finished importing of boot images from 1 source(s).
Sep 10 13:57:24 ubuntu-maas-test3 maas.import-images: [INFO] Started importing boot images.
Sep 10 13:57:24 ubuntu-maas-test3 maas.import-images: [WARNING] I/O error while syncing boot images. If this problem persists, verify network connectivity and disk usage.
Sep 10 13:57:24 ubuntu-maas-test3 maas.import-images: [WARNING] Finished importing boot images, the region does not have any boot images available.

sysadmin@ubuntu-maas-test3:~$ ping google.com
PING google.com (216.58.211.174) 56(84) bytes of data.
64 bytes from dub08s01-in-f174.1e100.net (216.58.211.174): icmp_seq=1 ttl=56 time=4.19 ms
64 bytes from dub08s01-in-f174.1e100.net (216.58.211.174): icmp_seq=2 ttl=56 time=3.84 ms
64 bytes from dub08s01-in-f174.1e100.net (216.58.211.174): icmp_seq=3 ttl=56 time=4.04 ms
^C
--- google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2006ms
rtt min/avg/max/mdev = 3.848/4.031/4.198/0.161 ms

[James Tunnicliffe (dooferlad)]

Same bug here.

Just installed a MAAS server in LXC:

root@maaster:/var/log/maas# tail -f maas.log
Nov 11 11:03:42 maaster maas.import-images: [WARNING] I/O error while syncing boot images. If this problem persists, verify network connectivity and disk usage.
Nov 11 11:03:42 maaster maas.import-images: [WARNING] Finished importing boot images, the region does not have any boot images available.
Nov 11 11:03:42 maaster maas.tftp: [WARNING] No boot images have been imported from the region.
Nov 11 11:04:42 maaster maas.lease_upload_service: [INFO] Uploading 2 DHCP leases to region controller.
Nov 11 11:05:56 maaster maas.service_monitor: [INFO] Service 'maas-dhcpd' has been restarted. Its current state is 'on' and 'running'.
Nov 11 11:06:57 maaster maas.bootsources: [INFO] Updated boot sources cache.
Nov 11 11:06:57 maaster maas.bootresources: [INFO] Started importing of boot images from 1 source(s).
Nov 11 11:06:58 maaster maas.bootresources: [INFO] Importing images from source: http://maas.ubuntu.com/images/ephemeral-v2/releases/
Nov 11 11:06:59 maaster maas.bootresources: [INFO] Finished importing of boot images from 1 source(s).
Nov 11 11:06:59 maaster maas.import-images: [INFO] Started importing boot images.
Nov 11 11:08:42 maaster maas.import-images: [INFO] Started importing boot images.
Nov 11 11:08:42 maaster maas.import-images: [WARNING] I/O error while syncing boot images. If this problem persists, verify network connectivity and disk usage.
Nov 11 11:08:42 maaster maas.import-images: [WARNING] Finished importing boot images, the region does not have any boot images available.
Nov 11 11:08:42 maaster maas.tftp: [WARNING] No boot images have been imported from the region.
^C
root@maaster:/var/log/maas# ping 192.168.0.2
PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
64 bytes from 192.168.0.2: icmp_seq=1 ttl=63 time=0.333 ms
64 bytes from 192.168.0.2: icmp_seq=2 ttl=63 time=0.369 ms
^C
--- 192.168.0.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.333/0.351/0.369/0.018 ms
root@maaster:/var/log/maas# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=52 time=18.1 ms
^C
--- 8.8.8.8 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 18.188/18.188/18.188/0.000 ms
root@maaster:/var/log/maas# ping nanosheep.org
PING nanosheep.org (104.28.16.104) 56(84) bytes of data.
64 bytes from 104.28.16.104: icmp_seq=1 ttl=54 time=14.2 ms
64 bytes from 104.28.16.104: icmp_seq=2 ttl=54 time=14.1 ms
^C
--- nanosheep.org ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 14.189/14.212/14.236/0.121 ms
root@maaster:/var/log/maas# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sdb1 229508316 163648356 54178580 76% /
cgroup 12 0 12 0% /sys/fs/cgroup
none 100 4 96 4% /dev
tmpfs 8164772 0 8164772 0% /sys/fs/cgroup/cgmanager
none 1632956 96 1632860 1% /run
none 5120 0 5120 0% /run/lock...

[Andres Rodriguez (andreserl)]

Hi James,

Are you behind a proxy?

Thanks.

[James Tunnicliffe (dooferlad)]

No, but I looked further and spotted this:

Nov 11 11:15:21 maaster maas.boot_image_download_service: [ERROR] Failed to download images: Command `sudo /usr/bin/uec2roottar --user=maas /var/lib/maas/boot-resources/cache/root-image-da293bb56890ea54a339bef967c375c32bed51b8e2496bf410ccc79da246f309 /var/lib/maas/boot-resources/cache/root-tgz-da293bb56890ea54a339bef967c375c32bed51b8e2496bf410ccc79da246f309` returned non-zero exit status 1:#012mount: Could not find any loop device. Maybe this kernel does not know#012 about the loop device? (If so, recompile or `modprobe loop'.)#012Traceback (most recent call last):#012 File "/usr/bin/uec2roottar", line 27, in <module>#012 uec2roottar.main(args)#012 File "/usr/lib/python2.7/dist-packages/provisioningserver/import_images/uec2roottar.py", line 181, in main#012 extract_image(image, output)#012 File "/usr/lib/python2.7/dist-packages/provisioningserver/import_images/uec2roottar.py", line 150, in extract_image#012 with loop_mount(image, mountpoint):#012 File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__#012 return self.gen.next()#012 File "/usr/lib/python2.7/dist-packages/provisioningserver/import_images/uec2roottar.py", line 130, in loop_mount#012 check_call(['mount', '-o', 'ro', image, mountpoint])#012 File "/usr/lib/python2.7/subprocess.py", line 540, in check_call#012 raise CalledProcessError(retcode, cmd)#012subprocess.CalledProcessError: Command '[u'mount', u'-o', u'ro', '/var/lib/maas/boot-resources/cache/root-image-da293bb56890ea54a339bef967c375c32bed51b8e2496bf410ccc79da246f309', u'/tmp/maas-w9R7ki']' returned non-zero exit status 2
Nov 11 11:15:21 maaster maas.tftp: [WARNING] No boot images have been imported from the region.

So it looks like LXC + loop device problems.

[Andres Rodriguez (andreserl)]

Indeed, that seems that app armor might be preventing you from creating loop devices. Marking this as invalid again.

Some info here:

http://askubuntu.com/questions/376345/allow-loop-mounting-files-inside-lxc-containers