HTTP Error in celery.log when maas-import-pxe-files is run

[Ramon Acedo]

- Last version of MAAS from ppa:maas-maintainers/stable (1.2+bzr1351+dfs at this moment)
- Ubuntu 12.04

- Releases configured in /etc/maas/import_pxe_files:
RELEASES="precise quantal"

Whenever maas-import-pxe-files is run I see the following in /var/log/maas/celery.log:

[2013-02-13 11:23:48,001: INFO/MainProcess] Got task from broker: provisioningserver.tasks.report_boot_images[c6b5dc87-e886-46f0-a7d6-3dca94c4cd9a]
[2013-02-13 11:23:48,200: ERROR/MainProcess] Task provisioningserver.tasks.report_boot_images[c6b5dc87-e886-46f0-a7d6-3dca94c4cd9a] raised exception: IOError()
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/celery/execute/trace.py", line 47, in trace
    return cls(states.SUCCESS, retval=fun(*args, **kwargs))
  File "/usr/lib/python2.7/dist-packages/celery/app/task/__init__.py", line 247, in __call__
    return self.run(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/celery/app/__init__.py", line 175, in run
    return fun(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/provisioningserver/tasks.py", line 339, in report_boot_images
    boot_images.report_to_server()
  File "/usr/lib/python2.7/dist-packages/provisioningserver/boot_images.py", line 72, in report_to_server
    submit(maas_url, api_credentials, images)
  File "/usr/lib/python2.7/dist-packages/provisioningserver/boot_images.py", line 60, in submit
    nodegroup=get_cluster_uuid(), images=json.dumps(images))
  File "/usr/lib/python2.7/dist-packages/apiclient/maas_client.py", line 191, in post
    url, method="POST", headers=headers, data=body)
  File "/usr/lib/python2.7/dist-packages/apiclient/maas_client.py", line 86, in dispatch_query
    return urllib2.urlopen(req)
  File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 406, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 519, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 444, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/celery/execute/trace.py", line 47, in trace
    return cls(states.SUCCESS, retval=fun(*args, **kwargs))
  File "/usr/lib/python2.7/dist-packages/celery/app/task/__init__.py", line 247, in __call__
    return self.run(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/celery/app/__init__.py", line 175, in run
    return fun(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/provisioningserver/tasks.py", line 339, in report_boot_images
    boot_images.report_to_server()
  File "/usr/lib/python2.7/dist-packages/provisioningserver/boot_images.py", line 72, in report_to_server
    submit(maas_url, api_credentials, images)
  File "/usr/lib/python2.7/dist-packages/provisioningserver/boot_images.py", line 60, in submit
    nodegroup=get_cluster_uuid(), images=json.dumps(images))
  File "/usr/lib/python2.7/dist-packages/apiclient/maas_client.py", line 191, in post
    url, method="POST", headers=headers, data=body)
  File "/usr/lib/python2.7/dist-packages/apiclient/maas_client.py", line 86, in dispatch_query
    return urllib2.urlopen(req)
  File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 406, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 519, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 444, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError

The output of maas-import-pxe-files is always the following:

$ sudo maas-import-pxe-files
Downloading to temporary location /tmp/tmp.Hc7KVaHHgt.
/tmp/tmp.Hc7KVaHHgt ~
2013-02-13 11:41:09 URL:http://archive.ubuntu.com/ubuntu//dists/precise/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64//linux [4965840/4965840] -> "linux" [1]
2013-02-13 11:41:12 URL:http://archive.ubuntu.com/ubuntu//dists/precise/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64//initrd.gz [17446386/17446386] -> "initrd.gz" [1]
2013-02-13 11:41:13 URL:http://archive.ubuntu.com/ubuntu//dists/quantal/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64//linux [5130968/5130968] -> "linux" [1]
2013-02-13 11:41:17 URL:http://archive.ubuntu.com/ubuntu//dists/quantal/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64//initrd.gz [18668122/18668122] -> "initrd.gz" [1]
2013-02-13 11:41:18 URL:http://archive.ubuntu.com/ubuntu//dists/precise/main/installer-i386/current/images/netboot/ubuntu-installer/i386//linux [5015840/5015840] -> "linux" [1]
2013-02-13 11:41:21 URL:http://archive.ubuntu.com/ubuntu//dists/precise/main/installer-i386/current/images/netboot/ubuntu-installer/i386//initrd.gz [15977428/15977428] -> "initrd.gz" [1]
2013-02-13 11:41:23 URL:http://archive.ubuntu.com/ubuntu//dists/quantal/main/installer-i386/current/images/netboot/ubuntu-installer/i386//linux [5171760/5171760] -> "linux" [1]
2013-02-13 11:41:26 URL:http://archive.ubuntu.com/ubuntu//dists/quantal/main/installer-i386/current/images/netboot/ubuntu-installer/i386//initrd.gz [17086667/17086667] -> "initrd.gz" [1]
2013-02-13 11:41:28 URL:http://ports.ubuntu.com/ubuntu-ports//dists/precise-updates/main/installer-armhf/current/images/highbank/netboot//vmlinuz [2973904/2973904] -> "vmlinuz" [1]
2013-02-13 11:41:32 URL:http://ports.ubuntu.com/ubuntu-ports//dists/precise-updates/main/installer-armhf/current/images/highbank/netboot//initrd.gz [4971790/4971790] -> "initrd.gz" [1]
2013-02-13 11:41:35 URL:http://ports.ubuntu.com/ubuntu-ports//dists/quantal/main/installer-armhf/current/images/highbank/netboot//vmlinuz [3738504/3738504] -> "vmlinuz" [1]
2013-02-13 11:41:39 URL:http://ports.ubuntu.com/ubuntu-ports//dists/quantal/main/installer-armhf/current/images/highbank/netboot//initrd.gz [6213909/6213909] -> "initrd.gz" [1]
~
precise/amd64: up to date [maas-precise-12.04-amd64-ephemeral-20121008]
precise/i386: up to date [maas-precise-12.04-i386-ephemeral-20121008]
precise/armhf: up to date [maas-precise-12.04-armhf-ephemeral-20121008]
quantal/amd64: up to date [maas-quantal-12.10-amd64-ephemeral-20121017]
quantal/i386: up to date [maas-quantal-12.10-i386-ephemeral-20121017]
quantal/armhf: up to date [maas-quantal-12.10-armhf-ephemeral-20121017]

When I log on the UI I always get this message in a yellow square:

"The region controller does not know whether any boot images have been imported yet. If this message does not disappear in 5 minutes, there may be a communication problem between the region worker process and the region controller. Check the region worker's logs for signs that it was unable to report to the MAAS API."

In the /var/log/maas/maas.log I get this:

Exception: 'HttpResponse' object has no attribute '_is_string'

And this thread mentions that piston 0.2.3 is not compatible with django 1.4: https://lists.launchpad.net/maas-devel/msg00572.html and in precise we have these versions:

$ dpkg -l python-django-piston python-django
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Description
+++-=====================================-=====================================-==========================================================================================
ii python-django 1.4.1-2ubuntu0.2~cloud0 High-level Python web development framework
ii python-django-piston 0.2.3-1ubuntu1 Django mini-framework creating RESTful APIs

I just backported python-django-piston from quantal which seems to have compatibility with django 1.4 but that breaks it and I get a "ValueError: bad marshal data (unknown type code)" in apache from mod_wsgi.

[Raphaël Badin (rvb)]

Hi, I think the problem you're having, as you found out, is due to the incompatibility between the version of python-django-piston in precise (0.2.3-1ubuntu1) and the version of django you're using. You're using django 1.4 and the version that works with the maas precise package in ppa:maas-maintainers/stable is the one that is in this ppa as well: 1.3.1-4ubuntu1.5~ppa1.
I'm going to mark this bug as invalid but feel free to re-open it if I'm wrong.

[Michael Jeanson (mjeanson)]

I had the same problem when I enabled the folsom cloud archive source on my Maas server, which pulled in a new version of Django.

[Ramon Acedo (ramon-linux-labs)]

As Michael mentioned in #2 this happens when enabling the folsom repository in the maas server and django 1.4 is installed from there (but not a patched piston).