celery from 12.04+cloudarchive is incompatible with python-amqp from cloud-tools-next

[Alexander List]

We have HP DLxxx servers with iLO3/iLO4 cards.

Before upgrading MAAS from maas 1.4+bzr1693+dfsg-0ubuntu2.3~ctools0 to 1.5+bzr2252-0ubuntu1~ctools0, IPMI power control worked fine.

After the upgrade, MAAS does not power on servers e.g. when commissioning, bootstrapping or deploying via Juju.

We verified that IPMI works using the credentials stored in the MAAS DB and ipmitool.

ii maas 1.5+bzr2252-0ubuntu1~ctools0 MAAS server all-in-one metapackage
ii maas-cli 1.5+bzr2252-0ubuntu1~ctools0 MAAS command line API tool
ii maas-cluster-controller 1.5+bzr2252-0ubuntu1~ctools0 MAAS server cluster controller
ii maas-common 1.5+bzr2252-0ubuntu1~ctools0 MAAS server common files
ii maas-dhcp 1.5+bzr2252-0ubuntu1~ctools0 MAAS DHCP server
ii maas-dns 1.5+bzr2252-0ubuntu1~ctools0 MAAS DNS server
ii maas-region-controller 1.5+bzr2252-0ubuntu1~ctools0 MAAS server complete region controller
ii maas-region-controller-min 1.5+bzr2252-0ubuntu1~ctools0 MAAS Server minimum region controller

[Julian Edwards (julian-edwards)]

Please upgrade to the supported package in the main archive.

[James Troup (elmo)]

Julian Edwards <email address hidden> writes:

> Please upgrade to the supported package in the main archive.

We're providing MAAS 1.5 for precise customers in the cloud tools
archive. Surely it's either supported, or it should be removed;
having it there but somehow unsupported, isn't sane?

--
James

[Julian Edwards (julian-edwards)]

On 13/06/14 17:12, James Troup wrote:
> Julian Edwards <email address hidden> writes:
>
>> Please upgrade to the supported package in the main archive.
>
> We're providing MAAS 1.5 for precise customers in the cloud tools
> archive. Surely it's either supported, or it should be removed;
> having it there but somehow unsupported, isn't sane?

Let me clarify, it's not supported by the maas core team, it's a
server-built package, and none of us have ever run it on precise. I
would hope, in that case, the server guys are providing the support.

[Alexander List (alexlist)]

James Page asked for celery logs from MAAS 1.5, and it turns out that after the upgrade, there are none.

Celery doesn't start:

# /usr/bin/authbind --deep /usr/sbin/maas-provision start-cluster-controller $MAAS_URL -u maas -g maas
Traceback (most recent call last):
  File "/usr/bin/celeryd", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2711, in <module>
    parse_requirements(__requires__), Environment()
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 584, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: amqp>=1.0.12,<1.1.0

[James Page (james-page)]

This install is coming from the cloud-tools-next repository, which provider amqp 1.3.3; this is not compatible with celery as found in base 12.04, hence the error in #5.

Either we need to drop the new version of python-amqp OR we need to backport celery as well for 12.04.

[Diogo Matsubara (matsubara)]

Hi Alexander, could you paste the python-celery and python-amqp versions you're using in that system?

I did the upgrade path on a pristine precise VM, going from MAAS 1.4+bzr1693+dfsg-0ubuntu2.3~ctools0 to 1.5+bzr2252-0ubuntu1~ctools0 and couldn't reproduce that celery failure.

Thanks!

[Alexander List (alexlist)]

ii python-amqp 1.3.3-1ubuntu1~ctools0 Low-level AMQP client
ii python-celery 2.4.6-1ubuntu0.1 async task/job queue based on message passing

[Diogo Matsubara (matsubara)]

I debugged this on IRC with Alexander and I think we found the issue. The machine in question has both cloud archive and cloud tools archive enabled. It turned out that the python-kombu package installed came from the cloud archive but python-celery used by MAAS from ctools requires an older version of python-kombu. python-kombu 1.4.3 doesn't have a dependency on amqp>=1.0.12,<1.1.0 but python-kombu 2.5.12-0ubuntu2~cloud1 does triggering the error in #5

root@viburnum:/var/lib/dpkg/info# apt-cache policy python-kombu
python-kombu:
  Installed: 2.5.12-0ubuntu2~cloud1
  Candidate: 2.5.12-0ubuntu2~cloud1
  Version table:
 *** 2.5.12-0ubuntu2~cloud1 0
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu/ precise-updates/havana/main amd64 Packages
        100 /var/lib/dpkg/status
     1.4.3-1 0
        500 http://archive.ubuntu.com/ubuntu/ precise/main amd64 Packages

[Scott Moser (smoser)]

cloud-tools and cloud-archive are not supported together.
What was the reason for putting them on the same system?

[Alexander List (alexlist)]

The reason is that we need recent versions of the python clients for managing the openstack cloud we deploy.

Putting them into a VM/container is not really an option for us.

We will upgrade the MAAS controller machine to trusty to get more recent versions of the clients from the main archive, and drop cloud-archive from our apt sources.

However, I don't know if that works for other users who are on precise.

$ apt-cache policy python-keystoneclient
python-keystoneclient:
  Installed: 1:0.3.2-0ubuntu1~cloud0
  Candidate: 1:0.3.2-0ubuntu1~cloud0
  Version table:
 *** 1:0.3.2-0ubuntu1~cloud0 0
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu/ precise-updates/havana/main amd64 Packages
        100 /var/lib/dpkg/status
     2012.1-0ubuntu1 0
        500 http://archive.ubuntu.com/ubuntu/ precise/main amd64 Packages

[Alexander List (alexlist)]

Upgrading to trusty worked for us.