Ubuntu
maas package

UI URL displays "200 Error" page after CD install

Bug #1065763 reported by dann frazier
20
This bug affects 4 people
Affects Status Importance Assigned to Milestone
maas (Ubuntu)
Fix Released
Critical
Andres Rodriguez

Bug Description

I installed a MAAS server into a VM w/ the beta2 quantal cd:
  ubuntu-12.10-beta2-server-amd64.iso

I did pass console=ttyS0, and I am in an environment w/ no NTP server, but other than that a very "vanilla" install. After the intall completed, browsing to the UI page gave me a "200 Error" page. I dug through the apache logs, and found messages like this whenever I'd hit refresh on the UI page:

==> error.log <==
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] mod_wsgi (pid=1173): Target WSGI script '/usr/share/maas/wsgi.py' cannot be loaded as Python module.
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] mod_wsgi (pid=1173): Exception occurred processing WSGI script '/usr/share/maas/wsgi.py'.
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] Traceback (most recent call last):
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] File "/usr/share/maas/wsgi.py", line 30, in <module>
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] start_up()
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] File "/usr/lib/python2.7/dist-packages/maasserver/start_up.py", line 59, in start_up
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] inner_start_up()
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] File "/usr/lib/python2.7/dist-packages/maasserver/start_up.py", line 87, in inner_start_up
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] write_full_dns_config(reload_retry=True)
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] File "/usr/lib/python2.7/dist-packages/maasserver/dns.py", line 309, in write_full_dns_config
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] args=[['reload'], reload_retry]))
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] File "/usr/lib/python2.7/dist-packages/celery/app/task/__init__.py", line 353, in delay
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] return self.apply_async(args, kwargs)
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] File "/usr/lib/python2.7/dist-packages/celery/app/task/__init__.py", line 449, in apply_async
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] publish = publisher or self.app.amqp.publisher_pool.acquire(block=True)
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 657, in acquire
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] R = self.prepare(R)
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] File "/usr/lib/python2.7/dist-packages/kombu/pools.py", line 54, in prepare
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] p = p()
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] File "/usr/lib/python2.7/dist-packages/kombu/pools.py", line 45, in <lambda>
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] return lambda: self.create_producer()
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] File "/usr/lib/python2.7/dist-packages/celery/app/amqp.py", line 265, in create_producer
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] pub = self.app.amqp.TaskPublisher(conn, auto_declare=False)
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] File "/usr/lib/python2.7/dist-packages/celery/app/amqp.py", line 328, in TaskPublisher
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] return TaskPublisher(*args, **self.app.merge(defaults, kwargs))
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] File "/usr/lib/python2.7/dist-packages/celery/app/amqp.py", line 158, in __init__
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] super(TaskPublisher, self).__init__(*args, **kwargs)
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] File "/usr/lib/python2.7/dist-packages/kombu/compat.py", line 61, in __init__
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] super(Publisher, self).__init__(connection, self.exchange, **kwargs)
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 79, in __init__
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] self.revive(self.channel)
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 168, in revive
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] channel = channel.default_channel
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 581, in default_channel
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] self.connection
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 574, in connection
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] self._connection = self._establish_connection()
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 533, in _establish_connection
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] conn = self.transport.establish_connection()
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] File "/usr/lib/python2.7/dist-packages/kombu/transport/amqplib.py", line 279, in establish_connection
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] connect_timeout=conninfo.connect_timeout)
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] File "/usr/lib/python2.7/dist-packages/kombu/transport/amqplib.py", line 89, in __init__
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] super(Connection, self).__init__(*args, **kwargs)
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/connection.py", line 144, in __init__
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] (10, 30), # tune
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 95, in wait
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] self.channel_id, allowed_methods)
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/connection.py", line 202, in _wait_method
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] self.method_reader.read_method()
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 221, in read_method
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] raise m
[Thu Oct 11 18:03:33 2012] [error] [client 10.250.35.107] error: [Errno 104] Connection reset by peer

==> access.log <==
10.250.35.107 - - [11/Oct/2012:18:03:30 -0500] "GET /MAAS/ HTTP/1.1" 500 630 "-" "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:16.0) Gecko/20100101 Firefox/16.0"

==> error.log <==
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] mod_wsgi (pid=1172): Target WSGI script '/usr/share/maas/wsgi.py' cannot be loaded as Python module.
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] mod_wsgi (pid=1172): Exception occurred processing WSGI script '/usr/share/maas/wsgi.py'.
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] Traceback (most recent call last):
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] File "/usr/share/maas/wsgi.py", line 30, in <module>
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] start_up()
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] File "/usr/lib/python2.7/dist-packages/maasserver/start_up.py", line 59, in start_up
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] inner_start_up()
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] File "/usr/lib/python2.7/dist-packages/maasserver/start_up.py", line 87, in inner_start_up
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] write_full_dns_config(reload_retry=True)
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] File "/usr/lib/python2.7/dist-packages/maasserver/dns.py", line 309, in write_full_dns_config
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] args=[['reload'], reload_retry]))
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] File "/usr/lib/python2.7/dist-packages/celery/app/task/__init__.py", line 353, in delay
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] return self.apply_async(args, kwargs)
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] File "/usr/lib/python2.7/dist-packages/celery/app/task/__init__.py", line 449, in apply_async
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] publish = publisher or self.app.amqp.publisher_pool.acquire(block=True)
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 657, in acquire
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] R = self.prepare(R)
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] File "/usr/lib/python2.7/dist-packages/kombu/pools.py", line 54, in prepare
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] p = p()
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] File "/usr/lib/python2.7/dist-packages/kombu/pools.py", line 45, in <lambda>
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] return lambda: self.create_producer()
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] File "/usr/lib/python2.7/dist-packages/celery/app/amqp.py", line 265, in create_producer
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] pub = self.app.amqp.TaskPublisher(conn, auto_declare=False)
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] File "/usr/lib/python2.7/dist-packages/celery/app/amqp.py", line 328, in TaskPublisher
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] return TaskPublisher(*args, **self.app.merge(defaults, kwargs))
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] File "/usr/lib/python2.7/dist-packages/celery/app/amqp.py", line 158, in __init__
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] super(TaskPublisher, self).__init__(*args, **kwargs)
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] File "/usr/lib/python2.7/dist-packages/kombu/compat.py", line 61, in __init__
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] super(Publisher, self).__init__(connection, self.exchange, **kwargs)
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 79, in __init__
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] self.revive(self.channel)
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 168, in revive
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] channel = channel.default_channel
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 581, in default_channel
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] self.connection
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 574, in connection
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] self._connection = self._establish_connection()
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 533, in _establish_connection
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] conn = self.transport.establish_connection()
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] File "/usr/lib/python2.7/dist-packages/kombu/transport/amqplib.py", line 279, in establish_connection
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] connect_timeout=conninfo.connect_timeout)
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] File "/usr/lib/python2.7/dist-packages/kombu/transport/amqplib.py", line 89, in __init__
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] super(Connection, self).__init__(*args, **kwargs)
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/connection.py", line 144, in __init__
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] (10, 30), # tune
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 95, in wait
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] self.channel_id, allowed_methods)
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/connection.py", line 202, in _wait_method
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] self.method_reader.read_method()
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 221, in read_method
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] raise m
[Thu Oct 11 18:03:38 2012] [error] [client 127.0.0.1] error: [Errno 104] Connection reset by peer

==> access.log <==
127.0.0.1 - - [11/Oct/2012:18:03:35 -0500] "POST /MAAS/api/1.0/nodegroups/?op=register HTTP/1.1" 500 797 "-" "Python-urllib/2.7"

Looking at other logs that appeared at the same time, I see this in RabbiqMQ's log:
=ERROR REPORT==== 11-Oct-2012::18:12:39 ===
closing AMQP connection <0.561.0> (10.35.11.4:36784 -> 10.35.11.4:5672):
{channel0_error,starting,
                {amqp_error,access_refused,
                            "AMQPLAIN login refused: user 'maas_workers' - invalid credentials",
                            'connection.start_ok'}}

'dpkg-reconfigure maas-region-controller' (w/o changing any settings) seems to fix this up.

I was able to reproduce this twice, each with a fresh install from the CD. rbasak attempted to reproduce w/ a clean quantal install + apt-get install maas juju, but that did not demonstrate the problem.

See original description

Related branches

lp:ubuntu/quantal/maas
Revision history for this message
dann frazier (dannf) wrote :

Oh - and my maas packages are all versioned 0.1+bzr1264+dfsg-0ubuntu1. dpkg.log contains no reference to older versions (e.g., I do not see evidence that it installed one version than upgraded to the latest).

Revision history for this message
Robie Basak (racb) wrote :

It sounds like postinsts are assuming to be run in a particular order but package dependencies provide no guarantee of this. This is just a guess.

Andres Rodriguez (andreserl)
Changed in maas:
status: New → Confirmed
importance: Undecided → High
assignee: nobody → Andres Rodriguez (andreserl)
Revision history for this message
Andres Rodriguez (andreserl) wrote :

The reason of the bug is because that the RabbitMQ user is not created during CD install.

In order for maas to create the user on installation, it needs RabbitMQ running. RabbitMQ is started and successfully creates the credentials for longpoll, however, it seems that it then gets killed and the credentials for celery worker are not created.

affects: maas → maas (Ubuntu)
Andres Rodriguez (andreserl)
tags: added: release-p-tracking
tags: added: rls-p-tracking
removed: release-p-tracking
dann frazier (dannf)
summary: - UI URL gives HTTP error 200 after CD install
+ UI URL displays "200 Error" page after CD install
description: updated
Andres Rodriguez (andreserl)
Changed in maas (Ubuntu):
importance: High → Critical
Revision history for this message
Ubuntu QA Website (ubuntuqa) wrote :

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1065763

tags: added: iso-testing
Andres Rodriguez (andreserl)
Changed in maas (Ubuntu):
status: Confirmed → In Progress
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package maas - 0.1+bzr1264+dfsg-0ubuntu2

---------------
maas (0.1+bzr1264+dfsg-0ubuntu2) quantal; urgency=low

  * Fix MAAS not working after CD/preseed install (LP: #1065763)
    - debian/extras/maas-region-celeryd: Add wrapper to run celery from
      the upstart job.
    - debian/extras/maas-region-controller.maas-region-celery.upstart: Create
      rabbitmq user if it does not exists.
  * debian/control: Remove unneeded deps on python-txtftp for some binary pkgs.
  * debian/maas-region-controller.postinst: Minor fix referencing variable.
 -- Andres Rodriguez <email address hidden> Thu, 11 Oct 2012 17:38:08 -0400

Changed in maas (Ubuntu):
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.