Mirantis OpenStack

Heat-engine fails to start during deployment due to incorrect version of oslo.utils python package

Bug #1449007 reported by Oleksiy Butenko
18
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Confirmed
High
Oleksiy Butenko
Mirantis OpenStack 6.1

Bug Description

{"build_id": "2015-04-26_22-54-38", "build_number": "352", "release_versions": {"2014.2.2-6.1": {"VERSION": {"build_id": "2015-04-26_22-54-38", "build_number": "352", "api": "1.0", "fuel-library_sha": "6bdf783e2bffdce80ecffcca2915e6d32a8ccdd7", "nailgun_sha": "f9f17fe395f3e38c27d3d484302728a37ad8012d", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "2b311b3b82a1e2df1dc3484a0f37e282273cd988", "astute_sha": "c1793f982fda7e3fc7b937ccaa613c649be6a144", "fuel-ostf_sha": "b38602c841deaa03ddffc95c02f319360462cbe3", "release": "6.1", "fuelmain_sha": "01288380950bc89d572cf5902141c9a393ada950"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "6bdf783e2bffdce80ecffcca2915e6d32a8ccdd7", "nailgun_sha": "f9f17fe395f3e38c27d3d484302728a37ad8012d", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "2b311b3b82a1e2df1dc3484a0f37e282273cd988", "astute_sha": "c1793f982fda7e3fc7b937ccaa613c649be6a144", "fuel-ostf_sha": "b38602c841deaa03ddffc95c02f319360462cbe3", "release": "6.1", "fuelmain_sha": "01288380950bc89d572cf5902141c9a393ada950"}

Ubuntu HA, 3 controller, 1 compute, 1 cinder, Neutron with VLAN

OSTF - Platform services functional tests:
Typical stack actions: create, delete, show details, etc.
Advanced stack actions: suspend, resume and check.
Check stack rollback
Update stack actions: inplace, replace and update whole template.
All this tests failed on first step - Create stack

In Horizon:
Navigate: Project->Orchestration->Stacks

Error: Unable to retrieve stack list.

On Controller:

heat stack-list
ERROR: <html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>

cat /var/log/heat-all.log |grep ERROR
<131>Apr 27 12:06:16 node-1 heat-api 2015-04-27 12:06:16.059 31999 ERROR root [req-052cf5ba-5f91-447b-be75-c60d2b412bf0 ] Unexpected error occurred serving API: Timed out waiting for a reply to message ID 9a97215e3db64011ada46daabc6beea8
<131>Apr 27 12:12:34 node-1 heat-api 2015-04-27 12:12:34.144 31999 ERROR root [req-b120e1ba-1b95-447b-9fdd-0ade19f30047 ] Unexpected error occurred serving API: Timed out waiting for a reply to message ID 55863e72173b43e89c34d5e3efdbd566
<131>Apr 27 12:13:34 node-1 heat-api 2015-04-27 12:13:34.727 31999 ERROR root [req-58f24ac4-13c1-4387-a80c-b63c25ffc1ad ] Unexpected error occurred serving API: Timed out waiting for a reply to message ID 13241087fc834de39571d49be23a8733
<131>Apr 27 12:13:35 node-1 heat-api 2015-04-27 12:13:35.158 31999 ERROR root [req-a0488f50-91d1-4ab6-9d3d-ccfe9ce51e77 ] Unexpected error occurred serving API: Timed out waiting for a reply to message ID 808f2871872e4c19a785ea172f241517

Revision history for this message
Oleksiy Butenko (obutenko) wrote :
Changed in mos:
importance: Undecided → High
Sergey Kraynev (skraynev)
Changed in mos:
milestone: none → 6.1
assignee: nobody → MOS Heat (mos-heat)
Revision history for this message
Sergey Kraynev (skraynev) wrote :
Download full text (3.4 KiB)

Regarding logs from heat-api it looks like issue with rabbit queue:

  15 2015-04-27 12:06:16.059 31999 TRACE root Traceback (most recent call last):
 16 2015-04-27 12:06:16.059 31999 TRACE root File "/usr/lib/python2.7/dist-packages/heat/common/wsgi.py", line 633, in __call__
 17 2015-04-27 12:06:16.059 31999 TRACE root request, **action_args)
 18 2015-04-27 12:06:16.059 31999 TRACE root File "/usr/lib/python2.7/dist-packages/heat/common/wsgi.py", line 707, in dispatch
 19 2015-04-27 12:06:16.059 31999 TRACE root return method(*args, **kwargs)
 20 2015-04-27 12:06:16.059 31999 TRACE root File "/usr/lib/python2.7/dist-packages/heat/api/openstack/v1/util.py", line 37, in handle_stack_method
 21 2015-04-27 12:06:16.059 31999 TRACE root return handler(controller, req, **kwargs)
 22 2015-04-27 12:06:16.059 31999 TRACE root File "/usr/lib/python2.7/dist-packages/heat/api/openstack/v1/stacks.py", line 291, in create
 23 2015-04-27 12:06:16.059 31999 TRACE root data.args())
 24 2015-04-27 12:06:16.059 31999 TRACE root File "/usr/lib/python2.7/dist-packages/heat/rpc/client.py", line 160, in create_stack
 25 2015-04-27 12:06:16.059 31999 TRACE root args)
 26 2015-04-27 12:06:16.059 31999 TRACE root File "/usr/lib/python2.7/dist-packages/heat/rpc/client.py", line 174, in _create_stack
 27 2015-04-27 12:06:16.059 31999 TRACE root owner_id=owner_id))
 28 2015-04-27 12:06:16.059 31999 TRACE root File "/usr/lib/python2.7/dist-packages/heat/rpc/client.py", line 50, in call
 29 2015-04-27 12:06:16.059 31999 TRACE root return client.call(ctxt, method, **kwargs)
 30 2015-04-27 12:06:16.059 31999 TRACE root File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 389, in call
 31 2015-04-27 12:06:16.059 31999 TRACE root return self.prepare().call(ctxt, method, **kwargs)
 32 2015-04-27 12:06:16.059 31999 TRACE root File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call
 33 2015-04-27 12:06:16.059 31999 TRACE root retry=self.retry)
 34 2015-04-27 12:06:16.059 31999 TRACE root File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
 35 2015-04-27 12:06:16.059 31999 TRACE root timeout=timeout, retry=retry)
 36 2015-04-27 12:06:16.059 31999 TRACE root File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 434, in send
 37 2015-04-27 12:06:16.059 31999 TRACE root retry=retry)
 38 2015-04-27 12:06:16.059 31999 TRACE root File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 423, in _send
 39 2015-04-27 12:06:16.059 31999 TRACE root result = self._waiter.wait(msg_id, timeout)
 40 2015-04-27 12:06:16.059 31999 TRACE root File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 289, in wait
 41 2015-04-27 12:06:16.059 31999 TRACE root reply, ending = self._poll_connection(msg_id, timeout)
 42 2015-04-27 12:06:16.059 31999 TRACE root File "/usr/lib/python2.7/dist-packages/oslo/messaging/_driv...

Read more...

Revision history for this message
Sergey Kraynev (skraynev) wrote :

Please check rabbit's logs on deployment. Currently I don't see any issue on Heat side.

Changed in mos:
status: New → Invalid
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

So, we don't know the exact reason of this fail, Heat team investigated the issue and found that this is not Heat issue. Ok, probably it is some issue with RabbitMQ?

Changed in mos:
status: Invalid → Incomplete
Revision history for this message
Sergey Kraynev (skraynev) wrote :

Right. So I suggest to assign this issue to another team who more familiar with Rabbit

Changed in mos:
assignee: MOS Heat (mos-heat) → nobody
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

Oleksiy, we need to investigate the root of this issue, assign it to the correct team and change status to Confirmed, if this issue will be reproduced or to 'invalid' if it is some temporary issue which doesn't reproduced.

Changed in mos:
assignee: nobody → Oleksiy Butenko (obutenko)
Revision history for this message
Oleksiy Butenko (obutenko) wrote :

Heat-engine fails to start during deployment due to incorrect version of oslo.utils python package

Revision history for this message
Oleksiy Butenko (obutenko) wrote :

Apr 28 13:45:06 node-1 heat-engine 2015-04-28 13:45:06.676 4185 CRITICAL heat.engine [-] Could not load AWSTemplateFormatVersion.2010-09-09: (oslo.utils 1.2.1 (/usr/lib/python2.7/d
ist-packages), Requirement.parse('oslo.utils==1.4.0'))

summary: - HEAT: Unexpected error occurred serving API
+ Heat-engine fails to start during deployment due to incorrect version of
+ oslo.utils python package
Changed in mos:
status: Incomplete → Confirmed
Revision history for this message
Oleksiy Butenko (obutenko) wrote :

https://review.fuel-infra.org/#/c/6248/

Revision history for this message
Sam Stoelinga (sammiestoel) wrote :

I'm on an MOS 6.1 environment and also seeing this. The root cause seems that the queue didn't get created:

=ERROR REPORT==== 20-Oct-2015::01:57:24 ===
connection <0.25905.72>, channel 1 - soft error:
{amqp_error,not_found,
            "no queue 'reply_736c2c019bf642c8a3e0df6e25db7598' in vhost '/'",
            'basic.consume'}

=ERROR REPORT==== 20-Oct-2015::01:57:25 ===
connection <0.25905.72>, channel 1 - soft error:
{amqp_error,not_found,
            "no queue 'reply_736c2c019bf642c8a3e0df6e25db7598' in vhost '/'",
            'basic.consume'}

=ERROR REPORT==== 20-Oct-2015::01:57:26 ===
connection <0.25905.72>, channel 1 - soft error:
{amqp_error,not_found,
            "no queue 'reply_736c2c019bf642c8a3e0df6e25db7598' in vhost '/'",
            'basic.consume'}

=ERROR REPORT==== 20-Oct-2015::01:57:27 ===
connection <0.25905.72>, channel 1 - soft error:
{amqp_error,not_found,
            "no queue 'reply_736c2c019bf642c8a3e0df6e25db7598' in vhost '/'",
            'basic.consume'}

Any way I can apply an hotfix on MOS 6.1?

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.