Autoscaling test failed on newly deployed cluster since 8.0

Bug #1568825 reported by Vladimir Grujic
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Won't Fix
High
Kyrylo Galanov
8.0.x
Won't Fix
High
Kyrylo Galanov
Mitaka
Invalid
High
Kyrylo Galanov

Bug Description

Created a new environment on freshly deployed fuel 8.0.
When running health checks, auto scaling test is failing when creating cpu_alarm_low resource. The error is Forbidden: You are not authorized to perform the requested action. (HTTP 403)

diagnostic snapshot is at :
http://owncloud.cloudex.rs/public.php?service=files&t=e4add1b91373e85fb7b57a954515dd2b

Tags: area-library
Changed in fuel:
status: New → Confirmed
importance: Undecided → High
assignee: nobody → Fuel Library Team (fuel-library)
milestone: none → 8.0-updates
tags: added: area-library
Revision history for this message
Sergey Kolekonov (skolekonov) wrote :

Here's the diagnostic snapshot uploaded to another storage
https://drive.google.com/file/d/0B4uE8p8Gea5QeGc4bHRlbjhaQVE

Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Matthew Mosesohn (raytrac3r)
Changed in fuel:
assignee: Matthew Mosesohn (raytrac3r) → Kyrylo Galanov (kgalanov)
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
milestone: 8.0-updates → 10.0
summary: - Autoscaling test failed on newly deployed 8.0
+ Autoscaling test failed on newly deployed cluster since 8.0
Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
Kyrylo Galanov (kgalanov) wrote :
Revision history for this message
Kyrylo Galanov (kgalanov) wrote :

9.0 - could not test - https://custom-ci.infra.mirantis.net/view/9.0/job/9.0.custom.system_test/195/
  - Check stack autoscaling (fuel_health.tests.tests_platform.test_heat.HeatSmokeTests.test_autoscaling). Test status: skipped, message: This test requires more hardware resources of your OpenStack cluster: your cloud should allow to create at least 1 VM with 7000 MB of RAM, 40 HDD and 2 vCPUs. You need to remove some resources or add compute nodes to have an ability to run this OSTF test.

Revision history for this message
Kyrylo Galanov (kgalanov) wrote :
Changed in fuel:
status: In Progress → Confirmed
Revision history for this message
Kyrylo Galanov (kgalanov) wrote :
Revision history for this message
magicboiz (magicboiz) wrote :
Download full text (7.0 KiB)

Hi

I've deployed MOS 8.0 and fuel autoscaling test (also https://github.com/openstack/heat-templates/blob/master/hot/autoscaling.yaml) doesn't work.

I get

Check stack autoscaling
Failed to get to expected status. Currently in CREATE_FAILED status. Please refer to OpenStack logs for more details.
Target component: Heat

Scenario:
1. Create test flavor.
2. Create a keypair.
3. Save generated private key to file on Controller node.
4. Create a security group.
5. Create a stack.
6. Wait for the stack status to change to 'CREATE_COMPLETE'.
7. Create a floating IP.
8. Assign the floating IP to the instance of the stack.
9. Wait when the instance is ready to connect.
10. Wait for the 2nd instance to be launched.
11. Wait for the 2nd instance to be terminated.
12. Delete the file with private key.
13. Delete the stack.
14. Wait for the stack to be deleted.

and checking heat logs, I can see this problem: ERROR heat.engine.resource Forbidden: You are not authorized to perform the requested action.

Complete log heat-all.log:
...
<134>May 10 10:03:08 node-4 heat-engine: 2016-05-10 10:03:08.572 3513 INFO heat.engine.resource [-] CREATE: CeilometerAlarm "cpu_alarm_low" Stack "ost1_test-heat-stack-2108765664" [9f674d01-eccc-4cf0-8f0b-1345ae60777b]
2016-05-10 10:03:08.572 3513 ERROR heat.engine.resource Traceback (most recent call last):
2016-05-10 10:03:08.572 3513 ERROR heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 640, in _action_recorder
2016-05-10 10:03:08.572 3513 ERROR heat.engine.resource yield
2016-05-10 10:03:08.572 3513 ERROR heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 710, in _do_action
2016-05-10 10:03:08.572 3513 ERROR heat.engine.resource yield self.action_handler_task(action, args=handler_args)
2016-05-10 10:03:08.572 3513 ERROR heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/scheduler.py", line 306, in wrapper
2016-05-10 10:03:08.572 3513 ERROR heat.engine.resource step = next(subtask)
2016-05-10 10:03:08.572 3513 ERROR heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 681, in action_handler_task
2016-05-10 10:03:08.572 3513 ERROR heat.engine.resource handler_data = handler(*args)
2016-05-10 10:03:08.572 3513 ERROR heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/resources/openstack/ceilometer/alarm.py", line 335, in handle_create
2016-05-10 10:03:08.572 3513 ERROR heat.engine.resource alarm = self.client().alarms.create(**props)
2016-05-10 10:03:08.572 3513 ERROR heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 564, in client
2016-05-10 10:03:08.572 3513 ERROR heat.engine.resource return self.stack.clients.client(client_name)
2016-05-10 10:03:08.572 3513 ERROR heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/clients/__init__.py", line 70, in client
2016-05-10 10:03:08.572 3513 ERROR heat.engine.resource return client_plugin.client()
2016-05-10 10:03:08.572 3513 ERROR heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/clients/client...

Read more...

Revision history for this message
Kyrylo Galanov (kgalanov) wrote :

Hi magicboiz,

Please report your issue properly as a separate bug.

Thanks,
Kyrylo

Changed in fuel:
status: Confirmed → Incomplete
Revision history for this message
magicboiz (magicboiz) wrote :

Hi Kyrylo

I don't think this bug should be tagged as duplicate of #1576520.
In my case, it looks like there are some problems with rights/privileges needed to create the alarm (The error is Forbidden: You are not authorized to perform the requested action).
On the other hand, in bug #1576520 the alarms are created (but with a http/https problem).

Thanks.

Revision history for this message
Dmitrii (dsivaev) wrote :

Greetings everyone!

Got the same error then trying to validate my cloud using OSTF. I'm using MOS 8.0+MU2, TLS is disabled (so this problem isn't like described in bug #1576520).

Part of the heat-all.log file from OSTF test "Check stack autoscaling" run is in attachment.

Changed in fuel:
status: Incomplete → In Progress
Revision history for this message
Kyrylo Galanov (kgalanov) wrote :

The fix was introduced in commit https://github.com/openstack/puppet-heat/commit/48b80077e83203c2dc40894e3adc6f3bd1ac170d

Unfortunately, it was not included in 8.0 Release ISO, however it was pushed to updates.

$ git tag --contains 48b80077e83203c2dc40894e3adc6f3bd1ac170d
8.0.0
8.1.0
8.2.0
9.0.0
9.1.0

Please install the latest updates (https://docs.mirantis.com/openstack/fuel/fuel-8.0/maintenance-updates.html#mu8-0-how-to-update)

OR

fix /etc/heat/heat.conf manually by adding in [trustee] section (http://docs.openstack.org/liberty/install-guide-rdo/heat-install.html#install-and-configure-components)

Changed in fuel:
status: In Progress → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.