Referencing a lb-healthmonitor ID for the first time from Heat would fail

Bug #1439696 reported by Amit Ugol on 2015-04-02
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
octavia
Incomplete
Low
Doug Wiegley

Bug Description

Creating a stack with heat that creates a lb-healthmonitor would result in a 404 for that ID.
This happens only at the first attempt to do so. Deleting the heat stack and recreating it using the same template would result in a success so it does not look like an issue originating from heat. Later operations either by neutron or Heat would succeed and the only way to reproduce this specific issue is to unstack and re-stack.

From heat's log (has neutron's answer):

REQ: curl -i http://10.35.160.83:9696//v2.0/lb/health_monitors.json -X POST -H "User-Agent: python-neutronclient" -H "X-Auth-Token: 40357276a5b34f1bb4980d566d36e9c4" -d '{"health_monitor": {"delay": 5, "max_retr
 from (pid=10195) http_log_req /usr/lib/python2.7/site-packages/neutronclient/common/utils.py:130
2015-04-02 15:24:19.791 DEBUG neutronclient.client [-] RESP:404 {'date': 'Thu, 02 Apr 2015 12:24:19 GMT', 'connection': 'keep-alive', 'content-type': 'text/plain; cha

The resource could not be found.

 from (pid=10195) http_log_resp /usr/lib/python2.7/site-packages/neutronclient/common/utils.py:139
2015-04-02 15:24:19.791 DEBUG neutronclient.v2_0.client [-] Error message: 404 Not Found

The resource could not be found.

    from (pid=10195) _handle_fault_response /usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py:173
2015-04-02 15:24:19.792 INFO heat.engine.resource [-] CREATE: HealthMonitor "monitor" Stack "test-001-load_balancer-ukmrf56u2dm4" [7aab3fa0-b71d-47b3-acc5-4767cb23b99
2015-04-02 15:24:19.792 TRACE heat.engine.resource Traceback (most recent call last):
2015-04-02 15:24:19.792 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/resource.py", line 466, in _action_recorder
2015-04-02 15:24:19.792 TRACE heat.engine.resource yield
2015-04-02 15:24:19.792 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/resource.py", line 536, in _do_action
2015-04-02 15:24:19.792 TRACE heat.engine.resource yield self.action_handler_task(action, args=handler_args)
2015-04-02 15:24:19.792 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/scheduler.py", line 295, in wrapper
2015-04-02 15:24:19.792 TRACE heat.engine.resource step = next(subtask)
2015-04-02 15:24:19.792 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/resource.py", line 507, in action_handler_task
2015-04-02 15:24:19.792 TRACE heat.engine.resource handler_data = handler(*args)
2015-04-02 15:24:19.792 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/resources/neutron/loadbalancer.py", line 146, in handle_create
2015-04-02 15:24:19.792 TRACE heat.engine.resource {'health_monitor': properties})['health_monitor']
2015-04-02 15:24:19.792 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 99, in with_params
2015-04-02 15:24:19.792 TRACE heat.engine.resource ret = self.function(instance, *args, **kwargs)
2015-04-02 15:24:19.792 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1064, in create_health_monitor
2015-04-02 15:24:19.792 TRACE heat.engine.resource return self.post(self.health_monitors_path, body=body)
2015-04-02 15:24:19.792 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 295, in post
2015-04-02 15:24:19.792 TRACE heat.engine.resource headers=headers, params=params)
2015-04-02 15:24:19.792 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 208, in do_request
2015-04-02 15:24:19.792 TRACE heat.engine.resource self._handle_fault_response(status_code, replybody)
2015-04-02 15:24:19.792 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 182, in _handle_fault_response
2015-04-02 15:24:19.792 TRACE heat.engine.resource exception_handler_v20(status_code, des_error_body)
2015-04-02 15:24:19.792 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 80, in exception_handler_v20
2015-04-02 15:24:19.792 TRACE heat.engine.resource message=message)
2015-04-02 15:24:19.792 TRACE heat.engine.resource NeutronClientException: 404 Not Found
2015-04-02 15:24:19.792 TRACE heat.engine.resource
2015-04-02 15:24:19.792 TRACE heat.engine.resource The resource could not be found.

Amit Ugol (amit-ugol) wrote :

On further note, Just running neutron --debug lb-healthmonitor-list would do the same thing

summary: - Referencing a network ID for the first time from Heat would fail
+ Referencing a lb-healthmonitor ID for the first time from Heat would
+ fail
description: updated
Vikram Hosakote (vhosakot) wrote :

After unstacking and re-stacking, when the heat template is run for the first time, does it start the neutron lbaas agent ?

I have seen the 404 error (The resource could not be found) for "neutron lb-healthmonitor-create" when the neutron lbaas agent is not running.

Refer https://wiki.openstack.org/wiki/Neutron/LBaaS/HowToRun to setup and run the lbaas agent. Ideally, the heat template should do this.

I can run the commands successfully when the neutron lbaas agent is running.

# ps aux | grep lb
neutron 24794 0.0 0.0 276740 31152 ? Ss Apr01 0:20 /usr/bin/python /usr/bin/neutron-lbaas-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/lbaas_agent.ini --log-file /var/log/neutron/lbaas-agent.log

# neutron lb-healthmonitor-create --delay 3 --type HTTP --max-retries 3 --timeout 3
Created a new health_monitor:
+----------------+--------------------------------------+
| Field | Value |
+----------------+--------------------------------------+
| admin_state_up | True |
| delay | 3 |
| expected_codes | 200 |
| http_method | GET |
| id | 771eca34-42c1-44e9-a96c-56de7afd2b7e |
| max_retries | 3 |
| pools | |
| tenant_id | ab81207b56754cafac46f183dc86529e |
| timeout | 3 |
| type | HTTP |
| url_path | / |
+----------------+--------------------------------------+

# neutron lb-healthmonitor-list
+--------------------------------------+------+----------------+
| id | type | admin_state_up |
+--------------------------------------+------+----------------+
| 771eca34-42c1-44e9-a96c-56de7afd2b7e | HTTP | True |
+--------------------------------------+------+----------------+

tags: added: api lbaas
Eugene Nikanorov (enikanorov) wrote :

I'm moving this to confirmed (even though i didn't checked it myself)

It seems like a severe issue, API availability should not depend on whether the agent is running.
However I might be missing some important design decision.

Changed in neutron:
importance: Undecided → High
status: New → Confirmed
Doug Wiegley (dougwig) on 2015-04-29
tags: added: lbaasv1
Andrea (andreagardiman) wrote :

The bug affect me too. Moreover, the problem happens to me not only the first time, but every time I create the stack or I execute neutron --debug lb-healthmonitor-list.

Kyle Mestery (mestery) wrote :

Assigning to Doug to have the LBaaS team triage this one more.

Changed in neutron:
assignee: nobody → Doug Wiegley (dougwig)
Kyle Mestery (mestery) on 2015-09-14
Changed in neutron:
importance: High → Medium
Changed in neutron:
importance: Medium → Low

Can we confirm that this does not affect also lbaas v2?

Changed in neutron:
milestone: none → mitaka-1

I mean assuming that Heat would try to do the same thing but using the v2 API instead?

Changed in neutron:
milestone: mitaka-1 → mitaka-2
Changed in neutron:
milestone: mitaka-2 → mitaka-3
Changed in neutron:
milestone: mitaka-3 → mitaka-rc1

No low bugs in RC1 window

Changed in neutron:
milestone: mitaka-rc1 → newton-1
Changed in neutron:
milestone: newton-1 → newton-2
Changed in neutron:
milestone: newton-2 → newton-3
Changed in neutron:
milestone: newton-3 → newton-rc1
Changed in neutron:
milestone: newton-rc1 → ocata-1
Changed in neutron:
milestone: ocata-1 → ocata-2
Michael Johnson (johnsom) wrote :

Can we confirm this is an issue with LBaaSv2 and it is still occurring?
If so, what OpenStack release is being used?

affects: neutron → octavia
Changed in octavia:
milestone: ocata-2 → none
status: Confirmed → Incomplete
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers