Referencing a lb-healthmonitor ID for the first time from Heat would fail

[Removed by request]

Creating a stack with heat that creates a lb-healthmonitor would result in a 404 for that ID.
This happens only at the first attempt to do so. Deleting the heat stack and recreating it using the same template would result in a success so it does not look like an issue originating from heat. Later operations either by neutron or Heat would succeed and the only way to reproduce this specific issue is to unstack and re-stack.

From heat's log (has neutron's answer):

REQ: curl -i http://10.35.160.83:9696//v2.0/lb/health_monitors.json -X POST -H "User-Agent: python-neutronclient" -H "X-Auth-Token: 40357276a5b34f1bb4980d566d36e9c4" -d '{"health_monitor": {"delay": 5, "max_retr
 from (pid=10195) http_log_req /usr/lib/python2.7/site-packages/neutronclient/common/utils.py:130
2015-04-02 15:24:19.791 DEBUG neutronclient.client [-] RESP:404 {'date': 'Thu, 02 Apr 2015 12:24:19 GMT', 'connection': 'keep-alive', 'content-type': 'text/plain; cha

The resource could not be found.

 from (pid=10195) http_log_resp /usr/lib/python2.7/site-packages/neutronclient/common/utils.py:139
2015-04-02 15:24:19.791 DEBUG neutronclient.v2_0.client [-] Error message: 404 Not Found

The resource could not be found.

    from (pid=10195) _handle_fault_response /usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py:173
2015-04-02 15:24:19.792 INFO heat.engine.resource [-] CREATE: HealthMonitor "monitor" Stack "test-001-load_balancer-ukmrf56u2dm4" [7aab3fa0-b71d-47b3-acc5-4767cb23b99
2015-04-02 15:24:19.792 TRACE heat.engine.resource Traceback (most recent call last):
2015-04-02 15:24:19.792 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/resource.py", line 466, in _action_recorder
2015-04-02 15:24:19.792 TRACE heat.engine.resource yield
2015-04-02 15:24:19.792 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/resource.py", line 536, in _do_action
2015-04-02 15:24:19.792 TRACE heat.engine.resource yield self.action_handler_task(action, args=handler_args)
2015-04-02 15:24:19.792 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/scheduler.py", line 295, in wrapper
2015-04-02 15:24:19.792 TRACE heat.engine.resource step = next(subtask)
2015-04-02 15:24:19.792 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/resource.py", line 507, in action_handler_task
2015-04-02 15:24:19.792 TRACE heat.engine.resource handler_data = handler(*args)
2015-04-02 15:24:19.792 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/resources/neutron/loadbalancer.py", line 146, in handle_create
2015-04-02 15:24:19.792 TRACE heat.engine.resource {'health_monitor': properties})['health_monitor']
2015-04-02 15:24:19.792 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 99, in with_params
2015-04-02 15:24:19.792 TRACE heat.engine.resource ret = self.function(instance, *args, **kwargs)
2015-04-02 15:24:19.792 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1064, in create_health_monitor
2015-04-02 15:24:19.792 TRACE heat.engine.resource return self.post(self.health_monitors_path, body=body)
2015-04-02 15:24:19.792 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 295, in post
2015-04-02 15:24:19.792 TRACE heat.engine.resource headers=headers, params=params)
2015-04-02 15:24:19.792 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 208, in do_request
2015-04-02 15:24:19.792 TRACE heat.engine.resource self._handle_fault_response(status_code, replybody)
2015-04-02 15:24:19.792 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 182, in _handle_fault_response
2015-04-02 15:24:19.792 TRACE heat.engine.resource exception_handler_v20(status_code, des_error_body)
2015-04-02 15:24:19.792 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 80, in exception_handler_v20
2015-04-02 15:24:19.792 TRACE heat.engine.resource message=message)
2015-04-02 15:24:19.792 TRACE heat.engine.resource NeutronClientException: 404 Not Found
2015-04-02 15:24:19.792 TRACE heat.engine.resource
2015-04-02 15:24:19.792 TRACE heat.engine.resource The resource could not be found.

[Removed by request (removed3381998)]

On further note, Just running neutron --debug lb-healthmonitor-list would do the same thing

[Vikram Hosakote (vhosakot)]

After unstacking and re-stacking, when the heat template is run for the first time, does it start the neutron lbaas agent ?

I have seen the 404 error (The resource could not be found) for "neutron lb-healthmonitor-create" when the neutron lbaas agent is not running.

Refer https://wiki.openstack.org/wiki/Neutron/LBaaS/HowToRun to setup and run the lbaas agent. Ideally, the heat template should do this.

I can run the commands successfully when the neutron lbaas agent is running.

# ps aux | grep lb
neutron 24794 0.0 0.0 276740 31152 ? Ss Apr01 0:20 /usr/bin/python /usr/bin/neutron-lbaas-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/lbaas_agent.ini --log-file /var/log/neutron/lbaas-agent.log

# neutron lb-healthmonitor-create --delay 3 --type HTTP --max-retries 3 --timeout 3
Created a new health_monitor:
+----------------+--------------------------------------+
| Field | Value |
+----------------+--------------------------------------+
| admin_state_up | True |
| delay | 3 |
| expected_codes | 200 |
| http_method | GET |
| id | 771eca34-42c1-44e9-a96c-56de7afd2b7e |
| max_retries | 3 |
| pools | |
| tenant_id | ab81207b56754cafac46f183dc86529e |
| timeout | 3 |
| type | HTTP |
| url_path | / |
+----------------+--------------------------------------+

# neutron lb-healthmonitor-list
+--------------------------------------+------+----------------+
| id | type | admin_state_up |
+--------------------------------------+------+----------------+
| 771eca34-42c1-44e9-a96c-56de7afd2b7e | HTTP | True |
+--------------------------------------+------+----------------+

[Eugene Nikanorov (enikanorov)]

I'm moving this to confirmed (even though i didn't checked it myself)

It seems like a severe issue, API availability should not depend on whether the agent is running.
However I might be missing some important design decision.

[Andrea (andreagardiman)]

The bug affect me too. Moreover, the problem happens to me not only the first time, but every time I create the stack or I execute neutron --debug lb-healthmonitor-list.

[Kyle Mestery (mestery)]

Assigning to Doug to have the LBaaS team triage this one more.

[Armando Migliaccio (armando-migliaccio)]

Can we confirm that this does not affect also lbaas v2?

[Armando Migliaccio (armando-migliaccio)]

I mean assuming that Heat would try to do the same thing but using the v2 API instead?

[Armando Migliaccio (armando-migliaccio)]

No low bugs in RC1 window

[Michael Johnson (johnsom)]

Can we confirm this is an issue with LBaaSv2 and it is still occurring?
If so, what OpenStack release is being used?

[Gregory Thiemonge (gthiemonge)]

Abandoned after re-enabling the Octavia launchpad.