Incorrect update on "networks" parameter change

[Volodymyr Litovka]

Hi colleagues,

facing an annoying behaviour - when updating stack, deleting one from two attached ports:

  n1:
    type: OS::Nova::Server
    properties:
      flavor: B-cup
      image: xenial
      key_name: doka-key
      name: lbt-n1
      scheduler_hints: { group: { get_resource: af-group } }
      networks:
        - port: { get_resource: n1-lan }
        - port: { get_resource: n1-wan }

by either commenting or deleting row with n1-wan or n1-lan (thus leaving one port), I get FAIL with the following message:

{"badRequest": {"message": "Multiple possible networks found, use a Network ID to be more specific.", "code": 400}}

2017-10-19 13:54:42.883 24717 DEBUG heat.engine.stack [req-90c3f46e-dafe-4d9f-96b4-c17723a4d9fe - bush - default default] Persisting stack amp status UPDATE FAILED _send_notification_and_add_event /usr/lib/python2.7/dist-packages/heat/engine/stack.py:1010

in log, there is empty "interfaceAttachment" parameter in request before fail:

REQ: curl -g -i -X POST http://controller:8774/v2.1/s
ervers/658a9141-f09a-4a94-a6ea-047afc111935/os-interface -H "User-Agent: python-novaclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-OpenStack-Nova
-API-Version: 2.1" -H "X-Auth-Token: {SHA1}bb231126a1f7d5c39f902b91565e334cf6a6ffa9" -d '{"interfaceAttachment": {}}' _http_log_request /usr/lib/python2.7/dist-packages/keystonea
uth1/session.py:375

The interesting is that despite the fail:

$ openstack server list
+--------------------------------------+--------+--------+--------------------+--------+--------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+--------+--------+--------------------+--------+--------+
| ac1bf0f4-537c-4b32-bdb8-d5db4525d988 | lbt-n1 | ACTIVE | e-net=X.X.X.X | xenial | B-cup |
+--------------------------------------+--------+--------+--------------------+--------+--------+

I'm using the following versions of modules:

$ dpkg -l |egrep "nova|heat"
 heat-api 1:9.0.0-0ubuntu1~cloud0
 heat-api-cfn 1:9.0.0-0ubuntu1~cloud0
 heat-common 1:9.0.0-0ubuntu1~cloud0
 heat-engine 1:9.0.0-0ubuntu1~cloud0
 nova-api 2:16.0.0-0ubuntu2~cloud0
 nova-common 2:16.0.0-0ubuntu2~cloud0
 nova-compute 2:16.0.0-0ubuntu2~cloud0
 nova-compute-kvm 2:16.0.0-0ubuntu2~cloud0
 nova-compute-libvirt 2:16.0.0-0ubuntu2~cloud0
 nova-conductor 2:16.0.0-0ubuntu2~cloud0
 nova-consoleauth 2:16.0.0-0ubuntu2~cloud0
 nova-novncproxy 2:16.0.0-0ubuntu2~cloud0
 nova-placement-api 2:16.0.0-0ubuntu2~cloud0
 nova-scheduler 2:16.0.0-0ubuntu2~cloud0
 python-heat 1:9.0.0-0ubuntu1~cloud0
 python-heatclient 1.11.0-0ubuntu1~cloud0
 python-nova 2:16.0.0-0ubuntu2~cloud0
 python-novaclient 2:9.1.0-0ubuntu1~cloud0

on Ubuntu 16.04.3.

Is it bug or problem somewhere in my configuration?

Thanks.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/513611

[Volodymyr Litovka (doka.ua)]

Hi Rabi,

thanks for the fix, it solved the problem.

During tests I found another issue, probably related. I understand this
is a kind of synthetic situation, but can be indication of some logic
uncertainty in Heat Engine.

1) I create server using the following template:

   n1:
     type: OS::Nova::Server
     properties:
       flavor: B-cup
       image: amphora
       key_name: doka-key
       name: lbt-n1
       networks:
         - port: { get_resource: n1-lan }
         - port: { get_resource: n1-wan }
       scheduler_hints: { group: { get_resource: af-group } }

and stack creation is ok, server shows two attached interfaces which are
accessible (ping is ok on both interfaces)

2) then I delete entire "network" declaration (networks: { port: {...},
port: {...} }, what is *expected* leads to error upon update (stack
update failed) but clears ports list of server:

$ openstack stack list
+--------------------------------------+------------+---------------+----------------------+----------------------+
| ID                                   | Stack Name | Stack Status  |
Creation Time        | Updated Time         |
+--------------------------------------+------------+---------------+----------------------+----------------------+
| c61969bb-4ab7-4b5c-bef2-f6f88c9eed31 | amp        | UPDATE_FAILED |
2017-10-20T09:23:09Z | *2017-10-20T09:24:43Z* |
+--------------------------------------+------------+---------------+----------------------+----------------------+

$ openstack server list
+--------------------------------------+--------+--------+----------+---------+--------+
| ID                                   | Name   | Status | Networks |
Image   | Flavor |
+--------------------------------------+--------+--------+----------+---------+--------+
| d9bf220d-952e-4870-bcfa-f8b517227f95 | lbt-n1 | ACTIVE |          |
amphora | B-cup  |
+--------------------------------------+--------+--------+----------+---------+--------+

3) then I return back "network" declaration in the template and launch
stack update, which finishes successfully, but still no ports on server,
while they're available in stack. Take a note that stack update time is
later than update time of the server (n1) which is actually time of
previous (failed) update:

$ openstack stack list
+--------------------------------------+------------+-----------------+----------------------+----------------------+
| ID                                   | Stack Name | Stack Status    |
Creation Time        | Updated Time         |
+--------------------------------------+------------+-----------------+----------------------+----------------------+
| c61969bb-4ab7-4b5c-bef2-f6f88c9eed31 | amp        | UPDATE_COMPLETE |
2017-10-20T09:23:09Z | 2017-10-20T09:27:39Z |
+--------------------------------------+------------+-----------------+----------------------+----------------------+

$ openstack server list
+--------------------------------------+--------+--------+----------+---------+--------+
| ID                                   | Name   | Status | Networks |
Image   | Flavor |
+--------------------------------------+--------+--------+----------+---------+--------+
| d9bf...

[Rabi Mishra (rabi)]

> then I delete entire "network" declaration (networks: { port: {...},
port: {...} }, what is *expected* leads to error upon update (stack
update failed) but clears ports list of server

If there are no 'networks' for the server, heat would try to create an implicit interface on a network in the tenant. The reason it's failing here is because you've two networks and it does not know what to use.

Once you update with the old template (both interfaces) again, heat finds the FAILED server is in good state [1](ACTIVE though without any interfaces, which is probably a bug) and does not replace it. Then compares the networks properties and finds no change (create and 2nd update templates are same), hence does not do any update.

[1] https://github.com/openstack/heat/blob/master/heat/engine/resources/openstack/nova/server.py#L1280

[Volodymyr Litovka (doka.ua)]

Thanks, Rabi, for the explanation.

On 10/23/17 9:04 AM, Rabi Mishra wrote:
>> then I delete entire "network" declaration (networks: { port: {...},
> port: {...} }, what is *expected* leads to error upon update (stack
> update failed) but clears ports list of server
>
> If there are no 'networks' for the server, heat would try to create an
> implicit interface on a network in the tenant. The reason it's failing
> here is because you've two networks and it does not know what to use.
>
> Once you update with the old template (both interfaces) again, heat
> finds the FAILED server is in good state [1](ACTIVE though without any
> interfaces, which is probably a bug) and does not replace it. Then
> compares the networks properties and finds no change (create and 2nd
> update templates are same), hence does not do any update.
>
> [1]
> https://github.com/openstack/heat/blob/master/heat/engine/resources/openstack/nova/server.py#L1280
>

--
Volodymyr Litovka
   "Vision without Execution is Hallucination." -- Thomas Edison

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/513611
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=40cf08915ceb677908831e00987fddd9e2bbaf86
Submitter: Zuul
Branch: master

commit 40cf08915ceb677908831e00987fddd9e2bbaf86
Author: rabi <email address hidden>
Date: Fri Oct 20 10:21:29 2017 +0530

    Check for existing interfaces before adding implicit one

    We seem to be adding an implicit interface/port for the server
    irrespective of the fact that there are existing non-updated
    interfaces already there.

    Change-Id: Ifce2301c4d47c8462f2280955c665bdad72dd417
    Closes-Bug: #1724843

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/heat 10.0.0.0b2 development milestone.