VIM creation is in PENDING state

[Amit Kumar]

setup : Ubuntu 18.04.5 LTS (GNU/Linux 4.15.0-112-generic x86_64)
release stable/ussri
installed via devstack

local.conf file
------------------------------------------------------------------------------
[[local|localrc]]
HOST_IP=127.0.0.1
SERVICE_HOST=127.0.0.1
SERVICE_PASSWORD=devstack
ADMIN_PASSWORD=devstack
SERVICE_TOKEN=devstack
DATABASE_PASSWORD=root
RABBIT_PASSWORD=rabbitmq
ENABLE_HTTPD_MOD_WSGI_SERVICES=True
KEYSTONE_USE_MOD_WSGI=True

# Logging
LOGFILE=$DEST/logs/stack.sh.log
VERBOSE=True
ENABLE_DEBUG_LOG_LEVEL=True
ENABLE_VERBOSE_LOG_LEVEL=True
GIT_BASE=${GIT_BASE:-https://opendev.org}

TACKER_MODE=standalone
USE_BARBICAN=True
#TACKER_BRANCH="stable/ussuri"
enable_plugin networking-sfc ${GIT_BASE}/openstack/networking-sfc stable/ussuri
enable_plugin barbican ${GIT_BASE}/openstack/barbican stable/ussuri
enable_plugin mistral ${GIT_BASE}/openstack/mistral stable/ussuri
enable_plugin tacker ${GIT_BASE}/openstack/tacker stable/ussuri
----------------------------------------------------------------------------------

Setup completed but getting issue during vim registration
vim registration state is in pending

---------------------------------------------------------------------------------
+----------------+-------------------------------------------------+
| Field | Value |
+----------------+-------------------------------------------------+
| auth_cred | { |
| | "username": "admin", |
| | "user_domain_name": "Default", |
| | "cert_verify": "True", |
| | "project_id": null, |
| | "project_name": "admin", |
| | "project_domain_name": "Default", |
| | "auth_url": "http://192.168.32.28:5000/v3", |
| | "key_type": "barbican_key", |
| | "secret_uuid": "***", |
| | "password": "***" |
| | } |
| auth_url | http://192.168.32.28:5000/v3 |
| created_at | 2020-08-21 16:27:36.891541 |
| description | |
| id | 93d7f0e4-2a6a-4755-8e9f-ec951f209c6a |
| is_default | True |
| name | testvim |
| placement_attr | { |
| | "regions": [ |
| | "RegionOne" |
| | ] |
| | } |
| project_id | 13e2f3ff0b9c4dbaa35451d353c3ab06 |
| status | PENDING |
| type | openstack |
| updated_at | None |
| vim_project | { |
| | "name": "admin", |
| | "project_domain_name": "Default" |
| | } |
+----------------+-------------------------------------------------+
--------------------------------------------------------------------------------

getting below error in tacker-conductor.log

2020-08-21 16:31:39.800 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 111] ECONNREFUSED (retrying in 32.0 seconds): ConnectionRefusedError: [Errno 111] ECONNREFUSED

similar error in mistral-api.log
2020-08-21 16:40:28.558655 Connection failed: [Errno 111] Connection refused (retrying in 32.0 seconds)
2020-08-21 16:41:00.608540 Connection failed: [Errno 111] Connection refused (retrying in 32.0 seconds)

tacker-server.log

---------------------------------------------------------------------------------
2020-08-21 16:27:36.969 DEBUG keystoneclient.session [req-2dddd758-7ea2-4c0d-bf50-e166447a49da demo demo] RESP: [200] Date: Fri, 21 Aug 2020 16:27:36 GMT Server: Apache/2.4.29 (Ubuntu) Content-Type: application/json Content-Length: 253 Vary: X-Auth-Token x-openstack-request-id: req-d46d3bab-2408-4125-99b0-2c9b13ba81b7 Connection: close
RESP BODY: {"version": {"id": "v3.14", "status": "stable", "updated": "2020-04-07T00:00:00Z", "links": [{"rel": "self", "href": "http://127.0.0.1/identity/v3/"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}]}}
 from (pid=1107) _http_log_response /usr/local/lib/python3.6/dist-packages/keystoneclient/session.py:255
2020-08-21 16:27:37.024 DEBUG tacker.nfvo.workflows.vim_monitor.vim_monitor_utils [req-2dddd758-7ea2-4c0d-bf50-e166447a49da demo demo] vim monitor workflow: version: '2.0'
vim_id_93d7f0e4-2a6a-4755-8e9f-ec951f209c6a:
  input:
  - ping_vim
  tasks:
    monitor_ping_vimPingVIMTASK:
      action: tacker.vim_ping_action
      input:
        count: '1'
        interval: '1'
        targetip: 192.168.32.28
        timeout: '1'
        vim_id: 93d7f0e4-2a6a-4755-8e9f-ec951f209c6a
  type: direct
 from (pid=1107) prepare_and_create_workflow /opt/stack/tacker/tacker/nfvo/workflows/vim_monitor/vim_monitor_utils.py:39
2020-08-21 16:27:44.594 DEBUG mistralclient.api.httpclient [req-2dddd758-7ea2-4c0d-bf50-e166447a49da demo demo] HTTP POST http://127.0.0.1:8989/v2/workflows?scope=private&namespace= 201 from (pid=1107) decorator /usr/local/lib/python3.6/dist-packages/mistralclient/api/httpclient.py:56
2020-08-21 16:32:44.676 DEBUG mistralclient.api.httpclient [req-2dddd758-7ea2-4c0d-bf50-e166447a49da demo demo] HTTP POST http://127.0.0.1:8989/v2/executions 504 from (pid=1107) decorator /usr/local/lib/python3.6/dist-packages/mistralclient/api/httpclient.py:56
2020-08-21 16:32:44.676 WARNING tacker.nfvo.nfvo_plugin [req-2dddd758-7ea2-4c0d-bf50-e166447a49da demo demo] Failed to set up vim monitoring: mistralclient.api.base.APIException: b'<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>504 Gateway Timeout</title>\n</head><body>\n<h1>Gateway Timeout</h1>\n<p>The gateway did not receive a timely response\nfrom the upstream server or application.</p>\n<hr>\n<address>Apache/2.4.29 (Ubuntu) Server at 127.0.0.1 Port 8989</address>\n</body></html>\n'
2020-08-21 16:32:44.682 INFO tacker.wsgi [req-2dddd758-7ea2-4c0d-bf50-e166447a49da demo demo] 127.0.0.1 - - [21/Aug/2020 16:32:44] "POST /v1.0/vims.json HTTP/1.1" 201 912 310.305382
------------------------------------------------------------------------------------

[Amit Kumar (amit-netwk)]

Hi,

During debugging I figured out that it is only issue with the "vim_ping_action" and VIM is added in tacker, I am even able to instantiate VNF to VIM.

the issue that need attention is as below:

mistral.exceptions.InvalidActionException: Failed to find action [action_name=tacker.vim_ping_action] in [namespace=]

Failed to run task [error=Failed to find action [action_name=tacker.vim_ping_action] in [namespace=], wf=vim_id_f829c238-5f61-48c8-a627-b5fdc0fdba70, task=monitor_ping_vimPingVIMTASK]: |
| | Traceback (most recent call last): |
| | File "/opt/stack/mistral/mistral/engine/task_handler.py", line 68, in run_task |
| | task.run() |
| | File "/usr/local/lib/python3.6/dist-packages/osprofiler/profiler.py", line 160, in wrapper |
| | result = f(*args, **kwargs) |
| | File "/opt/stack/mistral/mistral/engine/tasks.py", line 508, in run |
| | self._run_new() |
| | File "/usr/local/lib/python3.6/dist-packages/osprofiler/profiler.py", line 160, in wrapper |
| | result = f(*args, **kwargs) |
| | File "/opt/stack/mistral/mistral/engine/tasks.py", line 540, in _run_new |
| | self._schedule_actions() |
| | File "/opt/stack/mistral/mistral/engine/tasks.py", line 616, in _schedule_actions |
| | action = self._build_action() ...

[Manpreet Kaur (manpreetk)]

Hi Amit,

Could you please share tacker configuration and mistral configuration files.
Additionally please share rabbitmq logs.
Please find path:
RabbitMQ: /var/log/rabbitmq/rabbit@<NODE NAME>.log
Tacker configuration file: /etc/tacker/tacker.conf
Mistral configuration file: /etc/mistral/mistral.conf

[Amit Kumar (amit-netwk)]

Hi Manpreet,

I figured out the root cause, actually during populating mistral actions and workflow in db

mistral-db-manage --config-file /etc/mistral/mistral.conf populate

the tacker.vim_ping_action was not able to populate into the db because 'base' class wasn't found in mistral.
the class mistral.actions.base.Action is moved to mistral_lib.actions.Action.

So I had to modify the /opt/stack/tacker/tacker/nfvo/workflows/vim_monitor/vim_ping_action.py file with below update:

--------------------------------------------------------
import netaddr

from mistral_lib import actions ---> #1
from oslo_config import cfg
from oslo_log import log as logging

LOG = logging.getLogger(__name__)

class PingVimAction(actions.Action): ----> #2

    def __init__(self, count, targetip, vim_id,
                 interval, timeout):
 def _update(self, status):
                          vim_id=self.vim_id,
                          status=status)

    def run(self, action_ctx): ----> #3
        servers = []
        try:
            rpc.init_action_rpc(cfg.CONF)

--------------------------------------------------------

After making this change, I am able to populate mistral DB with tacker.vim_ping_action and everything is working fine now.

[Manpreet Kaur (manpreetk)]

Thanks amit for sharing your finding, this issue has been address in victoria release.
Please find details in refer bug detail,
https://bugs.launchpad.net/tacker/+bug/1874007

[Manpreet Kaur (manpreetk)]

This is a backport bug, as mention in #4, code changes need to merge in ussuri release.

[Yasufumi Ogawa (yasufum)]

Hi Manpreet,
Could I confirm the current status? If it's already merged, please tell me URL of the change on gerrit. Thanks.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/tacker/+/803083

[Manpreet Kaur (manpreetk)]

Hi Yasufumi Ogawa san,

Sorry for the late response on bug closure.

The solution of the bug was merge in the Victoria release, please refer,
https://review.opendev.org/c/openstack/tacker/+/722209

I have cherry-picked changes to the stable/ussuri branch, please find the URL,
https://review.opendev.org/c/openstack/tacker/+/803083

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tacker/+/803083
Committed: https://opendev.org/openstack/tacker/commit/de9fdf5fc6c7c04a77442f6bc7e39331f55578f0
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit de9fdf5fc6c7c04a77442f6bc7e39331f55578f0
Author: Quang Hiep <email address hidden>
Date: Thu Apr 23 06:07:21 2020 +0000

    Fix create custom Mistral action

    Class mistral.actions.base.Action is moved to
    mistral_lib.actions.Action accoring to
    https://docs.openstack.org/mistral/latest/developer/extensions/creating_custom_action.html

    This change imports the new action class and
    add context parameter for the run method.

    Closes-Bug: #1892528
    Change-Id: I7858f69ca051c64a8ac52e6bfde353d55a42a43e
    (cherry picked from commit b38ed56020ee403e11ef60e3ebd35ae0fca9b868)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tacker ussuri-eol release.