SM provision of r4.1-5 ocata cluster failed in provision_control.py for all non-openstack nodes

[wenqing liang]

This was seen twice in half a dozen retries.

From: Nitish Krishna Kaveri Poompatnam
Sent: Monday, November 27, 2017 10:22 AM
To: Wenqing Liang <email address hidden>; Kamlesh Parmar <email address hidden>; Sachin Bansal <email address hidden>
Cc: dl-contrail-server-manager <email address hidden>
Subject: Re: r4.1-5 ocata provision failure

Hey Sachin,

Can you look into the error on this setup?

It seems like all the BGP routers were added correctly:

{
    "bgp-routers": [
        {
            "fq_name": [
                "default-domain",
                "default-project",
                "ip-fabric",
                "__default__",
                "server6"
            ],
            "href": "http://localhost:8095/bgp-router/6e52b7a0-ce10-41a3-b76f-8bfad3964b59",
            "uuid": "6e52b7a0-ce10-41a3-b76f-8bfad3964b59"
        },
        {
            "fq_name": [
                "default-domain",
                "default-project",
                "ip-fabric",
                "__default__",
                "server5"
            ],
            "href": "http://localhost:8095/bgp-router/e001fca1-42fb-4dc6-ab52-2307c7e9899d",
            "uuid": "e001fca1-42fb-4dc6-ab52-2307c7e9899d"
        },
        {
            "fq_name": [
                "default-domain",
                "default-project",
                "ip-fabric",
                "__default__",
                "server4"
            ],
            "href": "http://localhost:8095/bgp-router/ca4f0f97-1d03-43fa-b632-7bd478ece6c9",
            "uuid": "ca4f0f97-1d03-43fa-b632-7bd478ece6c9"
        }
    ]
}

And the run of provision_control.py shows no error:

root@server4(controller):/usr/share/contrail-utils# python provision_control.py --host_name server4 --host_ip 10.0.0.7 --oper add --use_admin_api --admin_user admin --admin_password c0ntrail123 --admin_tenant_name admin --router_asn 64512
BGP Router [u'default-domain',
u'default-project',
u'ip-fabric',
u'__default__',
'server4'] already exists with uuid ca4f0f97-1d03-43fa-b632-7bd478ece6c9
root@server4(controller):/usr/share/contrail-utils# echo $?
0

But according to traceback in below mail, the provision is not able to process the above result.

Can you take a look please?
Thanks,
Nitish

From: Wenqing Liang <email address hidden>
Date: Monday, November 27, 2017 at 8:00 AM
To: dl-contrail-server-manager <email address hidden>
Subject: r4.1-5 ocata provision failure

Hi,
On one full (openstack/contrail) HA 10-node setup, the latest r4.1-5 ocata provision failed in provision_control.py (see below). Can someone pls have a look? The SM is 10.87.118.54. FYI, provision did succeed on another setup albeit after 2 hrs.

+----------+---------------------+------------+--------------------------------------------------------------------------+
| id | status | ip_address | roles |
+----------+---------------------+------------+--------------------------------------------------------------------------+
| server10 | provision_failed | 10.0.0.13 | [u'contrail-compute'] |
| server3 | provision_completed | 10.0.0.6 | [u'openstack'] |
| server4 | provision_failed | 10.0.0.7 | [u'contrail-controller', u'contrail-analytics', u'contrail-analyticsdb'] |
| server7 | provision_failed | 10.0.0.10 | [u'contrail-lb'] |
| server5 | provision_failed | 10.0.0.8 | [u'contrail-controller', u'contrail-analytics', u'contrail-analyticsdb'] |
| server2 | provision_completed | 10.0.0.5 | [u'openstack'] |
| server6 | provision_failed | 10.0.0.9 | [u'contrail-controller', u'contrail-analytics', u'contrail-analyticsdb'] |
| server1 | provision_completed | 10.0.0.4 | [u'openstack'] |
| server8 | provision_failed | 10.0.0.11 | [u'contrail-compute'] |
| server9 | provision_failed | 10.0.0.12 | [u'contrail-compute'] |
+----------+---------------------+------------+--------------------------------------------------------------------------+

"2017-11-27 07:40:58,799-INFO-sm_ansible_callback.py:53-append(): fatal: [10.0.0.8]Traceback (most recent call last):
  File "/usr/share/contrail-utils/provision_control.py", line 226, in <module>
    main()
  File "/usr/share/contrail-utils/provision_control.py", line 222, in main
    ControlProvisioner(args_str)
  File "/usr/share/contrail-utils/provision_control.py", line 64, in __init__
    self._args.address_families, self._args.md5)
  File "/usr/share/contrail-utils/provision_bgp.py", line 109, in add_bgp_router
    cur_obj = vnc_lib.bgp_router_read(fq_name=fq_name)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 42, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 531, in _object_read
    res_type, fq_name, fq_name_str, id, ifmap_id)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 858, in _read_args_to_id
    return (True, self.fq_name_to_id(res_type, fq_name))
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 42, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 1117, in fq_name_to_id
    content = self._request_server(rest.OP_POST, uri, data=json_body)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 872, in _request_server
    retry_after_authn=retry_after_authn, retry_count=retry_count)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 956, in _request
    % (op, url, data, content))
cfgm_common.exceptions.NoIdError: Unknown id: Error: oper 1 url /fqname-to-id body {"fq_name": ["default-domain", "default-project", "ip-fabric", "__default__", "server4"], "type": "bgp-router"} response Name ['default-domain', 'default-project', 'ip-fabric', '__default__', 'server4'] not found"
"2017-11-27 07:40:58,800-INFO-sm_ansible_callback.py:53-append(): changed: [10.0.0.9] => (item={'key': u'10.10.0.7', 'value': u'server4'})"
"2017-11-27 07:41:00,107-INFO-sm_ansible_callback.py:53-append(): changed: [10.0.0.9] => (item={'key': u'10.10.0.9', 'value': u'server6'})"
"2017-11-27 07:41:00,145-INFO-sm_ansible_callback.py:53-append(): changed: [10.0.0.7] => (item={'key': u'10.10.0.9', 'value': u'server6'})"
"2017-11-27 07:41:00,243-INFO-sm_ansible_callback.py:53-append(): changed: [10.0.0.8] => (item={'key': u'10.10.0.9', 'value': u'server6'})"
"2017-11-27 07:41:01,600-INFO-sm_ansible_callback.py:53-append(): changed: [10.0.0.9] => (item={'key': u'10.10.0.8', 'value': u'server5'})"
"2017-11-27 07:41:01,603-INFO-sm_ansible_callback.py:53-append(): changed: [10.0.0.9]"
"2017-11-27 07:41:01,638-INFO-sm_ansible_callback.py:53-append(): fatal: [10.0.0.8]: FAILED! => (item - None) {"changed": true, "failed": true, "msg": "One or more items failed"}"

Thanks,
Wenqing

[wenqing liang (wliang)]

seen in FB r4.1-7 mitaka as well.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/38076
Submitter: Sachin Bansal (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/38076
Committed: http://github.com/Juniper/contrail-controller/commit/c4b4dae3230daba3396694bd154c6270f36c5e85
Submitter: Zuul (<email address hidden>)
Branch: master

commit c4b4dae3230daba3396694bd154c6270f36c5e85
Author: Sachin Bansal <email address hidden>
Date: Fri Dec 1 14:04:49 2017 -0800

Avoid reading bgp-router object after creating it

We need to read the object only if creation failed with RefsExistError
and we need to update md5 params. Otherwise, we don't need to read
the object.

Change-Id: I3bf2d2ab139f449513c43619837116dcd233ee60
Partial-Bug: 1734809

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/38558
Submitter: Sachin Bansal (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/38558
Committed: http://github.com/Juniper/contrail-controller/commit/822a4dd0b7aabb08c3ef5af47b966f91198c837d
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit 822a4dd0b7aabb08c3ef5af47b966f91198c837d
Author: Sachin Bansal <email address hidden>
Date: Fri Dec 1 14:04:49 2017 -0800

Avoid reading bgp-router object after creating it

We need to read the object only if creation failed with RefsExistError
and we need to update md5 params. Otherwise, we don't need to read
the object.

Change-Id: I3bf2d2ab139f449513c43619837116dcd233ee60
Partial-Bug: 1734809
(cherry picked from commit c4b4dae3230daba3396694bd154c6270f36c5e85)