FAILED - RETRYING: TASK: contrail/config : Wait till config

[Prashanth Nageshappa]

R4.0.1 build #43 installation went thru fine. But, when I wanted to change the config using contrailctl, I hit an issue described below.

I added the following lines to the end of /etc/contrailctl/controller.conf file:
[API]
aaa_mode = no-auth
[ANALYTICS_API]
aaa_mode = no-auth

And ran the command:
docker exec controller contrailctl config sync -c controller -Fv

Following was the failure:
TASK [contrail/config : Make sure contrail svc-monitor service is in desired state] ***
ok: [localhost]

TASK [contrail/config : include] ***********************************************
included: /contrail-ansible-internal/playbooks/roles/contrail/config/tasks/provision/main.yml for localhost

TASK [contrail/config : Include cloud_orchestrator specific code to provision config objects] ***
included: /contrail-ansible-internal/playbooks/roles/contrail/config/tasks/provision/keystone.yml for localhost

TASK [contrail/config : Wait till config api server answers before creating config objects] ***
FAILED - RETRYING: TASK: contrail/config : Wait till config api server answers before creating config objects (300 retries left).
FAILED - RETRYING: TASK: contrail/config : Wait till config api server answers before creating config objects (299 retries left).
:
:
FAILED - RETRYING: TASK: contrail/config : Wait till config api server answers before creating config objects (2 retries left).
FAILED - RETRYING: TASK: contrail/config : Wait till config api server answers before creating config objects (1 retries left).
fatal: [localhost]: FAILED! => {"attempts": 300, "changed": false, "content": "", "failed": true, "msg": "Status code was not [200]: Request failed: <urlopen error [Errno 111] Connection refused>", "redirected": false, "status": -1, "url": "http://127.0.0.1:8095"}
        to retry, use: --limit @/contrail-ansible-internal/playbooks/contrail_controller.retry

PLAY RECAP *********************************************************************
localhost : ok=85 changed=9 unreachable=0 failed=1

I also see controller docker is restarting every few mins. When I look into the /var/logs/contrail/ I see these errors:

==> contrail-webui-middleware-stdout.log <==
08/23/2017 08:44:42 AM - debug: Redis: reconnecting
08/23/2017 08:44:42 AM - debug: Redis: reconnecting
08/23/2017 08:44:42 AM - debug: Redis: reconnecting
08/23/2017 08:44:42 AM - debug: Redis: reconnecting
08/23/2017 08:44:42 AM - debug: Created a new Job with jobName:getVRoutersSummary
08/23/2017 08:44:42 AM - debug: Created a new Job with jobName:getVRoutersGenerators
08/23/2017 08:44:42 AM - debug: Redis: reconnecting
08/23/2017 08:44:42 AM - error: Redis: error String: Error: Redis connection to 10.213.17.153:6379 failed - connect ECONNREFUSED
08/23/2017 08:44:42 AM - debug: Redis: reconnecting
08/23/2017 08:44:43 AM - debug: Redis: reconnecting
08/23/2017 08:44:43 AM - debug: Redis: reconnecting
08/23/2017 08:44:43 AM - debug: Redis: reconnecting
08/23/2017 08:44:43 AM - debug: Redis: reconnecting

/usr/lib64/node_modules/redis/index.js:1046
                    throw new Error(err);
                          ^
Error: Redis connection gone from close event.
    at Command.callback (/usr/lib64/node_modules/redis/index.js:1046:27)
    at RedisClient.flush_and_error (/usr/lib64/node_modules/redis/index.js:127:25)
    at RedisClient.connection_gone (/usr/lib64/node_modules/redis/index.js:435:10)
    at Socket.<anonymous> (/usr/lib64/node_modules/redis/index.js:96:14)
    at Socket.emit (events.js:95:17)
    at TCP.close (net.js:466:12)
08/23/2017 08:44:43 AM - debug: The worker #12 has disconnected.
08/23/2017 08:44:43 AM - error: The worker #12 has exited with exit code 8
08/23/2017 08:44:43 AM - error: Redis: error String: Error: Redis connection to 10.213.17.153:6379 failed - connect ECONNREFUSED

08/23/2017 08:44:43 AM - debug: Redis: reconnecting
08/23/2017 08:44:43 AM - error: Redis: error String: Error: Redis connection to 10.213.17.153:6379 failed - connect ECONNREFUSED
08/23/2017 08:44:43 AM - debug: Redis: reconnecting
08/23/2017 08:44:44 AM - debug: Contrail Service Response Updated by process:11972 {"dns-server":[{"@publisher-id":"10.213.17.153","ip-address":"10.213.17.153","port":"8092","status":"up"}]}
08/23/2017 08:44:44 AM - debug: Got the token successfully for tenant:admin
08/23/2017 08:44:44 AM - debug: Contrail Service Response Updated by process:11972 {"ApiServer":[{"@publisher-id":"10.213.17.153","ip-address":"10.213.17.153","port":"8082","status":"up"}]}
08/23/2017 08:44:44 AM - debug: Got the token successfully for tenant:admin
08/23/2017 08:44:44 AM - debug: Contrail Service Response Updated by process:11972 {"OpServer":[{"@publisher-id":"10.213.17.153","ip-address":"10.213.17.153","port":"8081","status":"up"}]}

==> contrail-api-0-stdout.log <==
10.213.17.153 - - [2017-08-23 08:44:54] "GET / HTTP/1.1" 200 20549 0.002582
10.213.17.153 - - [2017-08-23 08:44:54] "GET /physical-routers?count=False&shared=False&detail=False HTTP/1.1" 200 132 0.004821

==> supervisord.log <==
2017-08-23 08:45:00,106 WARN received SIGHUP indicating restart request
2017-08-23 08:45:00,107 INFO waiting for supervisord_control, supervisord_config, supervisord_webui to die
2017-08-23 08:45:00,110 INFO stopped: supervisord_control (terminated by SIGKILL)

[Ignatious Johnson Christopher (ijohnson-x)]

Hi Prashanth,

I am trying to reproduce the issue locally.

I will request you to recreate, if i am not able to reproduce.
If you still have the setup in failed state. can you please share he setup details.

Thanks,
Ignatious

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35213
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35214
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/35214
Committed: http://github.com/Juniper/contrail-ansible-internal/commit/937972eae70d2027aaee0e36465deef1a56d7b66
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 937972eae70d2027aaee0e36465deef1a56d7b66
Author: Ignatious Johnson Christopher <email address hidden>
Date: Fri Sep 1 17:13:59 2017 -0700

Connect to localhost:8082 when aaa_mode is

configured as no-auth in the api-server.

Change-Id: Ibf699beda523747d0691aeda5d9e355cdd51b64f
Closes-Bug: 1712509

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/35213
Committed: http://github.com/Juniper/contrail-ansible-internal/commit/2fb304738c959b0cda6d0ed18929a990d08f7b3e
Submitter: Zuul (<email address hidden>)
Branch: master

commit 2fb304738c959b0cda6d0ed18929a990d08f7b3e
Author: Ignatious Johnson Christopher <email address hidden>
Date: Fri Sep 1 17:13:59 2017 -0700

Connect to localhost:8082 when aaa_mode is

configured as no-auth in the api-server.

Change-Id: Ibf699beda523747d0691aeda5d9e355cdd51b64f
Closes-Bug: 1712509