Redis server service is not running after provisioning in R3.2

[musharani]

After provisioning R3.2-47 mitaka build redis-server service is not running. Because of this webui page is not opening.

root@nodec41:~# service redis-server status
redis-server stop/waiting

root@nodec41:~# contrail-status
== Contrail vRouter ==
supervisor-vrouter: active
contrail-vrouter-agent active (Collector connection down)
contrail-vrouter-nodemgr initializing (Collector connection down)

== Contrail Control ==
supervisor-control: active
contrail-control initializing (Collector connection down)
contrail-control-nodemgr initializing (Collector connection down)
contrail-dns active
contrail-named active

== Contrail Analytics ==
supervisor-analytics: active
contrail-alarm-gen initializing (Collector, Redis-UVE:AggregateRedis[None] connection down)
contrail-analytics-api initializing (Collector, UvePartitions:UVE-Aggregation[Partitions:0], Redis-UVE:LOCAL[None] connection down)
contrail-analytics-nodemgr initializing (Collector connection down)
contrail-collector initializing (Number of connections:6, Expected:7 Missing: Discovery:Collector)
contrail-query-engine initializing (Number of connections:3, Expected:4 Missing: Redis-Query:Query)
contrail-snmp-collector initializing (Collector connection down)
contrail-topology initializing (Collector connection down)

== Contrail Config ==
supervisor-config: active
contrail-api:0 initializing (Collector connection down)
contrail-config-nodemgr initializing (Collector connection down)
contrail-device-manager initializing (Collector connection down)
contrail-discovery:0 initializing (Collector connection down)
contrail-schema initializing (Collector connection down)
contrail-svc-monitor initializing (Collector connection down)
ifmap active

== Contrail Web UI ==
supervisor-webui: active
contrail-webui active
contrail-webui-middleware active

== Contrail Database ==
contrail-database: active

== Contrail Supervisor Database ==
supervisor-database: active
contrail-database-nodemgr active
kafka active

== Contrail Support Services ==
supervisor-support-service: active
rabbitmq-server active

redis-server log:
=================
[876] 17 Aug 18:54:03.863 # Creating Server TCP listening socket 10.204.217.81:6379: bind: Cannot assign requested address
[879] 17 Aug 18:54:03.866 # Unable to set the max number of files limit to 10032 (Operation not permitted), setting the max clients configuration to 3984.
[879] 17 Aug 18:54:03.866 # Creating Server TCP listening socket 10.204.217.81:6379: bind: Cannot assign requested address

[Naga Kiran (nagakiran)]

From: Sundaresan Rajangam
Sent: Wednesday, August 23, 2017 12:07 PM
To: Usha Rani Mani
Cc: Naga Kiran K Y S; Anish Mehta; Manoj Naik; Biswajit Mandal; Megh Bhatt
Subject: Re: Reg: https://bugs.launchpad.net/juniperopenstack/+bug/1711366

+Megh

From the fab log, I see that redis was started twice during provisioning time

setup_all_2017_08_22_17_04_57_009210.log:2017-08-22 17:11:44:340936: [root@10.204.217.81] sudo: service redis start
setup_all_2017_08_22_17_04_57_009210.log:2017-08-22 17:12:12:420994: [root@10.204.217.81] sudo: service redis start

The corresponding logs from redis.log:

[27217] 22 Aug 17:11:44.512 # Server started, Redis version 2.8.19
[28824] 22 Aug 17:12:12.483 # Server started, Redis version 2.8.19

However, the system was rebooted @ 17:17

[root@nodec41 fabric-utils]# last reboot
reboot system boot 3.10.0-229.el7.x Tue Aug 22 17:17 - 10:45 (17:27)
reboot system boot 3.10.0-229.el7.x Tue Aug 22 17:04 - 10:45 (17:40)
reboot system boot 3.10.0-229.el7.x Tue Aug 22 16:49 - 10:45 (17:55)

From the /var/log/messages, it seems that redis started before the network was up and hence failed to start.
redis service was not started later after the network was up.

Aug 22 17:17:39 nodec41 kernel: IPv6: ADDRCONF(NETDEV_UP): p1p0p0: link is not ready
Aug 22 17:17:40 nodec41 network: Bringing up interface p1p0p0: [ OK ]
Aug 22 17:17:40 nodec41 network: Bringing up interface vhost0: ERROR : [/etc/sysconfig/network-scripts/ifup-eth] Device vhost0 does not seem to be present, delaying initialization.
Aug 22 17:17:41 nodec41 /etc/sysconfig/network-scripts/ifup-eth: Device vhost0 does not seem to be present, delaying initialization. <<<<<<<
Aug 22 17:17:41 nodec41 network: [FAILED] <<<<<<<<
Aug 22 17:17:41 nodec41 systemd-sysctl: Overwriting earlier assignment of net/ipv4/ip_local_reserved_ports in file '/usr/lib/sysctl.d/openstack-keystone.conf'.
Aug 22 17:17:41 nodec41 systemd-sysctl: Overwriting earlier assignment of net/ipv4/ip_local_reserved_ports in file '/etc/sysctl.conf'.
Aug 22 17:17:41 nodec41 systemd: network.service: control process exited, code=exited status=1 <<<<<<<<<<<<
Aug 22 17:17:41 nodec41 systemd: Failed to start LSB: Bring up/down networking.
Aug 22 17:17:41 nodec41 systemd: Starting Redis persistent key-value database… <<<<<<<<<<<
Aug 22 17:17:41 nodec41 systemd: Started Redis persistent key-value database. <<<<<
Aug 22 17:17:43 nodec41 redis-shutdown: Could not connect to Redis at 10.204.217.81:6379: Network is unreachable <<<<<<<
Aug 22 17:17:43 nodec41 systemd: redis.service: control process exited, code=exited status=1
Aug 22 17:17:43 nodec41 systemd: Unit redis.service entered failed state.
...
Aug 22 17:18:06 nodec41 systemd-sysctl: Overwriting earlier assignment of net/ipv4/ip_local_reserved_ports in file '/etc/sysctl.conf'.
Aug 22 17:18:06 nodec41 NetworkManager[704]: <info> (vhost0): carrier is OFF
Aug 22 17:18:06 nodec41 NetworkManager[704]: <info> (vhost0): new Generic device (driver: 'vhost' ifindex: 7)
Aug 22 17:18:06 nodec41 NetworkManager[704]: <info> (vhost0): exported as /org/freedesktop/NetworkManager/Devices/6
Aug 22 17:18:06 nodec41 NetworkManager[704]: <info> (v...

[Sundaresan Rajangam (srajanga)]

This is a corner case and not a typical use case, where compute and analytics run on the same server. Hence marking it as Won't Fix.