SMLite/SM:R3.2build 31: Openstack HA: Provisioning gets stuck at pre_exec_vnc_galera_completedwith error pymysql.err.OperationalError) (1045, u"Access denied for user 'nova_api'

[sundarkh]

SMLite/SM:R3.2build 31: Openstack HA: Provisioning gets stuck at pre_exec_vnc_galera_completedwith error pymysql.err.OperationalError) (1045, u"Access denied for user 'nova_api'

1) Install SM Package 3.2.2-build 31
2) Add Ubuntu OS 14-4-5; COntrail Mitaka Package for ubuntu 3.2.2 build 31
3) Add cluster/Server of target which has mention of kernel version 3.13.0-106, and kernel_upgrade set as yes
4) Reimage happen successfully
5) Initiate provision using provision.sh with https://github.com/Juniper/contrail-tools/blob/master/testbeds/testbed_smgr_multi_interface_HA.py.ubuntu-14.04
6) Provision gets stuck with openstack state

 Issue #1
----------

Provision gets stuck at openstack_completed state

syslog
-------

Mar 20 05:40:27 nodei17 puppet-agent[2288]: (/Stage[openstack]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns) error: (pymysql.err.OperationalError) (1045, u"Access denied for user 'nova_api'@'nodei20.englab.juniper.net' (using password: YES)")

Mar 20 05:40:27 nodei17 puppet-agent[2288]: (/Stage[openstack]/Nova::Db::Sync_api/Exec[nova-db-sync-api]) Failed to call refresh: /usr/bin/nova-manage api_db sync returned 1 instead of one of [0]

Issue #2
---------

Contrail repo on the SM Lite box does not have the required linux headers

root@nodei17:/var/www/html/contrail/repo/image_contrail_packages_3_2_2_0_31_mitaka# ls *linux*
ls: cannot access *linux*: No such file or directory >>>> No linux headers for the required kernel version
root@nodei17:/var/www/html/contrail/repo/image_contrail_packages_3_2_2_0_31_mitaka# ls
conf contrail-install-packages_3.2.2.0-31-mitaka.tgz db dists dpdk_depends pool

to fetch some archives, maybe run apt-get update or try with --fix-missing?
Mar 20 05:13:39 nodei19 puppet-agent[4302]: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install linux-image-extra-3.13.0-106-generic' returned 100: Reading package lists...
Mar 20 05:13:39 nodei19 puppet-agent[4302]: Building dependency tree...
Mar 20 05:13:39 nodei19 puppet-agent[4302]: Reading state information...
Mar 20 05:13:39 nodei19 puppet-agent[4302]: The following extra packages will be installed:
Mar 20 05:13:39 nodei19 puppet-agent[4302]: linux-image-3.13.0-106-generic
Mar 20 05:13:39 nodei19 puppet-agent[4302]: Suggested packages:
Mar 20 05:13:39 nodei19 puppet-agent[4302]: fdutils linux-doc-3.13.0 linux-source-3.13.0 linux-tools
Mar 20 05:13:39 nodei19 puppet-agent[4302]: The following NEW packages will be installed:
Mar 20 05:13:39 nodei19 puppet-agent[4302]: linux-image-3.13.0-106-generic linux-image-extra-3.13.0-106-generic
Mar 20 05:13:39 nodei19 puppet-agent[4302]: 0 upgraded, 2 newly installed, 0 to remove and 15 not upgraded.
Mar 20 05:13:39 nodei19 puppet-agent[4302]: Need to get 52.1 MB of archives.
Mar 20 05:13:39 nodei19 puppet-agent[4302]: After this operation, 195 MB of additional disk space will be used.
Mar 20 05:13:39 nodei19 puppet-agent[4302]: WARNING: The following packages cannot be authenticated!
Mar 20 05:13:39 nodei19 puppet-agent[4302]: linux-image-3.13.0-106-generic linux-image-extra-3.13.0-106-generic
Mar 20 05:13:39 nodei19 puppet-agent[4302]: Authentication warning overridden.
Mar 20 05:13:39 nodei19 puppet-agent[4302]: Err http://puppet/contrail/repo/image_contrail_packages_3_2_2_0_31_mitaka/ contrail/main linux-image-3.13.0-106-generic amd64 3.13.0-106.153
Mar 20 05:13:39 nodei19 puppet-agent[4302]: Could not connect to puppet:80 (10.204.217.129). - connect (111: Connection refused)
Mar 20 05:13:39 nodei19 puppet-agent[4302]: Err http://puppet/contrail/repo/image_contrail_packages_3_2_2_0_31_mitaka/ contrail/main linux-image-extra-3.13.0-106-generic amd64 3.13.0-106.153
Mar 20 05:13:39 nodei19 puppet-agent[4302]: Unable to connect to puppet:http:
Mar 20 05:13:39 nodei19 puppet-agent[4302]: E: Failed to fetch http://puppet/contrail/repo/image_contrail_packages_3_2_2_0_31_mitaka/pool/main/l/linux/linux-image-3.13.0-106-generic_3.13.0-106.153_amd64.deb Could not connect to puppet:80 (10.204.217.129). - connect (111: Connection refused)
Mar 20 05:13:39 nodei19 puppet-agent[4302]:

Though reachability to SM+conf node (10.204.217.129) , other nodes are unable to download the linux header;

root@nodei19:~# ping puppet
PING puppet (10.204.217.129) 56(84) bytes of data.
64 bytes from puppet (10.204.217.129): icmp_seq=1 ttl=64 time=0.166 ms
64 bytes from puppet (10.204.217.129): icmp_seq=2 ttl=64 time=0.164 ms
64 bytes from puppet (10.204.217.129): icmp_seq=3 ttl=64 time=0.110 ms

7)Issue #2 seems similar to https://bugs.launchpad.net/juniperopenstack/+bug/1662079 SM:3.2.1-build21: Provision of Ubuntu Targets does not initilialise with error in "Unable to locate package linux-headers-3.13.0-100-generic" Edit

8) This issue is seen only in SM Lite setup ;

[Abhay Joshi (abhayj)]

Not applicable to mainline.

[Dheeraj Gautam (dgautam)]

#2 is a non-issue as packages are under pool/... folder, and packages under debian has been removed after repo is created, as these were redundant and not required.

[Dheeraj Gautam (dgautam)]

#1 is not observed into SM based installation. for R3.1-33-Ubuntu-Mitaka

[Jeba Paulaiyan (jebap)]

As the provision stuck issue is not happening, reducing severity for now.

[Dheeraj Gautam (dgautam)]

Re-assigning to Jeba to validate the issue in R3.2.3, as might be issue of intermediate builds.