[9.2] [updates] failed on task [contrail-os-controller-prov]

[vnaumov]

Fuel 9.0 -> 9.1 -> 9.2, Contrail 5.0.0(updated to 5.0.1), LDAP 3.0.1 [1]

Steps to reproduce:
  1. Deploy 9.0 env + contrail plugin (9 slaves) [2]
  2. Install and configure LDAP
  3. Reset env (to activate ldap)
  4. Deploy changes
  5. Update to 9.1 state master node [3]
  6. Add mos-updates
"deb http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/ mos9.0-updates main restricted"
     into Settings -> General on Fuel UI
  7. Run `fuel2 update --env {env_id} install --repos mos-updates --restart-rabbit --restart-mysql` on master Node
  8. Deploy changes
  9. scp contrail-5.0-5.0.1-1-5.noarch.rpm root@10.109.0.2:/tmp
  10. fuel plugins --update /tmp/contrail-5.0-5.0.1-1-5.noarch.rpm (as a workaround to bug [4])
  11. start update to 9.2 [5]
  12. cd /etc/yum.repos.d/
  13.vim mos92-updates.repo
  14.switch update mechanism in repo-url to snapshots-case, and use MOS_CENTOS_PROPOSED_MIRROR_ID=proposed-2017-01-28-114420 from snap 822 (rc2)
  15. continue updating to 9.2 [6]
  16.replace step 4 in above guide by following command
ansible-playbook playbooks/mos9_prepare_env.yml -e '
{"env_id":1, "snapshot_repo":"snapshots/9.0-2017-01-28-114420", "snapshot_suite":"mos9.0-proposed" }'
  17.check that's above snap-repo matched with repo-mos-updates in fuel UI
  18. go to step 9(from guide), fuel2 update --env 1 install --repos mos9.2-updates

Expected results:
successful deploy and continue updating env

Actual result:
Deployment has failed. All nodes are finished. Failed tasks: Task[contrail-os-controller-prov/6] Stopping the deployment process!

log from controller node
``` Could not prefetch neutron_router_interface provider 'neutron': Can't retrieve subnet-list because Neutron or Keystone API is not available.```

Workaround:
after step 17, need to disable Security TLS in General Settings and deploy

installed plugins: Contrail 5.0.0(updated to 5.0.1), LDAP 3.0.1
link to diagnostic snapshot: [7]

[1] http://plugins.mirantis.com/repository/l/d/ldap/ldap-3.0-3.0.1-1.noarch.rpm
[2] CI: http://jenkins-tpi.bud.mirantis.net:8080/view/9.0-custom/job/9.0.contrail.5.0.0.custom.dev/8/console
[3] http://docs.openstack.org/developer/fuel-docs/userdocs/fuel-install-guide/update-fuel.html#update-fuel
[4]https://bugs.launchpad.net/fuel/+bug/1658652
[5]https://docs.mirantis.com/openstack/fuel/fuel-master/release-notes/update-product/update-prerequisites-9-2.html
[6]https://docs.mirantis.com/openstack/fuel/fuel-master/release-notes/update-product/update-env-9-2.html#update-env-9-2
[7]https://drive.google.com/open?id=0B-AO_niPlWzFOHFKdTBaNDZDcG8

[Illia Polliul (ipolliul)]

Plugin version 5.0.1 is not intended to work on Fuel 9.1. *Please do not install it unless you are planning to do update to Fuel 9.2 immediately after that.*

[vnaumov (vnaumov-b)]

reproduced twice on another env and lab

[vnaumov (vnaumov-b)]

during reproduce this bug, there was filed sereveral logs on failed nodes

 Checksumfile /var/lib/puppet/state/state.yaml is corrupt ((<unknown>): could not find expected ':' while scanning a simple key at line 8327 column 3); replacing

seems root cause in amount of ssl certs

i decided to deploy without function "Use TLS" in ldap plugin

result = deploy failed

tasks = [pkg_upgrade] on nodes

tpi121

do we need to file another bug @ipolliul?

[Illia Polliul (ipolliul)]

Fix committed in https://review.openstack.org/#/c/429719/

[vnaumov (vnaumov-b)]

seems we need to reopen bug
9.2 Contrail 5.0.1 + LDAP (https://plugin-ci.fuel-infra.org/job/9.x.fuel-plugin.ldap.build/70/artifact/plugin/ldap-3.0-3.0.1-1.noarch.rpm)
problem reproduced again during "Deploy changes" on 9.2 env

[Yevgeniy (yshapovalov)]

Bug was reproduced with contrail-plugin 5.1.0 (contrail 3.2, fuel 9.2)
http://jenkins-tpi.bud.mirantis.net:8080/job/9.x.contrail.5.1.0.contrail_no_default.juniper/3/console

[Illia Polliul (ipolliul)]

2017-03-03 08:18:34 +0000 /Stage[main]/Contrail::Provision::Controller/Exec[prov_route_target]/returns (notice):(notice): cfgm_common.exceptions.BadRequest: Configured route target must use ASN that is different from global ASN or route target value must be less than 8000000

root@node-3:~# hiera -h contrail | grep route_target
 "contrail_route_target"=>"4294967295",

Please change test value for route_target back to something reasonable.

[Illia Polliul (ipolliul)]

Fix committed in https://review.openstack.org/#/c/441234/