Ussury upgrade job fails to lookup hiera value

[Bogdan Dobrelya]

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e1e/757829/8/gate/tripleo-ci-centos-8-standalone-upgrade-ussuri/e1e917a/logs/undercloud/var/log/extra/podman/containers/container-puppet-neutron/stdout.log

<13>Nov 27 00:05:00 puppet-user: Error: Evaluation Error: Error while evaluating a Function Call, 'downcase' parameter 'arg' expects a value of type Numeric, String, Array, or Hash, got Undef (file: /etc/puppet/modules/tripleo/manifests/profile/base/neutron/plugins/ml2.pp, line: 47, column: 21) on node standalone.localdomain

Indeed we can see that hieradata/all_nodes.json does not contain neutron_plugin_ml2_short_bootstrap_node_name https://opendev.org/openstack/puppet-tripleo/src/branch/stable/ussuri/manifests/profile/base/neutron/plugins/ml2.pp#L42 and that's why Undef makes downcase() failing

But there is hieradata/all_nodes.json: "neutron_plugin_ml2_ovn_short_bootstrap_node_name": "standalone",

[Marios Andreou (marios-b)]

12:03 < marios> still not clear if this is a consistent thing
                ...https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-standalone-upgrade-ussuri
12:03 < marios> bogdando: there is only one example so far the one you point to in the bug afaik
12:03 < marios> bogdando: we discussed this earlier with arx (commented on /+/757829/
12:04 < bogdando> so that happens on the way from Ipuppet-tripleo.noarch 11.5.0-0.20201126221846.f34afbe.el8 /
                  openstack-tripleo-heat-templates.noarch 11.3.2-0.20201124102500.1a31770.el8 to
                  Ipuppet-tripleo-0:12.4.2-0.20201116133848.25c6b71.el8.noarch /
                  openstack-tripleo-heat-templates-0:12.4.2-0.20201107032927.55eafcb.el8.noarch
12:04 < marios> bogdando: but the error you point to in the bug doesn't seem to be something that is
                inconsistent i.e. we should be seeing it all the time
12:07 < bogdando> marios: other commits for those packages was there, on the passed one
12:07 < bogdando> so that gives us bisect prolly
12:09 < bogdando> marios: hm, only "from" is different. "to" has same versions
12:10 < bogdando> anyway, in tht there is f296b67..1a31770 to analyze for stable/ussury
12:11 < bogdando> nothing interesting in the diff of puppet tripleo

12:12 < bogdando> and I cannot see actually nothing suspicious for tht as well. So have no poking idea why it
                  has passed fristly, then failed with that hiera error
12:12 < bogdando> ramishra: ^^ perchance
12:13 < bogdando> tl;dr disable voting, merge content providers, put it back voting
12:13 < marios> bogdando: yeah possibly, if it happens on next run
12:13 < marios> bogdando: yeah possibly, if it happens on next run

12:14 < marios> bogdando: s/next/current
12:16 < bogdando> and hiera data is equal for both jobs
12:17 < bogdando> how come that neutron_plugin_ml2_short* pops up
12:17 < bogdando> instead of neutron_plugin_ml2_ovn
12:17 < bogdando> ¯\_(ツ)_/¯

[Bogdan Dobrelya (bogdando)]

https://review.opendev.org/c/openstack/puppet-tripleo/+/764442

[Marios Andreou (marios-b)]

thanks ykarel & bogdan on irc just now

seems this is related to bug https://bugs.launchpad.net/tripleo/+bug/1904565 and should be addressed with https://review.opendev.org/c/openstack/tripleo-ci/+/764359/1/playbooks/tripleo-ci/install-built-repo.yml

i.e. explicitly use the correct provider ip address

[Bogdan Dobrelya (bogdando)]

Abandoned the backport since downcase(Undef) works OK for puppet 5.5.10 in Train...

[yatin (yatinkarel)]

The issue happened because package built[1] in train provider was used in ussuri-upgrade jobs, so puppet-tripleo from train got installed with other packages from ussuri leading to the issue in neutron container-puppet run.

There are two issues:-
1) both train and ussuri provider building package, ideally package should be built only for the on review change.
2) ussuri-upgrade job using package build from train-provider instead of ussuri-provider, the WIP patch https://review.opendev.org/c/openstack/tripleo-ci/+/764359/1/playbooks/tripleo-ci/install-built-repo.yml should address the issue.

[1] https://b452999f9cf0b74ec5d3-cc9a180b756778dd4f9ff9a8ddf16569.ssl.cf5.rackcdn.com/757829/8/gate/tripleo-ci-centos-8-content-provider-train/abd007d/logs/undercloud/home/zuul/dlrn.log

[Marios Andreou (marios-b)]

the patch at https://review.opendev.org/c/openstack/tripleo-ci/+/764359/ merged so I am moving this to fix-released

if there is another issue then please move back to in progress (or possibly file a new bug) and please ping the ruck|rover in tripleo or oooq

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 11.6.0 release.