standalone-upgrade fails during deploy steps (generate config) step1 for "Evaluation Error:... Could not find class ::tripleo::profile::base::placement::api"

[Marios Andreou]

In [1][2] the tripleo-ci-centos-7-standalone-upgrade fails during upgrade but in the deploy steps "TASK [Run container-puppet tasks (generate config) during step 1]" with a trace like:

  2019-04-24 08:14:59 | 2019-04-24 08:14:59.418 108926 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] "2019-04-24 08:12:32,176 ERROR: 133938 -- ['/usr/bin/podman', 'run', '--user', 'root', '--name', u'container-puppet-placement', '--env', u'PUPPET_TAGS=file,file_line,concat,augeas,cron,placement_config', '--env', u'NAME=placement', '--env', u'HOSTNAME=standalone', '--env', 'NO_ARCHIVE=', '--env', 'STEP=6', '--env', 'NET_HOST=true', '--log-driver', 'json-file', '--volume', '/etc/localtime:/etc/localtime:ro', '--volume', '/tmp/tmpBTkLXn:/etc/config.pp:ro', '--volume', '/etc/puppet/:/tmp/puppet-etc/:ro', '--volume', '/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume', '/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume', '/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume', '/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume', '/var/lib/config-data:/var/lib/config-data/:rw', '--volume', '/dev/log:/dev/log:rw', '--log-opt', u'path=/var/log/containers/stdouts/container-puppet-placement.log', '--security-opt', 'label=disable', '--volume', '/usr/share/openstack-puppet/modules/:/usr/share/openstack-puppet/modules/:ro', '--entrypoint', '/var/lib/container-puppet/container-puppet.sh', '--net', 'host', '--volume', '/etc/hosts:/etc/hosts:ro', '--volume', '/var/lib/container-puppet/container-puppet.sh:/var/lib/container-puppet/container-puppet.sh:ro', u'docker.io/tripleomaster/centos-binary-placement-api:current-tripleo'] run failed after + mkdir -p /etc/puppet",
  2019-04-24 08:14:59 | 2019-04-24 08:14:59.418 108926 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] "+ '[' -n file,file_line,concat,augeas,cron,placement_config ']'",
  2019-04-24 08:14:59 | 2019-04-24 08:14:59.419 108926 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] "+ TAGS='--tags file,file_line,concat,augeas,cron,placement_config'",
  2019-04-24 08:14:59 | 2019-04-24 08:14:59.419 108926 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] "+ origin_of_time=/var/lib/config-data/placement.origin_of_time",
  2019-04-24 08:14:59 | 2019-04-24 08:14:59.420 108926 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] "+ touch /var/lib/config-data/placement.origin_of_time",
  2019-04-24 08:14:59 | 2019-04-24 08:14:59.421 108926 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] "+ /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --logdest syslog --logdest console --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags file,file_line,concat,augeas,cron,placement_config /etc/config.pp",
  2019-04-24 08:14:59 | 2019-04-24 08:14:59.422 108926 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] "Error: Evaluation Error: Error while evaluating a Function Call, Could not find class ::tripleo::profile::base::placement::api for standalone.localdomain (file: /etc/config.pp, line: 3, column: 1) on node standalone.localdomain",

Not sure yet if this is related to [3] which is a recent change for placement in puppet-tripleo

[1] http://logs.openstack.org/63/653363/2/check/tripleo-ci-centos-7-standalone-upgrade/054554f/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz#_2019-04-24_08_14_59
[2] http://logs.openstack.org/20/655120/1/check/tripleo-ci-centos-7-standalone-upgrade/338c482/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz#_2019-04-24_08_27_45
[3] https://review.opendev.org/#/c/624335/

[Quique Llorente (quiquell)]

13:22 <mschuppert> quiquell|rover: there is additional work to be done to support upgrades from nova-placement in Stein to an extracted service in Train https://review.opendev.org/#/q/topic:tripleo-placement-upgrade-from-nova-placement+(status:open++OR+status:merged)

So upgrades are now WIP.

[Martin Schuppert (mschuppert)]

[1] add the needed upgrade step to extract the DB data from nova_api db to the new placement db, but the upgrade tasks still fail with this error [2].

We seem to miss the links in /etc/puppet/modules to /usr/share/openstack-puppet/modules/* :

[root@standalone openstack-tripleo-heat-templates]# paunch debug --file /var/lib/tripleo-config/hashed-container-startup-config-step_2.json --interactive --shell --user root --container placement_init_log --action run
()[root@bd8e30391735 /]# ls -la /etc/puppet/modules/
total 0
drwxr-xr-x. 2 root root 6 Oct 31 2018 .
drwxr-xr-x. 3 root root 57 May 4 00:23 ..

[root@standalone openstack-tripleo-heat-templates]# paunch debug --file /var/lib/tripleo-config/hashed-container-startup-config-step_2.json --interactive --shell --user root --container cinder_scheduler_init_logs --action run
()[root@bde471ce86c0 /]# ls -la /etc/puppet/modules/
total 0
drwxr-xr-x. 2 root root 6 Oct 31 2018 .
drwxr-xr-x. 3 root root 57 May 4 00:23 ..

[1] https://review.opendev.org/#/c/635134/
[2] http://logs.openstack.org/34/635134/13/check/tripleo-ci-centos-7-standalone-upgrade/0d17721/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz#_2019-05-04_00_03_21

[Piotr Kopec (pkopec)]

I've reproduced the CI and the links seams fine but the problem seems to be with packaging or CI configuration. Package that is installed does not contain placement puppet manifests as seen below.

[zuul@standalone manifests]$ yum provides "*manifests/profile/base/placement/api.pp"
Loaded plugins: fastestmirror, priorities
Loading mirror speeds from cached hostfile
124 packages excluded due to repository priority protections
puppet-tripleo-10.4.1-0.20190505174907.2678a4c.el7.noarch : TripleO Puppet Module
Repo : delorean-current
Matched from:
Filename : /usr/share/openstack-puppet/modules/tripleo/manifests/profile/base/placement/api.pp

[zuul@standalone manifests]$ less /usr/share/openstack-puppet/modules/tripleo/manifests/profile/base/placement/api.pp
/usr/share/openstack-puppet/modules/tripleo/manifests/profile/base/placement/api.pp: No such file or directory
[zuul@standalone manifests]$ rpm -qa | grep puppet-tripleo
puppet-tripleo-10.4.2-0.20190502215846.02cd12e.el7.noarch

[Martin Schuppert (mschuppert)]

good catch, so the issue is the that master repo has no master puppet-tripleo rpm containing the placement manifest. 10.4.2 is still stein.

https://trunk.rdoproject.org/centos7-master-head/report.html
https://trunk.rdoproject.org/centos7-master-head/38/8f/388f96b348dc6c41aab892f7ed59aa64ecbff146_a8eef76b/

[Martin Schuppert (mschuppert)]

so the issue is that in the master repo [1] we have 10.4.1 version and in the stein repo [2] we have 10.4.2 . For the upgrade ci tests we get 10.4.2 (which does not include the placement manifest) and when we do the upgrade to master the files from master gets not installed as the rpm has a lower version.

[1] https://trunk.rdoproject.org/centos7-master/current/
[2] https://trunk.rdoproject.org/centos7-stein/current/

[Martin Schuppert (mschuppert)]

https://review.opendev.org/#/c/657758/