featureset035 master failing with ERROR: Protected resource registry overrides detected

[Ananya Banerjee]

periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master:

https://logserver.rdoproject.org/21/35121/12/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master/121f78b/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master/5177268/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master:

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/e42975a/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

fails at overcloud deploy with:

2021-09-14 10:45:31 | 2021-09-14 10:45:31.663 186827 INFO tripleoclient.utils.utils [-] Temporary directory [ /tmp/tripleohnoz66sw ] cleaned up
2021-09-14 10:45:31 | 2021-09-14 10:45:31.664 186827 INFO tripleoclient.utils.utils [-] Ansible execution success. playbook: /usr/share/ansible/tripleo-playbooks/cli-overcloud-node-growvols.yaml
2021-09-14 10:45:31 | 2021-09-14 10:45:31.665 186827 INFO tripleoclient.utils.utils [-] Temporary directory [ /tmp/tripleoscysgym3 ] cleaned up
2021-09-14 10:45:31 | 2021-09-14 10:45:31.843 186827 ERROR openstack [-] ERROR: Protected resource registry overrides detected! These entries are used in internal environments and should not be overridden in the user environment. Please remove these overrides from the environment files.
2021-09-14 10:45:31 | Conflict detected for resource_registry entry: OS::TripleO::DeployedServer::ControlPlanePort.
2021-09-14 10:45:31 | User environment: /usr/share/openstack-tripleo-heat-templates/environments/deployed-server-environment.yaml.
2021-09-14 10:45:31 | Internal environment: ['/home/zuul/overcloud-deploy/overcloud/tripleo-heat-templates/user-environments/baremetal-deployed.yaml']
2021-09-14 10:45:31 | Conflict detected for resource_registry entry: OS::TripleO::DeployedServer::ControlPlanePort.
2021-09-14 10:45:31 | User environment: /home/zuul/overcloud-baremetal-deployed.yaml.
2021-09-14 10:45:31 | Internal environment: ['/home/zuul/overcloud-deploy/overcloud/tripleo-heat-templates/user-environments/baremetal-deployed.yaml']
2021-09-14 10:45:31 | : tripleoclient.exceptions.DeploymentError: ERROR: Protected resource registry overrides detected! These entries are used in internal environments and should not be overridden in the user environment. Please remove these overrides from the environment files.
2021-09-14 10:45:31 | 2021-09-14 10:45:31.847 186827 INFO osc_lib.shell [-] END return value: 1
2021-09-14 10:45:32 | + status_code=1
2021-09-14 10:45:32 | + openstack stack list
2021-09-14 10:45:32 | + grep -q overcloud
2021-09-14 10:46:03 | + echo 'overcloud deployment not started. Check the deploy configurations'
2021-09-14 10:46:03 | overcloud deployment not started. Check the deploy configurations
2021-09-14 10:46:03 | + exit 1

[Ronelle Landy (rlandy)]

Possible change with impact: https://github.com/openstack/python-tripleoclient/commit/81cd105620b90d6cda29c0c3fa1c2fd09de6c61d

[wes hayutin (weshayutin)]

also hitting periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master

[Michele Baldessari (michele)]

I started to hit this as well:
2021-09-14 09:54:25.602 97272 ERROR openstack [-] ERROR: Protected resource registry overrides detected! These entries are used in internal environments and should not be overridden in the user environment. Please remove these overrides from the environment files.
Conflict detected for resource_registry entry: OS::TripleO::Controller::Ports::InternalApiPort.
        User environment: /home/stack/overcloud-bm-deployed.yaml.
        Internal environment: ['/home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/user-environments/baremetal-deployed.yaml']
Conflict detected for resource_registry entry: OS::TripleO::Compute::Ports::StoragePort.
        User environment: /home/stack/overcloud-bm-deployed.yaml.
        Internal environment: ['/home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/user-environments/baremetal-deployed.yaml']
Conflict detected for resource_registry entry: OS::TripleO::DeployedServer::ControlPlanePort.
        User environment: /home/stack/overcloud-bm-deployed.yaml.
        Internal environment: ['/home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/user-environments/baremetal-deployed.yaml']
Conflict detected for resource_registry entry: OS::TripleO::Controller::Ports::StorageMgmtPort.
        User environment: /home/stack/overcloud-bm-deployed.yaml.
        Internal environment: ['/home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/user-environments/baremetal-deployed.yaml']
Conflict detected for resource_registry entry: OS::TripleO::Compute::Ports::TenantPort.
        User environment: /home/stack/overcloud-bm-deployed.yaml.
        Internal environment: ['/home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/user-environments/baremetal-deployed.yaml']
Conflict detected for resource_registry entry: OS::TripleO::Controller::Ports::TenantPort.
        User environment: /home/stack/overcloud-bm-deployed.yaml.
        Internal environment: ['/home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/user-environments/baremetal-deployed.yaml']
Conflict detected for resource_registry entry: OS::TripleO::Compute::Ports::InternalApiPort.
        User environment: /home/stack/overcloud-bm-deployed.yaml.
        Internal environment: ['/home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/user-environments/baremetal-deployed.yaml']
Conflict detected for resource_registry entry: OS::TripleO::Controller::Ports::StoragePort.
        User environment: /home/stack/overcloud-bm-deployed.yaml.
        Internal environment: ['/home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/user-environments/baremetal-deployed.yaml']
Conflict detected for resource_registry entry: OS::TripleO::Controller::Ports::ExternalPort.
        User environment: /home/stack/overcloud-bm-deployed.yaml.
        Internal environment: ['/home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/user-environments/baremetal-deployed.yaml']
Conflict detected for resource_registry entry: OS::TripleO::Compute::Ports::ExternalPort.
        User environment: /home/stack/overcloud-bm-deployed.yaml.
        Internal environment: ['/home/stack/overcloud-deploy/overcloud/tripleo-h...

[Ronelle Landy (rlandy)]

Added fs035 and fs020 to https://review.opendev.org/c/openstack/tripleo-quickstart/+/807703.
with this change, deploy passes.

[Ronelle Landy (rlandy)]

Testing https://review.opendev.org/c/openstack/tripleo-quickstart/+/807703 as a fix

and then https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/806562

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/807703
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/dbb21969d2eb8d012fbd645111f7ea1d4d76d067
Submitter: "Zuul (22348)"
Branch: master

commit dbb21969d2eb8d012fbd645111f7ea1d4d76d067
Author: Harald Jensås <email address hidden>
Date: Tue Sep 7 13:14:33 2021 +0200

    Disable protected resourced types

    This review adds and extra-vars to
    disable protected resourced types in
    releases wallaby and after for
    fs010, fs037, fs035 and fs020.

    There is a need to disable the resource
    registry override protection to merge:
    https://review.opendev.org/798642

    Closes-Bug: #1943579
    Change-Id: I618c0a0d99c934fb65a6af30bc76096d52d4679d

[wes hayutin (weshayutin)]

Also impacting jobs in centos

2021-09-20 19:52:08 | 2021-09-20 19:52:08.579966 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-09-20 19:52:08 | 2021-09-20 19:52:08.775 123124 INFO tripleoclient.utils.utils [-] Temporary directory [ /tmp/tripleo2owfus8z ] cleaned up
2021-09-20 19:52:08 | 2021-09-20 19:52:08.776 123124 INFO tripleoclient.utils.utils [-] Ansible execution success. playbook: /usr/share/ansible/tripleo-playbooks/cli-overcloud-node-growvols.yaml
2021-09-20 19:52:08 | 2021-09-20 19:52:08.777 123124 INFO tripleoclient.utils.utils [-] Temporary directory [ /tmp/tripleoez9frlsh ] cleaned up
2021-09-20 19:52:08 | 2021-09-20 19:52:08.874 123124 ERROR openstack [-] ERROR: Protected resource registry overrides detected! These entries are used in internal environments and should not be overridden in the user environment. Please remove these overrides from the environment files.
2021-09-20 19:52:08 | Conflict detected for resource_registry entry: OS::TripleO::DeployedServer::ControlPlanePort.
2021-09-20 19:52:08 | User environment: /usr/share/openstack-tripleo-heat-templates/environments/deployed-server-environment.yaml.
2021-09-20 19:52:08 | Internal environment: ['/home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/user-environments/baremetal-deployed.yaml']
2021-09-20 19:52:08 | Conflict detected for resource_registry entry: OS::TripleO::DeployedServer::ControlPlanePort.
2021-09-20 19:52:08 | User environment: /home/stack/overcloud-baremetal-deployed.yaml.
2021-09-20 19:52:08 | Internal environment: ['/home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/user-environments/baremetal-deployed.yaml']
2021-09-20 19:52:08 | : tripleoclient.exceptions.DeploymentError: ERROR: Protected resource registry overrides detected! These entries are used in internal environments and should not be overridden in the user environment. Please remove these overrides from the environment files.
2021-09-20 19:52:08 | 2021-09-20 19:52:08.875 123124 INFO osc_lib.shell [-] END return value: 1

https://artifacts.ci.centos.org/rdo/jenkins-tripleo-quickstart-promote-master-current-tripleo-delorean-minimal-29/undercloud/home/stack/overcloud_deploy.log

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/810310

[wes hayutin (weshayutin)]

https://review.opendev.org/c/openstack/tripleo-quickstart/+/810310

[Bogdan Dobrelya (bogdando)]

why is this a CI only issue? what about changing defaults https://opendev.org/openstack/python-tripleoclient/src/branch/master/tripleoclient/v1/overcloud_deploy.py#L1034 for tripleo as well?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/810310
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/966e96951042c0255ea4632bf08b4df768805bdd
Submitter: "Zuul (22348)"
Branch: master

commit 966e96951042c0255ea4632bf08b4df768805bdd
Author: Wesley Hayutin <email address hidden>
Date: Tue Sep 21 13:15:50 2021 -0600

    disable protected resources for centos ci deployments

    Related-Change: https://review.opendev.org/c/openstack/tripleo-quickstart/+/807703
    Related-Bug: #1943579
    Change-Id: I30553e352b351a3551f6c2a5c354b91aacf3112d

[Harald Jensås (harald-jensas)]

Question:

Why is there no '--skip-nodes-and-networks' in the overcloud-deploy-command? Both featuresets [2] and [3] set network_provision == true. So the overcloud-deploy.sh should render with that option[1]?

'--skip-nodes-and-networks' should always be used when pre-deploying networks and baremetal nodes.
When it is used the "protected resources" code to add the protected resources won't run. The protected_overrides[4] would remain an empty dict, i.e there should be no conflicts.

[1] https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/overcloud-deploy/templates/overcloud-deploy.sh.j2#L21
[2] https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/general_config/featureset035.yml#L99-L104
[3] https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/general_config/featureset020.yml#L118-L124
[4] https://opendev.org/openstack/python-tripleoclient/src/branch/master/tripleoclient/v1/overcloud_deploy.py#L273

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/811037

[Ronelle Landy (rlandy)]

Also impacting fs030

 https://review.opendev.org/c/openstack/tripleo-quickstart/+/811037 Disable protected resourced types - fs030

[Bhagyashri Shewale (bhagyashri-shewale)]

Also impacting on below jobs:

1. periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-master
2. periodic-tripleo-ci-centos-8-scenario007-multinode-oooq-container-master
3. periodic-tripleo-ci-centos-8-containers-multinode-master

[1]: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-master/9eed6f9/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
[2]: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario007-multinode-oooq-container-master/cd8eb99/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
[3]: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-containers-multinode-master/f0182ff/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

[Grzegorz Grasza (xek)]

We have also seen this on periodic-tripleo-ci-centos-8-containers-multinode-security-master

(afaranha) I got to this, for this one: 2021-09-25 15:39:27 | : tripleoclient.exceptions.DeploymentError: ERROR: Protected resource registry overrides detected! These entries are used in internal environments and should not be overridden in the user environment. Please remove these overrides from the environment files.
(afaranha) https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-containers-multinode-security-master/1cb77f5/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
(afaranha) Conflict detected for resource_registry entry: OS::TripleO::Network::Ports::ControlPlaneVipPort

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/811176

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/811188

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/811188
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/0a513f6984cada29da625412d753939a2bfe1691
Submitter: "Zuul (22348)"
Branch: master

commit 0a513f6984cada29da625412d753939a2bfe1691
Author: Ronelle Landy <email address hidden>
Date: Mon Sep 27 11:52:50 2021 -0400

    Add --disable-protected-resource-types to extra_args

    Removing duplicate exatr_args from cloud
    provider settings will allow this extra_args
    definition to take.

    Related patch:
    https://review.opendev.org/c/openstack/tripleo-ci/+/811176

    Change-Id: Ic21c8101a669a9c917d49177bcd0f38fdd5e674d
    Related-Bug: #1943579

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Sandeep Yadav <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/811176
Reason: tripleo-tox-molecule timed_out, Abandoning to save time. will restore

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ci/+/811176
Committed: https://opendev.org/openstack/tripleo-ci/commit/abbdcfad5b900f4822aa3ada2e1d48dceea14452
Submitter: "Zuul (22348)"
Branch: master

commit abbdcfad5b900f4822aa3ada2e1d48dceea14452
Author: Sandeep Yadav <email address hidden>
Date: Mon Sep 27 20:00:12 2021 +0530

    Replace extra-args with workers_args

    We have assigned extra-args in multiple featureset files[1].
    Using extra-args here override the value we defines in featureset
    files.

    Example: If we compare periodic job deploy command[2] vs
    check job[3] deploy command for same job
    scenario007-multinode-oooq-container - we are not passing
    environments/services/neutron-ovs.yaml because we overriden
    the extra_args from featureset file[4].

    [1] https://codesearch.opendev.org/?q=extra_args&i=nope&literal=nope&files=&excludeFiles=&repos=openstack/tripleo-quickstart
    [2] https://e5dbd9d96734c74a1cf0-391ca9f25982feff002f89e6b05bcdaa.ssl.cf2.rackcdn.com/810256/3/check/tripleo-ci-centos-8-scenario007-multinode-oooq-container/7bc4f34/logs/undercloud/home/zuul/overcloud-deploy.sh
    [3] https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario007-multinode-oooq-container-master/75482de/logs/undercloud/home/zuul/overcloud-deploy.sh.txt.gz
    [4] https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset030.yml#L38-L45

    Related-Bug: #1943579

    Depends-On: https://review.opendev.org/c/openstack/tripleo-quickstart/+/811188

    Change-Id: I71e9a33ff63b004a948852ea4385d1b1ecbd4527

[Ronelle Landy (rlandy)]

we have a fix - closing this out

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Harald Jensås <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/807704

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Ghanshyam <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/811037
Reason: TrieplO project is retiring now, for details, please see https://review.opendev.org/c/openstack/governance/+/905145 or reach out to OpenStack TC.