Overcloud deployment fails with permission issues

[Lance Bragstad]

I'm noticing the following issue when I deploy and overcloud:

    /usr/share/ansible/tripleo-playbooks/cli-grant-local-access.yaml:34 -----------
    Check for required inputs ----------------------------------------------- 0.29s
    /usr/share/ansible/tripleo-playbooks/cli-grant-local-access.yaml:24 -----------
    Temporary directory [ /tmp/tripleo2musqho6 ] cleaned up
    Ansible execution success. playbook: /usr/share/ansible/tripleo-playbooks/cli-grant-local-access.yaml
    Temporary directory [ /tmp/tripleo5e8rsika ] cleaned up
    Checking for blacklisted hosts from stack: overcloud
    Retrieving configuration for stack: overcloud
    Generating configuration under the directory: /tmp/tripleo-0ai0bzyt-config
    Getting deployment data from Heat...
    The TripleO configuration has been successfully generated into: /tmp/tripleo-0ai0bzyt-config
    Snapshotting /tmp/tripleo-0ai0bzyt-config
    Created commit 32db3d535e07a1e0f743d29a958435809d2b0d32
    Downloading configuration for stack: overcloud
    Retrieving keyfile for stack: overcloud
    Generating information for stack: overcloud
    Executing deployment playbook for stack: overcloud
    Running Ansible playbook with timeout 231m: /var/lib/mistral/overcloud/deploy_steps_playbook.yaml, Working directory: /var/lib/mistral/overcloud/deploy_steps_playbook.yaml, Playbook directory: /tmp/tripleo3oywx_dv
    Temporary directory [ /tmp/tripleo0clr93xn ] cleaned up
    Temporary directory [ /tmp/tripleo3oywx_dv ] cleaned up
    Exception occured while running the command
    Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 34, in run
        super(Command, self).run(parsed_args)
      File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
        return super(Command, self).run(parsed_args)
      File "/usr/lib/python3.6/site-packages/cliff/command.py", line 187, in run
        return_code = self.take_action(parsed_args) or 0
      File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1052, in take_action
        deployment_timeout=timeout
      File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/deployment.py", line 496, in config_download
        timeout=deployment_timeout,
      File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 644, in run_ansible_playbook
        os.chmod(command_path, 0o750)
    PermissionError: [Errno 1] Operation not permitted: '/var/lib/mistral/overcloud/ansible-playbook-command.sh'
    [Errno 1] Operation not permitted: '/var/lib/mistral/overcloud/ansible-playbook-command.sh'
    END return value: 1
    (undercloud) [stack@undercloud ~]$

I originally stood up the environment without any issues and I'm seeing the above issue during a redeployment of the overcloud.

I'm installing the overcloud using:

$ openstack overcloud deploy \
--verbose \
--templates /usr/share/openstack-tripleo-heat-templates \
--ntp-server clock.redhat.com \
--deployed-server \
--disable-validations \
--overcloud-ssh-user stack \
--overcloud-ssh-key ~/.ssh/id_rsa \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-everywhere-endpoints-dns.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/haproxy-public-tls-certmonger.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-internal-tls.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/deployed-server-environment.yaml \
-e ~/container-parameters.yaml \
-e ~/parameters.yaml

These are the parameters I'm using in parameters.yaml:

[stack@undercloud ~]$ cat parameters.yaml

resource_registry:
  OS::TripleO::Services::IpaClient: /usr/share/openstack-tripleo-heat-templates/deployment/ipa/ipaservices-baremetal-ansible.yaml
  OS::TripleO::Controller::Net::SoftwareConfig: /usr/share/openstack-tripleo-heat-templates/net-config-static-bridge.yaml
  OS::TripleO::Compute::Net::SoftwareConfig: /usr/share/openstack-tripleo-heat-templates/net-config-static-bridge.yaml
  OS::TripleO::DeployedServer::ControlPlanePort: /usr/share/openstack-tripleo-heat-templates/deployed-server/deployed-neutron-port.yaml
  OS::TripleO::Network::Ports::ControlPlaneVipPort: /usr/share/openstack-tripleo-heat-templates/deployed-server/deployed-neutron-port.yaml
  OS::TripleO::Network::Ports::RedisVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
  OS::TripleO::Network::Ports::OVNDBsVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml

parameter_defaults:
  RootStackName: overcloud
  ControllerCount: 1
  ComputeCount: 1
  NtpServer:
    - clock.redhat.com
    - clock2.redhat.com
  EC2MetadataIp: 192.168.24.1
  ControlPlaneDefaultRoute: 192.168.24.15
  DockerInsecureRegistryAddress:
    - undercloud.ctlplane.ooo.test:8787
  # This corresponds to the provision network interface.
  NeutronPublicInterface: eth1

  DnsSearchDomains: ["ooo.test", "lbragstad-master-tls-pre-prov.test"]
  IdMServer: ipa.lbragstad-master-tls-pre-prov.test
  IdMDomain: lbragstad-master-tls-pre-prov.test
  DnsServers: ["192.168.1.13"]
  CloudDomain: ooo.test
  CloudName: overcloud.ooo.test
  CloudNameInternal: overcloud.internalapi.ooo.test
  CloudNameStorage: overcloud.storage.ooo.test
  CloudNameStorageManagement: overcloud.storagemgmt.ooo.test
  CloudNameCtlplane: overcloud.ctlplane.ooo.test

  KeystoneLDAPDomainEnable: true
  KeystoneLDAPBackendConfigs:
    tripleo:
      url: ldap://192.168.1.13
      user: uid=admin,cn=users,cn=accounts,dc=ooo,dc=test
      password: password
      suffix: dc=ooo,dc=test
      user_tree_dn: cn=users,cn=accounts,dc=ooo,dc=test
      user_objectclass: person
      user_id_attribute: cn

  # This maps the hostname values from each role to the hostname for each server.
  # For example, the default value for hostnames in the Controller role is
  # overcloud-controller-$N, where $N is the controller number (e.g., 0, 1, 2).
  # That value needs to map to the actual host name specified in Upshift. If you
  # supply an override to the hostname through role data, you need to update this
  # map. It's setup to assume the defaults. If this mapping isn't correct, the
  # overcloud install will fail because the hosts aren't discoverable.
  HostnameMap:
    overcloud-controller-0: controller-0
    overcloud-novacompute-0: compute-0

  # This is a list of all the infrastructure we want to use for the
  # pre-provisioned deployment. Each IP address is associated to the
  # provisioning netowrk in Upshift. This ensures the overcloud and undercloud
  # are using the same interface to communicate.
  DeployedServerPortMap:
    control_virtual_ip:
      fixed_ips:
        - ip_address: 192.168.24.100
      subnets:
        - cidr: 192.168.24.0/24
      network:
        tags:
          - 192.168.24.0/24
    controller-0-ctlplane:
      fixed_ips:
        - ip_address: 192.168.24.16
      subnets:
        - cidr: 192.168.24.0/24
      network:
        tags:
          - 192.168.24.0/24
    compute-0-ctlplane:
      fixed_ips:
        - ip_address: 192.168.24.10
      subnets:
        - cidr: 192.168.24.0/24
      network:
        tags:
          - 192.168.24.0/24