rocky undercloud fails to install

Bug #1811713 reported by Keith Plant
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
tripleo
Incomplete
High
Unassigned

Bug Description

Description
===========
Undercloud fails to install, error messages reference iscsid failures.

Steps to reproduce
==================
1. Clean install of CentOS
2. Bring all packages up to date then reboot
3. Install python2-tripleo-repos from trunk.rdoproject.org/centos7/current/
   - python2-tripleo-repos-0.0.1-0.20181218212820.a5b709e.el7.noarch.rpm
4. Set repositories for Rocky and Ceph: sudo -E tripleo-repos -b rocky current ceph
5. Install TripleO client and ceph-ansible: sudo yum install -y python-tripleoclient ceph-ansible
6. Paste in undercloud.conf (detailed below)
7. Attempt Undercloud installation: openstack undercloud install

Expected result
===============
Successful undercloud installation

Actual result
=============
Installation failed

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Deployment Failed!

ERROR: Heat log files: /var/log/heat-launcher/undercloud_deploy-tWkpy4

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Deployment failed.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

An error has occured while deploying the Undercloud.

See the previous output for details about what went wrong.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Command '['sudo', 'openstack', 'tripleo', 'deploy', '--standalone', '--standalone-role', 'Undercloud', '--stack', 'undercloud', '--local-domain=localdomain', '--local-ip=192.168.24.1/24', '--templates=/usr/share/openstack-tripleo-heat-templates/', '--heat-native', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/docker.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/undercloud.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/use-dns-for-vips.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/services/masquerade-networks.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/services/ironic.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/services/ironic-inspector.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/services/mistral.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/services/zaqar-swift-backend.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/services/tripleo-ui.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/services/tempest.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/no-tls-endpoints-public-ip.yaml', '--deployment-user', 'stack', '--output-dir=/home/stack', '--cleanup', '-e', '/home/stack/tripleo-config-generated-env-files/undercloud_parameters.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/tripleo-validations.yaml', '--log-file=install-undercloud.log', '-e', '/usr/share/openstack-tripleo-heat-templates/undercloud-stack-vstate-dropin.yaml']' returned non-zero exit status 1
Command '['sudo', 'openstack', 'tripleo', 'deploy', '--standalone', '--standalone-role', 'Undercloud', '--stack', 'undercloud', '--local-domain=localdomain', '--local-ip=192.168.24.1/24', '--templates=/usr/share/openstack-tripleo-heat-templates/', '--heat-native', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/docker.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/undercloud.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/use-dns-for-vips.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/services/masquerade-networks.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/services/ironic.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/services/ironic-inspector.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/services/mistral.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/services/zaqar-swift-backend.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/services/tripleo-ui.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/services/tempest.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/no-tls-endpoints-public-ip.yaml', '--deployment-user', 'stack', '--output-dir=/home/stack', '--cleanup', '-e', '/home/stack/tripleo-config-generated-env-files/undercloud_parameters.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/tripleo-validations.yaml', '--log-file=install-undercloud.log', '-e', '/usr/share/openstack-tripleo-heat-templates/undercloud-stack-vstate-dropin.yaml']' returned non-zero exit status 1

Logs & Configs
==============

Install log: http://paste.openstack.org/show/742344/
I only included the part I thought would be meaningful here as the entire log is far too large for pastebin or paste.openstack.org

undercloud.conf:
[DEFAULT]
undercloud_hostname = chrnc-dev-undercloud-01.dev.chtrse.com
local_interface = bond0.240
local_mtu = 1500
local_ip = 192.168.24.1/24
undercloud_public_host = 192.168.24.2
undercloud_admin_host = 192.168.24.3
undercloud_service_certificate =
generate_service_certificate = False
scheduler_max_attempts = 10

[ctlplane-subnet]
cidr = 192.168.24.0/24
gateway = 192.168.24.1
dhcp_start = 192.168.24.10
dhcp_end = 192.168.24.50
inspection_iprange = 192.168.24.60,192.168.24.100
masquerade = true

Environment
===========
- CentOS Linux release 7.6.1810 x86_64
- OpenStack Rocky
- Undercloud install target is baremetal

Revision history for this message
Alex Schultz (alex-schultz) wrote :

This appears to be affecting current rocky

2019-01-14 12:29:28.726 78587 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] "Notice: /Stage[main]/Tripleo::Profile::Base::Iscsid/Exec[reset-iscsi-initiator-name]/returns: sh: /etc/iscsi/initiatorname.iscsi: Permission denied",

Changed in tripleo:
status: New → Confirmed
importance: Undecided → Critical
importance: Critical → High
milestone: none → stein-3
Revision history for this message
Alex Schultz (alex-schultz) wrote :
Changed in tripleo:
importance: High → Critical
tags: added: alert ci
wes hayutin (weshayutin)
Changed in tripleo:
status: Confirmed → Triaged
Revision history for this message
Alex Schultz (alex-schultz) wrote :

Oh nevermind, that failed for a different reason.

tags: removed: alert ci
Changed in tripleo:
importance: Critical → High
Revision history for this message
Alan Bishop (alan-bishop) wrote :

I am able to reproduce the problem. I traced the problem to an selinux issue, and ultimately to [1], which was recently introduced to stable/rocky.

[1] https://review.openstack.org/625262

If I (manually) undo the patch, the undercloud installs just fine.

Earlier, I thought the new python2-tripleo-repos-0.0.1-0.20190114223251.6470f0c.el7.noarch.rpm behaved differently, but it does not.
- Undercloud install fails the same way (selinux issue with iscsid container during puppet_config)
- Undercloud install succeeds if [1] is removed

Revision history for this message
Alex Schultz (alex-schultz) wrote :

Ok so it's an selinux enforcing problem. You can set undercloud_selinux to permissive and install the undercloud and it should be fine.

Revision history for this message
Alex Schultz (alex-schultz) wrote :

Sorry, set undercloud_enable_selinux to False in undercloud.conf

Revision history for this message
Keith Plant (kplant) wrote :

I was able to successfully complete undercloud installations, without error, using both Alan's and Alex's workarounds.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/rocky)

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/631354

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/631355

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (stable/rocky)

Reviewed: https://review.openstack.org/631354
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=666573d15b44428856bbfe953ac34458d7219dde
Submitter: Zuul
Branch: stable/rocky

commit 666573d15b44428856bbfe953ac34458d7219dde
Author: Emilien Macchi <email address hidden>
Date: Thu Aug 23 12:28:46 2018 -0400

    Create missing directories before mounting them

    When deploying with podman, we need to create directories if they don't
    exist before trying to mount them later when containers are starting.
    Otherwise, podman fails with this kind of error:
    error checking path \"/etc/iscsi\": stat /etc/iscsi: no such file or directory"

    Related-Bug: #1811713
    Change-Id: I7dbdc7f3646dda99c8014b4c8ca2edd48778b392
    (cherry picked from commit 24f5a255c954a94d69768b139a113113d4e37c80)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/631355
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=606ce4bc564e8cd1d5da67f0df15c38782327cd4
Submitter: Zuul
Branch: stable/rocky

commit 606ce4bc564e8cd1d5da67f0df15c38782327cd4
Author: Cédric Jeanneret <email address hidden>
Date: Wed Sep 5 17:28:06 2018 +0200

    Set proper setype for service directories

    This will allow proper access from the containers without any
    new SELinux policy

    Conflicts:
     docker/services/cinder-volume.yaml
     docker/services/haproxy.yaml
     docker/services/nova-api.yaml
     docker/services/swift-storage.yaml
    Related-Bug: #1811713
    Depends-On: Ie9f5d3b6380caa6824ca940ca48ed0fcf6308608
    Change-Id: I284126db5dcf9dc31ee5ee640b2684643ef3a066
    (cherry picked from commit 3eeece2d290eef7ba41e7a82765481fd4e732c12)

Revision history for this message
Alex Schultz (alex-schultz) wrote :

This appears to be an issue with the openstack-selinux package available in rocky.

Revision history for this message
Alex Schultz (alex-schultz) wrote :

openstack-selinux-0.8.15-1.el7ost.noarch works, openstack-selinux-0.8.17-0.20190116193749.faef39f.el7.noarch does not.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/rocky)

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/631869

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/631874

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/rocky)

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/631875

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/631876

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-heat-templates (stable/rocky)

Change abandoned by Alex Schultz (<email address hidden>) on branch: stable/rocky
Review: https://review.openstack.org/631869
Reason: we're just backing out the selinux items

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/rocky)

Reviewed: https://review.openstack.org/631874
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=4208b04741bb3cf581afe55f71be9b4008e18f4e
Submitter: Zuul
Branch: stable/rocky

commit 4208b04741bb3cf581afe55f71be9b4008e18f4e
Author: Alex Schultz <email address hidden>
Date: Fri Jan 18 19:37:40 2019 +0000

    Revert "docker: wire SELinuxMode with Ansible vars"

    This reverts commit 220cb399846fa2e6fd2b650e807d432c092a4b7b.

    Rocky wasn't ready for containers to be selinux enforcing.

    Change-Id: Ia6934902bd4526902a71082c3315d814bed36ca0
    Closes-Bug: #1811713

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (stable/rocky)

Reviewed: https://review.openstack.org/631875
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=7d24a21b892a19c6a56c1ff49d3f9eae68253587
Submitter: Zuul
Branch: stable/rocky

commit 7d24a21b892a19c6a56c1ff49d3f9eae68253587
Author: Alex Schultz <email address hidden>
Date: Fri Jan 18 19:39:14 2019 +0000

    Revert "Set proper setype for service directories"

    This reverts commit 606ce4bc564e8cd1d5da67f0df15c38782327cd4.

    Rather than include this large change to try and get selinux
    working. Let's just back it all out.

    Change-Id: I85ceb7c170fffb88f22cd4ac9f58ed75a733786d
    Related-Bug: #1811713

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/631876
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=d95b5b9a71955137cc9590ba4b6d1bafe3b1b080
Submitter: Zuul
Branch: stable/rocky

commit d95b5b9a71955137cc9590ba4b6d1bafe3b1b080
Author: Alex Schultz <email address hidden>
Date: Fri Jan 18 19:39:44 2019 +0000

    Revert "Create missing directories before mounting them"

    This reverts commit 666573d15b44428856bbfe953ac34458d7219dde.

    Backing this out since we don't need this for Rocky.

    Change-Id: If78667c12a5882c1280e7d24e351fd67dcb23da9
    Related-Bug: #1811713

Changed in tripleo:
milestone: stein-3 → stein-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 9.3.0

This issue was fixed in the openstack/tripleo-heat-templates 9.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/rocky)

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/644257

Changed in tripleo:
milestone: stein-rc1 → train-1
Changed in tripleo:
milestone: train-1 → train-2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-heat-templates (stable/rocky)

Change abandoned by Emilien Macchi (<email address hidden>) on branch: stable/rocky
Review: https://review.opendev.org/644257
Reason: won't fix it for rocky.

Changed in tripleo:
milestone: train-2 → train-3
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.opendev.org/675358

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (stable/queens)

Reviewed: https://review.opendev.org/675358
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=84beb53b79f813b7b974cc6bc3bb36000f1c5de2
Submitter: Zuul
Branch: stable/queens

commit 84beb53b79f813b7b974cc6bc3bb36000f1c5de2
Author: Emilien Macchi <email address hidden>
Date: Thu Aug 23 12:28:46 2018 -0400

    Create missing directories before mounting them

    When deploying with podman, we need to create directories if they don't
    exist before trying to mount them later when containers are starting.
    Otherwise, podman fails with this kind of error:
    error checking path \"/etc/iscsi\": stat /etc/iscsi: no such file or directory"

    Related-Bug: #1811713
    Change-Id: I7dbdc7f3646dda99c8014b4c8ca2edd48778b392
    (cherry picked from commit 24f5a255c954a94d69768b139a113113d4e37c80)

tags: added: in-stable-queens
Changed in tripleo:
milestone: train-3 → ussuri-1
Changed in tripleo:
milestone: ussuri-1 → ussuri-2
wes hayutin (weshayutin)
Changed in tripleo:
milestone: ussuri-2 → ussuri-3
wes hayutin (weshayutin)
Changed in tripleo:
milestone: ussuri-3 → ussuri-rc3
wes hayutin (weshayutin)
Changed in tripleo:
status: Triaged → Incomplete
wes hayutin (weshayutin)
Changed in tripleo:
milestone: ussuri-rc3 → victoria-1
Changed in tripleo:
milestone: victoria-1 → victoria-3
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.