stable/mitaka to newton undercloud upgrade fails with various errors related to o-p-m packaging

[Marios Andreou]

[UPDATE 05 July 2016]:

Updating to reflect that this will track a number of errors that were seen here and that were all related to the openstack-puppet-modules packaging (and fixed by reinstalling it as commented below). The bugs and errors are:

this.bug, was previously for tracking "Error: Could not find class ::tripleo::selinux for instack on node instack"
https://bugs.launchpad.net/tripleo/+bug/1594890 "Error: Could not find class ::ironic::drivers::deploy for instack on node instack "
https://bugs.launchpad.net/tripleo/+bug/1594893 "Error: Invalid parameter pxe_bootfile_name on Class[Ironic::Drivers::Pxe] at /etc/puppet/manifests/puppet-stack-config.pp:437 on node instack"
https://bugs.launchpad.net/tripleo/+bug/1594895 "Error: Invalid parameter dport on Tripleo::Firewall::Rule[101 mongodb_config] on node instack"
https://bugs.launchpad.net/tripleo/+bug/1594896 "Error: Invalid parameter destination on Tripleo::Firewall::Rule[140 network cidr nat] on node instack"

Those bugs will all be closed as duplicated pointing here.

Original bug description, for the 'selinux' issue follows.
****************************************

tl;dr do we need to update puppet-tripleo and release openstack-puppet-modules because '::tripleo::selinux' ? The bug @ https://bugs.launchpad.net/tripleo/+bug/1593182 also needs o-p-m update so this may get fixed if that happens there.

During upgrade of a stable/mitaka environment to latest newton (delorean repos), and after applying a fix for bug https://bugs.launchpad.net/tripleo/+bug/1593182, the "openstack undercloud upgrade" fails with:

    2016-06-17 11:09:51 - Error: Could not find class ::tripleo::selinux for instack on node instack
    2016-06-17 11:09:51 - Error: Could not find class ::tripleo::selinux for instack on node instack
    + rc=1
    + set -e
    + echo 'puppet apply exited with exit code 1'
    puppet apply exited with exit code 1
    + '[' 1 '!=' 2 -a 1 '!=' 0 ']'
    + exit 1
    [2016-06-17 11:09:51,792] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run- parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1]

    [2016-06-17 11:09:51,792] (os-refresh-config) [ERROR] Aborting...
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1039, in install
        _run_orc(instack_env)
    File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 917, in _run_orc
      _run_live_command(args, instack_env, 'os-refresh-config')
      File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 450, in _run_live_command
        raise RuntimeError('%s failed. See log for details.' % name)
RuntimeError: os-refresh-config failed. See log for details.
    Command 'instack-install-undercloud' returned non-zero exit status 1

some combination of grep and git blame told me that '::tripleo::selinux' was introduced by

"Add tripleo::selinux " https://review.openstack.org/#/c/309408/ If214005df733d41c2fa4e197df247d8a14baaa14 puppet-tripleo

so I _think_ we need an update of puppet-tripleo in the openstack-puppet-modules and a new release of o-p-m? Note that https://bugs.launchpad.net/tripleo/+bug/1593182 also requires an update of openstack-puppet-modules (but for a puppet-nova fix).

This env has

    Version : 8.1.1
    Release : 0.20160609150428.ab63b38.el7.centos
    Size : 14 M
    Repo : installed
    From repo : delorean
    Summary : Puppet modules used to deploy OpenStack

For now I worked around this by manually patching tripleo in the openstack-puppet-modules

    https://review.openstack.org/changes/309408/revisions/8e533aaf447022c62865130f2ffc88690f06aef1/patch?zip

    sudo su
    yum install unzip
    # backup the templates incase:
    cp -r /usr/share/openstack-puppet/modules/tripleo /usr/share/openstack-puppet/modules/tripleo.BACKUP
    pushd /usr/share/openstack-puppet/modules/tripleo

    # Add tripleo::selinux https://review.openstack.org/#/c/309408/
    curl -o "patch1.diff.zip" https://review.openstack.org/changes/309408/revisions/8e533aaf447022c62865130f2ffc88690f06aef1/patch?zip
    unzip patch1.diff.zip
    patch -p1 < 8e533aaf.diff

    popd
    exit # the sudo su

and with that the openstack undercloud upgrade got further (but hitting another issue, will file another bug for that \o/ ... )

[Marios Andreou (marios-b)]

I am having trouble understanding how rdo openstack-puppet-modules is pulling in the puppet-tripleo module (more below) so a summary here I can point at for help; I have openstack-puppet-modules-8.1.1-0.20160609150428.ab63b38.el7.centos.noarch after the yum update with latest repos, but still don't have the selinux class in /usr/share/openstack-puppet/modules/tripleo/manifests/. That is, the change at https://review.openstack.org/#/c/309408/ isn't included in that package build it seems/afaics.

To reiterate, I deployed stable/mitaka. Then to upgrade, I disable stable/mitaka, enable latest repos using tripleo.sh, stop all services (thanks bnemec) like https://review.openstack.org/#/c/331804 and then yum update:

        sudo systemctl stop openstack-*
        sudo systemctl stop neutron-*

        # Depending on when the base image was created, the pinned delorean repo may
        # actually be older, which causes yum to ignore its metadata.
        sudo rm -f /etc/yum.repos.d/delorean*
        sudo yum clean all && sudo yum clean metadata && sudo yum clean dbcache && sudo yum makecache
        if ! [[ -d ~/tripleo-ci ]]; then git clone http://github.com/openstack-infra/tripleo-ci.git ~/tripleo-ci ;fi;
        ./tripleo-ci/scripts/tripleo.sh --repo-setup
        nice -n -10 sudo yum update -y

        openstack undercloud upgrade

        2016-06-22 10:48:01 - Error: Could not find class ::tripleo::selinux for instack on node instack
        2016-06-22 10:48:01 - Error: Could not find class ::tripleo::selinux for instack on node instack

I assume the openstack-puppet-modules package I get from the latest delorean repos (from ./tripleo-ci/scripts/tripleo.sh --repo-setup) is made via https://github.com/rdo-packages/openstack-puppet-modules-distgit

The version I have is openstack-puppet-modules-8.1.1-0.20160609150428.ab63b38.el7.centos.noarch - I can confirm that there is no selinux.pp in the installed code:

        [stack@instack ~]$ ll /usr/share/openstack-puppet/modules/tripleo/manifests/
        total 96
        drwxr-xr-x. 2 root root 4096 Jun 14 13:15 cluster
        -rw-r--r--. 1 root root 1559 Jun 9 15:05 fencing.pp
        drwxr-xr-x. 2 root root 4096 Jun 14 13:15 firewall
        -rw-r--r--. 1 root root 2418 Jun 9 15:05 firewall.pp
        -rw-r--r--. 1 root root 688 Jun 9 15:05 init.pp
        -rw-r--r--. 1 root root 51401 Jun 9 15:05 loadbalancer.pp
        drwxr-xr-x. 3 root root 4096 Jun 14 13:15 network
        -rw-r--r--. 1 root root 1632 Jun 9 15:05 noop.pp
        -rw-r--r--. 1 root root 2477 Jun 9 15:05 packages.pp
        drwxr-xr-x. 4 root root 4096 Jun 14 13:15 profile
        -rw-r--r--. 1 root root 1133 Jun 9 15:05 redis_notification.pp
        drwxr-xr-x. 2 root root 4096 Jun 14 13:15 ssl

My question is, what version of https://github.com/openstack/puppet-tripleo is pulled into that openstack-puppet-modules package and how is that governed? Do we need to add a version constraint to https://github.com/rdo-packages/openstack-puppet-modules-distgit/blob/9d3bcf2d35c4b816fb310ec74c2aa1ad008ba767/openstack-puppet-modules.spec#L74 ?

One other thing I noticed - does it matter that there is a difference b...

[Marios Andreou (marios-b)]

so... it looks like you have to remove and re-install the openstack-puppet-modules package alltogether in order to pull in the dependencies properly, as listed in the https://github.com/rdo-packages/openstack-puppet-modules-distgit/blob/9d3bcf2d35c4b816fb310ec74c2aa1ad008ba767/openstack-puppet-modules.spec#L74

At least when I removed and re-installed openstack-puppet-modules, it pulled in all the puppet-* things, like puppet-tripleo @ v 5.0.0 - am trying to see how far I can now get in the undercloud upgrade... but at least it looks like it has pulled in the selinux class this time:
        [stack@instack ~]$ ll /usr/share/openstack-puppet/modules/tripleo/manifests/selinux.pp
-rw-r--r--. 1 root root 2466 May 10 14:15 /usr/share/openstack-puppet/modules/tripleo/manifests/selinux.pp

        [stack@instack ~]$ sudo yum remove openstack-puppet-modules
        [stack@instack ~]$ sudo yum install -y openstack-puppet-modules python-tripleoclient instack-undercloud

         puppet-tripleo noarch 5.0.0-0.20160610144302.3ea4259.el7.centos delorean 59 k

        [stack@instack ~]$ yum info openstack-puppet-modules
        Loaded plugins: fastestmirror, priorities
        Loading mirror speeds from cached hostfile
         * base: mirrors.coreix.net
         * centos7: mirrors.coreix.net
         * epel: nl.mirror.babylon.network
         * extras: mirrors.coreix.net
         * updates: mirrors.coreix.net
        618 packages excluded due to repository priority protections
        Installed Packages
        Name : openstack-puppet-modules
        Arch : noarch
        Epoch : 1
        Version : 8.0.0
        Release : 0.20160520142355.6a3e8bf.el7.centos
        Size : 0.0
        Repo : installed
        From repo : delorean
        Summary : Puppet modules to deploy OpenStack
        URL : https://github.com/redhat-openstack/openstack-puppet-modules
        License : Apache-2.0
        Description : Metapackage for OpenStack Puppet Modules

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/332889

[Emilien Macchi (emilienm)]

I confirm undercloud upgrade now works.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Marios Andreou (<email address hidden>) on branch: master
Review: https://review.openstack.org/332889
Reason: this was fixed in packaging