oooq: undercloud lacks network-environment and tls files to deploy overcloud

[Tom Barron]

When deploying via oooq against current master, overcloud-deploy.sh fails because /home/stack/{network_environment.yaml,enable-tls.yaml,inject-trust-anchor.yaml} are missing.

Steps to reproduce:
------------------
1. run ooq with master branch: ./tripleo-quickstart/quickstart.sh -R tripleo-ci/master -T all $HOSTNAME
2. ssh to undercloud and run overcloud_deploy.sh
3. observe that the script fails with:
Error: The following files were not found: /home/stack/network-environment.yaml, /home/stack/enable-tls.yaml, /home/stack/inject-trust-anchor.yaml

Expected Result
---------------
overcloud deploy will succeed

Etherpad with fuller details:
----------------------------

https://paste.fedoraproject.org/paste/FV~ydXUF1~UNDU4ln87AmA

ooq info
--------
[test@ibm-x3550m3-15 tripleo-quickstart]$ git status
# On branch master
nothing to commit, working directory clean
[test@ibm-x3550m3-15 tripleo-quickstart]$ git show HEAD
commit b800002c133a1adb95fa1fcf68c5df701b78e7e0
Merge: 60d66f3 cdbed04
Author: Zuul <email address hidden>
Date: Sat Dec 9 06:08:48 2017 +0000

[Tom Barron (tpb)]

Probably can work around this by copying these files from old undercloud deployments but this bug would seem to block developers who don't have a reference undercloud to work from.

[Carlos Camacho (ccamacho)]

From my dev box here you have them:

https://paste.fedoraproject.org/paste/So4YONWevuCl7mFuZvkX0g

[Tom Barron (tpb)]

Carlos, let me see if I can reproduce this today and if not we can close this as invalid. IIRC I was deploying with a ceph storage node when I hit this.

[Tom Barron (tpb)]

Here's a current repro:

[test@sealusa16 ~]$ cat mynodes.yml
control_memory: 8192
compute_memory: 4096
ceph_memory: 4096

undercloud_memory: 12288
undercloud_vcpu: 4

default_vcpu: 2

node_count: 7

overcloud_nodes:
  - name: control_0
    flavor: control
    virtualbmc_port: 6230

  - name: control_1
    flavor: control
    virtualbmc_port: 6231

  - name: control_2
    flavor: control
    virtualbmc_port: 6232

  - name: compute_0
    flavor: compute
    virtualbmc_port: 6240

  - name: ceph_0
    flavor: ceph
    virtualbmc_port: 6250

  - name: ceph_1
    flavor: ceph
    virtualbmc_port: 6251

  - name: ceph_2
    flavor: ceph
    virtualbmc_port: 6252

topology: >-
  --control-scale 3
  --compute-scale 1
  --ceph-storage-scale 3

extradisks_list:
  - vdb
  - vdc
  - vdd
  - vde
  - vdf
  - vdg
[test@sealusa16 ~]$ ./quickstart.sh -R tripleo-ci/master -T all -S tripleoui-validate -N mynodes.yml -X 127.0.0.2 |& tee quickstart_sh.log
...
    ssh -F /home/test/.quickstart/ssh.config.ansible undercloud

Follow the documentation in the link below to complete your deployment.

    http://ow.ly/c44w304begR

##################################
Virtual Environment Setup Complete
[test@sealusa16 ~]$ ssh -F /home/test/.quickstart/ssh.config.ansible undercloud
Warning: Permanently added '127.0.0.2' (ECDSA) to the list of known hosts.
Warning: Permanently added 'undercloud' (ECDSA) to the list of known hosts.
Last login: Thu Feb 8 02:11:14 2018 from gateway
[stack@undercloud ~]$ ls
cloud-names.yaml overcloud-full.initrd overcloud_prep_network.log
hieradata-overrides-classic-undercloud.yaml overcloud-full.qcow2 overcloud-prep-network.sh
hieradata-overrides-t-h-t-undercloud.yaml overcloud-full.vmlinuz stackrc
instackenv.json overcloud-prep-containers.sh undercloud.conf
ironic-python-agent.initramfs overcloud_prep_flavors.log undercloud_install.log
ironic-python-agent.kernel overcloud-prep-flavors.sh undercloud-install.sh
overcloud-deploy-post.sh overcloud_prep_images.log undercloud-parameter-defaults.yaml
overcloud-deploy.sh overcloud-prep-images.sh undercloud-passwords.conf
[stack@undercloud ~]$

[Ricardo Noriega (rnoriega-b)]

I keep hitting this issue very frequently:

$ bash quickstart.sh --release master --config ~/.quickstart/config/general_config/containers_minimal.yml --nodes config/nodes/3ctlr_1comp.yml 127.0.0.2

(undercloud) [stack@undercloud ~]$ ll
total 2434404
-rw-rw-r--. 1 stack stack 357 Feb 13 19:30 cloud-names.yaml
-rw-rw-r--. 1 stack stack 7915 Feb 13 20:13 containers-default-parameters.yaml
-rw-rw-r--. 1 stack stack 1419938 Feb 13 20:13 docker_journalctl.log
-rw-rw-r--. 1 stack stack 1060 Feb 14 12:13 enable-fencing.yaml
-rw-------. 1 stack stack 550 Feb 13 19:08 hieradata-overrides-classic-undercloud.yaml
-rw-------. 1 stack stack 474 Feb 13 19:08 hieradata-overrides-t-h-t-undercloud.yaml
-rw-rw-r--. 1 stack stack 1898 Feb 13 19:30 inject-trust-anchor.yaml
-rw-rw-r--. 1 stack stack 1745 Feb 13 19:05 instackenv.json
-rw-rw-r--. 1 stack stack 377490788 Feb 13 19:05 ironic-python-agent.initramfs
-rwxr-xr-x. 1 stack stack 5890720 Feb 13 19:05 ironic-python-agent.kernel
-rw-rw-r--. 1 stack stack 4097 Feb 13 20:13 overcloud_containers.yaml
-rwxr-xr-x. 1 stack stack 1109 Feb 13 19:30 overcloud-deploy-post.sh
-rwxr-xr-x. 1 stack stack 3411 Feb 14 12:22 overcloud-deploy.sh
-rw-r--r--. 1 stack stack 52126866 Feb 13 19:05 overcloud-full.initrd
-rw-r--r--. 1 stack stack 2049507328 Feb 13 19:05 overcloud-full.qcow2
-rwxr-xr-x. 1 stack stack 5890720 Feb 13 19:05 overcloud-full.vmlinuz
-rwxr-xr-x. 1 stack stack 1586 Feb 13 19:24 overcloud-prep-containers.sh
-rw-rw-r--. 1 stack stack 10760 Feb 13 19:29 overcloud_prep_flavors.log
-rwxr-xr-x. 1 stack stack 3672 Feb 13 19:28 overcloud-prep-flavors.sh
-rw-rw-r--. 1 stack stack 7571 Feb 13 19:28 overcloud_prep_images.log
-rwxr-xr-x. 1 stack stack 643 Feb 13 19:24 overcloud-prep-images.sh
-rw-rw-r--. 1 stack stack 4225 Feb 13 19:29 overcloud_prep_network.log
-rwxr-xr-x. 1 stack stack 1459 Feb 13 19:29 overcloud-prep-network.sh
-rw-------. 1 stack stack 1233 Feb 13 19:22 stackrc
-rw-------. 1 stack stack 9393 Feb 13 19:08 undercloud.conf
-rw-rw-r--. 1 stack stack 6229 Feb 13 19:09 undercloud-containers-default-parameters.yaml
-rw-rw-r--. 1 stack stack 339186 Feb 13 19:23 undercloud_install.log
-rwxr-xr-x. 1 stack stack 153 Feb 13 19:08 undercloud-install.sh
-rw-------. 1 stack stack 526 Feb 13 19:08 undercloud-parameter-defaults.yaml
-rw-------. 1 stack stack 1994 Feb 13 19:09 undercloud-passwords.conf
-rw-rw-r--. 1 stack stack 12312 Feb 13 19:09 undercloud_prep_containers.log
-rwxr-xr-x. 1 stack stack 1478 Feb 13 19:08 undercloud-prep-containers.sh

network-environment.yaml is not present.
Of course, the latest patch I have is:

be788f02e85a8bca8249194f331360380c737055

[wes hayutin (weshayutin)]

Ricardo,
So afaict the issue is the ansible tags. You are running quickstart with the default tags and Tom is running w/ all. I'll look into it this morning and put in a fix.

Until a fix is merged, you could also do the following.
If you would like to be more precise in each step, I would execute quickstart via the following docs https://docs.openstack.org/tripleo-quickstart/latest/getting-started.html where you execute each step individually and w/ all the tags.

[Bogdan Dobrelya (bogdando)]

Please disregard my comment for https://review.openstack.org/#/c/540377/1/roles/undercloud-deploy/defaults/main.yml@82 . That patch has nothing to overcloud deployments. I misunderstood the bug as if it was for the missing undercloud network configuration config used to deploy *undercloud* and not overcloud.

[Ricardo Noriega (rnoriega-b)]

The issue seems to happen with containers_minimal.yaml config file.

[Miguel Angel Ajo (mangelajo)]

I'm experiencing this, my quickstart settings were:

  bash ./quickstart.sh --teardown all --release master-tripleo-ci \
   --nodes config/nodes/3ctlr_1comp.yml \
   --config config/general_config/pacemaker.yml 127.0.0.2

[Juan Badia Payno (jbadiapa)]

By default the executed tags for ./quickstart.sh are:
 untagged,provision,environment,libvirt,undercloud-scripts,undercloud-inventory,overcloud-scripts,undercloud-setup,undercloud-install,undercloud-post-install,tripleoui-validate

To have the /home/stack/network-environment.yaml on the undercloud. The tag "overcloud-prep-config"
is needed.

Adding the following arguments on the ./quickstart.sh script worked for me:
 -t untagged,provision,environment,libvirt,undercloud-scripts,undercloud-inventory,overcloud-scripts,undercloud-setup,undercloud-install,undercloud-post-install,tripleoui-validate,overcloud-prep-config

[Giulio Fidente (gfidente)]

Can we add overcloud-prep-config into the default tags list then?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/553895

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/553892
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=4fbcf0f078604b16de279431c8d306713653f1d8
Submitter: Zuul
Branch: master

commit 4fbcf0f078604b16de279431c8d306713653f1d8
Author: Wes Hayutin <email address hidden>
Date: Fri Mar 16 15:51:37 2018 -0400

    consolidate tq configurations

    There are several configurations that are no longer
    very well tested due to limitation in ci.centos.
    Consolidating on configurations that are tested in upstream
    and rdo.

    The patch rebases minimal and pacemaker configs to be
    exactly like fs001 w/ single-nic-vlans instead of
    multiple nics

    Related-Bug: #1737602
    Change-Id: Id26eb558800b8fccc5e5f2c2f9684545e445bc6d

[Bogdan Dobrelya (bogdando)]

To generate the network config as a separate step w/o affecting the provisioned nodes or installed undercloud, the example command with the proper tags is:

./quickstart.sh --retain-inventory --teardown none --tags 'overcloud-prep-config,overcloud-scripts,undercloud-inventory' -R queens --nodes oooq/nodes.yaml --config oooq/config.yaml 127.0.0.2

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/572107

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/572107
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=60229bdded1535eaf2a9bbaafa0633162df8df28
Submitter: Zuul
Branch: master

commit 60229bdded1535eaf2a9bbaafa0633162df8df28
Author: Sagi Shnaidman <email address hidden>
Date: Mon Jun 4 15:31:42 2018 +0200

    Add generation of network yaml to default tags

    Generate network yaml file as part of overcloud scripts generation.

    Resolves-Bug: #1737602
    Change-Id: I5ca4f6aef34aeebda10809857804d7e3f0d22d47

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/553895
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=fff7bd3060668d5e395a04026e87e7b55aa3028f
Submitter: Zuul
Branch: master

commit fff7bd3060668d5e395a04026e87e7b55aa3028f
Author: Wes Hayutin <email address hidden>
Date: Fri Mar 16 16:21:06 2018 -0400

    add default overcloud-prep tags

    Users may not know about which tags to use or not use.
    Users may also not be running in a step by step method.
    Allow users three basic options.

    1. machine prep and undercloud
    2. machine prep, undercloud, and overcloud prep
    3. all steps

    Resolves-Bug: #1737602
    Change-Id: Ie337e465c5a24db7c18fbb61f2ecb4e1dd258c23