cephadm_command.sh does not work due to tripleo_common package missing

[Juan Badia Payno]

I was not able to deploy a full master centos8-stream deployment: (3xcontroller 1xcompute 3xceph).

It seems to me that the package tripleo-common is not installed on the controller-0.
The package tripleo-common was not installed on the controller-0.

The error at
/home/stack/overcloud-deploy/overcloud/config-download/overcloud/cephadm/cephadm_command.log

    2022-02-17 16:12:29,171 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.171379 | 00bbf8a1-5ffa-5929-be30-0000000000c6 | TASK | Get certificate from remote node if needed
    2022-02-17 16:12:29,187 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.186777 | 00bbf8a1-5ffa-5929-be30-0000000000c6 | SKIPPED | Get certificate from remote node if needed | overcloud-controller-0
    2022-02-17 16:12:29,200 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.200123 | 00bbf8a1-5ffa-5929-be30-0000000000c7 | TASK | Create the RGW Daemon spec definition
    2022-02-17 16:12:29,487 p=157229 u=stack n=ansible | An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ModuleNotFoundError: No module named 'tripleo_common'
    2022-02-17 16:12:29,487 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.487677 | 00bbf8a1-5ffa-5929-be30-0000000000c7 | FATAL | Create the RGW Daemon spec definition | overcloud-controller-0 | error={"changed": false, "module_stderr": "Traceback (most recent call last):\n File \"<stdin>\", line 102, in <module>\n File \"<stdin>\", line 94, in _ansiballz_main\n File \"<stdin>\", line 40, in invoke_module\n File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_ceph_mkspec_payload_yc7boarj/ansible_ceph_mkspec_payload.zip/ansible/modules/ceph_mkspec.py\", line 24, in <module>\nModuleNotFoundError: No module named 'tripleo_common'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
    2022-02-17 16:12:29,490 p=157229 u=stack n=ansible | PLAY RECAP *********************************************************************
    2022-02-17 16:12:29,490 p=157229 u=stack n=ansible | overcloud-controller-0 : ok=26 changed=10 unreachable=0 failed=1 skipped=30 rescued=0 ignored=0
    2022-02-17 16:12:29,491 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.490829 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    2022-02-17 16:12:29,491 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.491079 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 57 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    2022-02-17 16:12:29,491 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.491359 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:03:27.663940 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    2022-02-17 16:12:29,491 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.491605 | UUID | Info | Host | Task Name | Run Time
    2022-02-17 16:12:29,492 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.491854 | 00bbf8a1-5ffa-5929-be30-00000000028d | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Wait for expected number of osds to be running | 93.63s
    2022-02-17 16:12:29,492 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.492095 | 00bbf8a1-5ffa-5929-be30-000000000057 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Run cephadm bootstrap | 52.47s
    2022-02-17 16:12:29,492 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.492356 | 00bbf8a1-5ffa-5929-be30-0000000000b6 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Set trash interval | 31.99s
    2022-02-17 16:12:29,492 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.492617 | 00bbf8a1-5ffa-5929-be30-000000000083 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Apply spec | 9.89s
    2022-02-17 16:12:29,493 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.492856 | 00bbf8a1-5ffa-5929-be30-0000000000b3 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Create pool(s) | 9.57s
    2022-02-17 16:12:29,493 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.493090 | 00bbf8a1-5ffa-5929-be30-000000000014 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Install cephadm package | 1.74s
    2022-02-17 16:12:29,493 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.493350 | 00bbf8a1-5ffa-5929-be30-000000000080 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Get the ceph orchestrator status | 0.99s
    2022-02-17 16:12:29,493 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.493577 | 00bbf8a1-5ffa-5929-be30-00000000022f | SUMMARY | overcloud-controller-0 | tripleo_cephadm : set default container image in ceph configuration | 0.95s
    2022-02-17 16:12:29,493 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.493812 | 00bbf8a1-5ffa-5929-be30-000000000231 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : set container image base in ceph configuration | 0.86s
    2022-02-17 16:12:29,494 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.494042 | 00bbf8a1-5ffa-5929-be30-000000000017 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : List Ceph daemon instances on this host | 0.53s
    2022-02-17 16:12:29,494 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.494298 | 00bbf8a1-5ffa-5929-be30-000000000022 | SUMMARY | overcloud-controller-0 | push tripleo_cephadm_spec to bootstrap node if spec file exsits | 0.50s
    2022-02-17 16:12:29,494 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.494522 | 00bbf8a1-5ffa-5929-be30-000000000053 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Stat necessary files to bootstrap with cephadm | 0.36s
    2022-02-17 16:12:29,494 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.494746 | 00bbf8a1-5ffa-5929-be30-00000000007d | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Stat spec file on bootstrap node | 0.33s
    2022-02-17 16:12:29,495 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.494975 | 00bbf8a1-5ffa-5929-be30-000000000027 | SUMMARY | overcloud-controller-0 | generate tripleo_cephadm_assimilate_conf on bootstrap node | 0.31s
    2022-02-17 16:12:29,495 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.495223 | 00bbf8a1-5ffa-5929-be30-000000000015 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Stat cephadm file | 0.30s
    2022-02-17 16:12:29,495 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.495460 | 00bbf8a1-5ffa-5929-be30-0000000000c7 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Create the RGW Daemon spec definition | 0.29s
    2022-02-17 16:12:29,495 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.495708 | 00bbf8a1-5ffa-5929-be30-00000000001f | SUMMARY | overcloud-controller-0 | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists | 0.24s
    2022-02-17 16:12:29,496 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.495949 | 00bbf8a1-5ffa-5929-be30-000000000055 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Stat pre ceph conf file in case we should bootrap with it | 0.19s
    2022-02-17 16:12:29,496 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.496201 | 00bbf8a1-5ffa-5929-be30-000000000020 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Ensure specs directory exists | 0.18s
    2022-02-17 16:12:29,496 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.496433 | 00bbf8a1-5ffa-5929-be30-000000000016 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Fail if cephadm is not available | 0.07s
    2022-02-17 16:12:29,496 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.496656 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    2022-02-17 16:12:29,497 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.496906 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    2022-02-17 16:12:29,497 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.497139 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~
    2022-02-17 16:12:29,497 p=157229 u=stack n=ansible | 2022-02-17 16:12:29.497415 | The following node(s) had failures: overcloud-controller-0

[John Fulton (jfulton-org)]

WORKAROUND: install tripleo-common package on overcloud

[John Fulton (jfulton-org)]

These changes caused the issue and CI didn't catch it because it was using standalone only:

1. Add ceph_mkspec.py to tripleo-common:
   https://review.opendev.org/c/openstack/tripleo-common/+/824203

2. Remove ceph_mkspec.py from tripleo-ansible and call it from tripleo-common:
   https://review.opendev.org/c/openstack/tripleo-ansible/+/822829

3. Update tripleo-client to use ceph_mkspec.py:
   https://review.opendev.org/c/openstack/python-tripleoclient/+/822835

If you revert 1 and 2, then you'll break 3. We could revert 1 and 2, and change 3 so it's stuck with only the ansible interface to ceph_mkspec.py.

Part of the motivation for moving that library was that ansible wouldn't be there and that the tripleo client would need to be smarter. But we seem to have moved away from that goal.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/831019

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/python-tripleoclient/+/831023

[John Fulton (jfulton-org)]

This bug can be resolved by merging the following patches in the following order:

1. https://review.opendev.org/c/openstack/tripleo-ansible/+/830573
2. https://review.opendev.org/c/openstack/tripleo-ansible/+/831019
3. https://review.opendev.org/c/openstack/python-tripleoclient/+/831023
4. https://review.opendev.org/c/openstack/tripleo-common/+/830572

The strategy is to:

- add the ceph_spec library back to tripleo-ansible with 1
- give tripleo-ansible the ability to create a standalone ceph_spec with 2
- have tripleo-client call tripleo-ansible to create the standalone ceph_spec with 3
- clean up the redundant (after the above merge) copy of the ceph_spec library with 4

The patches have the following relations to each other in gerrit:

- 4 depends-on 3
- 3 depends-on 2
- 2 is stacked on top of 1 (same repo)

These relations are used since when 1 and 2 merge they shouldn't break anything and when 3 merges the bug will technically be fixed, though we need 4 to clean up.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/830573
Committed: https://opendev.org/openstack/tripleo-ansible/commit/afd3374f46a33483c1da0692d5c4b51ac6acea74
Submitter: "Zuul (22348)"
Branch: master

commit afd3374f46a33483c1da0692d5c4b51ac6acea74
Author: Francesco Pantano <email address hidden>
Date: Wed Feb 23 09:55:01 2022 +0000

    Revert "Import ceph_spec from tripleo-common"

    This reverts commit f7ab245202921a7866c8a4d1b3e5c2e29f8035bb.

    Reason for revert: Issues resolving the ceph_spec library on the overcloud nodes.

    Related-Bug: 1961325

    Change-Id: I72fc8ef8af8d35f8f9bc0deceb9c823e0c3020dd

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/831019
Committed: https://opendev.org/openstack/tripleo-ansible/commit/3951c074e91c628d4b4325623fe1ff5b8592f1a7
Submitter: "Zuul (22348)"
Branch: master

commit 3951c074e91c628d4b4325623fe1ff5b8592f1a7
Author: John Fulton <email address hidden>
Date: Fri Feb 25 18:48:37 2022 +0000

    Support standalone in ceph_spec_bootstrap module

    Add features from Icd18de08ebc818441e45ff7ce9f348fbc5813253
    to tripleo-ansible's ceph_spec_bootstrap module so that the
    ceph_spec library does not need to be in tripleo-common and
    the related bug can be avoided.

    Related-Bug: 1961325
    Change-Id: I90f9e16476f5a986a87c9e678e455846b1d02bf9

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/python-tripleoclient/+/831023
Committed: https://opendev.org/openstack/python-tripleoclient/commit/ec411b93297f2c841b03f7afb55a51bf2a0f0d0e
Submitter: "Zuul (22348)"
Branch: master

commit ec411b93297f2c841b03f7afb55a51bf2a0f0d0e
Author: John Fulton <email address hidden>
Date: Fri Feb 25 19:10:34 2022 +0000

    Build standalone ceph_spec with Ansible

    Instead of calling the ceph_spec library directly from
    the python-tripleoclient call an ansible wrapper to it
    instead so that the library does not need to live in
    tripleo-common and bug 1961325 can be avoided.

    Closes-Bug: 1961325
    Depends-On: I90f9e16476f5a986a87c9e678e455846b1d02bf9
    Change-Id: I7514c95b5c3c24a82e8703556d4205b272270511

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-common/+/830572
Committed: https://opendev.org/openstack/tripleo-common/commit/e2bc5dddcd7a8a890aa103212a6b4a53c85ea556
Submitter: "Zuul (22348)"
Branch: master

commit e2bc5dddcd7a8a890aa103212a6b4a53c85ea556
Author: Francesco Pantano <email address hidden>
Date: Wed Feb 23 09:54:20 2022 +0000

    Revert "Add ceph_spec library to tripleo-common"

    This reverts commit c09ef2c9a1da567612d7d6242e05925523b9fb06.

    Reason for revert: Issues resolving the tripleo ceph_spec library on the overcloud nodes

    Related-Bug: 1961325

    Depends-On: I7514c95b5c3c24a82e8703556d4205b272270511
    Change-Id: I0d14179516b5b5e43b46249d8270aedb178f7ab5

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-tripleoclient 18.0.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/python-tripleoclient/+/845482

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/845595

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/python-tripleoclient/+/847980

[OpenStack Infra (hudson-openstack)]

Change abandoned by "John Fulton <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/python-tripleoclient/+/847980
Reason: Created new patch by accident, instead wanted to update https://review.opendev.org/c/openstack/python-tripleoclient/+/845482

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/845595
Committed: https://opendev.org/openstack/tripleo-ansible/commit/5b6ee92531c8ffb4ced1b1bed8bdf3f67971d9a1
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 5b6ee92531c8ffb4ced1b1bed8bdf3f67971d9a1
Author: John Fulton <email address hidden>
Date: Wed Dec 22 16:43:08 2021 +0000

    Squash standalone deployed_ceph for tripleo-ansible to Wallaby

    This squash backports the standalone deployed_ceph feature to
    Wallaby without introducing related bugs. It has 6 commits.

    1. Make deployed ceph baremetal file optional

    Only require deployed_ceph_tht_path and
    baremetal_deployed_path if a dynamic spec
    needs to be generated. If a spec is passed
    then the playbook works fine without them.

    Change-Id: I9a6bd27f825d226b508053e4b7d2670a14f6f773
    (cherry picked from commit b18e0e7e21b11c5a9b821195f793bb415201045c)

    2. Add tripleo_cephadm_standalone boolean

    If tripleo_cephadm_standalone is true, then a standalone
    deployment is being run, which requires the THT variable
    CephSpecFqdn to be true. The python-tripleoclient patch
    which depends on this one passes this parameter and always
    creates Ceph specs with the FQDN set if --standalone is
    used.

    Change-Id: Idfe4f334f59a3a40363b7127f33ed4996f09362f
    (cherry picked from commit 18f4ce78ba6c2547c0a162b23c5b50584b80d82c)

    3. Support standalone in ceph_spec_bootstrap module

    Add features from Icd18de08ebc818441e45ff7ce9f348fbc5813253
    to tripleo-ansible's ceph_spec_bootstrap module so that the
    ceph_spec library does not need to be in tripleo-common and
    the related bug can be avoided.

    Related-Bug: 1961325
    Change-Id: I90f9e16476f5a986a87c9e678e455846b1d02bf9
    (cherry picked from commit 3951c074e91c628d4b4325623fe1ff5b8592f1a7)

    4. Support both inventory and metal file in ceph_spec_bootstrap

    The deployed metal file comes from the output of 'openstack
    overcloud node provision'. If this file's DeployedSeverPortMap
    is empty, we cannot map hostnames to their provisioning IPs.
    This information is in the inventory however so, if we have
    an inventory, then use it to complete the mapping. We do this
    by adding an option to ceph_spec_bootstrap to use both the
    inventory and the metal file as a source to build the spec.

    The cli-deployed-ceph.yaml playbook was designed to work with
    an inventory containing only the undercloud and a populated
    DeployedSeverPortMap. The ceph_spec_bootstrap module still
    supports this behavior though the playbook now passes the
    "both" option and thus requires an inventory with all hosts.
    Such an inventory is genereated by 'openstack overcloud node
    provision'. When the "both" case is used, it is assumed the
    inventory does not have the ceph_mon group so the playbook
    still dynamically expands its inventory to add it.

    Closes-Bug: 1962226
    Change-Id: I17f4303636b08d2be2d0f6e248242ad53c5fddf1
    (cherry picked from commit 1cf374b63bd4d2639f677f9a8591b3bf41bb458f)

    5. Pass method to ceph_spec...

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/python-tripleoclient/+/845482
Committed: https://opendev.org/openstack/python-tripleoclient/commit/46919b64db6764fa74c7500422cce85728d04f4a
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 46919b64db6764fa74c7500422cce85728d04f4a
Author: John Fulton <email address hidden>
Date: Tue Dec 21 17:04:50 2021 +0000

    Squash standalone deployed_ceph for pythonclient to Wallaby

    This squash backports the standalone deployed_ceph feature to
    Wallaby without introducing related bugs. It has 5 commits.

    Depends-On: https://review.opendev.org/c/openstack/tripleo-ansible/+/845595

    1. Add --mon-ip option to deployed ceph

    Add --mon-ip option to 'openstack overcloud
    ceph deploy', to easily override the variable
    tripleo_cephadm_first_mon_ip from the CLI.

    This option makes deployed ceph easier when
    using standalone.

    Change-Id: Id9a61356e8fd4618ce68faa2677f4ed30981b7d6
    (cherry picked from commit 2ac3953410337361777974cb2dccde3231ee3116)

    2. Make deployed ceph baremetal file optional

    If the --ceph-spec option is passed to 'openstack
    overcloud ceph deploy', then it is not necessary
    for the path to the environment file output from
    'openstack overcloud node provision' to be passed.

    Add conditional to ensure that either --ceph-spec
    or <deployed_baremetal.yaml> are used and add two
    unit tests to cover all three scenarios: a. metal
    is passed, b. spec is passed and c. neither are
    passed so a failure is asserted.

    Change-Id: Ibd350956f401fa48aaeae89fdd5e0bd9c28d0685
    (cherry picked from commit 84055814f9303a3fea6a252a24c2d9d123535bea)

    3. Introduce "openstack overcloud ceph spec"

    This command may be used to create a cephadm spec file as
    a function of the output of metalsmith and a TripleO roles
    file.

    Calls cli-deployed-ceph.yaml with ceph_spec tag so users
    may create the Ceph spec in a separate step from the Ceph
    Deployment. The Ceph spec file may be then be passed to
    "openstack overcloud ceph deploy".

    Also, supports a --standalone feature which directly calls
    the ceph_spec library in tripleo-common in the depends-on
    chain. As Ansible is used less the tripleo-pythonclient can
    use the ceph_spec library directly for more than just the
    --standalone feature.

    Change-Id: Icd18de08ebc818441e45ff7ce9f348fbc5813253
    (cherry picked from commit 3701f2ac318373250210c85f3483cd4f1be17dc7)

    4. Add --standalone to overcloud ceph user and deploy

    Both 'openstack overcloud ceph user enable' and 'openstack
    overcloud ceph deploy' currently use Ansible and require an
    inventory with hosts matching those in the Ceph spec. The
    inventory is created in the working directory when 'openstack
    overcloud node provision' runs but for a standalone deployment
    the user would need to create it manually. Abstract this detail
    away by providing a --standalone option for single node deploys
    which do not use metalsmith.

    Change-Id: Icb785f015ee69d45dd513aef09fac625de421250
    (cherry...