Zun install stable/xena

Bug #1979208 reported by Davide De Pasquale
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack-Ansible
Fix Released
Undecided
Unassigned

Bug Description

Dear all,

after a complete and successfully installed "stable/xena" with just the minimum services for operating some experiments (just nova, neutron, ceph, ceilometer, aodh, keystone, cinder, glance, heat, horizon) I have decided to upgrade the installation including some new services, such as masakari, barbican, octavia, mistral, zun.
Just because I am still in a pre-production environment, I have executed successfully
openstack-ansible setup-hosts.yml
openstack-ansible setup-infrastructure-yml

but when I execute openstack-ansible setup-openstack.yml, I received an error at os-zun-install.yml
Here the ansible error log:

2022-06-19 19:24:58,963 p=248021 u=root n=ansible | TASK [os_zun : Generate zun config] **************************************************************************************************************
2022-06-19 19:24:59,443 p=248021 u=root n=ansible | fatal: [infra1_zun_api_container-ca03ba7a]: FAILED! => {"msg": "{{ [[(ansible_facts['processor_vcpus']//ansible_facts['processor_threads_per_core'])|default(1), 1] | max * 2, zun_api_threads_max] | min }}: 'dict object' has no attribute 'processor_vcpus'"}
2022-06-19 19:24:59,445 p=248021 u=root n=ansible | RUNNING HANDLER [os_zun : Stop services] *********************************************************************************************************
2022-06-19 19:25:12,996 p=248021 u=root n=ansible | failed: [infra1_zun_api_container-ca03ba7a] (item={'group': 'zun_api', 'service_name': 'zun-wsproxy', 'start_order': 2, 'execstarts': '/openstack/venvs/zun-24.3.1.dev1/bin/zun-wsproxy --config-dir /etc/zun', 'enabled': 'yes', 'state': 'started', 'config_overrides': {}}) => {"ansible_loop_var": "item", "attempts": 5, "changed": false, "item": {"config_overrides": {}, "enabled": "yes", "execstarts": "/openstack/venvs/zun-24.3.1.dev1/bin/zun-wsproxy --config-dir /etc/zun", "group": "zun_api", "service_name": "zun-wsproxy", "start_order": 2, "state": "started"}, "msg": "Could not find the requested service zun-wsproxy: host"}
2022-06-19 19:25:12,999 p=248021 u=root n=ansible | RUNNING HANDLER [os_zun : Start services] ********************************************************************************************************
2022-06-19 19:25:26,162 p=248021 u=root n=ansible | failed: [infra1_zun_api_container-ca03ba7a] (item={'group': 'zun_api', 'service_name': 'zun-wsproxy', 'start_order': 2, 'execstarts': '/openstack/venvs/zun-24.3.1.dev1/bin/zun-wsproxy --config-dir /etc/zun', 'enabled': 'yes', 'state': 'started', 'config_overrides': {}}) => {"ansible_loop_var": "item", "attempts": 5, "changed": false, "item": {"config_overrides": {}, "enabled": "yes", "execstarts": "/openstack/venvs/zun-24.3.1.dev1/bin/zun-wsproxy --config-dir /etc/zun", "group": "zun_api", "service_name": "zun-wsproxy", "start_order": 2, "state": "started"}, "msg": "Could not find the requested service zun-wsproxy: host"}
2022-06-19 19:25:26,165 p=248021 u=root n=ansible | RUNNING HANDLER [python_venv_build : venv changed] ***********************************************************************************************
2022-06-19 19:25:26,260 p=248021 u=root n=ansible | RUNNING HANDLER [uwsgi : Execute service restart] ************************************************************************************************
2022-06-19 19:25:39,089 p=248021 u=root n=ansible | failed: [infra1_zun_api_container-ca03ba7a] (item={'key': 'zun-api', 'value': {'group': 'zun_api', 'service_name': 'zun-api', 'init_config_overrides': {}, 'start_order': 1, 'wsgi_app': True, 'wsgi_path': '/openstack/venvs/zun-24.3.1.dev1/bin/zun-api-wsgi', 'uwsgi_bind_address': '172.29.236.200', 'uwsgi_port': 9517, 'uwsgi_overrides': {}, 'uwsgi_uid': 'zun', 'uwsgi_guid': 'zun', 'wsgi_venv': '/openstack/venvs/zun-24.3.1.dev1', 'uwsgi_processes': '2', 'uwsgi_threads': 1}}) => {"ansible_loop_var": "service", "attempts": 5, "changed": false, "msg": "Could not find the requested service zun-api: host", "service": {"key": "zun-api", "value": {"group": "zun_api", "init_config_overrides": {}, "service_name": "zun-api", "start_order": 1, "uwsgi_bind_address": "172.29.236.200", "uwsgi_guid": "zun", "uwsgi_overrides": {}, "uwsgi_port": 9517, "uwsgi_processes": "2", "uwsgi_threads": 1, "uwsgi_uid": "zun", "wsgi_app": true, "wsgi_path": "/openstack/venvs/zun-24.3.1.dev1/bin/zun-api-wsgi", "wsgi_venv": "/openstack/venvs/zun-24.3.1.dev1"}}}
2022-06-19 19:25:39,094 p=248021 u=root n=ansible | PLAY RECAP ***************************************************************************************************************************************
2022-06-19 19:25:39,094 p=248021 u=root n=ansible | infra1_zun_api_container-ca03ba7a : ok=56 changed=34 unreachable=0 failed=4 skipped=16 rescued=0 ignored=0

any suggestion is very welcome.
Regards,
Davide

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible (master)
Changed in openstack-ansible:
status: New → In Progress
Revision history for this message
Dmitriy Rabotyagov (noonedeadpunk) wrote :

I believe this happens because of missing cache facts for zun containers. We take care of facts gathering inside playbook, but it seems we had a type in the playbook which lead to that failure.

As another workaround, you can try running simple ad-hoc against zun_all group, like:

  /opt/openstack-ansible$ ANSIBLE_GATHER_SUBSET="!all,hardware" ansible -m setup zun_all

This should also populate facts and cache them IIRC.

Revision history for this message
Davide De Pasquale (davidedepasquale) wrote :
Download full text (3.9 KiB)

Dear Dmitriy,

I have modified as per your current change proposal. The playbook continues to run until the following task where it seems etcd cluster (probably implemented inside the container and not using the infrastructure one) is not able to communicate between all containers.

TASK [systemd_service : Load service] ************************************************************************************************************
failed: [infra1_zun_api_container-ca03ba7a] (item={'service_name': 'etcd', 'service_type': 'notify', 'enabled': True, 'state': 'started', 'execstarts': '/usr/local/bin/etcd', 'config_overrides': {'Unit': {'Description': 'etcd - highly-available key value store', 'Documentation': 'https://github.com/coreos/etcd', 'Wants': {'network-online.target': None}}, 'Service': {'EnvironmentFile': '-/etc/default/%p', 'LimitNOFILE': 65536}}}) => {"ansible_loop_var": "item", "changed": false, "item": {"config_overrides": {"Service": {"EnvironmentFile": "-/etc/default/%p", "LimitNOFILE": 65536}, "Unit": {"Description": "etcd - highly-available key value store", "Documentation": "https://github.com/coreos/etcd", "Wants": {"network-online.target": null}}}, "enabled": true, "execstarts": "/usr/local/bin/etcd", "service_name": "etcd", "service_type": "notify", "state": "started"}, "msg": "Unable to start service etcd.service: Job for etcd.service failed because a timeout was exceeded.\nSee \"systemctl status etcd.service\" and \"journalctl -xe\" for details.\n"}

from infra1-zun container I have checked the "journalctl -xe" and collected the following fragment that seem to be pertinent for the error clarification:

Jun 20 22:02:52 infra1-zun-api-container-ca03ba7a etcd[4273]: publish error: etcdserver: request timed out
Jun 20 22:02:53 infra1-zun-api-container-ca03ba7a etcd[4273]: e8c46a04d116fb6 is starting a new election at term 134
Jun 20 22:02:53 infra1-zun-api-container-ca03ba7a etcd[4273]: e8c46a04d116fb6 became candidate at term 135
Jun 20 22:02:53 infra1-zun-api-container-ca03ba7a etcd[4273]: e8c46a04d116fb6 received MsgVoteResp from e8c46a04d116fb6 at term 135
Jun 20 22:02:53 infra1-zun-api-container-ca03ba7a etcd[4273]: e8c46a04d116fb6 [logterm: 1, index: 3] sent MsgVote request to 42216b9f4e65b1c at term 135
Jun 20 22:02:53 infra1-zun-api-container-ca03ba7a etcd[4273]: e8c46a04d116fb6 [logterm: 1, index: 3] sent MsgVote request to 7e13711afbb18a91 at term 135
Jun 20 22:02:55 infra1-zun-api-container-ca03ba7a etcd[4273]: health check for peer 42216b9f4e65b1c could not connect: dial tcp 172.29.236.52:2380: connect: connection refused (prober "ROUND_TRIPPER_SNAPSHOT")
Jun 20 22:02:55 infra1-zun-api-container-ca03ba7a etcd[4273]: health check for peer 42216b9f4e65b1c could not connect: dial tcp 172.29.236.52:2380: connect: connection refused (prober "ROUND_TRIPPER_RAFT_MESSAGE")
Jun 20 22:02:55 infra1-zun-api-container-ca03ba7a etcd[4273]: health check for peer 7e13711afbb18a91 could not connect: dial tcp 172.29.236.218:2380: connect: connection refused (prober "ROUND_TRIPPER_SNAPSHOT")
Jun 20 22:02:55 infra1-zun-api-container-ca03ba7a etcd[4273]: health check for peer 7e13711afbb18a91 could not connect: dial tcp 172.29.236.218:2380...

Read more...

Revision history for this message
Dmitriy Rabotyagov (noonedeadpunk) wrote :

I have a guess, that it's related to the serial manner of role execution.

Can you try running `openstack-ansible os-zun-install.yml -e zun_serial="100%"`?

Or define `zun_serial: "100%"` in user_variables.yml?

Revision history for this message
Davide De Pasquale (davidedepasquale) wrote :

Ciao Dmitriy,

let me confirm that with the command you proposed the playbook completes without any problem.
Just for future readers, and for my personal understand :), what does it means "zun_serial" in this context?

We can also close as solved this issue if you agree.
Best regards,
Davide

Revision history for this message
Dmitriy Rabotyagov (noonedeadpunk) wrote :

I can't say issue is solved, it's just a workaround. In the meanwhile we need to introduce a way to run this in serial mode and likely re-arrange some workflow.

zun_serial controls serial execution in the playbook. You can check Ansible documentation on serial here: https://docs.ansible.com/ansible/latest/user_guide/playbooks_strategies.html#setting-the-batch-size-with-serial

Changed in openstack-ansible:
status: In Progress → Confirmed
Revision history for this message
Davide De Pasquale (davidedepasquale) wrote :

Thanks Dmitriy,

just for my understanding when I complete the os-zun-install.yml playbook I see only api containers are involved in the installation process.
Can you confirm that also compute nodes have to be "touched" by the installation process and so I am wrong in something?

Sorry I am still new too new to openstack-ansible to identify any hidden configuration paramenter that shall be taken in consideration. It would be great if someone (eventually) will publish the configuration of a real scenario ready to use.
I promise I will do it if I will be able to complete and configure a whole openstack "as I wish" :)

I can still manually modify my computes by following the manual installation procedure, but this is in open contrast with the will to use an automated and verified tool such openstack-ansible

Thanks for helping me.
Davide

Revision history for this message
Davide De Pasquale (davidedepasquale) wrote :

Please forget comment #7 I was wrong.
I found in the aio zun example the answer to my question...

here the correct conf.d/zun.yml (for any reader that eventually will have my same doubt)

---
# The infra nodes that will be running the zun services
zun-infra_hosts:
  infra1:
    ip: 172.29.236.11
  infra2:
    ip: 172.29.236.12
  infra3:
    ip: 172.29.236.13
zun-compute_hosts:
  compute1:
    ip: 172.29.236.31
  compute2:
    ip: 172.29.236.32
  compute3:
    ip: 172.29.236.33
  compute4:
    ip: 172.29.236.34

Revision history for this message
Dmitriy Rabotyagov (noonedeadpunk) wrote :

I can also encourage you joining our IRC channel #openstack-ansible on OFTC network. There's quite friendly community of operators that will try to answer your questions. Just mind that ppl could be in different time zones and not constantly monitoring it, so your question might be noticed not instantly, and it's important to stay connected otherwise you won't get reply.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/openstack-ansible/+/847261

Changed in openstack-ansible:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-ansible (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/openstack-ansible/+/847261
Committed: https://opendev.org/openstack/openstack-ansible/commit/0359a21ccaaf80cfc86f635e5fafcb7cfe3df8cf
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 0359a21ccaaf80cfc86f635e5fafcb7cfe3df8cf
Author: Dmitriy Rabotyagov <email address hidden>
Date: Mon Jun 20 21:48:07 2022 +0200

    Fix facts gathering for zun

    We have wrong/non-existing group defined for facts gathering in zun
    playbook.

    Closes-Bug: #1979208
    Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/847668
    Change-Id: Ic7f6f2446ea5f1264c44493ca1b329fe07eb9e30

tags: added: in-stable-yoga
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/openstack-ansible 25.0.0.0rc2

This issue was fixed in the openstack/openstack-ansible 25.0.0.0rc2 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-ansible (master)

Reviewed: https://review.opendev.org/c/openstack/openstack-ansible/+/846799
Committed: https://opendev.org/openstack/openstack-ansible/commit/244134b94bb6bffd063fe567e087adb7f5a75b93
Submitter: "Zuul (22348)"
Branch: master

commit 244134b94bb6bffd063fe567e087adb7f5a75b93
Author: Dmitriy Rabotyagov <email address hidden>
Date: Mon Jun 20 21:48:07 2022 +0200

    Fix facts gathering for zun

    We have wrong/non-existing group defined for facts gathering in zun
    playbook.

    Closes-Bug: #1979208
    Change-Id: Ic7f6f2446ea5f1264c44493ca1b329fe07eb9e30

Changed in openstack-ansible:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/openstack-ansible/+/849630

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/openstack-ansible/+/849631

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/openstack-ansible/+/849632

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-ansible (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/openstack-ansible/+/849632
Committed: https://opendev.org/openstack/openstack-ansible/commit/1523a194a39bd014288b8c9254b4da216d216e3f
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 1523a194a39bd014288b8c9254b4da216d216e3f
Author: Dmitriy Rabotyagov <email address hidden>
Date: Mon Jun 20 21:48:07 2022 +0200

    Fix facts gathering for zun

    We have wrong/non-existing group defined for facts gathering in zun
    playbook.

    Closes-Bug: #1979208
    Change-Id: Ic7f6f2446ea5f1264c44493ca1b329fe07eb9e30
    (cherry picked from commit 244134b94bb6bffd063fe567e087adb7f5a75b93)

tags: added: in-stable-victoria
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-ansible (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/openstack-ansible/+/849631
Committed: https://opendev.org/openstack/openstack-ansible/commit/4d8d9e8bf73d8da953d9055966f0bb0dd818e044
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 4d8d9e8bf73d8da953d9055966f0bb0dd818e044
Author: Dmitriy Rabotyagov <email address hidden>
Date: Mon Jun 20 21:48:07 2022 +0200

    Fix facts gathering for zun

    We have wrong/non-existing group defined for facts gathering in zun
    playbook.

    Closes-Bug: #1979208
    Change-Id: Ic7f6f2446ea5f1264c44493ca1b329fe07eb9e30
    (cherry picked from commit 244134b94bb6bffd063fe567e087adb7f5a75b93)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/openstack-ansible 23.4.0

This issue was fixed in the openstack/openstack-ansible 23.4.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-ansible (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/openstack-ansible/+/849630
Committed: https://opendev.org/openstack/openstack-ansible/commit/76752fda361d6623b6c9d54cc8cd75489070ef34
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 76752fda361d6623b6c9d54cc8cd75489070ef34
Author: Dmitriy Rabotyagov <email address hidden>
Date: Mon Jun 20 21:48:07 2022 +0200

    Fix facts gathering for zun

    We have wrong/non-existing group defined for facts gathering in zun
    playbook.

    Closes-Bug: #1979208
    Depends-On: https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/850233
    Change-Id: Ic7f6f2446ea5f1264c44493ca1b329fe07eb9e30
    (cherry picked from commit 244134b94bb6bffd063fe567e087adb7f5a75b93)

tags: added: in-stable-xena
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/openstack-ansible 24.4.0

This issue was fixed in the openstack/openstack-ansible 24.4.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/openstack-ansible 26.0.0.0rc1

This issue was fixed in the openstack/openstack-ansible 26.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/openstack-ansible victoria-eom

This issue was fixed in the openstack/openstack-ansible victoria-eom release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.