nova scheduler & compute healthchecks don't test the right port

Bug #1824805 reported by Emilien Macchi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Medium
Martin Schuppert

Bug Description

    "nova_scheduler": {
        "environment": [
            "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS"
        ],
        "healthcheck": {
            "test": "/openstack/healthcheck null"
        },
        "image": "192.168.24.1:8787/tripleomaster/centos-binary-nova-scheduler:be90d93c3c5f77f428d12a9a8a2ef97b9dada8f3_5b53d5ba-updated-20190412030455",
        "net": "host",
        "privileged": false,
        "restart": "always",
        "volumes": [
            "/etc/hosts:/etc/hosts:ro",
            "/etc/localtime:/etc/localtime:ro",
            "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro",
            "/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro",
            "/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro",
            "/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro",
            "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro",
            "/dev/log:/dev/log",
            "/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro",
            "/etc/puppet:/etc/puppet:ro",
            "/var/log/containers/nova:/var/log/nova",
            "/var/lib/kolla/config_files/nova_scheduler.json:/var/lib/kolla/config_files/config.json:ro",
            "/var/lib/config-data/puppet-generated/nova/:/var/lib/kolla/config_files/src:ro",
            "/run:/run"
        ]
    },
    "nova_compute": {
        "environment": [
            "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS"
        ],
        "healthcheck": {
            "test": "/openstack/healthcheck null"
        },
        "image": "192.168.24.1:8787/tripleomaster/centos-binary-nova-compute:be90d93c3c5f77f428d12a9a8a2ef97b9dada8f3_5b53d5ba-updated-20190412030455",
        "ipc": "host",
        "net": "host",
        "privileged": true,
        "restart": "always",
        "start_order": 3,
        "ulimit": [
            "nofile=1024"
        ],
        "user": "nova",
        "volumes": [
            "/etc/hosts:/etc/hosts:ro",
            "/etc/localtime:/etc/localtime:ro",
            "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro",
            "/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro",
            "/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro",
            "/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro",
            "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro",
            "/dev/log:/dev/log",
            "/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro",
            "/etc/puppet:/etc/puppet:ro",
            "/var/log/containers/nova:/var/log/nova",
            "/var/lib/kolla/config_files/nova_compute.json:/var/lib/kolla/config_files/config.json:ro",
            "/var/lib/config-data/puppet-generated/nova_libvirt/:/var/lib/kolla/config_files/src:ro",
            "/etc/iscsi:/var/lib/kolla/config_files/src-iscsid:ro",
            "/etc/ceph:/var/lib/kolla/config_files/src-ceph:ro",
            "/dev:/dev",
            "/lib/modules:/lib/modules:ro",
            "/run:/run",
            "/var/lib/iscsi:/var/lib/iscsi:z",
            "/var/lib/nova:/var/lib/nova:shared,z",
            "/var/lib/libvirt:/var/lib/libvirt:shared,z",
            "/sys/class/net:/sys/class/net",
            "/sys/bus/pci:/sys/bus/pci"
        ]
    },

healthcheck null is wrong, it should be the rabbitmq port, which failed to be lookup because nova::rabbit_port doens't exist in puppet-nova. We need to lookup the parameter from somewhere else.
Note that it seems we can't use RpcPort because the parameter is marked as deprecated.

Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

Note, we have more places using that deprecated nova::rabbit_port and much more that just doing that for nova: http://codesearch.openstack.org/?q=%3A%3Arabbit_port&i=nope&files=&repos=tripleo-heat-templates

Revision history for this message
Martin Schuppert (mschuppert) wrote :

405366fa32583e88c34417e5f46fa574ed8f4e98 removed nova::rabbit_port from deployment/nova/nova-base-puppet.yaml . checking the options.

Changed in tripleo:
assignee: nobody → Martin Schuppert (mschuppert)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.openstack.org/652964

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/652964
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=8ff04029f5bc4cc2db4b3575229ce7bafb4eb358
Submitter: Zuul
Branch: master

commit 8ff04029f5bc4cc2db4b3575229ce7bafb4eb358
Author: Martin Schuppert <email address hidden>
Date: Tue Apr 16 12:19:27 2019 +0200

    Use oslo_messaging_rpc_port for nova rpc healthchecks

    With 405366fa32583e88c34417e5f46fa574ed8f4e98 the parameters RpcPort,
    RpcUserName, RpcPassword and RpcUseSSL got deprecated and
    nova::rabbitmq_port removed. As a result the healtcheck get called with
    null parameter and fail.
    We now get the global_config_settings from RabbitMQService and use
    oslo_messaging_rpc_port for the healthcheck.

    Change-Id: I1849926b1d6256de5f4d677de5a9b34d78aad5d0
    Closes-Bug: #1824805

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/654445

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/stein)

Reviewed: https://review.opendev.org/654445
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=847a2057c2ef89fad5bb9f46025a6e3c2a7e4a98
Submitter: Zuul
Branch: stable/stein

commit 847a2057c2ef89fad5bb9f46025a6e3c2a7e4a98
Author: Martin Schuppert <email address hidden>
Date: Tue Apr 16 12:19:27 2019 +0200

    Use oslo_messaging_rpc_port for nova rpc healthchecks

    With 405366fa32583e88c34417e5f46fa574ed8f4e98 the parameters RpcPort,
    RpcUserName, RpcPassword and RpcUseSSL got deprecated and
    nova::rabbitmq_port removed. As a result the healtcheck get called with
    null parameter and fail.
    We now get the global_config_settings from RabbitMQService and use
    oslo_messaging_rpc_port for the healthcheck.

    Change-Id: I1849926b1d6256de5f4d677de5a9b34d78aad5d0
    Closes-Bug: #1824805
    (cherry picked from commit 8ff04029f5bc4cc2db4b3575229ce7bafb4eb358)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/656488

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/656488
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=372ca22483a1988a363b9ed8ab0ef319b1ab0f16
Submitter: Zuul
Branch: master

commit 372ca22483a1988a363b9ed8ab0ef319b1ab0f16
Author: Martin Schuppert <email address hidden>
Date: Tue Apr 30 11:45:30 2019 +0200

    Use oslo_messaging_rpc_port for nova rpc healthchecks

    With 405366fa32583e88c34417e5f46fa574ed8f4e98 the parameters RpcPort,
    RpcUserName, RpcPassword and RpcUseSSL got deprecated and
    nova::rabbitmq_port removed. As a result the healtcheck get called with
    null parameter and fail.
    We now get the global_config_settings from RabbitMQService and use
    oslo_messaging_rpc_port for the healthcheck.

    This is an addition to 8ff04029f5bc4cc2db4b3575229ce7bafb4eb358 for
    the undercloud compute service.

    Related-Bug: #1824805

    Change-Id: Ic5f545b3bda571e028452d5c4123d9eca4493054

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/stein)

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/657139

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (stable/stein)

Reviewed: https://review.opendev.org/657139
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=9096b494185279b7e7cfcc21c965c08b8a8a0c26
Submitter: Zuul
Branch: stable/stein

commit 9096b494185279b7e7cfcc21c965c08b8a8a0c26
Author: Martin Schuppert <email address hidden>
Date: Tue Apr 30 11:45:30 2019 +0200

    Use oslo_messaging_rpc_port for nova rpc healthchecks

    With 405366fa32583e88c34417e5f46fa574ed8f4e98 the parameters RpcPort,
    RpcUserName, RpcPassword and RpcUseSSL got deprecated and
    nova::rabbitmq_port removed. As a result the healtcheck get called with
    null parameter and fail.
    We now get the global_config_settings from RabbitMQService and use
    oslo_messaging_rpc_port for the healthcheck.

    This is an addition to 8ff04029f5bc4cc2db4b3575229ce7bafb4eb358 for
    the undercloud compute service.

    Related-Bug: #1824805

    Change-Id: Ic5f545b3bda571e028452d5c4123d9eca4493054
    (cherry picked from commit 372ca22483a1988a363b9ed8ab0ef319b1ab0f16)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 11.0.0

This issue was fixed in the openstack/tripleo-heat-templates 11.0.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 10.6.0

This issue was fixed in the openstack/tripleo-heat-templates 10.6.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.