nova-manage run on compute to discover nodes fail with permission error

Bug #1832482 reported by Martin Schuppert
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Undecided
Martin Schuppert

Bug Description

[1] switched to run nova-manage discovery as non root user.

                command: "/usr/bin/bootstrap_host_exec nova_compute su nova -s /bin/bash -c '/container-config-scripts/pyshim.sh /container-config-scripts/nova_cell_v2_discover_hosts.py'"

In case of updates where we already can have a nova-manage log owned by root make the overcloud deploy to fail with:

2019-06-12 05:30:08,789 p=13529 u=mistral | fatal: [compute-0]: FAILED! => {
    "failed_when_result": true,
    "outputs.stdout_lines | default([]) | union(outputs.stderr_lines | default([]))": [
        "Error running ['docker', 'run', '--name', 'nova_cell_v2_discover_hosts', '--label', 'config_id=tripleo_step5', '--label', 'container_name=nova_cell_v2_discover_hosts', '--label', 'managed_by=paunch', '--label', 'config_data={\"start_order\": 0, \"image\": \"192.168.24.1:87
87/rhosp14/openstack-nova-compute:2019-04-23.1\", \"environment\": [\"TRIPLEO_DEPLOY_IDENTIFIER=1560329712\"], \"command\": \"su nova -s /bin/bash -c \\'/docker-config-scripts/nova_cell_v2_discover_hosts.py\\'\", \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/et
c/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.
crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/var/lib/config-data/nova_libvirt/etc/my.cnf.d/:/etc/my.cnf.d/:ro\
", \"/var/lib/config-data/nova_libvirt/etc/nova/:/etc/nova/:ro\", \"/var/log/containers/nova:/var/log/nova\", \"/var/lib/docker-config-scripts/:/docker-config-scripts/\"], \"net\": \"host\", \"detach\": false}', '--env=TRIPLEO_DEPLOY_IDENTIFIER=1560329712', '--net=host', '--user=ro
ot', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:
/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro
', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/var/lib/config-data/nova_libvirt/etc/my.cnf.d/:/etc/my.cnf.d/:ro', '--volume=/var/lib/config-data/nova_libvirt/etc/nova/:/etc/nova/:ro', '--volume=/var/log/containers/nova:/var/log/nova', '--volume=/var/lib/docker-config-scripts/
:/docker-config-scripts/', '192.168.24.1:8787/rhosp14/openstack-nova-compute:2019-04-23.1', 'su', 'nova', '-s', '/bin/bash', '-c', \"'/docker-config-scripts/nova_cell_v2_discover_hosts.py'\"]. [1]",
        "",
        "stdout: ERROR:nova_cell_v2_discover_hosts:Cell v2 discovery failed with exit code 1, retrying",
        "ERROR:nova_cell_v2_discover_hosts:Cell v2 discovery failed with exit code 1, retrying",
        "stderr: Traceback (most recent call last):",
        " File \"/usr/bin/nova-manage\", line 10, in <module>",
        " sys.exit(main())",
        " File \"/usr/lib/python2.7/site-packages/nova/cmd/manage.py\", line 2325, in main",
        " logging.setup(CONF, \"nova\")",
        " File \"/usr/lib/python2.7/site-packages/oslo_log/log.py\", line 264, in setup",
        " _setup_logging_from_conf(conf, product_name, version)",
        " File \"/usr/lib/python2.7/site-packages/oslo_log/log.py\", line 353, in _setup_logging_from_conf",
        " filelog = file_handler(logpath)",
        " File \"/usr/lib64/python2.7/logging/handlers.py\", line 392, in __init__",
        " logging.FileHandler.__init__(self, filename, mode, encoding, delay)",
        " File \"/usr/lib64/python2.7/logging/__init__.py\", line 902, in __init__",
        " StreamHandler.__init__(self, self._open())",
        " File \"/usr/lib64/python2.7/logging/__init__.py\", line 925, in _open",
        " stream = open(self.baseFilename, self.mode)",
        "IOError: [Errno 13] Permission denied: '/var/log/nova/nova-manage.log'",
        "Traceback (most recent call last):"
    ]
}

We need to fix log owner ship on every deploy run on the computes, like we have on the
controller.

[1] https://review.opendev.org/#/c/652039/13/deployment/nova/nova-compute-container-puppet.yaml

Changed in tripleo:
assignee: nobody → Martin Schuppert (mschuppert)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.opendev.org/664850

Changed in tripleo:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/664853

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/664854

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/664856

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/664850
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=ccabc8f8e68c12321b98e4ada12c005d61b724a1
Submitter: Zuul
Branch: master

commit ccabc8f8e68c12321b98e4ada12c005d61b724a1
Author: Martin Schuppert <email address hidden>
Date: Wed Jun 12 12:11:56 2019 +0200

    Fix log owner on computes during overcloud deploy runs

    [1] switched to run nova-manage discovery as non root user. In case of
    updates there can be already a nove-manage log owned by root from previous
    runs. This change make sure we change the owner of nova-manage log to
    nova:nova on overcloud deploy runs on the computes

    [1] https://review.opendev.org/#/c/652039/13/deployment/nova/nova-compute-container-puppet.yaml

    Closes-Bug: #1832482

    Change-Id: Ie6d3f5262f7eb20ef92b07f2f359d88cb77b8752

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/stein)

Reviewed: https://review.opendev.org/664853
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=1d02e964db7ed7864f9ec13338c39057f82c95dd
Submitter: Zuul
Branch: stable/stein

commit 1d02e964db7ed7864f9ec13338c39057f82c95dd
Author: Martin Schuppert <email address hidden>
Date: Wed Jun 12 12:11:56 2019 +0200

    Fix log owner on computes during overcloud deploy runs

    [1] switched to run nova-manage discovery as non root user. In case of
    updates there can be already a nove-manage log owned by root from previous
    runs. This change make sure we change the owner of nova-manage log to
    nova:nova on overcloud deploy runs on the computes

    [1] https://review.opendev.org/#/c/652039/13/deployment/nova/nova-compute-container-puppet.yaml

    Closes-Bug: #1832482

    Change-Id: Ie6d3f5262f7eb20ef92b07f2f359d88cb77b8752
    (cherry picked from commit ccabc8f8e68c12321b98e4ada12c005d61b724a1)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/rocky)

Reviewed: https://review.opendev.org/664854
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=d574fddb1425c9e8166863bf1bc832690be66270
Submitter: Zuul
Branch: stable/rocky

commit d574fddb1425c9e8166863bf1bc832690be66270
Author: Martin Schuppert <email address hidden>
Date: Wed Jun 12 12:11:56 2019 +0200

    Fix log owner on computes during overcloud deploy runs

    [1] switched to run nova-manage discovery as non root user. In case of
    updates there can be already a nove-manage log owned by root from previous
    runs. This change make sure we change the owner of nova-manage log to
    nova:nova on overcloud deploy runs on the computes

    [1] https://review.opendev.org/#/c/652039/13/deployment/nova/nova-compute-container-puppet.yaml

    Closes-Bug: #1832482

    Conflicts:
     deployment/nova/nova-compute-container-puppet.yaml

    Change-Id: Ie6d3f5262f7eb20ef92b07f2f359d88cb77b8752
    (cherry picked from commit ccabc8f8e68c12321b98e4ada12c005d61b724a1)
    (cherry picked from commit 1d02e964db7ed7864f9ec13338c39057f82c95dd)

tags: added: in-stable-rocky
tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/queens)

Reviewed: https://review.opendev.org/664856
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=f6883a0bf1012f92261e760dee6973c458672b42
Submitter: Zuul
Branch: stable/queens

commit f6883a0bf1012f92261e760dee6973c458672b42
Author: Martin Schuppert <email address hidden>
Date: Wed Jun 12 12:11:56 2019 +0200

    Fix log owner on computes during overcloud deploy runs

    [1] switched to run nova-manage discovery as non root user. In case of
    updates there can be already a nove-manage log owned by root from previous
    runs. This change make sure we change the owner of nova-manage log to
    nova:nova on overcloud deploy runs on the computes

    [1] https://review.opendev.org/#/c/652039/13/deployment/nova/nova-compute-container-puppet.yaml

    Closes-Bug: #1832482

    Conflicts:
     deployment/nova/nova-compute-container-puppet.yaml

    Change-Id: Ie6d3f5262f7eb20ef92b07f2f359d88cb77b8752
    (cherry picked from commit ccabc8f8e68c12321b98e4ada12c005d61b724a1)
    (cherry picked from commit 1d02e964db7ed7864f9ec13338c39057f82c95dd)
    (cherry picked from commit d574fddb1425c9e8166863bf1bc832690be66270)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 9.4.0

This issue was fixed in the openstack/tripleo-heat-templates 9.4.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 8.4.0

This issue was fixed in the openstack/tripleo-heat-templates 8.4.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 10.6.0

This issue was fixed in the openstack/tripleo-heat-templates 10.6.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 11.1.0

This issue was fixed in the openstack/tripleo-heat-templates 11.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.