glance-api privsep error with cinder backend [not affecting only centos binary]

[Radosław Piliszek]

https://bugs.launchpad.net/kolla-ansible/+bug/1855695 continued:

There were two issues:
1) source images missing sudoers config file
2) source images missing rootwrap config files and ubu/deb binary having weird path /etc/glance/glance for them

Only CentOS binary was not affected here.

Issue #1 log:
2020-03-25 17:45:30.418 20 INFO oslo.privsep.daemon [req-e2d4f418-0012-4db5-a7dc-3f65eeb07054 609684595a074556be10f586f5e52fbc 15daa19f9fd742948eda9651dbccd3b4 - default default] Running privsep helper: ['sudo', 'glance-rootwrap', '/etc/glance/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/glance/glance-api.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmp2c0ocddc/privsep.sock']
2020-03-25 17:45:30.430 20 WARNING oslo.privsep.daemon [-] privsep log: sudo: no tty present and no askpass program specified
2020-03-25 17:45:30.434 20 CRITICAL oslo.privsep.daemon [req-e2d4f418-0012-4db5-a7dc-3f65eeb07054 609684595a074556be10f586f5e52fbc 15daa19f9fd742948eda9651dbccd3b4 - default default] privsep helper command exited non-zero (1)
2020-03-25 17:45:30.435 20 DEBUG os_brick.utils [req-e2d4f418-0012-4db5-a7dc-3f65eeb07054 609684595a074556be10f586f5e52fbc 15daa19f9fd742948eda9651dbccd3b4 - default default] <== get_connector_properties: exception (17ms) FailedToDropPrivileges('privsep helper command exited non-zero (1)',) trace_logging_wrapper /var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/utils.py:156
2020-03-25 17:45:30.435 20 ERROR glance_store._drivers.cinder [req-e2d4f418-0012-4db5-a7dc-3f65eeb07054 609684595a074556be10f586f5e52fbc 15daa19f9fd742948eda9651dbccd3b4 - default default] Failed to write to volume 305f97f2-3a52-4810-87ba-71df7834617e.: oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (1)
2020-03-25 17:45:35.641 20 ERROR glance.api.v2.image_data [req-e2d4f418-0012-4db5-a7dc-3f65eeb07054 609684595a074556be10f586f5e52fbc 15daa19f9fd742948eda9651dbccd3b4 - default default] Failed to upload image data due to internal error: oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (1)
2020-03-25 17:45:35.685 20 ERROR glance.common.wsgi [req-e2d4f418-0012-4db5-a7dc-3f65eeb07054 609684595a074556be10f586f5e52fbc 15daa19f9fd742948eda9651dbccd3b4 - default default] Caught error: privsep helper command exited non-zero (1): oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (1)

Issue #2 log in comment #5.

[Radosław Piliszek (yoctozepto)]

affects sudoers, hence kolla

[Radosław Piliszek (yoctozepto)]

doing https://review.opendev.org/715051

[Radosław Piliszek (yoctozepto)]

testing https://review.opendev.org/699910

[Radosław Piliszek (yoctozepto)]

oopsie: https://review.opendev.org/714999

[Radosław Piliszek (yoctozepto)]

2020-03-25 21:02:18.141 21 DEBUG os_brick.utils [req-da12a42a-5306-4f6a-9c24-346e6f5fe753 0dae8deae9e148edb8cc98946544dcc0 709285dcad0c4ee4b73f4ab2122608ea - default default] ==> get_connector_properties: call "{'root_helper': 'sudo glance-rootwrap /etc/glance/rootwrap.conf', 'my_ip': 'secondary2', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" trace_logging_wrapper /var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/utils.py:146
2020-03-25 21:02:18.143 21 INFO oslo.privsep.daemon [req-da12a42a-5306-4f6a-9c24-346e6f5fe753 0dae8deae9e148edb8cc98946544dcc0 709285dcad0c4ee4b73f4ab2122608ea - default default] Running privsep helper: ['sudo', 'glance-rootwrap', '/etc/glance/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/glance/glance-api.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmp2czc5r92/privsep.sock']
2020-03-25 21:02:18.371 21 WARNING oslo.privsep.daemon [-] privsep log: /var/lib/kolla/venv/bin/glance-rootwrap: Incorrect configuration file: /etc/glance/rootwrap.conf
2020-03-25 21:02:18.403 21 CRITICAL oslo.privsep.daemon [req-da12a42a-5306-4f6a-9c24-346e6f5fe753 0dae8deae9e148edb8cc98946544dcc0 709285dcad0c4ee4b73f4ab2122608ea - default default] privsep helper command exited non-zero (97)
2020-03-25 21:02:18.403 21 DEBUG os_brick.utils [req-da12a42a-5306-4f6a-9c24-346e6f5fe753 0dae8deae9e148edb8cc98946544dcc0 709285dcad0c4ee4b73f4ab2122608ea - default default] <== get_connector_properties: exception (262ms) FailedToDropPrivileges('privsep helper command exited non-zero (97)',) trace_logging_wrapper /var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/utils.py:156
2020-03-25 21:02:18.404 21 ERROR glance_store._drivers.cinder [req-da12a42a-5306-4f6a-9c24-346e6f5fe753 0dae8deae9e148edb8cc98946544dcc0 709285dcad0c4ee4b73f4ab2122608ea - default default] Failed to write to volume b33244e0-0d9f-41d5-be51-b924394d7733.: oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (97)
2020-03-25 21:02:23.575 21 ERROR glance.api.v2.image_data [req-da12a42a-5306-4f6a-9c24-346e6f5fe753 0dae8deae9e148edb8cc98946544dcc0 709285dcad0c4ee4b73f4ab2122608ea - default default] Failed to upload image data due to internal error: oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (97)
2020-03-25 21:02:23.624 21 ERROR glance.common.wsgi [req-da12a42a-5306-4f6a-9c24-346e6f5fe753 0dae8deae9e148edb8cc98946544dcc0 709285dcad0c4ee4b73f4ab2122608ea - default default] Caught error: privsep helper command exited non-zero (97): oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (97)

[Radosław Piliszek (yoctozepto)]

Source builds are missing rootwrap.conf as well beacause it is in glance_store (oh well).

CentOS binary build looks fine.

Ubuntu&Debian binary has weird /etc/glance/glance path.

Binary builds have sudoers in place.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/715051
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=d7d40d6c8695f79da5aad58c45231de7e9f738a3
Submitter: Zuul
Branch: master

commit d7d40d6c8695f79da5aad58c45231de7e9f738a3
Author: Radosław Piliszek <email address hidden>
Date: Wed Mar 25 21:01:38 2020 +0100

    Fix glance-api privsep errors

    This fixes usage of cinder as glance backend.

    Glance API source containers were missing both sudoers and
    rootwrap config. rootwrap config gets included in Kolla
    because otherwise it is clunky to get it from glance_store
    where it lives. The positive side is that it has not
    changed since July 2016.

    Debian/Ubuntu binary have weird double-glance path to
    rootwrap which gets fixed now by symbolic link.

    Closes-bug: #1869072

    Change-Id: Ia53657a415249882b93acca86b5e2a3bfa167820