Missing os-brick commands in debug mode

[Gorka Eguileor]

To debug os-brick's attach and detach code developers and system administrators rely on seeing what commands are actually being executed by os-brick.

The os-brick library relies on the DEBUG logs from the libraries (such as ``oslo_concurrency.processutils``) for this purpose instead of duplicating log entries by logging the calls and stdout-stderr itself.

The default configuration in Nova no longer logs those os-brick commands when running on debug mode.

This issue was introduced when fixing bug #1784062, as the fix was to set ALL privsep calls to log only INFO level messages.

The current solution is to set the ``default_log_levels`` configuration option in nova and include ``oslo.privsep.daemon=DEBUG`` in it.

The default for os-brick should be the other way around, it should allow emitting DEBUG messages on debug mode.

[Gorka Eguileor (gorka)]

Related fix proposed to oslo.privsep master
Review: https://review.opendev.org/c/openstack/oslo.privsep/+/784098

Once that is merged, released, and we update our requirements, we can fix this issue by making the following 2 changes:

--- a/nova/config.py
+++ b/nova/config.py
@@ -71,7 +71,7 @@ def set_log_defaults():
         extra_default_log_levels = ['glanceclient=WARN']
     # NOTE(danms): DEBUG logging in privsep will result in some large
     # and potentially sensitive things being logged.
- extra_default_log_levels.append('oslo.privsep.daemon=INFO')
+ extra_default_log_levels.append('nova.privsep=INFO')

--- a/nova/privsep/__init__.py
+++ b/nova/privsep/__init__.py
@@ -22,6 +22,7 @@ sys_admin_pctxt = priv_context.PrivContext(
     'nova',
     cfg_section='nova_sys_admin',
     pypath=__name__ + '.sys_admin_pctxt',
+ logger_name=__name__,
     capabilities=[capabilities.CAP_CHOWN,
                   capabilities.CAP_DAC_OVERRIDE,
                   capabilities.CAP_DAC_READ_SEARCH,

[Balazs Gibizer (balazs-gibizer)]

The privsep change is proposed in https://review.opendev.org/c/openstack/oslo.privsep/+/784098

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/oslo.privsep/+/784098
Committed: https://opendev.org/openstack/oslo.privsep/commit/71c732c8337309b3dc1b5a1957255da617d8dc3f
Submitter: "Zuul (22348)"
Branch: master

commit 71c732c8337309b3dc1b5a1957255da617d8dc3f
Author: Gorka Eguileor <email address hidden>
Date: Tue Mar 30 18:59:18 2021 +0200

    Allow finer grained log levels

    Currently all privsep calls use the same logging level as they use the
    ``oslo_privsep.daemon`` logger.

    This creates a problem for services like nova where we don't want to
    log its own privsep debug messages by default (even in debug mode) but
    we want them for os-brick privsep calls.

    This patch introduces a new string parameter to ``PrivContext`` called
    ``logger_name`` to control the name of the logger to use, which defaults
    to ``oslo_privsep.daemon``.

    Related-Bug: #1922052
    Change-Id: Ie7cc25ed53563dd500f7b10ff43a48518d46a2e0

[Gorka Eguileor (gorka)]

Fix available in Xena (2.6.0)

[Gorka Eguileor (gorka)]

Moving bug to os-brick since that's where we are going to fix it, although the issue was caused by a change in Nova's code and it will only be visible on Nova.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/os-brick/+/871835

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/os-brick/+/871835
Committed: https://opendev.org/openstack/os-brick/commit/a82b2ffce5ad11b591130df40500b2fb1673a129
Submitter: "Zuul (22348)"
Branch: master

commit a82b2ffce5ad11b591130df40500b2fb1673a129
Author: Gorka Eguileor <email address hidden>
Date: Thu Jan 26 15:41:15 2023 +0100

    Support separate privsep logging levels

    Currently we don't have os_brick DEBUG log levels in Nova when setting
    the service to debug mode.

    That happens because Nova is forcefully setting oslo.privsep.daemon
    levels to INFO to prevent leaking instance XML details (bug #1784062).

    Oslo Privsep now supports per-context debug log levels, so this patch
    sets the log level name for its only os_brick privsep context to
    "os_brick.privileged" to differentiate it from the service it runs under
    which uses the default "oslo_privsep.daemon".

    This way even though Nova is still disabling its own privileged calls it
    won't affect os-brick privileged calls, allowing us to properly debug
    block device attach/detach operations.

    Closes-Bug: #1922052
    Related-Bug: #1784062
    Change-Id: I0de32021eb90ca045845a6c7c7e3d27e52895948

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/os-brick/+/873100

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-brick 6.2.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/os-brick/+/883480

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/os-brick/+/873100
Committed: https://opendev.org/openstack/os-brick/commit/30ddb1761045dfc551e0b218b375842a14c42b0e
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 30ddb1761045dfc551e0b218b375842a14c42b0e
Author: Gorka Eguileor <email address hidden>
Date: Thu Jan 26 15:41:15 2023 +0100

    Support separate privsep logging levels

    Currently we don't have os_brick DEBUG log levels in Nova when setting
    the service to debug mode.

    That happens because Nova is forcefully setting oslo.privsep.daemon
    levels to INFO to prevent leaking instance XML details (bug #1784062).

    Oslo Privsep now supports per-context debug log levels, so this patch
    sets the log level name for its only os_brick privsep context to
    "os_brick.privileged" to differentiate it from the service it runs under
    which uses the default "oslo_privsep.daemon".

    This way even though Nova is still disabling its own privileged calls it
    won't affect os-brick privileged calls, allowing us to properly debug
    block device attach/detach operations.

    Closes-Bug: #1922052
    Related-Bug: #1784062
    Change-Id: I0de32021eb90ca045845a6c7c7e3d27e52895948
    (cherry picked from commit a82b2ffce5ad11b591130df40500b2fb1673a129)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/os-brick/+/883480
Committed: https://opendev.org/openstack/os-brick/commit/e57201a2843c2dd9cc6744cb8148dd6e3ed8f52f
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit e57201a2843c2dd9cc6744cb8148dd6e3ed8f52f
Author: Gorka Eguileor <email address hidden>
Date: Thu Jan 26 15:41:15 2023 +0100

    Support separate privsep logging levels

    Currently we don't have os_brick DEBUG log levels in Nova when setting
    the service to debug mode.

    That happens because Nova is forcefully setting oslo.privsep.daemon
    levels to INFO to prevent leaking instance XML details (bug #1784062).

    Oslo Privsep now supports per-context debug log levels, so this patch
    sets the log level name for its only os_brick privsep context to
    "os_brick.privileged" to differentiate it from the service it runs under
    which uses the default "oslo_privsep.daemon".

    This way even though Nova is still disabling its own privileged calls it
    won't affect os-brick privileged calls, allowing us to properly debug
    block device attach/detach operations.

    Closes-Bug: #1922052
    Related-Bug: #1784062
    Change-Id: I0de32021eb90ca045845a6c7c7e3d27e52895948
    (cherry picked from commit a82b2ffce5ad11b591130df40500b2fb1673a129)
    (cherry picked from commit 30ddb1761045dfc551e0b218b375842a14c42b0e)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-brick 5.2.4 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-brick 6.1.2 release.