novaclient stats all files in /usr/bin

[Eric Larese]

It appears that novaclient is searching Python's sys.path to find novaclient's own executable, and a side effect of this is an operating system security package will log hundreds of errors each time this happens. For example, this stack trace:

  /usr/lib/python2.7/site-packages/neutron/manager.py(244)get_plugin()
-> return weakref.proxy(cls.get_instance().plugin)
  /usr/lib/python2.7/site-packages/neutron/manager.py(238)get_instance()
-> cls._create_instance()
  /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py(252)inner()
-> return f(*args, **kwargs)
  /usr/lib/python2.7/site-packages/neutron/manager.py(224)_create_instance()
-> cls._instance = cls()
  /usr/lib/python2.7/site-packages/neutron/manager.py(120)__init__()
-> plugin_provider)
  /usr/lib/python2.7/site-packages/neutron/manager.py(157)_get_plugin_instance()
-> return plugin_class()
  /usr/lib/python2.7/site-packages/neutron/quota/resource_registry.py(121)wrapper()
-> return f(*args, **kwargs)
  /usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py(145)__init__()
-> super(Ml2Plugin, self).__init__()
  /usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py(103)__init__()
-> self.nova_notifier = nova.Notifier()
  /usr/lib/python2.7/site-packages/neutron/notifiers/nova.py(98)__init__()
-> ext for ext in nova_client.discover_extensions(NOVA_API_VERSION)
> /usr/lib/python2.7/site-packages/novaclient/client.py(724)discover_extensions()
-> _discover_via_contrib_path(version)

This stack trace is during neutron server startup, a novaclient call is made which results in _discover_via_python_path() being invoked here: https://github.com/openstack/python-novaclient/blob/master/novaclient/client.py#L723

This method uses pkgutil.iter_modules() which will search all of /usr/bin (among many other places). An operating system security package such as SELinux on RedHat will log hundreds of errors like this to /var/log/audit/audit.log:

type=AVC msg=audit(10/23/2015 15:41:08.766:368903) : avc: denied { getattr } for pid=13716 comm=neutron-server path=/usr/bin/virsh dev="dm-5" ino=138258059 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:virsh_exec_t:s0 tclass=file

One error is logged for every searched file in /usr/bin, about 1,300 messages each time neutron-server restarts on my test system. This generates a huge amount of noise in audit.log. I have not attempted to reproduce this with Ubuntu / AppArmor to verify if the issue is the same.

Is this something the novaclient code would worry about? Is there some way I could submit a patch to fix this?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/280725

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/280725
Committed: https://git.openstack.org/cgit/openstack/python-novaclient/commit/?id=c18ccb1bfae574b4b496c138e9192fc737ed9c20
Submitter: Jenkins
Branch: master

commit c18ccb1bfae574b4b496c138e9192fc737ed9c20
Author: Andrey Kurilin <email address hidden>
Date: Tue Feb 16 15:35:57 2016 +0200

    Add a way to discover only contrib extensions

    Several OS projects(cinder, neutron, osc...) use
    `novaclient.discover_extensions` for initialization novaclient.client.Client
    with novaclient.v2.contrib extensions. In this case, it would be nice to
    provide a way to not discover extension via python path an entry-point.

    Change-Id: I030f4c55c2795c7f7973f5f12e54b9819c4a5578
    Closes-Bug: #1509500

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/python-novaclient 3.3.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/287449

[Armando Migliaccio (armando-migliaccio)]

This bug is > 180 days without activity. We are unsetting assignee and milestone and setting status to Incomplete in order to allow its expiry in 60 days.

If the bug is still valid, then update the bug status.

[Armando Migliaccio (armando-migliaccio)]

https://review.openstack.org/#/c/287449/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/287449
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1db573ff12172f0a278b7318fe8094527e2bc72a
Submitter: Jenkins
Branch: master

commit 1db573ff12172f0a278b7318fe8094527e2bc72a
Author: Eric Larese <email address hidden>
Date: Wed Mar 2 15:41:36 2016 -0500

    Use only_contrib option for nova_client calls

    Use the only_contrib option that was added by
    I030f4c55c2795c7f7973f5f12e54b9819c4a5578 to speed up nova_client calls
    and skip the search for nova_client extensions to reduce
    /var/log/audit noise.

    Change-Id: Ic97b342a3633ffdf05b02ddd81baad88e1605a75
    Closes-Bug: #1509500

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/430317

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/430317
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0fb35ea59068b58be7e922314e66b530ffdea2ee
Submitter: Jenkins
Branch: stable/ocata

commit 0fb35ea59068b58be7e922314e66b530ffdea2ee
Author: Eric Larese <email address hidden>
Date: Wed Mar 2 15:41:36 2016 -0500

    Use only_contrib option for nova_client calls

    Use the only_contrib option that was added by
    I030f4c55c2795c7f7973f5f12e54b9819c4a5578 to speed up nova_client calls
    and skip the search for nova_client extensions to reduce
    /var/log/audit noise.

    Change-Id: Ic97b342a3633ffdf05b02ddd81baad88e1605a75
    Closes-Bug: #1509500
    (cherry picked from commit 1db573ff12172f0a278b7318fe8094527e2bc72a)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 10.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 11.0.0.0b1 development milestone.