monasca-api use wrong mod_wsgi Python version on Debian/Ubuntu source

[Ole Kleinschmidt]

**Bug Report**

It seems there is a bug in the monasca-api config and/or the docker image. After we have enabled monasca in globals.yaml and the deployment has been completed successfully, the error log fills up with the following entries:

tail -f /var/log/kolla/monasca/monasca-api-error.log
[Fri Apr 17 07:31:21.438942 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] mod_wsgi (pid=26): Target WSGI script '/monasca-api/monasca_api/api/wsgi.py' cannot be loaded as Python module.
[Fri Apr 17 07:31:21.438996 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] mod_wsgi (pid=26): Exception occurred processing WSGI script '/monasca-api/monasca_api/api/wsgi.py'.
[Fri Apr 17 07:31:21.439017 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] Traceback (most recent call last):
[Fri Apr 17 07:31:21.439033 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] File "/monasca-api/monasca_api/api/wsgi.py", line 18, in <module>
[Fri Apr 17 07:31:21.439084 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] from monasca_api.api import server
[Fri Apr 17 07:31:21.439095 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] File "/var/lib/kolla/venv/lib/python3.6/site-packages/monasca_api/api/server.py", line 21, in <module>
[Fri Apr 17 07:31:21.439144 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] from oslo_config import cfg
[Fri Apr 17 07:31:21.439153 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_config/cfg.py", line 39, in <module>
[Fri Apr 17 07:31:21.439575 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] import enum
[Fri Apr 17 07:31:21.439593 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] ImportError: No module named enum

We use the kolla/ubuntu-source-*:train docker-images. It looks to me like Apache in the monasca_api container is using python2.7:

# docker exec -it monasca_api /bin/bash
(monasca-api)[root@cXXXX /]# ll /usr/lib/apache2/modules/mod_wsgi.so
lrwxrwxrwx 1 root root 15 Feb 14 11:08 /usr/lib/apache2/modules/mod_wsgi.so -> mod_wsgi.so-2.7
(monasca-api)[root@cXXXX /]#

But in the log above, python3.6 modules are called. Also the central-logging seems broken now:

# docker logs fluentd
2020-04-16 11:08:36 +0000 [warn]: parameter 'type' in <match syslog.local0.**>
  @type copy
  type copy
  <store>
    @type "file"
    path "/var/log/kolla/swift/swift_latest.*.log"
    utc
    append true
    compress gzip
    <format>
      utc
      localtime false
    </format>
    <inject>
      localtime false
    </inject>
    <buffer time>
      path "/var/log/kolla/swift/swift_latest.*.log"
    </buffer>
  </store>
  <store>
    @type "monasca"
    keystone_url "http://10.38.1.253:5000"
    monasca_log_api "http://10.38.1.253:5607"
    monasca_log_api_version "v3.0"
    username "monasca-agent"
    password xxxxxx
    domain_id "default"
    project_name "monasca_control_plane"
    message_field_name "Payload"
    <buffer>
      flush_mode interval
      retry_type exponential_backoff
    </buffer>
  </store>
</match> is not used.
2020-04-16 11:08:36 +0000 [warn]: parameter 'type' in <match syslog.local1.**>
  @type copy
  type copy
  <store>
    @type "file"
    path "/var/log/kolla/haproxy/haproxy_latest.*.log"
    output_tag false
    output_time false
    utc
    append true
    compress gzip
    <format>
      output_time false
      output_tag false
      utc
      localtime false
    </format>
    <inject>
      localtime false
    </inject>
    <buffer time>
      path "/var/log/kolla/haproxy/haproxy_latest.*.log"
    </buffer>
  </store>
  <store>
    @type "monasca"
    keystone_url "http://10.38.1.253:5000"
    monasca_log_api "http://10.38.1.253:5607"
    monasca_log_api_version "v3.0"
    username "monasca-agent"
    password xxxxxx
    domain_id "default"
    project_name "monasca_control_plane"
    message_field_name "Payload"
    <buffer>
      flush_mode interval
      retry_type exponential_backoff
    </buffer>
  </store>
</match> is not used.
2020-04-16 11:08:36 +0000 [info]: #0 starting fluentd worker pid=13280 ppid=6 worker=0
2020-04-16 11:08:36 +0000 [error]: #0 unexpected error error_class=RestClient::Unauthorized error="401 Unauthorized: {\"error\":{\"code\":401,\"message\":\"The request you have made requires authentication.\",\"title\":\"Unauthorized\"}}\n"
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/rest-client-1.8.0/lib/restclient/abstract_response.rb:74:in `return!'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/rest-client-1.8.0/lib/restclient/request.rb:495:in `process_result'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/rest-client-1.8.0/lib/restclient/request.rb:421:in `block in transmit'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/2.4.0/net/http.rb:877:in `start'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/rest-client-1.8.0/lib/restclient/request.rb:413:in `transmit'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/rest-client-1.8.0/lib/restclient/request.rb:176:in `execute'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/rest-client-1.8.0/lib/restclient/request.rb:41:in `execute'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/rest-client-1.8.0/lib/restclient/resource.rb:67:in `post'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-monasca-output-0.0.1/lib/fluent/plugin/keystone/keystone_client.rb:38:in `request'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-monasca-output-0.0.1/lib/fluent/plugin/keystone/keystone_client.rb:31:in `authenticate'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-monasca-output-0.0.1/lib/fluent/plugin/out_monasca.rb:82:in `authenticate'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-monasca-output-0.0.1/lib/fluent/plugin/out_monasca.rb:42:in `start'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/compat/call_super_mixin.rb:42:in `start'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/root_agent.rb:200:in `block in start'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/root_agent.rb:189:in `block (2 levels) in lifecycle'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/root_agent.rb:188:in `each'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/root_agent.rb:188:in `block in lifecycle'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/root_agent.rb:175:in `each'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/root_agent.rb:175:in `lifecycle'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/root_agent.rb:199:in `start'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/engine.rb:248:in `start'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/engine.rb:147:in `run'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/supervisor.rb:592:in `block in run_worker'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/supervisor.rb:823:in `main_process'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/supervisor.rb:586:in `run_worker'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/command/fluentd.rb:338:in `<top (required)>'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/bin/fluentd:8:in `<top (required)>'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/bin/fluentd:23:in `load'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/bin/fluentd:23:in `<top (required)>'
  2020-04-16 11:08:36 +0000 [error]: #0 /usr/sbin/td-agent:7:in `load'
  2020-04-16 11:08:36 +0000 [error]: #0 /usr/sbin/td-agent:7:in `<main>'
2020-04-16 11:08:36 +0000 [error]: #0 unexpected error error_class=RestClient::Unauthorized error="401 Unauthorized: {\"error\":{\"code\":401,\"message\":\"The request you have made requires authentication.\",\"title\":\"Unauthorized\"}}\n"
  2020-04-16 11:08:36 +0000 [error]: #0 suppressed same stacktrace

**Environment**:
OS: Ubuntu 18.04
Kernel: 4.15.0-91-generic
Docker: 19.03.8
Type: source
Official images from Docker Hub, but retagged and pushed to our local nexus.

cheers,
Ole

[Mark Goddard (mgoddard)]

Thanks for raising this. It looks like we need to switch to the python 3 version of the mod_wsgi package on Debian/Ubuntu. I'll propose a fix.

[Mark Goddard (mgoddard)]

Fix proposed to kolla: https://review.opendev.org/#/c/725221/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/725221
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=f5d31da12fa66cedfc2e270b8cd3be8e2bb9254b
Submitter: Zuul
Branch: master

commit f5d31da12fa66cedfc2e270b8cd3be8e2bb9254b
Author: Mark Goddard <email address hidden>
Date: Mon May 4 10:32:47 2020 +0100

    Use Python 3 for mod_wsgi in cyborg and monasca on Debian/Ubuntu

    In Train kolla switched to Python 3 in Debian and Ubuntu source images.
    For services running under httpd with mod_wsgi, they should install
    libapache2-mod-wsgi-py3 rather than libapache2-mod-wsgi. This was done
    for most images, but cyborg and monasca were omitted. This change fixes
    that.

    Change-Id: I25cfa62dbf20490685617b4a4185bf95ab91725e
    Closes-Bug: #1873421

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/726460

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/726460
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=f221b4f7ab7745ee20c8e97d6a1a2d3c7e74f466
Submitter: Zuul
Branch: stable/train

commit f221b4f7ab7745ee20c8e97d6a1a2d3c7e74f466
Author: Mark Goddard <email address hidden>
Date: Mon May 4 10:32:47 2020 +0100

    Use Python 3 for mod_wsgi in cyborg and monasca on Debian/Ubuntu

    In Train kolla switched to Python 3 in Debian and Ubuntu source images.
    For services running under httpd with mod_wsgi, they should install
    libapache2-mod-wsgi-py3 rather than libapache2-mod-wsgi. This was done
    for most images, but cyborg and monasca were omitted. This change fixes
    that.

    Change-Id: I25cfa62dbf20490685617b4a4185bf95ab91725e
    Closes-Bug: #1873421
    (cherry picked from commit f5d31da12fa66cedfc2e270b8cd3be8e2bb9254b)