kolla

monasca-api use wrong mod_wsgi Python version on Debian/Ubuntu source

Series train
Bug #1873421

Bug #1873421 reported by Ole Kleinschmidt on 2020-04-17

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
kolla	Fix Released	Medium	Unassigned	kolla 10.0.0 "ussuri"
Train	Fix Committed	Medium	Radosław Piliszek	kolla 9.1.0 "Train"
Ussuri	Fix Released	Medium	Unassigned	kolla 10.0.0 "ussuri"

Bug Description

**Bug Report**

It seems there is a bug in the monasca-api config and/or the docker image. After we have enabled monasca in globals.yaml and the deployment has been completed successfully, the error log fills up with the following entries:

tail -f /var/log/kolla/monasca/monasca-api-error.log
[Fri Apr 17 07:31:21.438942 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] mod_wsgi (pid=26): Target WSGI script '/monasca-api/monasca_api/api/wsgi.py' cannot be loaded as Python module.
[Fri Apr 17 07:31:21.438996 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] mod_wsgi (pid=26): Exception occurred processing WSGI script '/monasca-api/monasca_api/api/wsgi.py'.
[Fri Apr 17 07:31:21.439017 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] Traceback (most recent call last):
[Fri Apr 17 07:31:21.439033 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] File "/monasca-api/monasca_api/api/wsgi.py", line 18, in <module>
[Fri Apr 17 07:31:21.439084 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] from monasca_api.api import server
[Fri Apr 17 07:31:21.439095 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] File "/var/lib/kolla/venv/lib/python3.6/site-packages/monasca_api/api/server.py", line 21, in <module>
[Fri Apr 17 07:31:21.439144 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] from oslo_config import cfg
[Fri Apr 17 07:31:21.439153 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_config/cfg.py", line 39, in <module>
[Fri Apr 17 07:31:21.439575 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] import enum
[Fri Apr 17 07:31:21.439593 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] ImportError: No module named enum

We use the kolla/ubuntu-source-*:train docker-images. It looks to me like Apache in the monasca_api container is using python2.7:

# docker exec -it monasca_api /bin/bash
(monasca-api)[root@cXXXX /]# ll /usr/lib/apache2/modules/mod_wsgi.so
lrwxrwxrwx 1 root root 15 Feb 14 11:08 /usr/lib/apache2/modules/mod_wsgi.so -> mod_wsgi.so-2.7
(monasca-api)[root@cXXXX /]#

But in the log above, python3.6 modules are called. Also the central-logging seems broken now:

# docker logs fluentd
2020-04-16 11:08:36 +0000 [warn]: parameter 'type' in <match syslog.local0.**>
  @type copy
  type copy
  <store>
    @type "file"
    path "/var/log/kolla/swift/swift_latest.*.log"
    utc
    append true
    compress gzip
    <format>
      utc
      localtime false
    </format>
    <inject>
      localtime false
    </inject>
    <buffer time>
      path "/var/log/kolla/swift/swift_latest.*.log"
    </buffer>
  </store>
  <store>
    @type "monasca"
    keystone_url "http://10.38.1.253:5000"
    monasca_log_api "http://10.38.1.253:5607"
    monasca_log_api_version "v3.0"
    username "monasca-agent"
    password xxxxxx
    domain_id "default"
    project_name "monasca_control_plane"
    message_field_name "Payload"
    <buffer>
      flush_mode interval
      retry_type exponential_backoff
    </buffer>
  </store>
</match> is not used.
2020-04-16 11:08:36 +0000 [warn]: parameter 'type' in <match syslog.local1.**>
  @type copy
  type copy
  <store>
    @type "file"
    path "/var/log/kolla/haproxy/haproxy_latest.*.log"
    output_tag false
    output_time false
    utc
    append true
    compress gzip
    <format>
      output_time false
      output_tag false
      utc
      localtime false
    </format>
    <inject>
      localtime false
    </inject>
    <buffer time>
      path "/var/log/kolla/haproxy/haproxy_latest.*.log"
    </buffer>
  </store>
  <store>
    @type "monasca"
    keystone_url "http://10.38.1.253:5000"
    monasca_log_api "http://10.38.1.253:5607"
    monasca_log_api_version "v3.0"
    username "monasca-agent"
    password xxxxxx
    domain_id "default"
    project_name "monasca_control_plane"
    message_field_name "Payload"
    <buffer>
      flush_mode interval
      retry_type exponential_backoff
    </buffer>
  </store>
</match> is not used.
2020-04-16 11:08:36 +0000 [info]: #0 starting fluentd worker pid=13280 ppid=6 worker=0
2020-04-16 11:08:36 +0000 [error]: #0 unexpected error error_class=RestClient::Unauthorized error="401 Unauthorized: {\"error\":{\"code\":401,\"message\":\"The request you have made requires authentication.\",\"title\":\"Unauthorized\"}}\n"
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/rest-client-1.8.0/lib/restclient/abstract_response.rb:74:in `return!'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/rest-client-1.8.0/lib/restclient/request.rb:495:in `process_result'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/rest-client-1.8.0/lib/restclient/request.rb:421:in `block in transmit'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/2.4.0/net/http.rb:877:in `start'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/rest-client-1.8.0/lib/restclient/request.rb:413:in `transmit'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/rest-client-1.8.0/lib/restclient/request.rb:176:in `execute'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/rest-client-1.8.0/lib/restclient/request.rb:41:in `execute'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/rest-client-1.8.0/lib/restclient/resource.rb:67:in `post'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-monasca-output-0.0.1/lib/fluent/plugin/keystone/keystone_client.rb:38:in `request'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-monasca-output-0.0.1/lib/fluent/plugin/keystone/keystone_client.rb:31:in `authenticate'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-monasca-output-0.0.1/lib/fluent/plugin/out_monasca.rb:82:in `authenticate'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-monasca-output-0.0.1/lib/fluent/plugin/out_monasca.rb:42:in `start'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/compat/call_super_mixin.rb:42:in `start'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/root_agent.rb:200:in `block in start'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/root_agent.rb:189:in `block (2 levels) in lifecycle'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/root_agent.rb:188:in `each'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/root_agent.rb:188:in `block in lifecycle'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/root_agent.rb:175:in `each'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/root_agent.rb:175:in `lifecycle'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/root_agent.rb:199:in `start'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/engine.rb:248:in `start'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/engine.rb:147:in `run'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/supervisor.rb:592:in `block in run_worker'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/supervisor.rb:823:in `main_process'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/supervisor.rb:586:in `run_worker'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/command/fluentd.rb:338:in `<top (required)>'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/bin/fluentd:8:in `<top (required)>'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/bin/fluentd:23:in `load'
  2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/bin/fluentd:23:in `<top (required)>'
  2020-04-16 11:08:36 +0000 [error]: #0 /usr/sbin/td-agent:7:in `load'
  2020-04-16 11:08:36 +0000 [error]: #0 /usr/sbin/td-agent:7:in `<main>'
2020-04-16 11:08:36 +0000 [error]: #0 unexpected error error_class=RestClient::Unauthorized error="401 Unauthorized: {\"error\":{\"code\":401,\"message\":\"The request you have made requires authentication.\",\"title\":\"Unauthorized\"}}\n"
  2020-04-16 11:08:36 +0000 [error]: #0 suppressed same stacktrace

**Environment**:
OS: Ubuntu 18.04
Kernel: 4.15.0-91-generic
Docker: 19.03.8
Type: source
Official images from Docker Hub, but retagged and pushed to our local nexus.

cheers,
Ole

See original description

Ole Kleinschmidt (oklhost) on 2020-04-17

description:

updated

Revision history for this message

Mark Goddard (mgoddard) wrote on 2020-05-04:

Thanks for raising this. It looks like we need to switch to the python 3 version of the mod_wsgi package on Debian/Ubuntu. I'll propose a fix.

Revision history for this message

Mark Goddard (mgoddard) wrote on 2020-05-05:

Fix proposed to kolla: https://review.opendev.org/#/c/725221/

no longer affects:	kolla-ansible
no longer affects:	kolla-ansible/train
no longer affects:	kolla-ansible/ussuri
summary:	- monasca-api seems broken + monasca-api use wrong mod_wsgi Python version on Debian/Ubuntu source

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-05-08: Fix merged to kolla (master)

Reviewed: https://review.opendev.org/725221
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=f5d31da12fa66cedfc2e270b8cd3be8e2bb9254b
Submitter: Zuul
Branch: master

commit f5d31da12fa66cedfc2e270b8cd3be8e2bb9254b
Author: Mark Goddard <email address hidden>
Date: Mon May 4 10:32:47 2020 +0100

Use Python 3 for mod_wsgi in cyborg and monasca on Debian/Ubuntu

    In Train kolla switched to Python 3 in Debian and Ubuntu source images.
    For services running under httpd with mod_wsgi, they should install
    libapache2-mod-wsgi-py3 rather than libapache2-mod-wsgi. This was done
    for most images, but cyborg and monasca were omitted. This change fixes
    that.

Change-Id: I25cfa62dbf20490685617b4a4185bf95ab91725e
Closes-Bug: #1873421

Changed in kolla:
status:	Triaged → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-05-08: Fix proposed to kolla (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/726460

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-05-11: Fix merged to kolla (stable/train)

Reviewed: https://review.opendev.org/726460
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=f221b4f7ab7745ee20c8e97d6a1a2d3c7e74f466
Submitter: Zuul
Branch: stable/train

commit f221b4f7ab7745ee20c8e97d6a1a2d3c7e74f466
Author: Mark Goddard <email address hidden>
Date: Mon May 4 10:32:47 2020 +0100

Use Python 3 for mod_wsgi in cyborg and monasca on Debian/Ubuntu

    Change-Id: I25cfa62dbf20490685617b4a4185bf95ab91725e
    Closes-Bug: #1873421
    (cherry picked from commit f5d31da12fa66cedfc2e270b8cd3be8e2bb9254b)

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.