Activity log for bug #1873421

Date Who What changed Old value New value Message
2020-04-17 09:28:46 Ole Kleinschmidt bug added bug
2020-04-17 09:43:19 Ole Kleinschmidt description **Bug Report** It seems there is a bug in the monasca-api config and/or the docker image. After we have enabled monasca in globals.yaml and the deployment has been completed successfully, the error log fills up with the following entries: tail -f /var/log/kolla/monasca/monasca-api-error.log [Fri Apr 17 07:31:21.438942 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] mod_wsgi (pid=26): Target WSGI script '/monasca-api/monasca_api/api/wsgi.py' cannot be loaded as Python module. [Fri Apr 17 07:31:21.438996 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] mod_wsgi (pid=26): Exception occurred processing WSGI script '/monasca-api/monasca_api/api/wsgi.py'. [Fri Apr 17 07:31:21.439017 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] Traceback (most recent call last): [Fri Apr 17 07:31:21.439033 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] File "/monasca-api/monasca_api/api/wsgi.py", line 18, in <module> [Fri Apr 17 07:31:21.439084 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] from monasca_api.api import server [Fri Apr 17 07:31:21.439095 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] File "/var/lib/kolla/venv/lib/python3.6/site-packages/monasca_api/api/server.py", line 21, in <module> [Fri Apr 17 07:31:21.439144 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] from oslo_config import cfg [Fri Apr 17 07:31:21.439153 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_config/cfg.py", line 39, in <module> [Fri Apr 17 07:31:21.439575 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] import enum [Fri Apr 17 07:31:21.439593 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] ImportError: No module named enum We use the kolla/ubuntu-source-*:train docker-images. It looks to me like Apache in the monasca_api container is using python2.7: # docker exec -it monasca_api /bin/bash (monasca-api)[root@cXXXX /]# ll /usr/lib/apache2/modules/mod_wsgi.so lrwxrwxrwx 1 root root 15 Feb 14 11:08 /usr/lib/apache2/modules/mod_wsgi.so -> mod_wsgi.so-2.7 (monasca-api)[root@cXXXX /]# **Environment**: OS: Ubuntu 18.04 Kernel: 4.15.0-91-generic Docker: 19.03.8 Type: source Official images from Docker Hub, but retagged and pushed to our local nexus. But in the log above, python3.6 modules are called. cheers, Ole **Bug Report** It seems there is a bug in the monasca-api config and/or the docker image. After we have enabled monasca in globals.yaml and the deployment has been completed successfully, the error log fills up with the following entries: tail -f /var/log/kolla/monasca/monasca-api-error.log [Fri Apr 17 07:31:21.438942 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] mod_wsgi (pid=26): Target WSGI script '/monasca-api/monasca_api/api/wsgi.py' cannot be loaded as Python module. [Fri Apr 17 07:31:21.438996 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] mod_wsgi (pid=26): Exception occurred processing WSGI script '/monasca-api/monasca_api/api/wsgi.py'. [Fri Apr 17 07:31:21.439017 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] Traceback (most recent call last): [Fri Apr 17 07:31:21.439033 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] File "/monasca-api/monasca_api/api/wsgi.py", line 18, in <module> [Fri Apr 17 07:31:21.439084 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] from monasca_api.api import server [Fri Apr 17 07:31:21.439095 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] File "/var/lib/kolla/venv/lib/python3.6/site-packages/monasca_api/api/server.py", line 21, in <module> [Fri Apr 17 07:31:21.439144 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] from oslo_config import cfg [Fri Apr 17 07:31:21.439153 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_config/cfg.py", line 39, in <module> [Fri Apr 17 07:31:21.439575 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] import enum [Fri Apr 17 07:31:21.439593 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] ImportError: No module named enum We use the kolla/ubuntu-source-*:train docker-images. It looks to me like Apache in the monasca_api container is using python2.7: # docker exec -it monasca_api /bin/bash (monasca-api)[root@cXXXX /]# ll /usr/lib/apache2/modules/mod_wsgi.so lrwxrwxrwx 1 root root 15 Feb 14 11:08 /usr/lib/apache2/modules/mod_wsgi.so -> mod_wsgi.so-2.7 (monasca-api)[root@cXXXX /]# But in the log above, python3.6 modules are called. Also the central-logging seems broken now: # docker logs fluentd 2020-04-16 11:08:36 +0000 [warn]: parameter 'type' in <match syslog.local0.**> @type copy type copy <store> @type "file" path "/var/log/kolla/swift/swift_latest.*.log" utc append true compress gzip <format> utc localtime false </format> <inject> localtime false </inject> <buffer time> path "/var/log/kolla/swift/swift_latest.*.log" </buffer> </store> <store> @type "monasca" keystone_url "http://10.38.1.253:5000" monasca_log_api "http://10.38.1.253:5607" monasca_log_api_version "v3.0" username "monasca-agent" password xxxxxx domain_id "default" project_name "monasca_control_plane" message_field_name "Payload" <buffer> flush_mode interval retry_type exponential_backoff </buffer> </store> </match> is not used. 2020-04-16 11:08:36 +0000 [warn]: parameter 'type' in <match syslog.local1.**> @type copy type copy <store> @type "file" path "/var/log/kolla/haproxy/haproxy_latest.*.log" output_tag false output_time false utc append true compress gzip <format> output_time false output_tag false utc localtime false </format> <inject> localtime false </inject> <buffer time> path "/var/log/kolla/haproxy/haproxy_latest.*.log" </buffer> </store> <store> @type "monasca" keystone_url "http://10.38.1.253:5000" monasca_log_api "http://10.38.1.253:5607" monasca_log_api_version "v3.0" username "monasca-agent" password xxxxxx domain_id "default" project_name "monasca_control_plane" message_field_name "Payload" <buffer> flush_mode interval retry_type exponential_backoff </buffer> </store> </match> is not used. 2020-04-16 11:08:36 +0000 [info]: #0 starting fluentd worker pid=13280 ppid=6 worker=0 2020-04-16 11:08:36 +0000 [error]: #0 unexpected error error_class=RestClient::Unauthorized error="401 Unauthorized: {\"error\":{\"code\":401,\"message\":\"The request you have made requires authentication.\",\"title\":\"Unauthorized\"}}\n" 2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/rest-client-1.8.0/lib/restclient/abstract_response.rb:74:in `return!' 2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/rest-client-1.8.0/lib/restclient/request.rb:495:in `process_result' 2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/rest-client-1.8.0/lib/restclient/request.rb:421:in `block in transmit' 2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/2.4.0/net/http.rb:877:in `start' 2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/rest-client-1.8.0/lib/restclient/request.rb:413:in `transmit' 2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/rest-client-1.8.0/lib/restclient/request.rb:176:in `execute' 2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/rest-client-1.8.0/lib/restclient/request.rb:41:in `execute' 2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/rest-client-1.8.0/lib/restclient/resource.rb:67:in `post' 2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-monasca-output-0.0.1/lib/fluent/plugin/keystone/keystone_client.rb:38:in `request' 2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-monasca-output-0.0.1/lib/fluent/plugin/keystone/keystone_client.rb:31:in `authenticate' 2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-monasca-output-0.0.1/lib/fluent/plugin/out_monasca.rb:82:in `authenticate' 2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-monasca-output-0.0.1/lib/fluent/plugin/out_monasca.rb:42:in `start' 2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/compat/call_super_mixin.rb:42:in `start' 2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/root_agent.rb:200:in `block in start' 2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/root_agent.rb:189:in `block (2 levels) in lifecycle' 2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/root_agent.rb:188:in `each' 2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/root_agent.rb:188:in `block in lifecycle' 2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/root_agent.rb:175:in `each' 2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/root_agent.rb:175:in `lifecycle' 2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/root_agent.rb:199:in `start' 2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/engine.rb:248:in `start' 2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/engine.rb:147:in `run' 2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/supervisor.rb:592:in `block in run_worker' 2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/supervisor.rb:823:in `main_process' 2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/supervisor.rb:586:in `run_worker' 2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/command/fluentd.rb:338:in `<top (required)>' 2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require' 2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require' 2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/bin/fluentd:8:in `<top (required)>' 2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/bin/fluentd:23:in `load' 2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/bin/fluentd:23:in `<top (required)>' 2020-04-16 11:08:36 +0000 [error]: #0 /usr/sbin/td-agent:7:in `load' 2020-04-16 11:08:36 +0000 [error]: #0 /usr/sbin/td-agent:7:in `<main>' 2020-04-16 11:08:36 +0000 [error]: #0 unexpected error error_class=RestClient::Unauthorized error="401 Unauthorized: {\"error\":{\"code\":401,\"message\":\"The request you have made requires authentication.\",\"title\":\"Unauthorized\"}}\n" 2020-04-16 11:08:36 +0000 [error]: #0 suppressed same stacktrace **Environment**: OS: Ubuntu 18.04 Kernel: 4.15.0-91-generic Docker: 19.03.8 Type: source Official images from Docker Hub, but retagged and pushed to our local nexus. cheers, Ole
2020-05-04 09:16:05 Mark Goddard bug added subscriber Doug Szumski
2020-05-04 09:31:40 Mark Goddard nominated for series kolla-ansible/train
2020-05-04 09:31:40 Mark Goddard bug task added kolla-ansible/train
2020-05-04 09:31:40 Mark Goddard nominated for series kolla-ansible/ussuri
2020-05-04 09:31:40 Mark Goddard bug task added kolla-ansible/ussuri
2020-05-04 09:31:44 Mark Goddard kolla-ansible/ussuri: importance Undecided Critical
2020-05-04 09:31:46 Mark Goddard kolla-ansible/ussuri: importance Critical Medium
2020-05-04 09:31:48 Mark Goddard kolla-ansible/train: importance Undecided Medium
2020-05-04 09:31:51 Mark Goddard kolla-ansible/ussuri: milestone 10.0.0
2020-05-04 09:31:53 Mark Goddard kolla-ansible/train: milestone 9.1.0
2020-05-04 09:31:58 Mark Goddard kolla-ansible/ussuri: status New Triaged
2020-05-04 09:32:01 Mark Goddard kolla-ansible/train: status New Triaged
2020-05-05 10:21:15 Mark Goddard kolla-ansible/train: milestone 9.1.0 9.1.1
2020-05-05 10:21:49 Mark Goddard bug task added kolla
2020-05-05 10:22:05 Mark Goddard nominated for series kolla/train
2020-05-05 10:22:05 Mark Goddard bug task added kolla/train
2020-05-05 10:22:05 Mark Goddard nominated for series kolla/ussuri
2020-05-05 10:22:05 Mark Goddard bug task added kolla/ussuri
2020-05-05 10:22:12 Mark Goddard kolla/ussuri: status New Triaged
2020-05-05 10:22:14 Mark Goddard kolla/train: status New Triaged
2020-05-05 10:22:17 Mark Goddard bug task deleted kolla-ansible
2020-05-05 10:22:23 Mark Goddard kolla/ussuri: importance Undecided Medium
2020-05-05 10:22:26 Mark Goddard kolla/train: importance Undecided Medium
2020-05-05 10:22:32 Mark Goddard bug task deleted kolla-ansible/train
2020-05-05 10:22:35 Mark Goddard bug task deleted kolla-ansible/ussuri
2020-05-05 10:22:43 Mark Goddard kolla/ussuri: milestone 10.0.0
2020-05-05 10:22:51 Mark Goddard kolla/train: milestone 9.1.0
2020-05-05 10:23:34 Mark Goddard summary monasca-api seems broken monasca-api use wrong mod_wsgi Python version on Debian/Ubuntu source
2020-05-08 19:07:51 OpenStack Infra kolla: status Triaged Fix Released
2020-05-08 19:14:00 OpenStack Infra kolla/train: status Triaged In Progress
2020-05-08 19:14:00 OpenStack Infra kolla/train: assignee Radosław Piliszek (yoctozepto)
2020-05-11 12:28:50 OpenStack Infra kolla/train: status In Progress Fix Committed