| 2020-04-17 09:43:19 |
Ole Kleinschmidt |
description |
**Bug Report**
It seems there is a bug in the monasca-api config and/or the docker image. After we have enabled monasca in globals.yaml and the deployment has been completed successfully, the error log fills up with the following entries:
tail -f /var/log/kolla/monasca/monasca-api-error.log
[Fri Apr 17 07:31:21.438942 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] mod_wsgi (pid=26): Target WSGI script '/monasca-api/monasca_api/api/wsgi.py' cannot be loaded as Python module.
[Fri Apr 17 07:31:21.438996 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] mod_wsgi (pid=26): Exception occurred processing WSGI script '/monasca-api/monasca_api/api/wsgi.py'.
[Fri Apr 17 07:31:21.439017 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] Traceback (most recent call last):
[Fri Apr 17 07:31:21.439033 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] File "/monasca-api/monasca_api/api/wsgi.py", line 18, in <module>
[Fri Apr 17 07:31:21.439084 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] from monasca_api.api import server
[Fri Apr 17 07:31:21.439095 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] File "/var/lib/kolla/venv/lib/python3.6/site-packages/monasca_api/api/server.py", line 21, in <module>
[Fri Apr 17 07:31:21.439144 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] from oslo_config import cfg
[Fri Apr 17 07:31:21.439153 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_config/cfg.py", line 39, in <module>
[Fri Apr 17 07:31:21.439575 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] import enum
[Fri Apr 17 07:31:21.439593 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] ImportError: No module named enum
We use the kolla/ubuntu-source-*:train docker-images. It looks to me like Apache in the monasca_api container is using python2.7:
# docker exec -it monasca_api /bin/bash
(monasca-api)[root@cXXXX /]# ll /usr/lib/apache2/modules/mod_wsgi.so
lrwxrwxrwx 1 root root 15 Feb 14 11:08 /usr/lib/apache2/modules/mod_wsgi.so -> mod_wsgi.so-2.7
(monasca-api)[root@cXXXX /]#
**Environment**:
OS: Ubuntu 18.04
Kernel: 4.15.0-91-generic
Docker: 19.03.8
Type: source
Official images from Docker Hub, but retagged and pushed to our local nexus.
But in the log above, python3.6 modules are called.
cheers,
Ole |
**Bug Report**
It seems there is a bug in the monasca-api config and/or the docker image. After we have enabled monasca in globals.yaml and the deployment has been completed successfully, the error log fills up with the following entries:
tail -f /var/log/kolla/monasca/monasca-api-error.log
[Fri Apr 17 07:31:21.438942 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] mod_wsgi (pid=26): Target WSGI script '/monasca-api/monasca_api/api/wsgi.py' cannot be loaded as Python module.
[Fri Apr 17 07:31:21.438996 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] mod_wsgi (pid=26): Exception occurred processing WSGI script '/monasca-api/monasca_api/api/wsgi.py'.
[Fri Apr 17 07:31:21.439017 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] Traceback (most recent call last):
[Fri Apr 17 07:31:21.439033 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] File "/monasca-api/monasca_api/api/wsgi.py", line 18, in <module>
[Fri Apr 17 07:31:21.439084 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] from monasca_api.api import server
[Fri Apr 17 07:31:21.439095 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] File "/var/lib/kolla/venv/lib/python3.6/site-packages/monasca_api/api/server.py", line 21, in <module>
[Fri Apr 17 07:31:21.439144 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] from oslo_config import cfg
[Fri Apr 17 07:31:21.439153 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_config/cfg.py", line 39, in <module>
[Fri Apr 17 07:31:21.439575 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] import enum
[Fri Apr 17 07:31:21.439593 2020] [wsgi:error] [pid 26:tid 139675178346240] [remote 10.38.1.14:35394] ImportError: No module named enum
We use the kolla/ubuntu-source-*:train docker-images. It looks to me like Apache in the monasca_api container is using python2.7:
# docker exec -it monasca_api /bin/bash
(monasca-api)[root@cXXXX /]# ll /usr/lib/apache2/modules/mod_wsgi.so
lrwxrwxrwx 1 root root 15 Feb 14 11:08 /usr/lib/apache2/modules/mod_wsgi.so -> mod_wsgi.so-2.7
(monasca-api)[root@cXXXX /]#
But in the log above, python3.6 modules are called. Also the central-logging seems broken now:
# docker logs fluentd
2020-04-16 11:08:36 +0000 [warn]: parameter 'type' in <match syslog.local0.**>
@type copy
type copy
<store>
@type "file"
path "/var/log/kolla/swift/swift_latest.*.log"
utc
append true
compress gzip
<format>
utc
localtime false
</format>
<inject>
localtime false
</inject>
<buffer time>
path "/var/log/kolla/swift/swift_latest.*.log"
</buffer>
</store>
<store>
@type "monasca"
keystone_url "http://10.38.1.253:5000"
monasca_log_api "http://10.38.1.253:5607"
monasca_log_api_version "v3.0"
username "monasca-agent"
password xxxxxx
domain_id "default"
project_name "monasca_control_plane"
message_field_name "Payload"
<buffer>
flush_mode interval
retry_type exponential_backoff
</buffer>
</store>
</match> is not used.
2020-04-16 11:08:36 +0000 [warn]: parameter 'type' in <match syslog.local1.**>
@type copy
type copy
<store>
@type "file"
path "/var/log/kolla/haproxy/haproxy_latest.*.log"
output_tag false
output_time false
utc
append true
compress gzip
<format>
output_time false
output_tag false
utc
localtime false
</format>
<inject>
localtime false
</inject>
<buffer time>
path "/var/log/kolla/haproxy/haproxy_latest.*.log"
</buffer>
</store>
<store>
@type "monasca"
keystone_url "http://10.38.1.253:5000"
monasca_log_api "http://10.38.1.253:5607"
monasca_log_api_version "v3.0"
username "monasca-agent"
password xxxxxx
domain_id "default"
project_name "monasca_control_plane"
message_field_name "Payload"
<buffer>
flush_mode interval
retry_type exponential_backoff
</buffer>
</store>
</match> is not used.
2020-04-16 11:08:36 +0000 [info]: #0 starting fluentd worker pid=13280 ppid=6 worker=0
2020-04-16 11:08:36 +0000 [error]: #0 unexpected error error_class=RestClient::Unauthorized error="401 Unauthorized: {\"error\":{\"code\":401,\"message\":\"The request you have made requires authentication.\",\"title\":\"Unauthorized\"}}\n"
2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/rest-client-1.8.0/lib/restclient/abstract_response.rb:74:in `return!'
2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/rest-client-1.8.0/lib/restclient/request.rb:495:in `process_result'
2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/rest-client-1.8.0/lib/restclient/request.rb:421:in `block in transmit'
2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/2.4.0/net/http.rb:877:in `start'
2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/rest-client-1.8.0/lib/restclient/request.rb:413:in `transmit'
2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/rest-client-1.8.0/lib/restclient/request.rb:176:in `execute'
2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/rest-client-1.8.0/lib/restclient/request.rb:41:in `execute'
2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/rest-client-1.8.0/lib/restclient/resource.rb:67:in `post'
2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-monasca-output-0.0.1/lib/fluent/plugin/keystone/keystone_client.rb:38:in `request'
2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-monasca-output-0.0.1/lib/fluent/plugin/keystone/keystone_client.rb:31:in `authenticate'
2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-monasca-output-0.0.1/lib/fluent/plugin/out_monasca.rb:82:in `authenticate'
2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-monasca-output-0.0.1/lib/fluent/plugin/out_monasca.rb:42:in `start'
2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/compat/call_super_mixin.rb:42:in `start'
2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/root_agent.rb:200:in `block in start'
2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/root_agent.rb:189:in `block (2 levels) in lifecycle'
2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/root_agent.rb:188:in `each'
2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/root_agent.rb:188:in `block in lifecycle'
2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/root_agent.rb:175:in `each'
2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/root_agent.rb:175:in `lifecycle'
2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/root_agent.rb:199:in `start'
2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/engine.rb:248:in `start'
2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/engine.rb:147:in `run'
2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/supervisor.rb:592:in `block in run_worker'
2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/supervisor.rb:823:in `main_process'
2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/supervisor.rb:586:in `run_worker'
2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/lib/fluent/command/fluentd.rb:338:in `<top (required)>'
2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.9.2/bin/fluentd:8:in `<top (required)>'
2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/bin/fluentd:23:in `load'
2020-04-16 11:08:36 +0000 [error]: #0 /opt/td-agent/embedded/bin/fluentd:23:in `<top (required)>'
2020-04-16 11:08:36 +0000 [error]: #0 /usr/sbin/td-agent:7:in `load'
2020-04-16 11:08:36 +0000 [error]: #0 /usr/sbin/td-agent:7:in `<main>'
2020-04-16 11:08:36 +0000 [error]: #0 unexpected error error_class=RestClient::Unauthorized error="401 Unauthorized: {\"error\":{\"code\":401,\"message\":\"The request you have made requires authentication.\",\"title\":\"Unauthorized\"}}\n"
2020-04-16 11:08:36 +0000 [error]: #0 suppressed same stacktrace
**Environment**:
OS: Ubuntu 18.04
Kernel: 4.15.0-91-generic
Docker: 19.03.8
Type: source
Official images from Docker Hub, but retagged and pushed to our local nexus.
cheers,
Ole |
|
