neutron

Cannot run neutron-api via mod_wsgi

Bug #1864418 reported by yj.bai
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
kolla
Invalid
Undecided
Unassigned
neutron
Confirmed
Undecided
Unassigned

Bug Description

I started the neutron api with apache in the docker container use kolla.There was no problem with the first three requests (use openstack networking list) after startup. but the fourth started to be problematic.
I started the RPC service:
# /usr/bin/neutron-rpc-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini

I refer to this article : https://docs.openstack.org/neutron/latest/admin/config-wsgi.html

/etc/httpd/conf.d/wsgi-neutron.conf

Listen [fd00:1001::101]:9696

<Directory "/var/lib/kolla/venv/bin">
    <FilesMatch "neutron-api">
        AllowOverride None
        Options None
        Require all granted
    </FilesMatch>
</Directory>

<VirtualHost *:9696>
    WSGIDaemonProcess neutron-server processes=1 threads=1 user=neutron group=neutron display-name=%{GROUP}
    WSGIProcessGroup neutron-server
    WSGIScriptAlias / /var/lib/kolla/venv/bin/neutron-api
    WSGIApplicationGroup %{GLOBAL}
    WSGIPassAuthorization On
    <IfVersion >= 2.4>
      ErrorLogFormat "%{cu}t %M"
    </IfVersion>
    ErrorLog "/var/log/neutron/neutron-error.log"
    LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b %D \"%{Referer}i\" \"%{User-Agent}i\"" logformat
    CustomLog "/var/log/neutron/neutron-error.log" logformat
</VirtualHost>

Alias /networking /var/lib/kolla/venv/bin/neutron-api
<Location /networking>
    SetHandler wsgi-script
    Options +ExecCGI
    WSGIProcessGroup neutron-server
    WSGIApplicationGroup %{GLOBAL}
    WSGIPassAuthorization On
</Location>

WSGISocketPrefix /var/run/apache2

The following problem occurs after the fourth request is sent

var/log/kolla/neutron/neutron-api.log

2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors [req-dc4bf26f-7078-4857-978e-60bffa96df16 51a6efff6aa34b44926fcb95c5a01b3b e474c27f32bf462d87eecb392f373424 - default default] An error occurred during processing the request: GET /v2.0/networks HTTP/1.1^M
Accept: application/json^M
Accept-Encoding: gzip, deflate^M
Connection: keep-alive^M
Host: [fd00:1001::111]:9696^M
User-Agent: openstacksdk/0.38.0 keystoneauth1/3.18.0 python-requests/2.22.0 CPython/2.7.5^M
X-Auth-Token: *****
X-Forwarded-For: fd00:1001::111^M
X-Forwarded-Proto: https: AttributeError: 'module' object has no attribute 'poll'
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors Traceback (most recent call last):
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_middleware/catch_errors.py", line 40, in __call__
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors response = req.get_response(self.application)
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/request.py", line 1314, in send
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors application, catch_exc_info=False)
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/request.py", line 1278, in call_application
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors app_iter = application(self.environ, start_response)
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/dec.py", line 129, in __call__
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors resp = self.call_func(req, *args, **kw)
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/dec.py", line 193, in call_func
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors return self.func(req, *args, **kwargs)
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/osprofiler/web.py", line 112, in __call__
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors return request.get_response(self.application)
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/request.py", line 1314, in send
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors application, catch_exc_info=False)
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/request.py", line 1278, in call_application
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors app_iter = application(self.environ, start_response)
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/dec.py", line 129, in __call__
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors resp = self.call_func(req, *args, **kw)
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/webob/dec.py", line 193, in call_func
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors return self.func(req, *args, **kwargs)
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 338, in __call__
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors response = self.process_request(req)
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 659, in process_request
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors resp = super(AuthProtocol, self).process_request(request)
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 411, in process_request
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors allow_expired=allow_expired)
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 445, in _do_fetch_token
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors data = self.fetch_token(token, **kwargs)
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 754, in fetch_token
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors allow_expired=allow_expired)
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystonemiddleware/auth_token/_identity.py", line 180, in verify_token
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors allow_expired=allow_expired)
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystonemiddleware/auth_token/_identity.py", line 75, in verify_token
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors access_rules_support=ACCESS_RULES_SUPPORT)
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneclient/v3/tokens.py", line 120, in validate
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors access_rules_support=access_rules_support)
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneclient/v3/tokens.py", line 93, in get_token_data
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors resp, body = self._client.get(url, headers=headers)
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 386, in get
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors return self.request(url, 'GET', **kwargs)
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 545, in request
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 248, in request
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors return self.session.request(url, method, **kwargs)
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 248, in request
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors return self.session.request(url, method, **kwargs)
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/session.py", line 888, in request
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors resp = send(**kwargs)
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/session.py", line 979, in _send_request
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors resp = self.session.request(method, url, **kwargs)
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/requests/sessions.py", line 533, in request
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors resp = self.send(prep, **send_kwargs)
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/requests/sessions.py", line 646, in send
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors r = adapter.send(request, **kwargs)
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/requests/adapters.py", line 449, in send
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors timeout=timeout
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/urllib3/connectionpool.py", line 591, in urlopen
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors conn = self._get_conn(timeout=pool_timeout)
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/urllib3/connectionpool.py", line 240, in _get_conn
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors if conn and is_connection_dropped(conn):
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/urllib3/util/connection.py", line 24, in is_connection_dropped
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors return wait_for_read(sock, timeout=0.0)
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/urllib3/util/wait.py", line 143, in wait_for_read
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors return wait_for_socket(sock, read=True, timeout=timeout)
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python2.7/site-packages/urllib3/util/wait.py", line 95, in poll_wait_for_socket
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors poll_obj = select.poll()
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors AttributeError: 'module' object has no attribute 'poll'
2020-02-23 22:04:39.272 861 ERROR oslo_middleware.catch_errors

I add httpd and mod_wsgi packages to neutron server image.
I success to use openstack network list command three times or two times.
but form the third or fourth times this wrong appeared.

See original description
Revision history for this message
Bernard Cafarelli (bcafarel) wrote :

From a quick read it looks like some issue in kolla packages, adding them

yj.bai (baiyj)
description: updated
Revision history for this message
yj.bai (baiyj) wrote :

What packages do you think are missing here
https://review.opendev.org/#/c/708810/

Revision history for this message
YAMAMOTO Takashi (yamamoto) wrote :

it seems that "module" here is "select" module urllib3 imported.
for some reasons your environment doesn't have select.poll?

Revision history for this message
yj.bai (baiyj) wrote :

my environment have select.poll my enviroment can use openstack network list three times
if doesn`t have select.poll can not be seccess three times.

Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

yj.bai is working on that for kolla, it is not a bug in the current state of kolla.

Indeed it is weird that the 4th request fails. Could it be the case that WSGI pool gets poisoned by the first three requests?
I believe this request is about neutron having difficulty being run behind apache/mod_wsgi.

I subscribed to this report, please mention me if you need help from kolla side.

Changed in kolla:
status: New → Invalid
Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

So, it has been redone for Kolla by James Kirsch and we hit the same issue.
Seemingly one cannot put Neutron API behind mod_wsgi (or exactly urllib3 behind mod_wsgi).
Could it be missing Apache config?

Changed in neutron:
status: New → Confirmed
summary: - has wrong with use apache to start neutron api in docker container
+ Cannot run neutron-api via mod_wsgi
Revision history for this message
Radosław Piliszek (yoctozepto) wrote :
Download full text (5.9 KiB)

With advanced memcache pool we get:

2020-09-24 00:04:18.797 946 ERROR oslo_middleware.catch_errors Traceback (most recent call last):
2020-09-24 00:04:18.797 946 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_middleware/catch_errors.py", line 40, in __call__
2020-09-24 00:04:18.797 946 ERROR oslo_middleware.catch_errors response = req.get_response(self.application)
2020-09-24 00:04:18.797 946 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python3.8/site-packages/webob/request.py", line 1313, in send
2020-09-24 00:04:18.797 946 ERROR oslo_middleware.catch_errors status, headers, app_iter = self.call_application(
2020-09-24 00:04:18.797 946 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python3.8/site-packages/webob/request.py", line 1278, in call_application
2020-09-24 00:04:18.797 946 ERROR oslo_middleware.catch_errors app_iter = application(self.environ, start_response)
2020-09-24 00:04:18.797 946 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python3.8/site-packages/webob/dec.py", line 129, in __call__
2020-09-24 00:04:18.797 946 ERROR oslo_middleware.catch_errors resp = self.call_func(req, *args, **kw)
2020-09-24 00:04:18.797 946 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python3.8/site-packages/webob/dec.py", line 193, in call_func
2020-09-24 00:04:18.797 946 ERROR oslo_middleware.catch_errors return self.func(req, *args, **kwargs)
2020-09-24 00:04:18.797 946 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python3.8/site-packages/osprofiler/web.py", line 112, in __call__
2020-09-24 00:04:18.797 946 ERROR oslo_middleware.catch_errors return request.get_response(self.application)
2020-09-24 00:04:18.797 946 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python3.8/site-packages/webob/request.py", line 1313, in send
2020-09-24 00:04:18.797 946 ERROR oslo_middleware.catch_errors status, headers, app_iter = self.call_application(
2020-09-24 00:04:18.797 946 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python3.8/site-packages/webob/request.py", line 1278, in call_application
2020-09-24 00:04:18.797 946 ERROR oslo_middleware.catch_errors app_iter = application(self.environ, start_response)
2020-09-24 00:04:18.797 946 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python3.8/site-packages/webob/dec.py", line 129, in __call__
2020-09-24 00:04:18.797 946 ERROR oslo_middleware.catch_errors resp = self.call_func(req, *args, **kw)
2020-09-24 00:04:18.797 946 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python3.8/site-packages/webob/dec.py", line 193, in call_func
2020-09-24 00:04:18.797 946 ERROR oslo_middleware.catch_errors return self.func(req, *args, **kwargs)
2020-09-24 00:04:18.797 946 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python3.8/site-packages/keystonemiddleware/auth_token/__init__.py", line 338, in __call__
2020-09-24 00:04:18.797 946 ERROR oslo_middleware.catch_errors response = self.process_request(req)
2020-09-24 00:04:18.797 946 ERROR o...

Read more...

Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

Considering the memcache pool issues that only neutron seems to be experiencing and this being both related to keystone middleware, I suspect there is some subtle threading issue

Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

^ I'm thinking about: https://bugs.launchpad.net/oslo.cache/+bug/1883659

Revision history for this message
Rabi Mishra (rabi) wrote :

This seems to be due to eventlet monkey-patching[1]

[1] https://github.com/openstack/neutron/blob/master/neutron/cmd/eventlet/__init__.py#L15

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.