Cinder-WSGI cannot find api-paste.ini - Ubuntu

Bug #1715024 reported by Jon Davis on 2017-09-04
30
This bug affects 7 people
Affects Status Importance Assigned to Milestone
Cinder
Undecided
Unassigned
Ubuntu Cloud Archive
High
James Page
Pike
High
James Page
cinder (Ubuntu)
High
James Page

Bug Description

I upgraded Ocata to Pike and am getting error in apache2 cinder_error.log when trying to list volumes via API or Horizon:

root@cinder-test:/var/log/apache2# tail -f cinder_error.log
2017-09-04 15:30:52.587013 mod_wsgi (pid=1847): Target WSGI script '/usr/bin/cinder-wsgi' cannot be loaded as Python module.
2017-09-04 15:30:52.587099 mod_wsgi (pid=1847): Exception occurred processing WSGI script '/usr/bin/cinder-wsgi'.
2017-09-04 15:30:52.587126 Traceback (most recent call last):
2017-09-04 15:30:52.587147 File "/usr/bin/cinder-wsgi", line 51, in <module>
2017-09-04 15:30:52.587220 application = initialize_application()
2017-09-04 15:30:52.587235 File "/usr/lib/python2.7/dist-packages/cinder/wsgi/wsgi.py", line 46, in initialize_application
2017-09-04 15:30:52.588456 return wsgi.Loader(CONF).load_app(name='osapi_volume')
2017-09-04 15:30:52.588480 File "/usr/lib/python2.7/dist-packages/oslo_service/wsgi.py", line 340, in __init__
2017-09-04 15:30:52.589984 raise ConfigNotFound(path=config_path)
2017-09-04 15:30:52.590011 ConfigNotFound: Could not find config at api-paste.ini

Also did fresh controller install (Pike) on a test machine to see if I could replicate, using instructions: https://docs.openstack.org/install-guide/openstack-services.html#minimal-deployment

I get the same error, following docs closely.

Here is some output of the environment:

Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-93-generic x86_64)

################# PERMISSIONS ########################

root@cinder-test:/etc/cinder# ls -aFl
total 36
drwxr-x--- 3 root cinder 4096 Sep 4 15:10 ./
drwxr-xr-x 105 root root 4096 Sep 4 15:30 ../
-rw-r--r-- 1 root cinder 2711 Aug 30 11:07 api-paste.ini
-rw-r--r-- 1 root cinder 812 Sep 4 15:10 cinder.conf
-rw-r--r-- 1 root cinder 1359 Aug 30 13:11 logging.conf
-rw-r--r-- 1 root cinder 7708 Aug 30 11:07 policy.json
-rw-r--r-- 1 root root 991 Aug 30 11:07 rootwrap.conf
drwxr-xr-x 2 root root 4096 Aug 31 11:49 rootwrap.d/

################# API-PASTE.INI ########################

#############
# OpenStack #
#############

[composite:osapi_volume]
use = call:cinder.api:root_app_factory
/: apiversions
/v1: openstack_volume_api_v1
/v2: openstack_volume_api_v2
/v3: openstack_volume_api_v3

[composite:openstack_volume_api_v1]
use = call:cinder.api.middleware.auth:pipeline_factory
noauth = cors http_proxy_to_wsgi request_id faultwrap sizelimit osprofiler noauth apiv1
keystone = cors http_proxy_to_wsgi request_id faultwrap sizelimit osprofiler authtoken keystonecontext apiv1
keystone_nolimit = cors http_proxy_to_wsgi request_id faultwrap sizelimit osprofiler authtoken keystonecontext apiv1

[composite:openstack_volume_api_v2]
use = call:cinder.api.middleware.auth:pipeline_factory
noauth = cors http_proxy_to_wsgi request_id faultwrap sizelimit osprofiler noauth apiv2
keystone = cors http_proxy_to_wsgi request_id faultwrap sizelimit osprofiler authtoken keystonecontext apiv2
keystone_nolimit = cors http_proxy_to_wsgi request_id faultwrap sizelimit osprofiler authtoken keystonecontext apiv2

[composite:openstack_volume_api_v3]
use = call:cinder.api.middleware.auth:pipeline_factory
noauth = cors http_proxy_to_wsgi request_id faultwrap sizelimit osprofiler noauth apiv3
keystone = cors http_proxy_to_wsgi request_id faultwrap sizelimit osprofiler authtoken keystonecontext apiv3
keystone_nolimit = cors http_proxy_to_wsgi request_id faultwrap sizelimit osprofiler authtoken keystonecontext apiv3

[filter:request_id]
paste.filter_factory = oslo_middleware.request_id:RequestId.factory

[filter:http_proxy_to_wsgi]
paste.filter_factory = oslo_middleware.http_proxy_to_wsgi:HTTPProxyToWSGI.factory

[filter:cors]
paste.filter_factory = oslo_middleware.cors:filter_factory
oslo_config_project = cinder

[filter:faultwrap]
paste.filter_factory = cinder.api.middleware.fault:FaultWrapper.factory

[filter:osprofiler]
paste.filter_factory = osprofiler.web:WsgiMiddleware.factory

[filter:noauth]
paste.filter_factory = cinder.api.middleware.auth:NoAuthMiddleware.factory

[filter:sizelimit]
paste.filter_factory = oslo_middleware.sizelimit:RequestBodySizeLimiter.factory

[app:apiv1]
paste.app_factory = cinder.api.v1.router:APIRouter.factory

[app:apiv2]
paste.app_factory = cinder.api.v2.router:APIRouter.factory

[app:apiv3]
paste.app_factory = cinder.api.v3.router:APIRouter.factory

[pipeline:apiversions]
pipeline = cors http_proxy_to_wsgi faultwrap osvolumeversionapp

[app:osvolumeversionapp]
paste.app_factory = cinder.api.versions:Versions.factory

##########
# Shared #
##########

[filter:keystonecontext]
paste.filter_factory = cinder.api.middleware.auth:CinderKeystoneContext.factory

[filter:authtoken]
paste.filter_factory = keystonemiddleware.auth_token:filter_factory

################# CINDER.CONF ########################

[DEFAULT]
rootwrap_config = /etc/cinder/rootwrap.conf
api_paste_confg = /etc/cinder/api-paste.ini
iscsi_helper = tgtadm
volume_name_template = volume-%s
volume_group = cinder-volumes
verbose = True
auth_strategy = keystone
state_path = /var/lib/cinder
lock_path = /var/lock/cinder
volumes_dir = /var/lib/cinder/volumes
enabled_backends = lvm
transport_url = rabbit://openstack:openstack@controller
my_ip = 10.1.1.22

[database]
connection = mysql+pymysql://cinder:openstack@controller/cinder

[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = cinder
password = openstack

[oslo_concurrency]
lock_path = /var/lib/cinder/tmp

################# APACHE WSGI ########################

Listen 8776
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" %D(us)" cinder_combined

<VirtualHost *:8776>
    WSGIDaemonProcess cinder-wsgi processes=5 threads=1 user=cinder display-name=%{GROUP}
    WSGIProcessGroup cinder-wsgi
    WSGIScriptAlias / /usr/bin/cinder-wsgi
    WSGIApplicationGroup %{GLOBAL}
    WSGIPassAuthorization On
    <IfVersion >= 2.4>
      ErrorLogFormat "%{cu}t %M"
    </IfVersion>

    ErrorLog /var/log/apache2/cinder_error.log
    CustomLog /var/log/apache2/cinder.log cinder_combined

    <Directory /usr/bin>
<IfVersion >= 2.4>
            Require all granted
        </IfVersion>
        <IfVersion < 2.4>
            Order allow,deny
            Allow from all
        </IfVersion>
    </Directory>
</VirtualHost>

Jon Davis (aggiering) on 2017-09-04
description: updated
Christoph Fiehe (cfiehe) wrote :

I can confirm this bug... I made a fresh OpenStack Pike installation on Ubuntu 16.04.3 LTS and receive the same error.

Some version information:
$ pip list | grep cinder
cinder (11.0.0)
python-cinderclient (3.1.0)

Changed in cinder:
status: New → Confirmed
Christoph Fiehe (cfiehe) wrote :

I got it working, even when it took me some hours.

It's just a permission problem that can be fixed as follows:
(1) Ensure that the cinder configuration folder is owned by the 'cinder' group:
    $ chown root:cinder /etc/cinder
(2) Modify the apache configuration and ensure that the process is executed using the 'cinder' group.
    WSGIDaemonProcess cinder-wsgi processes=5 threads=1 user=cinder group=cinder display-name=%{GROUP}
(3) Restart Apache:
    $ service apache2 restart

These steps fixed the problem described above.

Eric Harney (eharney) on 2017-09-05
Changed in cinder:
status: Confirmed → Invalid
David Seelbach (davedastrange) wrote :

can confirm
thanks wolle5050

Rufus Deponian (deponian) wrote :

Christoph Fiehe, thank you so much

Thomas Dreibholz (dreibh) wrote :
Download full text (4.2 KiB)

I can confirm the bug. On a fresh install of OpenStack Pike:

1. Run "cinder --debug list":
...
DEBUG:keystoneauth:REQ: curl -g -i -X GET http://nisse.simula.nornet:8776/v3/168529e95adc4284a40c28f648d48d6f/volumes/detail -H "User-Agent: python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}fc794cac0686f40616da08b48c22144c58d2745b"
DEBUG:keystoneauth:RESP: [500] Date: Thu, 14 Sep 2017 20:30:08 GMT Server: Apache/2.4.18 (Ubuntu) Content-Length: 618 Connection: close Content-Type: text/html; charset=iso-8859-1
RESP BODY: Omitted, Content-Type is set to text/html; charset=iso-8859-1. Only application/json responses have their bodies logged.

DEBUG (shell:974) Internal Server Error (HTTP 500)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cinderclient/shell.py", line 969, in main
    for item in sys.argv[1:]])
  File "/usr/lib/python2.7/dist-packages/cinderclient/shell.py", line 787, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.7/dist-packages/cinderclient/v3/shell.py", line 328, in do_list
    sort_dir=args.sort_dir, sort=args.sort)
  File "/usr/lib/python2.7/dist-packages/cinderclient/v2/volumes.py", line 321, in list
    return self._list(url, resource_type, limit=limit)
  File "/usr/lib/python2.7/dist-packages/cinderclient/base.py", line 84, in _list
    resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 177, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 168, in _cs_request
    return self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 154, in request
    raise exceptions.from_response(resp, body)
ClientException: Internal Server Error (HTTP 500)
DEBUG:cinderclient.shell:Internal Server Error (HTTP 500)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cinderclient/shell.py", line 969, in main
    for item in sys.argv[1:]])
  File "/usr/lib/python2.7/dist-packages/cinderclient/shell.py", line 787, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.7/dist-packages/cinderclient/v3/shell.py", line 328, in do_list
    sort_dir=args.sort_dir, sort=args.sort)
  File "/usr/lib/python2.7/dist-packages/cinderclient/v2/volumes.py", line 321, in list
    return self._list(url, resource_type, limit=limit)
  File "/usr/lib/python2.7/dist-packages/cinderclient/base.py", line 84, in _list
    resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 177, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 168, in _cs_request
    return self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 154, in request
    raise exceptions.from_response(resp, body)
ClientException: Internal Server Error (HTTP 500)
ERROR: Internal Server Error (HTTP 500)

2. Check /var/log/apache2/cinder_error.log
017-09-14 20:40:59.764418 mod_wsgi (pid=20203): Target WSGI script '/usr/bin/cinder-wsgi' cannot be loade...

Read more...

Thomas Dreibholz (dreibh) wrote :

/etc/apache2/conf-available/cinder-wsgi.conf is from the package "cinder-api".

Version:
nornetpp@nisse:~$ apt-show-versions | grep cinder-api
cinder-api:all/xenial-updates 2:11.0.0-0ubuntu1~cloud0 uptodate

Repository:
nornetpp@nisse:~$ cat /etc/apt/sources.list.d/cloudarchive-pike.list
deb http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/pike main
deb-src http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/pike main

tags: added: xenial
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in cinder (Ubuntu):
status: New → Confirmed

I am also facing the same issue in a fresh pike installation, However the solution "Christoph Fiehe (cfiehe)" Suggested Works Fine.

James Page (james-page) on 2017-09-18
Changed in cinder (Ubuntu):
importance: Undecided → High
status: Confirmed → Triaged
assignee: nobody → James Page (james-page)
James Page (james-page) on 2017-09-18
Changed in cinder (Ubuntu):
status: Triaged → In Progress
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cinder - 2:11.0.0-0ubuntu2

---------------
cinder (2:11.0.0-0ubuntu2) artful; urgency=medium

  * d/cinder-wsgi.conf: Ensure apache processes run under cinder group,
    allowing access to /etc/cinder/* (LP: #1715024).

 -- James Page <email address hidden> Mon, 18 Sep 2017 09:24:40 +0100

Changed in cinder (Ubuntu):
status: In Progress → Fix Released

Hello Jon, or anyone else affected,

Accepted cinder into pike-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:pike-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-pike-needed to verification-pike-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-pike-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-pike-needed

The verification of the Stable Release Update for cinder has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

James Page (james-page) wrote :

This bug was fixed in the package cinder - 2:11.0.0-0ubuntu2~cloud0
---------------

 cinder (2:11.0.0-0ubuntu2~cloud0) xenial-pike; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 cinder (2:11.0.0-0ubuntu2) artful; urgency=medium
 .
   * d/cinder-wsgi.conf: Ensure apache processes run under cinder group,
     allowing access to /etc/cinder/* (LP: #1715024).

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers