glance-image-import.conf not parsed when running under wsgi

[Abhishek Kekane]

When development environment is set to run under wsgi mode (by setting WSGI_MODE=mod_wsgi in local.conf) glance-image-import.conf file is not parsed.

It is working properly if glance is running under uwsgi.

Steps to reproduce:
NOTE: Here I am trying to use plugin "inject_image_metadata" to inject metadata properties to the image.

1. Add glance-image-import.conf as mentioned at /etc/glance
   [image_import_opts]
   image_import_plugins = ["inject_image_metadata"]

   [inject_metadata_properties]
   inject = "property1":"value"
   ignore_user_roles = demo

2. Restart glance api service using "<email address hidden>"
3. Source using admin credentials (as we are ignoring demo role in glance-image-import.conf)
   $ source devstack/openrc admin admin
3. Create image using import api
   $ glance image-create-via-import --container-format bare --disk-format qcow2 --name ceph_image_default --file temp.qcow2

Expected Output:
Properties mentioned in glance-image-import.conf should be injected to the image

Actual Output:
Properties are not injected as plugin is not loaded.

[Abhishek Kekane (abhishek-kekane)]

Workaround:
we can fix this issue by changing ExecStart mentioned in /<email address hidden> unit file to

ExecStart = /usr/local/bin/glance-api --config-dir /etc/glance

or

Make changes in glance/cmd/api.py as mentioned in [1] and then changing ExecStart mentioned in /<email address hidden> unit file to

ExecStart = /usr/local/bin/glance-api

NOTE:
After modifying <email address hidden> we need to relaod deamon using "sudo systemctl daemon-reload" command

[1] http://paste.openstack.org/show/722963/

[Erno Kuvaja (jokke)]

Taken into consideration that we don't currently support nor encourage anyone running under wsgi mode either mod_wsgi nor uwsgi , I'd say the priority for this bug is very low, specially as you mentioned in your comment the fix needs to happen in Devstack rather than in Glance.

[Paul Bourke (pauldbourke)]

Just to add a slightly different perspective on this:

I'm deploying Glance via kolla-ansible, which does *not* use any of the wsgi deployment modes, sticking to the default out of the box eventlet model. After following the guide to enable metadata injection[0], the config appears to be completely ignored, and none of the relevant code paths are touched (verified via debugging).

The only part of the code I can see that touches glance-image-import.conf is wsgi_app.py[1], which runs contrary to the statement that this only works via eventlet.

Can someone verify this has been tested using eventlet?

[0] https://docs.openstack.org/glance/latest/admin/interoperable-image-import.html#configuring-the-glance-direct-method
[1] https://github.com/openstack/glance/blob/master/glance/common/wsgi_app.py#L29

[Abhishek Kekane (abhishek-kekane)]

This bug is fixed in devstack by rosmaita.
Please refer https://review.openstack.org/#/c/625628/

Marking invalid as it no more affects glance.

[Brian Rosmaita (brian-rosmaita)]

Actually, I don't think this is fixed. Re-opening.

[Brian Rosmaita (brian-rosmaita)]

This is a real bug.

@Erno: setting WSGI_MODE=mod_wsgi in local.conf makes glance NOT use uwsgi and instead use the eventlet-based server, it doesn't really use mod_wsgi. So this problem is occurring when Glance is run in the recommended way.

[0] https://github.com/openstack-dev/devstack/blob/13e260ea2c3220a217626b12bff538916fdd2caf/lib/glance#L345-L349

[Brian Rosmaita (brian-rosmaita)]

@Abhishek: forgive me for asking, but are you sure that just changing the --config-dir as you mention in comment #2 is enough to get this working?

[Abhishek Kekane (abhishek-kekane)]

Hi Brian,
I have tested that, it was working, and sure it will work.

Abhishek

On Mon, 1 Apr 2019 at 10:01 PM, Brian Rosmaita <email address hidden>
wrote:

> @Abhishek: forgive me for asking, but are you sure that just changing
> the --config-dir as you mention in comment #2 is enough to get this
> working?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1775782
>
> Title:
> glance-image-import.conf not parsed when running under wsgi
>
> Status in Glance:
> Triaged
>
> Bug description:
> When development environment is set to run under wsgi mode (by setting
> WSGI_MODE=mod_wsgi in local.conf) glance-image-import.conf file is not
> parsed.
>
> It is working properly if glance is running under uwsgi.
>
> Steps to reproduce:
> NOTE: Here I am trying to use plugin "inject_image_metadata" to inject
> metadata properties to the image.
>
> 1. Add glance-image-import.conf as mentioned at /etc/glance
> [image_import_opts]
> image_import_plugins = ["inject_image_metadata"]
>
> [inject_metadata_properties]
> inject = "property1":"value"
> ignore_user_roles = demo
>
> 2. Restart glance api service using "<email address hidden>"
> 3. Source using admin credentials (as we are ignoring demo role in
> glance-image-import.conf)
> $ source devstack/openrc admin admin
> 3. Create image using import api
> $ glance image-create-via-import --container-format bare
> --disk-format qcow2 --name ceph_image_default --file temp.qcow2
>
> Expected Output:
> Properties mentioned in glance-image-import.conf should be injected to
> the image
>
> Actual Output:
> Properties are not injected as plugin is not loaded.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/glance/+bug/1775782/+subscriptions
>
--
Thanks & Best Regards,

Abhishek Kekane

[Brian Rosmaita (brian-rosmaita)]

We need to reopen this. I couldn't get devstack to start earlier this week (with a cherry-pick of my don-t-use-uwsgi patch) because glance-api kept timing out when stack.sh was checking to see if the API was available. Turns out that the API was being started on the same port as the registry (19191, I had tls-proxy enabled; g-api should be listening on 19292). I think that just telling oslo-config what directory to look in is not specific enough, and it's reading the DEFAULT/bind_port from the wrong file (could be it's reading all the files, and since r > a, the registry setting wins.)

Anyway, using --config-file=/etc/glance/glance-api.conf allowed the API (and devstack) to start correctly. I didn't experiment with using both --config-dir and --donfig-file, and I'm not inclined to. I think we need to use code to find the config-dir that the loaded api.conf is in, and then look for the import.conf there, just like the uwsgi code is doing.

I don't think this is related to the problems being reported in #openstack-infra earlier this week, though they did sound kind of similar ... the difference is that they're using default devstack (which uses uwsgi) and they're apparently only seeing this happen on vexxhost. More info: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2019-04-23.log.html#t2019-04-23T13:28:43

[Brian Rosmaita (brian-rosmaita)]

@Abhishek: I don't doubt that you tested this, I thought it was working, too. I think it has to be tested in a completely clean environment (which is why I didn't mess with trying both --config-file and --config-dir in the exec statement in the systemd file). There also could be an element of randomness depending on the order oslo.config reads the files, though it was systematically bringing up the API on the registry port in my environment.

[Brian Rosmaita (brian-rosmaita)]

Probably need to revert devstack change https://review.openstack.org/#/c/625628/

[Abhishek Kekane (abhishek-kekane)]

As I mention in comment #1,

Make changes in glance/cmd/api.py as mentioned in [1] and then changing ExecStart mentioned in /<email address hidden> unit file to

ExecStart = /usr/local/bin/glance-api

NOTE:
After modifying <email address hidden> we need to relaod deamon using "sudo systemctl daemon-reload" command

[1] http://paste.openstack.org/show/722963/

So rosmaita, we need to change the devstack script to use

run_process g-api "$GLANCE_BIN_DIR/glance-api

instead of

run_process g-api "$GLANCE_BIN_DIR/glance-api --config-dir=$GLANCE_CONF_DIR"

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/678015