m-api uwsgi deployment broken with setuptools

[Goutham Pacha Ravi]

Manila CI jobs are perma-failing with an issue starting the manila-api service:

Apr 28 15:32:48.216443 np0040561028 systemd[1]: Starting <email address hidden> - Devstack <email address hidden>...
Apr 28 15:32:48.229464 np0040561028 <email address hidden>[83053]: [uWSGI] getting INI configuration from /etc/manila/manila-uwsgi.ini
Apr 28 15:32:48.248227 np0040561028 <email address hidden>[83053]: *** Starting uWSGI 2.0.24-debian (64bit) on [Mon Apr 28 15:32:48 2025] ***
Apr 28 15:32:48.248227 np0040561028 <email address hidden>[83053]: compiled with version: 13.2.0 on 17 April 2024 16:23:48
Apr 28 15:32:48.248227 np0040561028 <email address hidden>[83053]: os: Linux-6.8.0-58-generic #60-Ubuntu SMP PREEMPT_DYNAMIC Fri Mar 14 18:29:48 UTC 2025
Apr 28 15:32:48.248227 np0040561028 <email address hidden>[83053]: nodename: np0040561028
Apr 28 15:32:48.248227 np0040561028 <email address hidden>[83053]: machine: x86_64
Apr 28 15:32:48.248227 np0040561028 <email address hidden>[83053]: clock source: unix
Apr 28 15:32:48.248227 np0040561028 <email address hidden>[83053]: pcre jit disabled
Apr 28 15:32:48.248227 np0040561028 <email address hidden>[83053]: detected number of CPU cores: 8
Apr 28 15:32:48.248227 np0040561028 <email address hidden>[83053]: current working directory: /
Apr 28 15:32:48.248227 np0040561028 <email address hidden>[83053]: detected binary path: /usr/bin/uwsgi-core
Apr 28 15:32:48.248227 np0040561028 <email address hidden>[83053]: your processes number limit is 31476
Apr 28 15:32:48.248227 np0040561028 <email address hidden>[83053]: your memory page size is 4096 bytes
Apr 28 15:32:48.248227 np0040561028 <email address hidden>[83053]: detected max file descriptor number: 2048
Apr 28 15:32:48.248227 np0040561028 <email address hidden>[83053]: lock engine: pthread robust mutexes
Apr 28 15:32:48.249074 np0040561028 <email address hidden>[83053]: thunder lock: enabled
Apr 28 15:32:48.249074 np0040561028 <email address hidden>[83053]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/manila-wsgi.socket fd 4
Apr 28 15:32:48.249074 np0040561028 <email address hidden>[83053]: Python version: 3.12.3 (main, Feb 4 2025, 14:48:35) [GCC 13.3.0]
Apr 28 15:32:48.249074 np0040561028 <email address hidden>[83053]: PEP 405 virtualenv detected: /opt/stack/data/venv
Apr 28 15:32:48.249074 np0040561028 <email address hidden>[83053]: Set PythonHome to /opt/stack/data/venv
Apr 28 15:32:48.311589 np0040561028 <email address hidden>[83053]: Python main interpreter initialized at 0x7306372a2f68
Apr 28 15:32:48.311589 np0040561028 <email address hidden>[83053]: python threads support enabled
Apr 28 15:32:48.311589 np0040561028 <email address hidden>[83053]: your server socket listen backlog is limited to 100 connections
Apr 28 15:32:48.311589 np0040561028 <email address hidden>[83053]: your mercy for graceful operations on workers is 80 seconds
Apr 28 15:32:48.312101 np0040561028 <email address hidden>[83053]: mapped 403077 bytes (393 KB) for 2 cores
Apr 28 15:32:48.312229 np0040561028 <email address hidden>[83053]: *** Operational MODE: preforking ***
Apr 28 15:32:48.312522 np0040561028 <email address hidden>[83053]: *** uWSGI is running in multiple interpreter mode ***
Apr 28 15:32:48.312522 np0040561028 <email address hidden>[83053]: spawned uWSGI master process (pid: 83053)
Apr 28 15:32:48.312806 np0040561028 systemd[1]: Started <email address hidden> - Devstack <email address hidden>.
Apr 28 15:32:48.313217 np0040561028 <email address hidden>[83053]: spawned uWSGI worker 1 (pid: 83054, cores: 1)
Apr 28 15:32:48.313818 np0040561028 <email address hidden>[83053]: spawned uWSGI worker 2 (pid: 83055, cores: 1)
Apr 28 15:32:48.313818 np0040561028 <email address hidden>[83053]: running "unix_signal:15 gracefully_kill_them_all" (master-start)...
Apr 28 15:32:48.319112 np0040561028 <email address hidden>[83055]: failed to open python file /opt/stack/data/venv/bin/manila-wsgi
Apr 28 15:32:48.319112 np0040561028 <email address hidden>[83055]: unable to load app 0 (mountpoint='') (callable not found or import error)
Apr 28 15:32:48.319112 np0040561028 <email address hidden>[83055]: *** no app loaded. going in full dynamic mode ***
Apr 28 15:32:48.321113 np0040561028 <email address hidden>[83054]: failed to open python file /opt/stack/data/venv/bin/manila-wsgi
Apr 28 15:32:48.321113 np0040561028 <email address hidden>[83054]: unable to load app 0 (mountpoint='') (callable not found or import error)
Apr 28 15:32:48.321113 np0040561028 <email address hidden>[83054]: *** no app loaded. going in full dynamic mode ***
Apr 28 15:32:48.528994 np0040561028 <email address hidden>[83055]: --- no python application found, check your startup logs for errors ---
Apr 28 15:32:48.528994 np0040561028 <email address hidden>[83055]: [pid: 83055|app: -1|req: -1/1] 2001:4801:7825:103:be76:4eff:fe10:1da1 () {60 vars in 1111 bytes} [Mon Apr 28 15:32:48 2025] GET /share => generated 21 bytes in 0 msecs (HTTP/1.1 500) 3 headers in 102 bytes (1 switches on core 0)
Apr 28 15:32:55.784142 np0040561028 <email address hidden>[83054]: --- no python application found, check your startup logs for errors ---
Apr 28 15:32:55.784142 np0040561028 <email address hidden>[83054]: [pid: 83054|app: -1|req: -1/2] 2001:4801:7825:103:be76:4eff:fe10:1da1 () {66 vars in 1419 bytes} [Mon Apr 28 15:32:55 2025] GET /share/ => generated 21 bytes in 0 msecs (HTTP/1.1 500) 3 headers in 102 bytes (0 switches on core 0)
Apr 28 15:32:56.065538 np0040561028 <email address hidden>[83055]: --- no python application found, check your startup logs for errors ---
Apr 28 15:32:56.065538 np0040561028 <email address hidden>[83055]: [pid: 83055|app: -1|req: -1/3] 2001:4801:7825:103:be76:4eff:fe10:1da1 () {66 vars in 1419 bytes} [Mon Apr 28 15:32:56 2025] GET /share/ => generated 21 bytes in 0 msecs (HTTP/1.1 500) 3 headers in 102 bytes (0 switches on core 0)

We don't ship a pyproject.toml file and so we see a warning in the devstacklog:

2025-04-28 15:24:49.566 | Installing collected packages: SQLAlchemy-Utils, manila
2025-04-28 15:24:49.690 | DEPRECATION: Legacy editable install of manila==20.1.0.dev18 from file:///opt/stack/manila (setup.py develop) is deprecated. pip 25.3 will enforce this behaviour change. A possible replacement is to add a pyproject.toml or enable --use-pep517, and use setuptools >= 64. If the resulting installation is not behaving as expected, try using --config-settings editable_mode=compat. Please consult the setuptools documentation for more information. Discussion can be found at https://github.com/pypa/pip/issues/11457
2025-04-28 15:24:49.691 | Running setup.py develop for manila

This seems to cause the wsgi binary file to not be in the location the configuration expects.

Similar bugs in other OpenStack projects:
 https://bugs.launchpad.net/barbican/+bug/2109584
 https://bugs.launchpad.net/openstack-cyborg/+bug/2109583
 https://bugs.launchpad.net/designate/+bug/2109577

In parallel, there's been a cross project effort to stop relying on wsgi paths: https://governance.openstack.org/tc/goals/proposed/migrate-from-wsgi-scripts-to-module-paths.html

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/manila/+/948491

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/manila/+/948588

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2025.1
Review: https://review.opendev.org/c/openstack/manila/+/949279

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/949279
Committed: https://opendev.org/openstack/manila/commit/eaf4c0a15b62fc00633005924c9a1a450ee1b765
Submitter: "Zuul (22348)"
Branch: stable/2025.1

commit eaf4c0a15b62fc00633005924c9a1a450ee1b765
Author: Goutham Pacha Ravi <email address hidden>
Date: Tue Apr 29 15:05:29 2025 -0700

    Add new manila.wsgi module

    We also update our devstack plugin to take advantage of it. This aligns
    Manila to a proposed OpenStack goal [1].

    [1] https://governance.openstack.org/tc/goals/proposed/migrate-from-wsgi-scripts-to-module-paths.html

    Change-Id: Icec9a8765eb7c40ec27da539a8bd544f487b61af
    Co-authored-by: Stephen Finucane <email address hidden>
    Closes-Bug: #2109645
    (cherry picked from commit a13d6c80627a3394ffd6e2f5f416bad49a2f1d5c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/948491
Committed: https://opendev.org/openstack/manila/commit/a13d6c80627a3394ffd6e2f5f416bad49a2f1d5c
Submitter: "Zuul (22348)"
Branch: master

commit a13d6c80627a3394ffd6e2f5f416bad49a2f1d5c
Author: Goutham Pacha Ravi <email address hidden>
Date: Tue Apr 29 15:05:29 2025 -0700

    Add new manila.wsgi module

    We also update our devstack plugin to take advantage of it. This aligns
    Manila to a proposed OpenStack goal [1].

    [1] https://governance.openstack.org/tc/goals/proposed/migrate-from-wsgi-scripts-to-module-paths.html

    Change-Id: Icec9a8765eb7c40ec27da539a8bd544f487b61af
    Co-authored-by: Stephen Finucane <email address hidden>
    Closes-Bug: #2109645

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2025.1
Review: https://review.opendev.org/c/openstack/manila/+/949611

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2024.2
Review: https://review.opendev.org/c/openstack/manila/+/949612

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/manila/+/949613

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/948588
Committed: https://opendev.org/openstack/manila/commit/db97a9f322a136604bdcf6846a2c4020cecdc868
Submitter: "Zuul (22348)"
Branch: master

commit db97a9f322a136604bdcf6846a2c4020cecdc868
Author: Goutham Pacha Ravi <email address hidden>
Date: Wed Apr 30 13:24:50 2025 -0700

    Add pyproject.toml

    pip 23.1 removed the "setup.py install" fallback for projects that do
    not have pyproject.toml. This change adds the minimal pyproject.toml
    to enable pbr to be properly used to build editable wheels.

    [1] https://docs.openstack.org/pbr/latest/user/using.html#pyproject-toml

    Change-Id: Ie14fb4db0da3b20daadfd55647c8563e65a338d2
    Closes-Bug: #2109645

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/949611
Committed: https://opendev.org/openstack/manila/commit/b4168f42dfe7ddc23a81a00b4f471f803de465cb
Submitter: "Zuul (22348)"
Branch: stable/2025.1

commit b4168f42dfe7ddc23a81a00b4f471f803de465cb
Author: Goutham Pacha Ravi <email address hidden>
Date: Wed Apr 30 13:24:50 2025 -0700

    Add pyproject.toml

    pip 23.1 removed the "setup.py install" fallback for projects that do
    not have pyproject.toml. This change adds the minimal pyproject.toml
    to enable pbr to be properly used to build editable wheels.

    [1] https://docs.openstack.org/pbr/latest/user/using.html#pyproject-toml

    Change-Id: Ie14fb4db0da3b20daadfd55647c8563e65a338d2
    Closes-Bug: #2109645
    (cherry picked from commit db97a9f322a136604bdcf6846a2c4020cecdc868)

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Stephen Finucane <email address hidden>" on branch: stable/2024.2
Review: https://review.opendev.org/c/openstack/manila/+/949612
Reason: We don't actually need this here right now. I suspect things will break come October (when setuptools removes `setup.py install` support) but we'll have to come up with a larger, OpenStack-wide fix when it does.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Stephen Finucane <email address hidden>" on branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/manila/+/949613
Reason: We don't actually need this here right now. I suspect things will break come October (when setuptools removes setup.py install support) but we'll have to come up with a larger, OpenStack-wide fix when it does.