stack.sh fails because uWSGI directory not found in lib/apache

[Hirotaka Wakabayashi]

Hello devstack-team,

stack.sh fails because "uWSGI directory not found" error:
```
Successfully downloaded uwsgi
+ lib/apache:install_apache_uwsgi:97 : local uwsgi
++ lib/apache:install_apache_uwsgi:98 : ls 'uwsgi*'
ls: cannot access 'uwsgi*': No such file or directory
```

Uwsgi upstream seems to have changed the source package name and the list the contents[1].
https://pypi.org/project/uWSGI/2.0.18/#files
https://pypi.org/project/uWSGI/2.0.19/#files

Here is a simple workaround for me.
```
$ git diff lib/apache
diff --git a/lib/apache b/lib/apache
index 84cec732..68d6a4b7 100644
--- a/lib/apache
+++ b/lib/apache
@@ -95,8 +95,9 @@ function install_apache_uwsgi {
     pip_install uwsgi
     pip download uwsgi -c $REQUIREMENTS_DIR/upper-constraints.txt
     local uwsgi
- uwsgi=$(ls uwsgi*)
- tar xvf $uwsgi
+ if [ -f uW*.tar.gz ] ; then mv -f uW*.tar.gz uwsgi-latest.tar.gz; mkdir uwsgi-latest ; fi
+ uwsgi=$(ls uwsgi*.tar.gz)
+ tar xvf $uwsgi -C uwsgi-latest
     cd uwsgi*/apache2
```

Thanks in advance,
Hirotaka Wakabayashi

[1] uwsgi's changes(filename and list of the contents)
```
$ curl -O https://files.pythonhosted.org/packages/e7/1e/3dcca007f974fe4eb369bf1b8629d5e342bb3055e2001b2e5340aaefae7a/uwsgi-2.0.18.tar.gz && gunzip -f uwsgi-2.0.18.tar.gz && tar tvf uwsgi-2.0.18.tar | grep uwsgi_main.c
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 782k 100 782k 0 0 3418k 0 --:--:-- --:--:-- --:--:-- 3403k
-rw-rw-r-- root/root 134 2019-02-09 23:48 uwsgi-2.0.18/uwsgi_main.c

$ curl -O https://files.pythonhosted.org/packages/6f/e9/729f0eb15f4847c729c1a841bcbb5e327d08d208de0cbe661224955c9297/uWSGI-2.0.19.tar.gz && gunzip -f uWSGI-2.0.19.tar.gz && tar tvf uWSGI-2.0.19.tar | grep uwsgi_main.c
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 785k 100 785k 0 0 4758k 0 --:--:-- --:--:-- --:--:-- 4758k
-rw-rw-r-- root/root 134 2020-06-15 02:06 uwsgi_main.c
```

[Hirotaka Wakabayashi (hiwkby)]

Hello,

My stack.sh works fine with the patch below. I can submit a patch if no problem.

```
diff --git a/lib/apache b/lib/apache
index 84cec732..453d7d90 100644
--- a/lib/apache
+++ b/lib/apache
@@ -95,8 +95,14 @@ function install_apache_uwsgi {
     pip_install uwsgi
     pip download uwsgi -c $REQUIREMENTS_DIR/upper-constraints.txt
     local uwsgi
- uwsgi=$(ls uwsgi*)
- tar xvf $uwsgi
+ if [ -f uW*.tar.gz ] ; then
+ mkdir uwsgi-latest
+ uwsgi=$(ls uW*.tar.gz)
+ tar xvf $uwsgi -C uwsgi-latest
+ else
+ uwsgi=$(ls uwsgi*)
+ tar xvf $uwsgi
+ fi
     cd uwsgi*/apache2
     sudo $apxs -i -c mod_proxy_uwsgi.c
     popd
```

Thanks in advance,
Hirotaka Wakabayahsi

[Dr. Jens Harbott (j-harbott)]

There's already a fix proposed at https://review.opendev.org/735494 but it seems to be incomplete.

[Hirotaka Wakabayashi (hiwkby)]

Dr. Jens Harbott, Thank you for your comment! I wait for the review to be closed. Let me know if there is anything I can do.

Best,
Hirotaka Wakabayashi

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/735517

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/735517
Committed: https://git.openstack.org/cgit/openstack/devstack/commit/?id=84737ebd96327310ec5f8f7c312aeae12cbeb234
Submitter: Jens Harbott (frickler) (<email address hidden>)
Branch: master

commit 84737ebd96327310ec5f8f7c312aeae12cbeb234
Author: Jens Harbott <email address hidden>
Date: Mon Jun 15 09:48:46 2020 +0200

    Work around uwsgi breakage

    uwsgi broke installation from source with their latest release [0].
    Since we want to move away from source installation anyway, make
    grenade based jobs non-voting for the moment so that we can backport
    [1] properly.

    [0] https://bugs.launchpad.net/bugs/1883468
    [1] https://review.opendev.org/577955

    Related-Bug: 1883468
    Change-Id: I8e47bb7c70031a4df7f1af6b811df4c6cc784b2a

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/735522

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/735736

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jens Harbott (frickler) (<email address hidden>) on branch: stable/ussuri
Review: https://review.opendev.org/735522
Reason: Merged into https://review.opendev.org/735523

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/735535
Committed: https://git.openstack.org/cgit/openstack/devstack/commit/?id=5c07f3b1b5edc7ebc19cf265d5ada0acbfec6409
Submitter: Zuul
Branch: stable/stein

commit 5c07f3b1b5edc7ebc19cf265d5ada0acbfec6409
Author: Jens Harbott <email address hidden>
Date: Mon Jun 15 09:48:46 2020 +0200

    Work around gate breakage

    This is merged from three cherry-picked patches, all needed in order to
    pass gate testing. Make neutron-grenade non-voting in gate because
    dropping it there would be complicated.

    Use packaged uwsgi on Fedora and Ubuntu

    Building uwsgi from source was a workaround that was introduced a long
    time ago, it doesn't seem like it is needed anymore and will actually
    fail for Ubuntu 20.04.

    Also it doesn't match what will happen for most real-world
    installations, so let's try to get back to using distro packages. We'll
    still use the source install for RHEL/Centos, it remains to be tested
    whether we can get back to using distro versions there, too.

    Use uwsgi binary from path

    All these uwsgi invocations assume that the uwsgi binary is in the
    same directory as their project binaries are installed into (probably
    /usr/bin). That may not be correct -- for example if using a packaged
    uwsgi on Fedora the binary will live in /usr/sbin/uwsgi (not /usr/bin
    where the project files from pip are).

    Switch invocations to just find it in the path.

    Related-Bug: 1883468
    Change-Id: I82f539bfa533349293dd5a8ce309c9cc0ffb0393
    (cherry picked from commit 2d903568ed4158aa668bbda6986307a8780c71a4)
    (cherry picked from commit 84737ebd96327310ec5f8f7c312aeae12cbeb234)
    (cherry picked from commit 312517d5101206b33d3c395d27ec93d385b7fd24)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/735536
Committed: https://git.openstack.org/cgit/openstack/devstack/commit/?id=20e04092c8ce311017b2d490c4041cfb9b3a9395
Submitter: Zuul
Branch: stable/train

commit 20e04092c8ce311017b2d490c4041cfb9b3a9395
Author: Ian Wienand <email address hidden>
Date: Thu May 3 10:51:30 2018 +1000

    Fix gate issues

    This is a combination of 3 commits.
    This is the 1st commit message:

    Work around uwsgi breakage

    uwsgi broke installation from source with their latest release [0].
    Since we want to move away from source installation anyway, make
    grenade based jobs non-voting for the moment so that we can backport
    [1] properly.

    [0] https://bugs.launchpad.net/bugs/1883468
    [1] https://review.opendev.org/577955

    This is the commit message #2:

    Use packaged uwsgi on Fedora and Ubuntu

    Building uwsgi from source was a workaround that was introduced a long
    time ago, it doesn't seem like it is needed anymore and will actually
    fail for Ubuntu 20.04.

    Also it doesn't match what will happen for most real-world
    installations, so let's try to get back to using distro packages. We'll
    still use the source install for RHEL/Centos, it remains to be tested
    whether we can get back to using distro versions there, too.

    This is the commit message #3:

    Use uwsgi binary from path

    All these uwsgi invocations assume that the uwsgi binary is in the
    same directory as their project binaries are installed into (probably
    /usr/bin). That may not be correct -- for example if using a packaged
    uwsgi on Fedora the binary will live in /usr/sbin/uwsgi (not /usr/bin
    where the project files from pip are).

    Switch invocations to just find it in the path.

    While all this was happening, Zuul updated to Ansible 2.9 as well, so
    we require a fix for Ansible syntax in 736006 as well.

    Depends-On: https://review.opendev.org/735535
    Depends-On: https://review.opendev.org/736006
    Related-Bug: 1883468

    Change-Id: I82f539bfa533349293dd5a8ce309c9cc0ffb0393
    (cherry picked from commit 84737ebd96327310ec5f8f7c312aeae12cbeb234)
    (cherry picked from commit 2d903568ed4158aa668bbda6986307a8780c71a4)
    (cherry picked from commit 312517d5101206b33d3c395d27ec93d385b7fd24)

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jens Harbott (frickler) (<email address hidden>) on branch: stable/stein
Review: https://review.opendev.org/735736
Reason: Merged into https://review.opendev.org/#/c/735535/2

[Hirotaka Wakabayashi (hiwkby)]

Hello Dr. Jens Harbott,

Thank you very much for your support!

I don't know when and where I should add my comment on this issue, but I think the following lines still have problem for users of CentOS, which is officially supported by OpenStack.
https://opendev.org/openstack/devstack/src/branch/stable/ussuri/lib/apache#L116-L134

I hope this helps us.
```
diff --git a/lib/apache b/lib/apache
index 41c2e3d8..f9829c90 100644
--- a/lib/apache
+++ b/lib/apache
@@ -100,7 +100,7 @@ function install_apache_uwsgi {
                         uwsgi-plugin-python \
                         uwsgi-plugin-python3 \
                         libapache2-mod-proxy-uwsgi
- elif [[ $os_VENDOR == "Fedora" ]]; then
+ elif is_fedora; then
         # Note httpd comes with mod_proxy_uwsgi and it is loaded by
         # default; the mod_proxy_uwsgi package actually conflicts now.
         # See:
@@ -109,7 +109,7 @@ function install_apache_uwsgi {
         # Thus there is nothing else to do after this install
         install_package uwsgi \
                         uwsgi-plugin-python3
- elif [[ $os_VENDOR =~ openSUSE ]]; then
+ elif is_suse; then
         install_package uwsgi \
                         uwsgi-python3 \
                         apache2-mod_uwsgi
@@ -123,8 +123,14 @@ function install_apache_uwsgi {
         pip_install uwsgi
         pip download uwsgi -c $REQUIREMENTS_DIR/upper-constraints.txt
         local uwsgi
- uwsgi=$(ls uwsgi*)
- tar xvf $uwsgi
+ if [ -f uW*.tar.gz ] ; then
+ mkdir uwsgi-latest
+ uwsgi=$(ls uW*.tar.gz)
+ tar xvf $uwsgi -C uwsgi-latest
+ else
+ uwsgi=$(ls uwsgi*)
+ tar xvf $uwsgi
+ fi
         cd uwsgi*/apache2
         sudo $apxs -i -c mod_proxy_uwsgi.c
         popd
```

Thanks in advance,
Hirotaka Wakabayashi

[Ryan Liang (ryan-liang)]

It affects the rocky release too.

Here is my review for backporting some bugfix into rocky https://review.opendev.org/#/c/727602/.
But with several zuul failures due to this issue.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/739072

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/739072
Committed: https://git.openstack.org/cgit/openstack/devstack/commit/?id=c431116ceb14dc3b75796309a28c33287fcf9a72
Submitter: Zuul
Branch: stable/rocky

commit c431116ceb14dc3b75796309a28c33287fcf9a72
Author: Jens Harbott <email address hidden>
Date: Mon Jun 15 09:48:46 2020 +0200

    Use uwsgi binary from path

    All these uwsgi invocations assume that the uwsgi binary is in the
    same directory as their project binaries are installed into (probably
    /usr/bin). That may not be correct -- for example if using a packaged
    uwsgi on Fedora the binary will live in /usr/sbin/uwsgi (not /usr/bin
    where the project files from pip are).

    Switch invocations to just find it in the path.

    NOTE(lyarwood): This change was unfortunatley missing from the original
    backport of I82f539bfa533349293dd5a8ce309c9cc0ffb0393 so while uwsgi is
    now installed from packages it wasn't actually used. This was only
    noticed as g-api was being installed on subnodes in multinode jobs where
    uwsgi was not being installed from pip, leading to failures when
    attempting to use the project bin path.

    Related-Bug: 1883468
    (cherry picked from commit 2d903568ed4158aa668bbda6986307a8780c71a4)
    (cherry picked from commit 84737ebd96327310ec5f8f7c312aeae12cbeb234)
    (cherry picked from commit 312517d5101206b33d3c395d27ec93d385b7fd24)
    (cherry picked from commit 5c07f3b1b5edc7ebc19cf265d5ada0acbfec6409)

    Change-Id: Ieb63516b7208d63c4ae9b181aabb762387a6882a

[Ghanshyam Mann (ghanshyammann)]

this is fixed on all stable now, let's close this.

[Spencer (spencerharmon)]

Hi, sorry. This isn't resolved for centos in any of the stable branches. The proposed changes in the bug description and in comment number 12 would both resolve my issue. Commit c431116ceb14dc3b75796309a28c33287fcf9a72 does not address this problem. Can this be reopened to backport the fix proposed by Hirotaka Wakabayashi?

[Radosław Piliszek (yoctozepto)]

testing https://review.opendev.org/c/openstack/devstack/+/788196

[Radosław Piliszek (yoctozepto)]

It seems it works just fine. Which part of the above fix do you consider necessary?

[Spencer (spencerharmon)]

The crux of the problem is that, on every instance I've tested, if you "pip download uwsgi" as what's performed in this script, you will end up with a file called "uWSGI-2.0.19.1.tar.gz", which won't be found by "ls uwsgi*", obviously. The script needs to be updated to reflect the correct name of the file. As well, it seems that the internal directory structure doesn't match what's implied by this script.

Are you certain that what you've tested is reaching the relevant line? Control must reach line 122 os /lib apache to see the bug. If I'm reading it right, this bug should not affect either Fedora or Ubuntu.

This is the patch I'm using for centos 7, branch stable/stein:

diff --git a/lib/apache b/lib/apache
index a31188be..8fe213c8 100644
--- a/lib/apache
+++ b/lib/apache
@@ -119,9 +119,9 @@ function install_apache_uwsgi {
         pip_install uwsgi
         pip download uwsgi -c $REQUIREMENTS_DIR/upper-constraints.txt
         local uwsgi
- uwsgi=$(ls uwsgi*)
+ uwsgi=$(ls uWSGI*)
         tar xvf $uwsgi
- cd uwsgi*/apache2
+ cd apache2
         sudo $apxs -i -c mod_proxy_uwsgi.c
         popd
         # delete the temp directory

[Spencer (spencerharmon)]

Sorry; just saw the zuul link. It does appear that control reached line 110 in that test, so the relevant line isn't covered.

2021-04-27 08:11:26.581409 | controller | + functions-common:is_fedora:453 : '[' CentOS = CentOS ']'
2021-04-27 08:11:26.583959 | controller | + lib/apache:install_apache_uwsgi:110 : install_package uwsgi uwsgi-plugin-python3
2021-04-27 08:11:26.586533 | controller | + functions-common:install_package:1333 : update_package_repo

It looks like that test is for stable/usurri, which uses the is_fedora test, causing different behavior. Ussuri is the first version which is compatible with centos 8 as opposed to centos 7, which is required for Stein, right?

Thank you so much for looking into this!