| 2018-01-06 14:17:53 |
Jean-Philippe Evrard |
bug |
|
|
added bug |
| 2018-01-06 14:21:24 |
Jean-Philippe Evrard |
description |
virtualenv-tools has been unmaintained for a while, or at least unmodified with pending PR.
In the past we raised a few issues against it [2], and had to do conditional behavior depending on distros [1].
virtualenv-tools is a python2 only tool, and will break with python3 (it would be easy to move it to python3, because the code is simple, but I foresee issues with the print statements already!).
Anyway, on top of that we are victim of a bug while changing paths.
Our repo build process builds a virtualenv and pip install software on it.
By default, the path of python in the virtualenv for any bin in the tgz is something like #!/tmp/openstack-venv-builder/venvs/keystone/bin/python.
This lines comes virtualenv itself, as proven by the first line of the venv build's log (for example /var/log/repo/venv_build_keystone-master-x86_64.log):
"New python executable in /tmp/openstack-venv-builder/venvs/keystone/bin/python"
This line comes is logged during our venv-build-script.sh process, ${VENV_CREATE_COMMAND} is the first command to log on the ${ROLE_VENV_LOG}, and generates a few lines.
This venv-build-script.sh script takes the virtualenv options, for example /var/www/repo/os-releases/master/ubuntu-16.04-x86_64/venv-build-options-os_keystone.txt, which contains the virtualenv command, in my example: "/usr/local/bin/virtualenv --always-copy --extra-search-dir /var/www/repo/os-releases/master/ubuntu-16.04-x86_64 --never-download").
Long story short, the "New executable" line come from [3], which formats the py_executable, whose value is (unless python interpreter path is set with -p), sys.executable [4].
I haven't dugged deeper on that part, but I expect that some sys.executable can give different versions on different systems,
some could return python2 instead of python.
So on those systems, the generated line would be #!/tmp/openstack-venv-builder/venvs/keystone/bin/python2 as shown in the log [5].
virtualenv-tools wouldn't recognize those shabangs [6] because they don't end with python, and therefore won't modify them.
The impact is now that the pip installed packages in the venv are pointing to a wrong executable: the path #!/tmp/openstack-venv-builder/venvs/keystone/bin/python2 won't exist on the destination machine.
We should probably get rid of virtualenv-tools, or be more explicit with the executable used, all the way in the process, and replace it with simply line in files (sedding) the py files. The binary files should not be shipped in the virtualenv tgz anyway.
[1]: https://github.com/openstack/openstack-ansible-os_keystone/blob/a48a73089286a370312a35fc7df54a6a3a513fa2/tasks/keystone_install.yml#L165-L171
[2]: https://github.com/fireteam/virtualenv-tools/issues/5
[3]: https://github.com/evrardjp/virtualenv/blob/master/virtualenv.py#L1217
[4]: https://github.com/evrardjp/virtualenv/blob/master/virtualenv.py#L1205
[5]: http://logs.openstack.org/24/504224/40/check/openstack-ansible-deploy-all_metal-ubuntu-xenial/9b0dec0/logs/host/repo/venv_build_keystone-master-x86_64.log.txt.gz
[6]: https://github.com/fireteam/virtualenv-tools/blob/master/virtualenv_tools.py#L68-L69 |
virtualenv-tools has been unmaintained for a while, or at least it has not changed for a while, while still having pending PR.
In the past we raised a few issues against it [2], and had to do conditional behavior depending on distros [1].
virtualenv-tools is also a python2 only tool, and will break with python3 (it would be easy to move it to python3, because the code is simple, but I foresee issues with the print statements already!).
Anyway, on top of that we are victim of a bug while changing paths.
Our repo-build process builds a virtualenv and pip install software into it. By default, the shabang of python binaries in the virtualenv's archive is something like #!/tmp/openstack-venv-builder/venvs/keystone/bin/python.
This lines comes virtualenv itself, as proven by the first line of the venv build's log (for example /var/log/repo/venv_build_keystone-master-x86_64.log): "New python executable in /tmp/openstack-venv-builder/venvs/keystone/bin/python"
This line comes is logged during our venv-build-script.sh process, whose ${VENV_CREATE_COMMAND} is the first command to log on the ${ROLE_VENV_LOG}, and generates a few lines.
This venv-build-script.sh script takes the virtualenv options, for example /var/www/repo/os-releases/master/ubuntu-16.04-x86_64/venv-build-options-os_keystone.txt, which contains the virtualenv command, in my example: "/usr/local/bin/virtualenv --always-copy --extra-search-dir /var/www/repo/os-releases/master/ubuntu-16.04-x86_64 --never-download").
Long story short, the "New executable" line come from [3], which formats the py_executable, whose value is (unless python interpreter path is set with -p), coming from sys.executable [4], if I am not mistaken.
I haven't dugged deeper on the following part, but I expect that some sys.executable can be different on different systems, which would explain why some could return python2 instead of python.
So on those systems, the generated line would be #!/tmp/openstack-venv-builder/venvs/keystone/bin/python2 as shown in the log [5].
virtualenv-tools wouldn't recognize those shabangs [6] because they don't end with python, and therefore won't modify them.
The impact is now that the pip installed packages in the venv are pointing to a wrong executable and therefore can't execute: the path #!/tmp/openstack-venv-builder/venvs/keystone/bin/python2 won't exist on the destination machine.
We should probably get rid of virtualenv-tools, or be more explicit with the executable used, all the way in the process, and replace it with simply line in files (sedding) the py files. The binary files should not be shipped in the virtualenv tgz anyway.
[1]: https://github.com/openstack/openstack-ansible-os_keystone/blob/a48a73089286a370312a35fc7df54a6a3a513fa2/tasks/keystone_install.yml#L165-L171
[2]: https://github.com/fireteam/virtualenv-tools/issues/5
[3]: https://github.com/evrardjp/virtualenv/blob/master/virtualenv.py#L1217
[4]: https://github.com/evrardjp/virtualenv/blob/master/virtualenv.py#L1205
[5]: http://logs.openstack.org/24/504224/40/check/openstack-ansible-deploy-all_metal-ubuntu-xenial/9b0dec0/logs/host/repo/venv_build_keystone-master-x86_64.log.txt.gz
[6]: https://github.com/fireteam/virtualenv-tools/blob/master/virtualenv_tools.py#L68-L69 |
|
| 2018-01-07 17:59:13 |
Jean-Philippe Evrard |
description |
virtualenv-tools has been unmaintained for a while, or at least it has not changed for a while, while still having pending PR.
In the past we raised a few issues against it [2], and had to do conditional behavior depending on distros [1].
virtualenv-tools is also a python2 only tool, and will break with python3 (it would be easy to move it to python3, because the code is simple, but I foresee issues with the print statements already!).
Anyway, on top of that we are victim of a bug while changing paths.
Our repo-build process builds a virtualenv and pip install software into it. By default, the shabang of python binaries in the virtualenv's archive is something like #!/tmp/openstack-venv-builder/venvs/keystone/bin/python.
This lines comes virtualenv itself, as proven by the first line of the venv build's log (for example /var/log/repo/venv_build_keystone-master-x86_64.log): "New python executable in /tmp/openstack-venv-builder/venvs/keystone/bin/python"
This line comes is logged during our venv-build-script.sh process, whose ${VENV_CREATE_COMMAND} is the first command to log on the ${ROLE_VENV_LOG}, and generates a few lines.
This venv-build-script.sh script takes the virtualenv options, for example /var/www/repo/os-releases/master/ubuntu-16.04-x86_64/venv-build-options-os_keystone.txt, which contains the virtualenv command, in my example: "/usr/local/bin/virtualenv --always-copy --extra-search-dir /var/www/repo/os-releases/master/ubuntu-16.04-x86_64 --never-download").
Long story short, the "New executable" line come from [3], which formats the py_executable, whose value is (unless python interpreter path is set with -p), coming from sys.executable [4], if I am not mistaken.
I haven't dugged deeper on the following part, but I expect that some sys.executable can be different on different systems, which would explain why some could return python2 instead of python.
So on those systems, the generated line would be #!/tmp/openstack-venv-builder/venvs/keystone/bin/python2 as shown in the log [5].
virtualenv-tools wouldn't recognize those shabangs [6] because they don't end with python, and therefore won't modify them.
The impact is now that the pip installed packages in the venv are pointing to a wrong executable and therefore can't execute: the path #!/tmp/openstack-venv-builder/venvs/keystone/bin/python2 won't exist on the destination machine.
We should probably get rid of virtualenv-tools, or be more explicit with the executable used, all the way in the process, and replace it with simply line in files (sedding) the py files. The binary files should not be shipped in the virtualenv tgz anyway.
[1]: https://github.com/openstack/openstack-ansible-os_keystone/blob/a48a73089286a370312a35fc7df54a6a3a513fa2/tasks/keystone_install.yml#L165-L171
[2]: https://github.com/fireteam/virtualenv-tools/issues/5
[3]: https://github.com/evrardjp/virtualenv/blob/master/virtualenv.py#L1217
[4]: https://github.com/evrardjp/virtualenv/blob/master/virtualenv.py#L1205
[5]: http://logs.openstack.org/24/504224/40/check/openstack-ansible-deploy-all_metal-ubuntu-xenial/9b0dec0/logs/host/repo/venv_build_keystone-master-x86_64.log.txt.gz
[6]: https://github.com/fireteam/virtualenv-tools/blob/master/virtualenv_tools.py#L68-L69 |
virtualenv-tools has been unmaintained for a while, or at least it has not changed for a while, while still having pending PR.
In the past we raised a few issues against it [2], and had to do conditional behavior depending on distros [1].
virtualenv-tools is also a python2 only tool, and will break with python3 (it would be easy to move it to python3, because the code is simple, but I foresee issues with the print statements already!).
Anyway, on top of that we are victim of a bug while changing paths.
Our repo-build process builds a virtualenv and pip install software into it. By default, the shabang of python binaries in the virtualenv's archive is something like #!/tmp/openstack-venv-builder/venvs/keystone/bin/python.
This lines comes virtualenv itself, as proven by the first line of the venv build's log (for example /var/log/repo/venv_build_keystone-master-x86_64.log): "New python executable in /tmp/openstack-venv-builder/venvs/keystone/bin/python"
This line comes is logged during our venv-build-script.sh process, whose ${VENV_CREATE_COMMAND} is the first command to log on the ${ROLE_VENV_LOG}, and generates a few lines.
This venv-build-script.sh script takes the virtualenv options, for example /var/www/repo/os-releases/master/ubuntu-16.04-x86_64/venv-build-options-os_keystone.txt, which contains the virtualenv command, in my example: "/usr/local/bin/virtualenv --always-copy --extra-search-dir /var/www/repo/os-releases/master/ubuntu-16.04-x86_64 --never-download").
Long story short, the "New executable" line come from [3], which formats the py_executable, whose value is (unless python interpreter path is set with -p), coming from sys.executable [4], if I am not mistaken.
I didn't dig deeper on the following part, but I expect that some sys.executable can be different on different systems, which would explain why some could return python2 instead of python.
So on those systems, the generated line would be #!/tmp/openstack-venv-builder/venvs/keystone/bin/python2 as shown in the log [5].
virtualenv-tools wouldn't recognize those shabangs [6] because they don't end with python, and therefore won't modify them.
The impact is now that the pip installed packages in the venv are pointing to a wrong executable and therefore can't execute: the path #!/tmp/openstack-venv-builder/venvs/keystone/bin/python2 won't exist on the destination machine.
We should probably get rid of virtualenv-tools, or be more explicit with the executable used, all the way in the process, and replace it with simply line in files (sedding) the py files. The binary files should not be shipped in the virtualenv tgz anyway.
[1]: https://github.com/openstack/openstack-ansible-os_keystone/blob/a48a73089286a370312a35fc7df54a6a3a513fa2/tasks/keystone_install.yml#L165-L171
[2]: https://github.com/fireteam/virtualenv-tools/issues/5
[3]: https://github.com/evrardjp/virtualenv/blob/master/virtualenv.py#L1217
[4]: https://github.com/evrardjp/virtualenv/blob/master/virtualenv.py#L1205
[5]: http://logs.openstack.org/24/504224/40/check/openstack-ansible-deploy-all_metal-ubuntu-xenial/9b0dec0/logs/host/repo/venv_build_keystone-master-x86_64.log.txt.gz
[6]: https://github.com/fireteam/virtualenv-tools/blob/master/virtualenv_tools.py#L68-L69 |
|
| 2018-01-08 08:43:35 |
Jean-Philippe Evrard |
description |
virtualenv-tools has been unmaintained for a while, or at least it has not changed for a while, while still having pending PR.
In the past we raised a few issues against it [2], and had to do conditional behavior depending on distros [1].
virtualenv-tools is also a python2 only tool, and will break with python3 (it would be easy to move it to python3, because the code is simple, but I foresee issues with the print statements already!).
Anyway, on top of that we are victim of a bug while changing paths.
Our repo-build process builds a virtualenv and pip install software into it. By default, the shabang of python binaries in the virtualenv's archive is something like #!/tmp/openstack-venv-builder/venvs/keystone/bin/python.
This lines comes virtualenv itself, as proven by the first line of the venv build's log (for example /var/log/repo/venv_build_keystone-master-x86_64.log): "New python executable in /tmp/openstack-venv-builder/venvs/keystone/bin/python"
This line comes is logged during our venv-build-script.sh process, whose ${VENV_CREATE_COMMAND} is the first command to log on the ${ROLE_VENV_LOG}, and generates a few lines.
This venv-build-script.sh script takes the virtualenv options, for example /var/www/repo/os-releases/master/ubuntu-16.04-x86_64/venv-build-options-os_keystone.txt, which contains the virtualenv command, in my example: "/usr/local/bin/virtualenv --always-copy --extra-search-dir /var/www/repo/os-releases/master/ubuntu-16.04-x86_64 --never-download").
Long story short, the "New executable" line come from [3], which formats the py_executable, whose value is (unless python interpreter path is set with -p), coming from sys.executable [4], if I am not mistaken.
I didn't dig deeper on the following part, but I expect that some sys.executable can be different on different systems, which would explain why some could return python2 instead of python.
So on those systems, the generated line would be #!/tmp/openstack-venv-builder/venvs/keystone/bin/python2 as shown in the log [5].
virtualenv-tools wouldn't recognize those shabangs [6] because they don't end with python, and therefore won't modify them.
The impact is now that the pip installed packages in the venv are pointing to a wrong executable and therefore can't execute: the path #!/tmp/openstack-venv-builder/venvs/keystone/bin/python2 won't exist on the destination machine.
We should probably get rid of virtualenv-tools, or be more explicit with the executable used, all the way in the process, and replace it with simply line in files (sedding) the py files. The binary files should not be shipped in the virtualenv tgz anyway.
[1]: https://github.com/openstack/openstack-ansible-os_keystone/blob/a48a73089286a370312a35fc7df54a6a3a513fa2/tasks/keystone_install.yml#L165-L171
[2]: https://github.com/fireteam/virtualenv-tools/issues/5
[3]: https://github.com/evrardjp/virtualenv/blob/master/virtualenv.py#L1217
[4]: https://github.com/evrardjp/virtualenv/blob/master/virtualenv.py#L1205
[5]: http://logs.openstack.org/24/504224/40/check/openstack-ansible-deploy-all_metal-ubuntu-xenial/9b0dec0/logs/host/repo/venv_build_keystone-master-x86_64.log.txt.gz
[6]: https://github.com/fireteam/virtualenv-tools/blob/master/virtualenv_tools.py#L68-L69 |
virtualenv-tools has been unmaintained for a while, or at least it has not changed for a while, while still having pending PR.
(Edit: It's a dead project, according to its author [7])
In the past we raised a few issues against it [2], and had to do conditional behavior depending on distros [1].
virtualenv-tools is also a python2 only tool, and will break with python3 (it would be easy to move it to python3, because the code is simple, but I foresee issues with the print statements already!).
Anyway, on top of that we are victim of a bug while changing paths.
Our repo-build process builds a virtualenv and pip install software into it. By default, the shabang of python binaries in the virtualenv's archive is something like #!/tmp/openstack-venv-builder/venvs/keystone/bin/python.
This lines comes virtualenv itself, as proven by the first line of the venv build's log (for example /var/log/repo/venv_build_keystone-master-x86_64.log): "New python executable in /tmp/openstack-venv-builder/venvs/keystone/bin/python"
This line comes is logged during our venv-build-script.sh process, whose ${VENV_CREATE_COMMAND} is the first command to log on the ${ROLE_VENV_LOG}, and generates a few lines.
This venv-build-script.sh script takes the virtualenv options, for example /var/www/repo/os-releases/master/ubuntu-16.04-x86_64/venv-build-options-os_keystone.txt, which contains the virtualenv command, in my example: "/usr/local/bin/virtualenv --always-copy --extra-search-dir /var/www/repo/os-releases/master/ubuntu-16.04-x86_64 --never-download").
Long story short, the "New executable" line come from [3], which formats the py_executable, whose value is (unless python interpreter path is set with -p), coming from sys.executable [4], if I am not mistaken.
I didn't dig deeper on the following part, but I expect that some sys.executable can be different on different systems, which would explain why some could return python2 instead of python.
So on those systems, the generated line would be #!/tmp/openstack-venv-builder/venvs/keystone/bin/python2 as shown in the log [5].
virtualenv-tools wouldn't recognize those shabangs [6] because they don't end with python, and therefore won't modify them.
The impact is now that the pip installed packages in the venv are pointing to a wrong executable and therefore can't execute: the path #!/tmp/openstack-venv-builder/venvs/keystone/bin/python2 won't exist on the destination machine.
We should probably get rid of virtualenv-tools, or be more explicit with the executable used, all the way in the process, and replace it with simply line in files (sedding) the py files. The binary files should not be shipped in the virtualenv tgz anyway.
[1]: https://github.com/openstack/openstack-ansible-os_keystone/blob/a48a73089286a370312a35fc7df54a6a3a513fa2/tasks/keystone_install.yml#L165-L171
[2]: https://github.com/fireteam/virtualenv-tools/issues/5
[3]: https://github.com/evrardjp/virtualenv/blob/master/virtualenv.py#L1217
[4]: https://github.com/evrardjp/virtualenv/blob/master/virtualenv.py#L1205
[5]: http://logs.openstack.org/24/504224/40/check/openstack-ansible-deploy-all_metal-ubuntu-xenial/9b0dec0/logs/host/repo/venv_build_keystone-master-x86_64.log.txt.gz
[6]: https://github.com/fireteam/virtualenv-tools/blob/master/virtualenv_tools.py#L68-L69
[7]: https://twitter.com/evrardjp/status/950069466112692224 |
|
| 2018-01-08 08:44:53 |
Jean-Philippe Evrard |
description |
virtualenv-tools has been unmaintained for a while, or at least it has not changed for a while, while still having pending PR.
(Edit: It's a dead project, according to its author [7])
In the past we raised a few issues against it [2], and had to do conditional behavior depending on distros [1].
virtualenv-tools is also a python2 only tool, and will break with python3 (it would be easy to move it to python3, because the code is simple, but I foresee issues with the print statements already!).
Anyway, on top of that we are victim of a bug while changing paths.
Our repo-build process builds a virtualenv and pip install software into it. By default, the shabang of python binaries in the virtualenv's archive is something like #!/tmp/openstack-venv-builder/venvs/keystone/bin/python.
This lines comes virtualenv itself, as proven by the first line of the venv build's log (for example /var/log/repo/venv_build_keystone-master-x86_64.log): "New python executable in /tmp/openstack-venv-builder/venvs/keystone/bin/python"
This line comes is logged during our venv-build-script.sh process, whose ${VENV_CREATE_COMMAND} is the first command to log on the ${ROLE_VENV_LOG}, and generates a few lines.
This venv-build-script.sh script takes the virtualenv options, for example /var/www/repo/os-releases/master/ubuntu-16.04-x86_64/venv-build-options-os_keystone.txt, which contains the virtualenv command, in my example: "/usr/local/bin/virtualenv --always-copy --extra-search-dir /var/www/repo/os-releases/master/ubuntu-16.04-x86_64 --never-download").
Long story short, the "New executable" line come from [3], which formats the py_executable, whose value is (unless python interpreter path is set with -p), coming from sys.executable [4], if I am not mistaken.
I didn't dig deeper on the following part, but I expect that some sys.executable can be different on different systems, which would explain why some could return python2 instead of python.
So on those systems, the generated line would be #!/tmp/openstack-venv-builder/venvs/keystone/bin/python2 as shown in the log [5].
virtualenv-tools wouldn't recognize those shabangs [6] because they don't end with python, and therefore won't modify them.
The impact is now that the pip installed packages in the venv are pointing to a wrong executable and therefore can't execute: the path #!/tmp/openstack-venv-builder/venvs/keystone/bin/python2 won't exist on the destination machine.
We should probably get rid of virtualenv-tools, or be more explicit with the executable used, all the way in the process, and replace it with simply line in files (sedding) the py files. The binary files should not be shipped in the virtualenv tgz anyway.
[1]: https://github.com/openstack/openstack-ansible-os_keystone/blob/a48a73089286a370312a35fc7df54a6a3a513fa2/tasks/keystone_install.yml#L165-L171
[2]: https://github.com/fireteam/virtualenv-tools/issues/5
[3]: https://github.com/evrardjp/virtualenv/blob/master/virtualenv.py#L1217
[4]: https://github.com/evrardjp/virtualenv/blob/master/virtualenv.py#L1205
[5]: http://logs.openstack.org/24/504224/40/check/openstack-ansible-deploy-all_metal-ubuntu-xenial/9b0dec0/logs/host/repo/venv_build_keystone-master-x86_64.log.txt.gz
[6]: https://github.com/fireteam/virtualenv-tools/blob/master/virtualenv_tools.py#L68-L69
[7]: https://twitter.com/evrardjp/status/950069466112692224 |
virtualenv-tools has been unmaintained for a while, or at least it has not changed for a while, while still having pending PR.
(Edit: It's a dead project, according to its author [7])
In the past we raised a few issues against it [2], and had to do conditional behavior depending on distros [1].
virtualenv-tools is also a python2 only tool, and will break with python3 (it would be easy to move it to python3, because the code is simple, but I foresee issues with the print statements already!).
Anyway, on top of that we are victim of a bug while changing paths.
Our repo-build process builds a virtualenv and pip install software into it. By default, the shabang of python binaries in the virtualenv's archive is something like #!/tmp/openstack-venv-builder/venvs/keystone/bin/python.
This lines comes virtualenv itself, as proven by the first line of the venv build's log (for example /var/log/repo/venv_build_keystone-master-x86_64.log): "New python executable in /tmp/openstack-venv-builder/venvs/keystone/bin/python"
This line comes is logged during our venv-build-script.sh process, whose ${VENV_CREATE_COMMAND} is the first command to log on the ${ROLE_VENV_LOG}, and generates a few lines.
This venv-build-script.sh script takes the virtualenv options, for example /var/www/repo/os-releases/master/ubuntu-16.04-x86_64/venv-build-options-os_keystone.txt, which contains the virtualenv command, in my example: "/usr/local/bin/virtualenv --always-copy --extra-search-dir /var/www/repo/os-releases/master/ubuntu-16.04-x86_64 --never-download").
Long story short, the "New executable" line come from [3], which formats the py_executable, whose value is (unless python interpreter path is set with -p), coming from sys.executable [4], if I am not mistaken.
I didn't dig deeper on the following part, but I expect that some sys.executable can be different on different systems, which would explain why some could return python2 instead of python.
So on those systems, the generated line would be #!/tmp/openstack-venv-builder/venvs/keystone/bin/python2 as shown in the log [5].
(Edit: This probably comes from the shabang of /usr/bin/virtualenv, which
could contain python2 instead of python)
virtualenv-tools wouldn't recognize those shabangs [6] because they don't end with python, and therefore won't modify them.
The impact is now that the pip installed packages in the venv are pointing to a wrong executable and therefore can't execute: the path #!/tmp/openstack-venv-builder/venvs/keystone/bin/python2 won't exist on the destination machine.
We should probably get rid of virtualenv-tools, or be more explicit with the executable used, all the way in the process, and replace it with simply line in files (sedding) the py files. The binary files should not be shipped in the virtualenv tgz anyway.
[1]: https://github.com/openstack/openstack-ansible-os_keystone/blob/a48a73089286a370312a35fc7df54a6a3a513fa2/tasks/keystone_install.yml#L165-L171
[2]: https://github.com/fireteam/virtualenv-tools/issues/5
[3]: https://github.com/evrardjp/virtualenv/blob/master/virtualenv.py#L1217
[4]: https://github.com/evrardjp/virtualenv/blob/master/virtualenv.py#L1205
[5]: http://logs.openstack.org/24/504224/40/check/openstack-ansible-deploy-all_metal-ubuntu-xenial/9b0dec0/logs/host/repo/venv_build_keystone-master-x86_64.log.txt.gz
[6]: https://github.com/fireteam/virtualenv-tools/blob/master/virtualenv_tools.py#L68-L69
[7]: https://twitter.com/evrardjp/status/950069466112692224 |
|
| 2018-01-08 08:50:29 |
Jean-Philippe Evrard |
description |
virtualenv-tools has been unmaintained for a while, or at least it has not changed for a while, while still having pending PR.
(Edit: It's a dead project, according to its author [7])
In the past we raised a few issues against it [2], and had to do conditional behavior depending on distros [1].
virtualenv-tools is also a python2 only tool, and will break with python3 (it would be easy to move it to python3, because the code is simple, but I foresee issues with the print statements already!).
Anyway, on top of that we are victim of a bug while changing paths.
Our repo-build process builds a virtualenv and pip install software into it. By default, the shabang of python binaries in the virtualenv's archive is something like #!/tmp/openstack-venv-builder/venvs/keystone/bin/python.
This lines comes virtualenv itself, as proven by the first line of the venv build's log (for example /var/log/repo/venv_build_keystone-master-x86_64.log): "New python executable in /tmp/openstack-venv-builder/venvs/keystone/bin/python"
This line comes is logged during our venv-build-script.sh process, whose ${VENV_CREATE_COMMAND} is the first command to log on the ${ROLE_VENV_LOG}, and generates a few lines.
This venv-build-script.sh script takes the virtualenv options, for example /var/www/repo/os-releases/master/ubuntu-16.04-x86_64/venv-build-options-os_keystone.txt, which contains the virtualenv command, in my example: "/usr/local/bin/virtualenv --always-copy --extra-search-dir /var/www/repo/os-releases/master/ubuntu-16.04-x86_64 --never-download").
Long story short, the "New executable" line come from [3], which formats the py_executable, whose value is (unless python interpreter path is set with -p), coming from sys.executable [4], if I am not mistaken.
I didn't dig deeper on the following part, but I expect that some sys.executable can be different on different systems, which would explain why some could return python2 instead of python.
So on those systems, the generated line would be #!/tmp/openstack-venv-builder/venvs/keystone/bin/python2 as shown in the log [5].
(Edit: This probably comes from the shabang of /usr/bin/virtualenv, which
could contain python2 instead of python)
virtualenv-tools wouldn't recognize those shabangs [6] because they don't end with python, and therefore won't modify them.
The impact is now that the pip installed packages in the venv are pointing to a wrong executable and therefore can't execute: the path #!/tmp/openstack-venv-builder/venvs/keystone/bin/python2 won't exist on the destination machine.
We should probably get rid of virtualenv-tools, or be more explicit with the executable used, all the way in the process, and replace it with simply line in files (sedding) the py files. The binary files should not be shipped in the virtualenv tgz anyway.
[1]: https://github.com/openstack/openstack-ansible-os_keystone/blob/a48a73089286a370312a35fc7df54a6a3a513fa2/tasks/keystone_install.yml#L165-L171
[2]: https://github.com/fireteam/virtualenv-tools/issues/5
[3]: https://github.com/evrardjp/virtualenv/blob/master/virtualenv.py#L1217
[4]: https://github.com/evrardjp/virtualenv/blob/master/virtualenv.py#L1205
[5]: http://logs.openstack.org/24/504224/40/check/openstack-ansible-deploy-all_metal-ubuntu-xenial/9b0dec0/logs/host/repo/venv_build_keystone-master-x86_64.log.txt.gz
[6]: https://github.com/fireteam/virtualenv-tools/blob/master/virtualenv_tools.py#L68-L69
[7]: https://twitter.com/evrardjp/status/950069466112692224 |
virtualenv-tools has been unmaintained for a while, or at least it has not changed for a while, while still having pending PR.
(Edit: It's a dead project, according to its author [7])
In the past we raised a few issues against it [2], and had to do conditional behavior depending on distros [1].
virtualenv-tools is also a python2 only tool, and will break with python3 (it would be easy to move it to python3, because the code is simple, but I foresee issues with the print statements already!).
Anyway, on top of that we are victim of a bug while changing paths.
Our repo-build process builds a virtualenv and pip install software into it. By default, the shabang of python binaries in the virtualenv's archive is something like #!/tmp/openstack-venv-builder/venvs/keystone/bin/python.
This lines comes virtualenv itself, as proven by the first line of the venv build's log (for example /var/log/repo/venv_build_keystone-master-x86_64.log): "New python executable in /tmp/openstack-venv-builder/venvs/keystone/bin/python"
This line comes is logged during our venv-build-script.sh process, whose ${VENV_CREATE_COMMAND} is the first command to log on the ${ROLE_VENV_LOG}, and generates a few lines.
This venv-build-script.sh script takes the virtualenv options, for example /var/www/repo/os-releases/master/ubuntu-16.04-x86_64/venv-build-options-os_keystone.txt, which contains the virtualenv command, in my example: "/usr/local/bin/virtualenv --always-copy --extra-search-dir /var/www/repo/os-releases/master/ubuntu-16.04-x86_64 --never-download").
Long story short, the "New executable" line come from [3], which formats the py_executable, whose value is (unless python interpreter path is set with -p), coming from sys.executable [4], if I am not mistaken.
I didn't dig deeper on the following part, but I expect that some sys.executable can be different on different systems, which would explain why some could return python2 instead of python.
So on those systems, the generated line would be #!/tmp/openstack-venv-builder/venvs/keystone/bin/python2 as shown in the log [5].
(Edit: This probably comes from the shabang of /usr/bin/virtualenv, which
could contain python2 instead of python
Edit 2: That is exact, see [8] for gate tests to prove this)
virtualenv-tools wouldn't recognize those shabangs [6] because they don't end with python, and therefore won't modify them.
The impact is now that the pip installed packages in the venv are pointing to a wrong executable and therefore can't execute: the path #!/tmp/openstack-venv-builder/venvs/keystone/bin/python2 won't exist on the destination machine.
We should probably get rid of virtualenv-tools, or be more explicit with the executable used, all the way in the process, and replace it with simply line in files (sedding) the py files. The binary files should not be shipped in the virtualenv tgz anyway.
[1]: https://github.com/openstack/openstack-ansible-os_keystone/blob/a48a73089286a370312a35fc7df54a6a3a513fa2/tasks/keystone_install.yml#L165-L171
[2]: https://github.com/fireteam/virtualenv-tools/issues/5
[3]: https://github.com/evrardjp/virtualenv/blob/master/virtualenv.py#L1217
[4]: https://github.com/evrardjp/virtualenv/blob/master/virtualenv.py#L1205
[5]: http://logs.openstack.org/24/504224/40/check/openstack-ansible-deploy-all_metal-ubuntu-xenial/9b0dec0/logs/host/repo/venv_build_keystone-master-x86_64.log.txt.gz
[6]: https://github.com/fireteam/virtualenv-tools/blob/master/virtualenv_tools.py#L68-L69
[7]: https://twitter.com/evrardjp/status/950069466112692224
[8]: https://review.openstack.org/#/c/531687/ |
|
| 2018-01-08 08:51:50 |
Jean-Philippe Evrard |
description |
virtualenv-tools has been unmaintained for a while, or at least it has not changed for a while, while still having pending PR.
(Edit: It's a dead project, according to its author [7])
In the past we raised a few issues against it [2], and had to do conditional behavior depending on distros [1].
virtualenv-tools is also a python2 only tool, and will break with python3 (it would be easy to move it to python3, because the code is simple, but I foresee issues with the print statements already!).
Anyway, on top of that we are victim of a bug while changing paths.
Our repo-build process builds a virtualenv and pip install software into it. By default, the shabang of python binaries in the virtualenv's archive is something like #!/tmp/openstack-venv-builder/venvs/keystone/bin/python.
This lines comes virtualenv itself, as proven by the first line of the venv build's log (for example /var/log/repo/venv_build_keystone-master-x86_64.log): "New python executable in /tmp/openstack-venv-builder/venvs/keystone/bin/python"
This line comes is logged during our venv-build-script.sh process, whose ${VENV_CREATE_COMMAND} is the first command to log on the ${ROLE_VENV_LOG}, and generates a few lines.
This venv-build-script.sh script takes the virtualenv options, for example /var/www/repo/os-releases/master/ubuntu-16.04-x86_64/venv-build-options-os_keystone.txt, which contains the virtualenv command, in my example: "/usr/local/bin/virtualenv --always-copy --extra-search-dir /var/www/repo/os-releases/master/ubuntu-16.04-x86_64 --never-download").
Long story short, the "New executable" line come from [3], which formats the py_executable, whose value is (unless python interpreter path is set with -p), coming from sys.executable [4], if I am not mistaken.
I didn't dig deeper on the following part, but I expect that some sys.executable can be different on different systems, which would explain why some could return python2 instead of python.
So on those systems, the generated line would be #!/tmp/openstack-venv-builder/venvs/keystone/bin/python2 as shown in the log [5].
(Edit: This probably comes from the shabang of /usr/bin/virtualenv, which
could contain python2 instead of python
Edit 2: That is exact, see [8] for gate tests to prove this)
virtualenv-tools wouldn't recognize those shabangs [6] because they don't end with python, and therefore won't modify them.
The impact is now that the pip installed packages in the venv are pointing to a wrong executable and therefore can't execute: the path #!/tmp/openstack-venv-builder/venvs/keystone/bin/python2 won't exist on the destination machine.
We should probably get rid of virtualenv-tools, or be more explicit with the executable used, all the way in the process, and replace it with simply line in files (sedding) the py files. The binary files should not be shipped in the virtualenv tgz anyway.
[1]: https://github.com/openstack/openstack-ansible-os_keystone/blob/a48a73089286a370312a35fc7df54a6a3a513fa2/tasks/keystone_install.yml#L165-L171
[2]: https://github.com/fireteam/virtualenv-tools/issues/5
[3]: https://github.com/evrardjp/virtualenv/blob/master/virtualenv.py#L1217
[4]: https://github.com/evrardjp/virtualenv/blob/master/virtualenv.py#L1205
[5]: http://logs.openstack.org/24/504224/40/check/openstack-ansible-deploy-all_metal-ubuntu-xenial/9b0dec0/logs/host/repo/venv_build_keystone-master-x86_64.log.txt.gz
[6]: https://github.com/fireteam/virtualenv-tools/blob/master/virtualenv_tools.py#L68-L69
[7]: https://twitter.com/evrardjp/status/950069466112692224
[8]: https://review.openstack.org/#/c/531687/ |
virtualenv-tools has been unmaintained for a while, or at least it has not changed for a while, while still having pending PR.
(Edit: It's a dead project, according to its author [7])
In the past we raised a few issues against it [2], and had to do conditional behavior depending on distros [1].
virtualenv-tools is also a python2 only tool, and will break with python3 (it would be easy to move it to python3, because the code is simple, but I foresee issues with the print statements already!).
Anyway, on top of that we are victim of a bug while changing paths.
Our repo-build process builds a virtualenv and pip install software into it. By default, the shabang of python binaries in the virtualenv's archive is something like #!/tmp/openstack-venv-builder/venvs/keystone/bin/python.
This lines comes virtualenv itself, as proven by the first line of the venv build's log (for example /var/log/repo/venv_build_keystone-master-x86_64.log): "New python executable in /tmp/openstack-venv-builder/venvs/keystone/bin/python"
This line comes is logged during our venv-build-script.sh process, whose ${VENV_CREATE_COMMAND} is the first command to log on the ${ROLE_VENV_LOG}, and generates a few lines.
This venv-build-script.sh script takes the virtualenv options, for example /var/www/repo/os-releases/master/ubuntu-16.04-x86_64/venv-build-options-os_keystone.txt, which contains the virtualenv command, in my example: "/usr/local/bin/virtualenv --always-copy --extra-search-dir /var/www/repo/os-releases/master/ubuntu-16.04-x86_64 --never-download").
Long story short, the "New executable" line come from [3], which formats the py_executable, whose value is (unless python interpreter path is set with -p), coming from sys.executable [4], if I am not mistaken.
I didn't dig deeper on the following part, but I expect that some sys.executable can be different on different systems, which would explain why some could return python2 instead of python.
So on those systems, the generated line would be #!/tmp/openstack-venv-builder/venvs/keystone/bin/python2 as shown in the log [5].
(Edit: This probably comes from the shabang of /usr/bin/virtualenv, which
could contain python2 instead of python
Edit 2: That is exact, see [8] for gate tests to prove this)
virtualenv-tools wouldn't recognize those shabangs [6] because they don't end with python (they would end with python2), and therefore won't modify them.
The impact is now that the pip installed packages in the venv are pointing to a wrong executable and therefore can't execute: the path #!/tmp/openstack-venv-builder/venvs/keystone/bin/python2 won't exist on the destination machine.
We should probably get rid of virtualenv-tools, or be more explicit with the executable used, all the way in the process, and replace it with simply line in files (sedding) the py files. The binary files should not be shipped in the virtualenv tgz anyway.
[1]: https://github.com/openstack/openstack-ansible-os_keystone/blob/a48a73089286a370312a35fc7df54a6a3a513fa2/tasks/keystone_install.yml#L165-L171
[2]: https://github.com/fireteam/virtualenv-tools/issues/5
[3]: https://github.com/evrardjp/virtualenv/blob/master/virtualenv.py#L1217
[4]: https://github.com/evrardjp/virtualenv/blob/master/virtualenv.py#L1205
[5]: http://logs.openstack.org/24/504224/40/check/openstack-ansible-deploy-all_metal-ubuntu-xenial/9b0dec0/logs/host/repo/venv_build_keystone-master-x86_64.log.txt.gz
[6]: https://github.com/fireteam/virtualenv-tools/blob/master/virtualenv_tools.py#L68-L69
[7]: https://twitter.com/evrardjp/status/950069466112692224
[8]: https://review.openstack.org/#/c/531687/ |
|
| 2018-01-08 08:51:54 |
Jean-Philippe Evrard |
openstack-ansible: importance |
Undecided |
Critical |
|
| 2018-01-08 08:52:03 |
Jean-Philippe Evrard |
openstack-ansible: assignee |
|
Jean-Philippe Evrard (jean-philippe-evrard) |
|
| 2018-01-08 12:34:21 |
OpenStack Infra |
openstack-ansible: status |
New |
In Progress |
|
| 2018-01-09 11:32:09 |
Jean-Philippe Evrard |
openstack-ansible: status |
In Progress |
New |
|
| 2018-01-09 16:34:03 |
Jean-Philippe Evrard |
openstack-ansible: status |
New |
Confirmed |
|
| 2018-01-09 16:34:11 |
Jean-Philippe Evrard |
openstack-ansible: status |
Confirmed |
In Progress |
|
| 2018-02-07 19:35:19 |
OpenStack Infra |
openstack-ansible: assignee |
Jean-Philippe Evrard (jean-philippe-evrard) |
Mohammed Naser (mnaser) |
|
| 2018-02-08 08:50:58 |
OpenStack Infra |
openstack-ansible: assignee |
Mohammed Naser (mnaser) |
Jean-Philippe Evrard (jean-philippe-evrard) |
|
| 2018-02-09 16:43:52 |
OpenStack Infra |
openstack-ansible: assignee |
Jean-Philippe Evrard (jean-philippe-evrard) |
Mohammed Naser (mnaser) |
|
| 2018-02-13 21:51:33 |
OpenStack Infra |
openstack-ansible: assignee |
Mohammed Naser (mnaser) |
Jean-Philippe Evrard (jean-philippe-evrard) |
|
| 2018-02-16 16:16:39 |
OpenStack Infra |
tags |
|
in-stable-queens |
|
| 2018-03-27 10:50:16 |
OpenStack Infra |
tags |
in-stable-queens |
in-stable-pike in-stable-queens |
|
| 2018-04-11 11:31:09 |
Jean-Philippe Evrard |
openstack-ansible: status |
In Progress |
Fix Released |
|
