Value Error for Multiple Swift Services

[Oliver]

The following behaviour occured on OpenStack Ansible Victoria (22.1.0):

On my swift host, for the swift-container-updater service, the following error appears in the log:
(the same is true for the swift-container-auditor, swift-account-auditor and swift-account-reaper services.)

Mar 22 08:42:19 bc1bl12 systemd[1]: swift-container-updater.service: Main
process exited, code=exited, status=1/FAILURE
Mar 22 08:42:19 bc1bl12 systemd[1]: swift-container-updater.service: Failed with
result 'exit-code'.
Mar 22 08:42:21 bc1bl12 systemd[1]: swift-container-updater.service: Scheduled
restart job, restart counter is at 162982.
Mar 22 08:42:21 bc1bl12 systemd[1]: Stopped swift-container-updater service.
Mar 22 08:42:21 bc1bl12 systemd[1]: Started swift-container-updater service.
Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: Traceback (most recent
call last):
Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: File
"/openstack/venvs/swift-22.1.0/lib/python3.8/site-packages/swift/common/utils.py",
line 803, in config_fallocate_value
Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: reserve_value =
float(reserve_value[:-1])
Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: ValueError: could not
convert string to float: '1%'
Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: During handling of the
above exception, another exception occurred:
Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: Traceback (most recent
call last):
Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: File
"/openstack/venvs/swift-22.1.0/bin/swift-container-updater", line 23, in
<module>
Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]:
    run_daemon(ContainerUpdater, conf_file, **options)
Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: File
"/openstack/venvs/swift-22.1.0/lib/python3.8/site-packages/swift/common/daemon.py",
line 304, in run_daemon
Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]:
    utils.config_fallocate_value(conf.get('fallocate_reserve', '1%'))
Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: File
"/openstack/venvs/swift-22.1.0/lib/python3.8/site-packages/swift/common/utils.py",
line 809, in config_fallocate_value
Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: raise
ValueError('Error: %s is an invalid value for fallocate'
Mar 22 08:42:22 bc1bl12 swift-container-updater[50699]: ValueError: Error: 1%%
is an invalid value for fallocate_reserve.

On the swift host, I found that '1%%' had been written to the respective .conf files under /etc/swift/container-server, /etc/swift/account-server/ and /etc/swift/object-server. When manually changing these to '1%', the errors disappear.

In the user_variables.yml I didn't override any swift values except for 'swift_allow_all_users: True'

Kind regards,
Oliver

[Dmitriy Rabotyagov (noonedeadpunk)]

Having a fresh look into swift role, I found what caused that. So I just tried to revert wrong commit in ML.
So the correct revert is https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/782956

Would be great if you could try it out (preferably in the sandbox :P)

[Oliver (ciwi)]

I tried it out but couldn't get setup-openstack.yml to run:

TASK [python_venv_build : Show venv install failure message] *****************************************************************************************************************************************
fatal: [swift-node1]: FAILED! => {"changed": false, "msg": "The python packages have failed to install, please check the log file located at /var/log/python_venv_build.log for more information.\n"}
fatal: [infra1_swift_proxy_container-27169fa7]: FAILED! => {"changed": false, "msg": "The python packages have failed to install, please check the log file located at /var/log/python_venv_build.log for more information.\n"}

/var/log/python_venv_build.log on swift-node1 showed this:

2021-03-25T16:40:39,726 Using pip 20.2.3 from /openstack/venvs/swift-22.1.0/lib/python3.8/site-packages/pip (python 3.8)
2021-03-25T16:40:39,726 Non-user install because user site-packages disabled
2021-03-25T16:40:39,837 Created temporary directory: /tmp/pip-ephem-wheel-cache-ze3zoly2
2021-03-25T16:40:39,838 Created temporary directory: /tmp/pip-req-tracker-tavr0jtw
2021-03-25T16:40:39,838 Initialized build tracking at /tmp/pip-req-tracker-tavr0jtw
2021-03-25T16:40:39,838 Created build tracker: /tmp/pip-req-tracker-tavr0jtw
2021-03-25T16:40:39,838 Entered build tracker: /tmp/pip-req-tracker-tavr0jtw
2021-03-25T16:40:39,838 Created temporary directory: /tmp/pip-install-l6n19cb3
2021-03-25T16:40:39,946 Looking in links: http://192.168.110.211:8181/os-releases/22.1.0/ubuntu-20.04-x86_64
2021-03-25T16:40:39,949 Setting ceilometermiddleware==2.1.0 (from -c /openstack/venvs/swift-22.1.0/constraints.txt (line 47)) extras to: ()
2021-03-25T16:40:39,949 Setting cryptography==3.1 (from -c /openstack/venvs/swift-22.1.0/constraints.txt (line 71)) extras to: ()
2021-03-25T16:40:39,949 Setting dnspython==1.16.0 (from -c /openstack/venvs/swift-22.1.0/constraints.txt (line 1)) extras to: ()
2021-03-25T16:40:39,949 Setting ecdsa==0.16.1 (from -c /openstack/venvs/swift-22.1.0/constraints.txt (line 5)) extras to: ()
2021-03-25T16:40:39,950 ERROR: Could not satisfy constraints for 'swift': installation from path or url cannot be constrained to a version
2021-03-25T16:40:39,950 Exception information:
2021-03-25T16:40:39,950 Traceback (most recent call last):
2021-03-25T16:40:39,950 File "/openstack/venvs/swift-22.1.0/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 228, in _main
2021-03-25T16:40:39,950 status = self.run(options, args)
2021-03-25T16:40:39,950 File "/openstack/venvs/swift-22.1.0/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
2021-03-25T16:40:39,950 return func(self, options, args)
2021-03-25T16:40:39,950 File "/openstack/venvs/swift-22.1.0/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 323, in run
2021-03-25T16:40:39,950 requirement_set = resolver.resolve(
2021-03-25T16:40:39,950 File "/openstack/venvs/swift-22.1.0/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 173, in resolve
2021-03-25T16:40:39,950 requirement_set.add_requirement(req)
2021-03-25T16:40:39,950 File "/openstack/venvs/s...

[Dmitriy Rabotyagov (noonedeadpunk)]

Can you also kindly attach output 2 or 3 tasks before `Show venv install failure message` as it should contain another failed task with command that has been issued but failed.

As I'd expect seing such issue on master with new pip resolver, but not on V.

[Oliver (ciwi)]

Sure, here's the output:

TASK [python_venv_build : Install python packages into the venv] *************************************************************************************************************************************
FAILED - RETRYING: Install python packages into the venv (5 retries left).
FAILED - RETRYING: Install python packages into the venv (5 retries left).
FAILED - RETRYING: Install python packages into the venv (4 retries left).
FAILED - RETRYING: Install python packages into the venv (4 retries left).
FAILED - RETRYING: Install python packages into the venv (3 retries left).
FAILED - RETRYING: Install python packages into the venv (3 retries left).
FAILED - RETRYING: Install python packages into the venv (2 retries left).
FAILED - RETRYING: Install python packages into the venv (2 retries left).
FAILED - RETRYING: Install python packages into the venv (1 retries left).
FAILED - RETRYING: Install python packages into the venv (1 retries left).
fatal: [swift-node1]: FAILED! => {"attempts": 5, "changed": false, "cmd": ["/openstack/venvs/swift-22.1.0/bin/pip3", "install", "-U", "--constraint", "/openstack/venvs/swift-22.1.0/global-constraints.txt", "--constraint", "/openstack/venvs/swift-22.1.0/constraints.txt", "--pre", "--log", "/var/log/python_venv_build.log", "--find-links", "http://192.168.110.211:8181/os-releases/22.1.0/ubuntu-20.04-x86_64", "--trusted-host", "192.168.110.211", "ceilometermiddleware", "cryptography", "dnspython", "ecdsa", "git+https://opendev.org/openstack/swift@26a20516005b1eca162da7f1d203c413e27c6104#egg=swift", "keystonemiddleware", "osprofiler", "pyeclib", "pymemcache", "python-keystoneclient", "python-memcached", "python-swiftclient", "systemd-python"], "msg": "stdout: Looking in links: http://192.168.110.211:8181/os-releases/22.1.0/ubuntu-20.04-x86_64\n\n:stderr: ERROR: Could not satisfy constraints for 'swift': installation from path or url cannot be constrained to a version\nWARNING: You are using pip version 20.2.3; however, version 21.0.1 is available.\nYou should consider upgrading via the '/openstack/venvs/swift-22.1.0/bin/python -m pip install --upgrade pip' command.\n"}
fatal: [infra1_swift_proxy_container-27169fa7]: FAILED! => {"attempts": 5, "changed": false, "cmd": ["/openstack/venvs/swift-22.1.0/bin/pip3", "install", "-U", "--constraint", "/openstack/venvs/swift-22.1.0/global-constraints.txt", "--constraint", "/openstack/venvs/swift-22.1.0/constraints.txt", "--pre", "--log", "/var/log/python_venv_build.log", "--find-links", "http://192.168.110.211:8181/os-releases/22.1.0/ubuntu-20.04-x86_64", "--trusted-host", "192.168.110.211", "ceilometermiddleware", "cryptography", "dnspython", "ecdsa", "git+https://opendev.org/openstack/swift@26a20516005b1eca162da7f1d203c413e27c6104#egg=swift", "keystonemiddleware", "osprofiler", "pyeclib", "pymemcache", "python-keystoneclient", "python-memcached", "python-swiftclient", "systemd-python"], "msg": "stdout: Looking in links: http://192.168.110.211:8181/os-releases/22.1.0/ubuntu-20.04-x86_64\n\n:stderr: ERROR: Could not satisfy constraints for 'swift': installation from path or url cannot be constrained to a version\nWARNING: You are using pip version 20....

[Dmitriy Rabotyagov (noonedeadpunk)]

According to the output, I thing you have pulled/checkout to the patch, instead of just cherry-picking it? Because `git+https://opendev.org/openstack/swift@26a20516005b1eca162da7f1d203c413e27c6104#egg=swift` placed in installation arguments is smth from master, implemented with https://opendev.org/openstack/openstack-ansible-os_swift/commit/baa3dc357801ca0631308a939259feeb0e237099

So you should ensure that you're running role from Victoria while jsut applying specific patch.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/782956
Committed: https://opendev.org/openstack/openstack-ansible-os_swift/commit/24bbb72ac97bbd2683d0eae04b8ab4c3674005bc
Submitter: "Zuul (22348)"
Branch: master

commit 24bbb72ac97bbd2683d0eae04b8ab4c3674005bc
Author: Dmitriy Rabotyagov <email address hidden>
Date: Thu Mar 25 10:08:57 2021 +0000

    Revert "split templates to work around configparser bug"

    This reverts commit 3e151d97adbff78be1b146938374a0dced5970e9.

    Reason for revert: Upstream bug has been fixed

    Change-Id: I5498030487677523ae71ae2f9a13074ab4552204
    Related-Bug: #1872553
    Closes-Bug: #1921354

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/786790

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/786790
Committed: https://opendev.org/openstack/openstack-ansible-os_swift/commit/7c1e3547190fb2c9294b47499646ee04e20b600e
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 7c1e3547190fb2c9294b47499646ee04e20b600e
Author: Dmitriy Rabotyagov <email address hidden>
Date: Thu Mar 25 10:08:57 2021 +0000

    Revert "split templates to work around configparser bug"

    This reverts commit 3e151d97adbff78be1b146938374a0dced5970e9.

    Reason for revert: Upstream bug has been fixed

    Change-Id: I5498030487677523ae71ae2f9a13074ab4552204
    Related-Bug: #1872553
    Closes-Bug: #1921354
    (cherry picked from commit 24bbb72ac97bbd2683d0eae04b8ab4c3674005bc)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/806210

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/806210
Committed: https://opendev.org/openstack/openstack-ansible-os_swift/commit/fc55b10b8771b603be90a304a2c1906a89c7bae2
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit fc55b10b8771b603be90a304a2c1906a89c7bae2
Author: Dmitriy Rabotyagov <email address hidden>
Date: Thu Mar 25 10:08:57 2021 +0000

    Revert "split templates to work around configparser bug"

    This reverts commit 3e151d97adbff78be1b146938374a0dced5970e9.

    Reason for revert: Upstream bug has been fixed

    Change-Id: I5498030487677523ae71ae2f9a13074ab4552204
    Related-Bug: #1872553
    Closes-Bug: #1921354
    (cherry picked from commit 24bbb72ac97bbd2683d0eae04b8ab4c3674005bc)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_swift ussuri-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_swift yoga-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_swift victoria-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_swift wallaby-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_swift xena-eom release.