Had to put a waitall() on the greenpool when the task was being spawned. That gave me some useful logs and I can watch the task going to failure. Not sure yet exactly why without the waitall() the tasks are just sitting there.
This is definitely a problem though:
Aug 29 11:47:02 br-virtual-machine <email address hidden>[65490]: DEBUG oslo_concurrency.processutils [None req-81a7fc37-af25-499f-9190-70851c43eda5 admin admin] Running cmd (subprocess): /usr/local/bin/uwsgi -m oslo_concurrency.prlimit --as=1073741824 --cpu=2 -- qemu-img info --output=json file:///tmp/glance_workdir/0d397d5e-0662-40c5-ba24-230d11f5788c {{(pid=65491) execute /usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:355}}
Aug 29 11:47:02 br-virtual-machine <email address hidden>[65490]: DEBUG oslo_concurrency.processutils [None req-81a7fc37-af25-499f-9190-70851c43eda5 admin admin] CMD "/usr/local/bin/uwsgi -m oslo_concurrency.prlimit --as=1073741824 --cpu=2 -- qemu-img info --output=json file:///tmp/glance_workdir/0d397d5e-0662-40c5-ba24-230d11f5788c" returned: 1 in 0.016s {{(pid=65491) execute /usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:385}}
Aug 29 11:47:02 br-virtual-machine <email address hidden>[65490]: DEBUG oslo_concurrency.processutils [None req-81a7fc37-af25-499f-9190-70851c43eda5 admin admin] None
Aug 29 11:47:02 br-virtual-machine <email address hidden>[65490]: command: u'/usr/local/bin/uwsgi -m oslo_concurrency.prlimit --as=1073741824 --cpu=2 -- qemu-img info --output=json file:///tmp/glance_workdir/0d397d5e-0662-40c5-ba24-230d11f5788c'
Aug 29 11:47:02 br-virtual-machine <email address hidden>[65490]: exit code: 1
Aug 29 11:47:02 br-virtual-machine <email address hidden>[65490]: stdout: u''
Aug 29 11:47:02 br-virtual-machine <email address hidden>[65490]: stderr: u'unable to load configuration from oslo_concurrency.prlimit\n' {{(pid=65491) execute /usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:424}}
Aug 29 11:47:02 br-virtual-machine <email address hidden>[65490]: DEBUG oslo_concurrency.processutils [None req-81a7fc37-af25-499f-9190-70851c43eda5 admin admin] u'/usr/local/bin/uwsgi -m oslo_concurrency.prlimit --as=1073741824 --cpu=2 -- qemu-img info --output=json file:///tmp/glance_workdir/0d397d5e-0662-40c5-ba24-230d11f5788c' failed. Not Retrying. {{(pid=65491) execute /usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:433}}
The command being run in the subprocess seems kind of weird, i.e., /usr/local/bin/uwsgi
Had to put a waitall() on the greenpool when the task was being spawned. That gave me some useful logs and I can watch the task going to failure. Not sure yet exactly why without the waitall() the tasks are just sitting there.
This is definitely a problem though:
Aug 29 11:47:02 br-virtual-machine <email address hidden>[65490]: DEBUG oslo_concurrenc y.processutils [None req-81a7fc37- af25-499f- 9190-70851c43ed a5 admin admin] Running cmd (subprocess): /usr/local/ bin/uwsgi -m oslo_concurrenc y.prlimit --as=1073741824 --cpu=2 -- qemu-img info --output=json file:// /tmp/glance_ workdir/ 0d397d5e- 0662-40c5- ba24-230d11f578 8c {{(pid=65491) execute /usr/local/ lib/python2. 7/dist- packages/ oslo_concurrenc y/processutils. py:355} } y.processutils [None req-81a7fc37- af25-499f- 9190-70851c43ed a5 admin admin] CMD "/usr/local/ bin/uwsgi -m oslo_concurrenc y.prlimit --as=1073741824 --cpu=2 -- qemu-img info --output=json file:// /tmp/glance_ workdir/ 0d397d5e- 0662-40c5- ba24-230d11f578 8c" returned: 1 in 0.016s {{(pid=65491) execute /usr/local/ lib/python2. 7/dist- packages/ oslo_concurrenc y/processutils. py:385} } y.processutils [None req-81a7fc37- af25-499f- 9190-70851c43ed a5 admin admin] None local/bin/ uwsgi -m oslo_concurrenc y.prlimit --as=1073741824 --cpu=2 -- qemu-img info --output=json file:// /tmp/glance_ workdir/ 0d397d5e- 0662-40c5- ba24-230d11f578 8c' y.prlimit\ n' {{(pid=65491) execute /usr/local/ lib/python2. 7/dist- packages/ oslo_concurrenc y/processutils. py:424} } y.processutils [None req-81a7fc37- af25-499f- 9190-70851c43ed a5 admin admin] u'/usr/ local/bin/ uwsgi -m oslo_concurrenc y.prlimit --as=1073741824 --cpu=2 -- qemu-img info --output=json file:// /tmp/glance_ workdir/ 0d397d5e- 0662-40c5- ba24-230d11f578 8c' failed. Not Retrying. {{(pid=65491) execute /usr/local/ lib/python2. 7/dist- packages/ oslo_concurrenc y/processutils. py:433} }
Aug 29 11:47:02 br-virtual-machine <email address hidden>[65490]: DEBUG oslo_concurrenc
Aug 29 11:47:02 br-virtual-machine <email address hidden>[65490]: DEBUG oslo_concurrenc
Aug 29 11:47:02 br-virtual-machine <email address hidden>[65490]: command: u'/usr/
Aug 29 11:47:02 br-virtual-machine <email address hidden>[65490]: exit code: 1
Aug 29 11:47:02 br-virtual-machine <email address hidden>[65490]: stdout: u''
Aug 29 11:47:02 br-virtual-machine <email address hidden>[65490]: stderr: u'unable to load configuration from oslo_concurrenc
Aug 29 11:47:02 br-virtual-machine <email address hidden>[65490]: DEBUG oslo_concurrenc
The command being run in the subprocess seems kind of weird, i.e., /usr/local/ bin/uwsgi