| 2018-02-18 12:56:03 |
Shannon Mitchell |
description |
I have been seeing the dynamic_inventory.py runs randomly break more recently. It seems to happen
more often on larger environments with multiple people/scripts accessing the inventory for various
tasks. It looks like the /etc/openstack_deploy/backup_openstack_inventory.tar archive is getting
corrupted causing the inventory script to bomb on all future calls. The workaround is to blow it
away the tar file and rerun the dynamic_inventory.py script to re-generate.
You can simulate by running multiple calls to it at once.
root@shan5464-aio:/opt/openstack-ansible/inventory# cat test.sh
#!/bin/bash
rm /etc/openstack_deploy/backup_openstack_inventory.tar
for ITER in {1..40}; do
/opt/openstack-ansible/inventory/dynamic_inventory.py > /dev/null 2>&1 &
done
wait
root@shan5464-aio:/opt/openstack-ansible/inventory# ./test.sh
root@shan5464-aio:/opt/openstack-ansible/inventory# /opt/openstack-ansible/inventory/dynamic_inventory.py
Traceback (most recent call last):
File "/opt/openstack-ansible/inventory/dynamic_inventory.py", line 80, in <module>
output = generate.main(**all_args)
File "/opt/openstack-ansible/osa_toolkit/generate.py", line 1075, in main
inventory, inv_path = filesys.load_inventory(config, INVENTORY_SKEL)
File "/opt/openstack-ansible/osa_toolkit/filesystem.py", line 247, in load_inventory
_make_backup(load_path, file_loaded)
File "/opt/openstack-ansible/osa_toolkit/filesystem.py", line 156, in _make_backup
with tarfile.open(inventory_backup_file, 'a') as tar:
File "/usr/lib/python2.7/tarfile.py", line 1711, in open
return cls.taropen(name, mode, fileobj, **kwargs)
File "/usr/lib/python2.7/tarfile.py", line 1721, in taropen
return cls(name, mode, fileobj, **kwargs)
File "/usr/lib/python2.7/tarfile.py", line 1601, in __init__
raise ReadError(str(e))
tarfile.ReadError: empty header
root@shan5464-aio:/opt/openstack-ansible/inventory# rm /etc/openstack_deploy/backup_openstack_inventory.tar
root@shan5464-aio:/opt/openstack-ansible/inventory# /opt/openstack-ansible/inventory/dynamic_inventory.py | head -n 4
{
"_meta": {
"hostvars": {
"shan5464-aio": {
Looks like it may need to handle locking in osa_toolkit/filesystem.py or a way to run it read-only to keep the inventory/backup files from being generated on every call. |
I have been seeing the dynamic_inventory.py runs randomly break more recently. It seems to happen more often on larger environments with
multiple people/scripts accessing the inventory for various tasks.
It looks like the /etc/openstack_deploy/backup_openstack_inventory.tar archive is getting corrupted causing the inventory script to bomb on
all future calls. The workaround is to blow it away the tar file and
rerun the dynamic_inventory.py script to re-generate.
You can simulate by running multiple calls to it at once.
root@shan5464-aio:/opt/openstack-ansible/inventory# cat test.sh
#!/bin/bash
rm /etc/openstack_deploy/backup_openstack_inventory.tar
for ITER in {1..40}; do
/opt/openstack-ansible/inventory/dynamic_inventory.py > /dev/null 2>&1 &
done
wait
root@shan5464-aio:/opt/openstack-ansible/inventory# ./test.sh
root@shan5464-aio:/opt/openstack-ansible/inventory# /opt/openstack-ansible/inventory/dynamic_inventory.py
Traceback (most recent call last):
File "/opt/openstack-ansible/inventory/dynamic_inventory.py", line 80, in <module>
output = generate.main(**all_args)
File "/opt/openstack-ansible/osa_toolkit/generate.py", line 1075, in main
inventory, inv_path = filesys.load_inventory(config, INVENTORY_SKEL)
File "/opt/openstack-ansible/osa_toolkit/filesystem.py", line 247, in load_inventory
_make_backup(load_path, file_loaded)
File "/opt/openstack-ansible/osa_toolkit/filesystem.py", line 156, in _make_backup
with tarfile.open(inventory_backup_file, 'a') as tar:
File "/usr/lib/python2.7/tarfile.py", line 1711, in open
return cls.taropen(name, mode, fileobj, **kwargs)
File "/usr/lib/python2.7/tarfile.py", line 1721, in taropen
return cls(name, mode, fileobj, **kwargs)
File "/usr/lib/python2.7/tarfile.py", line 1601, in __init__
raise ReadError(str(e))
tarfile.ReadError: empty header
root@shan5464-aio:/opt/openstack-ansible/inventory# rm /etc/openstack_deploy/backup_openstack_inventory.tar
root@shan5464-aio:/opt/openstack-ansible/inventory# /opt/openstack-ansible/inventory/dynamic_inventory.py | head -n 4
{
"_meta": {
"hostvars": {
"shan5464-aio": {
Looks like it may need to handle locking in osa_toolkit/filesystem.py or a way to run it read-only to keep the inventory/backup files from being generated on every call. |
|
