Comment 0 for bug 1750233

I have been seeing the dynamic_inventory.py runs randomly break more recently. It seems to happen
more often on larger environments with multiple people/scripts accessing the inventory for various
tasks. It looks like the /etc/openstack_deploy/backup_openstack_inventory.tar archive is getting
corrupted causing the inventory script to bomb on all future calls. The workaround is to blow it
away the tar file and rerun the dynamic_inventory.py script to re-generate.

You can simulate by running multiple calls to it at once.

root@shan5464-aio:/opt/openstack-ansible/inventory# cat test.sh
#!/bin/bash

rm /etc/openstack_deploy/backup_openstack_inventory.tar
for ITER in {1..40}; do
  /opt/openstack-ansible/inventory/dynamic_inventory.py > /dev/null 2>&1 &
done
wait

root@shan5464-aio:/opt/openstack-ansible/inventory# ./test.sh

root@shan5464-aio:/opt/openstack-ansible/inventory# /opt/openstack-ansible/inventory/dynamic_inventory.py
Traceback (most recent call last):
  File "/opt/openstack-ansible/inventory/dynamic_inventory.py", line 80, in <module>
    output = generate.main(**all_args)
  File "/opt/openstack-ansible/osa_toolkit/generate.py", line 1075, in main
    inventory, inv_path = filesys.load_inventory(config, INVENTORY_SKEL)
  File "/opt/openstack-ansible/osa_toolkit/filesystem.py", line 247, in load_inventory
    _make_backup(load_path, file_loaded)
  File "/opt/openstack-ansible/osa_toolkit/filesystem.py", line 156, in _make_backup
    with tarfile.open(inventory_backup_file, 'a') as tar:
  File "/usr/lib/python2.7/tarfile.py", line 1711, in open
    return cls.taropen(name, mode, fileobj, **kwargs)
  File "/usr/lib/python2.7/tarfile.py", line 1721, in taropen
    return cls(name, mode, fileobj, **kwargs)
  File "/usr/lib/python2.7/tarfile.py", line 1601, in __init__
    raise ReadError(str(e))
tarfile.ReadError: empty header

root@shan5464-aio:/opt/openstack-ansible/inventory# rm /etc/openstack_deploy/backup_openstack_inventory.tar

root@shan5464-aio:/opt/openstack-ansible/inventory# /opt/openstack-ansible/inventory/dynamic_inventory.py | head -n 4
{
    "_meta": {
        "hostvars": {
            "shan5464-aio": {

Looks like it may need to handle locking in osa_toolkit/filesystem.py or a way to run it read-only to keep the inventory/backup files from being generated on every call.