auditor recon dump is not consistent on new nodes

[clayg]

When you first bring up a node, and there's no data to audit the recon dump includes an empty disk:

ubuntu@saio:/etc/swift$ cat /var/cache/swift/node4/object.recon | python -m json.tool
{
    "object_auditor_stats_ALL": {},
    "object_auditor_stats_ZBF": {},
    "object_reconstruction_last": 1500319192.22745,
    "object_reconstruction_time": 0.00022380352020263673,
    "object_replication_last": 1500319107.354281,
    "object_replication_time": 9.981393814086914e-05,
    "replication_last": 1500319107.354281,
    "replication_stats": {
        "attempted": 0,
        "failure": 0,
        "hashmatch": 0,
        "remove": 0,
        "rsync": 0,
        "start": 1500319107.347603,
        "success": 0
    },
    "replication_time": 9.981393814086914e-05
}

Then if you run it *again* the keys are missing:

ubuntu@saio:/etc/swift$ cat /var/cache/swift/node4/object.recon | python -m json.tool
{
    "object_reconstruction_last": 1500319192.22745,
    "object_reconstruction_time": 0.00022380352020263673,
    "object_replication_last": 1500319107.354281,
    "object_replication_time": 9.981393814086914e-05,
    "replication_last": 1500319107.354281,
    "replication_stats": {
        "attempted": 0,
        "failure": 0,
        "hashmatch": 0,
        "remove": 0,
        "rsync": 0,
        "start": 1500319107.347603,
        "success": 0
    },
    "replication_time": 9.981393814086914e-05
}

Easiest way to observe is just running a single auditor on a brand new saio:

swift-init object-auditor once -nv -c 4

Then check the recon dump between passes, you can also set your run_pause to 0 and watch it toggle.

I think it would be better if once the auditor finished it wrote empty stats and the keys stayed there.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/484375
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=c2e59b9b8b43e5846c0fe20cfd7810bbc0291cad
Submitter: Jenkins
Branch: master

commit c2e59b9b8b43e5846c0fe20cfd7810bbc0291cad
Author: Alistair Coles <email address hidden>
Date: Mon Jul 17 14:29:53 2017 +0100

    Make dict deletion idempotent in dump_recon_cache

    Calling dump_recon_cache with a key mapped to an empty dict value
    causes the key to be removed from the cache entry. Doing the same
    again causes the key to be added back and mapped an empty dict, and
    the key continues to toggle as calls are repeated. This behavior is
    seen on the Related-Bug report.

    This patch fixes dump_recon_cache to make deletion of a key
    idempotent. This fix is needed for the Related-Change which makes use
    of empty dicts with dump_recon_cache to clear unwanted keys from the
    cache.

    The only caller that currently set empty dict values is
    obj/auditor.py where the current intended behavior would appear to be
    as per this patch.

    Related-Change: I28925a37f3985c9082b5a06e76af4dc3ec813abe
    Related-Bug: #1704858
    Change-Id: If9638b4e7dba0ec2c7bd95809cec6c5e18e9301e