ItemAllocator class can throw a ValueError when file is corrupted

[Brian Haley]

While running tests for an unrelated change, I saw this failure:

neutron.tests.unit.agent.l3.test_dvr_fip_ns.TestDvrFipNs.test_subscribe
-----------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "neutron/tests/unit/agent/l3/test_dvr_fip_ns.py", line 40, in setUp
        use_ipv6=True)
      File "neutron/agent/l3/dvr_fip_ns.py", line 57, in __init__
        FIP_PR_END)
      File "neutron/agent/l3/fip_rule_priority_allocator.py", line 53, in __init__
        pool)
      File "neutron/agent/l3/item_allocator.py", line 46, in __init__
        key, saved_value = line.strip().split(delimiter)
    ValueError: need more than 1 value to unpack

This is due to the fact that all the tests in this file using the ItemAllocator class are using/re-using the same files in /tmp, and when the right ones are run simultaneously, the file can be truncated, leading to a corruption. To recreate it you just have to do this:

$ echo "111.2.3.466652" > /tmp/fip-priorities
$ tox -e py27 neutron.tests.unit.agent.l3.test_dvr_fip_ns.TestDvrFipNs.test_subscribe

So the tests need to change to use unique files, and clean them up on exit.

But it also shows that the ItemAllocator class cannot deal with a corrupt file. In this case we should catch the error and skip the invalid entry, since it could cause the l3-agent to not start.

The same is probably true of other files in /tmp the tests are using as well.

[Brandon Logan (brandon-logan)]

i can reproduce as well. thanks for the easy steps to follow :)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/341794

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/341794
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ed34f18916c62d77358d4c5910ff17694828576a
Submitter: Jenkins
Branch: master

commit ed34f18916c62d77358d4c5910ff17694828576a
Author: Brian Haley <email address hidden>
Date: Wed Jul 13 16:06:06 2016 -0400

    DVR: Fix ItemAllocator class to handle exceptions

    The ItemAllocator class blindly reads from a file
    when initialized, assuming each line is always in
    "key,value" syntax. It's possible that the file
    may become corrupted, leading to ValueErrors being
    thrown which it doesn't handle.

    This was found running the unit tests, more specifically
    those doing DVR FIP namespace operations, since they were
    always reading/writing the same files in /tmp. Typically
    the failure was random and would go away with a recheck,
    but could be the cause of check job instability. All
    occurrences were fixed to use a test-created temporary
    directory to avoid collision with each other.

    Change-Id: I39d116aa8261b50bfcb3269416c1a307cafa134e
    Closes-bug: #1602794

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 9.0.0.0b3 development milestone.