Unscheduling a router(HA/legacy) from an agent not scheduled for that router causes an error.

[Yoni Shafrir]

A legacy router was created and automatically scheduled to both agents.
Given this state of the agent scheduling :

[stack@vpn-6-98 devstack (master %=)]$ neutron l3-agent-list-hosting-router R5
+--------------------------------------+-------------------------+----------------+-------+
| id | host | admin_state_up | alive |
+--------------------------------------+-------------------------+----------------+-------+
| fda97b55-813b-486e-8a74-e2c7aa830f52 | vpn-6-41.tlv.redhat.com | True | :-) |
+--------------------------------------+-------------------------+----------------+-------+

When trying to remove a router from the same agent twice an error occurs:

[stack@vpn-6-98 devstack (master %=)]$ neutron l3-agent-router-remove fda97b55-813b-486e-8a74-e2c7aa830f52 R5
Removed router R5 from L3 agent
[stack@vpn-6-98 devstack (master %=)]$ neutron l3-agent-router-remove fda97b55-813b-486e-8a74-e2c7aa830f52 R5
Conflict (HTTP 409) (Request-ID: req-9a96e8a2-70f8-4c24-b256-6dffb933282c)

For reference, when trying to re-schedule the router on the same agent the client notifies the user:

neutron l3-agent-router-add fda97b55-813b-486e-8a74-e2c7aa830f52 R5
Added router R5 to L3 agent
neutron l3-agent-router-add fda97b55-813b-486e-8a74-e2c7aa830f52 R5
Added router R5 to L3 agent

There is no 'real' effect on the DB or anything but the client should indicate the operation failed
since the agent is already scheduled.

The second removal of the same agent shouldn't fail as we try to avoid
unnecessary errors.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/148883

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/148883
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f78ce6782a57d6fd63c6e1dabd6a45cbd5c458cc
Submitter: Jenkins
Branch: master

commit f78ce6782a57d6fd63c6e1dabd6a45cbd5c458cc
Author: Yoni Shafrir <email address hidden>
Date: Wed Jan 21 08:25:50 2015 +0200

    Removing a router twice from the same agent shouldn't cause an error

    When we remove a router from an agent that has already been
    unscheduled from we raise an exception that eventually causes an error.
    The method '_unbind_router' raises a 'RouterNotHostedByL3Agent' exception
    on failure. In both cases the actual removal of the router
    from the same agent has no effect.

    The solution is to stop raising 'RouterNotHostedByL3Agent' so
    that _unbind_router() being invoked without error can indicate that
    the router is no longer bound.

    This solution matches the behaviour found when trying
    to schedule a router to the same agent twice.

    This change is a result of the discussion in:

    https://review.openstack.org/#/c/144681/2

    Closes-Bug: #1406705

    Change-Id: I015bfc0fde11ba4f39417e4c134faa8132cb3eac