OK and now:
2018-02-13 14:18:08.281 2065380 DEBUG neutron.agent.l3.agent [-] Starting router update for 213b6544-ab4b-4e46-a5c6-5d8d587a0c6d, action 1, priority 0 _process_router_update /usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py:483
which I think is the actual processing of the message from the work queue from 14:12:57 (wow).
action 1 is a DELETE_ROUTER.
this fails:
2018-02-13 14:18:09.959 2065380 ERROR neutron.agent.l3.agent [-] Error while deleting router 213b6544-ab4b-4e46-a5c6-5d8d587a0c6d 2018-02-13 14:18:09.959 2065380 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2018-02-13 14:18:09.959 2065380 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 366, in _safe_router_removed 2018-02-13 14:18:09.959 2065380 ERROR neutron.agent.l3.agent self._router_removed(router_id) 2018-02-13 14:18:09.959 2065380 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 385, in _router_removed 2018-02-13 14:18:09.959 2065380 ERROR neutron.agent.l3.agent ri.delete() 2018-02-13 14:18:09.959 2065380 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/ha_router.py", line 420, in delete 2018-02-13 14:18:09.959 2065380 ERROR neutron.agent.l3.agent self.destroy_state_change_monitor(self.process_monitor) 2018-02-13 14:18:09.959 2065380 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/ha_router.py", line 355, in destroy_state_change_monitor 2018-02-13 14:18:09.959 2065380 ERROR neutron.agent.l3.agent pm = self._get_state_change_monitor_process_manager() 2018-02-13 14:18:09.959 2065380 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/ha_router.py", line 326, in _get_state_change_monitor_process_manager 2018-02-13 14:18:09.959 2065380 ERROR neutron.agent.l3.agent default_cmd_callback=self._get_state_change_monitor_callback()) 2018-02-13 14:18:09.959 2065380 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/ha_router.py", line 330, in _get_state_change_monitor_callback 2018-02-13 14:18:09.959 2065380 ERROR neutron.agent.l3.agent ha_cidr = self._get_primary_vip() 2018-02-13 14:18:09.959 2065380 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/ha_router.py", line 166, in _get_primary_vip 2018-02-13 14:18:09.959 2065380 ERROR neutron.agent.l3.agent return self._get_keepalived_instance().get_primary_vip() 2018-02-13 14:18:09.959 2065380 ERROR neutron.agent.l3.agent AttributeError: 'NoneType' object has no attribute 'get_primary_vip' 2018-02-13 14:18:09.959 2065380 ERROR neutron.agent.l3.agent 2018-02-13 14:18:09.960 2065380 DEBUG neutron.agent.l3.agent [-] Finished a router update for 213b6544-ab4b-4e46-a5c6-5d8d587a0c6d _process_router_update /usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py:513
OK and now:
2018-02-13 14:18:08.281 2065380 DEBUG neutron. agent.l3. agent [-] Starting router update for 213b6544- ab4b-4e46- a5c6-5d8d587a0c 6d, action 1, priority 0 _process_ router_ update /usr/lib/ python2. 7/dist- packages/ neutron/ agent/l3/ agent.py: 483
which I think is the actual processing of the message from the work queue from 14:12:57 (wow).
action 1 is a DELETE_ROUTER.
this fails:
2018-02-13 14:18:09.959 2065380 ERROR neutron. agent.l3. agent [-] Error while deleting router 213b6544- ab4b-4e46- a5c6-5d8d587a0c 6d agent.l3. agent Traceback (most recent call last): agent.l3. agent File "/usr/lib/ python2. 7/dist- packages/ neutron/ agent/l3/ agent.py" , line 366, in _safe_router_ removed agent.l3. agent self._router_ removed( router_ id) agent.l3. agent File "/usr/lib/ python2. 7/dist- packages/ neutron/ agent/l3/ agent.py" , line 385, in _router_removed agent.l3. agent ri.delete() agent.l3. agent File "/usr/lib/ python2. 7/dist- packages/ neutron/ agent/l3/ ha_router. py", line 420, in delete agent.l3. agent self.destroy_ state_change_ monitor( self.process_ monitor) agent.l3. agent File "/usr/lib/ python2. 7/dist- packages/ neutron/ agent/l3/ ha_router. py", line 355, in destroy_ state_change_ monitor agent.l3. agent pm = self._get_ state_change_ monitor_ process_ manager( ) agent.l3. agent File "/usr/lib/ python2. 7/dist- packages/ neutron/ agent/l3/ ha_router. py", line 326, in _get_state_ change_ monitor_ process_ manager agent.l3. agent default_ cmd_callback= self._get_ state_change_ monitor_ callback( )) agent.l3. agent File "/usr/lib/ python2. 7/dist- packages/ neutron/ agent/l3/ ha_router. py", line 330, in _get_state_ change_ monitor_ callback agent.l3. agent ha_cidr = self._get_ primary_ vip() agent.l3. agent File "/usr/lib/ python2. 7/dist- packages/ neutron/ agent/l3/ ha_router. py", line 166, in _get_primary_vip agent.l3. agent return self._get_ keepalived_ instance( ).get_primary_ vip() agent.l3. agent AttributeError: 'NoneType' object has no attribute 'get_primary_vip' agent.l3. agent agent.l3. agent [-] Finished a router update for 213b6544- ab4b-4e46- a5c6-5d8d587a0c 6d _process_ router_ update /usr/lib/ python2. 7/dist- packages/ neutron/ agent/l3/ agent.py: 513
2018-02-13 14:18:09.959 2065380 ERROR neutron.
2018-02-13 14:18:09.959 2065380 ERROR neutron.
2018-02-13 14:18:09.959 2065380 ERROR neutron.
2018-02-13 14:18:09.959 2065380 ERROR neutron.
2018-02-13 14:18:09.959 2065380 ERROR neutron.
2018-02-13 14:18:09.959 2065380 ERROR neutron.
2018-02-13 14:18:09.959 2065380 ERROR neutron.
2018-02-13 14:18:09.959 2065380 ERROR neutron.
2018-02-13 14:18:09.959 2065380 ERROR neutron.
2018-02-13 14:18:09.959 2065380 ERROR neutron.
2018-02-13 14:18:09.959 2065380 ERROR neutron.
2018-02-13 14:18:09.959 2065380 ERROR neutron.
2018-02-13 14:18:09.959 2065380 ERROR neutron.
2018-02-13 14:18:09.959 2065380 ERROR neutron.
2018-02-13 14:18:09.959 2065380 ERROR neutron.
2018-02-13 14:18:09.959 2065380 ERROR neutron.
2018-02-13 14:18:09.959 2065380 ERROR neutron.
2018-02-13 14:18:09.960 2065380 DEBUG neutron.