clean_interfaces in asr1k_cfg_syncer fails due to HA assumption

[James Denton]

Environment: RDO
Version: Ocata-3
Networking-Cisco: 5.3.0
Device: Cisco ASR 1001 (Standalone)
Version: 03.16.06.S

In my environment, I have the following router type defined:

[root@rdo02-ocata ~(keystone_admin)]# neutron cisco-router-type-show 00000000-0000-0000-0000-000000000003
+--------------------------+------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+--------------------------+------------------------------------------------------------------------------------------------------------------------+
| cfg_agent_driver | networking_cisco.plugins.cisco.cfg_agent.device_drivers.asr1k.asr1k_routing_driver.ASR1kRoutingDriver |
| cfg_agent_service_helper | networking_cisco.plugins.cisco.cfg_agent.service_helpers.routing_svc_helper.RoutingServiceHelper |
| description | Neutron router implemented in Cisco ASR1k device |
| driver | networking_cisco.plugins.cisco.l3.drivers.asr1k.asr1k_routertype_driver.ASR1kL3RouterDriver |
| ha_enabled_by_default | False |
| id | 00000000-0000-0000-0000-000000000003 |
| name | ASR1k_router |
| project_id | c39ec022ee8b4b8f90e025fa3e13f682 |
| scheduler | networking_cisco.plugins.cisco.l3.schedulers.l3_router_hosting_device_scheduler.L3RouterHostingDeviceHARandomScheduler |
| shared | True |
| slot_need | 2 |
| template_id | 00000000-0000-0000-0000-000000000003 |
| tenant_id | c39ec022ee8b4b8f90e025fa3e13f682 |
+--------------------------+------------------------------------------------------------------------------------------------------------------------+

I used Neutron to configure a VLAN-based tenant network, VLAN-based external network, and a router object. The driver appeared to configure all necessary objects on the ASR without much trouble. The interface for the tenant network was configured like so:

interface GigabitEthernet0/0/2.794
 description OPENSTACK_NEUTRON_INTF
 encapsulation dot1Q 794
 vrf forwarding nrouter-93dce6
 ip address 172.16.0.1 255.255.255.0
 ip nat inside

The interface for the external network was configured like so (HSRP is questionable since HA is False, but that may be another bug):

interface GigabitEthernet0/0/2.70
 description OPENSTACK_NEUTRON_EXTERNAL_INTF
 encapsulation dot1Q 70
 ip address 192.168.70.5 255.255.255.0
 ip nat outside
 standby delay minimum 30 reload 60
 standby version 2
 standby 1064 ip 192.168.70.9
 standby 1064 timers 1 3
 standby 1064 name neutron-hsrp-1064-70

Upon creating a second router and attaching the respective interfaces with the Neutron CLI, it was observed that nothing was built on the ASR. The neutron-cisco-cfg-agent service was later restarted, at which point the following traceback was seen:

2017-12-22 11:56:51.073 28115 INFO networking_cisco.plugins.cisco.cfg_agent.device_drivers.asr1k.asr1k_cfg_syncer [-] neutron router db records
2017-12-22 11:56:51.073 28115 INFO networking_cisco.plugins.cisco.cfg_agent.device_drivers.asr1k.asr1k_cfg_syncer [-] ROUTER_ID: 93dce6
2017-12-22 11:56:51.073 28115 INFO networking_cisco.plugins.cisco.cfg_agent.device_drivers.asr1k.asr1k_cfg_syncer [-] ROUTER_ID: 8fe4b6
2017-12-22 11:56:51.073 28115 INFO networking_cisco.plugins.cisco.cfg_agent.device_drivers.asr1k.asr1k_cfg_syncer [-] ROUTER_ID: f77761
2017-12-22 11:56:51.074 28115 INFO networking_cisco.plugins.cisco.cfg_agent.device_drivers.asr1k.asr1k_cfg_syncer [-] ROUTER_ID: 7f7455
2017-12-22 11:56:51.074 28115 INFO networking_cisco.plugins.cisco.cfg_agent.device_drivers.asr1k.asr1k_cfg_syncer [-]
2017-12-22 11:56:51.074 28115 INFO networking_cisco.plugins.cisco.cfg_agent.device_drivers.asr1k.asr1k_cfg_syncer [-] SEGMENT_ID: 793
2017-12-22 11:56:51.074 28115 INFO networking_cisco.plugins.cisco.cfg_agent.device_drivers.asr1k.asr1k_cfg_syncer [-] INTF: 172.18.0.1, f77761, network:router_interface
2017-12-22 11:56:51.074 28115 INFO networking_cisco.plugins.cisco.cfg_agent.device_drivers.asr1k.asr1k_cfg_syncer [-] SEGMENT_ID: 794
2017-12-22 11:56:51.075 28115 INFO networking_cisco.plugins.cisco.cfg_agent.device_drivers.asr1k.asr1k_cfg_syncer [-] INTF: 172.16.0.1, 93dce6, network:router_interface
2017-12-22 11:56:51.075 28115 INFO networking_cisco.plugins.cisco.cfg_agent.device_drivers.asr1k.asr1k_cfg_syncer [-] SEGMENT_ID: 70
2017-12-22 11:56:51.075 28115 INFO networking_cisco.plugins.cisco.cfg_agent.device_drivers.asr1k.asr1k_cfg_syncer [-] INTF: 192.168.70.5, 7f7455, network:router_interface
2017-12-22 11:56:51.075 28115 INFO ncclient.operations.rpc [-] Requesting 'GetConfig'
2017-12-22 11:56:51.345 28115 INFO networking_cisco.plugins.cisco.cfg_agent.device_drivers.asr1k.asr1k_cfg_syncer [-]
nat overload rule: <LineItem 'ip nat inside source list neutron_acl_794_8e64dd2c pool nrouter-93dce6_nat_pool vrf nrouter-93dce6 overload'>
2017-12-22 11:56:51.347 28115 INFO networking_cisco.plugins.cisco.cfg_agent.device_drivers.asr1k.asr1k_cfg_syncer [-]
NAT pool: <LineItem 'ip nat pool nrouter-93dce6_nat_pool 192.168.70.6 192.168.70.6 netmask 255.255.255.0'>
2017-12-22 11:56:51.349 28115 INFO networking_cisco.plugins.cisco.cfg_agent.device_drivers.asr1k.asr1k_cfg_syncer [-]
default route: <LineItem 'ip route vrf nrouter-93dce6 0.0.0.0 0.0.0.0 GigabitEthernet0/0/2.70 192.168.70.1'>
2017-12-22 11:56:51.349 28115 INFO networking_cisco.plugins.cisco.cfg_agent.device_drivers.asr1k.asr1k_cfg_syncer [-] router_id: 93dce6, segment_id: 70, next_hop: 192.168.70.1
2017-12-22 11:56:51.351 28115 INFO networking_cisco.plugins.cisco.cfg_agent.device_drivers.asr1k.asr1k_cfg_syncer [-]
acl: <LineItem 'ip access-list standard neutron_acl_794_8e64dd2c'>
2017-12-22 11:56:51.351 28115 INFO networking_cisco.plugins.cisco.cfg_agent.device_drivers.asr1k.asr1k_cfg_syncer [-] segment_id: 794, port_id: 8e64dd2c
2017-12-22 11:56:51.352 28115 INFO networking_cisco.plugins.cisco.cfg_agent.device_drivers.asr1k.asr1k_cfg_syncer [-] permit rule: <LineItem 'permit 172.16.0.0 0.0.0.255'>
2017-12-22 11:56:51.352 28115 INFO networking_cisco.plugins.cisco.cfg_agent.device_drivers.asr1k.asr1k_cfg_syncer [-] cfg_subnet: 172.16.0.0/24, db_subnet: 172.16.0.0/24
2017-12-22 11:56:51.355 28115 INFO networking_cisco.plugins.cisco.cfg_agent.device_drivers.asr1k.asr1k_cfg_syncer [-] ASR interface: <LineItem 'interface GigabitEthernet0/0/2.70'>
2017-12-22 11:56:51.355 28115 INFO networking_cisco.plugins.cisco.cfg_agent.device_drivers.asr1k.asr1k_cfg_syncer [-] segment_id: 70
2017-12-22 11:56:51.356 28115 INFO networking_cisco.plugins.cisco.cfg_agent.device_drivers.asr1k.asr1k_cfg_syncer [-] VRF: None
2017-12-22 11:56:51.356 28115 INFO networking_cisco.plugins.cisco.cfg_agent.device_drivers.asr1k.asr1k_cfg_syncer [-] NAT Type: outside
2017-12-22 11:56:51.357 28115 INFO networking_cisco.plugins.cisco.cfg_agent.device_drivers.asr1k.asr1k_cfg_syncer [-] target_ip: 192.168.70.9, actual_ip: 192.168.70.9
2017-12-22 11:56:51.357 28115 INFO networking_cisco.plugins.cisco.cfg_agent.device_drivers.asr1k.asr1k_cfg_syncer [-] ASR interface: <LineItem 'interface GigabitEthernet0/0/2.794'>
2017-12-22 11:56:51.357 28115 INFO networking_cisco.plugins.cisco.cfg_agent.device_drivers.asr1k.asr1k_cfg_syncer [-] segment_id: 794
2017-12-22 11:56:51.358 28115 INFO networking_cisco.plugins.cisco.cfg_agent.device_drivers.asr1k.asr1k_cfg_syncer [-] VRF: <LineItem 'vrf forwarding nrouter-93dce6'>
2017-12-22 11:56:51.358 28115 ERROR networking_cisco.plugins.cisco.cfg_agent.service_helpers.routing_svc_helper [-] Failed processing routers
2017-12-22 11:56:51.358 28115 ERROR networking_cisco.plugins.cisco.cfg_agent.service_helpers.routing_svc_helper Traceback (most recent call last):
2017-12-22 11:56:51.358 28115 ERROR networking_cisco.plugins.cisco.cfg_agent.service_helpers.routing_svc_helper File "/usr/lib/python2.7/site-packages/networking_cisco/plugins/cisco/cfg_agent/service_helpers/routing_svc_helper.py", line 267, in process_service
2017-12-22 11:56:51.358 28115 ERROR networking_cisco.plugins.cisco.cfg_agent.service_helpers.routing_svc_helper self._cleanup_invalid_cfg(routers)
2017-12-22 11:56:51.358 28115 ERROR networking_cisco.plugins.cisco.cfg_agent.service_helpers.routing_svc_helper File "/usr/lib/python2.7/site-packages/networking_cisco/plugins/cisco/cfg_agent/service_helpers/routing_svc_helper.py", line 372, in _cleanup_invalid_cfg
2017-12-22 11:56:51.358 28115 ERROR networking_cisco.plugins.cisco.cfg_agent.service_helpers.routing_svc_helper routers[0]['hosting_device'], routers)
2017-12-22 11:56:51.358 28115 ERROR networking_cisco.plugins.cisco.cfg_agent.service_helpers.routing_svc_helper File "/usr/lib/python2.7/site-packages/networking_cisco/plugins/cisco/cfg_agent/device_drivers/asr1k/asr1k_routing_driver.py", line 354, in cleanup_invalid_cfg
2017-12-22 11:56:51.358 28115 ERROR networking_cisco.plugins.cisco.cfg_agent.service_helpers.routing_svc_helper cfg_syncer.delete_invalid_cfg()
2017-12-22 11:56:51.358 28115 ERROR networking_cisco.plugins.cisco.cfg_agent.service_helpers.routing_svc_helper File "/usr/lib/python2.7/site-packages/networking_cisco/plugins/cisco/cfg_agent/device_drivers/asr1k/asr1k_cfg_syncer.py", line 320, in delete_invalid_cfg
2017-12-22 11:56:51.358 28115 ERROR networking_cisco.plugins.cisco.cfg_agent.service_helpers.routing_svc_helper parsed_cfg)
2017-12-22 11:56:51.358 28115 ERROR networking_cisco.plugins.cisco.cfg_agent.service_helpers.routing_svc_helper File "/usr/lib/python2.7/site-packages/networking_cisco/plugins/cisco/cfg_agent/device_drivers/asr1k/asr1k_cfg_syncer.py", line 1290, in clean_interfaces
2017-12-22 11:56:51.358 28115 ERROR networking_cisco.plugins.cisco.cfg_agent.service_helpers.routing_svc_helper correct_grp_num = int(db_intf[ha.HA_INFO]['group'])
2017-12-22 11:56:51.358 28115 ERROR networking_cisco.plugins.cisco.cfg_agent.service_helpers.routing_svc_helper KeyError: 'ha_info'
2017-12-22 11:56:51.358 28115 ERROR networking_cisco.plugins.cisco.cfg_agent.service_helpers.routing_svc_helper

The offending code snippet is here:

https://github.com/openstack/networking-cisco/blob/5.3.0/networking_cisco/plugins/cisco/cfg_agent/device_drivers/asr1k/asr1k_cfg_syncer.py#L1290

In line 1271, a check is already being done to test the interface for HA_INFO. Setting correct_grp_num within that if statement may be a viable workaround.

Please let me know if you need any additional information.

[James Denton (james-denton)]

Ignore my recommendation - didn't realize that test was being done within an else block for internal interfaces only. I just added a simple test to get it working for both external/internal:

if (ha.HA_INFO in db_intf):
    correct_grp_num = int(db_intf[ha.HA_INFO]['group'])

[Timothy Swanson (tiswanso)]

Thanks for reporting this and the analysis, James! Indeed our test environments and customers environments for ASR1K and CSR are HA pairs.

Great ASR1K articles in your blog, btw!

[Leon Zachery (lzachery)]

Lower priority due to recommended HA configuration.