While looking through logs I came across this ERROR:
http://logs.openstack.org/30/370430/6/check/gate-tempest-dsvm-neutron-dvr-ubuntu-xenial/c853e94/logs/screen-q-l3.txt.gz#_2016-09-15_16_27_41_804
Pasting here since that log could go away (even if the paste is horribly wrapped, but tried to format it a little):
ERROR neutron.agent.l3.dvr_router_base [-]
DVR: SNAT port not found in the list
[{u'allowed_address_pairs': [], u'extra_dhcp_opts': [],
u'updated_at': u'2016-09-15T16:26:02',
u'device_owner': u'network:router_centralized_snat',
u'revision_number': 6,
u'port_security_enabled': False, u'binding:profile': {},
u'binding:vnic_type': u'normal',
u'fixed_ips':
[{u'subnet_id': u'af427fca-9194-440f-87d6-e74e4d1c8a27',
u'prefixlen': 28, u'ip_address': u'10.1.0.4'}],
u'id': u'54df3773-e6cf-4d9c-b1b9-0e43f77ba21a',
u'security_groups': [],
u'binding:vif_details': {u'port_filter': True,
u'ovs_hybrid_plug': True},
u'address_scopes': {u'4': None, u'6': None},
u'binding:vif_type': u'ovs',
u'mac_address': u'fa:16:3e:ed:f7:77', u'project_id': u'',
u'status': u'ACTIVE',
u'subnets': [{u'dns_nameservers': [],
u'ipv6_ra_mode': None,
u'gateway_ip': u'10.1.0.1',
u'cidr': u'10.1.0.0/28',
u'id': u'af427fca-9194-440f-87d6-e74e4d1c8a27',
u'subnetpool_id': None}],
u'binding:host_id': u'ubuntu-xenial-osic-cloud1-4314226',
u'description': u'',
u'device_id': u'6e1fd121-cf73-4e72-b6fa-431761591de6',
u'name': u'', u'admin_state_up': True,
u'network_id': u'86a63ce4-50b6-48ee-b192-c77fe2481db9',
u'tenant_id': u'', u'created_at': u'2016-09-15T16:25:51',
u'mtu': 1450, u'extra_subnets': []},
{u'allowed_address_pairs': [], u'extra_dhcp_opts': [],
u'updated_at': u'2016-09-15T16:26:06',
u'device_owner': u'network:router_centralized_snat',
u'revision_number': 7,
u'port_security_enabled': False, u'binding:profile': {},
u'binding:vnic_type': u'normal',
u'fixed_ips':
[{u'subnet_id': u'b433a3c3-eb7c-404b-aea3-5e454b4dd0bd',
u'prefixlen': 64, u'ip_address': u'2003:0:0:1::2'},
{u'subnet_id': u'3dd250b4-eade-4239-9011-9860ccf31364',
u'prefixlen': 64, u'ip_address': u'2003::4'}],
u'id': u'7454b08a-3431-4099-a3bc-7e46c56f310b',
u'security_groups': [],
u'binding:vif_details': {u'port_filter': True,
u'ovs_hybrid_plug': True},
u'address_scopes': {u'4': None, u'6': None},
u'binding:vif_type': u'ovs',
u'mac_address': u'fa:16:3e:ce:db:80', u'project_id': u'',
u'status': u'ACTIVE',
u'subnets': [{u'dns_nameservers': [],
u'ipv6_ra_mode': u'dhcpv6-stateless',
u'gateway_ip': u'2003:0:0:1::1',
u'cidr': u'2003:0:0:1::/64',
u'id': u'b433a3c3-eb7c-404b-aea3-5e454b4dd0bd',
u'subnetpool_id': None},
{u'dns_nameservers': [],
u'ipv6_ra_mode': u'dhcpv6-stateless',
u'gateway_ip': u'2003::1',
u'cidr': u'2003::/64',
u'id': u'3dd250b4-eade-4239-9011-9860ccf31364',
u'subnetpool_id': None}],
u'binding:host_id': u'ubuntu-xenial-osic-cloud1-4314226',
u'description': u'',
u'device_id': u'6e1fd121-cf73-4e72-b6fa-431761591de6',
u'name': u'', u'admin_state_up': True,
u'network_id': u'7862083c-a928-4170-b99e-3cfd0fb3ae77',
u'tenant_id': u'', u'created_at': u'2016-09-15T16:25:55',
u'mtu': 1450, u'extra_subnets': []}]
for the given router internal port
{u'allowed_address_pairs': [], u'extra_dhcp_opts': [],
u'updated_at': u'2016-09-15T16:27:37',
u'device_owner': u'network:router_interface_distributed',
u'revision_number': 8,
u'port_security_enabled': False, u'binding:profile': {},
u'binding:vnic_type': u'normal',
u'fixed_ips':
[{u'subnet_id': u'3dd250b4-eade-4239-9011-9860ccf31364',
u'prefixlen': 64, u'ip_address': u'2003::1'}],
u'id': u'6092e3b0-f833-4611-82e4-7fe5d1d31021',
u'security_groups': [],
u'binding:vif_details': {},
u'address_scopes': {u'4': None, u'6': None},
u'binding:vif_type': u'distributed',
u'mac_address': u'fa:16:3e:ea:5f:f0',
u'project_id': u'b557a0938e3748d6a34b0ca2efdee658',
u'status': u'ACTIVE',
u'subnets': [{u'dns_nameservers': [],
u'ipv6_ra_mode': u'dhcpv6-stateless',
u'gateway_ip': u'2003::1',
u'cidr': u'2003::/64',
u'id': u'3dd250b4-eade-4239-9011-9860ccf31364',
u'subnetpool_id': None}],
u'binding:host_id': u'', u'description': u'',
u'device_id': u'6e1fd121-cf73-4e72-b6fa-431761591de6',
u'name': u'', u'admin_state_up': True,
u'network_id': u'7862083c-a928-4170-b99e-3cfd0fb3ae77',
u'tenant_id': u'b557a0938e3748d6a34b0ca2efdee658',
u'created_at': u'2016-09-15T16:25:54', u'mtu': 1450,
u'extra_subnets': [{u'dns_nameservers': [],
u'ipv6_ra_mode': u'dhcpv6-stateless',
u'gateway_ip': u'2003:0:0:1::1',
u'cidr': u'2003:0:0:1::/64',
u'id': u'b433a3c3-eb7c-404b-aea3-5e454b4dd0bd',
u'subnetpool_id': None}]}
From looking at the trace, I do see the subnet in question in the list, but since the code is only checking it's in the first port it doesn't find it. I'm curious why it doesn't look in them all, so will propose a patch to verify it works and get some feedback.
Fix proposed to branch: master
Review: https:/