Thanks for all the questions and input, both here and on the l3 and drivers meetings.

I hope to answer all questions, but I will need multiple takes to get there.

Let me start with exploring alternative (3) first, that is: Can we use router interfaces instead of external gateway networks?

For a long time I believed that you can only add an external network into a router's external_gw_info. And I also believed that you can only add an internal network's subnet as a router interface.

Now it seems to me I was wrong in the latter. (So now I'm careful with not using a router's external gw and external net interchangeably.) I tested it with l3-agent and it is actually possible:

* to add an external network's subnet as a router interface
* to add multiple external networks' subnets as router interfaces
* to have connectivity from an internal network to all of these external networks

For me this is quite in contrast with our documentation. Or at least I don't understand what "internal" means below:

https://docs.openstack.org/api-ref/network/v2/index.html?expanded=add-interface-to-router-detail#routers-routers

"""
PUT /v2.0/routers/{router_id}/add_router_interface
Adds an internal interface to a logical router. This means a specified subnet is attached to a router as an internal router interface.
"""

So before I dive into alternative (3) deeper, I must ask: Do we consider this (adding an external net's subnet as a router interface) a feature or an accident of the neutron API?

If it is a feature then alternative (3) may be realistic for me given that:

* the number of router interfaces to external nets are not limited to 1
* my use case does not require Floating IPs or SNAT

However there are multiple further considerations:

* neutron-dynamic-routing does not advertise a single route in a configuration like this. I checked the implementation and it looks for ports with device_owner='router_gateway':

https://opendev.org/openstack/neutron-dynamic-routing/src/commit/4212b705386153c5ff9986068c78ee9c7b4fbe4c/neutron_dynamic_routing/db/bgp_db.py#L839

Of which we don't have any of course, that's why there are no routes advertised. If we take the path of alternative (3) can we consider this a bug of neutron-dynamic-routing? Changing neutron-dynamic-routing to advertise additional routes sounds significantly smaller work and has way less impact than extending core l3 APIs. That sounds good to me.

* Can we allow a router to have the same external network in its external_gw_info and added as a router interface? Can we ever allow/implement an external net as a router interface on a distributed router? My use case does not require any of these. However at the moment I don't have good answers here.

* Shall we change our api-ref to explicit support external networks as router interfaces?

That's all I have about alternative (3) now.

For the record this is what I tested (it all worked):

openstack address scope create scope0
openstack subnet pool create --address-scope scope0 --pool-prefix 10.0.0.0/8 --default-prefix-length 24 pool0

openstack router create router0

devstack # sudo ip link set up dev br-physnet0
devstack # sudo ip address add 10.0.0.1/24 dev br-physnet0

openstack network create --external --provider-network-type flat --provider-physical-network physnet0 net0
openstack subnet create --network net0 --subnet-pool pool0 --subnet-range 10.0.0.0/24 --gateway 10.0.0.1 --no-dhcp subnet0

openstack router add subnet router0 subnet0

devstack # sudo ip link set up dev br-physnet1
devstack # sudo ip address add 10.0.1.1/24 dev br-physnet1

openstack network create --external --provider-network-type flat --provider-physical-network physnet1 net1
openstack subnet create --network net1 --subnet-pool pool0 --subnet-range 10.0.1.0/24 --gateway 10.0.1.1 --no-dhcp subnet1

openstack router add subnet router0 subnet1

openstack network create net2
openstack subnet create --network net2 --subnet-pool pool0 --subnet-range 10.0.2.0/24 subnet2

openstack router add subnet router0 subnet2

openstack server create --flavor cirros256 --image cirros-0.5.1-x86_64-disk --nic net-id=net2 --wait vm0
sudo virsh console "$( openstack server show vm0 -f value -c OS-EXT-SRV-ATTR:instance_name )"

ping -c3 10.0.2.1
ping -c3 10.0.1.1
ping -c3 10.0.0.1