network ports for l3 get created in random AZ even availability-zone-hint is specified

[Yoshi Kadokawa]

In a DVR environment, when you create a router with an internal network and external network attached, several ports will be created.
- dhcp
- router_gateway
- floatingip_agent_gateway
- router_centralized_snat
- router_interface_distributed

However, in an environment with multiple AZ, it looks like that the router_centralized_snat and router_gateway ports get created in random AZ, even if you have specified availability-zone-hint for router and network(both internal and external).

If all of the AZs are in the same overlay network and provider network, then there will be no problem, however, if those are independent in each AZ, instances are still reachable and can go out to the internet via floating IP, but if SNAT namespace is in different AZ(no network reachability) DNS forwarder will not be reachable and instances without floating IP, they will not be able to go outside.

According to these[0][1], in order to have these ports created in the specified availability zone, you will need to add the following in neutron.conf

router_scheduler_driver = neutron.scheduler.l3_agent_scheduler.AZLeastRoutersScheduler

I managed to reproduce the issue with the following bundle[2],
and by adding the above configuration via config-flags resolved the issue.
However, I think it would be better to have a new config in neutron-api charm,
like "enable-az-aware-scheduler" to add the above scheduler configuration.

[0] https://docs.openstack.org/neutron/queens/admin/config-az.html#router-scheduler
[1] https://github.com/openstack/neutron/blob/ef97c83e8753a668e0c0f405ef4beb027efa02cd/neutron/scheduler/l3_agent_scheduler.py#L360
[2] https://pastebin.canonical.com/p/HpPY2vHD6p/

[Trent Lloyd (lathiat)]

Ideally we would just enable this by default, whether or not we add a config option to allow disabling it.

I think users would expect this behaviour. So as long as the option works in all deployments with or without AZs, I'm thinking we should just enable it by default.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/760104